--- name: computer-vision description: | Computer vision implementation covering image classification, object detection (YOLO), image segmentation, OCR (Tesseract), face detection, image preprocessing, data augmentation, transfer learning, model deployment, and edge inference. Use when the user asks about computer vision, computer vision best practices, or needs guidance on computer vision implementation. Do NOT use when the user needs a different specialized skill or is asking about an unrelated technology domain. license: Apache-2.0 metadata: author: foundry-skills version: "1.0.0" tags: "ai-ml deep-learning guide" category: "ai-machine-learning" subcategory: "applied-ai" depends: "" disclaimer: "none" difficulty: "intermediate" --- # Computer Vision ## Overview Computer vision enables machines to interpret and act on visual data. This skill covers practical implementation of core CV tasks -- classification, detection, segmentation, and OCR -- using modern frameworks and pre-trained models, with guidance on data augmentation, transfer learning, and deployment. ## Image Preprocessing ### Standard Pipeline ```python import cv2 import numpy as np from PIL import Image import torchvision.transforms as T class ImagePreprocessor: """Standard image preprocessing for CV models.""" def __init__(self, target_size: tuple = (224, 224), normalize: bool = True): self.target_size = target_size self.normalize = normalize def preprocess_opencv(self, image_path: str) -> np.ndarray: """Preprocess with OpenCV.""" img = cv2.imread(image_path) img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # Resize # ... (condensed) ... transforms.extend([ T.ToTensor(), T.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), ]) return T.Compose(transforms) ``` ### Common Preprocessing Operations | Operation | When to Use | Library | |-----------|------------|---------| | Resize | Always (model input size) | cv2, PIL, torchvision | | Normalize | Always (ImageNet stats for pretrained) | torchvision | | Grayscale | OCR, edge detection | cv2 | | Histogram equalization | Low contrast images | cv2 (CLAHE) | | Denoising | Noisy images | cv2.fastNlMeansDenoising | | Crop | Focus on region of interest | cv2, PIL | ## Data Augmentation ### Albumentations (Recommended) ```python import albumentations as A from albumentations.pytorch import ToTensorV2 def get_training_augmentations(image_size: int = 224) -> A.Compose: """Production training augmentation pipeline.""" return A.Compose([ A.RandomResizedCrop(height=image_size, width=image_size, scale=(0.8, 1.0)), A.HorizontalFlip(p=0.5), A.VerticalFlip(p=0.1), A.ShiftScaleRotate( shift_limit=0.1, scale_limit=0.15, rotate_limit=15, p=0.5 ), A.OneOf([ A.GaussNoise(var_limit=(10.0, 50.0)), A.GaussianBlur(blur_limit=(3, 7)), A.MotionBlur(blur_limit=5), ], p=0.3), A.OneOf([ # ... (condensed) ... A.RandomResizedCrop(height=image_size, width=image_size, scale=(0.5, 1.0)), A.HorizontalFlip(p=0.5), A.RandomBrightnessContrast(p=0.3), A.HueSaturationValue(p=0.3), A.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225]), ToTensorV2(), ], bbox_params=A.BboxParams(format='pascal_voc', label_fields=['labels'])) ``` ### Augmentation Strategy by Dataset Size ``` Dataset size < 500 images: -> Heavy augmentation + transfer learning essential -> Use MixUp, CutMix, Mosaic -> Consider synthetic data generation Dataset size 500-5000: -> Moderate augmentation -> Standard flips, crops, color jitter -> Transfer learning highly recommended Dataset size > 5000: -> Light augmentation -> Basic flips and normalization -> Can train from scratch for simple tasks ``` ## Image Classification ### Transfer Learning with PyTorch ```python import torch import torch.nn as nn from torchvision import models def create_classifier( num_classes: int, backbone: str = "resnet50", pretrained: bool = True, freeze_backbone: bool = True, ) -> nn.Module: """Create a classifier with transfer learning.""" if backbone == "resnet50": model = models.resnet50(weights="IMAGENET1K_V2" if pretrained else None) num_features = model.fc.in_features model.fc = nn.Sequential( nn.Dropout(0.3), nn.Linear(num_features, num_classes), # ... (condensed) ... # Freeze backbone layers if freeze_backbone: for param in list(model.parameters())[:-2]: param.requires_grad = False return model ``` ### Hugging Face Image Classification ```python from transformers import pipeline, AutoModelForImageClassification, AutoImageProcessor # Quick inference with pipeline classifier = pipeline("image-classification", model="google/vit-base-patch16-224") result = classifier("photo.jpg") # [{"label": "golden retriever", "score": 0.95}, ...] # Fine-tuning from transformers import TrainingArguments, Trainer from datasets import load_dataset dataset = load_dataset("imagefolder", data_dir="./data") model = AutoModelForImageClassification.from_pretrained( "google/vit-base-patch16-224", num_labels=len(dataset["train"].features["label"].names), ignore_mismatched_sizes=True, ) # ... (condensed) ... load_best_model_at_end=True, ), train_dataset=dataset["train"], eval_dataset=dataset["test"], ) trainer.train() ``` ### Model Selection Guide | Model | Params | Accuracy (ImageNet) | Speed | Best For | |-------|--------|-------------------|-------|----------| | MobileNetV3-Small | 2.5M | 67.4% | Very fast | Mobile/edge | | EfficientNet-B0 | 5.3M | 77.1% | Fast | Balanced | | ResNet-50 | 25M | 80.4% | Medium | General purpose | | EfficientNet-V2-S | 21M | 84.2% | Medium | High accuracy | | ViT-B/16 | 86M | 84.5% | Slower | Max accuracy | | ConvNeXt-Base | 88M | 85.8% | Slower | SOTA accuracy | ## Object Detection ### YOLOv8 (Ultralytics) ```python from ultralytics import YOLO # Load pretrained model model = YOLO("yolov8n.pt") # nano (fastest) # model = YOLO("yolov8s.pt") # small # model = YOLO("yolov8m.pt") # medium # model = YOLO("yolov8l.pt") # large # Inference results = model("image.jpg") for result in results: boxes = result.boxes for box in boxes: cls = int(box.cls[0]) conf = float(box.conf[0]) xyxy = box.xyxy[0].tolist() # [x1, y1, x2, y2] label = model.names[cls] # ... (condensed) ... patience=20, device="0", # GPU ) # Export for deployment model.export(format="onnx") model.export(format="tflite") ``` ### YOLO Dataset Format ```yaml # dataset.yaml path: /path/to/dataset train: images/train val: images/val test: images/test names: 0: person 1: car 2: bicycle ``` ``` # Label format (one .txt per image) # class_id center_x center_y width height (all normalized 0-1) 0 0.5 0.6 0.3 0.4 1 0.2 0.3 0.15 0.2 ``` ### Detection Model Comparison | Model | mAP@50 (COCO) | Speed (ms) | Params | Best For | |-------|---------------|------------|--------|----------| | YOLOv8n | 37.3 | 1.2 | 3.2M | Real-time, edge | | YOLOv8s | 44.9 | 2.0 | 11.2M | Balanced | | YOLOv8m | 50.2 | 4.2 | 25.9M | High accuracy | | YOLOv8x | 53.9 | 8.7 | 68.2M | Max accuracy | | RT-DETR-L | 53.0 | 9.3 | 32M | Transformer-based | ## Image Segmentation ### Semantic Segmentation ```python from transformers import pipeline segmenter = pipeline("image-segmentation", model="nvidia/segformer-b0-finetuned-ade-512-512") results = segmenter("street_scene.jpg") for segment in results: print(f"{segment['label']}: score={segment['score']:.2f}") ``` ### Instance Segmentation with SAM (Segment Anything) ```python from segment_anything import sam_model_registry, SamPredictor, SamAutomaticMaskGenerator # Load SAM model sam = sam_model_registry["vit_h"](checkpoint="sam_vit_h.pth") sam.to("cuda") # Automatic mask generation mask_generator = SamAutomaticMaskGenerator(sam) masks = mask_generator.generate(image) # Point-prompted segmentation predictor = SamPredictor(sam) predictor.set_image(image) # Click a point to segment the object there masks, scores, logits = predictor.predict( point_coords=np.array([[500, 375]]), point_labels=np.array([1]), # 1 = foreground multimask_output=True, ) # Use the highest-scoring mask best_mask = masks[np.argmax(scores)] ``` ### YOLO Segmentation ```python from ultralytics import YOLO model = YOLO("yolov8n-seg.pt") # Instance segmentation results = model("image.jpg") for result in results: if result.masks: for mask, box in zip(result.masks.data, result.boxes): cls = int(box.cls[0]) label = model.names[cls] binary_mask = mask.cpu().numpy() # H x W binary mask ``` ## OCR (Optical Character Recognition) ### Tesseract OCR ```python import pytesseract from PIL import Image import cv2 def extract_text(image_path: str, preprocess: bool = True) -> str: """Extract text from image using Tesseract.""" img = cv2.imread(image_path) if preprocess: # Convert to grayscale gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY) # Adaptive thresholding binary = cv2.adaptiveThreshold( gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2 ) # ... (condensed) ... 'x': data['left'][i], 'y': data['top'][i], 'width': data['width'][i], 'height': data['height'][i], }, }) return results ``` ### EasyOCR (Multi-language) ```python import easyocr reader = easyocr.Reader(['en', 'fr', 'de']) results = reader.readtext('document.jpg') for (bbox, text, confidence) in results: print(f"'{text}' (confidence: {confidence:.2f})") ``` ## Face Detection ### MediaPipe (Fastest) ```python import mediapipe as mp mp_face = mp.solutions.face_detection mp_drawing = mp.solutions.drawing_utils def detect_faces(image_path: str) -> list[dict]: """Detect faces using MediaPipe.""" with mp_face.FaceDetection(model_selection=1, min_detection_confidence=0.5) as detector: image = cv2.imread(image_path) rgb = cv2.cvtColor(image, cv2.COLOR_BGR2RGB) results = detector.process(rgb) faces = [] if results.detections: h, w = image.shape[:2] for detection in results.detections: bbox = detection.location_data.relative_bounding_box faces.append({ "confidence": detection.score[0], "bbox": { "x": int(bbox.xmin * w), "y": int(bbox.ymin * h), "width": int(bbox.width * w), "height": int(bbox.height * h), }, }) return faces ``` ## Model Deployment ### ONNX Export ```python import torch def export_to_onnx(model, sample_input, output_path: str = "model.onnx"): """Export PyTorch model to ONNX format.""" model.cpu() torch.onnx.export( model, sample_input, output_path, export_params=True, opset_version=17, input_names=['input'], output_names=['output'], dynamic_axes={ 'input': {0: 'batch_size'}, 'output': {0: 'batch_size'}, }, ) # ... (condensed) ... session = ort.InferenceSession( model_path, providers=['CUDAExecutionProvider', 'CPUExecutionProvider'] ) input_name = session.get_inputs()[0].name output = session.run(None, {input_name: input_data}) return output[0] ``` ### Edge Inference with TFLite ```python import tensorflow as tf def convert_to_tflite(saved_model_path: str, output_path: str, quantize: bool = True): """Convert model to TFLite for edge deployment.""" converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_path) if quantize: converter.optimizations = [tf.lite.Optimize.DEFAULT] converter.target_spec.supported_types = [tf.float16] tflite_model = converter.convert() with open(output_path, 'wb') as f: f.write(tflite_model) ``` ### Deployment Decision Tree ``` Target platform? CLOUD (GPU available): -> PyTorch/TensorRT (best performance) -> ONNX Runtime (portable) EDGE (mobile, IoT): -> TFLite (Android, Raspberry Pi) -> CoreML (iOS, macOS) -> ONNX Runtime Mobile BROWSER: -> TensorFlow.js -> ONNX Runtime Web Latency requirement? < 10ms: TensorRT or hardware-specific optimization 10-100ms: ONNX Runtime with GPU > 100ms: Any framework works ``` ## Checklist - [ ] Set up image preprocessing pipeline with proper normalization - [ ] Implement data augmentation appropriate to dataset size - [ ] Choose model architecture based on accuracy vs speed tradeoffs - [ ] Use transfer learning for datasets under 10K images - [ ] Set up proper train/val/test splits - [ ] Track training metrics with experiment tracker - [ ] Export model to deployment format (ONNX, TFLite, TensorRT) - [ ] Benchmark inference latency on target hardware - [ ] Implement preprocessing in the deployment pipeline (not just training) - [ ] Test with edge cases (poor lighting, occlusion, unusual angles) - [ ] Monitor model performance in production ## When to Use **Use this skill when:** - Designing or implementing computer vision solutions - Reviewing or improving existing computer vision approaches - Making architectural or implementation decisions about computer vision - Learning computer vision patterns and best practices - Troubleshooting computer vision-related issues **Do NOT use this skill when:** - The question is about a fundamentally different technology domain - A more specific sibling skill covers the exact topic needed - The user needs a complete hands-on tutorial rather than expert guidance ## Output Format ```markdown # Computer Vision Analysis ## Context Assessment [Situation summary and constraints] ## Recommended Approach [Primary recommendation with rationale] ## Implementation Steps 1. [Step with specific details] 2. [Step with specific details] 3. [Step with specific details] ## Trade-offs and Considerations - [Key trade-off 1] - [Key trade-off 2] ## Next Steps - [Immediate action item] - [Follow-up action item] ``` ## Example **Input:** "Help me implement computer vision for a medium-scale production application" **Output:** A structured analysis covering current state assessment, recommended computer vision approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints. ## Edge Cases - **Legacy system integration:** When computer vision must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite - **Scale mismatch:** When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit - **Team skill gaps:** When the team lacks experience with the recommended approach, include learning resources and simpler alternatives - **Conflicting requirements:** When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities