SkillAgentSearch skills...

vision-framework

Implement computer vision features including text recognition (OCR), face detection, barcode scanning, image segmentation, object tracking, and document scanning in iOS apps.

Install / Use

npx skills add dpearson2699/swift-ios-skills --skill vision-framework

Installs into whichever agent you are using.

About this skill
📄

SKILL.md

Installable skill definition

Quality Score

89/100

Supported Platforms

Universal

Our assessment of vision-framework

vision-framework scores 89/100 on our quality scale, 421st of 947 AI & Machine Learning skills we index (top 45%).

Its SKILL.md is 18 KB long, well organised into 27 sections with 13 code examples: a thorough specification that gives an agent plenty to work with.

With 1,143 GitHub stars, it is one of the more widely adopted skills in the catalogue.

Substance
30/30
Structure
20/20
Description
15/15
Adoption
13/20
Freshness
11/15

Maintenance, license and trust

  • The repository was last updated about 4 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.
  • Our last check on 2026-10-31 found the source still online.
  • No license is declared. By default that means all rights are reserved: you can read it, but reusing or redistributing it is not clearly permitted. Ask the author before building on it commercially.
  • Its trust signals score 86/100, with 1 caution from licensing, adoption, age or documentation. These come from repository metadata, not a code audit — read the skill file before letting an agent act on it.

Safety scan

No issues found

Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful.

AI review by kimi-k2.7-code on 2026-10-04. Automated pattern scan on 2026-10-04. It catches known dangerous patterns, not every risk — read a skill before letting an agent act on it.

vision-framework compared with similar skills

All 4 of these similar skills score higher than vision-framework; compare them before choosing.

SkillScoreStarsUpdatedFormat
vision-framework (this skill)by dpearson2699891.1k4mo agoSKILL.md
claude-memby thedotmack100117.6ktodayCLAUDE.md
taste-skillby Leonxlnx100116.8k8d agoCLAUDE.md
Agent-Reachby Panniantong100112.3k8d agoCLAUDE.md
Understand-Anythingby Egonex-AI10092.8k1d agoCLAUDE.md

Frequently asked questions

How do I install vision-framework?
Run npx skills add dpearson2699/swift-ios-skills --skill vision-framework. The install tabs above show the steps for each supported agent.
Which AI agents does vision-framework work with?
It is written for Universal, as a SKILL.md file. Other agents that read the same format can often use it too.
Is vision-framework safe to use?
Our scan of the whole file found no instruction hijacking, hidden characters, credential access, data exfiltration or destructive commands. An AI review of the same text found nothing harmful. It declares no license and scores 86/100 on trust signals. Skills are instructions an agent will follow, so read the file before installing it and do not approve commands you do not understand.
Is vision-framework still maintained?
The repository was last updated about 4 months ago. That is recent enough to be usable, but agent tooling moves fast, so check the instructions against your agent's current version.

name: vision-framework description: "Implement computer vision features including text recognition (OCR), face detection, barcode scanning, image segmentation, object tracking, and document scanning in iOS apps. Covers both the modern Swift-native Vision API (iOS 18+) and legacy VNRequest patterns, VisionKit DataScannerViewController for live camera scanning, and CoreMLRequest/VNCoreMLRequest for custom model inference. Use when adding OCR, barcode scanning, face detection, or custom Core ML model inference with Vision."

Vision Framework

Detect text, faces, barcodes, objects, and body poses in images and video using on-device computer vision. Prefer the modern iOS 18+ request APIs and load the legacy reference only when the deployment target requires it.

See references/vision-requests.md for complete code patterns and references/visionkit-scanner.md for DataScannerViewController integration.

Contents

Two API Generations

Vision has two distinct API layers. Prefer the modern API for new code: Swift-native request types plus try await request.perform(on:). Keep VN*, VNImageRequestHandler, VNSequenceRequestHandler, completion handlers, and legacy CGRect helpers inside explicit legacy fallback sections or files.

| Aspect | Modern (iOS 18+) | Legacy | |---|---|---| | Pattern | let result = try await request.perform(on: image) | VNImageRequestHandler + completion handler | | Request types | Swift types — structs and classes (RecognizeTextRequest, DetectFaceRectanglesRequest) | ObjC classes (VNRecognizeTextRequest, VNDetectFaceRectanglesRequest) | | Concurrency | Native async/await | Completion handlers or synchronous perform | | Observations | Typed return values | Cast results from [Any] | | Availability | iOS 18+ / macOS 15+ | iOS 11+ |

The modern API uses the ImageProcessingRequest protocol. Each request type has a perform(on:orientation:) method that accepts CGImage, CIImage, CVPixelBuffer, CMSampleBuffer, Data, or URL. Most requests are structs; stateful requests such as GeneratePersonSegmentationRequest, TrackObjectRequest, TrackRectangleRequest, and DetectTrajectoriesRequest are final classes.

Request Pattern (Modern API)

All modern Vision requests follow the same pattern: create a request, call perform(on:), and handle the typed result.

import Vision

func recognizeText(in image: CGImage) async throws -> [String] {
    var request = RecognizeTextRequest()
    request.recognitionLevel = .accurate
    request.recognitionLanguages = [Locale.Language(identifier: "en-US")]

    let observations = try await request.perform(on: image)
    return observations.compactMap { observation in
        observation.topCandidates(1).first?.string
    }
}

Legacy Pattern (Pre-iOS 18)

For pre-iOS 18 targets, use the corresponding VNRequest with VNImageRequestHandler or VNSequenceRequestHandler. Load references/vision-requests.md for complete legacy request and handler patterns.

Text Recognition (OCR)

Modern: RecognizeTextRequest (iOS 18+)

var request = RecognizeTextRequest()
request.recognitionLevel = .accurate       // .fast for real-time
request.recognitionLanguages = [
    Locale.Language(identifier: "en-US"),
    Locale.Language(identifier: "fr-FR"),
]
request.usesLanguageCorrection = true
request.customWords = ["SwiftUI", "Xcode"] // domain-specific terms

let observations = try await request.perform(on: cgImage)
for observation in observations {
    guard let candidate = observation.topCandidates(1).first else { continue }
    let text = candidate.string
    let confidence = candidate.confidence  // 0.0 ... 1.0
    let bounds = observation.boundingBox   // NormalizedRect
}

Legacy: VNRecognizeTextRequest

The legacy request uses string language identifiers and the handler pattern in the reference; both generations support accurate and fast recognition levels.

Face Detection

Detect face rectangles, landmarks (eyes, nose, mouth), and capture quality.

// Modern API
let faceRequest = DetectFaceRectanglesRequest()
let faces = try await faceRequest.perform(on: cgImage)

for face in faces {
    let boundingBox = face.boundingBox   // NormalizedRect
    let roll = face.roll                 // Measurement<UnitAngle>
    let yaw = face.yaw                  // Measurement<UnitAngle>
}

// Landmarks (eyes, nose, mouth contours)
var landmarkRequest = DetectFaceLandmarksRequest()
let landmarkFaces = try await landmarkRequest.perform(on: cgImage)
for face in landmarkFaces {
    let landmarks = face.landmarks
    let leftEye = landmarks?.leftEye.points
    let nose = landmarks?.nose.points
}

Coordinate System

Vision uses a normalized coordinate system with origin at the bottom-left. Convert to UIKit (top-left origin) before display:

import Vision

func imageRectForDisplay(_ rect: NormalizedRect, imageSize: CGSize) -> CGRect {
    rect.toImageCoordinates(imageSize, origin: .upperLeft)
}

Barcode Detection

Detect 1D and 2D barcodes including QR codes.

var request = DetectBarcodesRequest()
let symbologies: [BarcodeSymbology] = [.qr, .ean13, .code128, .pdf417]
request.symbologies = symbologies

let barcodes = try await request.perform(on: cgImage)
for barcode in barcodes {
    let payload = barcode.payloadString          // decoded content
    let symbology = barcode.symbology            // .qr, .ean13, etc.
    let bounds = barcode.boundingBox             // NormalizedRect
}

Type annotate local values first, then assign request properties separately.

Document Scanning (iOS 26+)

RecognizeDocumentsRequest provides structured document reading with layout understanding beyond basic OCR. Returns DocumentObservation objects with a nested Container structure for paragraphs, tables, lists, and barcodes. Currently, Vision returns one document observation for each image.

var request = RecognizeDocumentsRequest()
let documents = try await request.perform(on: cgImage)

for observation in documents {
    let container = observation.document

    // Full text content
    let fullText = container.text

    // Structured access to paragraphs
    for paragraph in container.paragraphs {
        let paragraphText = paragraph.text
    }

    // Tables and lists
    for table in container.tables { /* structured table data */ }
    for list in container.lists { /* structured list data */ }

    // Embedded barcodes detected within the document
    for barcode in container.barcodes { /* barcode data */ }

    // Document title if detected
    if let title = container.title { print(title) }
}

For simpler document camera scanning, use VisionKit's VNDocumentCameraViewController which provides a full-screen camera UI with auto-capture, perspective correction, and multi-page scanning.

Image Segmentation

Modern: GeneratePersonSegmentationRequest (iOS 18+)

var request = GeneratePersonSegmentationRequest()
request.qualityLevel = .accurate  // .balanced, .fast

let mask = try await request.perform(on: cgImage)
// mask is a PixelBufferObservation with a pixelBuffer property
let maskBuffer = mask.pixelBuffer
// Apply mask using Core Image: CIFilter.blendWithMask()

Legacy: VNGeneratePersonSegmentationRequest

For older targets, VNGeneratePersonSegmentationRequest exposes its mask through the first pixel-buffer observation; use the reference's handler and mask-composition recipe.

Quality levels:

  • .accurate -- best quality, slowest (~1s), full resolution
  • .balanced -- good quality, moderate speed (~100ms), 960x540
  • .fast -- lowest quality, fastest (~10ms), 256x144, suitable for real-time

Instance Segmentation (iOS 18+)

Separate masks per person for individual effects.

// Modern API (iOS 18+)
let request = GeneratePersonInstanceMaskRequest()
let observation = try await request.perform(on: cgImage)
let indices = observation.allInstances

for index in indices {
    let mask = try observation.generateMask(for: IndexSet(integer: index))
    // mask is a CVPixelBuffer with only this person visible
}
// Legacy API (iOS 17+)
let request = VNGeneratePersonInstanceMaskRequest()
let handler = VNImageRequestHandler(cgImage: cgImage)
try handler.perform([request])

guard let result = request.results?.first else { return }
let indices = result.allInstances
for index in indices {
    let instanceMask = try result.generateMaskedImage(
        ofInstances: IndexSet(integer: index),
        from: handler,
        croppedToInstancesExtent: false
    )
}

See references/vision-requests.md for mask composition and Core Image filter integration patterns.

Object Tracking

Modern: TrackObjectRequest (iOS 18+)

TrackObjectRequest is a stateful request that maintains tracking context across frames.

// Initialize with a detected object's bounding box
let initialObservation = DetectedObjectObservation(boundingBox: detectedBox)
let request = TrackObjectRequest(detectedObject: initialObservation)

for pixelBuffer in framePixelBuffers {
    let results = try await request.perform(on: pixelBuffer)
    if let tracked = results.first {
        let updatedBounds = tracked.boundingBox  // NormalizedRect
    }
}

Modern TrackObjectRequest has no trackingLevel or qualityLevel.

Legacy: VNTrackObjectRequest

For older targets, use VNTrackObjectRequest with one retained VNSequenceRequestHandler and feed each result back as the next input observation. The reference contains the complete loop.

Other Request Types

Vision provides additional requests covered in references/vision-requests.md:

| Request | Purpose | |---|---| | ClassifyImageRequest | Classify scene content (outdoor, food, animal, etc.) | | GenerateAttentionBasedSaliencyImageRequest | Single SaliencyImageObservation for where viewers focus attention | | GenerateObjectnessBasedSaliencyImageRequest | Single SaliencyImageObservation for object-like regions | | GenerateForegroundInstanceMaskRequest | Foreground object segmentation (not person-specific) | | DetectRectanglesRequest | Detect rectangular shapes (documents, cards, screens) | | DetectHorizonRequest | Detect horizon angle for auto-leveling photos | | DetectHumanBodyPoseRequest | Detect body joints (shoulders, elbows, knees) | | DetectHumanBodyPose3DRequest | 3D human body pose estimation | | DetectHumanHandPoseRequest | Detect hand joints and finger positions | | DetectAnimalBodyPoseRequest | Detect animal body joint positions | | DetectFaceCaptureQualityRequest | Face capture quality scoring (0–1) for photo selection | | TrackRectangleRequest | Track rectangular objects across video frames | | TrackOpticalFlowRequest | Optical flow between video frames | | DetectTrajectoriesRequest | Detect object trajectories in video |

All modern request types above are iOS 18+ / macOS 15+.

Core ML Integration

Run custom Core ML models through Vision for automatic image preprocessing.

Vision runs already-prepared models with CoreMLRequest or VNCoreMLRequest; hand conversion, profiling, packaging,

Truncated for display — read the full file on GitHub.

Related Skills

View on GitHub
GitHub Stars1.1k
CategoryAI
Updated3mo ago
Forks58

Languages

Python

Trust signals

86/100

From repository metadata: license, adoption, age and documentation. Not a code audit — see the Safety scan above for what the skill file itself contains.

1 medium1 info