{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-237", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/237", "wordCount": 2243, "segments": [{"start": 6.0, "text": "Hi, I'm Megan Williams, from the Vision framework team."}, {"start": 10.0, "text": "This year there are some powerful advancements in image understanding,"}, {"start": 14.0, "text": "which you can now use to create incredible experiences in your apps."}, {"start": 18.0, "text": "I'll talk about a few of them, starting with what's new -"}, {"start": 23.0, "text": "Oh, that's weird, my agenda is missing."}, {"start": 28.0, "text": "Let me see if I can create one quickly."}, {"start": 31.0, "text": "I made notes of the topics I want to cover."}, {"start": 34.0, "text": "I bet AI can help me make an agenda."}, {"start": 36.0, "text": "I'll use this picture of my notes."}, {"start": 38.0, "text": "And ask a large language model to generate an agenda."}, {"start": 42.0, "text": "This is pretty easy to do with the Foundation Models framework."}, {"start": 45.0, "text": "Thankfully, this year Foundation Models is supporting image inputs."}, {"start": 50.0, "text": "Great! The model made me an agenda."}, {"start": 52.0, "text": "Now I can get back to the presentation."}, {"start": 55.0, "text": "This year, there's more ways than ever to bring image understanding to your apps."}, {"start": 60.0, "text": "Starting with what's new in Vision,"}, {"start": 62.0, "text": "the tap-to-segment API allows you to isolate"}, {"start": 65.0, "text": "any object in an image just by tapping on it."}, {"start": 69.0, "text": "There's also a new and powerful way"}, {"start": 71.0, "text": "to analyze images, using large language models."}, {"start": 75.0, "text": "I'll talk about how to do this with the Foundation Models framework."}, {"start": 79.0, "text": "Then I'll show how you can create image-based tools for LLMs"}, {"start": 83.0, "text": "that unlock even more possibilities in image understanding."}, {"start": 88.0, "text": "Finally, Vision is also now available on watchOS."}, {"start": 92.0, "text": "I'll show how to use Vision to enhance your watch apps."}, {"start": 96.0, "text": "But first, I'll show you some of the awesome things you can do"}, {"start": 99.0, "text": "with the tap-to-segment API."}, {"start": 102.0, "text": "Vision already has several image segmentation capabilities."}, {"start": 106.0, "text": "For example, there's person segmentation,"}, {"start": 109.0, "text": "which will isolate all of the people in an image."}, {"start": 112.0, "text": "But, what if I want to segment something else in the image?"}, {"start": 116.0, "text": "Like this flower vase?"}, {"start": 118.0, "text": "Now with Vision's tap-to-segment API,"}, {"start": 121.0, "text": "I can choose any object in the image to segment."}, {"start": 124.0, "text": "Like this board game, a piece of clothing, or even the floor."}, {"start": 130.0, "text": "There are multiple ways that I can select an object to segment."}, {"start": 134.0, "text": "I'll demonstrate a few of them."}, {"start": 139.0, "text": "In my app, I have a photo from a cafe"}, {"start": 142.0, "text": "and I want to segment the coffee cup on the table."}, {"start": 145.0, "text": "First, I can choose a point on the cup."}, {"start": 148.0, "text": "And now the cup is isolated."}, {"start": 150.0, "text": "This works well for simple objects,"}, {"start": 152.0, "text": "but if my subject is complex,"}, {"start": 154.0, "text": "then selecting just one point may not be enough."}, {"start": 157.0, "text": "For example, maybe I want to include the plate as well."}, {"start": 161.0, "text": "I can instead draw a bounding box"}, {"start": 163.0, "text": "around all of the objects I want to segment."}, {"start": 167.0, "text": "And now I'm able to get both the cup and the plate."}, {"start": 171.0, "text": "I can also draw a lasso around an object."}, {"start": 174.0, "text": "I'll use a lasso to segment this croissant."}, {"start": 180.0, "text": "Another great option is to draw a scribble."}, {"start": 183.0, "text": "I can scribble over multiple objects to easily segment all of them at once."}, {"start": 190.0, "text": "Once I have a mask,"}, {"start": 191.0, "text": "I can refine the mask by adding or subtracting more points."}, {"start": 196.0, "text": "I've already segmented this cup with an initial point,"}, {"start": 199.0, "text": "and now I want to include the plate."}, {"start": 202.0, "text": "I'll just tap the plate, and now it's included too."}, {"start": 206.0, "text": "I can also remove sections from my mask."}, {"start": 210.0, "text": "Maybe I only want the coffee, not the cup."}, {"start": 213.0, "text": "I can choose a point on the cup to exclude from my mask,"}, {"start": 218.0, "text": "and now I get just the coffee."}, {"start": 221.0, "text": "To use the API, you'll start with an image."}, {"start": 224.0, "text": "You can use an ImageRequestHandler to hold your image."}, {"start": 229.0, "text": "In Vision, images are processed using requests."}, {"start": 233.0, "text": "To segment an object, you'll use GenerateIterativeSegmentationRequest."}, {"start": 239.0, "text": "Use the ImageRequestHandler to perform the request."}, {"start": 243.0, "text": "This generates a mask of the segmented object."}, {"start": 248.0, "text": "The mask is a PixelBuffer that shows"}, {"start": 250.0, "text": "which pixels belong to the segmented object."}, {"start": 254.0, "text": "Here's the code."}, {"start": 255.0, "text": "I'll start with an image and create an ImageRequestHandler."}, {"start": 260.0, "text": "Now I'll create the request, using a point inside the object I want"}, {"start": 264.0, "text": "to segment as a starting seed."}, {"start": 266.0, "text": "Then I'll use the ImageRequestHandler to perform the request on the image."}, {"start": 272.0, "text": "This produces a segmentation mask of the object."}, {"start": 276.0, "text": "I can now refine this mask with a new point if I want."}, {"start": 281.0, "text": "To do this, I'll just include the point in the request."}, {"start": 284.0, "text": "And then I'll perform the request again."}, {"start": 287.0, "text": "There are a few other things to keep in mind."}, {"start": 291.0, "text": "Vision uses a normalized coordinates system with the coordinate origin"}, {"start": 295.0, "text": "in the lower left hand corner."}, {"start": 298.0, "text": "Points should be normalized to the image width and height,"}, {"start": 301.0, "text": "with coordinate values between 0 and 1."}, {"start": 305.0, "text": "It's also important when you're drawing a lasso,"}, {"start": 308.0, "text": "that your stroke width is wide enough."}, {"start": 311.0, "text": "Thin strokes may not produce the best result."}, {"start": 315.0, "text": "The line width should be at least 1% of the total image width."}, {"start": 320.0, "text": "Lastly, I want to mention that before you perform a segmentation request"}, {"start": 324.0, "text": "for the first time on a device, you'll have to download the model."}, {"start": 329.0, "text": "You can use the downloadAssets API to begin a download."}, {"start": 334.0, "text": "And if you're not sure whether the model is downloaded or not,"}, {"start": 337.0, "text": "you can check assetStatus to see if the model is ready to use."}, {"start": 343.0, "text": "With tap-to-segment,"}, {"start": 344.0, "text": "you can now interactively segment any part of an image you'd like."}, {"start": 350.0, "text": "Now, I want to talk about a new and exciting way to analyze images"}, {"start": 354.0, "text": "using the Foundation Models framework."}, {"start": 357.0, "text": "I showed an example earlier of using a large language model"}, {"start": 361.0, "text": "to put together an agenda, using a photo of my sticky notes."}, {"start": 365.0, "text": "But large language models can do a lot more."}, {"start": 369.0, "text": "I can also ask a model"}, {"start": 371.0, "text": "to help generate captions for the images in my app."}, {"start": 375.0, "text": "Models tend to do well with descriptive tasks."}, {"start": 379.0, "text": "It can even help me with my interior decorating,"}, {"start": 382.0, "text": "and provide some helpful suggestions for my living room."}, {"start": 387.0, "text": "And my personal favorite,"}, {"start": 388.0, "text": "I can use a large language model to create a recipe from a picture of my fridge."}, {"start": 394.0, "text": "The possibilities are endless."}, {"start": 397.0, "text": "And the API to do this is really simple."}, {"start": 400.0, "text": "Here's the code to generate a caption for an image."}, {"start": 405.0, "text": "I'm using the prompt builder syntax from Foundation Models."}, {"start": 409.0, "text": "I have a text prompt with instructions"}, {"start": 411.0, "text": "for the model about how to process the image."}, {"start": 414.0, "text": "Then I'll include the image as an attachment in my prompt."}, {"start": 419.0, "text": "Now I can ask the model to respond to the prompt,"}, {"start": 422.0, "text": "and it will generate a caption for me."}, {"start": 425.0, "text": "Now you can try this with your own prompts."}, {"start": 429.0, "text": "There are now multiple ways to analyze images, each with their own benefits."}, {"start": 434.0, "text": "The Foundation Models framework leverages large language models,"}, {"start": 439.0, "text": "which can do almost anything you ask them."}, {"start": 442.0, "text": "By comparison, traditional image processing frameworks, like Vision,"}, {"start": 447.0, "text": "use a fixed set of computer vision APIs."}, {"start": 451.0, "text": "Vision APIs are fined tuned for specific tasks,"}, {"start": 455.0, "text": "which they do really well."}, {"start": 457.0, "text": "And Vision is fast."}, {"start": 459.0, "text": "Often fast enough to analyze video frames in real time."}, {"start": 464.0, "text": "But you don't always have to choose between Vision and Foundation Models"}, {"start": 467.0, "text": "to analyze your images."}, {"start": 470.0, "text": "There's a way to leverage Vision's expertise"}, {"start": 472.0, "text": "with Foundation Model's versatility using tool calling."}, {"start": 477.0, "text": "I'll demonstrate how you can give large language models access"}, {"start": 481.0, "text": "to tools that run traditional image processing APIs,"}, {"start": 484.0, "text": "like those in Vision,"}, {"start": 486.0, "text": "to take image understanding to new levels."}, {"start": 489.0, "text": "But first, I'll give a quick refresher on tool calling."}, {"start": 494.0, "text": "Earlier I showed how you can give a prompt to a model"}, {"start": 497.0, "text": "and have it generate a response."}, {"start": 500.0, "text": "With tool calling,"}, {"start": 501.0, "text": "your model can now invoke a tool to run external code"}, {"start": 505.0, "text": "and get back a result."}, {"start": 507.0, "text": "The model can use this result in its response."}, {"start": 512.0, "text": "For example, I can have a weather tool,"}, {"start": 514.0, "text": "which fetches a weather forecast for a specific day."}, {"start": 518.0, "text": "My prompt is a question about the weather."}, {"start": 521.0, "text": "The model can't answer the question by itself,"}, {"start": 524.0, "text": "so it makes a tool call to the weather tool."}, {"start": 527.0, "text": "When a model makes a tool call,"}, {"start": 529.0, "text": "it will generate arguments needed by the tool."}, {"start": 532.0, "text": "In this case, the argument will be the date"}, {"start": 535.0, "text": "that the model wants to fetch the weather for."}, {"start": 538.0, "text": "The tool will then fetch the weather for the requested date"}, {"start": 541.0, "text": "and report back to the model."}, {"start": 543.0, "text": "Now the model can respond to the question about the weather."}, {"start": 548.0, "text": "For more information on tool calling,"}, {"start": 550.0, "text": "check out \"Deep dive into the Foundation Models framework\"."}, {"start": 555.0, "text": "This year, tool calling supports image arguments."}, {"start": 559.0, "text": "For example, now you can give a photo of a plant and ask a question about it."}, {"start": 564.0, "text": "If the model can't identify the plant by itself,"}, {"start": 567.0, "text": "I could create my own plant identification tool"}, {"start": 570.0, "text": "and give the model access."}, {"start": 572.0, "text": "The model would call the tool on the image to identify the plant."}, {"start": 577.0, "text": "Rather than passing the whole image as an argument,"}, {"start": 580.0, "text": "the model would instead pass a reference to the image."}, {"start": 584.0, "text": "The tool will analyze the image and return the name of the plant."}, {"start": 589.0, "text": "Now the model can respond with the correct information."}, {"start": 592.0, "text": "Here's the code."}, {"start": 594.0, "text": "My tool conforms to the tool protocol from the Foundation Models framework."}, {"start": 599.0, "text": "Tools must define input arguments."}, {"start": 602.0, "text": "For my plant identifier tool,"}, {"start": 604.0, "text": "I want the argument to be an ImageReference."}, {"start": 608.0, "text": "This signals to the model that the argument needs to be a reference"}, {"start": 612.0, "text": "to an existing image from the current chat session."}, {"start": 616.0, "text": "Tools also need to define a call method,"}, {"start": 619.0, "text": "which is invoked when the model calls the tool."}, {"start": 623.0, "text": "Inside the call method,"}, {"start": 624.0, "text": "I can access the imageReference from the tool arguments."}, {"start": 629.0, "text": "But now I need to resolve this reference into an actual image."}, {"start": 633.0, "text": "Each imageReference is only valid in the context of the transcript"}, {"start": 638.0, "text": "from which it was generated."}, {"start": 640.0, "text": "To access this transcript, use the history session property."}, {"start": 646.0, "text": "Using the transcript, the imageReference is resolved back into an imageAttachment."}, {"start": 652.0, "text": "Now the attachment is converted into a pixelBuffer, so it can be analyzed."}, {"start": 657.0, "text": "Tools can provide a lot of utility for models,"}, {"start": 661.0, "text": "particularly in tasks that models don't do well."}, {"start": 665.0, "text": "While you can write own tools,"}, {"start": 666.0, "text": "for common tasks, Vision is providing some tools for you."}, {"start": 671.0, "text": "Some models struggle to read barcodes and QR codes."}, {"start": 676.0, "text": "I have an event flyer here,"}, {"start": 678.0, "text": "and I'm asking the model to extract information"}, {"start": 680.0, "text": "like the date, location, and website registration."}, {"start": 684.0, "text": "Without tools enabled, the model can find the location and the date,"}, {"start": 689.0, "text": "but it can't read the QR code."}, {"start": 692.0, "text": "Vision provides a barcode reader tool which will help the model."}, {"start": 696.0, "text": "Now the model can make a tool call to the barcode reader."}, {"start": 700.0, "text": "The tool will analyze the image"}, {"start": 702.0, "text": "and return the website from the QR code."}, {"start": 705.0, "text": "Now the model can read all of the information correctly"}, {"start": 710.0, "text": "Vision gives you two tools."}, {"start": 713.0, "text": "You've already seen the barcode reader tool"}, {"start": 715.0, "text": "for scanning barcodes and QR codes."}, {"start": 717.0, "text": "There's also an OCR tool"}, {"start": 720.0, "text": "which is good for helping models read really fine or dense text."}, {"start": 725.0, "text": "It can read text in over 30 languages."}, {"start": 729.0, "text": "To use the tools, all you need to do is import Vision,"}, {"start": 733.0, "text": "and then configure your language model session"}, {"start": 736.0, "text": "with the tools you want to use."}, {"start": 739.0, "text": "Now the model will be able to call the tool"}, {"start": 741.0, "text": "to help answer your prompts."}, {"start": 744.0, "text": "It's also important that you give attached images a label"}, {"start": 747.0, "text": "when you want the model to make an image-based tool call."}, {"start": 751.0, "text": "This label is how the model will identify which image to pass to the tool."}, {"start": 757.0, "text": "You can also make your own tools using Vision."}, {"start": 761.0, "text": "Vision supports over 30 different types of image analysis."}, {"start": 765.0, "text": "I've mentioned image segmentation, but I'll highlight a few more."}, {"start": 770.0, "text": "Vision can also do facial analysis,"}, {"start": 774.0, "text": "pose estimation,"}, {"start": 776.0, "text": "detection and image classification,"}, {"start": 778.0, "text": "and even trajectory analysis and object tracking."}, {"start": 782.0, "text": "Check out \"Discover Swift enhancements in the Vision framework\""}, {"start": 786.0, "text": "to see the full list."}, {"start": 789.0, "text": "This year, Vision is available in more places than ever."}, {"start": 793.0, "text": "You can even use Vision to enhance your watchOS apps."}, {"start": 798.0, "text": "I have a watch app that displays information"}, {"start": 801.0, "text": "about local wildlife I can look at when I'm on a hike."}, {"start": 804.0, "text": "It has a bunch of different animals I might encounter,"}, {"start": 807.0, "text": "and I can select an animal to learn more about it."}, {"start": 811.0, "text": "The app displays a photo of the animal,"}, {"start": 814.0, "text": "but because the watch screen is so small,"}, {"start": 816.0, "text": "it's hard to see."}, {"start": 818.0, "text": "Vision can help."}, {"start": 820.0, "text": "I can use Vision's saliency analysis"}, {"start": 823.0, "text": "to identify subjects of interest in the photo."}, {"start": 827.0, "text": "Then I can crop the image to feature the main subject more prominently."}, {"start": 833.0, "text": "Here's the code to generate a crop using Vision."}, {"start": 837.0, "text": "First I'll the create the request."}, {"start": 840.0, "text": "I'm using GenerateObjectnessBasedSaliencyImageRequest."}, {"start": 846.0, "text": "Now I'll perform the request on the image."}, {"start": 849.0, "text": "This produces a saliency observation."}, {"start": 853.0, "text": "From this observation I can access the bounding boxes of the salient objects"}, {"start": 857.0, "text": "detected in the image."}, {"start": 860.0, "text": "I'll take most prominent object, and use this for my crop."}, {"start": 866.0, "text": "I've updated my app to display only the salient portion of the image."}, {"start": 871.0, "text": "Now when I select an animal, I can get a zoomed in view."}, {"start": 876.0, "text": "That looks much better."}, {"start": 879.0, "text": "I've covered a lot in this video."}, {"start": 881.0, "text": "Here's a quick recap."}, {"start": 884.0, "text": "Vision's new tap-to-segment API lets you interactively segment objects in an image."}, {"start": 890.0, "text": "Foundation Models now supports image inputs for large language models,"}, {"start": 895.0, "text": "which lets you analyze images in new ways"}, {"start": 898.0, "text": "that weren't possible before."}, {"start": 901.0, "text": "And you can create tools that use frameworks like Vision"}, {"start": 904.0, "text": "to make your image analysis even better."}, {"start": 907.0, "text": "You can also use Vision to enhance your apps on all platforms, including watchOS."}, {"start": 915.0, "text": "You can download the watchOS"}, {"start": 916.0, "text": "and tap-to-segment sample apps I showed earlier"}, {"start": 919.0, "text": "from the developer website."}, {"start": 921.0, "text": "And don't forget to watch \"Discover Swift enhancements in the Vision framework\""}, {"start": 925.0, "text": "to learn about more Vision APIs."}, {"start": 928.0, "text": "You can also check out \"What's new in the Foundation Models framework\""}, {"start": 932.0, "text": "to learn about the other ways large language models can enhance your apps."}, {"start": 937.0, "text": "Thanks for watching."}]}