{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-297", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/297", "wordCount": 2316, "segments": [{"start": 7.0, "text": "Hi, I'm David,"}, {"start": 9.0, "text": "an ML engineer on System Experience."}, {"start": 13.0, "text": "Let's build something with Visual Intelligence."}, {"start": 16.0, "text": "In this session, I'll take you step by step"}, {"start": 19.0, "text": "through integrating your app with Visual Intelligence"}, {"start": 23.0, "text": "and share some best practices along the way."}, {"start": 26.0, "text": "Since Visual Intelligence was introduced, people have been using it"}, {"start": 30.0, "text": "to quickly learn more about what's around them"}, {"start": 33.0, "text": "whether in their physical surroundings or on their iPhone screen."}, {"start": 39.0, "text": "This year, we're adding new capabilities like adding to contacts,"}, {"start": 43.0, "text": "saving multiple calendar events, and medical device logging,"}, {"start": 48.0, "text": "as well as bringing Visual Intelligence to iPad and macOS."}, {"start": 55.0, "text": "So how do you bring your app into this experience?"}, {"start": 58.0, "text": "I'll show you by building one."}, {"start": 60.0, "text": "I love listening to and discovering new music."}, {"start": 64.0, "text": "So I want to create an app that helps me discover albums"}, {"start": 67.0, "text": "and find upcoming concerts."}, {"start": 70.0, "text": "Here's what we'll build today."}, {"start": 72.0, "text": "This is my music app."}, {"start": 74.0, "text": "I can browse albums, check out upcoming concerts,"}, {"start": 77.0, "text": "and start listening to anything with a tap."}, {"start": 81.0, "text": "If I take a picture or screenshot of some album artwork,"}, {"start": 85.0, "text": "and highlight to search,"}, {"start": 86.0, "text": "my app shows matching albums and concerts right in Visual Intelligence."}, {"start": 91.0, "text": "I can even capture a post about an upcoming concert,"}, {"start": 96.0, "text": "use Visual Intelligence to add the event to my calendar"}, {"start": 101.0, "text": "and the concert shows up automatically in my app."}, {"start": 105.0, "text": "By the end of this session, you'll know how to build all of this."}, {"start": 109.0, "text": "There are a few steps"}, {"start": 110.0, "text": "to making the most of your integration with Visual Intelligence,"}, {"start": 113.0, "text": "which I'll go over today."}, {"start": 115.0, "text": "First, we'll define the content we want to return from our app"}, {"start": 119.0, "text": "using App entities."}, {"start": 122.0, "text": "Next, we'll implement a query so Visual Intelligence can find"}, {"start": 127.0, "text": "and return our app's content."}, {"start": 130.0, "text": "Then, we'll take our integration beyond iOS — bringing it to Mac and iPad,"}, {"start": 137.0, "text": "covering some considerations for each platform along the way."}, {"start": 142.0, "text": "And to wrap things up, we'll explore system store integrations"}, {"start": 147.0, "text": "where information extracted by Visual Intelligence"}, {"start": 150.0, "text": "can be read by your app automatically,"}, {"start": 152.0, "text": "from common data stores you may have already adopted."}, {"start": 156.0, "text": "Let's start with the basics of Image Search."}, {"start": 159.0, "text": "Integrating Image Search leverages both the App Intents"}, {"start": 163.0, "text": "and Visual Intelligence frameworks."}, {"start": 166.0, "text": "If you're new to App Intents,"}, {"start": 167.0, "text": "I'd recommend checking out these sessions from WWDC25."}, {"start": 174.0, "text": "The first step to integrating Image Search is defining the content we want to return."}, {"start": 180.0, "text": "We'll use App entities from the App Intents framework for this."}, {"start": 184.0, "text": "App entities are the nouns within your app."}, {"start": 187.0, "text": "In our app, I want our Image Search to first return visually similar albums,"}, {"start": 192.0, "text": "so I'll define an album entity."}, {"start": 196.0, "text": "Let's see how this looks in code."}, {"start": 200.0, "text": "We'll start by defining an AlbumEntity"}, {"start": 202.0, "text": "that Visual Intelligence can display in its search results."}, {"start": 207.0, "text": "First I'll add a default EntityQuery"}, {"start": 210.0, "text": "and a typeDisplayRepresentation, which are standard for any App entity."}, {"start": 217.0, "text": "Then I'll define the content of the entity."}, {"start": 221.0, "text": "Each AlbumEntity has an identifier,"}, {"start": 223.0, "text": "name, artistName, and thumbnail data for the album artwork."}, {"start": 230.0, "text": "And I'll add a displayRepresentation"}, {"start": 232.0, "text": "which tells Visual Intelligence how to present each result."}, {"start": 237.0, "text": "Let's talk about the display representation we just defined."}, {"start": 241.0, "text": "This is the first thing people will see in the Image Search results."}, {"start": 245.0, "text": "And there's not a lot of room —"}, {"start": 247.0, "text": "you get about three lines of text for a title and subtitle,"}, {"start": 251.0, "text": "as well as a thumbnail image."}, {"start": 255.0, "text": "It's good practice to put the most important"}, {"start": 257.0, "text": "identifying information here."}, {"start": 259.0, "text": "In my case, the album name and the artist."}, {"start": 264.0, "text": "And if you initialize a display representation"}, {"start": 267.0, "text": "with an image URL,"}, {"start": 269.0, "text": "I'd recommend serving a thumbnail-sized image when appropriate,"}, {"start": 273.0, "text": "rather than pointing to your full-resolution asset."}, {"start": 278.0, "text": "For example, if you always expect to return multiple results,"}, {"start": 282.0, "text": "using smaller images can help your results load faster"}, {"start": 286.0, "text": "and still look good in a two column layout."}, {"start": 290.0, "text": "However, if you only return one result,"}, {"start": 294.0, "text": "keep in mind that this image will take up the full width of the results sheet."}, {"start": 299.0, "text": "Now that I have my entity defined,"}, {"start": 302.0, "text": "how does Visual Intelligence actually query my app for results?"}, {"start": 306.0, "text": "That's where the Intent value query comes in."}, {"start": 310.0, "text": "An Intent value query is a lightweight query protocol"}, {"start": 313.0, "text": "that provides entity values to the system."}, {"start": 317.0, "text": "You might already have one"}, {"start": 318.0, "text": "if you've adopted App Intents to make your app work with Siri."}, {"start": 323.0, "text": "For Visual Intelligence, the key difference is the input,"}, {"start": 328.0, "text": "the system passes a SemanticContentDescriptor"}, {"start": 331.0, "text": "containing information about the captured image."}, {"start": 335.0, "text": "Let's jump back to the code to build this."}, {"start": 339.0, "text": "I'll adopt the IntentValueQuery protocol"}, {"start": 342.0, "text": "and implement its values for requirement with a SemanticContentDescriptor as input."}, {"start": 349.0, "text": "In the body, I'll grab the pixelBuffer from the input"}, {"start": 353.0, "text": "and pass it to my catalog.search method, which returns matching albums."}, {"start": 359.0, "text": "But how does that search actually work?"}, {"start": 361.0, "text": "Let's look at that next."}, {"start": 365.0, "text": "For this app, I'll search on device using a local catalog of saved albums."}, {"start": 371.0, "text": "I'll use the Vision framework for this,"}, {"start": 374.0, "text": "which provides pre-trained machine learning models for computer vision tasks."}, {"start": 380.0, "text": "Each entry in our catalog will have a featurePrint,"}, {"start": 383.0, "text": "a compact numerical representation of the image,"}, {"start": 386.0, "text": "which we can use to compare image similarity."}, {"start": 391.0, "text": "I'll define a function to compute feature prints"}, {"start": 394.0, "text": "using GenerateImageFeaturePrintRequest."}, {"start": 397.0, "text": "I'll make sure to pre-compute these for albums in my catalog,"}, {"start": 401.0, "text": "so we don't need to do this computation at query time."}, {"start": 405.0, "text": "For our query, I'll first convert the pixelBuffer to a CGImage"}, {"start": 409.0, "text": "using VideoToolbox."}, {"start": 412.0, "text": "Then, I'll generate a new feature print for this image."}, {"start": 417.0, "text": "I'll compare that against the pre-computed feature prints in my catalog,"}, {"start": 421.0, "text": "applying a maximum distance threshold to filter out dissimilar results."}, {"start": 428.0, "text": "Finally, I sort by similarity and return the top results."}, {"start": 434.0, "text": "A few things to note."}, {"start": 436.0, "text": "I pre-compute feature prints for my album catalog,"}, {"start": 439.0, "text": "to keep the query fast."}, {"start": 441.0, "text": "And I sort results by similarity so the best match appears first."}, {"start": 446.0, "text": "Whether you're searching on device or hitting a server,"}, {"start": 450.0, "text": "the same principles apply — return results fast and ranked."}, {"start": 456.0, "text": "I'd also recommend limiting the number of results returned"}, {"start": 459.0, "text": "to ensure they're relevant."}, {"start": 462.0, "text": "If you don't find any good matches, you can return an empty array."}, {"start": 466.0, "text": "The system will handle displaying an empty response."}, {"start": 470.0, "text": "And I encourage you to check out the Vision framework APIs"}, {"start": 474.0, "text": "to learn more about image processing techniques you can use in your app."}, {"start": 479.0, "text": "We just scratched the surface with feature prints,"}, {"start": 481.0, "text": "but you can do so much more like extract text,"}, {"start": 485.0, "text": "scan barcodes, detect faces, and classify images, just to name a few."}, {"start": 491.0, "text": "These can be incredibly useful techniques"}, {"start": 493.0, "text": "for extending the capabilities of your app's visual search."}, {"start": 498.0, "text": "Now, how can we land people on the right screen of the app"}, {"start": 502.0, "text": "when they tap on a result?"}, {"start": 504.0, "text": "For that, we need an OpenIntent."}, {"start": 507.0, "text": "When someone taps an album in the Image Search results,"}, {"start": 510.0, "text": "the system calls this intent with the selected entity."}, {"start": 515.0, "text": "My perform method navigates to the album detail page."}, {"start": 520.0, "text": "Your OpenIntent should take people straight to the content they selected."}, {"start": 525.0, "text": "If you already have an OpenIntent for your entity"}, {"start": 528.0, "text": "from adopting App Intents to power other features,"}, {"start": 531.0, "text": "you can reuse it here too."}, {"start": 534.0, "text": "You don't need a separate one just for Visual Intelligence."}, {"start": 538.0, "text": "And it's recommended to keep this lightweight."}, {"start": 541.0, "text": "This method runs as the app comes to the foreground,"}, {"start": 544.0, "text": "so do your navigation and save any heavy loading"}, {"start": 547.0, "text": "for after the view appears."}, {"start": 551.0, "text": "And that's everything you need for a basic Image Search integration."}, {"start": 555.0, "text": "Let's take a look at what we've built so far."}, {"start": 559.0, "text": "My friend sent me this recommendation,"}, {"start": 562.0, "text": "let's use Visual Intelligence to start listening to it in our app."}, {"start": 566.0, "text": "I'll take a screenshot, highlight to search,"}, {"start": 570.0, "text": "and choose our app from the available providers."}, {"start": 575.0, "text": "Our query worked,"}, {"start": 576.0, "text": "and we were able to find the album and return it as the top result."}, {"start": 581.0, "text": "It's worth mentioning that your app appears here alongside other adopting apps."}, {"start": 587.0, "text": "The system decides the ordering based"}, {"start": 589.0, "text": "on which Image Search providers are available on the device."}, {"start": 593.0, "text": "If I tap on this result, it takes me right to the album page in my app."}, {"start": 598.0, "text": "That's our entity, our query, and our OpenIntent all working together."}, {"start": 605.0, "text": "Now, let's bring this to more platforms."}, {"start": 608.0, "text": "This year,"}, {"start": 609.0, "text": "Visual Intelligence is also available on iPadOS and macOS."}, {"start": 615.0, "text": "The same APIs are available on these new platforms as well,"}, {"start": 619.0, "text": "with minimal changes needed to your app."}, {"start": 623.0, "text": "Your IntentValueQuery, your entities,"}, {"start": 626.0, "text": "and your OpenIntent all work across iOS, iPadOS, and macOS."}, {"start": 631.0, "text": "That's the same code we just wrote."}, {"start": 635.0, "text": "That said,"}, {"start": 636.0, "text": "there are a few platform differences worth keeping in mind."}, {"start": 640.0, "text": "On iOS, people often use Visual Intelligence through the camera —"}, {"start": 644.0, "text": "capturing physical objects like vinyl records or concert posters."}, {"start": 650.0, "text": "On macOS and iPad, the primary entry point is screenshots — capturing digital media."}, {"start": 657.0, "text": "Make sure your search handles both kinds of content well."}, {"start": 661.0, "text": "Also keep in mind that on Mac,"}, {"start": 664.0, "text": "the input pixel buffer can be much larger than what you'd encounter on iPhone."}, {"start": 669.0, "text": "Consider if resizing is necessary for your use case."}, {"start": 674.0, "text": "Let's build our app for macOS and see how it looks."}, {"start": 679.0, "text": "I'll take a screenshot of that same image,"}, {"start": 682.0, "text": "and with no changes to our query or entity code,"}, {"start": 686.0, "text": "our app's Image Search works on macOS."}, {"start": 690.0, "text": "The result looks great."}, {"start": 693.0, "text": "Now that we've covered the basics,"}, {"start": 695.0, "text": "I want to add even more capabilities to our app."}, {"start": 699.0, "text": "What if we could search not only for visually similar albums,"}, {"start": 703.0, "text": "but also for upcoming concerts by artists of those albums?"}, {"start": 707.0, "text": "For that, we can use UnionValue."}, {"start": 710.0, "text": "Since our app can only have one IntentValueQuery"}, {"start": 713.0, "text": "that accepts a SemanticContentDescriptor,"}, {"start": 716.0, "text": "I'll define a @UnionValue enum with a case for each entity type — album and concert."}, {"start": 724.0, "text": "And since I have two entity types now, I'll need an OpenIntent for each one."}, {"start": 732.0, "text": "Then I'll update my query to return this union type."}, {"start": 736.0, "text": "I'll search for the top matching albums first,"}, {"start": 740.0, "text": "then use the artists from those albums to find nearby concerts,"}, {"start": 744.0, "text": "and combine them into a single results list."}, {"start": 748.0, "text": "Consider if it makes sense for your app to return multiple types of results."}, {"start": 753.0, "text": "And it's worth thinking about the different types of content"}, {"start": 756.0, "text": "your app can return beyond simply matching pixels."}, {"start": 761.0, "text": "I found albums through image similarity,"}, {"start": 763.0, "text": "then used those artist names to surface nearby concerts,"}, {"start": 766.0, "text": "a completely different kind of result."}, {"start": 770.0, "text": "Feel free to be creative about the type of content you return"}, {"start": 773.0, "text": "based on the context."}, {"start": 776.0, "text": "As a final touch, if people don't find the result they're looking for immediately,"}, {"start": 781.0, "text": "I want to provide an easy way for them to continue the search inside the app."}, {"start": 787.0, "text": "We can use the semanticContentSearch schema to do that."}, {"start": 791.0, "text": "I'll create an intent conforming to the semanticContentSearch schema."}, {"start": 795.0, "text": "The system provides the semanticContent property automatically."}, {"start": 800.0, "text": "That's the same SemanticContentDescriptor we saw before with the pixel buffer."}, {"start": 805.0, "text": "In perform, I'll navigate to an in-app search view"}, {"start": 809.0, "text": "with some pre-populated search results."}, {"start": 813.0, "text": "Now when someone taps More results,"}, {"start": 816.0, "text": "they'll land in my app's full search experience."}, {"start": 819.0, "text": "It's good practice to use semantic content search"}, {"start": 822.0, "text": "to give people a way to continue into your full search experience."}, {"start": 827.0, "text": "And you can pre-populate your search view based on the input context,"}, {"start": 831.0, "text": "rather than starting from scratch."}, {"start": 834.0, "text": "Your app can show much more than the Visual Intelligence results view —"}, {"start": 837.0, "text": "filters, categories, the full depth of your content."}, {"start": 842.0, "text": "Take advantage of that."}, {"start": 844.0, "text": "Let's see everything we've built in action."}, {"start": 847.0, "text": "I'll take another screenshot of this album,"}, {"start": 850.0, "text": "and my app returns matching albums and concerts right in Visual Intelligence."}, {"start": 856.0, "text": "And if I want to browse more,"}, {"start": 858.0, "text": "tapping the More results button takes me into my app's full search."}, {"start": 863.0, "text": "We've talked about your app providing results to Visual Intelligence."}, {"start": 867.0, "text": "But there's another side to this story."}, {"start": 870.0, "text": "Your app can also receive data from Visual Intelligence"}, {"start": 872.0, "text": "through system store integrations."}, {"start": 875.0, "text": "Providing results to Visual Intelligence"}, {"start": 877.0, "text": "is done through the Image Search integration,"}, {"start": 880.0, "text": "which is everything we've built so far."}, {"start": 883.0, "text": "Other Visual Intelligence actions write data to system stores,"}, {"start": 887.0, "text": "which provide developers with a bridge to shared system data."}, {"start": 891.0, "text": "Events can be read with EventKit, contact information with Contacts,"}, {"start": 897.0, "text": "and medical device readings with HealthKit."}, {"start": 900.0, "text": "If your app already reads from the data stores in these frameworks,"}, {"start": 904.0, "text": "Visual Intelligence becomes a new source of input automatically."}, {"start": 910.0, "text": "For our app, I want to know upcoming concerts people are interested in"}, {"start": 914.0, "text": "so we can suggest songs for them to listen to beforehand."}, {"start": 918.0, "text": "So let's add an EventKit integration to access these events."}, {"start": 923.0, "text": "This is my UpcomingConcertManager,"}, {"start": 926.0, "text": "which uses EKEventStore."}, {"start": 930.0, "text": "I'll request read access to calendar, then query for upcoming events."}, {"start": 937.0, "text": "For our app, I'll simply filter for events in the near future"}, {"start": 941.0, "text": "that match artists in my catalog."}, {"start": 945.0, "text": "I'll also add a notification observer"}, {"start": 947.0, "text": "so new events, including ones created by Visual Intelligence,"}, {"start": 951.0, "text": "appear automatically."}, {"start": 954.0, "text": "Now let's see the final piece."}, {"start": 957.0, "text": "When I capture this social media post about an upcoming concert,"}, {"start": 961.0, "text": "Visual Intelligence detects the event so I can add it to my calendar."}, {"start": 966.0, "text": "When I open my app,"}, {"start": 968.0, "text": "it's already there in Upcoming Concerts, with a suggestion to start listening."}, {"start": 974.0, "text": "The same pattern applies to other system stores."}, {"start": 978.0, "text": "Contacts added through Visual Intelligence,"}, {"start": 980.0, "text": "for example from a business card, can be accessed through CNContactStore."}, {"start": 987.0, "text": "And medical device readings captured by Visual Intelligence"}, {"start": 991.0, "text": "from displays on blood pressure monitors,"}, {"start": 993.0, "text": "glucose meters or weight scales can be queried using HKHealthStore."}, {"start": 1000.0, "text": "If your health or fitness app reads from HealthKit,"}, {"start": 1004.0, "text": "Visual Intelligence becomes another way for people to log data"}, {"start": 1007.0, "text": "without manual entry."}, {"start": 1010.0, "text": "We've covered a lot today."}, {"start": 1012.0, "text": "To recap, Visual Intelligence offers two powerful integration points for your app."}, {"start": 1018.0, "text": "You can provide results to Visual Intelligence"}, {"start": 1020.0, "text": "through Image Search,"}, {"start": 1022.0, "text": "and you can receive data from Visual Intelligence"}, {"start": 1024.0, "text": "through system store integrations."}, {"start": 1027.0, "text": "With Visual Intelligence now available on iOS, iPadOS, and macOS,"}, {"start": 1033.0, "text": "your integration can reach people across their devices."}, {"start": 1037.0, "text": "If you want to learn more,"}, {"start": 1039.0, "text": "check out the documentation available on the developer website."}, {"start": 1044.0, "text": "You can also view these related sessions to explore further capabilities"}, {"start": 1048.0, "text": "in App Intents and the Vision framework."}, {"start": 1052.0, "text": "Thanks for watching."}, {"start": 1053.0, "text": "I can't wait to see what you build with Visual Intelligence."}]}