{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-343", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/343", "wordCount": 3568, "segments": [{"start": 7.0, "text": "Hi, I'm Antonio Cancio,"}, {"start": 9.0, "text": "a software engineer on the App Intents team."}, {"start": 12.0, "text": "Let's \"Explore advanced App Intents features for Siri and Apple Intelligence.\""}, {"start": 18.0, "text": "Today, you'll learn techniques"}, {"start": 19.0, "text": "to take the experience of your app in Siri and Apple Intelligence"}, {"start": 22.0, "text": "beyond the basics, so it feels polished, personal, and unmistakably yours."}, {"start": 28.0, "text": "This talk assumes a basic understanding of App Intents and App Schemas."}, {"start": 32.0, "text": "If they're new to you, I recommend starting with these sessions"}, {"start": 36.0, "text": "which cover the fundamentals."}, {"start": 38.0, "text": "Let's go through today's plan."}, {"start": 40.0, "text": "First, we'll explore how you can make people's conversations"}, {"start": 44.0, "text": "with your app through Siri more intuitive and familiar."}, {"start": 47.0, "text": "You'll learn how to build custom responses that match your app's look and feel,"}, {"start": 51.0, "text": "and how to set up interaction donations"}, {"start": 53.0, "text": "that help Apple Intelligence feel more personal."}, {"start": 57.0, "text": "Then, we'll discuss how you can make your content more widely available."}, {"start": 61.0, "text": "From semantic index integration to structured search, in-app search,"}, {"start": 66.0, "text": "and onscreen awareness, you'll come away knowing how to help Siri find your content"}, {"start": 70.0, "text": "and connect what's visible on screen to actions it can understand."}, {"start": 75.0, "text": "Finally, people can use Siri from anywhere in the system."}, {"start": 80.0, "text": "We'll cover how to add entity annotations"}, {"start": 82.0, "text": "to existing integrations with notifications,"}, {"start": 84.0, "text": "Now Playing, and alarms,"}, {"start": 87.0, "text": "so people can act on your content wherever they encounter it."}, {"start": 91.0, "text": "To show this in practice,"}, {"start": 92.0, "text": "I wanna share a few apps that my colleagues and I have been building:"}, {"start": 96.0, "text": "CosmoTunes, which lets me play music,"}, {"start": 98.0, "text": "and create alarms and timers with my favorite songs;"}, {"start": 102.0, "text": "UnicornChat, a messaging app to stay in touch with friends;"}, {"start": 106.0, "text": "and CometCal to manage my calendar."}, {"start": 109.0, "text": "Hm."}, {"start": 110.0, "text": "Hey, there's something about App Intents that encourages celestial symbolism."}, {"start": 115.0, "text": "You can download these samples from the links below and follow along."}, {"start": 120.0, "text": "Let's start with shaping the Siri conversation using custom responses."}, {"start": 125.0, "text": "Siri does the heavy lifting."}, {"start": 127.0, "text": "It understands natural language, picks the right action,"}, {"start": 130.0, "text": "and crafts a helpful response."}, {"start": 133.0, "text": "The App Intents framework gives you the tools to shape what Siri does,"}, {"start": 137.0, "text": "and refine how it responds."}, {"start": 139.0, "text": "By tailoring how it responds,"}, {"start": 141.0, "text": "you can let your app's unique personality shine through."}, {"start": 144.0, "text": "Let me illustrate this with some code."}, {"start": 147.0, "text": "In CosmoTunes, .addToPlaylistIntent lets people add songs to a playlist."}, {"start": 152.0, "text": "To start, I want Siri to handle the response."}, {"start": 155.0, "text": "In the perform method of the intent, I'll add the song to the playlist,"}, {"start": 158.0, "text": "and return an empty IntentResult."}, {"start": 161.0, "text": "This tells Siri to take care of the response when the intent runs."}, {"start": 165.0, "text": "After trying that out, I want the response to better match the app's personality."}, {"start": 170.0, "text": "I call songs tracks and playlists mix tapes."}, {"start": 174.0, "text": "To customize this, I mark the perform method as providing a dialog response"}, {"start": 178.0, "text": "by adding the ProvidesDialog protocol."}, {"start": 181.0, "text": "I'll also change the IntentResult by passing an IntentDialog"}, {"start": 185.0, "text": "containing both full and supporting strings."}, {"start": 188.0, "text": "Siri can show the supporting string with UI,"}, {"start": 190.0, "text": "and read the full dialog on voice-only devices like AirPods."}, {"start": 194.0, "text": "Because of this, the full string should describe what happened on its own."}, {"start": 198.0, "text": "That covers responses when your intent has finished."}, {"start": 202.0, "text": "But what if you want to ask people a question while your intent is running?"}, {"start": 206.0, "text": "A well-placed clarifying question lets people finish the action"}, {"start": 209.0, "text": "they meant to take."}, {"start": 211.0, "text": "To ask a question before your intent result,"}, {"start": 213.0, "text": "use a dialog request within your perform method."}, {"start": 217.0, "text": "People can create timers in CosmoTunes that start or stop audio playback"}, {"start": 221.0, "text": "after a set amount of time."}, {"start": 223.0, "text": "This intent adopts a schema with required and optional parameters."}, {"start": 227.0, "text": "If there's a timer already running,"}, {"start": 229.0, "text": "I want to ask the person to name this new timer to avoid confusion."}, {"start": 234.0, "text": "I'll request a value for the optional label parameter"}, {"start": 236.0, "text": "when one hasn't been provided."}, {"start": 239.0, "text": "If you want to ask people to choose from a list of items,"}, {"start": 241.0, "text": "or ask for a confirmation,"}, {"start": 243.0, "text": "check out the sample app and documentation"}, {"start": 245.0, "text": "to learn about other kinds of dialog requests."}, {"start": 248.0, "text": "Next, I want Siri's visuals to match the app's look and feel."}, {"start": 252.0, "text": "Entity display representation and custom views in intent responses"}, {"start": 256.0, "text": "are a great opportunity to visually present information"}, {"start": 259.0, "text": "and surface your app's identity alongside the dialog."}, {"start": 264.0, "text": "Defining an entity's DisplayRepresentation can tell Siri"}, {"start": 266.0, "text": "how your entity should look and read when showing your content."}, {"start": 270.0, "text": "Entity display representation can be used in responses,"}, {"start": 273.0, "text": "like when an entity has been created or updated."}, {"start": 276.0, "text": "They are also used when asking someone to choose between similar entities,"}, {"start": 280.0, "text": "or when answering questions about content in your app."}, {"start": 283.0, "text": "Spotlight and Shortcuts can use them, too."}, {"start": 287.0, "text": "To define a basic DisplayRepresentation, provide a title."}, {"start": 291.0, "text": "You can make these even richer by providing a subtitle and an image."}, {"start": 297.0, "text": "I'll provide an image associated with my songs that Siri can show"}, {"start": 300.0, "text": "when people ask about their songs in the app."}, {"start": 303.0, "text": "Entity DisplayRepresentation lets you refine the visual identity"}, {"start": 307.0, "text": "of your entities across the system."}, {"start": 310.0, "text": "To define the visual responses of some specific actions,"}, {"start": 314.0, "text": "you can use custom view snippets."}, {"start": 316.0, "text": "Back in AddToPlaylistIntent,"}, {"start": 318.0, "text": "Siri already responds automatically using the entity display representation."}, {"start": 323.0, "text": "To use a custom view,"}, {"start": 325.0, "text": "I add the ShowsSnippetView return type to my perform method."}, {"start": 329.0, "text": "This lets me return an IntentResult with a SwiftUI view,"}, {"start": 332.0, "text": "like my PlaylistSnippetView which displays the playlist details in familiar colors."}, {"start": 338.0, "text": "When approaching customization, test your intents and decide"}, {"start": 341.0, "text": "where customization actually makes sense for your app."}, {"start": 345.0, "text": "Make sure your responses are accurate and sound natural across all platforms,"}, {"start": 349.0, "text": "including voice-only devices like AirPods."}, {"start": 352.0, "text": "Remember to ask clarifying questions sparingly to avoid friction."}, {"start": 356.0, "text": "Finally, use custom visuals to bring your app's identity to Siri,"}, {"start": 361.0, "text": "keeping in mind how they'll scale across the ecosystem."}, {"start": 365.0, "text": "Your intent responses typically come at the end of a Siri interaction."}, {"start": 369.0, "text": "But, Siri can ask additional questions before even calling your intent."}, {"start": 374.0, "text": "For example, if I ask to message someone,"}, {"start": 377.0, "text": "Siri may ask me to choose from a list of similarly named contacts,"}, {"start": 380.0, "text": "because the system isn't sure which one I mean."}, {"start": 384.0, "text": "That brings us to Interaction Donations,"}, {"start": 386.0, "text": "an API you can adopt to help Apple Intelligence be even smarter"}, {"start": 390.0, "text": "in handling those kinds of requests."}, {"start": 393.0, "text": "Here's the good news."}, {"start": 395.0, "text": "When people interact with your app through Siri or Shortcuts,"}, {"start": 397.0, "text": "the system already knows about it."}, {"start": 400.0, "text": "But, Apple Intelligence can't learn from actions"}, {"start": 402.0, "text": "people take through your app's UI without your help."}, {"start": 405.0, "text": "That's where donations come in."}, {"start": 407.0, "text": "Each donation is a hint that a person took a specific action in your app's UI."}, {"start": 413.0, "text": "The system stores these as schema-conforming App Intents"}, {"start": 415.0, "text": "in a temporary transcript,"}, {"start": 417.0, "text": "giving Siri the context it needs to make smarter decisions."}, {"start": 421.0, "text": "The transcript contains UI interaction donations over time."}, {"start": 425.0, "text": "Say someone opens UnicornChat and sends a message from the compose view."}, {"start": 430.0, "text": "Right then, the app donates the send message action to the system,"}, {"start": 433.0, "text": "using the SendMessageIntent schema."}, {"start": 436.0, "text": "After messaging them frequently in the app,"}, {"start": 439.0, "text": "eventually, when someone says:"}, {"start": 441.0, "text": "\"Send a message to a contact from the Home Screen,\""}, {"start": 444.0, "text": "Siri might infer the right app to use for that contact."}, {"start": 447.0, "text": "To adopt interaction donations for message sending,"}, {"start": 450.0, "text": "I started by looking at the ConversationView in UnicornChat."}, {"start": 454.0, "text": "The ConversationView and the sendMessage intent"}, {"start": 457.0, "text": "both call the same helper to send the message."}, {"start": 460.0, "text": "That looks like a good place to start donating the sendMessage intent."}, {"start": 464.0, "text": "I'll add a donateIntent parameter"}, {"start": 465.0, "text": "so I know whether the helper was called from the intent or from the UI."}, {"start": 470.0, "text": "Apple Intelligence already learns from Siri interactions,"}, {"start": 473.0, "text": "so I only need to donate UI interactions."}, {"start": 477.0, "text": "Then, I'll create the intent, populate its parameters and the intent result,"}, {"start": 482.0, "text": "and donate it via the IntentDonationManager API."}, {"start": 486.0, "text": "Now when a person opens the app and sends a message,"}, {"start": 489.0, "text": "the system can learn when they prefer to use UnicornChat."}, {"start": 493.0, "text": "Beyond learning preferences,"}, {"start": 494.0, "text": "Interaction Donations also keep Siri aware of ongoing activities in your app."}, {"start": 499.0, "text": "This is especially useful for activities people might start or stop with Siri."}, {"start": 504.0, "text": "In an app in the Maps domain,"}, {"start": 506.0, "text": "a person could start a NavigationSession with the app's UI"}, {"start": 509.0, "text": "which donates that interaction."}, {"start": 511.0, "text": "Then, the person gets in the car, and asks Siri to add a stop on their way."}, {"start": 516.0, "text": "Thanks to the Interaction Donation,"}, {"start": 519.0, "text": "Siri can know what NavigationSession is active in the app,"}, {"start": 522.0, "text": "and help the person with their request."}, {"start": 525.0, "text": "This pattern applies to intents that start"}, {"start": 527.0, "text": "or stop NavigationSessions in the Maps domain,"}, {"start": 529.0, "text": "and stop, start, pause, or lap stopwatches in the Clock domain."}, {"start": 535.0, "text": "Your interaction donations should accurately represent"}, {"start": 538.0, "text": "real user behavior in your app."}, {"start": 540.0, "text": "If your app donates excessively, the system may ignore those donations."}, {"start": 545.0, "text": "Once Siri has gathered all the parameter values"}, {"start": 547.0, "text": "and is ready to call your App Intent, there is one final step: confirmation."}, {"start": 554.0, "text": "Asking people to confirm that the action looks right"}, {"start": 556.0, "text": "keeps them informed and protects them from unintended side effects,"}, {"start": 560.0, "text": "which are a known risk with Large Language Models."}, {"start": 563.0, "text": "This matters most for intents that could have meaningful side effects on your data,"}, {"start": 566.0, "text": "or the outside world."}, {"start": 569.0, "text": "That's why Siri can automatically confirm these kinds of intents."}, {"start": 572.0, "text": "For example, it can confirm when I say:"}, {"start": 575.0, "text": "\"Cancel my expedition next week in CometCal.\""}, {"start": 579.0, "text": "This matters even more for intents that update app content"}, {"start": 583.0, "text": "which a person has made public or shared with others."}, {"start": 586.0, "text": "For example, Siri may not confirm when I update a personal event,"}, {"start": 591.0, "text": "but it may confirm when I ask it to update Crew Lunch"}, {"start": 593.0, "text": "since I'm updating an event with attendees."}, {"start": 596.0, "text": "By default, Siri assumes your entities are private to the person,"}, {"start": 600.0, "text": "and may skip confirmations for them."}, {"start": 603.0, "text": "To tell Siri the owner has made an entity public or shared it with others,"}, {"start": 607.0, "text": "conform the relevant entities to the new OwnershipProvidingEntity protocol."}, {"start": 612.0, "text": "Only add the protocol to entities"}, {"start": 614.0, "text": "that people can share or make public in your app."}, {"start": 617.0, "text": "Then, provide the ownership state."}, {"start": 620.0, "text": "Keep the ownership state up to date"}, {"start": 622.0, "text": "whenever the system requests an entity from your app."}, {"start": 625.0, "text": "This ensures Siri has the necessary information when deciding to confirm."}, {"start": 631.0, "text": "Remember those entity display representations"}, {"start": 633.0, "text": "we customized earlier?"}, {"start": 635.0, "text": "Siri can use them as visuals in these intent confirmations, too."}, {"start": 639.0, "text": "Giving people a chance to confirm actions when appropriate"}, {"start": 642.0, "text": "builds trust in your app's experience with Siri."}, {"start": 646.0, "text": "To learn more about other ways to establish trust and mitigate risks,"}, {"start": 650.0, "text": "check out \"Secure your app: Mitigate risks to agentic features.\""}, {"start": 656.0, "text": "So far, I've defined how the actions in these apps work with Siri,"}, {"start": 660.0, "text": "given Apple Intelligence context about how people use my apps,"}, {"start": 663.0, "text": "and helped Siri protect people from unintended side effects."}, {"start": 667.0, "text": "Next, let's talk about how Siri finds content in the first place."}, {"start": 672.0, "text": "There are three paths I want to cover:"}, {"start": 674.0, "text": "the semantic index, structured search, and in-app search."}, {"start": 679.0, "text": "Playlists in CosmoTunes are all available locally on device."}, {"start": 683.0, "text": "To help Apple Intelligence find them,"}, {"start": 685.0, "text": "I'll adopt IndexedEntity and index those entities in Spotlight."}, {"start": 690.0, "text": "I'll use the .indexAppEntities method on CSSearchableIndex,"}, {"start": 694.0, "text": "which populates the Spotlight semantic index."}, {"start": 698.0, "text": "Now I can ask Siri: \"Play my WWDC playlist in CosmoTunes.\""}, {"start": 705.0, "text": "I can also search for my playlists in the Spotlight search UI."}, {"start": 709.0, "text": "And depending on the App Intents domain,"}, {"start": 711.0, "text": "indexing entities in Spotlight provides semantic search capabilities."}, {"start": 715.0, "text": "This means Apple Intelligence and Siri can understand your entities based on meaning,"}, {"start": 720.0, "text": "not just exact keywords."}, {"start": 723.0, "text": "Adding to the index is the first step."}, {"start": 725.0, "text": "Keeping it up to date is key to helping Siri find your content."}, {"start": 730.0, "text": "Index new entities as people add content to your app."}, {"start": 733.0, "text": "Update existing entries when key properties change,"}, {"start": 736.0, "text": "especially those used in your display representation."}, {"start": 739.0, "text": "When people remove content, delete those index entries too."}, {"start": 744.0, "text": "Spotlight may need your app to reindex its entities."}, {"start": 747.0, "text": "Your app can support reindexing by adopting the new IndexedEntityQuery."}, {"start": 752.0, "text": "Check out IndexedEntityQuery in the sample project."}, {"start": 755.0, "text": "If your project already supports reindexing with Core Spotlight-level APIs,"}, {"start": 759.0, "text": "you do not need to define an IndexedEntityQuery."}, {"start": 763.0, "text": "However, you might not index your entities if your content dataset is large,"}, {"start": 767.0, "text": "lives on a server, or changes too frequently to index ahead of time."}, {"start": 771.0, "text": "For example, I decided to index all the app's playlists, but not songs."}, {"start": 776.0, "text": "To still give people the flexibility in playing songs with Siri,"}, {"start": 779.0, "text": "I reached for an IntentValueQuery."}, {"start": 783.0, "text": "IntentValueQuery is suitable if you don't index all your entities ahead of time."}, {"start": 788.0, "text": "This is very similar to EntityQuery."}, {"start": 790.0, "text": "The key differences are that"}, {"start": 792.0, "text": "your app receives a structured search input from the system,"}, {"start": 795.0, "text": "and you can return more than one entity type."}, {"start": 798.0, "text": "Siri needs an entity for the audioEntity parameter"}, {"start": 801.0, "text": "on the PlayAudioIntent in CosmoTunes."}, {"start": 804.0, "text": "To find the entity, Siri calls the IntentValueQuery with an AudioSearch."}, {"start": 809.0, "text": "The query maps the structured properties of that search input"}, {"start": 812.0, "text": "to audio entities in the app."}, {"start": 815.0, "text": "In the IntentValueQuery, I implemented the values for method"}, {"start": 818.0, "text": "to handle the AudioSearch input, and return an AudioEntity."}, {"start": 823.0, "text": "AudioEntity is a UnionValue type that includes both songs and playlists."}, {"start": 828.0, "text": "The AudioSearch value has a .criteria property"}, {"start": 830.0, "text": "that describes the person's query."}, {"start": 833.0, "text": "The .searchQuery case contains the relevant part of what the person said,"}, {"start": 836.0, "text": "and I use that to find matching entities."}, {"start": 839.0, "text": "The app also supports an unspecified search."}, {"start": 842.0, "text": "For example: \"Play CosmoTunes\""}, {"start": 845.0, "text": "which isn't specific about what I want to play."}, {"start": 848.0, "text": "In that case, the app jumps straight into playing songs I've previously liked."}, {"start": 853.0, "text": "There's also a URL case for when someone references a link from your app."}, {"start": 857.0, "text": "Like: \"Play that playlist Glow sent me.\""}, {"start": 861.0, "text": "Check out the documentation for the full set of AudioSearch criteria."}, {"start": 867.0, "text": "Sometimes people aren't asking Siri to take action,"}, {"start": 870.0, "text": "they just want to find something."}, {"start": 872.0, "text": "When I ask: \"Show me running playlists in CosmoTunes.\""}, {"start": 875.0, "text": "Siri can display a list of entity search results."}, {"start": 878.0, "text": "That's a nice default."}, {"start": 880.0, "text": "But I spent a lot of time crafting the app's own search experience,"}, {"start": 883.0, "text": "and I'd love to show these results there."}, {"start": 886.0, "text": "To do this, I'll adopt the system .searchInApp schema."}, {"start": 890.0, "text": "The .system search schema introduced in iOS 17 is now named .system.searchInApp."}, {"start": 896.0, "text": "It is part of the System App Schema domain,"}, {"start": 899.0, "text": "and it lets people search in your app with Siri,"}, {"start": 901.0, "text": "no matter which other domains you adopt, and even if you don't index your entities."}, {"start": 908.0, "text": "Siri calls this intent with the same string it searched for,"}, {"start": 911.0, "text": "and the intent's perform method finds and shows those results in the app."}, {"start": 915.0, "text": "Spotlight and structured search let Siri reason about your content."}, {"start": 919.0, "text": "That's great if people ask Siri to play content in the app by name"}, {"start": 923.0, "text": "but just like in everyday conversations, people often refer to what they see."}, {"start": 928.0, "text": "That's why I want to allow people to interact with the audio content"}, {"start": 931.0, "text": "they're looking at on their screen."}, {"start": 934.0, "text": "Onscreen awareness is how your app connects what's visible on screen"}, {"start": 937.0, "text": "to structured information and actions the system understands."}, {"start": 942.0, "text": "Siri can then resolve references like: \"Play the third one\" or \"that conversation\""}, {"start": 949.0, "text": "without the person naming them explicitly."}, {"start": 952.0, "text": "When people start a Siri request, Siri has an understanding of text on screen,"}, {"start": 957.0, "text": "but it's limited to exactly what's in the pixels."}, {"start": 961.0, "text": "For example, Siri can't act on the tracks shown,"}, {"start": 964.0, "text": "and it may not be able to tell you about the artist"}, {"start": 967.0, "text": "because the artist isn't currently shown on screen."}, {"start": 970.0, "text": "Adopting onscreen awareness APIs"}, {"start": 973.0, "text": "provides Siri with additional context of what entities are on screen,"}, {"start": 976.0, "text": "and where they are on screen."}, {"start": 978.0, "text": "This onscreen context means Siri can answer detailed questions"}, {"start": 982.0, "text": "about those entities, and take action on them."}, {"start": 985.0, "text": "When adopting onscreen awareness, the NSUserActivity and View Annotation APIs"}, {"start": 990.0, "text": "are where you should start."}, {"start": 992.0, "text": "With NSUserActivity, attach .userActivity to the view"}, {"start": 996.0, "text": "representing your primary onscreen content."}, {"start": 999.0, "text": "Use the View Entity annotation"}, {"start": 1001.0, "text": "when the entity is one item among many on screen."}, {"start": 1005.0, "text": "Attach .appEntityIdentifier to each view that represents an entity."}, {"start": 1009.0, "text": "In CosmoTunes, AlbumView uses a View Entity annotation"}, {"start": 1013.0, "text": "because both the album and the containing tracks are visible."}, {"start": 1016.0, "text": "NowPlayingView uses NSUserActivity"}, {"start": 1019.0, "text": "because the screen is dedicated to the currently playing item."}, {"start": 1023.0, "text": "NSUserActivity and View Entity annotations"}, {"start": 1026.0, "text": "are enough when a screen has a handful of entities."}, {"start": 1029.0, "text": "But there are two more onscreen awareness APIs."}, {"start": 1032.0, "text": "The first is for lists and collections, where you display many entities at once."}, {"start": 1038.0, "text": "Tracks in CosmoTunes are displayed in lists in a few views of the app."}, {"start": 1042.0, "text": "Collection annotations help me avoid the overhead"}, {"start": 1045.0, "text": "of attaching an annotation to every single row."}, {"start": 1049.0, "text": "Instead, the system fetches identifiers lazily, as it needs them."}, {"start": 1054.0, "text": "Collection annotations also let Siri discover entities that have been selected"}, {"start": 1058.0, "text": "and scrolled off screen."}, {"start": 1060.0, "text": "Per row annotations disappear as soon as the view leaves the view hierarchy."}, {"start": 1064.0, "text": "In SwiftUI, use the .appEntityIdentifier (forSelectionType:) modifier on a List,"}, {"start": 1071.0, "text": "returning the EntityIdentifier for each item's selection ID."}, {"start": 1076.0, "text": "The second API is the custom canvas view annotation."}, {"start": 1079.0, "text": "I built this custom canvas view that looks like a piano roll."}, {"start": 1083.0, "text": "It illustrates the notes in the current track"}, {"start": 1085.0, "text": "and brings the unique retro look CosmoTunes is known for."}, {"start": 1089.0, "text": "I want people to be able to act on the associated song"}, {"start": 1092.0, "text": "using Siri whenever this canvas is visible."}, {"start": 1095.0, "text": "To help the system understand this non-standard subview,"}, {"start": 1098.0, "text": "I used the custom canvas view annotation."}, {"start": 1101.0, "text": "If you're using SwiftUI, check out how I adopted this in the PianoRollView"}, {"start": 1105.0, "text": "in the CosmoTunes sample code."}, {"start": 1108.0, "text": "UIKit and AppKit also support all of the onscreen awareness APIs."}, {"start": 1113.0, "text": "Check out the documentation for: AppEntityAnnotatable,"}, {"start": 1117.0, "text": "UICollectionViewAppIntentsDataSource, and appEntityUIElementProvider."}, {"start": 1122.0, "text": "And to learn more"}, {"start": 1123.0, "text": "about how these entity annotations help power contextual menu items in UIKit apps,"}, {"start": 1129.0, "text": "check out, \"Modernize your UIKit app.\""}, {"start": 1133.0, "text": "After adopting onscreen awareness,"}, {"start": 1134.0, "text": "some of the app's views show many entities at once."}, {"start": 1139.0, "text": "Siri needs to quickly understand"}, {"start": 1141.0, "text": "if the on-screen entities relate to a request."}, {"start": 1144.0, "text": "For example, someone asks Siri to play the third one."}, {"start": 1148.0, "text": "If Siri can't understand my on-screen entities quickly enough,"}, {"start": 1152.0, "text": "it may ask to clarify or play something else entirely."}, {"start": 1156.0, "text": "People can abandon the request when that happens."}, {"start": 1159.0, "text": "The entity display representations you customized earlier can help."}, {"start": 1163.0, "text": "In CosmoTunes, I enabled display representation querying"}, {"start": 1166.0, "text": "on the playlist entity query"}, {"start": 1169.0, "text": "by implementing the displayRepresentations method."}, {"start": 1173.0, "text": "Now, when Siri is trying to understand the content on screen,"}, {"start": 1176.0, "text": "it can query just the text representation of the entity"}, {"start": 1180.0, "text": "and skip the overhead of fetching the full content from the database."}, {"start": 1184.0, "text": "Onscreen awareness provides Siri with additional context"}, {"start": 1187.0, "text": "when people are looking at your app."}, {"start": 1189.0, "text": "Beyond the UI, your app already integrates closely with other parts of the system."}, {"start": 1195.0, "text": "To give Siri even more context,"}, {"start": 1197.0, "text": "you can connect entities to integrations that you already adopt,"}, {"start": 1200.0, "text": "like user notifications."}, {"start": 1202.0, "text": "With this added context, your app entities act as a universal language."}, {"start": 1208.0, "text": "They let Siri understand not just what's on screen,"}, {"start": 1211.0, "text": "but how other system integrations"}, {"start": 1213.0, "text": "and time-sensitive events relate to your content."}, {"start": 1217.0, "text": "I'll add entities to three integrations the apps already use:"}, {"start": 1221.0, "text": "UserNotifications, NowPlaying, and AlarmKit."}, {"start": 1225.0, "text": "When I'm done, I'll be able to say: \"Play the live version\""}, {"start": 1229.0, "text": "letting me easily switch to a different version of the currently playing song."}, {"start": 1234.0, "text": "When a UnicornChat notification is announced on AirPods"}, {"start": 1244.0, "text": "I can say: \"Reply, 'ok,"}, {"start": 1247.0, "text": "I'll swing by the unicorn supply store and pick those up.'\""}, {"start": 1252.0, "text": "And Snooze it to snooze an alarm from CosmoTunes."}, {"start": 1256.0, "text": "All three use the same pattern, and we call these entity annotations."}, {"start": 1263.0, "text": "Annotating notifications with entities gives Siri concrete entity context"}, {"start": 1267.0, "text": "when announcing a notification on AirPods."}, {"start": 1270.0, "text": "When listening to the announced notification,"}, {"start": 1273.0, "text": "the person might want to act on the entity behind it,"}, {"start": 1276.0, "text": "like replying to a message or checking off a reminder."}, {"start": 1280.0, "text": "To give the additional context to Siri"}, {"start": 1282.0, "text": "about what entity is associated with the UnicornChat notification,"}, {"start": 1286.0, "text": "I'll update the posting flow."}, {"start": 1288.0, "text": "After importing AppIntents I assign the persistent message EntityIdentifier"}, {"start": 1293.0, "text": "to the .appEntityIdentifiers property on UNMutableNotificationContent."}, {"start": 1298.0, "text": "Note, that with the three entity annotation APIs I'm describing,"}, {"start": 1302.0, "text": "you can't use TransientAppEntity."}, {"start": 1305.0, "text": "Transient entities are temporary model objects,"}, {"start": 1307.0, "text": "so they don't have persistent identifiers."}, {"start": 1310.0, "text": "To add entity annotations to NowPlaying in CosmoTunes,"}, {"start": 1313.0, "text": "I followed the same pattern."}, {"start": 1315.0, "text": "I'm already providing song attributes using MusicContent"}, {"start": 1317.0, "text": "in the app's MediaSessionRepresentable conformance."}, {"start": 1321.0, "text": "To enhance this state, I'll take the existing song,"}, {"start": 1324.0, "text": "artist, and playlist entities, and add them to the appEntityIdentifiers property"}, {"start": 1330.0, "text": "in order of most specific to least specific."}, {"start": 1334.0, "text": "This enables contextual requests like: \"Play the live version.\""}, {"start": 1339.0, "text": "With AlarmKit, I add a single EntityIdentifier"}, {"start": 1341.0, "text": "to the appEntityIdentifier parameter on AlarmConfiguration"}, {"start": 1345.0, "text": "when creating an alarm or a timer."}, {"start": 1348.0, "text": "With this, people can act on firing alarms and timers."}, {"start": 1352.0, "text": "That's all it takes"}, {"start": 1353.0, "text": "to connect your entities to notifications, Now Playing, and alarms."}, {"start": 1359.0, "text": "We've covered several advanced ways to make your app work even better with Siri."}, {"start": 1363.0, "text": "As you think about next steps,"}, {"start": 1365.0, "text": "a great place to start is by customizing your entity display representations."}, {"start": 1370.0, "text": "They are used to display your entities across the system."}, {"start": 1373.0, "text": "From there, add your entities to the semantic index,"}, {"start": 1376.0, "text": "and keep the index up to date, so Siri can always find your freshest content."}, {"start": 1382.0, "text": "You might also consider making your entities accessible through Siri"}, {"start": 1384.0, "text": "with an IntentValueQuery and in-app search."}, {"start": 1388.0, "text": "And annotating your views, activities,"}, {"start": 1390.0, "text": "and your existing system integrations with entities"}, {"start": 1393.0, "text": "can give Apple Intelligence even more context to work with."}, {"start": 1397.0, "text": "When you're ready, look into donating UI interactions to help Apple Intelligence"}, {"start": 1401.0, "text": "understand how people use your app, making for a more personalized experience."}, {"start": 1407.0, "text": "To see any of these concepts applied, take a look at the sample projects."}, {"start": 1412.0, "text": "For a hands-on look at adopting App Schemas,"}, {"start": 1415.0, "text": "check out the \"Code-Along: Make your app available to Siri.\""}, {"start": 1419.0, "text": "With the 27 releases, Apple Intelligence is transforming what Siri can do,"}, {"start": 1424.0, "text": "and App Intents puts that transformative power directly in your hands."}, {"start": 1429.0, "text": "You now have everything you need to build delightful, elevated experiences"}, {"start": 1434.0, "text": "that feel like a natural extension of the system."}, {"start": 1436.0, "text": "I can't wait to see what you create."}, {"start": 1439.0, "text": "Until next time."}]}