{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-246", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/246", "wordCount": 2830, "segments": [{"start": 7.0, "text": "Hi, I'm Jennifer,"}, {"start": 8.0, "text": "from the Spotlight engineering team."}, {"start": 10.0, "text": "This year, we're taking search to a whole new level,"}, {"start": 13.0, "text": "with Foundation Models and Core Spotlight."}, {"start": 15.0, "text": "You can build rich, conversational experiences in your app,"}, {"start": 19.0, "text": "simply by making your app content available"}, {"start": 21.0, "text": "to a large language model for reasoning and response generation."}, {"start": 25.0, "text": "Now I'm from California, and there are so many beautiful hikes in the area."}, {"start": 30.0, "text": "I've been slowly making my way through some of the nicest trails,"}, {"start": 33.0, "text": "so I thought I'd build an app to help me along."}, {"start": 36.0, "text": "In my hiking trails app, I can already browse through state parks and trails."}, {"start": 40.0, "text": "And once I've completed a trail,"}, {"start": 42.0, "text": "I like to write my own notes on what I enjoyed most about the hike."}, {"start": 46.0, "text": "But it would be really great to be able to ask a language model"}, {"start": 49.0, "text": "all about the hikes I've already gone on,"}, {"start": 51.0, "text": "or even about new hikes I should try."}, {"start": 54.0, "text": "Well, the Foundation Models framework makes it easy to get started."}, {"start": 57.0, "text": "By introducing a language model session into the app,"}, {"start": 60.0, "text": "I can ask broad questions,"}, {"start": 62.0, "text": "and the model will answer just by drawing on its own knowledge of the world."}, {"start": 66.0, "text": "Now, I really only want answers about hikes that my app knows about."}, {"start": 70.0, "text": "And this is where Spotlight can help."}, {"start": 72.0, "text": "The hiking trails app has indexed all these great hikes"}, {"start": 75.0, "text": "into a Core Spotlight search index."}, {"start": 77.0, "text": "So to help the model answer questions about those particular hikes,"}, {"start": 81.0, "text": "we can use the app's Core Spotlight search index,"}, {"start": 83.0, "text": "through tool-calling from the Foundation Models framework."}, {"start": 87.0, "text": "The Tool protocol from Foundation Models is a powerful concept"}, {"start": 90.0, "text": "that can be used to extend a model's capabilities,"}, {"start": 93.0, "text": "both by taking actions for a request,"}, {"start": 95.0, "text": "or by looking up context that a model needs to generate a response."}, {"start": 100.0, "text": "A tool works by declaring its arguments and output,"}, {"start": 103.0, "text": "along with some instructions on what the tool does."}, {"start": 106.0, "text": "And then, when the model decides it needs to use a tool,"}, {"start": 109.0, "text": "it will simply generate the arguments to call up that tool,"}, {"start": 112.0, "text": "and use that output for response generation."}, {"start": 116.0, "text": "If you haven't already, there are some great sessions,"}, {"start": 118.0, "text": "such as the \"Deep dive into the Foundation Models framework\","}, {"start": 121.0, "text": "to learn more about how tool-calling works."}, {"start": 124.0, "text": "So what if we had a tool that lets a model generate a search"}, {"start": 127.0, "text": "on an app's Core Spotlight index?"}, {"start": 129.0, "text": "Well, today, we're introducing SpotlightSearchTool."}, {"start": 132.0, "text": "It's a tool that adopts the tool protocol,"}, {"start": 134.0, "text": "to let a language model directly search your app's content in Core Spotlight"}, {"start": 138.0, "text": "for contextual response generation."}, {"start": 141.0, "text": "SpotlightSearchTool is available on iOS, iPadOS, macOS, and visionOS."}, {"start": 147.0, "text": "Before we get started,"}, {"start": 148.0, "text": "you'll want to make sure your app donates searchable content with Core Spotlight."}, {"start": 152.0, "text": "Take a look at our past session"}, {"start": 153.0, "text": "on \"Supporting semantic search with Core Spotlight\","}, {"start": 156.0, "text": "where we talk through how to donate searchable content to Spotlight,"}, {"start": 160.0, "text": "how to manage donations with a delegate and reindex extension,"}, {"start": 163.0, "text": "and how to perform structured search over item attributes,"}, {"start": 166.0, "text": "and search against the semantic index."}, {"start": 169.0, "text": "Once your app has donated searchable items to Core Spotlight,"}, {"start": 172.0, "text": "or indexed entities for Apple Intelligence,"}, {"start": 175.0, "text": "we're ready to begin."}, {"start": 177.0, "text": "We have a lot to cover in this video!"}, {"start": 179.0, "text": "We'll show you how to provide the new SpotlightSearchTool"}, {"start": 183.0, "text": "to your language model session."}, {"start": 185.0, "text": "Then we'll explore how to customize SpotlightSearchTool with guidance,"}, {"start": 189.0, "text": "knowledge providers, and specialized capabilities ."}, {"start": 192.0, "text": "And finally, we'll look at ways to evaluate model responses,"}, {"start": 196.0, "text": "with the evaluations framework."}, {"start": 197.0, "text": "Alright let's get started."}, {"start": 200.0, "text": "First, let's see how SpotlightSearchTool"}, {"start": 202.0, "text": "can be used for contextual response generation."}, {"start": 205.0, "text": "In our hiking trails app, we've donated searchable items,"}, {"start": 208.0, "text": "to the Spotlight index that represent hiking trails."}, {"start": 212.0, "text": "Each trail has metadata like the trail's name and location."}, {"start": 216.0, "text": "And on some of the hikes, there's also some personal details"}, {"start": 218.0, "text": "such as the date when a hike was completed,"}, {"start": 221.0, "text": "and some notes I wrote about how those hikes went."}, {"start": 224.0, "text": "If I wanted to ask: What hikes have I gone on?,"}, {"start": 227.0, "text": "the model will need to search for items by attributes, like completion date"}, {"start": 230.0, "text": "and location, to be able to formulate its response."}, {"start": 234.0, "text": "So let's build this functionality into our app."}, {"start": 236.0, "text": "There's three things we'll look at when adopting SpotlightSearchTool."}, {"start": 240.0, "text": "We'll need to configure the tool,"}, {"start": 241.0, "text": "for the kind of search we want the model to perform."}, {"start": 244.0, "text": "Then we'll want to add additional context to the model,"}, {"start": 247.0, "text": "while the search is active, to get the best response."}, {"start": 250.0, "text": "And finally,"}, {"start": 251.0, "text": "we'll explore different ways to display results in our app's user interface."}, {"start": 256.0, "text": "Configuring the tool is not too different from performing a Spotlight query directly."}, {"start": 260.0, "text": "We'll start by importing both CoreSpotlight and FoundationModels."}, {"start": 264.0, "text": "Then, in one line of code, the tool is ready to search your app's Core Spotlight index."}, {"start": 269.0, "text": "You can also provide SpotlightSearchTool with a custom configuration."}, {"start": 273.0, "text": "Here we're specifying a FileSource"}, {"start": 274.0, "text": "to perform a search against file paths in your app's sandbox."}, {"start": 278.0, "text": "Next you'll want to choose the right model for your app,"}, {"start": 281.0, "text": "whether it's the SystemLanguageModel or a model of your choosing,"}, {"start": 285.0, "text": "which you can do using the new Model Provider APIs."}, {"start": 288.0, "text": "Once you've chosen the model,"}, {"start": 290.0, "text": "add the new SpotlightSearchTool instance to your LanguageModelSession"}, {"start": 293.0, "text": "to start getting a response."}, {"start": 296.0, "text": "It feels like magic,"}, {"start": 297.0, "text": "but the response follows a path of tool calling and generation."}, {"start": 301.0, "text": "For a question like: What hikes have I gone on?,"}, {"start": 304.0, "text": "the trajectory might start with the model deciding it needs to use SpotlightSearchTool"}, {"start": 309.0, "text": "the model will invoke the tool with a generated query"}, {"start": 312.0, "text": "Spotlight will execute that query"}, {"start": 314.0, "text": "and return a description of the result set back,"}, {"start": 317.0, "text": "and the model will reason over that output"}, {"start": 319.0, "text": "and generate its final response."}, {"start": 323.0, "text": "Now when I ask: What hikes have I gone on?,"}, {"start": 326.0, "text": "the model can generate an answer grounded in the app's content."}, {"start": 331.0, "text": "You might notice from some responses,"}, {"start": 333.0, "text": "that the model was not able to see all of the metadata,"}, {"start": 335.0, "text": "that was donated for the items."}, {"start": 338.0, "text": "That's because some metadata in the Spotlight index,"}, {"start": 340.0, "text": "like text content and HTML,"}, {"start": 342.0, "text": "is stored in a highly-compact representation that can be searched,"}, {"start": 346.0, "text": "but not recovered in a way that a language model can read it."}, {"start": 349.0, "text": "For these cases, you'll want to consider providing additional metadata for an item,"}, {"start": 354.0, "text": "while SpotlightSearchTool is performing a search."}, {"start": 357.0, "text": "If your app donates searchable content to Core Spotlight,"}, {"start": 360.0, "text": "you'll already be familiar with the index delegate protocol."}, {"start": 363.0, "text": "Your app would set an index delegate on your CSSearchableIndex"}, {"start": 367.0, "text": "to handle reindex requests,"}, {"start": 369.0, "text": "such as when Spotlight needs to perform migration or recovery."}, {"start": 372.0, "text": "For SpotlightSearchTool,"}, {"start": 374.0, "text": "we've added a method to the delegate to recover the full CSSearchableItem"}, {"start": 377.0, "text": "by its unique identifier."}, {"start": 379.0, "text": "This allows the model"}, {"start": 380.0, "text": "to efficiently manage responses over potentially millions of results."}, {"start": 384.0, "text": "On your index delegate,"}, {"start": 385.0, "text": "simply adopt the new searchableItems (forIdentifiers:)"}, {"start": 388.0, "text": "to return the complete CSSearchableItem."}, {"start": 391.0, "text": "If your app has metadata that doesn't make sense to donate for search,"}, {"start": 395.0, "text": "but might be useful for the model to reason about,"}, {"start": 397.0, "text": "this is the right time to set any additional attributes on an item"}, {"start": 400.0, "text": "for the model to see."}, {"start": 402.0, "text": "Now that we've configured the tool to perform searches,"}, {"start": 405.0, "text": "we'll want to think about how to display results"}, {"start": 407.0, "text": "and responses in our user interface."}, {"start": 409.0, "text": "The session response is a concise description over the result set."}, {"start": 413.0, "text": "And in an assistant-style interface,"}, {"start": 415.0, "text": "this response is typically what an app would want to display."}, {"start": 419.0, "text": "But search results are also available directly on SpotlightSearchTool itself."}, {"start": 423.0, "text": "For a list-style display, this is the best way to access searchable items,"}, {"start": 427.0, "text": "especially when the result set is large."}, {"start": 430.0, "text": "Search replies pass back results in batches during the search,"}, {"start": 433.0, "text": "so query tokens can be used to manage the conversation stream,"}, {"start": 437.0, "text": "ensuring that user interface stays up-to-date with the model."}, {"start": 440.0, "text": "To access results from the SpotlightSearchTool,"}, {"start": 443.0, "text": "your app can wait for search replies"}, {"start": 445.0, "text": "and check for CSSearchableItem in the content of the reply."}, {"start": 448.0, "text": "Search replies come as an async sequence of events,"}, {"start": 451.0, "text": "where each reply may include a batch of results, until the tool call completes."}, {"start": 456.0, "text": "Keep in mind that for any given response,"}, {"start": 458.0, "text": "the model may call SpotlightSearchTool more than once,"}, {"start": 460.0, "text": "before generating its final response."}, {"start": 462.0, "text": "For that reason, use the queryToken on each reply,"}, {"start": 465.0, "text": "to determine when the user interface should refresh."}, {"start": 469.0, "text": "SpotlightSearchTool provides a host of search capabilities,"}, {"start": 472.0, "text": "from semantic search over text, to structured search over metadata,"}, {"start": 475.0, "text": "like dates, persons, locations and more."}, {"start": 478.0, "text": "But depending on the language model you choose,"}, {"start": 480.0, "text": "you may want to customize SpotlightSearchTool"}, {"start": 483.0, "text": "both for the model, and your app content."}, {"start": 485.0, "text": "There's a few ways to customize SpotlightSearchTool."}, {"start": 488.0, "text": "Guidance profiles can be used to scope the tool's search capabilities."}, {"start": 493.0, "text": "Providing the tool with world knowledge can help with reference resolution."}, {"start": 496.0, "text": "And implementing custom pipeline stages,"}, {"start": 499.0, "text": "can improve model reasoning over your app's content."}, {"start": 502.0, "text": "SpotlightSearchTool provides its entire set of search capabilities"}, {"start": 505.0, "text": "to a model for guided generation."}, {"start": 508.0, "text": "But guidance profiles can help scope that guidance to only what an app needs."}, {"start": 512.0, "text": "The hiking trails app doesn't donate person relationships,"}, {"start": 515.0, "text": "so guiding the model on how to search for authors and recipients,"}, {"start": 518.0, "text": "could be skipped for limited-context models."}, {"start": 521.0, "text": "To selectively enable guidance on search capabilities like people and dates,"}, {"start": 526.0, "text": "use a GuidanceProfile."}, {"start": 527.0, "text": "You can even specify the exact list of metadata attributes,"}, {"start": 530.0, "text": "that the model should consider during a search."}, {"start": 533.0, "text": "Then set a dynamic guide level using the profile,"}, {"start": 536.0, "text": "when creating SpotlightSearchTool."}, {"start": 538.0, "text": "On-device models have a more restricted model context size,"}, {"start": 541.0, "text": "so it's best to use focused guidance for simpler search capabilities."}, {"start": 546.0, "text": "Reference resolution is another way for your app"}, {"start": 548.0, "text": "to provide context that's not directly available in the search index."}, {"start": 552.0, "text": "As an example, if the hiking trails app did donate person relationships,"}, {"start": 556.0, "text": "the person using the app might want to ask about other participants on the trail."}, {"start": 560.0, "text": "In that case, the model needs to know who that person refers to in a prompt."}, {"start": 564.0, "text": "If the app already knows who that person is,"}, {"start": 566.0, "text": "use a contact resolver to help the tool filter to the right set of results."}, {"start": 571.0, "text": "A contactResolver should return"}, {"start": 572.0, "text": "any contact information related to the user's identity,"}, {"start": 576.0, "text": "that can be matched against metadata in the search index."}, {"start": 579.0, "text": "And at last, your app can take advantage of custom pipeline stages,"}, {"start": 583.0, "text": "that take document reasoning even further."}, {"start": 586.0, "text": "For really complex requests,"}, {"start": 588.0, "text": "the language model might forgo a simple search query,"}, {"start": 590.0, "text": "in favor of a pipeline search."}, {"start": 592.0, "text": "A pipeline search brings together queries to the index,"}, {"start": 595.0, "text": "plus computation over a result set, for maximal efficiency."}, {"start": 600.0, "text": "I could ask:"}, {"start": 601.0, "text": "how many trails have I hiked this year, and for each month,"}, {"start": 604.0, "text": "how many miles have I gone on average?"}, {"start": 607.0, "text": "Now, the model could perform a simple search"}, {"start": 609.0, "text": "and keep a tally in memory to answer the question."}, {"start": 611.0, "text": "Or, if the result set is likely to be large,"}, {"start": 614.0, "text": "SpotlightSearchTool allows the model to request"}, {"start": 617.0, "text": "that Spotlight run a pipeline of search and computation stages."}, {"start": 621.0, "text": "With a pipeline search,"}, {"start": 622.0, "text": "the model can break down this complex query into a set of steps."}, {"start": 626.0, "text": "The model might generate a search for completed hikes,"}, {"start": 630.0, "text": "along with a counting stage that builds a table by month,"}, {"start": 633.0, "text": "then a stage that computes an average over all counts."}, {"start": 636.0, "text": "Pipeline stages allow the tool to perform efficient computation, or transformation,"}, {"start": 641.0, "text": "over a search result set on behalf of the model."}, {"start": 644.0, "text": "And your app can participate by registering its own custom stages."}, {"start": 648.0, "text": "Pipeline stages are Generable,"}, {"start": 649.0, "text": "so the model will generate a stage on-demand based on the user's prompt."}, {"start": 654.0, "text": "And whenever a stage is generated,"}, {"start": 656.0, "text": "the model may choose to return data back to the app when it makes sense."}, {"start": 659.0, "text": "The Foundation Models deep dive has a great segment on Guided Generation"}, {"start": 663.0, "text": "and Generable types that I highly recommend."}, {"start": 666.0, "text": "Let's take a look at the hiking trails app again."}, {"start": 668.0, "text": "Some trails includes personal notes on how each hike went,"}, {"start": 671.0, "text": "so I might want to ask:"}, {"start": 673.0, "text": "I remember being really happy on some of my hikes. Which ones were they?"}, {"start": 677.0, "text": "On its own, the model could make its best guess at my happiness level,"}, {"start": 680.0, "text": "just by reading my notes."}, {"start": 683.0, "text": "Or, the app could register a custom stage,"}, {"start": 685.0, "text": "that computes a happiness score over each item,"}, {"start": 688.0, "text": "allowing the model to generate a response,"}, {"start": 689.0, "text": "solely on the computed top-scoring results."}, {"start": 693.0, "text": "To build a custom stage that computes a happiness score,"}, {"start": 696.0, "text": "we'll want to operate on CSSearchableItem as the input,"}, {"start": 699.0, "text": "and return a scored version as the output."}, {"start": 701.0, "text": "The score could be computed by running a sentiment analysis model"}, {"start": 704.0, "text": "over the notes attribute on the item,"}, {"start": 706.0, "text": "or by some other custom logic,"}, {"start": 708.0, "text": "perhaps taking into account hikes rated with 5 stars."}, {"start": 711.0, "text": "And since this is a Generable type,"}, {"start": 713.0, "text": "we can add properties with Guides"}, {"start": 715.0, "text": "to inform the model on which results to prefer."}, {"start": 718.0, "text": "Then we simply register the stage by adding it to the tool's configuration."}, {"start": 723.0, "text": "There's one more thing:"}, {"start": 724.0, "text": "remember how SpotlightSearchTool returns replies with search results for display?"}, {"start": 729.0, "text": "Well, the model may decide to send back"}, {"start": 731.0, "text": "a search reply with the output data of a pipeline stage,"}, {"start": 734.0, "text": "as another kind of partial result."}, {"start": 737.0, "text": "From aggregate counts and tables,"}, {"start": 739.0, "text": "to free-form text or computed numeric values,"}, {"start": 742.0, "text": "your app can display some or all of these data types."}, {"start": 745.0, "text": "And each reply comes with a handy LLM-generated label"}, {"start": 748.0, "text": "describing the content,"}, {"start": 750.0, "text": "giving your app the most flexibility for its user interface."}, {"start": 754.0, "text": "With so many options for customization,"}, {"start": 756.0, "text": "from the model we choose"}, {"start": 757.0, "text": "and the searchable content our app donates,"}, {"start": 759.0, "text": "to guidance levels and custom reasoning,"}, {"start": 762.0, "text": "how can we verify, in a broad way,"}, {"start": 764.0, "text": "how well the model is responding in our app?"}, {"start": 767.0, "text": "Well, the Evaluations framework can help us in a few important ways."}, {"start": 771.0, "text": "Not only can we quickly build evaluations"}, {"start": 773.0, "text": "to see how well the model is calling the tool,"}, {"start": 776.0, "text": "and how meaningful the response;"}, {"start": 778.0, "text": "we can also rapidly iterate on our app's searchable content"}, {"start": 781.0, "text": "paired with different guidance profiles on SpotlightSearchTool itself."}, {"start": 785.0, "text": "The Evaluations framework has some great APIs"}, {"start": 787.0, "text": "for building an end-to-end evaluation suite,"}, {"start": 790.0, "text": "from large-scale dataset generation,"}, {"start": 793.0, "text": "to evaluation runs using custom metrics, and reporting."}, {"start": 797.0, "text": "There are some great sessions that go in-depth on sample data generation APIs,"}, {"start": 801.0, "text": "and the video on creating robust evaluations for an agentic app"}, {"start": 805.0, "text": "is a great resource to get started"}, {"start": 807.0, "text": "on evaluating model responses with tool-calling."}, {"start": 811.0, "text": "For our purposes, we're going to focus on result coverage"}, {"start": 814.0, "text": "as a way to evaluate the hiking trails conversational experience."}, {"start": 818.0, "text": "We want to know, given a dataset that's indexed in Core Spotlight,"}, {"start": 822.0, "text": "how well does the model generate responses based on the items we expect it to find."}, {"start": 827.0, "text": "We'll start by defining a dataset that adopts the ModelSampleProtocol."}, {"start": 831.0, "text": "Our TrailRequest already includes the natural language input"}, {"start": 834.0, "text": "that a person might ask about trails in our app,"}, {"start": 837.0, "text": "the output is a language model response"}, {"start": 840.0, "text": "and an expectation of the trajectory of the request."}, {"start": 843.0, "text": "We'll also be adding a set of unique identifiers of searchable items"}, {"start": 846.0, "text": "that we expect the tool to return for that prompt."}, {"start": 850.0, "text": "If we have real data to test against, that's great;"}, {"start": 854.0, "text": "but if not, we can use Sample Generation APIs"}, {"start": 857.0, "text": "to generate data based on a prompt."}, {"start": 859.0, "text": "Let's take a look at this in Xcode."}, {"start": 862.0, "text": "For our evaluations, we can define a set of hiking trails"}, {"start": 865.0, "text": "with the metadata that our app is expected to donate to Core Spotlight."}, {"start": 870.0, "text": "Then we'll build a set of seed samples to use in our evaluations."}, {"start": 873.0, "text": "Samples can be serialized in any Codable format,"}, {"start": 876.0, "text": "and JSON works well for that purpose."}, {"start": 879.0, "text": "Our samples include the query and the set of item identifiers"}, {"start": 882.0, "text": "we expect to be returned for the search."}, {"start": 884.0, "text": "We can also provide a sample response that we can use later"}, {"start": 887.0, "text": "in a quality comparison with the model's actual response."}, {"start": 891.0, "text": "Using the Sample Generation APIs in a command line tool,"}, {"start": 894.0, "text": "I can expand this seed set to many more variations,"}, {"start": 898.0, "text": "to get broad coverage on how people might want to ask about trails."}, {"start": 902.0, "text": "The next step is to define our evaluation with metrics and trajectory."}, {"start": 906.0, "text": "For our samples, we expect the trajectory of a response"}, {"start": 909.0, "text": "to include a call to SpotlightSearchTool to perform a query,"}, {"start": 912.0, "text": "so here's how we might define that expectation."}, {"start": 916.0, "text": "And here's an overview of an evaluation flow"}, {"start": 919.0, "text": "that takes into account how many expected items were included in the final response."}, {"start": 924.0, "text": "In our test target,"}, {"start": 925.0, "text": "our evaluation will load the trail items and samples from our generated datasets."}, {"start": 930.0, "text": "Then, we'll donate the trail items to Core Spotlight,"}, {"start": 933.0, "text": "and configure SpotlightSearchTool for this evaluation."}, {"start": 936.0, "text": "Once the evaluation completes its run,"}, {"start": 939.0, "text": "we can set the expectation for any metric we've included,"}, {"start": 942.0, "text": "like result coverage."}, {"start": 945.0, "text": "This is just the start towards building comprehensive evaluations"}, {"start": 948.0, "text": "that will help you craft the best experience possible for your app."}, {"start": 952.0, "text": "It's a big year for Foundation Models, and we hope you'll make the most of it."}, {"start": 956.0, "text": "Download our sample code to see the hiking trails app in action."}, {"start": 959.0, "text": "Try adding your own custom functionality to the app to really see what's possible."}, {"start": 964.0, "text": "You might also want to add your own evaluation suite,"}, {"start": 966.0, "text": "with some inspiration from the evaluations agentic deep dive."}, {"start": 971.0, "text": "And remember, we're not writing search queries anymore."}, {"start": 974.0, "text": "We're providing the content, and letting intelligence do the rest."}]}