{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-243", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/243", "wordCount": 2101, "segments": [{"start": 7.0, "text": "Hi, I'm Erik an AI Tools Engineer."}, {"start": 10.0, "text": "In this session, I'll show you how to use Instruments to debug"}, {"start": 13.0, "text": "and develop features built with the Foundation Models framework."}, {"start": 17.0, "text": "The Foundation Models APIs give your app direct access"}, {"start": 21.0, "text": "to on-device and server-based generative AI."}, {"start": 24.0, "text": "With them, you can build features that understand natural language,"}, {"start": 28.0, "text": "generate content, and respond to what the person is doing."}, {"start": 32.0, "text": "The features that create the best experiences aren't static."}, {"start": 36.0, "text": "They adapt based on context."}, {"start": 38.0, "text": "That's what the Foundation Models APIs are designed for."}, {"start": 41.0, "text": "DynamicInstructions lets you specify exactly which instructions"}, {"start": 45.0, "text": "and tools the model can access."}, {"start": 48.0, "text": "It re-evaluates before every request,"}, {"start": 51.0, "text": "so the model always has the right context for the task at hand."}, {"start": 54.0, "text": "That flexibility is what makes these features so responsive,"}, {"start": 58.0, "text": "and also what makes them harder to debug."}, {"start": 61.0, "text": "Building with Large Language Models or LLMs"}, {"start": 64.0, "text": "is different from traditional development."}, {"start": 66.0, "text": "Traditional code is predictable."}, {"start": 68.0, "text": "LLMs are non-deterministic -"}, {"start": 70.0, "text": "the same input can produce different outputs."}, {"start": 73.0, "text": "When a feature loses context or responds too slowly,"}, {"start": 76.0, "text": "tracking down the cause isn't straightforward."}, {"start": 78.0, "text": "Good tooling makes the difference."}, {"start": 81.0, "text": "By the end of this session,"}, {"start": 82.0, "text": "you'll know how to use Instruments to identify"}, {"start": 85.0, "text": "and fix those issues"}, {"start": 86.0, "text": "and ship fast, reliable experiences with confidence."}, {"start": 91.0, "text": "First, we'll start by comparing and contrasting"}, {"start": 93.0, "text": "traditional versus LLM app development concepts"}, {"start": 96.0, "text": "to get us into the right mindset."}, {"start": 99.0, "text": "Then, we'll use Instruments to inspect and debug an agentic experience"}, {"start": 102.0, "text": "I'm developing in my Craft app."}, {"start": 105.0, "text": "Before getting started,"}, {"start": 106.0, "text": "we recommend you check out \"What's new in the Foundation Models framework\""}, {"start": 110.0, "text": "and \"Build agentic app experiences with the Foundation Models framework\""}, {"start": 114.0, "text": "to gain a better understanding of the latest additions."}, {"start": 118.0, "text": "Building apps with LLMs introduces three challenges"}, {"start": 120.0, "text": "you won't find in traditional software development."}, {"start": 124.0, "text": "The first is probabilistic output."}, {"start": 127.0, "text": "Give a traditional function the same input twice,"}, {"start": 129.0, "text": "and you get the same output."}, {"start": 131.0, "text": "LLMs don't work that way."}, {"start": 133.0, "text": "The same prompt can produce two completely different responses"}, {"start": 137.0, "text": "which means standard unit testing breaks down."}, {"start": 140.0, "text": "You can't assert that an output matches a hardcoded string."}, {"start": 144.0, "text": "You have to evaluate the quality and intent of the response instead."}, {"start": 148.0, "text": "The second is model-to-model communication."}, {"start": 151.0, "text": "Powerful features often rely on multiple models working together."}, {"start": 155.0, "text": "For example, in a recipe app,"}, {"start": 157.0, "text": "one model might identify ingredients in a photo,"}, {"start": 160.0, "text": "while a second generates a recipe from that result."}, {"start": 163.0, "text": "Getting data to flow reliably between those models,"}, {"start": 166.0, "text": "and recovering gracefully when something goes wrong,"}, {"start": 168.0, "text": "is where real complexity lives."}, {"start": 171.0, "text": "And the third is observability."}, {"start": 174.0, "text": "When something breaks in a multi-model pipeline,"}, {"start": 176.0, "text": "it can be very hard to know where it went wrong."}, {"start": 179.0, "text": "You need visibility into each step:"}, {"start": 181.0, "text": "what the model received,"}, {"start": 183.0, "text": "what it decided, and why."}, {"start": 185.0, "text": "That's exactly what this session is about."}, {"start": 188.0, "text": "At its core, an LLM application does three things:"}, {"start": 191.0, "text": "a person sends a prompt,"}, {"start": 193.0, "text": "the model reasons about it,"}, {"start": 194.0, "text": "and the person gets a response."}, {"start": 196.0, "text": "Simple, fast, and for many features"}, {"start": 199.0, "text": "(a summarization tool,"}, {"start": 200.0, "text": "a writing assistant,"}, {"start": 201.0, "text": "a Q&A interface),"}, {"start": 202.0, "text": "exactly what you need."}, {"start": 204.0, "text": "Many useful features need more than text generation."}, {"start": 207.0, "text": "Sometimes the model needs information it doesn't have:"}, {"start": 210.0, "text": "the current time,"}, {"start": 211.0, "text": "a database record,"}, {"start": 212.0, "text": "or a search result."}, {"start": 213.0, "text": "That's where tool calls come in."}, {"start": 215.0, "text": "The loop works like this:"}, {"start": 217.0, "text": "the person sends a prompt,"}, {"start": 218.0, "text": "the model reasons about it and calls a tool,"}, {"start": 221.0, "text": "that tool performs an action,"}, {"start": 223.0, "text": "the model takes the result"}, {"start": 224.0, "text": "and generates a final response,"}, {"start": 226.0, "text": "which can kick off the loop again."}, {"start": 229.0, "text": "Each extra step adds latency."}, {"start": 231.0, "text": "Each step is a new place for failure."}, {"start": 233.0, "text": "Understanding this loop is the basis for everything"}, {"start": 236.0, "text": "the Foundation Models Instrument shows you."}, {"start": 239.0, "text": "Now that I have covered the mindset required for LLM app development,"}, {"start": 242.0, "text": "I'll use Instruments to debug and inspect the brainstorming feature"}, {"start": 245.0, "text": "I'm developing for my Craft app."}, {"start": 248.0, "text": "I'm working on a crafting companion app"}, {"start": 250.0, "text": "where you can keep a journal of your craft projects."}, {"start": 254.0, "text": "The app lets you record craft progress,"}, {"start": 256.0, "text": "ask questions about specific crafts,"}, {"start": 258.0, "text": "and generate tutorials."}, {"start": 260.0, "text": "Recently, I had an idea for an interactive brainstorming feature"}, {"start": 264.0, "text": "that gives people suggestions on what to craft."}, {"start": 267.0, "text": "The crafter can speak with the model to refine its ideas"}, {"start": 270.0, "text": "and when they're ready to commit,"}, {"start": 271.0, "text": "the app generates a detailed tutorial for that craft."}, {"start": 275.0, "text": "This feature uses two sets of instructions:"}, {"start": 278.0, "text": "one for brainstorming ideas,"}, {"start": 279.0, "text": "and a second for tutorial generation."}, {"start": 282.0, "text": "The brainstorming instructions include two tools:"}, {"start": 284.0, "text": "a GenerateCraftIdeaTool"}, {"start": 286.0, "text": "and a SwitchToTutorialModeTool."}, {"start": 288.0, "text": "Both sets of instructions use the server model on Private Cloud Compute,"}, {"start": 292.0, "text": "one for quick idea generation"}, {"start": 294.0, "text": "and the other to generate more detailed tutorials."}, {"start": 297.0, "text": "Let's see this in action with Instruments."}, {"start": 302.0, "text": "The project is already open in Xcode."}, {"start": 304.0, "text": "To begin profiling, I'll open the Product menu and select Profile."}, {"start": 308.0, "text": "Xcode will build the app locally."}, {"start": 311.0, "text": "From the template chooser,"}, {"start": 312.0, "text": "I'll select the Foundation Models template and click Record."}, {"start": 316.0, "text": "This instrument captures prompt and response data from your device,"}, {"start": 320.0, "text": "which can include sensitive information."}, {"start": 322.0, "text": "Logging is off in production"}, {"start": 324.0, "text": "but it's on for the duration of your trace"}, {"start": 326.0, "text": "so keep your trace files somewhere safe."}, {"start": 329.0, "text": "Select \"Record Anyway\" to get started."}, {"start": 332.0, "text": "Now that the app has launched, let's give it a try."}, {"start": 335.0, "text": "As soon as we land here,"}, {"start": 337.0, "text": "the model suggests a few project ideas:"}, {"start": 339.0, "text": "Yarn PomPom,"}, {"start": 341.0, "text": "Fabric Pouch,"}, {"start": 342.0, "text": "and Paper Butterfly."}, {"start": 343.0, "text": "Paper Butterfly sounds fun - let's go with that."}, {"start": 352.0, "text": "Hm."}, {"start": 353.0, "text": "That's not right."}, {"start": 354.0, "text": "The model was supposed to kick off a tutorial"}, {"start": 357.0, "text": "but instead it just offered more ideas."}, {"start": 359.0, "text": "Something's off."}, {"start": 360.0, "text": "Let's end the recording and dig into the trace to find out what happened."}, {"start": 364.0, "text": "Instruments shows a lot at once,"}, {"start": 366.0, "text": "so let's walk through it together."}, {"start": 368.0, "text": "The top section holds the tracks."}, {"start": 371.0, "text": "Tracks show activity on the timeline,"}, {"start": 373.0, "text": "and each track can contain multiple lanes"}, {"start": 376.0, "text": "with charts that show levels or regions."}, {"start": 379.0, "text": "Below the timeline is the detail view."}, {"start": 382.0, "text": "It shows summary information about the range you're currently inspecting."}, {"start": 386.0, "text": "If you click a bar in the timeline or a row in the detail view,"}, {"start": 389.0, "text": "the inspector opens up on the right"}, {"start": 392.0, "text": "giving you a closer look at what you've selected."}, {"start": 396.0, "text": "The Foundation Models Instrument has 6 lanes in the timeline."}, {"start": 400.0, "text": "These give you a quick overview of session structure and latencies."}, {"start": 403.0, "text": "Alongside the timeline,"}, {"start": 405.0, "text": "there's a tree detail view."}, {"start": 407.0, "text": "That's where you can really dig into the model's chain of thought."}, {"start": 411.0, "text": "The Instructions lane"}, {"start": 412.0, "text": "shows how long a given set of instructions and tools was active."}, {"start": 416.0, "text": "One set can cover multiple requests."}, {"start": 419.0, "text": "Looking at this lane,"}, {"start": 420.0, "text": "it's clear only one set of instructions was active for the entire session"}, {"start": 425.0, "text": "but the feature was supposed to use two,"}, {"start": 427.0, "text": "so something went wrong during the handoff."}, {"start": 431.0, "text": "The Model Inference lane has two types of bars:"}, {"start": 434.0, "text": "yellow and orange."}, {"start": 436.0, "text": "Yellow bars represent how long the system spent"}, {"start": 439.0, "text": "processing the input prompt."}, {"start": 442.0, "text": "Orange bars represent how long it took to generate the response."}, {"start": 449.0, "text": "The timeline gives you a quick overview"}, {"start": 451.0, "text": "but the real power is in the tree view."}, {"start": 453.0, "text": "It takes everything logged during this recording"}, {"start": 455.0, "text": "and organizes it into a hierarchy:"}, {"start": 457.0, "text": "sessions, requests, model inferences, instructions, prompts, and responses."}, {"start": 462.0, "text": "Let's use it to track down why the instruction set never changed."}, {"start": 468.0, "text": "Session 1 had two requests."}, {"start": 470.0, "text": "The first one was kicked off by the prompt starting with"}, {"start": 472.0, "text": "\"Please generate 3 craft ideas.\""}, {"start": 478.0, "text": "That request was made up of two model inferences and a few tool calls."}, {"start": 482.0, "text": "Every model inference should have instructions,"}, {"start": 484.0, "text": "a prompt,"}, {"start": 485.0, "text": "and either a response or an error."}, {"start": 487.0, "text": "Click any node in the tree to pull it up in the inspector."}, {"start": 495.0, "text": "The model inference detail shows a summary of the instructions,"}, {"start": 499.0, "text": "prompt, and response that made up this call."}, {"start": 504.0, "text": "Scroll down and you'll find duration visualizations and token usage metrics."}, {"start": 508.0, "text": "We'll come back to those later"}, {"start": 510.0, "text": "when we talk about optimizing for reliability and performance."}, {"start": 516.0, "text": "Getting back to the failure,"}, {"start": 518.0, "text": "the timeline already told us the instruction set never changed,"}, {"start": 521.0, "text": "and here in the inspector for this model inference node,"}, {"start": 524.0, "text": "I can see the prompt tied to those instructions."}, {"start": 526.0, "text": "Let's select the Instructions node to see how they're set up."}, {"start": 534.0, "text": "The inspector shows that this instruction only had one tool associated with it."}, {"start": 539.0, "text": "The prompt references the switchToTutorialMode tool"}, {"start": 545.0, "text": "but that tool isn't actually configured with this instruction."}, {"start": 549.0, "text": "Without it, the app has no way to switch from brainstorm mode to tutorial mode,"}, {"start": 554.0, "text": "so the crafter gets stuck in a loop."}, {"start": 560.0, "text": "Looking at the subsequent nodes in the tree,"}, {"start": 562.0, "text": "this was a silent failure."}, {"start": 564.0, "text": "The model kept accepting input"}, {"start": 565.0, "text": "and making tool calls but never threw an error."}, {"start": 569.0, "text": "There was no clear signal that anything had gone wrong."}, {"start": 572.0, "text": "That makes it a hard bug to catch."}, {"start": 574.0, "text": "Now that the root cause is clear,"}, {"start": 576.0, "text": "I'll jump into Xcode to fix it."}, {"start": 578.0, "text": "Based on what I found in Instruments,"}, {"start": 580.0, "text": "I'll look at the BrainstormDynamicInstructions definition."}, {"start": 583.0, "text": "In the Instructions block,"}, {"start": 584.0, "text": "the SwitchToTutorialMode tool is mentioned in the prompt"}, {"start": 587.0, "text": "but only the GenerateCraftIdeasTool is listed in the toolset,"}, {"start": 594.0, "text": "so let's add it."}, {"start": 600.0, "text": "Now, I'll recompile and re-run with Instruments"}, {"start": 603.0, "text": "to make sure the fix actually worked."}, {"start": 608.0, "text": "Back in the app, I'll head to the Ideas tab,"}, {"start": 610.0, "text": "and just like before, the model suggests some new crafts."}, {"start": 614.0, "text": "I'll go with..."}, {"start": 616.0, "text": "necklace."}, {"start": 622.0, "text": "And there it is."}, {"start": 623.0, "text": "The UI has switched to tutorial mode."}, {"start": 625.0, "text": "The model made the transition"}, {"start": 627.0, "text": "and generated a full tutorial for this craft."}, {"start": 630.0, "text": "Now let's jump back into Instruments and take a look at"}, {"start": 633.0, "text": "this new recording to make sure everything ran efficiently."}, {"start": 637.0, "text": "The Instructions lane now shows two distinct instructions"}, {"start": 641.0, "text": "active during this experience."}, {"start": 646.0, "text": "The first is a brainstorming instruction"}, {"start": 649.0, "text": "and the second is a tutorial generation instruction."}, {"start": 654.0, "text": "That lines up exactly with the brainstorm experience design we covered earlier."}, {"start": 659.0, "text": "Let's dig into the tree view to see how that transition actually happened."}, {"start": 665.0, "text": "The first set of instructions now includes both the generateCraftIdea"}, {"start": 669.0, "text": "and switchToTutorialMode tools."}, {"start": 672.0, "text": "That confirms the model had everything it needed to make the switch. The fix worked."}, {"start": 677.0, "text": "The instruction change happened after the second model inference of Request 2."}, {"start": 683.0, "text": "That inference resulted in a tool call to switchToTutorialMode,"}, {"start": 687.0, "text": "passing the selected craft as an argument."}, {"start": 691.0, "text": "And in the following request, the instructions correctly switched over"}, {"start": 694.0, "text": "to the tutorial generator,"}, {"start": 696.0, "text": "with the selected craft passed along as context."}, {"start": 700.0, "text": "The info column is a great way to quickly flag nodes worth a closer look:"}, {"start": 704.0, "text": "things like errors, long durations, and large token counts."}, {"start": 708.0, "text": "Request 1's first model inference took a bit longer than I was expecting,"}, {"start": 713.0, "text": "so let's take a look."}, {"start": 717.0, "text": "The metrics and duration sections break down token usage for this inference."}, {"start": 721.0, "text": "These numbers are your starting point for understanding"}, {"start": 723.0, "text": "and improving the efficiency of an experience."}, {"start": 727.0, "text": "You can measure performance using three key metrics."}, {"start": 730.0, "text": "Time to First Token measures how long it takes for the model"}, {"start": 733.0, "text": "to begin generating a response after receiving a prompt."}, {"start": 737.0, "text": "A high Time to First Token means people are staring at a blank screen."}, {"start": 741.0, "text": "To reduce it, shorten your prompt."}, {"start": 744.0, "text": "Tokens per Second measures overall generation speed of the response."}, {"start": 748.0, "text": "Use it to benchmark performance across different prompt configurations"}, {"start": 751.0, "text": "and catch regressions after changes."}, {"start": 755.0, "text": "Total Latency is the complete time from sending the request"}, {"start": 759.0, "text": "to receiving the final response."}, {"start": 761.0, "text": "This is the number people feel most directly."}, {"start": 764.0, "text": "To reduce perceived Total Latency,"}, {"start": 766.0, "text": "utilize streaming to surface partial results sooner."}, {"start": 771.0, "text": "Running a trace is where optimization starts."}, {"start": 774.0, "text": "These metrics tell you exactly where time and resources are going"}, {"start": 777.0, "text": "and point you toward the right fix."}, {"start": 780.0, "text": "Use the model inference node to get a clear picture of your token usage."}, {"start": 784.0, "text": "In this session, I showed you how to use Instruments"}, {"start": 787.0, "text": "to debug an agentic experience developed with the Foundation Models framework."}, {"start": 791.0, "text": "Once you've ironed out the bugs, the next thing to explore is evaluation."}, {"start": 795.0, "text": "Watch \"Meet the Evaluations framework\""}, {"start": 797.0, "text": "to see how you can measure and improve the quality of your prompts"}, {"start": 800.0, "text": "by using structured evaluation."}, {"start": 803.0, "text": "To get started with the improved Foundation Models Instrument,"}, {"start": 806.0, "text": "install Xcode 27."}, {"start": 808.0, "text": "Then, on the device you'd like to run and profile your app on,"}, {"start": 811.0, "text": "update to the latest OS releases."}, {"start": 814.0, "text": "Its important to note that this Instrument supports using any model you use"}, {"start": 818.0, "text": "with the Foundation Models framework."}, {"start": 821.0, "text": "The Foundation Models APIs are your starting point."}, {"start": 824.0, "text": "Experiment, build, and see what's possible."}, {"start": 826.0, "text": "When something isn't working as expected,"}, {"start": 828.0, "text": "the Foundation Models Instrument is there to help you debug,"}, {"start": 831.0, "text": "giving you direct visibility into framework behavior right in context."}, {"start": 836.0, "text": "Go further with related sessions on agentic app experiences"}, {"start": 839.0, "text": "and the Evaluations framework"}, {"start": 841.0, "text": "and explore the full documentation to unlock everything the framework can do."}, {"start": 845.0, "text": "Thank you for joining us!"}, {"start": 846.0, "text": "We're excited to see you develop and debug your intelligent experiences"}, {"start": 849.0, "text": "using the improved Foundation Models Instrument."}]}