{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-299", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/299", "wordCount": 3385, "segments": [{"start": 8.0, "text": "Hi, I'm Ada!"}, {"start": 10.0, "text": "And I'm Kyle!"}, {"start": 11.0, "text": "And we're engineers on the Evaluations team!"}, {"start": 14.0, "text": "Today, we are so excited to walk you"}, {"start": 16.0, "text": "through some of the advanced features in the Evaluations framework!"}, {"start": 20.0, "text": "The Evaluations framework introduces a way"}, {"start": 22.0, "text": "to assess intelligence-powered features in Swift apps,"}, {"start": 24.0, "text": "track improvements over time, and ensure quality in production."}, {"start": 28.0, "text": "This framework is new in Xcode 27 and supports macOS, iOS, watchOS and visionOS."}, {"start": 35.0, "text": "If you haven't already, check out the \"Meet the Evaluations framework\" video"}, {"start": 39.0, "text": "to learn about the building blocks of the Evaluations framework"}, {"start": 42.0, "text": "and our other video \"Improve your prompts by hill climbing with Evaluations\""}, {"start": 46.0, "text": "to explore different strategies to improve your intelligence features."}, {"start": 51.0, "text": "In this video, we'll discuss how to address complexity"}, {"start": 54.0, "text": "and scalability with your evaluations."}, {"start": 57.0, "text": "We'll begin by exploring how to grow your evaluation dataset"}, {"start": 60.0, "text": "by generating and validating synthetic data."}, {"start": 63.0, "text": "And then we'll cover how to build robust evaluations for agentic workflows"}, {"start": 67.0, "text": "that incorporate a special kind of model behavior known as tool calling."}, {"start": 72.0, "text": "In the \"Meet the Evaluations framework\" video,"}, {"start": 74.0, "text": "we introduced the hill-climbing process."}, {"start": 77.0, "text": "This illustrates the process of how we build, test and ship intelligent features."}, {"start": 83.0, "text": "In this video, we will primarily focus on the Develop and Evaluate step."}, {"start": 88.0, "text": "In the Develop step, we often start with a handful of samples for our evaluations,"}, {"start": 94.0, "text": "but your feature is almost always more complex than your dataset can cover."}, {"start": 99.0, "text": "It takes time to build, it's harder to scale,"}, {"start": 102.0, "text": "and it rarely captures the variety you need"}, {"start": 104.0, "text": "to truly understand how your feature behaves in the real world."}, {"start": 108.0, "text": "The quality of your evaluation results is only as good as the data behind them."}, {"start": 112.0, "text": "And writing good evaluation data is hard."}, {"start": 115.0, "text": "That's where synthetic data comes in."}, {"start": 117.0, "text": "The Evaluations framework exposes APIs"}, {"start": 120.0, "text": "that let you define sample generation entirely in code,"}, {"start": 123.0, "text": "so you can build your own generation pipeline, run it from the command line,"}, {"start": 127.0, "text": "or plug it directly into your existing workflows."}, {"start": 130.0, "text": "It supports text-based data"}, {"start": 132.0, "text": "and leverages the generable macro to generate structured synthetic data."}, {"start": 136.0, "text": "My colleagues and I have been working on BookTracker"}, {"start": 138.0, "text": "which is a personal library app that uses intelligence-backed features"}, {"start": 142.0, "text": "to auto-tag books based on written reviews."}, {"start": 145.0, "text": "Let's examine how we define each book."}, {"start": 148.0, "text": "We have a class named Book that includes the title,"}, {"start": 152.0, "text": "author, review, tags, and rating."}, {"start": 155.0, "text": "We define other variables used to support the cover design."}, {"start": 159.0, "text": "We also define sampleBooks which is an array of 13 Book samples,"}, {"start": 164.0, "text": "Like this one here about Pride and Prejudice."}, {"start": 167.0, "text": "These 13 samples might feel like a reasonable starting point,"}, {"start": 170.0, "text": "but this small dataset only give us a narrow window"}, {"start": 173.0, "text": "into how our feature performs."}, {"start": 176.0, "text": "Our evaluation results could look great and still be completely misleading."}, {"start": 180.0, "text": "Think about the variety of possible data used"}, {"start": 183.0, "text": "to evaluate our tag generation feature!"}, {"start": 186.0, "text": "There are countless books."}, {"start": 188.0, "text": "Hundreds of genres."}, {"start": 191.0, "text": "And a wide variety of ways a user might review what they just read."}, {"start": 195.0, "text": "We're also talking about the real world"}, {"start": 196.0, "text": "where summaries can be vague or incomplete."}, {"start": 199.0, "text": "Thirteen samples can't capture all of that."}, {"start": 202.0, "text": "We need more coverage,"}, {"start": 203.0, "text": "and we need it without spending days writing examples by hand!"}, {"start": 207.0, "text": "So let's discuss how to expand our dataset to capture more of that variety."}, {"start": 211.0, "text": "We'll start simple."}, {"start": 212.0, "text": "The makeSamples API requires three components:"}, {"start": 215.0, "text": "a prompt, a dataset, and a target count,"}, {"start": 218.0, "text": "which is the number of samples"}, {"start": 220.0, "text": "you'd like to synthetically generate including the dataset you provide."}, {"start": 223.0, "text": "Here, I've defined a prompt"}, {"start": 225.0, "text": "that asks the model to suggest more diverse book review samples."}, {"start": 229.0, "text": "To write a well-defined prompt,"}, {"start": 231.0, "text": "consider what information the model may need in order to best understand the task"}, {"start": 235.0, "text": "and handle the range of inputs your users might provide."}, {"start": 239.0, "text": "For our dataset, I'm passing in our sampleBooks"}, {"start": 241.0, "text": "which includes our 13 initial samples."}, {"start": 245.0, "text": "Here we leverage the new ModelSamples API which includes the book's review"}, {"start": 248.0, "text": "as the prompt and the book's tags as the expected output."}, {"start": 253.0, "text": "And for the target count, I've set it to one hundred samples to start!"}, {"start": 257.0, "text": "Remember, the targetCount is the size of the full resulting dataset,"}, {"start": 261.0, "text": "including the samples we started with,"}, {"start": 263.0, "text": "so the model will actually generate 87 new ones."}, {"start": 266.0, "text": "Now you might be wondering how much data is enough?"}, {"start": 270.0, "text": "And the answer is, it depends."}, {"start": 272.0, "text": "For our BookTracker app, a target count of one hundred is just the starting point!"}, {"start": 277.0, "text": "Synthetic data generation is often an iterative process"}, {"start": 280.0, "text": "of defining an initial dataset,"}, {"start": 282.0, "text": "generating synthetic data, validating the samples,"}, {"start": 286.0, "text": "then, analyzing whether or not the data is representative enough"}, {"start": 289.0, "text": "and continuing this cycle until you are confident!"}, {"start": 292.0, "text": "So the right target count for your evaluation dataset"}, {"start": 295.0, "text": "depends entirely on your feature."}, {"start": 297.0, "text": "What it does, who uses it, and how many different ways people interact with it."}, {"start": 303.0, "text": "What matters far more than quantity is coverage!"}, {"start": 308.0, "text": "So instead of asking how many samples do I need?"}, {"start": 311.0, "text": "Ask yourself,"}, {"start": 312.0, "text": "have I covered the meaningful variety of ways this feature will actually be used?"}, {"start": 317.0, "text": "Now that I've defined the required variables,"}, {"start": 319.0, "text": "I can use the makeSamples method,"}, {"start": 321.0, "text": "which returns an async stream of newly generated samples."}, {"start": 325.0, "text": "As I iterate over it,"}, {"start": 327.0, "text": "each new sample gets appended to a variable called expandedDataset"}, {"start": 330.0, "text": "that I've initialized with the starting dataset."}, {"start": 333.0, "text": "By default, the framework uses the on device model for generation."}, {"start": 337.0, "text": "The on-device model is a great option in most cases,"}, {"start": 340.0, "text": "but you might want to bring your own model,"}, {"start": 342.0, "text": "or customize the instructions the model operates under."}, {"start": 345.0, "text": "The framework provides the flexibility"}, {"start": 347.0, "text": "to define your own configurations for sample generation."}, {"start": 350.0, "text": "Lets go over how to do that!"}, {"start": 353.0, "text": "For more complex configurations beyond the prompt, dataset, and target count,"}, {"start": 357.0, "text": "the framework provides the SampleGenerator"}, {"start": 360.0, "text": "Which gives you full control over the generation process."}, {"start": 364.0, "text": "Let's go over some of these configurations!"}, {"start": 368.0, "text": "The sessionProvider is a closure that returns a LanguageModelSession."}, {"start": 372.0, "text": "This is where you control which model drives generation"}, {"start": 375.0, "text": "and what system-level instructions frame the task."}, {"start": 378.0, "text": "For our synthetic data generation,"}, {"start": 381.0, "text": "I'll use the PrivateCloudComputeLanguageModel"}, {"start": 383.0, "text": "since the context size is larger and then I'll add"}, {"start": 387.0, "text": "custom instructions to focus generation on specific books, genres and moods."}, {"start": 393.0, "text": "I also specify a list of rules for expectations on the samples generated."}, {"start": 397.0, "text": "I'll go over these later."}, {"start": 399.0, "text": "Just a note about how the session is used."}, {"start": 402.0, "text": "The framework handles batch size automatically"}, {"start": 404.0, "text": "which is the number of samples processed during generation."}, {"start": 408.0, "text": "The generator calls your sessionProvider once at the start of a run"}, {"start": 412.0, "text": "and then reuses that session across batches"}, {"start": 415.0, "text": "which helps the model maintain context as generation progresses."}, {"start": 419.0, "text": "But a session has a limit for how large it can grow."}, {"start": 423.0, "text": "The one exception is if you're making a lot of requests,"}, {"start": 426.0, "text": "giving it a large prompt, or getting large outputs,"}, {"start": 429.0, "text": "You can exhaust the session's context window mid-run which will throw an error."}, {"start": 434.0, "text": "In that case, the generator calls sessionProvider again"}, {"start": 437.0, "text": "to get a fresh one to continue generation"}, {"start": 441.0, "text": "but this won't contain context from the previous session."}, {"start": 444.0, "text": "So make sure your instructions in your sessionProvider is self-contained"}, {"start": 447.0, "text": "and doesn't assume it'll only be called once."}, {"start": 450.0, "text": "To learn more ways to mitigate against context size limits,"}, {"start": 454.0, "text": "watch the video \"Build agentic app experiences with Foundation Models\"."}, {"start": 459.0, "text": "Now with the custom session provider, you can also use the SampleGenerator"}, {"start": 462.0, "text": "to customize samplingStrategy,"}, {"start": 465.0, "text": "which controls how the generator selects examples"}, {"start": 468.0, "text": "from your initial dataset to show the model as in-context examples."}, {"start": 472.0, "text": "There are two types of sampling strategies you can specify,"}, {"start": 474.0, "text": "the first one is random sampling."}, {"start": 478.0, "text": "This strategy selects a random subset of your initial samples"}, {"start": 481.0, "text": "as examples to show the model making sure there are no duplicates."}, {"start": 485.0, "text": "This keeps the output varied without requiring us to think carefully"}, {"start": 488.0, "text": "about the order of our initial samples."}, {"start": 491.0, "text": "The second type of sampling strategy you can use is sliding window."}, {"start": 496.0, "text": "This strategy steps through your initial samples sequentially,"}, {"start": 499.0, "text": "skipping duplicates as it goes."}, {"start": 501.0, "text": "If your dataset has meaningful order,"}, {"start": 503.0, "text": "consider using this sliding window strategy."}, {"start": 507.0, "text": "For our generator, we'll use the random strategy"}, {"start": 510.0, "text": "because our initial samples are not meaningfully ordered."}, {"start": 513.0, "text": "And since it's the default strategy we don't need to explicitly define it here."}, {"start": 518.0, "text": "So, now that we've configured the generator with our custom sessionProvider,"}, {"start": 522.0, "text": "we can call the .run function,"}, {"start": 524.0, "text": "which returns a stream of newly synthesized samples."}, {"start": 527.0, "text": "As we iterate through each one,"}, {"start": 529.0, "text": "it gets added to our expandedDataset defined earlier."}, {"start": 533.0, "text": "Now that we've setup our configuration,"}, {"start": 536.0, "text": "let's explore how we can ensure our synthetic data is the way we expect it."}, {"start": 540.0, "text": "That's where the validator closure comes in hand."}, {"start": 543.0, "text": "The validator lets you define your own logic"}, {"start": 545.0, "text": "to accept or reject every generated sample."}, {"start": 548.0, "text": "We've already defined a set of rules"}, {"start": 550.0, "text": "in the instructions in the session provider earlier,"}, {"start": 553.0, "text": "but that doesn't guarantee the output will actually follow the rules."}, {"start": 557.0, "text": "Let's review them."}, {"start": 559.0, "text": "The first rule we defined"}, {"start": 560.0, "text": "is that the review must be at least 100 characters long"}, {"start": 563.0, "text": "Each review should also cover a wide range of genres, moods, and tones."}, {"start": 568.0, "text": "And the review needs to vary in length."}, {"start": 571.0, "text": "The model should also generate between 3 and 8 book tags."}, {"start": 575.0, "text": "And tags must be lowercase."}, {"start": 578.0, "text": "In order to understand what to validate your samples on,"}, {"start": 580.0, "text": "we need to consider what we can systematically check based on these rules."}, {"start": 585.0, "text": "Also, the validation closure validates per sample generation in isolation"}, {"start": 590.0, "text": "and doesn't have context to the other samples."}, {"start": 592.0, "text": "Reviewing these rules,"}, {"start": 593.0, "text": "I can tell that the diversity of reviews will require more judgement"}, {"start": 597.0, "text": "beyond a simple validation check"}, {"start": 599.0, "text": "and the length of reviews requires assessing across all samples."}, {"start": 604.0, "text": "For the other rules, we can assess them systematically"}, {"start": 607.0, "text": "using the validation closure."}, {"start": 609.0, "text": "For the first rule, we can define a review length validation."}, {"start": 614.0, "text": "Let's take a classic book we all know,"}, {"start": 616.0, "text": "\"Frankenstein\" by Mary Shelley, for example."}, {"start": 619.0, "text": "We can check if the generated sample defines a review"}, {"start": 622.0, "text": "with at least a length of 100 characters."}, {"start": 624.0, "text": "The model also generates tags for each review."}, {"start": 627.0, "text": "This means we can validate when there are between 3 and 8 tags."}, {"start": 632.0, "text": "And lastly, we can check if the tags are all lowercase."}, {"start": 637.0, "text": "Here I've already defined these 3 validation metrics in the SampleGenerator"}, {"start": 641.0, "text": "to check that the samples meet our expected structure."}, {"start": 644.0, "text": "So where do the results end up?"}, {"start": 647.0, "text": "Well, as generation progresses,"}, {"start": 649.0, "text": "valid samples are collected in the samples property on the SyntheticGenerator."}, {"start": 653.0, "text": "Any sample that fails these validators"}, {"start": 655.0, "text": "gets set aside automatically as invalidSamples."}, {"start": 658.0, "text": "Both are updated in real time throughout the run,"}, {"start": 660.0, "text": "so you can access them at any point."}, {"start": 662.0, "text": "Either during iteration to check progress or after the loop completes."}, {"start": 667.0, "text": "You can then use these results directly in your app or save the dataset locally."}, {"start": 672.0, "text": "Now let's review our evaluation with the 13 initial samples."}, {"start": 676.0, "text": "In Xcode 27, we introduced a new Evaluations Report"}, {"start": 679.0, "text": "to visualize your results."}, {"start": 681.0, "text": "This is the BookTaggingEvaluation with the 13 initial samples."}, {"start": 685.0, "text": "As you can see we got pretty high scores for tag quality"}, {"start": 689.0, "text": "evaluating both relevance and usefulness."}, {"start": 692.0, "text": "I've went ahead and ran the evaluation with our new dataset of 100 samples."}, {"start": 697.0, "text": "Now, we can compare the two evaluations using the Compare button"}, {"start": 701.0, "text": "and we're expecting the scores to drop!"}, {"start": 705.0, "text": "And we were correct!"}, {"start": 707.0, "text": "The quality scores have dropped."}, {"start": 709.0, "text": "Our tag generation feature looked like it was performing well earlier"}, {"start": 712.0, "text": "because we weren't testing it with a comprehensive dataset."}, {"start": 715.0, "text": "By running our evaluation on a larger dataset,"}, {"start": 718.0, "text": "a drop in scores could signal many different things."}, {"start": 721.0, "text": "Consider what this signal could suggest."}, {"start": 723.0, "text": "Score changes could be due to problems with our prompt or instructions."}, {"start": 728.0, "text": "You could refine one or both to better capture your needs."}, {"start": 732.0, "text": "You could also consider gaps in your intelligence feature."}, {"start": 735.0, "text": "Or you may want to adjust your evaluation to understand"}, {"start": 738.0, "text": "what you are actually evaluating on."}, {"start": 741.0, "text": "And lastly, your dataset may still not be representative enough"}, {"start": 745.0, "text": "and need to capture more variation."}, {"start": 747.0, "text": "You can continue to increase the dataset or include more edge cases"}, {"start": 751.0, "text": "using the synthetic data APIs."}, {"start": 754.0, "text": "These are the core ways to further improve your results."}, {"start": 758.0, "text": "Now that we have a solid approach for building a robust evaluation dataset"}, {"start": 761.0, "text": "using synthetic data,"}, {"start": 763.0, "text": "I want to take it one step further."}, {"start": 765.0, "text": "So far we've been evaluating our book tagging feature,"}, {"start": 769.0, "text": "but what happens when our app becomes more complex"}, {"start": 771.0, "text": "and needs to take multiple actions to complete a task like search?"}, {"start": 775.0, "text": "That's where tool calling comes in."}, {"start": 777.0, "text": "I'll hand it over to Kyle to show how that works!"}, {"start": 780.0, "text": "Thanks Ada!"}, {"start": 781.0, "text": "Now let's keep our evaluation driven development going"}, {"start": 783.0, "text": "and cover tool evaluations."}, {"start": 785.0, "text": "So far, we've been evaluating what the model generates —"}, {"start": 789.0, "text": "for our feature that's tags for books."}, {"start": 792.0, "text": "But intelligence features often take many behind-the-scenes steps"}, {"start": 795.0, "text": "to create their output."}, {"start": 796.0, "text": "They perform multiple actions in your app that each contribute to the results."}, {"start": 801.0, "text": "Tools add structure to model workflows"}, {"start": 803.0, "text": "when they're completing a task for people using your app."}, {"start": 807.0, "text": "You use them to operate on real data that people use daily."}, {"start": 810.0, "text": "They can operate using any custom business logic you define."}, {"start": 815.0, "text": "They can call functionality a user can invoke directly"}, {"start": 818.0, "text": "or entirely new logic for your intelligence feature,"}, {"start": 820.0, "text": "or a combo of both."}, {"start": 823.0, "text": "Here's the thing."}, {"start": 824.0, "text": "A model might give you a reasonable-sounding answer"}, {"start": 826.0, "text": "without ever calling the right tool."}, {"start": 829.0, "text": "The final output can look correct while the path to get there isn't right."}, {"start": 832.0, "text": "So let's talk about those challenges"}, {"start": 834.0, "text": "and how tool evaluations can help you handle them"}, {"start": 837.0, "text": "First, instruction following:"}, {"start": 840.0, "text": "you need to tell a model how to use each tool,"}, {"start": 842.0, "text": "and the attention you pay to the details matters."}, {"start": 845.0, "text": "Try following the instructions word-by-word yourself"}, {"start": 848.0, "text": "to see if you miss a step."}, {"start": 850.0, "text": "Then there's tool complexity,"}, {"start": 852.0, "text": "they can accept simple instructions or require fine-tuning parameter ranges."}, {"start": 857.0, "text": "Then there are edge cases."}, {"start": 858.0, "text": "A tool might seem to work well on common inputs,"}, {"start": 861.0, "text": "but behave surprisingly on the rare ones."}, {"start": 864.0, "text": "That's why we need tool evaluations."}, {"start": 866.0, "text": "They let you verify the how, not just the what."}, {"start": 870.0, "text": "The model should call the correct tools,"}, {"start": 872.0, "text": "with the correct arguments in the order you expect."}, {"start": 875.0, "text": "And along the way,"}, {"start": 877.0, "text": "you'll double check that there weren't any unexpected tool calls in the middle."}, {"start": 881.0, "text": "Let's take a look at this in practice and build our first tool evaluation."}, {"start": 884.0, "text": "In the BookTracker app, we've added a library assistant."}, {"start": 888.0, "text": "A user can search for a book"}, {"start": 889.0, "text": "and instead of just filtering books based on the title and other strings,"}, {"start": 893.0, "text": "the model uses our app's custom tools to find relevant books."}, {"start": 897.0, "text": "There's a searchBooks tool to find books that might have similar tags."}, {"start": 901.0, "text": "Then there's a getBookDetails tool to extract book metadata,"}, {"start": 904.0, "text": "like publication date from the searches."}, {"start": 908.0, "text": "Then there's the findSimilarBooks tool"}, {"start": 909.0, "text": "that performs a semantic search for similar books,"}, {"start": 912.0, "text": "so we're chaining together multiple steps, each one a tool call."}, {"start": 916.0, "text": "Here's SearchBooksTool."}, {"start": 919.0, "text": "It conforms to the Tool protocol, it has a name the model sees"}, {"start": 924.0, "text": "and a description that tells it when this tool is useful."}, {"start": 928.0, "text": "The arguments are a Generable struct."}, {"start": 930.0, "text": "Notice these are all optional,"}, {"start": 932.0, "text": "the model decides which filters to use based on what the user asked for."}, {"start": 937.0, "text": "If you prompt a model with find gothic books,"}, {"start": 940.0, "text": "we'd expect it to populate the tag argument."}, {"start": 943.0, "text": "If you prompt a model with show me something cheerful,"}, {"start": 946.0, "text": "we'd expect to generate a mood search."}, {"start": 949.0, "text": "These are exactly the kinds of decisions we want to evaluate."}, {"start": 952.0, "text": "OK, so that's a refresher on the tools."}, {"start": 955.0, "text": "Now let's write our first tool evaluation and see how they perform."}, {"start": 959.0, "text": "The main component of a tool evaluation is a trajectory expectation."}, {"start": 964.0, "text": "A session transcript has tool calls among the prompts and responses,"}, {"start": 968.0, "text": "A trajectory expectation checks the order"}, {"start": 970.0, "text": "and kind of each tool call in a language model session."}, {"start": 973.0, "text": "You can think of a trajectory expectation check"}, {"start": 976.0, "text": "like going over the list of decisions you made when planning a route."}, {"start": 979.0, "text": "Cars, bikes, and buses are all tools"}, {"start": 981.0, "text": "that have their time and place in getting somewhere,"}, {"start": 983.0, "text": "but you can evaluate their utility for each segment in a specific trip."}, {"start": 988.0, "text": "The expectation looks for all of the tool calls."}, {"start": 991.0, "text": "Then for each one,"}, {"start": 993.0, "text": "runs it against the expectations you write into your evaluations."}, {"start": 996.0, "text": "Here's a simple case in code form."}, {"start": 999.0, "text": "Our prompt is \"Find books tagged gothic\"."}, {"start": 1002.0, "text": "We expect one tool call \"searchBooks\"."}, {"start": 1006.0, "text": "This is a TrajectoryExpectation."}, {"start": 1008.0, "text": "It describes the tool calls we expect to see in the model's transcript."}, {"start": 1012.0, "text": "The unordered here means we don't care when this tool call happens,"}, {"start": 1016.0, "text": "just that it happens."}, {"start": 1018.0, "text": "We can further refine this by adding arguments to the expectation."}, {"start": 1022.0, "text": "Here I'm adding an argument to expect the tag \"gothic\"."}, {"start": 1026.0, "text": "An exact match isn't always what you want."}, {"start": 1029.0, "text": "If the prompt is \"Find something cheerful\","}, {"start": 1031.0, "text": "the model might pass uplifting, happy, cheerful — any of those are fine."}, {"start": 1037.0, "text": "The .naturalLanguage matcher checks whether the value matches the intent,"}, {"start": 1040.0, "text": "not the exact string."}, {"start": 1042.0, "text": "And there's a whole set of matchers for different situations —"}, {"start": 1045.0, "text": "contains, oneOf, pattern, range, and more."}, {"start": 1051.0, "text": "Check out the developer documentation for more information."}, {"start": 1054.0, "text": "For multistep tasks, order matters."}, {"start": 1057.0, "text": "Here the model must first call \"searchBooks\","}, {"start": 1060.0, "text": "then call \"getBookDetails\"."}, {"start": 1063.0, "text": "If an agent tries to get details first, it doesn't have a bookId yet —"}, {"start": 1067.0, "text": "that's a bug."}, {"start": 1068.0, "text": "Trajectory expectations catch it because we're checking the journey,"}, {"start": 1071.0, "text": "not just the destination."}, {"start": 1075.0, "text": "Sometimes what an agent shouldn't do is just as important."}, {"start": 1079.0, "text": "If a prompt includes ideas like don't look for similar books,"}, {"start": 1082.0, "text": "the model should follow instructions."}, {"start": 1085.0, "text": "The disallowed parameter specifies tools that must not appear in the transcript."}, {"start": 1089.0, "text": "If an agent calls \"findSimilarBooks\" anyway — that's a failure."}, {"start": 1094.0, "text": "Here's where all of the trajectory expectations come together"}, {"start": 1096.0, "text": "in the full evaluation."}, {"start": 1099.0, "text": "We define a dataset of samples,"}, {"start": 1100.0, "text": "each with a prompt and a trajectory expectation"}, {"start": 1104.0, "text": "and use ToolCallEvaluator to score them."}, {"start": 1107.0, "text": "The ToolCallEvaluator combines a LanguageModelSession with the tools,"}, {"start": 1111.0, "text": "gets a response, and captures the structured transcript."}, {"start": 1116.0, "text": "Tool call evaluation results show up in the Xcode assistant"}, {"start": 1120.0, "text": "alongside the rest of your results,"}, {"start": 1122.0, "text": "and you can get the whole picture of how your intelligence-based feature behaves."}, {"start": 1126.0, "text": "But wait!"}, {"start": 1127.0, "text": "We can also use the Evaluations APIs"}, {"start": 1129.0, "text": "to generate synthetic data for your tool evaluations!"}, {"start": 1133.0, "text": "Oh yes let's do that!"}, {"start": 1135.0, "text": "Trajectory expectations are generable too."}, {"start": 1137.0, "text": "Expanding a dataset for your tool evaluations can be quite complex,"}, {"start": 1142.0, "text": "and with the Evaluations framework we've made it a lot easier to do just that!"}, {"start": 1147.0, "text": "Since our Tool Call evaluation leverages ModelSample"}, {"start": 1150.0, "text": "and TrajectoryExpectation that are generable,"}, {"start": 1153.0, "text": "we can synthetically generate more samples using Sample generator like before."}, {"start": 1158.0, "text": "I've went ahead and defined a prompt"}, {"start": 1160.0, "text": "and custom instructions for the sessionProvider."}, {"start": 1163.0, "text": "Keep in mind when creating synthetic data for tool evaluations,"}, {"start": 1167.0, "text": "the model doesn't know what tools you've defined"}, {"start": 1169.0, "text": "or what order the tools need to be called in."}, {"start": 1172.0, "text": "So here I've specified the available tools explaining their purpose,"}, {"start": 1177.0, "text": "any order expectations, and other context the model might need."}, {"start": 1182.0, "text": "Then we can define the sampleGenerator"}, {"start": 1185.0, "text": "and use our existing dataset as our initial samples,"}, {"start": 1188.0, "text": "and a targetCount of 100."}, {"start": 1191.0, "text": "We can also specify validation metrics here as well!"}, {"start": 1195.0, "text": "Here I've made sure there's always an expectation"}, {"start": 1198.0, "text": "and I've also made sure the synthetic samples include at least one tool."}, {"start": 1202.0, "text": "And lastly any tools called are actual tools we've already defined."}, {"start": 1208.0, "text": "And that's how you can generate"}, {"start": 1209.0, "text": "and validate synthetic samples for your tool evaluations!"}, {"start": 1212.0, "text": "The synthetic data APIs are a powerful way"}, {"start": 1215.0, "text": "to expand your existing dataset beyond your capabilities!"}, {"start": 1218.0, "text": "And the more representative your data, the more your scores reflect reality."}, {"start": 1223.0, "text": "Alright Kyle, back to you!"}, {"start": 1225.0, "text": "This is where it all comes together."}, {"start": 1227.0, "text": "Earlier we built book tagging evaluation, it checks what the model produces."}, {"start": 1232.0, "text": "Tag count, genre coverage, quality scores."}, {"start": 1236.0, "text": "Now we have tool evaluations — they check how the model gets there."}, {"start": 1240.0, "text": "The right tools, right arguments and right order."}, {"start": 1244.0, "text": "Run both in the same evaluation suite"}, {"start": 1246.0, "text": "and you'll have built end-to-end confidence in your feature."}, {"start": 1249.0, "text": "Now that we've covered some ways to make your evaluations even more robust,"}, {"start": 1253.0, "text": "you can start applying these ideas to your apps and evaluation datasets."}, {"start": 1257.0, "text": "To get started, try making your own synthetic data,"}, {"start": 1261.0, "text": "evaluate the custom tools in your app"}, {"start": 1263.0, "text": "and check out the sample app and other articles in the developer documentation."}, {"start": 1268.0, "text": "Wow Ada, we've covered a lot today!"}, {"start": 1270.0, "text": "Yeah, we definitely did!"}, {"start": 1272.0, "text": "But the real plot twist is what you build with it."}, {"start": 1275.0, "text": "No spoilers though!"}, {"start": 1277.0, "text": "And we hope you enjoyed learning about the Evaluations framework!"}]}