{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-298", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/298", "wordCount": 3703, "segments": [{"start": 8.0, "text": "Hi, I'm Yada."}, {"start": 9.0, "text": "And I'm Rob."}, {"start": 10.0, "text": "We're excited to introduce the Evaluations framework."}, {"start": 13.0, "text": "A new framework that measures the quality of your intelligent features"}, {"start": 16.0, "text": "so you can deliver your apps with confidence."}, {"start": 20.0, "text": "Last year we introduced the Foundation Models framework,"}, {"start": 23.0, "text": "which helped you add intelligent features to your apps, using our on-device models."}, {"start": 28.0, "text": "The same models which power Apple Intelligence."}, {"start": 32.0, "text": "Building app features with generative AI poses new testing challenges,"}, {"start": 36.0, "text": "because the same input can produce different outputs."}, {"start": 40.0, "text": "These models break a contract that is fundamental to software testing."}, {"start": 45.0, "text": "Consider traditional software,"}, {"start": 47.0, "text": "where a particular input always produces a particular output."}, {"start": 51.0, "text": "You can easily verify this behavior with a unit test."}, {"start": 56.0, "text": "You're guaranteed the same input will produce the same output"}, {"start": 59.0, "text": "on any device, including your customers'."}, {"start": 63.0, "text": "With intelligent software,"}, {"start": 65.0, "text": "you cannot rely on functional consistency to verify behavior."}, {"start": 69.0, "text": "Which means that unit tests are insufficient."}, {"start": 72.0, "text": "Unverified behavior can erode customer confidence."}, {"start": 76.0, "text": "Your customers expect intelligent features in your app,"}, {"start": 79.0, "text": "like any feature, to be safe, trustworthy, and reliable."}, {"start": 83.0, "text": "Shipping a feature with unpredictable behavior,"}, {"start": 86.0, "text": "can have adverse consequences on your app's reputation."}, {"start": 90.0, "text": "It's important we measure our intelligent features"}, {"start": 93.0, "text": "and understand how they respond to different inputs."}, {"start": 96.0, "text": "And since functional tests can't verify probabilistic behavior,"}, {"start": 100.0, "text": "we need new a form of test that is more robust."}, {"start": 104.0, "text": "We need to know: how often does my app produce unexpected results?"}, {"start": 109.0, "text": "How often does the agent take an unexpected path to generate answers?"}, {"start": 114.0, "text": "And under what circumstances does the feature produce unsafe results?"}, {"start": 119.0, "text": "The challenges of testing intelligence features"}, {"start": 122.0, "text": "powered by generative AI"}, {"start": 123.0, "text": "is exactly why we built the Evaluations framework."}, {"start": 128.0, "text": "The Evaluations framework is a flexible system"}, {"start": 131.0, "text": "of provided types and protocols."}, {"start": 134.0, "text": "This video will focus on evaluating intelligent features"}, {"start": 137.0, "text": "powered by language models."}, {"start": 140.0, "text": "But you can evaluate any stochastic system,"}, {"start": 142.0, "text": "such as classifiers and linear regression models."}, {"start": 146.0, "text": "Yada and I will introduce you to several types in the framework."}, {"start": 151.0, "text": "We'll cover data loading and building a diverse dataset."}, {"start": 155.0, "text": "Building quantitative metrics with Evaluator and Metric."}, {"start": 159.0, "text": "And refining your measurements using model judges and score dimensions"}, {"start": 163.0, "text": "to create qualitative metrics."}, {"start": 166.0, "text": "In this video, you'll get started with Evaluations."}, {"start": 169.0, "text": "After building your first evaluation,"}, {"start": 171.0, "text": "we'll show you how to scale that evaluation,"}, {"start": 173.0, "text": "with more data, and more measurements."}, {"start": 176.0, "text": "Then we'll teach you how to build powerful model judges using our simple API."}, {"start": 181.0, "text": "Let's get started with Evaluations."}, {"start": 185.0, "text": "Yada and I are building an app called Book Tracker."}, {"start": 188.0, "text": "We both love books and wanted an app to manage our libraries."}, {"start": 192.0, "text": "Yada just added a new feature, called BookTaggingService."}, {"start": 196.0, "text": "It automatically tags books based on a review we've written in Book Tracker."}, {"start": 201.0, "text": "I can't wait to open Xcode and try it out."}, {"start": 206.0, "text": "Let's add a #Playground macro to BookTaggingService.swift."}, {"start": 210.0, "text": "Here's the review of \"Pride & Prejudice\" Yada added to Book Tracker."}, {"start": 214.0, "text": "Have to say I'm a fan myself."}, {"start": 217.0, "text": "Let's see what tags we get back."}, {"start": 219.0, "text": "This a good start, but as I'm reading some of the tags,"}, {"start": 223.0, "text": "I can see our service will need a little work."}, {"start": 226.0, "text": "9 tags is more than I was expecting."}, {"start": 229.0, "text": "And I don't want the book's name as a tag, either."}, {"start": 233.0, "text": "Multi-word tags are gonna be a problem in the UI,"}, {"start": 236.0, "text": "so we should avoid those as well."}, {"start": 239.0, "text": "Let's see if we have better luck with another review: \"Dracula\"."}, {"start": 248.0, "text": "7 tags is within our expected amount."}, {"start": 251.0, "text": "Let's take a closer look at them."}, {"start": 253.0, "text": "There are some behaviors that I'd like to see more of."}, {"start": 257.0, "text": "It identified literary genres,"}, {"start": 260.0, "text": "and some categories that would help me browse a larger library."}, {"start": 264.0, "text": "Okay, we've just completed our first evaluation of the service."}, {"start": 268.0, "text": "We created a list of expectations"}, {"start": 270.0, "text": "and used our human judgement to measure how the service performed."}, {"start": 274.0, "text": "Every evaluation measures how well an intelligent feature"}, {"start": 277.0, "text": "performs against our expectations."}, {"start": 280.0, "text": "Unfortunately human judgement doesn't scale."}, {"start": 285.0, "text": "But we've created a way to automate and scale evaluations."}, {"start": 288.0, "text": "All you have to do is add import Evaluations,"}, {"start": 291.0, "text": "and implement the Evaluation protocol."}, {"start": 294.0, "text": "Let's build an evaluation in code."}, {"start": 298.0, "text": "And we'll start with our first expectation:"}, {"start": 300.0, "text": "measuring that our service generates the correct number of tags."}, {"start": 304.0, "text": "There are five steps to building and running an evaluation."}, {"start": 307.0, "text": "You define what code you're measuring."}, {"start": 310.0, "text": "Then, define what data you're sending the code."}, {"start": 313.0, "text": "Next, define what measurements you're making and how."}, {"start": 318.0, "text": "Then, summarize your measurements."}, {"start": 320.0, "text": "And then, finally, create a test to run your evaluation."}, {"start": 324.0, "text": "First, we add the call to the BookTaggingService,"}, {"start": 327.0, "text": "and return it's output inside of the subject(from:) method."}, {"start": 331.0, "text": "These generated tags are the subject of our evaluation."}, {"start": 335.0, "text": "Next, define the input samples we'll feed the code we're measuring."}, {"start": 340.0, "text": "Then, we'll use ModelSample to wrap the same reviews"}, {"start": 343.0, "text": "we tested in the #Playground earlier:"}, {"start": 345.0, "text": "\"Pride & Prejudice\" and \"Dracula\"."}, {"start": 348.0, "text": "Notice we define expected tags as well."}, {"start": 351.0, "text": "These are the ideal tags we'd like to see from the service."}, {"start": 356.0, "text": "Now, its time to define our measurements, using the Metric type."}, {"start": 361.0, "text": "We add a Metric called \"TagCount\","}, {"start": 363.0, "text": "which will track the number of generated tags returned by the service."}, {"start": 367.0, "text": "We need something to measure the generated tags."}, {"start": 369.0, "text": "Evaluator takes a closure,"}, {"start": 371.0, "text": "that gets passed the output from the service, for a given sample."}, {"start": 375.0, "text": "We can check the number of generated tags by using the count of the tags property."}, {"start": 381.0, "text": "If the length of the tags array is between 3 and 8,"}, {"start": 384.0, "text": "we return a passing metric from our Evaluator."}, {"start": 388.0, "text": "If not, we return a failing metric."}, {"start": 392.0, "text": "Evaluators run over a single sample at a time."}, {"start": 396.0, "text": "But we can measure trends and look for patterns"}, {"start": 398.0, "text": "measured over all of our samples in the aggregateMetrics(using:) method."}, {"start": 404.0, "text": "Let's calculate the average number of times"}, {"start": 406.0, "text": "the service generates the correct number of tags."}, {"start": 411.0, "text": "Then we'll have a ratio for how often the service behaves correctly."}, {"start": 416.0, "text": "Okay, we've written our first evaluation."}, {"start": 419.0, "text": "Next, let's write some code to run it."}, {"start": 423.0, "text": "Evaluations integrates with Swift Testing,"}, {"start": 426.0, "text": "so you can run your evaluations in your app's test targets."}, {"start": 430.0, "text": "Here we instantiate our BookTaggingEvaluation"}, {"start": 432.0, "text": "inside of a Test Suite."}, {"start": 434.0, "text": "We add some notes to our evaluation run,"}, {"start": 436.0, "text": "so we can keep track of the configuration we're evaluating."}, {"start": 440.0, "text": "This will be helpful later,"}, {"start": 441.0, "text": "when we compare across different evaluation runs."}, {"start": 445.0, "text": "Next, we add a test function, using the @Test macro,"}, {"start": 448.0, "text": "and a new @Test trait .evaluates."}, {"start": 451.0, "text": "This trait takes our evaluation and a notes dictionary,"}, {"start": 454.0, "text": "like the one we've created earlier in the @Suite."}, {"start": 458.0, "text": "Inside our @Test, we can access an evaluation results bundle."}, {"start": 462.0, "text": "This contains all of the metrics"}, {"start": 464.0, "text": "and aggregate metrics from our evaluation run."}, {"start": 467.0, "text": "Let's grab all of our tagCount metrics from the results,"}, {"start": 470.0, "text": "and assert against its average value."}, {"start": 472.0, "text": "We'll use the aggregateValue method on the results bundle."}, {"start": 476.0, "text": "Then, assert against the average in an #expect macro."}, {"start": 480.0, "text": "Here, I expect the service to produce the correct number of tags 80% of the time."}, {"start": 485.0, "text": "Why 80%?"}, {"start": 487.0, "text": "If the service performance dips below 80%, I want to know"}, {"start": 490.0, "text": "and a failing test is great signal."}, {"start": 493.0, "text": "But what if I want even more insight into what happened during the evaluation?"}, {"start": 498.0, "text": "We have a new test report for evaluations."}, {"start": 501.0, "text": "It's a great way to dive into the details of your evaluation and analyze further."}, {"start": 507.0, "text": "Let's run our test, and I'll walk you through the report."}, {"start": 510.0, "text": "Based on what the service returned earlier in my #Playground,"}, {"start": 513.0, "text": "specifically how many tags it generated for \"Pride & Prejudice\","}, {"start": 517.0, "text": "I don't expect the test to pass."}, {"start": 521.0, "text": "Okay, the test didn't pass."}, {"start": 523.0, "text": "Let's go to the report and review what happened."}, {"start": 526.0, "text": "Click on the report navigator,"}, {"start": 527.0, "text": "and then select Evaluations in the test report."}, {"start": 531.0, "text": "Here's the evaluation report for the test suite."}, {"start": 534.0, "text": "Let's double click the row to find out more."}, {"start": 537.0, "text": "And I see my TagCount metric only passed 50% of the time."}, {"start": 542.0, "text": "And a quick look at the full results table shows me"}, {"start": 545.0, "text": "that my \"Pride & Prejudice\" sample produced a failure."}, {"start": 547.0, "text": "But my \"Dracula\" sample produced the correct number of tags."}, {"start": 552.0, "text": "I can select each row in the table to see more details,"}, {"start": 556.0, "text": "using the assistant editor in Xcode."}, {"start": 559.0, "text": "The detail panel shows the prompt, and each measurement for the ModelSample."}, {"start": 563.0, "text": "At the bottom, you see the entire response from the model."}, {"start": 568.0, "text": "Let's recap a little."}, {"start": 570.0, "text": "We built an evaluation for BookTaggingService."}, {"start": 573.0, "text": "Ran that evaluation and it failed to meet our optimization target."}, {"start": 578.0, "text": "Remember back in our test definition?"}, {"start": 580.0, "text": "This is where we defined our optimization target."}, {"start": 583.0, "text": "We're saying the feature behaves as expected,"}, {"start": 586.0, "text": "if the correct number of tags were generated, 80% of the time."}, {"start": 591.0, "text": "Beyond the automated check of our optimization target"}, {"start": 594.0, "text": "we need to analyze deeper into our results and gather insights."}, {"start": 598.0, "text": "Specifically, think about the changes"}, {"start": 600.0, "text": "that could be made to improve the feature's performance."}, {"start": 604.0, "text": "I have a hunch, so I look back at the @Generable type, BookTags,"}, {"start": 608.0, "text": "that the service is generating."}, {"start": 610.0, "text": "We already have a @Guide macro giving the model additional instructions"}, {"start": 614.0, "text": "for the tags property."}, {"start": 616.0, "text": "I could specify a count property in that @Guide,"}, {"start": 619.0, "text": "which can take a range."}, {"start": 620.0, "text": "That should instruct the model to only generate between 3 and 8 tags."}, {"start": 625.0, "text": "This is an interesting theory."}, {"start": 627.0, "text": "Let's make that change."}, {"start": 631.0, "text": "Then re-run the evaluation to see if I'm right."}, {"start": 633.0, "text": "We call this process hill-climbing."}, {"start": 637.0, "text": "All right, I made the change and I re-ran the evaluation."}, {"start": 641.0, "text": "My test passed, and my TagCount passes a 100% of the time."}, {"start": 645.0, "text": "But I notice a potentially strange behavior:"}, {"start": 648.0, "text": "after my change, the service always generates eight tags."}, {"start": 652.0, "text": "Hmmm."}, {"start": 654.0, "text": "Now that we have the Evaluations set up,"}, {"start": 657.0, "text": "let's collect more measurements across more samples,"}, {"start": 660.0, "text": "and let's see if that strange behavior persists."}, {"start": 663.0, "text": "We started our evaluation with only two data samples."}, {"start": 666.0, "text": "As we saw, that only gave us two measurements to extract trends."}, {"start": 671.0, "text": "Good evaluations have thousands of samples to extract trends,"}, {"start": 675.0, "text": "but also to exercise your feature in many different ways."}, {"start": 680.0, "text": "We should consider variety in our dataset."}, {"start": 682.0, "text": "For example…"}, {"start": 684.0, "text": "We want the service to recognize different genres."}, {"start": 687.0, "text": "We can't assume every user will give it a verbose review,"}, {"start": 690.0, "text": "so our reviews should be different lengths."}, {"start": 693.0, "text": "You browse for fiction and non-fiction using different categories,"}, {"start": 697.0, "text": "your samples should represent that variety."}, {"start": 700.0, "text": "Finally, you should consider different forms:"}, {"start": 703.0, "text": "novels, short stories, and essays."}, {"start": 707.0, "text": "Let's makes it hard on the model too."}, {"start": 709.0, "text": "Sprinkle in personal opinions,"}, {"start": 710.0, "text": "so we can measure how well the service ignores those in the reviews."}, {"start": 715.0, "text": "If you want to teach the feature how to write tags like you,"}, {"start": 718.0, "text": "start by including more of your personal style"}, {"start": 721.0, "text": "in the expected values of the samples."}, {"start": 724.0, "text": "Let's look at a few examples in code."}, {"start": 727.0, "text": "This review of \"The Secret Garden\""}, {"start": 729.0, "text": "reads very different than the reviews we started with"}, {"start": 732.0, "text": "because we wrote it as though we were an avid gardener."}, {"start": 735.0, "text": "Here we challenge the model,"}, {"start": 737.0, "text": "including a personal review from a mother reading \"Treasure Island\" to her son."}, {"start": 741.0, "text": "Lots of personal opinions in this review."}, {"start": 745.0, "text": "This board game enthusiast needed multiple paragraphs"}, {"start": 748.0, "text": "for their review of the Chinese classic, \"Romance of the Three Kingdoms\"."}, {"start": 754.0, "text": "While this casual reader described a famous British detective's sidekick"}, {"start": 758.0, "text": "in a single sentence."}, {"start": 760.0, "text": "The game is afoot, when the model tries to decipher this one."}, {"start": 766.0, "text": "And while it's fun to come up with these examples,"}, {"start": 768.0, "text": "human data creation doesn't scale, either."}, {"start": 771.0, "text": "Consider these sentence completion pairs,"}, {"start": 774.0, "text": "where the output of the feature"}, {"start": 775.0, "text": "is compared directly to the expected answer."}, {"start": 778.0, "text": "You need thousands of examples for this evaluation to be effective."}, {"start": 783.0, "text": "Fortunately, we include a SampleGenerator as part of the Evaluations framework."}, {"start": 788.0, "text": "You can call it directly on an array of ModelSamples"}, {"start": 791.0, "text": "and it will synthetically generate more samples using a model of your choice."}, {"start": 796.0, "text": "To hear more about how you can synthesize larger datasets,"}, {"start": 799.0, "text": "and learn more about advanced uses of ModelSample,"}, {"start": 802.0, "text": "please check out our video"}, {"start": 804.0, "text": "\"Create robust evaluations for agentic apps\"."}, {"start": 809.0, "text": "Back to BookTagging."}, {"start": 810.0, "text": "I'm going to update my dataset property"}, {"start": 812.0, "text": "to include all of the book reviews from our library,"}, {"start": 815.0, "text": "including the four we showed earlier."}, {"start": 818.0, "text": "When I re-run my evaluation with the expanded dataset,"}, {"start": 822.0, "text": "my test passes, my TagCount average is still 100%,"}, {"start": 826.0, "text": "and the service generated eight tags for all of them."}, {"start": 829.0, "text": "Now we know there's a weird behavior in the service."}, {"start": 833.0, "text": "Looking back at my expectations,"}, {"start": 835.0, "text": "I've built an evaluator to track if the number of tags are in range."}, {"start": 839.0, "text": "I think I still need to refine that a little."}, {"start": 842.0, "text": "Here's my current Metric and Evaluators setup."}, {"start": 845.0, "text": "First, I define a new Metric, \"TagTotal\","}, {"start": 848.0, "text": "that will record the number of generated tags."}, {"start": 851.0, "text": "Then I build a simple Evaluator,"}, {"start": 853.0, "text": "which records the length of the generated tags array."}, {"start": 856.0, "text": "Then, we record a measurement using a scoring value,"}, {"start": 859.0, "text": "instead of a pass/fail value."}, {"start": 862.0, "text": "Using the \"TagTotal\" and \"TagCount\" metrics we evaluate range compliance"}, {"start": 868.0, "text": "and the distribution of generated tags."}, {"start": 871.0, "text": "We can follow a similar pattern for checking the number of words in tags."}, {"start": 876.0, "text": "Here, we check each tag for a space,"}, {"start": 878.0, "text": "then returning a failing metric if it does."}, {"start": 881.0, "text": "Identifying a literary genre is equally straightforward"}, {"start": 885.0, "text": "assuming you're looking for a known set of genres."}, {"start": 888.0, "text": "We check the BookTaggingService for knownGenres."}, {"start": 891.0, "text": "Then compare each of the generated tags for a match."}, {"start": 895.0, "text": "Our evaluation is really filling out."}, {"start": 897.0, "text": "We can already measure three of our original five expectations."}, {"start": 902.0, "text": "And our evaluation report provides a rich picture"}, {"start": 905.0, "text": "of how our tagging service is performing."}, {"start": 907.0, "text": "We track our three expectations using five aggregate metrics."}, {"start": 912.0, "text": "Here, we can see the distribution of tags,"}, {"start": 914.0, "text": "along with range compliance and containing genre tags."}, {"start": 919.0, "text": "Using our hill-climbing methodology,"}, {"start": 921.0, "text": "we've iterated on our instructions for the service."}, {"start": 924.0, "text": "Here's where we started at the beginning."}, {"start": 927.0, "text": "After several updates to our evaluation and multiple runs through our loop."}, {"start": 933.0, "text": "And we can track each change to our instructions,"}, {"start": 936.0, "text": "by an expectation we added to our evaluation to verify that change."}, {"start": 941.0, "text": "When you take our hill-climbing feedback loop,"}, {"start": 944.0, "text": "and center your development process around it,"}, {"start": 947.0, "text": "we call it evaluation-driven development."}, {"start": 951.0, "text": "But we're not done getting our service up to spec."}, {"start": 954.0, "text": "We still expect our tags to be informative,"}, {"start": 957.0, "text": "relevant to the book and helpful for browsing your library."}, {"start": 961.0, "text": "Here's Yada to tell you about model judges,"}, {"start": 964.0, "text": "and how they'll take your evaluation to the next level."}, {"start": 967.0, "text": "Thanks Rob."}, {"start": 968.0, "text": "Model judges are how we measure qualitative metrics at scale."}, {"start": 972.0, "text": "Let me show you how to build and refine one."}, {"start": 975.0, "text": "Let's take a look at a concrete example."}, {"start": 979.0, "text": "Here's a review of \"Alice in Wonderland\" that Rob wrote in Book Tracker."}, {"start": 985.0, "text": "And here are the tags that our service generated."}, {"start": 989.0, "text": "Six tags, single word or hyphenated, with tags identifying genre."}, {"start": 995.0, "text": "Every quantitative metric we built with Rob passed."}, {"start": 999.0, "text": "But look closer."}, {"start": 1001.0, "text": "'Overrated' and 'pretentious' doesn't describe the book —"}, {"start": 1005.0, "text": "they describe how the reader felt about it."}, {"start": 1008.0, "text": "And 'whodunit' isn't even the right genre."}, {"start": 1011.0, "text": "The model picked it up from 'riddles he never answers.'"}, {"start": 1014.0, "text": "It latched onto the language of the review without understanding the book."}, {"start": 1019.0, "text": "Our metrics are passing, but they're not giving us the right signals back."}, {"start": 1025.0, "text": "But, I think we can ask a model to help us here."}, {"start": 1028.0, "text": "If a person can read these tags and tell us which ones work,"}, {"start": 1032.0, "text": "maybe a model can too."}, {"start": 1035.0, "text": "Oh nice!"}, {"start": 1036.0, "text": "The model actually captured that certain tags are not helpful."}, {"start": 1040.0, "text": "I think I can ask the model to evaluate all of the tags"}, {"start": 1044.0, "text": "that my feature generated!"}, {"start": 1047.0, "text": "And that's exactly what a Model Judge is."}, {"start": 1050.0, "text": "A Model Judge is a language model used to score your feature's output."}, {"start": 1055.0, "text": "It gives you a subjective rating —"}, {"start": 1058.0, "text": "the kind of judgment call a person would make —"}, {"start": 1060.0, "text": "but applied consistently across your entire dataset."}, {"start": 1065.0, "text": "So let's talk about how this works."}, {"start": 1068.0, "text": "Here's the model powering your intelligence feature."}, {"start": 1072.0, "text": "Our BookTaggingService runs on-device"}, {"start": 1075.0, "text": "because it needs to be fast and local for every user interaction."}, {"start": 1080.0, "text": "You can use a second model as a judge to evaluate your feature."}, {"start": 1084.0, "text": "Your judge should be at least as capable as the model you're evaluating."}, {"start": 1089.0, "text": "In our case,"}, {"start": 1090.0, "text": "we can use a more capable model from Private Cloud Compute."}, {"start": 1095.0, "text": "The model judge has a few key components."}, {"start": 1098.0, "text": "The instruction tells the model it will be given book reviews,"}, {"start": 1102.0, "text": "and how it should evaluate it."}, {"start": 1104.0, "text": "The feature input is the prompt given to the feature being judged,"}, {"start": 1109.0, "text": "in our case, its the book review."}, {"start": 1112.0, "text": "The feature output is the tags our service generated."}, {"start": 1117.0, "text": "And finally, the scoring guide tells the model how to evaluate"}, {"start": 1121.0, "text": "and score the feature."}, {"start": 1123.0, "text": "The Evaluations framework handles most of this for you,"}, {"start": 1126.0, "text": "so you can focus on the scoring guide."}, {"start": 1131.0, "text": "Putting it all together, here's a simple model judge."}, {"start": 1135.0, "text": "We've defined a \"TagQuality\" metric on a 1 to 4 scale,"}, {"start": 1140.0, "text": "with each level describing what that score means."}, {"start": 1143.0, "text": "An even number of options prevents the judge from defaulting"}, {"start": 1147.0, "text": "to a neutral middle score."}, {"start": 1149.0, "text": "Four levels provides just enough distinction"}, {"start": 1153.0, "text": "without diluting the meaning of each rating."}, {"start": 1156.0, "text": "And finally, we've specified Private Cloud Compute as our judge model,"}, {"start": 1161.0, "text": "giving us a more capable evaluator"}, {"start": 1164.0, "text": "than the on-device model we're evaluating."}, {"start": 1168.0, "text": "In the Evaluations framework, a model judge is just another Evaluator."}, {"start": 1173.0, "text": "It conforms to the same protocol as the quantitative evaluators"}, {"start": 1177.0, "text": "and produces the same Metric type."}, {"start": 1180.0, "text": "So you can mix them freely within a single evaluation."}, {"start": 1184.0, "text": "Alright, let's run it!"}, {"start": 1187.0, "text": "Every sample received a 3 or 4 quality score."}, {"start": 1192.0, "text": "Lets go back to our \"Alice in Wonderland\" sample."}, {"start": 1196.0, "text": "The model judge gave this a quality score of 3."}, {"start": 1200.0, "text": "If we look at the rationale,"}, {"start": 1202.0, "text": "we can identify that the model flagged 'whodunit'"}, {"start": 1205.0, "text": "and 'detective-fiction' as not relevant to the book."}, {"start": 1209.0, "text": "But, we also expected it to flag all of these other tags"}, {"start": 1214.0, "text": "that either reflect the reader's opinion or are not helpful for browsing."}, {"start": 1220.0, "text": "With model judges, rationales are essential."}, {"start": 1223.0, "text": "They give you a window into why the judge scored what it scored."}, {"start": 1227.0, "text": "And here's the thing:"}, {"start": 1229.0, "text": "by the scale we wrote, the judge is actually right."}, {"start": 1233.0, "text": "Every tag connects to something that the user wrote."}, {"start": 1236.0, "text": "The judge is faithfully following the scoring guide we provided."}, {"start": 1240.0, "text": "We meant something specific by relevant and useful for browsing,"}, {"start": 1245.0, "text": "and the judge interpreted those words differently than we did."}, {"start": 1250.0, "text": "By asking the model to provide judgement for my feature, in my place,"}, {"start": 1255.0, "text": "I expected it to provide a similar score"}, {"start": 1258.0, "text": "to how I would have scored these tags."}, {"start": 1261.0, "text": "When there is a mismatch between the model judge and us,"}, {"start": 1265.0, "text": "we can refine the model judge"}, {"start": 1267.0, "text": "until it can stand in for our own judgement."}, {"start": 1271.0, "text": "Looking back, the problem with our first model judge"}, {"start": 1275.0, "text": "was that it was too broad."}, {"start": 1277.0, "text": "It was asking two different questions."}, {"start": 1280.0, "text": "When you find yourself disagreeing with a score,"}, {"start": 1282.0, "text": "you should try and see if you can split the questions."}, {"start": 1286.0, "text": "In our case, relevance and usefulness are actually two different metrics."}, {"start": 1292.0, "text": "Lets take a look at defining \"Relevance\" as a ScoreDimension."}, {"start": 1297.0, "text": "When we say the tags are relevant we mean that each tag describes a quality,"}, {"start": 1303.0, "text": "theme, or tone of the book itself"}, {"start": 1305.0, "text": "rather than small details or the reader's personal reactions."}, {"start": 1310.0, "text": "And we can write that as the description for our ScoreDimension."}, {"start": 1315.0, "text": "To score these tags, you'd walk through each one."}, {"start": 1319.0, "text": "Identify which tags are bad and which are good,"}, {"start": 1322.0, "text": "based on whether or not they meaningfully describe the book."}, {"start": 1326.0, "text": "You'd repeat this for every tag."}, {"start": 1329.0, "text": "In this case, all of the tags are good,"}, {"start": 1331.0, "text": "which earns a score of 4 on our 1 to 4 scale."}, {"start": 1336.0, "text": "You would repeat the same process to define each scale in the scoring guide."}, {"start": 1341.0, "text": "And that's our \"Relevance\" metric with the metric name, description,"}, {"start": 1346.0, "text": "and scale that the model judge can use."}, {"start": 1349.0, "text": "I can use the same process to define \"Usefulness\"."}, {"start": 1352.0, "text": "Now, I can add both dimensions to the ModelJudgeEvaluator."}, {"start": 1358.0, "text": "But dimensions alone aren't enough."}, {"start": 1361.0, "text": "They tell the judge what to measure, but not how to think about your app."}, {"start": 1365.0, "text": "Without that context, a judge evaluating tags for Book Tracker"}, {"start": 1370.0, "text": "might treat a reader's criticism as a valid book descriptor."}, {"start": 1374.0, "text": "It has no way to know that Book Tracker is a personal library,"}, {"start": 1378.0, "text": "not a review platform."}, {"start": 1381.0, "text": "And that's where the ModelJudgePrompt comes in."}, {"start": 1384.0, "text": "This is an example of a ModelJudgePrompt."}, {"start": 1388.0, "text": "We can tell the judge its evaluating tags"}, {"start": 1391.0, "text": "for a personal library app in the instructions."}, {"start": 1394.0, "text": "Format the response in the evaluationTarget,"}, {"start": 1397.0, "text": "and pass the expectedTags as reference for the model to compare against."}, {"start": 1403.0, "text": "For more details on ModelJudgePrompt please see our documentation."}, {"start": 1408.0, "text": "Now that our model judge has the context it needs,"}, {"start": 1411.0, "text": "lets rerun our evaluation."}, {"start": 1413.0, "text": "In place of Quality we now have a relevance and usefulness score."}, {"start": 1419.0, "text": "And here is the evaluation result of our \"Alice in Wonderland\" book sample."}, {"start": 1424.0, "text": "Notice how the two rationales separate the diagnosis."}, {"start": 1428.0, "text": "Relevance tells us what kind of tag is wrong."}, {"start": 1431.0, "text": "And Usefulness tells us how the wrong tags fail at browsing."}, {"start": 1436.0, "text": "With these results, I now have a clear path forward."}, {"start": 1439.0, "text": "I can update my BookTaggingService instructions,"}, {"start": 1443.0, "text": "run the evaluation again, and watch the scores change."}, {"start": 1446.0, "text": "That's the feedback loop Rob walked us through,"}, {"start": 1449.0, "text": "now powered by qualitative metrics."}, {"start": 1452.0, "text": "When are you uploading to TestFlight?"}, {"start": 1454.0, "text": "Well Rob, I've been a little busy!"}, {"start": 1458.0, "text": "Let's wrap-up with a few best practices for evaluating your apps."}, {"start": 1463.0, "text": "Start small."}, {"start": 1464.0, "text": "A focused dataset of 20 to 30 samples is a great place to get started."}, {"start": 1468.0, "text": "Spec out your app by thinking about how you want the model to behave."}, {"start": 1473.0, "text": "Use heuristics to measure quantifiable traits."}, {"start": 1476.0, "text": "These rule-of-thumb metrics are a great way to start understanding your feature."}, {"start": 1481.0, "text": "The rule-of-thumb is: if you can measure it in code, then it's quantitative."}, {"start": 1486.0, "text": "And if you can only describe it in words, then you need a qualitative metric,"}, {"start": 1491.0, "text": "using a ModelJudgeEvaluator."}, {"start": 1494.0, "text": "Start simple with your model judge."}, {"start": 1496.0, "text": "Define your scoring dimension, run it, and read the rationales."}, {"start": 1500.0, "text": "You'll learn more from a single run than from hours of careful planning."}, {"start": 1505.0, "text": "Use rationales to drive your next change."}, {"start": 1508.0, "text": "If the scores are all the same, your question is too broad."}, {"start": 1512.0, "text": "If you can't isolate the problem, split the dimensions."}, {"start": 1516.0, "text": "And if the judge doesn't understand your app, add context."}, {"start": 1520.0, "text": "Well, I guess we should get back to work."}, {"start": 1523.0, "text": "Be sure to check out our documentation."}, {"start": 1525.0, "text": "And our sample code."}, {"start": 1526.0, "text": "And check out our other videos featuring the Evaluations framework:"}, {"start": 1530.0, "text": "\"Improve your prompts by hill climbing with Evaluations\","}, {"start": 1533.0, "text": "and \"Create robust evaluations for agentic apps\"."}, {"start": 1536.0, "text": "Later!"}, {"start": 1537.0, "text": "Bye!"}]}