{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-335", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/335", "wordCount": 4175, "segments": [{"start": 7.0, "text": "HI! My name is Marcus, a manager on the Evaluations framework team."}, {"start": 12.0, "text": "I'm excited to show you"}, {"start": 14.0, "text": "how to use Evaluations to improve your intelligence-powered feature."}, {"start": 17.0, "text": "As you probably know by now,"}, {"start": 19.0, "text": "using AI in your app is a powerful way"}, {"start": 21.0, "text": "to provide new levels of personalization to your users."}, {"start": 25.0, "text": "This technology can add a level of depth to your app"}, {"start": 27.0, "text": "that was not previously possible with traditional software."}, {"start": 30.0, "text": "However,"}, {"start": 31.0, "text": "it's also a challenge to know whether or not your intelligence-powered feature"}, {"start": 35.0, "text": "behaves as you'd expect in all cases."}, {"start": 38.0, "text": "To help with this, we're releasing the Evaluations framework"}, {"start": 42.0, "text": "to provide you with the tools you need to ship with confidence."}, {"start": 46.0, "text": "Shipping with confidence takes more than just a framework."}, {"start": 50.0, "text": "The Evaluations framework also allows you to hill climb,"}, {"start": 53.0, "text": "which is a process of iteratively improving"}, {"start": 56.0, "text": "the quality of your feature using the scores of your evaluation as a guide."}, {"start": 61.0, "text": "Hill-climbing starts with development,"}, {"start": 64.0, "text": "that is making some change you want to measure against your existing feature."}, {"start": 69.0, "text": "Once all your changes are made,"}, {"start": 70.0, "text": "you then need to run the evaluation."}, {"start": 72.0, "text": "And see if the results have passed your expectations."}, {"start": 76.0, "text": "From there, you analyze the results to understand"}, {"start": 79.0, "text": "how your feature could be further improved."}, {"start": 82.0, "text": "Leveraging the hill-climbing process"}, {"start": 84.0, "text": "is a great way to systematically improve your feature,"}, {"start": 88.0, "text": "but effective hill-climbing takes a little bit more than just following the loop."}, {"start": 92.0, "text": "It also takes a little bit of… Science!"}, {"start": 95.0, "text": "So, in this video,"}, {"start": 97.0, "text": "I'll walk you through how to improve prompts"}, {"start": 99.0, "text": "by following the hill-climbing loop,"}, {"start": 101.0, "text": "while incorporating some scientific thinking along the way."}, {"start": 105.0, "text": "Next, I'll walk you through how to conduct comparative Evaluations"}, {"start": 108.0, "text": "to make the process of hill-climbing easier."}, {"start": 112.0, "text": "And finally, we'll go beyond changing prompts,"}, {"start": 115.0, "text": "by improving other aspects of an intelligence-powered feature."}, {"start": 119.0, "text": "But, before we go any further,"}, {"start": 121.0, "text": "this video is about the process of hill-climbing an existing evaluation."}, {"start": 125.0, "text": "That means you've already written the foundations of an evaluation pipeline,"}, {"start": 130.0, "text": "which provides a wholistic understanding of the strengths"}, {"start": 132.0, "text": "and weaknesses of your intelligence-powered feature."}, {"start": 136.0, "text": "If you're unfamiliar with how to do that,"}, {"start": 138.0, "text": "go check out our other video,"}, {"start": 140.0, "text": "\"Meet the Evaluations framework\"."}, {"start": 142.0, "text": "It covers everything you need to know"}, {"start": 144.0, "text": "in order to build a great evaluation pipeline."}, {"start": 148.0, "text": "With that covered, let's get started."}, {"start": 151.0, "text": "In the \"Meet the Evaluations framework\" video we introduced you to Book Tracker."}, {"start": 155.0, "text": "In case you forgot, Book Tracker allows readers to catalog and review books."}, {"start": 160.0, "text": "Recently, I've been reading a lot of classics,"}, {"start": 163.0, "text": "so I've added the books to my catalog."}, {"start": 166.0, "text": "In fact, I just finished reading \"Treasure Island\"!"}, {"start": 169.0, "text": "It's such a thought-provoking read"}, {"start": 171.0, "text": "that covers the tension between loyalty and betrayal."}, {"start": 174.0, "text": "One of Book Tracker's new features is a tagging service,"}, {"start": 178.0, "text": "which uses a model to generate tags based on the readers review."}, {"start": 182.0, "text": "While the tags for this review cover the general themes of the book,"}, {"start": 186.0, "text": "I feel like something is missing."}, {"start": 189.0, "text": "I would have expected to see tags like \"tense\" or \"morally grey\","}, {"start": 193.0, "text": "which speak to the themes of the story."}, {"start": 196.0, "text": "The tags generated for \"Little Women\" had a similar problem."}, {"start": 200.0, "text": "Tags like poignant has more to do with how the reader felt"}, {"start": 203.0, "text": "than the contents of the book."}, {"start": 205.0, "text": "Emotion in book reviews is great,"}, {"start": 207.0, "text": "but it should not make it into the list of tags."}, {"start": 211.0, "text": "Also, a tag like quiet-steadiness,"}, {"start": 213.0, "text": "which is pulled directly from the review,"}, {"start": 216.0, "text": "isn't going to be a very useful tag when I want to search my library later."}, {"start": 221.0, "text": "It seems like Book Tracker's tag generator"}, {"start": 223.0, "text": "isn't as good as I think it could be."}, {"start": 226.0, "text": "Fortunately, when my colleagues wrote this feature,"}, {"start": 229.0, "text": "they wrote an Evaluation,"}, {"start": 231.0, "text": "which measures the tags against a set of criteria."}, {"start": 234.0, "text": "And here is my Evaluation for Book Tracker's book tags."}, {"start": 238.0, "text": "I'm particularly interested in how we are judging the quality of my tags"}, {"start": 242.0, "text": "so I'll scroll down and take a look."}, {"start": 245.0, "text": "The qualitative aspects of my app"}, {"start": 247.0, "text": "are captured by the score dimensions type."}, {"start": 250.0, "text": "Relevance tracks the how well the tags represent information"}, {"start": 254.0, "text": "about the book's plot,"}, {"start": 255.0, "text": "theme, or other relevant information."}, {"start": 258.0, "text": "Usefulness, measures how good the tags are as search terms."}, {"start": 264.0, "text": "The ModelJudgeEvaluator uses the score dimensions"}, {"start": 267.0, "text": "and a prompt to generate a score for each set of tags."}, {"start": 272.0, "text": "My plan is to add these two books to my Evaluation,"}, {"start": 276.0, "text": "and review the tags I get back."}, {"start": 279.0, "text": "It will also be a good opportunity to see how my ratings stack up"}, {"start": 283.0, "text": "against the ratings of my model judge."}, {"start": 286.0, "text": "Wanting to improve some part of an intelligence powered feature"}, {"start": 290.0, "text": "is how the hill-climbing process starts."}, {"start": 292.0, "text": "So, to start the loop you begin in the development phase."}, {"start": 297.0, "text": "Here, you make any changes you need to feature as well as your Evaluation."}, {"start": 303.0, "text": "In this case,"}, {"start": 304.0, "text": "I'll add my review of \"Treasure Island\" to my Evaluation dataset."}, {"start": 308.0, "text": "And I've done the same thing for \"Little Women\"."}, {"start": 311.0, "text": "With those two entries now in my dataset,"}, {"start": 314.0, "text": "I now want to run my Evaluation"}, {"start": 316.0, "text": "and see what tags the model produces"}, {"start": 318.0, "text": "and how the judge scores them."}, {"start": 320.0, "text": "But, to get there we need to first ask if the Evaluation we ran"}, {"start": 325.0, "text": "met our expectations."}, {"start": 328.0, "text": "As a reminder,"}, {"start": 329.0, "text": "your expectations can be defined using Swift Testing's expect macro."}, {"start": 334.0, "text": "That way, you can tell if your expectations are met"}, {"start": 337.0, "text": "by whether or not your tests pass."}, {"start": 340.0, "text": "In this case, my Evaluation met all of my expectations;"}, {"start": 344.0, "text": "however, because I know the tags aren't as good as I'd like them to be,"}, {"start": 349.0, "text": "I need to investigate further."}, {"start": 351.0, "text": "And that brings us to the analyze phase."}, {"start": 355.0, "text": "Xcode's new evaluations report gives me in depth information"}, {"start": 359.0, "text": "about my last Evaluation run."}, {"start": 362.0, "text": "To see more details I can click on my BookTaggingEvaluation run."}, {"start": 367.0, "text": "This brings up the Evaluation detail view."}, {"start": 370.0, "text": "On the top are our aggregate metric charts."}, {"start": 374.0, "text": "And below is our table of results."}, {"start": 377.0, "text": "What I'd like to do now is compare the response of the model"}, {"start": 381.0, "text": "to the list of expected tags I generated before."}, {"start": 384.0, "text": "I can do that by opening up the Assistant Editor."}, {"start": 388.0, "text": "And now I can see detailed information"}, {"start": 391.0, "text": "about the tags generated for each item in my dataset."}, {"start": 395.0, "text": "I want to focus on the difference between what the model generated"}, {"start": 399.0, "text": "and what I expected."}, {"start": 401.0, "text": "I can review that in detail in this table."}, {"start": 404.0, "text": "The collection of terms isn't bad"}, {"start": 407.0, "text": "but it leaves out a number of key details"}, {"start": 409.0, "text": "from the story which the user might want to search for."}, {"start": 413.0, "text": "Because of that, I would have scored these tags a 4 for relevance"}, {"start": 417.0, "text": "and a 2 for usefulness."}, {"start": 420.0, "text": "My model judge also gave the tags a relevance score of 4 which is great,"}, {"start": 425.0, "text": "but it also gave usefulness a score of 4, which isn't right."}, {"start": 430.0, "text": "I should see if I feel the same way about the tags for my review of \"Little Women\"."}, {"start": 436.0, "text": "The tags don't really contain all the useful information that I'd expect."}, {"start": 442.0, "text": "And it turns out I disagree with the judge's scores here as well."}, {"start": 447.0, "text": "Once again, I think relevance should be a 4"}, {"start": 450.0, "text": "and usefulness should be a 2."}, {"start": 453.0, "text": "Doing this analysis has made it clear to me"}, {"start": 455.0, "text": "that there is a discrepancy between how my model judge and I rate tags."}, {"start": 461.0, "text": "This discrepancy between model and human is known as drift,"}, {"start": 465.0, "text": "and it is a problem faced by all developers"}, {"start": 467.0, "text": "trying to evaluate intelligent features."}, {"start": 469.0, "text": "Here's why."}, {"start": 471.0, "text": "Say I have an evaluation with 10 samples."}, {"start": 474.0, "text": "I then ask a model judge and a person to rate each sample."}, {"start": 479.0, "text": "The model and person then give their ratings on a scale from 1 to 4,"}, {"start": 484.0, "text": "and at the end we average those scores to build an aggregate."}, {"start": 488.0, "text": "If the model and the human tend to disagree in their ratings,"}, {"start": 492.0, "text": "then their average scores will diverge from one another, hence the name drift."}, {"start": 497.0, "text": "As your data set continues to grow and grow"}, {"start": 500.0, "text": "the drift will get wider and wider."}, {"start": 502.0, "text": "At which point, it'll be hard for you to know whether or not"}, {"start": 505.0, "text": "your feature is being properly evaluated."}, {"start": 508.0, "text": "To help with this,"}, {"start": 509.0, "text": "you can align your judge to a person's expert opinion."}, {"start": 513.0, "text": "Now that we know drift is a problem,"}, {"start": 515.0, "text": "we need a way to know how much"}, {"start": 517.0, "text": "our model judge has drifted from our expert ratings."}, {"start": 521.0, "text": "One way to accomplish this"}, {"start": 522.0, "text": "would be to line up the ratings of the expert"}, {"start": 524.0, "text": "and mark where the two match."}, {"start": 526.0, "text": "You can then use this to generate a percentage."}, {"start": 529.0, "text": "This percentage is called accuracy,"}, {"start": 532.0, "text": "and it is a great way to measure alignment"}, {"start": 534.0, "text": "if every value in your scoring scale is equally likely to appear."}, {"start": 539.0, "text": "However, it's more likely that your dataset will contain values"}, {"start": 543.0, "text": "that have an uneven distribution of scores."}, {"start": 546.0, "text": "Think about it, datasets often contain examples of high quality output."}, {"start": 551.0, "text": "Therefore it is often the case that a human rater"}, {"start": 554.0, "text": "is likely to rate items in the dataset with higher scores."}, {"start": 559.0, "text": "If a model then happens to judge your smaller dataset"}, {"start": 562.0, "text": "with high scores,"}, {"start": 563.0, "text": "it may seem like the two are aligned."}, {"start": 565.0, "text": "But then when unleashed on a larger dataset"}, {"start": 568.0, "text": "with more variations in scores,"}, {"start": 570.0, "text": "it's tendency to score high will still result in drift."}, {"start": 574.0, "text": "So we need an alternative to accuracy,"}, {"start": 577.0, "text": "one that accounts for the weighted nature of our dataset"}, {"start": 580.0, "text": "and the chance that the model might guess the right answer."}, {"start": 583.0, "text": "Fortunately there is a solution!"}, {"start": 587.0, "text": "Cohen's kappa coefficient is a mathematical formula"}, {"start": 590.0, "text": "made popular by statistician and psychologist Jacob Cohen in 1960."}, {"start": 595.0, "text": "Cohen's kappa measures alignment,"}, {"start": 597.0, "text": "that is how often do two raters agree."}, {"start": 601.0, "text": "To do that, we need to know what percentage of the time the raters agreed,"}, {"start": 606.0, "text": "better known as accuracy."}, {"start": 608.0, "text": "And this is exactly what the accuracy metric from before was calculating."}, {"start": 613.0, "text": "But now we need to calculate a new value."}, {"start": 616.0, "text": "Coincidence, which represents the chance that one rater"}, {"start": 620.0, "text": "might get lucky and happen to align."}, {"start": 623.0, "text": "This luck is then weighted based on the chances"}, {"start": 626.0, "text": "certain answers are more likely to appear."}, {"start": 628.0, "text": "So now the question is, how do you calculate it?"}, {"start": 631.0, "text": "To calculate alignment,"}, {"start": 633.0, "text": "we start with our accuracy score."}, {"start": 636.0, "text": "From the accuracy score"}, {"start": 638.0, "text": "we subtract the possibility of two raters randomly agreeing."}, {"start": 642.0, "text": "Finally, we divide the difference by the inverse of random agreement,"}, {"start": 647.0, "text": "namely the chance that the two raters intentionally agreed."}, {"start": 652.0, "text": "The result of that gives us alignment."}, {"start": 655.0, "text": "Cohen's kappa is a powerful way to measure the alignment"}, {"start": 658.0, "text": "between a model judge and your expert opinion."}, {"start": 661.0, "text": "I can use this to hill climb the alignment scores"}, {"start": 664.0, "text": "between me and my model judge."}, {"start": 666.0, "text": "So now, we start back at the beginning of hill-climbing loop"}, {"start": 670.0, "text": "in the develop phase."}, {"start": 672.0, "text": "To do this, I am going to set up an evaluation"}, {"start": 675.0, "text": "to compare my ratings against my judge and produce and alignment score."}, {"start": 680.0, "text": "To do that, I need to write an evaluation,"}, {"start": 682.0, "text": "which is made up of four components."}, {"start": 685.0, "text": "First is my dataset."}, {"start": 687.0, "text": "Then the subject of my evaluation."}, {"start": 689.0, "text": "Then, I need to define my evaluators."}, {"start": 692.0, "text": "And finally, I need to aggregate my results."}, {"start": 695.0, "text": "So let's start with the dataset."}, {"start": 698.0, "text": "For this evaluation to work properly,"}, {"start": 700.0, "text": "both my model judge and I need to evaluate the exact same dataset."}, {"start": 705.0, "text": "In this case the model judge reviews tags,"}, {"start": 708.0, "text": "so I need to produce a common set of tags"}, {"start": 711.0, "text": "for the judge and I to review."}, {"start": 713.0, "text": "And I have just the perfect dataset."}, {"start": 715.0, "text": "My evaluation from before"}, {"start": 717.0, "text": "contains a collection of reviews and tags."}, {"start": 721.0, "text": "Because I ran this evaluation in a test,"}, {"start": 723.0, "text": "Xcode generated an attachment"}, {"start": 725.0, "text": "containing all the of evaluation data that was generated."}, {"start": 729.0, "text": "I can retrieve that attachment and extract summary and tag pairs."}, {"start": 734.0, "text": "Now, with the summary and tag pairs extracted,"}, {"start": 737.0, "text": "I need to add my ratings."}, {"start": 740.0, "text": "After that,"}, {"start": 741.0, "text": "I can pass the contents of this file as the input to my evaluation."}, {"start": 746.0, "text": "Next, I need to capture the subject of my evaluation."}, {"start": 750.0, "text": "Normally, the subject method is for calling API"}, {"start": 754.0, "text": "related to your feature,"}, {"start": 755.0, "text": "but since the generated model responses are part of our dataset,"}, {"start": 759.0, "text": "we can simply return the already generated tags."}, {"start": 763.0, "text": "Now, I need to define my evaluators."}, {"start": 767.0, "text": "As you might have guessed,"}, {"start": 768.0, "text": "my evaluator is the exact same model judge evaluator"}, {"start": 772.0, "text": "as in our book tags evaluation."}, {"start": 774.0, "text": "This is where the judge provides its rating."}, {"start": 777.0, "text": "Finally, I need to aggregate my results."}, {"start": 781.0, "text": "Here is where we compare my ratings against the judge's."}, {"start": 785.0, "text": "To do that, we need to calculate Cohen's kappa,"}, {"start": 788.0, "text": "which I can do that with a custom aggregation method."}, {"start": 792.0, "text": "In addition to just Cohen's kappa,"}, {"start": 794.0, "text": "I'll also calculate the mean"}, {"start": 795.0, "text": "and standard deviation of each score dimension."}, {"start": 799.0, "text": "This will be helpful to know if the scores of the judge"}, {"start": 801.0, "text": "are going up or down."}, {"start": 804.0, "text": "Now, I can setup my test with my evaluation."}, {"start": 807.0, "text": "For this test, I've set an expectation"}, {"start": 810.0, "text": "that my ratings and the judges ratings"}, {"start": 813.0, "text": "should produce an alignment score of 0.6."}, {"start": 816.0, "text": "We've chosen this number because according to statisticians,"}, {"start": 819.0, "text": "an alignment score of 0.6 represents a meaningful level of agreement."}, {"start": 824.0, "text": "Now, it's time to evaluate"}, {"start": 826.0, "text": "and get a baseline for our alignment."}, {"start": 829.0, "text": "And then determine if my evaluation has passed my expectations."}, {"start": 834.0, "text": "It appears that the tests failed,"}, {"start": 836.0, "text": "which means my expectations weren't met."}, {"start": 839.0, "text": "So once again, it's time to analyze the results in detail."}, {"start": 843.0, "text": "I now know that my alignment scores didn't match my expectations."}, {"start": 848.0, "text": "I can now go to the evaluation report to get more information."}, {"start": 852.0, "text": "As I expected, the scores for both usefulness"}, {"start": 856.0, "text": "and relevance are quite low,"}, {"start": 858.0, "text": "meaning my model judge and I aren't aligned."}, {"start": 861.0, "text": "Now, I want to get more information"}, {"start": 864.0, "text": "about how each sample in my dataset performed."}, {"start": 867.0, "text": "To do that, I need to open the assistant"}, {"start": 869.0, "text": "and view the results in detail."}, {"start": 872.0, "text": "As I scanned through the results,"}, {"start": 874.0, "text": "this review of \"Frankenstein\" caught my eye."}, {"start": 877.0, "text": "I can see a pretty large discrepancy between my rating of the tags"}, {"start": 881.0, "text": "and the judge's."}, {"start": 883.0, "text": "It seems like our judge thinks tags like self-help"}, {"start": 886.0, "text": "and self-improvement are relevant to the story."}, {"start": 889.0, "text": "Also psychological is an okay search term,"}, {"start": 893.0, "text": "but probably not a term a user is likely to search for."}, {"start": 897.0, "text": "I then started looking through other items in my dataset"}, {"start": 900.0, "text": "that had a similar problem"}, {"start": 901.0, "text": "and came across this review of \"The Ramakien\"."}, {"start": 905.0, "text": "The judge and I agree that these collection of terms are helpful"}, {"start": 909.0, "text": "and relevant to the contents of the book."}, {"start": 911.0, "text": "Where we disagree is on usefulness."}, {"start": 914.0, "text": "Terms like visual-dimension and quaint-dignity are way too specific."}, {"start": 920.0, "text": "So what's the problem here?"}, {"start": 922.0, "text": "I believe the model doesn't have enough knowledge"}, {"start": 925.0, "text": "on it's own to distinguish between a good tag and bad one."}, {"start": 930.0, "text": "That's likely because the prompt of my judge doesn't provide enough context."}, {"start": 935.0, "text": "To do that, I need to develop a new prompt."}, {"start": 939.0, "text": "That way I can compare the alignment scores of my current prompt"}, {"start": 942.0, "text": "against my scores of my new prompt."}, {"start": 945.0, "text": "Fortunately, in Xcode 27,"}, {"start": 947.0, "text": "we've made so you can compare the results of two evaluations against each other."}, {"start": 952.0, "text": "When doing comparisons,"}, {"start": 953.0, "text": "some scientific thinking can go a long way."}, {"start": 957.0, "text": "In a science experiment, you have two groups."}, {"start": 959.0, "text": "The control group, which represents the baseline"}, {"start": 963.0, "text": "and the experimental group"}, {"start": 964.0, "text": "which represents the change we are trying to compare against."}, {"start": 968.0, "text": "We can think of the two versions of our instructions in the same way,"}, {"start": 972.0, "text": "where the control group is represented by our base prompt"}, {"start": 975.0, "text": "and our experimental group is represented by our newly changed prompt."}, {"start": 979.0, "text": "I now need to create a second version of our evaluation"}, {"start": 983.0, "text": "with an experimental prompt."}, {"start": 985.0, "text": "For our baseline,"}, {"start": 986.0, "text": "we will use the same evaluation with the same model judge prompt as before."}, {"start": 991.0, "text": "For our experimental prompt,"}, {"start": 993.0, "text": "I've written a more thorough description"}, {"start": 995.0, "text": "about how to judge the set of tags."}, {"start": 998.0, "text": "It starts by providing the judge context about the app"}, {"start": 1002.0, "text": "and what it's about to be judging."}, {"start": 1004.0, "text": "Then it gives examples of good tags."}, {"start": 1008.0, "text": "As well as ways to identify bad tags."}, {"start": 1011.0, "text": "With both prompts written,"}, {"start": 1013.0, "text": "I can add both evaluations to a test suite,"}, {"start": 1016.0, "text": "which will run both evaluations."}, {"start": 1018.0, "text": "So, I'll run that suite now and compare the results."}, {"start": 1023.0, "text": "With the evaluation now finished"}, {"start": 1025.0, "text": "I can return to the evaluation report."}, {"start": 1028.0, "text": "Looks like my alignment scores for relevance improved."}, {"start": 1031.0, "text": "While my alignment score for usefulness dropped considerably."}, {"start": 1035.0, "text": "Balancing tradeoffs like this are tricky"}, {"start": 1038.0, "text": "so I need to think carefully how to proceed."}, {"start": 1041.0, "text": "But before in depth analysis comes checking if we passed."}, {"start": 1045.0, "text": "And my test confirms the obvious, we haven't."}, {"start": 1049.0, "text": "After thinking about it further,"}, {"start": 1051.0, "text": "I am going to keep this prompt change"}, {"start": 1053.0, "text": "and focus the next round of iteration on improving my usefulness score."}, {"start": 1057.0, "text": "Therefore, the most effective way to review my results"}, {"start": 1060.0, "text": "is to compare the usefulness scores of both judges against one another."}, {"start": 1065.0, "text": "To do that, I can use the new comparison view in the evaluation report."}, {"start": 1070.0, "text": "From the evaluation report"}, {"start": 1072.0, "text": "I can open the comparison button"}, {"start": 1074.0, "text": "and open my baseline evaluation."}, {"start": 1077.0, "text": "Here, I can review the scores of the two prompts side by side."}, {"start": 1082.0, "text": "One thing that jumped out to me immediately"}, {"start": 1084.0, "text": "is the discrepancy between usefulness scores"}, {"start": 1086.0, "text": "of this review of \"Picture of Dorian Gray\"."}, {"start": 1089.0, "text": "It seems to me that the model may be judging too harshly on usefulness."}, {"start": 1094.0, "text": "The usefulness column of the experimental evaluation seems to corroborate my guess."}, {"start": 1100.0, "text": "I noticed that all the scores are either a 3 or 2,"}, {"start": 1104.0, "text": "which is way too harsh."}, {"start": 1106.0, "text": "I think what could help here"}, {"start": 1108.0, "text": "is being more specific about how to grade each scoring dimension."}, {"start": 1113.0, "text": "To do that, I'll need to make some changes to my experimental evaluation."}, {"start": 1118.0, "text": "But before I can make changes to the experimental evaluation,"}, {"start": 1121.0, "text": "I applied the new prompt"}, {"start": 1123.0, "text": "from my experimental evaluation into my baseline."}, {"start": 1126.0, "text": "This ensures there's only one different variable."}, {"start": 1130.0, "text": "Namely, the changes to my scoring dimensions."}, {"start": 1133.0, "text": "For relevance, I've provided a slightly longer description"}, {"start": 1137.0, "text": "which emphasizes the need for a genre tag."}, {"start": 1140.0, "text": "And here is the one for usefulness."}, {"start": 1142.0, "text": "Which emphasizes being more critical of overly specific tags."}, {"start": 1147.0, "text": "And once again I'll wait for my evaluation to run."}, {"start": 1151.0, "text": "And the scores both improved greatly over the baseline."}, {"start": 1155.0, "text": "It looks like these specific scoring dimensions"}, {"start": 1157.0, "text": "are going to be a lot more helpful."}, {"start": 1160.0, "text": "But, we've still not quite hit our alignment goals."}, {"start": 1163.0, "text": "So now, I need to do another comparison"}, {"start": 1166.0, "text": "to see where we might be able to make improvements."}, {"start": 1169.0, "text": "To analyze further, I've gone back to the experimental evaluation."}, {"start": 1174.0, "text": "I want to review the results in detail so I'll bring up the assistant view."}, {"start": 1178.0, "text": "Thumbing through the results brought me to the review of \"Moby Dick\"."}, {"start": 1182.0, "text": "My relevance score is starting to align."}, {"start": 1186.0, "text": "But my usefulness score could still use some work."}, {"start": 1190.0, "text": "While some results are looking promising, others are still way off."}, {"start": 1194.0, "text": "This review of \"Frankenstein\" continues to give our judge trouble."}, {"start": 1198.0, "text": "What I think our judge needs now is some examples of the way I judge things,"}, {"start": 1204.0, "text": "which should give it a pattern for how to judge according to my scale."}, {"start": 1208.0, "text": "That means we need another round of hill-climbing."}, {"start": 1212.0, "text": "I've already added the new score dimensions to my baseline evaluation."}, {"start": 1216.0, "text": "Now, I've reworked my main judge prompt"}, {"start": 1219.0, "text": "to give it more detail about the goal"}, {"start": 1221.0, "text": "of the tag generation feature"}, {"start": 1223.0, "text": "to help ground the model in the problem space."}, {"start": 1226.0, "text": "From there, I've written out a number of examples"}, {"start": 1229.0, "text": "for the model to use as a guideline for reviewing."}, {"start": 1232.0, "text": "I've made sure to only give the model a few examples."}, {"start": 1236.0, "text": "By giving it a longer list"}, {"start": 1237.0, "text": "I am prone to overfit the alignment score,"}, {"start": 1240.0, "text": "which would make it hard to tell"}, {"start": 1241.0, "text": "if my judge is actually aligned with me."}, {"start": 1244.0, "text": "Now that it's a fair comparison,"}, {"start": 1246.0, "text": "I need to run my evaluation and view the results."}, {"start": 1249.0, "text": "And now finally my scores are over my expected value!"}, {"start": 1253.0, "text": "Which means I've finally passed"}, {"start": 1255.0, "text": "and can exit out of the loop!"}, {"start": 1258.0, "text": "This now means I can be confident that when my model judge provides ratings,"}, {"start": 1263.0, "text": "I can confidently say that the tags are good or bad according my standards."}, {"start": 1268.0, "text": "That means I can now put my judge to work"}, {"start": 1271.0, "text": "on evaluating Book Tracker's Book Tagging Service."}, {"start": 1274.0, "text": "So far we've seen how to hill climb on prompts,"}, {"start": 1277.0, "text": "making them incrementally better and better,"}, {"start": 1280.0, "text": "now I'd like to show you how to improve your feature"}, {"start": 1283.0, "text": "through something other than your prompts."}, {"start": 1286.0, "text": "To generate its tags,"}, {"start": 1287.0, "text": "Book Tracker uses the on-device model."}, {"start": 1290.0, "text": "We use it because readers tend to be in all kinds of places when cataloging books,"}, {"start": 1295.0, "text": "so using the on-device model ensures they can generate tags"}, {"start": 1299.0, "text": "no matter where they are."}, {"start": 1301.0, "text": "What I want to do is give the model some more context about the book"}, {"start": 1305.0, "text": "it's generating tags for."}, {"start": 1308.0, "text": "I think the additional context will help the model generate more relevant"}, {"start": 1312.0, "text": "and useful tags."}, {"start": 1314.0, "text": "Better still, Book Tracker already has the data needed for this"}, {"start": 1318.0, "text": "because we store the author's name"}, {"start": 1320.0, "text": "and book title when they write their review."}, {"start": 1323.0, "text": "So, to help the tag generator,"}, {"start": 1325.0, "text": "I've created a tool to get additional information on the book,"}, {"start": 1329.0, "text": "which provides the book title and author if they are available."}, {"start": 1333.0, "text": "Adding this tool is a form of hill-climbing"}, {"start": 1336.0, "text": "because we are attempting to improve the quality of our feature"}, {"start": 1339.0, "text": "through an incremental change."}, {"start": 1341.0, "text": "For this evaluation we will use the book tagging Evaluation,"}, {"start": 1345.0, "text": "now with an improved model judge."}, {"start": 1348.0, "text": "But I need a way to compare the quality of my feature"}, {"start": 1351.0, "text": "without the tool to the quality of my feature with it."}, {"start": 1354.0, "text": "So to do that, I'll need to make a change to Book Tagging Service"}, {"start": 1358.0, "text": "BookTaggingService now takes a list of tools as input."}, {"start": 1363.0, "text": "I also set the default to an empty array"}, {"start": 1365.0, "text": "so my existing evaluation won't need any changes."}, {"start": 1369.0, "text": "But now I need to write a new evaluation"}, {"start": 1372.0, "text": "to compare the service with the tool to the service without the tool."}, {"start": 1376.0, "text": "Here is the new evaluation I wrote."}, {"start": 1378.0, "text": "It's exactly the same as the other evaluation."}, {"start": 1382.0, "text": "The only difference is I now pass my new lookup tool in the tools array."}, {"start": 1387.0, "text": "So all I have to do is define two instances of my evaluation."}, {"start": 1392.0, "text": "One without the tool"}, {"start": 1393.0, "text": "and one with it."}, {"start": 1396.0, "text": "And now, let's evaluate it"}, {"start": 1398.0, "text": "and determine if I'm ready to ship."}, {"start": 1401.0, "text": "Well, my service which uses tools met all my expectations,"}, {"start": 1406.0, "text": "so things are looking good."}, {"start": 1408.0, "text": "But, my dataset for Book Tracker contains only 13 book and review pairs,"}, {"start": 1414.0, "text": "that doesn't cover the wide variety of books and reviews"}, {"start": 1417.0, "text": "a user might submit for tagging."}, {"start": 1420.0, "text": "In addition,"}, {"start": 1421.0, "text": "I was looking at the results of the evaluation of my service"}, {"start": 1424.0, "text": "with the tool."}, {"start": 1425.0, "text": "I can see that the service with the tool is performing better,"}, {"start": 1429.0, "text": "however it does seem like my tool"}, {"start": 1431.0, "text": "isn't being called in all the places I think it needs to."}, {"start": 1435.0, "text": "What I really need is a way to tell whether or not"}, {"start": 1437.0, "text": "my tool has been called in the right situations."}, {"start": 1441.0, "text": "Fortunately, the Evaluations framework can help with both of those problems."}, {"start": 1445.0, "text": "To learn more about our APIs for evaluating tool usage"}, {"start": 1449.0, "text": "and generating comprehensive datasets,"}, {"start": 1451.0, "text": "take a look at the \"Create robust evaluations for agentic apps\" video."}, {"start": 1456.0, "text": "There, you'll learn about tool call Evaluators"}, {"start": 1459.0, "text": "and how to use the Sample Generator API"}, {"start": 1461.0, "text": "to test the wide variety of uses cases your app might see."}, {"start": 1465.0, "text": "Before we wrap up, I'd like to recap what we covered today."}, {"start": 1469.0, "text": "Hill-climbing works best when you focus on making one change at a time."}, {"start": 1473.0, "text": "To do this, treat every iteration of the loop like a science experiment."}, {"start": 1478.0, "text": "Being able to isolate your changes will help you to understand"}, {"start": 1482.0, "text": "how each part of your feature contributes to the overall quality."}, {"start": 1486.0, "text": "Knowing how each part works individually"}, {"start": 1488.0, "text": "will also help you to know where you might need to make changes"}, {"start": 1491.0, "text": "to resolve a bug or unwanted pattern later down the line."}, {"start": 1496.0, "text": "Second, this process takes time."}, {"start": 1499.0, "text": "Not every change you make will result in positive change."}, {"start": 1503.0, "text": "However, failed experiments tell you just as much as successful ones."}, {"start": 1508.0, "text": "Third, good experiments require creativity."}, {"start": 1512.0, "text": "In an intelligent feature there are so many things you can change."}, {"start": 1517.0, "text": "In your feature you can change"}, {"start": 1519.0, "text": "the instructions,"}, {"start": 1520.0, "text": "the tools,"}, {"start": 1522.0, "text": "as well as the model or models you use to generate responses."}, {"start": 1527.0, "text": "On the evaluation side you can change the dataset,"}, {"start": 1530.0, "text": "aggregation methods,"}, {"start": 1532.0, "text": "and even the evaluators themselves."}, {"start": 1535.0, "text": "Everything is fair game."}, {"start": 1536.0, "text": "Make sure to consider all of these"}, {"start": 1538.0, "text": "when thinking about how to hill climb."}, {"start": 1541.0, "text": "Finally, watch out for drift."}, {"start": 1544.0, "text": "It can feel a bit meta to evaluate your evaluators"}, {"start": 1547.0, "text": "but a well tuned model evaluator will save you time in the long run."}, {"start": 1552.0, "text": "Models can generate ratings much faster than humans can."}, {"start": 1555.0, "text": "So by keeping them aligned,"}, {"start": 1557.0, "text": "you get useful signal as your dataset grows to cover more and more use cases."}, {"start": 1563.0, "text": "If you want to learn more about what we've covered here today,"}, {"start": 1566.0, "text": "you can review the Book Tracker app I've been using"}, {"start": 1569.0, "text": "as well as the evaluations for aligning the model judge."}, {"start": 1573.0, "text": "You can also get a comprehensive rundown of all our new APIs"}, {"start": 1577.0, "text": "on the developer documentation website."}, {"start": 1579.0, "text": "Thank you for taking the time to learn about how to improve"}, {"start": 1582.0, "text": "your evaluation scores by hill-climbing."}, {"start": 1585.0, "text": "Your dedication will pay off"}, {"start": 1587.0, "text": "as you deliver high quality experiences to your users."}, {"start": 1591.0, "text": "Thanks for watching and happy hill-climbing!"}]}