{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-334", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/334", "wordCount": 2605, "segments": [{"start": 7.0, "text": "Hi! I'm Eric Gourlaouen,"}, {"start": 9.0, "text": "an engineer on the Foundation Models Framework team."}, {"start": 12.0, "text": "Today, I'd like to introduce new ways you can leverage the Apple Foundation Models"}, {"start": 16.0, "text": "on macOS."}, {"start": 18.0, "text": "At WWDC25, we introduced the Foundation Models Framework in Swift."}, {"start": 23.0, "text": "You can use it to prompt the on-device Apple Foundation Model in your apps."}, {"start": 27.0, "text": "It was introduced along with features like guided generation,"}, {"start": 30.0, "text": "to generate structured outputs, and Tool Calling,"}, {"start": 34.0, "text": "to let the model interact with the context of your app."}, {"start": 38.0, "text": "With macOS 27 and iOS 27 come a number of new features to the framework."}, {"start": 44.0, "text": "Like support for passing images in your prompt."}, {"start": 46.0, "text": "And access to server models,"}, {"start": 48.0, "text": "so that your app can leverage any large language model with the same Swift API."}, {"start": 53.0, "text": "Using the Foundation Models Framework"}, {"start": 55.0, "text": "lets you easily tap into the power of Apple Foundation Models."}, {"start": 59.0, "text": "You can use those models on problems from just text extraction and analysis"}, {"start": 63.0, "text": "up to building advanced agentic workflows."}, {"start": 66.0, "text": "And it's easy to set up, with no API key needed and no cloud API costs."}, {"start": 72.0, "text": "But until now, those models were only available from Swift code."}, {"start": 76.0, "text": "This year,"}, {"start": 77.0, "text": "we're introducing new ways you can access Apple Foundation Models on macOS."}, {"start": 82.0, "text": "We're introducing a new command line tool called fm,"}, {"start": 85.0, "text": "and a new Foundation Models SDK for Python."}, {"start": 88.0, "text": "The fm command line tool comes pre-installed with macOS 27."}, {"start": 92.0, "text": "It's a fantastic tool to quickly test prompts,"}, {"start": 95.0, "text": "right from a terminal, or to incorporate it in automation."}, {"start": 99.0, "text": "It makes it really easy to test the model with some prompts"}, {"start": 102.0, "text": "without rebuilding your project in Xcode."}, {"start": 105.0, "text": "Using this command line tool is as easy as opening a terminal window,"}, {"start": 109.0, "text": "typing fm respond, typing my prompt,"}, {"start": 119.0, "text": "and pressing enter."}, {"start": 121.0, "text": "And after a bit, I'll see the response from the model."}, {"start": 127.0, "text": "The Foundation Models SDK for Python"}, {"start": 129.0, "text": "is our other new way to access the on-device model."}, {"start": 132.0, "text": "It supports the Foundation Models Framework's core features,"}, {"start": 135.0, "text": "like tool calling and guided generation."}, {"start": 138.0, "text": "If you're a Swift developer and you've used the Foundation Models framework,"}, {"start": 142.0, "text": "you'll find the API very familiar to you."}, {"start": 144.0, "text": "And if you're a machine learning engineer, you might use more Python than Swift."}, {"start": 149.0, "text": "In that case,"}, {"start": 150.0, "text": "using this SDK makes it easy to use the on-device model in your Python code."}, {"start": 155.0, "text": "Python has a rich ecosystem of open-source packages"}, {"start": 158.0, "text": "for machine learning and data science."}, {"start": 161.0, "text": "With the Python SDK, you can write evaluation pipelines in Python"}, {"start": 165.0, "text": "and leverage those packages to quantify the quality of your feature."}, {"start": 168.0, "text": "And because Python is a scripting language,"}, {"start": 171.0, "text": "it makes it easy to quickly test prompts, see results, and iterate."}, {"start": 176.0, "text": "Let's dive into what's possible with those new options."}, {"start": 179.0, "text": "I'll start by going over the new fm command line tool."}, {"start": 182.0, "text": "We'll go over the basics of using it,"}, {"start": 184.0, "text": "and then I'll show you how you can use it to create an automation script."}, {"start": 187.0, "text": "Next, I'll introduce the Python SDK."}, {"start": 190.0, "text": "I'll show you how to interact with the model,"}, {"start": 192.0, "text": "and then how to leverage advanced SDK options."}, {"start": 195.0, "text": "I'll then show you through a case study"}, {"start": 197.0, "text": "how you can use Python tools to analyze your prompt outputs,"}, {"start": 200.0, "text": "and improve the quality of your app."}, {"start": 203.0, "text": "Let's start by discussing the command line tool fm."}, {"start": 206.0, "text": "Starting from macOS 27, this command line tool comes pre-installed on your Mac."}, {"start": 210.0, "text": "It's available right from your Terminal app."}, {"start": 213.0, "text": "To get started with fm, just open the Terminal and type fm."}, {"start": 222.0, "text": "You can see a list of commands that are available."}, {"start": 224.0, "text": "For example, you can use respond to prompt the model and return a response,"}, {"start": 230.0, "text": "chat to start an interactive interface, schema to create a schema, and more."}, {"start": 236.0, "text": "To show you what fm is capable of, let's try using fm chat."}, {"start": 242.0, "text": "With this new terminal interface, I can start a conversation"}, {"start": 245.0, "text": "with the on-device model, right from my terminal."}, {"start": 248.0, "text": "I can start with a first question,"}, {"start": 260.0, "text": "then ask a follow-up question."}, {"start": 271.0, "text": "fm chat comes with a number of commands."}, {"start": 274.0, "text": "For example, with /model,"}, {"start": 277.0, "text": "I can switch the conversation to use the Private Cloud Compute model."}, {"start": 283.0, "text": "Or, with /save, I can save the current conversation to resume later."}, {"start": 292.0, "text": "Interactive sessions with fm chat"}, {"start": 294.0, "text": "are great for getting a first pulse of the model."}, {"start": 297.0, "text": "So if you're exploring a new idea,"}, {"start": 299.0, "text": "you can pry the model and see how it performs with your prompts."}, {"start": 302.0, "text": "When you'd rather have inline responses, like in scripts,"}, {"start": 305.0, "text": "use the command fm respond instead."}, {"start": 307.0, "text": "Run fm respond with a prompt in a terminal,"}, {"start": 310.0, "text": "and you'll receive the response from the model as output."}, {"start": 314.0, "text": "fm respond has a number of options,"}, {"start": 316.0, "text": "like the model option, that lets you prompt the Private Cloud Compute model."}, {"start": 321.0, "text": "Or the image option, to include an image in your prompt."}, {"start": 325.0, "text": "Just like with the Swift framework,"}, {"start": 326.0, "text": "I can use the model to produce structured outputs."}, {"start": 330.0, "text": "Using the command fm schema object, I can create a schema,"}, {"start": 334.0, "text": "and I can then use it with fm respond with the schema option."}, {"start": 339.0, "text": "There's more options that could be useful to you."}, {"start": 341.0, "text": "To check out all the options, use the help option."}, {"start": 345.0, "text": "As you've seen so far,"}, {"start": 346.0, "text": "the fm command line tool lets you use either the on-device model,"}, {"start": 350.0, "text": "or the Apple Foundation Model on Private Cloud Compute."}, {"start": 353.0, "text": "By default, it uses the on-device model that comes with macOS,"}, {"start": 357.0, "text": "and that's always available."}, {"start": 360.0, "text": "You can also use the Apple Foundation Model on Private Cloud Compute,"}, {"start": 363.0, "text": "which has usage limits."}, {"start": 365.0, "text": "It's a much bigger model than the on-device model,"}, {"start": 368.0, "text": "so it will perform better on complex problems."}, {"start": 371.0, "text": "Let's put together what we learned to solve a practical problem."}, {"start": 375.0, "text": "I just completed a presentation project on my Mac."}, {"start": 379.0, "text": "The folder where I was storing my assets is full of drafts,"}, {"start": 382.0, "text": "and I'd like to free up space on disk."}, {"start": 384.0, "text": "I'd like to clean up this folder to only keep the final versions of my assets."}, {"start": 389.0, "text": "I'll use Foundation Models to sort out my files, keep only the essentials,"}, {"start": 394.0, "text": "back them up, and move the old ones to my archive disk."}, {"start": 398.0, "text": "I'd like to automate this in a script so that whenever this happens again,"}, {"start": 402.0, "text": "I can just rerun this script."}, {"start": 404.0, "text": "Using fm here lets me call into a language model"}, {"start": 406.0, "text": "that can sort draft versus final files in my script."}, {"start": 410.0, "text": "So that the script works even if the names are messy"}, {"start": 413.0, "text": "and are difficult to sort predictably."}, {"start": 418.0, "text": "I've prepared a script that uses fm to distinguish draft files from final files,"}, {"start": 423.0, "text": "and moves the files accordingly."}, {"start": 425.0, "text": "It's going to sort the working folder."}, {"start": 428.0, "text": "Right now, this folder has both draft and final files."}, {"start": 433.0, "text": "I'll go ahead and execute the script."}, {"start": 441.0, "text": "Now that it's complete,"}, {"start": 442.0, "text": "I can see that the old files were correctly moved out of the folder."}, {"start": 446.0, "text": "I'll now open the archive folder."}, {"start": 454.0, "text": "I can see that the draft files were moved there,"}, {"start": 456.0, "text": "and, I'll open the backup directory."}, {"start": 465.0, "text": "I can see that the final files were copied there too as backup."}, {"start": 469.0, "text": "Let's go over the script together"}, {"start": 471.0, "text": "to understand how I used fm to sort those files."}, {"start": 475.0, "text": "We start by loading a list of the files in the working directory."}, {"start": 478.0, "text": "Next, we prompt the model to sort this list,"}, {"start": 481.0, "text": "and provide me with a list of draft files, as well as a list of the final files."}, {"start": 486.0, "text": "I can do this with the fm respond command, passing my instructions and my prompt."}, {"start": 492.0, "text": "To get a structured result from the model,"}, {"start": 494.0, "text": "I define a schema using the fm schema object command further up."}, {"start": 498.0, "text": "The structured output will have two fields,"}, {"start": 501.0, "text": "a list of final files, and a list of draft files."}, {"start": 505.0, "text": "I then use fm respond's schema option to use this schema to generate the output."}, {"start": 512.0, "text": "The output of fm respond contains a result in a JSON that's generated by the model."}, {"start": 517.0, "text": "I can then use this result to first,"}, {"start": 520.0, "text": "copy the final files to my backup, and move the draft files to the archive."}, {"start": 525.0, "text": "There's more to discover with fm, so check out the tool on macOS 27 today,"}, {"start": 530.0, "text": "and try using the tool in automation."}, {"start": 532.0, "text": "Let's talk now about the Python SDK."}, {"start": 535.0, "text": "The Python SDK gives you access to Apple Foundation Models right from Python code."}, {"start": 540.0, "text": "You can install it on a Python environment on your Mac,"}, {"start": 542.0, "text": "provided that the Python version is at least Python 3.10,"}, {"start": 546.0, "text": "that you have Xcode installed, and that you're using an Apple Silicon Mac."}, {"start": 550.0, "text": "It's installed through pip, or any other package manager of your choice."}, {"start": 554.0, "text": "The Python SDK includes the core features of the framework."}, {"start": 558.0, "text": "If you've already used it in Swift,"}, {"start": 560.0, "text": "the APIs and abstractions will quickly feel familiar."}, {"start": 563.0, "text": "You can use it to prompt a model with text inputs and image inputs,"}, {"start": 567.0, "text": "and you can use it to stream responses."}, {"start": 570.0, "text": "Just like in Swift,"}, {"start": 571.0, "text": "you can use guided generation to have the model generate structured outputs."}, {"start": 575.0, "text": "And you can use tool calling to enable the model to interact with code."}, {"start": 580.0, "text": "Let's go over a practical example."}, {"start": 582.0, "text": "I'm building an app to order groceries,"}, {"start": 584.0, "text": "and I'd like to let the user prompt the app using the on-device model."}, {"start": 589.0, "text": "As I'm starting to add features, I'd like to evaluate the accuracy of my prompts."}, {"start": 593.0, "text": "So I'll prototype them in Python, before implementing them in Swift."}, {"start": 597.0, "text": "Prompting the model is done just like in Swift."}, {"start": 600.0, "text": "I start by creating a LanguageModelSession,"}, {"start": 603.0, "text": "to which I can pass instructions if I'd like."}, {"start": 606.0, "text": "Then, I call session.respond, passing my prompt as an argument."}, {"start": 611.0, "text": "The result of method contains the output of the model."}, {"start": 615.0, "text": "Just like in the Swift Framework,"}, {"start": 617.0, "text": "I can expose tools to the model to interact with the user's context."}, {"start": 621.0, "text": "For example,"}, {"start": 622.0, "text": "I can define a tool that the model can call to fetch the last few orders,"}, {"start": 625.0, "text": "so that it can provide more personalized information."}, {"start": 629.0, "text": "Just like in the Swift Framework and in the command line tool,"}, {"start": 632.0, "text": "I can also constrain the model to produce structured outputs."}, {"start": 635.0, "text": "For example, in this code,"}, {"start": 637.0, "text": "I'm using guided generation to ensure the output of the model"}, {"start": 640.0, "text": "is captured in an ItemsSuggestion object."}, {"start": 644.0, "text": "Here, using the fm.generable decorator, I define the desired output structure,"}, {"start": 649.0, "text": "and I pass it to fm.respond as the generating argument."}, {"start": 654.0, "text": "One of the main benefits of our Python SDK"}, {"start": 656.0, "text": "is easy integration with Python's ecosystem."}, {"start": 659.0, "text": "Let me illustrate this with a use case"}, {"start": 661.0, "text": "where we'll use some open-source Python packages to set up an evaluation pipeline."}, {"start": 666.0, "text": "As I'm designing an app to order groceries,"}, {"start": 668.0, "text": "one feature I'm working on is the ability to prepare the user's next order."}, {"start": 673.0, "text": "Using a language model,"}, {"start": 675.0, "text": "I'd like to predict what users would like to add to their cart"}, {"start": 678.0, "text": "based on their previous orders."}, {"start": 680.0, "text": "As I'm designing this feature,"}, {"start": 682.0, "text": "I'd like to make sure the output reliably works off of the previous orders."}, {"start": 686.0, "text": "And also, that the prediction accounts for any items already in the cart."}, {"start": 691.0, "text": "I've prepared a few different implementations for this feature,"}, {"start": 694.0, "text": "each with different prompts."}, {"start": 697.0, "text": "And I'd like to quantify their accuracy,"}, {"start": 699.0, "text": "so I can select the best one and make sure it performs well."}, {"start": 703.0, "text": "To evaluate their prompt and iterate,"}, {"start": 706.0, "text": "Swift developers can leverage the Evaluations framework."}, {"start": 709.0, "text": "It's available with Xcode 27, and it makes it easy to create evaluations,"}, {"start": 714.0, "text": "and track the accuracy of your features across multiple iterations."}, {"start": 718.0, "text": "But many data scientists might be more familiar with Python than with Swift."}, {"start": 722.0, "text": "If you fall under this scenario,"}, {"start": 724.0, "text": "let me show you how I can perform this analysis in Python"}, {"start": 727.0, "text": "by using the Python SDK from a Jupyter Notebook."}, {"start": 731.0, "text": "First, I used a large server model to generate evaluation data."}, {"start": 735.0, "text": "I now have some inputs, and for each of those,"}, {"start": 738.0, "text": "data on what I expect in the output."}, {"start": 741.0, "text": "I'll write a number of implementations that use different prompts."}, {"start": 745.0, "text": "Then, for each of my evaluation inputs,"}, {"start": 748.0, "text": "I'll generate outputs using each of those different implementations."}, {"start": 751.0, "text": "I'll then save this data as rows in a Pandas DataFrame."}, {"start": 755.0, "text": "Next, I've designed some judge functions that rely on a server model."}, {"start": 759.0, "text": "They will score each output on the criteria of my choice."}, {"start": 763.0, "text": "I'll then save those metrics in the Pandas DataFrame."}, {"start": 766.0, "text": "I can now generate some charts to see them visually."}, {"start": 770.0, "text": "Let's see it in action."}, {"start": 774.0, "text": "My notebook contains evaluation data, with inputs and expected outputs."}, {"start": 780.0, "text": "I prepared three different implementations of how to complete the user's cart."}, {"start": 784.0, "text": "Each of those leverage the on-device model by prompting it differently."}, {"start": 789.0, "text": "The first method uses a very minimal prompt."}, {"start": 794.0, "text": "The second one uses a more descriptive prompt,"}, {"start": 797.0, "text": "and describes the task more in detail."}, {"start": 801.0, "text": "And the third one has the most comprehensive prompts,"}, {"start": 804.0, "text": "and describes a list of rules to the model."}, {"start": 810.0, "text": "For each row in my evaluation dataset,"}, {"start": 813.0, "text": "I went ahead and generated the outputs for each of those implementations."}, {"start": 817.0, "text": "I then stored those inputs and outputs in a Pandas DataFrame."}, {"start": 821.0, "text": "I passed this data to a third party model that I'm using as a judge model."}, {"start": 826.0, "text": "Which will score each result on a set of criteria."}, {"start": 830.0, "text": "With the gradings generated,"}, {"start": 831.0, "text": "I can use matplotlib to generate charts."}, {"start": 834.0, "text": "So that I can quickly see how each set of prompts performs."}, {"start": 838.0, "text": "Here, since the data has already been generated and graded,"}, {"start": 841.0, "text": "I can run this cell and the below to generate the charts."}, {"start": 847.0, "text": "Let's look at the charts."}, {"start": 850.0, "text": "First, by looking at the errors generated by setup,"}, {"start": 853.0, "text": "I can see that the detailed prompt leads to a high percentage of generation errors."}, {"start": 857.0, "text": "This can happen, for example, when we reach the model's max context window size."}, {"start": 863.0, "text": "Next, we can see that the two less detailed prompts tend to lead to excess"}, {"start": 867.0, "text": "items added to the cart, while the more detailed one has less excess items."}, {"start": 873.0, "text": "However, with the more detailed prompts,"}, {"start": 876.0, "text": "we tend to miss more items that were expected."}, {"start": 879.0, "text": "The first prompt also tends to lead to more hallucinated items added to the cart."}, {"start": 885.0, "text": "I can use those insights to iterate on those prompts."}, {"start": 888.0, "text": "With Python, I can make those iterations quickly right from my notebook"}, {"start": 892.0, "text": "without having to rebuild the whole project."}, {"start": 894.0, "text": "It makes it so convenient to test and make changes!"}, {"start": 898.0, "text": "With this example, we saw how you could generate outputs,"}, {"start": 902.0, "text": "grade them, and create charts with the Python SDK."}, {"start": 905.0, "text": "Python has a strong open-source ecosystem"}, {"start": 907.0, "text": "of machine learning and data science packages,"}, {"start": 910.0, "text": "and we used some of those today in our automation."}, {"start": 913.0, "text": "If you develop automation in Python, I encourage you to explore the ecosystem"}, {"start": 917.0, "text": "and see if you can reuse existing packages."}, {"start": 920.0, "text": "Let's wrap this up."}, {"start": 922.0, "text": "We just went over the new ways you can interact with Apple's Foundation Models."}, {"start": 926.0, "text": "I encourage you to try them out today on macOS 27."}, {"start": 930.0, "text": "You can use those tools alongside your Xcode project,"}, {"start": 932.0, "text": "as a way to prototype and evaluate prompts."}, {"start": 935.0, "text": "Or you can use them on their own, to use the model in novel ways."}, {"start": 940.0, "text": "To get more familiar with those tools,"}, {"start": 942.0, "text": "here are a few next steps that I recommend."}, {"start": 945.0, "text": "First, start by exploring the command line tool from the Terminal app."}, {"start": 949.0, "text": "Explore the different options and features, and try them out."}, {"start": 953.0, "text": "Next, to learn more about how to use the Python SDK,"}, {"start": 956.0, "text": "head to the GitHub repository."}, {"start": 958.0, "text": "You'll find some example snippets and some documentation"}, {"start": 961.0, "text": "that you can use as a reference on how to build advanced workflows."}, {"start": 965.0, "text": "Once you've gotten the hang of the Python SDK,"}, {"start": 968.0, "text": "put this new knowledge in practice to create an evaluation pipeline."}, {"start": 971.0, "text": "Think of a way you can use the model, and after you find some working prompts,"}, {"start": 975.0, "text": "quantify the results of the model against an evaluation dataset"}, {"start": 978.0, "text": "to measure the effectiveness of your prompts."}, {"start": 981.0, "text": "I hope using those new tools"}, {"start": 983.0, "text": "will inspire you to use language models in new and exciting ways."}, {"start": 987.0, "text": "Happy building!"}]}