{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-241", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/241", "wordCount": 2697, "segments": [{"start": 7.0, "text": "Hello everyone, and welcome!"}, {"start": 9.0, "text": "My name is Erik!"}, {"start": 10.0, "text": "And my name is Zhen!"}, {"start": 12.0, "text": "Last year we introduced the Foundation Models framework"}, {"start": 15.0, "text": "with features like guided generation, snapshot streaming,"}, {"start": 19.0, "text": "and the powerful tool protocol."}, {"start": 21.0, "text": "We were blown away by your excitement"}, {"start": 23.0, "text": "for the Foundation Models framework in year one,"}, {"start": 26.0, "text": "and we think you're going to love what we've lined up for you this year, even more!"}, {"start": 32.0, "text": "We're going to walk through everything that's new in the framework this year."}, {"start": 35.0, "text": "And let me tell you, this release is packed!"}, {"start": 40.0, "text": "Our 2027 release is all about integrations into and beyond the OS,"}, {"start": 45.0, "text": "a wider variety of models,"}, {"start": 47.0, "text": "and new primitives for building agentic experiences."}, {"start": 51.0, "text": "Let's kick it off with one of our most exciting updates."}, {"start": 55.0, "text": "The Foundation Models framework,"}, {"start": 57.0, "text": "including many of the brand new APIs that we're announcing today,"}, {"start": 61.0, "text": "is going open source!"}, {"start": 63.0, "text": "And doing it in style!"}, {"start": 66.0, "text": "In addition to the core framework, we're also releasing a new package,"}, {"start": 70.0, "text": "Foundation Models framework utilities, that will be updated between OS releases"}, {"start": 76.0, "text": "to give you access to emerging and experimental building blocks."}, {"start": 80.0, "text": "And during the course of this session, you'll be hearing about"}, {"start": 83.0, "text": "multiple other packages all joining the Foundation Models framework ecosystem."}, {"start": 89.0, "text": "So, are you ready to jump in with us?"}, {"start": 92.0, "text": "I'm going to cover everything related to the new models,"}, {"start": 95.0, "text": "new modalities, and new tools we're adding to the framework."}, {"start": 99.0, "text": "And I'm going to show you the brand new APIs we've made"}, {"start": 103.0, "text": "to help you get the most out of them!"}, {"start": 105.0, "text": "We've got a full schedule!"}, {"start": 108.0, "text": "First up, we've got a whole pile of model updates for you,"}, {"start": 111.0, "text": "including updates to the on-device model and access to server models."}, {"start": 117.0, "text": "We've also added new system tools that supercharge your session"}, {"start": 121.0, "text": "with help from Spotlight and the Vision framework."}, {"start": 124.0, "text": "Afterwards, Zhen will rejoin us"}, {"start": 127.0, "text": "to tell you about a powerful new API called dynamic profiles,"}, {"start": 131.0, "text": "our new primitive for creating agentic experiences."}, {"start": 135.0, "text": "Zhen will also dig into the brand new Evaluations framework"}, {"start": 138.0, "text": "and its tight integration with Foundation Models."}, {"start": 143.0, "text": "And to wrap up,"}, {"start": 144.0, "text": "we have some exciting news to share about Mac-specific productivity tools,"}, {"start": 148.0, "text": "so make sure to stick around!"}, {"start": 151.0, "text": "Let's jump right in with our new model!"}, {"start": 154.0, "text": "This release comes with a new on-device model,"}, {"start": 157.0, "text": "rebuilt from the ground up, and better across the board."}, {"start": 161.0, "text": "It's more intelligent; better at logic and tool calling."}, {"start": 165.0, "text": "In iOS 26.4, we released new APIs for inspecting the model's context size"}, {"start": 172.0, "text": "and counting the tokens in instructions, prompts, and transcripts."}, {"start": 176.0, "text": "You'll want to use these going forward"}, {"start": 179.0, "text": "to adapt your app to the hardware it's running on."}, {"start": 182.0, "text": "We've also been hard at work on refining our guardrails."}, {"start": 186.0, "text": "You may have noticed adjustments in iOS 26.4"}, {"start": 189.0, "text": "to reduce the number of false positives,"}, {"start": 191.0, "text": "and we're continuing to make even more improvements in iOS 27."}, {"start": 196.0, "text": "In addition the on-device model is also gaining Vision capabilities,"}, {"start": 201.0, "text": "which unlocks entire new categories of applications."}, {"start": 206.0, "text": "The API is simple, a natural extension of the existing prompt builders."}, {"start": 213.0, "text": "Here we've created a session,"}, {"start": 215.0, "text": "and we want to ask about the photo of the origami on the right."}, {"start": 219.0, "text": "Simply insert an image attachment into your prompt, together with text."}, {"start": 225.0, "text": "Now, the model can answer questions about the image."}, {"start": 232.0, "text": "Image attachments can be created from a variety of types including;"}, {"start": 236.0, "text": "UIImage,"}, {"start": 238.0, "text": "NSImage,"}, {"start": 241.0, "text": "CGImage"}, {"start": 244.0, "text": "Core Image types,"}, {"start": 246.0, "text": "CoreVideo Pixel Buffers,"}, {"start": 249.0, "text": "and file URLs."}, {"start": 252.0, "text": "The model supports images in any size and aspect ratio,"}, {"start": 255.0, "text": "so you don't need to crop or pad to any particular shape."}, {"start": 259.0, "text": "Arbitrary image sizes are allowed, but bear in mind"}, {"start": 263.0, "text": "that larger images will consume more tokens and incur more latency."}, {"start": 268.0, "text": "Thanks to all these upgrades,"}, {"start": 269.0, "text": "the on-device system language model is more capable than ever."}, {"start": 274.0, "text": "But if you need even more horsepower, we're giving you access to a brand new"}, {"start": 279.0, "text": "PrivateCloudComputeLanguageModel."}, {"start": 281.0, "text": "The Private Cloud Compute model is the very same one that"}, {"start": 284.0, "text": "powers many of the Apple Intelligence features you know and love."}, {"start": 288.0, "text": "It is a much bigger model than the on-device models,"}, {"start": 291.0, "text": "and has a 32,000 token context window,"}, {"start": 295.0, "text": "and it comes with a powerful new capability, reasoning."}, {"start": 300.0, "text": "Reasoning models are trained to spend time carefully thinking through"}, {"start": 303.0, "text": "their answers before providing a response,"}, {"start": 306.0, "text": "which results in significantly better outcomes."}, {"start": 310.0, "text": "Using Private Cloud Compute couldn't be easier."}, {"start": 314.0, "text": "Just create an instance of the model"}, {"start": 316.0, "text": "and use it to initialize your language model session."}, {"start": 319.0, "text": "When prompting the session,"}, {"start": 321.0, "text": "I can now specify a reasoning level on the new contextOptions argument."}, {"start": 326.0, "text": "reasoningLevel controls how much the model is allowed think before responding."}, {"start": 331.0, "text": "Deep reasoning produces better responses in exchange for additional compute."}, {"start": 337.0, "text": "One of the best things about Private Cloud Compute"}, {"start": 341.0, "text": "is that you don't have to worry about account setup,"}, {"start": 344.0, "text": "you don't have to deal with authentication,"}, {"start": 347.0, "text": "and you don't have to store API keys, it's all completely seamless!"}, {"start": 353.0, "text": "And of course, Private Cloud Compute is above all else, private."}, {"start": 359.0, "text": "No prompts are ever stored,"}, {"start": 361.0, "text": "and we make it possible for independent researchers to verify these claims."}, {"start": 366.0, "text": "And to top it all off, Private Cloud Compute makes it possible for us"}, {"start": 370.0, "text": "to bring the Foundation Models framework to watchOS."}, {"start": 374.0, "text": "Starting in watchOS 27,"}, {"start": 376.0, "text": "you can wear your most powerful intelligence features right on your wrist."}, {"start": 381.0, "text": "PCC is available with no cloud API costs"}, {"start": 385.0, "text": "to developers who have less than 2 million first time downloads."}, {"start": 389.0, "text": "Your users will have access to PCC every day"}, {"start": 392.0, "text": "and if they are subscribed to iCloud+, their limit will be even higher!"}, {"start": 398.0, "text": "To learn more about the ins and outs of PrivateCloudComputeLanguageModel,"}, {"start": 402.0, "text": "including the entitlement you'll need to use it,"}, {"start": 404.0, "text": "make sure to tune in to our video about building with Private Cloud Compute."}, {"start": 408.0, "text": "In addition to an overhauled on-device model"}, {"start": 411.0, "text": "and the new Private Cloud Compute model,"}, {"start": 414.0, "text": "we're opening up our model abstraction layer"}, {"start": 416.0, "text": "to make it possible for nearly any language model"}, {"start": 419.0, "text": "to be used with the Foundation Model's framework."}, {"start": 422.0, "text": "The abstraction layer is built around a new LanguageModel protocol"}, {"start": 426.0, "text": "that allows both local and servers models to back a LanguageModelSession."}, {"start": 432.0, "text": "Existing models like SystemLanguageModel and PrivateCloudComputeLanguageModel"}, {"start": 437.0, "text": "already conform to this protocol."}, {"start": 440.0, "text": "And, we're open sourcing two additional implementations:"}, {"start": 445.0, "text": "CoreAILanguageModel and MLXLanguageModel,"}, {"start": 449.0, "text": "for running a myriad of local models on the Apple Neural Engine in your Mac's GPU."}, {"start": 454.0, "text": "We're also excited to share that we've been hard at work behind the scenes"}, {"start": 459.0, "text": "to ensure you have access to a variety of frontier server models!"}, {"start": 463.0, "text": "Anthropic, and Google are both publishing Swift packages"}, {"start": 467.0, "text": "to provide you with access to their latest and greatest models!"}, {"start": 471.0, "text": "The model abstraction layer makes using third party models simple."}, {"start": 475.0, "text": "I'll just import a language model package using Swift Package Manager,"}, {"start": 480.0, "text": "initialize the model that I want to use,"}, {"start": 484.0, "text": "and pass it when creating my session."}, {"start": 495.0, "text": "Everything downstream stays the same."}, {"start": 501.0, "text": "Bear in mind that if you use third party server models,"}, {"start": 504.0, "text": "you'll probably have to deal with both authentication and billing."}, {"start": 510.0, "text": "Remember, never store private keys in your app binary."}, {"start": 515.0, "text": "Always fetch access tokens with a secure mechanism like OAuth,"}, {"start": 519.0, "text": "and store them securely using KeyChain."}, {"start": 522.0, "text": "As a developer, you'll typically be billed per-token when using 3rd party models,"}, {"start": 528.0, "text": "so we've made it easy to keep track of your usage."}, {"start": 531.0, "text": "Sessions and responses now have a usage property"}, {"start": 536.0, "text": "that tells you precisely how many tokens were used."}, {"start": 539.0, "text": "You can also check how many of the input tokens were read from cache,"}, {"start": 543.0, "text": "and how many of the response tokens were used for reasoning."}, {"start": 548.0, "text": "If you'd like to learn more about using LanguageModels,"}, {"start": 551.0, "text": "or how you can author your own LanguageModel package,"}, {"start": 554.0, "text": "check out \"Bring an LLM provider to the Foundation Models framework\"."}, {"start": 559.0, "text": "We have finally made it through all our model updates!"}, {"start": 563.0, "text": "Next up is system tools!"}, {"start": 565.0, "text": "In this release, we're introducing several built-in tools"}, {"start": 569.0, "text": "that supercharge your LanguageModelSessions"}, {"start": 571.0, "text": "with system provided functionality."}, {"start": 574.0, "text": "FoundationModels now contains two native tools"}, {"start": 577.0, "text": "backed by the Vision framework's powerful capabilities."}, {"start": 581.0, "text": "The BarcodeReaderTool allows the model read information from barcodes,"}, {"start": 585.0, "text": "and the OCRTool allows the model to extract structured text from images."}, {"start": 590.0, "text": "Both enhance a model's ability to reason about visual information"}, {"start": 593.0, "text": "in ways it can't natively."}, {"start": 596.0, "text": "The \"What's new in image understanding\" video has more detail about"}, {"start": 599.0, "text": "how to leverage these tools, so queue that one up for more info."}, {"start": 604.0, "text": "Similarly, we're also introducing a search tool powered by Spotlight"}, {"start": 608.0, "text": "for implementing fully local Retrieval-Augmented Generation."}, {"start": 613.0, "text": "This has been one of your most most requested features."}, {"start": 617.0, "text": "Retrieval-Augmented Generation, or RAG,"}, {"start": 620.0, "text": "is a technique that gives the model access to up-to-date personal or domain knowledge"}, {"start": 625.0, "text": "by leveraging a Spotlight index and specially processed queries."}, {"start": 631.0, "text": "If this sounds like just what you've been waiting for,"}, {"start": 634.0, "text": "\"LLM search using Core Spotlight\" should be at the top of your watch list."}, {"start": 638.0, "text": "Now that we've looked at all the new models in this release,"}, {"start": 642.0, "text": "and the new system tools we're adding to the SDK, I'm going to hand off to Zhen"}, {"start": 646.0, "text": "to tell you all about our new APIs for building agentic app experiences."}, {"start": 651.0, "text": "Get excited!"}, {"start": 652.0, "text": "Take it away, Zhen!"}, {"start": 654.0, "text": "Thanks Erik!"}, {"start": 656.0, "text": "I'm going to introduce you to dynamic profiles,"}, {"start": 659.0, "text": "our new primitive for building agentic experiences."}, {"start": 664.0, "text": "Let's begin by walking through the crafts app"}, {"start": 667.0, "text": "and looking at the kinds of experiences dynamic profiles make possible."}, {"start": 672.0, "text": "Inside the app, I can create a journal entry with some origami photos."}, {"start": 677.0, "text": "The app creates a session that starts in craft analysis mode."}, {"start": 681.0, "text": "The instructions tell the model to analyze the images, and record what it finds."}, {"start": 687.0, "text": "It identifies the craft type, colors, and materials,"}, {"start": 692.0, "text": "then saves them back to the journal, through a tool call."}, {"start": 696.0, "text": "Next, the app switches to brainstorming mode"}, {"start": 700.0, "text": "using Private Cloud Compute's reasoning capability,"}, {"start": 703.0, "text": "it takes everything it just learned"}, {"start": 705.0, "text": "and suggests a list of creative origami projects."}, {"start": 709.0, "text": "Pretty cool, right?"}, {"start": 711.0, "text": "To implement this feature, I'd start by creating a LanguageModelSession."}, {"start": 716.0, "text": "Then I'll add more sessions, each with its own models, instructions, and tools."}, {"start": 724.0, "text": "But what if I want the model to autonomously switch modes?"}, {"start": 730.0, "text": "Things start to get hairy."}, {"start": 734.0, "text": "Managing context and orchestrating an agentic system like this"}, {"start": 738.0, "text": "can involve a lot of boilerplates."}, {"start": 741.0, "text": "That's why Foundation Models is introducing a new declarative API,"}, {"start": 747.0, "text": "dynamic profiles, so you can focus on what matters in the context,"}, {"start": 752.0, "text": "and worry less about imperative controls,"}, {"start": 755.0, "text": "all within a single language model session."}, {"start": 759.0, "text": "To create a simple dynamic profile, I can declare a struct,"}, {"start": 764.0, "text": "and conform it to the DynamicProfile protocol"}, {"start": 769.0, "text": "with a body property that contains a Profile."}, {"start": 774.0, "text": "Language model sessions now can be initialized with a DynamicProfile."}, {"start": 779.0, "text": "I can specify the instructions and tools,"}, {"start": 783.0, "text": "that should be present in the context, at that very moment."}, {"start": 788.0, "text": "This is the simplest form of a DynamicProfile,"}, {"start": 792.0, "text": "a data structure made up of instructions, and tools."}, {"start": 797.0, "text": "I want to implement two different modes,"}, {"start": 799.0, "text": "one for craft analysis, and one for brainstorm."}, {"start": 803.0, "text": "My app has an observable object that stores a mode variable,"}, {"start": 808.0, "text": "so I can switch on it."}, {"start": 811.0, "text": "In the different branches,"}, {"start": 813.0, "text": "the LanguageModelSession should have different instructions and tools."}, {"start": 818.0, "text": "I can even give the model a tool,"}, {"start": 821.0, "text": "to intelligently switch to the context for brainstorm mode."}, {"start": 827.0, "text": "Sometimes it's not enough to just manage the context,"}, {"start": 831.0, "text": "you may also want different models and configurations for different tasks,"}, {"start": 837.0, "text": "while still maintaining the conversation history."}, {"start": 842.0, "text": "In my crafts app, there are 2 scenarios: craft analysis and brainstorm."}, {"start": 849.0, "text": "Each of them already has a different set of instructions and tools."}, {"start": 855.0, "text": "For quick tasks like analyzing a craft, SystemLanguageModel is probably enough."}, {"start": 862.0, "text": "Now, if I want to switch to brainstorming,"}, {"start": 865.0, "text": "I can also specify Private Cloud Compute, configured with deep reasoning."}, {"start": 871.0, "text": "To describe those configurations, I can use modifiers."}, {"start": 877.0, "text": "A model modifier to specify PCC,"}, {"start": 881.0, "text": "And a reasoningLevel modifier to ask the model think thoroughly."}, {"start": 887.0, "text": "Now we have a LanguageModelSession"}, {"start": 889.0, "text": "with dynamically configured model, tools, and instructions."}, {"start": 894.0, "text": "When the app needs to handle different context with different model capabilities,"}, {"start": 899.0, "text": "dynamic profiles are a great fit."}, {"start": 902.0, "text": "The important thing to understand is that a DynamicProfile"}, {"start": 906.0, "text": "resolves to a single active Profile at any given time."}, {"start": 911.0, "text": "You use conditionals to pick which Profile is active,"}, {"start": 915.0, "text": "and the framework handles the transition for you."}, {"start": 919.0, "text": "Now let's try it out in the crafts app."}, {"start": 922.0, "text": "As I select the idea, the model switches to Private Cloud Compute."}, {"start": 928.0, "text": "It still has the full context from the analysis, but generating creative"}, {"start": 933.0, "text": "project ideas benefits from the larger model's capabilities,"}, {"start": 937.0, "text": "like better tool calling, and broader world knowledge."}, {"start": 941.0, "text": "Profiles make it so much easier to manage context"}, {"start": 945.0, "text": "and dynamically configure sessions."}, {"start": 948.0, "text": "When using this API, consider privacy boundaries, model capabilities, and cost."}, {"start": 955.0, "text": "To explore more, check out the deep dive session:"}, {"start": 958.0, "text": "\"Build agentic app experiences with Foundation Models framework\"."}, {"start": 963.0, "text": "As powerful as these capabilities are,"}, {"start": 966.0, "text": "language models are inherently non-deterministic,"}, {"start": 970.0, "text": "which makes their behaviors hard to predict."}, {"start": 974.0, "text": "The Evaluations framework is a new Swift framework"}, {"start": 977.0, "text": "that measures the quality of your intelligence features."}, {"start": 982.0, "text": "With the Evaluations framework,"}, {"start": 984.0, "text": "you can quantify accuracy as you tweak your prompts."}, {"start": 988.0, "text": "Evaluations is built to help app developers like you,"}, {"start": 992.0, "text": "understand the statistical impact of changes,"}, {"start": 996.0, "text": "and deliver your app with confidence."}, {"start": 1000.0, "text": "To learn more about Evaluations, check out these sessions."}, {"start": 1006.0, "text": "Now, I want to shift gears, and talk about our tooling and open source efforts."}, {"start": 1012.0, "text": "In macOS 27, the models are coming to the command line."}, {"start": 1017.0, "text": "The fm CLI is a brand new way to use Apple Foundation Models"}, {"start": 1022.0, "text": "for everyday productivity."}, {"start": 1025.0, "text": "You can access the on-device model and PCC from the terminal,"}, {"start": 1030.0, "text": "just by using the fm command."}, {"start": 1034.0, "text": "fm has a nice helper and it lists all the features it supports."}, {"start": 1040.0, "text": "I've been using fm chat to experiment with models, for my app features."}, {"start": 1046.0, "text": "Let me show you."}, {"start": 1048.0, "text": "I want to know, what does valley fold mean in the context origami?"}, {"start": 1058.0, "text": "Easy."}, {"start": 1059.0, "text": "Just like that."}, {"start": 1060.0, "text": "I can even plug fm into shell scripts to summarize documents,"}, {"start": 1065.0, "text": "extract information, or generate content."}, {"start": 1069.0, "text": "For example, I have some pictures with random names"}, {"start": 1073.0, "text": "like this one, IMG_1234."}, {"start": 1078.0, "text": "Let me just ask fm to generate a file name based on the content inside the image."}, {"start": 1087.0, "text": "Look at that!"}, {"start": 1089.0, "text": "It just came up with a nice, descriptive name!"}, {"start": 1093.0, "text": "And if you're a data scientist or researcher"}, {"start": 1096.0, "text": "working in the Python ecosystem,"}, {"start": 1099.0, "text": "the FoundationModels SDK for Python has you covered, too."}, {"start": 1105.0, "text": "The Python SDK gives you direct access to the very same on-device model"}, {"start": 1111.0, "text": "that powers the Swift Foundation Models framework."}, {"start": 1115.0, "text": "You can check model availability,"}, {"start": 1118.0, "text": "or generate a response with just a few lines of Python."}, {"start": 1122.0, "text": "The SDK has the core feature of the Swift framework"}, {"start": 1126.0, "text": "so you can go from a prompt to a structured response in seconds."}, {"start": 1131.0, "text": "To learn more, check out the session:"}, {"start": 1134.0, "text": "\"Build AI-powered scripts with the fm CLI and Python SDK\"."}, {"start": 1140.0, "text": "Now that we have looked at productivity on Mac,"}, {"start": 1143.0, "text": "let's talk about open source."}, {"start": 1146.0, "text": "Starting with Foundation Models framework utilities."}, {"start": 1151.0, "text": "Utilities contains a collection of building blocks"}, {"start": 1155.0, "text": "to help you explore emerging practices in working with LLMs."}, {"start": 1161.0, "text": "It provides profile modifiers for transcript management,"}, {"start": 1165.0, "text": "a skill API for procedural knowledge loading,"}, {"start": 1169.0, "text": "and a language model that can interface with servers"}, {"start": 1173.0, "text": "using the Chat Completions standard."}, {"start": 1176.0, "text": "These are just the starting points."}, {"start": 1179.0, "text": "Tools and trends evolve,"}, {"start": 1181.0, "text": "and the Foundation Models framework utilities"}, {"start": 1183.0, "text": "is there to grow with you."}, {"start": 1187.0, "text": "In addition to the utilities package,"}, {"start": 1189.0, "text": "the core of the FoundationModels framework will also be open source."}, {"start": 1195.0, "text": "Open sourcing the Foundation Models framework"}, {"start": 1198.0, "text": "makes it a great solution for interacting with LLMs"}, {"start": 1202.0, "text": "everywhere Swift runs, including Linux servers."}, {"start": 1206.0, "text": "Together with other model providers like Anthropic and Google,"}, {"start": 1211.0, "text": "alongside CoreAI and MLX integrations,"}, {"start": 1215.0, "text": "you'll be able to run any model, anywhere."}, {"start": 1219.0, "text": "Welcome back Erik!"}, {"start": 1220.0, "text": "Are you ready to wrap up?"}, {"start": 1221.0, "text": "Way to bring it home Zhen!"}, {"start": 1224.0, "text": "We hope you're as jazzed as we are about all these new capabilities,"}, {"start": 1228.0, "text": "models, and APIs."}, {"start": 1230.0, "text": "We've only just scratched the surface."}, {"start": 1233.0, "text": "Yeah, to get the full scoop, make sure to check out other videos,"}, {"start": 1237.0, "text": "for deep dives on all the topics we've introduced here,"}, {"start": 1240.0, "text": "from the Evaluations framework to Private Cloud Compute,"}, {"start": 1244.0, "text": "the enhanced Xcode instrument, and the nitty gritty on dynamic profiles."}, {"start": 1250.0, "text": "Some great next steps would be to explore our sample app"}, {"start": 1253.0, "text": "to learn more about dynamic profiles,"}, {"start": 1256.0, "text": "and to start getting familiar with the Evaluations framework."}, {"start": 1259.0, "text": "On behalf of the whole team, thanks for joining us!"}, {"start": 1263.0, "text": "Thank you!"}]}