{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-339", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/339", "wordCount": 3146, "segments": [{"start": 7.0, "text": "Hi there!"}, {"start": 8.0, "text": "I'm Christopher Webb,"}, {"start": 9.0, "text": "an engineer on the Machine Learning Research team,"}, {"start": 11.0, "text": "and I'm excited to talk to you"}, {"start": 13.0, "text": "about a new way to use the Foundation Models framework."}, {"start": 16.0, "text": "We previously introduced the Foundation Models framework"}, {"start": 18.0, "text": "to give you access to Apple's on-device language model."}, {"start": 22.0, "text": "And now, we're opening up the framework to work with nearly any LLM,"}, {"start": 26.0, "text": "local or serverbased."}, {"start": 28.0, "text": "This allows anyone, from large companies to individual developers,"}, {"start": 31.0, "text": "to easily build their own model-integration on top of the framework."}, {"start": 36.0, "text": "The on-device System Language Model has been rebuilt from the ground up:"}, {"start": 40.0, "text": "it's smarter,"}, {"start": 41.0, "text": "better at instruction following,"}, {"start": 43.0, "text": "and accepts images directly in your prompts."}, {"start": 46.0, "text": "Beyond the system model,"}, {"start": 47.0, "text": "we've added three more options."}, {"start": 49.0, "text": "Private Cloud Compute brings you the model behind many Apple Intelligence features:"}, {"start": 54.0, "text": "now with reasoning,"}, {"start": 55.0, "text": "a 32K token context window,"}, {"start": 58.0, "text": "and the privacy guarantees you'd expect."}, {"start": 62.0, "text": "Core AI lets you run local models efficiently"}, {"start": 65.0, "text": "and take advantage of the ANE."}, {"start": 68.0, "text": "And MLX unlocks the thousands of models available"}, {"start": 71.0, "text": "via the MLX-Community on Hugging Face."}, {"start": 75.0, "text": "And because these are built on top of a brand new public protocol,"}, {"start": 78.0, "text": "developers can bring frontier AI models"}, {"start": 80.0, "text": "into their apps using the same framework."}, {"start": 83.0, "text": "Anthropic and Google will soon extend"}, {"start": 86.0, "text": "the Foundation Models framework with Swift packages of their own,"}, {"start": 89.0, "text": "making state-of-the-art Claude and Gemini models"}, {"start": 92.0, "text": "available to all Swift developers."}, {"start": 95.0, "text": "Which ever model you use,"}, {"start": 97.0, "text": "Apple's, yours, or the community's,"}, {"start": 99.0, "text": "you call them the same way,"}, {"start": 101.0, "text": "because every model conforms to the Language Model protocol."}, {"start": 105.0, "text": "For app developers,"}, {"start": 106.0, "text": "I'll show you how to call any of these models"}, {"start": 108.0, "text": "through the same familiar API."}, {"start": 110.0, "text": "For model providers,"}, {"start": 111.0, "text": "I'll walk you through how to create a Language Model package of your own."}, {"start": 115.0, "text": "But first, let me show you a preview of how easy it is to use."}, {"start": 120.0, "text": "Here's our on-device Foundation Model."}, {"start": 122.0, "text": "Create it,"}, {"start": 123.0, "text": "pass it into a session,"}, {"start": 125.0, "text": "and call the respond function."}, {"start": 126.0, "text": "And there are even more model options."}, {"start": 129.0, "text": "If you need more horsepower,"}, {"start": 131.0, "text": "try Private Cloud Compute."}, {"start": 132.0, "text": "Just swap the model."}, {"start": 134.0, "text": "If you want to ship your own model,"}, {"start": 136.0, "text": "just point CoreAI at your resources."}, {"start": 139.0, "text": "And if you want to try the latest open source models,"}, {"start": 143.0, "text": "simply pass in a model ID,"}, {"start": 145.0, "text": "and let the framework handle the rest."}, {"start": 148.0, "text": "And using a model built on top of the Language Model protocol"}, {"start": 151.0, "text": "means you get access to all kinds of great Foundation Models features,"}, {"start": 154.0, "text": "like Dynamic Profiles."}, {"start": 157.0, "text": "For an overview of everything we’re adding,"}, {"start": 159.0, "text": "check out \"What’s new in the Foundation Models framework\"."}, {"start": 164.0, "text": "The reason it’s possible to swap out models so easily"}, {"start": 167.0, "text": "is that every LanguageModel honors the same protocol,"}, {"start": 171.0, "text": "the System Language Model,"}, {"start": 172.0, "text": "PCC,"}, {"start": 173.0, "text": "Core AI,"}, {"start": 174.0, "text": "MLX,"}, {"start": 175.0, "text": "and those built by the community."}, {"start": 177.0, "text": "If you're a model provider,"}, {"start": 179.0, "text": "you should join the fun!"}, {"start": 180.0, "text": "Let me show you how."}, {"start": 182.0, "text": "There are four steps to bring your model into the framework."}, {"start": 185.0, "text": "We’ll start with packaging."}, {"start": 187.0, "text": "A well-crafted Swift package makes it easy for developers to get started."}, {"start": 191.0, "text": "Then you implement the protocol"}, {"start": 193.0, "text": "by defining the types that describe your model"}, {"start": 196.0, "text": "and the EXECutor that runs it."}, {"start": 199.0, "text": "Next, we’ll talk about how to implement"}, {"start": 201.0, "text": "authentication for server-based models,"}, {"start": 203.0, "text": "including some best practices."}, {"start": 205.0, "text": "And finally, customization."}, {"start": 207.0, "text": "If you need to tailor"}, {"start": 208.0, "text": "the protocol’s building blocks to meet your needs,"}, {"start": 210.0, "text": "you can do that."}, {"start": 211.0, "text": "From attaching response metadata,"}, {"start": 213.0, "text": "all the way to defining entirely new modalities."}, {"start": 217.0, "text": "First up, packaging."}, {"start": 219.0, "text": "We recommend using Swift package manager"}, {"start": 221.0, "text": "so that developers can simply add your package"}, {"start": 224.0, "text": "as a dependency of their app."}, {"start": 226.0, "text": "We'll cover how to set up Package.swift,"}, {"start": 228.0, "text": "and how to publish a release."}, {"start": 231.0, "text": "An important consideration"}, {"start": 232.0, "text": "is which platforms you'll want to support."}, {"start": 235.0, "text": "Foundation Models supports iOS,"}, {"start": 237.0, "text": "macOS,"}, {"start": 238.0, "text": "visionOS,"}, {"start": 239.0, "text": "and watchOS,"}, {"start": 240.0, "text": "allowing developers to create a variety of experiences."}, {"start": 244.0, "text": "We recommend you try to do the same."}, {"start": 246.0, "text": "And because the Foundation Models framework"}, {"start": 248.0, "text": "is being released as open source,"}, {"start": 250.0, "text": "your package could also be useful to developers"}, {"start": 253.0, "text": "who deploy Swift on their servers,"}, {"start": 255.0, "text": "so consider supporting Linux too."}, {"start": 258.0, "text": "Third, your dependencies."}, {"start": 260.0, "text": "Every dependency translates to bytes"}, {"start": 262.0, "text": "that a developer ships to their users."}, {"start": 264.0, "text": "Carefully consider what dependencies are linked by your package."}, {"start": 269.0, "text": "Publishing your package is as easy as creating a git tag."}, {"start": 273.0, "text": "Swift Package Manager is decentralized,"}, {"start": 275.0, "text": "so your repo URL is your distribution channel."}, {"start": 278.0, "text": "Developers can paste the URL"}, {"start": 281.0, "text": "into Xcode and start integrating your model"}, {"start": 283.0, "text": "into their apps."}, {"start": 284.0, "text": "For more, see \"Creating Swift Packages”."}, {"start": 288.0, "text": "With the package in place,"}, {"start": 289.0, "text": "we move on to the protocol."}, {"start": 291.0, "text": "The protocol is the bridge between your model,"}, {"start": 293.0, "text": "and the Foundation Models framework."}, {"start": 296.0, "text": "The protocol has two key pieces."}, {"start": 298.0, "text": "The first is LanguageModel."}, {"start": 300.0, "text": "It describes the model to the framework."}, {"start": 302.0, "text": "It declares what the model can do,"}, {"start": 304.0, "text": "through capabilities,"}, {"start": 305.0, "text": "and provides the configuration the framework needs"}, {"start": 308.0, "text": "to set up the model's EXECutor."}, {"start": 312.0, "text": "The second piece is LanguageModelExecutor where the work happens."}, {"start": 316.0, "text": "It has an initializer that takes a Configuration,"}, {"start": 319.0, "text": "a prewarm function for preparing resources"}, {"start": 322.0, "text": "ahead of the first request,"}, {"start": 324.0, "text": "and a respond function"}, {"start": 325.0, "text": "that streams generation back to the session."}, {"start": 329.0, "text": "The Configuration is what links the two types:"}, {"start": 332.0, "text": "the Model provides it,"}, {"start": 333.0, "text": "and the framework uses it to construct the EXECutor."}, {"start": 336.0, "text": "Now you've seen the protocol in code,"}, {"start": 338.0, "text": "let's build an intuition for how the model's configuration"}, {"start": 342.0, "text": "links it to the EXECutor."}, {"start": 344.0, "text": "Each session holds an EXECutor store."}, {"start": 348.0, "text": "When Model1 arrives,"}, {"start": 349.0, "text": "the framework checks the store using the model's configuration,"}, {"start": 353.0, "text": "but there's no matching EXECutor."}, {"start": 355.0, "text": "So, the LanguageModelSession creates a new EXECutor and stores it."}, {"start": 361.0, "text": "Model2 produces the same configuration,"}, {"start": 364.0, "text": "and because Configuration is Hashable,"}, {"start": 366.0, "text": "the framework knows it matches,"}, {"start": 367.0, "text": "and resolves to the same EXECutor."}, {"start": 370.0, "text": "The configuration is the lookup key, not the model."}, {"start": 374.0, "text": "Model3 produces a different configuration,"}, {"start": 377.0, "text": "so it gets its own EXECutor."}, {"start": 379.0, "text": "Each unique configuration maps to exactly one EXECutor in the store."}, {"start": 385.0, "text": "So what does this look like in your code?"}, {"start": 387.0, "text": "Here's a LanguageModel implementation."}, {"start": 389.0, "text": "It declares its capabilities"}, {"start": 391.0, "text": "and returns the configuration the framework uses to find its EXECutor."}, {"start": 397.0, "text": "The Executor is where the real work lives,"}, {"start": 399.0, "text": "loading weights,"}, {"start": 400.0, "text": "managing resources,"}, {"start": 402.0, "text": "and streaming tokens back to the session."}, {"start": 404.0, "text": "The framework constructs it from a configuration your model provides,"}, {"start": 408.0, "text": "then hands the model in on every request."}, {"start": 412.0, "text": "That split is what keeps your Model trivial to construct."}, {"start": 416.0, "text": "When the session deallocates, the store goes with it."}, {"start": 418.0, "text": "Every stored EXECutor gets released,"}, {"start": 421.0, "text": "your deinit runs,"}, {"start": 422.0, "text": "weights are freed,"}, {"start": 423.0, "text": "and connections closed,"}, {"start": 424.0, "text": "all automatically."}, {"start": 426.0, "text": "You don't write any of that teardown code yourself."}, {"start": 429.0, "text": "Within that lifecycle,"}, {"start": 430.0, "text": "your EXECutor has one more function: prewarm."}, {"start": 434.0, "text": "Before a request arrives,"}, {"start": 435.0, "text": "the developer can ask the framework to prewarm."}, {"start": 437.0, "text": "It's your chance to do expensive setup ahead of time,"}, {"start": 440.0, "text": "like loading weights, opening connections,"}, {"start": 442.0, "text": "or anything that would otherwise slow down that first response."}, {"start": 446.0, "text": "Let's look at how to use it."}, {"start": 448.0, "text": "One approach is to put that setup in a private helper"}, {"start": 451.0, "text": "that loads the weights once and caches them."}, {"start": 454.0, "text": "prewarm calls the helper eagerly,"}, {"start": 456.0, "text": "so the weights are ready before the first request arrives."}, {"start": 459.0, "text": "But prewarm isn't guaranteed to run."}, {"start": 463.0, "text": "Either way, weights load exactly once,"}, {"start": 465.0, "text": "and if your EXECutor has no expensive setup,"}, {"start": 468.0, "text": "like a server-backed model,"}, {"start": 469.0, "text": "prewarm can simply be a no-op."}, {"start": 472.0, "text": "Once your respond function is called,"}, {"start": 474.0, "text": "your EXECutor goes to work."}, {"start": 476.0, "text": "It converts the transcript of the conversation"}, {"start": 478.0, "text": "into the format your model expects."}, {"start": 480.0, "text": "It applies the options the developer has set"}, {"start": 483.0, "text": "and it streams generation events to the session."}, {"start": 487.0, "text": "From the developer's side,"}, {"start": 489.0, "text": "the session is the entire interaction surface."}, {"start": 491.0, "text": "They initialize the model,"}, {"start": 492.0, "text": "create the session,"}, {"start": 494.0, "text": "call respond, and wait."}, {"start": 496.0, "text": "Your EXECutor and the rest of your package,"}, {"start": 498.0, "text": "all of that lives behind the session, out of sight."}, {"start": 501.0, "text": "The developer never sees that machinery,"}, {"start": 503.0, "text": "but here's what's happening behind the scenes."}, {"start": 506.0, "text": "The framework hands you transcript entries,"}, {"start": 509.0, "text": "but your inference engine can only process its native types."}, {"start": 513.0, "text": "So your EXECutor sits in the middle,"}, {"start": 516.0, "text": "translates the entries into messages your inference engine understands,"}, {"start": 520.0, "text": "and passes them along for inference."}, {"start": 524.0, "text": "When your inference engine answers,"}, {"start": 526.0, "text": "the same translation runs in reverse:"}, {"start": 528.0, "text": "your messages back to transcript entries, streamed to the session."}, {"start": 533.0, "text": "For now, let's focus on the transcripts"}, {"start": 536.0, "text": "that flow in and out of the EXECutor."}, {"start": 540.0, "text": "A transcript is the conversation so far,"}, {"start": 542.0, "text": "expressed as a sequence of entries."}, {"start": 544.0, "text": "Each entry plays a role."}, {"start": 546.0, "text": "Instructions, set by the developer,"}, {"start": 549.0, "text": "prompts, from the user,"}, {"start": 552.0, "text": "tool calls your model made,"}, {"start": 553.0, "text": "and the outputs they returned,"}, {"start": 555.0, "text": "and the responses your model has produced."}, {"start": 559.0, "text": "Zooming back out:"}, {"start": 560.0, "text": "your EXECutor's job is to turn each transcript entry"}, {"start": 564.0, "text": "into a message your inference engine knows how to read."}, {"start": 567.0, "text": "So, what's inside a transcript?"}, {"start": 570.0, "text": "Foundation Models defines these six entry types."}, {"start": 574.0, "text": "Your model defines its own roles."}, {"start": 576.0, "text": "Your EXECutor's job is to map between the two,"}, {"start": 579.0, "text": "no matter the shape your model takes."}, {"start": 582.0, "text": "In this example, instructions, prompt, and response"}, {"start": 585.0, "text": "map to system, user, and assistant."}, {"start": 588.0, "text": "Here, tool calls, tool outputs,"}, {"start": 591.0, "text": "and reasoning all map to assistant too."}, {"start": 593.0, "text": "They're part of what the model did during its turn,"}, {"start": 596.0, "text": "and since this model doesn't have dedicated roles for these,"}, {"start": 599.0, "text": "we just map them to assistant."}, {"start": 601.0, "text": "If your model does define something like a dedicated tool role,"}, {"start": 605.0, "text": "you can route there instead."}, {"start": 607.0, "text": "Either way, your EXECutor stays in control."}, {"start": 611.0, "text": "Your EXECutor reads the conversation."}, {"start": 613.0, "text": "But every request carries more than history,"}, {"start": 616.0, "text": "it carries the developer's intent for how the model should respond,"}, {"start": 619.0, "text": "expressed through two additional properties."}, {"start": 623.0, "text": "Every request object can include ContextOptions"}, {"start": 626.0, "text": "and GenerationOptions."}, {"start": 628.0, "text": "ContextOptions control what goes into the prompt,"}, {"start": 631.0, "text": "like the reasoning level you want the model to use,"}, {"start": 633.0, "text": "or a response schema."}, {"start": 635.0, "text": "GenerationOptions control the decoder loop:"}, {"start": 637.0, "text": "sampling strategy,"}, {"start": 639.0, "text": "temperature,"}, {"start": 640.0, "text": "and maximum response length."}, {"start": 642.0, "text": "Here's what that looks like inside respond."}, {"start": 645.0, "text": "Both types of options come in on the request,"}, {"start": 648.0, "text": "your EXECutor pulls them out"}, {"start": 650.0, "text": "and passes them along when calling the model."}, {"start": 653.0, "text": "So that's everything coming in:"}, {"start": 654.0, "text": "transcript, options, all parsed."}, {"start": 657.0, "text": "Now for the half your developer sees: the response."}, {"start": 660.0, "text": "On the response side, there are a few things to send:"}, {"start": 663.0, "text": "the text your inference engine generates,"}, {"start": 665.0, "text": "any tool calls or reasoning,"}, {"start": 667.0, "text": "and the metadata that travels with them."}, {"start": 669.0, "text": "They all go out as events on the channel."}, {"start": 672.0, "text": "Each chunk that the inference engine emits,"}, {"start": 675.0, "text": "a token or tool-call fragment, becomes an event."}, {"start": 678.0, "text": "A textDelta, a toolCallDelta, and so on."}, {"start": 682.0, "text": "The framework writes them to the transcript."}, {"start": 684.0, "text": "Foundation Models exposes both one-shot and streaming responses,"}, {"start": 688.0, "text": "but the implementation is always streaming;"}, {"start": 690.0, "text": "the one-shot API just collects the deltas internally."}, {"start": 695.0, "text": "So far we've looked at this from your model's side,"}, {"start": 697.0, "text": "events going out as the model produces them."}, {"start": 700.0, "text": "But put yourself in the developer's seat for a moment."}, {"start": 702.0, "text": "They've called respond and they're waiting."}, {"start": 705.0, "text": "What do they need first?"}, {"start": 707.0, "text": "Here's your EXECutor's side of the handshake with the developer."}, {"start": 710.0, "text": "There's a deliberate order to it."}, {"start": 712.0, "text": "First, a metadata update,"}, {"start": 714.0, "text": "model and request IDs the developer can use for logging and debugging."}, {"start": 719.0, "text": "Then a usage update,"}, {"start": 720.0, "text": "prompt token counts for accounting."}, {"start": 723.0, "text": "Sending these upfront means the developer isn't waiting through the whole stream"}, {"start": 726.0, "text": "to learn what each request costs."}, {"start": 729.0, "text": "Finally, for each token your model produces,"}, {"start": 732.0, "text": "send a text delta the moment it arrives."}, {"start": 734.0, "text": "The framework streams those deltas to the session as they arrive,"}, {"start": 738.0, "text": "so users see the response appear word-by-word instead of all at once."}, {"start": 742.0, "text": "Earlier we saw how the framework caches EXECutors by configuration."}, {"start": 746.0, "text": "If your integration is stateful,"}, {"start": 749.0, "text": "holding a KV cache or persistent session between calls,"}, {"start": 752.0, "text": "that caching is what lets you minimize network churn and avoid redoing work."}, {"start": 757.0, "text": "Now let's look at how to design yours to take advantage of that,"}, {"start": 760.0, "text": "and how your EXECutor can preserve work across calls."}, {"start": 764.0, "text": "Your EXECutor receives the full transcript on every call to respond."}, {"start": 768.0, "text": "Here's what you processed last time,"}, {"start": 770.0, "text": "an instruction,"}, {"start": 771.0, "text": "a prompt,"}, {"start": 772.0, "text": "and the response you generated."}, {"start": 775.0, "text": "When the next call comes in"}, {"start": 776.0, "text": "you compare the new transcript to the one you saved from last time."}, {"start": 780.0, "text": "In most cases,"}, {"start": 782.0, "text": "new entries have simply been appended,"}, {"start": 784.0, "text": "a new prompt after the last response."}, {"start": 787.0, "text": "When that's the case,"}, {"start": 788.0, "text": "you can preserve your existing state"}, {"start": 790.0, "text": "and only process what's new."}, {"start": 793.0, "text": "But sometimes your comparison finds that entries have been removed or modified,"}, {"start": 797.0, "text": "for example, when the developer trims older entries to save context."}, {"start": 802.0, "text": "When that happens,"}, {"start": 803.0, "text": "you'll need to invalidate back to where the transcripts diverge."}, {"start": 806.0, "text": "The framework gives you the full transcript on every call."}, {"start": 809.0, "text": "Your EXECutor decides what counts as a match,"}, {"start": 811.0, "text": "and how to handle any changes."}, {"start": 813.0, "text": "Sometimes your model can't do exactly what the developer asked."}, {"start": 817.0, "text": "When that happens, your EXECutor has two choices: approximate or throw."}, {"start": 822.0, "text": "Be flexible where you can, and honor the developer's intent."}, {"start": 826.0, "text": "But sometimes there's no honest approximation."}, {"start": 828.0, "text": "If a developer sets a token limit,"}, {"start": 830.0, "text": "but also specifies a schema with required fields,"}, {"start": 833.0, "text": "there might not be a way to satisfy both."}, {"start": 836.0, "text": "So you throw."}, {"start": 837.0, "text": "Foundation Models ships LanguageModelError for exactly these cases:"}, {"start": 842.0, "text": "context window overflows, rate limits, refusals, and more."}, {"start": 847.0, "text": "Throw one of these,"}, {"start": 848.0, "text": "and any developer who's used the framework"}, {"start": 850.0, "text": "already knows how to handle it."}, {"start": 853.0, "text": "When the built-in LanguageModelError cases don't cover your situation,"}, {"start": 857.0, "text": "define your own error type."}, {"start": 859.0, "text": "Some failures only make sense in the context of your service:"}, {"start": 862.0, "text": "your subscription tiers, your features, your account states."}, {"start": 866.0, "text": "A purpose-built case name carries the intent,"}, {"start": 869.0, "text": "so a developer catching it knows exactly what happened."}, {"start": 872.0, "text": "Custom errors are powerful, and sometimes you need them."}, {"start": 875.0, "text": "But each one is a new case developers must learn,"}, {"start": 878.0, "text": "catch, and handle in their app."}, {"start": 880.0, "text": "Try to use a built-in LanguageModelError when it fits,"}, {"start": 883.0, "text": "and save the custom ones for failures only your service can produce."}, {"start": 886.0, "text": "We've finished implementing the protocol requirements."}, {"start": 890.0, "text": "Let's discuss how to handle authentication next."}, {"start": 893.0, "text": "Your job as a package author is to make it easy for developers to do the right thing."}, {"start": 898.0, "text": "If your initializer takes an API key as a string,"}, {"start": 901.0, "text": "developers will be tempted to take the path of least resistance."}, {"start": 904.0, "text": "Instead, help developers do the right thing"}, {"start": 907.0, "text": "by offering a token provider or sign in flow."}, {"start": 910.0, "text": "And if your package fetches access tokens on behalf of developers,"}, {"start": 914.0, "text": "make sure to persist them securely using Keychain."}, {"start": 917.0, "text": "Credential handling is half the story."}, {"start": 920.0, "text": "Device at-test-ation is the other half."}, {"start": 922.0, "text": "If you're shipping a cloud-based LanguageModel package,"}, {"start": 925.0, "text": "this is worth a deep look."}, {"start": 927.0, "text": "This related session walks through verifying the device,"}, {"start": 930.0, "text": "catching tampered builds,"}, {"start": 932.0, "text": "signing payloads,"}, {"start": 933.0, "text": "and using Apple's fraud signal to keep bad traffic off your service."}, {"start": 938.0, "text": "Check it out in \"Secure your apps with App Attest\"."}, {"start": 941.0, "text": "You've packaged your model,"}, {"start": 943.0, "text": "implemented the protocol,"}, {"start": 944.0, "text": "and handled authentication."}, {"start": 946.0, "text": "That means you've got a solid package for your LanguageModel,"}, {"start": 949.0, "text": "with all the fundamentals covered."}, {"start": 951.0, "text": "Now it's time to differentiate."}, {"start": 953.0, "text": "The protocol gives you room to shape LanguageModelSession"}, {"start": 956.0, "text": "around the abilities only your model offers."}, {"start": 959.0, "text": "Response metadata is a lightweight option"}, {"start": 962.0, "text": "to attach additional information to your responses,"}, {"start": 964.0, "text": "and give developers clear ways to access it."}, {"start": 968.0, "text": "You can attach your own custom metadata to the response."}, {"start": 971.0, "text": "Here, after streaming completes,"}, {"start": 973.0, "text": "our EXECutor sends tokensPerSecond"}, {"start": 976.0, "text": "and timeToFirstToken through the channel."}, {"start": 979.0, "text": "We recommend providing utilities or documentation"}, {"start": 981.0, "text": "that make it easy for developers to work with your metadata;"}, {"start": 984.0, "text": "clear keys,"}, {"start": 985.0, "text": "typed accessors,"}, {"start": 986.0, "text": "whatever makes sense."}, {"start": 988.0, "text": "Underneath, metadata is just a dictionary."}, {"start": 991.0, "text": "It can contain strings, numbers, and other built-in types."}, {"start": 995.0, "text": "But in some cases, you may need something more flexible."}, {"start": 999.0, "text": "Custom segments are the answer."}, {"start": 1001.0, "text": "You'll define a new segment type,"}, {"start": 1003.0, "text": "receive it in your EXECutor,"}, {"start": 1005.0, "text": "and stream results back through the same channel,"}, {"start": 1008.0, "text": "and the developer never has to leave LanguageModelSession to use them."}, {"start": 1012.0, "text": "Custom segment types let you extend the protocol."}, {"start": 1015.0, "text": "When a new modality comes along,"}, {"start": 1017.0, "text": "audio, video, whatever's next,"}, {"start": 1019.0, "text": "developers have a typed, structured way to send that data to your model."}, {"start": 1023.0, "text": "Here's how it works."}, {"start": 1025.0, "text": "First, you'll define a type that conforms to custom segment."}, {"start": 1029.0, "text": "Because custom segments are required to be PromptRepresentable,"}, {"start": 1033.0, "text": "developers can pass it directly in their prompts,"}, {"start": 1035.0, "text": "just like text."}, {"start": 1037.0, "text": "In your EXECutor, you'll receive this as a customSegment in the transcript,"}, {"start": 1041.0, "text": "alongside the text entries you're already handling."}, {"start": 1045.0, "text": "When your model responds,"}, {"start": 1046.0, "text": "you emit the result back through the channel"}, {"start": 1048.0, "text": "as a custom segment update."}, {"start": 1051.0, "text": "The segment ID controls whether you're adding a new segment,"}, {"start": 1053.0, "text": "or updating one you've already started streaming."}, {"start": 1056.0, "text": "This gives you full control over how results stream into the app."}, {"start": 1061.0, "text": "With custom segments in hand,"}, {"start": 1062.0, "text": "there's one more thing worth calling out:"}, {"start": 1064.0, "text": "a recommendation for server-side tools."}, {"start": 1067.0, "text": "Server-side tools are capabilities your model runs on its own,"}, {"start": 1071.0, "text": "like web search,"}, {"start": 1072.0, "text": "code execution,"}, {"start": 1073.0, "text": "or image generation."}, {"start": 1075.0, "text": "The model invokes them,"}, {"start": 1077.0, "text": "the server runs them,"}, {"start": 1078.0, "text": "and your EXECutor watches the results stream in."}, {"start": 1082.0, "text": "We'll walk through three levels of detail,"}, {"start": 1084.0, "text": "each surfacing more of the tool's work,"}, {"start": 1086.0, "text": "using web search as an example."}, {"start": 1089.0, "text": "Server-side tools are named, typed values on your model."}, {"start": 1093.0, "text": "The developer constructs the model with the tools they want,"}, {"start": 1096.0, "text": "and your EXECutor receives them through the model on every request,"}, {"start": 1099.0, "text": "the same way it receives every other capability the model declares."}, {"start": 1104.0, "text": "First, the simplest pattern:"}, {"start": 1106.0, "text": "run the tool privately and stream only the answer back."}, {"start": 1110.0, "text": "The tool grounds the model's response,"}, {"start": 1113.0, "text": "but its work stays inside your EXECutor."}, {"start": 1117.0, "text": "Each text delta you append gets streamed into the transcript by the framework,"}, {"start": 1121.0, "text": "with no trace of the tool that produced it."}, {"start": 1124.0, "text": "In addition to grounding the answer on the tool's output,"}, {"start": 1127.0, "text": "you can also attach additional metadata to the response."}, {"start": 1132.0, "text": "When a text delta carries metadata,"}, {"start": 1134.0, "text": "like a citation,"}, {"start": 1136.0, "text": "forward both to the channel,"}, {"start": 1138.0, "text": "and the framework attaches the metadata to the text segment in the transcript."}, {"start": 1143.0, "text": "And finally, you can choose to surface the tool's work itself."}, {"start": 1147.0, "text": "With custom segments,"}, {"start": 1148.0, "text": "you forward the tool's structured output to the channel,"}, {"start": 1151.0, "text": "alongside the text and any metadata,"}, {"start": 1154.0, "text": "giving apps everything the model produced along the way."}, {"start": 1158.0, "text": "Through one channel, the events you forward,"}, {"start": 1161.0, "text": "the metadata you attach,"}, {"start": 1162.0, "text": "and the custom segments you design,"}, {"start": 1164.0, "text": "server-side tools shape what apps using your package"}, {"start": 1167.0, "text": "can show their users."}, {"start": 1170.0, "text": "There's one more thing to keep in mind:"}, {"start": 1171.0, "text": "whether you're choosing a package or shipping one,"}, {"start": 1174.0, "text": "make sure everyone in the chain understands"}, {"start": 1176.0, "text": "the privacy implications of the model behind it."}, {"start": 1179.0, "text": "On-device and cloud-based models have very different privacy characteristics,"}, {"start": 1183.0, "text": "and your users deserve to know which they're getting."}, {"start": 1187.0, "text": "You've seen how to bring your model to the framework."}, {"start": 1189.0, "text": "These sessions show what developers will build with it."}, {"start": 1193.0, "text": "Check out \"Integrate On-Device AI Models into Your App Using Core AI\""}, {"start": 1198.0, "text": "for bundling local models directly into an app."}, {"start": 1201.0, "text": "\"Build with the new Apple Foundation Model on Private Cloud Compute\""}, {"start": 1205.0, "text": "goes deep on serverscale inference with Apple's privacy guarantees."}, {"start": 1209.0, "text": "And \"Build agentic app experiences with the Foundation Models framework\""}, {"start": 1213.0, "text": "shows how developers use dynamic profiles"}, {"start": 1216.0, "text": "to build multi-step,"}, {"start": 1217.0, "text": "tool-using workflows"}, {"start": 1219.0, "text": "on top of models like yours."}, {"start": 1221.0, "text": "We're excited about what's ahead."}, {"start": 1223.0, "text": "We hope to see a thriving ecosystem of LanguageModel packages,"}, {"start": 1227.0, "text": "giving Swift developers the freedom to choose"}, {"start": 1229.0, "text": "the model that's right for their app."}, {"start": 1231.0, "text": "We can't wait to see what you build."}]}