{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-382", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/382", "wordCount": 12677, "segments": [{"start": 2.0, "text": "Inside Apple Intelligence and Xcode: Special Presentation"}, {"start": 8.0, "text": "Welcome, everyone."}, {"start": 11.0, "text": "Wow, this is amazing."}, {"start": 14.0, "text": "It is genuinely great to have you here."}, {"start": 18.0, "text": "WWDC is one of our favorite moments of the year"}, {"start": 21.0, "text": "when we get to share what we've built for you,"}, {"start": 24.0, "text": "and then watch what you build with it."}, {"start": 27.0, "text": "Now quick show of hands."}, {"start": 28.0, "text": "How many of you are in the Steve Jobs Theater"}, {"start": 31.0, "text": "for the first time?"}, {"start": 33.0, "text": "Okay, yeah, I see you."}, {"start": 35.0, "text": "I hear you. Yes."}, {"start": 36.0, "text": "Today we're going to be"}, {"start": 38.0, "text": "going deep on Intelligence."}, {"start": 40.0, "text": "One of the biggest things"}, {"start": 42.0, "text": "that we've been working on this year."}, {"start": 44.0, "text": "Across the industry,"}, {"start": 45.0, "text": "the AI story is often complicated,"}, {"start": 48.0, "text": "with different components"}, {"start": 49.0, "text": "coming from different companies and providers."}, {"start": 52.0, "text": "And then you as the developer,"}, {"start": 54.0, "text": "are left to stitch it all together."}, {"start": 57.0, "text": "It's a lot of complexity"}, {"start": 58.0, "text": "before you get to the good part,"}, {"start": 60.0, "text": "building something that people love."}, {"start": 62.0, "text": "And that's not the story"}, {"start": 64.0, "text": "we want for you."}, {"start": 65.0, "text": "At Apple,"}, {"start": 66.0, "text": "AI isn't a layer you bolt on."}, {"start": 68.0, "text": "It's integrated into our platforms."}, {"start": 71.0, "text": "The silicon that runs it,"}, {"start": 73.0, "text": "the OS that integrates it,"}, {"start": 76.0, "text": "the frameworks and models that enable it,"}, {"start": 78.0, "text": "the tools that help"}, {"start": 80.0, "text": "you create with it."}, {"start": 81.0, "text": "And this year, it"}, {"start": 83.0, "text": "reaches all new levels"}, {"start": 84.0, "text": "so your ideas can go further."}, {"start": 87.0, "text": "Now we know that everyone is"}, {"start": 89.0, "text": "at different places with AI right now."}, {"start": 91.0, "text": "Some of you are just beginning to ask,"}, {"start": 94.0, "text": "what could this mean for my app?"}, {"start": 96.0, "text": "Others are deep in model development,"}, {"start": 99.0, "text": "pushing Apple Silicon to its limits."}, {"start": 102.0, "text": "And most of you"}, {"start": 103.0, "text": "are probably somewhere in between."}, {"start": 105.0, "text": "Curious, digging in with agentic coding"}, {"start": 108.0, "text": "and looking for the right place to start."}, {"start": 111.0, "text": "Apple's AI platform has something"}, {"start": 113.0, "text": "for all of you."}, {"start": 115.0, "text": "Like those of you"}, {"start": 116.0, "text": "who want to build faster."}, {"start": 118.0, "text": "You got into this because you love creating things."}, {"start": 122.0, "text": "You have ideas,"}, {"start": 123.0, "text": "you have users, and you want to move."}, {"start": 126.0, "text": "And that happens when AI becomes"}, {"start": 128.0, "text": "part of how you actually work."}, {"start": 131.0, "text": "Xcode 27 brings agentic coding,"}, {"start": 134.0, "text": "not as an isolated chat interface,"}, {"start": 136.0, "text": "but as a collaborator"}, {"start": 138.0, "text": "that sees your whole project."}, {"start": 140.0, "text": "It thinks in Swift"}, {"start": 141.0, "text": "and speaks in Apple frameworks."}, {"start": 143.0, "text": "Whether you're sketching out a new feature"}, {"start": 145.0, "text": "or asking it to write"}, {"start": 147.0, "text": "and validate a test,"}, {"start": 148.0, "text": "it meets you where you are"}, {"start": 150.0, "text": "accelerating the work"}, {"start": 152.0, "text": "without ever taking"}, {"start": 153.0, "text": "control of the wheel away from you."}, {"start": 156.0, "text": "Think of it as a development multiplier,"}, {"start": 158.0, "text": "the same craft"}, {"start": 159.0, "text": "with dramatically more momentum."}, {"start": 162.0, "text": "Then, for those of you who"}, {"start": 164.0, "text": "want to integrate your app throughout the system,"}, {"start": 167.0, "text": "you want your app to feel like it's connected"}, {"start": 169.0, "text": "and the way that people actually think"}, {"start": 171.0, "text": "and work throughout the day."}, {"start": 173.0, "text": "App Intents is how you get there."}, {"start": 175.0, "text": "Connecting your app"}, {"start": 177.0, "text": "to Apple Intelligence, and more."}, {"start": 179.0, "text": "When you express your app's content"}, {"start": 182.0, "text": "and its actions through them,"}, {"start": 183.0, "text": "something powerful happens."}, {"start": 185.0, "text": "Your app becomes part of the fabric of the system"}, {"start": 188.0, "text": "itself, woven into"}, {"start": 190.0, "text": "how people discover things,"}, {"start": 192.0, "text": "get things done, and move throughout their day."}, {"start": 195.0, "text": "It's searchable in Spotlight,"}, {"start": 197.0, "text": "available through natural language and in Siri."}, {"start": 200.0, "text": "Then integrated in Shortcuts and the action button."}, {"start": 203.0, "text": "That's AI"}, {"start": 204.0, "text": "that helps you surface your app"}, {"start": 206.0, "text": "in more places across the system."}, {"start": 208.0, "text": "Your app becomes more findable,"}, {"start": 210.0, "text": "more useful, and more"}, {"start": 212.0, "text": "present in people's lives."}, {"start": 214.0, "text": "Then, for those of you who"}, {"start": 216.0, "text": "want to ship intelligent experiences"}, {"start": 218.0, "text": "in your app,"}, {"start": 219.0, "text": "you're ready to build features"}, {"start": 221.0, "text": "that are powered by language models."}, {"start": 224.0, "text": "The Foundation Models framework"}, {"start": 226.0, "text": "gives you a single native Swift API"}, {"start": 228.0, "text": "to choose the right model"}, {"start": 230.0, "text": "for the right moment."}, {"start": 231.0, "text": "You can use the on device"}, {"start": 233.0, "text": "Apple Foundation Model for tasks that need to be fast,"}, {"start": 236.0, "text": "private, and available offline."}, {"start": 238.0, "text": "And many developers in the App Store"}, {"start": 240.0, "text": "Small Business program can use the Apple Foundation Model"}, {"start": 243.0, "text": "on Private Cloud Compute for deeper reasoning."}, {"start": 246.0, "text": "This is frontier-level"}, {"start": 248.0, "text": "Intelligence with unparalleled"}, {"start": 250.0, "text": "privacy protections,"}, {"start": 251.0, "text": "all at no cloud API cost."}, {"start": 254.0, "text": "Or you can connect to models from companies"}, {"start": 256.0, "text": "like Anthropic and Google and more"}, {"start": 258.0, "text": "for specialized use cases."}, {"start": 260.0, "text": "The Foundation Models framework enables"}, {"start": 263.0, "text": "language understanding, visual reasoning,"}, {"start": 265.0, "text": "and agentic workflows directly in the experiences"}, {"start": 268.0, "text": "that you ship."}, {"start": 270.0, "text": "This is the best way"}, {"start": 271.0, "text": "to build intelligent experiences"}, {"start": 273.0, "text": "with any language model,"}, {"start": 274.0, "text": "which gives you the flexibility"}, {"start": 276.0, "text": "you need to swap the right one"}, {"start": 277.0, "text": "in as your needs evolve."}, {"start": 280.0, "text": "Then,"}, {"start": 281.0, "text": "for those of you with custom models,"}, {"start": 283.0, "text": "sometimes you might have"}, {"start": 284.0, "text": "a specific model in mind"}, {"start": 286.0, "text": "that's trained for your domain"}, {"start": 288.0, "text": "or optimized for your use case,"}, {"start": 290.0, "text": "and you want it to run"}, {"start": 292.0, "text": "entirely on-device,"}, {"start": 293.0, "text": "with no external dependencies"}, {"start": 295.0, "text": "and no compromises."}, {"start": 298.0, "text": "Core AI is built for exactly that."}, {"start": 301.0, "text": "It's a brand new framework,"}, {"start": 302.0, "text": "integrated into our platforms,"}, {"start": 304.0, "text": "and is designed to be"}, {"start": 306.0, "text": "the best way to bring"}, {"start": 307.0, "text": "and run models on-device in your apps."}, {"start": 310.0, "text": "It delivers uncompromising performance"}, {"start": 313.0, "text": "through a modern Swift API"}, {"start": 315.0, "text": "with extensive tuning capabilities,"}, {"start": 317.0, "text": "taking full advantage"}, {"start": 319.0, "text": "of Apple Silicon."}, {"start": 320.0, "text": "No cloud calls."}, {"start": 322.0, "text": "No third party services."}, {"start": 324.0, "text": "Your models will perform"}, {"start": 325.0, "text": "efficiently across all devices,"}, {"start": 328.0, "text": "and because Core AI is built"}, {"start": 329.0, "text": "right into the platform,"}, {"start": 331.0, "text": "your apps automatically"}, {"start": 332.0, "text": "benefit from the latest fixes and enhancements"}, {"start": 335.0, "text": "with no extra work from you."}, {"start": 338.0, "text": "Then, if you're an AI enthusiast,"}, {"start": 340.0, "text": "MLX is an open source"}, {"start": 342.0, "text": "framework that lets you experiment with, train,"}, {"start": 345.0, "text": "research, or fine tune generative models,"}, {"start": 349.0, "text": "or you can run a local inference server."}, {"start": 351.0, "text": "It's a flexible, high performance framework"}, {"start": 354.0, "text": "that scales from numerical experiments"}, {"start": 356.0, "text": "to running the largest frontier models."}, {"start": 359.0, "text": "And with distributed training and inference,"}, {"start": 361.0, "text": "you can build a local compute cluster that delivers"}, {"start": 364.0, "text": "the bandwidth and latency"}, {"start": 366.0, "text": "that serious ML workloads demand."}, {"start": 368.0, "text": "This is AI for exploration"}, {"start": 371.0, "text": "and scaling."}, {"start": 372.0, "text": "Open source, optimized"}, {"start": 374.0, "text": "and built for wherever your curiosity takes you."}, {"start": 378.0, "text": "What makes Apple's AI platform"}, {"start": 380.0, "text": "distinct isn't any single piece,"}, {"start": 383.0, "text": "it's that every layer"}, {"start": 385.0, "text": "reinforces the others."}, {"start": 387.0, "text": "The silicon that powers Core AI and MLX"}, {"start": 390.0, "text": "is the same silicon"}, {"start": 391.0, "text": "that makes the on-device"}, {"start": 392.0, "text": "Apple Foundation Model practical."}, {"start": 395.0, "text": "The privacy architecture protecting"}, {"start": 396.0, "text": "Apple Foundation Models on-device"}, {"start": 398.0, "text": "and on Private Cloud Compute is the same one"}, {"start": 402.0, "text": "protecting App Intents."}, {"start": 403.0, "text": "The Swift APIs that surface"}, {"start": 405.0, "text": "intelligence in your apps are the same ones"}, {"start": 408.0, "text": "that Xcode agents use"}, {"start": 410.0, "text": "to help you build it."}, {"start": 412.0, "text": "This isn't a collection of features."}, {"start": 413.0, "text": "It's a coherent system"}, {"start": 415.0, "text": "designed from the ground up"}, {"start": 417.0, "text": "so that wherever you enter,"}, {"start": 419.0, "text": "you're working with something that makes AI faster,"}, {"start": 421.0, "text": "safer, and more capable at every level."}, {"start": 424.0, "text": "So whether you come to this"}, {"start": 425.0, "text": "as a builder, an integrator,"}, {"start": 427.0, "text": "a product engineer or a researcher,"}, {"start": 430.0, "text": "there is a place for you here."}, {"start": 432.0, "text": "The only question is"}, {"start": 434.0, "text": "where do you want to start?"}, {"start": 435.0, "text": "Today we're going to help you answer that"}, {"start": 437.0, "text": "by going into each of these areas in depth."}, {"start": 440.0, "text": "starting off with Xcode 27"}, {"start": 443.0, "text": "and the new agentic workflows"}, {"start": 444.0, "text": "that will change how you build."}, {"start": 447.0, "text": "Here's Ken and Jerome!"}, {"start": 454.0, "text": "Hey, good morning!"}, {"start": 457.0, "text": "Hey, I'm Ken."}, {"start": 459.0, "text": "Hi, I'm Jerome."}, {"start": 460.0, "text": "And today we're going to show you what agentic coding"}, {"start": 463.0, "text": "looks like in Xcode 27."}, {"start": 466.0, "text": "Now, for decades, writing software"}, {"start": 469.0, "text": "meant telling a computer"}, {"start": 470.0, "text": "exactly what to do."}, {"start": 472.0, "text": "One keystroke at a time."}, {"start": 476.0, "text": "But we built tools to help,"}, {"start": 477.0, "text": "like autocomplete,"}, {"start": 480.0, "text": "predictive code completion,"}, {"start": 483.0, "text": "and more recently"}, {"start": 484.0, "text": "LLMs that can edit single files."}, {"start": 487.0, "text": "Now each one of those shifted"}, {"start": 488.0, "text": "what a computer could do."}, {"start": 491.0, "text": "What's happening right now"}, {"start": 492.0, "text": "with agentic coding"}, {"start": 493.0, "text": "that is a much bigger shift."}, {"start": 497.0, "text": "From a single prompt,"}, {"start": 498.0, "text": "Xcode 27 works"}, {"start": 499.0, "text": "across your entire codebase."}, {"start": 502.0, "text": "It plans."}, {"start": 503.0, "text": "It applies changes across files."}, {"start": 505.0, "text": "It uses tools like previews,"}, {"start": 508.0, "text": "playgrounds, device"}, {"start": 509.0, "text": "integration, and more"}, {"start": 511.0, "text": "to bring your ideas to life."}, {"start": 513.0, "text": "You're no longer limited by how fast"}, {"start": 516.0, "text": "you can type or how much of an API you can remember."}, {"start": 519.0, "text": "So today"}, {"start": 521.0, "text": "we're going to build something fun,"}, {"start": 524.0, "text": "live, on stage together."}, {"start": 528.0, "text": "But first, Ken will give you a quick tour of Xcode 27."}, {"start": 532.0, "text": "All right!"}, {"start": 536.0, "text": "Here is Xcode 27,"}, {"start": 539.0, "text": "which takes the next big"}, {"start": 540.0, "text": "step in agentic coding."}, {"start": 543.0, "text": "And you work with agents"}, {"start": 544.0, "text": "through conversations."}, {"start": 546.0, "text": "You can start one"}, {"start": 547.0, "text": "right up here in the toolbar."}, {"start": 549.0, "text": "and each conversation now"}, {"start": 551.0, "text": "gets the full main editor area."}, {"start": 555.0, "text": "And right next to each conversation"}, {"start": 557.0, "text": "you get a dedicated space"}, {"start": 558.0, "text": "for everything that Xcode produces."}, {"start": 560.0, "text": "So plans, files, previews"}, {"start": 564.0, "text": "all together in one spot."}, {"start": 567.0, "text": "And you can have as many conversations going as you want."}, {"start": 570.0, "text": "It makes it really easy to"}, {"start": 572.0, "text": "learn and experiment with new ideas."}, {"start": 575.0, "text": "Okay, Jerome, let's build something."}, {"start": 578.0, "text": "All right!"}, {"start": 579.0, "text": "WWDC"}, {"start": 580.0, "text": "it's such an incredible moment"}, {"start": 582.0, "text": "each year. You get to learn"}, {"start": 583.0, "text": "about new technologies,"}, {"start": 585.0, "text": "meet developers from around the world,"}, {"start": 587.0, "text": "and get"}, {"start": 589.0, "text": "the pins."}, {"start": 593.0, "text": "Every year."}, {"start": 594.0, "text": "Yeah."}, {"start": 597.0, "text": "Every year there's a new design."}, {"start": 599.0, "text": "People trade them, people collect them, people"}, {"start": 602.0, "text": "put them on their jacket."}, {"start": 603.0, "text": "You know, I got a lot of those."}, {"start": 605.0, "text": "And I always wanted to build an app"}, {"start": 607.0, "text": "to track my collection."}, {"start": 609.0, "text": "Well, luckily for me,"}, {"start": 611.0, "text": "my colleague Summer, she gave me all the pins"}, {"start": 614.0, "text": "artwork."}, {"start": 615.0, "text": "I put them in a folder"}, {"start": 616.0, "text": "right here, organized by year."}, {"start": 618.0, "text": "And also have a subfolder"}, {"start": 620.0, "text": "“collected” for the pins I already have."}, {"start": 623.0, "text": "I also drew a sketch"}, {"start": 624.0, "text": "of what I have in mind."}, {"start": 626.0, "text": "It's a grid of pins organized by year."}, {"start": 628.0, "text": "The ones I have are in"}, {"start": 630.0, "text": "full color with a checkmark, and the ones I don't"}, {"start": 632.0, "text": "have are grayed out."}, {"start": 634.0, "text": "When I tap the pins,"}, {"start": 635.0, "text": "it opens a brand new view."}, {"start": 637.0, "text": "The ones I have, I see in full color,"}, {"start": 639.0, "text": "and the ones that I don't have yet,"}, {"start": 640.0, "text": "I can mark them as “collected.”"}, {"start": 642.0, "text": "So I'm going to use that sketch"}, {"start": 644.0, "text": "as, the beginning of our app here."}, {"start": 646.0, "text": "I'm going to copy it."}, {"start": 648.0, "text": "I'm going to start Xcode."}, {"start": 650.0, "text": "Start a new conversation."}, {"start": 653.0, "text": "Add the sketch"}, {"start": 655.0, "text": "and ask for what I want."}, {"start": 657.0, "text": "The “Pins” directory"}, {"start": 658.0, "text": "in my project folder contains"}, {"start": 660.0, "text": "my collection of pin images."}, {"start": 662.0, "text": "They are grouped by year, and each year has a “Collected”"}, {"start": 664.0, "text": "subdirectory, matching"}, {"start": 666.0, "text": "the current state of my collection."}, {"start": 668.0, "text": "And I like to create an app that can be used to track"}, {"start": 671.0, "text": "my collection based on the attached design."}, {"start": 674.0, "text": "And now Xcode jumps into plan mode."}, {"start": 677.0, "text": "Now plan mode is one of my favorite features"}, {"start": 678.0, "text": "of Xcode 27."}, {"start": 680.0, "text": "When working with coding agents,"}, {"start": 682.0, "text": "the best result comes from collaborating"}, {"start": 684.0, "text": "on the implementation and design first."}, {"start": 687.0, "text": "Right now, Xcode is parsing the prompt"}, {"start": 688.0, "text": "and exploring the pins folder pulling out anything"}, {"start": 691.0, "text": "that needs clarification before it starts writing any code."}, {"start": 696.0, "text": "Oh, now we"}, {"start": 697.0, "text": "have a few questions I need to answer, and we built this"}, {"start": 699.0, "text": "great Q&A experience directly in Xcode,"}, {"start": 702.0, "text": "so you can stay in the flow of your conversations."}, {"start": 705.0, "text": "All right, let's see what we have."}, {"start": 707.0, "text": "First questions."}, {"start": 708.0, "text": "How should we handle the pin images?"}, {"start": 710.0, "text": "I'm going to use an Asset Catalog for that."}, {"start": 712.0, "text": "This sounds like a good idea."}, {"start": 714.0, "text": "How should we persist"}, {"start": 715.0, "text": "which pins are collected?"}, {"start": 717.0, "text": "I think I could use user default."}, {"start": 719.0, "text": "It's a small app."}, {"start": 720.0, "text": "I can always implement Swift Data a little bit later."}, {"start": 723.0, "text": "And... Should user be able"}, {"start": 725.0, "text": "to uncollect a pin?"}, {"start": 727.0, "text": "Why would I want that!"}, {"start": 728.0, "text": "I have no plan to give away my collection"}, {"start": 730.0, "text": "so I'm going to say no to that one."}, {"start": 732.0, "text": "All right. And I'm going to send it."}, {"start": 735.0, "text": "Now Xcode is"}, {"start": 737.0, "text": "working with the agent to build a plan"}, {"start": 740.0, "text": "using the answers I just gave."}, {"start": 742.0, "text": "Think of this as Xcode and the agent"}, {"start": 744.0, "text": "doing a white-boarding session."}, {"start": 746.0, "text": "And this is just exploration phase,"}, {"start": 749.0, "text": "no code is being written yet."}, {"start": 751.0, "text": "So I can always add more details,"}, {"start": 753.0, "text": "change my mind on the answers I just gave,"}, {"start": 756.0, "text": "and more."}, {"start": 757.0, "text": "Let’s wait for the plan to finish."}, {"start": 762.0, "text": "And here it is."}, {"start": 764.0, "text": "We have a plan to create our app!"}, {"start": 766.0, "text": "Rendering in markdown."}, {"start": 767.0, "text": "Fully render markdown, right next to the conversation!"}, {"start": 770.0, "text": "All right let's have a look."}, {"start": 772.0, "text": "So we're going to build a SwiftUI app"}, {"start": 774.0, "text": "Perfect."}, {"start": 775.0, "text": "It's using my sketch as a starting point."}, {"start": 778.0, "text": "That's great."}, {"start": 779.0, "text": "I'm going to have filter"}, {"start": 780.0, "text": "at the top all “Collected” and “Missing.”"}, {"start": 782.0, "text": "I love that."}, {"start": 783.0, "text": "It’s going to use pin images from the Pins"}, {"start": 786.0, "text": "folder right next to our project."}, {"start": 787.0, "text": "And look, you know, it's adding all the decisions we make,"}, {"start": 790.0, "text": "with the questions before."}, {"start": 792.0, "text": "That sounds like a good plan."}, {"start": 794.0, "text": "Let's kick it off."}, {"start": 795.0, "text": "And it's going to take a few minutes."}, {"start": 799.0, "text": "All right."}, {"start": 800.0, "text": "While Jerome keeps an eye on that,"}, {"start": 802.0, "text": "let me tell you about what Xcode does behind the scenes"}, {"start": 805.0, "text": "when you're working with an agent."}, {"start": 807.0, "text": "Xcode leverages"}, {"start": 808.0, "text": "the full power of the best"}, {"start": 810.0, "text": "models and agents in Xcode"}, {"start": 811.0, "text": "so you can get the best possible results."}, {"start": 816.0, "text": "And we worked with Anthropic,"}, {"start": 818.0, "text": "OpenAI, Google"}, {"start": 820.0, "text": "to bring their agents right into Xcode, so you can choose"}, {"start": 822.0, "text": "the model that works best for you."}, {"start": 826.0, "text": "All right."}, {"start": 832.0, "text": "And with the ACPs"}, {"start": 833.0, "text": "so that's the Agent Client Protocol"}, {"start": 835.0, "text": "you can bring in any agent,"}, {"start": 837.0, "text": "any model, Including ones that run"}, {"start": 840.0, "text": "locally right on your Mac."}, {"start": 842.0, "text": "Now when you type a prompt,"}, {"start": 844.0, "text": "Xcode adds a lot of details"}, {"start": 847.0, "text": "and it gives agents the ability to use different tools."}, {"start": 849.0, "text": "So that you get the best"}, {"start": 851.0, "text": "possible results."}, {"start": 853.0, "text": "Now, the first thing Xcode does"}, {"start": 855.0, "text": "it layers in everything"}, {"start": 856.0, "text": "the model needs to know about your project."}, {"start": 859.0, "text": "So how it's configured."}, {"start": 861.0, "text": "How it's organized."}, {"start": 862.0, "text": "Your build settings, and more."}, {"start": 865.0, "text": "Then Xcode adds in"}, {"start": 866.0, "text": "context about itself."}, {"start": 868.0, "text": "So what tabs are open."}, {"start": 870.0, "text": "Current selection."}, {"start": 871.0, "text": "Your most visited files."}, {"start": 874.0, "text": "So when you say something like"}, {"start": 875.0, "text": "“fix this”, the word “this”"}, {"start": 878.0, "text": "well it actually means something."}, {"start": 881.0, "text": "And then finally Xcode"}, {"start": 882.0, "text": "gives agents a set of tools"}, {"start": 884.0, "text": "like building and testing, rendering previews"}, {"start": 887.0, "text": "with variants, interacting with the simulator"}, {"start": 889.0, "text": "localizing your app, debugging"}, {"start": 891.0, "text": "and more."}, {"start": 894.0, "text": "Now, for example,"}, {"start": 895.0, "text": "when you’re using a specific API,"}, {"start": 897.0, "text": "the agent can use the documentation tool."}, {"start": 900.0, "text": "We created a version"}, {"start": 901.0, "text": "of the documentation that's optimized for model"}, {"start": 903.0, "text": "use with vector embeddings"}, {"start": 905.0, "text": "for fast access."}, {"start": 908.0, "text": "Yeah."}, {"start": 910.0, "text": "Works really, really well."}, {"start": 913.0, "text": "And that gives the models more context"}, {"start": 915.0, "text": "when there using Apple APIs and frameworks."}, {"start": 919.0, "text": "Some of these tools"}, {"start": 920.0, "text": "are things that you could type into Terminal yourself"}, {"start": 924.0, "text": "like copy, move or delete."}, {"start": 926.0, "text": "They need to be used carefully."}, {"start": 929.0, "text": "So Xcode implements a new security approach"}, {"start": 931.0, "text": "that leverages new features"}, {"start": 933.0, "text": "in macOS 27."}, {"start": 936.0, "text": "There’s a new setting in Xcode that enables"}, {"start": 938.0, "text": "file access at the kernel level."}, {"start": 941.0, "text": "So when a sub-process"}, {"start": 942.0, "text": "like an agent wants to read or write a file,"}, {"start": 945.0, "text": "the kernel asks Xcode first."}, {"start": 948.0, "text": "Xcode can apply policies"}, {"start": 950.0, "text": "to those requests."}, {"start": 951.0, "text": "And it can trust your project"}, {"start": 953.0, "text": "and all of its files."}, {"start": 955.0, "text": "And the agent can run,"}, {"start": 957.0, "text": "but it can only touch"}, {"start": 959.0, "text": "what Xcode allows it to."}, {"start": 962.0, "text": "And when something does"}, {"start": 963.0, "text": "need your approval, Xcode asks."}, {"start": 966.0, "text": "The work keeps moving"}, {"start": 968.0, "text": "you always stay in control."}, {"start": 971.0, "text": "So you've always got the"}, {"start": 972.0, "text": "control that you need."}, {"start": 974.0, "text": "All right,"}, {"start": 975.0, "text": "Jerome, how are we looking over there?"}, {"start": 977.0, "text": "You know, Ken, we've been done for a while."}, {"start": 978.0, "text": "But I really didn't want to interrupt you."}, {"start": 980.0, "text": "That was super interesting."}, {"start": 981.0, "text": "I do appreciate- I appreciate that, Jerome."}, {"start": 983.0, "text": "Thank you."}, {"start": 984.0, "text": "All right. Our app is ready."}, {"start": 986.0, "text": "Let's just run it."}, {"start": 990.0, "text": "And here it is!"}, {"start": 992.0, "text": "Right."}, {"start": 998.0, "text": "Pretty good already."}, {"start": 1000.0, "text": "It's already seeded with my collection"}, {"start": 1002.0, "text": "the pins I have in full color, and the ones"}, {"start": 1003.0, "text": "I unfortunately don't have yet are still greyed out."}, {"start": 1006.0, "text": "I can tap on a pin."}, {"start": 1007.0, "text": "I see it in full color."}, {"start": 1009.0, "text": "Now, this looks really great."}, {"start": 1011.0, "text": "123 pins,"}, {"start": 1013.0, "text": "zero set up, one prompt,"}, {"start": 1015.0, "text": "and we already have a pretty good app, right?"}, {"start": 1018.0, "text": "And that's really what agentic coding unlocks"}, {"start": 1020.0, "text": "from an idea to a working app."}, {"start": 1023.0, "text": "You describe what you want,"}, {"start": 1025.0, "text": "you make the decisions, and the agent"}, {"start": 1027.0, "text": "does the heavy lifting."}, {"start": 1028.0, "text": "But it doesn't stop here."}, {"start": 1030.0, "text": "Once you have that solid foundation,"}, {"start": 1032.0, "text": "you can keep going,"}, {"start": 1034.0, "text": "keep exploring, keep creating."}, {"start": 1037.0, "text": "Now I have really fun ideas"}, {"start": 1040.0, "text": "to make our app even better!"}, {"start": 1043.0, "text": "I think that, when I collect a pin"}, {"start": 1046.0, "text": "it should burst into color."}, {"start": 1048.0, "text": "It should scale up,"}, {"start": 1049.0, "text": "It should spin."}, {"start": 1051.0, "text": "It should- Hey, Jerome?"}, {"start": 1053.0, "text": "I've got an idea."}, {"start": 1054.0, "text": "Do you think we could add"}, {"start": 1057.0, "text": "some sparkles?"}, {"start": 1058.0, "text": "Ken-"}, {"start": 1059.0, "text": "I love sparkles."}, {"start": 1060.0, "text": "It's WWDC."}, {"start": 1061.0, "text": "Of course we have sparkles, right?"}, {"start": 1063.0, "text": "I love it."}, {"start": 1064.0, "text": "And when I touch a pin I want to see a really cool"}, {"start": 1066.0, "text": "holographic 3D effect. Right?"}, {"start": 1069.0, "text": "So let's just build that."}, {"start": 1073.0, "text": "We're going to add an animation when marking a pin as collected."}, {"start": 1076.0, "text": "I want the pin to gain a color,"}, {"start": 1077.0, "text": "scale and spin rapidly."}, {"start": 1079.0, "text": "Also, add sparkles."}, {"start": 1082.0, "text": "And add a holographic 3D effect"}, {"start": 1083.0, "text": "when dragging a pin on screen."}, {"start": 1086.0, "text": "Actually I have a second idea."}, {"start": 1089.0, "text": "I'm opening a new conversation here"}, {"start": 1092.0, "text": "and this is what I will ask."}, {"start": 1094.0, "text": "I want to be able"}, {"start": 1095.0, "text": "to see the pins I collected"}, {"start": 1096.0, "text": "on a corkboard."}, {"start": 1097.0, "text": "That's how I used to collect pins when I was a kid."}, {"start": 1100.0, "text": "So we'll just do that."}, {"start": 1102.0, "text": "And right now, this is what I ask."}, {"start": 1104.0, "text": "I want to see my current collection"}, {"start": 1106.0, "text": "on a realistic corkboard background."}, {"start": 1108.0, "text": "The pins should be scattered randomly without overlapping."}, {"start": 1110.0, "text": "That would actually make a great wallpaper."}, {"start": 1113.0, "text": "And now both conversations"}, {"start": 1114.0, "text": "are working in parallel."}, {"start": 1116.0, "text": "Here in the Navigator,"}, {"start": 1118.0, "text": "I can track the progress of each of them."}, {"start": 1120.0, "text": "And if an agent requires my attention for example"}, {"start": 1123.0, "text": "to allow specific tools to be used,"}, {"start": 1125.0, "text": "I get a status message right here."}, {"start": 1127.0, "text": "I can switch back and forth"}, {"start": 1129.0, "text": "to track the progress of each."}, {"start": 1131.0, "text": "And I use this a lot"}, {"start": 1132.0, "text": "to read documentation about new APIs,"}, {"start": 1135.0, "text": "or get some explanation on the code"}, {"start": 1136.0, "text": "that's just been generated."}, {"start": 1138.0, "text": "I think it's a great way to build"}, {"start": 1139.0, "text": "and learn at the same time."}, {"start": 1142.0, "text": "All right."}, {"start": 1143.0, "text": "Apparently the corkboard is building... already done."}, {"start": 1146.0, "text": "And, the animation,"}, {"start": 1147.0, "text": "one is building, too."}, {"start": 1149.0, "text": "I'm excited to see the result of that."}, {"start": 1154.0, "text": "All right."}, {"start": 1156.0, "text": "Looks like we're ready."}, {"start": 1158.0, "text": "Let's just run it."}, {"start": 1163.0, "text": "All right."}, {"start": 1164.0, "text": "There's a new icon right here."}, {"start": 1166.0, "text": "I'm going to click on it."}, {"start": 1167.0, "text": "And here it is."}, {"start": 1172.0, "text": "That looks really great!"}, {"start": 1173.0, "text": "Right?"}, {"start": 1175.0, "text": "Lots of great memories right here."}, {"start": 1178.0, "text": "And then I'm going to tap a pin like,"}, {"start": 1180.0, "text": "you know, the new Dogcow from this year."}, {"start": 1182.0, "text": "Mark it as collected,"}, {"start": 1185.0, "text": "and here they are."}, {"start": 1186.0, "text": "Nice- the sparkles."}, {"start": 1188.0, "text": "Love it."}, {"start": 1190.0, "text": "Now watch this"}, {"start": 1192.0, "text": "When I drag the pin around,"}, {"start": 1194.0, "text": "I get this really cool"}, {"start": 1195.0, "text": "3D holographic effect on the pin."}, {"start": 1198.0, "text": "How cool is that."}, {"start": 1205.0, "text": "So agentic coding"}, {"start": 1206.0, "text": "together with Apple platforms,"}, {"start": 1208.0, "text": "frameworks and tools,"}, {"start": 1210.0, "text": "helps bring ideas to life."}, {"start": 1213.0, "text": "And you can build those ideas for yourself,"}, {"start": 1216.0, "text": "for your friends, your communities."}, {"start": 1218.0, "text": "Agents in Xcode can help"}, {"start": 1220.0, "text": "with all kinds of things"}, {"start": 1222.0, "text": "like adopting new APIs,"}, {"start": 1223.0, "text": "making your app more accessible, and more."}, {"start": 1227.0, "text": "For example,"}, {"start": 1229.0, "text": "when using an app,"}, {"start": 1230.0, "text": "there is nothing that makes"}, {"start": 1232.0, "text": "it feel more personal"}, {"start": 1234.0, "text": "than when it’s localized in your own"}, {"start": 1235.0, "text": "language."}, {"start": 1237.0, "text": "Let’s talk about how agentic coding"}, {"start": 1239.0, "text": "can help you with that."}, {"start": 1240.0, "text": "Localizing an app"}, {"start": 1241.0, "text": "is way more than just"}, {"start": 1244.0, "text": "translating strings word for word."}, {"start": 1246.0, "text": "It’s about understanding how each string is used"}, {"start": 1249.0, "text": "in the context of your app."}, {"start": 1252.0, "text": "For example, the same word can mean"}, {"start": 1254.0, "text": "many different things depending on how it’s used."}, {"start": 1258.0, "text": "And when you ask the agent in Xcode"}, {"start": 1260.0, "text": "to localize your app,"}, {"start": 1262.0, "text": "there’s three stages that it goes through"}, {"start": 1264.0, "text": "to ensure maximum accuracy."}, {"start": 1267.0, "text": "So first, Xcode adds"}, {"start": 1269.0, "text": "the language to your project settings."}, {"start": 1271.0, "text": "Then it builds your project"}, {"start": 1273.0, "text": "so that it can discover all the localizable strings."}, {"start": 1277.0, "text": "It creates a strings catalog"}, {"start": 1278.0, "text": "if you didn't already have one,"}, {"start": 1280.0, "text": "and then it adds slots for each and every string."}, {"start": 1284.0, "text": "Then the agent does the actual translation."}, {"start": 1287.0, "text": "And to get that done"}, {"start": 1289.0, "text": "as fast as possible"}, {"start": 1290.0, "text": "it creates subagents"}, {"start": 1292.0, "text": "to do that translation in parallel."}, {"start": 1295.0, "text": "Xcode provides a lot of context"}, {"start": 1297.0, "text": "about the strings themselves."}, {"start": 1298.0, "text": "Things like the surrounding code"}, {"start": 1301.0, "text": "or the UI that they appear in."}, {"start": 1304.0, "text": "And some strings can even vary"}, {"start": 1306.0, "text": "by plural or device."}, {"start": 1307.0, "text": "So Xcode makes that info"}, {"start": 1309.0, "text": "available to the agents as well, so you get the correct"}, {"start": 1312.0, "text": "plural and device variations."}, {"start": 1315.0, "text": "Once the translation is complete,"}, {"start": 1317.0, "text": "the primary agent goes back, checks"}, {"start": 1319.0, "text": "everything that the subagents did."}, {"start": 1323.0, "text": "And then pulls it all back into Xcode"}, {"start": 1326.0, "text": "where you can review and refine"}, {"start": 1328.0, "text": "all of those strings."}, {"start": 1330.0, "text": "You know, Ken"}, {"start": 1331.0, "text": "I would love to localize the app"}, {"start": 1332.0, "text": "for all the developers in the audience"}, {"start": 1335.0, "text": "but we have developers who have traveled"}, {"start": 1337.0, "text": "from 65 different countries, right?"}, {"start": 1339.0, "text": "It would probably take a while."}, {"start": 1341.0, "text": "So I'll do that later."}, {"start": 1342.0, "text": "But for today,"}, {"start": 1343.0, "text": "how about I add a feature"}, {"start": 1345.0, "text": "that let's me collect pins"}, {"start": 1348.0, "text": "from a picture."}, {"start": 1349.0, "text": "Oh, I love that idea."}, {"start": 1350.0, "text": "That's a great idea. All right."}, {"start": 1352.0, "text": "I'm going to do that."}, {"start": 1353.0, "text": "So here's what I simply ask."}, {"start": 1355.0, "text": "Allow me to collect a new pin"}, {"start": 1357.0, "text": "by taking a photo."}, {"start": 1361.0, "text": "All right. While we let that run,"}, {"start": 1363.0, "text": "let me tell you about how you can extend"}, {"start": 1365.0, "text": "Xcode’s capabilities."}, {"start": 1368.0, "text": "Now, everything we’ve shown you"}, {"start": 1369.0, "text": "so far is built right into Xcode."}, {"start": 1372.0, "text": "But you can also bring your own skills, your own tools,"}, {"start": 1375.0, "text": "and your own agents through plugins."}, {"start": 1378.0, "text": "Now, that's a format created by Anthropic."}, {"start": 1380.0, "text": "Already widely adopted by the community."}, {"start": 1383.0, "text": "Now, a plugin"}, {"start": 1384.0, "text": "can contain skills which are just simple markdown files"}, {"start": 1387.0, "text": "and those capture how to do a particular task."}, {"start": 1390.0, "text": "It can contain MCP tools,"}, {"start": 1392.0, "text": "for connecting to anything"}, {"start": 1393.0, "text": "from a database to a design system."}, {"start": 1397.0, "text": "And then we added"}, {"start": 1398.0, "text": "something really cool on top of that"}, {"start": 1400.0, "text": "With the Agent Client Protocol,"}, {"start": 1401.0, "text": "a plugin can also bring in"}, {"start": 1403.0, "text": "the agent of your choice."}, {"start": 1406.0, "text": "Installing one?"}, {"start": 1407.0, "text": "Easy. Paste a GitHub URL"}, {"start": 1409.0, "text": "Xcode clones it, installs it, manages it."}, {"start": 1414.0, "text": "And partners like Figma and GitHub"}, {"start": 1416.0, "text": "already have shipped their own plugins."}, {"start": 1418.0, "text": "One click to set it up,"}, {"start": 1420.0, "text": "super simple."}, {"start": 1422.0, "text": "And like I mentioned before,"}, {"start": 1424.0, "text": "you are always in control."}, {"start": 1425.0, "text": "You decide what each plugin can see,"}, {"start": 1428.0, "text": "everything else needs your approval."}, {"start": 1431.0, "text": "Okay."}, {"start": 1432.0, "text": "How are we looking over there, Jerome?"}, {"start": 1434.0, "text": "It's still building."}, {"start": 1435.0, "text": "And, you know, it was a simple prompt,"}, {"start": 1438.0, "text": "but actually, there's a lot of things behind it, right?"}, {"start": 1441.0, "text": "Think of this as,"}, {"start": 1442.0, "text": "we're going to create"}, {"start": 1443.0, "text": "an experience with the camera, right?"}, {"start": 1445.0, "text": "And then we're going to have a vision model"}, {"start": 1447.0, "text": "that will compare the image we're taking"}, {"start": 1450.0, "text": "with all the images from my assets catalog."}, {"start": 1452.0, "text": "Right."}, {"start": 1453.0, "text": "And then we're going to create the entire experience around it."}, {"start": 1456.0, "text": "In the app we already built."}, {"start": 1457.0, "text": "So that single prompt is actually a very complex features"}, {"start": 1460.0, "text": "that leverages lots of technology and platforms."}, {"start": 1464.0, "text": "So let's wait for it to finish."}, {"start": 1465.0, "text": "Now it's updating the info"}, {"start": 1467.0, "text": "pin list to use the camera."}, {"start": 1468.0, "text": "That's perfect."}, {"start": 1469.0, "text": "And now it’s building the project."}, {"start": 1471.0, "text": "All right. Build succeeded."}, {"start": 1472.0, "text": "I'm excited. We're going to run that"}, {"start": 1474.0, "text": "on a real device, so we can use the camera."}, {"start": 1476.0, "text": "So I'm going to select my phone here,"}, {"start": 1478.0, "text": "and just build it."}, {"start": 1481.0, "text": "All right."}, {"start": 1483.0, "text": "I'm going to lunch it on my phone."}, {"start": 1485.0, "text": "Perfect."}, {"start": 1487.0, "text": "And we're going to tap the camera icon."}, {"start": 1490.0, "text": "And here I have this Finder"}, {"start": 1492.0, "text": "pin from 2025. I love this design."}, {"start": 1494.0, "text": "I'm going to snap a picture of it."}, {"start": 1497.0, "text": "Here's a picture."}, {"start": 1499.0, "text": "And here it is."}, {"start": 1500.0, "text": "Recognized immediately."}, {"start": 1508.0, "text": "I think I'm going to keep that one."}, {"start": 1509.0, "text": "I'm going to add it to my collection."}, {"start": 1511.0, "text": "And here they are. The sparkles."}, {"start": 1514.0, "text": "Love it."}, {"start": 1515.0, "text": "There's so many more ideas I have for this app, but"}, {"start": 1518.0, "text": "unfortunately that’s all the time we have here today."}, {"start": 1522.0, "text": "Hey, Jerome, before we leave today,"}, {"start": 1525.0, "text": "I do have a little surprise for you."}, {"start": 1527.0, "text": "Oh."}, {"start": 1528.0, "text": "I have..."}, {"start": 1531.0, "text": "a never..."}, {"start": 1534.0, "text": "before seen special edition"}, {"start": 1538.0, "text": "WWDC26 pin for your collection!"}, {"start": 1542.0, "text": "I think I still have room for it"}, {"start": 1543.0, "text": "on my jacket right here."}, {"start": 1545.0, "text": "Okay."}, {"start": 1546.0, "text": "That's pretty good"}, {"start": 1547.0, "text": "That looks amazing."}, {"start": 1550.0, "text": "Is anybody else interested in one?"}, {"start": 1554.0, "text": "I thought so. Great news, we've got one for everyone here."}, {"start": 1566.0, "text": "All right. Well, 20 minutes ago, we started with a folder of images,"}, {"start": 1570.0, "text": "and an idea."}, {"start": 1572.0, "text": "Now we have a real app."}, {"start": 1574.0, "text": "Pin collection by year."}, {"start": 1577.0, "text": "Tap one,"}, {"start": 1578.0, "text": "colors, drag it,"}, {"start": 1581.0, "text": "holographic."}, {"start": 1582.0, "text": "Snap a picture of a new pin,"}, {"start": 1584.0, "text": "it gets recognized."}, {"start": 1587.0, "text": "All built live,"}, {"start": 1588.0, "text": "in the time that it took"}, {"start": 1589.0, "text": "to explain this."}, {"start": 1591.0, "text": "And best of all,"}, {"start": 1592.0, "text": "it even included a few sparkles"}, {"start": 1595.0, "text": "Next, here’s Michael"}, {"start": 1597.0, "text": "to talk about Siri and App Intents!"}, {"start": 1607.0, "text": "Welcome, everyone,"}, {"start": 1608.0, "text": "and thanks so much for coming out today!"}, {"start": 1611.0, "text": "My name is Michael Gorbach and I run the App Intents “core team”"}, {"start": 1614.0, "text": "here at Apple."}, {"start": 1616.0, "text": "Today, I'm super excited to show you"}, {"start": 1618.0, "text": "how you can bring your app’s content and actions"}, {"start": 1621.0, "text": "to Siri."}, {"start": 1623.0, "text": "This year, Siri is taking a huge step forward."}, {"start": 1627.0, "text": "It's more capable, more contextual,"}, {"start": 1629.0, "text": "and more personal than ever before."}, {"start": 1631.0, "text": "Siri can now"}, {"start": 1632.0, "text": "take action inside your app, driven entirely"}, {"start": 1635.0, "text": "by natural language."}, {"start": 1638.0, "text": "It's got a rich understanding"}, {"start": 1639.0, "text": "of people’s personal context,"}, {"start": 1641.0, "text": "allowing it to search and reason"}, {"start": 1643.0, "text": "over your app's content in a safe and private way."}, {"start": 1647.0, "text": "And Siri's aware of what's"}, {"start": 1649.0, "text": "on screen when you interact with it,"}, {"start": 1651.0, "text": "so people naturally can refer"}, {"start": 1652.0, "text": "to what's right in front of them."}, {"start": 1654.0, "text": "You can make your app a part of all of this"}, {"start": 1657.0, "text": "by adopting APIs"}, {"start": 1658.0, "text": "from the App Intents framework."}, {"start": 1661.0, "text": "That's the foundation for integrating"}, {"start": 1663.0, "text": "your app with Siri and Apple Intelligence."}, {"start": 1666.0, "text": "It provides a structured way"}, {"start": 1667.0, "text": "to describe what your app can do"}, {"start": 1669.0, "text": "and the content within it."}, {"start": 1672.0, "text": "By adopting these APIs,"}, {"start": 1674.0, "text": "you're really modeling your app's data and capabilities"}, {"start": 1677.0, "text": "so they become first-class citizens"}, {"start": 1679.0, "text": "in the system."}, {"start": 1681.0, "text": "Once your app's plugged in,"}, {"start": 1682.0, "text": "Siri can handle the heavy lifting"}, {"start": 1684.0, "text": "finding content,"}, {"start": 1686.0, "text": "taking action with natural language,"}, {"start": 1688.0, "text": "and understanding context to let people do"}, {"start": 1691.0, "text": "amazing things with your software"}, {"start": 1693.0, "text": "and their devices."}, {"start": 1696.0, "text": "Let's start with a demo."}, {"start": 1705.0, "text": "I've been working on"}, {"start": 1706.0, "text": "an app called HotTickety that handles"}, {"start": 1709.0, "text": "alarms, timers,"}, {"start": 1711.0, "text": "and stopwatches."}, {"start": 1713.0, "text": "I do like to eat, but sometimes I get"}, {"start": 1715.0, "text": "caught up in things and kind of need a reminder."}, {"start": 1719.0, "text": "Let me show you how I can interact"}, {"start": 1721.0, "text": "with HotTickety through natural language"}, {"start": 1723.0, "text": "with Siri."}, {"start": 1725.0, "text": "If I open the Siri app,"}, {"start": 1727.0, "text": "let's start a new conversation."}, {"start": 1728.0, "text": "I can say something like “2 p.m. alarm"}, {"start": 1733.0, "text": "for panini lunch"}, {"start": 1739.0, "text": "in"}, {"start": 1741.0, "text": "HotTickety."}, {"start": 1747.0, "text": "And Siri would do that for me."}, {"start": 1749.0, "text": "Notice that Siri is able to take action"}, {"start": 1752.0, "text": "on my behalf within this app."}, {"start": 1755.0, "text": "If I tap on that alarm,"}, {"start": 1757.0, "text": "Siri will open it for me in HotTickety"}, {"start": 1759.0, "text": "so I can see the details."}, {"start": 1762.0, "text": "Cool."}, {"start": 1763.0, "text": "If I go back to Siri"}, {"start": 1765.0, "text": "we can try something like,"}, {"start": 1767.0, "text": "“Let me know"}, {"start": 1770.0, "text": "when 30 seconds are up.”"}, {"start": 1779.0, "text": "And Siri will start a timer in HotTickety for me."}, {"start": 1781.0, "text": "Notice it's running up there at the top."}, {"start": 1784.0, "text": "Also notice I'm just using very natural phrasing."}, {"start": 1787.0, "text": "HotTickety didn't have to do anything special"}, {"start": 1789.0, "text": "to handle all the myriad"}, {"start": 1790.0, "text": "different ways someone could want to start a timer."}, {"start": 1794.0, "text": "And if I type something like,"}, {"start": 1796.0, "text": "“Cancel it.”"}, {"start": 1799.0, "text": "Siri will go ahead and do that for me as well."}, {"start": 1801.0, "text": "Siri has context"}, {"start": 1802.0, "text": "on the conversation and remembers"}, {"start": 1804.0, "text": "what just happened."}, {"start": 1807.0, "text": "Let’s see if the can find some information"}, {"start": 1809.0, "text": "from HotTickety for me."}, {"start": 1811.0, "text": "I can ask it"}, {"start": 1812.0, "text": "something like, “Check my alarms"}, {"start": 1815.0, "text": "in HotTickety.”"}, {"start": 1820.0, "text": "And it will search for them"}, {"start": 1823.0, "text": "and find all of my alarms."}, {"start": 1826.0, "text": "One last thing."}, {"start": 1827.0, "text": "If I go back to the app, I'm looking at the schedule"}, {"start": 1830.0, "text": "and I think I'm a little bit still too full from breakfast."}, {"start": 1833.0, "text": "So I'm going to put off lunch."}, {"start": 1836.0, "text": "“Delete the fourth one.”"}, {"start": 1842.0, "text": "Siri knows what's on screen,"}, {"start": 1843.0, "text": "will confirm and delete it for me."}, {"start": 1847.0, "text": "And that is how I use"}, {"start": 1848.0, "text": "HotTickety with natural language through Siri."}, {"start": 1857.0, "text": "So today I'm going to walk you through"}, {"start": 1859.0, "text": "how all the features you just saw"}, {"start": 1861.0, "text": "break down into straightforward"}, {"start": 1863.0, "text": "adoptions of powerful APIs"}, {"start": 1865.0, "text": "that you can try in your app."}, {"start": 1867.0, "text": "We'll start by talking about"}, {"start": 1868.0, "text": "entities, intents, and schemas"}, {"start": 1871.0, "text": "which are the basic building blocks that you use when you adopt"}, {"start": 1873.0, "text": "App Intents to build a Siri experience."}, {"start": 1876.0, "text": "Then we're going to cover"}, {"start": 1877.0, "text": "how you can use entity annotations to augment"}, {"start": 1880.0, "text": "Siri’s on screen awareness. Like you just saw."}, {"start": 1883.0, "text": "And finally,"}, {"start": 1884.0, "text": "we'll talk about interaction donations,"}, {"start": 1886.0, "text": "and how they allow Siri to"}, {"start": 1887.0, "text": "learn from user behavior."}, {"start": 1891.0, "text": "Let's start with entity schemas."}, {"start": 1895.0, "text": "Say you're cooking dinner,"}, {"start": 1896.0, "text": "and you ask Siri to start a 12 minute pasta timer."}, {"start": 1899.0, "text": "The noun “timer”, as I use it here,"}, {"start": 1902.0, "text": "does have a direct representation"}, {"start": 1904.0, "text": "in the App Intents framework."}, {"start": 1905.0, "text": "It is an app entity."}, {"start": 1907.0, "text": "When you adopt App Intents,"}, {"start": 1909.0, "text": "your app will have its own key"}, {"start": 1910.0, "text": "types of content that will become"}, {"start": 1912.0, "text": "your entities."}, {"start": 1914.0, "text": "An event if you've got a calendar app."}, {"start": 1916.0, "text": "A mail message for your mail app,"}, {"start": 1918.0, "text": "Or an album or photo"}, {"start": 1920.0, "text": "if you work on a photos app."}, {"start": 1925.0, "text": "Entities have an identifier"}, {"start": 1927.0, "text": "that the system knows them by,"}, {"start": 1929.0, "text": "plus a set of properties."}, {"start": 1931.0, "text": "For example, HotTickety’s timer entity includes a duration,"}, {"start": 1934.0, "text": "and a label."}, {"start": 1935.0, "text": "It's also got an enum that describe its state,"}, {"start": 1937.0, "text": "showing that it's currently running."}, {"start": 1941.0, "text": "In code,"}, {"start": 1942.0, "text": "there easy to create."}, {"start": 1945.0, "text": "It's just a Swift struct, and each property"}, {"start": 1947.0, "text": "is marked with @Property."}, {"start": 1951.0, "text": "Okay, now that we have"}, {"start": 1953.0, "text": "an entity, let's talk about how Siri can work with it."}, {"start": 1956.0, "text": "That's where App Schemas come in."}, {"start": 1958.0, "text": "App Schemas give Siri"}, {"start": 1959.0, "text": "an understanding of common concepts, like messages,"}, {"start": 1962.0, "text": "albums, events, or timers."}, {"start": 1966.0, "text": "Groups of schemas"}, {"start": 1967.0, "text": "are organized into domains like this"}, {"start": 1968.0, "text": "clock domain, which covers alarms, timers,"}, {"start": 1971.0, "text": "and stopwatches."}, {"start": 1972.0, "text": "Super useful"}, {"start": 1973.0, "text": "for HotTickety and also great for your app"}, {"start": 1975.0, "text": "if it includes similar features."}, {"start": 1979.0, "text": "HotTickety’s timer entity adopts the timer"}, {"start": 1983.0, "text": "AppEntity schema,"}, {"start": 1984.0, "text": "which lets Siri recognize it as a timer,"}, {"start": 1986.0, "text": "and defines the properties that it needs to have."}, {"start": 1989.0, "text": "Apple's Clock app does too,"}, {"start": 1991.0, "text": "and so can your app,"}, {"start": 1992.0, "text": "if you have a timer feature."}, {"start": 1994.0, "text": "When I add the schema at the top of that entity struct,"}, {"start": 1997.0, "text": "Xcode helps me get the properties just right."}, {"start": 2000.0, "text": "I can even delete the ‘@Property’ wrappers"}, {"start": 2002.0, "text": "because the schema is taking care of them for me."}, {"start": 2006.0, "text": "Okay, let's try another phrase."}, {"start": 2009.0, "text": "Responding to this one"}, {"start": 2010.0, "text": "means Siri needs to figure out"}, {"start": 2012.0, "text": "the right timer entity to answer"}, {"start": 2014.0, "text": "this question."}, {"start": 2016.0, "text": "The good news is that"}, {"start": 2017.0, "text": "when receiving a request like this"}, {"start": 2018.0, "text": "Siri actually already knows"}, {"start": 2020.0, "text": "what's on your device."}, {"start": 2022.0, "text": "Because Apple Intelligence"}, {"start": 2023.0, "text": "has a datastore for entities:"}, {"start": 2025.0, "text": "the semantic index."}, {"start": 2027.0, "text": "Siri can search"}, {"start": 2028.0, "text": "not just based on keywords,"}, {"start": 2030.0, "text": "but based on the meanings of words,"}, {"start": 2032.0, "text": "or the values of specific properties."}, {"start": 2034.0, "text": "And your entities can be included in it too."}, {"start": 2039.0, "text": "Adding HotTickety’s timers"}, {"start": 2041.0, "text": "to the semantic index is simple."}, {"start": 2043.0, "text": "I start by adopting"}, {"start": 2044.0, "text": "the ‘IndexedEntity’ protocol."}, {"start": 2046.0, "text": "And then I call ‘indexAppEntities’"}, {"start": 2048.0, "text": "whenever my timers are created, renamed, or deleted,"}, {"start": 2051.0, "text": "so Siri can know about them and be ready to take action."}, {"start": 2055.0, "text": "This works super well for an app"}, {"start": 2057.0, "text": "like HotTickety where all the data is local."}, {"start": 2059.0, "text": "For apps that have huge catalogs of data,"}, {"start": 2061.0, "text": "or data stored on servers,"}, {"start": 2063.0, "text": "Siri can also retrieve entities"}, {"start": 2065.0, "text": "from them directly, through queries."}, {"start": 2067.0, "text": "You can learn more about those in our sessions"}, {"start": 2069.0, "text": "and documentation."}, {"start": 2073.0, "text": "Schemas provide a common structure"}, {"start": 2075.0, "text": "for entities representing the same kind of content."}, {"start": 2078.0, "text": "The cool thing here"}, {"start": 2079.0, "text": "is that since HotTickety’s timer entry conforms to the schema,"}, {"start": 2083.0, "text": "each timers are first class citizens"}, {"start": 2085.0, "text": "in the semantic index."}, {"start": 2087.0, "text": "They’re found and shown together with timers"}, {"start": 2089.0, "text": "from the Clock app, or other apps on the system."}, {"start": 2095.0, "text": "Now that we’ve created some entities"}, {"start": 2097.0, "text": "and we can find them, it's"}, {"start": 2098.0, "text": "time to take action."}, {"start": 2103.0, "text": "Looks like I made"}, {"start": 2104.0, "text": "a different brand of spaghetti this time."}, {"start": 2107.0, "text": "The verb “change” here also has"}, {"start": 2109.0, "text": "a direct representation in the App Intents APIs."}, {"start": 2112.0, "text": "It is an app intent."}, {"start": 2114.0, "text": "Intents are the functions that your app performs."}, {"start": 2117.0, "text": "They may open your app when they run or work completely"}, {"start": 2120.0, "text": "in the background."}, {"start": 2122.0, "text": "Here's one that updates a timer."}, {"start": 2125.0, "text": "Intent to take specific inputs"}, {"start": 2127.0, "text": "like in this case, the timer to change,"}, {"start": 2129.0, "text": "the new duration, and the label."}, {"start": 2131.0, "text": "They produce a specific output,"}, {"start": 2133.0, "text": "like the newly updated timer entity."}, {"start": 2138.0, "text": "In code, an intent is just another ‘struct’."}, {"start": 2142.0, "text": "The inputs are declared"}, {"start": 2143.0, "text": "with ‘@Parameter’"}, {"start": 2145.0, "text": "and you can see the output type shown"}, {"start": 2147.0, "text": "in the return value of the perform method that's doing"}, {"start": 2150.0, "text": "the update."}, {"start": 2154.0, "text": "Like it does for entities,"}, {"start": 2155.0, "text": "the App Intents framework also defines schemas"}, {"start": 2158.0, "text": "for every intent that Siri can understand,"}, {"start": 2160.0, "text": "including what the expected inputs"}, {"start": 2162.0, "text": "and outputs are."}, {"start": 2163.0, "text": "The update timer"}, {"start": 2164.0, "text": "intent schema specifies the parameters like the duration,"}, {"start": 2168.0, "text": "the label, and the timer to change"}, {"start": 2170.0, "text": "as well as the result:"}, {"start": 2172.0, "text": "In this case, an entity"}, {"start": 2173.0, "text": "conforming to that timer schema."}, {"start": 2176.0, "text": "HotTickety adopts this schema, and again,"}, {"start": 2179.0, "text": "so does Apple’s Clock app."}, {"start": 2181.0, "text": "Doing this makes their intents available"}, {"start": 2183.0, "text": "to Siri."}, {"start": 2187.0, "text": "The adoption is again"}, {"start": 2188.0, "text": "one line of code at the top of our intent struct."}, {"start": 2191.0, "text": "And I can skip writing the ‘@Parameter’,"}, {"start": 2194.0, "text": "because the schema is handling it for me."}, {"start": 2199.0, "text": "A couple more super cool"}, {"start": 2200.0, "text": "things about schemas."}, {"start": 2201.0, "text": "First, they automatically"}, {"start": 2203.0, "text": "configure your Spotlight keys for you"}, {"start": 2205.0, "text": "so the semantic index can search"}, {"start": 2207.0, "text": "over granular information about your entities"}, {"start": 2210.0, "text": "with minimal adoption from you."}, {"start": 2214.0, "text": "Second, they allow Siri to automatically"}, {"start": 2216.0, "text": "handle lots of privacy and safety features for you,"}, {"start": 2219.0, "text": "like confirming actions"}, {"start": 2220.0, "text": "with people when it's appropriate, to do so, and intelligently"}, {"start": 2223.0, "text": "handling authentication and redaction"}, {"start": 2225.0, "text": "for locked devices."}, {"start": 2228.0, "text": "If you adopt schemas,"}, {"start": 2229.0, "text": "your app will also automatically"}, {"start": 2231.0, "text": "benefit from future Siri updates,"}, {"start": 2234.0, "text": "as Siri's understanding"}, {"start": 2235.0, "text": "improves to handle new languages or regional dialects."}, {"start": 2241.0, "text": "Okay, now let's talk Entity Annotations."}, {"start": 2245.0, "text": "Everything you say to Siri"}, {"start": 2247.0, "text": "is interpreted in context."}, {"start": 2249.0, "text": "This context is another area"}, {"start": 2251.0, "text": "where the App Intents platform, and your adoption of it,"}, {"start": 2254.0, "text": "can really help empower Siri and the people"}, {"start": 2256.0, "text": "that use it."}, {"start": 2258.0, "text": "The App Intents framework has a rich set of entity"}, {"start": 2261.0, "text": "annotation APIs that you can use to augment"}, {"start": 2263.0, "text": "Siri's understanding of what's on screen."}, {"start": 2266.0, "text": "Let me show you an example."}, {"start": 2270.0, "text": "What if you say something like this"}, {"start": 2271.0, "text": "while looking at HotTickety"}, {"start": 2273.0, "text": "in the foreground."}, {"start": 2275.0, "text": "Handling this one requires"}, {"start": 2276.0, "text": "Siri to understand what's on screen on your device"}, {"start": 2279.0, "text": "right now, and be able to take action on it."}, {"start": 2285.0, "text": "App Intents includes simple APIs to let you tag"}, {"start": 2288.0, "text": "your app's, views, or user activities"}, {"start": 2289.0, "text": "with the specific entities that correspond to them."}, {"start": 2292.0, "text": "So Siri knows exactly what's going on,"}, {"start": 2294.0, "text": "right at the point where people make a request."}, {"start": 2299.0, "text": "It's really easy to add annotations to your views."}, {"start": 2302.0, "text": "In HotTickety,"}, {"start": 2303.0, "text": "I added a single SwiftUI modifier to specify the ID"}, {"start": 2306.0, "text": "of the associated entity for each of my alarm rows."}, {"start": 2310.0, "text": "And there are other flavors of this API"}, {"start": 2312.0, "text": "to handle collections of views"}, {"start": 2314.0, "text": "with a selection or custom"}, {"start": 2315.0, "text": "drawn canvases, and even support for UIKit"}, {"start": 2318.0, "text": "and AppKit."}, {"start": 2322.0, "text": "But Siri's understanding"}, {"start": 2323.0, "text": "goes a bit beyond the views in your app's UI."}, {"start": 2326.0, "text": "There’s a whole set of APIs that allow you to add entities"}, {"start": 2329.0, "text": "to your adoption of core system functionality,"}, {"start": 2332.0, "text": "like notifications, now playing sessions,"}, {"start": 2335.0, "text": "and AlarmKit alarms."}, {"start": 2337.0, "text": "Adopting these can let Siri read your app's"}, {"start": 2339.0, "text": "messages to folks using AirPods,"}, {"start": 2341.0, "text": "or take action on an audio track"}, {"start": 2343.0, "text": "someone's listening to."}, {"start": 2347.0, "text": "In HotTickety,"}, {"start": 2348.0, "text": "I added an entity annotation to my AlarmKit alarms,"}, {"start": 2351.0, "text": "in just a single line of code."}, {"start": 2352.0, "text": "And that means Siri can cancel"}, {"start": 2354.0, "text": "a firing timer, or snooze an alarm,"}, {"start": 2357.0, "text": "even if my app isn’t visible onscreen."}, {"start": 2361.0, "text": "Lastly, let's cover how Siri is more intelligent"}, {"start": 2365.0, "text": "and more delightful because it can learn"}, {"start": 2367.0, "text": "from previous interactions,"}, {"start": 2368.0, "text": "and how you can contribute through Interaction Donations."}, {"start": 2374.0, "text": "You might have noticed"}, {"start": 2375.0, "text": "that in my demo, I sometimes wrote out"}, {"start": 2377.0, "text": "the name of the app: HotTickety"}, {"start": 2379.0, "text": "and sometimes I didn't."}, {"start": 2381.0, "text": "People often use language"}, {"start": 2383.0, "text": "that doesn't specify the app like that."}, {"start": 2384.0, "text": "In fact, people often use language"}, {"start": 2386.0, "text": "that doesn't specify a lot of things."}, {"start": 2389.0, "text": "If you ask to message your work colleague,"}, {"start": 2391.0, "text": "you might not even specify that you want to do that on Slack."}, {"start": 2394.0, "text": "It's just implied. You expect Siri"}, {"start": 2396.0, "text": "to figure it out."}, {"start": 2399.0, "text": "Siri can automatically learn"}, {"start": 2401.0, "text": "from previous interactions a person has had with it."}, {"start": 2404.0, "text": "Each time you say “Start a Timer in HotTickety”"}, {"start": 2407.0, "text": "Siri saves a receipt of that action"}, {"start": 2409.0, "text": "into a temporary transcript."}, {"start": 2411.0, "text": "Eventually, if you use HotTickety with Siri"}, {"start": 2413.0, "text": "enough like I did during my development of it,"}, {"start": 2416.0, "text": "it'll start to default to that app automatically"}, {"start": 2419.0, "text": "because it starts to know that's the one you prefer."}, {"start": 2423.0, "text": "But Siri and Apple Intelligence do need your help"}, {"start": 2426.0, "text": "to see the interactions"}, {"start": 2427.0, "text": "that happen within your apps,"}, {"start": 2429.0, "text": "so they can learn from those too."}, {"start": 2432.0, "text": "You can teach Siri how people"}, {"start": 2433.0, "text": "actually use your apps to make"}, {"start": 2435.0, "text": "Siri even more personal for them."}, {"start": 2437.0, "text": "To do so, App Intents"}, {"start": 2438.0, "text": "has a set of “Interaction Donation” APIs"}, {"start": 2441.0, "text": "that let you add to this transcript"}, {"start": 2442.0, "text": "based on what actually happened"}, {"start": 2444.0, "text": "in your app's UI."}, {"start": 2446.0, "text": "Siri can use these donations"}, {"start": 2447.0, "text": "to do a better job routing"}, {"start": 2449.0, "text": "user requests to just the right app"}, {"start": 2451.0, "text": "based on the parameters,"}, {"start": 2452.0, "text": "like always contacting that work colleague"}, {"start": 2455.0, "text": "over Slack."}, {"start": 2459.0, "text": "Interaction donations"}, {"start": 2460.0, "text": "also help Siri improve its knowledge"}, {"start": 2462.0, "text": "of what entities are relevant on the device."}, {"start": 2465.0, "text": "If I've started a stopwatch"}, {"start": 2467.0, "text": "in HotTickety with my voice,"}, {"start": 2469.0, "text": "Siri can understand"}, {"start": 2470.0, "text": "there's a stopwatch now running in the app."}, {"start": 2472.0, "text": "And when I go to HotTickety and tap,"}, {"start": 2474.0, "text": "to reset that stopwatch..."}, {"start": 2476.0, "text": "HotTickety donates that interaction and reading it,"}, {"start": 2479.0, "text": "Siri can know the stopwatch has stopped running."}, {"start": 2485.0, "text": "Donating an interaction"}, {"start": 2486.0, "text": "is as simple as creating the app"}, {"start": 2488.0, "text": "intent equivalent of what the user did in-app."}, {"start": 2491.0, "text": "In this case resetting the stopwatch"}, {"start": 2493.0, "text": "and then calling ‘IntentDonationManager’’s"}, {"start": 2495.0, "text": "donate method with the intent."}, {"start": 2499.0, "text": "OK, let's pull all of this together."}, {"start": 2502.0, "text": "The way to make your apps"}, {"start": 2504.0, "text": "work with Siri is through the App Intents framework"}, {"start": 2506.0, "text": "which powers a lot"}, {"start": 2508.0, "text": "more than just intents!"}, {"start": 2509.0, "text": "With all these APIs we covered today,"}, {"start": 2511.0, "text": "you can model your content with entities,"}, {"start": 2513.0, "text": "and tap into a whole set of powerful capabilities"}, {"start": 2516.0, "text": "like entity annotations,"}, {"start": 2517.0, "text": "interactions donations, and search."}, {"start": 2521.0, "text": "Getting started with App Intents"}, {"start": 2523.0, "text": "is actually pretty easy!"}, {"start": 2524.0, "text": "Schemas can let you do a lot less work."}, {"start": 2527.0, "text": "And you can quickly start"}, {"start": 2528.0, "text": "getting functioning features,"}, {"start": 2530.0, "text": "even before you've adopted everything."}, {"start": 2534.0, "text": "Now, I only had a little bit"}, {"start": 2536.0, "text": "of time today, and there's so much more"}, {"start": 2538.0, "text": "to this platform that I could not get a chance to cover."}, {"start": 2540.0, "text": "For example, there's a Transferable API"}, {"start": 2542.0, "text": "that lets Siri translate"}, {"start": 2544.0, "text": "entities into the different formats"}, {"start": 2546.0, "text": "needed for cross-app actions."}, {"start": 2548.0, "text": "There are all kinds of ways to customize dialog"}, {"start": 2551.0, "text": "and visuals to show off your app's unique style,"}, {"start": 2554.0, "text": "and the ability"}, {"start": 2555.0, "text": "to mark your entities as syncable"}, {"start": 2557.0, "text": "with the Cloud so conversations just work across"}, {"start": 2559.0, "text": "all of a person’s devices."}, {"start": 2562.0, "text": "There's even a super cool new framework"}, {"start": 2564.0, "text": "called ‘AppIntentsTesting’"}, {"start": 2565.0, "text": "to help you get all of your adoption working,"}, {"start": 2568.0, "text": "and keep it working."}, {"start": 2571.0, "text": "We have a ton of talks,"}, {"start": 2572.0, "text": "documentation, and sample code this year"}, {"start": 2574.0, "text": "that you can check out to learn about all of this."}, {"start": 2579.0, "text": "Every year you folks blow our minds"}, {"start": 2582.0, "text": "by building things with App Intents"}, {"start": 2583.0, "text": "that we couldn't even imagine."}, {"start": 2586.0, "text": "That is what gets us up and out of bed"}, {"start": 2589.0, "text": "every morning."}, {"start": 2591.0, "text": "This year is a big one,"}, {"start": 2592.0, "text": "and we are setting our alarms for release day"}, {"start": 2595.0, "text": "to check out all the great apps"}, {"start": 2596.0, "text": "you're going to build."}, {"start": 2598.0, "text": "And now I'll hand you over to Louis."}, {"start": 2608.0, "text": "Thank you Michael."}, {"start": 2609.0, "text": "Hi everyone I’m Louis."}, {"start": 2611.0, "text": "Now Michael showed you how"}, {"start": 2612.0, "text": "you can bring your app’s, content and actions to Siri,"}, {"start": 2615.0, "text": "and make your app work seamlessly"}, {"start": 2616.0, "text": "with Apple Intelligence."}, {"start": 2618.0, "text": "Now I’ll show you how to integrate"}, {"start": 2620.0, "text": "AI into your apps."}, {"start": 2621.0, "text": "So let’s talk about the Foundation Models framework."}, {"start": 2624.0, "text": "It gives you direct access"}, {"start": 2625.0, "text": "to a large language model"}, {"start": 2627.0, "text": "built right into the system"}, {"start": 2629.0, "text": "all through a powerful native Swift API."}, {"start": 2632.0, "text": "At its heart,"}, {"start": 2633.0, "text": "the framework is very straightforward."}, {"start": 2635.0, "text": "Your app sends a prompt"}, {"start": 2637.0, "text": "to the large language model."}, {"start": 2639.0, "text": "The LLM then reasons about your prompt"}, {"start": 2641.0, "text": "and generates a response."}, {"start": 2643.0, "text": "For example,"}, {"start": 2644.0, "text": "you can ask it to create a three day"}, {"start": 2646.0, "text": "itinerary for visiting Paris,"}, {"start": 2648.0, "text": "and the model will respond"}, {"start": 2650.0, "text": "with a natural language plan,"}, {"start": 2653.0, "text": "very detailed."}, {"start": 2655.0, "text": "But Foundation Models goes"}, {"start": 2656.0, "text": "far beyond basic text generation."}, {"start": 2658.0, "text": "It also lets you control"}, {"start": 2660.0, "text": "exactly how the model behaves."}, {"start": 2662.0, "text": "For instance, using Guided Generation,"}, {"start": 2664.0, "text": "you can guarantee"}, {"start": 2665.0, "text": "perfectly structured output,"}, {"start": 2667.0, "text": "simply by providing a Swift struct."}, {"start": 2671.0, "text": "And You can also use Tool Calling"}, {"start": 2672.0, "text": "to extend the model’s capabilities,"}, {"start": 2675.0, "text": "allowing it to retrieve"}, {"start": 2676.0, "text": "real-time or dynamic data from your apps."}, {"start": 2680.0, "text": "And you can guide the model’s tone"}, {"start": 2681.0, "text": "from custom Instructions and it can provide real-time"}, {"start": 2683.0, "text": "UI updates by Streaming the model’s output."}, {"start": 2687.0, "text": "There is so much more."}, {"start": 2689.0, "text": "When you put these pieces"}, {"start": 2690.0, "text": "together, the possibilities become endless."}, {"start": 2693.0, "text": "You can build features that analyze"}, {"start": 2694.0, "text": "complex data,"}, {"start": 2696.0, "text": "synthesize information"}, {"start": 2697.0, "text": "into concise summaries,"}, {"start": 2699.0, "text": "or create entirely new"}, {"start": 2700.0, "text": "conversational interfaces."}, {"start": 2704.0, "text": "But to build the next generation"}, {"start": 2705.0, "text": "of truly agentic experiences,"}, {"start": 2708.0, "text": "you need more scale,"}, {"start": 2710.0, "text": "more context, and more control."}, {"start": 2712.0, "text": "So today,"}, {"start": 2713.0, "text": "we are bringing five big updates"}, {"start": 2715.0, "text": "to the Foundation Models framework."}, {"start": 2718.0, "text": "We'll cover Image Input,"}, {"start": 2720.0, "text": "scaling up with Private Cloud Compute,"}, {"start": 2722.0, "text": "integrating third party models,"}, {"start": 2725.0, "text": "new APIs"}, {"start": 2726.0, "text": "for letting you build agentic experiences in your apps,"}, {"start": 2729.0, "text": "and testing it all"}, {"start": 2730.0, "text": "with our new Evaluations framework."}, {"start": 2734.0, "text": "So let's start with my favorite new feature,"}, {"start": 2737.0, "text": "Image Input."}, {"start": 2739.0, "text": "You can now"}, {"start": 2740.0, "text": "use both text and images in your prompts."}, {"start": 2743.0, "text": "And this opens up so many possibilities."}, {"start": 2747.0, "text": "Your app can now be multimodal."}, {"start": 2751.0, "text": "If you combine that with the Generable macro,"}, {"start": 2753.0, "text": "you can easily ask the LLM to get structured"}, {"start": 2755.0, "text": "data from an image."}, {"start": 2758.0, "text": "For tasks like this, it used to be"}, {"start": 2760.0, "text": "that you had to train a custom ML model..."}, {"start": 2763.0, "text": "but now this is all possible"}, {"start": 2764.0, "text": "with the built in model."}, {"start": 2766.0, "text": "And yes, I can tell you,"}, {"start": 2767.0, "text": "he really is a good boy."}, {"start": 2771.0, "text": "So that’s Image Input."}, {"start": 2772.0, "text": "Now let’s talk about Private Cloud Compute."}, {"start": 2775.0, "text": "The on-device model"}, {"start": 2776.0, "text": "we were just looking at is"}, {"start": 2778.0, "text": "incredibly useful for so many tasks,"}, {"start": 2780.0, "text": "especially with the image input"}, {"start": 2782.0, "text": "that we're adding this year."}, {"start": 2784.0, "text": "But we know that sometimes"}, {"start": 2786.0, "text": "you might want a larger model"}, {"start": 2787.0, "text": "for deeper reasoning and more complex tasks."}, {"start": 2791.0, "text": "So this year we’re also giving you access"}, {"start": 2793.0, "text": "to a new server model,"}, {"start": 2794.0, "text": "which runs on Private Cloud Compute."}, {"start": 2797.0, "text": "Now, developers in the App Store Small Business Program"}, {"start": 2800.0, "text": "with less than 2 million downloads"}, {"start": 2802.0, "text": "can get access to this new server model."}, {"start": 2805.0, "text": "And just like with the on-device model, access"}, {"start": 2807.0, "text": "to Private Cloud Compute is integrated"}, {"start": 2809.0, "text": "in the OS."}, {"start": 2811.0, "text": "So there's no account setup,"}, {"start": 2814.0, "text": "no authentication, and no API keys!"}, {"start": 2819.0, "text": "And even better,"}, {"start": 2821.0, "text": "there are no token cost to you, the developer."}, {"start": 2824.0, "text": "There is a daily usage limit per user."}, {"start": 2827.0, "text": "So this lets you scale"}, {"start": 2828.0, "text": "your number of users,"}, {"start": 2830.0, "text": "without worrying about the total cost"}, {"start": 2833.0, "text": "to you, the developer."}, {"start": 2835.0, "text": "And users can upgrade to iCloud+"}, {"start": 2837.0, "text": "to get even higher limits if they want."}, {"start": 2840.0, "text": "So let's look at the code."}, {"start": 2841.0, "text": "With the on-device LLM,"}, {"start": 2843.0, "text": "it takes just a few lines of code to get started."}, {"start": 2845.0, "text": "Create a session and give it a prompt."}, {"start": 2848.0, "text": "And now with just one line of code,"}, {"start": 2850.0, "text": "you can switch to the new server model"}, {"start": 2852.0, "text": "running on Private Cloud Compute."}, {"start": 2855.0, "text": "It’s that easy!"}, {"start": 2859.0, "text": "One line of code."}, {"start": 2863.0, "text": "Now of course"}, {"start": 2864.0, "text": "Private Cloud Compute is private."}, {"start": 2867.0, "text": "It’s architected in a way"}, {"start": 2868.0, "text": "that guarantees that user data"}, {"start": 2870.0, "text": "is never stored."}, {"start": 2871.0, "text": "Your data is only used for your requests."}, {"start": 2873.0, "text": "And it comes with a verifiable privacy promise."}, {"start": 2878.0, "text": "So that’s Private Cloud Compute."}, {"start": 2881.0, "text": "Now let's talk about 3rd-party models."}, {"start": 2883.0, "text": "There are a lot of LLMs out there."}, {"start": 2885.0, "text": "Maybe even some new ones"}, {"start": 2886.0, "text": "by the time we’re done here."}, {"start": 2889.0, "text": "And now any LLM provider"}, {"start": 2891.0, "text": "can create a Swift package"}, {"start": 2893.0, "text": "that conforms to the new LanguageModel protocol."}, {"start": 2895.0, "text": "And that lets you run virtually any LLM"}, {"start": 2898.0, "text": "with the Foundation Models framework."}, {"start": 2901.0, "text": "And this year,"}, {"start": 2903.0, "text": "Apple's on-device AI inference is getting"}, {"start": 2905.0, "text": "a big upgrade with the new Core AI framework"}, {"start": 2907.0, "text": "and upgraded MLX."}, {"start": 2910.0, "text": "And both of these now"}, {"start": 2911.0, "text": "have an open source package,"}, {"start": 2912.0, "text": "letting you run a variety"}, {"start": 2914.0, "text": "of on device models,"}, {"start": 2915.0, "text": "all with our unified Swift API"}, {"start": 2918.0, "text": "in the Foundation Models framework."}, {"start": 2920.0, "text": "And here is Matt to show you"}, {"start": 2921.0, "text": "this in action."}, {"start": 2928.0, "text": "Thanks, Louis."}, {"start": 2929.0, "text": "Hi everyone."}, {"start": 2930.0, "text": "I'd like to start"}, {"start": 2931.0, "text": "by looking at some of the APIs"}, {"start": 2933.0, "text": "we’ve all come to love and write since last year."}, {"start": 2935.0, "text": "So I have a Playground open here where"}, {"start": 2937.0, "text": "I've created a instance of the SystemLanguageModel."}, {"start": 2940.0, "text": "I then pass that model"}, {"start": 2942.0, "text": "to a LanguageModelSession,"}, {"start": 2943.0, "text": "and finally,"}, {"start": 2944.0, "text": "I prompt the model via the session."}, {"start": 2947.0, "text": "In this case, my prompt is"}, {"start": 2948.0, "text": "tell me about the history of origami."}, {"start": 2950.0, "text": "And on the right"}, {"start": 2951.0, "text": "I can see the model’s response."}, {"start": 2953.0, "text": "So tell me about the origin of origami"}, {"start": 2955.0, "text": "and its cultural significance."}, {"start": 2958.0, "text": "Now we LOVE the SystemLanguageModel"}, {"start": 2960.0, "text": "and seeing the capabilities it’s added to your apps."}, {"start": 2963.0, "text": "But wouldn’t it be great"}, {"start": 2964.0, "text": "if I could use other general-purpose"}, {"start": 2966.0, "text": "models like Qwen?"}, {"start": 2968.0, "text": "So I actually"}, {"start": 2969.0, "text": "have a version of Qwen here in my project."}, {"start": 2972.0, "text": "and I’ve also added the Core AI models package"}, {"start": 2975.0, "text": "that has an implementation"}, {"start": 2976.0, "text": "that conforms to the LanguageModel protocol."}, {"start": 2979.0, "text": "This allows for the Foundation Models framework"}, {"start": 2981.0, "text": "to reach for models that are ready for Core AI."}, {"start": 2984.0, "text": "We’ll talk more about"}, {"start": 2985.0, "text": "Core AI later. But for now, let's see what it takes"}, {"start": 2988.0, "text": "to bring this model into my project."}, {"start": 2990.0, "text": "All I have to do"}, {"start": 2992.0, "text": "is add the import for the package."}, {"start": 2995.0, "text": "Change out the SystemLanguageModel"}, {"start": 2997.0, "text": "for the Core AI language model,"}, {"start": 2999.0, "text": "And have it target my Qwen model."}, {"start": 3002.0, "text": "And that’s it."}, {"start": 3003.0, "text": "I don't have to change"}, {"start": 3004.0, "text": "any other code and I don't have to change my prompt"}, {"start": 3007.0, "text": "That's the power of the LanguageModel protocol."}, {"start": 3010.0, "text": "It lets you plug in"}, {"start": 3011.0, "text": "any model you'd like directly"}, {"start": 3012.0, "text": "into the Foundation Models framework."}, {"start": 3014.0, "text": "And on the right"}, {"start": 3015.0, "text": "I can see the new model’s response."}, {"start": 3018.0, "text": "It's that easy."}, {"start": 3019.0, "text": "We are so excited"}, {"start": 3021.0, "text": "about the LanguageModel protocol"}, {"start": 3022.0, "text": "and what it allows you to do."}, {"start": 3024.0, "text": "We can’t wait to see the models"}, {"start": 3026.0, "text": "you'll bring to the FoundationModels framework"}, {"start": 3028.0, "text": "this year. Thanks."}, {"start": 3030.0, "text": "Back to you, Louis."}, {"start": 3036.0, "text": "Awesome, thank you, Matt!"}, {"start": 3038.0, "text": "And beyond on-device models,"}, {"start": 3040.0, "text": "as we announced yesterday at the Platforms State of the Union,"}, {"start": 3043.0, "text": "Google and Anthropic are also extending"}, {"start": 3046.0, "text": "the Foundation Models framework with Swift packages"}, {"start": 3048.0, "text": "of their own,"}, {"start": 3050.0, "text": "making state-of-the-art"}, {"start": 3051.0, "text": "Gemini and Claude models"}, {"start": 3053.0, "text": "available to all Swift developers."}, {"start": 3057.0, "text": "Now, with all these models"}, {"start": 3058.0, "text": "just a few lines of code away,"}, {"start": 3059.0, "text": "the AI experiences you can build"}, {"start": 3062.0, "text": "are more capable than ever."}, {"start": 3064.0, "text": "And to enable you to build agentic experiences,"}, {"start": 3066.0, "text": "we're also giving you new APIs!"}, {"start": 3068.0, "text": "So imagine you're"}, {"start": 3070.0, "text": "building a travel app."}, {"start": 3072.0, "text": "You could use the on-device model"}, {"start": 3074.0, "text": "as an offline planner, especially useful"}, {"start": 3077.0, "text": "when you don’t have internet. Like in an airport."}, {"start": 3080.0, "text": "And in the same app,"}, {"start": 3081.0, "text": "you might want to use a server model,"}, {"start": 3083.0, "text": "like the one on PCC,"}, {"start": 3084.0, "text": "for more complex tasks like rebooking hotels."}, {"start": 3089.0, "text": "With agentic features using multiple models,"}, {"start": 3091.0, "text": "you typically want to share"}, {"start": 3092.0, "text": "that context in some way. Because here's Session B"}, {"start": 3095.0, "text": "with that server model can’t see what time my flight is."}, {"start": 3099.0, "text": "Now sharing that context is typically complicated."}, {"start": 3102.0, "text": "And hard to manage-"}, {"start": 3103.0, "text": "like my two kids at home."}, {"start": 3106.0, "text": "and I wish I was kidding."}, {"start": 3110.0, "text": "So this year we’re introducing new APIs,"}, {"start": 3112.0, "text": "letting you manage multiple sessions"}, {"start": 3114.0, "text": "with shared context."}, {"start": 3117.0, "text": "Now, If you had to write"}, {"start": 3118.0, "text": "this would imperative code,"}, {"start": 3120.0, "text": "you have to write all this boilerplate!"}, {"start": 3122.0, "text": "And nobody wants that."}, {"start": 3124.0, "text": "But with our new DynamicProfile API,"}, {"start": 3126.0, "text": "you get a high level Swift builder syntax"}, {"start": 3129.0, "text": "to easily use models together."}, {"start": 3131.0, "text": "And this reduces the overhead"}, {"start": 3133.0, "text": "of managing state,"}, {"start": 3134.0, "text": "and lets you focus"}, {"start": 3136.0, "text": "on the specific details of your features."}, {"start": 3139.0, "text": "And we know this space moves fast."}, {"start": 3142.0, "text": "So to give you even more APIs, we’re also releasing"}, {"start": 3145.0, "text": "an open source utilities package!"}, {"start": 3149.0, "text": "This contains custom skills."}, {"start": 3151.0, "text": "To let a model dynamically"}, {"start": 3153.0, "text": "discover tools in your app."}, {"start": 3155.0, "text": "And we have other utilities"}, {"start": 3156.0, "text": "like context management."}, {"start": 3158.0, "text": "And you can clone this package"}, {"start": 3159.0, "text": "from Github today."}, {"start": 3162.0, "text": "And to help you profile and debug,"}, {"start": 3165.0, "text": "we’ve updated Instruments this year."}, {"start": 3167.0, "text": "You can now see everything the models see"}, {"start": 3169.0, "text": "at runtime."}, {"start": 3171.0, "text": "Each prompt, tool call model output"}, {"start": 3174.0, "text": "can be inspected in detail."}, {"start": 3177.0, "text": "But what if a model isn’t"}, {"start": 3178.0, "text": "always doing what you want it to do?"}, {"start": 3181.0, "text": "We've all been there, right?"}, {"start": 3183.0, "text": "Well, that brings us"}, {"start": 3184.0, "text": "to evaluating your apps."}, {"start": 3185.0, "text": "We’re also making that easy for you this year."}, {"start": 3188.0, "text": "We've created a new Swift framework,"}, {"start": 3191.0, "text": "that's integrated right in Xcode."}, {"start": 3193.0, "text": "And here's Rob to tell you all about that."}, {"start": 3201.0, "text": "Thanks, Louis."}, {"start": 3202.0, "text": "Hello, everyone."}, {"start": 3205.0, "text": "If you’ve spent any time"}, {"start": 3206.0, "text": "building features with Language Models,"}, {"start": 3208.0, "text": "you already know the biggest challenge:"}, {"start": 3210.0, "text": "their output can be unpredictable."}, {"start": 3213.0, "text": "Traditional code"}, {"start": 3214.0, "text": "does exactly what you tell it to do."}, {"start": 3216.0, "text": "Language models"}, {"start": 3219.0, "text": "not so much."}, {"start": 3220.0, "text": "Let me show you what I mean."}, {"start": 3223.0, "text": "I’m building a book tracker app"}, {"start": 3225.0, "text": "that uses a model."}, {"start": 3227.0, "text": "And I wrote a simple prompt"}, {"start": 3228.0, "text": "asking the model to act as a librarian"}, {"start": 3231.0, "text": "and generate descriptive"}, {"start": 3232.0, "text": "tags based on the users book review."}, {"start": 3236.0, "text": "When I test it myself,"}, {"start": 3238.0, "text": "it looks great."}, {"start": 3239.0, "text": "It added tags to this book"}, {"start": 3242.0, "text": "and it handled a few other books"}, {"start": 3243.0, "text": "with no issues."}, {"start": 3246.0, "text": "But then I shared the app"}, {"start": 3247.0, "text": "with a friend."}, {"start": 3249.0, "text": "They wrote a review for Dracula,"}, {"start": 3251.0, "text": "and the model tagged it as"}, {"start": 3253.0, "text": "I mean, cuddly?"}, {"start": 3255.0, "text": "Family friendly."}, {"start": 3258.0, "text": "I don't know what that is about kittens."}, {"start": 3262.0, "text": "Clearly these are problems."}, {"start": 3264.0, "text": "And these are the kind of problems"}, {"start": 3266.0, "text": "that Evaluations can solve for you."}, {"start": 3269.0, "text": "The Evaluations framework provides a flexible system"}, {"start": 3272.0, "text": "to measure the quality of your model output."}, {"start": 3274.0, "text": "You can define metrics,"}, {"start": 3276.0, "text": "automatically grade outputs"}, {"start": 3277.0, "text": "using language models,"}, {"start": 3279.0, "text": "and analyze statistical trends across"}, {"start": 3281.0, "text": "large data sets."}, {"start": 3283.0, "text": "And you can tailor it to your app"}, {"start": 3286.0, "text": "so you ensure your intelligent features"}, {"start": 3288.0, "text": "perform reliably."}, {"start": 3292.0, "text": "The framework lets you scale your evaluation over time,"}, {"start": 3295.0, "text": "and across a variety of samples."}, {"start": 3298.0, "text": "If you need help creating more samples"}, {"start": 3300.0, "text": "the Evaluations framework"}, {"start": 3302.0, "text": "has your back too."}, {"start": 3303.0, "text": "It can synthesize samples for you"}, {"start": 3305.0, "text": "using language models."}, {"start": 3308.0, "text": "Evaluations are easy to run,"}, {"start": 3311.0, "text": "integrating directly"}, {"start": 3312.0, "text": "into your test targets."}, {"start": 3314.0, "text": "We have a new trait for Swift Testing,"}, {"start": 3316.0, "text": "called ‘evaluates’"}, {"start": 3317.0, "text": "that will automatically set up your evaluation harness,"}, {"start": 3321.0, "text": "run it,"}, {"start": 3322.0, "text": "and provide the results inside of your test."}, {"start": 3325.0, "text": "And our metric"}, {"start": 3326.0, "text": "aggregation makes it easy to assert against trends"}, {"start": 3329.0, "text": "in your evaluation."}, {"start": 3332.0, "text": "We've added a new Test report"}, {"start": 3334.0, "text": "in Xcode 27 for evaluation results."}, {"start": 3337.0, "text": "It's a great way to dive into the details"}, {"start": 3339.0, "text": "and analyze your results further."}, {"start": 3342.0, "text": "Let's look at this report"}, {"start": 3343.0, "text": "in more detail."}, {"start": 3345.0, "text": "This report lets you analyze"}, {"start": 3347.0, "text": "every metric in your evaluation."}, {"start": 3350.0, "text": "Each metric"}, {"start": 3351.0, "text": "is measured by an evaluator"}, {"start": 3353.0, "text": "for each data sample."}, {"start": 3354.0, "text": "Let's consider the “tag count” metric."}, {"start": 3358.0, "text": "And this is the code for the evaluator."}, {"start": 3360.0, "text": "It's a simple swift comparison of the tags array"}, {"start": 3363.0, "text": "returned from the model for a given sample."}, {"start": 3366.0, "text": "Evaluators can be any function of the model"}, {"start": 3369.0, "text": "response and sample passed to the model."}, {"start": 3372.0, "text": "Here I return a passing measurement"}, {"start": 3374.0, "text": "if the tag count is within my desired range."}, {"start": 3378.0, "text": "But Metrics don't have to be pass fail."}, {"start": 3381.0, "text": "They could also return a Double value,"}, {"start": 3383.0, "text": "using the scoring measurement."}, {"start": 3385.0, "text": "Here I return the raw number of tags"}, {"start": 3387.0, "text": "generated for each sample."}, {"start": 3391.0, "text": "I can combine both metrics to determine"}, {"start": 3393.0, "text": "if the model created the right number of tags,"}, {"start": 3395.0, "text": "and observe the distribution of tags."}, {"start": 3398.0, "text": "This aggregation"}, {"start": 3399.0, "text": "looks across all data samples in the evaluation."}, {"start": 3402.0, "text": "You can use aggregate metrics"}, {"start": 3404.0, "text": "in your tests to assert"}, {"start": 3405.0, "text": "against statistical patterns in your evaluation results."}, {"start": 3408.0, "text": "These aggregates are particularly useful"}, {"start": 3410.0, "text": "when evaluating a large data set."}, {"start": 3414.0, "text": "But not all metrics"}, {"start": 3415.0, "text": "are calculated in code."}, {"start": 3417.0, "text": "Oftentimes, you want an assessment,"}, {"start": 3420.0, "text": "using a qualitative scale."}, {"start": 3422.0, "text": "Instead of a hard-coded measurement,"}, {"start": 3424.0, "text": "I use a ScoreDimension here"}, {"start": 3425.0, "text": "to define one or more scoring scales for a model judge."}, {"start": 3430.0, "text": "Model judges are powerful evaluators,"}, {"start": 3433.0, "text": "which use language models"}, {"start": 3435.0, "text": "to approximate human judgement."}, {"start": 3437.0, "text": "This model judge determines"}, {"start": 3438.0, "text": "if the generated tags are relevant to the book,"}, {"start": 3441.0, "text": "describing the book instead of repeating"}, {"start": 3443.0, "text": "personal reactions from my review."}, {"start": 3447.0, "text": "My model judge uses to ScoreDimensions,"}, {"start": 3449.0, "text": "relevance and usefulness,"}, {"start": 3451.0, "text": "to describe the quality of the generated tags."}, {"start": 3454.0, "text": "When aggregated,"}, {"start": 3455.0, "text": "this gives me a top-level"}, {"start": 3457.0, "text": "understanding of my feature's quality."}, {"start": 3460.0, "text": "This helps me understand how"}, {"start": 3461.0, "text": "well the model performs at tasks"}, {"start": 3463.0, "text": "my app needs."}, {"start": 3464.0, "text": "And with this knowledge,"}, {"start": 3465.0, "text": "I can engineer"}, {"start": 3467.0, "text": "my system instructions"}, {"start": 3468.0, "text": "and prompts to maximize performance."}, {"start": 3472.0, "text": "I've been using my evaluation"}, {"start": 3473.0, "text": "for a few days to help me"}, {"start": 3475.0, "text": "refine the tagging service"}, {"start": 3476.0, "text": "so it consistently behaves"}, {"start": 3478.0, "text": "as I expect."}, {"start": 3480.0, "text": "You can see the effect of these iterations"}, {"start": 3482.0, "text": "on the instructions passed"}, {"start": 3483.0, "text": "through the LanguageModelSession"}, {"start": 3485.0, "text": "inside of my tagging feature."}, {"start": 3488.0, "text": "If you remember,"}, {"start": 3489.0, "text": "this is what I started with in the beginning."}, {"start": 3492.0, "text": "And here's my instructions now,"}, {"start": 3494.0, "text": "after several evaluation runs and tweaks."}, {"start": 3497.0, "text": "Here you can see each"}, {"start": 3498.0, "text": "expectation of the system"}, {"start": 3500.0, "text": "I verified next to the instructions"}, {"start": 3502.0, "text": "I added to the system based on evaluating"}, {"start": 3505.0, "text": "those expectations."}, {"start": 3509.0, "text": "This process of iterative prompt engineering,"}, {"start": 3511.0, "text": "based on evaluation"}, {"start": 3513.0, "text": "results is called “hill-climbing”."}, {"start": 3515.0, "text": "And when you integrate a hill-climbing feedback loop"}, {"start": 3518.0, "text": "into your app's development process, we call it"}, {"start": 3521.0, "text": "Evaluation-driven Development."}, {"start": 3524.0, "text": "Developing with evaluations helps"}, {"start": 3526.0, "text": "you determine when you're ready to ship."}, {"start": 3528.0, "text": "It helps you maintain quality"}, {"start": 3530.0, "text": "over every future release."}, {"start": 3533.0, "text": "And that's the Evaluations framework."}, {"start": 3535.0, "text": "Now back to Louis."}, {"start": 3543.0, "text": "Amazing, right."}, {"start": 3544.0, "text": "Thank you, Rob."}, {"start": 3545.0, "text": "So the Foundation Models framework"}, {"start": 3547.0, "text": "now lets you plug in a variety of models."}, {"start": 3549.0, "text": "And as you just saw the evaluations framework lets"}, {"start": 3552.0, "text": "you measure your features, no matter which model you’re using."}, {"start": 3557.0, "text": "So putting this all together"}, {"start": 3558.0, "text": "this year we're giving you all the building blocks"}, {"start": 3560.0, "text": "to create great AI features!"}, {"start": 3563.0, "text": "But we didn’t stop there."}, {"start": 3566.0, "text": "The Foundation Models framework"}, {"start": 3567.0, "text": "is going to open source later this summer."}, {"start": 3577.0, "text": "So that means you'll be able"}, {"start": 3578.0, "text": "to use it on other platforms, like on Linux"}, {"start": 3581.0, "text": "Swift on Server."}, {"start": 3583.0, "text": "And that will make the Foundation Models framework"}, {"start": 3585.0, "text": "THE Swift API for any LLM"}, {"start": 3589.0, "text": "on any platform."}, {"start": 3592.0, "text": "Now the team got so excited,"}, {"start": 3594.0, "text": "that we did even more."}, {"start": 3596.0, "text": "So new in macOS 27,"}, {"start": 3598.0, "text": "there is the ‘fm’ command line tool,"}, {"start": 3600.0, "text": "which lets you easily"}, {"start": 3601.0, "text": "access both on-device"}, {"start": 3603.0, "text": "and the new server model on PCC."}, {"start": 3606.0, "text": "Now this is so great"}, {"start": 3607.0, "text": "for prototyping and scripting."}, {"start": 3612.0, "text": "And there’s even more that we didn’t have time for."}, {"start": 3614.0, "text": "Like the CoreSpotlight RAG tool,"}, {"start": 3617.0, "text": "We have Vision tools, dynamic instructions API,"}, {"start": 3620.0, "text": "and a lot more."}, {"start": 3621.0, "text": "But that's all the time we have right now."}, {"start": 3623.0, "text": "So now, back to Rachel."}, {"start": 3630.0, "text": "Thanks, Louis."}, {"start": 3632.0, "text": "All right. How's everyone doing?"}, {"start": 3634.0, "text": "Great. Great."}, {"start": 3635.0, "text": "Okay."}, {"start": 3636.0, "text": "Are you excited to try some of this stuff out?"}, {"start": 3638.0, "text": "Amazing. Okay."}, {"start": 3640.0, "text": "So far, you've seen"}, {"start": 3641.0, "text": "how you could build faster with Xcode,"}, {"start": 3644.0, "text": "how you can integrate your apps"}, {"start": 3645.0, "text": "content and capabilities"}, {"start": 3647.0, "text": "across the system,"}, {"start": 3648.0, "text": "including through natural language in Siri"}, {"start": 3651.0, "text": "And how you can ship intelligent features"}, {"start": 3654.0, "text": "in your app through the Foundation Models framework."}, {"start": 3657.0, "text": "Then finally, we get to test"}, {"start": 3658.0, "text": "some of these features through the Evaluations framework."}, {"start": 3661.0, "text": "But we're not done yet."}, {"start": 3663.0, "text": "Many of you might want to take your own models,"}, {"start": 3665.0, "text": "or take custom generative models"}, {"start": 3667.0, "text": "and run them on-device."}, {"start": 3669.0, "text": "For that, we have two more exciting technologies"}, {"start": 3673.0, "text": "starting with Core AI."}, {"start": 3674.0, "text": "Here's Raziel to tell you more."}, {"start": 3679.0, "text": "Thank you, Rachel."}, {"start": 3682.0, "text": "Hi everyone,"}, {"start": 3683.0, "text": "my name is Raziel."}, {"start": 3684.0, "text": "I lead Core AI."}, {"start": 3687.0, "text": "Today Geppy and I will give"}, {"start": 3689.0, "text": "you an overview of the technology."}, {"start": 3692.0, "text": "But let's start with some context"}, {"start": 3694.0, "text": "and hopefully something"}, {"start": 3696.0, "text": "that is not controversial at this point."}, {"start": 3698.0, "text": "Is that the world of AI"}, {"start": 3699.0, "text": "is vast and growing."}, {"start": 3701.0, "text": "Just look at this list."}, {"start": 3702.0, "text": "There is a large amount of models"}, {"start": 3705.0, "text": "and a large amount"}, {"start": 3707.0, "text": "of potential applications for each one of them."}, {"start": 3710.0, "text": "It's things like image captioning."}, {"start": 3712.0, "text": "Voice cancellation."}, {"start": 3714.0, "text": "Gesture control. Background removal."}, {"start": 3717.0, "text": "So as the AI landscape"}, {"start": 3719.0, "text": "becomes more complex"}, {"start": 3720.0, "text": "and your ideas grow more specialized"}, {"start": 3723.0, "text": "we want to make sure that the paths to bring them"}, {"start": 3725.0, "text": "to production remain as smooth as possible."}, {"start": 3728.0, "text": "And that's why we created Core AI."}, {"start": 3731.0, "text": "It marks the next evolution of on-device AI execution"}, {"start": 3734.0, "text": "across Apple platforms."}, {"start": 3737.0, "text": "It lets you bring cutting-edge AI"}, {"start": 3739.0, "text": "into your apps."}, {"start": 3741.0, "text": "Enabling some truly magical experiences."}, {"start": 3745.0, "text": "And of course, this comes"}, {"start": 3746.0, "text": "with the benefits of local execution:"}, {"start": 3749.0, "text": "Your data stays private on-device."}, {"start": 3752.0, "text": "Your features are always available"}, {"start": 3754.0, "text": "because there is no reliance"}, {"start": 3755.0, "text": "on connectivity,"}, {"start": 3758.0, "text": "and there is no inference cost to you,"}, {"start": 3759.0, "text": "or your users."}, {"start": 3763.0, "text": "Core AI is built"}, {"start": 3764.0, "text": "from the ground up with modern workloads in mind."}, {"start": 3767.0, "text": "And it provides the fast inference"}, {"start": 3769.0, "text": "that you need to build"}, {"start": 3770.0, "text": "advanced AI features."}, {"start": 3773.0, "text": "And it does this by taking advantage"}, {"start": 3775.0, "text": "of all of the Apple Silicon."}, {"start": 3777.0, "text": "Across the CPU,"}, {"start": 3778.0, "text": "the GPU, and the Neural Engine."}, {"start": 3782.0, "text": "Now, Core AI is not just the framework"}, {"start": 3784.0, "text": "that we build into the operating systems,"}, {"start": 3786.0, "text": "but a comprehensive suite"}, {"start": 3788.0, "text": "of technologies and tools"}, {"start": 3789.0, "text": "that expand the full model"}, {"start": 3791.0, "text": "deployment lifecycle."}, {"start": 3793.0, "text": "Starting with model creation,"}, {"start": 3795.0, "text": "optimization, and inference fine tuning,"}, {"start": 3798.0, "text": "all the way to app integration and debugging."}, {"start": 3801.0, "text": "We designed these"}, {"start": 3802.0, "text": "to enable the fast iterations"}, {"start": 3804.0, "text": "that you need to build great AI features."}, {"start": 3809.0, "text": "Now, to give you a better sense"}, {"start": 3810.0, "text": "of how all these pieces click together,"}, {"start": 3813.0, "text": "let’s go a little bit deeper"}, {"start": 3814.0, "text": "into four key areas of Core AI."}, {"start": 3817.0, "text": "Starting with a foundational one"}, {"start": 3819.0, "text": "which is our OS Framework."}, {"start": 3822.0, "text": "Core AI delivers great performance"}, {"start": 3824.0, "text": "through a clean, expressive,"}, {"start": 3826.0, "text": "and modern Swift API."}, {"start": 3829.0, "text": "This API is designed to progressively disclose"}, {"start": 3832.0, "text": "its capabilities."}, {"start": 3833.0, "text": "What this means is that we want to make it very easy"}, {"start": 3835.0, "text": "for you to get things up and running,"}, {"start": 3837.0, "text": "while still providing"}, {"start": 3839.0, "text": "the layers of flexibility that are necessary"}, {"start": 3841.0, "text": "to support performance"}, {"start": 3843.0, "text": "critical applications."}, {"start": 3846.0, "text": "And it's also designed"}, {"start": 3847.0, "text": "with modern Swift in mind."}, {"start": 3849.0, "text": "And this allows us to give capabilities"}, {"start": 3851.0, "text": "like memory-safety without sacrificing performance."}, {"start": 3856.0, "text": "In its simplest form, the API is very straightforward."}, {"start": 3859.0, "text": "It always starts with an AI model."}, {"start": 3861.0, "text": "This AI model gets loaded into your app,"}, {"start": 3864.0, "text": "into the device, and it’s specialized."}, {"start": 3867.0, "text": "The model contains one"}, {"start": 3868.0, "text": "or more functions"}, {"start": 3870.0, "text": "and each function receives inputs"}, {"start": 3871.0, "text": "and produces output in the form of"}, {"start": 3874.0, "text": "NDArray objects."}, {"start": 3876.0, "text": "This is even simple to to understand"}, {"start": 3878.0, "text": "when you look at the code."}, {"start": 3879.0, "text": "There is just one line"}, {"start": 3881.0, "text": "to instantiate your model,"}, {"start": 3883.0, "text": "another line to select the function"}, {"start": 3885.0, "text": "that you want to load and run,"}, {"start": 3888.0, "text": "and then another line to just simply run it."}, {"start": 3891.0, "text": "That's it."}, {"start": 3892.0, "text": "We take care of all"}, {"start": 3893.0, "text": "the complexity underneath to make this fast"}, {"start": 3896.0, "text": "and efficient."}, {"start": 3899.0, "text": "Now, the second area"}, {"start": 3900.0, "text": "is a comprehensive suite of Python Tools,"}, {"start": 3903.0, "text": "that enable preparing"}, {"start": 3904.0, "text": "your models for a framework and runtime."}, {"start": 3908.0, "text": "These tools integrate seamlessly"}, {"start": 3910.0, "text": "into familiar PyTorch workflows,"}, {"start": 3913.0, "text": "providing rich APIs"}, {"start": 3914.0, "text": "to prepare your model."}, {"start": 3916.0, "text": "What this means is that it's very easy to go"}, {"start": 3918.0, "text": "from a trained PyTorch model"}, {"start": 3920.0, "text": "into a Core AI model"}, {"start": 3922.0, "text": "that is optimized for efficient inference"}, {"start": 3925.0, "text": "in Apple Silicon."}, {"start": 3928.0, "text": "Let's look at the code. It's also very simple."}, {"start": 3930.0, "text": "We start first producing what we call"}, {"start": 3932.0, "text": "an exported version of the PyTorch model."}, {"start": 3935.0, "text": "In order to do this, we just use"}, {"start": 3937.0, "text": "standard PyTorch APIs."}, {"start": 3940.0, "text": "Then we use the Core AI Torch package."}, {"start": 3943.0, "text": "In this case, the TorchConverter,"}, {"start": 3945.0, "text": "to produce a Core AI"}, {"start": 3947.0, "text": "representation of the model."}, {"start": 3950.0, "text": "And finally,"}, {"start": 3951.0, "text": "we take this representation"}, {"start": 3952.0, "text": "of the Core AI model and we save it into an asset."}, {"start": 3956.0, "text": "This is the asset that you will load into your application"}, {"start": 3958.0, "text": "and you will run on device."}, {"start": 3962.0, "text": "Now we understand that each application needs are unique,"}, {"start": 3966.0, "text": "so Core AI offers"}, {"start": 3967.0, "text": "extensive Customization Capabilities."}, {"start": 3970.0, "text": "There is APIs to control different aspects"}, {"start": 3973.0, "text": "across model creation, optimization,"}, {"start": 3975.0, "text": "and execution."}, {"start": 3978.0, "text": "For example, on the author inside"}, {"start": 3979.0, "text": "we just show you the very simple"}, {"start": 3981.0, "text": "conversion workflow."}, {"start": 3983.0, "text": "But the Python package"}, {"start": 3984.0, "text": "is much richer than that."}, {"start": 3986.0, "text": "For example, it allows you to directly"}, {"start": 3988.0, "text": "author your model using these APIs."}, {"start": 3991.0, "text": "We also provide a comprehensive"}, {"start": 3994.0, "text": "suite of techniques and tools that allow you"}, {"start": 3996.0, "text": "to optimize your model for Apple Silicon."}, {"start": 4000.0, "text": "And you can go even all the way"}, {"start": 4002.0, "text": "and provide your own custom kernels to optimize your model"}, {"start": 4005.0, "text": "according to your specific needs."}, {"start": 4009.0, "text": "On the execution side,"}, {"start": 4010.0, "text": "the Core AI framework"}, {"start": 4011.0, "text": "allows you the control that you need"}, {"start": 4013.0, "text": "to optimize your application as you want it."}, {"start": 4016.0, "text": "You can go as deep as you want."}, {"start": 4018.0, "text": "For example, you can control"}, {"start": 4020.0, "text": "how cached management works,"}, {"start": 4022.0, "text": "when to purchase resources,"}, {"start": 4024.0, "text": "you can also share cached models"}, {"start": 4026.0, "text": "across different app groups."}, {"start": 4028.0, "text": "You can also query the formats of inputs"}, {"start": 4030.0, "text": "and outputs to device-specific layouts."}, {"start": 4034.0, "text": "So there is no there is no cost of copies"}, {"start": 4037.0, "text": "or conversions and inference time."}, {"start": 4040.0, "text": "Similarly, we allow you to"}, {"start": 4041.0, "text": "pre-allocate the values of states"}, {"start": 4044.0, "text": "and outputs and have functions write directly into them"}, {"start": 4047.0, "text": "and this is critical for optimizing Large Language Models."}, {"start": 4052.0, "text": "And we have a very flexible asynchronous API."}, {"start": 4055.0, "text": "So you can create function pipelines"}, {"start": 4058.0, "text": "and take advantage of parallel execution."}, {"start": 4061.0, "text": "Again you can go"}, {"start": 4062.0, "text": "as deep as you want depending on your needs"}, {"start": 4064.0, "text": "and the goals that you have for your particular application."}, {"start": 4069.0, "text": "Finally, all this is tied together"}, {"start": 4071.0, "text": "by our new Developer Toolchain."}, {"start": 4074.0, "text": "We’ve built deep integration"}, {"start": 4075.0, "text": "in the into the tools that you use every day"}, {"start": 4078.0, "text": "while still providing some cool new features."}, {"start": 4080.0, "text": "So let me show you a couple."}, {"start": 4083.0, "text": "First,"}, {"start": 4084.0, "text": "with Core AI, now we are enabling you to pre-compile"}, {"start": 4086.0, "text": "the models ahead of time"}, {"start": 4088.0, "text": "producing a pre-compiled asset."}, {"start": 4092.0, "text": "When this asset gets"}, {"start": 4093.0, "text": "loaded into the device,"}, {"start": 4094.0, "text": "it will still be optimized,"}, {"start": 4097.0, "text": "but this will be very fast."}, {"start": 4100.0, "text": "So this allows you to work with very large models."}, {"start": 4102.0, "text": "Again, very critical these days."}, {"start": 4106.0, "text": "And to complete that seamless loop"}, {"start": 4108.0, "text": "between modeling and development,"}, {"start": 4110.0, "text": "We're introducing the new"}, {"start": 4111.0, "text": "Core AI Debugger application."}, {"start": 4115.0, "text": "This allows you to visualize"}, {"start": 4116.0, "text": "computation graphs,"}, {"start": 4118.0, "text": "inspect tensor values,"}, {"start": 4120.0, "text": "and even trace those values"}, {"start": 4122.0, "text": "from the Core AI model"}, {"start": 4123.0, "text": "back into the original Python source code."}, {"start": 4126.0, "text": "Making everything very actionable."}, {"start": 4131.0, "text": "With these four pillars, Core AI"}, {"start": 4133.0, "text": "now lets you deploy powerful"}, {"start": 4134.0, "text": "AI to your users at scale."}, {"start": 4137.0, "text": "And talking about"}, {"start": 4138.0, "text": "a different type of scalability,"}, {"start": 4140.0, "text": "Core AI is engineered to seamlessly scale"}, {"start": 4143.0, "text": "with the compute available on the device."}, {"start": 4146.0, "text": "This means that you can run"}, {"start": 4147.0, "text": "a highly responsive vision model"}, {"start": 4150.0, "text": "on an iPhone for doing something"}, {"start": 4152.0, "text": "like real time camera capture,"}, {"start": 4154.0, "text": "inquiries, or push the limits"}, {"start": 4157.0, "text": "with a multi-billion-parameter model"}, {"start": 4159.0, "text": "running on a Mac Studio"}, {"start": 4161.0, "text": "for doing some complex agentic workflow."}, {"start": 4164.0, "text": "Now, to give you a concrete"}, {"start": 4165.0, "text": "example of what it’s possible,"}, {"start": 4167.0, "text": "here is Geppy."}, {"start": 4172.0, "text": "Thanks, Raziel."}, {"start": 4174.0, "text": "Hello, my name is Geppy Parziale"}, {"start": 4176.0, "text": "and I'm a creative technologist"}, {"start": 4178.0, "text": "here at Apple."}, {"start": 4180.0, "text": "Today I want to show you an iPad app I’ve built."}, {"start": 4184.0, "text": "It’s an ideation creative"}, {"start": 4186.0, "text": "tool for working with images"}, {"start": 4189.0, "text": "and generative AI."}, {"start": 4193.0, "text": "Let me show you. The workflow"}, {"start": 4195.0, "text": "I’m going to show you"}, {"start": 4197.0, "text": "is style re-targeting."}, {"start": 4199.0, "text": "You bring in an image, write a prompt,"}, {"start": 4203.0, "text": "and generate multiple variations of the source image."}, {"start": 4208.0, "text": "So let me load an image."}, {"start": 4211.0, "text": "I'll pick up this one."}, {"start": 4216.0, "text": "Let me add a prompt"}, {"start": 4219.0, "text": "thatI have prepared."}, {"start": 4233.0, "text": "So “Create variations from this image across different camera"}, {"start": 4236.0, "text": "angles and lighting conditions.”"}, {"start": 4239.0, "text": "I hit generate,"}, {"start": 4241.0, "text": "and it starts to work."}, {"start": 4244.0, "text": "This is running fully on device."}, {"start": 4246.0, "text": "No cloud APIs, no external services."}, {"start": 4249.0, "text": "All the inference runs locally using Core AI."}, {"start": 4253.0, "text": "Now what's interesting is"}, {"start": 4255.0, "text": "what's, what's happening under the hood."}, {"start": 4258.0, "text": "Let me show you."}, {"start": 4262.0, "text": "So first, I use a Visual Language Model"}, {"start": 4265.0, "text": "that analyzed the image"}, {"start": 4267.0, "text": "and combines it with a user prompt,"}, {"start": 4270.0, "text": "and an internal art direction"}, {"start": 4272.0, "text": "prompt to produce a creative brief."}, {"start": 4275.0, "text": "That brief feeds into FLUX,"}, {"start": 4278.0, "text": "and image generation model,"}, {"start": 4281.0, "text": "that generates multiple variations."}, {"start": 4284.0, "text": "I converted both FLUX"}, {"start": 4285.0, "text": "and the VLM from PyTorch"}, {"start": 4288.0, "text": "and wired them in"}, {"start": 4289.0, "text": "with just a few lines of code."}, {"start": 4292.0, "text": "Both models are run using Core AI"}, {"start": 4295.0, "text": "completely on device."}, {"start": 4297.0, "text": "This ensures to preserve"}, {"start": 4298.0, "text": "the privacy of my users,"}, {"start": 4300.0, "text": "since the assets do not leave the device."}, {"start": 4305.0, "text": "In addition,"}, {"start": 4306.0, "text": "since I want to fine tune FLUX,"}, {"start": 4309.0, "text": "I need control of the full stack."}, {"start": 4312.0, "text": "Core AI gives me both privacy"}, {"start": 4315.0, "text": "and control."}, {"start": 4318.0, "text": "And here is the result."}, {"start": 4320.0, "text": "Beautiful."}, {"start": 4324.0, "text": "So multiple variations,"}, {"start": 4327.0, "text": "fully on device."}, {"start": 4330.0, "text": "So what's next,"}, {"start": 4332.0, "text": "I wanted to scale this up"}, {"start": 4333.0, "text": "to use larger, more capable models"}, {"start": 4336.0, "text": "for higher resolution"}, {"start": 4338.0, "text": "and more variations."}, {"start": 4340.0, "text": "What I did was simple."}, {"start": 4342.0, "text": "I took the iPad codebase,"}, {"start": 4344.0, "text": "I built a macOS app,"}, {"start": 4346.0, "text": "I'm showing you here,"}, {"start": 4348.0, "text": "Swapping in larger variants of FLUX and VLM."}, {"start": 4352.0, "text": "Same code, different models."}, {"start": 4355.0, "text": "Core AI made that swap"}, {"start": 4357.0, "text": "completely straightforward."}, {"start": 4360.0, "text": "Now, back to Raziel."}, {"start": 4366.0, "text": "Thank you, Geppy."}, {"start": 4368.0, "text": "This is just an example of the flexibility"}, {"start": 4370.0, "text": "and power of Core AI in action."}, {"start": 4372.0, "text": "And there is no better proof of this flexibility"}, {"start": 4374.0, "text": "and power than how we're using it ourselves."}, {"start": 4377.0, "text": "This is why now Core AI powers"}, {"start": 4379.0, "text": "Apple Intelligence and the all new Siri."}, {"start": 4382.0, "text": "With Core AI, every developer can"}, {"start": 4384.0, "text": "now tap into the very same"}, {"start": 4386.0, "text": "inference framework that powers"}, {"start": 4387.0, "text": "our most advanced AI features."}, {"start": 4390.0, "text": "One that is deeply optimized"}, {"start": 4392.0, "text": "to unlock the full potential"}, {"start": 4393.0, "text": "of Apple Silicon."}, {"start": 4399.0, "text": "With Core AI, we're giving you the tools"}, {"start": 4401.0, "text": "to remove the friction from AI deployment,"}, {"start": 4403.0, "text": "and we cannot wait to see what amazing new experiences"}, {"start": 4406.0, "text": "you create."}, {"start": 4408.0, "text": "Now, this is not all that"}, {"start": 4410.0, "text": "we have to tell you about AI frameworks."}, {"start": 4413.0, "text": "So next, let me hand off to Ronan,"}, {"start": 4414.0, "text": "who's going to tell you what's new on MLX."}, {"start": 4417.0, "text": "Thanks."}, {"start": 4423.0, "text": "Thanks, Raziel."}, {"start": 4424.0, "text": "And hello everyone."}, {"start": 4426.0, "text": "I'm, Ronan from the MLX team."}, {"start": 4429.0, "text": "And today,"}, {"start": 4430.0, "text": "I'm going to talk about MLX,"}, {"start": 4432.0, "text": "a Machine Learning and Numerical Computing framework"}, {"start": 4435.0, "text": "designed for Apple Silicon."}, {"start": 4439.0, "text": "And with MLX,"}, {"start": 4440.0, "text": "we are advancing modern science on Apple Silicon."}, {"start": 4443.0, "text": "The framework is built on"}, {"start": 4445.0, "text": "four core pillars."}, {"start": 4447.0, "text": "First is Numerical Computing,"}, {"start": 4449.0, "text": "at the foundation of modern science."}, {"start": 4452.0, "text": "MLX brings a flexible numerical computing library"}, {"start": 4454.0, "text": "to the Mac for fast linear algebra"}, {"start": 4457.0, "text": "and vectorized array operations."}, {"start": 4461.0, "text": "Second is Automatic Differentiation,"}, {"start": 4464.0, "text": "which handles gradient"}, {"start": 4465.0, "text": "computation for you."}, {"start": 4467.0, "text": "This is often used for model training"}, {"start": 4469.0, "text": "in machine learning."}, {"start": 4471.0, "text": "Thirdly, Distributed Computing"}, {"start": 4473.0, "text": "lets you scale massive workloads"}, {"start": 4476.0, "text": "across multiple Macs."}, {"start": 4479.0, "text": "And finally,"}, {"start": 4480.0, "text": "Machine Learning. With MLX"}, {"start": 4482.0, "text": "you can run powerful vision, speech,"}, {"start": 4484.0, "text": "or language models"}, {"start": 4486.0, "text": "right there on your Mac."}, {"start": 4489.0, "text": "And MLX is Apple Silicon first!"}, {"start": 4493.0, "text": "It’s built to take advantage"}, {"start": 4494.0, "text": "of the unified memory architecture."}, {"start": 4497.0, "text": "It's GPU accelerated directly"}, {"start": 4499.0, "text": "leveraging Metal,"}, {"start": 4500.0, "text": "and it supports the latest"}, {"start": 4502.0, "text": "advances in Apple Silicon,"}, {"start": 4504.0, "text": "including the GPU Neural Accelerator"}, {"start": 4507.0, "text": "from the M5."}, {"start": 4510.0, "text": "With MLX, we bring the same numerical"}, {"start": 4513.0, "text": "computing interface to several languages."}, {"start": 4517.0, "text": "First, in Python,"}, {"start": 4519.0, "text": "as Python is the most common programming language"}, {"start": 4522.0, "text": "of the Machine Learning community."}, {"start": 4525.0, "text": "Second is Swift,"}, {"start": 4527.0, "text": "because Swift is the language of choice"}, {"start": 4529.0, "text": "when it comes to Apple devices."}, {"start": 4532.0, "text": "And for people"}, {"start": 4533.0, "text": "who are into low-level programming,"}, {"start": 4535.0, "text": "we also have C and C++."}, {"start": 4539.0, "text": "And from there, MLX"}, {"start": 4540.0, "text": "brings also support for Machine Learning and AI,"}, {"start": 4543.0, "text": "with several higher-level packages"}, {"start": 4546.0, "text": "in particular,"}, {"start": 4547.0, "text": "MLX makes it easy"}, {"start": 4549.0, "text": "to train, fine tune, or simply run"}, {"start": 4552.0, "text": "language models locally on your Mac."}, {"start": 4555.0, "text": "In fact, here is all the Python code"}, {"start": 4558.0, "text": "it takes to use a state of"}, {"start": 4561.0, "text": "the art model on your Mac."}, {"start": 4564.0, "text": "One line to fetch a model"}, {"start": 4567.0, "text": "and one line"}, {"start": 4568.0, "text": "to generate text according to a prompt."}, {"start": 4571.0, "text": "It is that simple."}, {"start": 4573.0, "text": "And beyond Python,"}, {"start": 4574.0, "text": "we also provide a command line interface for ease of use"}, {"start": 4578.0, "text": "and a native Swift frontend as well."}, {"start": 4583.0, "text": "In fact"}, {"start": 4584.0, "text": "the whole MLX stack is open-source."}, {"start": 4587.0, "text": "From low-level kernels"}, {"start": 4588.0, "text": "to these high-level APIs."}, {"start": 4590.0, "text": "MLX is available on GitHub,"}, {"start": 4592.0, "text": "with a thriving community"}, {"start": 4594.0, "text": "and a lot of contributors."}, {"start": 4597.0, "text": "MLX has also a great presence on HuggingFace."}, {"start": 4600.0, "text": "HuggingFace is a good place"}, {"start": 4601.0, "text": "to find the latest"}, {"start": 4603.0, "text": "and greatest language models"}, {"start": 4606.0, "text": "from the machine learning community."}, {"start": 4609.0, "text": "And there are more"}, {"start": 4610.0, "text": "than 10,000 models from there"}, {"start": 4612.0, "text": "supported out of the box by MLX LM."}, {"start": 4617.0, "text": "Today, some of the most popular tools"}, {"start": 4621.0, "text": "for using language models on your Mac"}, {"start": 4624.0, "text": "are now powered by MLX."}, {"start": 4627.0, "text": "This includes Ollama, LM Studio,"}, {"start": 4629.0, "text": "or vLLM, to name just a few."}, {"start": 4632.0, "text": "So If you have been using one of these tools"}, {"start": 4634.0, "text": "on your Mac, chances are"}, {"start": 4636.0, "text": "you have been running on MLX already."}, {"start": 4640.0, "text": "But the community keeps building more."}, {"start": 4642.0, "text": "There are tons of open-source 3rd-party packages,"}, {"start": 4645.0, "text": "available out there,"}, {"start": 4647.0, "text": "like for training models from scratch,"}, {"start": 4649.0, "text": "fine tuning large language models,"}, {"start": 4651.0, "text": "support for speech,"}, {"start": 4652.0, "text": "image or video generation,"}, {"start": 4655.0, "text": "exploratory research work,"}, {"start": 4657.0, "text": "or even Large Language Model servers,"}, {"start": 4659.0, "text": "all that running locally on your Mac!"}, {"start": 4663.0, "text": "On that note, I'm"}, {"start": 4664.0, "text": "going to hand it over to Angelos"}, {"start": 4666.0, "text": "who is going to show you"}, {"start": 4668.0, "text": "what you can do with MLX on the MacBook Pro M5 Max."}, {"start": 4673.0, "text": "Thank you, Ronan."}, {"start": 4675.0, "text": "Hello everybody."}, {"start": 4677.0, "text": "Today I'm"}, {"start": 4678.0, "text": "going to show you a macOS application"}, {"start": 4680.0, "text": "that allows you to load the book, a whole book"}, {"start": 4683.0, "text": "and have a conversation about it"}, {"start": 4684.0, "text": "with a language model."}, {"start": 4686.0, "text": "Okay, let's load a book."}, {"start": 4688.0, "text": "I'm going to kick The Great Gatsby."}, {"start": 4693.0, "text": "And off we go."}, {"start": 4694.0, "text": "Here we are, feeding"}, {"start": 4695.0, "text": "the whole content of the book"}, {"start": 4697.0, "text": "to the KV cache of a language model."}, {"start": 4699.0, "text": "And because MLX takes great advantage"}, {"start": 4701.0, "text": "of the Neural Accelerator on this Mac,"}, {"start": 4704.0, "text": "on this M5 Max actually,"}, {"start": 4706.0, "text": "we are processing almost 2000 words per second"}, {"start": 4710.0, "text": "which is actually impressive"}, {"start": 4711.0, "text": "because we'll end up needing"}, {"start": 4713.0, "text": "only a little bit more than 30 seconds to process the whole book."}, {"start": 4716.0, "text": "Almost 200 page"}, {"start": 4718.0, "text": "and almost 100,000 tokens."}, {"start": 4720.0, "text": "And after we processed it,"}, {"start": 4722.0, "text": "our answers are going to be instant"}, {"start": 4723.0, "text": "because we don't need to process it"}, {"start": 4725.0, "text": "again and again."}, {"start": 4727.0, "text": "So let's give it one more second."}, {"start": 4729.0, "text": "And here we go."}, {"start": 4730.0, "text": "So just to make sure that the model has indeed read"}, {"start": 4733.0, "text": "the book, let's ask a simple question."}, {"start": 4735.0, "text": "I'm going to ask it to “List the top"}, {"start": 4738.0, "text": "then characters"}, {"start": 4741.0, "text": "in the book”."}, {"start": 4742.0, "text": "That looks correct."}, {"start": 4745.0, "text": "And off we go."}, {"start": 4746.0, "text": "Indeed, indeed."}, {"start": 4747.0, "text": "Starting with, Jay Gatsby and,"}, {"start": 4750.0, "text": "the narrator, Nick Carraway."}, {"start": 4752.0, "text": "And it looks like a pretty good list to me,"}, {"start": 4754.0, "text": "but that's a pretty simple question."}, {"start": 4757.0, "text": "Let's ask you to make something more interesting."}, {"start": 4760.0, "text": "How about to make a cover"}, {"start": 4762.0, "text": "image for the book?"}, {"start": 4769.0, "text": "So here, we're actually running a"}, {"start": 4771.0, "text": "12 billion parameter diffusion model"}, {"start": 4774.0, "text": "also on the Mac."}, {"start": 4775.0, "text": "And we- in mere seconds, we have a very nice cover"}, {"start": 4778.0, "text": "very much based on the context of the book."}, {"start": 4783.0, "text": "Thank you very much."}, {"start": 4784.0, "text": "And with that, I'll give it back to Ronan."}, {"start": 4792.0, "text": "Thanks Angelos."}, {"start": 4793.0, "text": "This was blazing fast!"}, {"start": 4795.0, "text": "But how can we make the experience"}, {"start": 4798.0, "text": "even faster?"}, {"start": 4800.0, "text": "Well, instead of using only one MacBook Pro M5 Max,"}, {"start": 4803.0, "text": "let's use two MacBook Pro M5 Max."}, {"start": 4806.0, "text": "And here we go."}, {"start": 4807.0, "text": "Angelos demo is now processing over"}, {"start": 4810.0, "text": "3000 words per second."}, {"start": 4812.0, "text": "So how did we make that happen?"}, {"start": 4815.0, "text": "Well,"}, {"start": 4816.0, "text": "we leveraged distributed computing."}, {"start": 4819.0, "text": "Distributed computing"}, {"start": 4820.0, "text": "is about parallelizing operations"}, {"start": 4822.0, "text": "over multiple Macs"}, {"start": 4824.0, "text": "By doing so, you scale up not only compute,"}, {"start": 4826.0, "text": "but also memory capacity"}, {"start": 4828.0, "text": "and memory bandwidth."}, {"start": 4830.0, "text": "But distributed computing"}, {"start": 4832.0, "text": "requires machines to be"}, {"start": 4834.0, "text": "efficiently interconnected"}, {"start": 4837.0, "text": "And that's where"}, {"start": 4838.0, "text": "Thunderbolt 5 comes into play."}, {"start": 4841.0, "text": "Last year we added Thunderbolt 5"}, {"start": 4843.0, "text": "RDMA support into MLX."}, {"start": 4845.0, "text": "Thunderbolt 5 enables high bandwidth"}, {"start": 4848.0, "text": "and low latency communication,"}, {"start": 4850.0, "text": "which are both critical for machine learning applications."}, {"start": 4856.0, "text": "Beyond speed,"}, {"start": 4857.0, "text": "another good reason to scale is to run"}, {"start": 4860.0, "text": "even larger language models."}, {"start": 4862.0, "text": "To solve the hardest problems on your Mac,"}, {"start": 4865.0, "text": "you may need the largest"}, {"start": 4867.0, "text": "language model available out there!"}, {"start": 4870.0, "text": "One of this language"}, {"start": 4871.0, "text": "model is Kimi 2.6"}, {"start": 4873.0, "text": "It comes with 1 trillion parameters."}, {"start": 4877.0, "text": "It won’t fit on a single machine,"}, {"start": 4879.0, "text": "but it will fit on several Macs."}, {"start": 4883.0, "text": "And here I'm going to hand it over"}, {"start": 4885.0, "text": "to a very special guest."}, {"start": 4887.0, "text": "I'm going to invite"}, {"start": 4889.0, "text": "Yagil from LM Studio on stage."}, {"start": 4892.0, "text": "He is going to demonstrate distributed"}, {"start": 4895.0, "text": "inference on a cluster of Macs."}, {"start": 4903.0, "text": "Thank you."}, {"start": 4904.0, "text": "Hi, everyone,"}, {"start": 4905.0, "text": "I'm Yagil, founder of LM Studio,"}, {"start": 4907.0, "text": "an app for running AI models"}, {"start": 4909.0, "text": "locally on your Mac."}, {"start": 4911.0, "text": "Today I’m going to show you"}, {"start": 4912.0, "text": "an upcoming feature in LM Studio,"}, {"start": 4914.0, "text": "that leverages MLX Distributed to run"}, {"start": 4917.0, "text": "much larger models on your local hardware."}, {"start": 4920.0, "text": "Over here I have four Mac Studios."}, {"start": 4923.0, "text": "They are all connected together"}, {"start": 4924.0, "text": "with Thunderbolt 5 to form"}, {"start": 4926.0, "text": "an ultra-low-latency communication network"}, {"start": 4929.0, "text": "using RDMA-over-Thunderbolt."}, {"start": 4932.0, "text": "With this setup in LM Studio,"}, {"start": 4934.0, "text": "I'm able to load and run"}, {"start": 4937.0, "text": "one of the largest models available today"}, {"start": 4939.0, "text": "Kimi K2.6 a frontier"}, {"start": 4942.0, "text": "open weight model with 1 Trillion parameters."}, {"start": 4946.0, "text": "So this is LM Studio"}, {"start": 4948.0, "text": "and as you can see,"}, {"start": 4949.0, "text": "I have Kimi loaded on the cluster,"}, {"start": 4951.0, "text": "distributed across these four Macs over here."}, {"start": 4954.0, "text": "And I’m actually"}, {"start": 4955.0, "text": "going to be using it remotely from this MacBook"}, {"start": 4958.0, "text": "using a feature in LM Studio called LM Link."}, {"start": 4961.0, "text": "LM Link allows you"}, {"start": 4962.0, "text": "to use your compute power"}, {"start": 4964.0, "text": "remotely over an end-to-end"}, {"start": 4965.0, "text": "encrypted connection."}, {"start": 4967.0, "text": "So we have the model loaded"}, {"start": 4968.0, "text": "1 trillion parameters"}, {"start": 4970.0, "text": "right here on these four Macs."}, {"start": 4972.0, "text": "And I'm going to demonstrate live using it."}, {"start": 4974.0, "text": "And we'll see how it goes."}, {"start": 4977.0, "text": "So I’ll paste the prompt and let it start,"}, {"start": 4980.0, "text": "let it start working."}, {"start": 4981.0, "text": "While I explain to you what it's doing."}, {"start": 4985.0, "text": "All right."}, {"start": 4986.0, "text": "So, I've taken The Great Gatsby and I've given it to the model"}, {"start": 4990.0, "text": "so this is the entire book"}, {"start": 4991.0, "text": "in the model’s context."}, {"start": 4993.0, "text": "And I asked it to read through the whole book."}, {"start": 4996.0, "text": "And when it's ready to tell me that, it's ready."}, {"start": 4998.0, "text": "And it did it."}, {"start": 4999.0, "text": "And, we're using a giant model."}, {"start": 5001.0, "text": "It's extremely capable."}, {"start": 5003.0, "text": "And when you’re using a capable model"}, {"start": 5005.0, "text": "you can expect to give it a very specific prompt."}, {"start": 5008.0, "text": "You can expect to give it tools, and you want to see it"}, {"start": 5010.0, "text": "following exactly what you ask it to do."}, {"start": 5012.0, "text": "So in this example we're giving it,"}, {"start": 5014.0, "text": "The Great Gatsby."}, {"start": 5015.0, "text": "And I've, I've told the model that I'm"}, {"start": 5017.0, "text": "a ninth grade schoolteacher and"}, {"start": 5020.0, "text": "I’m asking it to help me generate a quiz for my students."}, {"start": 5023.0, "text": "And I'm very specific about what I want."}, {"start": 5025.0, "text": "I want so and so many questions, and"}, {"start": 5029.0, "text": "so and so many questions"}, {"start": 5030.0, "text": "and when it's done, I would like it to be, an HTML page."}, {"start": 5033.0, "text": "And then I've also asked it to"}, {"start": 5035.0, "text": "open the page when it's done."}, {"start": 5037.0, "text": "It was obviously too quick."}, {"start": 5039.0, "text": "Here it is."}, {"start": 5040.0, "text": "So this is the quiz it was able to generate, here"}, {"start": 5043.0, "text": "live on stage, running on this cluster."}, {"start": 5046.0, "text": "Pretty cool."}, {"start": 5051.0, "text": "Cool."}, {"start": 5052.0, "text": "So this was done totally locally, right?"}, {"start": 5055.0, "text": "So, the model is running locally."}, {"start": 5057.0, "text": "Nothing went to the cloud while it did it."}, {"start": 5059.0, "text": "Now, when you're using a very capable model,"}, {"start": 5061.0, "text": "you can also expect to give it follow up"}, {"start": 5063.0, "text": "questions and follow up"}, {"start": 5065.0, "text": "tasks like this one."}, {"start": 5066.0, "text": "So we asked it to make a quiz for the ninth grade."}, {"start": 5068.0, "text": "But, maybe we want to make a quiz that's suitable"}, {"start": 5071.0, "text": "for university setting or something like that."}, {"start": 5073.0, "text": "So I've gone ahead, and,"}, {"start": 5076.0, "text": "asked it to, to do something additional."}, {"start": 5078.0, "text": "But while it's working,"}, {"start": 5080.0, "text": "I'll just explain to you a little bit how this works."}, {"start": 5082.0, "text": "So this is leveraging MLX Distributed."}, {"start": 5085.0, "text": "And this allows"}, {"start": 5086.0, "text": "you to run extremely, large models."}, {"start": 5089.0, "text": "And for some tasks you need frontier intelligence,"}, {"start": 5091.0, "text": "But the really incredible thing about this"}, {"start": 5094.0, "text": "thing is that, there's no token cost."}, {"start": 5096.0, "text": "You can run this thing 24/7,"}, {"start": 5098.0, "text": "and no data goes out of the cloud."}, {"start": 5099.0, "text": "So anything that I've showed here"}, {"start": 5101.0, "text": "today, nothing went out, to the cloud."}, {"start": 5104.0, "text": "And this is suitable for,"}, {"start": 5106.0, "text": "using sensitive data,"}, {"start": 5108.0, "text": "at work, at school or at home."}, {"start": 5112.0, "text": "MLX Distributed support"}, {"start": 5113.0, "text": "in LM Studio is coming later this year,"}, {"start": 5116.0, "text": "and you'll even be able to use your compute power"}, {"start": 5119.0, "text": "remotely from your other devices, using LM Link."}, {"start": 5122.0, "text": "Thank you. And back to Ronan."}, {"start": 5128.0, "text": "Thanks Yagil."}, {"start": 5131.0, "text": "Totally inspiring,"}, {"start": 5132.0, "text": "like a frontier model right at your fingertips."}, {"start": 5135.0, "text": "So, wrapping up with MLX"}, {"start": 5137.0, "text": "we are advancing modern science on Apple Silicon."}, {"start": 5140.0, "text": "And it’s all open source."}, {"start": 5143.0, "text": "So whether you want to contribute, back,"}, {"start": 5145.0, "text": "build on it, or simply experiment"}, {"start": 5147.0, "text": "with the largest language models out there, well,"}, {"start": 5149.0, "text": "come and check it out"}, {"start": 5151.0, "text": "on MLXframework.org."}, {"start": 5154.0, "text": "Thank you."}, {"start": 5155.0, "text": "And now back to Rachel."}, {"start": 5159.0, "text": "Thanks, Ronan!"}, {"start": 5161.0, "text": "Across our technologies,"}, {"start": 5163.0, "text": "no matter where you start"}, {"start": 5165.0, "text": "from tools to frameworks"}, {"start": 5167.0, "text": "to operating systems and hardware you're working with something"}, {"start": 5170.0, "text": "that makes AI faster, safer,"}, {"start": 5172.0, "text": "and more capable at every level."}, {"start": 5174.0, "text": "And while we went deep in these areas today,"}, {"start": 5177.0, "text": "there’s so much more to explore"}, {"start": 5179.0, "text": "with over 100 sessions to dive into everything"}, {"start": 5182.0, "text": "that we've covered today and all of our other announcements."}, {"start": 5185.0, "text": "You can find them on the Apple"}, {"start": 5187.0, "text": "Developer app and website,"}, {"start": 5189.0, "text": "YouTube, and on a new bilibili channel."}, {"start": 5192.0, "text": "You can also join us online"}, {"start": 5193.0, "text": "throughout the week in Group Labs"}, {"start": 5195.0, "text": "hosted by Apple engineers and designers"}, {"start": 5198.0, "text": "so drop in, bring your questions,"}, {"start": 5200.0, "text": "or just follow along."}, {"start": 5202.0, "text": "Anyone can watch live or on demand."}, {"start": 5205.0, "text": "Apple experts are also on the Apple Developer"}, {"start": 5208.0, "text": "Developer Forums all week long and beyond."}, {"start": 5211.0, "text": "And you can use"}, {"start": 5212.0, "text": "the new AI powered search on the developer website,"}, {"start": 5215.0, "text": "which brings together documentation, sample code,"}, {"start": 5218.0, "text": "sessions, group labs, and more."}, {"start": 5221.0, "text": "But right now"}, {"start": 5222.0, "text": "we'd love to continue this moment,"}, {"start": 5225.0, "text": "connecting with all of you throughout the afternoon"}, {"start": 5228.0, "text": "at the Apple Developer Center"}, {"start": 5229.0, "text": "across the street."}, {"start": 5231.0, "text": "You can join us"}, {"start": 5232.0, "text": "for lunch from a variety of food trucks,"}, {"start": 5234.0, "text": "and connect with Apple engineers"}, {"start": 5237.0, "text": "You can explore the Developer Center"}, {"start": 5238.0, "text": "and sit in on a live"}, {"start": 5240.0, "text": "viewing of our Swift Group Lab."}, {"start": 5242.0, "text": "Or you could vibe"}, {"start": 5244.0, "text": "with our Apple experts in lounges"}, {"start": 5247.0, "text": "for Icon Composer,"}, {"start": 5248.0, "text": "Xcode, and Reality Composer Pro."}, {"start": 5251.0, "text": "We hope to see all of you there!"}, {"start": 5252.0, "text": "Have a great WWDC!"}]}