{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-325", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/325", "wordCount": 3892, "segments": [{"start": 7.0, "text": "Hi. I'm Sachin,"}, {"start": 8.0, "text": "an engineer on the Core AI team,"}, {"start": 10.0, "text": "and later I will be joined by my colleague Nicole."}, {"start": 14.0, "text": "Today, we look forward to showing you how Core AI makes it easy,"}, {"start": 17.0, "text": "to get your model running efficiently on Apple Silicon"}, {"start": 21.0, "text": "from the basics to more advanced approaches."}, {"start": 24.0, "text": "As you saw in the \"Meet Core AI\" talk,"}, {"start": 27.0, "text": "Core AI is a complete suite of technologies,"}, {"start": 29.0, "text": "covering the model deployment lifecycle"}, {"start": 32.0, "text": "from model optimization and conversion to debugging and integration into your app."}, {"start": 39.0, "text": "In this talk, we will be zooming in on Core AI's Python ecosystem."}, {"start": 44.0, "text": "You will meet the different libraries and tools"}, {"start": 46.0, "text": "that Core AI provides to get your models deployed on Apple Silicon."}, {"start": 52.0, "text": "You will also meet the Core AI Debugger,"}, {"start": 53.0, "text": "which can be an integral part of your workflow"}, {"start": 57.0, "text": "for understanding and debugging key issues."}, {"start": 60.0, "text": "Lets dive in."}, {"start": 62.0, "text": "Here's our agenda."}, {"start": 63.0, "text": "I will first introduce you to the Core AI models repository,"}, {"start": 67.0, "text": "and Core AI skills, powerful tools that will jump-start your journey."}, {"start": 71.0, "text": "Then, I will show you the basic conversion and verification process."}, {"start": 76.0, "text": "Core AI is built around the Python and PyTorch workflows you already know"}, {"start": 81.0, "text": "and if you've used Core ML before, a lot of this will feel pretty familiar."}, {"start": 86.0, "text": "I will follow that with model optimization,"}, {"start": 88.0, "text": "and show how you can make the right choices"}, {"start": 91.0, "text": "for your use-case and target platform."}, {"start": 94.0, "text": "To help you get a deeper insight into your model,"}, {"start": 97.0, "text": "Nicole will introduce you to Core AI Debugger."}, {"start": 101.0, "text": "And finally, I will touch upon key ways"}, {"start": 103.0, "text": "that Core AI allows deep customization during model authoring and conversion."}, {"start": 108.0, "text": "Okay."}, {"start": 109.0, "text": "Lets start with models and skills."}, {"start": 113.0, "text": "At the forefront of Core AI's ecosystem, is the coreai-models repository."}, {"start": 118.0, "text": "It includes a Swift package for running LLMs in your app."}, {"start": 123.0, "text": "But at its core, its an open-source repository of models that are ready to go,"}, {"start": 128.0, "text": "including generative architectures like cutting-edge large language models."}, {"start": 133.0, "text": "We have examples engineered for various use-cases and constraints,"}, {"start": 138.0, "text": "along with components that you can use to bring your own models to Core AI."}, {"start": 144.0, "text": "And last but not the least,"}, {"start": 145.0, "text": "Core AI models ships with a set of agent skills."}, {"start": 149.0, "text": "You can install these skills into your favorite coding assistant,"}, {"start": 153.0, "text": "to get started with Core AI, just like an expert, from day one."}, {"start": 158.0, "text": "Core AI skills work with you and translate your high-level ideas"}, {"start": 162.0, "text": "into a clear deployment plan for downstream tasks."}, {"start": 166.0, "text": "They may get clarifications from you around the model you are interested in,"}, {"start": 169.0, "text": "the hardware families you are targeting, and the constraints your application has."}, {"start": 174.0, "text": "These requirements inform the Core AI features you need,"}, {"start": 177.0, "text": "all the way from any changes in the PyTorch model code"}, {"start": 180.0, "text": "to conversion, optimization and running the models."}, {"start": 184.0, "text": "AI skills give your coding agent access to the best practices"}, {"start": 188.0, "text": "and domain knowledge from our engineers."}, {"start": 191.0, "text": "This empowers you to leverage Core AI like a pro,"}, {"start": 194.0, "text": "and even understand it better with your coding assistant!"}, {"start": 198.0, "text": "In fact, most of the code you will see throughout this talk"}, {"start": 202.0, "text": "was co-developed with an agent actively leveraging these skills."}, {"start": 207.0, "text": "Now, lets dig into converting and running models in Core AI with Python."}, {"start": 213.0, "text": "The Core AI Python libraries, primarily Core AI PyTorch extensions,"}, {"start": 218.0, "text": "are your entry point into the ecosystem."}, {"start": 221.0, "text": "Installation is simple with pip install coreai-torch,"}, {"start": 225.0, "text": "this installs both the coreai package"}, {"start": 228.0, "text": "and the coreai-torch library building on top of it."}, {"start": 232.0, "text": "You hand coreai-torch a PyTorch exported program,"}, {"start": 235.0, "text": "and it converts directly to a Core AI model."}, {"start": 239.0, "text": "It supports advanced features that let you tailor the Core AI program"}, {"start": 242.0, "text": "to your exact use-case."}, {"start": 244.0, "text": "For example, you can assemble multiple models into a single artifact,"}, {"start": 249.0, "text": "register custom lowerings for specific operations,"}, {"start": 252.0, "text": "and inline Metal 4 kernels right into your converted model."}, {"start": 256.0, "text": "And finally, you can specialize models into optimized assets,"}, {"start": 260.0, "text": "and run them natively on Apple Silicon entirely from Python."}, {"start": 265.0, "text": "Here's the pipeline I just described."}, {"start": 267.0, "text": "Now, lets see this in practice."}, {"start": 270.0, "text": "I will walk you through a quick example."}, {"start": 273.0, "text": "Here I have a neural network — two linear layers with a relu activation."}, {"start": 278.0, "text": "Standard PyTorch."}, {"start": 280.0, "text": "Then, I run torch.export,"}, {"start": 282.0, "text": "I pass the model and an example_input, which gives me an exported_program."}, {"start": 288.0, "text": "This exported_program is the starting point"}, {"start": 290.0, "text": "for Core AI conversion."}, {"start": 292.0, "text": "It captures the full computational graph: weights, operations and shapes"}, {"start": 298.0, "text": "in a format that coreai-torch can work with."}, {"start": 302.0, "text": "And now, the Core AI side."}, {"start": 305.0, "text": "Core AI's TorchConverter takes my exported program,"}, {"start": 308.0, "text": "along with the input and output names, and converts it to a core_ai_ program."}, {"start": 313.0, "text": "If you've used CoreML tools before, this will feel familiar."}, {"start": 318.0, "text": "The converted model is then optimized and saved as an aimodel asset —"}, {"start": 323.0, "text": "an on-device format ready to run on Apple Silicon."}, {"start": 327.0, "text": "Once I have the specialized asset, I can load a function from the program"}, {"start": 331.0, "text": "and perform inference right from Python."}, {"start": 334.0, "text": "You can also pass specialization options at this point to customize the process."}, {"start": 339.0, "text": "To actually run inference,"}, {"start": 340.0, "text": "all you need to do is provide a dictionary mapping input names"}, {"start": 344.0, "text": "to corresponding numpy tensors!"}, {"start": 347.0, "text": "That's it… the whole workflow."}, {"start": 349.0, "text": "Conversion, optimization and execution — all from Python."}, {"start": 354.0, "text": "Now let's talk about making a model smaller,"}, {"start": 356.0, "text": "using Core AI's optimization library."}, {"start": 359.0, "text": "To showcase Core AI's optimization features,"}, {"start": 362.0, "text": "I'll be taking Segment Anything Model — SAM3 as the driving use-case."}, {"start": 368.0, "text": "SAM3 is an 850-million parameter model"}, {"start": 371.0, "text": "that performs prompt-based image segmentation."}, {"start": 374.0, "text": "Before I can optimize the model,"}, {"start": 376.0, "text": "it's key to understand its internals at a high level."}, {"start": 380.0, "text": "SAM3 has three main pieces."}, {"start": 383.0, "text": "An Image encoder that processes the image."}, {"start": 386.0, "text": "A Text encoder that handles the user's prompt."}, {"start": 389.0, "text": "These two components combined make up 96% of the model's parameters"}, {"start": 394.0, "text": "so getting these right is key."}, {"start": 397.0, "text": "And to complete the picture,"}, {"start": 398.0, "text": "a Detector module wrapping a DEtection TRansformer,"}, {"start": 401.0, "text": "combined with a mask decoder, produces the final output —"}, {"start": 405.0, "text": "the segmentation mask."}, {"start": 408.0, "text": "As you can see, SAM3 performs a complex, end-to-end task."}, {"start": 412.0, "text": "And this is exactly the kind of use-case"}, {"start": 414.0, "text": "developers increasingly want to execute on-device."}, {"start": 419.0, "text": "To optimize this, I'll leverage Core AI's optimization library — called coreai-opt."}, {"start": 425.0, "text": "Coreai-opt enables config-driven model compression,"}, {"start": 429.0, "text": "you describe what to compress and what to leave alone."}, {"start": 432.0, "text": "It supports various optimization schemes, from which you can choose one"}, {"start": 437.0, "text": "to optimize differently for macOS versus iOS, as an example."}, {"start": 442.0, "text": "It also supports int4, int8, FP4"}, {"start": 446.0, "text": "and FP8 weight compression with flexible granularity."}, {"start": 450.0, "text": "And finally, coreai-opt includes quantization APIs"}, {"start": 454.0, "text": "that you can either use with small amount of calibration data,"}, {"start": 458.0, "text": "or perform quantization aware training on larger data sets."}, {"start": 463.0, "text": "This is the simple pipeline I had previously."}, {"start": 466.0, "text": "Now I am adding a step."}, {"start": 468.0, "text": "Before conversion, I run the model through coreai-opt with a compression config"}, {"start": 473.0, "text": "or I can use one of their convenient presets."}, {"start": 475.0, "text": "This gives me a smaller model that still goes through the same export pipeline."}, {"start": 480.0, "text": "Let's try this on SAM3 and see what happens."}, {"start": 484.0, "text": "I start by wrapping SAM3 for export."}, {"start": 487.0, "text": "This wrapper defines the interface for torch export to capture"}, {"start": 491.0, "text": "the full computational graph of the model."}, {"start": 496.0, "text": "And here's the conversion pipeline from the slides,"}, {"start": 498.0, "text": "wrapped into a reusable helper."}, {"start": 501.0, "text": "A couple of interesting points though."}, {"start": 505.0, "text": "First, it runs decompositions in the PyTorch exported_program"}, {"start": 509.0, "text": "with Core AI's custom table."}, {"start": 511.0, "text": "This ensures that high-level semantics that Core AI supports, like attention,"}, {"start": 515.0, "text": "are preserved in the graph."}, {"start": 520.0, "text": "Second, it also supports casting the program"}, {"start": 522.0, "text": "to 16-bit floating point using coreai-opt's helper, if needed."}, {"start": 529.0, "text": "The full conversion takes a few minutes, so I have pre-computed the baseline asset."}, {"start": 537.0, "text": "What I do here is load the baseline 32-bit converted model and run it."}, {"start": 542.0, "text": "As you can see, it's over 3 gigs in size."}, {"start": 545.0, "text": "When I run, the default specialization kicks in to specialize and run the model."}, {"start": 552.0, "text": "This is my baseline."}, {"start": 554.0, "text": "In this image,"}, {"start": 555.0, "text": "I ask for a segmentation mask over all the flowers."}, {"start": 558.0, "text": "All are successfully detected based on the default threshold,"}, {"start": 561.0, "text": "running on-device."}, {"start": 563.0, "text": "This is what I need to preserve after compression."}, {"start": 567.0, "text": "Now let's look at compression."}, {"start": 569.0, "text": "coreai-opt ships with preset configurations."}, {"start": 573.0, "text": "presets.w4 gives me 4-bit per-channel, symmetric quantization in one line."}, {"start": 580.0, "text": "I set ExecutionMode to EAGER, which works great for weight compression."}, {"start": 585.0, "text": "For activations, I would use the GRAPH mode."}, {"start": 589.0, "text": "Then I initialize Coreai-opt's Quantizer with the config,"}, {"start": 593.0, "text": "pass example inputs and finalize —"}, {"start": 596.0, "text": "the model is then compressed."}, {"start": 602.0, "text": "As before, I load the model and run it on-device."}, {"start": 608.0, "text": "The model is now around 430 megabytes."}, {"start": 613.0, "text": "Look at the result."}, {"start": 614.0, "text": "One of the occluded flowers is no longer detected."}, {"start": 619.0, "text": "I applied the same aggressive compression to every single layer,"}, {"start": 623.0, "text": "and its likely that not every layer handles this equally well."}, {"start": 627.0, "text": "The question is — which layers are causing this?"}, {"start": 630.0, "text": "This is the kind of problem that's hard to diagnose from the output alone."}, {"start": 634.0, "text": "I need to see inside the model."}, {"start": 637.0, "text": "Let me hand it over to Nicole to show you how."}, {"start": 640.0, "text": "Thanks, Sachin!"}, {"start": 641.0, "text": "I'm excited to talk to you about Core AI Debugger."}, {"start": 645.0, "text": "Now we've seen how to create and optimize your Core AI model."}, {"start": 650.0, "text": "But, if you need a deeper understanding of your model and its behaviour"}, {"start": 655.0, "text": "you can use Core AI Debugger."}, {"start": 658.0, "text": "Core AI Debugger is a new standalone application"}, {"start": 662.0, "text": "that can help you inspect your models on Apple platforms."}, {"start": 666.0, "text": "With the debugger you can visualize your model's structure"}, {"start": 670.0, "text": "in an easy-to-understand graph format,"}, {"start": 673.0, "text": "execute your model on specific hardware for true runtime results,"}, {"start": 678.0, "text": "and validate inference correctness against a reference run — all in one place."}, {"start": 683.0, "text": "I'm excited to show you Core AI Debugger in action"}, {"start": 686.0, "text": "and figure out what happened when the SAM3 model was quantized."}, {"start": 691.0, "text": "I'll start by opening the original model"}, {"start": 695.0, "text": "and click Inspect to get started."}, {"start": 702.0, "text": "Now that the model is open, I can see the debugger workspace."}, {"start": 707.0, "text": "On the left is the navigator"}, {"start": 709.0, "text": "which contains a structured list of operations in the model."}, {"start": 713.0, "text": "These operations are grouped by their PyTorch module,"}, {"start": 716.0, "text": "which is especially powerful for larger models like SAM3"}, {"start": 721.0, "text": "and allows you to navigate your model in a way that feels familiar."}, {"start": 726.0, "text": "Selecting a PyTorch module in the navigator,"}, {"start": 728.0, "text": "like the detector decoder,"}, {"start": 730.0, "text": "will highlight all of the corresponding nodes"}, {"start": 733.0, "text": "in the structure viewer at the top of the workspace."}, {"start": 736.0, "text": "This view shows you a graphical representation of your model"}, {"start": 740.0, "text": "and provides a clear picture of operation connectivity,"}, {"start": 744.0, "text": "execution order, and data dependencies."}, {"start": 747.0, "text": "And, with the source viewer at the bottom,"}, {"start": 750.0, "text": "I'm always grounded in my model's original Python code"}, {"start": 754.0, "text": "down to the specific line."}, {"start": 757.0, "text": "Finally, I can learn even more about an operation by selecting it"}, {"start": 767.0, "text": "and opening the inspector on the right."}, {"start": 770.0, "text": "Here, I can find a description,"}, {"start": 772.0, "text": "and additional details on the operation's inputs and outputs."}, {"start": 779.0, "text": "Together, these views allow you to move fluidly"}, {"start": 782.0, "text": "between graph structure, source code,"}, {"start": 784.0, "text": "and execution details, which dramatically reduces"}, {"start": 787.0, "text": "the cognitive overhead of debugging complex models like SAM3."}, {"start": 792.0, "text": "Beyond static analysis,"}, {"start": 794.0, "text": "the debugger enables runtime analysis"}, {"start": 796.0, "text": "of how your model actually executes on-device."}, {"start": 800.0, "text": "This will be especially helpful for tracking down"}, {"start": 803.0, "text": "where quantization has caused a problem."}, {"start": 806.0, "text": "To run the model, I'll click device at the top of the workspace."}, {"start": 811.0, "text": "In the scheme settings, I'll pick my Mac from the list of targets,"}, {"start": 816.0, "text": "then specify the inputs I want to provide to the model."}, {"start": 820.0, "text": "Starting with the pixel values,"}, {"start": 823.0, "text": "then the input_IDs,"}, {"start": 829.0, "text": "and the attention_mask."}, {"start": 836.0, "text": "Finally, I'll click Run."}, {"start": 840.0, "text": "SAM3 is now being specialized to run on my device."}, {"start": 845.0, "text": "Now that it's ready,"}, {"start": 846.0, "text": "the structure viewer has updated to show me the model,"}, {"start": 849.0, "text": "exactly as it would run on my Mac."}, {"start": 852.0, "text": "And I can now click on any operation"}, {"start": 855.0, "text": "to see its output tensor directly in the inspector."}, {"start": 858.0, "text": "Without needing to modify anything."}, {"start": 862.0, "text": "Back to the problem at hand,"}, {"start": 864.0, "text": "I first want to verify the final detection masks."}, {"start": 868.0, "text": "So I'll scroll to the end of the model"}, {"start": 874.0, "text": "and select the final operation."}, {"start": 878.0, "text": "In the inspector, I'll click on the tensor preview to get a closer look at the mask."}, {"start": 883.0, "text": "I can see the flowers,"}, {"start": 884.0, "text": "but just like in the notebook, one is missing."}, {"start": 890.0, "text": "Now I want to understand how these results compare with the original PyTorch run."}, {"start": 895.0, "text": "I'll return to my notebook and use the NEW save intermediates API."}, {"start": 900.0, "text": "This API executes a PyTorch model"}, {"start": 902.0, "text": "and captures intermediate tensor values at each operation."}, {"start": 907.0, "text": "I want to compare my quantized results with the baseline Sachin showed earlier,"}, {"start": 912.0, "text": "so I'll pass in the int4 model"}, {"start": 915.0, "text": "alongside the original SAM3."}, {"start": 922.0, "text": "I'll let it run"}, {"start": 925.0, "text": "and now that the intermediates are saved,"}, {"start": 927.0, "text": "I'll return to the debugger to compare the results."}, {"start": 931.0, "text": "I'll start by clicking the comparison icon at the top of the workspace"}, {"start": 935.0, "text": "to initialize a new comparison session."}, {"start": 937.0, "text": "On the left is the existing configuration I specified earlier."}, {"start": 942.0, "text": "On the right, I can choose another configuration to compare against"}, {"start": 945.0, "text": "like a different Target or Compute Unit."}, {"start": 949.0, "text": "In this case, I'll click Target"}, {"start": 951.0, "text": "and load a reference run from an Intermediates File."}, {"start": 958.0, "text": "I'll use the file I just exported"}, {"start": 961.0, "text": "and start the comparison."}, {"start": 966.0, "text": "The navigator is now populated with operation pairs"}, {"start": 969.0, "text": "which combine an operation from the specialized model"}, {"start": 972.0, "text": "and PyTorch model."}, {"start": 979.0, "text": "These pairs are called sync points,"}, {"start": 981.0, "text": "places where the specialized model's output"}, {"start": 984.0, "text": "is expected to match the original PyTorch result."}, {"start": 987.0, "text": "The debugger automatically identifies these points throughout the model"}, {"start": 991.0, "text": "to make the comparison process easy."}, {"start": 995.0, "text": "Each sync point is paired with a metric indicating how similar the two outputs are"}, {"start": 1000.0, "text": "which makes it trivial to find where they diverge."}, {"start": 1003.0, "text": "The default metric is a peak signal-to-noise ratio or PSNR,"}, {"start": 1008.0, "text": "but this can be changed"}, {"start": 1009.0, "text": "to witchever similarity indicator suits your model best."}, {"start": 1013.0, "text": "For SAM3, I'll stick to PSNR."}, {"start": 1017.0, "text": "The value of the similarity metric"}, {"start": 1019.0, "text": "can also be quickly gleaned from the status indicator on the right"}, {"start": 1022.0, "text": "or from the graph itself:"}, {"start": 1025.0, "text": "green nodes indicate similar tensors,"}, {"start": 1027.0, "text": "red nodes would indicate significant differences."}, {"start": 1032.0, "text": "As I scroll through the operations,"}, {"start": 1034.0, "text": "I'm seeing several yellow sync points,"}, {"start": 1037.0, "text": "which indicates that parts of my model have moderately diverged"}, {"start": 1040.0, "text": "from the expected result."}, {"start": 1043.0, "text": "I'll sort by similarity,"}, {"start": 1044.0, "text": "and investigate the most dissimilar sync points."}, {"start": 1049.0, "text": "When I click on a sync point in the navigator,"}, {"start": 1051.0, "text": "the source viewer updates to show me the operation's PyTorch module hierarchy."}, {"start": 1057.0, "text": "For example, this operation came from the detector decoder."}, {"start": 1062.0, "text": "I'll use the up arrow key"}, {"start": 1063.0, "text": "to navigate through the low-PSNR sync points one-by-one"}, {"start": 1067.0, "text": "to see if a pattern emerges."}, {"start": 1076.0, "text": "I'm noticing that the vast majority of low-PSNR sync points"}, {"start": 1080.0, "text": "are actually coming from the detector decoder."}, {"start": 1084.0, "text": "This tells me that the quantization scheme applied earlier"}, {"start": 1087.0, "text": "has mildly corrupted the detector results."}, {"start": 1090.0, "text": "Since we previously identified that the detector block"}, {"start": 1093.0, "text": "only accounts for 4% of model parameters,"}, {"start": 1096.0, "text": "we're not getting much benefit from compressing it anyway."}, {"start": 1100.0, "text": "So, I'll return to the Jupyter notebook,"}, {"start": 1103.0, "text": "and try changing the quantization scheme to ignore the detector."}, {"start": 1111.0, "text": "Now that the new scheme has been applied,"}, {"start": 1113.0, "text": "I'll re-export the model"}, {"start": 1123.0, "text": "and verify if the change worked."}, {"start": 1126.0, "text": "Great!"}, {"start": 1127.0, "text": "I can see that we have once again reached baseline quality"}, {"start": 1130.0, "text": "where all flowers are detected"}, {"start": 1132.0, "text": "and the model is only a fraction of the size!"}, {"start": 1137.0, "text": "Core AI Debugger turned hours of manual tensor comparison into a visual diagnosis."}, {"start": 1143.0, "text": "I started with missing detections"}, {"start": 1145.0, "text": "and reached a revised quantization scheme in minutes."}, {"start": 1150.0, "text": "Beyond what I showcased today,"}, {"start": 1151.0, "text": "Core AI Debugger is capable of solving increasingly complex issues."}, {"start": 1156.0, "text": "It gives you deep visibility into how your models behave,"}, {"start": 1160.0, "text": "enabling greater confidence when bringing your model to Apple platforms."}, {"start": 1164.0, "text": "Now, back to Sachin."}, {"start": 1167.0, "text": "Thanks Nicole!"}, {"start": 1168.0, "text": "Now let's take things a step further."}, {"start": 1171.0, "text": "So far, I have been converting the model as a single, end-to-end unit."}, {"start": 1175.0, "text": "And for a lot of models, that works just fine."}, {"start": 1178.0, "text": "But it may not always be enough,"}, {"start": 1180.0, "text": "depending on your use-case"}, {"start": 1181.0, "text": "and especially your constraints."}, {"start": 1183.0, "text": "And this is where Core AI really empowers you to dig deeper."}, {"start": 1188.0, "text": "Concretely, I will now be zooming in to the PyTorch source itself"}, {"start": 1192.0, "text": "which defines a graph of computations from inputs to outputs."}, {"start": 1197.0, "text": "What advanced model authoring implies,"}, {"start": 1199.0, "text": "is that you look inside this computational graph"}, {"start": 1201.0, "text": "and really tune how it runs on the hardware."}, {"start": 1205.0, "text": "As a simple example, let's consider this series of operations."}, {"start": 1209.0, "text": "You can take a group of those ops"}, {"start": 1211.0, "text": "and fuse them into a single operation"}, {"start": 1213.0, "text": "This replaces several steps"}, {"start": 1215.0, "text": "with a single kernel dispatch within the graph."}, {"start": 1219.0, "text": "Core AI already ships with pre-packaged fast kernels"}, {"start": 1222.0, "text": "and primitives for heavy operations like Scaled Dot Product Attention,"}, {"start": 1226.0, "text": "commonly found in Transformers."}, {"start": 1228.0, "text": "You can find examples of how to leverage these operations"}, {"start": 1232.0, "text": "in the coreai-models repository."}, {"start": 1234.0, "text": "But if you live on the cutting edge"}, {"start": 1236.0, "text": "and want even more customization, we also have support for custom Metal 4 kernels."}, {"start": 1243.0, "text": "Coming back to my pipeline."}, {"start": 1245.0, "text": "Here's what changes with custom Metal kernels."}, {"start": 1248.0, "text": "I am adding a second input to coreai-torch."}, {"start": 1252.0, "text": "My kernel's source code written in the Metal Shading Language, or MSL."}, {"start": 1257.0, "text": "The converter takes both my PyTorch model and my custom kernel,"}, {"start": 1261.0, "text": "and bundles them together into a single asset."}, {"start": 1264.0, "text": "The MSL is embedded right inside."}, {"start": 1267.0, "text": "It ships with the model."}, {"start": 1269.0, "text": "Let me show you what that looks like in code."}, {"start": 1272.0, "text": "First, I define a PyTorch reference for our example."}, {"start": 1276.0, "text": "A standard Sigmoid Linear Unit, or SiLU."}, {"start": 1279.0, "text": "It's a common activation function used in generative transformer models."}, {"start": 1283.0, "text": "This is what torch.export sees during tracing."}, {"start": 1288.0, "text": "Below that, I implement the actual Metal kernel in MSL."}, {"start": 1292.0, "text": "This is a simple element-wise kernel,"}, {"start": 1294.0, "text": "one thread per element, that computes the fused activation directly on the GPU."}, {"start": 1300.0, "text": "With just these two pieces, I can now register"}, {"start": 1303.0, "text": "a Core AI TorchMetalKernel,"}, {"start": 1305.0, "text": "give it the Metal source,"}, {"start": 1307.0, "text": "the PyTorch reference"}, {"start": 1308.0, "text": "and the input and output names."}, {"start": 1310.0, "text": "In this case, the input and output names are \"x\" and \"y\" respectively,"}, {"start": 1314.0, "text": "and you can see those names being used in the MSL kernel above."}, {"start": 1319.0, "text": "So you write the Metal."}, {"start": 1320.0, "text": "You write the PyTorch reference."}, {"start": 1322.0, "text": "And Core AI binds them together."}, {"start": 1325.0, "text": "Using it in a model, I will just call it like any other Python function."}, {"start": 1329.0, "text": "Pass the input, specify the thread grid and I am done."}, {"start": 1334.0, "text": "One thing to note,"}, {"start": 1335.0, "text": "is that I pass in the result shapes to every instantiation"}, {"start": 1339.0, "text": "of the custom kernel in the PyTorch source."}, {"start": 1342.0, "text": "This allows Core AI to bake in the computation"}, {"start": 1345.0, "text": "of the output shapes of the kernel from the input shapes,"}, {"start": 1348.0, "text": "if your model has dynamic shaped inputs."}, {"start": 1351.0, "text": "When I convert with TorchConverter,"}, {"start": 1353.0, "text": "I register my custom kernels with the converter,"}, {"start": 1356.0, "text": "then add the exported program as before."}, {"start": 1359.0, "text": "The Metal source gets embedded directly in the asset"}, {"start": 1363.0, "text": "a single artifact."}, {"start": 1364.0, "text": "The kernel travels with the model."}, {"start": 1368.0, "text": "For more details on how you can write efficient Metal kernels for Core AI,"}, {"start": 1372.0, "text": "and to see an optimized kernel live in action with the SAM3 model please see"}, {"start": 1376.0, "text": "the \"Optimize custom machine learning operations with Metal tensors\" talk."}, {"start": 1381.0, "text": "So far, I showed how you can take multiple operations in the graph"}, {"start": 1385.0, "text": "and fuse them into one."}, {"start": 1387.0, "text": "But for more advanced optimizations,"}, {"start": 1389.0, "text": "especially for iOS,"}, {"start": 1390.0, "text": "you need to go further and rewrite"}, {"start": 1392.0, "text": "the entire model with a specific target in mind."}, {"start": 1396.0, "text": "We refer to this process, as model reauthoring."}, {"start": 1399.0, "text": "Back to our simple series of operations."}, {"start": 1401.0, "text": "Re-authoring typically involves replacing many aspects of this computational graph."}, {"start": 1406.0, "text": "This may imply using different operations,"}, {"start": 1409.0, "text": "novel tensor layouts,"}, {"start": 1410.0, "text": "and even modifying the interfaces of the model."}, {"start": 1413.0, "text": "Essentially, this is a completely different implementation"}, {"start": 1416.0, "text": "of the source code."}, {"start": 1419.0, "text": "Digging deeper, what does this kind of authoring involve?"}, {"start": 1422.0, "text": "One example, is using predefined patterns in the PyTorch code"}, {"start": 1426.0, "text": "that tell Core AI about a specific concept."}, {"start": 1429.0, "text": "This allows the framework to map these semantics"}, {"start": 1432.0, "text": "to an optimized implementation at runtime."}, {"start": 1435.0, "text": "An example of this, is in-place updates of the Key-Value cache"}, {"start": 1438.0, "text": "commonly used in Large Language models."}, {"start": 1441.0, "text": "Another mechanism used,"}, {"start": 1442.0, "text": "especially when targeting iOS, is the usage of static tensor shapes,"}, {"start": 1446.0, "text": "channels-first tensor layouts and convolutional op patterns."}, {"start": 1450.0, "text": "These enable Core AI to leverage powerful underlying primitives"}, {"start": 1454.0, "text": "and meet your on-device constraints."}, {"start": 1457.0, "text": "When you engineer a novel PyTorch implementation in this way,"}, {"start": 1460.0, "text": "its crucial that you employ rigorous testing"}, {"start": 1462.0, "text": "both at the module level and at the model level."}, {"start": 1466.0, "text": "This ensures that individual building blocks,"}, {"start": 1468.0, "text": "as well as the entire model work as intended."}, {"start": 1472.0, "text": "This testing can take the shape of unit tests or integration tests."}, {"start": 1476.0, "text": "To get you started, the Core AI models repository includes multiple examples"}, {"start": 1480.0, "text": "of such reusable components"}, {"start": 1482.0, "text": "and best practices across different models."}, {"start": 1486.0, "text": "Core AI skills also enable your coding assistant"}, {"start": 1489.0, "text": "to write PyTorch code optimized for Apple Silicon from day one."}, {"start": 1494.0, "text": "Let's continue with SAM3."}, {"start": 1496.0, "text": "Instead of converting the model as-is,"}, {"start": 1498.0, "text": "I can author a new PyTorch implementation that's hand-crafted for my goals."}, {"start": 1502.0, "text": "The biggest change I make is to have three separate functions"}, {"start": 1506.0, "text": "in the Core AI Model instead of one."}, {"start": 1508.0, "text": "Coreai-torch has APIs that lets you do this."}, {"start": 1512.0, "text": "Image Encode handles the image,"}, {"start": 1514.0, "text": "Text Encode processes the prompt,"}, {"start": 1516.0, "text": "and Detect wraps the final post-processing"}, {"start": 1518.0, "text": "to generate the output."}, {"start": 1520.0, "text": "Splitting the work this way allows me to run each bit"}, {"start": 1523.0, "text": "at a different cadence."}, {"start": 1525.0, "text": "For example, I may want to process a single prompt once"}, {"start": 1529.0, "text": "and use it across a variety of images in your application."}, {"start": 1532.0, "text": "It also gives each function a clean interface,"}, {"start": 1535.0, "text": "and lets me compress and author each one independently."}, {"start": 1539.0, "text": "Lets see this in practice."}, {"start": 1541.0, "text": "Here's the attention block from the Image Encoder transformer,"}, {"start": 1544.0, "text": "rewritten for power-efficient execution on iOS."}, {"start": 1550.0, "text": "Instead of standard Linear layers,"}, {"start": 1552.0, "text": "I use convolutional projections."}, {"start": 1554.0, "text": "This is one of the patterns that lets Core AI leverage native hardware primitives"}, {"start": 1558.0, "text": "on the right compute unit."}, {"start": 1560.0, "text": "The text encoder gets a similar treatment."}, {"start": 1563.0, "text": "The smaller decoder stays mostly unchanged."}, {"start": 1566.0, "text": "It's a small fraction of the compute,"}, {"start": 1568.0, "text": "so the payoff from re-authoring it is minimal."}, {"start": 1575.0, "text": "I structure the re-authored model as three independent modules."}, {"start": 1579.0, "text": "ImageEncoder,"}, {"start": 1582.0, "text": "TextEncoder"}, {"start": 1584.0, "text": "and the Detector."}, {"start": 1585.0, "text": "As mentioned earlier,"}, {"start": 1586.0, "text": "this separation lets me use different aspects of the model uniquely."}, {"start": 1596.0, "text": "For compression, I apply 4-bit palettization"}, {"start": 1599.0, "text": "with per-channel scales to the two encoders."}, {"start": 1603.0, "text": "There is a preset available for this,"}, {"start": 1604.0, "text": "but I use the lower-level representation here to showcase the APIs."}, {"start": 1609.0, "text": "This lookup-table-based compression,"}, {"start": 1611.0, "text": "is well-suited for power efficiency on iOS."}, {"start": 1618.0, "text": "As before, I construct a KMeansPalettizer"}, {"start": 1622.0, "text": "similar to the Quantizer,"}, {"start": 1623.0, "text": "and pass it the model and config."}, {"start": 1627.0, "text": "Then, I prepare and finalize."}, {"start": 1629.0, "text": "Also note, that I changed the input image size"}, {"start": 1632.0, "text": "from 1008 pixels to 336 to run on an iPhone."}, {"start": 1640.0, "text": "The detector stays uncompressed."}, {"start": 1642.0, "text": "I know that its sensitive to compression from our previous exercise."}, {"start": 1649.0, "text": "I then run each model through torch export."}, {"start": 1652.0, "text": "All of them get cast to half-precision."}, {"start": 1658.0, "text": "And here's where it comes together."}, {"start": 1660.0, "text": "A single TorchConverter, three exported programs,"}, {"start": 1664.0, "text": "each with its own entrypoint name."}, {"start": 1668.0, "text": "First, image_encode."}, {"start": 1670.0, "text": "Then, text_encode."}, {"start": 1672.0, "text": "And finally, detect."}, {"start": 1677.0, "text": "When saved, I get one model asset"}, {"start": 1681.0, "text": "with three callable functions inside."}, {"start": 1689.0, "text": "Now, lets load and run the pre-computed asset."}, {"start": 1698.0, "text": "First, I see all the flowers segmented as expected."}, {"start": 1704.0, "text": "And here's the payoff of the three-function split."}, {"start": 1708.0, "text": "I swapped the prompt to butterfly"}, {"start": 1710.0, "text": "and only re-ran the text encoder and the detector."}, {"start": 1717.0, "text": "As a result, the second inference is 76% faster, even after warmup."}, {"start": 1723.0, "text": "This shows the benefit of re-authoring."}, {"start": 1726.0, "text": "So, here's what you can do today."}, {"start": 1729.0, "text": "Convert your PyTorch models using Core AI's Python libraries."}, {"start": 1733.0, "text": "Optimize them with coreai-opt,"}, {"start": 1735.0, "text": "and use the debugger when you need to understand what's happening inside."}, {"start": 1739.0, "text": "Build on top of the examples in coreai-models."}, {"start": 1743.0, "text": "And plug in Core AI Skills into your favorite AI agent"}, {"start": 1746.0, "text": "to leverage the new framework like an expert."}, {"start": 1749.0, "text": "I look forward to seeing what models you bring to the platform!"}, {"start": 1752.0, "text": "Thank you!"}]}