{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-328", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/328", "wordCount": 2168, "segments": [{"start": 7.0, "text": "Hi! I'm David Koski and I work on MLX Swift."}, {"start": 11.0, "text": "Numerical computing, also called numerical analysis"}, {"start": 14.0, "text": "or scientific computing,"}, {"start": 16.0, "text": "is a set of techniques and algorithms used to solve mathematical problems."}, {"start": 20.0, "text": "These are often problems that are impractical"}, {"start": 22.0, "text": "to solve symbolically or by hand."}, {"start": 24.0, "text": "They need massive amounts of computation."}, {"start": 27.0, "text": "Some applications include simulations in chemistry,"}, {"start": 30.0, "text": "biology, physics and financial systems."}, {"start": 34.0, "text": "Other domains include audio and signal processing."}, {"start": 37.0, "text": "Visual applications include rendering, ray tracing, and fractals."}, {"start": 41.0, "text": "Large scale gradient descent can do arbitrary curve fitting."}, {"start": 44.0, "text": "This is the basis of training machine learning models."}, {"start": 48.0, "text": "Today we are going to discuss numerical computing using MLX Swift."}, {"start": 53.0, "text": "Apple platforms have a rich numerical-computing ecosystem."}, {"start": 57.0, "text": "Each existing framework is great at what it's designed for."}, {"start": 61.0, "text": "Accelerate gives you hand-tuned vector primitives on the CPU."}, {"start": 65.0, "text": "BNNS is the building block layer for neural networks."}, {"start": 68.0, "text": "Metal Performance Shaders give you direct access to GPU kernels."}, {"start": 73.0, "text": "Swift Numerics adds a Complex type and generic numeric protocols."}, {"start": 77.0, "text": "So when do you use MLX Swift?"}, {"start": 80.0, "text": "If your primary goal is writing mathematical code"}, {"start": 83.0, "text": "with an eye for performance,"}, {"start": 85.0, "text": "MLX Swift is a great solution."}, {"start": 87.0, "text": "The code you write looks like the math you are implementing"}, {"start": 90.0, "text": "without the programming overhead of some of the lower level libraries"}, {"start": 94.0, "text": "or the detailed bookkeeping required when manipulating arrays in plain Swift."}, {"start": 99.0, "text": "How does it look like math?"}, {"start": 101.0, "text": "The core idea is simple."}, {"start": 103.0, "text": "Mathematicians and numerical analysts work with vectors and matrices."}, {"start": 108.0, "text": "Rather than performing operations on single values"}, {"start": 111.0, "text": "you are doing it for the entire matrix at once."}, {"start": 113.0, "text": "MLX Swift uses n-dimensional arrays as the central abstraction,"}, {"start": 118.0, "text": "like NumPy and many others before it."}, {"start": 120.0, "text": "In fact if you've used NumPy, the API will look very familiar."}, {"start": 125.0, "text": "Most NumPy code can be translated to MLX Swift with minimal changes."}, {"start": 130.0, "text": "This is incredibly expressive without being complicated."}, {"start": 133.0, "text": "You can understand the code when you write it and read it later."}, {"start": 137.0, "text": "Array computing and lazy evaluation are what make two things possible:"}, {"start": 142.0, "text": "automatic GPU execution and automatic differentiation."}, {"start": 147.0, "text": "Best of all, mlx-swift and the entire MLX ecosystem"}, {"start": 151.0, "text": "are all open source with an MIT license."}, {"start": 154.0, "text": "We welcome issues and PRs and have a very active community,"}, {"start": 158.0, "text": "ask questions, fix bugs, and make it better!"}, {"start": 161.0, "text": "Here's the plan."}, {"start": 162.0, "text": "I will first introduce MLX Swift with some basic matrix-vector operations."}, {"start": 167.0, "text": "Then I will dive into three examples showing"}, {"start": 170.0, "text": "how MLX Swift makes it easy to translate math into code:"}, {"start": 173.0, "text": "computation of the Mandelbrot set,"}, {"start": 176.0, "text": "finding the steady state for heat distribution,"}, {"start": 179.0, "text": "and finally curve fitting."}, {"start": 181.0, "text": "First up, let's talk about MLX Swift."}, {"start": 185.0, "text": "Here we show some MLX Swift operations,"}, {"start": 188.0, "text": "using the power iteration as an example."}, {"start": 190.0, "text": "Let's walk through it."}, {"start": 192.0, "text": "We import MLX and set the matrix size and iteration count,"}, {"start": 196.0, "text": "then sample a random matrix and vector from a normal distribution."}, {"start": 200.0, "text": "Next we build a symmetric matrix by adding B and its transpose."}, {"start": 205.0, "text": "Notice how close this is to the math."}, {"start": 208.0, "text": ".T gives the transpose, and plus does matrix addition."}, {"start": 212.0, "text": "Inside the loop, matmul does matrix-vector multiplication,"}, {"start": 216.0, "text": "and norm gives the L2 norm."}, {"start": 219.0, "text": "Again, the code reads like the math."}, {"start": 221.0, "text": "Here we also see a key MLX feature: lazy evaluation."}, {"start": 225.0, "text": "Operations on MLX array objects build a compute graph,"}, {"start": 229.0, "text": "and nothing runs until you call ee-val or read a value."}, {"start": 233.0, "text": "In a loop like this, we call ee-val each step so the graph stays small."}, {"start": 238.0, "text": "Lazy evaluation is also what powers MLX's function transformations,"}, {"start": 243.0, "text": "like grad for automatic differentiation."}, {"start": 247.0, "text": "Finally, we recover the eigenvalue,"}, {"start": 249.0, "text": "and reading the value forces the computation."}, {"start": 252.0, "text": "Like other array frameworks,"}, {"start": 254.0, "text": "MLX Swift code reads almost like the math."}, {"start": 257.0, "text": "And if you actually need all the eigenvalues"}, {"start": 260.0, "text": "and eigenvectors of a matrix,"}, {"start": 261.0, "text": "the MLX Swift linear algebra package has functions for that too."}, {"start": 266.0, "text": "Let's move on to the next example."}, {"start": 268.0, "text": "Next, the Mandelbrot set."}, {"start": 270.0, "text": "It's a classic fractal,"}, {"start": 272.0, "text": "and it's also a perfect showcase for array computing"}, {"start": 275.0, "text": "where we apply a function over a large grid of points."}, {"start": 278.0, "text": "The definition is surprisingly simple."}, {"start": 281.0, "text": "For every point c in the complex plane,"}, {"start": 284.0, "text": "you iterate z = z² + c."}, {"start": 287.0, "text": "If the magnitude never exceeds 2,"}, {"start": 290.0, "text": "the point is in the set and it's colored black."}, {"start": 293.0, "text": "If it diverges, it's colored by how quickly it escapes."}, {"start": 297.0, "text": "The beauty of a fractal is that it's self-similar and infinitely detailed,"}, {"start": 301.0, "text": "you can pan and zoom forever and the patterns never repeat."}, {"start": 305.0, "text": "Let's start with a plain Swift implementation using scalars."}, {"start": 309.0, "text": "You loop over every pixel,"}, {"start": 311.0, "text": "and run the Mandelbrot iterations and check for divergence."}, {"start": 315.0, "text": "It works."}, {"start": 316.0, "text": "It's idiomatic Swift."}, {"start": 317.0, "text": "But you're managing a lot of bookkeeping that has nothing to do with the problem."}, {"start": 321.0, "text": "And it runs on the CPU, one point at a time."}, {"start": 324.0, "text": "Let's look at MLX Swift."}, {"start": 327.0, "text": "Here it is in MLX Swift."}, {"start": 329.0, "text": "Set up a grid of complex numbers, that's c."}, {"start": 333.0, "text": "Then the loop is just two lines."}, {"start": 336.0, "text": "z = z * z + c"}, {"start": 338.0, "text": "applied to every point at once."}, {"start": 340.0, "text": "Count how many iterations each point stays bounded."}, {"start": 344.0, "text": "That's it."}, {"start": 345.0, "text": "The code is a direct translation of the math."}, {"start": 347.0, "text": "The computation is performed over an entire grid of points"}, {"start": 350.0, "text": "as easily as it is for a single point."}, {"start": 353.0, "text": "By default, the GPU is used, giving us fast performance."}, {"start": 358.0, "text": "Plain Swift is expressive."}, {"start": 360.0, "text": "You can write numerical computing code naturally."}, {"start": 363.0, "text": "But you're working scalar-at-a-time,"}, {"start": 365.0, "text": "so you have to iterate over every point yourself."}, {"start": 368.0, "text": "The bookkeeping can obscure the math."}, {"start": 372.0, "text": "MLX Swift is built for numerical computing,"}, {"start": 375.0, "text": "you operate on arrays rather than scalars."}, {"start": 378.0, "text": "It looks like the math you are trying to express."}, {"start": 380.0, "text": "It runs faster on the GPU, processing all points in parallel."}, {"start": 385.0, "text": "How much faster depends on the exact algorithm,"}, {"start": 387.0, "text": "but 10x faster is certainly possible."}, {"start": 390.0, "text": "All this with smaller and simpler code."}, {"start": 394.0, "text": "Mandelbrot was embarrassingly parallel,"}, {"start": 396.0, "text": "every point independent."}, {"start": 398.0, "text": "The next example is different: each cell talks to its neighbors."}, {"start": 402.0, "text": "That pattern shows up all over physics, image processing, and neural networks."}, {"start": 407.0, "text": "MLX handles it with a single operation: convolution."}, {"start": 411.0, "text": "Imagine a room with walls and heat sources."}, {"start": 414.0, "text": "We want to know the steady-state temperature everywhere inside."}, {"start": 418.0, "text": "The simplest method to solve this is known as the Jacobi iteration."}, {"start": 423.0, "text": "Model the temperature as a 2D grid."}, {"start": 426.0, "text": "Each new iteration averages the neighboring values"}, {"start": 429.0, "text": "with a stencil like this."}, {"start": 432.0, "text": "You repeat this over and over and the heat spreads out"}, {"start": 435.0, "text": "until it reaches a steady state."}, {"start": 437.0, "text": "Notice the update only looks at a small neighborhood,"}, {"start": 440.0, "text": "and it's the same recipe at every point,"}, {"start": 442.0, "text": "that's exactly what a convolution is."}, {"start": 444.0, "text": "Let's see it in code."}, {"start": 447.0, "text": "Here's the core of the solver in MLX Swift."}, {"start": 450.0, "text": "Let me walk through it."}, {"start": 451.0, "text": "The kernel is the stencil from the previous slide, written out literally:"}, {"start": 455.0, "text": "the four quarters on the neighbors, zeros on the center and corners."}, {"start": 459.0, "text": "The temperature grid starts as the heat sources,"}, {"start": 462.0, "text": "a reasonable initial value."}, {"start": 464.0, "text": "Inside the loop, two lines."}, {"start": 466.0, "text": "The first is the physics:"}, {"start": 467.0, "text": "conv2d applies the stencil across the entire grid in one call."}, {"start": 472.0, "text": "The second line handles the boundary conditions:"}, {"start": 474.0, "text": "which is an elementwise ternary."}, {"start": 477.0, "text": "Wherever the mask says this is a heat source or a wall, keep the fixed value;"}, {"start": 481.0, "text": "everywhere else, take the new value from the convolution."}, {"start": 484.0, "text": "That's it."}, {"start": 485.0, "text": "The math said average the four neighbors"}, {"start": 487.0, "text": "and we implemented that as a single call to conv2d."}, {"start": 492.0, "text": "Jacobi iteration is fast to compute but slow to converge."}, {"start": 496.0, "text": "Heat can move one cell at a time"}, {"start": 498.0, "text": "and typically steady state requires N^2 iterations"}, {"start": 501.0, "text": "where N is the side of the grid."}, {"start": 503.0, "text": "Much like quicksort does less work than bubble sort,"}, {"start": 506.0, "text": "there are algorithms that reach steady state with less work."}, {"start": 510.0, "text": "One of these is called Successive Over-Relaxation, or SOR."}, {"start": 515.0, "text": "The equation looks similar to Jacobi iterations."}, {"start": 518.0, "text": "In fact it uses the same convolution kernel."}, {"start": 521.0, "text": "It uses a parameter, omega,"}, {"start": 523.0, "text": "which pushes each update further in the direction of change,"}, {"start": 526.0, "text": "overshooting slightly to get there faster."}, {"start": 529.0, "text": "The overshoot will recover as it iterates."}, {"start": 532.0, "text": "The omega parameter can be computed based on the size of the array"}, {"start": 536.0, "text": "and if the optimal value is used, this will converge in N iterations."}, {"start": 540.0, "text": "The other key to the technique is in-place updates."}, {"start": 543.0, "text": "MLX typically produces new arrays rather than updating in place,"}, {"start": 548.0, "text": "but a red/black checkerboard pattern"}, {"start": 550.0, "text": "where alternating cells are processed can be used to compute new values,"}, {"start": 554.0, "text": "giving the same effect."}, {"start": 556.0, "text": "On to the code!"}, {"start": 558.0, "text": "First, we compute the optimal omega based on the size of the grid."}, {"start": 562.0, "text": "You can see how we use omega here,"}, {"start": 564.0, "text": "it exactly matches the equation."}, {"start": 567.0, "text": "We use the checkerboard masks"}, {"start": 569.0, "text": "to update alternating cells in the array."}, {"start": 572.0, "text": "Now the black cells run the same update,"}, {"start": 574.0, "text": "but this time their red neighbors already have fresh values,"}, {"start": 578.0, "text": "which is exactly the in-place effect we needed."}, {"start": 581.0, "text": "Repeat this in a loop as before."}, {"start": 583.0, "text": "Let's see the difference."}, {"start": 586.0, "text": "The Jacobi and SOR code are nearly identical"}, {"start": 589.0, "text": "and they closely match the math."}, {"start": 591.0, "text": "Let's see how these run."}, {"start": 593.0, "text": "You can see Jacobi on the top"}, {"start": 595.0, "text": "slowly spread while SOR on the bottom quickly fills the area."}, {"start": 599.0, "text": "SOR has a striking ripple pattern as it runs,"}, {"start": 603.0, "text": "that's the overshooting and correcting in real time."}, {"start": 606.0, "text": "By the end, both converge to the same configuration."}, {"start": 610.0, "text": "One more thing."}, {"start": 611.0, "text": "I had to slow SOR down by a factor of 100 just to make it visible."}, {"start": 615.0, "text": "The power of choosing the right algorithm!"}, {"start": 617.0, "text": "The first two examples were forward computation:"}, {"start": 620.0, "text": "start with inputs, compute outputs."}, {"start": 623.0, "text": "The last one flips that around."}, {"start": 624.0, "text": "You have outputs, data points,"}, {"start": 626.0, "text": "and you want to find the parameters that produce them."}, {"start": 629.0, "text": "This is where we will apply a key function transformation"}, {"start": 632.0, "text": "provided by MLX Swift:"}, {"start": 634.0, "text": "'grad' for automatic differentiation."}, {"start": 636.0, "text": "Let's say you have some points"}, {"start": 638.0, "text": "and you want to find a function that approximates them."}, {"start": 641.0, "text": "You decide the structure of the function you want."}, {"start": 643.0, "text": "It could be a polynomial, a sum of sines, or whatever you like."}, {"start": 647.0, "text": "For this example we will use a polynomial,"}, {"start": 650.0, "text": "a quadratic that will give us a parabola."}, {"start": 652.0, "text": "You want to minimize the loss."}, {"start": 655.0, "text": "The mean of the squared differences between the output of your function"}, {"start": 659.0, "text": "and the actual data."}, {"start": 660.0, "text": "This is the same core idea behind training every ML model,"}, {"start": 664.0, "text": "just on a smaller scale."}, {"start": 667.0, "text": "To minimize the loss,"}, {"start": 668.0, "text": "we need the gradient with respect to the parameters"}, {"start": 671.0, "text": "and an optimization loop to update parameters."}, {"start": 674.0, "text": "We define the function f"}, {"start": 676.0, "text": "and the loss,"}, {"start": 677.0, "text": "which is mean squared error."}, {"start": 679.0, "text": "Next we make theta, the coefficients we are going to fit,"}, {"start": 682.0, "text": "and transform the loss function"}, {"start": 684.0, "text": "into a function that produces the exact gradient"}, {"start": 687.0, "text": "with respect to the parameters."}, {"start": 689.0, "text": "We didn't write any derivatives by hand."}, {"start": 691.0, "text": "They were derived by MLX."}, {"start": 694.0, "text": "In the optimization loop"}, {"start": 695.0, "text": "we evaluate the gradient at the current parameters,"}, {"start": 698.0, "text": "take a small step, and call eval to flush the computation graph"}, {"start": 702.0, "text": "each iteration so it doesn't grow without bound."}, {"start": 705.0, "text": "That's gradient descent."}, {"start": 708.0, "text": "Here is what that looks like."}, {"start": 710.0, "text": "The parabola quickly gets close and overshoots the data"}, {"start": 713.0, "text": "but settles in to closer and closer approximations."}, {"start": 716.0, "text": "Now this example was a simple polynomial"}, {"start": 719.0, "text": "and we could have used QR from the linear algebra package to fit the curve directly."}, {"start": 724.0, "text": "Gradients work with arbitrarily complex functions."}, {"start": 727.0, "text": "If you need more than just gradients,"}, {"start": 729.0, "text": "MLX has a suite of optimization algorithms like SGD,"}, {"start": 733.0, "text": "Adam, RMSprop, and more."}, {"start": 737.0, "text": "I've shown array computing, convolution and grad today,"}, {"start": 740.0, "text": "but MLX includes the full numericalcomputing toolkit."}, {"start": 744.0, "text": "Here is a sample."}, {"start": 746.0, "text": "Linear algebra,"}, {"start": 747.0, "text": "FFTs,"}, {"start": 749.0, "text": "N-dimensional convolutions,"}, {"start": 751.0, "text": "Reductions,"}, {"start": 752.0, "text": "Scans,"}, {"start": 753.0, "text": "Indexing,"}, {"start": 754.0, "text": "Random number generation, and many more."}, {"start": 757.0, "text": "There's already a healthy ecosystem of packages built on MLX Swift."}, {"start": 762.0, "text": "The core mlx-swift repo is the framework you've been seeing all session."}, {"start": 766.0, "text": "mlx-swift-lm is where the Swift language-model implementations live."}, {"start": 771.0, "text": "mlx-swift-examples has example programs that you can look at to get started."}, {"start": 776.0, "text": "Examples based on this session will be posted there."}, {"start": 780.0, "text": "All of these are open source and are installable with a few lines"}, {"start": 783.0, "text": "with Swift Package Manager."}, {"start": 785.0, "text": "MLX isn't only Swift."}, {"start": 787.0, "text": "It's one framework with four front-ends: Swift, Python, C++, and C."}, {"start": 793.0, "text": "Third parties have built even more front-ends if you have needs beyond that."}, {"start": 797.0, "text": "They share the same concepts,"}, {"start": 799.0, "text": "the same operations,"}, {"start": 801.0, "text": "and the same lazy-evaluation model."}, {"start": 803.0, "text": "The concepts and patterns transfer across them with minimal changes."}, {"start": 807.0, "text": "So you can prototype in Python and ship in Swift."}, {"start": 811.0, "text": "Python also has a broader research ecosystem around it."}, {"start": 814.0, "text": "Projects like mlx-lm and mlx-vlm are worth a look"}, {"start": 819.0, "text": "if you want to see what's been built on the Python side."}, {"start": 821.0, "text": "I encourage everybody to take a look at mlx-swift and mlx-swift-examples."}, {"start": 827.0, "text": "mlx-swift has documentation"}, {"start": 829.0, "text": "and tests you can explore to see how it works."}, {"start": 832.0, "text": "mlx-swift-examples has a variety of example applications"}, {"start": 835.0, "text": "that demonstrate LLM integration,"}, {"start": 838.0, "text": "stable diffusion,"}, {"start": 839.0, "text": "model training and fine-tuning,"}, {"start": 841.0, "text": "and of course the examples shown in this talk."}, {"start": 844.0, "text": "Do you have numerical computing needs?"}, {"start": 846.0, "text": "Or just want to play around"}, {"start": 847.0, "text": "with interesting simulations and visualizations?"}, {"start": 850.0, "text": "Give it a try!"}, {"start": 851.0, "text": "And if something sparks an idea"}, {"start": 853.0, "text": "I encourage everybody to participate and contribute."}, {"start": 856.0, "text": "There are open issues that you can try fixing"}, {"start": 858.0, "text": "or make a new example program."}, {"start": 861.0, "text": "Thanks for watching!"}]}