{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-359", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/359", "wordCount": 3243, "segments": [{"start": 7.0, "text": "Hi, I'm Yulia, a GPU Software Engineer here at Apple."}, {"start": 11.0, "text": "Today, I'll share how to bring machine learning"}, {"start": 13.0, "text": "to your real-time rendering pipeline with Metal 4."}, {"start": 17.0, "text": "You'll learn practical ways to integrate machine learning into your renderer,"}, {"start": 21.0, "text": "best practices for building high-performance pipelines,"}, {"start": 24.0, "text": "and two techniques to start adopting today."}, {"start": 27.0, "text": "Machine learning is moving from research into production in real-time rendering."}, {"start": 32.0, "text": "Across the rendering pipeline,"}, {"start": 34.0, "text": "many established techniques that have traditionally relied"}, {"start": 37.0, "text": "on analytical methods can also be implemented with machine learning."}, {"start": 41.0, "text": "Neural denoising, neural textures, learned tone mapping"}, {"start": 45.0, "text": "and many others are among the techniques that can leverage machine learning."}, {"start": 49.0, "text": "At every stage of the pipeline, these approaches can improve quality,"}, {"start": 53.0, "text": "performance, or memory footprint."}, {"start": 56.0, "text": "I'll share just how this works in Metal."}, {"start": 59.0, "text": "On Apple platforms, you have a complete machine learning toolset"}, {"start": 63.0, "text": "for your rendering needs."}, {"start": 64.0, "text": "At the highest level, MetalFX provides a ready-to-use neural denoising"}, {"start": 69.0, "text": "and upscaling API"}, {"start": 70.0, "text": "as a fully integrated, black box solution."}, {"start": 74.0, "text": "The Metal 4 ML command encoder"}, {"start": 76.0, "text": "lets you run pre-trained models directly in your command buffer,"}, {"start": 80.0, "text": "giving you more control over integration and scheduling."}, {"start": 84.0, "text": "And at the most flexible level,"}, {"start": 86.0, "text": "the TensorOps API provides the building blocks"}, {"start": 89.0, "text": "to design and run custom models directly in your shaders,"}, {"start": 93.0, "text": "enabling you to fully leverage the neural accelerator"}, {"start": 96.0, "text": "introduced in our M5 and A19 Pro Apple silicon GPUs."}, {"start": 102.0, "text": "Today, I'll talk about all of these in turn."}, {"start": 106.0, "text": "Here is the plan."}, {"start": 107.0, "text": "I'll cover how to adopt and achieve production quality results"}, {"start": 111.0, "text": "in your rendering pipeline with MetalFX,"}, {"start": 113.0, "text": "using Maxon's Redshift Live as an example"}, {"start": 116.0, "text": "of a modern real-time path tracing viewport"}, {"start": 118.0, "text": "that adopting MetalFX Denoising using Apple's best practices."}, {"start": 123.0, "text": "Then, I will describe how you can train"}, {"start": 125.0, "text": "a neural tone mapper and deploy it with Metal 4."}, {"start": 129.0, "text": "Finally, I will explain how to build a small network"}, {"start": 132.0, "text": "directly in a shader using the TensorOps API."}, {"start": 136.0, "text": "It starts with MetalFX."}, {"start": 138.0, "text": "In your path tracer your frame budget might only allow you one"}, {"start": 142.0, "text": "or few samples per pixel to stay interactive."}, {"start": 145.0, "text": "However one sample is naturally noisy."}, {"start": 149.0, "text": "To keep the quality bar, use MetalFX Denoising."}, {"start": 153.0, "text": "It is designed specifically"}, {"start": 154.0, "text": "for the low latency demands of a live viewport."}, {"start": 158.0, "text": "MetalFX Denoising is a combined neural upscaler and denoiser,"}, {"start": 162.0, "text": "the platform-integrated solution, optimized for Apple silicon."}, {"start": 167.0, "text": "You can integrate it easily in your pipeline."}, {"start": 170.0, "text": "You will need to generate a few extra auxiliary inputs like diffuse albedo,"}, {"start": 175.0, "text": "depth, and a few others."}, {"start": 177.0, "text": "Depending on your renderer, you might already have produced those."}, {"start": 181.0, "text": "You feed all these inputs to MetalFX, which produces a beautiful denoised image."}, {"start": 187.0, "text": "From there, you complete your pipeline"}, {"start": 189.0, "text": "with post processing and displaying the output."}, {"start": 193.0, "text": "This is Redshift Live, Maxon's modern real-time path tracer,"}, {"start": 197.0, "text": "rendering one of their high-quality 3D assets in Cinema 4D"}, {"start": 200.0, "text": "on Apple silicon."}, {"start": 202.0, "text": "You get all the benefits of path tracing directly in the viewport,"}, {"start": 205.0, "text": "but during camera movement you can see some noise"}, {"start": 208.0, "text": "from the one sample-per-pixel presentation."}, {"start": 211.0, "text": "Enable the MetalFX denoiser,"}, {"start": 213.0, "text": "and the image becomes dramatically more stable and noise-free."}, {"start": 217.0, "text": "Redshift Live can now deliver clean, near-final"}, {"start": 220.0, "text": "image quality at interactive frame rates,"}, {"start": 222.0, "text": "with real-time ray-traced lighting, shadows, and global illumination."}, {"start": 227.0, "text": "Now artists can see lighting effects take place in real-time in their viewport,"}, {"start": 232.0, "text": "like this tree being moved."}, {"start": 234.0, "text": "This becomes possible when you combine hardware-accelerated ray tracing"}, {"start": 238.0, "text": "with MetalFX neural denoising."}, {"start": 241.0, "text": "Here is an example of a one sample-per-pixel frame"}, {"start": 244.0, "text": "rendered by Redshift Live."}, {"start": 246.0, "text": "By leveraging both spatial and temporal techniques,"}, {"start": 249.0, "text": "MetalFX is able to transform the noisy one sample-per-pixel into an image"}, {"start": 254.0, "text": "with near final quality, in real time."}, {"start": 258.0, "text": "To get all the details on the inputs"}, {"start": 260.0, "text": "and how to leverage MetalFX in your application,"}, {"start": 263.0, "text": "check out \"Go further with Metal 4 games\" session."}, {"start": 266.0, "text": "I'll outline three key best practices"}, {"start": 269.0, "text": "that Maxon used to get the best quality from MetalFX,"}, {"start": 272.0, "text": "starting with denoiser inputs and noise."}, {"start": 276.0, "text": "The output quality of the denoiser is directly"}, {"start": 279.0, "text": "dependent on the quality of your inputs."}, {"start": 282.0, "text": "Normally your auxiliary inputs are noise free,"}, {"start": 285.0, "text": "do your best to keep them that way."}, {"start": 287.0, "text": "Among all the inputs, the diffuse albedo is the strongest signal for denoising."}, {"start": 292.0, "text": "When in doubt, make it as close as possible to a noise free version"}, {"start": 296.0, "text": "of the final result you want to see on the screen."}, {"start": 300.0, "text": "Consider building debug views for each input directly in your engine."}, {"start": 305.0, "text": "Use a GPU capture to inspect textures frame-by-frame."}, {"start": 309.0, "text": "This will allow you to validate your inputs"}, {"start": 311.0, "text": "and make sure they look the way the model expects."}, {"start": 316.0, "text": "You might have some noise-free layers in your scene,"}, {"start": 319.0, "text": "or some parts you don't want to denoise as strongly."}, {"start": 323.0, "text": "You have two tools at your disposal, the transparency overlay,"}, {"start": 327.0, "text": "and the denoiser strength mask,"}, {"start": 329.0, "text": "using them will help you to maximise the quality in these scenarios."}, {"start": 334.0, "text": "Particles, fog, volumetrics, and sky"}, {"start": 337.0, "text": "are effects that don't have a meaningful surface"}, {"start": 339.0, "text": "and might be already noise free based on your rendering pipeline."}, {"start": 343.0, "text": "MetalFX will denoise and upscale your noisy input."}, {"start": 348.0, "text": "For those noise free effects, you can leverage"}, {"start": 351.0, "text": "the MetalFX transparency overlay input instead."}, {"start": 354.0, "text": "The overlay input will only be upscaled"}, {"start": 357.0, "text": "and composited in the final result for you."}, {"start": 360.0, "text": "For areas that are already noise free, like the sky, you can configure MetalFX"}, {"start": 365.0, "text": "to skip denoising for those pixels,"}, {"start": 367.0, "text": "using the denoiser strength mask."}, {"start": 369.0, "text": "I'll share an example."}, {"start": 371.0, "text": "Here, the sky has been marked as not to be denoised."}, {"start": 375.0, "text": "The value can be tuned from zero,"}, {"start": 377.0, "text": "meaning no denoising, all the way to one,"}, {"start": 380.0, "text": "meaning denoise at max strength depending on your use case."}, {"start": 384.0, "text": "This gives you control over the denoising effect in the scene."}, {"start": 388.0, "text": "By now you should already have a great output by MetalFX,"}, {"start": 392.0, "text": "but there are a few tricky cases with reflection and transmission."}, {"start": 396.0, "text": "This is what this second best practice will help you with."}, {"start": 400.0, "text": "A mirror has no color of its own."}, {"start": 402.0, "text": "The viewer sees the reflected surface."}, {"start": 405.0, "text": "As discussed previously, your inputs"}, {"start": 407.0, "text": "and especially the diffuse albedo should represent"}, {"start": 410.0, "text": "the final desired output as close as possible."}, {"start": 413.0, "text": "Store your reflected geometry properties like albedo, normal,"}, {"start": 417.0, "text": "and roughness in the mirror-like objects."}, {"start": 421.0, "text": "Glass builds on the same foundational concepts and pushes it a bit further."}, {"start": 426.0, "text": "The viewer sees a combination of what is reflected"}, {"start": 429.0, "text": "and what is transmitted, which could be noisy."}, {"start": 432.0, "text": "One solution is to blend geometry properties like diffuse albedo,"}, {"start": 436.0, "text": "by the Fresnel term reducing substantially the noise of your inputs."}, {"start": 441.0, "text": "The Fresnel is the term telling you at a given intersection point"}, {"start": 445.0, "text": "how much light would be reflected versus refracted."}, {"start": 449.0, "text": "On the left, you can see the primary surface albedo"}, {"start": 452.0, "text": "while on the right,"}, {"start": 453.0, "text": "it is replaced by the combined reflected and refracted albedo."}, {"start": 459.0, "text": "This is a well known technique called primary surface replacement."}, {"start": 463.0, "text": "Getting this right will keep the reflection beautiful and sharp."}, {"start": 467.0, "text": "Now that your materials look rich,"}, {"start": 469.0, "text": "and your reflection and refraction are sharp,"}, {"start": 472.0, "text": "let's dive into the third best practice: get your motion vectors right."}, {"start": 476.0, "text": "Correct motion vectors are essential for temporal stability."}, {"start": 481.0, "text": "Motion vectors are per-pixel screen-space displacements"}, {"start": 484.0, "text": "from the current frame to the previous frame."}, {"start": 488.0, "text": "For every pixel, the motion vector should answer the question,"}, {"start": 491.0, "text": "where was this pixel in the previous frame?"}, {"start": 494.0, "text": "Motion vectors have been a staple of modern rendering technique."}, {"start": 499.0, "text": "Getting motion vectors right makes the difference between a blurry result"}, {"start": 502.0, "text": "and a sharp output under motion."}, {"start": 505.0, "text": "The model uses motion vectors"}, {"start": 507.0, "text": "to understand the scene under motion and over time."}, {"start": 510.0, "text": "MetalFX expects dejittered motion vectors, meaning without the sub-pixel shifts."}, {"start": 516.0, "text": "Without this, MetalFX might receive motion vectors"}, {"start": 518.0, "text": "that might be up to one pixel wrong,"}, {"start": 521.0, "text": "creating edge shimmering."}, {"start": 523.0, "text": "Here is how you can compute them correctly."}, {"start": 526.0, "text": "Here is the code to compute camera-only motion vectors for static objects."}, {"start": 531.0, "text": "You start by computing the projected position of the current vertex."}, {"start": 536.0, "text": "Then project the same position through the previous frame's matrix."}, {"start": 540.0, "text": "Your motion vector is the difference between the two."}, {"start": 544.0, "text": "However since the camera matrices were jittered,"}, {"start": 547.0, "text": "subtract the jitter deltas from the current"}, {"start": 549.0, "text": "and previous frame."}, {"start": 551.0, "text": "Finally, get a clean unjittered motion vector for a cleaner motion."}, {"start": 556.0, "text": "For objects that move and deforming geometry,"}, {"start": 558.0, "text": "the camera-only path won't see the displacement."}, {"start": 561.0, "text": "Store each vertex's previous-frame world position, or skin twice,"}, {"start": 566.0, "text": "and compute the actual motion vector."}, {"start": 569.0, "text": "For objects where motion is genuinely unreliable fast motion,"}, {"start": 572.0, "text": "like alpha-blended particles use the reactive mask."}, {"start": 576.0, "text": "For more on the reactive mask, check out \"Go further with Metal 4 games\"."}, {"start": 582.0, "text": "This is what it looks like in practice."}, {"start": 584.0, "text": "Redshift Live from Maxon ships every best practice I just covered,"}, {"start": 589.0, "text": "getting the most of MetalFX Denoising,"}, {"start": 591.0, "text": "running on Apple silicon and delivering near-final image quality."}, {"start": 597.0, "text": "Now, I'll take you beyond platform solutions,"}, {"start": 599.0, "text": "and share how you can build your own ML-powered solutions."}, {"start": 604.0, "text": "Neural rendering goes well beyond denoising."}, {"start": 607.0, "text": "More and more techniques across the pipeline"}, {"start": 609.0, "text": "are becoming machine learning based,"}, {"start": 611.0, "text": "and with Metal 4, you have the tools to build and deploy your own."}, {"start": 616.0, "text": "Metal 4 gives you two ways to bring your own"}, {"start": 619.0, "text": "machine leaning technique into the pipeline."}, {"start": 622.0, "text": "The machine learning command encoder lets you deploy a trained model"}, {"start": 625.0, "text": "right in your command buffer in the same pipeline"}, {"start": 628.0, "text": "without context switch."}, {"start": 630.0, "text": "The TensorOps API lets you build a small hardware-accelerated network"}, {"start": 634.0, "text": "directly in your shader."}, {"start": 636.0, "text": "For more details on both APIs,"}, {"start": 638.0, "text": "check out \"Combine Metal 4 machine learning and graphics\"."}, {"start": 643.0, "text": "Today I'll focus on tone mapping."}, {"start": 646.0, "text": "Most renderers have extended post-processing pipelines"}, {"start": 649.0, "text": "to correctly map the HDR image"}, {"start": 651.0, "text": "to something that can be displayed and matches the artistic vision,"}, {"start": 655.0, "text": "like tone mapping, color grade or film emulation."}, {"start": 659.0, "text": "The pipeline is composed of multiple stages,"}, {"start": 662.0, "text": "each with its own parameters and concatenated outputs."}, {"start": 667.0, "text": "The pipeline can grow arbitrarily complex."}, {"start": 670.0, "text": "The best results come from understanding the content of the image,"}, {"start": 674.0, "text": "and that's exactly what a neural network can learn."}, {"start": 677.0, "text": "The idea is simple."}, {"start": 679.0, "text": "Take your existing whole color pipeline or part of it"}, {"start": 683.0, "text": "and replace it with a single neural network."}, {"start": 685.0, "text": "The network will learn the color transformation."}, {"start": 689.0, "text": "An example of such a workflow is called HDRNet."}, {"start": 693.0, "text": "A 2017 architecture from Gharbi and colleagues."}, {"start": 697.0, "text": "Here's the bird's-eye view on how it works."}, {"start": 700.0, "text": "The network works on a small downsampled version of the image."}, {"start": 704.0, "text": "It performs two types of analysis,"}, {"start": 706.0, "text": "a global and local one to capture both scene level and small details."}, {"start": 712.0, "text": "This process allows the network to create color transformations"}, {"start": 715.0, "text": "for 16x16 tiles of the image."}, {"start": 719.0, "text": "These localized transformations are applied"}, {"start": 721.0, "text": "with smart, edge-aware techniques"}, {"start": 723.0, "text": "to produce the beautiful tone mapped final result."}, {"start": 728.0, "text": "To create this solution you would first develop"}, {"start": 731.0, "text": "and train the network in your framework of choice,"}, {"start": 733.0, "text": "for example PyTorch."}, {"start": 735.0, "text": "The training data could be deployed from manually tone mapped previous projects,"}, {"start": 740.0, "text": "or a lot of tone mapped images generated by your renderer."}, {"start": 744.0, "text": "Once the model is trained, export it to an MTLPackage."}, {"start": 749.0, "text": "In order to execute your network in Metal 4,"}, {"start": 752.0, "text": "there are a few steps that need to be done on both setup"}, {"start": 755.0, "text": "and on the actual execution."}, {"start": 758.0, "text": "First you need to setup the pipeline by loading an MTLPackage,"}, {"start": 762.0, "text": "specifying the network function with a function descriptor"}, {"start": 765.0, "text": "and creating a machine learning pipeline descriptor."}, {"start": 769.0, "text": "This process is very similar to loading regular pipelines."}, {"start": 774.0, "text": "The next step is to dispatch your network execution,"}, {"start": 777.0, "text": "to do that, you will create an encoder,"}, {"start": 780.0, "text": "create an argument table with the inputs and outputs"}, {"start": 784.0, "text": "and finally dispatch the command buffer."}, {"start": 787.0, "text": "That will kick off the execution,"}, {"start": 789.0, "text": "where you will have a mix of compute,"}, {"start": 791.0, "text": "machine learning, and rendering work happening at the same time."}, {"start": 795.0, "text": "Here's the updated pipeline."}, {"start": 797.0, "text": "First, your path tracer produces samples,"}, {"start": 800.0, "text": "followed by MetalFX denoising and the new neural tone mapper,"}, {"start": 804.0, "text": "all encoded in the same command buffer,"}, {"start": 806.0, "text": "executing in the same frame."}, {"start": 809.0, "text": "The ML encoder replaced your entire multi-stage"}, {"start": 812.0, "text": "post-processing chain with a single neural evaluation."}, {"start": 817.0, "text": "I've shared how you can train and deploy your networks."}, {"start": 820.0, "text": "Now, go one level deeper and build small networks"}, {"start": 823.0, "text": "directly in your shaders with the TensorOps API."}, {"start": 827.0, "text": "So far you have explored large general-purpose networks"}, {"start": 830.0, "text": "trained offline on a very large dataset."}, {"start": 833.0, "text": "Now I will show you the opposite approach: tiny networks for one specific task."}, {"start": 839.0, "text": "A few thousand parameters or less, trained on your scene data,"}, {"start": 844.0, "text": "sometimes even trained online every few frames."}, {"start": 847.0, "text": "The network only sees one scenario, it does not need to generalise."}, {"start": 852.0, "text": "So far you have learned how to execute ML"}, {"start": 855.0, "text": "in the same command buffer as a stand alone step."}, {"start": 859.0, "text": "Here it is executing alongside compute and render."}, {"start": 863.0, "text": "However a small network can fit inline in your shader,"}, {"start": 867.0, "text": "among the rest of your code,"}, {"start": 869.0, "text": "ALU and texture sampling instructions."}, {"start": 873.0, "text": "The key enabling technology is TensorOps,"}, {"start": 875.0, "text": "available in any stage of the rendering pipeline."}, {"start": 879.0, "text": "All this combined unlocks new possibilities and workflows"}, {"start": 883.0, "text": "that involve online training."}, {"start": 886.0, "text": "Here's an example, a skybox used for image based lighting."}, {"start": 890.0, "text": "The skybox is casting light on the geometry in the scene,"}, {"start": 893.0, "text": "creating a natural soft illumination."}, {"start": 896.0, "text": "The soft illumination is the result"}, {"start": 898.0, "text": "of the average light coming from all visible directions at a specific point."}, {"start": 903.0, "text": "Normally, this result is precomputed offline and sampled at runtime."}, {"start": 909.0, "text": "However, a scene is rarely static."}, {"start": 912.0, "text": "You might have a dynamic day-night cycle."}, {"start": 915.0, "text": "Your offline learned signal may be out of sync."}, {"start": 919.0, "text": "This is a learnable function for a neural network,"}, {"start": 922.0, "text": "and this is where online training comes into play."}, {"start": 925.0, "text": "Here is how you could recreate this technique."}, {"start": 928.0, "text": "Based on what you learned about the machine learning encoder so far,"}, {"start": 932.0, "text": "a simplified rendering loop might look like this,"}, {"start": 935.0, "text": "first, you update your world"}, {"start": 937.0, "text": "so that all the information is up to date for rendering."}, {"start": 941.0, "text": "Next, you dispatch the machine learning encoder"}, {"start": 943.0, "text": "to run the inference on the model,"}, {"start": 946.0, "text": "and produce the necessary lighting information"}, {"start": 949.0, "text": "that you will use later for shading."}, {"start": 952.0, "text": "Online training disrupts this paradigm."}, {"start": 954.0, "text": "By creating your own training and inference routines,"}, {"start": 958.0, "text": "you can run one or more training iterations per frame"}, {"start": 961.0, "text": "to improve the model accuracy."}, {"start": 964.0, "text": "This is how the online training loop"}, {"start": 966.0, "text": "would look like for the sky illumination model."}, {"start": 969.0, "text": "You start by generating a direction you wish to sample"}, {"start": 973.0, "text": "and run inference on your model to get the result."}, {"start": 976.0, "text": "Then you are able to compute the analytical solution"}, {"start": 979.0, "text": "to the sky illumination problem that you can use to compute the error,"}, {"start": 983.0, "text": "and finally, run a back propagation pass to progressively improve the model."}, {"start": 988.0, "text": "This is the same exact flow you could use to train offline,"}, {"start": 992.0, "text": "but this time, repeating training iteration over frames."}, {"start": 997.0, "text": "So, you are now running your own inference and training routines."}, {"start": 1001.0, "text": "This enables you to run the inference pass,"}, {"start": 1004.0, "text": "inline in your shading pass,"}, {"start": 1007.0, "text": "And TensorOps will allow you to implement this very efficiently."}, {"start": 1011.0, "text": "You now have a model that every frame adapts"}, {"start": 1013.0, "text": "to the new world condition"}, {"start": 1015.0, "text": "and can use this information for shading right away."}, {"start": 1019.0, "text": "This would not be possible"}, {"start": 1020.0, "text": "with the standard offline training workflow."}, {"start": 1023.0, "text": "This concept generalizes to any technique that can learn a signal."}, {"start": 1027.0, "text": "Here is how to start building your own solutions."}, {"start": 1030.0, "text": "At a high level, a neural network is composed"}, {"start": 1032.0, "text": "of three main building blocks:"}, {"start": 1035.0, "text": "the input layer,"}, {"start": 1036.0, "text": "which processes the network inputs, also known as input features."}, {"start": 1040.0, "text": "The output layer, which generates the network's final predictions,"}, {"start": 1044.0, "text": "and finally the hidden layers,"}, {"start": 1046.0, "text": "where the magic of learning happens."}, {"start": 1048.0, "text": "The sky probe is a small network,"}, {"start": 1051.0, "text": "the hidden layers group is composed of two hidden layers of four neurons each."}, {"start": 1056.0, "text": "The network takes as an input value three floats to encode a direction,"}, {"start": 1060.0, "text": "and produces three floats as an output"}, {"start": 1063.0, "text": "that represents the average illumination coming from that direction, as a color."}, {"start": 1068.0, "text": "This is called a fully connected multilayer perceptron,"}, {"start": 1072.0, "text": "or in short an MLP, a 3 - 4 - 4 -3 network."}, {"start": 1076.0, "text": "You can experiment with the input sizes,"}, {"start": 1079.0, "text": "amount and size of layers"}, {"start": 1080.0, "text": "to get the best result for your application."}, {"start": 1083.0, "text": "To be able to evaluate your network you need to prepare your input tensor."}, {"start": 1088.0, "text": "It's best to batch multiple inputs at the same time making it a 2D matrix."}, {"start": 1094.0, "text": "For the sky probe example, this will be a 2D matrix"}, {"start": 1097.0, "text": "of a batch of input directions you wish to evaluate."}, {"start": 1100.0, "text": "But the input can contain whatever data might be useful to the network,"}, {"start": 1105.0, "text": "like positional or material data."}, {"start": 1108.0, "text": "Same principle applies to the output tensor."}, {"start": 1111.0, "text": "For sky probe, make it a 2D matrix of a batch of colors."}, {"start": 1116.0, "text": "Now that you know the structure of an MLP,"}, {"start": 1118.0, "text": "here is how you can implement it in your shader"}, {"start": 1120.0, "text": "and evaluate it in a forward pass."}, {"start": 1123.0, "text": "Now you are ready to begin the evaluation."}, {"start": 1126.0, "text": "You have your input tensor and the first hidden layer weights tensor."}, {"start": 1131.0, "text": "You can multiply the two together using a matmul 2D tensor operation."}, {"start": 1137.0, "text": "You will obtain a pre-activation result"}, {"start": 1139.0, "text": "on which you want to apply your activation function."}, {"start": 1142.0, "text": "Before doing that, you will need to store your matrix multiplication result."}, {"start": 1147.0, "text": "I'll explain how to do that efficiently."}, {"start": 1150.0, "text": "You may be familiar with the thread execution scope,"}, {"start": 1152.0, "text": "where a single thread will be in charge of executing the whole tensor operation."}, {"start": 1157.0, "text": "This works great for executing divergent work or in pipeline stages"}, {"start": 1161.0, "text": "where you don't have full control of a thread group."}, {"start": 1164.0, "text": "However, when you do have full control, new possibilities arise."}, {"start": 1169.0, "text": "In a compute stage, you can use SIMD group execution scope,"}, {"start": 1174.0, "text": "where all participating threads will work on the same matrix multiplicaiton."}, {"start": 1178.0, "text": "This execution mode, will also give you access to cooperative tensors."}, {"start": 1183.0, "text": "Cooperative tensors storage is distributed"}, {"start": 1185.0, "text": "among multiple threads in the thread group,"}, {"start": 1188.0, "text": "avoiding an expensive round trip to main memory."}, {"start": 1192.0, "text": "By using a cooperative tensor as an output of your first multiplication,"}, {"start": 1196.0, "text": "the result will stay in fast thread storage memory."}, {"start": 1199.0, "text": "Then you can apply your activation function in place."}, {"start": 1203.0, "text": "You can now repeat the same operation of matrix multiplication"}, {"start": 1207.0, "text": "and activation for the next layer."}, {"start": 1210.0, "text": "And all the subsequent layers, all the way to the output layer,"}, {"start": 1214.0, "text": "where you can store the resulting tensor"}, {"start": 1216.0, "text": "and leverage the result in your compute shader immediately, or at a later stage."}, {"start": 1222.0, "text": "On the left, there is the ground truth render computed using raytracing."}, {"start": 1227.0, "text": "On the right the neural rendering version."}, {"start": 1230.0, "text": "The small neural network was capable of learning the signal efficiently."}, {"start": 1235.0, "text": "This was a high level overview of how you can construct an MLP"}, {"start": 1239.0, "text": "and evaluate it in your shader using TensorOps."}, {"start": 1242.0, "text": "The same exact building blocks can be used to create an efficient"}, {"start": 1246.0, "text": "back propagation pass needed for the online training step."}, {"start": 1250.0, "text": "For all the code details,"}, {"start": 1251.0, "text": "please check the \"Metal Performance Primitives (MPP) Programming Guide\"."}, {"start": 1255.0, "text": "To recap,"}, {"start": 1257.0, "text": "today, I have covered three levels of ML in your rendering pipeline."}, {"start": 1261.0, "text": "First, MetalFX gives you platform-integrated neural denoising,"}, {"start": 1264.0, "text": "with three best practices:"}, {"start": 1267.0, "text": "keep your inputs clean, store what the viewer sees,"}, {"start": 1270.0, "text": "get motion vectors right."}, {"start": 1272.0, "text": "Next, the MTLPackage lets you export"}, {"start": 1274.0, "text": "your offline trained models and deploy at runtime,"}, {"start": 1278.0, "text": "You learned how to replace an entire"}, {"start": 1280.0, "text": "post-processing pipeline with one neural evaluation."}, {"start": 1284.0, "text": "Finally, I covered the TensorOps API,"}, {"start": 1287.0, "text": "it lets you build tiny networks directly in your shaders,"}, {"start": 1290.0, "text": "running on the neural accelerator."}, {"start": 1292.0, "text": "Each level gives you more control."}, {"start": 1295.0, "text": "Pick the one that's right for your app."}, {"start": 1298.0, "text": "Download Xcode and explore the Metal 4 sample code."}, {"start": 1303.0, "text": "If your app has realtime requirements,"}, {"start": 1305.0, "text": "like viewports in pro-apps or games, adopt MetalFX Denoising and Upscaling."}, {"start": 1311.0, "text": "Try training a neural tone mapper with your own post-processing pipeline."}, {"start": 1316.0, "text": "And experiment with small specialized networks using the tensor API."}, {"start": 1321.0, "text": "Check out our sessions from previous years for more details."}, {"start": 1326.0, "text": "I can't wait to see what you build."}]}