{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-305", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/305", "wordCount": 2051, "segments": [{"start": 7.0, "text": "Welcome everyone."}, {"start": 8.0, "text": "My name is David Hayward"}, {"start": 10.0, "text": "and I am excited to talk about enhancements to Core Image,"}, {"start": 13.0, "text": "and its support for RAW image files."}, {"start": 16.0, "text": "I will talk about four main topics."}, {"start": 19.0, "text": "First, I will review how RAW image files from cameras,"}, {"start": 23.0, "text": "are supported by Core Image on Apple platforms."}, {"start": 27.0, "text": "And show you significant RAW quality improvements that are coming in iOS,"}, {"start": 32.0, "text": "iPadOS, macOS, and visionOS 27."}, {"start": 38.0, "text": "After that, I will discuss how to get optimal performance when rendering RAWs."}, {"start": 44.0, "text": "And lastly, I will describe features,"}, {"start": 47.0, "text": "that Apple added to the CIImageProcessor class for RAW."}, {"start": 52.0, "text": "To start, here is a quick summary of how Core Image supports RAW."}, {"start": 58.0, "text": "RAW files come from a diverse set of camera makes and models,"}, {"start": 62.0, "text": "but unlike HEIFs and JPEGs,"}, {"start": 64.0, "text": "they need special handling before they can be displayed."}, {"start": 69.0, "text": "The first step, is to parse the file's metadata,"}, {"start": 72.0, "text": "and unpack the RAW sensor values."}, {"start": 75.0, "text": "At this stage, each pixel location only has a red,"}, {"start": 79.0, "text": "green, or blue value arranged in a mosaic pattern."}, {"start": 83.0, "text": "This image is cropped way in, so that this pattern is visible."}, {"start": 88.0, "text": "The next step, is to demosaic the sensor data,"}, {"start": 91.0, "text": "so that each pixel contains red, green, and blue values."}, {"start": 97.0, "text": "The following stage is to denoise the pixels,"}, {"start": 100.0, "text": "so that the image is free of photon noise, read noise and thermal noise."}, {"start": 107.0, "text": "After this, convolutions are applied"}, {"start": 109.0, "text": "to sharpen edges and add local contrast."}, {"start": 113.0, "text": "The last major step is to adjust white balance, exposure,"}, {"start": 118.0, "text": "color and tone to make a pleasing final image."}, {"start": 122.0, "text": "The algorithms for all these steps are built in to iOS,"}, {"start": 126.0, "text": "iPadOS, macOS, and visionOS."}, {"start": 131.0, "text": "This means that people can view RAW files in system apps like Finder,"}, {"start": 136.0, "text": "Preview and Freeform."}, {"start": 139.0, "text": "In fact, any app or framework that uses the Image IO API"}, {"start": 144.0, "text": "gets RAW support automatically."}, {"start": 148.0, "text": "And beyond basic viewing, apps that use the CIRAWFilter API,"}, {"start": 153.0, "text": "can present advanced editing controls."}, {"start": 157.0, "text": "This API is used by apps like Photos, Pixelmator Pro,"}, {"start": 162.0, "text": "and others like Nitro, Acorn, and many more."}, {"start": 168.0, "text": "Starting back in 2006,"}, {"start": 170.0, "text": "the system included hand-tuned calibrations for just 21 camera models."}, {"start": 176.0, "text": "In the years since, that number has grown to 784 models,"}, {"start": 180.0, "text": "across all major camera vendors."}, {"start": 183.0, "text": "This even includes support for Apple Pro RAW files for iPhone cameras."}, {"start": 189.0, "text": "But the key feature of RAW, is that a beloved photo you took years ago,"}, {"start": 193.0, "text": "can be reprocessed using the latest state-of-the-art algorithms."}, {"start": 198.0, "text": "Apple's RAW processing pipeline has been updated eight times,"}, {"start": 202.0, "text": "each improving demosaic, denoise and color."}, {"start": 207.0, "text": "Some older versions have been maintained,"}, {"start": 209.0, "text": "so that people can still use them if desired."}, {"start": 213.0, "text": "Now, Apple has made its biggest update yet, RAW 9!"}, {"start": 218.0, "text": "This version dramatically improves the rendering of RAW files."}, {"start": 222.0, "text": "It is built atop a tiled CoreML model,"}, {"start": 225.0, "text": "that combines demosaic with denoise for best quality."}, {"start": 229.0, "text": "And the model is run on device using the Apple Neural Engine cores,"}, {"start": 233.0, "text": "for optimal performance."}, {"start": 236.0, "text": "Let me demonstrate the improvements up close."}, {"start": 240.0, "text": "This is a zoomed-in crop of a low noise image using RAW 8."}, {"start": 245.0, "text": "This Sony Alpha 7 II image of a vintage dial indicator actually looks quite good."}, {"start": 252.0, "text": "However, when you explore that same image under RAW 9,"}, {"start": 256.0, "text": "the image is sharper, clearer, and the fine text is easier to read."}, {"start": 262.0, "text": "The differences are even more dramatic, when you view high noise images."}, {"start": 267.0, "text": "First, observe the actual RAW data"}, {"start": 270.0, "text": "that is contained in this very noisy ISO 51,200 image."}, {"start": 276.0, "text": "In this example from a Canon 5D Mark III,"}, {"start": 280.0, "text": "the image is a 10x Crop of a box of crayons."}, {"start": 284.0, "text": "There is so much luma and chroma noise in the RAW data,"}, {"start": 287.0, "text": "that it's impossible to discern the unique color of each crayon."}, {"start": 292.0, "text": "Using our previous algorithms, this is the result!"}, {"start": 297.0, "text": "RAW 8 did an acceptable job of recovering the actual colors in the scene."}, {"start": 302.0, "text": "But if you examine the results under RAW 9,"}, {"start": 306.0, "text": "the output is significantly better."}, {"start": 308.0, "text": "The colors are accurate and well defined."}, {"start": 312.0, "text": "Even the shiny specular highlights on the crayons are visible."}, {"start": 317.0, "text": "This last example is a crop of a photo of embroidery yarn,"}, {"start": 321.0, "text": "shot with a Fujifilm X-T5 at ISO 12,800."}, {"start": 326.0, "text": "This camera has a non-traditional sensor pattern,"}, {"start": 329.0, "text": "which is challenging to demosaic."}, {"start": 332.0, "text": "In the RAW 8 results,"}, {"start": 334.0, "text": "there are some color artifacts and loss of detail in the yarn."}, {"start": 338.0, "text": "But if you observe the same image under RAW 9,"}, {"start": 341.0, "text": "the results are discernibly better."}, {"start": 344.0, "text": "The small text is more legible, and the texture in the yarn much clearer."}, {"start": 350.0, "text": "So here is how you can enable RAW 9 in your application."}, {"start": 355.0, "text": "First, use the CIRAWFilter API to load a RAW file."}, {"start": 360.0, "text": "This API is well described in the \"Capture and process ProRAW images\" from WWDC 21."}, {"start": 368.0, "text": "However, it is important to know that RAW 9 is not enabled by default."}, {"start": 373.0, "text": "You should check that the supportedDecoderVersions property"}, {"start": 377.0, "text": "contains the version9 enumeration."}, {"start": 380.0, "text": "If so,"}, {"start": 381.0, "text": "you can opt in by setting the decoderVersion property to version9."}, {"start": 387.0, "text": "To know what camera models support RAW 9,"}, {"start": 390.0, "text": "there's now a class method called supportedCameraModels."}, {"start": 394.0, "text": "This API will provide your app,"}, {"start": 397.0, "text": "an array of all the models that are supported with a given version."}, {"start": 401.0, "text": "In the release of iOS, iPadOS, macOS, and visionOS 27,"}, {"start": 407.0, "text": "there will be hundreds models that can use RAW 9,"}, {"start": 410.0, "text": "including all major professional camera vendors."}, {"start": 414.0, "text": "This camera list will grow,"}, {"start": 416.0, "text": "via over-the-air updates to the operating system."}, {"start": 419.0, "text": "RAW 9 is also automatically supported by cameras that shoot DNG natively,"}, {"start": 425.0, "text": "such as Apple iPhones."}, {"start": 428.0, "text": "The true power of the CIRAWFilter API is unleashed,"}, {"start": 432.0, "text": "when your app modifies its properties."}, {"start": 435.0, "text": "This allows people to customize how RAW files are displayed."}, {"start": 441.0, "text": "There are currently 20 calibrated properties that can be adjusted."}, {"start": 445.0, "text": "Adopt these adjustments in your app,"}, {"start": 447.0, "text": "to unlock the full power of editing RAW images."}, {"start": 451.0, "text": "These are the most important controls for editing RAW:"}, {"start": 455.0, "text": "exposure, which controls how much to brighten or darken the image,"}, {"start": 460.0, "text": "luminanceNoiseReductionAmount,"}, {"start": 462.0, "text": "which adjusts how much fine luma grain is visible,"}, {"start": 466.0, "text": "sharpnessAmount, which determines how much the edges are sharpened,"}, {"start": 471.0, "text": "and contrastAmount, which alters how much local contrast is applied near edges."}, {"start": 477.0, "text": "All these controls work even better in RAW 9 than they did in prior versions."}, {"start": 484.0, "text": "There are some properties that are no longer needed in RAW 9."}, {"start": 488.0, "text": "The colorNoiseReductionAmount property now has no effect,"}, {"start": 491.0, "text": "because the CoreML model handles color noise reduction automatically."}, {"start": 497.0, "text": "Also, the detailAmount and moireReductionAmount properties,"}, {"start": 501.0, "text": "are no longer needed nor supported in RAW 9."}, {"start": 505.0, "text": "You can call the is supported properties,"}, {"start": 508.0, "text": "to check if properties work for the filter instance."}, {"start": 513.0, "text": "Now that you have learned how RAW 9 appears and behaves,"}, {"start": 517.0, "text": "I would like to discuss its performance."}, {"start": 520.0, "text": "Fundamentally, in order to get such quality improvements,"}, {"start": 523.0, "text": "RAW 9 is more performance and resource intensive than prior versions."}, {"start": 529.0, "text": "Even though RAW 9 runs the CoreML model hundreds of times per image,"}, {"start": 534.0, "text": "when an app edits CIRAWFilter properties,"}, {"start": 538.0, "text": "subsequent renders are fast and responsive."}, {"start": 542.0, "text": "This is because Core Image caches intermediate results."}, {"start": 547.0, "text": "The best practices to get optimal performance,"}, {"start": 550.0, "text": "depends on how your app uses RAW files."}, {"start": 554.0, "text": "I will give some recommendations for the two most common use cases."}, {"start": 559.0, "text": "First, here is advice on the interactive editing use case."}, {"start": 564.0, "text": "Interactive editing,"}, {"start": 566.0, "text": "is when one RAW file is rendered multiple times at screen resolution."}, {"start": 571.0, "text": "Typically this is in response to the app adjusting a CIRAWFilter property,"}, {"start": 575.0, "text": "such as exposure or sharpness."}, {"start": 579.0, "text": "When interactive editing RAWs,"}, {"start": 581.0, "text": "use the scaleFactor property of the CIRAWFilter,"}, {"start": 585.0, "text": "when displaying the image at a reduced size."}, {"start": 588.0, "text": "This is important because it reduces the work to render images"}, {"start": 592.0, "text": "that have more megapixels than the display."}, {"start": 596.0, "text": "Use one CIContext per view and set its cacheIntermediates option to true."}, {"start": 602.0, "text": "Caching allows the intensive CoreML work to be skipped,"}, {"start": 606.0, "text": "while the app is adjusting CIRAWFilter properties."}, {"start": 610.0, "text": "Core Image will use more memory for caching between renders,"}, {"start": 614.0, "text": "if you add the \"Extended Virtual Addressing entitlement\" to your application."}, {"start": 619.0, "text": "There is detailed documentation for this on Apple's website."}, {"start": 623.0, "text": "Also, render directly to Metal-backed views."}, {"start": 626.0, "text": "This improves performance of repeated renders,"}, {"start": 629.0, "text": "because Metal can start work on the next frame before,"}, {"start": 632.0, "text": "the previous frame is completed."}, {"start": 635.0, "text": "For more information on rendering to a MTKView,"}, {"start": 639.0, "text": "consult the \"Display EDR content with Core Image, Metal, and SwiftUI\""}, {"start": 644.0, "text": "from WWDC 22."}, {"start": 648.0, "text": "After that advice on the interactive editing use case,"}, {"start": 652.0, "text": "here are the most important tips when exporting files:"}, {"start": 656.0, "text": "The exporting use case,"}, {"start": 657.0, "text": "is when multiple RAW files are each rendered once at full resolution"}, {"start": 662.0, "text": "to other formats like HEIF or JPEG."}, {"start": 667.0, "text": "The best practice for this case is to export with a CIContext,"}, {"start": 671.0, "text": "with the option cacheIntermediates set to false."}, {"start": 675.0, "text": "Also, you can tell Core Image to use more memory during each export,"}, {"start": 681.0, "text": "by setting the context memoryLimit option."}, {"start": 685.0, "text": "On iOS, the default limit is a conservative 256 megabytes."}, {"start": 691.0, "text": "Setting the limit to 512 or 1024 megabytes can significantly improve performance."}, {"start": 699.0, "text": "And you can get extra memory savings by using the context methods"}, {"start": 703.0, "text": "heifRepresentation or jpegRepresentation, instead of calling Image IO directly."}, {"start": 710.0, "text": "In the last section,"}, {"start": 711.0, "text": "I want to talk about improvements to the CIImageProcessor API."}, {"start": 716.0, "text": "RAW 9 uses the CIImageProcessor API,"}, {"start": 720.0, "text": "because it enables algorithms that use CoreML in conjunction with other CIKernels."}, {"start": 728.0, "text": "As a result of this effort, the Core Image team added two features"}, {"start": 732.0, "text": "to the CIImageProcessor API that you may want to use in your app."}, {"start": 737.0, "text": "The first I'll talk about, is support for explicit output tile sizes."}, {"start": 744.0, "text": "This is an example of a typical CIImageProcessor class."}, {"start": 748.0, "text": "It implements the region of interest callback,"}, {"start": 750.0, "text": "that defines how much of its input is needed for a given output rectangle."}, {"start": 756.0, "text": "Most importantly, within the process callback,"}, {"start": 759.0, "text": "you must be aware of the input.region and output.region,"}, {"start": 763.0, "text": "that the processor should operate on."}, {"start": 765.0, "text": "If there is plenty of memory,"}, {"start": 767.0, "text": "then CoreImage will call the process function,"}, {"start": 769.0, "text": "with an output.region for the entire image."}, {"start": 773.0, "text": "But when memory is limited,"}, {"start": 774.0, "text": "the output.region might be considerably smaller."}, {"start": 778.0, "text": "Now, a processor can explicitly control output tile sizes."}, {"start": 784.0, "text": "The code in your processor class remains as before,"}, {"start": 788.0, "text": "but you tell CoreImage to use the output regions of your choice."}, {"start": 794.0, "text": "First create an array,"}, {"start": 797.0, "text": "and then fill the array with all the desired tiles to cover your image."}, {"start": 802.0, "text": "In this example, the tiling strategy takes the input image extent,"}, {"start": 807.0, "text": "and breaks it up into 512x512 pixel tiles."}, {"start": 812.0, "text": "When you create an image which uses the processor,"}, {"start": 815.0, "text": "call its apply method and pass in the tile array that covers the image."}, {"start": 821.0, "text": "So now that I've described explicit output tile sizes,"}, {"start": 825.0, "text": "my last topic is the temporary buffers feature."}, {"start": 829.0, "text": "It is common for a CIImageProcessor that calls CoreML to use temporary buffers."}, {"start": 836.0, "text": "This is because CoreImage uses interleaved image buffers,"}, {"start": 840.0, "text": "which must be converted to planar data for CoreML."}, {"start": 845.0, "text": "When a processor callback is called for many tiles,"}, {"start": 848.0, "text": "the temporary buffers will be created and destroyed repeatedly."}, {"start": 853.0, "text": "This can impact performance."}, {"start": 856.0, "text": "The CIImageProcessorOutput class, now has methods to help."}, {"start": 861.0, "text": "Here's how this works!"}, {"start": 863.0, "text": "Here's an example of a simple CIImageProcessor class"}, {"start": 867.0, "text": "that uses the temporary buffer feature."}, {"start": 871.0, "text": "Instead of processing directly, from input to output,"}, {"start": 876.0, "text": "this callback requests a scratch buffer"}, {"start": 879.0, "text": "by asking the output object for a temporary CVPixelBuffer."}, {"start": 885.0, "text": "When doing so, provide an identifier."}, {"start": 888.0, "text": "This is vital for process callbacks that use more than one temporary buffer."}, {"start": 895.0, "text": "Then, this example processor"}, {"start": 898.0, "text": "copies from the input pixel buffer to the temporary buffer,"}, {"start": 902.0, "text": "alters the temporary buffer pixels in-place, and finally copies that to the output."}, {"start": 909.0, "text": "Core Image will manage the lifecycle of temporary buffers for you."}, {"start": 912.0, "text": "They will be released automatically and released at the correct time"}, {"start": 916.0, "text": "and recycled when the processor is called for the next tile."}, {"start": 920.0, "text": "Here are the key points to take away."}, {"start": 924.0, "text": "Try RAW 9 in your app."}, {"start": 927.0, "text": "It's a major quality improvement"}, {"start": 929.0, "text": "and it's just a few lines of code to enable."}, {"start": 933.0, "text": "Follow the performance best practices,"}, {"start": 935.0, "text": "I outlined for fast exporting and responsive editing."}, {"start": 940.0, "text": "Give your app access to the powerful editing properties of CIRAWFilter,"}, {"start": 945.0, "text": "that let people finetune how their RAW images appear."}, {"start": 950.0, "text": "And finally, for optimal performance in your CIImageProcessor,"}, {"start": 955.0, "text": "here use the explicit tiling and temporary buffer APIs."}, {"start": 960.0, "text": "Personally, I have really enjoyed revisiting my library,"}, {"start": 964.0, "text": "of over 7000 family photos that I have shot in RAW over the past 20 years."}, {"start": 969.0, "text": "It has been really great to see the improved quality of noise reduction."}, {"start": 974.0, "text": "I think that people using your app, will also enjoy these improvements."}, {"start": 979.0, "text": "Thank you for watching!"}]}