{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-338", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/338", "wordCount": 2314, "segments": [{"start": 7.0, "text": "Hi there and welcome to \"Build Live Production Tools,"}, {"start": 10.0, "text": "for Apple Immersive Video\"."}, {"start": 12.0, "text": "I'm Jared King and I lead the Apple Immersive Video Live Engineering team."}, {"start": 17.0, "text": "Apple Immersive Video is an incredibly exciting medium, and live streaming"}, {"start": 21.0, "text": "unlocks entirely new ways for customers to experience sports, music,"}, {"start": 25.0, "text": "and entertainment events!"}, {"start": 28.0, "text": "Earlier this year, Apple did something remarkable."}, {"start": 31.0, "text": "For the first time fans were transported, courtside to select LA Lakers games —"}, {"start": 36.0, "text": "live in Apple Vision Pro, through the Spectrum SportsNet and NBA apps."}, {"start": 41.0, "text": "Real games and arenas, experienced like you were there!"}, {"start": 45.0, "text": "Live Immersive cameras gave fans access to unreachable seats."}, {"start": 49.0, "text": "Data-driven graphics augmented the action"}, {"start": 52.0, "text": "and spatial audio, embedded customers inside the roaring crowd."}, {"start": 56.0, "text": "Behind the scenes, a live broadcast platform, built by Apple,"}, {"start": 60.0, "text": "powered the events, and transported this unique experience"}, {"start": 63.0, "text": "to customers around the world."}, {"start": 67.0, "text": "My goal is to inspire you to build the next generation of immersive tools,"}, {"start": 71.0, "text": "workflows, and live experiences."}, {"start": 74.0, "text": "For many, live broadcast technology might be new."}, {"start": 77.0, "text": "The systems are complex,"}, {"start": 79.0, "text": "but the opportunity to build for this space is enormous."}, {"start": 83.0, "text": "So to set the right foundation…"}, {"start": 85.0, "text": "First I'll provide a high level overview of the systems that make up"}, {"start": 88.0, "text": "a modern live production pipeline -"}, {"start": 90.0, "text": "along with some of the creative tools used across studios, production trucks,"}, {"start": 95.0, "text": "and broadcast facilities worldwide to create content."}, {"start": 99.0, "text": "Understanding this foundation will be invaluable"}, {"start": 101.0, "text": "as you begin building immersive production tools of your own."}, {"start": 105.0, "text": "Second, I'll cover what makes live immersive broadcast"}, {"start": 108.0, "text": "different from traditional 2D production."}, {"start": 111.0, "text": "Transporting customers from their homes, directly into an event —"}, {"start": 115.0, "text": "introduces an entirely new set of technical challenges."}, {"start": 118.0, "text": "The media formats, production tools, and the way content moves between tools"}, {"start": 123.0, "text": "within a production workflow, have been rebuilt."}, {"start": 126.0, "text": "I'll start today with a top level review"}, {"start": 128.0, "text": "of a live production pipeline."}, {"start": 130.0, "text": "Whether you're building for immersive,"}, {"start": 132.0, "text": "or traditional 2D broadcast,"}, {"start": 134.0, "text": "understanding the fundamental components"}, {"start": 136.0, "text": "of the end-to-end system is crucial."}, {"start": 139.0, "text": "A live production pipeline is a system"}, {"start": 142.0, "text": "where video audio and data is captured"}, {"start": 145.0, "text": "and creatively produced at one end -"}, {"start": 147.0, "text": "what I'll call the production domain,"}, {"start": 149.0, "text": "then encoded and streamed live to an audience"}, {"start": 152.0, "text": "through what I'll call the delivery domain."}, {"start": 155.0, "text": "Broadcasts can be large-scale productions,"}, {"start": 157.0, "text": "such as TV studios, and broadcast trucks,"}, {"start": 160.0, "text": "that manage many live cameras,"}, {"start": 162.0, "text": "audio sources, and graphics for sporting or entertainment events,"}, {"start": 167.0, "text": "Or, they can be smaller systems."}, {"start": 169.0, "text": "Such as podcast studios, theaters or local music venues"}, {"start": 173.0, "text": "where only a few of these elements might be used within a production."}, {"start": 176.0, "text": "Either way, the end goal is the same: live content is captured and produced"}, {"start": 181.0, "text": "in the production domain,"}, {"start": 183.0, "text": "then encoded and transmitted to the viewer in the delivery domain,"}, {"start": 186.0, "text": "broadcasting the event to an audience, in real time."}, {"start": 190.0, "text": "But to keep focused,"}, {"start": 191.0, "text": "I'll concentrate primarily on what makes the production domain unique,"}, {"start": 194.0, "text": "when building for immersive live."}, {"start": 197.0, "text": "Regardless of scale, most live pipelines"}, {"start": 199.0, "text": "rely on many of the same creative tools within the workflow"}, {"start": 203.0, "text": "simply scaled in quantity and sophistication,"}, {"start": 205.0, "text": "depending on the level of production."}, {"start": 208.0, "text": "Live Cameras are used to capture video of the scene, or event."}, {"start": 213.0, "text": "Oftentimes, multiple cameras are used on a production,"}, {"start": 216.0, "text": "to capture different angles, or points of view."}, {"start": 219.0, "text": "For example, in this production, here is camera 1."}, {"start": 223.0, "text": "This is camera 2."}, {"start": 225.0, "text": "And that is camera 3."}, {"start": 227.0, "text": "Graphics may be generated, and keyed onto the video to provide additional context,"}, {"start": 232.0, "text": "and creative flair to the production."}, {"start": 234.0, "text": "These can be elements like: a name -"}, {"start": 236.0, "text": "which just appeared as a lower third,"}, {"start": 239.0, "text": "a scoreboard, displayed in the top right,"}, {"start": 241.0, "text": "or a complex video animation."}, {"start": 243.0, "text": "Like the one down below."}, {"start": 246.0, "text": "Replay systems are used to record media and replay the footage back out"}, {"start": 250.0, "text": "when called for, as part of the live production."}, {"start": 253.0, "text": "Or, they can archive it for later use,"}, {"start": 255.0, "text": "in post production and editorial."}, {"start": 259.0, "text": "To assemble all these elements,"}, {"start": 260.0, "text": "video switchers let operators cut between cameras,"}, {"start": 263.0, "text": "overlay graphics, and produce the final creative stream the viewer sees."}, {"start": 268.0, "text": "On the audio front, microphones are used to pick up announcers, interviews,"}, {"start": 273.0, "text": "musical elements, and other sound sources on the production."}, {"start": 277.0, "text": "Audio consoles ingest all these sources!"}, {"start": 280.0, "text": "And artfully combine them together into the final product"}, {"start": 283.0, "text": "or \"mix\" the audience hears!"}, {"start": 285.0, "text": "Finally all of these tools need to exchange media with one another."}, {"start": 290.0, "text": "For example: camera feeds - connected into the video switcher inputs,"}, {"start": 294.0, "text": "or microphone sources - fed into an audio console."}, {"start": 299.0, "text": "To do that, every tool connects through a centralized media router"}, {"start": 302.0, "text": "that handles content exchange between them."}, {"start": 305.0, "text": "Think of it as a unified network layer,"}, {"start": 307.0, "text": "that lets every device send and receive signals"}, {"start": 309.0, "text": "to one another."}, {"start": 311.0, "text": "Now that you've got a handle on the basics of live production,"}, {"start": 314.0, "text": "here's where things start to diverge for immersive"}, {"start": 316.0, "text": "and where the real story begins!"}, {"start": 319.0, "text": "In this format, fidelity, presence,"}, {"start": 322.0, "text": "and preserving these through every step of the workflow"}, {"start": 325.0, "text": "is everything when you are transporting customers"}, {"start": 327.0, "text": "inside of the content."}, {"start": 329.0, "text": "And that translates into truly big numbers!"}, {"start": 333.0, "text": "Video resolution is 32 times larger"}, {"start": 336.0, "text": "than what is typically used in a 2D broadcast production -"}, {"start": 339.0, "text": "in order to match human visual acuity"}, {"start": 342.0, "text": "and it's produced at two times the frame rate!"}, {"start": 345.0, "text": "Audio mixes supporting Apple Immersive Live"}, {"start": 347.0, "text": "are far more resolute than traditional stereo audio"}, {"start": 351.0, "text": "or even 5.1 surround sound."}, {"start": 354.0, "text": "Apple Spatial Audio Format, or ASAF mixes,"}, {"start": 357.0, "text": "can contain 64 or more channels, in order to immerse the audience"}, {"start": 361.0, "text": "in a rich spatial audio experience."}, {"start": 364.0, "text": "These massive formats, and quality requirements - ripple through every part"}, {"start": 368.0, "text": "of the production pipeline."}, {"start": 370.0, "text": "Unfortunately, not all the traditional tools, transport methods, and formats,"}, {"start": 375.0, "text": "support media at this scale."}, {"start": 377.0, "text": "So, immersive live requires building an entirely different workflow."}, {"start": 383.0, "text": "I'll break down three key concepts that will enable you"}, {"start": 386.0, "text": "to begin building tools and exciting new workflows in the ecosystem!"}, {"start": 390.0, "text": "First, a media format standard - designed to deliver the required quality"}, {"start": 395.0, "text": "while remaining efficient, and practically useful,"}, {"start": 397.0, "text": "when building live production tools."}, {"start": 400.0, "text": "Second, a method for transporting immersive content in real time,"}, {"start": 404.0, "text": "between production devices, over an important standard,"}, {"start": 407.0, "text": "called SMPTE 2110."}, {"start": 410.0, "text": "Finally, saving a live stream to file, and playing it out again,"}, {"start": 413.0, "text": "is a fundamental function within any broadcast."}, {"start": 417.0, "text": "I'll go over how live streams from devices can be saved to file"}, {"start": 420.0, "text": "within an application, and played back out again without any compromise to quality."}, {"start": 425.0, "text": "I'll start at the top."}, {"start": 427.0, "text": "Within any production, there are three classes of devices."}, {"start": 431.0, "text": "Devices that output media, for example, a camera, a microphone,"}, {"start": 435.0, "text": "or a graphics generator."}, {"start": 438.0, "text": "Devices that ingest media such as a video encoder,"}, {"start": 440.0, "text": "or a color grading monitor."}, {"start": 443.0, "text": "And devices that do both, for example, a video switcher,"}, {"start": 446.0, "text": "receiving camera sources on its inputs, and switching the different angles to air,"}, {"start": 450.0, "text": "on its outputs!"}, {"start": 452.0, "text": "In any workflow, all devices have to agree on a unified set of media formats,"}, {"start": 458.0, "text": "so they can exchange content between them through the media router,"}, {"start": 461.0, "text": "seamlessly, in real time."}, {"start": 463.0, "text": "Like a common language."}, {"start": 465.0, "text": "To do that, three existing standards have been combined into one format"}, {"start": 469.0, "text": "to support live immersive production."}, {"start": 473.0, "text": "Apple Immersive Live Video is composed entirely of streamed ProRes frames,"}, {"start": 478.0, "text": "as opposed to uncompressed video frames, typical of regular broadcast cameras."}, {"start": 483.0, "text": "ProRes is a powerful video codec that strikes an exceptional balance"}, {"start": 487.0, "text": "between image quality and bandwidth, reducing video signals to a practical size"}, {"start": 492.0, "text": "that can be processed by tools, but maintains the high fidelity required"}, {"start": 496.0, "text": "of the image."}, {"start": 498.0, "text": "And because Apple Silicon is optimized for ProRes processing,"}, {"start": 501.0, "text": "it's the perfect platform for building production tools and pipelines!"}, {"start": 505.0, "text": "To learn more, refer to the \"Apple ProRes\" developer documentation."}, {"start": 511.0, "text": "ASAF audio mixes are composed of standard, uncompressed PCM audio tracks"}, {"start": 516.0, "text": "carrying high-order ambisonic beds and spatial audio objects."}, {"start": 521.0, "text": "Metadata is delivered as per-frame JSON objects that contain elements"}, {"start": 525.0, "text": "describing attributes of related video and audio feeds -"}, {"start": 529.0, "text": "such as lens calibrations, creative events, spatial audio behavior,"}, {"start": 534.0, "text": "and much more."}, {"start": 536.0, "text": "Together these three standards define the live, immersive, production format."}, {"start": 542.0, "text": "All tools and processes must be compliant with each media type -"}, {"start": 545.0, "text": "to ensure interoperability with the rest of the ecosystem!"}, {"start": 549.0, "text": "Next, devices need a standardized transport layer, to exchange live video,"}, {"start": 554.0, "text": "audio, and metadata feeds between them."}, {"start": 557.0, "text": "To achieve this, live feeds from devices are exchanged"}, {"start": 560.0, "text": "as individual SMPTE 2110 media streams —"}, {"start": 563.0, "text": "the industry standard, for professional media transport over IP."}, {"start": 567.0, "text": "\"2110\", as it's commonly known,"}, {"start": 569.0, "text": "is widely deployed across broadcast facilities worldwide,"}, {"start": 572.0, "text": "and is interoperable with a broad ecosystem of professional tools."}, {"start": 576.0, "text": "2110 uses multicast RTP or, Real-time Transport Protocol"}, {"start": 581.0, "text": "to move media across the network."}, {"start": 583.0, "text": "RTP streams carry timing information user flags,"}, {"start": 587.0, "text": "and other metadata, alongside a main media payload."}, {"start": 591.0, "text": "A 2110 stream will transmit either a video,"}, {"start": 593.0, "text": "an audio, or a metadata payload -"}, {"start": 596.0, "text": "and the transport of each media type"}, {"start": 598.0, "text": "is defined by a sub-standard within the broader 2110 specification."}, {"start": 602.0, "text": "I'll break down how each fits into that model."}, {"start": 605.0, "text": "Immersive ProRes video"}, {"start": 607.0, "text": "is transported as 2110-22 streams on the network,"}, {"start": 611.0, "text": "the defined standard for compressed media over IP."}, {"start": 615.0, "text": "This 2110-22 flow contains both the left and right eye"}, {"start": 620.0, "text": "of the immersive content transmitted as two separate data essences,"}, {"start": 624.0, "text": "but contained within the single stream."}, {"start": 627.0, "text": "This means there is no need to frame pack each eye,"}, {"start": 630.0, "text": "side by side into a single image raster"}, {"start": 632.0, "text": "or produce separate IP streams per eye."}, {"start": 636.0, "text": "This is hugely advantageous, as this eliminates the complex management"}, {"start": 640.0, "text": "of independent left and right eye video feeds within the production architecture."}, {"start": 646.0, "text": "ASAF Audio is transported as standard, 2110-30 streams on the network."}, {"start": 652.0, "text": "These contain the high order ambisonics and audio object channels that compose"}, {"start": 656.0, "text": "ASAF spatial audio mixes."}, {"start": 660.0, "text": "JSON objects are transmitted per-frame over 2110-41,"}, {"start": 666.0, "text": "the standard for the transport of user defined metadata over IP."}, {"start": 672.0, "text": "This carries the important metadata information —"}, {"start": 674.0, "text": "like lens calibrations, creative events, and motion data —"}, {"start": 678.0, "text": "in real time alongside the -22 video and -30 audio feeds"}, {"start": 683.0, "text": "within the production."}, {"start": 685.0, "text": "Lastly, the ability to record feeds, edit them,"}, {"start": 688.0, "text": "and play them back out again -"}, {"start": 690.0, "text": "for example Instant Replay is a crucial part of any live workflow."}, {"start": 695.0, "text": "In traditional 2D workflows, recording live video to file"}, {"start": 699.0, "text": "often introduces visual quality loss"}, {"start": 702.0, "text": "as content is encoded, decoded, and re-encoded many times"}, {"start": 707.0, "text": "throughout a typical workflow."}, {"start": 710.0, "text": "In Apple Immersive Video, even these small reductions in quality"}, {"start": 713.0, "text": "can significantly impact the customer experience"}, {"start": 717.0, "text": "especially as generational loss,"}, {"start": 718.0, "text": "due to multiple cycles of compression and decompression,"}, {"start": 721.0, "text": "compound over time."}, {"start": 724.0, "text": "Fortunately, this is solved by the immersive format:"}, {"start": 727.0, "text": "Everything is already ProRes!"}, {"start": 730.0, "text": "Because live media is natively generated in a file-friendly ProRes payload,"}, {"start": 735.0, "text": "recording to disk requires no additional encode, or decode steps"}, {"start": 738.0, "text": "as the content moves through the workflow."}, {"start": 741.0, "text": "The same ProRes frames are simply copied directly into MOV files,"}, {"start": 745.0, "text": "and read back out again, into live 2110 streams during playout — untouched."}, {"start": 750.0, "text": "Practically, this means that live content can be produced by a camera,"}, {"start": 755.0, "text": "transported between devices, recorded to disk,"}, {"start": 758.0, "text": "edited, and played back out live on repeat"}, {"start": 762.0, "text": "with no impact to quality throughout the entire process!"}, {"start": 767.0, "text": "Save video feeds to QuickTime MOV video tracks,"}, {"start": 770.0, "text": "using AVFoundation's AVAssetWriter."}, {"start": 773.0, "text": "The resulting MOV file, now contains the same untouched resolution, framerate,"}, {"start": 778.0, "text": "and stereo image data as the live stream,"}, {"start": 781.0, "text": "saved to a file that can be used in editorial,"}, {"start": 783.0, "text": "replay, or post production."}, {"start": 787.0, "text": "When writing the MOV video track,"}, {"start": 789.0, "text": "it's important to set the constant kVTProjectionKind_AppleImmersiveVideo"}, {"start": 794.0, "text": "in the AVVideoCompressionPropertiesKey."}, {"start": 797.0, "text": "This is a new VideoToolbox property"}, {"start": 799.0, "text": "and will add the correct video extended usage, or vexu,"}, {"start": 802.0, "text": "static metadata for Apple Immersive Video to the file,"}, {"start": 806.0, "text": "signaling it as immersive to other applications."}, {"start": 810.0, "text": "Save audio feeds in the usual way —"}, {"start": 812.0, "text": "uncompressed PCM carried in the 2110 stream"}, {"start": 815.0, "text": "is written directly into the MOV's audio tracks"}, {"start": 818.0, "text": "using AVAssetWriter."}, {"start": 821.0, "text": "Finally, the streamed JSON data"}, {"start": 823.0, "text": "is written into the Metadata, Box, Exchange format,"}, {"start": 826.0, "text": "or MEBX tracks, within the MOV container, using AVAssetWriter."}, {"start": 831.0, "text": "Prior to storage, the streamed JSON data must be deserialized, parsed"}, {"start": 836.0, "text": "and then Immersive Media Support framework —"}, {"start": 838.0, "text": "or IMS — is used to create lens calibration objects,"}, {"start": 842.0, "text": "camera IDs, and other metadata objects that are written into the MOV,"}, {"start": 846.0, "text": "synchronized with the video and audio."}, {"start": 849.0, "text": "IMS was first introduced in visionOS 26."}, {"start": 854.0, "text": "It enables reading and writing the essential metadata"}, {"start": 856.0, "text": "for Apple Immersive Video,"}, {"start": 858.0, "text": "and provides capabilities for previewing content in creative workflows."}, {"start": 863.0, "text": "While immersive video and audio is already powered by well known technologies —"}, {"start": 867.0, "text": "like AVFoundation, VideoToolbox, and Core Audio —"}, {"start": 871.0, "text": "IMS is a powerful framework,"}, {"start": 873.0, "text": "purpose-built for Apple Immersive Video."}, {"start": 877.0, "text": "As you're building the next generation of production tools,"}, {"start": 879.0, "text": "IMS will be one of the most important frameworks to understand!"}, {"start": 884.0, "text": "Check out the \"Immersive Media Support\" developer documentation to learn more."}, {"start": 889.0, "text": "Then, during a file playback scenario, all processes are reversed."}, {"start": 893.0, "text": "Video, audio, and metadata media types,"}, {"start": 897.0, "text": "are read directly from their tracks within the MOV,"}, {"start": 900.0, "text": "and retransmitted back into 2110 output streams"}, {"start": 903.0, "text": "for use within the wider production —"}, {"start": 905.0, "text": "using all the same frameworks and libraries."}, {"start": 908.0, "text": "Now that you understand the foundations of live immersive production,"}, {"start": 911.0, "text": "it's the perfect time to get started!"}, {"start": 914.0, "text": "Build your own immersive tools, using frameworks like AVFoundation,"}, {"start": 918.0, "text": "VideoToolbox, AudioToolbox, and Immersive Media Support."}, {"start": 922.0, "text": "Every layer of the stack is open for innovation."}, {"start": 927.0, "text": "Dive deeper into the world of 2110"}, {"start": 929.0, "text": "and connect your tools together into a true live workflow."}, {"start": 932.0, "text": "Level up, by visiting the SMPTE website"}, {"start": 935.0, "text": "to learn more about the various standards and best practices"}, {"start": 938.0, "text": "during network implementation."}, {"start": 940.0, "text": "Finally, be sure to watch:"}, {"start": 942.0, "text": "\"Learn about Apple Immersive Video technologies\""}, {"start": 945.0, "text": "and \"Support immersive video playback in visionOS apps\"."}, {"start": 948.0, "text": "Together, they provide valuable context"}, {"start": 951.0, "text": "around the creative and technical principles behind the format."}, {"start": 955.0, "text": "The future of immersive live has just started."}, {"start": 958.0, "text": "While it builds on the foundations of traditional broadcast"}, {"start": 961.0, "text": "it introduces entirely new creative and technical possibilities!"}, {"start": 965.0, "text": "And many of the best ideas haven't been invented yet!"}, {"start": 969.0, "text": "This is your chance to be a part of getting this new format"}, {"start": 971.0, "text": "off the ground, and on air."}, {"start": 973.0, "text": "I'll see you next time!"}]}