{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-286", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/286", "wordCount": 1960, "segments": [{"start": 7.0, "text": "Hi I'm Adrian and I'm an engineer at Apple."}, {"start": 11.0, "text": "visionOS allows you to bring people to new worlds"}, {"start": 14.0, "text": "and create rich spatial applications."}, {"start": 17.0, "text": "Some existing spatial apps require an external device, such as a PC."}, {"start": 22.0, "text": "Or they're built with alternative technologies like OpenXR."}, {"start": 26.0, "text": "Now there's a new way to enhance these apps for Apple Vision Pro."}, {"start": 31.0, "text": "It's called Foveated Streaming!"}, {"start": 35.0, "text": "Foveated Streaming helps Apple Vision Pro connect to external devices,"}, {"start": 39.0, "text": "like a PC,"}, {"start": 40.0, "text": "to stream OpenXR content."}, {"start": 43.0, "text": "visionOS automatically sends input data,"}, {"start": 46.0, "text": "like hands, controller positions, and microphone."}, {"start": 50.0, "text": "And the device streams your OpenXR content as video and audio."}, {"start": 55.0, "text": "We introduced the Foveated Streaming framework in visionOS 26.4"}, {"start": 60.0, "text": "and developers have already created some amazing experiences with it."}, {"start": 64.0, "text": "For example, X-Plane 12 from Laminar Research"}, {"start": 67.0, "text": "brings a premium flight simulation experience to Apple Vision Pro."}, {"start": 71.0, "text": "The X-Plane app on visionOS uses ARKit to understand your space and equipment,"}, {"start": 76.0, "text": "and streams the simulated experience from a PC."}, {"start": 79.0, "text": "It feels incredible to interact with a physical flight simulator"}, {"start": 83.0, "text": "while fully immersed in the virtual world."}, {"start": 85.0, "text": "You can look around the cockpit, or out the side windows."}, {"start": 89.0, "text": "visionOS seamlessly blends the real and the virtual"}, {"start": 92.0, "text": "to deliver an immersive and effective simulation."}, {"start": 96.0, "text": "iRacing, a motorsport racing game for PC,"}, {"start": 100.0, "text": "delivers an incredible sim racing experience on Apple Vision Pro."}, {"start": 105.0, "text": "The iRacing Connect app uses ARKit hand tracking"}, {"start": 108.0, "text": "to match the position of your physical racing wheel with the virtual cockpit."}, {"start": 112.0, "text": "So, you can see your hands holding the virtual wheel!"}, {"start": 116.0, "text": "And Innoactive brings Autodesk VRED to Apple Vision Pro."}, {"start": 119.0, "text": "This enables designers and engineers"}, {"start": 121.0, "text": "to visualize massive 3D assets and simulations."}, {"start": 126.0, "text": "In this video, designers at Kia use this technology"}, {"start": 129.0, "text": "to see their cars at one-to-one scale,"}, {"start": 132.0, "text": "and in rigorous detail."}, {"start": 134.0, "text": "So, Foveated Streaming already enables some fantastic experiences"}, {"start": 137.0, "text": "on Apple Vision Pro."}, {"start": 139.0, "text": "And the video quality is incredible,"}, {"start": 141.0, "text": "because our system intelligently optimizes the video stream."}, {"start": 146.0, "text": "Foveated Streaming compresses video based on where a person is looking."}, {"start": 151.0, "text": "We use the eye tracking technology in Apple Vision Pro"}, {"start": 154.0, "text": "to stream things you're focused on in higher detail."}, {"start": 158.0, "text": "This sophisticated stream processing is built in to visionOS,"}, {"start": 162.0, "text": "so we've done the heavy lifting for you."}, {"start": 165.0, "text": "It all happens so quickly that you don't even notice!"}, {"start": 168.0, "text": "Under the hood, visionOS comes with NVIDIA CloudXR™ streaming, built-in."}, {"start": 174.0, "text": "This means that your immersive content is streamed"}, {"start": 176.0, "text": "with a high performance streaming protocol and with minimal latency."}, {"start": 181.0, "text": "CloudXR™ is so performant that you can stream immersive content"}, {"start": 184.0, "text": "without any cables attached, over Wi-Fi."}, {"start": 187.0, "text": "It can stream content from a local PC over your home network,"}, {"start": 191.0, "text": "or from the cloud."}, {"start": 193.0, "text": "Your apps can also integrate with native visionOS frameworks."}, {"start": 197.0, "text": "So you can build user interfaces with SwiftUI,"}, {"start": 200.0, "text": "you can mix your content and the real world with ARKit,"}, {"start": 203.0, "text": "and you can combine on-device RealityKit rendering"}, {"start": 206.0, "text": "with your streamed content."}, {"start": 208.0, "text": "It's super easy to use."}, {"start": 210.0, "text": "We've found that you can start streaming your OpenXR application in just one day."}, {"start": 216.0, "text": "And in a week, you can enhance your application"}, {"start": 218.0, "text": "with capabilities that you can only find on visionOS."}, {"start": 222.0, "text": "So let's talk all about it!"}, {"start": 224.0, "text": "First, I'll describe how Foveated Streaming works,"}, {"start": 226.0, "text": "and show you how to get started in just a few hours."}, {"start": 230.0, "text": "Next, we will set up a visionOS app to display the streamed content."}, {"start": 235.0, "text": "Then we'll set up your OpenXR client so that it can stream to Apple Vision Pro."}, {"start": 240.0, "text": "And finally, I'll talk about how you can enhance your application with capabilities"}, {"start": 245.0, "text": "only possible on visionOS."}, {"start": 248.0, "text": "So let's get started!"}, {"start": 250.0, "text": "Here is a block diagram that outlines the frameworks you will build on."}, {"start": 254.0, "text": "Both for visionOS and for your streaming endpoint."}, {"start": 257.0, "text": "Your OpenXR app should implement Apple's Foveated Streaming Protocol,"}, {"start": 262.0, "text": "which manages pairing with the device."}, {"start": 265.0, "text": "Also, your app should use the OpenXR runtime"}, {"start": 269.0, "text": "provided by the NVIDIA CloudXR™ SDK."}, {"start": 272.0, "text": "On visionOS, your app uses the FoveatedStreaming framework"}, {"start": 276.0, "text": "to connect to the streaming endpoint."}, {"start": 279.0, "text": "And you can integrate with other visionOS frameworks"}, {"start": 281.0, "text": "such as ARKit, SwiftUI, and RealityKit."}, {"start": 286.0, "text": "Let's begin by setting up your Streaming endpoint."}, {"start": 289.0, "text": "We've provided an open source,"}, {"start": 291.0, "text": "end to end example on Apple's Github page."}, {"start": 294.0, "text": "Our windows sample code contains a reference implementation"}, {"start": 297.0, "text": "of Apple's Foveated Streaming Protocol."}, {"start": 301.0, "text": "We have also provided an example OpenXR application"}, {"start": 304.0, "text": "and helpful guides to help you set up the NVIDIA CloudXR™ runtime."}, {"start": 309.0, "text": "So to get started, check out our Github page."}, {"start": 312.0, "text": "And feel free to copy our reference implementation!"}, {"start": 315.0, "text": "This can help you set everything up in an afternoon."}, {"start": 318.0, "text": "Next, let's talk about how to build a visionOS receiver app."}, {"start": 324.0, "text": "The visionOS app you create welcomes people into your experience."}, {"start": 329.0, "text": "It uses the FoveatedStreaming.framework to connect to streaming endpoints."}, {"start": 334.0, "text": "And most importantly, it adds unique features to your experience"}, {"start": 338.0, "text": "by leveraging the great frameworks in visionOS."}, {"start": 342.0, "text": "To get started, download our visionOS sample on developer.apple.com."}, {"start": 347.0, "text": "You can use our sample code,"}, {"start": 349.0, "text": "both for visionOS and for Windows,"}, {"start": 351.0, "text": "to get your application streaming in an afternoon."}, {"start": 355.0, "text": "For the remainder of the talk, I'll dig into how these samples work."}, {"start": 359.0, "text": "To begin, let's talk more about the receiver app."}, {"start": 363.0, "text": "The Foveated Streaming framework uses a session-based API."}, {"start": 367.0, "text": "So your app should create a FoveatedStreamingSession."}, {"start": 371.0, "text": "When you call connect,"}, {"start": 373.0, "text": "the framework will automatically present a list of endpoints"}, {"start": 376.0, "text": "your app can connect to."}, {"start": 378.0, "text": "Apple Vision Pro must pair with your endpoint"}, {"start": 381.0, "text": "before beginning the stream."}, {"start": 382.0, "text": "To do this, the endpoint presents a QR code with pairing information."}, {"start": 387.0, "text": "We'll talk more about how to properly present this later."}, {"start": 391.0, "text": "When the code is on screen,"}, {"start": 392.0, "text": "the framework automatically presents a user interface to scan."}, {"start": 396.0, "text": "You scan the code by looking at it."}, {"start": 399.0, "text": "Once you're connected, it's time to present the streamed content."}, {"start": 403.0, "text": "You can do this with SwiftUI."}, {"start": 405.0, "text": "On visionOS, you present spatial content with an ImmersiveSpace."}, {"start": 410.0, "text": "If you pass a FoveatedStreamingSession to your ImmersiveSpace,"}, {"start": 413.0, "text": "it will include the streamed content."}, {"start": 417.0, "text": "You can add additional windows,"}, {"start": 419.0, "text": "just like any other SwiftUI application."}, {"start": 422.0, "text": "For example, our sample app adds a window to the scene"}, {"start": 425.0, "text": "to pause and resume the session."}, {"start": 428.0, "text": "It also adds views to the ImmersiveSpace itself,"}, {"start": 431.0, "text": "like this widget to re-open the main window."}, {"start": 435.0, "text": "And finally, it configures the immersive space"}, {"start": 437.0, "text": "to have a progressive immersion style."}, {"start": 440.0, "text": "Progressive immersion is a great fit for Foveated Streaming."}, {"start": 444.0, "text": "It allows people to view your experience through a portal,"}, {"start": 448.0, "text": "grounded in their physical environment."}, {"start": 450.0, "text": "In fact, you have access to all of SwiftUI's features"}, {"start": 454.0, "text": "when using Foveated Streaming."}, {"start": 456.0, "text": "So your streaming client can use native spatial gestures,"}, {"start": 459.0, "text": "and the visionOS look and feel."}, {"start": 462.0, "text": "You can add SwiftUI windows to your app, including volumetric windows."}, {"start": 467.0, "text": "And you can use immersion styles, like progressive immersion."}, {"start": 470.0, "text": "If you're new to SwiftUI,"}, {"start": 472.0, "text": "there are a ton of resources to help you get started."}, {"start": 475.0, "text": "I like the talk \"Get started with building apps for spatial computing\" from WWDC23."}, {"start": 482.0, "text": "Now let's look at what your streaming endpoint needs to do"}, {"start": 485.0, "text": "to integrate with Foveated Streaming."}, {"start": 488.0, "text": "I mentioned before that your OpenXR app needs to do two things."}, {"start": 492.0, "text": "First, it should implement the Foveated Streaming Protocol,"}, {"start": 496.0, "text": "which handles authentication and pairing."}, {"start": 499.0, "text": "Second, it should use the OpenXR runtime provided by the NVIDIA CloudXR™ SDK."}, {"start": 506.0, "text": "Let's begin by talking about the Foveated Streaming Protocol."}, {"start": 510.0, "text": "This is a lightweight, TCP-based connection"}, {"start": 513.0, "text": "which is established separately from the streaming connection."}, {"start": 517.0, "text": "It helps to authenticate the secure foveated stream."}, {"start": 521.0, "text": "It also communicates session state between visionOS and the endpoint."}, {"start": 526.0, "text": "For details on the protocol format, see our article on developer.apple.com."}, {"start": 532.0, "text": "We've also provided a reference implementation of the protocol"}, {"start": 535.0, "text": "on our Github page,"}, {"start": 536.0, "text": "so you can use that to get started."}, {"start": 539.0, "text": "For example, here's what happens when your app calls connect()."}, {"start": 544.0, "text": "Endpoints become visible to your local network with Bonjour."}, {"start": 549.0, "text": "When someone selects it, visionOS establishes a connection."}, {"start": 554.0, "text": "Then, pairing occurs."}, {"start": 556.0, "text": "And once that's done, the stream begins, and connect() returns."}, {"start": 561.0, "text": "Let's dive in to the specific messages that are sent"}, {"start": 564.0, "text": "and received during barcode pairing."}, {"start": 567.0, "text": "Messages in the Foveated Streaming Protocol are JSON-encoded."}, {"start": 572.0, "text": "The protocol uses a request-acknowledge pattern."}, {"start": 576.0, "text": "First, visionOS requests a connection,"}, {"start": 578.0, "text": "and if unpaired, it requests a pairing barcode."}, {"start": 582.0, "text": "The barcode is also JSON-encoded."}, {"start": 586.0, "text": "It contains two things:"}, {"start": 588.0, "text": "a client token,"}, {"start": 589.0, "text": "and a hash of the secure connection's certificate."}, {"start": 592.0, "text": "Both of these are provided by the NVIDIA CloudXR™ SDK."}, {"start": 599.0, "text": "visionOS will update your endpoint on the session status."}, {"start": 604.0, "text": "And once the endpoint reports the content is ready, streaming begins."}, {"start": 609.0, "text": "It's important that you monitor the session status"}, {"start": 612.0, "text": "to have a stable user experience."}, {"start": 615.0, "text": "For example, when you take off the device,"}, {"start": 617.0, "text": "it goes to sleep."}, {"start": 619.0, "text": "Just before, it informs your endpoint that the streaming session is paused."}, {"start": 625.0, "text": "While asleep, all connections are severed."}, {"start": 629.0, "text": "You should keep the endpoint available so that someone can reconnect"}, {"start": 632.0, "text": "when they put the device back on."}, {"start": 635.0, "text": "Now, let's talk about how NVIDIA CloudXR™ integrates with your OpenXR client."}, {"start": 640.0, "text": "NVIDIA CloudXR™ provides an OpenXR runtime for windows."}, {"start": 645.0, "text": "Your OpenXR application automatically connects to this runtime,"}, {"start": 649.0, "text": "and CloudXR™ handles the streaming details."}, {"start": 653.0, "text": "CloudXR provides visionOS input data to OpenXR automatically."}, {"start": 658.0, "text": "For example, you can use the OpenXR extension for hand tracking."}, {"start": 663.0, "text": "PlayStation VR2 Sense Controllers also pass through."}, {"start": 668.0, "text": "We recommend using a depth buffer for the best experience."}, {"start": 672.0, "text": "Also, provide an alpha channel to mix your content with the user's surroundings."}, {"start": 677.0, "text": "For more information, see our article on developer.apple.com."}, {"start": 682.0, "text": "You can also consult our sample code on Github"}, {"start": 684.0, "text": "to understand how to set up NVIDIA's runtime."}, {"start": 688.0, "text": "Now that you have everything set up,"}, {"start": 689.0, "text": "let's talk about how to evaluate"}, {"start": 691.0, "text": "the performance of your streamed experience."}, {"start": 694.0, "text": "We've provided an instrument in Xcode to measure statistics of the stream."}, {"start": 700.0, "text": "The Foveated Streaming instrument"}, {"start": 702.0, "text": "informs you about the stream's bandwidth,"}, {"start": 704.0, "text": "pose latency, frame rate, and more."}, {"start": 707.0, "text": "You can use this to diagnose issues with the streamed content."}, {"start": 712.0, "text": "To learn more, see our article on developer.apple.com."}, {"start": 716.0, "text": "Now that you have your streaming client up and running,"}, {"start": 718.0, "text": "I'll talk about some ways that you can enhance the experience"}, {"start": 721.0, "text": "with features you can only find on visionOS."}, {"start": 726.0, "text": "As we've seen, your streaming experience is composed of two apps:"}, {"start": 730.0, "text": "the receiver app on visionOS"}, {"start": 733.0, "text": "and the host app on your streaming endpoint."}, {"start": 737.0, "text": "The Foveated Streaming framework allows them"}, {"start": 739.0, "text": "to communicate with the message channels API."}, {"start": 744.0, "text": "On visionOS, message channels are an API on FoveatedStreamingSession."}, {"start": 749.0, "text": "And CloudXR provides an OpenXR extension for message channels."}, {"start": 755.0, "text": "These messages are completely Opaque Data blobs,"}, {"start": 758.0, "text": "so you can send whatever you want!"}, {"start": 761.0, "text": "For example, you can present a SwiftUI interface to select a level in your game,"}, {"start": 766.0, "text": "and send a command to the OpenXR application"}, {"start": 768.0, "text": "to initiate the load."}, {"start": 771.0, "text": "And while it's loading,"}, {"start": 772.0, "text": "the OpenXR app can report its progress."}, {"start": 775.0, "text": "Or, your visionOS app can send ARKit data,"}, {"start": 778.0, "text": "to align the scene in the users's real space."}, {"start": 782.0, "text": "X-Plane 12 is a great example of this."}, {"start": 786.0, "text": "The visionOS app uses ARKit"}, {"start": 788.0, "text": "to find the location of a physical flight simulator."}, {"start": 791.0, "text": "It uses message channels to sync this to the PC,"}, {"start": 794.0, "text": "so the real and the virtual are perfectly aligned."}, {"start": 798.0, "text": "It's easy to use ARKit and message channels"}, {"start": 800.0, "text": "to enhance your OpenXR experience."}, {"start": 803.0, "text": "FoveatedStreamingSession offers an API to convert between OpenXR"}, {"start": 807.0, "text": "and ARKit coordinate frames."}, {"start": 809.0, "text": "As a reminder,"}, {"start": 810.0, "text": "controller and hand tracking are already built in!"}, {"start": 813.0, "text": "And use an alpha channel,"}, {"start": 814.0, "text": "to blend your content with a person's environment."}, {"start": 817.0, "text": "You can also compose your streamed content with native RealityKit rendering!"}, {"start": 822.0, "text": "To do this, simply add a RealityView to your ImmersiveSpace."}, {"start": 826.0, "text": "RealityKit content seamlessly composes with your streamed content."}, {"start": 830.0, "text": "And if you supply depth to your OpenXR scene,"}, {"start": 833.0, "text": "the content will even occlude each other!"}, {"start": 836.0, "text": "And that concludes our session."}, {"start": 838.0, "text": "I invite you to download and review our sample code,"}, {"start": 841.0, "text": "which makes it super easy for you to get started."}, {"start": 844.0, "text": "Try to set up your own receiver app,"}, {"start": 846.0, "text": "and integrate your OpenXR client with our protocol."}, {"start": 850.0, "text": "Have fun, and enjoy the rest of WWDC!"}]}