{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-287", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/287", "wordCount": 4338, "segments": [{"start": 7.0, "text": "Hi, my name is Norman,"}, {"start": 8.0, "text": "and I'm a director in the Vision Products Group."}, {"start": 11.0, "text": "In this session, I'll take you through powerful new ways"}, {"start": 13.0, "text": "to build next-generation experiences with visionOS 27."}, {"start": 17.0, "text": "Looking back over this past year,"}, {"start": 20.0, "text": "we've been blown away by what you've built on the platform."}, {"start": 23.0, "text": "We've seen apps like YouTube, introducing world-class content"}, {"start": 26.0, "text": "in your living room,"}, {"start": 28.0, "text": "Valve, bringing Steam Link to Apple Vision Pro"}, {"start": 31.0, "text": "to seamlessly stream your 2D Mac and PC games on the infinite canvas,"}, {"start": 36.0, "text": "and Resolution Games, reimagining new ways"}, {"start": 39.0, "text": "to relive classic experiences."}, {"start": 42.0, "text": "We've also seen companies like Kia and Innoactive"}, {"start": 46.0, "text": "using Autodesk VRED, Nvidia's CloudXR SDK,"}, {"start": 50.0, "text": "and the new Foveated Streaming framework"}, {"start": 52.0, "text": "to design next generation vehicles,"}, {"start": 55.0, "text": "and Laminar Research,"}, {"start": 56.0, "text": "blending professionals' digital and physical worlds"}, {"start": 60.0, "text": "to deliver powerfully immersive training solutions with X-Plane."}, {"start": 64.0, "text": "Bringing these experiences to life is just the beginning,"}, {"start": 67.0, "text": "and the insights you've shared along the way have been invaluable."}, {"start": 71.0, "text": "Your feedback continues to shape the future of visionOS,"}, {"start": 75.0, "text": "guiding the features we release"}, {"start": 77.0, "text": "and accelerating the next era of computing."}, {"start": 80.0, "text": "Today, people aren't just working faster,"}, {"start": 83.0, "text": "they're inventing entirely new experiences and workflows."}, {"start": 87.0, "text": "To make these breakthroughs possible,"}, {"start": 89.0, "text": "the latest generation of Apple Vision Pro is powered by the M5 chip —"}, {"start": 94.0, "text": "delivering desktop-class compute for real-time rendering."}, {"start": 98.0, "text": "Ultra high-resolution displays provide over 4K pixels per eye,"}, {"start": 102.0, "text": "enabling stunning visual fidelity."}, {"start": 105.0, "text": "The system tracks your hand movements at 90Hz"}, {"start": 107.0, "text": "for highly responsive, low-latency control."}, {"start": 111.0, "text": "And with deep integration across the Apple ecosystem,"}, {"start": 115.0, "text": "your apps and games feel seamlessly connected across all your Apple devices."}, {"start": 120.0, "text": "Along with powerful hardware and platform capabilities,"}, {"start": 124.0, "text": "visionOS provides multiple ways to render your app's content…"}, {"start": 128.0, "text": "In the Shared Space, you can choose to render your app"}, {"start": 131.0, "text": "in a window or a volume,"}, {"start": 133.0, "text": "where your experience can coexist simultaneously with other applications."}, {"start": 139.0, "text": "This allows you to organize your workspace across the infinite canvas,"}, {"start": 143.0, "text": "taking productivity to the next level."}, {"start": 147.0, "text": "Or, you could launch an Immersive Space where your app runs exclusively,"}, {"start": 152.0, "text": "allowing you to render 2D and 3D content anywhere in your field of view."}, {"start": 158.0, "text": "In the Immersive Space,"}, {"start": 159.0, "text": "your content blends with the physical world,"}, {"start": 162.0, "text": "providing three immersion styles to choose from:"}, {"start": 165.0, "text": "The Mixed immersion style allows you"}, {"start": 167.0, "text": "to anchor 3D objects in real-world surroundings,"}, {"start": 171.0, "text": "like placing an object on a table."}, {"start": 173.0, "text": "The Progressive style lets your audience dial in the level of immersion"}, {"start": 178.0, "text": "that feels right for them."}, {"start": 180.0, "text": "Or you can enable Full immersion,"}, {"start": 183.0, "text": "to transport them into a completely virtual world."}, {"start": 187.0, "text": "Together, these scene types offer a spectrum of immersion for your content."}, {"start": 192.0, "text": "When building experiences on visionOS, there are three main paths to choose from,"}, {"start": 197.0, "text": "depending on your workflow."}, {"start": 199.0, "text": "The first path is for existing iOS or iPadOS apps."}, {"start": 204.0, "text": "If you already have an app or a game running on iPad or iPhone,"}, {"start": 209.0, "text": "it can very likely run on Apple Vision Pro today,"}, {"start": 212.0, "text": "with minimal changes to your code."}, {"start": 215.0, "text": "Compatibility is the simplest way to get onto the platform quickly —"}, {"start": 220.0, "text": "you can just check a box in App Store Connect!"}, {"start": 223.0, "text": "Or, you can recompile your iOS or iPadOS app to visionOS"}, {"start": 228.0, "text": "by adding visionOS as a deployment target in your Xcode settings."}, {"start": 232.0, "text": "Both of these paths offer low-friction ways"}, {"start": 235.0, "text": "to bring your experiences to visionOS."}, {"start": 239.0, "text": "The second path is for apps designed from the ground up for spatial computing."}, {"start": 244.0, "text": "This allows your experience"}, {"start": 245.0, "text": "to seamlessly blend with and react to people's environments."}, {"start": 251.0, "text": "In this vertical, you have 2 options:"}, {"start": 254.0, "text": "You can build for the platform using native frameworks"}, {"start": 256.0, "text": "like SwiftUI, RealityKit,"}, {"start": 259.0, "text": "and tools like Reality Composer Pro,"}, {"start": 262.0, "text": "or use your own custom rendering engine with CompositorServices."}, {"start": 266.0, "text": "Or, if you prefer to use a third-party game engine like Unity, Unreal, or Godot,"}, {"start": 272.0, "text": "these all support visionOS as a platform."}, {"start": 276.0, "text": "In addition to these options, we are introducing a third path"}, {"start": 280.0, "text": "that offers a way for you to bring your existing experiences"}, {"start": 283.0, "text": "on macOS or PC, to visionOS."}, {"start": 288.0, "text": "If you have a Mac app that renders spatial content today,"}, {"start": 292.0, "text": "visionOS 27 adds a powerful new tool with the Spatial Preview framework ,"}, {"start": 297.0, "text": "allowing you to extend images, documents, and 3D content"}, {"start": 302.0, "text": "directly from your Mac into Apple Vision Pro."}, {"start": 305.0, "text": "Alternatively, if you are bringing a 3D experience from a PC,"}, {"start": 310.0, "text": "the Foveated Streaming framework lets you display native spatial content"}, {"start": 315.0, "text": "alongside streamed content."}, {"start": 317.0, "text": "For example, a flight simulator crew rendered a highly detailed cockpit"}, {"start": 322.0, "text": "using our native rendering framework, RealityKit,"}, {"start": 325.0, "text": "while streaming a processor-intensive landscape from a remote computer."}, {"start": 330.0, "text": "I'll dive deeper into both of these technologies later in this session."}, {"start": 335.0, "text": "Together, these paths give you flexibility"}, {"start": 339.0, "text": "to build incredible experiences for visionOS,"}, {"start": 342.0, "text": "no matter where you are starting from."}, {"start": 344.0, "text": "I'll share some of the latest updates to these pathways."}, {"start": 348.0, "text": "First, I'll cover new capabilities in RealityKit and Reality Composer Pro"}, {"start": 353.0, "text": "that help you bring rich 3D experiences to life,"}, {"start": 356.0, "text": "with accelerated workflows."}, {"start": 359.0, "text": "I'll share updates on third party game engines like Unity, Unreal, and Godot."}, {"start": 365.0, "text": "Then, I'll take you through Spatial Preview,"}, {"start": 368.0, "text": "a feature that extends 3D content"}, {"start": 370.0, "text": "from your Mac app, into the immersive world of visionOS."}, {"start": 375.0, "text": "I'll also introduce the new Foveated Streaming framework"}, {"start": 378.0, "text": "that enables you to stream immersive content"}, {"start": 381.0, "text": "from a PC or cloud instance."}, {"start": 385.0, "text": "I'll cover incredible new ways to interact with your content."}, {"start": 389.0, "text": "And finally, I'll share updates to the immersive media pipeline to help you"}, {"start": 394.0, "text": "tell even more engaging stories on Apple Vision Pro."}, {"start": 399.0, "text": "Let's begin with updates to RealityKit and Reality Composer Pro."}, {"start": 404.0, "text": "RealityKit is the rendering engine behind some of the best native experiences"}, {"start": 409.0, "text": "on visionOS."}, {"start": 410.0, "text": "Reality Composer Pro is where you author content visually,"}, {"start": 414.0, "text": "letting you design, iterate, and preview without ever leaving the editor."}, {"start": 420.0, "text": "With visionOS 27 we're bringing the two even closer together."}, {"start": 426.0, "text": "With RealityKit you can build interactive, high-fidelity spatial experiences"}, {"start": 432.0, "text": "that seamlessly blend with your real world"}, {"start": 435.0, "text": "and visionOS 27 adds powerful new features in RealityKit"}, {"start": 440.0, "text": "to make your experiences more immersive than ever."}, {"start": 445.0, "text": "Let's take a look at a few of them: RealityKit's new physical space lighting"}, {"start": 450.0, "text": "deepens immersion by blending your virtual lighting with the real world."}, {"start": 455.0, "text": "Here's a virtual planetarium projector,"}, {"start": 457.0, "text": "built in RealityKit."}, {"start": 459.0, "text": "The stars and nebulae shine across the space, and as the projector spins,"}, {"start": 463.0, "text": "its light seamlessly conforms to every surface."}, {"start": 467.0, "text": "The stars and nebulae are powered by the new Projective Textures API,"}, {"start": 471.0, "text": "enabling you to add textures to your spotlights."}, {"start": 475.0, "text": "With this feature, you can simulate stunning effects"}, {"start": 478.0, "text": "like stained glass projections or underwater caustics."}, {"start": 483.0, "text": "Next, let's look at RealityKit's powerful new Cloth Simulation."}, {"start": 488.0, "text": "Here, a virtual mannequin wears a flowing dress"}, {"start": 492.0, "text": "made of highly realistic, simulated clothing."}, {"start": 495.0, "text": "As the mannequin walks, the fabric moves and folds naturally,"}, {"start": 499.0, "text": "all rendered in real time."}, {"start": 501.0, "text": "That same simulation also brings the virtual bed cover to life."}, {"start": 505.0, "text": "As it's pulled back, the fabric responds with realistic weight and drape."}, {"start": 510.0, "text": "RealityKit doesn't stop at visual realism."}, {"start": 513.0, "text": "With the new Custom Reverb Mesh,"}, {"start": 515.0, "text": "it brings that same authenticity to spatial audio."}, {"start": 519.0, "text": "Take this virtual band playing in a museum."}, {"start": 522.0, "text": "With RealityKit, the sound doesn't just originate in front of me,"}, {"start": 527.0, "text": "it fills the space from every direction."}, {"start": 530.0, "text": "The reverb system accurately simulates how sound is absorbed and scattered"}, {"start": 535.0, "text": "by the wood, metal, and stone in the hall, giving your virtual world an even richer,"}, {"start": 542.0, "text": "more believable sense of presence."}, {"start": 544.0, "text": "Another powerful new feature in RealityKit is Gaussian Splatting."}, {"start": 549.0, "text": "Here's a small potted plant,"}, {"start": 550.0, "text": "scanned and rendered as a 3D Gaussian splat."}, {"start": 553.0, "text": "Every detail comes through, down to the texture of the soil."}, {"start": 558.0, "text": "With RealityKit, you can now scan real-world objects"}, {"start": 561.0, "text": "and bring them straight into your virtual experiences."}, {"start": 565.0, "text": "It's a great way to capture objects that are hard to model by hand"}, {"start": 568.0, "text": "and use them to build incredibly realistic worlds."}, {"start": 572.0, "text": "To learn about even more features coming to RealityKit,"}, {"start": 576.0, "text": "check out the \"Explore advances in RealityKit\" session."}, {"start": 580.0, "text": "While RealityKit APIs enable you to create stunning, interactive 3D content,"}, {"start": 585.0, "text": "the process of putting that content together"}, {"start": 587.0, "text": "is greatly enhanced by Reality Composer Pro."}, {"start": 591.0, "text": "We're adding brand new tools and features in a major refresh,"}, {"start": 594.0, "text": "enabling fast, AI-powered, and collaborative workflows"}, {"start": 598.0, "text": "to the new Reality Composer Pro 3,"}, {"start": 601.0, "text": "so you can move faster and go further, without always needing to touch Xcode."}, {"start": 607.0, "text": "The new Reality Composer Pro brings a huge selection"}, {"start": 611.0, "text": "of new features to transform your workflows."}, {"start": 615.0, "text": "I'll take you through some of these brand-new capabilities,"}, {"start": 618.0, "text": "like Reality Composer Pro Assistant, Animation Graph,"}, {"start": 622.0, "text": "Script Graph, and Navigation Meshes."}, {"start": 625.0, "text": "Let's start with Reality Composer Pro Assistant,"}, {"start": 628.0, "text": "which integrates AI-capabilities into your creative workflow,"}, {"start": 632.0, "text": "right inside the editor."}, {"start": 634.0, "text": "Here's an example that shows how the AI assistant can be used"}, {"start": 638.0, "text": "to generate a dried fruit assortment,"}, {"start": 640.0, "text": "and then place them in a pre-existing empty bowl in the scene."}, {"start": 650.0, "text": "It even generated 3D candles and placed them on the table."}, {"start": 654.0, "text": "With Reality Composer Pro Assistant, you describe what you want,"}, {"start": 658.0, "text": "and it generates 3D models with detailed textures and materials,"}, {"start": 662.0, "text": "ready to place in your scene."}, {"start": 665.0, "text": "It's a great way to start telling your story, long before your final game assets"}, {"start": 669.0, "text": "are ready to ship."}, {"start": 672.0, "text": "Once your final assets and animations are ready,"}, {"start": 675.0, "text": "Animation Graph in Reality Composer Pro"}, {"start": 678.0, "text": "lets you control how they transition between states."}, {"start": 681.0, "text": "With a state machine, you can easily transition between idle and walking states"}, {"start": 686.0, "text": "at runtime,"}, {"start": 687.0, "text": "and visualize that transition live in the editor."}, {"start": 691.0, "text": "And — to help your character navigate around an environment"}, {"start": 695.0, "text": "while avoiding obstacles —"}, {"start": 697.0, "text": "you can generate and tweak a navigation mesh,"}, {"start": 699.0, "text": "right inside Reality Composer Pro!"}, {"start": 702.0, "text": "Simply start with an auto-generated mesh,"}, {"start": 705.0, "text": "as shown in blue, then add additional features such as jumps,"}, {"start": 709.0, "text": "ladders, and obstacles to complete the course."}, {"start": 712.0, "text": "To connect these pieces and make your scene interactive,"}, {"start": 715.0, "text": "Reality Composer Pro also brings Script Graph."}, {"start": 719.0, "text": "With Script Graph, you can add nodes that catch events like taps,"}, {"start": 723.0, "text": "driving where your character should move."}, {"start": 725.0, "text": "You can make real-time edits with live preview on Apple Vision Pro,"}, {"start": 730.0, "text": "all without opening Xcode."}, {"start": 733.0, "text": "Now, with everything in place, your character can find its way"}, {"start": 736.0, "text": "to its destination, avoiding obstacles"}, {"start": 739.0, "text": "and using its full animation set to walk, run, or climb."}, {"start": 745.0, "text": "Reality Composer Pro's node-based editor called Script Graph"}, {"start": 749.0, "text": "lets you build logic without ever leaving your visual workflow."}, {"start": 753.0, "text": "Here, Devs United Games uses a handful of nodes to animate their Aquascape character"}, {"start": 759.0, "text": "as it enters the scene."}, {"start": 761.0, "text": "As the fish swims through the environment, its animation playback even adjusts"}, {"start": 766.0, "text": "to match its exact speed."}, {"start": 769.0, "text": "Shader graph materials in Reality Composer Pro"}, {"start": 772.0, "text": "are also getting a major upgrade."}, {"start": 775.0, "text": "Last year we introduced a system environment"}, {"start": 778.0, "text": "featuring Jupiter's moon Amalthea,"}, {"start": 780.0, "text": "showcasing a subsurface scattering effect"}, {"start": 783.0, "text": "that makes the ice on the moon's surface look strikingly real."}, {"start": 788.0, "text": "Reality Composer Pro 3 now exposes subsurface scattering"}, {"start": 792.0, "text": "through Shader Graph,"}, {"start": 793.0, "text": "so you can bring a similar effect to your own scenes."}, {"start": 797.0, "text": "And with additional shader graph capabilities,"}, {"start": 800.0, "text": "you can now create lifelike skin, eyes, and hair for your characters,"}, {"start": 805.0, "text": "or even craft the otherworldly look of a portal."}, {"start": 809.0, "text": "Reality Composer Pro 3 brings so much more to explore,"}, {"start": 812.0, "text": "including Prototypes, Behavior Trees, Compute Graphs,"}, {"start": 816.0, "text": "and custom script graph nodes."}, {"start": 819.0, "text": "Check out these sessions to learn more."}, {"start": 822.0, "text": "Native frameworks like RealityKit and SwiftUI give you the deepest integration"}, {"start": 827.0, "text": "with visionOS."}, {"start": 828.0, "text": "But if you've already built your game in another engine,"}, {"start": 832.0, "text": "like Unity, Unreal, or Godot,"}, {"start": 834.0, "text": "it's easier than ever to bring it to Apple Vision Pro."}, {"start": 838.0, "text": "Unity has been compatible with visionOS since we first launched."}, {"start": 843.0, "text": "Here's LEGO Builder's Journey that was made with Unity"}, {"start": 847.0, "text": "and runs in a volume."}, {"start": 850.0, "text": "If you have a Unity Pro license,"}, {"start": 852.0, "text": "you can bring your Unity game to visionOS."}, {"start": 855.0, "text": "Windowed games made with Unity use RealityKit for native rendering;"}, {"start": 860.0, "text": "while immersive games on visionOS can be rendered either with RealityKit"}, {"start": 865.0, "text": "or CompositorServices framework, depending on your rendering needs."}, {"start": 869.0, "text": "We've released plugins to support spatial accessories"}, {"start": 872.0, "text": "such as the PSVR 2 Sense controller"}, {"start": 875.0, "text": "so that you can design tactile interactions for your Unity apps and games."}, {"start": 881.0, "text": "Unreal Engine is also available in immersive mode."}, {"start": 884.0, "text": "And Polyarc, the developer of Glassbreakers,"}, {"start": 888.0, "text": "brought their Unreal engine game to Apple Vision Pro"}, {"start": 891.0, "text": "with static foveation, for noticeably sharper visuals."}, {"start": 896.0, "text": "Games built with Godot, like DogWalk from Blender Studios,"}, {"start": 900.0, "text": "also run on Apple Vision Pro."}, {"start": 903.0, "text": "We've also added support for Godot rendering through CompositorServices,"}, {"start": 907.0, "text": "along with a plugin for rendering with RealityKit."}, {"start": 911.0, "text": "And we've published a PHASE audio plugin,"}, {"start": 914.0, "text": "so your Godot games can take full advantage of Apple's spatial audio."}, {"start": 919.0, "text": "You can download these game engine plugins from our GitHub page."}, {"start": 924.0, "text": "visionOS also supports experiences built on custom rendering engines."}, {"start": 930.0, "text": "With the CompositorServices framework, you can connect your proprietary engine"}, {"start": 935.0, "text": "to the system and render your content directly in an immersive space."}, {"start": 939.0, "text": "That was a sneak peek at supported games engines."}, {"start": 942.0, "text": "Next, let's talk about extending content from your Mac to the infinite canvas"}, {"start": 947.0, "text": "of Apple Vision Pro, using the Spatial Preview framework."}, {"start": 951.0, "text": "Mac Virtual Display transforms your workspace"}, {"start": 953.0, "text": "offering you the ability to stay in headset"}, {"start": 956.0, "text": "while you work on your Mac with complete privacy, wherever you are."}, {"start": 961.0, "text": "With visionOS 27, we're taking this capability even further"}, {"start": 965.0, "text": "with Spatial Preview."}, {"start": 967.0, "text": "This is a new macOS framework that lets you preview spatial content"}, {"start": 971.0, "text": "from your Mac directly on your Apple Vision Pro,"}, {"start": 974.0, "text": "and collaborate with others through SharePlay,"}, {"start": 977.0, "text": "— all without ever building a visionOS app!"}, {"start": 981.0, "text": "Leveraging Quick Look on visionOS,"}, {"start": 983.0, "text": "you can immediately preview and update content"}, {"start": 986.0, "text": "like spatial photos and Apple immersive video"}, {"start": 989.0, "text": "as well as edit 3D content live using USD."}, {"start": 994.0, "text": "You can move freely around 3D scenes,"}, {"start": 996.0, "text": "refine your content's placement,"}, {"start": 1000.0, "text": "adjust material overrides,"}, {"start": 1002.0, "text": "and share feedback with annotations,"}, {"start": 1005.0, "text": "all within a spatial environment."}, {"start": 1008.0, "text": "With real-time asset editing,"}, {"start": 1010.0, "text": "apps like Cinema4D and SketchUp transform the creative process —"}, {"start": 1015.0, "text": "unlocking real-time, collaborative 3D workflows"}, {"start": 1018.0, "text": "whether you're working side by side,"}, {"start": 1020.0, "text": "or across the globe."}, {"start": 1024.0, "text": "This capability is built directly into Preview on macOS 27,"}, {"start": 1029.0, "text": "so customers can experience these powerful features out of the box."}, {"start": 1033.0, "text": "We're also giving Preview new 3D editing tools"}, {"start": 1037.0, "text": "making 3D content as easy to work with"}, {"start": 1040.0, "text": "as images and PDFs."}, {"start": 1043.0, "text": "For a deeper dive, check out the \"Discover the Spatial Preview framework\" session."}, {"start": 1048.0, "text": "The Spatial Preview framework is a great way to stream"}, {"start": 1051.0, "text": "immersive Mac content to visionOS."}, {"start": 1053.0, "text": "Next, I'll introduce a new way to bring immersive PC content"}, {"start": 1057.0, "text": "to our platforms with Foveated Streaming."}, {"start": 1061.0, "text": "Foveated Streaming enables Apple Vision Pro"}, {"start": 1063.0, "text": "to connect to external devices —"}, {"start": 1065.0, "text": "like a PC —"}, {"start": 1066.0, "text": "to stream OpenXR content."}, {"start": 1069.0, "text": "visionOS automatically sends input data, like hands, controller positions,"}, {"start": 1074.0, "text": "and microphone."}, {"start": 1076.0, "text": "And the device streams your OpenXR content, like video and audio"}, {"start": 1081.0, "text": "with full-scale immersion, just like a native app."}, {"start": 1085.0, "text": "Foveated Streaming launched in visionOS 26.4,"}, {"start": 1089.0, "text": "and it's already unlocking incredible experiences."}, {"start": 1092.0, "text": "X-Plane 12 from Laminar Research"}, {"start": 1095.0, "text": "delivered a best-in-class flight simulation experience."}, {"start": 1098.0, "text": "The X-Plane app on visionOS uses ARKit to understand your space and equipment,"}, {"start": 1104.0, "text": "and streams the simulation from a PC."}, {"start": 1107.0, "text": "This means you can fly using a physical flight simulator"}, {"start": 1110.0, "text": "while being fully immersed in a virtual world."}, {"start": 1113.0, "text": "It's an integrated experience only Apple Vision Pro can deliver."}, {"start": 1118.0, "text": "iRacing, a motorsport racing game for PC, also streams to Apple Vision Pro"}, {"start": 1123.0, "text": "using Foveated Streaming."}, {"start": 1125.0, "text": "The iRacing Connect app matches the position of your physical racing wheel"}, {"start": 1130.0, "text": "with the virtual cockpit."}, {"start": 1131.0, "text": "Precise ARKit tracking makes this possible,"}, {"start": 1134.0, "text": "delivering an experience that is immersive and exhilarating."}, {"start": 1139.0, "text": "Innoactive also brings Autodesk VRED to Apple Vision Pro,"}, {"start": 1143.0, "text": "letting designers visualize massive, high-fidelity models with ray tracing,"}, {"start": 1148.0, "text": "at a 1:1 scale."}, {"start": 1150.0, "text": "PC-based rendering pairs seamlessly with SwiftUI,"}, {"start": 1154.0, "text": "so you can build advanced streaming applications quickly and intuitively."}, {"start": 1159.0, "text": "And the quality is exceptional, because it uses advanced technology"}, {"start": 1164.0, "text": "to optimize the video stream."}, {"start": 1166.0, "text": "Foveated streaming intelligently compresses video"}, {"start": 1169.0, "text": "based on where a person is looking."}, {"start": 1172.0, "text": "Areas in focus are streamed at higher quality,"}, {"start": 1176.0, "text": "while areas in your visual periphery use less bandwidth."}, {"start": 1180.0, "text": "It happens so quickly and seamlessly, that you don't even notice."}, {"start": 1185.0, "text": "The streaming protocol is powered by NVIDIA CloudXR."}, {"start": 1189.0, "text": "This is a top-ranked streaming technology, offering high quality and low latency."}, {"start": 1194.0, "text": "CloudXR is performant enough to stream over Wi-Fi,"}, {"start": 1197.0, "text": "with no dongles or cables needed,"}, {"start": 1200.0, "text": "whether from a local PC or a cloud instance."}, {"start": 1204.0, "text": "And it is incredibly easy to use."}, {"start": 1207.0, "text": "We've found that just in one day"}, {"start": 1209.0, "text": "you can start streaming your OpenXR applications"}, {"start": 1212.0, "text": "to Apple Vision Pro,"}, {"start": 1214.0, "text": "and in a week you can enhance your application"}, {"start": 1217.0, "text": "with features you can find only on visionOS."}, {"start": 1220.0, "text": "To learn more, see our dedicated session on Foveated Streaming."}, {"start": 1225.0, "text": "Those are the many ways you can bring your content to Apple Vision Pro."}, {"start": 1229.0, "text": "Next, let's look at some exciting new ways in which people can interact"}, {"start": 1233.0, "text": "with your content in visionOS."}, {"start": 1236.0, "text": "Let's start with enhancements to object tracking."}, {"start": 1240.0, "text": "We introduced object tracking in visionOS 2.0,"}, {"start": 1243.0, "text": "which lets you turn physical objects"}, {"start": 1245.0, "text": "into virtual anchors."}, {"start": 1247.0, "text": "To track an object, you can start with its USDZ model"}, {"start": 1252.0, "text": "and train a reference object in Create ML on your Mac."}, {"start": 1256.0, "text": "You pass that reference object to the object tracking API,"}, {"start": 1259.0, "text": "and your app receives updates about the position"}, {"start": 1263.0, "text": "and orientation of the physical object,"}, {"start": 1265.0, "text": "enabling immersive spatial experiences."}, {"start": 1269.0, "text": "Object tracking now supports high-frame-rate tracking,"}, {"start": 1272.0, "text": "giving your app more frequent pose updates"}, {"start": 1274.0, "text": "as objects move through space."}, {"start": 1277.0, "text": "We've added an extended training option in Create ML"}, {"start": 1280.0, "text": "that improves accuracy and robustness,"}, {"start": 1283.0, "text": "particularly for objects held in hand."}, {"start": 1287.0, "text": "There's a new API for obtaining the object pose in metric space,"}, {"start": 1290.0, "text": "without display corrections."}, {"start": 1293.0, "text": "This unlocks spatial measurement use cases"}, {"start": 1295.0, "text": "that require high-precision poses."}, {"start": 1298.0, "text": "And these features are now available on both visionOS and iOS."}, {"start": 1303.0, "text": "These enhancements to object tracking let you create dynamic experiences"}, {"start": 1307.0, "text": "that instantly react as people pick up and interact with objects"}, {"start": 1311.0, "text": "in the physical world."}, {"start": 1314.0, "text": "For example, you can now accurately track and measure physical spaces"}, {"start": 1318.0, "text": "using handheld items such as this medical probe."}, {"start": 1322.0, "text": "This opens up use cases like surgical navigation training."}, {"start": 1325.0, "text": "To bring object tracking to iOS, we're releasing an ARKit API"}, {"start": 1330.0, "text": "that supports the same functionality as visionOS."}, {"start": 1333.0, "text": "We designed the ML model training in Create ML to be platform agnostic."}, {"start": 1339.0, "text": "This means once you create a reference object,"}, {"start": 1342.0, "text": "you can use it in both your iOS and visionOS app"}, {"start": 1345.0, "text": "and get the same level of tracking quality."}, {"start": 1348.0, "text": "To learn more about developing with the object tracking API,"}, {"start": 1351.0, "text": "you can watch the WWDC24 session or explore our documentation."}, {"start": 1358.0, "text": "While object tracking is great for tracking regular objects,"}, {"start": 1362.0, "text": "visionOS also supports spatial accessories."}, {"start": 1366.0, "text": "In visionOS 26, we introduced our first set of spatial accessories:"}, {"start": 1371.0, "text": "the Logitech Muse and the PSVR2 Sense controller."}, {"start": 1375.0, "text": "These devices bring a new level of interactivity and immersion to your apps"}, {"start": 1380.0, "text": "through spatial tracking, button input, and haptic feedback."}, {"start": 1385.0, "text": "You can connect to them using the Game Controller framework,"}, {"start": 1388.0, "text": "and use RealityKit"}, {"start": 1389.0, "text": "or ARKit to track each accessory's movement and orientation in space."}, {"start": 1395.0, "text": "Now in visionOS 27, we are expanding support to enable you"}, {"start": 1399.0, "text": "to build your own accessory."}, {"start": 1401.0, "text": "A spatial accessory is an electronic device,"}, {"start": 1404.0, "text": "containing a board with the following components:"}, {"start": 1406.0, "text": "A constellation of LEDs visible to Apple Vision Pro"}, {"start": 1410.0, "text": "for tracking."}, {"start": 1411.0, "text": "An IMU to capture the orientation and acceleration"}, {"start": 1415.0, "text": "of the accessory."}, {"start": 1417.0, "text": "And a Bluetooth chip to send the signals to Apple Vision Pro."}, {"start": 1422.0, "text": "Spatial accessories can also host any variety of inputs"}, {"start": 1426.0, "text": "like buttons, touchpads,"}, {"start": 1428.0, "text": "along with haptic feedback."}, {"start": 1430.0, "text": "You can turn any object into a compatible spatial accessory"}, {"start": 1433.0, "text": "by just installing these components in it."}, {"start": 1437.0, "text": "To help you get started, manufacturers like DFRobot and MikroE"}, {"start": 1441.0, "text": "will release off-the-shelf reference hardware and development kits,"}, {"start": 1445.0, "text": "later this year."}, {"start": 1446.0, "text": "These can be a great starting point to include custom spatial accessory input"}, {"start": 1451.0, "text": "in your visionOS apps."}, {"start": 1452.0, "text": "For example, this is a 3D printed flashlight,"}, {"start": 1455.0, "text": "mounted with a DFRobot seeMote Cap."}, {"start": 1458.0, "text": "Thanks to low-latency tracking,"}, {"start": 1460.0, "text": "the virtual light it casts looks completely natural"}, {"start": 1463.0, "text": "on the physical walls around it."}, {"start": 1464.0, "text": "As the physical flashlight moves,"}, {"start": 1466.0, "text": "the virtual beam follows smoothly and accurately."}, {"start": 1471.0, "text": "Here's another example:"}, {"start": 1472.0, "text": "a MikroE Spatial Anchor R1"}, {"start": 1475.0, "text": "is mounted inside a physical steering wheel,"}, {"start": 1478.0, "text": "seamlessly anchoring a digital vehicle to it."}, {"start": 1481.0, "text": "When you grab the wheel, you feel like you're inside the car."}, {"start": 1485.0, "text": "This unlocks use cases like immersive racing simulations"}, {"start": 1488.0, "text": "and vehicle interior design."}, {"start": 1490.0, "text": "Apple Vision Pro tracks these devices at the highest possible frequency"}, {"start": 1494.0, "text": "with extremely low latency —"}, {"start": 1497.0, "text": "matching the display's native refresh rate"}, {"start": 1499.0, "text": "for a seamless experience."}, {"start": 1502.0, "text": "They support use cases that demand fast motion."}, {"start": 1505.0, "text": "Spatial accessories will continue to track robustly,"}, {"start": 1508.0, "text": "even when temporarily occluded."}, {"start": 1511.0, "text": "And they are also trackable under low-light conditions."}, {"start": 1514.0, "text": "Lastly, the physical buttons and haptics allow you to make your experiences"}, {"start": 1519.0, "text": "even more interactive and immersive."}, {"start": 1522.0, "text": "To learn more about integrating custom spatial accessories"}, {"start": 1525.0, "text": "into your spatial experiences,"}, {"start": 1527.0, "text": "check out the session \"Explore enhancements to visionOS object tracking\"."}, {"start": 1532.0, "text": "Next, let's talk about the workflow to bring immersive media experiences"}, {"start": 1537.0, "text": "to visionOS."}, {"start": 1538.0, "text": "visionOS supports many forms of spatial and immersive video types,"}, {"start": 1543.0, "text": "but Apple Immersive Video, or AIV,"}, {"start": 1545.0, "text": "is the highest fidelity immersive video experience,"}, {"start": 1548.0, "text": "available on visionOS."}, {"start": 1550.0, "text": "It has an extremely large field-of-view"}, {"start": 1553.0, "text": "and fully immersive audio that places you"}, {"start": 1556.0, "text": "in the experience, as if you were there."}, {"start": 1558.0, "text": "It represents a fundamental advancement in video media engineering,"}, {"start": 1562.0, "text": "built on high-resolution, high-frame-rate, stereoscopic 180 degree capture"}, {"start": 1568.0, "text": "that delivers unprecedented fidelity and dimensional accuracy."}, {"start": 1573.0, "text": "Apple Immersive Video supports both video-on-demand"}, {"start": 1576.0, "text": "and live broadcast streaming in visionOS."}, {"start": 1579.0, "text": "Achieving such a high quality experience requires video"}, {"start": 1582.0, "text": "that meets demanding specifications."}, {"start": 1585.0, "text": "Real-world stereoscopic scale is maintained"}, {"start": 1588.0, "text": "through metadata-driven lens calibration,"}, {"start": 1591.0, "text": "which provides accurate projection during playback."}, {"start": 1595.0, "text": "Video is captured and streamed at 90 frames per second"}, {"start": 1598.0, "text": "with a near-human visual acuity"}, {"start": 1600.0, "text": "greater than 100 megapixels per frame."}, {"start": 1604.0, "text": "That's over ten billion pixels per second."}, {"start": 1607.0, "text": "To handle all that video,"}, {"start": 1608.0, "text": "AIV already has a growing ecosystem of production"}, {"start": 1612.0, "text": "and post-production tools"}, {"start": 1614.0, "text": "that are now available with industry-leading broadcast hardware"}, {"start": 1617.0, "text": "and software developed for iOS, macOS, and visionOS."}, {"start": 1622.0, "text": "Many of those tools were built by developers like you"}, {"start": 1626.0, "text": "with the Immersive Media Support framework, or IMS."}, {"start": 1630.0, "text": "IMS enables reading and writing of rich metadata for Apple Immersive Video,"}, {"start": 1636.0, "text": "and provides support for authoring and modifying immersive content."}, {"start": 1640.0, "text": "You can get started with integrating IMS in your apps by watching these sessions:"}, {"start": 1646.0, "text": "\"Learn about Apple Immersive Video technologies\""}, {"start": 1649.0, "text": "and \"Support immersive video playback in visionOS apps\" from last year."}, {"start": 1654.0, "text": "In addition to recently introduced iOS support,"}, {"start": 1657.0, "text": "important new features have been added to IMS"}, {"start": 1660.0, "text": "in visionOS 27,"}, {"start": 1662.0, "text": "including camera presentation override commands,"}, {"start": 1666.0, "text": "an ImmersivePreviewRenderer API, and wide-aspect-ratio portals."}, {"start": 1672.0, "text": "We have also published some new sample code"}, {"start": 1674.0, "text": "that implements static foveation in dual track QuickTime,"}, {"start": 1678.0, "text": "and updates to the Apple Spatial Audio Format production suite."}, {"start": 1682.0, "text": "Let's first look at the impact of new camera presentation override commands."}, {"start": 1687.0, "text": "In live and complex production scenarios,"}, {"start": 1690.0, "text": "you may need to introduce new camera parameters"}, {"start": 1693.0, "text": "to override default camera configurations,"}, {"start": 1696.0, "text": "and in real time."}, {"start": 1698.0, "text": "That's where the new Set Camera Command Overrides come into play."}, {"start": 1703.0, "text": "If you would like to learn more about the practical application"}, {"start": 1706.0, "text": "of IMS in the AIV live production workflow with SMTPE 2110,"}, {"start": 1711.0, "text": "check out the WWDC 26 session,"}, {"start": 1714.0, "text": "\"Build live production tools for Apple Immersive Video\"."}, {"start": 1718.0, "text": "The new ImmersivePreviewRenderer enables real-time previewing"}, {"start": 1722.0, "text": "of Apple Immersive Video on Apple Vision Pro"}, {"start": 1725.0, "text": "directly from a Mac"}, {"start": 1726.0, "text": "during editorial or live production workflows."}, {"start": 1730.0, "text": "This gives editors, colorists, and directors"}, {"start": 1733.0, "text": "a more accurate representation of how immersive content will look,"}, {"start": 1737.0, "text": "and be experienced, at final delivery."}, {"start": 1740.0, "text": "We're also adding wide-aspect-ratio portal support for Apple Immersive Video."}, {"start": 1745.0, "text": "This lets you keep a very wide portal of the immersive video experience in viewing"}, {"start": 1751.0, "text": "when switching from full immersive mode to portal mode."}, {"start": 1755.0, "text": "The implementation for your apps is straightforward."}, {"start": 1758.0, "text": "Custom aspect ratios can be set"}, {"start": 1760.0, "text": "when using AVPlayerViewController in AVKit-based apps,"}, {"start": 1764.0, "text": "or VideoPlayerComponent in RealityKit-based apps."}, {"start": 1768.0, "text": "Next, I want to introduce you to a new sample that uses static foveation"}, {"start": 1773.0, "text": "to deliver a high-quality immersive video while keeping it streamable."}, {"start": 1778.0, "text": "It would be impractical,"}, {"start": 1780.0, "text": "even for a high-speed home internet connection,"}, {"start": 1782.0, "text": "to stream full resolution AIV, in stereo,"}, {"start": 1785.0, "text": "at 90 frames-per-second."}, {"start": 1787.0, "text": "But simply scaling the image down to 4K"}, {"start": 1790.0, "text": "would sacrifice too much pixel density."}, {"start": 1793.0, "text": "Instead, a smooth static foveation function can be applied to the image"}, {"start": 1798.0, "text": "before it is encoded for streaming."}, {"start": 1801.0, "text": "The new sample project demonstrates one way this technique can be used"}, {"start": 1805.0, "text": "to achieve high-acuity immersive video in a streamable frame size."}, {"start": 1810.0, "text": "And we can't forget that any good immersive video experience,"}, {"start": 1814.0, "text": "must also have an immersive spatial audio soundtrack."}, {"start": 1817.0, "text": "So we have released important new updates to the ASAF Production Suite."}, {"start": 1822.0, "text": "The suite of AAX plugins now has the ability to position objects"}, {"start": 1826.0, "text": "relative to a reference video,"}, {"start": 1829.0, "text": "with improvements to the ambisonics workflow"}, {"start": 1831.0, "text": "that include a new Scene Compressor plugin,"}, {"start": 1833.0, "text": "and enhancements to the heat map drawing"}, {"start": 1836.0, "text": "and the spatial filtering algorithm."}, {"start": 1838.0, "text": "The new sample project and the ASAF Production Suite"}, {"start": 1841.0, "text": "are both available for download on developer.apple.com."}, {"start": 1846.0, "text": "From deeper immersion, to richer media,"}, {"start": 1849.0, "text": "and powerful new ways to add interactivity,"}, {"start": 1852.0, "text": "visionOS 27 gives you so much to build with."}, {"start": 1856.0, "text": "To round things out, let's spotlight a few final additions,"}, {"start": 1859.0, "text": "starting with the Spatial Web."}, {"start": 1861.0, "text": "In Safari on visionOS 27,"}, {"start": 1864.0, "text": "windows can now be adjusted to a wider aspect ratio,"}, {"start": 1868.0, "text": "letting you take full advantage of the space around you."}, {"start": 1872.0, "text": "Larger windows will naturally curve, bringing more of your content"}, {"start": 1876.0, "text": "into comfortable view."}, {"start": 1879.0, "text": "And, Web Environments are now enabled by default"}, {"start": 1882.0, "text": "allowing websites to have backgrounds, just like apps."}, {"start": 1886.0, "text": "Here's an example showing a virtual environment"}, {"start": 1889.0, "text": "from the Apple TV show, Severance."}, {"start": 1892.0, "text": "We've streamlined notifications, system status, controls, and environments"}, {"start": 1897.0, "text": "all in one place via a newly improved Control Center,"}, {"start": 1901.0, "text": "along with High Quality Capture"}, {"start": 1904.0, "text": "enabling you to capture your apps, in stunning 4K video"}, {"start": 1908.0, "text": "from right inside Apple Vision Pro, no Mac required."}, {"start": 1912.0, "text": "And with accessory widget support on visionOS,"}, {"start": 1915.0, "text": "you can extend your app to Apple Vision Pro with smaller,"}, {"start": 1919.0, "text": "glanceable widgets that surface your most relevant information,"}, {"start": 1923.0, "text": "right where you need it."}, {"start": 1926.0, "text": "visionOS 27 brings so much more to explore,"}, {"start": 1929.0, "text": "including Siri enhancements, an all-new Iceland environment,"}, {"start": 1933.0, "text": "Spatial Panoramas, Personal Environments, and Freeform updates, to name a few."}, {"start": 1939.0, "text": "Together, they give you even more powerful tools to bring your ideas to life."}, {"start": 1945.0, "text": "We cannot wait to see what you'll build next."}]}