{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-283", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/283", "wordCount": 2406, "segments": [{"start": 6.0, "text": "Hello, welcome to explore enhancements to visionOS object track."}, {"start": 10.0, "text": "I'm Nathan Kong,"}, {"start": 11.0, "text": "a Strategic Partnerships Manager on the visionOS team."}, {"start": 14.0, "text": "In this session,"}, {"start": 15.0, "text": "we will share the new object tracking"}, {"start": 17.0, "text": "and spatial accessory capabilities"}, {"start": 19.0, "text": "that build upon features described in previous sessions."}, {"start": 22.0, "text": "As a reminder, in visionOS 2.0 we introduced object tracking,"}, {"start": 26.0, "text": "which allows you to turn real world objects into virtual anchors."}, {"start": 30.0, "text": "With only a USDZ model of the item you'd like to track,"}, {"start": 33.0, "text": "you can create a reference object"}, {"start": 35.0, "text": "through machine learning training in Create ML on your Mac."}, {"start": 38.0, "text": "By passing reference objects to our APIs,"}, {"start": 41.0, "text": "your app can get the position and orientation of the physical objects"}, {"start": 44.0, "text": "to create immersive spatial experiences."}, {"start": 47.0, "text": "In visionOS 27, we are expanding object tracking,"}, {"start": 50.0, "text": "giving you the ability to track objects in motion,"}, {"start": 52.0, "text": "including handheld items alongside other enhancements."}, {"start": 56.0, "text": "For example, you can now accurately track"}, {"start": 59.0, "text": "and measure physical spaces using handheld items such as this medical probe."}, {"start": 63.0, "text": "Here we can precisely measure the distance"}, {"start": 66.0, "text": "between the vertebrae on this physical spine model."}, {"start": 69.0, "text": "This can enable exciting use cases"}, {"start": 70.0, "text": "such surgical training, home remodeling, or guided assembly."}, {"start": 75.0, "text": "Switching to spatial accessories,"}, {"start": 77.0, "text": "in visionOS 26, we introduced the first set of spatial accessories"}, {"start": 81.0, "text": "including the Logitech Muse"}, {"start": 82.0, "text": "and the PSVR2 Sense controllers."}, {"start": 85.0, "text": "Spatial accessories are electronic devices that communicate with Apple Vision Pro,"}, {"start": 90.0, "text": "which tracks their realtime position and orientation in the real world."}, {"start": 94.0, "text": "These devices also enhance interactivity"}, {"start": 96.0, "text": "and immersion in your apps through buttons and haptics."}, {"start": 100.0, "text": "Now, in visionOS 27,"}, {"start": 101.0, "text": "we are expanding support to enable anyone to build your own accessory."}, {"start": 106.0, "text": "For example, we can mount an accessory"}, {"start": 108.0, "text": "like this Spatial Anchor inside of a physical steering wheel"}, {"start": 111.0, "text": "in order to seamlessly align a full scale digital vehicle to it."}, {"start": 115.0, "text": "When I reach out and grab the steering wheel,"}, {"start": 117.0, "text": "I truly feel like I'm inside this car."}, {"start": 120.0, "text": "This capability unlocks incredible experiences such as immersive racing"}, {"start": 124.0, "text": "and flight simulations or vehicle interior design."}, {"start": 128.0, "text": "In the remainder of this session"}, {"start": 129.0, "text": "we will cover improvements to object tracking,"}, {"start": 131.0, "text": "show you how to create a spatial accessory,"}, {"start": 134.0, "text": "and conclude with considerations you should keep in mind"}, {"start": 136.0, "text": "when deciding which approach is best for your visionOS app."}, {"start": 139.0, "text": "Now let's dive into object tracking."}, {"start": 142.0, "text": "Object tracking now supports high frame rate tracking."}, {"start": 145.0, "text": "This enables your app to have a better understanding of the position of objects"}, {"start": 148.0, "text": "as they move in space."}, {"start": 150.0, "text": "Along with this, we're adding a new extended training mode in Create ML"}, {"start": 153.0, "text": "for increased tracking accuracy and robustness,"}, {"start": 156.0, "text": "especially when holding objects in hand."}, {"start": 159.0, "text": "To enable even more accurate tracking use cases,"}, {"start": 161.0, "text": "we're introducing an API to obtain the object's pose in metric space,"}, {"start": 165.0, "text": "unaffected by any display corrections."}, {"start": 167.0, "text": "This is essential to enable measuring applications"}, {"start": 170.0, "text": "like the medical probe example from before."}, {"start": 173.0, "text": "Lastly, we're bringing object tracking to iOS this year."}, {"start": 176.0, "text": "Let's take a look at some of these updates in action using this flashlight."}, {"start": 180.0, "text": "If I have access to a photorealistic 3D model of the flashlight,"}, {"start": 183.0, "text": "I can use it as the reference object for object tracking."}, {"start": 186.0, "text": "Since my app understands the position of the flashlight,"}, {"start": 189.0, "text": "I can take advantage of the new Physical Surroundings Light in RealityKit"}, {"start": 193.0, "text": "to relight this room in real time."}, {"start": 195.0, "text": "I can also swap the digital pattern being projected onto the surfaces around me."}, {"start": 199.0, "text": "Even though my hand is occluding part of the flashlight,"}, {"start": 201.0, "text": "my app is still able to track the position of the object robustly."}, {"start": 205.0, "text": "Alternatively, I can 3D print a marker like this"}, {"start": 208.0, "text": "as the reference object and mount it to the flashlight."}, {"start": 211.0, "text": "This is an easy way to track any handheld object,"}, {"start": 214.0, "text": "even if you're unable to obtain a photorealistic 3D model of the device."}, {"start": 218.0, "text": "Now, let's discuss the specific updates"}, {"start": 220.0, "text": "to show you how to upgrade an existing object tracking experience"}, {"start": 223.0, "text": "or start a new one from scratch."}, {"start": 226.0, "text": "First I'm going to explain how to enable the new high frame rate tracking."}, {"start": 230.0, "text": "We're adding a new Reference Object Configuration API"}, {"start": 233.0, "text": "in ARKit on visionOS."}, {"start": 235.0, "text": "It let's you enable high frame rate tracking"}, {"start": 237.0, "text": "for an individual reference object"}, {"start": 239.0, "text": "before creating your object tracking session."}, {"start": 242.0, "text": "Once, configured, you can pass this configuration object"}, {"start": 245.0, "text": "as an additional parameter when loading the reference object as before."}, {"start": 249.0, "text": "Since this this is not a training setting, it can be applied to any reference object"}, {"start": 252.0, "text": "depending on the application's needs."}, {"start": 255.0, "text": "Next let's discuss the new extended training mode we added to Create ML."}, {"start": 259.0, "text": "When training a reference object in Create ML,"}, {"start": 262.0, "text": "you can now choose between the standard and the extended training mode."}, {"start": 266.0, "text": "The new extended training setting increases the accuracy"}, {"start": 268.0, "text": "and robustness of tracking,"}, {"start": 270.0, "text": "and is recommended to be used"}, {"start": 271.0, "text": "in combination with high frame rate tracking."}, {"start": 274.0, "text": "Please note that the extended training"}, {"start": 275.0, "text": "takes significantly longer compared to standard mode."}, {"start": 279.0, "text": "You will find the training mode setting in the Object Tracking template"}, {"start": 282.0, "text": "in the Create ML app,"}, {"start": 283.0, "text": "right below the viewing angle settings."}, {"start": 285.0, "text": "Everything else stays exactly the same"}, {"start": 287.0, "text": "when it comes to training a reference object."}, {"start": 290.0, "text": "If you prefer to train reference objects through the command line interface,"}, {"start": 293.0, "text": "you can configure the training mode there as well."}, {"start": 296.0, "text": "This gives you the flexibility to run the training on a remote machine."}, {"start": 299.0, "text": "Now we'll review the new benefits of obtaining object poses in metric space."}, {"start": 304.0, "text": "By default, object anchor transforms are optimized"}, {"start": 307.0, "text": "for placing virtual content aligned with the tracked object"}, {"start": 310.0, "text": "in the mixed immersion style."}, {"start": 312.0, "text": "To achieve this, the object poses are slightly altered"}, {"start": 315.0, "text": "to match the displayed camera images,"}, {"start": 317.0, "text": "impacting the accuracy in absolute world coordinates."}, {"start": 320.0, "text": "This can be a limitation when trying to use object tracking"}, {"start": 322.0, "text": "for spatial measuring tasks."}, {"start": 325.0, "text": "In visionOS 27, we're adding the ARKit Coordinate Space Correction API"}, {"start": 330.0, "text": "that let's you obtain the anchor transform with or without these corrections."}, {"start": 334.0, "text": "When querying a tracked object's pose, it provides two options:"}, {"start": 338.0, "text": "rendered, returns the pose with display corrections applied"}, {"start": 341.0, "text": "to keep virtual content visually aligned with the real-world object,"}, {"start": 344.0, "text": "and none, returns the object's pose in metric space, without any corrections."}, {"start": 349.0, "text": "This is useful for measuring the distance between tracked objects"}, {"start": 352.0, "text": "or determining where an object sits in physical space"}, {"start": 354.0, "text": "as shown in the medical probe demo earlier."}, {"start": 357.0, "text": "Last but not least, this year we're bringing object tracking to iOS!"}, {"start": 361.0, "text": "But first, let's bring back the globe from WWDC24!"}, {"start": 366.0, "text": "In iOS 27, we're adding support for reference objects in our ARKit APIs."}, {"start": 372.0, "text": "Machine learning training is not platform specific."}, {"start": 375.0, "text": "So once trained,"}, {"start": 376.0, "text": "all reference objects will be supported in both iOS and visionOS apps."}, {"start": 382.0, "text": "Here's all it takes to get object tracking running on iOS."}, {"start": 385.0, "text": "You load your reference objects and create a world tracking configuration."}, {"start": 389.0, "text": "These are the same reference object files you use on visionOS."}, {"start": 393.0, "text": "You assign objects to either detectionObjects"}, {"start": 396.0, "text": "for objects that are mostly stationary,"}, {"start": 398.0, "text": "or trackingObjects for high frame rate tracking of moving objects."}, {"start": 402.0, "text": "Then you run the session and handle the anchors in your delegate."}, {"start": 406.0, "text": "When ARKit recognizes an object,"}, {"start": 408.0, "text": "didAdd is called and you get an ARObjectAnchor"}, {"start": 411.0, "text": "that you can attach content to."}, {"start": 413.0, "text": "While the object gets tracked,"}, {"start": 414.0, "text": "didUpdate provides the latest poses,"}, {"start": 416.0, "text": "which you can use for your custom app behavior."}, {"start": 419.0, "text": "And if the object is removed from the scene,"}, {"start": 421.0, "text": "didRemove lets you clean up and remove the anchor entity created in didAdd."}, {"start": 426.0, "text": "Those are all the updates to object tracking in visionOS 27."}, {"start": 430.0, "text": "To learn more about how to develop an app with the object tracking API,"}, {"start": 434.0, "text": "you can watch the \"Explore object tracking for visionOS\" WWDC session"}, {"start": 438.0, "text": "or explore documentation."}, {"start": 440.0, "text": "Next, we'll explore how you can turn your object of interest"}, {"start": 442.0, "text": "into a spatial accessory."}, {"start": 445.0, "text": "I'll start by defining a spatial accessory and highlight some of their benefits."}, {"start": 449.0, "text": "Then I'll cover design considerations you should keep in mind"}, {"start": 452.0, "text": "when creating your accessory."}, {"start": 454.0, "text": "I'll explain the process to validate your design"}, {"start": 456.0, "text": "and prepare it for visionOS."}, {"start": 457.0, "text": "Share some easy plug-and-play accessories"}, {"start": 460.0, "text": "and finally show you how to prepare your visionOS app"}, {"start": 462.0, "text": "to take advantage of these devices."}, {"start": 464.0, "text": "Now let's get started with the basics."}, {"start": 467.0, "text": "A spatial accessory is an electronic device"}, {"start": 469.0, "text": "which must contain a board with the following components."}, {"start": 472.0, "text": "A constellation of LEDs visible to Apple Vision Pro for tracking."}, {"start": 476.0, "text": "An IMU to capture the orientation and acceleration of the accessory."}, {"start": 480.0, "text": "And a Bluetooth chip to send the signals to Vision Pro."}, {"start": 483.0, "text": "Spatial accessories can also host any variety of inputs"}, {"start": 486.0, "text": "including buttons or a touchpad, and outputs like haptics."}, {"start": 490.0, "text": "Any accessory that has these key components"}, {"start": 492.0, "text": "is compatible with visionOS."}, {"start": 494.0, "text": "Let's take a look at spatial accessories in action."}, {"start": 497.0, "text": "By installing the aforementioned components"}, {"start": 499.0, "text": "into the flashlight from before,"}, {"start": 501.0, "text": "we can make the device itself a spatial accessory."}, {"start": 503.0, "text": "Even when I quickly wave this flashlight,"}, {"start": 506.0, "text": "the digital beam of light follows smoothly"}, {"start": 508.0, "text": "due to the low latency tracking enabled by the embedded IMU."}, {"start": 512.0, "text": "And using this physical button I added to accessory,"}, {"start": 514.0, "text": "I can turn the digital light off and on,"}, {"start": 516.0, "text": "making my experience even more interactive."}, {"start": 519.0, "text": "Vision Pro is able to track this spatial accessory,"}, {"start": 521.0, "text": "by seeing the LEDs installed inside the flashlight."}, {"start": 524.0, "text": "Now we'll cover some of the benefits spatial accessories offer."}, {"start": 528.0, "text": "These devices can be tracked at high frequency"}, {"start": 530.0, "text": "up to the full display rate with low latency"}, {"start": 533.0, "text": "and support use-cases that demand fast motion."}, {"start": 536.0, "text": "Spatial accessories will continue to track robustly,"}, {"start": 539.0, "text": "even when temporarily occluded."}, {"start": 541.0, "text": "You can track the accessory under lower light conditions."}, {"start": 544.0, "text": "Lastly, the physical buttons and haptics allow you to make"}, {"start": 547.0, "text": "your experiences even more interactive and immersive."}, {"start": 550.0, "text": "Before you get started with creating your own accessory,"}, {"start": 553.0, "text": "there are important design considerations you should keep in mind."}, {"start": 556.0, "text": "It's important to spread the LEDs around the device"}, {"start": 558.0, "text": "in such a way that it creates a distinct"}, {"start": 560.0, "text": "and unique pattern when viewed from various angles."}, {"start": 563.0, "text": "Both the LEDs and IMU should be rigidly fixed"}, {"start": 566.0, "text": "to the board for accurate tracking."}, {"start": 568.0, "text": "Last but not least,"}, {"start": 569.0, "text": "you should consider the primary way users will interact with your accessory."}, {"start": 573.0, "text": "For example, a handheld accessory should position most of the LEDs"}, {"start": 577.0, "text": "in areas where users will not hold the device."}, {"start": 580.0, "text": "You should also consider the size and position of the battery"}, {"start": 582.0, "text": "to ensure the accessory is ergonomic."}, {"start": 585.0, "text": "For larger accessories used out of arms reach,"}, {"start": 587.0, "text": "you should consider the number of LEDs, LED size, and distance between them"}, {"start": 591.0, "text": "to ensure the accessory can be tracked accurately even when far away."}, {"start": 596.0, "text": "For more information on specific requirements and reference designs,"}, {"start": 599.0, "text": "please check out the \"Spatial Accessories\" chapter"}, {"start": 601.0, "text": "of the \"Accessory Design Guidelines\" for Apple Devices."}, {"start": 604.0, "text": "Now that you've designed a spatial accessory,"}, {"start": 606.0, "text": "let's explore how to validate that it works as expected"}, {"start": 608.0, "text": "and prepare it for your visionOS app."}, {"start": 611.0, "text": "The first step is to connect your accessory"}, {"start": 613.0, "text": "via Bluetooth to your Vision Pro"}, {"start": 615.0, "text": "and validate the signals sent from the device"}, {"start": 617.0, "text": "with the ARKit accessory tracking debug view."}, {"start": 620.0, "text": "This tool lets you examine how your Vision Pro sees the accessory"}, {"start": 623.0, "text": "and can be found in settings when your device is in developer mode."}, {"start": 626.0, "text": "It helps you with three things:"}, {"start": 628.0, "text": "One, verify your LEDs through the headset's IR camera,"}, {"start": 632.0, "text": "so you can confirm they're bright, distinct, and properly synchronized."}, {"start": 636.0, "text": "Two, validate your IMU with live metrics on frequency,"}, {"start": 639.0, "text": "latency, and per-axis values,"}, {"start": 641.0, "text": "in order to check scale, alignment, and motion response."}, {"start": 645.0, "text": "Three, debug timing between your accessory"}, {"start": 648.0, "text": "and the headset using the device's IR illuminators as a sync reference."}, {"start": 654.0, "text": "Next, in order for your visionOS app to track a spatial accessory,"}, {"start": 658.0, "text": "you will need to train it with the CreateML bundle."}, {"start": 662.0, "text": "This workflow uses information about both the physical appearance of the device"}, {"start": 665.0, "text": "and the location of the LEDs to create a reference accessory file."}, {"start": 669.0, "text": "To get started, create a USDZ of your design"}, {"start": 672.0, "text": "that contains a photorealistic 3D model"}, {"start": 674.0, "text": "of the device annotated with the positions of the IMU and LEDs."}, {"start": 679.0, "text": "Using this annotated USDZ,"}, {"start": 681.0, "text": "you can use the command line interface"}, {"start": 682.0, "text": "to generate the reference accessory file and add it to your app."}, {"start": 686.0, "text": "As the manufacturer of the accessory,"}, {"start": 688.0, "text": "you bundle this file in your app"}, {"start": 690.0, "text": "and declare it as an exported UTType in your Info.plist."}, {"start": 694.0, "text": "This registers your accessory system-wide,"}, {"start": 696.0, "text": "so any app on Apple Vision Pro can use it."}, {"start": 699.0, "text": "If you're a developer using a third-party accessory,"}, {"start": 702.0, "text": "you can also bundle the file yourself and declare it as an imported type,"}, {"start": 705.0, "text": "so your app works independently."}, {"start": 708.0, "text": "Before building a spatial accessory from scratch,"}, {"start": 711.0, "text": "you can also start testing"}, {"start": 712.0, "text": "and develop apps with plug-and-play accessories."}, {"start": 715.0, "text": "Manufacturers like DFRobot and MIKROE"}, {"start": 717.0, "text": "will release off-the-shelf reference hardware"}, {"start": 719.0, "text": "and development kits later this year."}, {"start": 722.0, "text": "These accessories can be immediately used for testing"}, {"start": 725.0, "text": "or implemented in your visionOS app."}, {"start": 727.0, "text": "Let's see one of these plug-and-play accessories in action."}, {"start": 731.0, "text": "As you can see here,"}, {"start": 732.0, "text": "I can simply mount a spatial accessory like the seeMote Cap to the flashlight"}, {"start": 736.0, "text": "and use the Spatial Accessories API"}, {"start": 738.0, "text": "to enable the same digital relighting experience as before."}, {"start": 742.0, "text": "Now that you have a spatial accessory,"}, {"start": 744.0, "text": "let's explore how to connect it to your app."}, {"start": 746.0, "text": "You discover accessories using the new GCSpatialAccessory class."}, {"start": 750.0, "text": "This works with any device that has a referenceaccessory bundle."}, {"start": 754.0, "text": "When you call Accessory(device), ARKit resolves it automatically."}, {"start": 758.0, "text": "From there, you can run an AccessoryTrackingProvider"}, {"start": 761.0, "text": "just like before."}, {"start": 763.0, "text": "We're also adding"}, {"start": 764.0, "text": "a new updateAccessories method"}, {"start": 765.0, "text": "to switch between accessories while your session is running"}, {"start": 768.0, "text": "to avoiding interruptions to tracking."}, {"start": 770.0, "text": "And with that you can create your own spatial accessory"}, {"start": 772.0, "text": "and connect it to your visionOS app."}, {"start": 775.0, "text": "To learn more about how to enable tracking with inputs and haptics in your app,"}, {"start": 779.0, "text": "you can watch the \"Explore spatial accessory input on visionOS\" session."}, {"start": 784.0, "text": "Let's wrap things up."}, {"start": 785.0, "text": "Today you've seen four different approaches"}, {"start": 787.0, "text": "for tracking an object in your app."}, {"start": 789.0, "text": "Let's review some considerations you should keep in mind"}, {"start": 792.0, "text": "when selecting which approach to use."}, {"start": 794.0, "text": "Object tracking excels in scenarios where accurate and precise tracking is required,"}, {"start": 798.0, "text": "such as measurement applications."}, {"start": 801.0, "text": "In case you do not have access to a photorealistic 3D model,"}, {"start": 804.0, "text": "you can train your reference object"}, {"start": 806.0, "text": "or a marker you mount on the object of interest."}, {"start": 809.0, "text": "Spatial accessories offer even higher refresh rates and lower latency,"}, {"start": 813.0, "text": "which is optimal for experiences that demand fast moving objects."}, {"start": 817.0, "text": "If you want to create an even more interactive"}, {"start": 819.0, "text": "and immersive experience with physical objects,"}, {"start": 821.0, "text": "you can design your own accessory with custom buttons and haptics."}, {"start": 825.0, "text": "The possibilities enabled by these new object tracking capabilities are endless."}, {"start": 830.0, "text": "We're excited to see the transformational experiences for work"}, {"start": 832.0, "text": "and play you can create."}, {"start": 834.0, "text": "Have a fantastic WWDC26!"}]}