{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-284", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/284", "wordCount": 4040, "segments": [{"start": 7.0, "text": "Welcome to Collaborate on Structured 3D Models in visionOS."}, {"start": 10.0, "text": "I'm Bill."}, {"start": 12.0, "text": "Today I am going to talk about building spatial experiences on Apple Vision Pro."}, {"start": 16.0, "text": "Specifically, how to work with complex assemblies and multi-dimensional data"}, {"start": 20.0, "text": "in ways that simply aren't possible on a flat screen."}, {"start": 24.0, "text": "Let's start with a look at what that can feel like."}, {"start": 27.0, "text": "Here we see a team in a SharePlay call, a design review of the AirPods Pro."}, {"start": 32.0, "text": "Everyone in the call sees the same asset, at the same fidelity, in the same space."}, {"start": 38.0, "text": "The case pulls closer."}, {"start": 39.0, "text": "It unlocks — the way it would on a workbench,"}, {"start": 42.0, "text": "except the workbench is wherever you happen to be."}, {"start": 45.0, "text": "The bottom assembly lifts out."}, {"start": 47.0, "text": "One person examines the interior, then rotates it"}, {"start": 50.0, "text": "so a colleague can see exactly the same thing."}, {"start": 53.0, "text": "A point — not an annotation, not a screenshot,"}, {"start": 56.0, "text": "just a gesture toward the part that matters and the group understands."}, {"start": 61.0, "text": "The model collapses back."}, {"start": 63.0, "text": "Clipping engages,"}, {"start": 65.0, "text": "a cross section opens, and the main logic board is right there —"}, {"start": 69.0, "text": "exposed in context, in a way that isn't feasible on a 2D screen."}, {"start": 74.0, "text": "Clipping disengages."}, {"start": 75.0, "text": "The full assembly expands."}, {"start": 77.0, "text": "Someone reaches in, pulls the motherboard free,"}, {"start": 80.0, "text": "and holds it up for the rest of the team."}, {"start": 82.0, "text": "Three people, one model, and the tools they needed to understand it."}, {"start": 86.0, "text": "What makes all of this possible comes down to three things"}, {"start": 89.0, "text": "Apple Vision Pro does really well."}, {"start": 92.0, "text": "Real-time collaboration:"}, {"start": 93.0, "text": "multiple people, in a shared space, at the same moment"}, {"start": 97.0, "text": "Manipulation of rich representations:"}, {"start": 100.0, "text": "because any data with enough dimensions deserves more than a flat screen."}, {"start": 104.0, "text": "Environment lighting:"}, {"start": 106.0, "text": "your physical world, grounding every virtual decision"}, {"start": 109.0, "text": "And those three capabilities aren't just useful for CAD."}, {"start": 113.0, "text": "Anywhere you have complex, multidimensional information to reason about —"}, {"start": 117.0, "text": "urban planning, logistics, real estate,"}, {"start": 120.0, "text": "production design — the same principles apply."}, {"start": 123.0, "text": "For a technical deep dive on making SharePlay work in your apps"}, {"start": 127.0, "text": "please see \"Share visionOS experiences with nearby people\""}, {"start": 131.0, "text": "from WWDC25."}, {"start": 134.0, "text": "There are 4 major considerations in building out the sample code."}, {"start": 138.0, "text": "I'll start with sharing some important aspects of preparing your 3D assets"}, {"start": 142.0, "text": "so that people can easily manipulate and work with them."}, {"start": 146.0, "text": "Complex spatial assets are structured collections of components,"}, {"start": 150.0, "text": "organized as assemblies."}, {"start": 153.0, "text": "I'll show you how to enable manipulating these assemblies."}, {"start": 157.0, "text": "Then, I'll go over clipping."}, {"start": 159.0, "text": "That allows the insides of an assembly to become as accessible as the outside."}, {"start": 164.0, "text": "And finally autoexpansion,"}, {"start": 167.0, "text": "where the asset expands itself,"}, {"start": 169.0, "text": "and every sub-assembly separates and moves into position,"}, {"start": 173.0, "text": "revealing the full structure."}, {"start": 175.0, "text": "On to asset preparation."}, {"start": 177.0, "text": "Assets without structure are hard to reason about and hard to use in code."}, {"start": 182.0, "text": "Without structure the code can't make decisions"}, {"start": 185.0, "text": "on what to hide or show, what to make manipulable."}, {"start": 189.0, "text": "When preparing an asset there are a lot of things to consider,"}, {"start": 193.0, "text": "what I'll focus on here is how to think about the structure of your asset,"}, {"start": 197.0, "text": "what assemblies contain what sub-assemblies."}, {"start": 200.0, "text": "In other words the hierarchy of the model."}, {"start": 203.0, "text": "Some of the other considerations, of a more technical aspect,"}, {"start": 206.0, "text": "are covered elsewhere."}, {"start": 209.0, "text": "For details, check out"}, {"start": 210.0, "text": "\"Optimize your 3D assets for spatial computing\""}, {"start": 213.0, "text": "from WWDC 24."}, {"start": 216.0, "text": "A 3D model is a part-whole relationship."}, {"start": 219.0, "text": "Flatten everything to the root,"}, {"start": 221.0, "text": "and even simple operations become surprisingly painful."}, {"start": 224.0, "text": "You've still got all the geometry — it's just not organized."}, {"start": 228.0, "text": "This engine block was exported without preserving its structure."}, {"start": 232.0, "text": "Everything flattened up to the root."}, {"start": 234.0, "text": "InteriorPart_01, InteriorPart_03, part 25, all the way down."}, {"start": 240.0, "text": "No sub-assemblies. No grouping."}, {"start": 243.0, "text": "And here's the thing."}, {"start": 244.0, "text": "This asset looks completely fine in the viewport."}, {"start": 247.0, "text": "The geometry is all there and it renders correctly."}, {"start": 250.0, "text": "But the structure that would make it interactive?"}, {"start": 253.0, "text": "Gone."}, {"start": 254.0, "text": "If I want to isolate one piston, it's in here somewhere."}, {"start": 258.0, "text": "Was it InteriorPart_47 or InteriorPart_18."}, {"start": 262.0, "text": "I don't know, and neither does my code."}, {"start": 265.0, "text": "Consider this updated asset, it's got a deep, nested hierarchy."}, {"start": 270.0, "text": "It's complex — and that's intentional."}, {"start": 273.0, "text": "Here we have hidden the outside of the engine"}, {"start": 276.0, "text": "and all the pistons but one."}, {"start": 278.0, "text": "We can see a piston and the crankshaft, alone."}, {"start": 281.0, "text": "Each one is its own node — named, organized, and grouped."}, {"start": 285.0, "text": "If I want to animate just the piston —"}, {"start": 288.0, "text": "isolate it, highlight it, let a person reach out and pull it free —"}, {"start": 291.0, "text": "I can."}, {"start": 292.0, "text": "It's organized, I can write code that can find it."}, {"start": 296.0, "text": "That's hierarchy doing exactly what it's supposed to do."}, {"start": 299.0, "text": "Now that we have a good hierarchy,"}, {"start": 301.0, "text": "I'll show you how to enable people to pull this hierarchy apart."}, {"start": 305.0, "text": "A good hierarchical structure will give people the ability"}, {"start": 308.0, "text": "to select an individual part,"}, {"start": 310.0, "text": "or move it using the natural input system on Apple Vision Pro."}, {"start": 314.0, "text": "This can be achieved using RealityKit's ManipulationComponent API correctly."}, {"start": 320.0, "text": "I'll show you how this is done."}, {"start": 322.0, "text": "Here is a demo of it in action."}, {"start": 324.0, "text": "This is the sample app that I showed you at the beginning of this presentation."}, {"start": 328.0, "text": "It showcases this AirPods Pro asset"}, {"start": 330.0, "text": "that I plan to use in a design review session later."}, {"start": 334.0, "text": "Watch what happens when we hit Open."}, {"start": 337.0, "text": "A moment ago, this was one object."}, {"start": 340.0, "text": "Now every part of it is individually interactive —"}, {"start": 344.0, "text": "grab one piece, leave the rest."}, {"start": 347.0, "text": "Let me show you how I got this working."}, {"start": 349.0, "text": "To enable people to move this assembly around,"}, {"start": 352.0, "text": "all we have to do is attach a ManipulationComponent to it."}, {"start": 355.0, "text": "That's our starting point."}, {"start": 356.0, "text": "Get the object manipulable so people can orient it,"}, {"start": 359.0, "text": "move it, and scale it with natural hand movements."}, {"start": 363.0, "text": "To learn more about ManipulationComponent"}, {"start": 365.0, "text": "see \"What's new in RealityKit\" from WWDC 25."}, {"start": 369.0, "text": "To enable people to move this assembly around,"}, {"start": 371.0, "text": "all we have to do is attach a ManipulationComponent to it."}, {"start": 376.0, "text": "To do that we move the ManipulationComponent down to the children."}, {"start": 379.0, "text": "Suddenly, the top enclosure can be pulled away"}, {"start": 382.0, "text": "while the bottom enclosure stays put."}, {"start": 384.0, "text": "A collaborator can rotate one of the ear buds"}, {"start": 387.0, "text": "while someone else is examining the other"}, {"start": 389.0, "text": "at the same time."}, {"start": 391.0, "text": "That shift, from \"thing to look at\" to \"thing to explore,\""}, {"start": 395.0, "text": "happens entirely because of where that component lives in the tree."}, {"start": 398.0, "text": "Nothing else changed."}, {"start": 401.0, "text": "Once you've pulled things apart,"}, {"start": 402.0, "text": "you can move ManipulationComponent back up to the root."}, {"start": 406.0, "text": "Now the whole spread moves as a single object again."}, {"start": 409.0, "text": "Reposition it together, rotate it, bring one section closer —"}, {"start": 413.0, "text": "the internal relationships stay exactly as you left them."}, {"start": 417.0, "text": "The hierarchy hasn't changed."}, {"start": 418.0, "text": "The geometry hasn't changed."}, {"start": 420.0, "text": "Just where the component lives in the tree."}, {"start": 423.0, "text": "And that's the whole idea here:"}, {"start": 425.0, "text": "component placement drives behavior."}, {"start": 427.0, "text": "Let's look at the code to make this work."}, {"start": 429.0, "text": "Remove the the ManipulationComponent and InputTargetComponent from the entity."}, {"start": 434.0, "text": "That makes this entity not manipulable."}, {"start": 437.0, "text": "Then the code iterates the sub-entities."}, {"start": 440.0, "text": "Add the InputTargetComponent"}, {"start": 442.0, "text": "and the ManipulationComponent to each sub-entity."}, {"start": 446.0, "text": "In the sample code I make sure"}, {"start": 448.0, "text": "to set the ManipulationComponent's releaseBehavior to .stay."}, {"start": 451.0, "text": "That makes it so the entity stays where the person puts it when they release it."}, {"start": 456.0, "text": "One more important note here,"}, {"start": 457.0, "text": "I'm specifically not showing the addition of the CollisionComponent."}, {"start": 461.0, "text": "But it is critical for event processing"}, {"start": 463.0, "text": "that your entities have collision components,"}, {"start": 466.0, "text": "don't forget to add them."}, {"start": 467.0, "text": "Of course, if you are going to open an assembly"}, {"start": 469.0, "text": "you probably want to close it too."}, {"start": 472.0, "text": "Closing an assembly follows the same process but in reverse."}, {"start": 476.0, "text": "Remove the Manipulation and InputTarget components"}, {"start": 479.0, "text": "from the sub-entities."}, {"start": 481.0, "text": "Add the Manipulation and InputTarget components"}, {"start": 484.0, "text": "back to the entity."}, {"start": 486.0, "text": "That's it, the tree is now able to be manipulated as one element"}, {"start": 490.0, "text": "as if it's closed or as independent entities if it's open."}, {"start": 494.0, "text": "Next up is clipping."}, {"start": 496.0, "text": "Any sufficiently complex asset has layers you can't see from the surface."}, {"start": 501.0, "text": "The internal structure of a building,"}, {"start": 503.0, "text": "the routing behind a panel,"}, {"start": 505.0, "text": "the infrastructure beneath a city block."}, {"start": 507.0, "text": "Clipping lets people see through the asset, literally,"}, {"start": 510.0, "text": "and it's a new RealityKit capability in visionOS 27."}, {"start": 515.0, "text": "Let me first show you a demo of it in action,"}, {"start": 517.0, "text": "then take you through how to edit the clipping planes."}, {"start": 521.0, "text": "Here you will notice the assembly, unclipped sitting in space,"}, {"start": 525.0, "text": "the clipping state is .off."}, {"start": 527.0, "text": "Then I turn clipping on and enable the clipped state of the assembly."}, {"start": 531.0, "text": "The clipping plane is inside the assembly perpendicular to one of the primary axis,"}, {"start": 536.0, "text": "the +z vector in this case."}, {"start": 539.0, "text": "It shows the internal structure."}, {"start": 541.0, "text": "The clipping state is .on."}, {"start": 543.0, "text": "Next the clipping plane moves around."}, {"start": 545.0, "text": "That is the really cool thing I'll show you how to do."}, {"start": 549.0, "text": "Before we do though, let's look at ClippingComponent."}, {"start": 552.0, "text": "ClippingComponent has 4 properties."}, {"start": 554.0, "text": "I'll go through the ones used in the sample code so you know what's there."}, {"start": 558.0, "text": "bounds is the property you'll be working with most —"}, {"start": 562.0, "text": "an axis-aligned bounding box in entity local space."}, {"start": 565.0, "text": "Anything outside it gets discarded by the renderer each frame."}, {"start": 570.0, "text": "shouldClipChildren defaults to false."}, {"start": 573.0, "text": "If you add this to a parent assembly and your children aren't clipping, that's why."}, {"start": 577.0, "text": "The sample code sets it to true."}, {"start": 580.0, "text": "shouldClipSelf defaults to true, which is almost always what you want."}, {"start": 584.0, "text": "Our goal is to make the bounds editable."}, {"start": 587.0, "text": "I'll show you how that's done after a quick look"}, {"start": 589.0, "text": "at the axis aligned bounding box."}, {"start": 592.0, "text": "The six faces of the bounding box become six interactive plane entities —"}, {"start": 596.0, "text": "one per axis, positive and negative."}, {"start": 599.0, "text": "Each face here represented by a different color."}, {"start": 605.0, "text": "Someone grabs the +x face"}, {"start": 607.0, "text": "and pulls it to reveal more of the interior,"}, {"start": 609.0, "text": "or pushes it back out to restore it."}, {"start": 612.0, "text": "Each plane controls exactly one scalar value in the bounds."}, {"start": 616.0, "text": "That's the entire interaction model."}, {"start": 617.0, "text": "Six planes, six numbers."}, {"start": 620.0, "text": "Keep that in mind as we work through the implementation."}, {"start": 623.0, "text": "To manage clipping we have a three-state machine."}, {"start": 626.0, "text": "Clipping can be .off — the assembly is not clipped."}, {"start": 629.0, "text": "When clipping is .on"}, {"start": 631.0, "text": "the model is clipped according to the bounding box,"}, {"start": 634.0, "text": "In the .editing state clipping planes are visible and interactive."}, {"start": 638.0, "text": "People can change the clipping bounds by moving them."}, {"start": 641.0, "text": "Let's look at it in action."}, {"start": 643.0, "text": "In the .off state there is no clipping of the assembly,"}, {"start": 646.0, "text": "only the outside of the assembly is visible."}, {"start": 650.0, "text": "In the .on state clipping is active"}, {"start": 652.0, "text": "and shows the inner workings and layout of the sub-assemblies."}, {"start": 656.0, "text": "In .editing state, clipping planes are on and people are able to move the planes."}, {"start": 661.0, "text": "As the planes move the clipping bounds are changed"}, {"start": 664.0, "text": "and more or less of the internal layout of the model becomes visible."}, {"start": 668.0, "text": "Let's look at the technical details that make this all work."}, {"start": 672.0, "text": "There are three components involved."}, {"start": 674.0, "text": "In the off state we have the ClippingBoundsCache,"}, {"start": 677.0, "text": "a custom component in the sample code."}, {"start": 680.0, "text": "It keeps track of the clipping bounds that were last edited"}, {"start": 683.0, "text": "and provides that value to the ClippingComponent"}, {"start": 685.0, "text": "when the state switches to .on."}, {"start": 688.0, "text": "In the .on state the ClippingComponent is created and added to the entity."}, {"start": 692.0, "text": "This is the RealityKit component we discussed earlier."}, {"start": 695.0, "text": "Geometry that's outside its bounds is discarded."}, {"start": 699.0, "text": "In .editing we add another custom component called ClippingTransformSync,"}, {"start": 703.0, "text": "the sample uses that to keep track of the assemblies transform"}, {"start": 706.0, "text": "and update the ClippingControl when the transform changes."}, {"start": 710.0, "text": "The ClippingControl is the entity we use to manage the clipping planes"}, {"start": 714.0, "text": "and make them interactive."}, {"start": 715.0, "text": "They are the visual affordance that allow people to see"}, {"start": 718.0, "text": "where the clipping planes are and edit them."}, {"start": 720.0, "text": "There are four coordinate frames involved."}, {"start": 723.0, "text": "The first is the world coordinate system, it's where everything else sits."}, {"start": 728.0, "text": "Model is where the model lives,"}, {"start": 730.0, "text": "and is the coordinate frame the clipping component operates in."}, {"start": 733.0, "text": "Changes to the bounds need to be made in this frame."}, {"start": 737.0, "text": "The clipping control frame is where we put the editing planes"}, {"start": 740.0, "text": "that allow people to change the ClippingComponent's bounding box."}, {"start": 744.0, "text": "The clipping plane coordinate frame is where the editing planes live"}, {"start": 748.0, "text": "and where the drag gesture events are expressed."}, {"start": 751.0, "text": "Changes to the position of the planes need to be in this frame"}, {"start": 754.0, "text": "and constrained to move in a direction expressed in the model frame."}, {"start": 759.0, "text": "World has two children."}, {"start": 761.0, "text": "The Clipping Control and Model Clipping Control contains the editing planes"}, {"start": 766.0, "text": "And Clipping Plane is the coordinate frame where the drag gestures are expressed."}, {"start": 771.0, "text": "The task is to get the change in drag gesture expressed in the Model frame"}, {"start": 775.0, "text": "constrained then converted back to the Clipping Plane."}, {"start": 779.0, "text": "To this point we've been talking about the clipping as a monolithic thing,"}, {"start": 783.0, "text": "but there are two distinct parts that are worth separating."}, {"start": 787.0, "text": "The ClippingComponent is in the model's coordinate space"}, {"start": 790.0, "text": "so to edit these bounds we need to have the change expressed"}, {"start": 793.0, "text": "and constrained in that coordinate frame."}, {"start": 796.0, "text": "The visual planes provided a visual understanding"}, {"start": 799.0, "text": "of what the movements do."}, {"start": 801.0, "text": "The planes need to move with the events as well,"}, {"start": 804.0, "text": "but they are expressed in the clipping plane coordinate system,"}, {"start": 807.0, "text": "so updates to their positions need to be expressed in their coordinate frame."}, {"start": 812.0, "text": "In both frames the change needs to be constrained"}, {"start": 815.0, "text": "to the direction normal to the bounding box plane."}, {"start": 819.0, "text": "Let me show you how all this fits together."}, {"start": 822.0, "text": "There are 4 distinct steps from the drag gesture"}, {"start": 825.0, "text": "to updates for the bounds and plane position."}, {"start": 827.0, "text": "I add a drag gesture to the clipping planes,"}, {"start": 830.0, "text": "one for each plane."}, {"start": 832.0, "text": "Again, this is the coordinate frame the events arrive in."}, {"start": 836.0, "text": "I transform that into the World coordinate frame."}, {"start": 839.0, "text": "From there, I transform to the Model frame."}, {"start": 842.0, "text": "Then I constrain the delta to the appropriate direction —"}, {"start": 846.0, "text": "+x, -y, etcetera — depending on which plane the person is moving."}, {"start": 851.0, "text": "Now I have the value I need,"}, {"start": 853.0, "text": "in the coordinate frame I need, constrained to the correct direction,"}, {"start": 858.0, "text": "and can update the clipping bounds."}, {"start": 860.0, "text": "To update the plane's location"}, {"start": 862.0, "text": "I'll have to convert this vector to the plane's coordinate frame."}, {"start": 865.0, "text": "We'll look at that in just a second."}, {"start": 867.0, "text": "For more information about the gesture component"}, {"start": 870.0, "text": "check out \"Better Together: SwiftUI and RealityKit\" from WWDC 25."}, {"start": 876.0, "text": "With the big picture in mind, let's look at each step in detail."}, {"start": 880.0, "text": "The gesture is expressed in the Clipping Plane coordinate frame."}, {"start": 884.0, "text": "It will look something like this."}, {"start": 886.0, "text": "The drag delta has values 0.5, -0.75, and 0.1."}, {"start": 892.0, "text": "These values are the expression of the drag delta"}, {"start": 895.0, "text": "in the Clipping Plane coordinate frame."}, {"start": 898.0, "text": "The task is to change the expression of this vector"}, {"start": 901.0, "text": "to the Model coordinate frame."}, {"start": 903.0, "text": "The drag delta vector is transformed"}, {"start": 906.0, "text": "from the Clipping Plane coordinate frame into the World coordinate frame."}, {"start": 910.0, "text": "Keep in mind, the vector hasn't changed,"}, {"start": 913.0, "text": "only the coordinate frame the vector is represented in."}, {"start": 916.0, "text": "Since Clipping Plane and World are not the same coordinate frame,"}, {"start": 920.0, "text": "the numbers change."}, {"start": 921.0, "text": "It's the same vector, just a different representation."}, {"start": 925.0, "text": "Before we can update the clipping bounds,"}, {"start": 927.0, "text": "the drag delta vector must be in the Model coordinate frame."}, {"start": 931.0, "text": "So, we do one more transform from the World coordinate frame"}, {"start": 934.0, "text": "into the Model coordinate frame."}, {"start": 936.0, "text": "Again, the vector hasn't changed,"}, {"start": 939.0, "text": "just the coordinate frame it is represented in."}, {"start": 942.0, "text": "Now comes the mathematical magic."}, {"start": 944.0, "text": "We project the drag delta onto the proper direction,"}, {"start": 948.0, "text": "which is just a fancy math way to say how long the drag delta is"}, {"start": 952.0, "text": "in the direction we care about."}, {"start": 955.0, "text": "Let's remove the AirPods Pro case so we can see the process better."}, {"start": 959.0, "text": "Projection sounds complex, but it really is just"}, {"start": 962.0, "text": "how long is the drag delta along the direction we care about."}, {"start": 967.0, "text": "It's like measuring the shadow of the drag delta vector cast"}, {"start": 971.0, "text": "on the direction vector."}, {"start": 973.0, "text": "Here is the math equation, don't let it scare you."}, {"start": 976.0, "text": "I'll break it down piece by piece."}, {"start": 978.0, "text": "First it's finding a vector in the direction we care about."}, {"start": 982.0, "text": "You might have done this before, it's the vector dived by its length squared."}, {"start": 986.0, "text": "In our case this is simple,"}, {"start": 988.0, "text": "the direction we care about is the normal to the plane,"}, {"start": 991.0, "text": "+x here, or {1, 0, 0}."}, {"start": 995.0, "text": "Then we do a dot product, which is a type of vector multiplication,"}, {"start": 999.0, "text": "with the drag delta and the direction vector."}, {"start": 1002.0, "text": "That gives us the amount we want the bounding box to change,"}, {"start": 1006.0, "text": "but it's just a number, we also need the direction."}, {"start": 1009.0, "text": "So we multiply the direction we care about, the normal to the plane,"}, {"start": 1014.0, "text": "by the amount we found in the last step."}, {"start": 1016.0, "text": "Now I have the constrained delta in the Model coordinate frame."}, {"start": 1020.0, "text": "That is exactly what I need"}, {"start": 1022.0, "text": "to updated the bounding box of the ClippingComponent."}, {"start": 1026.0, "text": "Here is the assembly clipped, the inside is as visible as the outside."}, {"start": 1030.0, "text": "That's pretty cool."}, {"start": 1032.0, "text": "Now, I go through the same constraint process."}, {"start": 1035.0, "text": "But this time to the Clipping Plane coordinate frame."}, {"start": 1038.0, "text": "We have to transform the constrained drag delta"}, {"start": 1041.0, "text": "from the Model coordinate frame into the Clipping Plane coordinate frame"}, {"start": 1046.0, "text": "then we project that down onto the plane's normal,"}, {"start": 1049.0, "text": "the same way we did last time."}, {"start": 1051.0, "text": "This gives us the value we need to move the plane."}, {"start": 1054.0, "text": "But, since it's been projected onto the normal"}, {"start": 1057.0, "text": "the change is constrained to move only in that direction,"}, {"start": 1060.0, "text": "instead of where ever the person moved their hand."}, {"start": 1062.0, "text": "That makes sure the changes from the gesture feel natural."}, {"start": 1067.0, "text": "Here we see the planes turned on and waiting for people to interact."}, {"start": 1071.0, "text": "We keep them on in the .editing state so that people know they can reach out"}, {"start": 1075.0, "text": "and move any one of these six planes."}, {"start": 1077.0, "text": "6 planes, 4 coordinate frames."}, {"start": 1080.0, "text": "Simple transformations between each"}, {"start": 1082.0, "text": "makes the individual calculations easier to reason about."}, {"start": 1086.0, "text": "Once you have the hierarchy of the coordinate systems in mind"}, {"start": 1089.0, "text": "and the really cool math trick of projection,"}, {"start": 1092.0, "text": "you can make this interaction feel natural in your apps."}, {"start": 1096.0, "text": "Now, let's talk about automatic expansion of the sub-assemblies"}, {"start": 1099.0, "text": "that make up a 3D model."}, {"start": 1101.0, "text": "People use this to reveal the model's inner structure."}, {"start": 1104.0, "text": "This feature can be great for a mechanical assembly, a building,"}, {"start": 1108.0, "text": "really any asset where understanding"}, {"start": 1111.0, "text": "how the parts relate to the whole would be helpful."}, {"start": 1113.0, "text": "I want the model to expand in an intuitive way."}, {"start": 1117.0, "text": "But I don't want to force the person to choose that direction."}, {"start": 1120.0, "text": "We are going to use a bit of math so the code can make the decision."}, {"start": 1124.0, "text": "I'll walk you through that, and show you exactly how it works."}, {"start": 1128.0, "text": "When an assembly loads,"}, {"start": 1130.0, "text": "its children sit exactly where they are defined in the file."}, {"start": 1134.0, "text": "For a well-constructed asset that means they're probably overlapping —"}, {"start": 1138.0, "text": "nested inside each other the way they exist in the real object."}, {"start": 1142.0, "text": "That's correct, but it's not useful for exploration."}, {"start": 1145.0, "text": "Expansion fans the children apart along a single axis,"}, {"start": 1150.0, "text": "giving each one space to be seen and grabbed independently."}, {"start": 1154.0, "text": "One tap, the assembly opens itself."}, {"start": 1157.0, "text": "The process of doing that is not complicated,"}, {"start": 1159.0, "text": "let's look at it a piece at a time."}, {"start": 1162.0, "text": "We could display the assembly spread along the x-axis, like this."}, {"start": 1166.0, "text": "It does capture the idea of the pieces pulling apart in space"}, {"start": 1169.0, "text": "to expose the interior layout."}, {"start": 1171.0, "text": "But, we'd like the expansion to feel more natural."}, {"start": 1175.0, "text": "More like what a person would expect in a design review."}, {"start": 1178.0, "text": "And have it expand along the y-axis like this."}, {"start": 1182.0, "text": "The question is, how does the code choose which axis to expand along?"}, {"start": 1186.0, "text": "For that I'll take a brief diversion into two concepts: variance and weighting."}, {"start": 1192.0, "text": "A low variance is just a way to say"}, {"start": 1195.0, "text": "all the values we have are basically in the same spot."}, {"start": 1199.0, "text": "For example, in this dartboard diagram the values are anything,"}, {"start": 1203.0, "text": "like ice cream sales to incidence of sunburn."}, {"start": 1206.0, "text": "It's just a way to get across the idea that values,"}, {"start": 1209.0, "text": "whatever values, are close together."}, {"start": 1213.0, "text": "A high variance will have the values spread out."}, {"start": 1216.0, "text": "Now that we have a feel for variance, let's look at it a little closer."}, {"start": 1220.0, "text": "I'll move to one dimension to make things a little clearer."}, {"start": 1223.0, "text": "Again these values are just numbers, they could represent anything,"}, {"start": 1227.0, "text": "the frequency of 6 guitar strings, or anything else"}, {"start": 1230.0, "text": "that has a single value per sample."}, {"start": 1233.0, "text": "Each of our points is placed on a number line"}, {"start": 1235.0, "text": "along with an indicator of how far the value is from the average."}, {"start": 1239.0, "text": "The distance from average is called deviation."}, {"start": 1243.0, "text": "This is the first step in finding the variance."}, {"start": 1246.0, "text": "Next, is to square each of the deviation values"}, {"start": 1249.0, "text": "then add them together,"}, {"start": 1250.0, "text": "and divide by the count to find the average."}, {"start": 1253.0, "text": "That's the variance."}, {"start": 1254.0, "text": "It's simple enough math,"}, {"start": 1256.0, "text": "the technical terms \"variance\" and \"deviation\""}, {"start": 1259.0, "text": "is usually what gets people."}, {"start": 1261.0, "text": "Conceptually, it's just a way to numerically specify"}, {"start": 1264.0, "text": "how much our set of values is spread out from the average."}, {"start": 1268.0, "text": "Now consider, some of these values might be more important than others."}, {"start": 1273.0, "text": "That's where weighting comes in."}, {"start": 1275.0, "text": "We use it to distinguish the importance of individual values."}, {"start": 1279.0, "text": "Each of the points now has a radius to represent a weight factor,"}, {"start": 1283.0, "text": "the larger the circle the more important the value."}, {"start": 1287.0, "text": "The weighting factor could be anything, saturation of a gradient,"}, {"start": 1290.0, "text": "or any other value that expresses importance."}, {"start": 1293.0, "text": "We are staying abstract to illustrate the process."}, {"start": 1297.0, "text": "We use the weight, or importance, to calculate a weighted variance."}, {"start": 1301.0, "text": "In addition to squaring the deviation values"}, {"start": 1304.0, "text": "we multiply each by its weight."}, {"start": 1307.0, "text": "And boom, now we have a weighted variance."}, {"start": 1310.0, "text": "With the weight, each value can have a different importance."}, {"start": 1313.0, "text": "And that's exactly what we'll do"}, {"start": 1315.0, "text": "to figure out what axis to expand our assembly along."}, {"start": 1319.0, "text": "We'll calculate the \"volume-weighted position variance\","}, {"start": 1322.0, "text": "what a mouthful,"}, {"start": 1323.0, "text": "along each axis and expand along the axis with the largest variance."}, {"start": 1329.0, "text": "Here, notice the table: one row for each of the sub-assemblies."}, {"start": 1334.0, "text": "It shows their volumes, and their positions."}, {"start": 1337.0, "text": "These are the values I'll use to find the natural axis to expand along."}, {"start": 1342.0, "text": "I'll use the x, y, and z values to calculate the variance in each direction."}, {"start": 1347.0, "text": "The dots are sized to reflect the volume of each element."}, {"start": 1351.0, "text": "I'll use the volume as the weighting factor."}, {"start": 1355.0, "text": "Here is the volume-weighted variance for the x axis."}, {"start": 1359.0, "text": "Most of the sub-assemblies lie at or very near the same position"}, {"start": 1362.0, "text": "along the x axis."}, {"start": 1364.0, "text": "That leaves us with a small variance along x."}, {"start": 1368.0, "text": "Since most of the sub-assemblies are at or near each other"}, {"start": 1371.0, "text": "the volume weighting factor doesn't make a difference along the x axis."}, {"start": 1375.0, "text": "The two ear buds are spread out"}, {"start": 1377.0, "text": "on the x axis and do their best to contribute,"}, {"start": 1380.0, "text": "but there volume is not enough to make up"}, {"start": 1382.0, "text": "for all the other elements at the same place."}, {"start": 1385.0, "text": "The result along the z axis is even smaller."}, {"start": 1389.0, "text": "The Bottom insert does have some volume to weight its contribution,"}, {"start": 1393.0, "text": "but it's too close to the average position to make much of a dent."}, {"start": 1397.0, "text": "The Hinge and Lid retention magnet have too small of a volume"}, {"start": 1401.0, "text": "to pull the variance very far."}, {"start": 1404.0, "text": "This takes us to the y axis, the clear winner here."}, {"start": 1407.0, "text": "The large parts are further away along the y axis"}, {"start": 1410.0, "text": "and the larger volume provides weight to their distance."}, {"start": 1413.0, "text": "With y being the clear winner"}, {"start": 1415.0, "text": "we assemble a set of FromToBy animations"}, {"start": 1418.0, "text": "to move the sub-assemblies into position along the y axis."}, {"start": 1423.0, "text": "And there we have it,"}, {"start": 1424.0, "text": "the interior of the model exposed for people to interact with."}, {"start": 1428.0, "text": "There is a lot I've covered today"}, {"start": 1430.0, "text": "from showing you how to prepare an asset hierarchy,"}, {"start": 1433.0, "text": "manipulate its parts,"}, {"start": 1434.0, "text": "use a clipping plane to look through a complex assembly,"}, {"start": 1438.0, "text": "and even expand the parts out along an axis"}, {"start": 1440.0, "text": "to give you a detailed view of the tiniest part of your assembly."}, {"start": 1445.0, "text": "This workflow can help you build design review apps"}, {"start": 1447.0, "text": "that can greatly enhance people's productivity."}, {"start": 1451.0, "text": "Download and explore the sample project from developer.apple.com"}, {"start": 1455.0, "text": "I'd highly encourage you"}, {"start": 1456.0, "text": "to familiarize yourself with concepts in statistics,"}, {"start": 1460.0, "text": "vector math, and linear algebra."}, {"start": 1462.0, "text": "Those are the parts of math we leaned heavily on today,"}, {"start": 1465.0, "text": "hopefully you find them a little less scary now than you did before."}, {"start": 1470.0, "text": "If you'd like to control models in real time from your Mac app,"}, {"start": 1473.0, "text": "the spatial preview framework might be a great choice for you."}, {"start": 1476.0, "text": "Do check out the session"}, {"start": 1478.0, "text": "\"Discover the spatial preview framework\" to learn more."}, {"start": 1481.0, "text": "Additionally, you could even augment a physical object"}, {"start": 1484.0, "text": "like a race car simulator cockpit"}, {"start": 1486.0, "text": "and overlay your virtual content on top of the simulator"}, {"start": 1489.0, "text": "and explore its internal structure with what you learned in this session."}, {"start": 1494.0, "text": "To learn more about augmenting physical objects"}, {"start": 1496.0, "text": "please check out the \"Explore enhancements to visionOS object tracking\" session."}, {"start": 1502.0, "text": "Thanks again for your attention today"}, {"start": 1503.0, "text": "and I look forward to seeing the cool stuff you do with these ideas in your apps."}]}