{"$schema": "https://raw.githubusercontent.com/guitaripod/wwdc-sessions/master/schema/transcript.schema.json", "id": "wwdc2026-279", "language": "eng", "source": "https://developer.apple.com/videos/play/wwdc2026/279", "wordCount": 3896, "segments": [{"start": 7.0, "text": "Hi, my name is Dennis"}, {"start": 9.0, "text": "and I'm a software engineer on the RealityKit team."}, {"start": 11.0, "text": "Welcome to my session: \"Explore advances in RealityKit\""}, {"start": 15.0, "text": "where I'll be introducing you to some of the latest advancements in RealityKit."}, {"start": 20.0, "text": "In 2019 we introduced RealityKit, a framework that enables you to easily build"}, {"start": 25.0, "text": "3D spatial experiences across a wide range of Apple platforms."}, {"start": 29.0, "text": "With RealityKit you can build your apps and games once and deploy them to"}, {"start": 33.0, "text": "visionOS,"}, {"start": 35.0, "text": "iOS,"}, {"start": 37.0, "text": "iPadOS,"}, {"start": 39.0, "text": "macOS,"}, {"start": 40.0, "text": "and tvOS."}, {"start": 44.0, "text": "And this year we are introducing a better way to build these apps and games"}, {"start": 48.0, "text": "with Reality Composer Pro 3."}, {"start": 51.0, "text": "Check out these sessions to learn more about Reality Composer Pro 3:"}, {"start": 54.0, "text": "from its fast, iterative scene editing tools to its powerful graph interfaces"}, {"start": 59.0, "text": "for modeling complex particle and character behaviors."}, {"start": 62.0, "text": "And thanks to the amazing feedback we got from all of you"}, {"start": 65.0, "text": "we are adding exciting new features to RealityKit this year."}, {"start": 68.0, "text": "These features will make it easier than ever"}, {"start": 70.0, "text": "to craft high-fidelity, immersive experiences."}, {"start": 73.0, "text": "To start I'll go over the advancements"}, {"start": 75.0, "text": "we made to RealityKit's lighting and shadows,"}, {"start": 77.0, "text": "enabling more realistic effects that blend seamlessly into your world."}, {"start": 81.0, "text": "Then I'll introduce RealityKit's navigation mesh as a way to help your players and NPCs"}, {"start": 85.0, "text": "navigate the worlds you have crafted."}, {"start": 88.0, "text": "I'll show you how to build elaborate cloth simulations"}, {"start": 91.0, "text": "that can be used to dress your virtual furnishings and characters."}, {"start": 95.0, "text": "If you're not careful, utilizing these features can hinder performance."}, {"start": 98.0, "text": "To help you tackle this I'll cover the tools available to you"}, {"start": 100.0, "text": "to both mitigate and monitor such issues."}, {"start": 104.0, "text": "Once we have a handle on performance I'll demonstrate"}, {"start": 106.0, "text": "how you can bring a piece of the real world"}, {"start": 108.0, "text": "into your apps and games with RealityKit's ability to render 3D gaussian splats."}, {"start": 113.0, "text": "Finally, I'll talk about RealityKit's immersive audio features"}, {"start": 116.0, "text": "and how they can improve the spatial realism of your virtual worlds."}, {"start": 120.0, "text": "Let's begin right away and go over what's new in RealityKit's lighting and shadows"}, {"start": 123.0, "text": "through a new game we have built: Chaparral Village."}, {"start": 126.0, "text": "In this game, a mysterious village is transported into the player's world."}, {"start": 131.0, "text": "They are then shrunk down by an owl and must navigate their way"}, {"start": 133.0, "text": "through this village in order to get to the alchemy area"}, {"start": 136.0, "text": "where they'll brew a number of potions under the owl's direction."}, {"start": 140.0, "text": "Let's take a look inside the alchemy area of Chaparral Village."}, {"start": 144.0, "text": "Hmmm, this looks pretty good but it seems a little dark in some of the corners."}, {"start": 149.0, "text": "Here, we can take advantage of RealityKit's support"}, {"start": 151.0, "text": "for lightmaps to enhance the lighting of the area's interior."}, {"start": 155.0, "text": "Using the light baker in Reality Composer Pro 3"}, {"start": 158.0, "text": "I'll create an indirect lighting lightmap and apply it to the scene."}, {"start": 162.0, "text": "There, the corners and shadows have brightened"}, {"start": 165.0, "text": "as they're now correctly seeing contributions from reflected light."}, {"start": 169.0, "text": "RealityKit's API supports attaching your own lightmap textures for:"}, {"start": 173.0, "text": "indirect lighting,"}, {"start": 174.0, "text": "ambient occlusion,"}, {"start": 176.0, "text": "and beauty."}, {"start": 177.0, "text": "But to get the best experience I recommend using Reality Composer Pro 3's light baker"}, {"start": 182.0, "text": "to generate these instead."}, {"start": 184.0, "text": "To learn more about light baking in Reality Composer Pro 3"}, {"start": 187.0, "text": "please check out the session:"}, {"start": 189.0, "text": "\"Iterate your spatial scenes faster with Reality Composer Pro 3.\""}, {"start": 193.0, "text": "Lightmaps help you render complex lighting effects,"}, {"start": 196.0, "text": "but only for static lighting."}, {"start": 198.0, "text": "We can, however, bring one of these complex effects to dynamic lights"}, {"start": 201.0, "text": "with RealityKit's soft shadows."}, {"start": 204.0, "text": "By default, the shadows in RealityKit have hard edges."}, {"start": 207.0, "text": "This is accurate in some scenarios. For instance,"}, {"start": 209.0, "text": "if the light source is infinitesimally small."}, {"start": 212.0, "text": "But this is no longer accurate when the light has some area."}, {"start": 215.0, "text": "Now, the shadow cast exhibits softer edges"}, {"start": 218.0, "text": "as only some of the light is obscured in these regions,"}, {"start": 222.0, "text": "also known as the penumbra of the shadow."}, {"start": 224.0, "text": "The size of the penumbra is influenced by the light's area."}, {"start": 228.0, "text": "The greater the light's area, the greater the penumbra."}, {"start": 231.0, "text": "Here is the hearth in the alchemy area."}, {"start": 234.0, "text": "I am currently simulating the light coming from the hearth using a spotlight."}, {"start": 238.0, "text": "Let's make the shadows cast by this spotlight soft."}, {"start": 242.0, "text": "I'll start by getting the hearth's spotlight's shadow."}, {"start": 246.0, "text": "I'll then update the shadow's lightSize variable."}, {"start": 249.0, "text": "This represents the diameter, in meters, of the light."}, {"start": 252.0, "text": "By default it is set to 0 producing a hard shadow."}, {"start": 256.0, "text": "I'll update it to 0.7 meters to produce a nice soft shadow."}, {"start": 262.0, "text": "I also have to set the shadow's quality."}, {"start": 264.0, "text": "This variable controls the number of samples used to calculate the soft shadow."}, {"start": 269.0, "text": "Setting the quality to high can produce more pleasing results"}, {"start": 273.0, "text": "but with a greater performance penalty."}, {"start": 275.0, "text": "For this reason, I'll stick to medium"}, {"start": 277.0, "text": "as this still looks good at the viewing distance."}, {"start": 281.0, "text": "Note that if I had set the quality to low,"}, {"start": 283.0, "text": "the shadow would be hard regardless of the light size."}, {"start": 286.0, "text": "The quality must be set to either medium or high to achieve soft shadows."}, {"start": 292.0, "text": "Finally, I add the shadow back to the hearthSpotLight entity."}, {"start": 297.0, "text": "Now, if I apply the changes to the shadow,"}, {"start": 300.0, "text": "we can see the shadows have softened,"}, {"start": 302.0, "text": "especially the shadow cast by the cauldron."}, {"start": 306.0, "text": "So far all of the lighting I've shown you is casting onto virtual objects."}, {"start": 310.0, "text": "But what if we wanted your lights to extent beyond the virtual world?"}, {"start": 313.0, "text": "To cast lights onto your world?"}, {"start": 316.0, "text": "Here is a planetarium projector I've built with RealityKit."}, {"start": 320.0, "text": "As the projector spins, my real world environment is painted"}, {"start": 323.0, "text": "with virtual stars and nebulae."}, {"start": 325.0, "text": "Notice how they realistically conform to the walls in my room."}, {"start": 329.0, "text": "I was able to achieve this effect with two RealityKit features:"}, {"start": 332.0, "text": "projective textures and physical space lighting."}, {"start": 336.0, "text": "Let's first take a look at projective textures."}, {"start": 339.0, "text": "Imagine taking a flashlight and a piece of film."}, {"start": 343.0, "text": "If I shined light through the film the image on the film will now appear"}, {"start": 347.0, "text": "on whatever surface the flashlight is pointing at."}, {"start": 350.0, "text": "This effect is what RealityKit's projective texture feature emulates."}, {"start": 354.0, "text": "With projective textures you can render various effects"}, {"start": 358.0, "text": "from the pattern caused by light shining through an intricate window"}, {"start": 361.0, "text": "or, when animated, the bright undulating caustics seen on the sea floor."}, {"start": 366.0, "text": "For my virtual planetarium,"}, {"start": 368.0, "text": "the stars and nebulae are attached as projective textures to rotating spotlights."}, {"start": 373.0, "text": "To build one of these spotlights I'll start by creating"}, {"start": 376.0, "text": "an Entity and attaching a SpotLightComponent to it."}, {"start": 379.0, "text": "I'll make sure the color is white"}, {"start": 381.0, "text": "so I don't apply a tint to the projective texture."}, {"start": 384.0, "text": "I'll tweak the other parameters as necessary for the room I am in."}, {"start": 387.0, "text": "For example, a larger room may need more intensity"}, {"start": 390.0, "text": "to make the projective texture visible on the walls."}, {"start": 394.0, "text": "I'll then generate a stars and nebulae texture."}, {"start": 396.0, "text": "This will be the texture that I project from the planetarium."}, {"start": 400.0, "text": "Finally, I'll create and attach my SpotLight's projectiveTexture component"}, {"start": 405.0, "text": "and attach the texture I just generated."}, {"start": 408.0, "text": "Here we can see the stars and nebulae projecting onto virtual walls."}, {"start": 412.0, "text": "But how do we get them to project onto the walls in my real room?"}, {"start": 415.0, "text": "For that, we must enable physical space lighting."}, {"start": 418.0, "text": "This feature enables virtual lights to interact with system environments"}, {"start": 422.0, "text": "or the world around you using RealityKit's scene understanding mesh."}, {"start": 426.0, "text": "Currently, physical space lighting is only supported for spotlights and point lights."}, {"start": 433.0, "text": "Let's enable this feature on our planetarium's spotlights."}, {"start": 436.0, "text": "Here we have the spotLightEntity from earlier."}, {"start": 439.0, "text": "To enable physical space lighting all we have to do"}, {"start": 442.0, "text": "is add the SpotLight's SurroundingsLight component."}, {"start": 445.0, "text": "That's it."}, {"start": 447.0, "text": "Now the the stars and nebulae project onto my real world room."}, {"start": 452.0, "text": "We have also enabled this physical space lighting effect"}, {"start": 455.0, "text": "in the alchemy area of the Chaparral village."}, {"start": 458.0, "text": "Speaking of which, how does the player's character"}, {"start": 461.0, "text": "choose the path to take to the alchemy area?"}, {"start": 464.0, "text": "This path is determined using RealityKit's navigation mesh."}, {"start": 468.0, "text": "Let me illustrate how such a mesh works with a simple example."}, {"start": 471.0, "text": "Imagine a scene where I start off on one side of the map"}, {"start": 475.0, "text": "and my goal is to make it to the flags on the other side."}, {"start": 477.0, "text": "Seems simple enough."}, {"start": 479.0, "text": "But oh no! Some obstacles have appeared in my way!"}, {"start": 482.0, "text": "I can use a navigation mesh to define the traversable parts of this scene,"}, {"start": 486.0, "text": "avoiding the dense forests."}, {"start": 488.0, "text": "RealityKit can then use this navigation mesh to calculate a path to the flag."}, {"start": 493.0, "text": "But what if I don't want to rule out the forests entirely?"}, {"start": 496.0, "text": "I can make my way through the forests, I'll just be a slower."}, {"start": 499.0, "text": "We can reflect this in the navigation mesh"}, {"start": 501.0, "text": "by giving these regions a different traversal cost."}, {"start": 505.0, "text": "This will reflect the lower speed I'll go at when going through the forest."}, {"start": 509.0, "text": "Now, if I calculate a path it will take into account this cost"}, {"start": 512.0, "text": "and pick a new route for me."}, {"start": 514.0, "text": "But what's this?"}, {"start": 515.0, "text": "A rift has broken up my scene and left the flag stranded."}, {"start": 519.0, "text": "This has led to two regions represented with two disconnected navigation meshes."}, {"start": 524.0, "text": "But not to worry, I can connect them using an off mesh connection,"}, {"start": 527.0, "text": "in this case a bridge."}, {"start": 529.0, "text": "The placement of this connection, however, means that a new path must be calculated."}, {"start": 535.0, "text": "To use RealityKit's navigation mesh,"}, {"start": 537.0, "text": "you first need to define a NavigationMeshResource."}, {"start": 541.0, "text": "This holds the geometric data for the navigation mesh, including labeled areas,"}, {"start": 545.0, "text": "custom flags for those areas, and the connection between areas."}, {"start": 549.0, "text": "This can either be defined using the Swift API or in Reality Composer Pro 3."}, {"start": 554.0, "text": "To learn how to build navigation meshes in Reality Composer Pro 3"}, {"start": 557.0, "text": "check out the session:"}, {"start": 558.0, "text": "\"Supercharge your spatial workflows with Reality Composer Pro 3.\""}, {"start": 563.0, "text": "This NavigationMeshResource is then"}, {"start": 565.0, "text": "fed into a NavigationComponent."}, {"start": 568.0, "text": "This component has a filter that is used to define the cost of areas,"}, {"start": 572.0, "text": "and which areas to include or exclude given the area's flag."}, {"start": 576.0, "text": "The NavigationComponent is then used"}, {"start": 579.0, "text": "by the NavigationController to calculate the path."}, {"start": 582.0, "text": "Either synchronously or asynchronously."}, {"start": 586.0, "text": "In Chaparral Village the navigation mesh is queried"}, {"start": 589.0, "text": "in the navigate entity extension function."}, {"start": 592.0, "text": "Within navigate I first create a NavigationController."}, {"start": 596.0, "text": "The controller requires an entity with a navigation component."}, {"start": 599.0, "text": "I'll get this from the entity itself."}, {"start": 602.0, "text": "I'll then use the async computePath function to get the path"}, {"start": 606.0, "text": "from the entity's current position"}, {"start": 608.0, "text": "to a desired position where the player tapped."}, {"start": 612.0, "text": "If the result is nil,"}, {"start": 614.0, "text": "then the NavigationController could not find a valid path"}, {"start": 617.0, "text": "and we return."}, {"start": 619.0, "text": "If this array is empty, then we have reached our destination"}, {"start": 623.0, "text": "and we return as well."}, {"start": 625.0, "text": "Otherwise, this array holds a collection of path nodes"}, {"start": 628.0, "text": "representing the path the NavigationController computed."}, {"start": 632.0, "text": "I'll iterate over these nodes so that I can determine"}, {"start": 634.0, "text": "the final path the entity should take."}, {"start": 637.0, "text": "If it's a node on the navigation mesh itself,"}, {"start": 640.0, "text": "I can just append its position to our path."}, {"start": 643.0, "text": "If it's an off-mesh connection, then we have to traverse a ladder."}, {"start": 647.0, "text": "I'll handle this situation separately."}, {"start": 650.0, "text": "Here, we can see the player's character navigating the village"}, {"start": 653.0, "text": "with RealityKit's navigation mesh."}, {"start": 655.0, "text": "Once they reach the top they must walk through two curtains"}, {"start": 658.0, "text": "that adorn the entrance to the alchemy area."}, {"start": 661.0, "text": "These curtains were built with RealityKit's advanced cloth simulation."}, {"start": 665.0, "text": "In RealityKit's cloth simulation the cloth is described through a mesh,"}, {"start": 670.0, "text": "where the vertices represent particles"}, {"start": 672.0, "text": "and the edges connecting the vertices represent springs."}, {"start": 676.0, "text": "Given a mesh with enough vertices,"}, {"start": 679.0, "text": "RealityKit can accurately simulate everything"}, {"start": 681.0, "text": "from the flow of this golden dress to these bed covers."}, {"start": 687.0, "text": "See how, as I pull off these covers, they realistically crease and fold,"}, {"start": 691.0, "text": "all in real time."}, {"start": 693.0, "text": "To use RealityKit's cloth simulation"}, {"start": 695.0, "text": "you need to add a cloth body component to your scene."}, {"start": 698.0, "text": "This component represents cloths themselves."}, {"start": 701.0, "text": "It contains a reference to its material properties"}, {"start": 703.0, "text": "and a cloth mesh resource that describes the layout of the particles and springs."}, {"start": 708.0, "text": "You can also add a cloth collider component which represent the rigid objects"}, {"start": 712.0, "text": "that a cloth can collide with, like the bed or mannequin from earlier."}, {"start": 716.0, "text": "Just as with the cloth body component,"}, {"start": 718.0, "text": "it contains a reference to its material properties"}, {"start": 721.0, "text": "and the geometry of the collider itself."}, {"start": 724.0, "text": "To run the simulation you need to add the cloth simulation component."}, {"start": 728.0, "text": "This component contains an array of materials"}, {"start": 731.0, "text": "that are referenced by the cloth bodies and colliders."}, {"start": 734.0, "text": "Each material has a set of properties like spring stiffness and friction."}, {"start": 739.0, "text": "The specific properties depend on whether"}, {"start": 741.0, "text": "it's describing a cloth or cloth collider material."}, {"start": 744.0, "text": "The simulation itself also has a number of properties"}, {"start": 748.0, "text": "that effect all descendent entities involved in the cloth simulation."}, {"start": 752.0, "text": "Some of these properties include which solver to use,"}, {"start": 755.0, "text": "the gravity to apply, and how big of a time step to take in the simulation."}, {"start": 760.0, "text": "In Chaparral Village, RealityKit's cloth simulation is used to adorn"}, {"start": 764.0, "text": "curtains to the entrance of the alchemy area."}, {"start": 767.0, "text": "But how do we implement the hoops that keep the curtains from falling down?"}, {"start": 771.0, "text": "For that we use the custom curtain pin component."}, {"start": 775.0, "text": "Here we iterate over pins, an array that holds a tuple of"}, {"start": 779.0, "text": "these custom components,"}, {"start": 780.0, "text": "and their corresponding entity."}, {"start": 783.0, "text": "We use this entity's position"}, {"start": 784.0, "text": "as the position at which we are pinning the curtain."}, {"start": 787.0, "text": "Then we build a sphere to represent the size of the pin itself."}, {"start": 791.0, "text": "The size is controlled through the custom component."}, {"start": 795.0, "text": "We use the position"}, {"start": 796.0, "text": "and the sphere we just created"}, {"start": 798.0, "text": "to get an array of all the vertices that should be pinned or made immovable."}, {"start": 803.0, "text": "We then set these vertices to kinematic."}, {"start": 806.0, "text": "Kinematic vertices can only be moved by an entity's transform"}, {"start": 809.0, "text": "and not by the cloth simulation itself."}, {"start": 812.0, "text": "This will essentially keep them in place."}, {"start": 814.0, "text": "And with that, we can pin the curtains to the alchemy area"}, {"start": 817.0, "text": "at the position of the hoops and keep them from falling over."}, {"start": 821.0, "text": "At this point I have gone over a number of features that,"}, {"start": 824.0, "text": "when not used with care, may come at a performance cost."}, {"start": 828.0, "text": "To help tackle this I'll first cover a technique"}, {"start": 830.0, "text": "that can be used to improve performance"}, {"start": 832.0, "text": "and then how you can track a performance indicator"}, {"start": 835.0, "text": "of your apps and games so that they can adapt accordingly."}, {"start": 839.0, "text": "Mesh level of detail refers to the process of rendering geometry"}, {"start": 842.0, "text": "at a lower detail such that the visual impact is negligible."}, {"start": 846.0, "text": "To demonstrate this I'll use the cauldron"}, {"start": 848.0, "text": "from the alchemy area in Chaparral Village."}, {"start": 851.0, "text": "By convention, these so-called \"levels of detail,\" or LODs,"}, {"start": 855.0, "text": "start at index 0."}, {"start": 857.0, "text": "Notice how the geometric complexity of the cauldron decreases"}, {"start": 860.0, "text": "as I go to"}, {"start": 861.0, "text": "LOD one,"}, {"start": 862.0, "text": "two,"}, {"start": 864.0, "text": "three,"}, {"start": 865.0, "text": "four,"}, {"start": 866.0, "text": "and finally five."}, {"start": 868.0, "text": "At LOD 5 the cauldron looks pretty bad."}, {"start": 871.0, "text": "However, if I scale it down as if the cauldron were very far away,"}, {"start": 874.0, "text": "I can now compare it to LOD 0."}, {"start": 877.0, "text": "The difference is negligible and we require less compute to render the cauldron."}, {"start": 882.0, "text": "Let's see how we can set up LODs in RealityKit."}, {"start": 886.0, "text": "The different LODs are specified as arrays of entities."}, {"start": 890.0, "text": "In this example I'll have 3 different LODs each made up of one entity."}, {"start": 895.0, "text": "Then I will create the entity that will hold the LODs themselves"}, {"start": 899.0, "text": "and switch between them."}, {"start": 901.0, "text": "But how does RealityKit know which LOD to use?"}, {"start": 904.0, "text": "For that I need to pick a switching algorithm."}, {"start": 907.0, "text": "I will walk you through two of RealityKit's switching algorithms."}, {"start": 911.0, "text": "The first one is based on the distance to the camera."}, {"start": 914.0, "text": "The further away from the camera the entity is,"}, {"start": 917.0, "text": "the higher the LOD we can select."}, {"start": 919.0, "text": "The second one is based on the screen area the entity takes up."}, {"start": 923.0, "text": "The less area the entity takes up on screen,"}, {"start": 925.0, "text": "the higher the LOD we can select."}, {"start": 928.0, "text": "Here, I am using the LevelOfDetailComponent's"}, {"start": 931.0, "text": "addByCameraDistance convenience function"}, {"start": 934.0, "text": "to setup camera distance based LODs."}, {"start": 937.0, "text": "For each LOD, I specify a max distance at which I will use this LOD."}, {"start": 942.0, "text": "Once the entity goes beyond this distance,"}, {"start": 944.0, "text": "it'll switch over to the next LOD."}, {"start": 947.0, "text": "For the last LOD, I want to set the max distance to infinity"}, {"start": 951.0, "text": "to indicate that, no matter how far the entity is beyond the previous threshold,"}, {"start": 956.0, "text": "I will be using this LOD."}, {"start": 958.0, "text": "If I want to switch LODs based on screen area"}, {"start": 962.0, "text": "then I use the addByScreenArea convenience function."}, {"start": 965.0, "text": "Here, I specify a minimum area as a fraction of the screen area."}, {"start": 969.0, "text": "If the entity takes up less than this specified screen area,"}, {"start": 973.0, "text": "it'll switch over to the next LOD."}, {"start": 976.0, "text": "Using LODs in your apps and games is a great way to improve performance."}, {"start": 980.0, "text": "However, it is also important to react in your apps and games"}, {"start": 983.0, "text": "when performance is becoming a problem."}, {"start": 986.0, "text": "This can be done by registering an observer on the thermalStateDidChange notification."}, {"start": 991.0, "text": "This will allow me to know if the device's processor is running too hot."}, {"start": 996.0, "text": "If a change in the thermal state occurred, I can query the current state."}, {"start": 1000.0, "text": "If it's nominal or fair, then my mitigation efforts succeeded"}, {"start": 1004.0, "text": "and the app or game can keep running as is."}, {"start": 1007.0, "text": "However, if it's serious or critical,"}, {"start": 1010.0, "text": "then I should take steps to improve performance,"}, {"start": 1012.0, "text": "like making the LOD switching thresholds more aggressive,"}, {"start": 1016.0, "text": "or lowering the quality of shadows."}, {"start": 1018.0, "text": "Keeping a handle on the performance of your apps and games is important"}, {"start": 1022.0, "text": "as it ensures the comfort of your users"}, {"start": 1024.0, "text": "and enables you to take advantage of other advanced RealityKit features."}, {"start": 1029.0, "text": "One such feature is RealityKit's ability to render 3D gaussian splats."}, {"start": 1033.0, "text": "3D gaussian splats are a high performance,"}, {"start": 1036.0, "text": "high quality technique to render volumetric data"}, {"start": 1039.0, "text": "captured from the real world."}, {"start": 1041.0, "text": "With this technique, 3D scenes are represented"}, {"start": 1043.0, "text": "as a collection of 3D gaussians."}, {"start": 1047.0, "text": "You can think of these as ellipsoids with different levels of opacity."}, {"start": 1051.0, "text": "To render such a scene we would need to evaluate,"}, {"start": 1054.0, "text": "at each pixel, a ray along all the gaussians."}, {"start": 1057.0, "text": "There are a number of optimizations that can be made when doing this,"}, {"start": 1061.0, "text": "and if you use RealityKit's API, it will handle them for you."}, {"start": 1064.0, "text": "To see this API in action,"}, {"start": 1066.0, "text": "download the gaussian splat sample from developer.apple.com."}, {"start": 1070.0, "text": "Here I have the sample running on an Apple Vision Pro."}, {"start": 1073.0, "text": "Notice the fine details of the potted succulent."}, {"start": 1076.0, "text": "We were able to capture the geometric complexity of the plants"}, {"start": 1079.0, "text": "and the soil they sit in, and RealityKit was able to render them both flawlessly."}, {"start": 1085.0, "text": "RealityKit API does not assume a specific file format for gaussian splats."}, {"start": 1090.0, "text": "Instead, you have to provide buffers"}, {"start": 1092.0, "text": "that describe the properties of the splats in a capture."}, {"start": 1095.0, "text": "Specifically:"}, {"start": 1096.0, "text": "their position,"}, {"start": 1098.0, "text": "scale,"}, {"start": 1099.0, "text": "rotation,"}, {"start": 1101.0, "text": "opacity,"}, {"start": 1102.0, "text": "and their spherical harmonics."}, {"start": 1105.0, "text": "The spherical harmonics allow you to control the color of an ellipsoid"}, {"start": 1109.0, "text": "depending on the viewing direction."}, {"start": 1111.0, "text": "We also need to specify a degree for the spherical harmonics."}, {"start": 1115.0, "text": "This articulates the number of color variations"}, {"start": 1117.0, "text": "as you move around the ellipsoid."}, {"start": 1119.0, "text": "For instance, a degree of 0 implies a solid color at all viewing directions."}, {"start": 1124.0, "text": "I can then assimilate all of these buffers into a BufferResource"}, {"start": 1127.0, "text": "and create a GaussianSplatResource."}, {"start": 1130.0, "text": "From this resource, I'll create a GaussianSplatComponent."}, {"start": 1133.0, "text": "Then, to render the 3D gaussian splats,"}, {"start": 1136.0, "text": "I will attach the component to an entity in my scene."}, {"start": 1139.0, "text": "With that, you are able to bring 3D gaussian splats into your virtual experiences,"}, {"start": 1144.0, "text": "enabling you to bring high-fidelity captures of real world objects to your users."}, {"start": 1148.0, "text": "Finally, let's go over RealityKit's new immersive audio features,"}, {"start": 1152.0, "text": "enabling enhanced realism"}, {"start": 1153.0, "text": "for your Apple Vision Pro apps and games through sound."}, {"start": 1157.0, "text": "Spatial audio rendering is an important aspect"}, {"start": 1159.0, "text": "of sound design for spatial computing."}, {"start": 1162.0, "text": "Accurate direction and timing of the direct path"}, {"start": 1165.0, "text": "and the reflection path are required for a realistic spatial audio experience."}, {"start": 1170.0, "text": "As a person and the audio source move around the environment,"}, {"start": 1173.0, "text": "the timing and orientation must be updated accordingly to maintain realism."}, {"start": 1178.0, "text": "The geometry and materials of your environment"}, {"start": 1181.0, "text": "have a big impact on how an audio source sounds."}, {"start": 1184.0, "text": "For example, the same audio source will sound very different"}, {"start": 1187.0, "text": "if it's in a small living room or a large museum."}, {"start": 1191.0, "text": "We can use RealityKit to simulate the reflections"}, {"start": 1194.0, "text": "and reverb of our environment using raytraced geometrical acoustics."}, {"start": 1199.0, "text": "For example, in this kitchen and dining room scene,"}, {"start": 1201.0, "text": "we can use RealityKit's custom reverb mesh"}, {"start": 1204.0, "text": "to acoustically model the wood floors, plaster walls, and stone countertop."}, {"start": 1209.0, "text": "Depending on where a person and the audio source are in the scene,"}, {"start": 1213.0, "text": "they will take on the reverb appropriate for their placement."}, {"start": 1217.0, "text": "To show this feature in action we are releasing a sample"}, {"start": 1219.0, "text": "that takes advantage of the custom reverb mesh."}, {"start": 1223.0, "text": "In this sample, you can hear a virtual band play"}, {"start": 1225.0, "text": "as you navigate around a large museum environment."}, {"start": 1228.0, "text": "You can even control the sound coming from each instrument independently."}, {"start": 1233.0, "text": "Thanks to RealityKit's custom reverb mesh,"}, {"start": 1235.0, "text": "the sounds emitted from these instruments realistically scatters around the museum"}, {"start": 1239.0, "text": "before they hit your ears."}, {"start": 1241.0, "text": "This can only truly be experienced on an Apple Vision Pro."}, {"start": 1244.0, "text": "Download the sample from developer.apple.com and give it a try!"}, {"start": 1249.0, "text": "To create a custom reverb mesh, you start by defining the geometry of the scene"}, {"start": 1253.0, "text": "with a ReverbMeshResource."}, {"start": 1255.0, "text": "This can be created from a mesh descriptor or mesh resource."}, {"start": 1259.0, "text": "But the easiest way to get started is with a shoebox,"}, {"start": 1262.0, "text": "which is a box with faces pointed inward."}, {"start": 1265.0, "text": "I'll set it to be 5 meters wide, 4 meters tall and 6 meters deep."}, {"start": 1270.0, "text": "I'll then combine this mesh with"}, {"start": 1272.0, "text": "the dryWall preset audio material to create"}, {"start": 1275.0, "text": "a simulated reverb."}, {"start": 1277.0, "text": "Finally I'll use this reverb to create a reverb component"}, {"start": 1280.0, "text": "and attach it to an entity in the scene."}, {"start": 1282.0, "text": "This will allow the reverb mesh to take effect."}, {"start": 1285.0, "text": "But I'm not limited to just the built-in preset materials."}, {"start": 1289.0, "text": "Let's see how I can create custom materials for my custom reverb mesh."}, {"start": 1293.0, "text": "I'll start by defining a thickCarpet material."}, {"start": 1296.0, "text": "I want this material to be more absorbent than the preset carpet material"}, {"start": 1300.0, "text": "so I call scalingAbsorption"}, {"start": 1303.0, "text": "and increasing the absorption a little bit for all frequencies."}, {"start": 1306.0, "text": "Next, I'll create a bookshelf material, but this time from scratch."}, {"start": 1311.0, "text": "This means we need to define both the absorption coefficients"}, {"start": 1314.0, "text": "and scattering coefficients."}, {"start": 1316.0, "text": "These define how much sound energy is absorbed"}, {"start": 1319.0, "text": "or scattered at different frequencies."}, {"start": 1321.0, "text": "I'll start by setting the absorption coefficients"}, {"start": 1323.0, "text": "for the 10-band center frequencies."}, {"start": 1326.0, "text": "But what if I only know the coefficients for specific frequencies?"}, {"start": 1329.0, "text": "Then, I'll define the scattering coefficients for a few specific frequencies."}, {"start": 1334.0, "text": "RealityKit will then extrapolate"}, {"start": 1336.0, "text": "and cover the entire audible frequency spectrum."}, {"start": 1339.0, "text": "Finally, I create the bookshelf material itself by composing the"}, {"start": 1343.0, "text": "absorption and"}, {"start": 1345.0, "text": "scattering data."}, {"start": 1347.0, "text": "Please note that this only works in immersive spaces."}, {"start": 1350.0, "text": "If you are in a shared space the system's room-sense reverb geometry"}, {"start": 1354.0, "text": "will be used instead."}, {"start": 1355.0, "text": "This is a reverb mesh that the Apple Vision Pro has built"}, {"start": 1359.0, "text": "based on your real-world surroundings."}, {"start": 1362.0, "text": "I've covered a lot of material in this session,"}, {"start": 1364.0, "text": "but this only scratches the surface of what RealityKit has to offer this year."}, {"start": 1368.0, "text": "There are many more features that we are also releasing this year:"}, {"start": 1372.0, "text": "Like coordinated multi-source audio that enables precise, synchronizing audio playback"}, {"start": 1376.0, "text": "across multiple entities;"}, {"start": 1378.0, "text": "high quality character rendering, that provides subsurface scattering"}, {"start": 1381.0, "text": "and advanced hair shaders to bring your characters to life;"}, {"start": 1385.0, "text": "portal customizations, where you can create custom portal materials"}, {"start": 1388.0, "text": "to alter a portal's opacity and shape;"}, {"start": 1390.0, "text": "and much more."}, {"start": 1392.0, "text": "I recommend that you visit the Apple developer portal at developer.apple.com"}, {"start": 1397.0, "text": "to download the samples from this session"}, {"start": 1398.0, "text": "so that you can see some of these exciting new RealityKit features in action."}, {"start": 1402.0, "text": "And while you are there, make sure to check out Reality Composer Pro 3,"}, {"start": 1406.0, "text": "a major new release of Reality Composer Pro"}, {"start": 1408.0, "text": "that enables you to better take advantage of RealityKit's features."}, {"start": 1412.0, "text": "You can check out these sessions to learn more about Reality Composer Pro 3"}, {"start": 1416.0, "text": "and all its new capabilities."}, {"start": 1418.0, "text": "I can't wait to see what amazing spatial experiences you build with RealityKit."}, {"start": 1422.0, "text": "Thank you for watching!"}]}