{ "cells": [ { "cell_type": "markdown", "id": "6a627acd-442f-4cd5-a319-6258cc34edd4", "metadata": {}, "source": [ "# Physics Simulation of 3D Gaussian Splats Using Simplicits - Now With Collisions!\n", "\n", "Let's simulate 3D Gaussian Splat objects using [Simplicits](https://research.nvidia.com/labs/toronto-ai/simplicits/), fully integrated into the Kaolin Library. We will be able to set up and interactively view the simulation directly in this Jupyter notebook.\n", "\n", "With v0.18.0, Kaolin also supports collision handling between objects, which we will also show here.\n", "\n", "\"image" ] }, { "cell_type": "markdown", "id": "f90cd2f2-584e-4d9f-bcca-220d6959b66f", "metadata": {}, "source": [ "## Installation and Requirements\n", "For splat rendering, we will be relying on [Nerfstudio Gsplat](https://github.com/nerfstudio-project/gsplat).\n", "\n", "We have recently tested this notebook with the following environment. Please follow [Kaolin Installation docs](https://kaolin.readthedocs.io/en/latest/notes/installation.html) to install Kaolin. \n", "- python 3.11.10\n", "- cuda 12.4\n", "- pytorch 2.5.1\n", "- setuptools 70.1.1" ] }, { "cell_type": "code", "execution_count": null, "id": "18fa3db8-ceae-41bf-947a-fb9060b17af0", "metadata": {}, "outputs": [], "source": [ "### Install necessary packages\n", "!pip install -q matplotlib gsplat" ] }, { "cell_type": "markdown", "id": "e3aa22b4-5df4-49bc-aae7-f0190341f8b3", "metadata": {}, "source": [ "### Import Kaolin Library and Other Requirements" ] }, { "cell_type": "code", "execution_count": null, "id": "73c496c5-1768-4990-bc1a-b9cdad751066", "metadata": {}, "outputs": [], "source": [ "import copy\n", "import ipywidgets\n", "import json\n", "import kaolin\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "import os\n", "import logging\n", "import sys\n", "import time\n", "import threading \n", "from pathlib import Path\n", "from functools import partial\n", "import warp as wp\n", "\n", "import torch\n", "import gsplat\n", "\n", "from IPython.display import display\n", "from ipywidgets import Button, HBox, VBox\n", "\n", "logging.basicConfig(level=logging.INFO, stream=sys.stdout, format=\"%(asctime)s|%(levelname)8s| %(message)s\")\n", "logger = logging.getLogger(__name__)\n", "\n", "%load_ext autoreload\n", "%autoreload 2\n", "\n", "sys.path.append(str(Path(\"..\")))\n", "from tutorial_common import COMMON_DATA_DIR, FILE_DIR\n", "\n", "def log_tensor(t, name, **kwargs):\n", " \"\"\" Debugging util, e.g. call: log_tensor(t, 'my tensor', print_stats=True) \"\"\"\n", " logger.info(kaolin.utils.testing.tensor_info(t, name=name, **kwargs))\n", "\n", "try:\n", " import k3d\n", "except ModuleNotFoundError:\n", " !pip install -q k3d\n", " import k3d\n", " raise RuntimeError(\"k3d has been installed. Do a hard refresh of the page (Ctrl + Shift + R) to avoid issue with k3d\")" ] }, { "cell_type": "markdown", "id": "f652d938-bcfb-498d-9407-8563f1645227", "metadata": {}, "source": [ "## Download Splat Models from AWS\n", "Lets grab two pre-trained 3D Gaussian Splat models from AWS.\n", "We can unzip and set the splat model path below to the correct `.ply` file." ] }, { "cell_type": "code", "execution_count": null, "id": "b5154170-a133-4df5-816a-1fdeb87788ac", "metadata": {}, "outputs": [], "source": [ "# Download and unzip the nerfsynthetic bulldozer\n", "!if test -d output/dozer; then echo \"Pretrained bulldozer splats already exist.\"; else wget https://nvidia-kaolin.s3.us-east-2.amazonaws.com/data/dozer.zip -P output/; unzip output/dozer.zip -d output/; fi;\n", "model_path = 'output/dozer/point_cloud/iteration_30000/point_cloud.ply'\n", "\n", "# Download and unzip the doll splat, captured and trained by the Kaolin team (please cite Kaolin if you use this model)\n", "!if [ -e output/BluehairRagdoll.usdc ]; then echo \"Pretrained doll splats already exist.\"; else wget https://nvidia-kaolin.s3.us-east-2.amazonaws.com/data/BluehairRagdoll.usdc -P output/; fi;\n", "model_path2 = 'output/BluehairRagdoll.usdc' " ] }, { "cell_type": "markdown", "id": "74dba058-4a2d-47f7-ada4-671a26a3cad5", "metadata": {}, "source": [ "### Load 3D Gaussian Splat Models\n", "\n", "After the setup, we can load and use Kaolin to display the splat model within the Jupyter notebook." ] }, { "cell_type": "code", "execution_count": null, "id": "ad5a0418-ba7f-4b84-9833-72093f120cf5", "metadata": { "scrolled": true }, "outputs": [], "source": [ "gaussians = kaolin.io.import_gaussiancloud(model_path).cuda()\n", "gaussians2 = kaolin.io.import_gaussiancloud(model_path2).cuda()" ] }, { "cell_type": "markdown", "id": "14852d04-c7b5-4d64-ae16-91f0c6e642bc", "metadata": {}, "source": [ "## Interactive Rendering Using Kaolin Visualizer\n", "\n", "In order to easily view splats in the notebook, let's set up Gaussian Splat rendering using Kaolin camera conventions.\n", "You should be able to see the rendering below this cell and to control the camera with your left mouse button." ] }, { "cell_type": "code", "execution_count": null, "id": "1005f2fc-f59d-430c-a4c5-70d7894edca8", "metadata": {}, "outputs": [], "source": [ "resolution = 512\n", "default_cam = kaolin.render.camera.Camera.from_args(\n", " eye=torch.ones((3,)) * 2, at=torch.zeros((3,)), up=torch.tensor([0., 0., 1.]),\n", " fov=torch.pi * 45 / 180, height=resolution, width=resolution)\n", "\n", "class GaussianRenderer:\n", " \"\"\" Define a rendering closure. \"\"\"\n", " def __init__(self, gaussians, downscale_factor=1):\n", " self.gaussians = gaussians\n", " self.downscale_factor = downscale_factor\n", "\n", " def downscale_camera(self, in_cam):\n", " lowres_cam = copy.deepcopy(in_cam)\n", " lowres_cam.width = in_cam.width // self.downscale_factor\n", " lowres_cam.height = in_cam.height // self.downscale_factor\n", " return lowres_cam\n", "\n", " def fast_render(self, camera):\n", " if self.downscale_factor > 1:\n", " camera = self.downscale_camera(camera)\n", " return self(camera)\n", "\n", " def __call__(self, camera):\n", " # Convert kaolin camera to nerfstudio gsplat camera\n", " cam_params = kaolin.render.camera.kaolin_camera_to_gsplat_nerfstudio(camera.cuda())\n", " # Render gaussians using the gsplat rendering function\n", " render_colors, render_alphas, info = gsplat.rendering.rasterization(\n", " self.gaussians.positions,\n", " self.gaussians.orientations,\n", " self.gaussians.scales,\n", " self.gaussians.opacities,\n", " self.gaussians.sh_coeff,\n", " sh_degree=self.gaussians.sh_degree,\n", " **cam_params\n", " )\n", " return (torch.clamp(render_colors[0], 0, 1) * 255).to(torch.uint8).detach().cpu()\n", " \n", "static_scene_viz = kaolin.visualize.IpyTurntableVisualizer(\n", " resolution, resolution, copy.deepcopy(default_cam), GaussianRenderer(gaussians), \n", " focus_at=None, world_up_axis=2, max_fps=12, img_quality=75, img_format='JPEG')\n", "static_scene_viz.show() " ] }, { "cell_type": "markdown", "id": "c0703b2d-4944-4f7c-8197-be14011a74c7", "metadata": {}, "source": [ "## Creating and Training Simplicits Objects from Points\n", "[Simplicits](https://research.nvidia.com/labs/toronto-ai/simplicits/) is a mesh-free, representation-agnostic method for simulating elastic deformations. We can use it to simulate Gaussian Splats at interactive rates within the Jupyter notebook. In order to simulate any point-sampled geometry, such as splats, Simplicits first\n", "_trains_ an object specific weight function representing the reduced degrees of freedom for the object. The physics solver then uses this reduced space to solve for deformations during simulation.\n", "\n", "Next, let's use the Simplicits API within Kaolin to create, train and simulate splat objects.\n", "\n", "First, let's set some material parameters." ] }, { "cell_type": "code", "execution_count": null, "id": "cfe06442-79cd-45c1-92c0-78395a847bbd", "metadata": {}, "outputs": [], "source": [ "# Physics material parameters (use approximated values, or look them up online)\n", "# We'll create a few presets that can be used\n", "youngs_modulus_presets = {\"softest\": 2000, \"soft\": 21000, \"medium\": 1e6, \"stiff\": 1e7}\n", "soft_youngs_modulus = youngs_modulus_presets[\"soft\"] # we will use this for training\n", "poisson_ratio = 0.45\n", "rhos = 100 # kg/m^3\n", "appx_vol = 3 # m^3" ] }, { "cell_type": "markdown", "id": "02e574e8-8d2f-4cf3-a19d-9f3cda7fcc9d", "metadata": {}, "source": [ "### Sampling Within Splat Volume\n", "\n", "Because splats tend to occupy the surface of the object, they provide poor sampling of the object's interior. This can affect the quality of the learned reduced space. To sample within the splat volume, we will use Kaolin's utility `kaolin.ops.gaussians.sample_points_in_volume`. " ] }, { "cell_type": "code", "execution_count": null, "id": "1d2bd1c6-727e-4183-bf80-618d1283796d", "metadata": {}, "outputs": [], "source": [ "densified_pos = kaolin.ops.gaussians.sample_points_in_volume(\n", " xyz=gaussians.positions,\n", " scale=gaussians.scales,\n", " rotation=gaussians.orientations,\n", " opacity=gaussians.opacities,\n", " clip_samples_to_input_bbox=False\n", ")\n", "log_tensor(gaussians.positions, 'original_pos', print_stats=True)\n", "log_tensor(densified_pos, 'densified_pos', print_stats=True)" ] }, { "cell_type": "code", "execution_count": null, "id": "84988936-25a0-4033-9e5f-fa0206c7e590", "metadata": {}, "outputs": [], "source": [ "def visualize_pts_k3d(densified_pos, pos):\n", " plot = k3d.plot()\n", " plot += k3d.points(densified_pos.detach().cpu().numpy(), point_size=0.01, color=0x00ff00)\n", " plot += k3d.points(pos.detach().cpu().numpy(), point_size=0.02, color=0xff0000)\n", " plot.display()\n", "\n", "visualize_pts_k3d(densified_pos, gaussians.positions)" ] }, { "cell_type": "markdown", "id": "e139e512-3072-4b0b-8981-a2b48378108c", "metadata": {}, "source": [ "# Storing physical materials info\n", "\n", "All the physical materials informations can be stored in `kaolin.physics.simplicits.PhysicsPoints`.\n", "We can save those in an USD file, and it's used to instantiate a `SimplicitsObject`" ] }, { "cell_type": "code", "execution_count": null, "id": "50e91d67-906e-4c57-ad0e-c6d2a47b4f2f", "metadata": {}, "outputs": [], "source": [ "physics_points = kaolin.physics.simplicits.PhysicsPoints(\n", " pts=densified_pos,\n", " yms=soft_youngs_modulus,\n", " prs=poisson_ratio,\n", " rhos=rhos,\n", " appx_vol=appx_vol\n", ")" ] }, { "cell_type": "markdown", "id": "a807d1eb-1be5-4464-82c8-0146a804fd36", "metadata": {}, "source": [ "### Training\n", "Next we create a `SimplicitsObject` and train its skinning weight functions using the volume samples, visualized above. The simulator will then use these reduced degrees of freedom to drive the simulation. **Note** there are different methods to create skinning weight functions. The original Simplicits method used to train an MLP, which could take a couple of minutes. Now we use FreeForm (CVPR 2026), which optimizes RKPM kernels to represent the eigenbasis of the object.\n", "\n", "We show how to save the object as a [baked representation](https://kaolin.readthedocs.io/en/latest/modules/kaolin.physics.simplicits.html#kaolin.physics.simplicits.SkinnedPhysicsPoints) into an USD file. While the baked object doesn't contain the skinning weight function, it's ready to be simulated in a scene.\n", "\n", "Use `ENABLE_SIMPLICITS_CACHING` to reuse the object, set it to `False` to force training." ] }, { "cell_type": "code", "execution_count": null, "id": "34b50fe3-13d3-42a5-9326-4b5e8d7f453a", "metadata": {}, "outputs": [], "source": [ "# # Whether to save reduced degrees of freedom used by the simulator and load from cache automatically\n", "ENABLE_SIMPLICITS_CACHING = True\n", "cache_dir = os.path.join(FILE_DIR, 'cache', 'gaussians_physics')\n", "os.makedirs(cache_dir, exist_ok=True)\n", "logger.info(f'Caching trained skinned physics points objects in {cache_dir}')" ] }, { "cell_type": "code", "execution_count": null, "id": "8b85eec7-f527-4b17-a873-2086996a899c", "metadata": {}, "outputs": [], "source": [ "def export_gaussians_and_physics(file_path,\n", " prim_paths: list[str],\n", " gaussianclouds: list[kaolin.rep.GaussianSplatModel],\n", " physics_points: list[kaolin.physics.simplicits.PhysicsPoints],\n", " skinned_physics_points: list[kaolin.physics.simplicits.SkinnedPhysicsPoints],\n", " init_transforms: list[torch.Tensor] = None,\n", " up_axis='Y'):\n", " \"\"\"Save Gaussians objects with their physics material and skinned physics points properties into an USD file\"\"\"\n", " assert len(prim_paths) == len(gaussianclouds) == len(physics_points) == len(skinned_physics_points)\n", " stage = kaolin.io.usd.create_stage(file_path, up_axis)\n", " try:\n", " if init_transforms is None:\n", " init_transforms = [None] * len(prim_paths)\n", " for i, gaussiancloud in enumerate(gaussianclouds):\n", " params = gaussiancloud.as_dict()\n", " del params['sh_degree']\n", " prim = kaolin.io.usd.add_gaussiancloud(stage, prim_paths[i], **params, local_to_world=init_transforms[i])\n", " kaolin.io.usd.add_physics_material(stage, prim, physics_points=physics_points[i])\n", " kaolin.io.usd.add_skinned_physics(stage, prim, skinned_physics_points=skinned_physics_points[i])\n", "\n", " stage.Save()\n", " finally:\n", " del stage\n", "\n", "def import_gaussians_and_physics(file_path):\n", " \"\"\"Load Gaussians, physics materials and skinned physics points properties from an USD file\"\"\"\n", " output = {}\n", " gaussianclouds = kaolin.io.usd.import_gaussianclouds(file_path, return_list=False)\n", " for path, gaussiancloud in gaussianclouds.items():\n", " # TODO(cfujitsang): support for subsets\n", " output[path] = {\n", " 'gaussiancloud': gaussiancloud,\n", " 'physics_points': kaolin.io.usd.get_physics_material(file_path, path, material_name='default'),\n", " 'skinned_physics_points': kaolin.io.usd.get_skinned_physics(file_path, path, instance_name='default')\n", " }\n", " return output\n", "\n", "def train_or_load_baked_simplicits_object(file_path,\n", " gaussianclouds: kaolin.rep.GaussianSplatModel = None,\n", " physics_points: kaolin.physics.simplicits.PhysicsPoints = None):\n", " if ENABLE_SIMPLICITS_CACHING and os.path.exists(file_path):\n", " logger.info(f\"loading cached object at '{file_path}'\")\n", " return import_gaussians_and_physics(file_path)['/World/Gaussians/obj']\n", " else:\n", " assert gaussianclouds is not None and physics_points is not None, \\\n", " f\"{file_path} doesn't exist, must provide 'gaussianclouds' and 'physics_points'\"\n", " sim_rkpm_obj = kaolin.physics.simplicits.SimplicitsObject.create_with_rkpm(physics_points=physics_points, num_handles=40, num_nodes=1024, num_points=10000)\n", " baked_obj = sim_rkpm_obj.bake(num_qps=1024, renderable_pts=gaussians.positions)\n", " export_gaussians_and_physics(file_path, ['/World/Gaussians/obj'], [gaussianclouds], [physics_points], [baked_obj])\n", " return {\n", " 'gaussiancloud': gaussianclouds,\n", " 'physics_points': physics_points,\n", " 'skinned_physics_points': baked_obj\n", " }\n", "\n", "data = train_or_load_baked_simplicits_object(os.path.join(cache_dir, 'simplicits_truck.usdc'), gaussians, physics_points)\n", "gaussians = data['gaussiancloud'].cuda()\n", "physics_points = data['physics_points'].cuda()\n", "baked_obj = data['skinned_physics_points'].cuda()" ] }, { "cell_type": "markdown", "id": "d5ca786d-d925-4e7c-b028-cbf1090e788f", "metadata": {}, "source": [ "## Setup Simulated Scene Using Simplicits Easy API\n", "Lets create an empty scene with default parameters, then reset the max number of newton steps for faster runtimes.\n", "\n", "**Note:** be patient, some of the steps below take time, as we need to build matrices used during simulation." ] }, { "cell_type": "code", "execution_count": null, "id": "570eba83-c02f-470d-b56a-c696798738f9", "metadata": {}, "outputs": [], "source": [ "scene = kaolin.physics.simplicits.SimplicitsScene() # Create a default scene # default empty scene\n", "\n", "scene.max_newton_steps = 3 #Convergence might not be guaranteed at few NM iterations, but runs very fast\n", "scene.timestep = 0.03\n", "scene.newton_hessian_regularizer = 1e-5" ] }, { "cell_type": "markdown", "id": "1d6f45f7-f7ff-4b9d-b27a-0a23a4d13f70", "metadata": {}, "source": [ "Now we add our object to the scene. We use 2048 cubature points to integrate over." ] }, { "cell_type": "code", "execution_count": null, "id": "c81b9030-e2de-4686-800c-0e16ab9240c4", "metadata": {}, "outputs": [], "source": [ "# The scene copies it into an internal SimulatableObject utility class\n", "obj_idx = scene.add_object(baked_obj)" ] }, { "cell_type": "markdown", "id": "199c2abc-96f5-45de-90af-220cedeb9e53", "metadata": {}, "source": [ "Lets set set gravity and floor forces on the scene" ] }, { "cell_type": "code", "execution_count": null, "id": "07868367-8b1a-4d38-8bfb-8227a1bda5e0", "metadata": { "scrolled": true }, "outputs": [], "source": [ "# Add gravity to the scene\n", "scene.set_scene_gravity(acc_gravity=torch.tensor([0, 0, 9.8]))\n", "# Add floor to the scene\n", "scene.set_scene_floor(floor_height=-0.7, floor_axis=2, floor_penalty=1000, flip_floor=False)" ] }, { "cell_type": "markdown", "id": "2d560e57-f5fc-4afc-9999-b259aa20f2f0", "metadata": {}, "source": [ "We can play around with the material parameters of the object, indicated via object_idx" ] }, { "cell_type": "markdown", "id": "d4368abe-41ac-44f1-9433-6751df4495e8", "metadata": {}, "source": [ "## Simulating and Interactive Visualizing \n", "\n", "That's it! We are ready to simulate. Let's just make sure we can visualize the simulation as it is running." ] }, { "cell_type": "markdown", "id": "16e55acb-e698-4e09-939c-2cb37da343e8", "metadata": {}, "source": [ "### Handling Splat Deformation\n", "\n", "As the splats deform, we must update their attributes using the transforms predicted by the simulator.\n", "For this, we will need the reduced degrees of freedom and ability to apply linear blend skinning to splats. These utilities can be found in `gaussian_utils.py` relative to this notebook." ] }, { "cell_type": "code", "execution_count": null, "id": "ddae56ff-f1b2-45ed-ab54-a2a6c5955847", "metadata": {}, "outputs": [], "source": [ "class SingleGaussianObjectDynamicRenderer(GaussianRenderer):\n", " def __init__(self, gaussians, scene, downscale_factor=1):\n", " self.rest_gaussians = copy.deepcopy(gaussians)\n", " self.gaussians = copy.deepcopy(gaussians)\n", " self.downscale_factor = downscale_factor\n", " self.scene = scene\n", " self.reset()\n", "\n", " def run_sim_step(self):\n", " self.scene.run_sim_step()\n", " self.update_gaussians()\n", "\n", " def update_gaussians(self):\n", " with torch.no_grad():\n", " per_pt_transforms = self.scene.get_object_point_transforms(0, 'rendered')\n", " self.gaussians = self.rest_gaussians.as_transformed(per_pt_transforms)\n", "\n", " def reset(self):\n", " self.scene.reset_scene()\n", " self.update_gaussians()\n", "\n", "renderer = SingleGaussianObjectDynamicRenderer(gaussians, scene, 8)" ] }, { "cell_type": "markdown", "id": "32699363-7667-42a7-b71a-0e8cd639cc21", "metadata": {}, "source": [ "### Threading\n", "\n", "We will run simulation in a separate thread, so it is possible to interact with the viewer as the simulation is running (in fact, it's encouraged). We'll reuse these utils for this and the multi-object simulation below." ] }, { "cell_type": "code", "execution_count": null, "id": "6efa4f1c-0a5a-4364-beed-d22efeab974e", "metadata": {}, "outputs": [], "source": [ "global sim_thread\n", "sim_thread = None\n", "\n", "def start_simulation(visualizer, num_iterations): \n", " global sim_thread\n", "\n", " def sim_function():\n", " for s in range(num_sim_iterations):\n", " with visualizer.out:\n", " visualizer.render.run_sim_step()\n", " print(\".\", end=\"\")\n", " visualizer.render_update()\n", "\n", " with visualizer.out:\n", " if sim_thread is not None:\n", " sim_thread.join()\n", " sim_thread = threading.Thread(target=sim_function, daemon=True)\n", " sim_thread.start()\n", "\n", "def reset_simulation(reset_function, visualizer):\n", " with visualizer.out:\n", " reset_function()\n", " visualizer.render_update()" ] }, { "cell_type": "markdown", "id": "d1b0f181-4917-4735-b2eb-196880e3a602", "metadata": {}, "source": [ "### Simulation: Let's bring everything together!" ] }, { "cell_type": "code", "execution_count": null, "id": "08e353fd-1d27-4ddb-ad11-4c77fb0f8c11", "metadata": {}, "outputs": [], "source": [ "num_sim_iterations = 100\n", "\n", "resolution = 512\n", "sim_visualizer = kaolin.visualize.IpyTurntableVisualizer(\n", " resolution, resolution, copy.deepcopy(default_cam),\n", " renderer, fast_render=renderer.fast_render,\n", " focus_at=torch.tensor([0, 0, 0.0]),\n", " world_up_axis=2, max_fps=12, img_quality=75, img_format='JPEG')\n", "\n", "buttons = [Button(description=x) for x in\n", " ['Run Sim', 'Reset']]\n", "buttons[0].on_click(lambda e: start_simulation(sim_visualizer, num_sim_iterations))\n", "buttons[1].on_click(lambda e: reset_simulation(renderer.reset, sim_visualizer))\n", "\n", "sim_visualizer.render_update()\n", "display(VBox([HBox([sim_visualizer.canvas, VBox(buttons)]), sim_visualizer.out]))" ] }, { "cell_type": "markdown", "id": "37a360af-788e-459e-837e-9f6ca193b511", "metadata": {}, "source": [ "# Part 2: Multiple Objects and Collisions\n", "\n", "It's time to make this simulation more exciting. Let's train and add the second object that we loaded above to the simulation." ] }, { "cell_type": "markdown", "id": "2ecf0011-d27b-4431-a1fb-40924681eabe", "metadata": {}, "source": [ "### Train Second Simplicits Object\n", "\n", "As before, we will sample and visualizer points in the object volume. Then, we'll train and cache a simplicits object." ] }, { "cell_type": "code", "execution_count": null, "id": "ca85da53-4ecf-44ad-bbdb-ec3b3b8fe323", "metadata": {}, "outputs": [], "source": [ "densified_pos2 = kaolin.ops.gaussians.sample_points_in_volume(\n", " xyz=gaussians2.positions.detach(), \n", " scale=gaussians2.scales.detach(),\n", " rotation=gaussians2.orientations.detach(),\n", " opacity=gaussians2.opacities.detach(),\n", " clip_samples_to_input_bbox=False\n", ")\n", "log_tensor(gaussians2.positions, 'original_pos', print_stats=True)\n", "log_tensor(densified_pos2, 'densified_pos', print_stats=True)" ] }, { "cell_type": "code", "execution_count": null, "id": "bae973ca-64fa-4ebf-850f-badd3fa3c380", "metadata": {}, "outputs": [], "source": [ "visualize_pts_k3d(densified_pos2, gaussians2.positions)" ] }, { "cell_type": "code", "execution_count": null, "id": "8a86b6c7-fc8d-451d-972a-904eecf38e70", "metadata": {}, "outputs": [], "source": [ "physics_points2 = kaolin.physics.simplicits.PhysicsPoints(\n", " pts=densified_pos2,\n", " yms=soft_youngs_modulus,\n", " prs=poisson_ratio,\n", " rhos=rhos,\n", " appx_vol=appx_vol\n", ")" ] }, { "cell_type": "markdown", "id": "b0e1d55c-b428-494e-9ae8-44c4882aea24", "metadata": {}, "source": [ "### Create the Second Object Quickly Using RKPM Bases " ] }, { "cell_type": "code", "execution_count": null, "id": "e2579b5b-c7ba-4f25-bf0d-8363fcfad9e1", "metadata": {}, "outputs": [], "source": [ "sim_rkpm_obj2 = kaolin.physics.simplicits.SimplicitsObject.create_with_rkpm(physics_points=physics_points2, num_handles=40, num_nodes=1024, num_points=10000)\n", "baked_obj2 = sim_rkpm_obj2.bake(num_qps=1024, renderable_pts=gaussians2.positions)" ] }, { "cell_type": "markdown", "id": "e55f2932-afd7-49a6-91fc-f23772bf32a6", "metadata": {}, "source": [ "### Set Up New Scene\n", "\n", "We'll set up a new scene to make sure the previous simulation cell is still functional." ] }, { "cell_type": "code", "execution_count": null, "id": "d65beddd-91d8-4be1-bec4-14a0ec529c2d", "metadata": {}, "outputs": [], "source": [ "init_transform = None\n", "init_transform2 = torch.tensor([[1,0,0,0],\n", " [0,1,0,0],\n", " [0,0,1,1], \n", " [0,0,0,1]], dtype=torch.float32, \n", " device=gaussians2.positions.device)" ] }, { "cell_type": "code", "execution_count": null, "id": "76a60794-c0ef-40e8-bd0e-814d193fbf4e", "metadata": {}, "outputs": [], "source": [ "scene2 = kaolin.physics.simplicits.SimplicitsScene() # Create a default scene # default empty scene\n", "\n", "scene2.max_newton_steps = 5 #Convergence might not be guaranteed at few NM iterations, but runs very fast\n", "scene2.timestep = 0.03\n", "scene2.newton_hessian_regularizer = 1e-4" ] }, { "cell_type": "markdown", "id": "b307cd08-334d-4dd7-a61f-2344430aa4c7", "metadata": {}, "source": [ "We'll add 2 objects this time, offsetting the doll in the z direction." ] }, { "cell_type": "code", "execution_count": null, "id": "e9675c1d-42a5-4267-a439-6aaf14769da9", "metadata": {}, "outputs": [], "source": [ "scene2_obj_idx = scene2.add_object(baked_obj, init_transform=init_transform)\n", "scene2_obj_idx2 = scene2.add_object(baked_obj2, init_transform=init_transform2) " ] }, { "cell_type": "markdown", "id": "f5d97a00-319a-4b6d-8edf-8ca45dee6e63", "metadata": {}, "source": [ "We'll set up forces as before." ] }, { "cell_type": "code", "execution_count": null, "id": "ddcb1608-ffc5-4f4e-b073-8729fe5f1ad6", "metadata": {}, "outputs": [], "source": [ "# Add gravity to the scene\n", "scene2.set_scene_gravity(acc_gravity=torch.tensor([0, 0, 9.8]))\n", "# Add floor to the scene\n", "scene2.set_scene_floor(floor_height=-0.7, floor_axis=2, floor_penalty=1000, flip_floor=False)" ] }, { "cell_type": "markdown", "id": "691e27be-a266-408d-8e04-5e15dadbcfa4", "metadata": {}, "source": [ "### Enable Collisions (new!)\n", "\n", "We will enable inter-object collisions here. " ] }, { "cell_type": "code", "execution_count": null, "id": "2a47be9b-8477-487f-9065-0ab18743fe9c", "metadata": {}, "outputs": [], "source": [ "scene2.enable_collisions(\n", " collision_particle_radius=0.1, # radius of each collision particle - energy starts accumulating at r\n", " detection_ratio=1.5, # radius * detection ratio is the area that is searched for potential contact\n", " impenetrable_barrier_ratio=0.25, # radius * barrier is the distance at which energy is infinite\n", " collision_penalty=1000.0, # coefficient of collision energy, force, gradient\n", " max_contact_pairs=10000, # the maximum number of particle contact pairs to allow\n", " friction=0.5, # friction coefficient\n", ")" ] }, { "cell_type": "markdown", "id": "d34b0a72-f19b-4caf-82e9-f6343787d06d", "metadata": {}, "source": [ "### Handle Deforming and Rendering Multiple Gaussians\n", "\n", "Because the renderer is not set up to render multi-object scenes, we need to do a little work in order to visualize the simulation. Let's concatenate both objects into a single GaussianSplatModel. The `SimplicitsScene` will provide per-object transform that we can concatenate to transform the whole scene. (We could also transform each object independently and concatenate them." ] }, { "cell_type": "code", "execution_count": null, "id": "0c7107bf-1da1-4d51-b446-07400015c591", "metadata": {}, "outputs": [], "source": [ "combined_gaussians = kaolin.rep.GaussianSplatModel.cat([gaussians, gaussians2])\n", "\n", "class MultiGaussianObjectDynamicRenderer(SingleGaussianObjectDynamicRenderer):\n", " def update_gaussians(self):\n", " with torch.no_grad():\n", " per_pt_transforms = []\n", " for k in sorted(self.scene.sim_obj_dict.keys()):\n", " per_pt_transforms.append(self.scene.get_object_point_transforms(k, 'rendered'))\n", " per_pt_transforms = torch.cat(per_pt_transforms, dim=0)\n", " self.gaussians = self.rest_gaussians.as_transformed(per_pt_transforms)\n", "\n", "multi_renderer = MultiGaussianObjectDynamicRenderer(combined_gaussians, scene2, 8)" ] }, { "cell_type": "markdown", "id": "a7f0409f-31a4-407b-a0cb-228353e11536", "metadata": {}, "source": [ "Let's make sure we can deform both objects using the learned degrees of freedom, which the Simplicits simulator is using to predict deformations." ] }, { "cell_type": "markdown", "id": "19c5dc40-04ae-4e04-95eb-36bdf7dbae14", "metadata": {}, "source": [ "### Simulate and Visualize\n", "\n", "Now we are ready to run the simulation and visualize it." ] }, { "cell_type": "code", "execution_count": null, "id": "b0b5ddfa-0ffe-4319-9ab0-690046f15424", "metadata": {}, "outputs": [], "source": [ "num_sim_iterations = 100\n", "\n", "resolution = 512\n", "multi_sim_visualizer = kaolin.visualize.IpyTurntableVisualizer(\n", " resolution, resolution, copy.deepcopy(default_cam),\n", " multi_renderer, fast_render=multi_renderer.fast_render,\n", " focus_at=torch.tensor([0, 0, 0.0]),\n", " world_up_axis=2, max_fps=12, img_quality=75, img_format='JPEG')\n", "\n", "buttons = [Button(description=x) for x in\n", " ['Run Sim', 'Reset']]\n", "buttons[0].on_click(lambda e: start_simulation(multi_sim_visualizer, num_sim_iterations))\n", "buttons[1].on_click(lambda e: reset_simulation(multi_renderer.reset, sim_visualizer))\n", "\n", "multi_sim_visualizer.render_update()\n", "display(VBox([HBox([multi_sim_visualizer.canvas, VBox(buttons)]), multi_sim_visualizer.out]))" ] }, { "cell_type": "markdown", "id": "562b9547-5078-45a1-add4-3a3545ca1bcf", "metadata": {}, "source": [ "# Saving / Loading the whole scene\n", "You can save and load a whole simulatable scene in USD (after loading, restart from `scene2 = kaolin.physics.simplicits.SimplicitsScene() ...`)" ] }, { "cell_type": "code", "execution_count": null, "id": "bd67ec6c-5b35-440e-aea7-c7acb1785acd", "metadata": {}, "outputs": [], "source": [ "multi_renderer.reset()\n", "export_gaussians_and_physics(os.path.join(cache_dir, 'whole_scene.usdc'),\n", " prim_paths=['/World/Gaussians/obj', '/World/Gaussians/obj2'],\n", " gaussianclouds=[gaussians, gaussians2],\n", " init_transforms=[init_transform, init_transform2],\n", " physics_points=[physics_points, physics_points2],\n", " skinned_physics_points=[baked_obj, baked_obj2])" ] }, { "cell_type": "code", "execution_count": null, "id": "61ff027c-a87e-45c0-b4f6-9d7ee044d3ee", "metadata": {}, "outputs": [], "source": [ "scene_gaussians_and_physics = import_gaussians_and_physics(os.path.join(cache_dir, \"whole_scene.usdc\"))\n", "gaussians = scene_gaussians_and_physics['/World/Gaussians/obj']['gaussiancloud'].cuda()\n", "gaussians2 = scene_gaussians_and_physics['/World/Gaussians/obj2']['gaussiancloud'].cuda()\n", "physics_points = scene_gaussians_and_physics['/World/Gaussians/obj']['physics_points'].cuda()\n", "physics_points2 = scene_gaussians_and_physics['/World/Gaussians/obj2']['physics_points'].cuda()\n", "baked_obj = scene_gaussians_and_physics['/World/Gaussians/obj']['skinned_physics_points'].cuda()\n", "baked_obj2 = scene_gaussians_and_physics['/World/Gaussians/obj2']['skinned_physics_points'].cuda()\n", "init_transform = gaussians.transform\n", "init_transform2 = gaussians2.transform\n", "gaussians.transform = None\n", "gaussians2.transform = None" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.12.3" } }, "nbformat": 4, "nbformat_minor": 5 }