{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# ๐จ Data Designer 101: Seeding Synthetic Data Generation with an External Dataset\n",
"\n",
"[Click here](https://raw.githubusercontent.com/NVIDIA-NeMo/DataDesigner/refs/heads/main/docs/notebooks/3-seeding-with-a-dataset.ipynb) to download this notebook to your computer.\n",
"#### ๐ What you'll learn\n",
"\n",
"In this notebook, we will demonstrate how to seed synthetic data generation in Data Designer with an external dataset.\n",
"\n",
"If this is your first time using Data Designer, we recommend starting with the [first notebook](/notebooks/1-the-basics/) in this 101 series."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ๐ฆ Import the essentials\n",
"\n",
"- The `essentials` module provides quick access to the most commonly used objects.\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"from data_designer.essentials import (\n",
" DataDesigner,\n",
" DataDesignerConfigBuilder,\n",
" InferenceParameters,\n",
" ModelConfig,\n",
" SeedConfig,\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### โ๏ธ Initialize the Data Designer interface\n",
"\n",
"- `DataDesigner` is the main object that is used to interface with the library.\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[16:33:56] [INFO] โป๏ธ Using default model providers from '/Users/amanoel/.data-designer/model_providers.yaml'\n"
]
}
],
"source": [
"data_designer_client = DataDesigner()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ๐๏ธ Define model configurations\n",
"\n",
"- Each `ModelConfig` defines a model that can be used during the generation process.\n",
"\n",
"- The \"model alias\" is used to reference the model in the Data Designer config (as we will see below).\n",
"\n",
"- The \"model provider\" is the external service that hosts the model (see the [model config](/models/default-model-settings.md) docs for more details).\n",
"\n",
"- By default, we use [build.nvidia.com](https://build.nvidia.com/models) as the model provider.\n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# This name is set in the model provider configuration.\n",
"MODEL_PROVIDER = \"nvidia\"\n",
"\n",
"# The model ID is from build.nvidia.com.\n",
"MODEL_ID = \"nvidia/nvidia-nemotron-nano-9b-v2\"\n",
"\n",
"# We choose this alias to be descriptive for our use case.\n",
"MODEL_ALIAS = \"nemotron-nano-v2\"\n",
"\n",
"# This sets reasoning to False for the nemotron-nano-v2 model.\n",
"SYSTEM_PROMPT = \"/no_think\"\n",
"\n",
"model_configs = [\n",
" ModelConfig(\n",
" alias=MODEL_ALIAS,\n",
" model=MODEL_ID,\n",
" provider=MODEL_PROVIDER,\n",
" inference_parameters=InferenceParameters(\n",
" temperature=0.5,\n",
" top_p=1.0,\n",
" max_tokens=1024,\n",
" ),\n",
" )\n",
"]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ๐๏ธ Initialize the Data Designer Config Builder\n",
"\n",
"- The Data Designer config defines the dataset schema and generation process.\n",
"\n",
"- The config builder provides an intuitive interface for building this configuration.\n",
"\n",
"- The list of model configs is provided to the builder at initialization.\n"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"config_builder = DataDesignerConfigBuilder(model_configs=model_configs)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## ๐ฅ Prepare a seed dataset\n",
"\n",
"- For this notebook, we'll create a synthetic dataset of patient notes.\n",
"\n",
"- We will _seed_ the generation process with a [symptom-to-diagnosis dataset](https://huggingface.co/datasets/gretelai/symptom_to_diagnosis).\n",
"\n",
"- We already have the dataset downloaded in the [data](../data) directory of this repository.\n",
"\n",
"
\n",
"\n",
"> ๐ฑ **Why use a seed dataset?**\n",
">\n",
"> - Seed datasets let you steer the generation process by providing context that is specific to your use case.\n",
">\n",
"> - Seed datasets are also an excellent way to inject real-world diversity into your synthetic data.\n",
">\n",
"> - During generation, prompt templates can reference any of the seed dataset fields.\n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"\n",
"DataDesignerConfigBuilder(\n",
" seed_dataset: 'gretelai_symptom_to_diagnosis.csv'\n",
" seed_dataset_columns: ['diagnosis', 'patient_summary']\n",
")\n",
"DataDesignerConfigBuilder(\n",
" seed_dataset: 'gretelai_symptom_to_diagnosis.csv'\n",
" seed_dataset_columns: ['diagnosis', 'patient_summary']\n",
" sampler_columns: [\n",
" "patient_sampler",\n",
" "doctor_sampler",\n",
" "patient_id",\n",
" "symptom_onset_date",\n",
" "date_of_visit"\n",
" ]\n",
" llm_text_columns: ['physician_notes']\n",
" expression_columns: [\n",
" "first_name",\n",
" "last_name",\n",
" "dob",\n",
" "physician"\n",
" ]\n",
")\n",
" \n",
" Seed Columns \n",
"โโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ Name โ Value โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ diagnosis โ cervical spondylosis โ\n",
"โโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_summary โ I've been having a lot of pain in my neck and back. I've also been having trouble with my โ\n",
"โ โ balance and coordination. I've been coughing a lot and my limbs feel weak. โ\n",
"โโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" Generated Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ Name โ Value โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ patient_sampler โ { โ\n",
"โ โ 'uuid': '68174e80-5e4b-4e5c-b68f-91bec63746ef', โ\n",
"โ โ 'locale': 'en_US', โ\n",
"โ โ 'first_name': 'Wendy', โ\n",
"โ โ 'last_name': 'Guzman', โ\n",
"โ โ 'middle_name': None, โ\n",
"โ โ 'sex': 'Female', โ\n",
"โ โ 'street_number': '7569', โ\n",
"โ โ 'street_name': 'Beth Ports', โ\n",
"โ โ 'city': 'West Lauren', โ\n",
"โ โ 'state': 'Arizona', โ\n",
"โ โ 'postcode': '62783', โ\n",
"โ โ 'age': 99, โ\n",
"โ โ 'birth_date': '1926-06-11', โ\n",
"โ โ 'country': 'Armenia', โ\n",
"โ โ 'marital_status': 'separated', โ\n",
"โ โ 'education_level': 'secondary_education', โ\n",
"โ โ 'unit': '', โ\n",
"โ โ 'occupation': 'Engineer, materials', โ\n",
"โ โ 'phone_number': '882-478-7638', โ\n",
"โ โ 'bachelors_field': 'no_degree' โ\n",
"โ โ } โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ doctor_sampler โ { โ\n",
"โ โ 'uuid': '6b0cc640-5322-4068-8e62-408648f2a01a', โ\n",
"โ โ 'locale': 'en_US', โ\n",
"โ โ 'first_name': 'Katherine', โ\n",
"โ โ 'last_name': 'Aguilar', โ\n",
"โ โ 'middle_name': None, โ\n",
"โ โ 'sex': 'Female', โ\n",
"โ โ 'street_number': '174', โ\n",
"โ โ 'street_name': 'Tran Shoals', โ\n",
"โ โ 'city': 'Lake Ryan', โ\n",
"โ โ 'state': 'Illinois', โ\n",
"โ โ 'postcode': '59489', โ\n",
"โ โ 'age': 106, โ\n",
"โ โ 'birth_date': '1919-07-02', โ\n",
"โ โ 'country': 'Mayotte', โ\n",
"โ โ 'marital_status': 'never_married', โ\n",
"โ โ 'education_level': 'secondary_education', โ\n",
"โ โ 'unit': '', โ\n",
"โ โ 'occupation': 'Archaeologist', โ\n",
"โ โ 'phone_number': '001-929-429-7385', โ\n",
"โ โ 'bachelors_field': 'no_degree' โ\n",
"โ โ } โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_id โ PT-CB852B6C โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ symptom_onset_date โ 2024-02-20 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ date_of_visit โ 2024-02-21 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ first_name โ Wendy โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ last_name โ Guzman โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ dob โ 1926-06-11 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician โ Dr. Aguilar โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician_notes โ **Dr. Katherine Aguilar** โ\n",
"โ โ **Primary Care Physician** โ\n",
"โ โ **Visit Date:** 2024-02-21 โ\n",
"โ โ **Patient:** Wendy Guzman โ\n",
"โ โ โ\n",
"โ โ --- โ\n",
"โ โ โ\n",
"โ โ **Chief Complaints:** โ\n",
"โ โ - Neck and back pain since 2024-02-20 โ\n",
"โ โ - Balance and coordination issues โ\n",
"โ โ - Frequent coughing โ\n",
"โ โ - Weakness in limbs โ\n",
"โ โ โ\n",
"โ โ **History of Present Illness:** โ\n",
"โ โ Wendy presents with worsening cervical spondylosis symptoms, including persistent neck and โ\n",
"โ โ back pain radiating to her shoulders and arms. She reports new-onset dizziness and โ\n",
"โ โ imbalance, which she attributes to her neck pain. Additionally, she has been coughing โ\n",
"โ โ frequently over the past few days and notes generalized weakness in her upper and lower โ\n",
"โ โ extremities. She denies fever, chest pain, or shortness of breath. โ\n",
"โ โ โ\n",
"โ โ **Past Medical History:** โ\n",
"โ โ - Chronic cervical spondylosis (diagnosed 2024-02-20) โ\n",
"โ โ - No prior surgeries or hospitalizations โ\n",
"โ โ - No known allergies โ\n",
"โ โ โ\n",
"โ โ **Medications:** โ\n",
"โ โ - Ibuprofen 400mg PRN for pain โ\n",
"โ โ - No other regular medications โ\n",
"โ โ โ\n",
"โ โ **Social History:** โ\n",
"โ โ - Sedentary lifestyle due to pain โ\n",
"โ โ - No smoking or alcohol use โ\n",
"โ โ - Denies recent travel or exposure to sick contacts โ\n",
"โ โ โ\n",
"โ โ **Review of Systems:** โ\n",
"โ โ - Constitutional: No fever, chills, or weight loss โ\n",
"โ โ - Neurological: Weakness in arms/legs, numbness in hands, occasional tingling โ\n",
"โ โ - Respiratory: Productive cough, no hemoptysis โ\n",
"โ โ - Musculoskeletal: Pain with neck movement, stiffness โ\n",
"โ โ โ\n",
"โ โ **Physical Exam:** โ\n",
"โ โ - **Vital Signs:** Temp 98.6ยฐF, HR 78, BP 128/80, RR 16, SpO2 98% on room air โ\n",
"โ โ - **Neck Exam:** Limited range of motion, tenderness at C5-C6, no focal deficits โ\n",
"โ โ - **Neurological:** Mild weakness in bilateral upper extremities (4/5 strength), no focal โ\n",
"โ โ deficits on reflex testing โ\n",
"โ โ - **Respiratory:** Clear lung fields, no wheezing or crackles โ\n",
"โ โ - **Musculoskeletal:** Mild tenderness in cervical spine, no joint swelling โ\n",
"โ โ โ\n",
"โ โ **Assessment:** โ\n",
"โ โ 1. Worsening cervical spondylosis with radicular symptoms (possible nerve root โ\n",
"โ โ irritation). โ\n",
"โ โ 2. New neurological symptoms (weakness, imbalance) possibly related to cervical spine โ\n",
"โ โ pathology or systemic issue. โ\n",
"โ โ 3. Productive cough โ differential includes upper respiratory infection, post-viral cough, โ\n",
"โ โ or exacerbation of cervical spine-related dysphagia/airway irritation. โ\n",
"โ โ โ\n",
"โ โ **Plan:** โ\n",
"โ โ 1. **Imaging:** Order MRI of the cervical spine to evaluate for disc herniation, stenosis, โ\n",
"โ โ or other structural abnormalities contributing to radiculopathy. โ\n",
"โ โ 2. **Neurology Referral:** For further evaluation of neurological deficits and balance โ\n",
"โ โ issues. โ\n",
"โ โ 3. **Respiratory Workup:** Consider chest X-ray or spirometry if cough persists or worsens โ\n",
"โ โ to rule out pneumonia or other pulmonary pathology. โ\n",
"โ โ 4. **Physical Therapy:** Referral for cervical spine mobilization and strengthening โ\n",
"โ โ exercises. โ\n",
"โ โ 5. **Pain Management:** Consider short-term prescription of a muscle relaxant or โ\n",
"โ โ neuropathic agent if symptoms persist. โ\n",
"โ โ 6. **Patient Education:** Advise activity modification, posture correction, and follow-up โ\n",
"โ โ in 1-2 weeks or sooner if symptoms escalate. โ\n",
"โ โ โ\n",
"โ โ **Next Appointment:** Scheduled for 2024-03-05 or sooner if symptoms worsen. โ\n",
"โ โ โ\n",
"โ โ **Signature:** โ\n",
"โ โ Dr. Katherine Aguilar โ\n",
"โ โ Primary Care Physician โ\n",
"โ โ [Contact Information] โ\n",
"โ โ โ\n",
"โโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" [index: 0] \n",
"\n"
],
"text/plain": [
" \n",
"\u001b[3m Seed Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1m \u001b[0m\u001b[1mName \u001b[0m\u001b[1m \u001b[0mโ\u001b[1m \u001b[0m\u001b[1mValue \u001b[0m\u001b[1m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ diagnosis โ cervical spondylosis โ\n",
"โโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_summary โ I've been having a lot of pain in my neck and back. I've also been having trouble with my โ\n",
"โ โ balance and coordination. I've been coughing a lot and my limbs feel weak. โ\n",
"โโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[3m Generated Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1m \u001b[0m\u001b[1mName \u001b[0m\u001b[1m \u001b[0mโ\u001b[1m \u001b[0m\u001b[1mValue \u001b[0m\u001b[1m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ patient_sampler โ \u001b[1m{\u001b[0m โ\n",
"โ โ \u001b[32m'uuid'\u001b[0m: \u001b[32m'68174e80-5e4b-4e5c-b68f-91bec63746ef'\u001b[0m, โ\n",
"โ โ \u001b[32m'locale'\u001b[0m: \u001b[32m'en_US'\u001b[0m, โ\n",
"โ โ \u001b[32m'first_name'\u001b[0m: \u001b[32m'Wendy'\u001b[0m, โ\n",
"โ โ \u001b[32m'last_name'\u001b[0m: \u001b[32m'Guzman'\u001b[0m, โ\n",
"โ โ \u001b[32m'middle_name'\u001b[0m: \u001b[3;35mNone\u001b[0m, โ\n",
"โ โ \u001b[32m'sex'\u001b[0m: \u001b[32m'Female'\u001b[0m, โ\n",
"โ โ \u001b[32m'street_number'\u001b[0m: \u001b[32m'7569'\u001b[0m, โ\n",
"โ โ \u001b[32m'street_name'\u001b[0m: \u001b[32m'Beth Ports'\u001b[0m, โ\n",
"โ โ \u001b[32m'city'\u001b[0m: \u001b[32m'West Lauren'\u001b[0m, โ\n",
"โ โ \u001b[32m'state'\u001b[0m: \u001b[32m'Arizona'\u001b[0m, โ\n",
"โ โ \u001b[32m'postcode'\u001b[0m: \u001b[32m'62783'\u001b[0m, โ\n",
"โ โ \u001b[32m'age'\u001b[0m: \u001b[1;36m99\u001b[0m, โ\n",
"โ โ \u001b[32m'birth_date'\u001b[0m: \u001b[32m'1926-06-11'\u001b[0m, โ\n",
"โ โ \u001b[32m'country'\u001b[0m: \u001b[32m'Armenia'\u001b[0m, โ\n",
"โ โ \u001b[32m'marital_status'\u001b[0m: \u001b[32m'separated'\u001b[0m, โ\n",
"โ โ \u001b[32m'education_level'\u001b[0m: \u001b[32m'secondary_education'\u001b[0m, โ\n",
"โ โ \u001b[32m'unit'\u001b[0m: \u001b[32m''\u001b[0m, โ\n",
"โ โ \u001b[32m'occupation'\u001b[0m: \u001b[32m'Engineer, materials'\u001b[0m, โ\n",
"โ โ \u001b[32m'phone_number'\u001b[0m: \u001b[32m'882-478-7638'\u001b[0m, โ\n",
"โ โ \u001b[32m'bachelors_field'\u001b[0m: \u001b[32m'no_degree'\u001b[0m โ\n",
"โ โ \u001b[1m}\u001b[0m โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ doctor_sampler โ \u001b[1m{\u001b[0m โ\n",
"โ โ \u001b[32m'uuid'\u001b[0m: \u001b[32m'6b0cc640-5322-4068-8e62-408648f2a01a'\u001b[0m, โ\n",
"โ โ \u001b[32m'locale'\u001b[0m: \u001b[32m'en_US'\u001b[0m, โ\n",
"โ โ \u001b[32m'first_name'\u001b[0m: \u001b[32m'Katherine'\u001b[0m, โ\n",
"โ โ \u001b[32m'last_name'\u001b[0m: \u001b[32m'Aguilar'\u001b[0m, โ\n",
"โ โ \u001b[32m'middle_name'\u001b[0m: \u001b[3;35mNone\u001b[0m, โ\n",
"โ โ \u001b[32m'sex'\u001b[0m: \u001b[32m'Female'\u001b[0m, โ\n",
"โ โ \u001b[32m'street_number'\u001b[0m: \u001b[32m'174'\u001b[0m, โ\n",
"โ โ \u001b[32m'street_name'\u001b[0m: \u001b[32m'Tran Shoals'\u001b[0m, โ\n",
"โ โ \u001b[32m'city'\u001b[0m: \u001b[32m'Lake Ryan'\u001b[0m, โ\n",
"โ โ \u001b[32m'state'\u001b[0m: \u001b[32m'Illinois'\u001b[0m, โ\n",
"โ โ \u001b[32m'postcode'\u001b[0m: \u001b[32m'59489'\u001b[0m, โ\n",
"โ โ \u001b[32m'age'\u001b[0m: \u001b[1;36m106\u001b[0m, โ\n",
"โ โ \u001b[32m'birth_date'\u001b[0m: \u001b[32m'1919-07-02'\u001b[0m, โ\n",
"โ โ \u001b[32m'country'\u001b[0m: \u001b[32m'Mayotte'\u001b[0m, โ\n",
"โ โ \u001b[32m'marital_status'\u001b[0m: \u001b[32m'never_married'\u001b[0m, โ\n",
"โ โ \u001b[32m'education_level'\u001b[0m: \u001b[32m'secondary_education'\u001b[0m, โ\n",
"โ โ \u001b[32m'unit'\u001b[0m: \u001b[32m''\u001b[0m, โ\n",
"โ โ \u001b[32m'occupation'\u001b[0m: \u001b[32m'Archaeologist'\u001b[0m, โ\n",
"โ โ \u001b[32m'phone_number'\u001b[0m: \u001b[32m'001-929-429-7385'\u001b[0m, โ\n",
"โ โ \u001b[32m'bachelors_field'\u001b[0m: \u001b[32m'no_degree'\u001b[0m โ\n",
"โ โ \u001b[1m}\u001b[0m โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_id โ PT-CB852B6C โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ symptom_onset_date โ 2024-02-20 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ date_of_visit โ 2024-02-21 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ first_name โ Wendy โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ last_name โ Guzman โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ dob โ 1926-06-11 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician โ Dr. Aguilar โ\n",
"โโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician_notes โ **Dr. Katherine Aguilar** โ\n",
"โ โ **Primary Care Physician** โ\n",
"โ โ **Visit Date:** 2024-02-21 โ\n",
"โ โ **Patient:** Wendy Guzman โ\n",
"โ โ โ\n",
"โ โ --- โ\n",
"โ โ โ\n",
"โ โ **Chief Complaints:** โ\n",
"โ โ - Neck and back pain since 2024-02-20 โ\n",
"โ โ - Balance and coordination issues โ\n",
"โ โ - Frequent coughing โ\n",
"โ โ - Weakness in limbs โ\n",
"โ โ โ\n",
"โ โ **History of Present Illness:** โ\n",
"โ โ Wendy presents with worsening cervical spondylosis symptoms, including persistent neck and โ\n",
"โ โ back pain radiating to her shoulders and arms. She reports new-onset dizziness and โ\n",
"โ โ imbalance, which she attributes to her neck pain. Additionally, she has been coughing โ\n",
"โ โ frequently over the past few days and notes generalized weakness in her upper and lower โ\n",
"โ โ extremities. She denies fever, chest pain, or shortness of breath. โ\n",
"โ โ โ\n",
"โ โ **Past Medical History:** โ\n",
"โ โ - Chronic cervical spondylosis (diagnosed 2024-02-20) โ\n",
"โ โ - No prior surgeries or hospitalizations โ\n",
"โ โ - No known allergies โ\n",
"โ โ โ\n",
"โ โ **Medications:** โ\n",
"โ โ - Ibuprofen 400mg PRN for pain โ\n",
"โ โ - No other regular medications โ\n",
"โ โ โ\n",
"โ โ **Social History:** โ\n",
"โ โ - Sedentary lifestyle due to pain โ\n",
"โ โ - No smoking or alcohol use โ\n",
"โ โ - Denies recent travel or exposure to sick contacts โ\n",
"โ โ โ\n",
"โ โ **Review of Systems:** โ\n",
"โ โ - Constitutional: No fever, chills, or weight loss โ\n",
"โ โ - Neurological: Weakness in arms/legs, numbness in hands, occasional tingling โ\n",
"โ โ - Respiratory: Productive cough, no hemoptysis โ\n",
"โ โ - Musculoskeletal: Pain with neck movement, stiffness โ\n",
"โ โ โ\n",
"โ โ **Physical Exam:** โ\n",
"โ โ - **Vital Signs:** Temp 98.6ยฐF, HR 78, BP 128/80, RR 16, SpO2 98% on room air โ\n",
"โ โ - **Neck Exam:** Limited range of motion, tenderness at C5-C6, no focal deficits โ\n",
"โ โ - **Neurological:** Mild weakness in bilateral upper extremities (4/5 strength), no focal โ\n",
"โ โ deficits on reflex testing โ\n",
"โ โ - **Respiratory:** Clear lung fields, no wheezing or crackles โ\n",
"โ โ - **Musculoskeletal:** Mild tenderness in cervical spine, no joint swelling โ\n",
"โ โ โ\n",
"โ โ **Assessment:** โ\n",
"โ โ 1. Worsening cervical spondylosis with radicular symptoms (possible nerve root โ\n",
"โ โ irritation). โ\n",
"โ โ 2. New neurological symptoms (weakness, imbalance) possibly related to cervical spine โ\n",
"โ โ pathology or systemic issue. โ\n",
"โ โ 3. Productive cough โ differential includes upper respiratory infection, post-viral cough, โ\n",
"โ โ or exacerbation of cervical spine-related dysphagia/airway irritation. โ\n",
"โ โ โ\n",
"โ โ **Plan:** โ\n",
"โ โ 1. **Imaging:** Order MRI of the cervical spine to evaluate for disc herniation, stenosis, โ\n",
"โ โ or other structural abnormalities contributing to radiculopathy. โ\n",
"โ โ 2. **Neurology Referral:** For further evaluation of neurological deficits and balance โ\n",
"โ โ issues. โ\n",
"โ โ 3. **Respiratory Workup:** Consider chest X-ray or spirometry if cough persists or worsens โ\n",
"โ โ to rule out pneumonia or other pulmonary pathology. โ\n",
"โ โ 4. **Physical Therapy:** Referral for cervical spine mobilization and strengthening โ\n",
"โ โ exercises. โ\n",
"โ โ 5. **Pain Management:** Consider short-term prescription of a muscle relaxant or โ\n",
"โ โ neuropathic agent if symptoms persist. โ\n",
"โ โ 6. **Patient Education:** Advise activity modification, posture correction, and follow-up โ\n",
"โ โ in 1-2 weeks or sooner if symptoms escalate. โ\n",
"โ โ โ\n",
"โ โ **Next Appointment:** Scheduled for 2024-03-05 or sooner if symptoms worsen. โ\n",
"โ โ โ\n",
"โ โ **Signature:** โ\n",
"โ โ Dr. Katherine Aguilar โ\n",
"โ โ Primary Care Physician โ\n",
"โ โ [Contact Information] โ\n",
"โ โ โ\n",
"โโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" [index: 0] \n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Run this cell multiple times to cycle through the 10 preview records.\n",
"preview.display_sample_record()"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" diagnosis | \n",
" patient_summary | \n",
" patient_sampler | \n",
" doctor_sampler | \n",
" patient_id | \n",
" symptom_onset_date | \n",
" date_of_visit | \n",
" first_name | \n",
" last_name | \n",
" dob | \n",
" physician | \n",
" physician_notes | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" cervical spondylosis | \n",
" I've been having a lot of pain in my neck and ... | \n",
" {'uuid': '68174e80-5e4b-4e5c-b68f-91bec63746ef... | \n",
" {'uuid': '6b0cc640-5322-4068-8e62-408648f2a01a... | \n",
" PT-CB852B6C | \n",
" 2024-02-20 | \n",
" 2024-02-21 | \n",
" Wendy | \n",
" Guzman | \n",
" 1926-06-11 | \n",
" Dr. Aguilar | \n",
" **Dr. Katherine Aguilar** \\n**Primary Care Ph... | \n",
"
\n",
" \n",
" | 1 | \n",
" impetigo | \n",
" I have a rash on my face that is getting worse... | \n",
" {'uuid': '4e98c43b-8f7d-4bb4-8bd7-a7eafa0b556d... | \n",
" {'uuid': '633df172-baa5-47bc-8e3b-d0b588542cdb... | \n",
" PT-CF78B0FA | \n",
" 2024-09-18 | \n",
" 2024-09-19 | \n",
" Amanda | \n",
" Smith | \n",
" 1994-09-27 | \n",
" Dr. Marsh | \n",
" **Dr. Sherri Marsh, MD** \\n**Primary Care Phy... | \n",
"
\n",
" \n",
" | 2 | \n",
" urinary tract infection | \n",
" I have been urinating blood. I sometimes feel ... | \n",
" {'uuid': 'c1af3224-3584-4e40-a995-3460f958848d... | \n",
" {'uuid': 'c59bda90-b568-4694-bf95-8fda720d6e79... | \n",
" PT-12A85CCD | \n",
" 2024-10-12 | \n",
" 2024-10-24 | \n",
" Timothy | \n",
" Simmons | \n",
" 1917-04-30 | \n",
" Dr. Carpenter | \n",
" **Dr. Elizabeth Carpenter** \\n**Primary Care ... | \n",
"
\n",
" \n",
" | 3 | \n",
" arthritis | \n",
" I have been having trouble with my muscles and... | \n",
" {'uuid': '0ba463b6-0b83-43b7-b689-78b6d25b2f48... | \n",
" {'uuid': '53f687e8-a9bd-4900-94ed-c9f23dfd0a07... | \n",
" PT-FFEF37D6 | \n",
" 2024-08-30 | \n",
" 2024-09-05 | \n",
" Patricia | \n",
" White | \n",
" 1984-11-27 | \n",
" Dr. Oliver | \n",
" **Dr. Maria Oliver** \\n**Primary Care Physici... | \n",
"
\n",
" \n",
" | 4 | \n",
" dengue | \n",
" I have been feeling really sick. My body hurts... | \n",
" {'uuid': 'f3ec1689-8b90-495c-b41a-fc64563a088c... | \n",
" {'uuid': '1e6ce90f-0907-4843-acad-7fc6325a4b5f... | \n",
" PT-DC0CCE6C | \n",
" 2024-12-19 | \n",
" 2024-12-28 | \n",
" Leah | \n",
" Weaver | \n",
" 1996-06-28 | \n",
" Dr. Wilkins | \n",
" **Dr. Christopher Wilkins** \\n**Primary Care ... | \n",
"
\n",
" \n",
" | 5 | \n",
" common cold | \n",
" I've been feeling really run down and weak. My... | \n",
" {'uuid': '99905039-a297-4539-8541-162c9bc0b944... | \n",
" {'uuid': '9824c4af-77bd-41ef-b3de-96bcef1a3afd... | \n",
" PT-79573082 | \n",
" 2024-09-19 | \n",
" 2024-09-24 | \n",
" Andrea | \n",
" Morrison | \n",
" 1944-07-30 | \n",
" Dr. Williams | \n",
" **Dr. Tristan Williams, Primary Care Physician... | \n",
"
\n",
" \n",
" | 6 | \n",
" dengue | \n",
" I have rashes all over my body. I also have a ... | \n",
" {'uuid': '94f729fc-c6f5-4f79-9930-cbe5b11591a5... | \n",
" {'uuid': 'f95cc47e-19c6-410e-9432-22bd7b3b264f... | \n",
" PT-CB395BC7 | \n",
" 2024-04-27 | \n",
" 2024-05-15 | \n",
" Holly | \n",
" Jackson | \n",
" 1939-07-24 | \n",
" Dr. Hoffman | \n",
" **Dr. Ashley Hoffman** \\n**Primary Care Physi... | \n",
"
\n",
" \n",
" | 7 | \n",
" dengue | \n",
" I have been feeling nauseous and have a consta... | \n",
" {'uuid': '32e44e19-e6da-4696-a7fd-d78ab2d31bd8... | \n",
" {'uuid': '629f6af3-f78c-4bf6-b253-c2b253d45813... | \n",
" PT-F673CE25 | \n",
" 2024-09-27 | \n",
" 2024-09-29 | \n",
" Lance | \n",
" Henderson | \n",
" 1981-11-17 | \n",
" Dr. Kennedy | \n",
" **Dr. Kristen Kennedy** \\n**Primary Care Phys... | \n",
"
\n",
" \n",
" | 8 | \n",
" impetigo | \n",
" I have a rash around my nose that is red and i... | \n",
" {'uuid': 'e61cefa1-254a-414b-baea-f9fa6696a854... | \n",
" {'uuid': 'cbec5207-bd43-4c5c-b8b6-5129065eae6f... | \n",
" PT-288E96A3 | \n",
" 2024-07-14 | \n",
" 2024-08-03 | \n",
" Megan | \n",
" Pena | \n",
" 1973-09-30 | \n",
" Dr. Moore | \n",
" **Dr. Pamela Moore** \\n**Primary Care Physici... | \n",
"
\n",
" \n",
" | 9 | \n",
" drug reaction | \n",
" I have a rash on my chest and back and it itch... | \n",
" {'uuid': 'c893e5ec-fa33-4acd-83de-2724eadab37e... | \n",
" {'uuid': 'eaf0d824-e65a-4e22-be8c-325f3d25bff7... | \n",
" PT-1DE48AF1 | \n",
" 2024-02-14 | \n",
" 2024-03-05 | \n",
" Melvin | \n",
" Thomas | \n",
" 1954-07-15 | \n",
" Dr. Leonard | \n",
" **Dr. Blake Leonard, Primary Care Physician** ... | \n",
"
\n",
" \n",
"
\n",
"
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ ๐จ Data Designer Dataset Profile โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"\n",
" \n",
" Dataset Overview \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ number of records โ number of columns โ percent complete records โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ 10 โ 12 โ 100.0% โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐ฑ Seed-Dataset Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ column name โ data type โ number unique values โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ diagnosis โ string โ 7 (70.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_summary โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐ฒ Sampler Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ column name โ data type โ number unique values โ sampler type โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ patient_sampler โ dict โ 10 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ doctor_sampler โ dict โ 10 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_id โ string โ 10 (100.0%) โ uuid โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ symptom_onset_date โ string โ 10 (100.0%) โ datetime โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ date_of_visit โ string โ 10 (100.0%) โ timedelta โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐ LLM-Text Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ โ โ โ prompt tokens โ completion tokens โ\n",
"โ column name โ data type โ number unique values โ per record โ per record โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ physician_notes โ string โ 10 (100.0%) โ 111.0 +/- 5.6 โ 725.5 +/- 102.8 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐งฉ Expression Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ column name โ data type โ number unique values โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ first_name โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ last_name โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ dob โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Table Notes โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ\n",
"โ โ\n",
"โ 1. All token statistics are based on a sample of max(1000, len(dataset)) records. โ\n",
"โ 2. Tokens are calculated using tiktoken's cl100k_base tokenizer. โ\n",
"โ โ\n",
"โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ\n",
" \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"\n"
],
"text/plain": [
"\u001b[1;38;2;118;185;0mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ \u001b[0m๐จ Data Designer Dataset Profile\u001b[1;38;2;118;185;0m โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\n",
"\n",
" \n",
"\u001b[1;38;2;118;185;0m Dataset Overview \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mnumber of records \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2mnumber of columns \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2mpercent complete records \u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ 10 โ 12 โ 100.0% โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐ฑ Seed-Dataset Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ diagnosis โ string โ 7 (70.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_summary โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐ฒ Sampler Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m sampler type\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ patient_sampler โ dict โ 10 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ doctor_sampler โ dict โ 10 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_id โ string โ 10 (100.0%) โ uuid โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ symptom_onset_date โ string โ 10 (100.0%) โ datetime โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ date_of_visit โ string โ 10 (100.0%) โ timedelta โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐ LLM-Text Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m prompt tokens\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m completion tokens\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m per record\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m per record\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ physician_notes โ string โ 10 (100.0%) โ 111.0 +/- 5.6 โ 725.5 +/- 102.8 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐งฉ Expression Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ first_name โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ last_name โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ dob โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician โ string โ 10 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[2mโญโ\u001b[0m\u001b[2mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\u001b[2m Table Notes \u001b[0m\u001b[2mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\u001b[2mโโฎ\u001b[0m\n",
"\u001b[2mโ\u001b[0m \u001b[2mโ\u001b[0m\n",
"\u001b[2mโ\u001b[0m 1. All token statistics are based on a sample of max(1000, len(dataset)) records. \u001b[2mโ\u001b[0m\n",
"\u001b[2mโ\u001b[0m 2. Tokens are calculated using tiktoken's cl100k_base tokenizer. \u001b[2mโ\u001b[0m\n",
"\u001b[2mโ\u001b[0m \u001b[2mโ\u001b[0m\n",
"\u001b[2mโฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ\u001b[0m\n",
" \n",
"\u001b[1;38;2;118;185;0mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Print the analysis as a table.\n",
"preview.analysis.to_report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### ๐ Scale up!\n",
"\n",
"- Happy with your preview data?\n",
"\n",
"- Use the `create` method to submit larger Data Designer generation jobs.\n"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[16:34:21] [INFO] ๐จ Creating Data Designer dataset\n",
"[16:34:21] [INFO] โ
Validation passed\n",
"[16:34:21] [INFO] โ๏ธ Sorting column configs into a Directed Acyclic Graph\n",
"[16:34:21] [INFO] ๐ฉบ Running health checks for models...\n",
"[16:34:21] [INFO] |-- ๐ Checking 'nvidia/nvidia-nemotron-nano-9b-v2' in provider named 'nvidia' for model alias 'nemotron-nano-v2'...\n",
"[16:34:22] [INFO] |-- โ
Passed!\n",
"[16:34:22] [INFO] โณ Processing batch 1 of 1\n",
"[16:34:22] [INFO] ๐ฑ Sampling 20 records from seed dataset\n",
"[16:34:22] [INFO] |-- seed dataset size: 820 records\n",
"[16:34:22] [INFO] |-- sampling strategy: ordered\n",
"[16:34:22] [INFO] ๐ฒ Preparing samplers to generate 20 records across 5 columns\n",
"[16:34:22] [INFO] (๐พ + ๐พ) Concatenating 2 datasets\n",
"[16:34:22] [INFO] ๐งฉ Generating column `first_name` from expression\n",
"[16:34:22] [INFO] ๐งฉ Generating column `last_name` from expression\n",
"[16:34:22] [INFO] ๐งฉ Generating column `dob` from expression\n",
"[16:34:22] [INFO] ๐งฉ Generating column `physician` from expression\n",
"[16:34:22] [INFO] ๐ Preparing llm-text column generation\n",
"[16:34:22] [INFO] |-- column name: 'physician_notes'\n",
"[16:34:22] [INFO] |-- model config:\n",
"{\n",
" \"alias\": \"nemotron-nano-v2\",\n",
" \"model\": \"nvidia/nvidia-nemotron-nano-9b-v2\",\n",
" \"inference_parameters\": {\n",
" \"temperature\": 0.5,\n",
" \"top_p\": 1.0,\n",
" \"max_tokens\": 1024,\n",
" \"max_parallel_requests\": 4,\n",
" \"timeout\": null,\n",
" \"extra_body\": null\n",
" },\n",
" \"provider\": \"nvidia\"\n",
"}\n",
"[16:34:22] [INFO] ๐ Processing llm-text column 'physician_notes' with 4 concurrent workers\n",
"[16:35:06] [INFO] ๐ Model usage summary:\n",
"{\n",
" \"nvidia/nvidia-nemotron-nano-9b-v2\": {\n",
" \"token_usage\": {\n",
" \"prompt_tokens\": 2581,\n",
" \"completion_tokens\": 15954,\n",
" \"total_tokens\": 18535\n",
" },\n",
" \"request_usage\": {\n",
" \"successful_requests\": 20,\n",
" \"failed_requests\": 0,\n",
" \"total_requests\": 20\n",
" },\n",
" \"tokens_per_second\": 416,\n",
" \"requests_per_minute\": 26\n",
" }\n",
"}\n",
"[16:35:06] [INFO] ๐ Measuring dataset column statistics:\n",
"[16:35:06] [INFO] |-- ๐ฑ column: 'diagnosis'\n",
"[16:35:06] [INFO] |-- ๐ฑ column: 'patient_summary'\n",
"[16:35:06] [INFO] |-- ๐ฒ column: 'patient_sampler'\n",
"[16:35:06] [INFO] |-- ๐ฒ column: 'doctor_sampler'\n",
"[16:35:06] [INFO] |-- ๐ฒ column: 'patient_id'\n",
"[16:35:06] [INFO] |-- ๐งฉ column: 'first_name'\n",
"[16:35:06] [INFO] |-- ๐งฉ column: 'last_name'\n",
"[16:35:06] [INFO] |-- ๐งฉ column: 'dob'\n",
"[16:35:06] [INFO] |-- ๐ฒ column: 'symptom_onset_date'\n",
"[16:35:06] [INFO] |-- ๐ฒ column: 'date_of_visit'\n",
"[16:35:06] [INFO] |-- ๐งฉ column: 'physician'\n",
"[16:35:06] [INFO] |-- ๐ column: 'physician_notes'\n"
]
}
],
"source": [
"job_results = data_designer_client.create(config_builder, num_records=20)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" diagnosis | \n",
" patient_summary | \n",
" patient_sampler | \n",
" doctor_sampler | \n",
" patient_id | \n",
" symptom_onset_date | \n",
" date_of_visit | \n",
" first_name | \n",
" last_name | \n",
" dob | \n",
" physician | \n",
" physician_notes | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" cervical spondylosis | \n",
" I've been having a lot of pain in my neck and ... | \n",
" {'age': 77, 'bachelors_field': 'no_degree', 'b... | \n",
" {'age': 80, 'bachelors_field': 'no_degree', 'b... | \n",
" PT-9B5AA9F0 | \n",
" 2024-11-12 | \n",
" 2024-11-21 | \n",
" Joseph | \n",
" Mcconnell | \n",
" 1948-09-04 | \n",
" Dr. Odonnell | \n",
" **Dr. Lori Odonnell, MD** \n",
"**Primary Care Phy... | \n",
"
\n",
" \n",
" | 1 | \n",
" impetigo | \n",
" I have a rash on my face that is getting worse... | \n",
" {'age': 18, 'bachelors_field': 'stem', 'birth_... | \n",
" {'age': 28, 'bachelors_field': 'no_degree', 'b... | \n",
" PT-879882E0 | \n",
" 2024-08-01 | \n",
" 2024-08-30 | \n",
" Ann | \n",
" Thomas | \n",
" 2007-09-30 | \n",
" Dr. Thomas | \n",
" **Dr. Melissa Thomas** \n",
"**Primary Care Physic... | \n",
"
\n",
" \n",
" | 2 | \n",
" urinary tract infection | \n",
" I have been urinating blood. I sometimes feel ... | \n",
" {'age': 35, 'bachelors_field': 'no_degree', 'b... | \n",
" {'age': 79, 'bachelors_field': 'education', 'b... | \n",
" PT-C8FAA660 | \n",
" 2024-11-09 | \n",
" 2024-11-24 | \n",
" Dalton | \n",
" Phillips | \n",
" 1990-01-31 | \n",
" Dr. Johns | \n",
" **Dr. Samantha Johns** \n",
"**Primary Care Physic... | \n",
"
\n",
" \n",
" | 3 | \n",
" arthritis | \n",
" I have been having trouble with my muscles and... | \n",
" {'age': 63, 'bachelors_field': 'stem_related',... | \n",
" {'age': 110, 'bachelors_field': 'no_degree', '... | \n",
" PT-ED015BD6 | \n",
" 2024-06-08 | \n",
" 2024-06-30 | \n",
" William | \n",
" Price | \n",
" 1962-09-19 | \n",
" Dr. Smith | \n",
" **Dr. Leah Smith, Primary Care Physician** \n",
"*... | \n",
"
\n",
" \n",
" | 4 | \n",
" dengue | \n",
" I have been feeling really sick. My body hurts... | \n",
" {'age': 94, 'bachelors_field': 'no_degree', 'b... | \n",
" {'age': 29, 'bachelors_field': 'no_degree', 'b... | \n",
" PT-0F6852CC | \n",
" 2024-11-09 | \n",
" 2024-12-06 | \n",
" Ashley | \n",
" Perez | \n",
" 1931-07-18 | \n",
" Dr. Koch | \n",
" **Dr. Michael Koch** \n",
"**Primary Care Physicia... | \n",
"
\n",
" \n",
"
\n",
"
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ ๐จ Data Designer Dataset Profile โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"\n",
" \n",
" Dataset Overview \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ number of records โ number of columns โ percent complete records โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ 20 โ 12 โ 100.0% โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐ฑ Seed-Dataset Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ column name โ data type โ number unique values โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ diagnosis โ string โ 12 (60.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_summary โ string โ 20 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐ฒ Sampler Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ column name โ data type โ number unique values โ sampler type โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ patient_sampler โ dict โ 20 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ doctor_sampler โ dict โ 20 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_id โ string โ 20 (100.0%) โ uuid โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ symptom_onset_date โ string โ 19 (95.0%) โ datetime โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ date_of_visit โ string โ 19 (95.0%) โ timedelta โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐ LLM-Text Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ โ โ โ prompt tokens โ completion tokens โ\n",
"โ column name โ data type โ number unique values โ per record โ per record โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ physician_notes โ string โ 20 (100.0%) โ 111.0 +/- 9.0 โ 760.0 +/- 111.1 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
" ๐งฉ Expression Columns \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ column name โ data type โ number unique values โ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ first_name โ string โ 19 (95.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ last_name โ string โ 19 (95.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ dob โ string โ 20 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician โ string โ 20 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ Table Notes โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ\n",
"โ โ\n",
"โ 1. All token statistics are based on a sample of max(1000, len(dataset)) records. โ\n",
"โ 2. Tokens are calculated using tiktoken's cl100k_base tokenizer. โ\n",
"โ โ\n",
"โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ\n",
" \n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"\n"
],
"text/plain": [
"\u001b[1;38;2;118;185;0mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ \u001b[0m๐จ Data Designer Dataset Profile\u001b[1;38;2;118;185;0m โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\n",
"\n",
" \n",
"\u001b[1;38;2;118;185;0m Dataset Overview \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mnumber of records \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2mnumber of columns \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2mpercent complete records \u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ 20 โ 12 โ 100.0% โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐ฑ Seed-Dataset Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ diagnosis โ string โ 12 (60.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_summary โ string โ 20 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐ฒ Sampler Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m sampler type\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ patient_sampler โ dict โ 20 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ doctor_sampler โ dict โ 20 (100.0%) โ person_from_faker โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ patient_id โ string โ 20 (100.0%) โ uuid โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ symptom_onset_date โ string โ 19 (95.0%) โ datetime โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ date_of_visit โ string โ 19 (95.0%) โ timedelta โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐ LLM-Text Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m prompt tokens\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m completion tokens\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m per record\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m per record\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ physician_notes โ string โ 20 (100.0%) โ 111.0 +/- 9.0 โ 760.0 +/- 111.1 โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[1;38;2;118;185;0m ๐งฉ Expression Columns \u001b[0m\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโณโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
"โ\u001b[1;2m \u001b[0m\u001b[1;2mcolumn name \u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m data type\u001b[0m\u001b[1;2m \u001b[0mโ\u001b[1;2m \u001b[0m\u001b[1;2m number unique values\u001b[0m\u001b[1;2m \u001b[0mโ\n",
"โกโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฉ\n",
"โ first_name โ string โ 19 (95.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ last_name โ string โ 19 (95.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ dob โ string โ 20 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโค\n",
"โ physician โ string โ 20 (100.0%) โ\n",
"โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\n",
" \n",
" \n",
"\u001b[2mโญโ\u001b[0m\u001b[2mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\u001b[2m Table Notes \u001b[0m\u001b[2mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\u001b[2mโโฎ\u001b[0m\n",
"\u001b[2mโ\u001b[0m \u001b[2mโ\u001b[0m\n",
"\u001b[2mโ\u001b[0m 1. All token statistics are based on a sample of max(1000, len(dataset)) records. \u001b[2mโ\u001b[0m\n",
"\u001b[2mโ\u001b[0m 2. Tokens are calculated using tiktoken's cl100k_base tokenizer. \u001b[2mโ\u001b[0m\n",
"\u001b[2mโ\u001b[0m \u001b[2mโ\u001b[0m\n",
"\u001b[2mโฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ\u001b[0m\n",
" \n",
"\u001b[1;38;2;118;185;0mโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ\u001b[0m\n"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"# Load the analysis results into memory.\n",
"analysis = job_results.load_analysis()\n",
"\n",
"analysis.to_report()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## โญ๏ธ Next Steps\n",
"\n",
"Use Data Designer to generate synthetic data for your specific use case!\n"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.9"
}
},
"nbformat": 4,
"nbformat_minor": 4
}