{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "nbsphinx": {
     "execute": "never"
    }
   },
   "source": [
    "# Tutorial: optimizing memory estimation\n",
    "\n",
    "(You can [⇩ download this notebook 💾](https://raw.githubusercontent.com/FCP-INDI/C-PAC_tutorials/main/observed_usage/index.ipynb) and run it locally.)\n",
    "\n",
    "C-PAC has some built-in heuristics for estimating the memory required for each node, but these estimates can be greatly improved by providing observed memory usage for a given configuration and comparable data. This tutorial will step through the process of an observation run and an optimized run.\n",
    "\n",
    "Note: The code cells in this notebook are using a [BASH kernel](https://pypi.org/project/bash_kernel) and [cpac (Python package)](https://fcp-indi.github.io/docs/nightly/user/cpac) v0.5.0\n",
    "\n",
    "## Observation run\n",
    "\n",
    "First, we need to run a single exemplar through our pipeline configuration to observe how much memory is used at each node. How many cores we provide to this run should be the same number we intend to use in our optimized runs, but we'll want to give this run lots of memory to allow for the built-in conservative memory estimates.\n",
    "\n",
    "For this tutorial, we'll use a subject from [ADHD-200](http://fcon_1000.projects.nitrc.org/indi/adhd200), from the FCP-INDI AWS S3 bucket, and we'll use the anatomical only preconfiguration."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading 🐳 Docker\n",
      "[…]\n",
      "220414-15:17:19,750 nipype.workflow INFO:\n",
      "\t Running in parallel.\n",
      "220414-15:17:19,751 nipype.workflow WARNING:\n",
      "\t The following nodes are estimated to exceed the total amount of memory available (10.00GB): \n",
      "\tresample_u: 14.799999999999999 GB\n",
      "\tresample_o: 14.799999999999999 GB\n",
      "\tresample_u: 14.799999999999999 GB\n",
      "\tresample_o: 14.799999999999999 GB\n",
      "220414-15:17:20,8 nipype.workflow INFO:\n",
      "\t \n",
      "\n",
      "Error of subject workflow cpac_sub-0010042_ses-1\n",
      "\n",
      "CPAC run error:\n",
      "\n",
      "    Pipeline configuration: cpac_anat\n",
      "    Subject workflow: cpac_sub-0010042_ses-1\n",
      "[…]\n",
      "RuntimeError: Insufficient resources available for job:\n",
      "\tresample_u: 14.799999999999999 GB\n",
      "\tresample_o: 14.799999999999999 GB\n",
      "\tresample_u: 14.799999999999999 GB\n",
      "\tresample_o: 14.799999999999999 GB\n",
      "[…]"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac run \\\n",
    "  s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/observation participant \\\n",
    "  --preconfig anat-only \\\n",
    "  --participant_label 0010042 \\\n",
    "  --n_cpus 4 \\\n",
    "  --mem_gb 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see the initial estimate requires at least 14.8 GB for some nodes, so we'll need to give it more memory. We have a couple options here.\n",
    "\n",
    "1. If we don't have hard memory limits (we expect our system will allow us to use more memory than we allocate), we can tell C-PAC to ignore the insufficient resources by adjusting our pipeline configuration, like\n",
    "\n",
    "[anat-only-insufficient.yml](./configs/pipeline/anat-only-insufficient.yml)\n",
    "```YAML\n",
    "FROM: anat-only\n",
    "\n",
    "pipeline_setup:\n",
    "  system_config:\n",
    "    raise_insufficient: Off\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading 🐳 Docker\n",
      "[…]\n",
      "Skipping bids-validator for S3 datasets...\n",
      "Loading the 'anat-only' pre-configured pipeline.\n",
      "#### Running C-PAC for 0010042\n",
      "Number of participants to run in parallel: 1\n",
      "Input directory: s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU\n",
      "Output directory: […]/cpac_runs/insufficient/output\n",
      "Working directory: […]/cpac_runs/insufficient/working\n",
      "Log directory: […]/cpac_runs/insufficient/log\n",
      "Remove working directory: False\n",
      "Available memory: 10.0 (GB)\n",
      "Available threads: 4\n",
      "Number of threads for ANTs: 1\n",
      "Parsing s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU..\n",
      "Connecting to AWS: fcp-indi anonymously...\n",
      "gathering files from S3 bucket (s3.Bucket(name='fcp-indi')) for data/Projects/ADHD200/RawDataBIDS/NYU\n",
      "Starting participant level processing\n",
      "Run called with config file […]/cpac_runs/insufficient/cpac_pipeline_config_2022-04-14T15-18-12Z.yml\n",
      "220414-15:18:19,245 nipype.workflow INFO:\n",
      "\t \n",
      "    Run command: run s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/insufficient participant --pipeline_file […]/configs/pipeline/anat-only-insufficient.yml --participant_label 0010042 --n_cpus 4 --mem_gb 10\n",
      "\n",
      "    C-PAC version: 1.8.4.dev\n",
      "\n",
      "    Setting maximum number of cores per participant to 4\n",
      "    Setting number of participants at once to 1\n",
      "    Setting OMP_NUM_THREADS to 1\n",
      "    Setting MKL_NUM_THREADS to 1\n",
      "    Setting ANTS/ITK thread usage to 1\n",
      "    Maximum potential number of cores that might be used during this run: 4\n",
      "[…]\n",
      "    End of subject workflow cpac_sub-0010042_ses-1\n",
      "\n",
      "    CPAC run complete:\n",
      "\n",
      "        Pipeline configuration: cpac_anat\n",
      "        Subject workflow: cpac_sub-0010042_ses-1\n",
      "[…]"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac run \\\n",
    "  s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/insufficient participant \\\n",
    "  --pipeline_file $PWD/configs/pipeline/anat-only-insufficient.yml \\\n",
    "  --participant_label 0010042 \\\n",
    "  --n_cpus 4 \\\n",
    "  --mem_gb 10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or\n",
    "\n",
    " 2. We can just allocate enough memory to satisfy the estimates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading 🐳 Docker\n",
      "[…]\n",
      "Skipping bids-validator for S3 datasets...\n",
      "Loading the 'anat-only' pre-configured pipeline.\n",
      "#### Running C-PAC for 0010042\n",
      "Number of participants to run in parallel: 1\n",
      "Input directory: s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU\n",
      "Output directory: […]/cpac_runs/enough/output\n",
      "Working directory: […]/cpac_runs/enough/working\n",
      "Log directory: […]/cpac_runs/enough/log\n",
      "Remove working directory: False\n",
      "Available memory: 14.8 (GB)\n",
      "Available threads: 4\n",
      "Number of threads for ANTs: 1\n",
      "Parsing s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU..\n",
      "Connecting to AWS: fcp-indi anonymously...\n",
      "gathering files from S3 bucket (s3.Bucket(name='fcp-indi')) for data/Projects/ADHD200/RawDataBIDS/NYU\n",
      "Starting participant level processing\n",
      "Run called with config file […]/cpac_runs/enough/cpac_pipeline_config_2022-04-14T15-25-19Z.yml\n",
      "220414-15:25:26,762 nipype.workflow INFO:\n",
      "\t \n",
      "    Run command: run s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/enough participant --preconfig anat-only --participant_label 0010042 --n_cpus 4 --mem_gb 14.8\n",
      "\n",
      "    C-PAC version: 1.8.4.dev\n",
      "\n",
      "    Setting maximum number of cores per participant to 4\n",
      "    Setting number of participants at once to 1\n",
      "    Setting OMP_NUM_THREADS to 1\n",
      "    Setting MKL_NUM_THREADS to 1\n",
      "    Setting ANTS/ITK thread usage to 1\n",
      "    Maximum potential number of cores that might be used during this run: 4\n",
      "[…]\n",
      "    End of subject workflow cpac_sub-0010042_ses-1\n",
      "\n",
      "    CPAC run complete:\n",
      "\n",
      "        Pipeline configuration: cpac_anat\n",
      "        Subject workflow: cpac_sub-0010042_ses-1\n",
      "[…]"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac run \\\n",
    "  s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/enough participant \\\n",
    "  --preconfig anat-only \\\n",
    "  --participant_label 0010042 \\\n",
    "  --n_cpus 4 \\\n",
    "  --mem_gb 14.8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a `callback.log` file with observed memory usage. We can peek at the observations with `cpac parse-resources`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: cpac parse-resources [-h]\n",
      "                            [--filter_field {runtime,estimate,efficiency}]\n",
      "                            [--filter_group {lowest,highest}]\n",
      "                            [--filter_count FILTER_COUNT]\n",
      "                            callback\n",
      "\n",
      "positional arguments:\n",
      "  callback              callback.log file found in the 'log' directory of the\n",
      "                        specified derivatives path\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help            show this help message and exit\n",
      "  --filter_field {runtime,estimate,efficiency}, -f {runtime,estimate,efficiency}\n",
      "  --filter_group {lowest,highest}, -g {lowest,highest}\n",
      "  --filter_count FILTER_COUNT, -n FILTER_COUNT\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources --help"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the 5 most efficient nodes in each successful run, we can see the resource usage varies from run to run, even with the same data, system, and configuration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.8167    │ 1.2235    │ 66.75 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.8133    │ 1.2235    │ 66.48 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7481    │ 1.8547    │ 40.33 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.WM_63.seg_tissue…\u001b[0m\u001b[2m \u001b[0m│ 0.6710    │ 1.7051    │ 39.35 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7274    │ 1.8547    │ 39.22 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field efficiency \\\n",
    "  --filter_group highest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/insufficient/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7283    │ 1.2235    │ 59.53 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7850    │ 1.8547    │ 42.32 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.CSF_64.seg_tissu…\u001b[0m\u001b[2m \u001b[0m│ 0.7159    │ 1.7051    │ 41.99 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.WM_64.seg_tissue…\u001b[0m\u001b[2m \u001b[0m│ 0.6658    │ 1.7051    │ 39.05 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.GM_64.seg_tissue…\u001b[0m\u001b[2m \u001b[0m│ 0.6486    │ 1.7051    │ 38.04 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field efficiency \\\n",
    "  --filter_group highest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the other bottom five efficiency and the top and bottom five for each of the other fields for one of these `callback.log`s."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.montage_mni_anat…\u001b[0m\u001b[2m \u001b[0m│ 0.2135    │ 13.8000   │ 1.55 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.anat_reorient_0  \u001b[0m\u001b[2m \u001b[0m│ 0.2233    │ 13.8000   │ 1.62 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.qc_skullstrip_63…\u001b[0m\u001b[2m \u001b[0m│ 0.2272    │ 13.8000   │ 1.65 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.qc_skullstrip_63…\u001b[0m\u001b[2m \u001b[0m│ 0.2272    │ 13.8000   │ 1.65 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.qc_skullstrip_63…\u001b[0m\u001b[2m \u001b[0m│ 0.2272    │ 13.8000   │ 1.65 %    │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field efficiency \\\n",
    "  --filter_group lowest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.montage_mni_anat…\u001b[0m\u001b[2m \u001b[0m│ 0.2135    │ 13.8000   │ 1.55 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.montage_mni_anat…\u001b[0m\u001b[2m \u001b[0m│ 0.2279    │ 13.8000   │ 1.65 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.anat_reorient_0  \u001b[0m\u001b[2m \u001b[0m│ 0.2233    │ 13.8000   │ 1.62 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.montage_mni_anat…\u001b[0m\u001b[2m \u001b[0m│ 0.2279    │ 13.8000   │ 1.65 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.qc_skullstrip_82…\u001b[0m\u001b[2m \u001b[0m│ 0.2348    │ 13.8000   │ 1.70 %    │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field estimate \\\n",
    "  --filter_group highest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.segment_64       \u001b[0m\u001b[2m \u001b[0m│ 1.1257    │ 3.4958    │ 32.20 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.9344    │ 3.0513    │ 30.62 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7850    │ 1.8547    │ 42.32 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7283    │ 1.2235    │ 59.53 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.CSF_64.seg_tissu…\u001b[0m\u001b[2m \u001b[0m│ 0.7159    │ 1.7051    │ 41.99 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field runtime \\\n",
    "  --filter_group highest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7283    │ 1.2235    │ 59.53 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.3601    │ 1.2806    │ 28.12 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.2212    │ 1.4200    │ 15.58 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.3732    │ 1.5000    │ 24.88 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.CSF_64.seg_tissu…\u001b[0m\u001b[2m \u001b[0m│ 0.7159    │ 1.7051    │ 41.99 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field estimate \\\n",
    "  --filter_group lowest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.CSF_64.seg_tissu…\u001b[0m\u001b[2m \u001b[0m│ 0.1778    │ 2.5000    │ 7.11 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.func_ingress_sub…\u001b[0m\u001b[2m \u001b[0m│ 0.1788    │ 2.0000    │ 8.94 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.func_ingress_sub…\u001b[0m\u001b[2m \u001b[0m│ 0.1789    │ 2.0000    │ 8.95 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.func_ingress_sub…\u001b[0m\u001b[2m \u001b[0m│ 0.1789    │ 2.0000    │ 8.95 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.func_ingress_sub…\u001b[0m\u001b[2m \u001b[0m│ 0.1796    │ 2.0000    │ 8.98 %    │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field runtime \\\n",
    "  --filter_group lowest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Memory optimized runs\n",
    "\n",
    "Now that we have observed memory usage specific to our system + configuration + data shape, we can use these observations to inform the estimates of our subesquent runs. Here we'll run another subject and re-run the same intial subject so we can compare the performance. We'll use our original desired memory constraints.\n",
    "\n",
    "In this example, we'll use a buffer of 25% to demonstrate adjusting the buffer. If you don't set a buffer, the default of 10% will be used. If you don't want a buffer, you can set the buffer to 0.\n",
    "\n",
    "We can do this with a pipeline configuration like\n",
    "\n",
    "`anat-only-optimized.yml`\n",
    "```YAML\n",
    "FROM: anat-only\n",
    "pipeline_setup:\n",
    "  system_config:\n",
    "    observed_usage:\n",
    "      callback_log: cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log\n",
    "      buffer: 25\n",
    "```\n",
    "\n",
    "or we can use commandline flags like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Loading 🐳 Docker\n",
      "[…]\n",
      "Skipping bids-validator for S3 datasets...\n",
      "Loading the 'anat-only' pre-configured pipeline.\n",
      "#### Running C-PAC for 0010042, 5971050\n",
      "Number of participants to run in parallel: 1\n",
      "Input directory: s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU\n",
      "Output directory: […]/cpac_runs/optimized/output\n",
      "Working directory: […]/cpac_runs/optimized/working\n",
      "Log directory: […]/cpac_runs/optimized/log\n",
      "Remove working directory: False\n",
      "Available memory: 10.0 (GB)\n",
      "Available threads: 4\n",
      "Number of threads for ANTs: 1\n",
      "Parsing s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU..\n",
      "Connecting to AWS: fcp-indi anonymously...\n",
      "gathering files from S3 bucket (s3.Bucket(name='fcp-indi')) for data/Projects/ADHD200/RawDataBIDS/NYU\n",
      "Starting participant level processing\n",
      "Run called with config file […]/cpac_runs/optimized/cpac_pipeline_config_2022-04-14T15-30-12Z.yml\n",
      "220414-15:30:20,887 nipype.workflow INFO:\n",
      "\t \n",
      "    Run command: run s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/optimized participant --preconfig anat-only --participant_label 0010042 5971050 --n_cpus 4 --mem_gb 10 --runtime_usage cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log --runtime_buffer 25\n",
      "\n",
      "    C-PAC version: 1.8.4.dev\n",
      "\n",
      "    Setting maximum number of cores per participant to 4\n",
      "    Setting number of participants at once to 1\n",
      "    Setting OMP_NUM_THREADS to 1\n",
      "    Setting MKL_NUM_THREADS to 1\n",
      "    Setting ANTS/ITK thread usage to 1\n",
      "    Maximum potential number of cores that might be used during this run: 4\n",
      "[…]\n",
      "\t The following nodes used excessive resources:\n",
      "---------------------------------------------\n",
      "\n",
      "cpac_sub-0010042_ses-1\n",
      "  .sinker_desc-brain_T1w_50\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.258327484375 > 0.23091316162109377\n",
      "\n",
      "cpac_sub-0010042_ses-1\n",
      "  .sinker_label-GM_probseg_71\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.258304595703125 > 0.22700309814453123\n",
      "\n",
      "cpac_sub-0010042_ses-1\n",
      "  .nii_space-template_desc-brain_T1w_57\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.25536346484375 > 0.22719383300781248\n",
      "\n",
      "cpac_sub-0010042_ses-1\n",
      "  .nii_label-WM_desc-preproc_mask_78\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.236785888671875 > 0.22455692260742188\n",
      "---------------------------------------------\n",
      "\n",
      "220414-15:31:25,9 nipype.workflow INFO:\n",
      "\t \n",
      "\n",
      "    End of subject workflow cpac_sub-0010042_ses-1\n",
      "\n",
      "    CPAC run complete:\n",
      "\n",
      "        Pipeline configuration: cpac_anat\n",
      "        Subject workflow: cpac_sub-0010042_ses-1\n",
      "[…]\n",
      "\n",
      "220414-15:31:25,33 nipype.workflow INFO:\n",
      "\t \n",
      "    Run command: run s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/optimized participant --preconfig anat-only --participant_label 0010042 5971050 --n_cpus 4 --mem_gb 10 --runtime_usage cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log --runtime_buffer 25\n",
      "\n",
      "    C-PAC version: 1.8.4.dev\n",
      "\n",
      "    Setting maximum number of cores per participant to 4\n",
      "    Setting number of participants at once to 1\n",
      "    Setting OMP_NUM_THREADS to 1\n",
      "    Setting MKL_NUM_THREADS to 1\n",
      "    Setting ANTS/ITK thread usage to 1\n",
      "    Maximum potential number of cores that might be used during this run: 4\n",
      "[…]\n",
      "\t The following nodes used excessive resources:\n",
      "---------------------------------------------\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .sinker_space-T1w_desc-brain_mask_47\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.239810943359375 > 0.23887157470703124\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .sinker_desc-brain_T1w_50\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.23128891015625 > 0.23091316162109377\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .nii_label-CSF_mask_73\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.23983383203125 > 0.22700309814453123\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .sinker_label-GM_probseg_71\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.23983383203125 > 0.22700309814453123\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .nii_space-template_desc-brain_T1w_57\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.2322998046875 > 0.22719383300781248\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .ANTS_T1_to_template_51\n",
      "  .inverse_all_transform_flags\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.2548370361328125 > 0.2273988720703125\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .nii_from-template_to-T1w_mode-image_xfm_59\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.255855560546875 > 0.22507190673828126\n",
      "\n",
      "cpac_sub-5971050_ses-1\n",
      "  .sinker_from-template_to-T1w_mode-image_xfm_59\n",
      "      **memory_gb**\n",
      "        runtime > estimated\n",
      "        0.255855560546875 > 0.22446632324218752\n",
      "---------------------------------------------\n",
      "\n",
      "220414-15:32:37,121 nipype.workflow INFO:\n",
      "\t \n",
      "\n",
      "    End of subject workflow cpac_sub-5971050_ses-1\n",
      "\n",
      "    CPAC run complete:\n",
      "\n",
      "        Pipeline configuration: cpac_anat\n",
      "        Subject workflow: cpac_sub-5971050_ses-1\n",
      "[…]\n"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac run \\\n",
    "  s3://fcp-indi/data/Projects/ADHD200/RawDataBIDS/NYU cpac_runs/optimized participant \\\n",
    "  --preconfig anat-only \\\n",
    "  --participant_label 0010042 5971050 \\\n",
    "  --n_cpus 4 \\\n",
    "  --mem_gb 10 \\\n",
    "  --runtime_usage cpac_runs/enough/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log \\\n",
    "  --runtime_buffer 25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see how our efficiency compares across runs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.sinker_label-GM_…\u001b[0m\u001b[2m \u001b[0m│ 0.2583    │ 0.2270    │ 113.79 %  │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.nii_space-templa…\u001b[0m\u001b[2m \u001b[0m│ 0.2554    │ 0.2272    │ 112.40 %  │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.sinker_desc-brai…\u001b[0m\u001b[2m \u001b[0m│ 0.2583    │ 0.2309    │ 111.87 %  │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.nii_label-WM_des…\u001b[0m\u001b[2m \u001b[0m│ 0.2368    │ 0.2246    │ 105.45 %  │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.montage_mni_anat…\u001b[0m\u001b[2m \u001b[0m│ 0.3656    │ 0.3714    │ 98.45 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field efficiency \\\n",
    "  --filter_group highest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/optimized/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.resampled_T1w-br…\u001b[0m\u001b[2m \u001b[0m│ 0.1883    │ 2.0000    │ 9.41 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.montage_mni_anat…\u001b[0m\u001b[2m \u001b[0m│ 0.2606    │ 0.5124    │ 50.87 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.CSF_64.seg_tissu…\u001b[0m\u001b[2m \u001b[0m│ 0.1830    │ 0.3170    │ 57.74 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.1926    │ 0.3283    │ 58.65 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.get_pve_wm_64    \u001b[0m\u001b[2m \u001b[0m│ 0.2566    │ 0.4225    │ 60.73 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field efficiency \\\n",
    "  --filter_group lowest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/optimized/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.resampled_T1w-br…\u001b[0m\u001b[2m \u001b[0m│ 0.1883    │ 2.0000    │ 9.41 %    │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.segment_64       \u001b[0m\u001b[2m \u001b[0m│ 1.1376    │ 1.4072    │ 80.84 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.9533    │ 1.1681    │ 81.62 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7189    │ 0.9812    │ 73.27 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7630    │ 0.9104    │ 83.81 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field estimate \\\n",
    "  --filter_group highest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/optimized/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.segment_64       \u001b[0m\u001b[2m \u001b[0m│ 1.1376    │ 1.4072    │ 80.84 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.9533    │ 1.1681    │ 81.62 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7630    │ 0.9104    │ 83.81 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.ANTS_T1_to_templ…\u001b[0m\u001b[2m \u001b[0m│ 0.7189    │ 0.9812    │ 73.27 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.WM_64.seg_tissue…\u001b[0m\u001b[2m \u001b[0m│ 0.6409    │ 0.8323    │ 77.00 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field runtime \\\n",
    "  --filter_group highest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/optimized/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.CSF_64.seg_tissu…\u001b[0m\u001b[2m \u001b[0m│ 0.1800    │ 0.2223    │ 80.96 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.func_ingress_sub…\u001b[0m\u001b[2m \u001b[0m│ 0.1815    │ 0.2245    │ 80.84 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.sinker_from-temp…\u001b[0m\u001b[2m \u001b[0m│ 0.2025    │ 0.2245    │ 90.24 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.func_ingress_sub…\u001b[0m\u001b[2m \u001b[0m│ 0.1815    │ 0.2245    │ 80.84 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.func_ingress_sub…\u001b[0m\u001b[2m \u001b[0m│ 0.1800    │ 0.2245    │ 80.17 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field estimate \\\n",
    "  --filter_group lowest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/optimized/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━┓\n",
      "┃\u001b[1;35m                                          \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mMemory   \u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┃\u001b[1;35m \u001b[0m\u001b[1;35mTask ID                                 \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mUsed     \u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEstimated\u001b[0m\u001b[1;35m \u001b[0m┃\u001b[1;35m \u001b[0m\u001b[1;35mEfficien…\u001b[0m\u001b[1;35m \u001b[0m┃\n",
      "┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━┩\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.id_string_from-T…\u001b[0m\u001b[2m \u001b[0m│ 0.1762    │ 0.2311    │ 76.28 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.json_desc-brain_…\u001b[0m\u001b[2m \u001b[0m│ 0.1762    │ 0.2311    │ 76.28 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.id_string_space-…\u001b[0m\u001b[2m \u001b[0m│ 0.1762    │ 0.2343    │ 75.22 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.json_desc-reorie…\u001b[0m\u001b[2m \u001b[0m│ 0.1762    │ 0.2257    │ 78.08 %   │\n",
      "│\u001b[2m \u001b[0m\u001b[2mcpac_sub-0010042_ses-1.id_string_from-t…\u001b[0m\u001b[2m \u001b[0m│ 0.1762    │ 0.2344    │ 75.18 %   │\n",
      "└──────────────────────────────────────────┴───────────┴───────────┴───────────┘\n",
      "\u001b[?2004h"
     ]
    },
    {
     "ename": "",
     "evalue": "1",
     "output_type": "error",
     "traceback": []
    }
   ],
   "source": [
    "cpac parse-resources \\\n",
    "  --filter_field runtime \\\n",
    "  --filter_group lowest \\\n",
    "  --filter_count 5 \\\n",
    "  cpac_runs/optimized/log/pipeline_cpac_anat/sub-0010042_ses-1/callback.log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here are some box-and-whisker plots (generated in [visualize_observed_usage.ipynb](./visualize_observed_usage.ipynb) comparing the efficiency, estimates, and observed memory usage of the four runs we completed above:\n",
    "\n",
    "![Box-and-whisker plot of efficiency](./images/efficiency.png)\n",
    "![Box-and-whisker plot of estimated memory usage](./images/estimated.png)\n",
    "![Box-and-whisker plot of observed memory usage](./images/runtime.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notes\n",
    "\n",
    "### Total memory usage\n",
    "These estimates and observations are per-node. At any given moment, you'll have up to `n_cpus` nodes running at once, plus C-PAC and Nipype will be using some memory, so a worst-case scenario for memory usage is approximately the heaviest `n_cpus` nodes + `1 GB` for overhead.\n",
    "\n",
    "In this example, we set `n_cpus` to `4`. For the optimized re-run in this example, the four greediest nodes + `1 GB` for overhead = `(1.1376 + 0.9533 + 0.7630 + 0.7189 + 1) GB` = `4.5728 GB`, which is well within the specified limit of `10 GB`. In most cases, the greediest nodes won't be peaking simultaneously, so we won't expect an out-of-memory error on an optimized run, but they are possible, even if no individual node is overruning by much."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}