{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"\n",
"# `cmdline_helper.py`: Multi-platform command-line helper functions\n",
"## Authors: Brandon Clark & Zach Etienne\n",
"\n",
"## This notebook presents the functionality of the [cmdline_helper.py](../edit/cmdline_helper.py) NRPy+ module.\n",
"\n",
"**Notebook Status:** Self-Validated \n",
"\n",
"**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). **Additional validation tests may have been performed, but are as yet, undocumented. (TODO)**\n",
"\n",
"### NRPy+ Python Module associated with this notebook: [cmdline_helper.py](../edit/cmdline_helper.py)\n",
"\n",
"## Introduction:\n",
"Throughout the NRPy+ tutorial there are a handful of modules that require interaction with the command line, to compile C code, manipulate files, execute code, etc. This module serves as a reference for Python functions that exist in [cmdline_helper](../edit/cmdline_helper.py), which is designed to be compatible with Linux, Windows, and Mac OS command line interfaces."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"# Table of Contents\n",
"$$\\label{toc}$$\n",
"\n",
"This notebook is organized as follows\n",
"\n",
"1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules\n",
"1. [Step 2](#functions): The Functions\n",
" 1. [Step 2.a](#checkexec): **`check_executable_exists()`**\n",
" 1. [Step 2.b](#compile): **`C_Compile()`**\n",
" 1. [Step 2.c](#execute): **`Execute()`**\n",
" 1. [Step 2.d](#output): **`Execute_input_string()`**\n",
" 1. [Step 2.e](#delete): **`delete_existing_files()`** & **`mkdir()`**\n",
"1. [Step 3](#code_validation): Code Validation against `cmdline_helper.py`NRPy+ module\n",
"1. [Step 4](#latex_pdf_output): Output this notebook to $\\LaTeX$-formatted PDF file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"# Step 1: Initialize core NRPy+ modules \\[Back to [top](#toc)\\]\n",
"$$\\label{initializenrpy}$$\n",
"\n",
"Let's start by importing the necessary Python modules."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile cmdline_helper-validation.py\n",
"# As documented in the NRPy+ tutorial notebook\n",
"# Tutorial-cmdline_helper.ipynb, this Python script\n",
"# provides a multi-platform means to run executables,\n",
"# remove files, and compile code.\n",
"\n",
"# Basic functions:\n",
"# check_executable_exists(): Check to see whether an executable exists.\n",
"# Error out or return False if not exists;\n",
"# return True if executable exists in PATH.\n",
"# C_compile(): Compile C code using gcc.\n",
"# Execute(): Execute generated executable file, using taskset\n",
"# if available. Calls Execute_input_string() to\n",
"# redirect output from stdout & stderr to desired\n",
"# destinations.\n",
"# Execute_input_string(): Executes an input string and redirects\n",
"# output from stdout & stderr to desired destinations.\n",
"# delete_existing_files(file_or_wildcard):\n",
"# Runs del file_or_wildcard in Windows, or\n",
"# rm file_or_wildcard in Linux/MacOS\n",
"\n",
"# Authors: Brandon Clark\n",
"# Zach Etienne\n",
"# zachetie **at** gmail **dot* com\n",
"# Kevin Lituchy\n",
"\n",
"import io, os, shlex, subprocess, sys, time, multiprocessing, getpass, platform"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"# Step 2: The Functions \\[Back to [top](#toc)\\]\n",
"$$\\label{functions}$$\n",
"\n",
"\n",
"\n",
"## Step 2.a: `check_executable_exists()` \\[Back to [top](#toc)\\]\n",
"$$\\label{checkexec}$$\n",
"\n",
"`check_executable_exists()` takes the required string `exec_name` (i.e., the name of the executable) as its first input. Its second input is the optional boolean `error_if_not_found`, which defaults to `True` (so that it exits with an error if the executable is not found).\n",
"\n",
"`check_executable_exists()` returns `True` if the executable exists, and `False` if the executable does not exist and `error_if_not_found` is set to `False`."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# check_executable_exists(): Check to see whether an executable exists.\n",
"# Error out or return False if not exists;\n",
"# return True if executable exists in PATH.\n",
"def check_executable_exists(exec_name,error_if_not_found=True):\n",
" cmd = \"where\" if os.name == \"nt\" else \"which\"\n",
" try:\n",
" subprocess.check_output([cmd, exec_name])\n",
" except subprocess.CalledProcessError:\n",
" if error_if_not_found:\n",
" print(\"Sorry, cannot execute the command: \" + exec_name)\n",
" sys.exit(1)\n",
" else:\n",
" return False\n",
" return True"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"## Step 2.b: `C_Compile()` \\[Back to [top](#toc)\\]\n",
"$$\\label{compile}$$\n",
"\n",
"The `C_Compile()` function takes the following inputs as **strings**\n",
"* Path name to the generated C_file, `\"main_C_output_path\"`, and\n",
"* Name of the executable playground file, `\"main_C_output_file\"`.\n",
"\n",
"The `C_Compile()` function first checks for a ***gcc compiler***, which is a must when compiling C code within the NRPy+ tutorial. The function then removes any existing executable file. After that, the function constructs a script `compile_string` to run the compilation based on the function inputs and the operating system (OS) in use.\n",
"\n",
"Finally, it runs the actual compilation, by passing the compilation script `compile_string` on to the `Execute_input_string()` function, see [Step 2.c](#output)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# C_compile(): Write a function to compile the Main C code into an executable file\n",
"def C_compile(main_C_output_path, main_C_output_file, compile_mode=\"optimized\", custom_compile_string=\"\"):\n",
" print(\"Compiling executable...\")\n",
" # Step 1: Check for gcc compiler\n",
" check_executable_exists(\"gcc\")\n",
"\n",
" # Step 2: Delete existing version of executable\n",
" if os.name == \"nt\":\n",
" main_C_output_file += \".exe\"\n",
" delete_existing_files(main_C_output_file)\n",
"\n",
" # Step 3: Compile the executable\n",
" if compile_mode==\"safe\":\n",
" compile_string = \"gcc -O2 -g -fopenmp \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" # Step 3.A: Revert to more compatible gcc compile option\n",
" print(\"Most safe failed. Removing -fopenmp:\")\n",
" compile_string = \"gcc -O2 \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"\n",
" Execute_input_string(compile_string, os.devnull)\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" elif compile_mode==\"icc\":\n",
" check_executable_exists(\"icc\")\n",
" compile_string = \"icc -O2 -xHost -qopenmp -unroll \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" elif compile_mode==\"custom\":\n",
" Execute_input_string(custom_compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" elif compile_mode==\"optimized\":\n",
" compile_string = \"gcc -Ofast -fopenmp -march=native -funroll-loops \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" # Step 3.A: Revert to more compatible gcc compile option\n",
" print(\"Most optimized compilation failed. Removing -march=native:\")\n",
" compile_string = \"gcc -Ofast -fopenmp -funroll-loops \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"\n",
" Execute_input_string(compile_string, os.devnull)\n",
" if not os.path.isfile(main_C_output_file):\n",
" # Step 3.B: Revert to maximally compatible gcc compile option\n",
" print(\"Next-to-most optimized compilation failed. Moving to maximally-compatible gcc compile option:\")\n",
" compile_string = \"gcc -O2 \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Step 3.C: If there are still missing components within the compiler, say compilation failed\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" else:\n",
" print(\"Sorry, compile_mode = \\\"\"+compile_mode+\"\\\" unsupported.\")\n",
" sys.exit(1)\n",
"\n",
" print(\"Finished compilation.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"## Step 2.c: `Execute()` \\[Back to [top](#toc)\\]\n",
"$$\\label{execute}$$\n",
"\n",
"The `Execute()` function takes the following inputs as **strings**\n",
"* Name of the executable file, `main_C_output_file`,\n",
"* **(Optional):** Any necessary arguments associated with the execuatble file output, `executable_output_arguments`, and \n",
"* **(Optional):** Name of a file to store output during execution, `executable_output_file_name`.\n",
"\n",
"The `Execute()` function first removes any exitsing output files. It then begins to construct the script `execute_string` in order to execute the executable file that has been genrated by the `C_compile()` function. `execute_string` is built based on the function inputs and the operating system (OS) in use.\n",
"\n",
"Finally, it runs the actual execution, by passing the execution script `execute_string` on to the `Execute_input_string()` function, see [Step 2.c](#output)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# Execute(): Execute generated executable file, using taskset\n",
"# if available. Calls Execute_input_string() to\n",
"# redirect output from stdout & stderr to desired\n",
"# destinations.\n",
"def Execute(executable, executable_output_arguments=\"\", file_to_redirect_stdout=os.devnull,verbose=True):\n",
" # Step 1: Delete old version of executable file\n",
" if file_to_redirect_stdout != os.devnull:\n",
" delete_existing_files(file_to_redirect_stdout)\n",
"\n",
" # Step 2: Build the script for executing the desired executable\n",
" execute_string = \"\"\n",
" # When in Windows...\n",
" # https://stackoverflow.com/questions/1325581/how-do-i-check-if-im-running-on-windows-in-python\n",
" if os.name == \"nt\":\n",
" # ... do as the Windows do\n",
" # https://stackoverflow.com/questions/49018413/filenotfounderror-subprocess-popendir-windows-7\n",
" execute_prefix = \"cmd /c \" # Run with cmd /c executable [options] on Windows\n",
" else:\n",
" execute_prefix = \"./\" # Run with ./executable [options] on Linux & Mac\n",
" taskset_exists = check_executable_exists(\"taskset\", error_if_not_found=False)\n",
" if taskset_exists:\n",
" execute_string += \"taskset -c 0\"\n",
" if getpass.getuser() != \"jovyan\": # on mybinder, username is jovyan, and taskset -c 0 is the fastest option.\n",
" # If not on mybinder and taskset exists:\n",
" has_HT_cores = False # Does CPU have hyperthreading cores?\n",
" if platform.processor() != '': # If processor string returns null, then assume CPU does not support hyperthreading.\n",
" # This will yield correct behavior on ARM (e.g., cell phone) CPUs.\n",
" has_HT_cores=True\n",
" if has_HT_cores == True:\n",
" # NOTE: You will observe a speed-up by using only *PHYSICAL* (as opposed to logical/hyperthreading) cores:\n",
" N_cores_to_use = int(multiprocessing.cpu_count()/2) # To account for hyperthreading cores\n",
" else:\n",
" N_cores_to_use = int(multiprocessing.cpu_count()) # Use all cores if none are hyperthreading cores.\n",
" # This will happen on ARM (e.g., cellphone) CPUs\n",
" for i in range(N_cores_to_use-1):\n",
" execute_string += \",\"+str(i+1)\n",
" execute_string += \" \"\n",
" execute_string += execute_prefix+executable+\" \"+executable_output_arguments\n",
"\n",
" # Step 3: Execute the desired executable\n",
" Execute_input_string(execute_string, file_to_redirect_stdout,verbose)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"## Step 2.d: `Execute_input_string()` \\[Back to [top](#toc)\\]\n",
"$$\\label{output}$$\n",
"\n",
"The `Execute_input_string()` function takes the following inputs as strings\n",
"* The script to be executed, `input_string`, and\n",
"* An output file name for any needed redirects, `executable_output_file_name`. \n",
"\n",
"The `Execute_input_string()` executes a script, outputting `stderr` to the screen and redirecting any additional outputs from the executable to the specified `executable_output_file_name`. \n"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# Execute_input_string(): Executes an input string and redirects\n",
"# output from stdout & stderr to desired destinations.\n",
"def Execute_input_string(input_string, file_to_redirect_stdout=os.devnull, verbose=True):\n",
"\n",
" if verbose:\n",
" print(\"(EXEC): Executing `\"+input_string+\"`...\")\n",
" start = time.time()\n",
" # https://docs.python.org/3/library/subprocess.html\n",
" if os.name != 'nt':\n",
" args = shlex.split(input_string)\n",
" else:\n",
" args = input_string\n",
"\n",
" # https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command\n",
" filename = \"tmp.txt\"\n",
" with io.open(filename, 'w') as writer, io.open(filename, 'rb', buffering=-1) as reader, io.open(file_to_redirect_stdout, 'wb') as rdirect:\n",
" process = subprocess.Popen(args, stdout=rdirect, stderr=writer)\n",
" while process.poll() is None:\n",
" # https://stackoverflow.com/questions/21689365/python-3-typeerror-must-be-str-not-bytes-with-sys-stdout-write/21689447\n",
" sys.stdout.write(reader.read().decode('utf-8'))\n",
" time.sleep(0.2)\n",
" # Read the remaining\n",
" sys.stdout.write(reader.read().decode('utf-8'))\n",
" delete_existing_files(filename)\n",
" end = time.time()\n",
" if verbose:\n",
" print(\"(BENCH): Finished executing in \"+str(end-start)+\" seconds.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"## Step 2.e: `delete_existing_files()` & `mkdir()` \\[Back to [top](#toc)\\]\n",
"$$\\label{delete}$$\n",
"\n",
"The `delete_existing_files()` function takes a string, `file_or_wildcard`, as input.\n",
"\n",
"`delete_existing_files()` deletes any existing files that match the pattern given by `file_or_wildcard`. Deleting files is important when running the same code multiple times, ensuring that you're not reusing old data from a previous run, or seeing the same plot from a previous output. \n",
"\n",
"The `mkdir()` function makes a directory if it does not yet exist. It passes the input string \"newpath\" through `os.path.join()` to ensure that forward slashes are replaced by backslashes in Windows environments."
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# delete_existing_files(file_or_wildcard):\n",
"# Runs del file_or_wildcard in Windows, or\n",
"# rm file_or_wildcard in Linux/MacOS\n",
"def delete_existing_files(file_or_wildcard):\n",
" delete_string = \"\"\n",
" if os.name == \"nt\":\n",
" delete_string += \"del \" + file_or_wildcard\n",
" else:\n",
" delete_string += \"rm -f \" + file_or_wildcard\n",
" os.system(delete_string)\n",
"\n",
"# https://stackoverflow.com/questions/1274405/how-to-create-new-folder\n",
"def mkdir(newpath):\n",
" if not os.path.exists(os.path.join(newpath)):\n",
" os.makedirs(os.path.join(newpath))"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"def output_Jupyter_notebook_to_LaTeXed_PDF(notebookname,location_of_template_file=os.path.join(\".\"),verbose=True):\n",
" Execute_input_string(r\"jupyter nbconvert --to latex --template \"\n",
" +os.path.join(location_of_template_file,\"latex_nrpy_style.tplx\")\n",
" +r\" --log-level='WARN' \"+notebookname+\".ipynb\",verbose=False)\n",
" for _i in range(3): # _i is an unused variable.\n",
" Execute_input_string(r\"pdflatex -interaction=batchmode \"+notebookname+\".tex\",verbose=False)\n",
" delete_existing_files(notebookname+\".out \"+notebookname+\".aux \"+notebookname+\".log\")\n",
" if verbose:\n",
" import textwrap\n",
" wrapper = textwrap.TextWrapper(initial_indent=\"\",subsequent_indent=\" \",width=75)\n",
" print(wrapper.fill(\"Created \"+notebookname+\".tex, and compiled LaTeX file to PDF file \"+notebookname+\".pdf\"))\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"# Step 3: Code Validation against `cmdline_helper.py` NRPy+ module\n",
"$$\\label{code_validation}$$\n",
"\n",
"To validate the code in this tutorial we check for agreement between the files\n",
"\n",
"1. `cmdline_helper-validation.py` (written in this tutorial) and\n",
"1. the NRPy+ [cmdline_helper.py](../edit/cmdline_helper.py) module\n"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Printing difference between original cmdline_helper.py and this code, cmdline_helper-validation.py.\n",
"No difference. TEST PASSED!\n"
]
}
],
"source": [
"import difflib\n",
"import sys\n",
"\n",
"print(\"Printing difference between original cmdline_helper.py and this code, cmdline_helper-validation.py.\")\n",
"# Open the files to compare\n",
"with open(\"cmdline_helper.py\") as file1, open(\"cmdline_helper-validation.py\") as file2:\n",
" # Read the lines of each file\n",
" file1_lines = file1.readlines()\n",
" file2_lines = file2.readlines()\n",
" num_diffs = 0\n",
" for line in difflib.unified_diff(file1_lines, file2_lines, fromfile=\"cmdline_helper.py\", tofile=\"cmdline_helper-validation.py\"):\n",
" sys.stdout.writelines(line)\n",
" num_diffs = num_diffs + 1\n",
" if num_diffs == 0:\n",
" print(\"No difference. TEST PASSED!\")\n",
" else:\n",
" print(\"ERROR: Disagreement found with .py file. See differences above.\")\n",
" sys.exit(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"\n",
"# Step 4: Output this notebook to $\\LaTeX$-formatted PDF file \\[Back to [top](#toc)\\]\n",
"$$\\label{latex_pdf_output}$$\n",
"\n",
"The following code cell converts this Jupyter notebook into a proper, clickable $\\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename\n",
"[Tutorial-cmdline_helper.pdf](Tutorial-cmdline_helper.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"import cmdline_helper as cmd # NRPy+: Multi-platform Python command-line interface\n",
"cmd.output_Jupyter_notebook_to_LaTeXed_PDF(\"Tutorial-cmdline_helper\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.2"
}
},
"nbformat": 4,
"nbformat_minor": 2
}