{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<script async src=\"https://www.googletagmanager.com/gtag/js?id=UA-59152712-8\"></script>\n",
"<script>\n",
" window.dataLayer = window.dataLayer || [];\n",
" function gtag(){dataLayer.push(arguments);}\n",
" gtag('js', new Date());\n",
"\n",
" gtag('config', 'UA-59152712-8');\n",
"</script>\n",
"\n",
"# `cmdline_helper.py`: Multi-platform command-line helper functions\n",
"## Authors: Brandon Clark & Zach Etienne\n",
"\n",
"## The notebook presents [cmdline_helper.py](../edit/cmdline_helper.py), a multi-platform Python module for interacting with the command-line within the NRPy+ tutorial framework. It provides functions for checking if an executable exists, compiling and executing C code, executing input strings, and file manipulations such as deletion or directory creation. Its functionality ensures compatibility across Linux, Windows, and Mac OS command-line interfaces.\n",
"\n",
"**Notebook Status:** <font color=orange><b> Self-Validated </b></font>\n",
"\n",
"**Validation Notes:** This tutorial notebook has been confirmed to be self-consistent with its corresponding NRPy+ module, as documented [below](#code_validation). **Additional validation tests may have been performed, but are as yet, undocumented. (TODO)**\n",
"\n",
"### NRPy+ Python Module associated with this notebook: [cmdline_helper.py](../edit/cmdline_helper.py)\n",
"\n",
"## Introduction:\n",
"Throughout the NRPy+ tutorial, there are a handful of modules that require interaction with the command line, to compile C code, manipulate files, execute code, etc. This module serves as a reference for Python functions that exist in [cmdline_helper](../edit/cmdline_helper.py), which is designed to be compatible with Linux, Windows, and Mac OS command line interfaces."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='toc'></a>\n",
"\n",
"# Table of Contents\n",
"$$\\label{toc}$$\n",
"\n",
"This notebook is organized as follows\n",
"\n",
"1. [Step 1](#initializenrpy): Initialize core Python/NRPy+ modules\n",
"1. [Step 2](#functions): The Functions\n",
" 1. [Step 2.a](#checkexec): **`check_executable_exists()`**\n",
" 1. [Step 2.b](#compile): **`C_compile()`** and **`new_C_compile()`**\n",
" 1. [Step 2.c](#execute): **`Execute()`**\n",
" 1. [Step 2.d](#output): **`Execute_input_string()`**\n",
" 1. [Step 2.e](#delete): **`delete_existing_files()`** & **`mkdir()`**\n",
"1. [Step 3](#code_validation): Code Validation against `cmdline_helper.py`NRPy+ module\n",
"1. [Step 4](#latex_pdf_output): Output this notebook to $\\LaTeX$-formatted PDF file"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='initializenrpy'></a>\n",
"\n",
"# Step 1: Initialize core NRPy+ modules \\[Back to [top](#toc)\\]\n",
"$$\\label{initializenrpy}$$\n",
"\n",
"Let's start by importing the necessary Python modules."
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.340446Z",
"iopub.status.busy": "2021-01-16T21:04:00.339979Z",
"iopub.status.idle": "2021-01-16T21:04:00.342709Z",
"shell.execute_reply": "2021-01-16T21:04:00.342249Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Overwriting cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile cmdline_helper-validation.py\n",
"# As documented in the NRPy+ tutorial notebook\n",
"# Tutorial-cmdline_helper.ipynb, this Python script\n",
"# provides a multi-platform means to run executables,\n",
"# remove files, and compile code.\n",
"\n",
"# Basic functions:\n",
"# check_executable_exists(): Check to see whether an executable exists.\n",
"# Error out or return False if it does not exist;\n",
"# return True if executable exists in PATH.\n",
"# C_compile(): Compile C code using gcc.\n",
"# Execute(): Execute generated executable file, using taskset\n",
"# if available. Calls Execute_input_string() to\n",
"# redirect output from stdout & stderr to desired\n",
"# destinations.\n",
"# Execute_input_string(): Executes an input string and redirects\n",
"# output from stdout & stderr to desired destinations.\n",
"# delete_existing_files(file_or_wildcard):\n",
"# Runs del file_or_wildcard in Windows, or\n",
"# rm file_or_wildcard in Linux/MacOS\n",
"\n",
"# Authors: Brandon Clark\n",
"# Zach Etienne\n",
"# zachetie **at** gmail **dot* com\n",
"# Kevin Lituchy\n",
"\n",
"import io, os, shlex, subprocess, sys, time, multiprocessing, getpass, platform, glob"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='functions'></a>\n",
"\n",
"# Step 2: The Functions \\[Back to [top](#toc)\\]\n",
"$$\\label{functions}$$\n",
"\n",
"<a id='checkexec'></a>\n",
"\n",
"## Step 2.a: `check_executable_exists()` \\[Back to [top](#toc)\\]\n",
"$$\\label{checkexec}$$\n",
"\n",
"`check_executable_exists()` takes the required string `exec_name` (i.e., the name of the executable) as its first input. Its second input is the optional boolean `error_if_not_found`, which defaults to `True` (so that it exits with an error if the executable is not found).\n",
"\n",
"`check_executable_exists()` returns `True` if the executable exists, `False` if the executable does not exist, and `error_if_not_found` is set to `False`."
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.345259Z",
"iopub.status.busy": "2021-01-16T21:04:00.344787Z",
"iopub.status.idle": "2021-01-16T21:04:00.347303Z",
"shell.execute_reply": "2021-01-16T21:04:00.346936Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# check_executable_exists(): Check to see whether an executable exists.\n",
"# Error out or return False if it does not exist;\n",
"# return True if executable exists in PATH.\n",
"def check_executable_exists(exec_name, error_if_not_found=True):\n",
" cmd = \"where\" if os.name == \"nt\" else \"which\"\n",
" try:\n",
" subprocess.check_output([cmd, exec_name])\n",
" except subprocess.CalledProcessError:\n",
" if error_if_not_found:\n",
" print(\"Sorry, cannot execute the command: \" + exec_name)\n",
" sys.exit(1)\n",
" else:\n",
" return False\n",
" return True"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='compile'></a>\n",
"\n",
"## Step 2.b: **`C_compile()`** and **`new_C_compile()`** \\[Back to [top](#toc)\\]\n",
"$$\\label{compile}$$\n",
"\n",
"### `C_compile()`\n",
"\n",
"The `C_compile()` function takes the following inputs as **strings**\n",
"* Path name to the generated C_file, `\"main_C_output_path\"`, and\n",
"* Name of the executable playground file, `\"main_C_output_file\"`.\n",
"\n",
"The `C_compile()` function first checks for a ***gcc compiler***, which is a must when compiling C code within the NRPy+ tutorial. The function then removes any existing executable file. After that, the function constructs a script `compile_string` to run the compilation based on the function inputs and the operating system (OS) in use.\n",
"\n",
"Finally, it runs the actual compilation, by passing the compilation script `compile_string` on to the `Execute_input_string()` function, see [Step 2.d](#output)."
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.350611Z",
"iopub.status.busy": "2021-01-16T21:04:00.350193Z",
"iopub.status.idle": "2021-01-16T21:04:00.353091Z",
"shell.execute_reply": "2021-01-16T21:04:00.352682Z"
},
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# C_compile(): Write a function to compile the Main C code into an executable file\n",
"def C_compile(main_C_output_path, main_C_output_file, compile_mode=\"optimized\", custom_compile_string=\"\", additional_libraries=\"\"):\n",
" print(\"Compiling executable...\")\n",
" # Step 1: Check for gcc compiler\n",
" check_executable_exists(\"gcc\")\n",
"\n",
" if additional_libraries != \"\":\n",
" additional_libraries = \" \" + additional_libraries\n",
"\n",
" # Step 2: Delete existing version of executable\n",
" if os.name == \"nt\":\n",
" main_C_output_file += \".exe\"\n",
" delete_existing_files(main_C_output_file)\n",
"\n",
" # Step 3: Compile the executable\n",
" if compile_mode==\"safe\":\n",
" compile_string = \"gcc -std=gnu99 -O2 -g -fopenmp \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"+additional_libraries\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" # Step 3.A: Maybe gcc is actually clang in disguise (as in MacOS)?!\n",
" # https://stackoverflow.com/questions/33357029/using-openmp-with-clang\n",
" print(\"Most safe failed. Probably on MacOS. Replacing -fopenmp with -fopenmp=libomp:\")\n",
" compile_string = \"gcc -std=gnu99 -O2 -fopenmp=libomp \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"+additional_libraries\n",
" Execute_input_string(compile_string, os.devnull)\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" elif compile_mode==\"icc\":\n",
" check_executable_exists(\"icc\")\n",
" compile_string = \"icc -std=gnu99 -O2 -xHost -qopenmp -unroll \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"+additional_libraries\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" elif compile_mode==\"custom\":\n",
" Execute_input_string(custom_compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" elif compile_mode==\"optimized\":\n",
" compile_string = \"gcc -std=gnu99 -Ofast -fopenmp -march=native -funroll-loops \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"+additional_libraries\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" if not os.path.isfile(main_C_output_file):\n",
" # Step 3.A: Revert to more compatible gcc compile option\n",
" print(\"Most optimized compilation failed. Removing -march=native:\")\n",
" compile_string = \"gcc -std=gnu99 -Ofast -fopenmp -funroll-loops \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"+additional_libraries\n",
" Execute_input_string(compile_string, os.devnull)\n",
" if not os.path.isfile(main_C_output_file):\n",
" # Step 3.B: Revert to maximally compatible gcc compile option\n",
" print(\"Next-to-most optimized compilation failed. Moving to maximally-compatible gcc compile option:\")\n",
" compile_string = \"gcc -std=gnu99 -O2 \"+str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm\"+additional_libraries\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Step 3.C: If there are still missing components within the compiler, say compilation failed\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" elif compile_mode==\"emscripten\":\n",
" compile_string = \"emcc -std=gnu99 -s -O3 -march=native -funroll-loops -s ALLOW_MEMORY_GROWTH=1 \"\\\n",
" +str(main_C_output_path)+\" -o \"+str(main_C_output_file)+\" -lm \"+additional_libraries\n",
" Execute_input_string(compile_string, os.devnull)\n",
" # Check if executable exists (i.e., compile was successful), if not, try with more conservative compile flags.\n",
" # If there are still missing components within the compiler, say compilation failed\n",
" if not os.path.isfile(main_C_output_file):\n",
" print(\"Sorry, compilation failed.\")\n",
" sys.exit(1)\n",
" else:\n",
" print(\"Sorry, compile_mode = \\\"\"+compile_mode+\"\\\" unsupported.\")\n",
" sys.exit(1)\n",
"\n",
" print(\"Finished compilation.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### `new_C_compile()`\n",
"\n",
"The `new_C_compile()` function first constructs a `Makefile` from all functions registered to `outputC`'s `outC_function_dict`, and then attempts to compile the code using a parallel `make`. If that fails (e.g., due to the `make` command not being found or optimizations not being supported), it instead constructs a script that builds the code in serial with most compiler optimizations disabled.\n",
"\n",
"`new_C_compile()` takes the following inputs:\n",
"* `Ccodesrootdir`: Path to the generated C code, `Makefile`, and executable.\n",
"* `exec_name`: File name of executable.\n",
"* `uses_free_parameters_h=False` (optional): Will cause the compilation of `main.c` to depend on `free_parameters.h`, such that if the latter is updated `main.c` will be recompiled due to `make` being run.\n",
"* `compiler_opt_option=\"fast\"` (optional): Optimization option for compilation. Choose from `fast`, `fastdebug`, or `debug`. The latter two will enable the `-g` flag.\n",
"* `addl_CFLAGS=None` (optional): A list of additional compiler flags. E.g., `[-funroll-loops,-ffast-math]`.\n",
"* `addl_libraries=None` (optional): A list of additional libraries against which to link.\n",
"* `mkdir_Ccodesrootdir=True` (optional): Attempt to make the `Ccodesrootdir` directory if it doesn't exist. If it does exist, this has no effect.\n",
"* `CC=\"gcc\"` (optional): Choose the compiler. `\"gcc\"` should be used for the C compiler and `\"g++\"` for C++. FIXME: Other compilers may throw warnings due to default compilation flags being incompatible.\n",
"* `attempt=1` (optional, **do not touch**): An internal flag used to recursively call this function again in case the `Makefile` build fails (runs a shell script in serial with debug options disabled, as a backup)."
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"\n",
"from outputC import construct_Makefile_from_outC_function_dict\n",
"def new_C_compile(Ccodesrootdir, exec_name, uses_free_parameters_h=False,\n",
" compiler_opt_option=\"fast\", addl_CFLAGS=None,\n",
" addl_libraries=None, mkdir_Ccodesrootdir=True, CC=\"gcc\", attempt=1):\n",
" check_executable_exists(\"gcc\")\n",
" use_make = check_executable_exists(\"make\", error_if_not_found=False)\n",
"\n",
" construct_Makefile_from_outC_function_dict(Ccodesrootdir, exec_name, uses_free_parameters_h,\n",
" compiler_opt_option, addl_CFLAGS,\n",
" addl_libraries, mkdir_Ccodesrootdir, use_make, CC=CC)\n",
" orig_working_directory = os.getcwd()\n",
" os.chdir(Ccodesrootdir)\n",
" if use_make:\n",
" Execute_input_string(\"make -j\" + str(int(multiprocessing.cpu_count()) + 2), os.devnull)\n",
" else:\n",
" Execute_input_string(os.path.join(\"./\", \"backup_script_nomake.sh\"))\n",
" os.chdir(orig_working_directory)\n",
"\n",
" if not os.path.isfile(os.path.join(Ccodesrootdir, exec_name)) and attempt == 1:\n",
" print(\"Optimized compilation FAILED. Removing optimizations (including OpenMP) and retrying with debug enabled...\")\n",
" # First clean up object files.\n",
" filelist = glob.glob(os.path.join(Ccodesrootdir, \"*.o\"))\n",
" for file in filelist:\n",
" os.remove(file)\n",
" # Then retry compilation (recursion)\n",
" new_C_compile(Ccodesrootdir, exec_name, uses_free_parameters_h,\n",
" compiler_opt_option=\"debug\", addl_CFLAGS=addl_CFLAGS,\n",
" addl_libraries=addl_libraries, mkdir_Ccodesrootdir=mkdir_Ccodesrootdir, CC=CC, attempt=2)\n",
" if not os.path.isfile(os.path.join(Ccodesrootdir, exec_name)) and attempt == 2:\n",
" print(\"Sorry, compilation failed\")\n",
" sys.exit(1)\n",
" print(\"Finished compilation.\")\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='execute'></a>\n",
"\n",
"## Step 2.c: `Execute()` \\[Back to [top](#toc)\\]\n",
"$$\\label{execute}$$\n",
"\n",
"The `Execute()` function takes the following inputs as **strings**\n",
"* Name of the executable file, `main_C_output_file`,\n",
"* **(Optional):** Any necessary arguments associated with the executable file output, `executable_output_arguments`, and \n",
"* **(Optional):** Name of a file to store output during execution, `executable_output_file_name`.\n",
"\n",
"The `Execute()` function first removes any existing output files. It then begins to construct the script `execute_string` in order to execute the executable file that has been generated by the `C_compile()` function. `execute_string` is built based on the function inputs and the operating system (OS) in use.\n",
"\n",
"Finally, it runs the actual execution, by passing the execution script `execute_string` on to the `Execute_input_string()` function, see [Step 2.d](#output)."
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.356004Z",
"iopub.status.busy": "2021-01-16T21:04:00.355584Z",
"iopub.status.idle": "2021-01-16T21:04:00.357965Z",
"shell.execute_reply": "2021-01-16T21:04:00.357613Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"\n",
"# Execute(): Execute generated executable file, using taskset\n",
"# if available. Calls Execute_input_string() to\n",
"# redirect output from stdout & stderr to desired\n",
"# destinations.\n",
"def Execute(executable, executable_output_arguments=\"\", file_to_redirect_stdout=os.devnull, verbose=True):\n",
" # Step 1: Delete old version of executable file\n",
" if file_to_redirect_stdout != os.devnull:\n",
" delete_existing_files(file_to_redirect_stdout)\n",
"\n",
" # Step 2: Build the script for executing the desired executable\n",
" execute_string = \"\"\n",
" # When in Windows...\n",
" # https://stackoverflow.com/questions/1325581/how-do-i-check-if-im-running-on-windows-in-python\n",
" if os.name == \"nt\":\n",
" # ... do as the Windows do\n",
" # https://stackoverflow.com/questions/49018413/filenotfounderror-subprocess-popendir-windows-7\n",
" execute_prefix = \"cmd /c \" # Run with cmd /c executable [options] on Windows\n",
" else:\n",
" execute_prefix = \"./\" # Run with ./executable [options] on Linux & Mac\n",
" taskset_exists = check_executable_exists(\"taskset\", error_if_not_found=False)\n",
" if taskset_exists:\n",
" execute_string += \"taskset -c 0\"\n",
" on_4900hs = False\n",
" if platform.system() == \"Linux\" and \\\n",
" \"AMD Ryzen 9 4900HS\" in str(subprocess.check_output(\"cat /proc/cpuinfo\", shell=True)):\n",
" on_4900hs = True\n",
" if not on_4900hs and getpass.getuser() != \"jovyan\": # on mybinder, username is jovyan, and taskset -c 0 is the fastest option.\n",
" # If not on mybinder and taskset exists:\n",
" has_HT_cores = False # Does CPU have hyperthreading cores?\n",
" if platform.processor() != '': # If processor string returns null, then assume CPU does not support hyperthreading.\n",
" # This will yield correct behavior on ARM (e.g., cell phone) CPUs.\n",
" has_HT_cores=True\n",
" if has_HT_cores == True:\n",
" # NOTE: You will observe a speed-up by using only *PHYSICAL* (as opposed to logical/hyperthreading) cores:\n",
" N_cores_to_use = int(multiprocessing.cpu_count()/2) # To account for hyperthreading cores\n",
" else:\n",
" N_cores_to_use = int(multiprocessing.cpu_count()) # Use all cores if none are hyperthreading cores.\n",
" # This will happen on ARM (e.g., cellphone) CPUs\n",
" for i in range(N_cores_to_use-1):\n",
" execute_string += \",\"+str(i+1)\n",
" if on_4900hs:\n",
" execute_string = \"taskset -c 1,3,5,7,9,11,13,15\"\n",
" execute_string += \" \"\n",
" execute_string += execute_prefix+executable+\" \"+executable_output_arguments\n",
"\n",
" # Step 3: Execute the desired executable\n",
" Execute_input_string(execute_string, file_to_redirect_stdout, verbose)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='output'></a>\n",
"\n",
"## Step 2.d: `Execute_input_string()` \\[Back to [top](#toc)\\]\n",
"$$\\label{output}$$\n",
"\n",
"The `Execute_input_string()` function takes the following inputs as strings\n",
"* The script to be executed, `input_string`, and\n",
"* An output file name for any needed redirects, `executable_output_file_name`. \n",
"\n",
"The `Execute_input_string()` executes a script, outputting `stderr` to the screen and redirecting any additional outputs from the executable to the specified `executable_output_file_name`. \n"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.360658Z",
"iopub.status.busy": "2021-01-16T21:04:00.360243Z",
"iopub.status.idle": "2021-01-16T21:04:00.362589Z",
"shell.execute_reply": "2021-01-16T21:04:00.362235Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# Execute_input_string(): Executes an input string and redirects\n",
"# output from stdout & stderr to desired destinations.\n",
"def Execute_input_string(input_string, file_to_redirect_stdout=os.devnull, verbose=True):\n",
"\n",
" if verbose:\n",
" print(\"(EXEC): Executing `\"+input_string+\"`...\")\n",
" start = time.time()\n",
" # https://docs.python.org/3/library/subprocess.html\n",
" if os.name != 'nt':\n",
" args = shlex.split(input_string)\n",
" else:\n",
" args = input_string\n",
"\n",
" # https://stackoverflow.com/questions/18421757/live-output-from-subprocess-command\n",
" filename = \"tmp.txt\"\n",
" with io.open(filename, 'w') as writer, io.open(filename, 'rb', buffering=-1) as reader, io.open(file_to_redirect_stdout, 'wb') as rdirect:\n",
" process = subprocess.Popen(args, stdout=rdirect, stderr=writer)\n",
" while process.poll() is None:\n",
" # https://stackoverflow.com/questions/21689365/python-3-typeerror-must-be-str-not-bytes-with-sys-stdout-write/21689447\n",
" sys.stdout.write(reader.read().decode('utf-8'))\n",
" time.sleep(0.2)\n",
" # Read the remaining\n",
" sys.stdout.write(reader.read().decode('utf-8'))\n",
" delete_existing_files(filename)\n",
" end = time.time()\n",
" if verbose:\n",
" print(\"(BENCH): Finished executing in \"+'{:#.2f}'.format(round(end-start, 2))+\" seconds.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='delete'></a>\n",
"\n",
"## Step 2.e: `delete_existing_files()` & `mkdir()` \\[Back to [top](#toc)\\]\n",
"$$\\label{delete}$$\n",
"\n",
"The `delete_existing_files()` function takes a string, `file_or_wildcard`, as input.\n",
"\n",
"`delete_existing_files()` deletes any existing files that match the pattern given by `file_or_wildcard`. Deleting files is important when running the same code multiple times, ensuring that you're not reusing old data from a previous run, or seeing the same plot from a previous output. \n",
"\n",
"The `mkdir()` function makes a directory if it does not yet exist. It passes the input string \"newpath\" through `os.path.join()` to ensure that forward slashes are replaced by backslashes in Windows environments."
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.365232Z",
"iopub.status.busy": "2021-01-16T21:04:00.364830Z",
"iopub.status.idle": "2021-01-16T21:04:00.367324Z",
"shell.execute_reply": "2021-01-16T21:04:00.366875Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# delete_existing_files(file_or_wildcard):\n",
"# Runs del file_or_wildcard in Windows, or\n",
"# rm file_or_wildcard in Linux/MacOS\n",
"def delete_existing_files(file_or_wildcard):\n",
" delete_string = \"\"\n",
" if os.name == \"nt\":\n",
" delete_string += \"del \" + file_or_wildcard\n",
" else:\n",
" delete_string += \"rm -f \" + file_or_wildcard\n",
" os.system(delete_string)\n",
"\n",
"# https://stackoverflow.com/questions/1274405/how-to-create-new-folder\n",
"def mkdir(newpath):\n",
" if not os.path.exists(os.path.join(newpath)):\n",
" os.makedirs(os.path.join(newpath))"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.369986Z",
"iopub.status.busy": "2021-01-16T21:04:00.369583Z",
"iopub.status.idle": "2021-01-16T21:04:00.372026Z",
"shell.execute_reply": "2021-01-16T21:04:00.371622Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Appending to cmdline_helper-validation.py\n"
]
}
],
"source": [
"%%writefile -a cmdline_helper-validation.py\n",
"\n",
"# TO BE RUN ONLY FROM nrpytutorial or nrpytutorial/subdir/\n",
"def output_Jupyter_notebook_to_LaTeXed_PDF(notebookname, verbose=True):\n",
" in_nrpytutorial_rootdir = os.getcwd().split(\"/\")[-1] == \"nrpytutorial\"\n",
" if sys.version_info[0] == 3:\n",
" location_of_template_file = \".\"\n",
" if not in_nrpytutorial_rootdir:\n",
" location_of_template_file = \"..\"\n",
" Execute_input_string(r\"jupyter nbconvert --to latex --template=\"\n",
" +os.path.join(location_of_template_file, \"nbconvert_latex_settings\")\n",
" +r\" --log-level='WARN' \"+notebookname+\".ipynb\",verbose=False)\n",
" else:\n",
" Execute_input_string(r\"jupyter nbconvert --to latex --log-level='WARN' \"+notebookname+\".ipynb\",verbose=False)\n",
" for _i in range(3): # _i is an unused variable.\n",
" Execute_input_string(r\"pdflatex -interaction=batchmode \"+notebookname+\".tex\",verbose=False)\n",
" delete_existing_files(notebookname+\".out \"+notebookname+\".aux \"+notebookname+\".log\")\n",
" if verbose:\n",
" import textwrap\n",
" wrapper = textwrap.TextWrapper(initial_indent=\"\",subsequent_indent=\" \",width=75)\n",
" print(wrapper.fill(\"Created \"+notebookname+\".tex, and compiled LaTeX file to PDF file \"+notebookname+\".pdf\"))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='code_validation'></a>\n",
"\n",
"# Step 3: Code Validation against `cmdline_helper.py` NRPy+ module \\[Back to [top](#toc)\\]\n",
"$$\\label{code_validation}$$\n",
"\n",
"To validate the code in this tutorial we check for agreement between the files\n",
"\n",
"1. `cmdline_helper-validation.py` (written in this tutorial) and\n",
"1. the NRPy+ [cmdline_helper.py](../edit/cmdline_helper.py) module\n"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Printing difference between original cmdline_helper.py and this code, cmdline_helper-validation.py.\n"
]
},
{
"ename": "FileNotFoundError",
"evalue": "[Errno 2] No such file or directory: 'cmdline_helper-validation.py'",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mFileNotFoundError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[1], line 13\u001b[0m\n\u001b[1;32m 11\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mPrinting difference between original cmdline_helper.py and this code, cmdline_helper-validation.py.\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 12\u001b[0m \u001b[38;5;66;03m# Open the files to compare\u001b[39;00m\n\u001b[0;32m---> 13\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28mopen\u001b[39m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcmdline_helper.py\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;28;01mas\u001b[39;00m file1, \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mcmdline_helper-validation.py\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m file2:\n\u001b[1;32m 14\u001b[0m \u001b[38;5;66;03m# Read the lines of each file\u001b[39;00m\n\u001b[1;32m 15\u001b[0m file1_lines \u001b[38;5;241m=\u001b[39m trim_lines(file1\u001b[38;5;241m.\u001b[39mreadlines())\n\u001b[1;32m 16\u001b[0m file2_lines \u001b[38;5;241m=\u001b[39m trim_lines(file2\u001b[38;5;241m.\u001b[39mreadlines())\n",
"File \u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/site-packages/IPython/core/interactiveshell.py:284\u001b[0m, in \u001b[0;36m_modified_open\u001b[0;34m(file, *args, **kwargs)\u001b[0m\n\u001b[1;32m 277\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[1;32m 278\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[1;32m 279\u001b[0m \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 280\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 281\u001b[0m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[1;32m 282\u001b[0m )\n\u001b[0;32m--> 284\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
"\u001b[0;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'cmdline_helper-validation.py'"
]
}
],
"source": [
"import difflib\n",
"import sys\n",
"def trim_lines(lines):\n",
" new_lines=[]\n",
" for line in lines:\n",
" x = line.rstrip()\n",
" if x != '':\n",
" new_lines += [x]\n",
" return new_lines\n",
"\n",
"print(\"Printing difference between original cmdline_helper.py and this code, cmdline_helper-validation.py.\")\n",
"# Open the files to compare\n",
"with open(\"cmdline_helper.py\") as file1, open(\"cmdline_helper-validation.py\") as file2:\n",
" # Read the lines of each file\n",
" file1_lines = trim_lines(file1.readlines())\n",
" file2_lines = trim_lines(file2.readlines())\n",
" num_diffs = 0\n",
" for line in difflib.unified_diff(file1_lines, file2_lines, fromfile=\"cmdline_helper.py\", tofile=\"cmdline_helper-validation.py\"):\n",
" sys.stdout.writelines(line)\n",
" num_diffs = num_diffs + 1\n",
" if num_diffs == 0:\n",
" print(\"No difference. TEST PASSED!\")\n",
" else:\n",
" print(\"ERROR: Disagreement found with .py file. See differences above.\")\n",
" sys.exit(1)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<a id='latex_pdf_output'></a>\n",
"\n",
"# Step 4: Output this notebook to $\\LaTeX$-formatted PDF file \\[Back to [top](#toc)\\]\n",
"$$\\label{latex_pdf_output}$$\n",
"\n",
"The following code cell converts this Jupyter notebook into a proper, clickable $\\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename\n",
"[Tutorial-cmdline_helper.pdf](Tutorial-cmdline_helper.pdf). (Note that clicking on this link may not work; you may need to open the PDF file through another means.)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"execution": {
"iopub.execute_input": "2021-01-16T21:04:00.380697Z",
"iopub.status.busy": "2021-01-16T21:04:00.380303Z",
"iopub.status.idle": "2021-01-16T21:04:02.621274Z",
"shell.execute_reply": "2021-01-16T21:04:02.621779Z"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Created Tutorial-cmdline_helper.tex, and compiled LaTeX file to PDF file\n",
" Tutorial-cmdline_helper.pdf\n"
]
}
],
"source": [
"import cmdline_helper as cmd # NRPy+: Multi-platform Python command-line interface\n",
"cmd.output_Jupyter_notebook_to_LaTeXed_PDF(\"Tutorial-cmdline_helper\")"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.11.1"
}
},
"nbformat": 4,
"nbformat_minor": 4
}