{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "80b37745",
   "metadata": {},
   "source": [
    "# Optimal Design Analysis\n",
    "---\n",
    "This notebook develops and investigates *optimal design analysis* (OPDA).\n",
    "\n",
    "OPDA provides a suite of tools for evaluating a deep learning model's design, and searching for an optimal one. For example, OPDA can answer how difficult it is to tune a model's hyperparameters, or whether the hyperparameters are fully tuned."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "304b6412",
   "metadata": {},
   "source": [
    "## Overview\n",
    "\n",
    "Many practical problems involve blackbox optimization. In these problems, you choose an input and then receive some amount of value. Investing more effort in the decision leads to better choices and greater value. For any given problem, it's important to understand this cost-benefit trade-off.\n",
    "\n",
    "The cost-benefit trade-off is captured by the *tuning curve*, or value as a function of the number of points tried [(Dodge et al., 2019)](https://arxiv.org/abs/1909.03004v1). Random search often provides a reasonable baseline for blackbox optimization, thus we can understand cost-benefit trade-offs by looking at its tuning curves.\n",
    "\n",
    "This notebook develops methods for constructing confidence intervals, plotting trends, and understanding the asymptotic behavior of tuning curves. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de97dc5a-5f77-4f79-a254-6c7f989c286a",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "Each notebook investigates a specific area or aspect of OPDA.\n",
    "\n",
    "### Theory\n",
    "\n",
    "The [**Theory**](#Theory) notebooks develop OPDA and the theory behind it.\n",
    "\n",
    "1. [**Formalizing the Problem**](./theory/formalizing-the-problem.ipynb): Notation and definitions for the tuning curve and random search.\n",
    "2. [**Nonparametric Analysis**](./theory/nonparametric-analysis.ipynb): Nonparametric, distribution-free analysis and confidence intervals for tuning curves.\n",
    "\n",
    "### Experiments\n",
    "\n",
    "The [**Experiments**](#Experiments) notebooks validate OPDA and explore its properties through empirical investigation.\n",
    "\n",
    "1. [**Comparing Point to Estimates**](./experiments/comparing-to-point-estimates.ipynb): A case study comparing optimal design analysis and point estimates for tuning curves.\n",
    "2. [**Evaluating DeBERTaV3 with the Nonparametric Analysis**](./experiments/evaluating-debertav3-with-the-nonparametric-analysis.ipynb): A nonparametric analysis of hyperparameter search for fine-tuning DeBERTa and DeBERTaV3."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd3373ed",
   "metadata": {},
   "source": [
    "## Contact\n",
    "\n",
    "See the [Git Repository](https://github.com/nalourie/opda) for more information on this work and how to cite it."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}