{ "cells": [ { "cell_type": "markdown", "id": "c5fd35b8-6808-477e-9618-926926e3d6ec", "metadata": {}, "source": [ "# Comparing two samples using `stambo`\n", "\n", "[![Binder](https://mybinder.org/badge_logo.svg)](https://mybinder.org/v2/gh/Oulu-IMEDS/stambo/main?labpath=notebooks%2FTwo_sample_test.ipynb)\n", "\n", "V1.1.5: © Aleksei Tiulpin, PhD, 2025\n", "\n", "There are many cases, when we develop models other than classification or regression, and we want to compute scores per datapoint, and then find their mean. We may often want to compare just two samples of measurements, and `stambo` allows to do this easily too.\n", "\n", "This example shows how to conduct a simple two-sample test. The example is synthetic, and we will just simply generate two Gaussian samples, and assess whether the mean of the second sample is greater than the mean of the first sample." ] }, { "cell_type": "markdown", "id": "ff6e13ca-dd76-44cd-9085-674af7927c34", "metadata": {}, "source": [ "## Importing the libraries" ] }, { "cell_type": "code", "execution_count": 1, "id": "f4da8f85-d04c-4232-b59d-fcb07025a9e7", "metadata": {}, "outputs": [ { "data": { "text/plain": [ "'0.1.5'" ] }, "execution_count": 1, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import numpy as np\n", "import stambo\n", "import matplotlib.pyplot as plt\n", "from scipy.linalg import block_diag\n", "\n", "SEED=2025\n", "\n", "stambo.__version__" ] }, { "cell_type": "markdown", "id": "ccdab62b-1320-4e4e-8877-97077b537ad6", "metadata": {}, "source": [ "## Data generation" ] }, { "cell_type": "code", "execution_count": 2, "id": "054ccde1-e5fd-415a-94c6-f537ad8fba10", "metadata": {}, "outputs": [], "source": [ "np.random.seed(SEED)\n", "n_samples = 50\n", "sample_1 = np.random.randn(n_samples)\n", "sample_2 = np.random.randn(n_samples)+0.7 # effect size is 0.7" ] }, { "cell_type": "markdown", "id": "5d4b527f-6765-4fb9-a208-a478751069b8", "metadata": {}, "source": [ "## Sample comparison\n", "\n", "Note that when it comes to a two-sample test, `stambo` does not require the statistic of choice to be a machine learning metric that is a subclass of `stambo.metrics.Metric`.\n", "Note that we have a non-paired design here (a regular two-sample test), so, it needs to be specified." ] }, { "cell_type": "code", "execution_count": 3, "id": "d29ac608-a618-4cfb-8547-509854ee4100", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "4950e740873a4c6caec761f58726ef43", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Bootstrapping: 0%| | 0/5000 [00:00" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def generate_covariance(size, rho, sigma=1.0):\n", " \"\"\"\n", " Generates an AR(1) covariance matrix for a single block.\n", " Formula: Cov(i, j) = sigma^2 * rho^|i-j|\n", " \"\"\"\n", " indices = np.arange(size)\n", " # Calculate |i - j| matrix\n", " diff = np.abs(np.subtract.outer(indices, indices))\n", " # Apply AR(1) formula\n", " cov_matrix = (sigma**2) * (rho**diff)\n", " return cov_matrix\n", "\n", "# --- Settings ---\n", "n_groups = 40 # Number of independent blocks\n", "N = 800\n", "p = 4 # Number of predictors (features)\n", "rho = 0.9 # Correlation strength within blocks (0 to 1)\n", "noise_sigma = 1.0 # Standard deviation of noise\n", "\n", "# --- 1. Generate Predictors (X) and True Coefficients (Beta) ---\n", "np.random.seed(42)\n", "X = np.random.normal(0, 1, size=(N, p))\n", "beta = np.array([2.5, -1.0, 3.2, 0.5])\n", "\n", "\n", "if N < n_groups:\n", " raise ValueError(\"N must be greater or equal to the number of groups\")\n", "\n", "base = np.ones(n_groups, dtype=int)\n", "remaining = N - n_groups\n", "\n", "probs = np.random.dirichlet(np.ones(n_groups) * 0.9)\n", "extra = np.random.multinomial(remaining, probs)\n", "counts = base + extra\n", "\n", "# Sanity check to ensure we can safely combine with X\n", "assert counts.sum() == N, \"Block sampling must result in exactly N samples\"\n", "\n", "group_errors_1 = []\n", "group_errors_2 = []\n", "block_covariances = [] # Storing these just to visualize later\n", "\n", "subject_ids = []\n", "for i in range(n_groups):\n", " # Create covariance for this specific block\n", " group_size = counts[i]\n", " subject_ids.extend([i] * group_size)\n", " sigma_block = generate_covariance(group_size, rho, noise_sigma)\n", " block_covariances.append(sigma_block)\n", " \n", " # Sample noise for this block: epsilon ~ N(0, Sigma_block)\n", " # We use Cholesky decomposition for sampling: L * standard_normal\n", " L = np.linalg.cholesky(sigma_block)\n", " u = np.random.normal(0, 1, size=group_size)\n", " e_block_1 = L @ u\n", "\n", " u = np.random.normal(0, 1, size=group_size)\n", " e_block_2 = L @ u\n", " \n", " \n", " group_errors_1.append(e_block_1)\n", " group_errors_2.append(e_block_2)\n", "\n", "# Concatenate all block errors to create the full epsilon vector\n", "epsilon_1 = np.concatenate(group_errors_1)\n", "epsilon_2 = np.concatenate(group_errors_2)\n", "\n", "# --- 3. Generate Target Variable (y) ---\n", "# Here we set no difference (NULL is true)\n", "group_1 = X @ beta + epsilon_1\n", "group_2 = X @ beta + epsilon_2\n", "\n", "if group_1.mean() > group_2.mean():\n", " group_1, group_2 = group_2, group_1\n", "\n", "print(f\"Generated data with shape: X={X.shape}, y={group_1.shape}\")\n", "print(f\"Structure: {n_groups} blocks of size {group_size}\")\n", "\n", "# --- Optional: Visualize the Full Covariance Matrix ---\n", "# Construct full matrix just for visualization\n", "Sigma_full = block_diag(*block_covariances)\n", "\n", "plt.figure(figsize=(6, 6))\n", "plt.imshow(Sigma_full, cmap='viridis', interpolation='none')\n", "#plt.colorbar(label='Covariance')\n", "plt.title(f\"Block Diagonal Sample-wise Covariance Matrix\\n({n_groups} blocks, rho={rho})\")\n", "plt.show()" ] }, { "cell_type": "markdown", "id": "35d76d15", "metadata": {}, "source": [ "## Grouped data & paired design\n", "\n", "Let us now cover another common case: we have a number of subjects with multiple measurements taken before and after treatment. We want to estimat the treatment effect, or rather - confidence intervals on it. \n", "This case is highly related to comparing two models. Model 1 could be seen as control, and improvements done on it (Model 2) could be seen as treatment. \n", "\n", "We now utilize our earlier generated data from the clustered design." ] }, { "cell_type": "code", "execution_count": 7, "id": "08f27af5", "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAgkAAADGCAYAAABYQxOwAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjcsIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvTLEjVAAAAAlwSFlzAAAPYQAAD2EBqD+naQAAG7RJREFUeJzt3Ql0VPX1wPGbhYQECJEtixIWZQmURdkEQYNEY6QohZalKGApVmUpe8GyRIpgQQFBltYqi5UitIqngngoEimrGOupspVgOOCBgIohgCYEmP+5v79vTib8AgnZZibfzznvzMx7j5mXN8O8O/e33ACXy+USAACAAgILrgAAACBIAAAAhSKTAAAArAgSAACAFUECAACwIkgAAABWBAkAAMCKIAEAAFgFiw+6evWqnDx5UmrUqCEBAQEVfTgAAPgMnUPx/PnzEhsbK4GBgf4XJGiAUL9+/Yo+DAAAfNaJEyfktttuK70gYc6cOfL222/LoUOHJCwsTLp06SJ//OMfpVmzZu59cnJyZPz48bJ27VrJzc2VpKQkWbp0qURFRbn3OX78uDz99NOybds2qV69ugwZMsQ8d3Bw0Q5HMwjOHxgREVGcPwEAgEotOzvb/NB2rqWlFiR89NFHMmLECOnQoYNcvnxZnn32WXnwwQflwIEDUq1aNbPP2LFjZePGjbJ+/XqpWbOmjBw5Uvr06SM7d+40269cuSI9e/aU6Oho2bVrl5w6dUoGDx4sVapUkdmzZxfpOJwmBg0QCBIAACi+ojTXB5SkwNPXX38t9erVM8HDvffeK+fOnZO6devKmjVr5Oc//7nZR7MO8fHxsnv3brn77rvl/fffl5/+9KemycDJLixfvlx+97vfmecLCQkpUhSkAYi+HkECAABFV5xraIlGN+gLqFq1apnbtLQ0ycvLk8TERPc+zZs3l7i4OBMkKL1t1aqVR/ODNknoQe/fv9/6OtpsodvzLwAAoGwFlmSEwZgxY+See+6Rn/zkJ2ZdZmamyQRERkZ67KsBgW5z9skfIDjbnW022l9Box5nodMiAABl76ZHN2jfhC+++EJ27NghZW3KlCkybty4azpdAABKRvuJaQYY/qNKlSoSFBRUcUGCdkZ87733ZPv27R7DJ7Qz4qVLlyQrK8sjm3D69Gmzzdnn448/9ng+3e5sswkNDTULAKB0aHc0zd7q9zX8T2RkpLmmlnQuoeDifqhGjRol77zzjqSmpkqjRo08trdr185EMFu3bpW+ffuadYcPHzZDHjt37mwe6+3zzz8vZ86cMZ0e1ZYtW0zniRYtWpTojwHgxVJSxCf4ynGWkBMg6PdweHg4E9P5CZfLJd9//725xqqYmJjyCxK0iUFHLrz77rtmfKXTh0D7Cei8CXo7bNgw0zSgnRn1wq9BhQYGOrJB6ZBJDQYef/xxmTt3rnmOqVOnmucmWwAA5dPE4AQItWvX5pT7mbCwMHPr/BgvSdNDsYKEZcuWmduEhASP9StWrJChQ4ea+wsWLDDTPGomIf9kSg49WG2q0MmUNHjQ+RV0MqWZM2fe9B8BwPt/hCekitco8BVW6Th9EDSDAP8U/uN7q+91uQUJRZlSoWrVqrJkyRKzFKZBgwayadOm4rw0AKCUUfvGfwWUUl0jqkACAAArggQAAOA/VSABAP4xuKMyDCZJTU2V7t27y3fffXfNZIPejkwCAMCn6Kg4HTnXuHFjMypOJ9fr1auXGX5fWrSD/pgxY6SyI5MAAPAZx44dM+UA9Bf5vHnzTC0g7cH/wQcfmKH0WlSwvLhcLjOcNDjYfy+lZBIAAD7jmWeeMT33deZeHWrftGlTadmypZmfZ8+ePWYfncDv0UcflerVq5v5evr16+ee2VelpKRI27Zt5Y033pCGDRuaOX4GDBgg58+fN9t1SL9WN3755ZfNa+miwYk2G+h9rWaskwdqFkNLE+hw/9GjR5s5CXSEX9euXWXfvn3iDwgSAAA+4ezZs7J582aTMdA5dgrS7IIWH9QAQffVC73O6Pvll19K//79PfY9evSobNiwwczbo4vu+8ILL5htGhzoPD7Dhw+XU6dOmSV/vaDJkyebfQ8ePCitW7eWSZMmyT/+8Q9ZtWqVfPrpp3LHHXeYOYL0GHyd/+ZIAAB+JT093aT4mzdvXug+2i/h888/l4yMDPeFffXq1SbboL/uO3ToYNZpMLFy5Uoze7DSWYD132rZAM0saEXj8PBwa00hnfzvgQceMPcvXrxoJhrU50pOTjbrXn31VROcvPbaazJx4kTxZWQSAAA+oSgT+umvew0O8v/y11IAmmXQbQ5tZnACBKfGgVPv4Ebat2/vkZHQPhHaT8KhNYw6duzo8Xq+iiABAOATmjRpYvoElEbnRL2Q56fPq9mFoqhmaerwVwQJAACfoIUDta1fp/3XNH9BWrQqPj5eTpw4YRbHgQMHzLbiVBrW5oYrV67ccL/bb7/d7Ltz5073Os0saNOGP1Q2JkgAAPgMDRD04q3pfO0seOTIEZPWX7RokelsmJiYaIZFDho0yHQi1FEQgwcPlvvuu8+jmeBGtDli7969ZlTDN998U2iWQbMKWrBQ+x5op0oNSLTDo5Zr1qrIvo6OiwAAn5kBUSdQ0ou/djAcP368GXlQt25dMyRROxBqs8G7775rJlu69957TVXihx56SBYvXlys15kwYYKpUKzZgB9++MF0hCyMjnTQIEI7P+owSg1GdN6GW265RXxdgKsoPUG8THZ2tul9eu7cOTMGFoAvlIpO8Y1S0d5+lSwFOTk55qLXqFEjM64fles9zi7GNZTmBgAAYEWQAAAArAgSAACAFUECAACwIkgAAABWBAkAAMCKIAEAAFgRJAAAACuCBAAAYMW0zAAqndTU62zzogkXK2Tyx/J+0Uoww6UvI5MAAPAJWpfhektKGQUcQ4cOld69e4u3WLlypURGRpbLa5FJAAD4BC3m5Hjrrbdk+vTpcvjwYfe66tWru+9rWSKtFhkczGWuXDMJ27dvl169eklsbKyJ3DZs2HBNxFUwutMKXPmdPXvWlPHUwhIaDWk5zQsXLpToDwEA+Lfo6Gj3ogWK9PriPD506JDUqFFD3n//fVMRMjQ0VHbs2GGqM86ZM8cUOgoLC5M2bdrI3//+d/dzaiCh1yBne7NmzeTll192b09JSZFVq1aZypLONS01NdWUkNb769atk27dupl/26FDB/nf//4n+/btM5UgNWhJTk6Wr7/+2uPv+Mtf/iLx8fGm8FLz5s1l6dKl7m3O87799tvSvXt3CQ8PN8e8e/dus11f+4knnjDFmco6g6KKHWJdvHjRHPCvfvUr6dOnj3UfDQpWrFjhfqxvVn4aIGhEuGXLFsnLyzN/8JNPPilr1qy5mb8BAABj8uTJ8uKLL5qS0lqqWQOEv/71r7J8+XJp0qSJ+aH72GOPmfLS9913nwkibrvtNlm/fr3Url1bdu3aZa5HMTEx0q9fP1My+uDBg6ZyonNdq1Wrlpw8edLcnzFjhixcuFDi4uLMdfGXv/ylCVY00NALvD6HZjy0jLV68803zeNXXnlF7rzzTvnPf/4jw4cPl2rVqpnS1I7f//735u/QY9b7AwcOlPT0dOnSpYt5vfxZlPwZlAoPEjQq0uV6NCjQyM5GT/bmzZvdkZbSOt8PP/ywOSGaoQAA4GbMnDlTHnjgAXM/NzdXZs+eLf/617+kc+fOZp0GD5ph+NOf/mSChCpVqshzzz3n/veaUdBf7Zoh6Nevn7kAa5ZAn8t2XdMgIikpydz/7W9/ay7mW7dulXvuuces0yyF9iFwaFDx0ksvuX9k6+sdOHDAHE/+IEGft2fPnua+Hl/Lli1NkKCZh/xZlLJWJo01mg6pV6+eieLuv/9+mTVrlonQlJ58bWJwAgSVmJgogYGBsnfvXvnZz352zfPpm6OLQyM6AAAKyn9t0Yvq999/7w4aHJcuXTK/4h1LliyR119/XY4fPy4//PCD2d62bdsindzWrVu770dFRZnbVq1aeaw7c+aMOxN/9OhREzho9sBx+fJlc+Ev7Hk1q6H0eTRIKE+lHiRoU4NGSBod6cl49tlnTeZBg4OgoCDJzMw0AYTHQQQHm/SNbrPRdFH+SA8AABtN2zucvm4bN26UW2+91WM/pxl87dq15le7/rrXbIM2FcybN8/8aC0KzUQ49Ne9bZ02aeQ/nldffVU6derk8Tx6fbzR8zrP49NBwoABA9z3NZrSaOj222832YUePXrc1HNOmTJFxo0b55FJqF+/fqkcL+AXvGSsecJ15h/wFQleNVFCYeu96Bi9WIsWLUwwoBkCbVqw2blzp2nnf+aZZ9zr9AdufiEhIaaDY0lpVkGb1L/88kvTN+9mldbxFEWZjw3R9p86deqYtI8GCdqG4qRe8qdadMRDYe0r+iYX7PwIAMD1aFZAswRjx441v8K7du1qRgVoYKCj67QPgHYMXL16tXzwwQcmA/7GG2+YPnN639GwYUOzXTsKatN5waaB4tCs+OjRo81zaOZdm9I/+eQT+e677zx+DF+PHo9mJbTvgw4k0A6SuvhkkPDVV1/Jt99+625T0XROVlaWpKWlmWEq6sMPPzRvYMH0C4Ci/XD0h1/w8BJ+lqX4wx/+YEYyaLO1/oLXPnF33XWXaQpXv/nNb8wIg/79+5u0vnY81KyCDqV0aP8BzYZrfwe9OG/bts1cqG/Gr3/9a3NB1yaNiRMnmuYRzbqPGTOmyM+hmY+nnnrKHLNeX7UzZFkNgwxw6YwTxaAnSLMCSjt+zJ8/34zl1D4FumiU1LdvX5MV0JTNpEmT5Pz58/L555+7swHaR+H06dNmSIozBFJPflGHQGpzg0ZhGhFqNAj4uxsHCf71xY7/l5BQyJko4QUhJydHMjIyzK9lHasP/5Nznfe4ONfQYk+mpGkRDQ6cnqGaHtH7OmZTO17897//lUceeUSaNm1qenBqtuDf//63R3OBjhPVHpra/KBDHzUF9Oc//7m4hwIAAMpQsZsbEhISzHSXhdF2mxvRjAMTJwEA4N0o8AQAAKwIEgAAgBVBAgBUUhUxOQ98672lhiYAVDI6GY9Oha9FinR4oD52ZvWDb3O5XGZaaa08qe+xvrclQZAAAJWMXjx0aJxW43WqGcK/hIeHm8qU+l6XBEECAFRC+gtTLyI64215TfGL8qHTEWhNpNLIDhEkAEAlpRcRLSSUv5gQkB8dFwEAgBVBAgAAsCJIAAAAVgQJAADAiiABAABYESQAAAArggQAAGBFkAAAAKwIEgAAgBVBAgAAsCJIAAAAVgQJAADAiiABAABYESQAAAArggQAAGBFkAAAAKwIEgAAgFWwfTVQeaSkXH97QuoNdigHCRV9AAAqJTIJAADAiiABAACUTpCwfft26dWrl8TGxkpAQIBs2LDBY7vL5ZLp06dLTEyMhIWFSWJiohw5csRjn7Nnz8qgQYMkIiJCIiMjZdiwYXLhwoXiHgoAAPCmIOHixYvSpk0bWbJkiXX73LlzZdGiRbJ8+XLZu3evVKtWTZKSkiQnJ8e9jwYI+/fvly1btsh7771nAo8nn3yyZH8JAACo2I6LycnJZrHRLMLChQtl6tSp8uijj5p1q1evlqioKJNxGDBggBw8eFA2b94s+/btk/bt25t9Fi9eLA8//LC8+OKLJkNRUG5urlkc2dnZxT1sAABQkX0SMjIyJDMz0zQxOGrWrCmdOnWS3bt3m8d6q00MToCgdP/AwECTebCZM2eOeR5nqV+/fmkeNgAAKOsgQQMEpZmD/PSxs01v69Wr57E9ODhYatWq5d6noClTpsi5c+fcy4kTJ0rzsAEAgK/OkxAaGmoWAADgo5mE6Ohoc3v69GmP9frY2aa3Z86c8dh++fJlM+LB2QcAAPhZkNCoUSNzod+6datHJ0Pta9C5c2fzWG+zsrIkLS3Nvc+HH34oV69eNX0XAACAjzY36HwG6enpHp0VP/vsM9OnIC4uTsaMGSOzZs2SJk2amKBh2rRpZsRC7969zf7x8fHy0EMPyfDhw80wyby8PBk5cqQZ+WAb2QAAAHwkSPjkk0+ke/fu7sfjxo0zt0OGDJGVK1fKpEmTzFwKOu+BZgy6du1qhjxWrVrV/W/efPNNExj06NHDjGro27evmVsBAAB4jwCXTm7gY7QJQ4dC6kgHnbUR8PcCT6icEhJu8kMLlNI1lNoNAADAiiABAAD47jwJAFAZpaYWst6LWhto+fBvZBIAAIAVQQIAALAiSAAAAFYECQAAwIogAQAAWBEkAAAAK4IEAABgRZAAAACsCBIAAIAVQQIAALBiWmYA8DFeVZm0sENhvma/QCYBAABYESQAAAArggQAAGBFkAAAAKwIEgAAgBVBAgAAsCJIAAAAVsyTgIrjJeOoE1Ir+ggAwDuRSQAAAFYECQAAwIogAQAAWBEkAACA8gkSUlJSJCAgwGNp3ry5e3tOTo6MGDFCateuLdWrV5e+ffvK6dOnS/swAACAN2YSWrZsKadOnXIvO3bscG8bO3as/POf/5T169fLRx99JCdPnpQ+ffqUxWEAAABvGwIZHBws0dHR16w/d+6cvPbaa7JmzRq5//77zboVK1ZIfHy87NmzR+6+++6yOBwAAOAtmYQjR45IbGysNG7cWAYNGiTHjx8369PS0iQvL08SExPd+2pTRFxcnOzevbvQ58vNzZXs7GyPBQAA+FiQ0KlTJ1m5cqVs3rxZli1bJhkZGdKtWzc5f/68ZGZmSkhIiERGRnr8m6ioKLOtMHPmzJGaNWu6l/r165f2YQMAgLJubkhOTnbfb926tQkaGjRoIOvWrZOwsLCbes4pU6bIuHHj3I81k0CgAACAjw+B1KxB06ZNJT093fRTuHTpkmRlZXnso6MbbH0YHKGhoRIREeGxAAAAHw8SLly4IEePHpWYmBhp166dVKlSRbZu3erefvjwYdNnoXPnzmV9KAAAoCKbGyZMmCC9evUyTQw6vHHGjBkSFBQkAwcONP0Jhg0bZpoOatWqZTICo0aNMgECIxsqX/0mCisBQCULEr766isTEHz77bdSt25d6dq1qxneqPfVggULJDAw0EyipKMWkpKSZOnSpaV9GAAAwNuChLVr1153e9WqVWXJkiVmAQAA3ovaDQAAwIogAQAAWBEkAACA8qvdAACoHFJTC9mQcIPhTeUsIeEmh2FVcmQSAACAFUECAACwIkgAAABWBAkAAMCKIAEAAFgRJAAAACuCBAAAYEWQAAAArAgSAACAFUECAACwIkgAAABWBAkAAMCKIAEAAFgRJAAAACuCBAAAYEWQAAAArAgSAACAVbB9NbxZSkpFHwEAoDIgkwAAAKzIJPiphFTSDQCAkiFIAAD4vdTUQjYkeNcPqoQE72pjJkjwwU4BCYV92AEAKEX0SQAAAN6VSViyZInMmzdPMjMzpU2bNrJ48WLp2LGj+EKCgF/yAIDKoEIyCW+99ZaMGzdOZsyYIZ9++qkJEpKSkuTMmTMVcTgAAMBbMgnz58+X4cOHyxNPPGEeL1++XDZu3Civv/66TJ48+Zr9c3NzzeI4d+6cuc3Ozi7V48r3Etd18XIRdwQAoBiybZeXUr7WOddOl8t1451d5Sw3N9cVFBTkeueddzzWDx482PXII49Y/82MGTP0L2HhHPAZ4DPAZ4DPAJ8BKZ1zcOLEiRtes8s9k/DNN9/IlStXJCoqymO9Pj506JD130yZMsU0TziuXr0qZ8+eldq1a0tAQECZH7M/0kiyfv36cuLECYmIiKjow/FpnEvOp7fis8n5tNEMwvnz5yU2Nlb8YghkaGioWfKLjIyssOPxJxogECRwLr0Rn03OpbeK8IPvzZo1a3pnx8U6depIUFCQnD592mO9Po6Oji7vwwEAAN4SJISEhEi7du1k69atHs0H+rhz587lfTgAAMCbmhu0f8GQIUOkffv2Zm6EhQsXysWLF92jHVD2tPlGh6AWbMYB57Ki8dnkXHqr0Er4vRmgvRcr4oVfeeUV92RKbdu2lUWLFkmnTp0q4lAAAIA3BQkAAMC7UbsBAABYESQAAAArggQAAGBFkAAAAKwIEiqh559/Xrp06SLh4eGFzlx5/Phx6dmzp9mnXr16MnHiRLl8+XK5H6uvadiwoZkqPP/ywgsvVPRh+QwtIa/nsGrVqma008cff1zRh+STUlJSrvkcNm/evKIPyyds375devXqZaYsDggIkA0bNnhs177+06dPl5iYGAkLC5PExEQ5cuSI+CuChEro0qVL8otf/EKefvpp63atraEBgu63a9cuWbVqlaxcudL8x8CNzZw5U06dOuVeRo0axWkrAkrIl66WLVt6fA537NjB57AIdM6eNm3amIDVZu7cuWbIvlYv3rt3r1SrVk2SkpIkJydH/FIpFXeED1qxYoWrZs2a16zftGmTKzAw0JWZmelet2zZMldERISp4onCNWjQwLVgwQJO0U3o2LGja8SIEe7HV65cccXGxrrmzJnD+SwmrZzbpk0bzlsJiYhHxeKrV6+6oqOjXfPmzXOvy8rKcoWGhrr+9re/+eX5JpOAa+zevVtatWrlUalTI2WtKLd//37O2A1o84JWKL3zzjvNhGE009yYZq3S0tJM6tYRGBhoHuvnEcWnKXBNmTdu3FgGDRpkmhBRMhkZGWYCwPyfUy2UpE1j/vo59YkqkChf+p/AVsrb2YbCjR49Wu666y6pVauWaarRMuea6p0/fz6nrZRLyKNwetHSJsJmzZqZz99zzz0n3bp1ky+++EJq1KjBqbtJmT9+/9k+p/763UgmwU9Mnjz5mo5KBRe+bMv+3GpdkoSEBGndurU89dRT8tJLL8nixYslNze3VN9v4HqSk5NNvyP9HGoWcNOmTZKVlSXr1q3jxKFYyCT4ifHjx8vQoUOvu4+mHYtCS3YX7FXulPaujOW8S3Ju9RedNjccO3bM/KqDHSXky5aOYmratKmkp6fzESyB6B+///T7UEc3OPSx1iDyRwQJfqJu3bpmKQ1asluHSZ45c8YMf1RbtmyRiIgIadGihVQ2JTm3n332mWlbd84jblxCvnfv3h4l5EeOHMlpK6ELFy7I0aNH5fHHH+dclkCjRo1MoKCfSyco0L5aOsqhsNFivo4goRLSDkxnz541t9oOrBcydccdd0j16tXlwQcfNMGAfqHocB9ta5s6daqMGDGiUpVILS7tuKRfFt27dzftvvp47Nix8thjj8ktt9xS0Yfn9SghX3omTJhgxvo3aNBATp48acobBwUFycCBA0vxVfw3oMqfccnIyDDfkdrPKC4uTsaMGSOzZs2SJk2amKBh2rRppoOoE9z6nYoeXoHyN2TIEDO0p+Cybds29z7Hjh1zJScnu8LCwlx16tRxjR8/3pWXl8fbdR1paWmuTp06mWGlVatWdcXHx7tmz57tysnJ4bwV0eLFi11xcXGukJAQMyRyz549nLub0L9/f1dMTIw5j7feeqt5nJ6ezrksAv0etH0/DhkyxD0Mctq0aa6oqCgz9LFHjx6uw4cP++25pVQ0AACwYnQDAACwIkgAAABWBAkAAMCKIAEAAFgRJAAAACuCBAAAYEWQAAAArAgSAACAFUECAACwIkgAAABWBAkAAEBs/g8iNYNSusHiawAAAABJRU5ErkJggg==", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "plt.figure(figsize=(6, 2))\n", "plt.hist(group_1, label=\"Control\", color=\"blue\", alpha=0.5)\n", "plt.hist(group_2, label=\"Treatment\", color=\"red\", alpha=0.5)\n", "plt.legend()\n", "plt.show()\n" ] }, { "cell_type": "markdown", "id": "1cc63ee0", "metadata": {}, "source": [ "Let us now first run the test without specifying the groups. " ] }, { "cell_type": "code", "execution_count": 8, "id": "b9975d69", "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "582e4122248740b4a896c9d14555453b", "version_major": 2, "version_minor": 0 }, "text/plain": [ "Bootstrapping: 0%| | 0/5000 [00:00