{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c7d90c1b-74ab-4590-800f-2c162452350c",
   "metadata": {},
   "source": [
    "# Module 1. Practice 2. Simple Parallel Codes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "37e3d58f-6bba-4b1f-bfae-7517af405b0b",
   "metadata": {},
   "source": [
    "## Introduction to simple Parallel Codes\n",
    "\n",
    "In High-Performance Computing (HPC), efficiently utilizing computational resources to solve problems faster is crucial. Python's `multiprocessing` module provides a robust and intuitive approach for parallel processing by allowing the creation and management of subprocesses. This is especially relevant in HPC where tasks are often CPU-bound and can benefit from the parallel execution across multiple cores of a processor.\n",
    "\n",
    "Parallel processing with the `multiprocessing` module effectively bypasses Python’s Global Interpreter Lock (GIL), which normally prevents multiple threads from executing Python bytecodes simultaneously. By using subprocesses instead of threads, each process gets its own Python interpreter and memory space, thus overcoming the limitations imposed by the GIL.\n",
    "\n",
    "In this practice, we will explore how to spawn multiple processes using the `multiprocessing` module. Each process will perform a simple computation—calculating the square of a number. This basic example serves as an introduction to the capabilities of parallel processing in Python, laying the groundwork for more complex parallel computations that are common in HPC applications, such as simulations, data analysis, and matrix operations. Understanding these fundamentals is essential for leveraging the full power of HPC resources to accelerate computation-intensive tasks.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4acf131b-7d9b-4e11-8ecf-99327f9029f0",
   "metadata": {},
   "source": [
    "## Understanding Serial vs. Parallel Execution in Python\n",
    "\n",
    "In this section, we explore the differences between serial and parallel execution using Python's `multiprocessing` module. We will compare how the execution time varies when calculating squares of numbers both serially and in parallel.\n",
    "\n",
    "### Serial Execution\n",
    "In serial execution, tasks are completed one after the other. This method does not utilize additional CPU cores, which can result in slower performance for CPU-bound tasks.\n",
    "\n",
    "### Parallel Execution\n",
    "Parallel execution allows multiple processes to run simultaneously, leveraging multiple CPU cores. This can significantly reduce the time required to complete CPU-intensive tasks by distributing the workload across available resources.\n",
    "\n",
    "## Estimating the Value of π Using Monte Carlo Simulation\n",
    "\n",
    "### What is Monte Carlo Simulation?\n",
    "Monte Carlo simulation is a statistical technique that allows us to compute an approximation of a value through random sampling. This method is often used in fields such as physics, finance, and engineering to solve problems that might be deterministic in principle but complex in practice.\n",
    "\n",
    "### Monte Carlo Simulation to Estimate π\n",
    "In this exercise, we use a Monte Carlo method to estimate the value of π. The principle behind this is simple: by randomly generating points within a square that encloses a quarter circle, we can estimate π based on the ratio of points that fall inside the circle to the total number of points. \n",
    "\n",
    "![Pi Monte carlo simulation](https://upload.wikimedia.org/wikipedia/commons/8/84/Pi_30K.gif)\n",
    "\n",
    "### Serial vs. Parallel Execution\n",
    "\n",
    "#### Serial Execution\n",
    "In serial execution, points are generated one at a time, and each point's position relative to the quarter circle is calculated sequentially. This approach does not leverage additional computational resources that might be available, such as multiple CPU cores, making it slower for a large number of points.\n",
    "\n",
    "#### Parallel Execution\n",
    "Parallel execution divides the task among multiple processes, allowing the simultaneous generation and evaluation of points. This method can significantly speed up the computation by utilizing multiple cores, thus demonstrating the power of parallel processing in computational tasks that are both independent and identically distributed.\n",
    "\n",
    "### Implementation\n",
    "We implement both serial and parallel approaches to estimate π. The parallel computation uses Python's `multiprocessing` module, which allows us to create multiple processes that can run on different cores and handle separate chunks of the task independently. The results from each process are then combined to get \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "d58a9410-5d1b-4d5c-bbc8-a158ccb5a7ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting serial calculation of π:\n",
      "Serial estimate of π: 3.1416214\n",
      "Serial execution time: 9.17483901977539 seconds\n",
      "\n",
      "Starting parallel calculation of π:\n",
      "Parallel estimate of π: 3.1410186\n",
      "Parallel execution time: 9.252578973770142 seconds\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "import random\n",
    "from multiprocessing import Process, Queue\n",
    "\n",
    "def monte_carlo_pi_part(n, queue):\n",
    "    count = 0\n",
    "    for _ in range(n):\n",
    "        x = random.random()\n",
    "        y = random.random()\n",
    "        if x**2 + y**2 <= 1:\n",
    "            count += 1\n",
    "    queue.put(count)\n",
    "\n",
    "def serial_monte_carlo_pi(n):\n",
    "    count = 0\n",
    "    for _ in range(n):\n",
    "        x = random.random()\n",
    "        y = random.random()\n",
    "        if x**2 + y**2 <= 1:\n",
    "            count += 1\n",
    "    return 4 * count / n\n",
    "\n",
    "def parallel_monte_carlo_pi(total_samples, num_processes):\n",
    "    queue = Queue()\n",
    "    processes = []\n",
    "    samples_per_process = total_samples // num_processes\n",
    "\n",
    "    start_time = time.time()\n",
    "    for _ in range(num_processes):\n",
    "        p = Process(target=monte_carlo_pi_part, args=(samples_per_process, queue))\n",
    "        processes.append(p)\n",
    "        p.start()\n",
    "\n",
    "    total_count = 0\n",
    "    for _ in range(num_processes):\n",
    "        total_count += queue.get()\n",
    "\n",
    "    for p in processes:\n",
    "        p.join()\n",
    "    end_time = time.time()\n",
    "\n",
    "    pi_estimate = 4 * total_count / total_samples\n",
    "    print(f\"Parallel estimate of π: {pi_estimate}\")\n",
    "    print(f\"Parallel execution time: {end_time - start_time} seconds\")\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    n_samples = 20_000_000\n",
    "    num_processes = 4\n",
    "\n",
    "    print(\"Starting serial calculation of π:\")\n",
    "    start_time = time.time()\n",
    "    pi_estimate = serial_monte_carlo_pi(n_samples)\n",
    "    end_time = time.time()\n",
    "    print(f\"Serial estimate of π: {pi_estimate}\")\n",
    "    print(f\"Serial execution time: {end_time - start_time} seconds\")\n",
    "\n",
    "    print(\"\\nStarting parallel calculation of π:\")\n",
    "    parallel_monte_carlo_pi(n_samples, num_processes)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58b8b410-4975-4529-afa5-d95819540719",
   "metadata": {},
   "source": [
    "## Parallelizing a Simple Loop with `multiprocessing.Pool`\n",
    "\n",
    "### Understanding `multiprocessing.Pool`\n",
    "The `multiprocessing.Pool` class is a powerful tool in Python's multiprocessing module that simplifies the process of distributing your work among multiple worker processes. This allows for parallel processing on multi-core machines which can lead to significant reductions in execution time, especially for CPU-bound tasks.\n",
    "\n",
    "### How Does a Pool Work?\n",
    "A `Pool` manages a number of worker processes and distributes tasks to them. When using a `Pool`, you don’t need to manage the worker processes yourself. Instead, you just specify the number of workers, and the pool automatically handles the task distribution, execution, and collection of results.\n",
    "\n",
    "### Use Case: Parallelizing Loops\n",
    "Often in programming, you encounter loops where each iteration is independent of the others. These are perfect candidates for parallel processing. By distributing iterations across multiple processes, you can complete the entire loop significantly faster than executing it serially.\n",
    "\n",
    "### Example: Computing Squares\n",
    "Consider a simple task where you need to compute the square of each number in a list. Serially, this would involve processing each number one after the other. In parallel, however, we can distribute these numbers across multiple processes, each calculating the square independently, thus completing the task more quickly.\n",
    "\n",
    "### Advantages of Using a Pool\n",
    "- **Efficiency**: Utilizes all available CPU cores, reducing overall processing time.\n",
    "- **Simplicity**: The API is straightforward, abstracting much of the complexity involved in process management.\n",
    "- **Flexibility**: Offers various ways to distribute tasks (e.g., `map`, `apply`, `starmap`).\n",
    "\n",
    "### Practical Example\n",
    "We will demonstrate this with a Python script that uses a `Pool` to compute the squares of numbers in a list in parallel. This example will help illustrate the reduction in execution time and the effective use of system resources.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "945a262a-8dd3-45fe-9b3c-ab61960aed72",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Squares: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n"
     ]
    }
   ],
   "source": [
    "from multiprocessing import Pool\n",
    "\n",
    "# Function to compute the square of a number\n",
    "def compute_square(num):\n",
    "    return num * num\n",
    "\n",
    "# Main execution block\n",
    "if __name__ == \"__main__\":\n",
    "    numbers = list(range(10))  # List of numbers from 0 to 9\n",
    "    \n",
    "    # Creating a pool of 4 worker processes\n",
    "    with Pool(4) as pool:\n",
    "        # Mapping 'compute_square' function to the numbers list\n",
    "        squares = pool.map(compute_square, numbers)\n",
    "    \n",
    "    # Printing the results\n",
    "    print(\"Squares:\", squares)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f42efd-70b3-4828-a79b-5aaaca0da617",
   "metadata": {},
   "source": [
    "## Parallel Execution with `concurrent.futures`\n",
    "\n",
    "### Introduction to `concurrent.futures`\n",
    "The `concurrent.futures` module provides a high-level interface for asynchronously executing callables. Introduced in Python 3.2, it abstracts away many of the complexities involved in directly managing threads or processes. The module includes `ThreadPoolExecutor` and `ProcessPoolExecutor` which encapsulate thread-based and process-based parallel execution, respectively.\n",
    "\n",
    "### Why Use `concurrent.futures`?\n",
    "The `concurrent.futures` module simplifies parallel execution by managing a pool of threads or processes, handling task submission, and returning futures. Futures represent the result of a computation that may not be complete yet, allowing the execution to continue without blocking.\n",
    "\n",
    "### Advantages of `ProcessPoolExecutor`\n",
    "- **Ease of Use**: The API simplifies running tasks in parallel and is easy to integrate into existing code.\n",
    "- **Flexibility**: Allows specifying the number of worker processes, letting the system allocate resources efficiently.\n",
    "- **Asynchronous Execution**: Returns future objects, enabling asynchronous programming patterns and non-blocking calls.\n",
    "\n",
    "### Use Case: Calculating Squares in Parallel\n",
    "A common use case for parallel processing is the independent computation of results from a list of inputs. Here, we will demonstrate using `ProcessPoolExecutor` to calculate the squares of numbers in a list. This example illustrates the ease of setup and potential speed improvements when using this method for CPU-intensive tasks.\n",
    "\n",
    "### Practical Example\n",
    "Next, we will provide a Python script using `ProcessPoolExecutor` to demonstrate how straightforward and powerful this tool can be for parallelizing a simple loop.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "9c17f927-3284-4c27-9989-fc99b36808eb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Squares: [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]\n"
     ]
    }
   ],
   "source": [
    "from concurrent.futures import ProcessPoolExecutor\n",
    "\n",
    "# Function to compute the square of a number\n",
    "def compute_square(num):\n",
    "    return num * num\n",
    "\n",
    "# Main execution block\n",
    "if __name__ == \"__main__\":\n",
    "    numbers = list(range(10))  # List of numbers from 0 to 9\n",
    "    \n",
    "    # Creating a ProcessPoolExecutor with 4 worker processes\n",
    "    with ProcessPoolExecutor(max_workers=4) as executor:\n",
    "        # Using executor.map to apply 'compute_square' function across the numbers list in parallel\n",
    "        squares = list(executor.map(compute_square, numbers))\n",
    "    \n",
    "    # Printing the results\n",
    "    print(\"Squares:\", squares)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b389404-717c-4c0b-b538-90078f8b4adf",
   "metadata": {},
   "source": [
    "## Case Study: Parallel Matrix Multiplication\n",
    "\n",
    "### Significance of Matrix Operations in Scientific Computing\n",
    "Matrix operations are fundamental to many scientific computations, including physics simulations, statistical analysis, and engineering calculations. These operations, especially matrix multiplication, are computationally intensive and often constitute the bottleneck in performance for algorithms in fields such as machine learning and numerical simulation.\n",
    "\n",
    "### Suitability for Parallel Processing\n",
    "Matrix multiplication can be effectively decomposed into smaller, independent computations, making it an ideal candidate for parallel processing. Since each element of the product matrix can be calculated independently of the others, parallel algorithms can distribute these calculations across multiple processors. This distribution significantly speeds up the computation as it leverages the computational power of multiple cores simultaneously.\n",
    "\n",
    "### Benefits of Parallel Matrix Multiplication\n",
    "- **Speed**: Parallel processing can drastically reduce computation time, which is crucial for handling large datasets or real-time processing.\n",
    "- **Efficiency**: Utilizing multiple cores or processors allows for more efficient use of hardware resources.\n",
    "- **Scalability**: As matrix size grows, parallel processing becomes increasingly advantageous, offering better scalability compared to serial computations.\n",
    "\n",
    "### Practical Implementation\n",
    "In the following section, we will explore both serial and parallel implementations of matrix multiplication using Python's built-in list data structures and the `multiprocessing` module to facilitate parallel computation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "430fad7b-9037-4270-b7e4-cadc5a6f1f95",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Resultant Matrix C (Serial): [[22.364729762344897, 22.99709314398664, 23.802940317496663, 23.84274360505822, 27.066650746588124, 26.41989608714333, 27.204964690696176, 24.936479934409256, 23.478729772510537, 24.961039935838254, 25.217552300467347, 22.296428639914232, 22.827204353371492, 22.57986439845265, 24.39177622969138, 26.25730202307473, 23.446585828569866, 29.16737536123621, 25.211074085972296, 26.19196396609341, 23.906807430113517, 24.742044321649445, 27.720114326323316, 24.571376198289432, 25.271399385727204, 28.123694086511087, 24.999834232127057, 26.35761351882074, 26.37226691828164, 22.095353218398305, 25.731704890020954, 27.98716840689283, 24.757515098959676, 25.29118325538527, 24.71499681875145, 25.863803158006945, 24.978375822148244, 25.167316746358566, 27.211661389593623, 23.157603642872377, 26.55409573628232, 26.0099974093898, 24.219091111367092, 25.29159110350913, 26.70641620719848, 25.33009857974611, 24.851541630730082, 27.153999697457003, 24.860524198424045, 23.653039727176495, 27.546368037773416, 27.13182659734745, 26.433259039808412, 25.24450595191513, 22.50647809345529, 25.467672033751604, 26.149042724994448, 27.27475277592002, 25.88433663752963, 28.59472184475393, 26.488199989830434, 27.793982676142548, 25.518914397542954, 22.16746121280962, 27.095836925718757, 22.237334431808083, 23.2150150781016, 26.682750916609237, 24.45344139279563, 24.06706855878724, 24.22759228074155, 25.971575210664, 26.060903796739716, 26.27070964008755, 23.851589818654915, 27.83379209939381, 26.10083801398121, 23.39321338394554, 25.70294268592262, 25.383638430673177, 26.531587270287623, 25.39233928118121, 26.039521416018925, 24.5653214652279, 24.30382214946832, 26.037635698873967, 24.169269230361323, 23.66035963353419, 25.346001752453756, 25.689542243514218, 25.78627267031202, 24.490664369023126, 26.110627219600495, 26.03821325811471, 25.804947159113368, 24.115886088716273, 26.331889309194285, 24.99375412559767, 21.873322791500755, 24.502314664107914], [20.893506069368208, 22.085859224277613, 23.45357028946027, 22.297639038688676, 26.27946763980254, 25.363134778314876, 23.69956816560993, 23.1602202488274, 22.808571864688243, 24.16277831627676, 23.850353546057146, 22.477651831513178, 20.364208948225897, 22.995615575869312, 23.277165626643942, 25.1485149818223, 21.84757506396078, 25.80731562979997, 23.13499222125475, 22.90099724860501, 21.157540005545854, 22.61792176524935, 24.534305763421685, 21.179674472832502, 22.48988731924208, 25.15811587376943, 21.64194017816883, 23.554484371363024, 23.742940080411845, 21.581376130383298, 25.193957686893384, 25.333748216809354, 20.928519152925645, 23.388277602967406, 25.05962117396302, 25.227409022328324, 23.262361909806103, 26.143727334704064, 25.23157619801403, 21.68694786080875, 22.18284875218181, 24.603861116063054, 23.071331513754227, 21.27664182147719, 24.456838118428305, 22.163451943136586, 22.16855326572819, 25.27443979575212, 23.485178117777313, 22.024340832643865, 23.688798832647294, 23.920167334390644, 23.528019927550723, 23.048872167070897, 20.199160232717045, 21.48587795936386, 25.11032612514378, 25.107727707736203, 22.364242626202575, 23.801444186393336, 21.440548337063646, 26.32503447118059, 22.74740586879196, 20.09455084766138, 23.417519935377896, 23.105694862018854, 20.38611257860953, 23.33970057421841, 22.919140589096564, 21.064102279412012, 21.88714192778885, 25.11480736301967, 24.218661473139942, 22.566861169545696, 22.60444330103761, 25.46892224117483, 26.081725039025372, 21.045403432198505, 19.93640372137657, 20.865741750235966, 25.05258731845121, 22.22781911897536, 24.805779057904854, 23.231726942482453, 22.62191119362238, 23.290598640445086, 22.930265975350732, 23.34553424413579, 23.415952580024317, 24.52884262709595, 22.285286177700442, 21.43950018484995, 22.76581857149364, 24.32713058599163, 22.651993307402762, 23.619090478047017, 23.572831887052672, 23.4689971262367, 19.65252365035124, 22.007973298837765], [24.009214726861362, 23.126692240762765, 26.767876393022934, 24.026331998694737, 27.26454131355984, 27.699044121621704, 27.09640719641684, 25.481828388878373, 24.804395585343407, 25.2314622675907, 26.35252136099373, 23.420438934269956, 23.828547270933004, 23.772823543355667, 25.013718779528578, 25.560152609965094, 24.66957349322401, 28.502438396945866, 26.839235401930775, 24.874954475758354, 24.034622702302357, 23.594535780664273, 25.038148128083417, 24.33782248473043, 23.8285646642467, 26.149270573212576, 25.968687818716386, 24.989512344145087, 26.53959484606202, 23.866802699982046, 24.895170018224423, 27.501635988299057, 24.249389027930103, 25.15674429407624, 26.49287179691116, 25.426265458624638, 24.99219260000033, 27.220394378557195, 27.525793496853925, 23.508559171318833, 26.38060145945558, 24.956043860668277, 25.136885801698686, 23.82200212780723, 26.823797642685182, 25.811078396976043, 24.741929883902163, 25.684534449022568, 25.227576539032246, 23.23720887724385, 25.88398937440856, 26.925797196406016, 26.074865439966786, 23.256163613448045, 23.337568645435642, 24.764052625792043, 27.78232552633416, 27.667673606591865, 24.106835043392195, 25.923970368826463, 24.5894498736145, 27.828131046322927, 23.889949647372973, 22.606757151001627, 24.962805997482235, 24.140035417659526, 22.1175917734775, 25.547975076182148, 24.67806088861391, 22.977276165296175, 22.1595764721894, 27.313348691761067, 28.197499120606704, 25.34007322378886, 24.1592400460872, 26.24342485661406, 27.30451917379691, 23.782160863771384, 24.74070441393281, 22.996103215941517, 25.297562443813046, 23.663353109833597, 26.073393058217928, 23.99491705761156, 23.18261148884636, 24.66744790322543, 23.034402745047956, 26.297898190081302, 25.321127839299834, 27.200047915334036, 24.28169774835816, 24.723056296432528, 25.217129619014763, 26.81335391558027, 24.43883653459414, 25.844659712382306, 24.635102688260922, 26.485869661981912, 22.02992959443731, 24.21744942626045], [21.35162173609271, 21.75798793766727, 25.428092862978875, 24.020656046543603, 27.627952901453153, 27.44290389086196, 27.10223791317261, 26.372063293593364, 23.497976175333562, 25.661742563109335, 26.227100698340372, 23.167119058038296, 23.937059960490483, 24.460774980087322, 25.446631651876075, 25.700901460307833, 24.120449485262824, 27.684349920715547, 25.029423888158238, 25.6370716919021, 24.872996822604534, 25.20573962834329, 26.91904658733459, 24.51496982981083, 24.391447383121147, 25.665035658860408, 24.734990934948367, 24.752765195248493, 26.45001776652225, 22.594882785406757, 24.58971666263992, 26.756494216365674, 22.36462206342414, 25.791348251039263, 24.913713737443064, 26.22994830562678, 25.233906081871748, 27.477199143861956, 27.219377929505693, 25.02950962075609, 27.555820277020725, 25.205295694769987, 24.73270808868377, 24.646494591082615, 26.15444773943101, 24.449265987129547, 25.710541415019247, 26.67683056668333, 24.600017208433115, 23.095705541566844, 26.241752347821254, 28.083876960611576, 26.528817751518808, 25.091690060535523, 21.42122235922848, 24.89087182328697, 25.75174108733417, 26.608854205483823, 25.954342020143546, 27.008311915749875, 25.755937785261352, 26.910663309953588, 22.650262073035996, 21.63030809498501, 25.723498574531863, 23.072936434603726, 25.01167586854695, 29.2761762943496, 23.66737670089168, 23.908599407661995, 23.144770268372426, 26.802166703827343, 27.81206448479574, 25.372747812979917, 25.231334368841317, 26.514610210751783, 26.17374249238733, 23.49270400464096, 23.43114480753412, 22.11894106910688, 26.439158775043484, 24.710103129317698, 26.39701902076296, 24.642438501856095, 24.505918299150174, 25.209767538844105, 23.90348439427675, 25.5510888943869, 26.769527043739426, 26.179304776476013, 26.722604548532704, 23.322977593193425, 24.52083201755463, 27.596006309166818, 25.107003319962168, 24.017505325094316, 26.164520132344624, 25.22793219280155, 20.486429057687463, 23.943046768716716], [21.97310570484223, 23.21154021113844, 25.19231308717936, 22.747726211242178, 29.026068343620796, 25.835885307017936, 26.618553996957115, 25.789010930510837, 25.743887500359033, 24.971028428216584, 24.9775723583545, 24.66683080111292, 22.494360929830968, 24.009928712950764, 23.944138289837618, 24.700089946053787, 23.53274058044855, 26.897807232964205, 25.559748245323817, 25.3568167075872, 23.056778576752425, 25.753488170899043, 26.60219652917825, 23.7784119013919, 23.799324055404124, 26.886580786185995, 25.463522803294865, 24.669603678288237, 25.13292278572479, 22.181317377276812, 24.751719043097435, 26.595475963312705, 23.104591380715988, 25.9647907798369, 24.32336853855053, 23.98352401243772, 23.716310455145596, 27.30877715532817, 27.994849267953278, 23.44682350295896, 24.170435250752494, 24.326864235464747, 25.125390731443677, 25.39057512663605, 26.599029840219284, 23.96311886359064, 24.708477964473143, 27.26208711951887, 23.49209948635553, 24.979438430961487, 24.671044918841396, 26.61316032166851, 26.198768609601576, 25.075260525556374, 20.439001003677088, 23.35743736273953, 27.373285908987373, 25.970289957705685, 25.139273607874166, 26.375074639270835, 25.324936479473752, 27.7634980158533, 22.735783232218477, 22.399458681111877, 24.959979800335052, 23.182460851628825, 23.224620168708714, 25.73161277545592, 23.08181684165142, 23.718133619409237, 22.142235614505022, 27.35013461897383, 27.421079218250426, 24.599123423907933, 23.563874401687386, 27.456162161997188, 28.333651492971406, 22.277297086184085, 25.115367778175695, 23.595764667039717, 26.841443602726002, 25.163508464747245, 26.04325313415067, 24.963517605121446, 22.755691766249594, 24.725269841853557, 23.42473864945184, 25.341818099594835, 25.12304406289895, 26.914275605855725, 25.296581146282385, 23.00378837635525, 24.70172002918129, 26.819071406525605, 25.295540533936382, 24.427922745018318, 25.463474834385842, 24.89058665070294, 19.902349990361333, 22.90335055186402]]\n",
      "Resultant Matrix C (Parallel): [[22.364729762344897, 22.99709314398664, 23.802940317496663, 23.84274360505822, 27.066650746588124, 26.41989608714333, 27.204964690696176, 24.936479934409256, 23.478729772510537, 24.961039935838254, 25.217552300467347, 22.296428639914232, 22.827204353371492, 22.57986439845265, 24.39177622969138, 26.25730202307473, 23.446585828569866, 29.16737536123621, 25.211074085972296, 26.19196396609341, 23.906807430113517, 24.742044321649445, 27.720114326323316, 24.571376198289432, 25.271399385727204, 28.123694086511087, 24.999834232127057, 26.35761351882074, 26.37226691828164, 22.095353218398305, 25.731704890020954, 27.98716840689283, 24.757515098959676, 25.29118325538527, 24.71499681875145, 25.863803158006945, 24.978375822148244, 25.167316746358566, 27.211661389593623, 23.157603642872377, 26.55409573628232, 26.0099974093898, 24.219091111367092, 25.29159110350913, 26.70641620719848, 25.33009857974611, 24.851541630730082, 27.153999697457003, 24.860524198424045, 23.653039727176495, 27.546368037773416, 27.13182659734745, 26.433259039808412, 25.24450595191513, 22.50647809345529, 25.467672033751604, 26.149042724994448, 27.27475277592002, 25.88433663752963, 28.59472184475393, 26.488199989830434, 27.793982676142548, 25.518914397542954, 22.16746121280962, 27.095836925718757, 22.237334431808083, 23.2150150781016, 26.682750916609237, 24.45344139279563, 24.06706855878724, 24.22759228074155, 25.971575210664, 26.060903796739716, 26.27070964008755, 23.851589818654915, 27.83379209939381, 26.10083801398121, 23.39321338394554, 25.70294268592262, 25.383638430673177, 26.531587270287623, 25.39233928118121, 26.039521416018925, 24.5653214652279, 24.30382214946832, 26.037635698873967, 24.169269230361323, 23.66035963353419, 25.346001752453756, 25.689542243514218, 25.78627267031202, 24.490664369023126, 26.110627219600495, 26.03821325811471, 25.804947159113368, 24.115886088716273, 26.331889309194285, 24.99375412559767, 21.873322791500755, 24.502314664107914], [20.893506069368208, 22.085859224277613, 23.45357028946027, 22.297639038688676, 26.27946763980254, 25.363134778314876, 23.69956816560993, 23.1602202488274, 22.808571864688243, 24.16277831627676, 23.850353546057146, 22.477651831513178, 20.364208948225897, 22.995615575869312, 23.277165626643942, 25.1485149818223, 21.84757506396078, 25.80731562979997, 23.13499222125475, 22.90099724860501, 21.157540005545854, 22.61792176524935, 24.534305763421685, 21.179674472832502, 22.48988731924208, 25.15811587376943, 21.64194017816883, 23.554484371363024, 23.742940080411845, 21.581376130383298, 25.193957686893384, 25.333748216809354, 20.928519152925645, 23.388277602967406, 25.05962117396302, 25.227409022328324, 23.262361909806103, 26.143727334704064, 25.23157619801403, 21.68694786080875, 22.18284875218181, 24.603861116063054, 23.071331513754227, 21.27664182147719, 24.456838118428305, 22.163451943136586, 22.16855326572819, 25.27443979575212, 23.485178117777313, 22.024340832643865, 23.688798832647294, 23.920167334390644, 23.528019927550723, 23.048872167070897, 20.199160232717045, 21.48587795936386, 25.11032612514378, 25.107727707736203, 22.364242626202575, 23.801444186393336, 21.440548337063646, 26.32503447118059, 22.74740586879196, 20.09455084766138, 23.417519935377896, 23.105694862018854, 20.38611257860953, 23.33970057421841, 22.919140589096564, 21.064102279412012, 21.88714192778885, 25.11480736301967, 24.218661473139942, 22.566861169545696, 22.60444330103761, 25.46892224117483, 26.081725039025372, 21.045403432198505, 19.93640372137657, 20.865741750235966, 25.05258731845121, 22.22781911897536, 24.805779057904854, 23.231726942482453, 22.62191119362238, 23.290598640445086, 22.930265975350732, 23.34553424413579, 23.415952580024317, 24.52884262709595, 22.285286177700442, 21.43950018484995, 22.76581857149364, 24.32713058599163, 22.651993307402762, 23.619090478047017, 23.572831887052672, 23.4689971262367, 19.65252365035124, 22.007973298837765], [24.009214726861362, 23.126692240762765, 26.767876393022934, 24.026331998694737, 27.26454131355984, 27.699044121621704, 27.09640719641684, 25.481828388878373, 24.804395585343407, 25.2314622675907, 26.35252136099373, 23.420438934269956, 23.828547270933004, 23.772823543355667, 25.013718779528578, 25.560152609965094, 24.66957349322401, 28.502438396945866, 26.839235401930775, 24.874954475758354, 24.034622702302357, 23.594535780664273, 25.038148128083417, 24.33782248473043, 23.8285646642467, 26.149270573212576, 25.968687818716386, 24.989512344145087, 26.53959484606202, 23.866802699982046, 24.895170018224423, 27.501635988299057, 24.249389027930103, 25.15674429407624, 26.49287179691116, 25.426265458624638, 24.99219260000033, 27.220394378557195, 27.525793496853925, 23.508559171318833, 26.38060145945558, 24.956043860668277, 25.136885801698686, 23.82200212780723, 26.823797642685182, 25.811078396976043, 24.741929883902163, 25.684534449022568, 25.227576539032246, 23.23720887724385, 25.88398937440856, 26.925797196406016, 26.074865439966786, 23.256163613448045, 23.337568645435642, 24.764052625792043, 27.78232552633416, 27.667673606591865, 24.106835043392195, 25.923970368826463, 24.5894498736145, 27.828131046322927, 23.889949647372973, 22.606757151001627, 24.962805997482235, 24.140035417659526, 22.1175917734775, 25.547975076182148, 24.67806088861391, 22.977276165296175, 22.1595764721894, 27.313348691761067, 28.197499120606704, 25.34007322378886, 24.1592400460872, 26.24342485661406, 27.30451917379691, 23.782160863771384, 24.74070441393281, 22.996103215941517, 25.297562443813046, 23.663353109833597, 26.073393058217928, 23.99491705761156, 23.18261148884636, 24.66744790322543, 23.034402745047956, 26.297898190081302, 25.321127839299834, 27.200047915334036, 24.28169774835816, 24.723056296432528, 25.217129619014763, 26.81335391558027, 24.43883653459414, 25.844659712382306, 24.635102688260922, 26.485869661981912, 22.02992959443731, 24.21744942626045], [21.35162173609271, 21.75798793766727, 25.428092862978875, 24.020656046543603, 27.627952901453153, 27.44290389086196, 27.10223791317261, 26.372063293593364, 23.497976175333562, 25.661742563109335, 26.227100698340372, 23.167119058038296, 23.937059960490483, 24.460774980087322, 25.446631651876075, 25.700901460307833, 24.120449485262824, 27.684349920715547, 25.029423888158238, 25.6370716919021, 24.872996822604534, 25.20573962834329, 26.91904658733459, 24.51496982981083, 24.391447383121147, 25.665035658860408, 24.734990934948367, 24.752765195248493, 26.45001776652225, 22.594882785406757, 24.58971666263992, 26.756494216365674, 22.36462206342414, 25.791348251039263, 24.913713737443064, 26.22994830562678, 25.233906081871748, 27.477199143861956, 27.219377929505693, 25.02950962075609, 27.555820277020725, 25.205295694769987, 24.73270808868377, 24.646494591082615, 26.15444773943101, 24.449265987129547, 25.710541415019247, 26.67683056668333, 24.600017208433115, 23.095705541566844, 26.241752347821254, 28.083876960611576, 26.528817751518808, 25.091690060535523, 21.42122235922848, 24.89087182328697, 25.75174108733417, 26.608854205483823, 25.954342020143546, 27.008311915749875, 25.755937785261352, 26.910663309953588, 22.650262073035996, 21.63030809498501, 25.723498574531863, 23.072936434603726, 25.01167586854695, 29.2761762943496, 23.66737670089168, 23.908599407661995, 23.144770268372426, 26.802166703827343, 27.81206448479574, 25.372747812979917, 25.231334368841317, 26.514610210751783, 26.17374249238733, 23.49270400464096, 23.43114480753412, 22.11894106910688, 26.439158775043484, 24.710103129317698, 26.39701902076296, 24.642438501856095, 24.505918299150174, 25.209767538844105, 23.90348439427675, 25.5510888943869, 26.769527043739426, 26.179304776476013, 26.722604548532704, 23.322977593193425, 24.52083201755463, 27.596006309166818, 25.107003319962168, 24.017505325094316, 26.164520132344624, 25.22793219280155, 20.486429057687463, 23.943046768716716], [21.97310570484223, 23.21154021113844, 25.19231308717936, 22.747726211242178, 29.026068343620796, 25.835885307017936, 26.618553996957115, 25.789010930510837, 25.743887500359033, 24.971028428216584, 24.9775723583545, 24.66683080111292, 22.494360929830968, 24.009928712950764, 23.944138289837618, 24.700089946053787, 23.53274058044855, 26.897807232964205, 25.559748245323817, 25.3568167075872, 23.056778576752425, 25.753488170899043, 26.60219652917825, 23.7784119013919, 23.799324055404124, 26.886580786185995, 25.463522803294865, 24.669603678288237, 25.13292278572479, 22.181317377276812, 24.751719043097435, 26.595475963312705, 23.104591380715988, 25.9647907798369, 24.32336853855053, 23.98352401243772, 23.716310455145596, 27.30877715532817, 27.994849267953278, 23.44682350295896, 24.170435250752494, 24.326864235464747, 25.125390731443677, 25.39057512663605, 26.599029840219284, 23.96311886359064, 24.708477964473143, 27.26208711951887, 23.49209948635553, 24.979438430961487, 24.671044918841396, 26.61316032166851, 26.198768609601576, 25.075260525556374, 20.439001003677088, 23.35743736273953, 27.373285908987373, 25.970289957705685, 25.139273607874166, 26.375074639270835, 25.324936479473752, 27.7634980158533, 22.735783232218477, 22.399458681111877, 24.959979800335052, 23.182460851628825, 23.224620168708714, 25.73161277545592, 23.08181684165142, 23.718133619409237, 22.142235614505022, 27.35013461897383, 27.421079218250426, 24.599123423907933, 23.563874401687386, 27.456162161997188, 28.333651492971406, 22.277297086184085, 25.115367778175695, 23.595764667039717, 26.841443602726002, 25.163508464747245, 26.04325313415067, 24.963517605121446, 22.755691766249594, 24.725269841853557, 23.42473864945184, 25.341818099594835, 25.12304406289895, 26.914275605855725, 25.296581146282385, 23.00378837635525, 24.70172002918129, 26.819071406525605, 25.295540533936382, 24.427922745018318, 25.463474834385842, 24.89058665070294, 19.902349990361333, 22.90335055186402]]\n"
     ]
    }
   ],
   "source": [
    "from multiprocessing import Pool\n",
    "\n",
    "# Function for serial matrix multiplication using basic Python lists\n",
    "def matrix_multiply(A, B):\n",
    "    # Get the dimensions of A and B\n",
    "    rows_A = len(A)\n",
    "    cols_A = len(A[0])\n",
    "    cols_B = len(B[0])\n",
    "\n",
    "    # Initialize the result matrix with zeros\n",
    "    result = [[0 for _ in range(cols_B)] for _ in range(rows_A)]\n",
    "\n",
    "    # Perform matrix multiplication\n",
    "    for i in range(rows_A):\n",
    "        for j in range(cols_B):\n",
    "            for k in range(cols_A):\n",
    "                result[i][j] += A[i][k] * B[k][j]\n",
    "    \n",
    "    return result\n",
    "\n",
    "# Function to multiply a matrix chunk by another matrix\n",
    "def multiply_chunk(args):\n",
    "    A_chunk, B = args\n",
    "    # Get the dimensions\n",
    "    rows_A_chunk = len(A_chunk)\n",
    "    cols_A_chunk = len(A_chunk[0])\n",
    "    cols_B = len(B[0])\n",
    "\n",
    "    # Initialize the result chunk with zeros\n",
    "    result_chunk = [[0 for _ in range(cols_B)] for _ in range(rows_A_chunk)]\n",
    "\n",
    "    # Perform matrix multiplication for the chunk\n",
    "    for i in range(rows_A_chunk):\n",
    "        for j in range(cols_B):\n",
    "            for k in range(cols_A_chunk):\n",
    "                result_chunk[i][j] += A_chunk[i][k] * B[k][j]\n",
    "\n",
    "    return result_chunk\n",
    "\n",
    "# Function for parallel matrix multiplication\n",
    "def parallel_matrix_multiply(A, B, n_processes):\n",
    "    chunk_size = len(A) // n_processes  # Determine the size of each chunk\n",
    "    chunks = [(A[i:i + chunk_size], B) for i in range(0, len(A), chunk_size)]\n",
    "    \n",
    "    # Create a pool of worker processes\n",
    "    with Pool(n_processes) as pool:\n",
    "        result_chunks = pool.map(multiply_chunk, chunks)\n",
    "    \n",
    "    # Combine the chunks back into a full result matrix\n",
    "    result = [row for chunk in result_chunks for row in chunk]\n",
    "    \n",
    "    return result\n",
    "\n",
    "# Helper function to create a random matrix of given size\n",
    "import random\n",
    "\n",
    "def create_random_matrix(rows, cols):\n",
    "    return [[random.random() for _ in range(cols)] for _ in range(rows)]\n",
    "\n",
    "# Creating two random matrices of size 100x100\n",
    "A = create_random_matrix(100, 100)\n",
    "B = create_random_matrix(100, 100)\n",
    "\n",
    "# Performing serial and parallel matrix multiplication\n",
    "C_serial = matrix_multiply(A, B)\n",
    "C_parallel = parallel_matrix_multiply(A, B, 4)\n",
    "\n",
    "# Output results (truncated for readability)\n",
    "print(\"Resultant Matrix C (Serial):\", C_serial[:5][:5])  # Print first 5 rows/columns\n",
    "print(\"Resultant Matrix C (Parallel):\", C_parallel[:5][:5])  # Print first 5 rows/columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a0149a8-dfad-4f6b-abf1-a824abea625d",
   "metadata": {},
   "source": [
    "## Advanced Topics in Parallel Programming\n",
    "\n",
    "### Scalability Considerations\n",
    "Scalability in parallel programming refers to the ability of a process or system to handle a growing amount of work or its potential to be enlarged to accommodate that growth. When developing parallel applications, it's crucial to design systems that can scale efficiently as the number of processors or tasks increases. Key considerations include:\n",
    "- **Load Balancing**: Distributing work evenly across all processors to avoid scenarios where some nodes are idle while others are overloaded.\n",
    "- **Overhead Management**: Keeping the communication and synchronization overhead to a minimum as the system scales up.\n",
    "\n",
    "### Understanding Deadlocks and Race Conditions\n",
    "- **Deadlocks**: A deadlock occurs when two or more processes are each waiting for the other to release a resource they need to continue execution. This situation results in a standstill where none of the processes can proceed.\n",
    "- **Race Conditions**: A race condition happens when multiple processes or threads manipulate shared data concurrently. The final value of the shared data depends on which process/thread completes last, leading to unpredictable results if not properly managed.\n",
    "\n",
    "### Synchronization Issues\n",
    "Synchronization is critical in parallel programming to ensure that multiple processes or threads can operate safely when sharing resources or data. Proper synchronization can prevent race conditions and ensure data integrity. Common synchronization mechanisms include:\n",
    "- **Locks**: Allow only one thread to access a resource at a time.\n",
    "- **Semaphores**: A more flexible mechanism that uses counters to control access to one or more shared resources.\n",
    "\n",
    "### Practical Example: Using Locks to Handle Race Conditions\n",
    "To demonstrate the importance of synchronization, we'll use a Python example where multiple processes increment a shared counter. Without proper synchronization, the final count could be incorrect due to race conditions. We'll use a `Lock` to ensure that only one process can increment the counter at a time.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9a0965b8-a5ca-4984-a75c-03cb72d2fd62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Final Value: 400\n"
     ]
    }
   ],
   "source": [
    "from multiprocessing import Process, Lock, Value\n",
    "\n",
    "# Function that increments a shared counter\n",
    "def increment(shared_value, lock):\n",
    "    with lock:\n",
    "        # Critical section: only one process can execute this block at a time\n",
    "        for _ in range(100):\n",
    "            shared_value.value += 1\n",
    "\n",
    "# Main block to set up and run processes\n",
    "if __name__ == \"__main__\":\n",
    "    # Shared value that all processes will increment\n",
    "    shared_value = Value('i', 0)\n",
    "    \n",
    "    # Lock to synchronize access to the shared value\n",
    "    lock = Lock()\n",
    "    \n",
    "    # List of processes that will increment the shared value\n",
    "    processes = [Process(target=increment, args=(shared_value, lock)) for _ in range(4)]\n",
    "    \n",
    "    # Start all processes\n",
    "    for p in processes:\n",
    "        p.start()\n",
    "    \n",
    "    # Wait for all processes to finish\n",
    "    for p in processes:\n",
    "        p.join()\n",
    "    \n",
    "    # Output the final value of the shared counter\n",
    "    print(\"Final Value:\", shared_value.value)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "017972e7-3460-4389-898d-345c85c3546f",
   "metadata": {},
   "source": [
    "### End of the practice."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a164834-6e68-421c-b610-d47915d8c92a",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}