{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "d700a42a-7611-4726-a3c2-6b788682dfab", "metadata": { "tags": [ "hide-input" ] }, "outputs": [], "source": [ "# Install the necessary dependencies\n", "\n", "import os\n", "import sys\n", "!{sys.executable} -m pip install --quiet seaborn pandas scikit-learn numpy matplotlib jupyterlab_myst ipython" ] }, { "cell_type": "markdown", "id": "5357e111", "metadata": { "tags": [ "remove-cell" ] }, "source": [ "---\n", "license:\n", " code: MIT\n", " content: CC-BY-4.0\n", "github: https://github.com/ocademy-ai/machine-learning\n", "venue: By Ocademy\n", "open_access: true\n", "bibliography:\n", " - https://raw.githubusercontent.com/ocademy-ai/machine-learning/main/open-machine-learning-jupyter-book/references.bib\n", "---" ] }, { "cell_type": "markdown", "id": "6195d6f2", "metadata": {}, "source": [ "# Overview\n", "\n", "Moving Machine Learning models into production is as important as building them, sometimes even harder. Maintaining data quality and model accuracy over time are just a few of the challenges. To achieve end-to-end system productionization as a whole, the various components and designs need to be identified, from defining a problem to serving the model as a service.\n", "\n", "According to the Algorithmia statistics, 55% of businesses working on ML models have yet to get them into production. " ] }, { "cell_type": "markdown", "id": "b13ac488", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/the%202020%20State%20of%20Enterprise%20ML%20by%20Algorithmia%20based%20on%20750%20businesses.png\n", "---\n", "name: the 2020 State of Enterprise ML by Algorithmia based on 750 businesses\n", "---\n", "the 2020 State of Enterprise ML by Algorithmia based on 750 businesses\n", ":::" ] }, { "cell_type": "markdown", "id": "294cbab7", "metadata": {}, "source": [ "This is because there are many problems and challenges between the theoretical study of the model and the actual production deployment:\n", "\n", "(1)**POC** to production gap:\n", "\n", "There is a huge gap from Proof of Concept (POC) to actual final product or service deployment in production, with only a tiny fraction of the complete machine learning service model actually invested consisting of ML code, and the surrounding infrastructure required for this is large and complex. At the same time, this gap may also involve challenges in technology, resources, security, stability and other aspects.\n" ] }, { "cell_type": "markdown", "id": "2bd5ba4a", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/the%20portion%20of%20ML%20code%20that%20is%20part%20of%20a%20complete%20ML%20system.jpg\n", "---\n", "name: the portion of ML code that is part of a complete ML system\n", "---\n", "the portion of ML code that is part of a complete ML system\n", ":::" ] }, { "cell_type": "markdown", "id": "92c348bc", "metadata": {}, "source": [ "(2) data drift and concept drift:\n", "\n", "Models do not last forever and sometimes degrade over time, even if the data itself is of good quality. Sometimes, model performance degrades due to data quality degradation:\n", "\n", "**Data drift** usually means that the variable distribution of the input data (**x**) changes, and the trained model is not related to this new data, so the performance will decline. For example, an e-commerce platform sets up a predictive model to predict the purchase possibility of users to push personalized offers, but at the beginning, the training and application of the model are based on the user data of spontaneous paid search. When the e-commerce platform launches a new advertising campaign, the users attracted by the new influx of advertisements do not adapt to the model previously analyzed.\n" ] }, { "cell_type": "markdown", "id": "23a0c31b", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/Continuous%20blue%20squares%20in%20the%20Data%20stream%20indicate%20the%20start%20of%20a%20Data%20Drift.jpg\n", "---\n", "name: Continuous blue squares in the Data stream indicate the start of a Data Drift\n", "---\n", "Continuous blue squares in the Data stream indicate the start of a Data Drift\n", ":::\n" ] }, { "cell_type": "markdown", "id": "fb685489", "metadata": {}, "source": [ "**Concept drift** usually means that the mapping between input and output changes (**x->y**), the pattern learned by the model is no longer valid, and what changes is not the data itself, but the statistical properties of the target domain have changed over time, that is, the so-called \"world has changed\". Sometimes these changes happen very quickly or even unexpectedly, as in the case of the COVID-19 outbreak, the Black Swan event, which dramatically increases the demand for gowns and masks in response to changes in government policies; Sometimes it is a slow change, for example, customers' online shopping preferences change with changes in personal interests, merchants' reputation, and service types.\n", "\n", "These data changes will affect the performance of the model and cause serious problems in the actual project landing process, so the model needs to be monitored and continuously deployed." ] }, { "cell_type": "markdown", "id": "02964563", "metadata": {}, "source": [ "This chapter combines the foundational concepts of Machine Learning with the functional expertise of modern software development and engineering to help you develop production-ready Machine Learning knowledge.\n", "\n", "Productionization of a Machine Learning solution is not a one-time thing. It is always under improving one-time through the iterative process continuously." ] }, { "cell_type": "markdown", "id": "f9a75a94", "metadata": {}, "source": [ "> \"Machine learning is a highly iterative process: you may try many dozens of ideas before finding one that you're satisfied with.\"\n", "-- Andrew Ng" ] }, { "cell_type": "markdown", "id": "218ca90a", "metadata": {}, "source": [ "The Machine Learning lifecycle, also known as MLOps(Machine Learning Operations), could be mapped and fit into the traditional software development process. A better understanding of Machine Learning will help you as you think about how to incorporate machine learning, including models, into your software development processes.\n", "\n", "A Machine Learning lifecycle consists of such major phases, including:\n", "\n", "- problem framing,\n", "- data engineering,\n", "- model training & evaluation,\n", "- deployment,\n", "- maintenance." ] }, { "cell_type": "markdown", "id": "65943c92", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/machine-learning-lifecycle.svg\n", "---\n", "name: Machine Learning Lifecycle\n", "---\n", "Machine Learning Lifecycle\n", ":::" ] }, { "cell_type": "markdown", "id": "8ac7baf9", "metadata": {}, "source": [ "In the below sections, we will walk through the Machine Learning lifecycle components with a real-world example." ] }, { "cell_type": "markdown", "id": "358a61e7", "metadata": {}, "source": [ "## Problem framing\n", "\n", "To bring a Machine Learning solution to production successfully, the first step is to define a valuable business objective and translate the objective into a Machine Learning solvable problem.\n", "\n", "**[COVID-19](https://en.wikipedia.org/wiki/COVID-19) Projections** is an artificial intelligence solution to accurately forecast infections, deaths, and recovery timelines of the COVID-19/coronavirus pandemic in the US and globally. By the end of April 2020, it was cited by the Centers for Disease Control & Prevention (CDC) as one of the first models to “help inform public health decision-making”." ] }, { "cell_type": "markdown", "id": "0f3eaf89", "metadata": {}, "source": [ ">\"I began estimating true infections in November 2020 because I couldn’t find any good models that were doing that in real-time during a critical moment in the pandemic (though there were 30+ models for forecasting deaths)... My goal when I started covid19-projections.com was to create the most accurate COVID-19 model.\"\n", "-- Youyang Gu, creator of covid19-projections.com" ] }, { "cell_type": "markdown", "id": "46692a2c", "metadata": {}, "source": [ "There have been three separate iterations of the covid19-projections.com model, which are Death Forecasts,\n", "Infections Estimates, and Vaccination Projections. We will use the [Death Forecasting model](https://covid19-projections.com/model-details/) as an example to explore how to frame a Machine Learning problem.\n", "\n", "Let's start with answering some Problem Framing related basic questions:\n", "\n", "1. What are the inputs?\n", " 1. time-series table of death data with geography and demography information. For example to United Status, each row of the data needs to have **a number of deaths $x$ at date $y$ in the region $z$**.\n", "2. What are the outputs?\n", " 1. **a number of deaths $x'$ at a given future date $y'$ in region $z'$**.\n", "3. What are the metrics to measure the success of the project? Such as,\n", " 1. projection accuracy, precision, etc. - comparing with existing Machine Learning models and real-world data,\n", " 2. model inference speed - comparing with existing Machine Learning models,\n", " 3. etc.\n", "4. What are the system architecture and required infrastructure?\n", " 1. a data pipeline to refresh the input data regularly,\n", " 2. a Machine Learning pipeline to regularly iterate the model by using the latest input data,\n", " 3. an event schedule module to manage the system communication and collaboration,\n", " 4. and a website to show the projected results and be accessible in real-time.\n", "5. Any other questions? Such as,\n", " 1. is the data generally available and easy to access,\n", " 2. what are the existing solutions,\n", " 3. etc." ] }, { "cell_type": "markdown", "id": "2d191a1b", "metadata": {}, "source": [ "## Data engineering" ] }, { "cell_type": "markdown", "id": "bbdfacdf", "metadata": {}, "source": [ "### Data ingestion" ] }, { "cell_type": "markdown", "id": "b3fc3be7", "metadata": {}, "source": [ "COVID-19 Projections Death Forecasting model uses the daily death total provided by [Johns Hopkins CSSE](https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data/csse_covid_19_time_series), which is considered by experts to be the “gold standard” reference data. It sometimes uses US testing data from the [COVID Tracking Project](https://covidtracking.com/) in our research and graphs. Below is a piece of sample data from CSSE." ] }, { "cell_type": "code", "execution_count": 2, "id": "5fbb347d", "metadata": { "tags": [ "output_scroll" ] }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
UIDiso2iso3code3FIPSAdmin2Province_StateCountry_RegionLatLong_...6/17/226/18/226/19/226/20/226/21/226/22/226/23/226/24/226/25/226/26/22
084001001USUSA8401001.0AutaugaAlabamaUS32.539527-86.644082...217217217217217217217217217217
184001003USUSA8401003.0BaldwinAlabamaUS30.727750-87.722071...683683683683683683683683683683
284001005USUSA8401005.0BarbourAlabamaUS31.868263-85.387129...99999999999999999999
384001007USUSA8401007.0BibbAlabamaUS32.996421-87.125115...105105105105105105105105105105
484001009USUSA8401009.0BlountAlabamaUS33.982109-86.567906...245245245245245245245245245245
\n", "

5 rows × 899 columns

\n", "
" ], "text/plain": [ " UID iso2 iso3 code3 FIPS Admin2 Province_State Country_Region \\\n", "0 84001001 US USA 840 1001.0 Autauga Alabama US \n", "1 84001003 US USA 840 1003.0 Baldwin Alabama US \n", "2 84001005 US USA 840 1005.0 Barbour Alabama US \n", "3 84001007 US USA 840 1007.0 Bibb Alabama US \n", "4 84001009 US USA 840 1009.0 Blount Alabama US \n", "\n", " Lat Long_ ... 6/17/22 6/18/22 6/19/22 6/20/22 6/21/22 \\\n", "0 32.539527 -86.644082 ... 217 217 217 217 217 \n", "1 30.727750 -87.722071 ... 683 683 683 683 683 \n", "2 31.868263 -85.387129 ... 99 99 99 99 99 \n", "3 32.996421 -87.125115 ... 105 105 105 105 105 \n", "4 33.982109 -86.567906 ... 245 245 245 245 245 \n", "\n", " 6/22/22 6/23/22 6/24/22 6/25/22 6/26/22 \n", "0 217 217 217 217 217 \n", "1 683 683 683 683 683 \n", "2 99 99 99 99 99 \n", "3 105 105 105 105 105 \n", "4 245 245 245 245 245 \n", "\n", "[5 rows x 899 columns]" ] }, "execution_count": 2, "metadata": {}, "output_type": "execute_result" } ], "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('https://static-1300131294.cos.ap-shanghai.myqcloud.com/data/sample_time_series_covid19_deaths_US.csv')\n", "df.head()" ] }, { "cell_type": "markdown", "id": "c4d09ba7", "metadata": {}, "source": [ "The above data is publicly available, and able to be automatically and easily sourced by a web crawler. There are also other ingestion approaches, including synthetic data simulation and manual collecting." ] }, { "cell_type": "markdown", "id": "1a548770", "metadata": {}, "source": [ "### Data processing" ] }, { "cell_type": "markdown", "id": "fe16cffd", "metadata": {}, "source": [ "Typical data processing includes data cleaning, labeling, feature engineering, and augmentation.\n", "\n", "For example, because the CSSE raw data may be noisy, a smoothing algorithm is first to smooth the data. For example, if a state reports 0 death on one day and 300 deaths the next day, the data is smoothed to show 150 deaths on each day. [Sigmoid Function](http://matlab.cheme.cmu.edu/2011/10/30/smooth-transitions-between-discontinuous-functions/#:~:text=Sigmoid%20functions,-A%20sigmoid%20function&text=There%20is%20no%20formal%20justification,or%20from%201%20to%20zero.) is used for this process as shown in below [code snippet](https://github.com/youyanggu/yyg-seir-simulator/blob/b511187a2d4273c92235fdb79017e7a6367e2f4c/region_model.py#L9)." ] }, { "cell_type": "code", "execution_count": 3, "id": "b7e58208", "metadata": {}, "outputs": [], "source": [ "def inv_sigmoid(shift=0, a=1, b=1, c=0):\n", " \"\"\"Returns a inverse sigmoid function based on the parameters.\"\"\"\n", " return lambda x: b * np.exp(-(a*(x-shift))) / (1 + np.exp(-(a*(x-shift)))) + c\n", "\n", "def get_transition_sigmoid(inflection_idx, inflection_rate, low_value, high_value,\n", " check_values=True):\n", " \"\"\"Returns a sigmoid function based on the specified parameters.\n", " A sigmoid helps smooth the transition between low_value and high_value,\n", " with the midpoint being inflection_idx.\n", " inflection_rate is typically a value between 0-1, with 1 being a very steep\n", " transition. We typically use 0.2-0.5 in our projections.\n", " \"\"\"\n", " if check_values:\n", " assert 0 < inflection_rate <= 1, inflection_rate\n", " assert 0 < low_value <= 10, low_value\n", " assert 0 <= high_value <= 10, high_value\n", " shift = inflection_idx\n", " a = inflection_rate\n", " b = low_value - high_value\n", " c = high_value\n", " return utils.inv_sigmoid(shift, a, b, c)" ] }, { "cell_type": "markdown", "id": "861cd050", "metadata": {}, "source": [ "## Model training & evaluation" ] }, { "cell_type": "markdown", "id": "dd3cc59e", "metadata": {}, "source": [ "### Model algorithm" ] }, { "cell_type": "markdown", "id": "39eb9eef", "metadata": {}, "source": [ "A [Grid Search](https://en.wikipedia.org/wiki/Hyperparameter_optimization#Grid_search) based approach is used as the model algorithm, which is similar to the traditional way of performing hyperparameter optimization. Grid Search builds a model on each parameter combination possible. It iterates through every different combination and stores a model for each of them.\n", "\n", "The algorithm is built on top of an SEIR(susceptible-exposed-infectious-recovered) simulator. SEIR is a modified [SIR](https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology) epidemiology model which is applied to the mathematical modeling of infectious diseases." ] }, { "cell_type": "markdown", "id": "979828f0", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/SEIR.jpeg\n", "---\n", "name: SEIR Model\n", "---\n", "SEIR Model\n", ":::" ] }, { "cell_type": "markdown", "id": "7fe45e88", "metadata": {}, "source": [ "The simulator generates infection and death data according to the pre-defined static parameters. By comparing the simulated data with real-world data, it gets the simulator performance under a certain parameter setting." ] }, { "cell_type": "markdown", "id": "e09c5212", "metadata": {}, "source": [ "::: {figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/death-forecasting-model-overview.png\n", "---\n", "name: Death Forecasting Model Overview\n", "---\n", "[Death Forecasting Model Overview](https://covid19-projections.com/model-details/)\n", ":::" ] }, { "cell_type": "markdown", "id": "3097dae0", "metadata": {}, "source": [ "The training of the Death Forecasting model is basically a Grid Search process. It is found that the brute-force Grid Search method that iterates through the entire parameter space is the most effective in finding an optimal set of parameters. So if there are $10$ values for one parameter and $10$ values for another parameter. Then there are $100$ different parameter combinations for those two parameters. For parameters not able to be estimated as lacking data, the values are considered all equally, resulting in a wider confidence interval." ] }, { "cell_type": "markdown", "id": "819c8fab", "metadata": {}, "source": [ ":::{seealso}\n", "[COVID-19 Death Forecasting - Model Details](https://covid19-projections.com/model-details/)\n", ":::" ] }, { "cell_type": "markdown", "id": "5a5ed1d3", "metadata": {}, "source": [ "Based on such a strategy, the Grid Search algorithm searches the optimized SEIR simulator parameters from thousands of different combinations. Then the simulator with the optimized parameters could be used to predict the future death data." ] }, { "cell_type": "markdown", "id": "d063cf71", "metadata": {}, "source": [ "### Model testing" ] }, { "cell_type": "markdown", "id": "e117101b", "metadata": {}, "source": [ "To evaluate the result, the trained Death Forecasting model has been compared with existing Machine Learning approaches, such as the popular [model](https://covid19.healthdata.org/) developed by the [Institute for Health Metrics and Evaluation (IHME)](https://en.wikipedia.org/wiki/Institute_for_Health_Metrics_and_Evaluation). The latter is commonly referred to by the White House and the media. The detailed result could be found on the [covid19-projections website](https://covid19-projections.com/about/#historical-performance)." ] }, { "cell_type": "markdown", "id": "c26e32df", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/covid19-model-performance-evaluation.png\n", "---\n", "name: Evaluation Between Different Machine Learning Model's COVID-19 Death Projections\n", "---\n", "Evaluation Between Different Machine Learning Model's COVID-19 Death Projections\n", ":::" ] }, { "cell_type": "markdown", "id": "682264f8", "metadata": {}, "source": [ "The evaluation is more focused on accuracy. There are other [Test-driven Machine Learning development](https://mlinproduction.com/testing-machine-learning-models-deployment-series-07/) methods, including unit testing, integration testing, etc." ] }, { "cell_type": "markdown", "id": "0a5e5372", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/test-pyramid.png\n", "---\n", "name: The Machine Learning Test Pyramid\n", "---\n", "The [Machine Learning Test Pyramid](https://martinfowler.com/articles/cd4ml.html) from Martin Fowler\n", ":::" ] }, { "cell_type": "markdown", "id": "7e3c13aa", "metadata": {}, "source": [ "## Deployment" ] }, { "cell_type": "markdown", "id": "ee2c3a6f", "metadata": {}, "source": [ "The model training itself is handled manually. Every day, raw daily projections for all 50 US states and select international countries will be uploaded onto the COVID-19 Projections [GitHub Pages](https://pages.github.com/). This will trigger the website building process, which is managed by Github Action and fully CI/CD." ] }, { "cell_type": "markdown", "id": "137f5345", "metadata": {}, "source": [ ":::{figure} https://static-1300131294.cos.ap-shanghai.myqcloud.com/images/covid19-projection-github-action.png\n", "---\n", "name: COVID-19 Projection Github Action\n", "---\n", "COVID-19 Projection Github Action\n", ":::" ] }, { "cell_type": "markdown", "id": "c40355b0", "metadata": {}, "source": [ "The Machine Learning model could be also deployed on an embedded device or as a service. You could refer to [AWS Greengrass](https://aws.amazon.com/greengrass/ml/) and [AWS SageMaker](https://aws.amazon.com/pm/sagemaker/) to learn more." ] }, { "cell_type": "markdown", "id": "c8ddcdab", "metadata": {}, "source": [ "## Maintenance" ] }, { "cell_type": "markdown", "id": "d8a462da", "metadata": {}, "source": [ "[Model drift](https://www.forbes.com/sites/forbestechcouncil/2021/09/23/model-drift-in-data-analytics-what-is-it-so-what-now-what/?sh=54ce17194862) refers to the degradation of model performance due to changes in data and relationships between input and output variables. In order to deal with model drift, continuous model monitoring is the key.\n", "\n", "The covid19-projection is a part-time project. During the author actively worked on it, the past performance is evaluated weekly. The [historical performance](https://covid19-projections.com/historical-performance/) along with other models are presented in the [COVID-19 Forecast Hub](https://github.com/reichlab/covid19-forecast-hub), which is a website that maintains the authoritative, up-to-date record for forecasts of COVID-19 cases, deaths, and hospitalizations in the US. The latest forecasts from these models were sent to the CDC weekly and presented on the CDC COVID-19 Forecasting page. Also, the evaluation results were used to guide the model iteration for better consistency and accuracy." ] }, { "cell_type": "markdown", "id": "62b44a41", "metadata": {}, "source": [ "## Your turn! 🚀" ] }, { "cell_type": "markdown", "id": "4fe980e7", "metadata": {}, "source": [ "It is time to start your own Machine Learning project!\n", "\n", "In this session/assignment, you should come up with a Machine Learning Project idea. Everyone should, in the next session, present their idea with:\n", "\n", "- either [a project plan following this template](../assignments/project-plan-template.ipynb),\n", "- or, [a slide of around 10 pages, like this one](https://docs.google.com/presentation/d/1gTK27XUOC12X8PpggB8_NGxVOAPeYV3I5W6QTYIp49U/edit#slide=id.gcb9a0b074_1_0).\n", "\n", "Your presentation should be limited to around 3 minutes. After the pitch, projects receiving the most votes would \"survive\". You will then form groups of 3-4 students, each group sharing one common project.\n", "\n", "You will present your mid-batch work in this manner:\n", "\n", "- [7. Deep fitting room - STAT 157, Spring 19 documentation](https://courses.d2l.ai/berkeley-stat-157/projects/7.html)\n", "- [M/19 Parameterizable Single GAN Multi-Style - YouTube](https://www.youtube.com/watch?v=pmDEWyf648c)\n", "\n", "And at the end of the batch (demo day), you will present your work in this manner:\n", "\n", "- [UCLA Statistics 102B Final Project Presentation - YouTube](https://www.youtube.com/watch?v=olhyQojuL5M)\n", "- [Unemployment Rate Forecasting using Machine Learning (Student Presentation, Group 3) - YouTube](https://www.youtube.com/watch?v=2F0GSnfKzY4)" ] }, { "cell_type": "markdown", "id": "2d5b782e", "metadata": {}, "source": [ ":::{seealso}\n", "Those links are for you to get some inspirations for a conceiving Machine Learning Project:\n", "\n", "- [Machine Learning Web App](https://www.bilibili.com/video/BV1244y1J7C7/)\n", "- [TensorFlow.js | Machine Learning for JavaScript Developers](https://www.tensorflow.org/js)\n", "- [Gallery • Streamlit](https://streamlit.io/gallery)\n", "- [Tensorflow Playground](https://playground.tensorflow.org)\n", "- [Digit Recognition WebApp](https://maneprajakta.github.io/Digit_Recognition_Web_App/)\n", "- [Google AI Experiments](https://experiments.withgoogle.com/collection/ai)\n", ":::" ] }, { "cell_type": "markdown", "id": "559250d4", "metadata": {}, "source": [ "## Self study\n", "\n", "- [Machine Learning Operations](https://ml-ops.org/)\n", "- [MLOps: Model management, deployment, lineage, and monitoring with Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/concept-model-management-and-deployment)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.11.5" } }, "nbformat": 4, "nbformat_minor": 5 }