{ "cells": [ { "cell_type": "markdown", "id": "5e9a9b6b-0f09-4b9e-8922-31848b2bd416", "metadata": {}, "source": [ "# Using the Python SDK for AutoML with the EON Tuner\n", "\n", "" ] }, { "cell_type": "markdown", "id": "14d2eaca-ddf2-4780-9ee1-3148e0e28df3", "metadata": {}, "source": [ "[![View in Edge Impulse docs](https://raw.githubusercontent.com/edgeimpulse/notebooks/main/.assets/images/ei-badge.svg)](https://docs.edgeimpulse.com/docs/tutorials/ml-and-data-engineering/ei-python-sdk/python-sdk-eon-tuner)\n", "[![Open in Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/edgeimpulse/notebooks/blob/main/notebooks/python-sdk-eon-tuner.ipynb)\n", "[![View on GitHub](https://raw.githubusercontent.com/edgeimpulse/notebooks/main/.assets/images/badge-view-on-github.svg)](https://github.com/edgeimpulse/notebooks/blob/main/notebooks/python-sdk-eon-tuner.ipynb)\n", "[![Download notebook](https://raw.githubusercontent.com/edgeimpulse/notebooks/main/.assets/images/badge-download-notebook.svg)](https://raw.githubusercontent.com/edgeimpulse/notebooks/main/notebooks/python-sdk-eon-tuner.ipynb) " ] }, { "cell_type": "markdown", "id": "53f980f3-366e-4811-ae30-a21d6c219aea", "metadata": {}, "source": [ "The [EON Tuner](https://docs.edgeimpulse.com/docs/edge-impulse-studio/eon-tuner) is Edge Impulse's automated machine learning (AutoML) tool to help you find the best combination of blocks and hyperparameters for your model and within your hardware constraints. This example will walk you through uploading data, running the EON Tuner, and interpreting the results.\n", "\n", "> **WARNING:** This notebook will add and delete data in your Edge Impulse project, so be careful! We recommend creating a throwaway project when testing this notebook.\n", "\n", "To start, create a new project in Edge Impulse. Do not add any data to it." ] }, { "cell_type": "code", "execution_count": null, "id": "280f9aea-c5fa-4fc6-a96d-6589e5e39011", "metadata": {}, "outputs": [], "source": [ "# If you have not done so already, install the following dependencies\n", "# !python -m pip install matplotlib pandas edgeimpulse" ] }, { "cell_type": "code", "execution_count": null, "id": "907e02bc-d39f-413e-82c3-f6399d771839", "metadata": {}, "outputs": [], "source": [ "import edgeimpulse as ei\n", "from edgeimpulse.experimental.data import (\n", " upload_directory\n", ")\n", "from edgeimpulse.experimental.tuner import (\n", " check_tuner,\n", " set_impulse_from_trial,\n", " start_tuner,\n", " start_custom_tuner,\n", " tuner_report_as_df,\n", ")\n", "from edgeimpulse.experimental.impulse import (\n", " build,\n", ")" ] }, { "cell_type": "markdown", "id": "3d58eddb-7677-4ecd-a7d4-d7048cbfab5a", "metadata": {}, "source": [ "You will need to obtain an API key from an Edge Impulse project. Log into [edgeimpulse.com](https://edgeimpulse.com/) and create a new project. Open the project, navigate to **Dashboard** and click on the **Keys** tab to view your API keys. Double-click on the API key to highlight it, right-click, and select **Copy**.\n", "\n", "![Copy API key from Edge Impulse project](https://raw.githubusercontent.com/edgeimpulse/notebooks/main/.assets/images/python-sdk-copy-ei-api-key.png)\n", "\n", "Note that you do not actually need to use the project in the Edge Impulse Studio. We just need the API Key.\n", "\n", "Paste that API key string in the `ei.API_KEY` value in the following cell:" ] }, { "cell_type": "code", "execution_count": null, "id": "71484567-736a-4e22-9068-23a2f90016b2", "metadata": {}, "outputs": [], "source": [ "# Settings\n", "ei.API_KEY = \"ei_dae2...\" # Change this to your Edge Impulse API key\n", "deploy_filename = \"my_model_cpp.zip\"" ] }, { "cell_type": "code", "execution_count": null, "id": "c262c6d1-85d4-47f9-ae8b-a279229fed1c", "metadata": {}, "outputs": [], "source": [ "# Get the project ID\n", "api = ei.experimental.api.EdgeImpulseApi()\n", "project_id = api.default_project_id()" ] }, { "cell_type": "markdown", "id": "fc9d18b1-c3f0-413c-bdd3-ed572b20efc8", "metadata": {}, "source": [ "## Upload dataset\n", "\n", "We start by downloading the [continuous motion dataset](https://docs.edgeimpulse.com/docs/pre-built-datasets/continuous-gestures) and uploading it to our project." ] }, { "cell_type": "code", "execution_count": null, "id": "80c0bf00-29ed-4ccb-a126-2f21e759313a", "metadata": {}, "outputs": [], "source": [ "# Download and unzip gesture dataset\n", "!mkdir -p dataset/\n", "!wget -P dataset -q https://cdn.edgeimpulse.com/datasets/gestures.zip\n", "!unzip -q dataset/gestures.zip -d dataset/gestures/" ] }, { "cell_type": "code", "execution_count": null, "id": "42eea0f4-029d-4894-9f5f-f1e0376e37c4", "metadata": {}, "outputs": [], "source": [ "# Upload training dataset\n", "resp = upload_directory(\n", " directory=\"dataset/gestures/training\",\n", " category=\"training\",\n", ")\n", "print(f\"Uploaded {len(resp.successes)} training samples\")\n", "\n", "# Upload test dataset\n", "resp = upload_directory(\n", " directory=\"dataset/gestures/testing\",\n", " category=\"testing\",\n", ")\n", "print(f\"Uploaded {len(resp.successes)} testing samples\")" ] }, { "cell_type": "code", "execution_count": null, "id": "40ff2db4-35d6-4320-86e5-9d03bb81beda", "metadata": {}, "outputs": [], "source": [ "# Uncomment the following if you want to delete the temporary dataset folder\n", "#!rm -rf dataset/" ] }, { "cell_type": "markdown", "id": "cfc9fdb6-acd8-43af-84e3-866eecc1f91a", "metadata": {}, "source": [ "## Run the Tuner\n", "\n", "To start, we need to list the possible target devices we can use for profiling. We need to pick from this list." ] }, { "cell_type": "code", "execution_count": null, "id": "161a63b2-cc86-4a4f-b079-a47c4b723a65", "metadata": {}, "outputs": [], "source": [ "# List the available profile targets\n", "ei.model.list_profile_devices()" ] }, { "cell_type": "markdown", "id": "74bd9c5f-8b49-4b8b-b394-9bd51ea4cb63", "metadata": {}, "source": [ "You should see a list printed such as:\n", "\n", "```\n", "['alif-he',\n", " 'alif-hp',\n", " 'arduino-nano-33-ble',\n", " 'arduino-nicla-vision',\n", " 'portenta-h7',\n", " 'brainchip-akd1000',\n", " 'cortex-m4f-80mhz',\n", " 'cortex-m7-216mhz',\n", " ...\n", " 'ti-tda4vm']\n", "```\n", "\n", "From there, we start the tuner with `start_tuner()` and wait for completion via `check_tuner()`. In this example, we configure the tuner to target for the *cortex-m4f-80mhz* device. Since we want to classify the motion, we choose `classification` for our `classifcation_type` and our dataset as motion continuous. We constrain our model to a latency of 100ms for running the impulse. \n", "\n", "> **NOTE:** We set the max trials to 3 here. In a real life situation, you will omit this so the tuner decides the best number of trials.\n", "\n", "Once the tuner is done, you can print out the results to determine the best combination of blocks and hyperparameters." ] }, { "cell_type": "code", "execution_count": null, "id": "6454ad7f-c0f7-42f8-aafd-384b3c8bef30", "metadata": {}, "outputs": [], "source": [ "# Choose a device from the list\n", "target_device = \"cortex-m4f-80mhz\"\n", "\n", "# Start tuner. This will take 15+ minutes.\n", "start_tuner(\n", " target_device=target_device,\n", " classification_type=\"classification\",\n", " dataset_category=\"motion_continuous\",\n", " target_latency=100,\n", " tuning_max_trials=3,\n", ")\n", "\n", "# Wait while checking the tuner's progress.\n", "state = check_tuner(\n", " wait_for_completion=True\n", ")" ] }, { "cell_type": "markdown", "id": "2695b5b8-2e7e-4d03-b9d0-59d44b53562e", "metadata": {}, "source": [ "## Print EON Tuner results\n", "\n", "To visualize the results of the tuner trials, you can head to the project page on Edge Impulse Studio.\n", "\n", "Alternatively, you can access the results programmatically: the configuration settings and output of the EON Tuner is stored in the variable `state`. You can access the results of the various trials with `state.trials`. Note that some trials can fail, so it's a good idea to test the status of each trial.\n", "\n", "From there, you will want to sort the results based on some metric. In this example, we will sort based on *int8* test set accuracy from highest to lowest.\n", "\n", "> Note: Edge Impulse supports only one learning block per project at this time (excluding anomaly detection blocks). As a result, we will use the first learning block (e.g. `learning_blocks[0]`) in the list to extract metrics." ] }, { "cell_type": "code", "execution_count": null, "id": "3fbb7ee3-2ef9-4c7c-9c83-af5e5e8147a8", "metadata": {}, "outputs": [], "source": [ "import json" ] }, { "cell_type": "code", "execution_count": null, "id": "2f523608-bc82-4283-bccb-30968bf180b8", "metadata": {}, "outputs": [], "source": [ "# The easiest way to view the results is to look at the EON Tuner page on your project\n", "print(f\"Navigate to https://studio.edgeimpulse.com/studio/{project_id}/tuner to see the results\")" ] }, { "cell_type": "code", "execution_count": null, "id": "14c6316a-9ed5-45a6-8c63-faaeadb30d3e", "metadata": {}, "outputs": [], "source": [ "# Set quantization (\"float32\" or \"int8\")\n", "qtzn = \"int8\"\n", "\n", "# Filter out all failed trials\n", "results = [r for r in state.trials if r.status == \"completed\"]\n", "\n", "# Extract int8 accuracies from the trial results\n", "accuracies = []\n", "for result in results:\n", " accuracy = result.impulse.learn_blocks[0][\"metrics\"][\"test\"][qtzn][\"accuracy\"]\n", " accuracies.append(accuracy)\n", "\n", "# Sort the results based on int8 accuracies\n", "acc_results = zip(accuracies, results)\n", "sorted_results = sorted(acc_results, reverse=True, key=lambda x: list(x)[0])\n", "sorted_results = [result for _, result in sorted_results]" ] }, { "cell_type": "markdown", "id": "75a74e14-abe5-4aac-972a-d45f446f6e4a", "metadata": {}, "source": [ "Now that we have the sorted results, we can extract the values we care about. We will print out the following metrics along with the impulse configuration (processing/learning block configuration and hyperparameters) of the top-performing trial.\n", "\n", "This will help you determine if the impulse can fit on your target hardware and run fast enough for your needs. The impulse configuration can be used to recreate the processing and learning blocks on Edge Impulse. Later, we will set the project impulse based on the trial ID to simply deploy (rather than re-train).\n", "\n", "> **Note:** we assume the first learning block has the metrics we care about." ] }, { "cell_type": "code", "execution_count": null, "id": "c282bf8a-fc21-4b1c-b293-5a5f1a6cc525", "metadata": {}, "outputs": [], "source": [ "def get_metrics(results, qtzn, idx):\n", " \"\"\"Calculate metrics for a given trial index\"\"\"\n", "\n", " metrics = {}\n", "\n", " # Get model accuracy results\n", " result_metrics = results[idx].impulse.learn_blocks[0][\"metrics\"]\n", " metrics[\"val-acc\"] = result_metrics['validation'][qtzn]['accuracy']\n", " metrics[\"test-acc\"] = result_metrics['test'][qtzn]['accuracy']\n", " \n", " # Calculate processing block RAM\n", " metrics[\"processing-block-ram\"] = 0\n", " for i, dsp_block in enumerate(results[idx].impulse.dsp_blocks):\n", " metrics[\"processing-block-ram\"] += dsp_block[\"performance\"][\"ram\"]\n", "\n", " # Get latency, RAM, and ROM usage\n", " device_performance = results[idx].device_performance[qtzn]\n", " metrics[\"learning-block-latency-ms\"] = device_performance['latency']\n", " metrics[\"learning-block-tflite-ram\"] = device_performance['tflite']['ramRequired']\n", " metrics[\"learning-block-tflite-rom\"] = device_performance['tflite']['romRequired']\n", " metrics[\"learning-block-eon-ram\"] = device_performance['eon']['ramRequired']\n", " metrics[\"learning-block-eon-rom\"] = device_performance['eon']['romRequired']\n", "\n", " return metrics" ] }, { "cell_type": "code", "execution_count": null, "id": "e89311e8-991a-4f8d-9547-57f2a34ffa44", "metadata": {}, "outputs": [], "source": [ "# The top performing impulse is the first element (sorted by highest int8 accuracy on test set)\n", "trial_idx = 0\n", "\n", "# Print info about the processing (DSP) blocks and store RAM usage\n", "print(\"Processing blocks\")\n", "print(\"===\")\n", "for i, dsp_block in enumerate(sorted_results[trial_idx].impulse.dsp_blocks):\n", " print(f\"Processing block {i}\")\n", " print(\"---\")\n", " print(\"Block:\")\n", " print(json.dumps(dsp_block[\"block\"], indent=2))\n", " print(\"Config:\")\n", " print(json.dumps(dsp_block[\"config\"], indent=2))\n", "print()\n", "\n", "# Print info about the learning blocks\n", "print(\"Learning blocks\")\n", "print(\"===\")\n", "for i, learn_block in enumerate(sorted_results[trial_idx].impulse.learn_blocks):\n", " print(f\"Learn block {i}\")\n", " print(\"---\")\n", " print(\"Block:\")\n", " print(json.dumps(learn_block[\"block\"], indent=2))\n", " print(\"Config:\")\n", " print(json.dumps(learn_block[\"config\"], indent=2))\n", " metadata = learn_block[\"metadata\"]\n", " qtzn_metadata = [m for m in metadata[\"modelValidationMetrics\"] if m.get(\"type\") == qtzn]\n", "print()\n", "\n", "# Print metrics\n", "metrics = get_metrics(sorted_results, qtzn, trial_idx)\n", "print(f\"Metrics ({qtzn}) for best trial\")\n", "print(\"===\")\n", "print(f\"Validation accuracy: {metrics['val-acc']}\")\n", "print(f\"Test accuracy: {metrics['test-acc']}\")\n", "print(f\"Estimated processing blocks RAM (bytes): {metrics['processing-block-ram']}\")\n", "print(f\"Estimated learning blocks latency (ms): {metrics['learning-block-latency-ms']}\")\n", "print(f\"Estimated learning blocks RAM (bytes): {metrics['learning-block-tflite-ram']}\")\n", "print(f\"Estimated learning blocks ROM (bytes): {metrics['learning-block-tflite-rom']}\")\n", "print(f\"Estimated learning blocks RAM with EON Compiler (bytes): {metrics['learning-block-eon-ram']}\")\n", "print(f\"Estimated learning blocks ROM with EON Compiler (bytes): {metrics['learning-block-eon-rom']}\")" ] }, { "cell_type": "markdown", "id": "bbd88e5f-f283-40ce-9bef-f521a738a92f", "metadata": {}, "source": [ "## Graph results\n", "\n", "You can optionally use a plotting package like matplotlib to graph the results from the top results to compare the metrics." ] }, { "cell_type": "code", "execution_count": null, "id": "d310f1af-9dc5-482a-9b56-2af82b3f279a", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt" ] }, { "cell_type": "code", "execution_count": null, "id": "2472096a-6932-4ba1-8ab7-d1036becda7b", "metadata": {}, "outputs": [], "source": [ "# Get metrics for the top 3 trials (sorted by int8 test set accuracy)\n", "num_trials = 3\n", "top_metrics = [get_metrics(sorted_results, qtzn, idx) for idx in range(num_trials)]\n", "\n", "# Construct metrics for plotting\n", "test_accs = [top_metrics[x]['test-acc'] for x in range(num_trials)]\n", "proc_rams = [top_metrics[x]['processing-block-ram'] for x in range(num_trials)]\n", "learn_latencies = [top_metrics[x]['learning-block-latency-ms'] for x in range(num_trials)]\n", "learn_tflite_rams = [top_metrics[x]['learning-block-tflite-ram'] for x in range(num_trials)]\n", "learn_tflite_roms = [top_metrics[x]['learning-block-tflite-rom'] for x in range(num_trials)]\n", "learn_eon_rams = [top_metrics[x]['learning-block-eon-ram'] for x in range(num_trials)]\n", "learn_eon_roms = [top_metrics[x]['learning-block-eon-rom'] for x in range(num_trials)]" ] }, { "cell_type": "code", "execution_count": null, "id": "6bc99e7f-96ec-41cf-9e44-3ddc2d31b0ac", "metadata": {}, "outputs": [], "source": [ "# Create plots\n", "fig, axs = plt.subplots(7, 1, figsize=(8, 15))\n", "indices = range(num_trials)\n", "\n", "# Plot test accuracies\n", "axs[0].barh(indices, test_accs)\n", "axs[0].set_title(\"Test set accuracy\")\n", "axs[0].set_xlabel(\"Accuracy\")\n", "axs[0].set_ylabel(\"Trial\")\n", "\n", "# Plot processing block RAM\n", "axs[1].barh(indices, proc_rams)\n", "axs[1].set_title(\"Processing block RAM\")\n", "axs[1].set_xlabel(\"RAM (bytes)\")\n", "axs[1].set_ylabel(\"Trial\")\n", "\n", "# Plot learning block latency\n", "axs[2].barh(indices, learn_latencies)\n", "axs[2].set_title(\"Learning block latency\")\n", "axs[2].set_xlabel(\"Latency (ms)\")\n", "axs[2].set_ylabel(\"Trial\")\n", "\n", "# Plot learning block RAM (TFLite)\n", "axs[3].barh(indices, learn_tflite_rams)\n", "axs[3].set_title(\"Learning block RAM (TFLite)\")\n", "axs[3].set_xlabel(\"RAM (bytes)\")\n", "axs[3].set_ylabel(\"Trial\")\n", "\n", "# Plot learning block ROM (TFLite)\n", "axs[4].barh(indices, learn_tflite_roms)\n", "axs[4].set_title(\"Learning block ROM (TFLite)\")\n", "axs[4].set_xlabel(\"ROM (bytes)\")\n", "axs[4].set_ylabel(\"Trial\")\n", "\n", "# Plot learning block RAM (EON)\n", "axs[5].barh(indices, learn_eon_rams)\n", "axs[5].set_title(\"Learning block RAM (EON)\")\n", "axs[5].set_xlabel(\"RAM (bytes)\")\n", "axs[5].set_ylabel(\"Trial\")\n", "\n", "# Plot learning block ROM (EON)\n", "axs[6].barh(indices, learn_eon_roms)\n", "axs[6].set_title(\"Learning block ROM (EON)\")\n", "axs[6].set_xlabel(\"ROM (bytes)\")\n", "axs[6].set_ylabel(\"Trial\")\n", "\n", "# Prevent overlap\n", "plt.tight_layout()" ] }, { "cell_type": "markdown", "id": "33cf1f9e-b3a2-4352-a195-0bb33a467e15", "metadata": {}, "source": [ "## Results as a DataFrame\n", "\n", "If you have [pandas](https://pandas.pydata.org/) installed, you can make the previous section much easier by reporting metrics as a DataFrame." ] }, { "cell_type": "code", "execution_count": null, "id": "e2b27f0b-fa98-41a4-a0ee-63a4079a6778", "metadata": {}, "outputs": [], "source": [ "import pandas as pd" ] }, { "cell_type": "code", "execution_count": null, "id": "fd19c9c5-41d6-4662-b4f2-3bb31e1ca0a8", "metadata": {}, "outputs": [], "source": [ "# Convert the state metrics into a DataFrame\n", "df = tuner_report_as_df(state)\n", "df.head()" ] }, { "cell_type": "code", "execution_count": null, "id": "2659a52e-3602-4829-9b7c-ac3c866c7d86", "metadata": {}, "outputs": [], "source": [ "# Print column names\n", "for col in df.columns:\n", " print(col)" ] }, { "cell_type": "code", "execution_count": null, "id": "5007aab3-095f-4070-b5b8-51564fe9426a", "metadata": {}, "outputs": [], "source": [ "# Sort the DataFrame by validation (int8) accuracy\n", "df = df.sort_values(by=\"test_int8_accuracy\", ascending=False)\n", "\n", "# Print out best trial metrics\n", "print(f\"Trial ID: {df.iloc[0]['id']}\")\n", "print(f\"Test accuracy (int8): {df.iloc[0]['test_int8_accuracy']}\")\n", "print(f\"Estimated learning blocks latency (ms): {df.iloc[0]['device_performance_int8_latency']}\")\n", "print(f\"Estimated learning blocks RAM (bytes): {df.iloc[0]['device_performance_int8_tflite_ram_required']}\")\n", "print(f\"Estimated learning blocks ROM (bytes): {df.iloc[0]['device_performance_int8_tflite_rom_required']}\")\n", "print(f\"Estimated learning blocks RAM with EON Compiler (bytes): {df.iloc[0]['device_performance_int8_eon_ram_required']}\")\n", "print(f\"Estimated learning blocks ROM with EON Compiler (bytes): {df.iloc[0]['device_performance_int8_eon_rom_required']}\")" ] }, { "cell_type": "markdown", "id": "e952e100-4cf0-454a-8033-400e831367c1", "metadata": {}, "source": [ "## Set trial as impulse and deploy\n", "\n", "We can replace the current impulse with the top performing trial from the EON Tuner. From there, we can deploy it, just like we would any impulse." ] }, { "cell_type": "code", "execution_count": null, "id": "31888f29-194c-4ccb-b3ca-6300f1179811", "metadata": {}, "outputs": [], "source": [ "# Get the ID for the top-performing trial and set that to our impulse. This will take about a minute.\n", "trial_id = df.iloc[0].trial_id\n", "response = set_impulse_from_trial(trial_id)\n", "job_id = response.id\n", "\n", "# Make sure the impulse update was successful\n", "if not hasattr(response, \"success\") or getattr(response, \"success\") == False:\n", " raise RuntimeError(\"Could not set project impulse to trial impulse\")" ] }, { "cell_type": "code", "execution_count": null, "id": "7cc04b19-b4a2-4605-9bf7-fc9679b3253f", "metadata": {}, "outputs": [], "source": [ "# List the available profile target devices\n", "ei.model.list_deployment_targets()" ] }, { "cell_type": "markdown", "id": "5fcbacef-4078-4562-bb6a-98bd639020c4", "metadata": {}, "source": [ "You should see a list printed such as:\n", "\n", "```\n", "['zip',\n", " 'arduino',\n", " 'cubemx',\n", " 'wasm',\n", " ...\n", " 'runner-linux-aarch64-jetson-orin-6-0']\n", "```\n", "\n", "The most generic target is to download a .zip file that holds a C++ library containing the inference runtime and your trained model, so we choose `'zip'` from the above list. To do that, we first need to create a Classification object which contains our label strings (and other optional information about the model). These strings will be added to the C++ library metadata so you can access them in your edge application.\n", "\n", "Note that instead of writing the raw bytes to a file, you can also specify an `output_directory` argument in the `.deploy()` function. Your deployment file(s) will be downloaded to that directory.\n", "\n", "**Important!** The deployment targets list will change depending on the values provided for `model`, `model_output_type`, and `model_input_type` in the next part. For example, you will not see `openmv` listed once you upload a model (e.g. using `.profile()` or `.deploy()`) if `model_input_type` is not set to `ei.model.input_type.ImageInput()`. If you attempt to deploy to an unavailable target, you will receive the error `Could not deploy: deploy_target: ...`. If `model_input_type` is not provided, it will default to [OtherInput](https://docs.edgeimpulse.com/reference/python-sdk/edgeimpulse/model/input_type#otherinput). See [this page](https://docs.edgeimpulse.com/reference/python-sdk/edgeimpulse/model/input_type) for more information about input types." ] }, { "cell_type": "code", "execution_count": null, "id": "9230c34e-6cdd-49d1-9cee-43c398c8f5d0", "metadata": {}, "outputs": [], "source": [ "# Build and download C++ library with the trained model\n", "deploy_bytes = None\n", "try:\n", " deploy_bytes = build(\n", " deploy_model_type=qtzn,\n", " engine=\"tflite\",\n", " deploy_target=\"zip\"\n", " )\n", "except Exception as e:\n", " print(f\"Could not deploy: {e}\")\n", " \n", "# Write the downloaded raw bytes to a file\n", "if deploy_bytes:\n", " with open(deploy_filename, 'wb') as f:\n", " f.write(deploy_bytes.getvalue())" ] }, { "cell_type": "markdown", "id": "46d9a4cb-72d3-48f0-99bd-40406caa7fd1", "metadata": {}, "source": [ "Your model C++ library should be downloaded as the file *my_model_cpp.zip* in the same directory as this notebook. You are now ready to use your C++ model in your embedded and edge device application! To use the C++ model for local inference, see our documentation [here](https://docs.edgeimpulse.com/docs/deployment/running-your-impulse-locally)." ] }, { "cell_type": "markdown", "id": "302fc299-d6ba-4a23-aef8-dfee1924ebee", "metadata": {}, "source": [ "## Configure custom search space\n", "\n", "By default, the EON Tuner will make a guess at a search space based on the type of data you uploaded (e.g. using spectral-analysis blocks for feature extraction). As a result, you can run the tuner without needing to construct a search space. However, you may want to define your own search space.\n", "\n", "The best way to define a search space is to open your project (after uploading data), head to the **EON Tuner** page, click **Run EON Tuner**, and select the **Space** tab.\n", "\n", "![EON Tuner search space](https://raw.githubusercontent.com/edgeimpulse/notebooks/main/.assets/images/python-sdk-eon-tuner-search-space.png)\n", "\n", "The search space is defined in JSON format, so we can just copy that to create a dictionary. This is a good place to start for tuning blocks and hyperparameters.\n", "\n", "> **Note:** Functions to get available blocks and search space parameters coming soon" ] }, { "cell_type": "code", "execution_count": null, "id": "c1d19d57-1b6f-4e1c-adfd-457140369e8b", "metadata": {}, "outputs": [], "source": [ "from edgeimpulse_api import (\n", " OptimizeConfig,\n", " TunerSpaceImpulse,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "b9232f88-e597-4f22-bbd4-5bfd2dbde71e", "metadata": {}, "outputs": [], "source": [ "# Configure the search space\n", "space = {\n", " \"inputBlocks\": [\n", " {\n", " \"type\": \"time-series\",\n", " \"window\": [\n", " {\"windowSizeMs\": 9000, \"windowIncreaseMs\": 9000},\n", " {\"windowSizeMs\": 10000, \"windowIncreaseMs\": 10000}\n", " ],\n", " \"frequencyHz\": [62.5],\n", " \"padZeros\": [True]\n", " }\n", " ],\n", " \"dspBlocks\": [\n", " {\n", " \"type\": \"spectral-analysis\",\n", " \"analysis-type\": [\"FFT\"],\n", " \"fft-length\": [16, 64],\n", " \"scale-axes\": [1],\n", " \"filter-type\": [\"none\"],\n", " \"filter-cutoff\": [3],\n", " \"filter-order\": [6],\n", " \"do-log\": [True],\n", " \"do-fft-overlap\": [True]\n", " },\n", " {\n", " \"type\": \"spectral-analysis\",\n", " \"analysis-type\": [\"Wavelet\"],\n", " \"wavelet\": [\"haar\", \"bior1.3\"],\n", " \"wavelet-level\": [1, 2]\n", " },\n", " {\"type\": \"raw\", \"scale-axes\": [1]}\n", " ],\n", " \"learnBlocks\": [\n", " {\n", " \"id\": 4,\n", " \"type\": \"keras\",\n", " \"dimension\": [\"dense\"],\n", " \"denseBaseNeurons\": [40, 20],\n", " \"denseLayers\": [2, 3],\n", " \"dropout\": [0.25, 0.5],\n", " \"learningRate\": [0.0005],\n", " \"trainingCycles\": [30]\n", " }\n", " ]\n", " }" ] }, { "cell_type": "code", "execution_count": null, "id": "dfa5b124-7ffe-4e25-b614-0b12a6b917e3", "metadata": {}, "outputs": [], "source": [ "# Wrap the search space\n", "ts = TunerSpaceImpulse.from_dict(space)\n", "\n", "# Create a custom configuration\n", "config = OptimizeConfig(\n", " name=None,\n", " target_device={\"name\": \"cortex-m4f-80mhz\"},\n", " classification_type=\"classification\",\n", " dataset_category=\"motion_continuous\",\n", " target_latency=100,\n", " tuning_max_trials=2,\n", " space=[ts]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "25303fc2-0caf-49dd-8e8b-7109eaeca564", "metadata": {}, "outputs": [], "source": [ "# Start tuner and wait for it to complete\n", "start_custom_tuner(\n", " config=config\n", ")\n", "state = check_tuner(\n", " wait_for_completion=True\n", ")" ] }, { "cell_type": "code", "execution_count": null, "id": "d25c5ae4-c6d0-4d9f-999c-9e4420f030a6", "metadata": {}, "outputs": [], "source": [ "# The easiest way to view the results is to look at the EON Tuner page on your project\n", "print(f\"Navigate to https://studio.edgeimpulse.com/studio/{project_id}/tuner to see the results\")" ] }, { "cell_type": "code", "execution_count": null, "id": "90a39aee-5d9a-443d-bc99-c5c7a9ea3a35", "metadata": {}, "outputs": [], "source": [ "# Set quantization (\"float32\" or \"int8\")\n", "qtzn = \"int8\"\n", "\n", "# Filter out all failed trials\n", "results = [r for r in state.trials if r.status == \"completed\"]\n", "\n", "# Extract float32 accuracies from the trial results\n", "accuracies = []\n", "for result in results:\n", " accuracy = result.impulse.learn_blocks[0][\"metrics\"][\"test\"][qtzn][\"accuracy\"]\n", " accuracies.append(accuracy)\n", "\n", "# Sort the results based on int8 accuracies\n", "acc_results = zip(accuracies, results)\n", "sorted_results = sorted(acc_results, reverse=True, key=lambda x: list(x)[0])\n", "sorted_results = [result for _, result in sorted_results]" ] }, { "cell_type": "code", "execution_count": null, "id": "3a7988c9-b6b9-49a4-a442-6303c57695fa", "metadata": {}, "outputs": [], "source": [ "# The top performing impulse is the first element (sorted by highest int8 accuracy on test set)\n", "trial_idx = 0\n", "\n", "# Print info about the processing (DSP) blocks and store RAM usage\n", "print(\"Processing blocks\")\n", "print(\"===\")\n", "for i, dsp_block in enumerate(sorted_results[trial_idx].impulse.dsp_blocks):\n", " print(f\"Processing block {i}\")\n", " print(\"---\")\n", " print(\"Block:\")\n", " print(json.dumps(dsp_block[\"block\"], indent=2))\n", " print(\"Config:\")\n", " print(json.dumps(dsp_block[\"config\"], indent=2))\n", "print()\n", "\n", "# Print info about the learning blocks\n", "print(\"Learning blocks\")\n", "print(\"===\")\n", "for i, learn_block in enumerate(sorted_results[trial_idx].impulse.learn_blocks):\n", " print(f\"Learn block {i}\")\n", " print(\"---\")\n", " print(\"Block:\")\n", " print(json.dumps(learn_block[\"block\"], indent=2))\n", " print(\"Config:\")\n", " print(json.dumps(learn_block[\"config\"], indent=2))\n", " metadata = learn_block[\"metadata\"]\n", " qtzn_metadata = [m for m in metadata[\"modelValidationMetrics\"] if m.get(\"type\") == qtzn]\n", "print()\n", "\n", "# Print metrics\n", "metrics = get_metrics(sorted_results, qtzn, trial_idx)\n", "print(f\"Metrics ({qtzn}) for best trial\")\n", "print(\"===\")\n", "print(f\"Validation accuracy: {metrics['val-acc']}\")\n", "print(f\"Test accuracy: {metrics['test-acc']}\")\n", "print(f\"Estimated processing blocks RAM (bytes): {metrics['processing-block-ram']}\")\n", "print(f\"Estimated learning blocks latency (ms): {metrics['learning-block-latency-ms']}\")\n", "print(f\"Estimated learning blocks RAM (bytes): {metrics['learning-block-tflite-ram']}\")\n", "print(f\"Estimated learning blocks ROM (bytes): {metrics['learning-block-tflite-rom']}\")\n", "print(f\"Estimated learning blocks RAM with EON Compiler (bytes): {metrics['learning-block-eon-ram']}\")\n", "print(f\"Estimated learning blocks ROM with EON Compiler (bytes): {metrics['learning-block-eon-rom']}\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.16" } }, "nbformat": 4, "nbformat_minor": 5 }