{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# Compare performance of 4 CNNs on Chromebook 2"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We compare **performance** of inference (forward propagation):\n",
"\n",
"- of 4 different CNN **models** (net architecture + weights):\n",
" - [AlexNet](https://github.com/BVLC/caffe/tree/master/models/bvlc_alexnet);\n",
" - [SqueezeNet 1.0](https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.0);\n",
" - [SqueezeNet 1.1](https://github.com/DeepScale/SqueezeNet/tree/master/SqueezeNet_v1.1);\n",
" - [GoogleNet](https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet);\n",
"\n",
"- with 4 different **libraries**:\n",
"\n",
" - [CPU] [OpenBLAS](https://github.com/xianyi/OpenBLAS) 0.2.18 (one thread per core);\n",
" - [GPU] [clBLAS](https://github.com/clMathLibraries/clBLAS) 2.4 (OpenCL 1.1 compliant);\n",
" - [GPU] [CLBlast](https://github.com/CNugteren/CLBlast) dev (35623cd > 0.8.0);\n",
" - [GPU] [CLBlast](https://github.com/CNugteren/CLBlast) dev (35623cd > 0.8.0) with Mali-optimized [overlay](https://github.com/intelfx/CLBlast/tree/mali-overlay) (641bb07);\n",
" \n",
"- on the [Samsung Chromebook 2](http://www.samsung.com/us/computing/chromebooks/under-12/samsung-chromebook-2-11-6-xe503c12-k01us/) **platform**:\n",
" - [CPU] quad-core ARM Cortex-A15 @ 1900 MHz;\n",
" - [GPU] quad-core ARM Mali-T628 @ 600 MHz;\n",
" - [GPU] OpenCL driver 6.0 (r6p0);\n",
" - [GPU] OpenCL standard 1.1;\n",
" - [RAM] 2 GB;\n",
" - Gentoo Linux [over](community.arm.com/groups/arm-mali-graphics/blog/2014/12/18/installing-opencl-on-chromebook-2-in-30-minutes) ChromeOS with `/etc/lsb-release`:\n",
"```\n",
"CHROMEOS_AUSERVER=https://tools.google.com/service/update2\n",
"CHROMEOS_BOARD_APPID={24E2E4F7-F92C-6115-3E26-02C7EAA02946}\n",
"CHROMEOS_CANARY_APPID={90F229CE-83E2-4FAF-8479-E368A34938B1}\n",
"CHROMEOS_DEVSERVER=\n",
"CHROMEOS_RELEASE_APPID={24E2E4F7-F92C-6115-3E26-02C7EAA02946}\n",
"CHROMEOS_RELEASE_BOARD=peach_pit-signed-mp-v2keys\n",
"CHROMEOS_RELEASE_BRANCH_NUMBER=68\n",
"CHROMEOS_RELEASE_BUILD_NUMBER=8350\n",
"CHROMEOS_RELEASE_BUILD_TYPE=Official Build\n",
"CHROMEOS_RELEASE_CHROME_MILESTONE=52\n",
"CHROMEOS_RELEASE_DESCRIPTION=8350.68.0 (Official Build) stable-channel peach_pit \n",
"CHROMEOS_RELEASE_NAME=Chrome OS\n",
"CHROMEOS_RELEASE_PATCH_NUMBER=0\n",
"CHROMEOS_RELEASE_TRACK=stable-channel\n",
"CHROMEOS_RELEASE_VERSION=8350.68.0\n",
"DEVICETYPE=CHROMEBOOK\n",
"GOOGLE_RELEASE=8350.68.0\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Includes"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Collective Knowledge"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CK version: 1.7.4dev\n"
]
}
],
"source": [
"import ck.kernel as ck\n",
"print ('CK version: %s' % ck.__version__)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Scientific"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import math\n",
"import IPython as ip\n",
"import numpy as np\n",
"import scipy as sp\n",
"import pandas as pd\n",
"import matplotlib as mp"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"IPython version: 4.1.1\n",
"NumPy version: 1.10.4\n",
"SciPy version: 0.17.0\n",
"Pandas version: 0.18.0+57.g101d81d.dirty\n",
"Matplotlib version: 1.5.1\n"
]
}
],
"source": [
"print ('IPython version: %s' % ip.__version__)\n",
"print ('NumPy version: %s' % np.__version__)\n",
"print ('SciPy version: %s' % sp.__version__)\n",
"print ('Pandas version: %s' % pd.__version__)\n",
"print ('Matplotlib version: %s' % mp.__version__)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from matplotlib import cm\n",
"%matplotlib inline"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Access experimental results"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def search_experimental_points_by_tags(tags):\n",
" r=ck.access({'action':'get', 'module_uoa':'experiment', 'tags':tags, 'load_json_files':['0001']})\n",
" if r['return']>0:\n",
" print (\"Error: %s\" % r['error'])\n",
" exit(1)\n",
" # FIXME: For now, assume a single entry per the given tags.\n",
" results = {}\n",
" for point in r['points']:\n",
" point_data_raw = point['0001']\n",
" point_data_dict = {}\n",
" time_fw_ms = [\n",
" characteristics['run']['time_fw_ms'] \n",
" for characteristics in point_data_raw['characteristics_list']\n",
" if characteristics['run']['run_success'] == 'yes'\n",
" ]\n",
" batch_size = point_data_raw['choices']['env']['CK_CAFFE_BATCH_SIZE']\n",
" results[batch_size] = time_fw_ms\n",
" return results"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"def get_min_time_per_image(results):\n",
" df = pd.DataFrame(data=results)\n",
" df.columns.name = 'Batch size'\n",
" df.index.name = 'Repetition'\n",
" df_stats = df.describe()\n",
" min_time_per_image_idx = (df_stats.ix['mean'] / df.columns).idxmin()\n",
" if not math.isnan(min_time_per_image_idx):\n",
" mean = (df_stats[min_time_per_image_idx] / min_time_per_image_idx)['mean']\n",
" std = (df_stats[min_time_per_image_idx] / min_time_per_image_idx)['std']\n",
" return (mean, std)\n",
" else:\n",
" return (None, None)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### AlexNet"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(535.13888888888891, 8.0335922042199481)"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"alexnet_clblas_tags = 'chromebook2,time,caffemodel,alexnet,clblas'\n",
"alexnet_clblas_results = search_experimental_points_by_tags(alexnet_clblas_tags)\n",
"alexnet_clblas_min_time_per_image = get_min_time_per_image(alexnet_clblas_results)\n",
"alexnet_clblas_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(565.09666666666669, 4.3450922122934852)"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"alexnet_clblast_development_tags = 'chromebook2,time,caffemodel,alexnet,clblast,vdevelopment'\n",
"alexnet_clblast_development_results = search_experimental_points_by_tags(alexnet_clblast_development_tags)\n",
"alexnet_clblast_development_min_time_per_image = get_min_time_per_image(alexnet_clblast_development_results)\n",
"alexnet_clblast_development_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(486.57500000000005, 20.60394066677539)"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"alexnet_clblast_mali_overlay_tags = 'chromebook2,time,caffemodel,alexnet,clblast,vmali-overlay'\n",
"alexnet_clblast_mali_overlay_results = search_experimental_points_by_tags(alexnet_clblast_mali_overlay_tags)\n",
"alexnet_clblast_mali_overlay_min_time_per_image = get_min_time_per_image(alexnet_clblast_mali_overlay_results)\n",
"alexnet_clblast_mali_overlay_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(330.08333333333331, 25.476067854622574)"
]
},
"execution_count": 10,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"alexnet_openblas_tags = 'chromebook2,time,caffemodel,alexnet,openblas'\n",
"alexnet_openblas_results = search_experimental_points_by_tags(alexnet_openblas_tags)\n",
"alexnet_openblas_min_time_per_image = get_min_time_per_image(alexnet_openblas_results)\n",
"alexnet_openblas_min_time_per_image"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SqueezeNet 1.0"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(665.42583333333334, 1.9714038990864458)"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_0_clblas_tags = 'chromebook2,time,caffemodel,squeezenet-1.0,clblas'\n",
"squeezenet_1_0_clblas_results = search_experimental_points_by_tags(squeezenet_1_0_clblas_tags)\n",
"squeezenet_1_0_clblas_min_time_per_image = get_min_time_per_image(squeezenet_1_0_clblas_results)\n",
"squeezenet_1_0_clblas_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(617.96333333333325, 7.1004512767382133)"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_0_clblast_development_tags = 'chromebook2,time,caffemodel,squeezenet-1.0,clblast,vdevelopment'\n",
"squeezenet_1_0_clblast_development_results = search_experimental_points_by_tags(squeezenet_1_0_clblast_development_tags)\n",
"squeezenet_1_0_clblast_development_min_time_per_image = get_min_time_per_image(squeezenet_1_0_clblast_development_results)\n",
"squeezenet_1_0_clblast_development_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(499.44666666666672, 42.00408088639854)"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_0_clblast_mali_overlay_tags = 'chromebook2,time,caffemodel,squeezenet-1.0,clblast,vmali-overlay'\n",
"squeezenet_1_0_clblast_mali_overlay_results = search_experimental_points_by_tags(squeezenet_1_0_clblast_mali_overlay_tags)\n",
"squeezenet_1_0_clblast_mali_overlay_min_time_per_image = get_min_time_per_image(squeezenet_1_0_clblast_mali_overlay_results)\n",
"squeezenet_1_0_clblast_mali_overlay_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(275.07666666666665, 26.41040116286257)"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_0_openblas_tags = 'chromebook2,time,caffemodel,squeezenet-1.0,openblas'\n",
"squeezenet_1_0_openblas_results = search_experimental_points_by_tags(squeezenet_1_0_openblas_tags)\n",
"squeezenet_1_0_openblas_min_time_per_image = get_min_time_per_image(squeezenet_1_0_openblas_results)\n",
"squeezenet_1_0_openblas_min_time_per_image"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### SqueezeNet 1.1"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(241.67033333333333, 0.95675089277895931)"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_1_clblas_tags = 'chromebook2,time,caffemodel,squeezenet-1.1,clblas'\n",
"squeezenet_1_1_clblas_results = search_experimental_points_by_tags(squeezenet_1_1_clblas_tags)\n",
"squeezenet_1_1_clblas_min_time_per_image = get_min_time_per_image(squeezenet_1_1_clblas_results)\n",
"squeezenet_1_1_clblas_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(395.6583333333333, 6.4850589498734195)"
]
},
"execution_count": 16,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_1_clblast_development_tags = 'chromebook2,time,caffemodel,squeezenet-1.1,clblast,vdevelopment'\n",
"squeezenet_1_1_clblast_development_results = search_experimental_points_by_tags(squeezenet_1_1_clblast_development_tags)\n",
"squeezenet_1_1_clblast_development_min_time_per_image = get_min_time_per_image(squeezenet_1_1_clblast_development_results)\n",
"squeezenet_1_1_clblast_development_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(None, None)"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_1_clblast_mali_overlay_tags = 'chromebook2,time,caffemodel,squeezenet-1.1,clblast,vmali-overlay'\n",
"squeezenet_1_1_clblast_mali_overlay_results = search_experimental_points_by_tags(squeezenet_1_1_clblast_mali_overlay_tags)\n",
"squeezenet_1_1_clblast_mali_overlay_min_time_per_image = get_min_time_per_image(squeezenet_1_1_clblast_mali_overlay_results)\n",
"squeezenet_1_1_clblast_mali_overlay_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(139.79100000000003, 3.5545299003525086)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"squeezenet_1_1_openblas_tags = 'chromebook2,time,caffemodel,squeezenet-1.1,openblas'\n",
"squeezenet_1_1_openblas_results = search_experimental_points_by_tags(squeezenet_1_1_openblas_tags)\n",
"squeezenet_1_1_openblas_min_time_per_image = get_min_time_per_image(squeezenet_1_1_openblas_results)\n",
"squeezenet_1_1_openblas_min_time_per_image"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### GoogleNet"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(836.8649999999999, 6.8739303713377886)"
]
},
"execution_count": 19,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"googlenet_clblas_tags = 'chromebook2,time,caffemodel,googlenet,clblas'\n",
"googlenet_clblas_results = search_experimental_points_by_tags(googlenet_clblas_tags)\n",
"googlenet_clblas_min_time_per_image = get_min_time_per_image(googlenet_clblas_results)\n",
"googlenet_clblas_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(1105.7725, 24.338692959976285)"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"googlenet_clblast_development_tags = 'chromebook2,time,caffemodel,googlenet,clblast,vdevelopment'\n",
"googlenet_clblast_development_results = search_experimental_points_by_tags(googlenet_clblast_development_tags)\n",
"googlenet_clblast_development_min_time_per_image = get_min_time_per_image(googlenet_clblast_development_results)\n",
"googlenet_clblast_development_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(None, None)"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"googlenet_clblast_mali_overlay_tags = 'chromebook2,time,caffemodel,googlenet,clblast,vmali-overlay'\n",
"googlenet_clblast_mali_overlay_results = search_experimental_points_by_tags(googlenet_clblast_mali_overlay_tags)\n",
"googlenet_clblast_mali_overlay_min_time_per_image = get_min_time_per_image(googlenet_clblast_mali_overlay_results)\n",
"googlenet_clblast_mali_overlay_min_time_per_image"
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"(652.70249999999999, 45.175774204765965)"
]
},
"execution_count": 22,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"googlenet_openblas_tags = 'chromebook2,time,caffemodel,googlenet,openblas'\n",
"googlenet_openblas_results = search_experimental_points_by_tags(googlenet_openblas_tags)\n",
"googlenet_openblas_min_time_per_image = get_min_time_per_image(googlenet_openblas_results)\n",
"googlenet_openblas_min_time_per_image"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Data frame"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def extract_by_component(idx):\n",
" data = {\n",
" 'AlexNet' : {\n",
" 'OpenBLAS: v0.2.18' : alexnet_openblas_min_time_per_image[idx],\n",
" 'clBLAS: v2.4' : alexnet_clblas_min_time_per_image[idx],\n",
" 'CLBlast: development' : alexnet_clblast_development_min_time_per_image[idx],\n",
" 'CLBlast: Mali overlay' : alexnet_clblast_mali_overlay_min_time_per_image[idx]\n",
" },\n",
" 'SqueezeNet 1.0' : {\n",
" 'OpenBLAS: v0.2.18' : squeezenet_1_0_openblas_min_time_per_image[idx],\n",
" 'clBLAS: v2.4' : squeezenet_1_0_clblas_min_time_per_image[idx],\n",
" 'CLBlast: development' : squeezenet_1_0_clblast_development_min_time_per_image[idx],\n",
" 'CLBlast: Mali overlay' : squeezenet_1_0_clblast_mali_overlay_min_time_per_image[idx]\n",
" },\n",
" 'SqueezeNet 1.1' : {\n",
" 'OpenBLAS: v0.2.18' : squeezenet_1_1_openblas_min_time_per_image[idx],\n",
" 'clBLAS: v2.4' : squeezenet_1_1_clblas_min_time_per_image[idx],\n",
" 'CLBlast: development' : squeezenet_1_1_clblast_development_min_time_per_image[idx],\n",
" 'CLBlast: Mali overlay' : squeezenet_1_1_clblast_mali_overlay_min_time_per_image[idx]\n",
" },\n",
" 'GoogleNet' : {\n",
" 'OpenBLAS: v0.2.18' : googlenet_openblas_min_time_per_image[idx],\n",
" 'clBLAS: v2.4' : googlenet_clblas_min_time_per_image[idx],\n",
" 'CLBlast: development' : googlenet_clblast_development_min_time_per_image[idx],\n",
" 'CLBlast: Mali overlay' : googlenet_clblast_mali_overlay_min_time_per_image[idx]\n",
" },\n",
" }\n",
" return data"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" AlexNet | \n",
" GoogleNet | \n",
" SqueezeNet 1.0 | \n",
" SqueezeNet 1.1 | \n",
"
\n",
" \n",
" \n",
" \n",
" | CLBlast: Mali overlay | \n",
" 486.575000 | \n",
" NaN | \n",
" 499.446667 | \n",
" NaN | \n",
"
\n",
" \n",
" | CLBlast: development | \n",
" 565.096667 | \n",
" 1105.7725 | \n",
" 617.963333 | \n",
" 395.658333 | \n",
"
\n",
" \n",
" | OpenBLAS: v0.2.18 | \n",
" 330.083333 | \n",
" 652.7025 | \n",
" 275.076667 | \n",
" 139.791000 | \n",
"
\n",
" \n",
" | clBLAS: v2.4 | \n",
" 535.138889 | \n",
" 836.8650 | \n",
" 665.425833 | \n",
" 241.670333 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" AlexNet | \n",
" GoogleNet | \n",
" SqueezeNet 1.0 | \n",
" SqueezeNet 1.1 | \n",
"
\n",
" \n",
" \n",
" \n",
" | CLBlast: Mali overlay | \n",
" 20.603941 | \n",
" NaN | \n",
" 42.004081 | \n",
" NaN | \n",
"
\n",
" \n",
" | CLBlast: development | \n",
" 4.345092 | \n",
" 24.338693 | \n",
" 7.100451 | \n",
" 6.485059 | \n",
"
\n",
" \n",
" | OpenBLAS: v0.2.18 | \n",
" 25.476068 | \n",
" 45.175774 | \n",
" 26.410401 | \n",
" 3.554530 | \n",
"
\n",
" \n",
" | clBLAS: v2.4 | \n",
" 8.033592 | \n",
" 6.873930 | \n",
" 1.971404 | \n",
" 0.956751 | \n",
"
\n",
" \n",
"
\n",
"