{ "cells": [ { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Welcome to the SETI Institute Code Challenge!\n", "\n", "This first tutorial will explain a little bit on what the data is and where to get it.\n", "\n", "# Introduction\n", "\n", "For the Code Challenge, you will be using the **\"primary\" data set**, as we've called it. The primary data set is \n", "\n", " * labeled data set of 350,000 simulated signals\n", " * 7 different labels, or \"signal classifications\"\n", " * total of about 128 GB of data\n", " \n", "This data set should be used to train your models. \n", "\n", "**You do not need to use all the data to train your models** if you do not want to or need to consume the entire set. \n", "\n", "There are also a **`small` and a `medium` sized subset** of these primary data files. \n", "\n", "\n", "## Simple Data Format\n", "\n", "Each data file has a simple format: \n", "\n", " * file name = .dat\n", " * a JSON header in the first line that contains:\n", " * UUID\n", " * signal_classification (label)\n", " * followed by stream complex-valued time-series data. \n", "\n", "The `ibmseti` Python package is available to assist in reading this data and performing some basic operations for you. \n", "\n", "## Basic Warmup Data Set.\n", "\n", "There is also a second, simple and clean data set that you may use for warmup, which we call the **\"basic\" data set**. This basic set should be used as a sanity check and for very early-stage prototyping. We recommend that everybody starts with this. \n", "\n", " * Only 4 different signal classifications\n", " * 1000 simulation files for each class: 4000 files total\n", " * Available as single zip file\n", " * ~1 GB in total. \n", " \n", "### Basic Set versus Primary Set\n", "\n", "> The difference between the `basic` and `primary` data sets is that the signals simulated in the `basic` set have, on average, much higher signal to noise ratio (they are larger amplitude signals). They also have other characteristics that will make the different signal classes very distinguishable. **You should be able to get very high signal classification accuracy with the basic data set.** The primary data set has smaller amplitude signals and can look more similar to each other, making classification accuracy more difficult with this data set. There are also only 4 classes in the basic data set and 7 classes in the primary set. \n", "\n", "\n", "## Primary Data Sets\n", "\n", "### Primary Small\n", "\n", "The `primary small` is a subset of the full primary data set. Use for early-stage prototyping.\n", "\n", " * All 7 signal classifications\n", " * 1000 simulations / class (7 classes = 7000 files)\n", " * Available as single zip file\n", " * ~2 GB in total\n", "\n", "### Primary Medium\n", "\n", "The `primary medium` is a subset of the full primary data set. Use for early-stage prototyping & model building.\n", "\n", " * All 7 signal classifications\n", " * 10000 simulations / class (7 classes = 70000 files)\n", " * Large enough for relatively robust model construction\n", " * Available in 6 separate zip files\n", " * ~20 GB in total\n", " \n", "### Primary Full\n", "\n", "The `primary full` is the entire primary data set. Use only if you want an enourmous training data set. You will need a small data center to process these data in a reasonable amount of time. \n", "\n", " * All 7 signal classifications\n", " * 50000 simulations / class (7 classes = 350000 files)\n", " * Only available in 350k individual files\n", " * one must read through the index file and download files individually, which will take some time from outside of IBM Cloud systems\n", " * ~130 GB in total\n", " \n", "\n", "## Index Files\n", "\n", "For all data sets, there exists an **index** file. That file is a CSV file. Each row holds the UUID, signal_classification (label) for a simulation file in the data set. You can use these index files in a few different ways (from using to keep track of your downloads, to facilitate parallelization of your analysis on Spark). \n", "\n", "\n", "\n", "## Direct Data URLs if you are working from outside of IBM Data Science Experience\n", "\n", "### Basic4\n", "Data: https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_basic_v2/basic4.zip\n", "\n", "Index: \n", "https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_files/public_list_basic_v2_26may_2017.csv\n", "\n", "\n", "### Primary Small\n", "Data: https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_small.zip\n", "\n", "Index: \n", "https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_files/public_list_primary_v2_small_1june_2017.csv\n", "\n", "### Primary Medium\n", "\n", "Data:\n", "1. https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_medium_1.zip\n", "\n", "2. https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_medium_2.zip\n", "\n", "3. https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_medium_3.zip\n", "\n", "4. https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_medium_4.zip\n", "\n", "5. https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_medium_5.zip\n", "\n", "6. https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_medium_6.zip\n", "\n", "Index: https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_files/public_list_primary_v2_medium_1june_2017.csv\n", "\n", "## Test Data Set\n", "\n", "There is one `primary_test` data set. Each data file is the same as the above training data except the JSON header does NOT contain the 'signal_classification' key. \n", "\n", " * All 7 classes\n", " * Roughly 1000 simulations per class (+- 50) (7014 total files)\n", " * JSON header with UUID only\n", " * Available as single zip file\n", " * ~2 GB in total\n", "\n", "### Direct Download Link\n", "\n", "Data: https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2_zipped/primary_testset.zip\n", "\n", "Index: https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_files/public_list_primary_testset_1k_1june_2017.csv\n", "\n", "### Submitting Classification Results\n", "\n", "See the [Judging Criteria](../Judging_Criteria.ipynb) notebook for information on submitting your test-set classifications." ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Programmatically Accessing the Data\n", "\n", "The data are stored in `containers` on IBM Object Storage. You can access these data with HTTP calls. Here we use system level `curl`, but you could easily use the Python `requests` package. \n", "\n", "The URL for all data files is composed of\n", "\n", " `base_url/container/objectname`.\n", " \n", "The `base_url` is:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "#If you are running this in IBM Apache Spark (via Data Science Experience)\n", "base_url = 'https://dal05.objectstorage.service.networklayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b'\n", "\n", "#ELSE, if you are outside of IBM:\n", "#base_url = 'https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b'\n", "\n", "#NOTE: if you are outside of IBM, pulling down data will be slower. :/" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "#Defining a local data folder to dump data\n", "\n", "import os\n", "\n", "mydatafolder = os.path.join( os.environ['PWD'], 'my_data_folder' )\n", "if os.path.exists(mydatafolder) is False:\n", " os.makedirs(mydatafolder)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Basic Data Set\n", "\n", "We'll start with the basic data set. Because the basic data set is small, we've created a `.zip` file of the full data set that you can download directly. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "import os" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "basic_container = 'simsignals_basic_v2'\n", "basic4_zip_file = 'basic4.zip'" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "os.system('curl {}/{}/{} > {}'.format(base_url, basic_container, basic4_zip_file, mydatafolder + '/' + basic4_zip_file))" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "!ls -al my_data_folder/basic4.zip" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "## Primary Data Set\n", "\n", "\n", "### Primary Small\n", "\n", "The `primary_small` subset can be found in a zip file in\n", "* contianer = 'simignals_v2_zipped'\n", "* objectname = 'primary_small.zip'" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "filename = 'primary_small.zip'\n", "primary_small_url = '{}/simsignals_v2_zipped/{}'.format(base_url, filename)\n", "os.system('curl {} > {}'.format(primary_small_url, mydatafolder +'/'+filename))" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "###### Primary Small Index File\n", "\n", "A CSV file containing the UUID, signal classifications for each file in the `primary_small` subset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "filename = 'public_list_primary_v2_small_1june_2017.csv'\n", "primary_small_csv_url = '{}/simsignals_files/{}'.format(base_url, filename)\n", "os.system('curl {} > {}'.format(primary_small_csv_url, mydatafolder +'/'+filename))" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### Primary Medium\n", "\n", "Similarly, the `primary_medium` subset can be found in a handful of zip files\n", "\n", "* contianer = 'simignals_v2_zipped'\n", "* objectname = 'primary_medium_N.zip'\n", "* for N = 1, 2, 3, 4, 5, 6" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "med_N = '{}/simsignals_v2_zipped/primary_medium_{}.zip'\n", "\n", "for i in range(1,7):\n", " med_url = med_N.format(base_url, i)\n", " output_file = mydatafolder + '/primary_medium_{}.zip'.format(i)\n", " print 'GETing', output_file\n", " os.system('curl {} > {}'.format(med_url, output_file ))" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "###### Primary Medium Index File\n", "\n", "Here too, there is a CSV file containing the UUID, signal classifications for each file in the `primary_medium` subset." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "filename = 'public_list_primary_v2_medium_1june_2017.csv'\n", "med_csv_url = '{}/simsignals_files/{}'.format(base_url, filename)\n", "os.system('curl {} > {}'.format(med_csv_url, mydatafolder +'/'+filename))" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### Primary Full set\n", "\n", "Because the full set is so incredibly large, we only have these 350,000 files available individually on object storage. \n", "\n", "The `primary_full` list can be found here: " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "filename = 'public_list_primary_v2_full_1june_2017.csv'\n", "prim_full = '{}/simsignals_files/{}'.format(base_url, filename)\n", "os.system('curl {} > {}'.format(prim_full, mydatafolder +'/'+filename))" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "One can download this list and begin to pull down files individually if desired. Warning, **however, this will take approximately a billion years if you are not running on IBM Apache Spark** -- IBM Apache Spark and Object Storage exist in the same data center and share a fast network connection. \n", "\n", "The data are found in \n", "\n", "`base_url/simsignals_v2/.dat`\n", "\n", "For example:\n", "\n", "https://dal.objectstorage.open.softlayer.com/v1/AUTH_cdbef52bdf7a449c96936e1071f0a46b/simsignals_v2/aa7d082f-9263-4533-a9d4-5595c5cdde25.dat\n", "\n", "**We are working to make the primary full data set more easily, as this current setup is less than ideal. You will be notified if that becomes available. The data will be directly available for participants of the hackathon, however.**\n", "\n", "If you wish to programmatically begin to download the full data set you may use the following code." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "import requests\n", "import copy" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "file_list_container = 'simsignals_files'\n", "file_list = 'public_list_primary_v2_full_1june_2017.csv'\n", "primary_data_container = 'simsignals_v2'" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "r = requests.get('{}/{}/{}'.format(base_url, file_list_container, file_list), timeout=(9.0, 21.0))\n", "filecontents = copy.copy(r.content)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "full_primary_files = [line.split(',') for line in filecontents.split('\\n')]\n", "full_primary_files = full_primary_files[1:-1] #strip the header and empty last element\n", "full_primary_files = map(lambda x: x[0]+\".dat\", full_primary_files) #now list of file names (.dat)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "#save your data into a local subfolder\n", "save_to_folder = mydatafolder + '/primary_data_set'\n", "if os.path.exists(save_to_folder) is False:\n", " os.mkdir(save_to_folder)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true, "scrolled": true }, "outputs": [], "source": [ "count = 0\n", "total = len(full_primary_files)\n", "for row in full_primary_files:\n", " r = requests.get('{}/{}/{}'.format(base_url, primary_data_container, row), timeout=(9.0, 21.0))\n", " \n", " if count % 100 == 0:\n", " print 'done ', count, ' out of ', total\n", " count += 1\n", " \n", " with open('{}/{}'.format(save_to_folder, row), 'w' ) as fout:\n", " fout.write(r.content)" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "### This is really a lot of data\n", "\n", "This will be a difficult data set to consume and process if you are using free-tier levels of software from any Cloud provider. You will likely want to have a robust machine, or sets of machines, with many threads and GPUs if you want to train models with such a large dat set. \n", "\n", "For example, if you have access to an IBM Spark Enterprise cluster, because the network connection between IBM Spark and IBM Object Storage is so fast, we recommend that you **do NOT** download each file. Instead you could parallelize the index file and then retrieve and process each file on a worker node. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "## Using Spark -- can parallelize the job across your worker nodes\n", "import ibmseti\n", "def retrieve_and_process(row):\n", " try:\n", " r = requests.get('{}/{}/{}'.format(base_url, primary_data_container, row), timeout=(9.0, 21.0))\n", " except Exception as e:\n", " return (row, 'failed', [])\n", " \n", " aca = ibmseti.compamp.SimCompamp(r.content)\n", " spectrogram = aca.get_spectrogram() # or do something else\n", " features = my_feature_extractor(spectrogram) #example external function for reducing the spectrogram into a handful of features, perhaps\n", " \n", " signal_class = aca.header()['signal_classifiation']\n", " \n", " return (row, signal_class, features)\n", "\n", "npartitions = 60 \n", "rdd = sc.parallelize(full_primary_files, npartitions)\n", "\n", "#Now ask Spark to run the job\n", "process_results = rdd.map(retrieve_and_process).collect()" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "# Test Data Set\n", "\n", "Once you've trained your model, done all of your testing, and tweaks and are ready to submit an entry to the contest, you'll need to download the test data set and apply your model to that. \n", "\n", "The test data set is similar to the labeled data, except that the JSON header is missing the 'signal_classification' key, and just contains the 'uuid'. \n", "\n", "Like the other sets, this set is found in a `.zip` file in the `simsignals_v2_zipped` container;" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "deletable": true, "editable": true }, "outputs": [], "source": [ "filename = 'primary_testset.zip'\n", "test_set_url = '{}/simsignals_v2_zipped/{}'.format(base_url, filename)\n", "os.system('curl {} > {}'.format(test_set_url, mydatafolder +'/'+filename))" ] }, { "cell_type": "markdown", "metadata": { "deletable": true, "editable": true }, "source": [ "There are approximately 1000 simulations of each of the 7 signal classes -- but not exactly 1000 (+- some largeish number) so you can't cheat :). \n", "\n", "See the [Judging Criteria document](https://github.com/setiQuest/ML4SETI/blob/master/JudgingCriteria.ipynb) for more details." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "deletable": true, "editable": true }, "outputs": [], "source": [ "filename = 'public_list_primary_testset_1k_1june_2017.csv'\n", "test_set_csv_url = '{}/simsignals_files/{}'.format(base_url, filename)\n", "os.system('curl {} > {}'.format(test_set_csv_url, mydatafolder + '/' + filename))" ] } ], "metadata": { "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.13" } }, "nbformat": 4, "nbformat_minor": 2 }