![](\"https://github.com/ibm-watson-data-lab/pixiedust/raw/master/docs/_static/pd_icon32.png\")
\n",
" \n",
" Pixiedust version 1.1.11\n",
" \n",
" \n",
" Pixiedust version 1.1.11\n",
" ![](\"https://github.com/ibm-cds-labs/pixiedust_node/raw/master/docs/_images/pdn_icon32.png\")
\n",
" \n",
" Pixiedust Node.js \n",
" \"Imagination is more important than knowledge\"\n", "Albert Einstein" ], "text/plain": [ "
\"Imagination is more important than knowledge\"\\nAlbert Einstein';\n", "html(str)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "If we have an image we want to render, we can do that with the `image` function:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
![](\"https://github.com/IBM/nodejs-in-notebooks/blob/master/notebooks/images/pixiedust_node_schematic.png?raw=true\")
"
],
"text/plain": [
"![](\"https://github.com/IBM/nodejs-in-notebooks/blob/master/notebooks/images/display_sin_cos.png?raw=true\")
\n",
"\n",
"PixieDust presents visualisations of DataFrames using Matplotlib, Bokeh, Brunel, d3, Google Maps and, MapBox. No code is required on your part because PixieDust presents simple pull-down menus and a friendly point-and-click interface, allowing you to configure how the data is presented:\n",
"\n",
"![](\"https://github.com/IBM/nodejs-in-notebooks/blob/master/notebooks/images/pd_chart_types.png?raw=true\")
"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Adding npm modules\n",
"There are thousands of libraries and tools in the npm repository, Node.js’s package manager. It’s essential that we can install npm libraries and use them in our notebook code.\n",
"Let’s say we want to make some HTTP calls to an external API service. We could deal with Node.js’s low-level HTTP library, or an easier option would be to use the ubiquitous `request` npm module.\n",
"Once we have pixiedust_node set up, installing an npm module is as simple as running `npm.install` in a Python cell:"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/opt/conda/envs/DSX-Python27/bin/npm install -s request\n",
"+ request@2.87.0\n",
"updated 1 package in 0.856s\n"
]
}
],
"source": [
"npm.install('request');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Once installed, you may `require` the module in your JavaScript code:"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"... ... ... ...\n",
"... ...\n",
"{ iss_position: { longitude: '42.2119', latitude: '51.1124' },\n",
"timestamp: 1531779053,\n",
"message: 'success' }\n"
]
}
],
"source": [
"%%node\n",
"var request = require('request');\n",
"var r = {\n",
" method:'GET',\n",
" url: 'http://api.open-notify.org/iss-now.json',\n",
" json: true\n",
"};\n",
"request(r, function(err, req, body) {\n",
" console.log(body);\n",
"});\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As an HTTP request is an asynchronous action, the `request` library calls our callback function when the operation has completed. Inside that function, we can call print to render the data.\n",
"We can organise our code into functions to encapsulate complexity and make it easier to reuse code. We can create a function to get the current position of the International Space Station in one notebook cell:"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"... ..... ..... ..... ..... ... ..... ..... ....... ....... ....... ....... ....... ..... ..... ...\n"
]
}
],
"source": [
"%%node\n",
"var request = require('request');\n",
"var getPosition = function(callback) {\n",
" var r = {\n",
" method:'GET',\n",
" url: 'http://api.open-notify.org/iss-now.json',\n",
" json: true\n",
" };\n",
" request(r, function(err, req, body) {\n",
" var obj = null;\n",
" if (!err) {\n",
" obj = body.iss_position\n",
" obj.latitude = parseFloat(obj.latitude);\n",
" obj.longitude = parseFloat(obj.longitude);\n",
" obj.time = new Date().getTime(); \n",
" }\n",
" callback(err, obj);\n",
" });\n",
"};"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And use it in another cell:"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"... ...\n",
"{ longitude: 42.9493, latitude: 51.1819, time: 1531779061073 }\n"
]
}
],
"source": [
"%%node\n",
"getPosition(function(err, data) {\n",
" console.log(data);\n",
"});"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Promises\n",
"If you prefer to work with JavaScript Promises when writing asynchronous code, then that’s okay too. Let’s rewrite our `getPosition` function to return a Promise. First we're going to install the `request-promise` module from npm:"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/opt/conda/envs/DSX-Python27/bin/npm install -s request request-promise\n",
"+ request@2.87.0\n",
"+ request-promise@4.2.2\n",
"updated 2 packages in 0.912s\n"
]
}
],
"source": [
"npm.install( ('request', 'request-promise') )"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Notice how you can install multiple modules in a single call. Just pass in a Python `list` or `tuple`.\n",
"Then we can refactor our function a little:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"... ..... ..... ..... ..... ... ..... ..... ..... ..... ..... ..... ..... ...\n"
]
}
],
"source": [
"%%node\n",
"var request = require('request-promise');\n",
"var getPosition = function(callback) {\n",
" var r = {\n",
" method:'GET',\n",
" url: 'http://api.open-notify.org/iss-now.json',\n",
" json: true\n",
" };\n",
" return request(r).then(function(body) {\n",
" var obj = null;\n",
" obj = body.iss_position;\n",
" obj.latitude = parseFloat(obj.latitude);\n",
" obj.longitude = parseFloat(obj.longitude);\n",
" obj.time = new Date().getTime(); \n",
" return obj;\n",
" });\n",
"};"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"And call it in the Promises style:"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"... ... ... ...\n",
"{ longitude: 44.0843, latitude: 51.2787, time: 1531779072259 }\n"
]
}
],
"source": [
"%%node\n",
"getPosition().then(function(data) {\n",
" console.log(data);\n",
"}).catch(function(err) {\n",
" console.error(err); \n",
"});"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or call it in a more compact form:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{ longitude: 44.6288, latitude: 51.3208, time: 1531779077984 }\n"
]
}
],
"source": [
"%%node\n",
"getPosition().then(console.log).catch(console.error);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"In the next part of this notebook we'll illustrate how you can access local and remote data sources from within the notebook."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"***\n",
"# Part 2: Working with data sources\n",
"\n",
"You can access any data source using your favorite public or home-grown packages. In the second part of this notebook you'll learn how to retrieve data from an Apache CouchDB (or Cloudant) database and visualize it using PixieDust or third-party libraries.\n",
"\n",
"## Accessing Cloudant data sources\n",
"\n",
"\n",
"To access data stored in an Apache CouchDB or Cloudant database, we can use the [`cloudant-quickstart`](https://www.npmjs.com/package/cloudant-quickstart) npm module:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/opt/conda/envs/DSX-Python27/bin/npm install -s cloudant-quickstart\n",
"+ cloudant-quickstart@1.25.5\n",
"updated 1 package in 0.983s\n"
]
}
],
"source": [
"npm.install('cloudant-quickstart')"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"With our Cloudant URL, we can start exploring the data in Node.js. First we make a connection to the remote Cloudant database:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"%%node\n",
"// connect to Cloudant using cloudant-quickstart\n",
"const cqs = require('cloudant-quickstart');\n",
"const cities = cqs('https://56953ed8-3fba-4f7e-824e-5498c8e1d18e-bluemix.cloudant.com/cities');"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"> For this code pattern example a remote database has been pre-configured to accept anonymous connection requests. If you wish to explore the `cloudant-quickstart` library beyond what is covered in this nodebook, we recommend you create your own replica and replace above URL with your own, e.g. `https://myid:mypassword@mycloudanthost/mydatabase`.\n",
"\n",
"Now we have an object named `cities` that we can use to access the database. \n",
"\n",
"### Exploring the data using Node.js in a notebook \n",
"\n",
"We can retrieve all documents using `all`."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ { _id: '1000501',\n",
"name: 'Grahamstown',\n",
"latitude: -33.30422,\n",
"longitude: 26.53276,\n",
"country: 'ZA',\n",
"population: 91548,\n",
"timezone: 'Africa/Johannesburg' },\n",
"{ _id: '1000543',\n",
"name: 'Graaff-Reinet',\n",
"latitude: -32.25215,\n",
"longitude: 24.53075,\n",
"country: 'ZA',\n",
"population: 62896,\n",
"timezone: 'Africa/Johannesburg' },\n",
"{ _id: '100077',\n",
"name: 'Abū Ghurayb',\n",
"latitude: 33.30563,\n",
"longitude: 44.18477,\n",
"country: 'IQ',\n",
"population: 900000,\n",
"timezone: 'Asia/Baghdad' } ]\n"
]
}
],
"source": [
"%%node\n",
"// If no limit is specified, 100 documents will be returned\n",
"cities.all({limit:3}).then(console.log).catch(console.error)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Specifying the optional `limit` and `skip` parameters we can paginate through the document list:\n",
"\n",
"```\n",
"cities.all({limit:10}).then(console.log).catch(console.error)\n",
"cities.all({skip:10, limit:10}).then(console.log).catch(console.error)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If we know the IDs of documents, we can retrieve them singly:"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{ _id: '2636749',\n",
"name: 'Stowmarket',\n",
"latitude: 52.18893,\n",
"longitude: 0.99774,\n",
"country: 'GB',\n",
"population: 15394,\n",
"timezone: 'Europe/London' }\n"
]
}
],
"source": [
"%%node\n",
"cities.get('2636749').then(console.log).catch(console.error);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or in bulk:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {
"scrolled": true
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[ { _id: '5913490',\n",
"name: 'Calgary',\n",
"latitude: 51.05011,\n",
"longitude: -114.08529,\n",
"country: 'CA',\n",
"population: 1019942,\n",
"timezone: 'America/Edmonton' },\n",
"{ _id: '4140963',\n",
"name: 'Washington, D.C.',\n",
"latitude: 38.89511,\n",
"longitude: -77.03637,\n",
"country: 'US',\n",
"population: 601723,\n",
"timezone: 'America/New_York' },\n",
"{ _id: '3520274',\n",
"name: 'Río Blanco',\n",
"latitude: 18.83036,\n",
"longitude: -97.156,\n",
"country: 'MX',\n",
"population: 39543,\n",
"timezone: 'America/Mexico_City' } ]\n"
]
}
],
"source": [
"%%node\n",
"cities.get(['5913490', '4140963','3520274']).then(console.log).catch(console.error);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Instead of just calling `print` to output the JSON, we can bring PixieDust's `display` function to bear by passing it an array of data to visualize. Using mapbox as renderer and satelite as basemap, we can display the location and population of the selected cities: "
]
},
{
"cell_type": "code",
"execution_count": 22,
"metadata": {
"pixiedust": {
"displayParams": {
"basemap": "satellite-v9",
"chartsize": "76",
"coloropacity": "53",
"colorrampname": "Orange to Purple",
"handlerId": "mapView",
"keyFields": "latitude,longitude",
"kind": "simple-cluster",
"legend": "false",
"mapboxtoken": "pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4M29iazA2Z2gycXA4N2pmbDZmangifQ.-g_vE53SD2WrJ6tFX7QHmA",
"rendererId": "mapbox",
"rowCount": "500",
"valueFields": "population,name"
}
},
"scrolled": false
},
"outputs": [],
"source": [
"%%node\n",
"cities.get(['5913490', '4140963','3520274']).then(display).catch(console.error);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"https://github.com/IBM/nodejs-in-notebooks/blob/master/notebooks/images/mapbox_americas.png?raw=true\")
\n",
"We can also query a subset of the data using the `query` function, passing it a [Cloudant Query](https://cloud.ibm.com/docs/services/Cloudant/api/cloudant_query.html#query) statement. Using mapbox as renderer, the customizable output looks as follows:"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {
"pixiedust": {
"displayParams": {
"basemap": "outdoors-v9",
"colorrampname": "Yellow to Blue",
"handlerId": "mapView",
"keyFields": "latitude,longitude",
"mapboxtoken": "pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4M29iazA2Z2gycXA4N2pmbDZmangifQ.-g_vE53SD2WrJ6tFX7QHmA",
"rowCount": "500",
"valueFields": "name,population"
}
},
"scrolled": false
},
"outputs": [],
"source": [
"%%node\n",
"// fetch cities in UK above latitude 54 degrees north\n",
"cities.query({country:'GB', latitude: { \"$gt\": 54}}).then(display).catch(console.error);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"![](\"https://github.com/IBM/nodejs-in-notebooks/blob/master/notebooks/images/mapbox_uk.png?raw=true\")
\n",
"\n",
"### Aggregating data\n",
"The `cloudant-quickstart` library also allows aggregations (sum, count, stats) to be performed in the Cloudant database.\n",
"Let’s calculate the sum of the population field:"
]
},
{
"cell_type": "code",
"execution_count": 24,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"2694222973\n",
"\n"
]
}
],
"source": [
"%%node\n",
"cities.sum('population').then(console.log).catch(console.error);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Or compute the sum of the `population`, grouped by the `country` field, displaying 10 countries with the largest population:"
]
},
{
"cell_type": "code",
"execution_count": 25,
"metadata": {
"pixiedust": {
"displayParams": {
"aggregation": "SUM",
"handlerId": "barChart",
"keyFields": "name",
"mapboxtoken": "pk.eyJ1IjoibWFwYm94IiwiYSI6ImNpejY4M29iazA2Z2gycXA4N2pmbDZmangifQ.-g_vE53SD2WrJ6tFX7QHmA",
"orientation": "vertical",
"rendererId": "google",
"rowCount": "100",
"sortby": "Values DESC",
"valueFields": "population"
}
},
"scrolled": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"... ... ... ..... ..... ... ... ..... ..... ... ... ..... ..... ...\n",
"CN 389,487,480\n",
"IN 269,553,896\n",
"US 190,515,768\n",
"BR 125,426,547\n",
"RU 108,885,695\n",
"JP 99,000,238\n",
"MX 80,474,387\n",
"ID 63,161,801\n",
"DE 58,884,999\n",
"TR 55,733,719\n"
]
}
],
"source": [
"%%node\n",
"\n",
"// helper function\n",
"function top10(data) {\n",
" // convert input data structure to array\n",
" var pop_array = [];\n",
" Object.keys(data).forEach(function(n,k) {\n",
" pop_array.push({name: n, population: data[n]});\n",
" });\n",
" // sort array by population in descending order\n",
" pop_array.sort(function(a,b) {\n",
" return b.population - a.population; \n",
" });\n",
" // display top 10 entries\n",
" pop_array.slice(0,10).forEach(function(e) {\n",
" console.log(e.name + ' ' + e.population.toLocaleString()); \n",
" });\n",
"}\n",
"\n",
"// fetch aggregated data and invoke helper routine\n",
"cities.sum('population','country').then(top10).catch(console.error);"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `cloudant-quickstart` package is just one of several Node.js libraries that you can use to access Apache CouchDB or Cloudant. Follow [this link](https://medium.com/ibm-watson-data-lab/choosing-a-cloudant-library-d14c06f3d714) to learn more about your options. "
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Visualizing data using custom charts\n",
"\n",
"If you prefer, you can also use third-party Node.js charting packages to visualize your data, such as [`quiche`](https://www.npmjs.com/package/quiche)."
]
},
{
"cell_type": "code",
"execution_count": 26,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"/opt/conda/envs/DSX-Python27/bin/npm install -s quiche\n",
"+ quiche@0.3.0\n",
"updated 1 package in 0.957s\n"
]
}
],
"source": [
"npm.install('quiche');"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"... ... ... ... ... ... ... ... ...\n"
]
},
{
"data": {
"text/html": [
"| \n", " | mpg | \n", "cylinders | \n", "engine | \n", "horsepower | \n", "weight | \n", "acceleration | \n", "year | \n", "origin | \n", "name | \n", "
|---|---|---|---|---|---|---|---|---|---|
| 0 | \n", "18.0 | \n", "8 | \n", "307.0 | \n", "130 | \n", "3504 | \n", "12.0 | \n", "70 | \n", "American | \n", "chevrolet chevelle malibu | \n", "
| 1 | \n", "15.0 | \n", "8 | \n", "350.0 | \n", "165 | \n", "3693 | \n", "11.5 | \n", "70 | \n", "American | \n", "buick skylark 320 | \n", "
| 2 | \n", "18.0 | \n", "8 | \n", "318.0 | \n", "150 | \n", "3436 | \n", "11.0 | \n", "70 | \n", "American | \n", "plymouth satellite | \n", "
| 3 | \n", "16.0 | \n", "8 | \n", "304.0 | \n", "150 | \n", "3433 | \n", "12.0 | \n", "70 | \n", "American | \n", "amc rebel sst | \n", "
| 4 | \n", "17.0 | \n", "8 | \n", "302.0 | \n", "140 | \n", "3449 | \n", "10.5 | \n", "70 | \n", "American | \n", "ford torino | \n", "