{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## JSON"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# JSON - Javascript Object Notation\n",
"#### Invented by Douglas Crockford when working at Yahoo in early 2000s.\n",
"\n",
"* Goal - Human Readable, Machine Parsable\n",
"\n",
"* Specification: https://www.json.org/"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"JSON — short for JavaScript Object Notation — format for sharing data. \n",
"\n",
"JSON is derived from the JavaScript programming language\n",
"\n",
"Available for use by many languages including Python \n",
"\n",
"usually file extension is .json when stored\n",
"\n"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {},
"outputs": [],
"source": [
"# Sample JSON below from https://json.org/example.html\n",
"# Question why is Syntax highlighting working properly ? :)"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'widget': {'debug': 'on',\n",
" 'image': {'alignment': 'center',\n",
" 'hOffset': 250,\n",
" 'name': 'sun1',\n",
" 'src': 'Images/Sun.png',\n",
" 'vOffset': 250},\n",
" 'text': {'alignment': 'center',\n",
" 'data': 'Click Here',\n",
" 'hOffset': 250,\n",
" 'name': 'text1',\n",
" 'onMouseUp': 'sun1.opacity = (sun1.opacity / 100) * 90;',\n",
" 'size': 36,\n",
" 'style': 'bold',\n",
" 'vOffset': 100},\n",
" 'window': {'height': 500,\n",
" 'name': 'main_window',\n",
" 'title': 'Sample Konfabulator Widget',\n",
" 'width': 500}}}"
]
},
"execution_count": 1,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"{\"widget\": {\n",
" \"debug\": \"on\",\n",
" \"window\": {\n",
" \"title\": \"Sample Konfabulator Widget\",\n",
" \"name\": \"main_window\",\n",
" \"width\": 500,\n",
" \"height\": 500\n",
" },\n",
" \"image\": { \n",
" \"src\": \"Images/Sun.png\",\n",
" \"name\": \"sun1\",\n",
" \"hOffset\": 250,\n",
" \"vOffset\": 250,\n",
" \"alignment\": \"center\"\n",
" },\n",
" \"text\": {\n",
" \"data\": \"Click Here\",\n",
" \"size\": 36,\n",
" \"style\": \"bold\",\n",
" \"name\": \"text1\",\n",
" \"hOffset\": 250,\n",
" \"vOffset\": 100,\n",
" \"alignment\": \"center\",\n",
" \"onMouseUp\": \"sun1.opacity = (sun1.opacity / 100) * 90;\"\n",
" }\n",
"}} \n"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [],
"source": [
"# if this was string starting with { it would be our json\n",
"mydata = {\n",
" \"firstName\": \"Jane\",\n",
" \"lastName\": \"Doe\",\n",
" \"hobbies\": [\"running\", \"sky diving\", \"dancing\"],\n",
" \"age\": 43,\n",
" \"children\": [\n",
" {\n",
" \"firstName\": \"Alice\",\n",
" \"age\": 7\n",
" },\n",
" {\n",
" \"firstName\": \"Bob\",\n",
" \"age\": 13\n",
" }\n",
" ]\n",
"}"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(mydata)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'age': 43,\n",
" 'children': [{'age': 7, 'firstName': 'Alice'},\n",
" {'age': 13, 'firstName': 'Bob'}],\n",
" 'firstName': 'Jane',\n",
" 'hobbies': ['running', 'sky diving', 'dancing'],\n",
" 'lastName': 'Doe'}"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mydata"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The process of encoding JSON is usually called serialization. This term refers to the transformation of data into a series of bytes (hence serial) to be stored or transmitted across a network. You may also hear the term marshaling, but that’s a whole other discussion. Naturally, deserialization is the reciprocal process of decoding data that has been stored or delivered in the JSON standard.\n",
"\n",
"All we’re talking about here is reading and writing. Think of it like this: encoding is for writing data to disk, while decoding is for reading data into memory.\n",
" https://realpython.com/python-json/"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {},
"outputs": [],
"source": [
"import json"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {},
"outputs": [],
"source": [
"with open(\"data_file.json\", \"w\") as write_file:\n",
" json.dump(mydata, write_file)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'{\"firstName\": \"Jane\", \"lastName\": \"Doe\", \"hobbies\": [\"running\", \"sky diving\", \"dancing\"], \"age\": 43, \"children\": [{\"firstName\": \"Alice\", \"age\": 7}, {\"firstName\": \"Bob\", \"age\": 13}]}'"
]
},
"execution_count": 7,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# use json string in our program\n",
"json_string = json.dumps(mydata)\n",
"json_string"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"str"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(json_string)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Avove example JSON and Python object have the same syntax but there are some differences"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Simple Python objects are translated to JSON according to a fairly intuitive conversion.\n",
"\n",
"Python\tJSON\n",
"\n",
"dict\tobject\n",
"\n",
"list, tuple\tarray\n",
"\n",
"str\tstring\n",
"\n",
"int, long, \n",
"\n",
"float\tnumber\n",
"\n",
"True\ttrue\n",
"\n",
"False\tfalse\n",
"\n",
"None\tnull"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'{\"firstName\": \"Jane\", \"lastName\": \"Doe\", \"hobbies\": [\"running\", \"sky diving\", \"dancing\"], \"age\": 43, \"children\": [{\"firstName\": \"Alice\", \"age\": 7}, {\"firstName\": \"Bob\", \"age\": 13}]}'"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# The first option most people want to change is whitespace. You can use the indent keyword argument to specify the indentation size for nested structures. Check out the difference for yourself by using data, which we defined above, and running the following commands in a console:\n",
"\n",
"json.dumps(mydata)\n"
]
},
{
"cell_type": "code",
"execution_count": 10,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{\n",
" \"firstName\": \"Jane\",\n",
" \"lastName\": \"Doe\",\n",
" \"hobbies\": [\n",
" \"running\",\n",
" \"sky diving\",\n",
" \"dancing\"\n",
" ],\n",
" \"age\": 43,\n",
" \"children\": [\n",
" {\n",
" \"firstName\": \"Alice\",\n",
" \"age\": 7\n",
" },\n",
" {\n",
" \"firstName\": \"Bob\",\n",
" \"age\": 13\n",
" }\n",
" ]\n",
"}\n"
]
}
],
"source": [
"# very useful for visibility!\n",
"print(json.dumps(mydata, indent=4))"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [],
"source": [
"with open(\"data_file.json\", \"w\") as write_file:\n",
" json.dump(mydata, write_file, indent=4)"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'age': 43,\n",
" 'children': [{'age': 7, 'firstName': 'Alice'},\n",
" {'age': 13, 'firstName': 'Bob'}],\n",
" 'firstName': 'Jane',\n",
" 'hobbies': ['running', 'sky diving', 'dancing'],\n",
" 'lastName': 'Doe'}"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"with open(\"data_file.json\", \"r\") as read_file:\n",
" data = json.load(read_file)\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict"
]
},
"execution_count": 13,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(data)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Keep in mind that the result of this method could return any of the allowed data types from the conversion table. This is only important if you’re loading in data you haven’t seen before. In most cases, the root object will be a dict or a list."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"If you've gotten JSON data in from another program or have otherwise obtained a string of JSON formatted data in Python, you can easily deserialize that with loads(), which naturally loads from a string:"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{'researcher': {'name': 'Ford Prefect',\n",
" 'relatives': [{'name': 'Zaphod Beeblebrox', 'species': 'Betelgeusian'}],\n",
" 'species': 'Betelgeusian'}}"
]
},
"execution_count": 14,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"json_string = \"\"\"\n",
"{\n",
" \"researcher\": {\n",
" \"name\": \"Ford Prefect\",\n",
" \"species\": \"Betelgeusian\",\n",
" \"relatives\": [\n",
" {\n",
" \"name\": \"Zaphod Beeblebrox\",\n",
" \"species\": \"Betelgeusian\"\n",
" }\n",
" ]\n",
" }\n",
"}\n",
"\"\"\"\n",
"data = json.loads(json_string)\n",
"data"
]
},
{
"cell_type": "code",
"execution_count": 15,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict"
]
},
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(data)"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'Zaphod Beeblebrox'"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"data['researcher']['relatives'][0]['name']"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [],
"source": [
"import json\n",
"import requests"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
"## Lets get some data https://jsonplaceholder.typicode.com/"
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"response = requests.get(\"https://jsonplaceholder.typicode.com/todos\")\n",
"todos = json.loads(response.text)\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"can open https://jsonplaceholder.typicode.com/todos in regular browser too.."
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"list"
]
},
"execution_count": 20,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(todos)"
]
},
{
"cell_type": "code",
"execution_count": 27,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"200"
]
},
"execution_count": 27,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"len(todos)"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[{'completed': False, 'id': 1, 'title': 'delectus aut autem', 'userId': 1},\n",
" {'completed': False,\n",
" 'id': 2,\n",
" 'title': 'quis ut nam facilis et officia qui',\n",
" 'userId': 1},\n",
" {'completed': False, 'id': 3, 'title': 'fugiat veniam minus', 'userId': 1},\n",
" {'completed': True, 'id': 4, 'title': 'et porro tempora', 'userId': 1},\n",
" {'completed': False,\n",
" 'id': 5,\n",
" 'title': 'laboriosam mollitia et enim quasi adipisci quia provident illum',\n",
" 'userId': 1},\n",
" {'completed': False,\n",
" 'id': 6,\n",
" 'title': 'qui ullam ratione quibusdam voluptatem quia omnis',\n",
" 'userId': 1},\n",
" {'completed': False,\n",
" 'id': 7,\n",
" 'title': 'illo expedita consequatur quia in',\n",
" 'userId': 1},\n",
" {'completed': True,\n",
" 'id': 8,\n",
" 'title': 'quo adipisci enim quam ut ab',\n",
" 'userId': 1},\n",
" {'completed': False,\n",
" 'id': 9,\n",
" 'title': 'molestiae perspiciatis ipsa',\n",
" 'userId': 1},\n",
" {'completed': True,\n",
" 'id': 10,\n",
" 'title': 'illo est ratione doloremque quia maiores aut',\n",
" 'userId': 1}]"
]
},
"execution_count": 21,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"todos[:10]"
]
},
{
"cell_type": "code",
"execution_count": 23,
"metadata": {},
"outputs": [],
"source": [
"myl = [('Valdis', 40), ('Alice',35), ('Bob', 23),('Carol',70)]"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Lambda = anonymous function"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"def myfun(el):\n",
" return el[1]\n",
"# same as myfun = lambda el: el[1]"
]
},
{
"cell_type": "code",
"execution_count": 28,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[('Carol', 70), ('Valdis', 40), ('Alice', 35), ('Bob', 23)]"
]
},
"execution_count": 28,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"sorted(myl, key = lambda el: el[1], reverse=True)"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# Exercise find out top 3 users with most tasks completed!\n",
"\n",
"# TIPS\n",
"# we need some sort of structure to store these user results before finding out top 3\n",
"# at least two good data structure choices here :)\n",
"# here the simplest might actually be the best if we consider userId values\n"
]
},
{
"cell_type": "code",
"execution_count": 29,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"(200, [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1])"
]
},
"execution_count": 29,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"users = [ el['userId'] for el in todos]\n",
"len(users),users[:15]"
]
},
{
"cell_type": "code",
"execution_count": 30,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}"
]
},
"execution_count": 30,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"uniqusers = set(users)\n",
"uniqusers"
]
},
{
"cell_type": "code",
"execution_count": 32,
"metadata": {},
"outputs": [],
"source": [
"# dictionary comprehension but could live without one\n",
"users = { el['userId'] : 0 for el in todos} "
]
},
{
"cell_type": "code",
"execution_count": 33,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"{1: 0, 2: 0, 3: 0, 4: 0, 5: 0, 6: 0, 7: 0, 8: 0, 9: 0, 10: 0}"
]
},
"execution_count": 33,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"users"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_keys([1, 2, 3, 4, 5, 6, 7, 8, 9, 10])"
]
},
"execution_count": 35,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"users.keys()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"users.value"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"#{'completed': True,\n",
"# 'id': 8,\n",
"# 'title': 'quo adipisci enim quam ut ab',\n",
"# 'userId': 1}"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [],
"source": [
"#idiomatic\n",
"for el in todos:\n",
" users[el['userId']] += el['completed'] # Boolean False is 0 True is 1 obviously this might not be too readable"
]
},
{
"cell_type": "code",
"execution_count": 35,
"metadata": {},
"outputs": [],
"source": [
"# same as above could be useful in more complicated cases\n",
"for el in todos:\n",
" if el['completed'] == True:\n",
" users[el['userId']] += 1"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# there could be a one liner or a solution with from collections import Counter"
]
},
{
"cell_type": "code",
"execution_count": 36,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"dict_items([(1, 11), (2, 8), (3, 7), (4, 6), (5, 12), (6, 6), (7, 9), (8, 11), (9, 8), (10, 12)])"
]
},
"execution_count": 36,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"users.items()"
]
},
{
"cell_type": "code",
"execution_count": 39,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(1, 11),\n",
" (2, 8),\n",
" (3, 7),\n",
" (4, 6),\n",
" (5, 12),\n",
" (6, 6),\n",
" (7, 9),\n",
" (8, 11),\n",
" (9, 8),\n",
" (10, 12)]"
]
},
"execution_count": 39,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"list(users.items())"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [],
"source": [
"userlist=list(users.items())"
]
},
{
"cell_type": "code",
"execution_count": 38,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"tuple"
]
},
"execution_count": 38,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(userlist[0])"
]
},
{
"cell_type": "code",
"execution_count": 40,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[(5, 12), (10, 12), (1, 11)]"
]
},
"execution_count": 40,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# we pass a key anonymous(lambda) function\n",
"sorted(userlist, key=lambda el: el[1], reverse=True)[:3]"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {},
"outputs": [],
"source": [
"# lets try a simple way"
]
},
{
"cell_type": "code",
"execution_count": 48,
"metadata": {},
"outputs": [],
"source": [
"mylist=[0]\n",
"mylist*=11"
]
},
{
"cell_type": "code",
"execution_count": 49,
"metadata": {},
"outputs": [],
"source": [
"for el in todos:\n",
" if el['completed'] == True:\n",
" mylist[el['userId']] +=1"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"[0, 11, 8, 7, 6, 12, 6, 9, 11, 8, 12]"
]
},
"execution_count": 50,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mylist"
]
},
{
"cell_type": "code",
"execution_count": 51,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"5"
]
},
"execution_count": 51,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"mylist.index(max(mylist))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# kind of hard to get more values need to get tricky"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": [
"# "
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.6.4"
}
},
"nbformat": 4,
"nbformat_minor": 2
}