object connecting to Twitter's API\n",
"client"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Twitter's REST APIs\n",
"\n",
"Twitter has a rich set of API calls (full list is listed at https://dev.twitter.com/rest/public). Today we'll be using these:\n",
"\n",
"* [GET friends/list](https://dev.twitter.com/rest/reference/get/friends/list) - who is user X is following?\n",
"* [GET followers/list](https://dev.twitter.com/rest/reference/get/followers/list) - who follows user X?\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### GET followers/list: let's find out who follows you!\n",
"\n",
"You'll see from the [GET followers/list](https://dev.twitter.com/rest/reference/get/followers/list) documentation that the URL to get the list of followers is:\n",
"\n",
" https://api.twitter.com/1.1/followers/list.json?screen_name=[screen_name]\n",
" \n",
"Which returns:\n",
"\n",
"1) A response body\n",
"\n",
"* JSON representing the data we requested\n",
"\n",
"2) A response header\n",
"\n",
"* There's a lot of stuff here, but one param to note are the **HTTP Response Codes**, which will tell you if the request was successful. Or if not, why. The ones you should note are:\n",
"\n",
"* **200** - **STATUS_OKAY** - Success :) . This is what you want.\n",
"* **429** - **RATE_LIMIT_EXCEEDED**. Uh-oh, slow it down :/. Twitter limits how frequently you can make requests, and you've exceeded it.\n",
"* **401** - **UNAUTHORIZED_USER**. Twitter isn't accepting your Consumer/Access tokens. Verify tokens were pasted correctly, or try generating new tokens.\n",
"\n",
" \n",
" \n",
"Now that we know what to expect, let's try it!"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import json\n",
"\n",
"FOLLOWERS_URL = 'https://api.twitter.com/1.1/followers/list.json'\n",
"\n",
"# TODO: put your twitter handle here\n",
"screen_name = 'my_twitter_handle'\n",
"\n",
"\n",
"url = FOLLOWERS_URL + '?screen_name=' + screen_name\n",
"header, response = client.request(url, method='GET')\n",
"\n",
"# let's save the whole response so you can take a look at it\n",
"with open('../materials/tutorial/my_followers.json', 'w') as f:\n",
" json.dump(json.loads(response), f, indent=2)\n",
" \n",
"print 'status:', header['status'] # should be 200 (STATUS_OKAY)\n",
"print response[:200] # a lot of data!"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"#### Extracting data from JSON result\n",
"\n",
"'my_followers.json' will look like the example below. Let's extract the values in RED:\n",
"\n",
"\n",
"{\n",
" \"previous_cursor\": 0, \n",
" \"previous_cursor_str\": \"0\", \n",
" \"next_cursor\": 1496386282559075381, # use next_cursor to get the next page of results\n",
" \"users\": [\n",
" {\n",
" ...\n",
" \"screen_name\": \"celiala\", # follower 1\n",
" ...\n",
" }, \n",
" {\n",
" ...\n",
" \"screen_name\": \"sarah_guido\", # follower 2\n",
" ...\n",
" }\n",
" ],\n",
" ...\n",
"}\n",
"
\n",
"\n",
"Let's extract **next_cursor** and the list of **followers**:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"data = json.loads(response) # convert JSON string into a dictionary object\n",
"\n",
"next_cursor = data['next_cursor']\n",
"followers = [u['screen_name'] for u in data['users']]\n",
"\n",
"# TODO: run this block to see what's in next_cursor and followers:\n",
"print 'next_cursor:', next_cursor\n",
"print len(followers), 'followers so far:', followers"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Generating the subsequent Twitter API call\n",
"\n",
"To get the next page of results, simply pass next_cursor as your next cursor value:"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"\n",
"# use next_cursor to get next 20 results\n",
"url = FOLLOWERS_URL + '?screen_name=' + screen_name + '&cursor=' + str(next_cursor)\n",
"header, response = client.request(url, method='GET')\n",
"\n",
"if header['status'] == '200': # STATUS_OKAY\n",
" \n",
" data = json.loads(response) # convert JSON to dictionary object\n",
"\n",
" next_cursor = data['next_cursor']\n",
" new_followers = [u['screen_name'] for u in data['users']]\n",
" followers.extend(new_followers)\n",
"\n",
" # save raw JSON\n",
" with open('../materials/tutorial/my_followers.' + str(next_cursor) + '.json', 'w') as f:\n",
" json.dump(data, f, indent=2)\n",
"\n",
" # save followers so far\n",
" with open('../materials/tutorial/my_followers.txt', 'w') as f:\n",
" f.write('\\n'.join(followers))\n",
"\n",
" print 'next_cursor:', next_cursor\n",
" print len(new_followers), 'new followers:', new_followers\n",
"\n",
"else:\n",
" print header, response"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Now we can just pass the screen_name and next_cursor, until we no longer get back a next_cursor.\n",
"\n",
"**Beware of Rate Limiting!** - GET followers/list only allows **15 calls in a 15-min window**. So, you may want to sleep between calls (`time.sleep(seconds_to_sleep)`)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"\n",
"### Twitter Interactive Console\n",
"\n",
"To explore the other API Endpoints, Twitter has a great interactive UI console, where you can tweak the inputs and see the outputs:\n",
"\n",
"https://dev.twitter.com/rest/tools/console\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# At this point, switch to [lesson.ipynb](lesson.ipynb)!!!\n",
"\n",
"Go to lesson iPython notebook: [/notebooks/notebooks/lesson.ipynb](lesson.ipynb)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# After the lesson:\n",
"\n",
"# Visualizations"
]
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import networkx as nx\n",
"\n",
"# we need this 'magic' command to draw graphs inline\n",
"%matplotlib inline \n",
"\n",
"g = nx.Graph()\n",
"\n",
"# let's attach a size attribute to each node to describe how big we want the node to be\n",
"g.add_node(1, {'size': 800})\n",
"g.add_node(2, {'size': 200})\n",
"g.add_node(3, {'size': 200})\n",
"g.add_node(4, {'size': 200})\n",
"g.add_node(5, {'size': 200})\n",
"\n",
"g.add_edge(1,2, { 'thickness': 20})\n",
"g.add_edge(1,3, { 'thickness': 20})\n",
"g.add_edge(1,4, { 'thickness': 20})\n",
"g.add_edge(1,5, { 'thickness': 20})\n",
"g.add_edge(2,3, { 'thickness': 5})\n",
"g.add_edge(3,4, { 'thickness': 5})\n",
"g.add_edge(4,5, { 'thickness': 5})\n",
"g.add_edge(5,2, { 'thickness': 5})\n",
"\n",
"# let's iterate through the nodes and edges and extract the list of node & edge sizes\n",
"node_size = [attribs['size'] for (node, attribs) in g.nodes(data=True)]\n",
"edge_thickness = [attribs['thickness'] for (v_from, v_to, attribs) in g.edges(data=True)]\n",
"\n",
"LIGHT_BLUE = '#A0CBE2'\n",
"\n",
"nx.draw(g, \n",
" node_size = node_size, # node_size can either take a single value (where all nodes will be size N),\n",
" # or a list of values, where Nth list value will be the size for the Nth node\n",
" width = edge_thickness, # similarly, the Nth value corresponds to the width for edge N\n",
" node_color = LIGHT_BLUE,\n",
" edge_color = LIGHT_BLUE,\n",
" font_size = 15,\n",
" with_labels = True\n",
")"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"import networkx as nx\n",
"import matplotlib.pyplot as plt\n",
"\n",
"edgelist_txt = '../../data/retweets.txt'\n",
"G = nx.read_edgelist(edgelist_txt, create_using=nx.DiGraph())"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "code",
"collapsed": false,
"input": [
"LAYOUTS = {\n",
" 'circular': nx.circular_layout,\n",
" 'fr': nx.fruchterman_reingold_layout,\n",
" 'random': nx.random_layout,\n",
" 'shell': nx.shell_layout,\n",
" 'spectral': nx.spectral_layout,\n",
" 'spring': nx.spring_layout\n",
"}\n",
"\n",
"def save_layout(G, layout_name):\n",
" elarge=[(u,v) for (u,v,d) in G.edges(data=True) if d['weight'] >1.5]\n",
" esmall=[(u,v) for (u,v,d) in G.edges(data=True) if d['weight'] <=1.5]\n",
" nlarge=[n for n in G.nodes() if n in ['PyTennessee']]\n",
" pos=LAYOUTS[layout_name](G) # positions for all nodes\n",
"\n",
" print nlarge\n",
" # nodes\n",
" nx.draw_networkx_nodes(G,pos,nodelist=nlarge,node_size=1)\n",
"\n",
" # edges\n",
" nx.draw_networkx_edges(G,pos,edgelist=elarge, width=1)\n",
" nx.draw_networkx_edges(G,pos,edgelist=esmall, width=1,alpha=0.5,edge_color='#cccccc')\n",
"\n",
" # labelsM\n",
" labels={}\n",
" labels['PyTennessee']='PyTennessee'\n",
" nx.draw_networkx_labels(G,pos,labels,font_size=6)\n",
" #nx.draw_networkx_labels(G,pos,nodelist=nlarge,font_size=6,font_family='sans-serif')\n",
"\n",
" plt.axis('off')\n",
" plt.savefig(layout_name + '.png', dpi=500)\n",
"\n",
"save_layout(G, 'spring')"
],
"language": "python",
"metadata": {},
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Spring\n",
"\n",
"Below is the same graph from above, just bigger:\n",
"\n",
"\n",
"## Other NetworkX Graphing Layouts\n",
"\n",
"### Circular\n",
"\n",
"### Fruchterman-Reingold\n",
"\n",
"### Random\n",
"\n",
"### Shell\n",
"\n",
"### Spectral\n",
""
]
}
],
"metadata": {}
}
]
}