{ "metadata": { "name": "", "signature": "sha256:27ef990ec46e9922c913136459813da23154aad631bdc54ae1138de24f9564ef" }, "nbformat": 3, "nbformat_minor": 0, "worksheets": [ { "cells": [ { "cell_type": "heading", "level": 1, "metadata": {}, "source": [ "Tutorial: Loading your social network data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This tutorial offers you some options to download your (actually not only your) social network data. Basically here I give you 3 ways to deal with it: \n", "1. Application with lots of buttons (Almost every SN)\n", "2. Web-app (Facebook)\n", "3. Social network API (VK)\n", "\n", "The way you get your data is not that important, however you should know that options **1** and **2** import incomplete set of additional features of the network elements." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "1. NodeXL (Application with lots of buttons)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "[NodeXL](http://nodexl.codeplex.com/) is a free, open-source template for Microsoft Excel 2007, 2010 and 2013 that makes it easy to explore network graphs. In fact that's quite nice addon to make fast network analysis routines (made in \u043f\u043e\u0434\u0432\u0430\u043b on \u043a\u043e\u043b\u0435\u043d\u043a\u0430). Moreover, it has some build-in importers of social-web data (Flickr, Twitter, Youtube) and you are able to download additional importers.\n", "\n", "
" ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Installation" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Simply download [installation file](http://nodexl.codeplex.com/downloads/get/806203) and follow the instructions. You may face an error, that states:\n", "\n", "*\"The value of the property 'type' cannot be parsed. Could not load file or assembly 'Microsoft.Office.BusinessApplications.Fba,Version=14.0.0.0,Culture=neutral, PublicKeyToken=... or one of it's dependencies. The system cannot find the file specified. (C:\\Program Files\\Common Files\\Microsoft Shared\\VSTO\\10.0\\VSTOInstaller.exeConfig line 10).\"*\n", "\n", "In this case you have to go to *C:\\Program Files\\Common Files\\Microsoft Shared\\VSTO\\10.0\\* and rename **VSTOInstaller.exe.config** to **VSTOInstaller.exe.config.old**" ] }, { "cell_type": "heading", "level": 3, "metadata": {}, "source": [ "Additional importers" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "You can find VK and Odnoklassniki importers [here](http://runetimporter.codeplex.com/releases). Simply download archive and save files in arbitrary folder. Then go to NodeXL -> Import -> Import Options. There indicate the folder and restart Application.\n", "\n", "Finally, go to Import -> From Vkontakte network importer. After you login with your account will get a notification to provide certain permissions to a vk.com app. Mark features you would like to import and push \"Download\" button.\n", "\n", "
\n", "\n", "Recently facebook have updated its private policy, so you cannot acquire personal impormation of your friend. Still there are some creepholes. As for NodeXL - you can download this [importer](http://socialnetimporter.codeplex.com/releases/view/134889) and download network of co-likes\\co-comments in Facebook fan-page or group-page.\n", "\n", "Be aware that NodeXL importers are quike buggy and issues raises with number of additional features you want to store.\n", "\n" ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "2. Web-app" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "This section generally associates with Facebook. As you remember on your first seminar we have used *Netvizz* app to download your network. Unfortunately with the [Facebook private policy](http://thepoliticsofsystems.net/2015/01/the-end-of-netvizz/) update Netvizz has lost his ability to get your network data.\n", "\n", "If you will be fast enough you can use [GetNet](http://snacourse.com/getnet/) which is still capable of downloading your friendship network. Probably it will also shut down after the 1st of April.\n", "\n", "Workflow here is simple: link the applications with your Facebook account, choose additional features and download resulted .gml file." ] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "3. Using VK API" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "An application-programming interface (API) is a set of programming instructions and standards for accessing a Web-based software application or Web tool. A software companies or web-services release their API to the public so that other software developers can design products that are powered to interact and with their services.\n", "\n", "So our **first step** here is to **create a Standalone Application** [here](https://vk.com/dev). In \"Settings\" menu you need to find and remeber your **Application ID**.\n", "\n", "
\n", "\n", "Actually from this moment you already can get your frindship network. But some of the [API Methods](https://vk.com/dev/methods) require addtional permissions to handle page data. So you need to **get an *access token*** -- special line of sumbols that identifies you and your app creditals and set of previlliges. On [this page](https://vk.com/dev/auth_mobile) you can find the template to authorisation request to get this token.\n", "\n", "The parameter of particular interest here is *scope* parameter, which identifies [the priviliges](https://vk.com/dev/permissions) of your app. An access to friends should be enough for your, so your authorisation request should be somewhat like this:\n", "\n", "https://oauth.vk.com/authorize?client_id=**APP_ID**&scope=friends&redirect_uri=https://oauth.vk.com/blank.html&display=wap&v=5.28&response_type=token\n", "\n", "Replace APP_ID with your app id and copy this link to your browser. You will see a pop-up window with confimation of the app priviliges. Afterall adress line of your browser will contain access token -- copy it and show to anybody! Note that your token has a lifetime. When it expires you should repeat your authorisation request.\n", "\n", "Let's try to run some API methods directly:" ] }, { "cell_type": "code", "collapsed": false, "input": [ "import urllib2, json, sys" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# Your token - sequence of symbols that goes after access_token=\n", "token = '' \n", "\n", "# Suppose you want to get a friendlist of a particular user.\n", "# You also want hometown, sex and education to be contained in this list.\n", "# Finnaly, I suggest to put lang=en to aviod cyrillic issues\n", "uid = ''\n", "url = u'https://api.vk.com/method/friends.get?fields=sex,city,education&uid=%s&lang=en&access_token=%s' %(uid ,token)\n", "\n", "res = urllib2.urlopen(url).read()\n", "data = json.loads( res )" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Variable *data* contains information in [json](https://ru.wikipedia.org/wiki/JSON) format" ] }, { "cell_type": "code", "collapsed": false, "input": [ "uid = ''\n", "url = u'https://api.vk.com/method/friends.get?fields=uids&uid=%s&lang=en&access_token=%s' %(uid ,token)\n", "\n", "res = urllib2.urlopen(url).read()\n", "data = json.loads( res )\n", "# Access to the first user in friendlist\n", "data['response']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "code", "collapsed": false, "input": [ "# Access to his\\her last name\n", "data['response'][0]['last_name']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In fact there is a [Python module](https://pypi.python.org/pypi/vk/1.5) that makes your life a bit easier. You can install in by typing \n", " *pip install vk* in your terminal window." ] }, { "cell_type": "code", "collapsed": false, "input": [ "import vk\n", "\n", "# Use app id, user email/phone and password for access to API\n", "vkapi = vk.API('my_app_id', 'user_login', 'user_password')\n", "vk.access_token = token\n", "\n", "profiles = vkapi.users.get(user_id=1)\n", "profiles[0]['last_name']" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "heading", "level": 2, "metadata": {}, "source": [ "Example: loading mutual friends" ] }, { "cell_type": "code", "collapsed": false, "input": [ "# Your uid\n", "me = ''\n", "\n", "# Using results of get.friends request\n", "uids = []\n", "for u in data['response']:\n", " uids.append(str(u['uid']))\n", "uids = ','.join(uids)\n", "\n", "# Mutual Friends Request\n", "url = u'https://api.vk.com/method/friends.getMutual?target_uids=%s&source_uid=%s&lang=en&access_token=%s' %(uids, me ,token)\n", "\n", "# Our Result\n", "res = urllib2.urlopen(url).read()\n", "data = json.loads( res )\n", "data['response'][0]" ], "language": "python", "metadata": {}, "outputs": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In fact VK API limits number of requests per second for apps with <= 10000 users. Thankfully, getMutual method has *target_uids* parameter for multiple target uids. In case you want to reduce number of requests I suggest you to implement your own Stored Function (find it in your app-management menu) and use [execute](https://vk.com/dev/execute) method." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That is pretty enough for your mid-term exam task.\n", "\n", "You also can find similar guidelines for [Facebook](http://nbviewer.ipython.org/github/chdoig/Mining-the-Social-Web-2nd-Edition/blob/master/ipynb/Chapter%202%20-%20Mining%20Facebook.ipynb) and [Twitter](http://mark-kay.net/2014/08/15/network-graph-of-twitter-followers/) APIs" ] } ], "metadata": {} } ] }