{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# LoL Churn Predictor [Part 1 - Data Collection]\n", "\n", "**David Skarbrevik - 2018**\n", "\n", "We will be getting all data from Riot's API. Their API allows us to query a Summoner's name (player username) to get match history from the last 3 years. Ultimately our goal is to build a model that predicts yes/no (1/0) will a player stop playing the game (based on some previous data) or not." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "
\n", "
\n", "
\n", "\n", "

Table of Contents

\n", "\n", "
\n", "\n", "
    \n", "

  1. Planning

    2. \n", "
    \n", "

  2. Getting Familiar with Riot's API

    3. \n", "
    \n", "

  3. Building functions to get our data

    4. \n", "
    \n", "

  4. Actually gathering the data

    5. \n", "
\n", "\n", "
\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "\n", "## Step 1) Planning \n", "\n", "Let's first make sure we have a clear vision of what kind of data we can (and want) to get.\n", "\n", "### 1) What kind of data can we collect from a match?\n", "\n", "Individual player data about how many kills, assists, at what time and location, what champion they chose, how long the match lasted, etc. \n", "\n", "### 2) Is there any other non-match data we can collect about a player to help with our prediction?\n", "\n", "Not really at the moment. Might think more about this in the future\n", "\n", "### 3) What players should we be targeting? \n", "\n", "Creating a random database may be difficult unless I can figure out how to get a list of random summoner IDs... **see \"Step 3\" of this notebook for more info.**\n", "\n", "\n", "### 4) How many players and matches do we need to collect to make an adequate test/train dataset?\n", "\n", "I think maybe 1000 players would be a good start. I'll just take the first match of each player to start but if this doesn't seem to be enough to build an accurate model I'll consider taking more. I think the biggest challenge for data collection right now will be finding enough players that fit my specifications (see \"Step 3\" of this notebook).\n", "\n", "### 5) How should our data be organized to feed it into a model?\n", "\n", "I'm planning to use either logistic regression or a simple feed-forward neural network. For these models I want each row of my dataset to represent all the data for a single player. By looking at just the first match for each player it should be easy to make sure the number of features in each row is the same. Later, if I wanted a more general model for churn that works for any player with any number of matches, an RNN might be worth looking into considering the sequential, non-equal length of input vectors in that case.\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " back to top\n", "
\n", "\n", "\n", "## Step 2) Getting familiar with Riot's API (using the Python wrapper Cassiopeia)\n", "\n", "Advantages of using Cassiopeia instead of making the API calls ourselves:\n", "* Useful pre-built functions for gathering data (e.g. `MatchParticipant.stats.to_dict()` gets a LOT of features at once)\n", "* Rate limiting is taken care of for us (sooo helpful since we want to make a lot of calls to build our database)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Import needed libraries**" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "data": { "application/javascript": [ "if (!(\"Notification\" in window)) {\n", " alert(\"This browser does not support desktop notifications, so the %%notify magic will not work.\");\n", "} else if (Notification.permission !== 'granted' && Notification.permission !== 'denied') {\n", " Notification.requestPermission(function (permission) {\n", " if(!('permission' in Notification)) {\n", " Notification.permission = permission;\n", " }\n", " })\n", "}\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import re\n", "import json\n", "import pandas as pd\n", "from collections import defaultdict, Counter\n", "import arrow\n", "import ipywidgets as widgets # for progress bar\n", "\n", "import jupyternotify # gives desktop notifications when cell blocks complete\n", "ip = get_ipython()\n", "ip.register_magics(jupyternotify.JupyterNotifyMagics(ip, require_interaction=True))\n", "\n", "import cassiopeia as cass # Riot API wrapper\n", "from cassiopeia import Summoner" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Set global parameters for Riot API**" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "no_print_calls = cass.get_default_config()\n", "no_print_calls['logging']['print_calls'] = False\n", "\n", "cass.apply_settings(no_print_calls)" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "cass.set_riot_api_key(\"RGAPI-975f0783-9b2f-4f69-adb0-000b71a0d9f2\") # API key that you get from your account\n", "cass.set_default_region(\"NA\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Let's try to get the summoner ids of some players (the best players)**" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/league/v3/challengerleagues/by-queue/RANKED_SOLO_5x5\n", "34402943\n", "24705131\n", "21103810\n", "47212115\n", "20598223\n", "93809082\n", "20823651\n", "21124626\n", "21623482\n", "20429469\n", "20202108\n", "29557073\n", "24449785\n", "28725693\n", "24671130\n", "30645632\n", "70132282\n", "21371454\n", "42133105\n", "41233933\n", "34225884\n", "49899264\n", "35097344\n", "22670229\n", "45186258\n", "19285521\n", "48193601\n", "19839806\n", "401063\n", "56901374\n", "30266347\n", "37532228\n", "19804632\n", "27251616\n", "79249642\n", "52371378\n", "22478427\n", "83339123\n", "60469878\n", "21542029\n", "30972130\n", "40661961\n", "93269094\n", "36119978\n", "19920510\n", "51580106\n", "77269277\n", "20947579\n", "30255016\n", "19787999\n", "20476893\n", "77520973\n", "21652056\n", "50877721\n", "90331209\n", "82962549\n", "32252808\n", "40279866\n", "36782900\n", "45079499\n", "19587365\n", "24823450\n", "20636732\n", "51949451\n", "32705024\n", "77720407\n", "71229345\n", "91419117\n", "22638316\n", "20389591\n", "52559520\n", "51075422\n", "88109144\n", "37481045\n", "20962399\n", "76609200\n", "19577112\n", "45038172\n", "35592112\n", "83089575\n", "43839117\n", "94819093\n", "20130821\n", "22403576\n", "71715\n", "35570736\n", "77489438\n", "40710913\n", "65389099\n", "77211704\n", "24326160\n", "32556358\n", "21276764\n", "19338315\n", "43079579\n", "20248040\n", "26065445\n", "19698094\n", "91636261\n", "21260950\n", "20414291\n", "24877618\n", "73543081\n", "41885666\n", "20877255\n", "51680686\n", "20056332\n", "19296564\n", "76602920\n", "22654107\n", "20902946\n", "88490180\n", "24777351\n", "19906739\n", "30366254\n", "77251686\n", "47401142\n", "21193669\n", "85889107\n", "23245665\n", "20132258\n", "37343071\n", "61542283\n", "19470447\n", "2648\n", "47836799\n", "21806931\n", "27470522\n", "91429131\n", "49159160\n", "48318695\n", "89909122\n", "25888946\n", "34925733\n", "91729099\n", "40540196\n", "67419639\n", "84509505\n", "19623941\n", "87330006\n", "25848742\n", "38375098\n", "36111109\n", "38880034\n", "579475\n", "21701239\n", "22292020\n", "91729110\n", "49290688\n", "75900909\n", "91419120\n", "19061980\n", "41292075\n", "35703236\n", "21498922\n", "20112175\n", "92949232\n", "19843806\n", "22079480\n", "24069119\n", "25851366\n", "20156599\n", "54986862\n", "46069412\n", "23552941\n", "91429100\n", "46052203\n", "21202019\n", "20270876\n", "29533033\n", "23965783\n", "44317220\n", "63668454\n", "42060215\n", "21014987\n", "74731419\n", "22350178\n", "87119170\n", "20024227\n", "72409957\n", "25232745\n", "22417049\n", "27222755\n", "19770082\n", "87912976\n", "19757901\n", "25108893\n", "38869932\n", "68401488\n", "19887289\n", "94189199\n", "19812679\n", "38091945\n", "22946266\n", "23934806\n", "89599264\n", "59991269\n", "30045762\n", "57029179\n", "51405\n" ] } ], "source": [ "challenger_league = cass.get_challenger_league(queue=cass.Queue.ranked_solo_fives)\n", "\n", "for challenger in challenger_league:\n", " print(challenger.summoner.id)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "That's cool but these are the ids of the best of the best... we want the newest of the new (unranked players) so we'll have to work a bit harder." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**How to access a single user**" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Msendak (37709821) is a level 55 summoner on the Region.north_america server.\n" ] } ], "source": [ "summoner = Summoner(name=\"Msendak\", region=\"NA\")\n", "print(\"{name} ({id}) is a level {level} summoner on the {region} server.\".format(name=summoner.name,\n", " level=summoner.level,\n", " id=summoner.id,\n", " region=summoner.region))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**We can also access the user with their summoner id**\n", "\n", "It is my understanding that new summoner names can be bought but that summoner ids may be more permenant so for the sake of stability we'll be associating players by their id when gathering data later." ] }, { "cell_type": "code", "execution_count": 35, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MSendak (37709821) is a level 49 summoner on the Region.north_america server.\n" ] } ], "source": [ "summoner = Summoner(id=37709821, region=\"NA\")\n", "print(\"{name} ({id}) is a level {level} summoner on the {region} server.\".format(name=summoner.name,\n", " level=summoner.level,\n", " id=summoner.id,\n", " region=summoner.region))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Get all the matches that user played**" ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "collapsed": true }, "outputs": [], "source": [ "matches = summoner.match_history" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "scrolled": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Total matches = 2779\n" ] } ], "source": [ "print(\"Total matches = {}\".format(len(matches)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Time of most recent and oldest match (in the last three years)**" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "MSendak's Oldest Match stored in the API: 2015-05-01\n", "MSendak's Most Recent Match: 2018-04-22\n" ] } ], "source": [ "first_match = matches[-1]\n", "latest_match = matches[0]\n", "\n", "time_first_match = re.search(\"\\d+-\\d+-\\d+\",str(first_match.creation))[0]\n", "time_latest_match = re.search(\"\\d+-\\d+-\\d+\",str(latest_match.creation))[0]\n", "\n", "print(\"{name}'s Oldest Match stored in the API: {date}\".format(name=summoner.name, date=time_first_match))\n", "print(\"{name}'s Most Recent Match: {date}\".format(name=summoner.name, date=time_latest_match))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Get stats from any player in that specific match**" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "collapsed": true }, "outputs": [], "source": [ "players = first_match.participants\n", "player = players[1] # selecting a specific player" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "scrolled": true }, "outputs": [ { "data": { "text/plain": [ "{'assists': 19,\n", " 'champLevel': 14,\n", " 'combatPlayerScore': 0,\n", " 'damageDealtToObjectives': 0,\n", " 'damageDealtToTurrets': 0,\n", " 'damageSelfMitigated': 0,\n", " 'deaths': 11,\n", " 'doubleKills': 3,\n", " 'firstBloodAssist': False,\n", " 'firstBloodKill': False,\n", " 'firstInhibitorAssist': True,\n", " 'firstInhibitorKill': False,\n", " 'firstTowerAssist': False,\n", " 'firstTowerKill': False,\n", " 'goldEarned': 8496,\n", " 'goldSpent': 7050,\n", " 'inhibitorKills': 0,\n", " 'item0': 3087,\n", " 'item1': 3006,\n", " 'item2': 2003,\n", " 'item3': 3153,\n", " 'item4': 0,\n", " 'item5': 0,\n", " 'item6': 2052,\n", " 'killingSprees': 3,\n", " 'kills': 9,\n", " 'largestCriticalStrike': 388,\n", " 'largestKillingSpree': 3,\n", " 'largestMultiKill': 3,\n", " 'longestTimeSpentLiving': 136,\n", " 'magicDamageDealt': 9021,\n", " 'magicDamageDealtToChampions': 3439,\n", " 'magicalDamageTaken': 1794,\n", " 'neutralMinionsKilled': 0,\n", " 'objectivePlayerScore': 0,\n", " 'participantId': 2,\n", " 'pentaKills': 0,\n", " 'physicalDamageDealt': 20675,\n", " 'physicalDamageDealtToChampions': 8278,\n", " 'physicalDamageTaken': 12276,\n", " 'playerScore0': 0,\n", " 'playerScore1': 0,\n", " 'playerScore2': 0,\n", " 'playerScore3': 0,\n", " 'playerScore4': 0,\n", " 'playerScore5': 0,\n", " 'playerScore6': 0,\n", " 'playerScore7': 0,\n", " 'playerScore8': 0,\n", " 'playerScore9': 0,\n", " 'quadraKills': 0,\n", " 'sightWardsBoughtInGame': 0,\n", " 'timeCCingOthers': 0,\n", " 'totalDamageDealt': 31406,\n", " 'totalDamageDealtToChampions': 13298,\n", " 'totalDamageTaken': 15399,\n", " 'totalHeal': 1222,\n", " 'totalMinionsKilled': 21,\n", " 'totalPlayerScore': 0,\n", " 'totalScoreRank': 0,\n", " 'totalTimeCrowdControlDealt': 25,\n", " 'totalUnitsHealed': 1,\n", " 'tripleKills': 1,\n", " 'trueDamageDealt': 1710,\n", " 'trueDamageDealtToChampions': 1580,\n", " 'trueDamageTaken': 1328,\n", " 'turretKills': 1,\n", " 'unrealKills': 0,\n", " 'visionScore': 0,\n", " 'visionWardsBoughtInGame': 0,\n", " 'wardsKilled': 0,\n", " 'wardsPlaced': 0,\n", " 'win': True}" ] }, "execution_count": 21, "metadata": {}, "output_type": "execute_result" } ], "source": [ "player_stats = player.stats.to_dict()\n", "\n", "player_stats" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "{'creepsPerMinDeltas': {'0-10': 1.5},\n", " 'csDiffPerMinDeltas': {'0-10': 0.1599999999999998},\n", " 'damageTakenDiffPerMinDeltas': {'0-10': -53.019999999999925},\n", " 'damageTakenPerMinDeltas': {'0-10': 964.6999999999999},\n", " 'goldPerMinDeltas': {'0-10': 467.29999999999995},\n", " 'id': 2,\n", " 'lane': 'MIDDLE',\n", " 'role': 'NONE',\n", " 'xpDiffPerMinDeltas': {'0-10': 36.0800000000001},\n", " 'xpPerMinDeltas': {'0-10': 672.8}}" ] }, "execution_count": 22, "metadata": {}, "output_type": "execute_result" } ], "source": [ "player_timeline = player.timeline.to_dict()\n", "\n", "player_timeline" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The \"stats\" and \"timeline\" attributes we see here are where to bulk of data from a match come from." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " back to top\n", "
\n", "\n", "\n", "## Step 3) Building functions to get our data\n", "\n", "Now that we're familiar with how the Cassiopeia library works, let's get the data we really want.\n", "\n", "I specifically want players that are new to the game (low level / few matches), but there is no way to directly access a list of players that meet that criteria through the API (at least not that I know of). So instead, I made a new account (summoner: 'OldTimeCandyBowl') and played a match. I then queried the API about the match to get the summoner ids of my teammates in that match (knowing they'd likely be low level players). My plan is to branch out from them, by taking the players in other matches they've had and then the other players _those_ players have played with, and so on and so forth until I've branched deep enough that I'm happy with the number of low level summoners I've gathered. Initially **I'll aim for 1,000 summoners**.\n", "\n", "**Targeted summoner for database I'm building:**\n", "\n", "* first match must be in 2018\n", "* first match must be at least 1 month ago (from today)\n", "* player level must not be higher than 5\n", "\n", "**Data to collect from target summoner:**\n", "\n", "* gameplay data from first match\n", "\n", "**Possible prediction tasks:**\n", "\n", "_(all predictions based on summoner's gameplay data from first match)_\n", "* Will the summoner get to level 3 or higher within the first month of play?\n", "* Did the summoner play more than 1 match?\n", "* Did the summoner play at least X matches?" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**We are interested in a very specific type of player so this function filters out players that don't fit our specifications.**" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "# limits the type of player we want in our database\n", "def is_candidate_player(summoner, first_match, level_cap):\n", " \n", " # NOTE: (int,int,int) that is output with True/False is a counter check to understand how many players are rejected from db.\n", " \n", " entry_game_types = ['BOT_5X5_INTRO', 'BOT_5X5_BEGINNER', 'BOT_5X5_INTERMEDIATE']\n", " time_now = arrow.now()\n", "\n", " # level cap restriction\n", " if summoner.level > level_cap:\n", " return False, (1,0,0)\n", " \n", " # must not be newer than this\n", " if first_match.creation.shift(months=1) > time_now:\n", " return False, (0,1,0)\n", " \n", " # must not be older than this\n", " if first_match.creation.year < 2018:\n", " return False, (0,0,1)\n", " \n", " return True, (0,0,0)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**This function actually gathers the game data from a player's first match.**" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [], "source": [ "# gather all player data from a single match\n", "def get_player_game_data(summoner, player_history): \n", "\n", " first_match = player_history[-1]\n", " recent_match = player_history[0]\n", " first_match_participant = first_match.participants[summoner]\n", " time_now = arrow.now()\n", " \n", " # we need to un-nest the timeline dict and standardize the format of this data\n", " timeline_dict = {}\n", " \n", " try:\n", " for key,val in first_match_participant.timeline.to_dict().items():\n", "\n", " if isinstance(val, dict):\n", " timeline_dict['{0}_0-10'.format(key)] = val.get('0-10',0)\n", " timeline_dict['{0}_10-20'.format(key)] = val.get('10-20',0)\n", " timeline_dict['{0}_20-30'.format(key)] = val.get('20-30',0)\n", " timeline_dict['{0}_30-end'.format(key)] = val.get('30-end',0) \n", " else:\n", " timeline_dict[key] = val \n", " except:\n", " print(\"Problem getting timeline data\")\n", " \n", " try:\n", " # stats data is already in good shape!\n", " stats_dict = first_match_participant.stats.to_dict()\n", " except:\n", " print(\"Problem getting stats data\")\n", " \n", " try:\n", " # get basic summoner info\n", " summoner_dict = {\"summoner_id\": summoner.id, \"summoner_name\": summoner.name, \"summoner_level\": summoner.level}\n", " except:\n", " print(\"Problem getting summoner info.\")\n", " \n", " try:\n", " # there is some other info we want that's not captured by the 'stats' or 'timeline\n", " extra_tid_bits = {'first_match_id': first_match.id, 'first_match_duration': first_match.duration, \n", " 'first_match_time': first_match.creation, 'latest_match_time': recent_match.creation, \n", " 'total_matches': len(player_history)}\n", " except:\n", " print(\"Problem getting extra info about player match.\")\n", " \n", " \n", " return {**extra_tid_bits, **timeline_dict, **stats_dict, **summoner_dict}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**This function takes in a list of summoner ids and produces a dataset of first match data from it.**\n", "\n", "Only really necessary if I'm in a situation where I have built a good list of summoner ids but I've lost my dataset for some reason." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "##############################\n", "# WE WILL CALL THIS DIRECTLY #\n", "############################## \n", "# given a list of summoner ids, will return dataframe with game data for each player\n", "# this is a backup, \"just in case\" function that I would only use if I lost my database but still had my list of summoners.\n", "def db_from_ids(summoner_ids: list):\n", " \n", " master_game_data_list = []\n", " issue_ids = []\n", "\n", " for i in range(len(summoner_ids)):\n", "\n", " if i % 10 == 0:\n", " print(\"Currently processing player {}\".format(i+1))\n", "\n", " try:\n", " summoner = Summoner(id=summoner_ids[i], region=\"NA\")\n", " summoner_history = summoner.match_history\n", " except Exception as inst:\n", " issue_ids.append(player)\n", " print(\"Problem getting player {}\".format(player))\n", " print(type(inst)) \n", " print(inst) \n", "\n", " try:\n", " game_data = get_player_game_data(summoner, summoner_history)\n", " master_game_data_db.append(game_data)\n", " except Exception as inst:\n", " print(\"Problem getting game_data\")\n", " print(type(inst)) \n", " print(inst) \n", "\n", " if issue_ids:\n", " print(\"Could not collect data on these players:\")\n", " print(issue_ids)\n", " \n", " master_df = pd.DataFrame([*master_game_data_db])\n", " \n", " return master_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**After each round of data collection this saves lists and data to files.**" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "##############################\n", "# WE WILL CALL THIS DIRECTLY #\n", "############################## \n", "# this ties the data from the end of a round to all the data collected to that point and saves it to disk\n", "def end_of_round_housekeeping(master_db, round_db: list, master_id_list: list, round_list: list):\n", " \n", " if isinstance(master_db, pd.core.frame.DataFrame):\n", " master_db = master_db.append(round_db)\n", "\n", " master_id_list += round_list # running list of all ids already in database\n", "\n", " # save the current seed list\n", " with open(\"./data/current_seed_list.txt\", \"w\") as file1:\n", " for summoner in round_list:\n", " file1.write(str(summoner)+\"\\n\")\n", "\n", " # save all summoner ids that have been added to database already\n", " with open(\"./data/summoners_in_db.txt\", \"a\") as file2:\n", " for summoner in round_list:\n", " file2.write(str(summoner)+\"\\n\")\n", " \n", " # save dataset to csv\n", " master_db.to_csv(\"./data/riot_master_df2.csv\", index=False)\n", " \n", " return master_db, master_id_list " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**If we need to restart the notebook or the kernel crashes for some reason, this will read our lists/data from disk.**" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "##############################\n", "# WE WILL CALL THIS DIRECTLY #\n", "##############################\n", "# if kernel crashes, call this to get most current lists and dataframe back\n", "def recover_lists(): \n", " current_seed_list = []\n", " summoners_in_db = []\n", "\n", " with open('./data/current_seed_list.txt', 'r') as file1:\n", " lines = file1.readlines()\n", " for line in lines:\n", " current_seed_list.append(int(line.rstrip('\\n')))\n", "\n", " with open('./data/summoners_in_db.txt', 'r') as file2:\n", " lines = file2.readlines()\n", " for line in lines:\n", " summoners_in_db.append(int(line.rstrip('\\n'))) \n", " \n", " data_df = pd.read_csv(\"./data/riot_master_df.csv\", encoding=\"ISO-8859-1\") \n", " \n", " print(\"Summoners in current seed list = {}\".format(len(current_seed_list)))\n", " print(\"All summoners gathered so far = {}\".format(len(summoners_in_db)))\n", " \n", " return data_df, current_seed_list, summoners_in_db" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**This will be the conductor of the orchestra. We'll call this to get new ids and data.**" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "##############################\n", "# WE WILL CALL THIS DIRECTLY #\n", "##############################\n", "# collect gameplay data from players\n", "def get_new_players(seed_list_of_summoners, summoners_already_in_db = None, cut_off = None, level_cap = 10):\n", "\n", " # PARAMETER(S) #\n", " # seed_list_of_summoners : (list) of summoner ids to branch from to find new players\n", " # cutt_off : (int) stops data mining if cut_off number of players get added to db\n", " \n", " \n", " # initialization for data we want to collect\n", " gameplay_database = [] # list of dictionaries containing player match data\n", " new_seeders = [] # new list of seeders for next call to this function\n", " back_up_seeders = [] # if there are less than 5 good candidates then return all players considered instead\n", " \n", " # tracking to troubleshoot issues and help validate data collection\n", " player_added_count = 0 # track how many players added to database\n", " considered_count = 0 # track how many players entered \"is_candidate_player\"\n", " denied_level_cap = 0 # track how many players denied b/c level too high\n", " denied_too_young = 0 # track how many players denied b/c account too new\n", " denied_too_old = 0 # track how many players denied b/c account too old\n", " \n", " \n", " if cut_off != None:\n", " if not isinstance(cut_off, int):\n", " cut_off = None\n", " print(\"Warning: cut_off must be type 'int'... cut_off was reset to none.\")\n", " \n", " # avoid unnecessary duplicate data mining\n", " if summoners_already_in_db == None:\n", " summoners_already_in_db = seed_list_of_summoners\n", " \n", " total_seeders = len(seed_list_of_summoners)\n", " \n", " # progress bar to make to track... progress\n", " interval = 100/total_seeders # for incrementing progress bar\n", " progress = widgets.FloatProgress(value=0.0, min=0.0, max=100.0, description='0%', bar_style='info')\n", " display(progress)\n", " \n", " # cycle through ids in \"seeder list\" to get new summoner ids and data\n", " for player in seed_list_of_summoners:\n", " \n", " try: \n", " seed_summoner = Summoner(id=player, region=\"NA\")\n", " seeder_history = seed_summoner.match_history\n", " seeder_total_matches = len(seeder_history)\n", " \n", " for i in range(seeder_total_matches):\n", " \n", " total_players = len(seeder_history[i].participants)\n", " \n", " for j in range(total_players):\n", " \n", " player = seeder_history[i].participants[j]\n", " \n", " # avoid bots and duplicates\n", " try:\n", " if player.is_bot == True:\n", " continue\n", " except:\n", " pass # not great practice but this occurs frequently and is meaningless for me\n", "\n", " try:\n", " if player.summoner.id in summoners_already_in_db:\n", " continue\n", " except Exception as inst:\n", " print(\"Issue checking if summoner in seed_list\")\n", " print(type(inst)) \n", " print(inst) \n", " \n", " player_history = player.summoner.match_history\n", " first_match = player_history[-1]\n", "\n", " try:\n", " put_in_db, denied_checks = is_candidate_player(player.summoner, first_match, level_cap)\n", " except Exception as inst:\n", " print(\"Problem checking if candidate player.\")\n", " print(type(inst)) \n", " print(inst) \n", "\n", " considered_count += 1\n", " denied_level_cap += denied_checks[0]\n", " denied_too_young += denied_checks[1]\n", " denied_too_old += denied_checks[2]\n", " \n", " back_up_seeders.append(player.summoner.id)\n", " \n", " if put_in_db == True:\n", " \n", " try:\n", " game_data = get_player_game_data(player.summoner, player_history)\n", " except:\n", " print(\"Problem getting game_data\")\n", "\n", " # throw our new valuable data into the treasure chest!\n", " gameplay_database.append(game_data)\n", " new_seeders.append(player.summoner.id)\n", " summoners_already_in_db.append(player.summoner.id)\n", " \n", " player_added_count += 1\n", " \n", " if player_added_count % 100 == 0:\n", " print(\"Currently {} players in database.\".format(player_added_count))\n", " \n", " if cut_off:\n", " if player_count >= cut_off:\n", " return gameplay_database\n", " \n", " # update progress bar (end of seeder player history) \n", " progress.value += interval\n", " progress.description = \"{}%\".format(round(progress.value,1))\n", " \n", " except Exception as inst:\n", " print(\"ERROR while processing player: {0} ({1}).\".format(seed_summoner.name, seed_summoner.id))\n", " print(type(inst)) \n", " print(inst)\n", " \n", " \n", " # made it to the end!!! Let's summarize what happened during collection.\n", " progress.bar_style = 'success'\n", " print(\"DONE! \\n\")\n", " print(\"Players that were considered for database = {}\".format(considered_count))\n", " print(\"Players that were added to database = {}\".format(player_added_count))\n", " print(\"Players denied b/c of level cap = {}\".format(denied_level_cap))\n", " print(\"Players denied b/c account too young = {}\".format(denied_too_young))\n", " print(\"Players denied b/c account too old = {}\".format(denied_too_old))\n", " \n", " if len(new_seeders) >= 5:\n", " return gameplay_database, new_seeders\n", " else:\n", " print(\"\\033[1;31m\" + \"WARNING:\" + \"\\033[0m\" + \" There weren't many good candidate players :( so look carefully at seed list that was returned.\")\n", " return gameplay_database, back_up_seeders" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " back to top\n", "
\n", "\n", "\n", "## Step 4) Actually gathering our data\n", "\n", "**Note about Step 4:** You'll notice as you look through Step 4 that I gather data in \"Rounds\". I'm essentially manually looping over a call to my function above until I have enough data. I'm doing this manually for now so that I can validate the data I get out of each loop. If I want to use this more in the future I'll probably write a loop in to the functions above." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Now that we have our functions ready, we just need a seed list of summoner ids to start from.**\n", "\n", "We'll use this new account that I played just a few games on to try and get new player ids." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/by-name/OldTimeCandyBowl\n", "OldTimeCandyBowl 3\n" ] } ], "source": [ "summoner = Summoner(name=\"OldTimeCandyBowl\", region=\"NA\")\n", "print(summoner.name, summoner.level)" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/247157416?beginIndex=0&endIndex=100\n", "Making call: https://ddragon.leagueoflegends.com/realms/na.json\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matches/2768433833\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/by-account/247157416\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/247157416?beginIndex=0&endIndex=100\n", "OldTimeCandyBowl (94643101) is level 3\n", "[94643101]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/76772327\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/233648039?beginIndex=0&endIndex=100\n", "FrozenVettel (76772327) is level 9\n", "[94643101]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/93799945\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246057408?beginIndex=0&endIndex=100\n", "INFO: Unexpected service rate limit, backing off for 35 seconds (from headers).\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246057408?beginIndex=0&endIndex=100\n", "LittyMcLitSauce (93799945) is level 8\n", "[94643101, 93799945]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/94533010\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/247155417?beginIndex=0&endIndex=100\n", "téétó (94533010) is level 6\n", "[94643101, 93799945, 94533010]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/90193433\n", "Y Z K E L (90193433) is level 11\n", "[94643101, 93799945, 94533010]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/90563962\n", "RioNad (90563962) is level 10\n", "[94643101, 93799945, 94533010]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/95079317\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/247179978?beginIndex=0&endIndex=100\n", "yawa akong lolo (95079317) is level 3\n", "[94643101, 93799945, 94533010, 95079317]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/94909450\n", "YesImFOB (94909450) is level 10\n", "[94643101, 93799945, 94533010, 95079317]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/95049679\n", "F1Na4lf3 (95049679) is level 10\n", "[94643101, 93799945, 94533010, 95079317]\n", "Making call: https://na1.api.riotgames.com/lol/summoner/v3/summoners/94859763\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/247219318?beginIndex=0&endIndex=100\n", "Accountnumber21 (94859763) is level 4\n", "[94643101, 93799945, 94533010, 95079317, 94859763]\n" ] } ], "source": [ "match_history = summoner.match_history\n", "seed_match = match_history[0]\n", "\n", "players = seed_match.participants\n", "\n", "seed_list = []\n", "\n", "for i in range(len(players)):\n", " \n", " if players[i].summoner.level < 10 and (players[i].summoner.match_history[-1].creation.year > 2017):\n", " seed_list.append(players[i].summoner.id)\n", " \n", " print(\"{0} ({1}) is level {2}\".format(players[i].summoner.name, \n", " players[i].summoner.id, \n", " players[i].summoner.level))\n", "\n", " \n" ] }, { "cell_type": "code", "execution_count": 22, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Seed list of summoner ids: [94643101, 93799945, 94533010, 95079317, 94859763]\n" ] } ], "source": [ "print(\"Seed list of summoner ids: {}\".format(seed_list))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Starting Round 1 of summoner id mining\n", "\n", "Let's use our seed list to try and get more new players." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "scrolled": true }, "outputs": [], "source": [ "game_data_dict, new_seed_list = get_new_players(seed_list)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Awesome! We found some new ids that fit our specifications! Let's verify that they seem right for our database.**" ] }, { "cell_type": "code", "execution_count": 20, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244707853?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244707853?beginIndex=0&endIndex=100\n", "Juanochi (92091071) is level 7 with 8 matches. First match was 01-16-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246050296?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246050296?beginIndex=0&endIndex=100\n", "BigDaddyMiles (93659918) is level 3 with 4 matches. First match was 03-10-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244444992?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244444992?beginIndex=0&endIndex=100\n", "ritowhy1 (93589089) is level 8 with 16 matches. First match was 03-09-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246049176?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246049176?beginIndex=0&endIndex=100\n", "Yanora (93759868) is level 10 with 21 matches. First match was 03-10-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246058398?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246058398?beginIndex=0&endIndex=100\n", "Catastrophz (93859982) is level 10 with 21 matches. First match was 03-10-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246038766?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246038766?beginIndex=0&endIndex=100\n", "roselovemoon (93769822) is level 8 with 21 matches. First match was 03-10-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244707853?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244707853?beginIndex=0&endIndex=100\n", "Juanochi (92091071) is level 7 with 8 matches. First match was 01-16-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245821906?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245821906?beginIndex=0&endIndex=100\n", "FeedzCx (93399527) is level 10 with 23 matches. First match was 03-02-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244410558?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244410558?beginIndex=0&endIndex=100\n", "Xeptorik (91508675) is level 6 with 11 matches. First match was 01-06-2018\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246226008?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246226008?beginIndex=0&endIndex=100\n", "mlxg is best (93641866) is level 4 with 4 matches. First match was 03-15-2018\n" ] } ], "source": [ "for id in new_seed_list:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " print(\"{0} ({1}) is level {2} with {3} matches. First match was {4}\".format(summoner.name, \n", " summoner.id, \n", " summoner.level, \n", " len(summoner.match_history),\n", " summoner.match_history[-1].creation.format(\"MM-DD-YYYY\")))\n" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**It looks like only two of the ten players are under level 5.** \n", "\n", "Since we want to have a lot of examples of players that didn't get to level 3 (or maybe at least not to level 5), we're going to just use the lowest level players from our new list to try and get more new players." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Starting Round 2 of data collection" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true, "scrolled": false }, "outputs": [], "source": [ "game_data_dict2, new_seed_list2 = get_new_players(new_seed_list, level_cap=5) # specifying level cap to be lower (default = 10)" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "**Yes! Looks like we may have found 21 players that are new (but at least a month old still) and low level!**\n", "\n", "Let's double check to see their info:" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "scrolled": true }, "outputs": [], "source": [ "match_count = 0\n", "for id in new_seed_list2:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " print(\"{0} ({1}) is level {2} with {3} matches. First match was {4}\".format(summoner.name, \n", " summoner.id, \n", " summoner.level, \n", " len(summoner.match_history),\n", " summoner.match_history[-1].creation.format(\"MM-DD-YYYY\")))\n", " match_count += len(summoner.match_history)\n", "\n", "print(\"\\n\")\n", "print(\"TOTAL MATCHES BETWEEN ALL NEW PLAYERS = {}\".format(match_count))" ] }, { "cell_type": "markdown", "metadata": { "collapsed": true }, "source": [ "So we have a total of 21 people in our dataset and with their combined 98 matches to search from, that's close to 1,000 possible new summoner ids we can get. Ultimately we'll want at least 1,000 so we'll need to rerun our data gathering functions at least once more (probably two more times).\n", "\n", "Also note that with our level cap of 5 most of the players in our dataset _are_ level 5 or 4... but very few are 3 or 2 or 1. Because one possible prediction task we'd like to do is decided if a players will get to level 3 (out of tutorial mode) or not, we'd like more cases where a player did not get out (still level 1 or 2). To accomplish this we could get a lot more data than 1,000 players or change our level_cap to 3 or 4. I'm going to opt to just collect more data for now and see how it goes." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Staring Round 3 of data collection\n", "\n", "This may take a while, so I'm going to use a cell magic command to notify me when this finishes." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%%notify\n", "game_data_dict23, new_seed_list3 = get_new_players(new_seed_list2, level_cap=5) # specifying level cap to be lower (default = 10)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Another successful run! We considered 564 players (not 1,000 like I estimated but still good!). This time with 48 summoners meeting our criteria. Our momentum is building!!**\n", "\n", "Let's take a quick look at some info from these players:" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# some quick checks\n", "match_count = 0\n", "below_level_3_count = 0\n", "level_3_count = 0\n", "above_level_3_count = 0\n", "\n", "for id in new_seed_list3:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " print(\"{0} ({1}) is level {2} with {3} matches. First match was {4}\".format(summoner.name, \n", " summoner.id, \n", " summoner.level, \n", " len(summoner.match_history),\n", " summoner.match_history[-1].creation.format(\"MM-DD-YYYY\")))\n", " match_count += len(summoner.match_history)\n", " if summoner.level < 3:\n", " below_level_3_count += 1\n", " elif summoner.level == 3:\n", " level_3_count += 1\n", " elif summoner.level > 3:\n", " above_level_3_count += 1\n", "\n", "print(\"\\n\")\n", "print(\"TOTAL MATCHES BETWEEN ALL NEW PLAYERS = {}\".format(match_count))\n", "print(\"NUMBER OF PLAYERS BELOW LEVEL 3 = {}\".format(below_level_3_count))\n", "print(\"NUMBER OF PLAYERS AT LEVEL 3 = {}\".format(level_3_count))\n", "print(\"NUMBER OF PLAYERS ABOVE LEVEL 3 = {}\".format(above_level_3_count))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see that we're starting to find players that didn't make it past level 3 but they are a relatively small percentage of the players we're collecting. This makes sense since low level players play fewer matches they will be harder for us to stumble upon. \n", "\n", "**Before we start Round 4, let's get together all the summoners already in our database:**" ] }, { "cell_type": "code", "execution_count": 57, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Round 2 seed list had 21 ids\n", "Round 3 seed list had 48 ids\n" ] } ], "source": [ "print(\"Round 2 seed list had {} ids\".format(len(new_seed_list2)))\n", "print(\"Round 3 seed list had {} ids\".format(len(new_seed_list3)))\n", "\n", "summoners_in_db = new_seed_list2 + new_seed_list3\n", "\n", "seed_list3 = new_seed_list3 # just want to change naming convention here" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Also because we've worked so hard to mine these summoner ids, let's save those to a file after each round just in case:**" ] }, { "cell_type": "code", "execution_count": 65, "metadata": { "collapsed": true }, "outputs": [], "source": [ "with open(\"current_seed_lists.txt\", \"w\") as file1:\n", " for summoner in seed_list3:\n", " file1.write(str(summoner)+\"\\n\")\n", " \n", "with open(\"summoners_in_db.txt\", \"w\") as file2:\n", " for summoner in summoners_in_db:\n", " file2.write(str(summoner)+\"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's also combine our databases for convenience:" ] }, { "cell_type": "code", "execution_count": 55, "metadata": { "collapsed": true }, "outputs": [], "source": [ "game_data_list_master = game_data_dict2 + game_data_dict23 # accidentally named game_data_dict3, \"game_data_dict23\"..." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Starting Round 4\n", "\n", "**note:** In this new round we're adding a `summoners_already_in_db` to avoid summoners we've already seen. Since we're starting to actually build momentum this will hopefully save us a lot of time in needless API calls." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%%notify -m \"Round 4 of Riot data collection complete!\"\n", "game_data_list4, seed_list4 = get_new_players(seed_list3, summoners_already_in_db = summoners_in_db, level_cap=5) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Another successful run!** Again let's take a look at the specific level breakdowns for these new players:" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244731542?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244704257?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244709001?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244708495?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244695549?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244676229?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244536171?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244676333?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244717994?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244707908?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246093994?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246331376?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246045381?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246039182?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246065394?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246036433?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246029981?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246063185?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246061249?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246052604?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246056397?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246058341?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246387338?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245257817?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246258521?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246228318?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246245769?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245232142?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245518637?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245986928?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246191991?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245971143?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245889024?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245972254?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246055070?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244929966?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245922268?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245789771?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245891572?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/242694162?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245866591?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246032536?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246048534?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246044738?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245981030?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246099272?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245949280?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246334958?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246314209?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246328496?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246305035?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246257381?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246182370?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246268548?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245992292?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246069209?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246053903?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/205968369?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246068857?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246036709?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246058520?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245994013?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245814118?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245885713?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245766781?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245899084?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245874946?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245897769?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245833976?beginIndex=0&endIndex=100\n", "\n", "\n", "TOTAL MATCHES BETWEEN ALL NEW PLAYERS = 281\n", "NUMBER OF PLAYERS BELOW LEVEL 3 = 24\n", "NUMBER OF PLAYERS AT LEVEL 3 = 12\n", "NUMBER OF PLAYERS ABOVE LEVEL 3 = 33\n" ] } ], "source": [ "# some quick checks\n", "match_count = 0\n", "below_level_3_count = 0\n", "level_3_count = 0\n", "above_level_3_count = 0\n", "\n", "for id in seed_list4:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " \n", " match_count += len(summoner.match_history)\n", " if summoner.level < 3:\n", " below_level_3_count += 1\n", " elif summoner.level == 3:\n", " level_3_count += 1\n", " elif summoner.level > 3:\n", " above_level_3_count += 1\n", "\n", "print(\"\\n\")\n", "print(\"TOTAL MATCHES BETWEEN ALL NEW PLAYERS = {}\".format(match_count))\n", "print(\"NUMBER OF PLAYERS BELOW LEVEL 3 = {}\".format(below_level_3_count))\n", "print(\"NUMBER OF PLAYERS AT LEVEL 3 = {}\".format(level_3_count))\n", "print(\"NUMBER OF PLAYERS ABOVE LEVEL 3 = {}\".format(above_level_3_count))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Great! There are actually quite a few new players below level 3.** But I'm aiming for a dataset of at least 1,000 players with at least 500 examples of a player below level 3 so this is about 0.5% of the data I need. Let's hope the next round is even better!" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Again, we'll do a bit of housekeeping before we're ready to move on to the next round of data collection:**" ] }, { "cell_type": "code", "execution_count": 72, "metadata": { "collapsed": true }, "outputs": [], "source": [ "game_data_list_master += game_data_list4 # current dataset... slowly building\n", "\n", "summoners_in_db += seed_list4 # running list of all ids already in database\n", "\n", "# save the current seed list\n", "with open(\"current_seed_lists.txt\", \"w\") as file1:\n", " for summoner in seed_list4:\n", " file1.write(str(summoner)+\"\\n\")\n", "\n", "# save all summoner ids that have been added to database already\n", "with open(\"summoners_in_db.txt\", \"w\") as file2:\n", " for summoner in summoners_in_db:\n", " file2.write(str(summoner)+\"\\n\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Starting Round 5 of data collection" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%%notify -m \"Round 5 of Riot data collection complete!\"\n", "game_data_list5, seed_list5 = get_new_players(seed_list4, summoners_already_in_db = summoners_in_db, level_cap=5) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Good News! It's still working!** We are still gaining momentum in building our database. Let's do another in-depth check to see how this is breaking down:" ] }, { "cell_type": "code", "execution_count": 79, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244730607?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244459338?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244603216?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/43573836?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244701400?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244704305?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244696435?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244714535?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246380132?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246374331?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246263152?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246063419?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245649347?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246061298?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245980711?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246012127?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246064583?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246037706?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245845772?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246024781?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/230828308?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245856386?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246361887?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246036931?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246059795?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246061647?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245533882?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246287990?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246303426?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245279370?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245277350?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246261196?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245469580?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246230027?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246260166?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244648410?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246067006?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246257836?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246237341?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246129748?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246061814?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246054826?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245840057?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245821638?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245750505?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245514719?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245476039?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245517765?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/51417249?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246217391?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246252579?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246016918?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246049808?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246128718?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245864736?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245997428?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245972797?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246075260?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246075110?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245802444?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246047483?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246040218?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246047796?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246128446?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246350386?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/42357163?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246334229?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246332967?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246334270?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/242250205?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246412066?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246325259?beginIndex=0&endIndex=100\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246122362?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245891544?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245143305?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246266881?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246258616?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246252665?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/241647014?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245998237?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246051784?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246055818?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245427878?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246056248?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/242773764?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/226837530?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246057129?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246042464?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246036854?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246105997?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246161410?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246114669?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246089285?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245990617?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246061668?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246055205?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246015745?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244696640?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245993387?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245995875?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245994298?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245909610?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245712837?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246005637?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246034414?beginIndex=0&endIndex=100\n", "\n", "\n", "TOTAL MATCHES BETWEEN ALL NEW PLAYERS = 428\n", "NUMBER OF PLAYERS BELOW LEVEL 3 = 33\n", "NUMBER OF PLAYERS AT LEVEL 3 = 12\n", "NUMBER OF PLAYERS ABOVE LEVEL 3 = 60\n" ] } ], "source": [ "# some quick checks\n", "match_count = 0\n", "below_level_3_count = 0\n", "level_3_count = 0\n", "above_level_3_count = 0\n", "\n", "for id in seed_list5:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " \n", " match_count += len(summoner.match_history)\n", " if summoner.level < 3:\n", " below_level_3_count += 1\n", " elif summoner.level == 3:\n", " level_3_count += 1\n", " elif summoner.level > 3:\n", " above_level_3_count += 1\n", "\n", "print(\"\\n\")\n", "print(\"TOTAL MATCHES BETWEEN ALL NEW PLAYERS = {}\".format(match_count))\n", "print(\"NUMBER OF PLAYERS BELOW LEVEL 3 = {}\".format(below_level_3_count))\n", "print(\"NUMBER OF PLAYERS AT LEVEL 3 = {}\".format(level_3_count))\n", "print(\"NUMBER OF PLAYERS ABOVE LEVEL 3 = {}\".format(above_level_3_count))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We see similar results as last round. Most players are level 4 or 5 but there are still a good amount of lower level players in this set. We've about doubled the amount of new ids from the last round to get a total of around 300. I should also note that this round took 2-3 hours to complete. **At this rate I expect to go 2-3 more rounds of data collection.** Each round should take significantly longer than the last (perhaps doubling each time?). So the last round might take the better part of a day. Because the Riot API keys I'm getting are only good for a day, I'll need to be mindful of when my key will expire before I start another round (maybe this is another good reason for not automating these rounds of collection)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Once again we have a few housekeeping steps.** We seem to have streamlined the process of housekeeping enough and we may have to do a few more rounds of data collection, so this time lets package it into a function to make it easier next time. _The function will be put up in \"Step 3\"_." ] }, { "cell_type": "code", "execution_count": 77, "metadata": { "collapsed": true }, "outputs": [], "source": [ "def end_of_round_housekeeping(master_db: list, round_db: list, master_id_list: list, round_list: list):\n", " master_db += round_db # current dataset... slowly building\n", "\n", " master_id_list += round_list # running list of all ids already in database\n", "\n", " # save the current seed list\n", " with open(\"current_seed_lists.txt\", \"w\") as file1:\n", " for summoner in round_list:\n", " file1.write(str(summoner)+\"\\n\")\n", "\n", " # save all summoner ids that have been added to database already\n", " with open(\"summoners_in_db.txt\", \"a\") as file2:\n", " for summoner in round_list:\n", " file2.write(str(summoner)+\"\\n\")\n", " \n", " # save last round's data to the other rounds as json\n", " with open(\"master_db.json\", 'a') as file3:\n", " json.dump(round_db, file3)\n", " \n", " return master_db, master_id_list" ] }, { "cell_type": "code", "execution_count": 78, "metadata": { "collapsed": true }, "outputs": [], "source": [ "game_data_list_master, summoners_in_db = end_of_round_housekeeping(master_db = game_data_list_master,\n", " round_db = game_data_list5,\n", " master_id_list = summoners_in_db,\n", " round_list = seed_list5)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Starting Round 6 of data collection " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%%notify -m \"Round 6 of Riot data collection complete!\"\n", "game_data_list6, seed_list6 = get_new_players(seed_list_of_summoners = seed_list5, \n", " summoners_already_in_db = summoners_in_db, \n", " level_cap=5) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**This round went smoothly, let's check out the breakdown:**" ] }, { "cell_type": "code", "execution_count": 85, "metadata": { "scrolled": true }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244626375?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244733460?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244464208?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244696963?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244704126?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244694447?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244661686?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244649651?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244707576?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244709609?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244710004?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244661418?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244706964?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244708157?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244651490?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244705240?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/243761357?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244591508?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244717872?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244717825?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244690674?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246419450?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246331656?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246332255?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/241233461?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245303081?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246036196?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245297873?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245981808?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246054467?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/219460977?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246063688?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246037332?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246036172?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246063549?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246080248?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245645517?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246057614?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245957087?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246258290?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246147544?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246044885?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246038118?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246057534?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245580868?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245585202?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245326708?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246261019?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246219472?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244646078?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244371364?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246312132?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246293646?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245907026?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245592552?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/243138582?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246231678?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246291216?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245634142?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246239082?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/230649037?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/243486628?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245982218?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246003547?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246050944?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245803596?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245847624?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245849861?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245809003?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245742752?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245624790?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245498115?beginIndex=0&endIndex=100\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245803463?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245825894?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245493143?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245414477?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245490646?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245411930?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245539968?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245527418?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245222406?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245507747?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245286277?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245723264?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246227477?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246250902?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246070812?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246044740?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246017689?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245944264?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245945584?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245896056?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245627187?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246088749?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246049310?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245898732?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246132862?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246129554?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246071308?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245561261?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246081762?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246077623?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246077722?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246076998?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245381798?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246090640?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246086897?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245997569?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246088057?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246177001?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246350232?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246179707?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246344472?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246359378?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246394274?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/227889574?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246102506?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246172324?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246151347?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246149046?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246145609?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246120738?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246123911?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244643649?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245950134?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244753036?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/243869397?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/244641116?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245047979?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245142768?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246255442?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246281270?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246282075?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246317094?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246318419?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246314558?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246292920?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245948391?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246169422?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246263571?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246278208?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246112150?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246124694?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245893181?beginIndex=0&endIndex=100\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246075111?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246075137?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246072477?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246014729?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246037095?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245432969?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245561240?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246057496?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/228114156?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245996591?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246057311?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246077324?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245551932?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246069494?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/240866542?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246005795?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246004575?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246019051?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/240769920?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245978553?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245742301?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246161048?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246156612?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246107332?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/211662479?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246121849?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246004429?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246117559?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246116610?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246118639?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246105282?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246074924?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246119337?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246106162?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246043235?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246027253?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246014862?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245993496?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246046224?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245718362?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245830782?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245933189?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245904468?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245868126?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/245715834?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246020366?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246034734?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246037724?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246030056?beginIndex=0&endIndex=100\n", "Making call: https://na1.api.riotgames.com/lol/match/v3/matchlists/by-account/246035500?beginIndex=0&endIndex=100\n", "\n", "\n", "TOTAL MATCHES BETWEEN ALL NEW PLAYERS = 920\n", "NUMBER OF PLAYERS BELOW LEVEL 3 = 49\n", "NUMBER OF PLAYERS AT LEVEL 3 = 32\n", "NUMBER OF PLAYERS ABOVE LEVEL 3 = 113\n" ] } ], "source": [ "# some quick checks\n", "match_count = 0\n", "below_level_3_count = 0\n", "level_3_count = 0\n", "above_level_3_count = 0\n", "\n", "for id in seed_list6:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " \n", " match_count += len(summoner.match_history)\n", " if summoner.level < 3:\n", " below_level_3_count += 1\n", " elif summoner.level == 3:\n", " level_3_count += 1\n", " elif summoner.level > 3:\n", " above_level_3_count += 1\n", "\n", "print(\"\\n\")\n", "print(\"TOTAL MATCHES BETWEEN ALL NEW PLAYERS = {}\".format(match_count))\n", "print(\"NUMBER OF PLAYERS BELOW LEVEL 3 = {}\".format(below_level_3_count))\n", "print(\"NUMBER OF PLAYERS AT LEVEL 3 = {}\".format(level_3_count))\n", "print(\"NUMBER OF PLAYERS ABOVE LEVEL 3 = {}\".format(above_level_3_count))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Looks good, let's do some housekeeping and move on to round 7.**" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "game_data_list_master, summoners_in_db = end_of_round_housekeeping(master_db = game_data_list_master,\n", " round_db = game_data_list6,\n", " master_id_list = summoners_in_db,\n", " round_list = seed_list6)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Oh no the kernel crashed! :(\n", "\n", "_Quick update:_\n", "\n", "I had an issue that forced me to restart the kernel (this is why we've been backing up our data!). So We're going to quickly read in the current seed list and complete list of summoners in db before starting round 7.\n", "\n", "ALSO, note how from this point on there are no printed output when a call is made to the Riot API... I realized that with so many calls being made/printed it was starting to slow down the notebook visually (scrolling and typing had slower response rates), I'm not sure if this was impacting the actual computational speed of the kernel but at any rate, we're not going to show those anymore. There are print statements built into my functions that will help us get a sense for how the data collection process is progressing instead." ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Summoners in current seed list = 194\n", "All summoners gathered so far = 506\n" ] } ], "source": [ "current_seed_list = []\n", "summoners_in_db = []\n", "\n", "with open('current_seed_lists.txt', 'r') as file1:\n", " lines = file1.readlines()\n", " for line in lines:\n", " current_seed_list.append(int(line.rstrip('\\n')))\n", " \n", "with open('summoners_in_db.txt', 'r') as file2:\n", " lines = file2.readlines()\n", " for line in lines:\n", " summoners_in_db.append(int(line.rstrip('\\n'))) \n", "\n", "print(\"Summoners in current seed list = {}\".format(len(current_seed_list)))\n", "print(\"All summoners gathered so far = {}\".format(len(summoners_in_db)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "OK great, we're back on track and ready for round 7." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Starting Round 7 of data collection" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%%notify -m \"Round 7 of Riot data collection complete!\"\n", "\n", "game_data_list7, seed_list7 = get_new_players(seed_list_of_summoners = current_seed_list, \n", " summoners_already_in_db = summoners_in_db, \n", " level_cap=5) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Great, another round done! And looks like we're still building momentum which is awesome. Let's be sure:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NUMBER OF PLAYERS BELOW LEVEL 3 = 81\n", "NUMBER OF PLAYERS AT LEVEL 3 = 76\n", "NUMBER OF PLAYERS ABOVE LEVEL 3 = 224\n" ] } ], "source": [ "# some quick checks\n", "below_level_3_count = 0\n", "level_3_count = 0\n", "above_level_3_count = 0\n", "\n", "for id in seed_list7:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " \n", " if summoner.level < 3:\n", " below_level_3_count += 1\n", " elif summoner.level == 3:\n", " level_3_count += 1\n", " elif summoner.level > 3:\n", " above_level_3_count += 1\n", "\n", "print(\"NUMBER OF PLAYERS BELOW LEVEL 3 = {}\".format(below_level_3_count))\n", "print(\"NUMBER OF PLAYERS AT LEVEL 3 = {}\".format(level_3_count))\n", "print(\"NUMBER OF PLAYERS ABOVE LEVEL 3 = {}\".format(above_level_3_count))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Perfect, we're still getting a good amount of all types of levels... let's do our housekeeping (saving and appending lists/data) before we move on." ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "collapsed": true }, "outputs": [], "source": [ "master_db, summoners_in_db = end_of_round_housekeeping(master_db = master_db,\n", " round_db = game_data_list7,\n", " master_id_list = summoners_in_db,\n", " round_list = seed_list7)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we're ready for round 8, but before that let's just see how big our database is right now:" ] }, { "cell_type": "code", "execution_count": 49, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "887 rows in our database with 116 columns each.\n" ] } ], "source": [ "print(\"{} rows in our database with {} columns each.\".format(master_db.shape[0],master_db.shape[1]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "So far it's looking like smooth sailing!... well... accept for the setbacks... and the slow, hackish building of our data collection functions... but otherwise compleeetely smooth!... let's just start round 8.\n", "\n", "### Starting Round 8 of data collection (possibly the final round)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Just kidding, slight hiccup... had to restart. So I put another function up in Step 3 so that I can quickly get back the most recent lists if the kernel ever crashes again." ] }, { "cell_type": "code", "execution_count": 38, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Summoners in current seed list = 381\n", "All summoners gathered so far = 887\n" ] } ], "source": [ "master_df, current_seed_list, summoners_in_db = recover_lists()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now we're ready!" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": true }, "outputs": [], "source": [ "%%notify -m \"Round 8 of Riot data collection complete!\"\n", "%%time\n", "\n", "game_data_list8, seed_list8 = get_new_players(seed_list_of_summoners = current_seed_list, \n", " summoners_already_in_db = summoners_in_db, \n", " level_cap=5) " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**We did it!** Over 8,000 players considered in this round. Unfortunately the vast majority of those were rejected due to being too high level, but we still got over 700. Let's see the level breakdown:" ] }, { "cell_type": "code", "execution_count": 40, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NUMBER OF PLAYERS BELOW LEVEL 3 = 189\n", "NUMBER OF PLAYERS AT LEVEL 3 = 142\n", "NUMBER OF PLAYERS ABOVE LEVEL 3 = 380\n" ] } ], "source": [ "# some quick checks\n", "below_level_3_count = 0\n", "level_3_count = 0\n", "above_level_3_count = 0\n", "\n", "for id in seed_list8:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " \n", " if summoner.level < 3:\n", " below_level_3_count += 1\n", " elif summoner.level == 3:\n", " level_3_count += 1\n", " elif summoner.level > 3:\n", " above_level_3_count += 1\n", "\n", "print(\"NUMBER OF PLAYERS BELOW LEVEL 3 = {}\".format(below_level_3_count))\n", "print(\"NUMBER OF PLAYERS AT LEVEL 3 = {}\".format(level_3_count))\n", "print(\"NUMBER OF PLAYERS ABOVE LEVEL 3 = {}\".format(above_level_3_count))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Last housekeeping step:**" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "collapsed": true }, "outputs": [], "source": [ "master_df, summoners_in_db = end_of_round_housekeeping(master_db = master_df,\n", " round_db = game_data_list8,\n", " master_id_list = summoners_in_db,\n", " round_list = seed_list8)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Quick peak at what we collected:**" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
assistschampLevelcombatPlayerScorecreepsPerMinDeltas_0-10creepsPerMinDeltas_10-20creepsPerMinDeltas_20-30creepsPerMinDeltas_30-endcsDiffPerMinDeltas_0-10csDiffPerMinDeltas_10-20csDiffPerMinDeltas_20-30...wardsPlacedwinxpDiffPerMinDeltas_0-10xpDiffPerMinDeltas_10-20xpDiffPerMinDeltas_20-30xpDiffPerMinDeltas_30-endxpPerMinDeltas_0-10xpPerMinDeltas_10-20xpPerMinDeltas_20-30xpPerMinDeltas_30-end
016.012.00.05.10.00.00.04.800.00.0...0.0True371.200.00.00.0583.90.00.00.0
14.011.00.03.50.00.00.01.720.00.0...3.0True84.660.00.00.0363.90.00.00.0
210.013.00.00.83.60.00.0NaNNaNNaN...0.0TrueNaNNaNNaNNaN331.8464.20.00.0
30.01.00.00.00.00.00.0NaNNaNNaN...0.0TrueNaNNaNNaNNaN0.00.00.00.0
47.017.00.04.14.03.10.04.104.03.0...1.0True538.60429.1783.30.0538.6429.1806.50.0
\n", "

5 rows × 116 columns

\n", "
" ], "text/plain": [ " assists champLevel combatPlayerScore creepsPerMinDeltas_0-10 \\\n", "0 16.0 12.0 0.0 5.1 \n", "1 4.0 11.0 0.0 3.5 \n", "2 10.0 13.0 0.0 0.8 \n", "3 0.0 1.0 0.0 0.0 \n", "4 7.0 17.0 0.0 4.1 \n", "\n", " creepsPerMinDeltas_10-20 creepsPerMinDeltas_20-30 \\\n", "0 0.0 0.0 \n", "1 0.0 0.0 \n", "2 3.6 0.0 \n", "3 0.0 0.0 \n", "4 4.0 3.1 \n", "\n", " creepsPerMinDeltas_30-end csDiffPerMinDeltas_0-10 \\\n", "0 0.0 4.80 \n", "1 0.0 1.72 \n", "2 0.0 NaN \n", "3 0.0 NaN \n", "4 0.0 4.10 \n", "\n", " csDiffPerMinDeltas_10-20 csDiffPerMinDeltas_20-30 ... \\\n", "0 0.0 0.0 ... \n", "1 0.0 0.0 ... \n", "2 NaN NaN ... \n", "3 NaN NaN ... \n", "4 4.0 3.0 ... \n", "\n", " wardsPlaced win xpDiffPerMinDeltas_0-10 xpDiffPerMinDeltas_10-20 \\\n", "0 0.0 True 371.20 0.0 \n", "1 3.0 True 84.66 0.0 \n", "2 0.0 True NaN NaN \n", "3 0.0 True NaN NaN \n", "4 1.0 True 538.60 429.1 \n", "\n", " xpDiffPerMinDeltas_20-30 xpDiffPerMinDeltas_30-end xpPerMinDeltas_0-10 \\\n", "0 0.0 0.0 583.9 \n", "1 0.0 0.0 363.9 \n", "2 NaN NaN 331.8 \n", "3 NaN NaN 0.0 \n", "4 783.3 0.0 538.6 \n", "\n", " xpPerMinDeltas_10-20 xpPerMinDeltas_20-30 xpPerMinDeltas_30-end \n", "0 0.0 0.0 0.0 \n", "1 0.0 0.0 0.0 \n", "2 464.2 0.0 0.0 \n", "3 0.0 0.0 0.0 \n", "4 429.1 806.5 0.0 \n", "\n", "[5 rows x 116 columns]" ] }, "execution_count": 3, "metadata": {}, "output_type": "execute_result" } ], "source": [ "master_df.head()" ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Number of summoners in dataset = 1598\n", "Number of features from each summoner = 116\n", "Average summoner level in dataset = 3.5\n" ] } ], "source": [ "print(\"Number of summoners in dataset = {}\".format(master_df.shape[0]))\n", "print(\"Number of features from each summoner = {}\".format(master_df.shape[1]))\n", "print(\"Average summoner level in dataset = {}\".format(round(np.average(master_df['summoner_level']),1)))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## BACK FROM THE GRAVE! (for round 9 of data collection)...\n", "\n", "After cleaning the data, performing analysis on the dataset, and building models with the dataset... I decided I want more data :)\n", "\n", "So here we are, back at the source to take a stab at round 9. Right now we have a little over a 1,000 players in the dataset, let's see how much we can gain in round 9!" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Summoners in current seed list = 711\n", "All summoners gathered so far = 1598\n" ] } ], "source": [ "master_df, current_seed_list, summoners_in_db = recover_lists()" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "fbcae695e9a945b18527cb4a6d548c6d", "version_major": 2, "version_minor": 0 }, "text/plain": [ "FloatProgress(value=0.0, bar_style='info', description='0%')" ] }, "metadata": {}, "output_type": "display_data" }, { "name": "stdout", "output_type": "stream", "text": [ "Currently 10 players in database.\n", "Currently 20 players in database.\n", "Currently 30 players in database.\n", "Currently 40 players in database.\n", "Currently 50 players in database.\n", "ERROR while processing player: saiuheadasirl (92681756).\n", "\n", "-1\n", "Currently 60 players in database.\n", "Currently 70 players in database.\n", "Currently 80 players in database.\n", "Currently 90 players in database.\n", "Currently 100 players in database.\n", "Currently 110 players in database.\n", "Currently 120 players in database.\n", "Currently 130 players in database.\n", "Currently 140 players in database.\n", "ERROR while processing player: Evanisugly (92300802).\n", "\n", "-1\n", "Currently 150 players in database.\n", "Currently 160 players in database.\n", "Currently 170 players in database.\n", "Currently 180 players in database.\n", "Currently 190 players in database.\n", "Currently 200 players in database.\n", "Currently 210 players in database.\n", "Currently 220 players in database.\n", "Currently 230 players in database.\n", "Currently 240 players in database.\n", "Currently 250 players in database.\n", "Currently 260 players in database.\n", "ERROR while processing player: Platinum Feed (93782806).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 500: \"Internal server error\"\n", "ERROR while processing player: ninaa1562 (93782025).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 500: \"Internal server error\"\n", "Currently 270 players in database.\n", "Currently 280 players in database.\n", "ERROR while processing player: Olion09 (93869427).\n", "\n", "-1\n", "Currently 290 players in database.\n", "Currently 300 players in database.\n", "Currently 310 players in database.\n", "Currently 320 players in database.\n", "ERROR while processing player: behnam1234 (93819607).\n", "\n", "-1\n", "Currently 330 players in database.\n", "ERROR while processing player: DeepShadow11 (93690000).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "Currently 340 players in database.\n", "Currently 350 players in database.\n", "Problem checking if candidate player.\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 500: \"Internal server error\"\n", "Currently 360 players in database.\n", "Currently 370 players in database.\n", "Currently 380 players in database.\n", "Currently 390 players in database.\n", "Currently 400 players in database.\n", "Currently 410 players in database.\n", "Currently 420 players in database.\n", "Currently 430 players in database.\n", "ERROR while processing player: DeezNutz209 (93802265).\n", "\n", "-1\n", "Currently 440 players in database.\n", "Currently 450 players in database.\n", "ERROR while processing player: supermarto (93642100).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "ERROR while processing player: GaZ3rN185 (93612133).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "Currently 460 players in database.\n", "Currently 470 players in database.\n", "Currently 480 players in database.\n", "Currently 490 players in database.\n", "Currently 500 players in database.\n", "Currently 510 players in database.\n", "Problem checking if candidate player.\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 500: \"Internal server error\"\n", "Currently 520 players in database.\n", "Currently 530 players in database.\n", "Currently 540 players in database.\n", "Currently 550 players in database.\n", "ERROR while processing player: Sslasherr (93649850).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 500: \"Internal server error\"\n", "Currently 560 players in database.\n", "ERROR while processing player: FucciSlot1 (93289688).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "ERROR while processing player: ThePugLord95 (93340465).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "Currently 570 players in database.\n", "ERROR while processing player: JCaulley7 (93311088).\n", "\n", "-1\n", "Currently 580 players in database.\n", "Currently 590 players in database.\n", "Currently 600 players in database.\n", "Currently 610 players in database.\n", "Currently 620 players in database.\n", "Currently 630 players in database.\n", "Currently 640 players in database.\n", "Currently 650 players in database.\n", "Currently 660 players in database.\n", "Currently 670 players in database.\n", "Currently 680 players in database.\n", "Currently 690 players in database.\n", "Currently 700 players in database.\n", "Currently 710 players in database.\n", "Currently 720 players in database.\n", "Currently 730 players in database.\n", "Currently 740 players in database.\n", "Currently 750 players in database.\n", "Currently 760 players in database.\n", "Currently 770 players in database.\n", "Currently 780 players in database.\n", "Currently 790 players in database.\n", "Currently 800 players in database.\n", "Currently 810 players in database.\n", "Currently 820 players in database.\n", "Currently 830 players in database.\n", "Currently 840 players in database.\n", "Currently 850 players in database.\n", "ERROR while processing player: NabeogreA (93660604).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "ERROR while processing player: FauxThunder7603 (93760271).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "Currently 860 players in database.\n", "Currently 870 players in database.\n", "Currently 880 players in database.\n", "Currently 890 players in database.\n", "Currently 900 players in database.\n", "Currently 910 players in database.\n", "Currently 920 players in database.\n", "Currently 930 players in database.\n", "Currently 940 players in database.\n", "Currently 950 players in database.\n", "Currently 960 players in database.\n", "Currently 970 players in database.\n", "Currently 980 players in database.\n", "Currently 990 players in database.\n", "ERROR while processing player: FauacakLoL (93802499).\n", "\n", "-1\n", "Currently 1000 players in database.\n", "Currently 1010 players in database.\n", "Currently 1020 players in database.\n", "Currently 1030 players in database.\n", "Currently 1040 players in database.\n", "Currently 1050 players in database.\n", "Currently 1060 players in database.\n", "Currently 1070 players in database.\n", "Currently 1080 players in database.\n", "Currently 1090 players in database.\n", "Currently 1100 players in database.\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Currently 1110 players in database.\n", "ERROR while processing player: TaraNoodlez (93839565).\n", "\n", "-1\n", "Currently 1120 players in database.\n", "Currently 1130 players in database.\n", "Currently 1140 players in database.\n", "Currently 1150 players in database.\n", "Currently 1160 players in database.\n", "Currently 1170 players in database.\n", "Currently 1180 players in database.\n", "Currently 1190 players in database.\n", "ERROR while processing player: BenDurDonDat (93060028).\n", "\n", "-1\n", "Currently 1200 players in database.\n", "Currently 1210 players in database.\n", "Currently 1220 players in database.\n", "ERROR while processing player: BigDaddyMeats (92110852).\n", "\n", "-1\n", "Currently 1230 players in database.\n", "Currently 1240 players in database.\n", "Currently 1250 players in database.\n", "ERROR while processing player: sindysus (93350379).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "ERROR while processing player: CorporalChemDawg (93029816).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "ERROR while processing player: LilninjaKilla619 (93230852).\n", "\n", "The Riot API experienced an internal error on the request. You may want to retry the request after a short wait or continue without the result. The received error was 503: \"Service unavailable\"\n", "Currently 1260 players in database.\n", "DONE! \n", "\n", "Players that were considered for database = 15227\n", "Players that were added to database = 1265\n", "Players denied b/c of level cap = 13796\n", "Players denied b/c account too young = 47\n", "Players denied b/c account too old = 119\n", "Wall time: 19h 36s\n" ] }, { "data": { "application/javascript": [ "$(document).ready(\n", " function() {\n", " function appendUniqueDiv(){\n", " // append a div with our uuid so we can check that it's already\n", " // been sent and avoid duplicates on page reload\n", " var notifiedDiv = document.createElement(\"div\")\n", " notifiedDiv.id = \"8d2b78dc-63b2-4d88-b5f3-47a07e1f1738\"\n", " element.append(notifiedDiv)\n", " }\n", "\n", " // only send notifications if the pageload is complete; this will\n", " // help stop extra notifications when a saved notebook is loaded,\n", " // which during testing gives us state \"interactive\", not \"complete\"\n", " if (document.readyState === 'complete') {\n", " // check for the div that signifies that the notification\n", " // was already sent\n", " if (document.getElementById(\"8d2b78dc-63b2-4d88-b5f3-47a07e1f1738\") === null) {\n", " var notificationPayload = {\"requireInteraction\": true, \"icon\": \"/static/base/images/favicon.ico\", \"body\": \"Round 9 of Riot data collection complete!\"};\n", " if (Notification.permission !== 'denied') {\n", " if (Notification.permission !== 'granted') { \n", " Notification.requestPermission(function (permission) {\n", " if(!('permission' in Notification)) {\n", " Notification.permission = permission\n", " }\n", " })\n", " }\n", " if (Notification.permission === 'granted') {\n", " var notification = new Notification(\"Jupyter Notebook\", notificationPayload)\n", " appendUniqueDiv()\n", " notification.onclick = function () {\n", " window.focus();\n", " this.close();\n", " };\n", " } \n", " } \n", " }\n", " }\n", " }\n", ")\n" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "%%notify -m \"Round 9 of Riot data collection complete!\"\n", "%%time\n", "\n", "game_data_list9, seed_list9 = get_new_players(seed_list_of_summoners = current_seed_list, \n", " summoners_already_in_db = summoners_in_db, \n", " level_cap=5) " ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "NUMBER OF PLAYERS BELOW LEVEL 3 = 335\n", "NUMBER OF PLAYERS AT LEVEL 3 = 253\n", "NUMBER OF PLAYERS ABOVE LEVEL 3 = 677\n" ] } ], "source": [ "# some quick checks\n", "below_level_3_count = 0\n", "level_3_count = 0\n", "above_level_3_count = 0\n", "\n", "for id in seed_list8:\n", " summoner = Summoner(id=id, region=\"NA\")\n", " \n", " if summoner.level < 3:\n", " below_level_3_count += 1\n", " elif summoner.level == 3:\n", " level_3_count += 1\n", " elif summoner.level > 3:\n", " above_level_3_count += 1\n", "\n", "print(\"NUMBER OF PLAYERS BELOW LEVEL 3 = {}\".format(below_level_3_count))\n", "print(\"NUMBER OF PLAYERS AT LEVEL 3 = {}\".format(level_3_count))\n", "print(\"NUMBER OF PLAYERS ABOVE LEVEL 3 = {}\".format(above_level_3_count))" ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [], "source": [ "master_df, summoners_in_db = end_of_round_housekeeping(master_db = master_df,\n", " round_db = game_data_list9,\n", " master_id_list = summoners_in_db,\n", " round_list = seed_list9)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Terrific! With this new round of data collection we've almost doubled the size of our previous dataset. Now we'll go send it through cleaning and see how it does on our models." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "***" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "
\n", " back to top\n", "
\n", "\n", "## End of Part 1\n", "\n", "### In the next notebook, we'll make sure the data is clean and do a little exploratory analysis." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.3" } }, "nbformat": 4, "nbformat_minor": 2 }