{ "cells": [ { "cell_type": "markdown", "id": "artificial-trauma", "metadata": {}, "source": [ "# Capturing Twitter data with Twarc\n", "\n", "\n", "# Background\n", "During the protests in July 2019 asking for the resignation of the Governor of Puerto Rico, Joel Blanco-Rivera began capturing tweets with twarc, a command line tool and Python library for archiving Twitter data. This tool was selected after reading about its application for archiving twitter data of other events by members of Documenting the Now. The goal was to capture tweets about the protests that used the hashtag #RickyRenuncia, which became the protest slogan in social media.\n", "\n", "This module uses the experience of archiving #RickyRenuncia Twitter data to explain how to capture Twitter data with twarc, generate reports and export the data to csv and geojson files. Through the module, you will be able to create and curate your own dataset.\n", "\n", "# Objectives\n", "
  • Capture and curate twitter data with twarc.
    • \n", "
  • Generate reports about the contents of a Twitter dataset.
    • \n", "
  • Export a JSON file to CSV and GeoJson.
    • \n", "## Learning goals\n", "
  • Understand the main requirements for installing twarc and getting access to Twitter's API.
    • \n", "
  • Learn about the structure of command lines to run twarc with Python.
    • \n", "
  • Learn about Twitter API for Academic Research and version two of Twarc.
    • \n", "## Expected Interaction\n", "The main function of this module is to provide information about installing and running twarc. To use twarc, users will need to install and set up the tool in their computers.\n", "\n" ] }, { "cell_type": "markdown", "id": "dense-registrar", "metadata": {}, "source": [ "## About Twarc\n", "Twarc is a command line tool and Python library for arhiving Twitter data. It uses Twitter API to search and capture tweets according to parameters given in the search command line. The captured tweets are saved as a JSON file. It was developed by Ed Summers from the Maryland Institute for Technology in the Humanities. It is also part of Documenting the Now, a project that creates tools for social media archiving to chronicle significant events, prioritizing ethical practices in the collection and preservation of social media content (https://www.docnow.io)." ] }, { "cell_type": "markdown", "id": "ecological-cloud", "metadata": {}, "source": [ "## Installing twarc\n", "There are two things to do before installing Twarc. First, you need to register an application at https://developer.twitter.com. Second, you need to install Python if you don't have it installed yet. After these two steps, follow the installation instructions in https://twarc-project.readthedocs.io/en/latest. It includes instructions for Mac and Windows. For this tutorial, twarc was used in a Mac and the commands are run in the application Terminal. The program is stored in a \"twarc-master\" folder. Once twarc is set up, we can begin collecting tweets.\n" ] }, { "cell_type": "markdown", "id": "royal-athens", "metadata": {}, "source": [ "### Capturing and saving tweets\n", "Open Terminal and change the directory to where the folder twarc-master is located: " ] }, { "cell_type": "code", "execution_count": null, "id": "innocent-question", "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image\n", "\n", "print ('Changing the directory in Terminal')\n", "\n", "local_image = Image (filename='./images/twarc-rickyrenuncia-1.gif')\n", "\n", "local_image" ] }, { "cell_type": "markdown", "id": "conditional-imaging", "metadata": {}, "source": [ "The basic command line for capturing tweets is \"twarc search\" and the search query. In this example the query is to search for tweets with #RickyRenuncia and save it as a JSON file: \n", "

    twarc search '#RickyRenuncia' > tweets_rickyrenuncia_20210208.jsonl


    \n", "\n", "The JSON file is saved in the twarc-master folder.\n", " " ] }, { "cell_type": "markdown", "id": "graphic-activation", "metadata": {}, "source": [ "

    Since Twitter API only captures tweets from seven to nine days back, it is very possible that you will have multiple json files from the same search parameters. In this case, you can combine all files and save them into one JSON file.

    " ] }, { "cell_type": "code", "execution_count": null, "id": "vietnamese-chest", "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image\n", "\n", "print ('Combine multiple files')\n", "\n", "local_image = Image (filename='./images/twarc-rickyrenuncia-2.gif')\n", "\n", "local_image" ] }, { "cell_type": "markdown", "id": "revised-belief", "metadata": {}, "source": [ "The above clip shows the use of the command line cat, where you can list the filename of all the files you want to combine. But you can also truncate the file name, as shown in the clip. This means that twarc will identify all the files that begin with tweetsRickyRenuncia and save them into one JSON file. Make sure that all the files are stored in the same folder." ] }, { "cell_type": "markdown", "id": "premier-surrey", "metadata": {}, "source": [ "Since there might be overlap in the tweets stored in the individual files, the combined JSON file will have duplicates. Here you can use one of the tools that are part of Twarc utilities, and saved in the /utils folder. To eliminate duplicates use the command line deduplicate.\n", "

    .utils/deduplicate.py tweetsRickyRenuncia.jsonl > tweetsRickyRenuncia_deduplicate.jsonl

    " ] }, { "cell_type": "markdown", "id": "02022a9d-4c6b-4806-8dda-46a7770afcc9", "metadata": {}, "source": [ "### Activity\n", "Identify a topic or event and create your own dataset of tweets using Twarc. \n", "
  • Did you find any difficulties installing and/or running twarc?
    • \n", "
  • If yes, which were the difficulties? How would you solve them?
    • \n" ] }, { "cell_type": "markdown", "id": "abroad-variety", "metadata": {}, "source": [ "## Converting to CSV and GeoJSON formats\n", "Once you have a dataset you can use a number of tools to create reports and visualizations. We will cover this later. You can also convert the json file to other formats, like a CSV table and a GeoJSON file. Converting to other formats will open the doors for additional data analysis and visualization strategies.\n", "\n", "To convert your dataset into a CSV table, use the json2csv.py script, located in the /utils folder. The command line is:
    \n", "

    ./utils/json2csv.py tweetsRickyRenuncia_deduplicate.jsonl > tweetsRickyRenuncia_deduplicate.csv

    \n", "GeoJSON is an open standard format used to represent geographical features. A GeoJSON file of the tweets can be used for data visualization, like the following example:" ] }, { "cell_type": "code", "execution_count": null, "id": "forbidden-colorado", "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image\n", "\n", "print ('Map with pins of tweets by location')\n", "\n", "local_image = Image (filename='./images/mapaTweetsRickyRenuncia1.png')\n", "\n", "local_image" ] }, { "cell_type": "markdown", "id": "separated-policy", "metadata": {}, "source": [ "

    To create a GeoJSON file you will use the command line geojson.py:

    \n", "

    ./utils/geojson.py tweetsRickyRenuncia_deduplicate.jsonl > tweetsRickyRenuncia.geojson

    " ] }, { "cell_type": "markdown", "id": "cross-discretion", "metadata": {}, "source": [ "It is important to mention that not all the collected tweets have coordinates. Indeed, is very probable that just a small percentage will include geo data. Therefore, the map example above is just a representation of part of all the collected tweets." ] }, { "cell_type": "markdown", "id": "2289d5cc-5344-4c91-b825-57dde2122cc1", "metadata": {}, "source": [ "### Activity\n", "Convert your dataset to CSV and GeoJSON data. What type of data analysis, representation or visualization you can do with these converted files?" ] }, { "cell_type": "markdown", "id": "clinical-scenario", "metadata": {}, "source": [ "## Twarc-report\n", "Twarc-report contains a number of utilities to generate reports about the dataset and for visualizations. You can download the file and read the installation instructions here: https://github.com/pbinkley/twarc-report. It is important to place the twarc-report folder within the twarc-master folder." ] }, { "cell_type": "code", "execution_count": null, "id": "64b76ada-89cd-4589-92a5-1e4957fe27c4", "metadata": {}, "outputs": [], "source": [ "from IPython.display import Image\n", "\n", "print ('Change directory')\n", "\n", "local_image = Image (filename='./images/twarc-rickyrenuncia-3.gif')\n", "\n", "local_image" ] }, { "cell_type": "markdown", "id": "polished-assets", "metadata": {}, "source": [ "

    The command line to generate a report is reportprofile.py.

    " ] }, { "cell_type": "markdown", "id": "located-rapid", "metadata": {}, "source": [ "

    For more information on the utilities of Twarc-report visit https://github.com/pbinkley/twarc-report.

    " ] }, { "cell_type": "markdown", "id": "bba31e5f-53e4-4401-9f18-5e560eaaf303", "metadata": {}, "source": [ "### Activity\n", "Create a report of your dataset. \n", "
  • How many tweets did you collect?
  • What are the three more shared images?
    • " ] }, { "cell_type": "markdown", "id": "e8104979-5010-45c2-9427-d2e540f8adfe", "metadata": {}, "source": [ "## twarc2\n", "In 2020, Twitter released version 2 of its API, which included the \"Twitter API for Academic Research\" or \"Academic API\". This service allows a user access to Twitter's full archive. One of the advantages this track is that you will not have the 7 to 9 days limit to capture data with twarc. Developers released version 2 of twarc, updating the tool to make it compatible with the Academic API.\n", "\n", "If you would like to capture data using the Academic API, there are two important things to do. First, you need to apply for an \"Academic project\" with Twitter. You can read more about the requirements and about how to apply for an Academic API [here](https://developer.twitter.com/en/products/twitter-api/academic-research).\n", "\n", "Second, you need to upgrade twarc. Follow the instructions [here](https://github.com/jeffcsauer/twarc-v2-tutorials/blob/master/twarc_fas.md).\n", "In the following video, Ed Summers provides a quick demo to use twarc2 and access the full Twitter archive with the Academic API:" ] }, { "cell_type": "markdown", "id": "661ae3f8-98b3-4d51-9923-2b27bf2be9ee", "metadata": {}, "source": [ "" ] }, { "cell_type": "code", "execution_count": null, "id": "20fe1f81-5d04-443c-9b8f-d37f7130616b", "metadata": {}, "outputs": [], "source": [ "from IPython.display import YouTubeVideo\n", "# quick demo of how to use twarc2 to talk to the Twitter V2 API and access the full historical archive using the Academic Research Product Track.\n", "# Video credit: Ed Summers.\n", "YouTubeVideo('t1kT719vxlQ')" ] }, { "cell_type": "markdown", "id": "fc54756b-9c0c-4625-aa33-1a62cbbdb75f", "metadata": {}, "source": [ "## Discussion questions\n", "\n", "
  • Researchers use data science tools to collect and analyze twitter data. In addition, Twitter already provides access to academic researchers to its whole archive. Do you think archivists should also collect Twitter data? Why?
    • \n", "
  • What considerations archivists should take into account when preserving Twitter data?
    • \n", "
  • What are the implications of APIs in relation to the preservation of social media content.
    • \n", "    " ] }, { "cell_type": "markdown", "id": "e576436a-5e55-4709-a750-5f8719278960", "metadata": {}, "source": [ "## What's next?\n", "\n", "Continue to [Evaluating Content](./3-Evaluating_Content.ipynb) to continue with the experience." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }