{ "cells": [ { "cell_type": "markdown", "id": "e20d58b8-4915-4c41-bd49-c7fa4f3fa27b", "metadata": {}, "source": [ "# **Evaluating Content**\n", "\n", "Although the search processed used by the team to retrieve the Tweet ID list was rigurous, some materials in the list may be unrelated. It is possible some posts match a hashtag but discuss subjects unrelated to the historical events of #RickyRenuncia movement from summer 2019.\n", "\n", "\n", "\n", "## Objectives\n", "\n", "This notebook presents a minimal IPython graphical user interface (GUI) where participants, researchers and members of the original team could interact with content and classify it.\n", "\n", "### Learning Goals\n", "- Interact with twitter embedings.\n", "- Update SQLite3 database.\n", "- Classify content (minimum of 20 tweets).\n", "- Visualize state of the database.\n", "\n", "## Requirements\n", "\n", "**Tweeter API Credentials**\n", "\n", "The user will need to have created the `twitter_secrets.py` file based on `twitter_secrets_example.py` and set the variables to his API specifications. See [Twitter API Credentials](./Developer_Registration.ipynb) section." ] }, { "cell_type": "markdown", "id": "fcf73d3a-e4b7-4e8e-ae1b-924aa44b016e", "metadata": {}, "source": [ "## Optional Requirements\n", "\n", "**OPTIONAL**\n", "\n", "**Google API Credentials**\n", "\n", "A `google.oauth2.service_account.Credentials` object is required to interact with the google translate API to automatically see translations of text. This should help non-Spanish speakers interact with content in Spanish.\n", "\n", "The user will need to have created/edited the `google_translate_keys.json` following the [Google API Credentials](./Developer_Registration.ipynb#Google-API-Credentials) section. This is **optional**, but will offer the user automatic translation of tweet text content to english (or other language)." ] }, { "cell_type": "markdown", "id": "14658cf3-dbb8-4cdb-9e4f-19865a7cecf1", "metadata": {}, "source": [ "## Import Libraries\n", "\n", "Add Library justifications" ] }, { "cell_type": "code", "execution_count": 2, "id": "34f2339e-93eb-4b7f-97c0-373f750b8568", "metadata": {}, "outputs": [], "source": [ "import ipywidgets as widgets\n", "from IPython.core.display import display, HTML, update_display\n", "import json, os, pickle\n", "from random import seed, randint\n", "# from tweet_requester.analysis import TweetJLAnalyzer, TweetAnalyzer\n", "from tweet_requester.display import TweetInteractiveClassifier, \\\n", "JsonLInteractiveClassifier, TSess, prepare_google_credentials, PROCESSING_STAGES\n", "# from twitter_secrets import C_BEARER_TOKEN \n", "from twitter_secrets_testing import C_BEARER_TOKEN \n", "JL_DATA=\"./tweetsRickyRenuncia-final.jsonl\"\n", "CACHE_DIR=\"tweet_cache\"\n", "SQLite_DB=\"tweets.db\"" ] }, { "cell_type": "markdown", "id": "7b8d5abf-cfaf-4648-9e00-a8376fe8b4e9", "metadata": {}, "source": [ "## Download Data Set\n", "\n", "A (dataset)[https://ia601005.us.archive.org/31/items/tweetsRickyRenuncia-final/tweetsRickyRenuncia-final.txt] of the tweets related to the investigation is public online. This list will be used as a basis for research.from os import isfile" ] }, { "cell_type": "code", "execution_count": 3, "id": "492268e2-481a-41ce-b455-02ce893a9288", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Data available at ./tweetsRickyRenuncia-final.jsonl.\n" ] } ], "source": [ "import requests\n", "from os.path import isfile, isdir\n", "\n", "tweet_list_url = \"https://ia601005.us.archive.org/31/items/tweetsRickyRenuncia-final/tweetsRickyRenuncia-final.txt\"\n", "\n", "# Download dataset if not present\n", "if not isfile(JL_DATA):\n", " response = requests.get(tweet_list_url)\n", " with open(JL_DATA, 'wb') as handler:\n", " handler.write(response.content)\n", " print(f\"Data downloaded at {JL_DATA}.\")\n", "else:\n", " print(f\"Data available at {JL_DATA}.\")" ] }, { "cell_type": "markdown", "id": "aee6766c-3b84-45d6-a068-21c6a995afd8", "metadata": {}, "source": [ "## Download Database and Cache\n", "\n", "The code bellow uses a combination of python logic and terminal commands to download the *compressed* **database** and **cache**, and then extracted only if needed. Terminals commands can be addentified by the `!` symbol at the beggining of the line. The commands `wget`, `tar` and `gzip` are [Bash](https://en.wikipedia.org/wiki/Bash_(Unix_shell)) commands available outside of python.\n", "\n", "A version of the database and cache is being shared publicly through the [#RickyRenuncia Project Scalar Book](https://libarchivist.com/rrp/rickyrenuncia/index). The database and cache should make the experience" ] }, { "cell_type": "code", "execution_count": 5, "id": "cb516037-2015-46d9-8453-0d46a0c40196", "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Database already available\n", "Database already extracted.\n", "Compressed Cache already available!\n", "Cache already extracted!\n" ] } ], "source": [ "from os.path import isfile, isdir\n", "\n", "tmp_file = \"tweet_cache.tar.gz\"\n", "tweets_db_url = \"https://libarchivist.com/rrp/rickyrenuncia//tweets.db.gz\"\n", "tweet_cache_url = \"https://libarchivist.com/rrp/rickyrenuncia//tweet_cache.tar.gz\"\n", "\n", "# Download database if not present\n", "if not isfile(\"tweets.db.gz\"):\n", " !wget \"$tweets_db_url\"\n", "else:\n", " print(\"Database already available\")\n", "\n", "# Extract database if not present\n", "if not isfile(\"tweets.db\"):\n", " !gzip -kd \"tweets.db.gz\"\n", "else:\n", " print(\"Database already extracted.\")\n", "\n", "# Download dataset if not present\n", "if not isfile(tmp_file):\n", " !wget \"$tweet_cache_url\"\n", "else:\n", " print(\"Compressed Cache already available!\")\n", "\n", "# Extract the cache if not present\n", "if not isdir(CACHE_DIR):\n", " !tar -xf $tmp_file\n", " print(\"Cache extracted!\")\n", "else:\n", " print(\"Cache already extracted!\")\n" ] }, { "cell_type": "markdown", "id": "69a9c916-785b-49d2-8160-db18af35b795", "metadata": {}, "source": [ "## Create a TSess\n", "The `TSess` object stores configuration and controls the connection used to retrieve content from the Twitter API. It is this object that requires your twitter credentials to create a connection.\n", "\n", "**Tweeter API Credentials** are required to create the session." ] }, { "cell_type": "code", "execution_count": 4, "id": "98ae133a-8172-4ef1-845c-36bc53415b4b", "metadata": {}, "outputs": [], "source": [ "tweet_session = TSess(\n", " C_BEARER_TOKEN, \n", " compression_level=5, \n", " sleep_time=3, # Minimal sleep between requests to avoid hitting rate limits\n", " cache_dir=CACHE_DIR, \n", " hash_split=True\n", " )" ] }, { "cell_type": "markdown", "id": "6102afde-5195-4737-a40d-70a4c6b6bc58", "metadata": {}, "source": [ "The session even include rate limiting for requests. For bearer token app authentication the limit is 300 tweet lookups each 15 minutes (900 seconds). In other words 3 seconds per tweet. Read more at \"[Rate limits | Docs | Twitter Developer Platform](https://developer.twitter.com/en/docs/twitter-api/rate-limits)\"." ] }, { "cell_type": "markdown", "id": "88b6d5e0-1b44-4d25-a80e-e4ffbf458004", "metadata": {}, "source": [ "## Create Google Translate Credentials\n", "\n", "After following the **optional** instructions from [Google API Credentials](./Developer_Registration.ipynb#Google-API-Credentials) run the code bellow. If the user did not acquire any credentials the code will default to no credentials." ] }, { "cell_type": "code", "execution_count": 5, "id": "714020af-b6cb-475a-afcd-622e4478a42e", "metadata": {}, "outputs": [], "source": [ "google_credentials = prepare_google_credentials(\n", " credentials_file=\"./google_translate_keys_testing.json\"\n", ")" ] }, { "cell_type": "markdown", "id": "15b5905b-dfe9-4a0b-9d03-c492d3fd4b29", "metadata": {}, "source": [ "## Create a JsonLInteractiveClassifier\n", "\n", "A JsonLInteractiveClassifier object handles interactions with a local SQLite database, Twitter API (importing a TSess) and a GUI for early data curation. \n", "\n", "In terms of the data integration process , the process of capturing the data from the API would fall under **Extraction**, the GUI for additional metadata and the SQLite database would fall under **Transform** as information is being stored in a different format for easier analysis. Any methods used to visualize the data or access attributes with less effort would fall under **Load**.\n", "\n", "In that sense the `JsonLInteractiveClassifier` handles multiple stages on the **ETL**. \n", "\n", "
\n", "If you downloaded the database and cache on the previous steps continue with the next code. The parameter `pre_initialized` is set as `True` to continue working with previous version of the database.\n", "
" ] }, { "cell_type": "code", "execution_count": 6, "id": "abbca9c3-5dee-4553-b19c-e76f438c0111", "metadata": {}, "outputs": [], "source": [ "classifier = JsonLInteractiveClassifier(\n", " tweet_ids_file=JL_DATA, \n", " session=tweet_session, mute=True, \n", " google_credentials=google_credentials,\n", " sqlite_db=SQLite_DB, pre_initialized=True)\n", "\n", "# classifier = JsonLInteractiveClassifier(\n", "# tweet_ids_file=JL_DATA, \n", "# session=tweet_session, mute=True, \n", "# google_credentials=google_credentials,\n", "# sqlite_db=SQLite_DB, pre_initialized=False)" ] }, { "cell_type": "markdown", "id": "3e3cf075-b353-451e-80c1-628722e03400", "metadata": {}, "source": [ "# **JsonLInteractiveClassifier**\n", "\n", "## Where are results and details stored?\n", "\n", "The `JsonLInteractiveClassifier` object includes a builtin SQLite3 database connection in the attribute `db`. It is possible to directly access the database after a connection is made.\n", "\n", "The code bellow displays the tables and columns of the relational database." ] }, { "cell_type": "code", "execution_count": null, "id": "3a3d3488-f4ad-44a1-b0fd-85a5686768ba", "metadata": {}, "outputs": [], "source": [ "# Connect to the database\n", "classifier.connect()\n", "\n", "# Extract database object and create a cursor\n", "database = classifier.db\n", "cursor = database.cursor()\n", "\n", "# SQL Command Display tables and columns\n", "# Total of tweets with slang among the processed.\n", "cursor.execute(\"\"\"SELECT name FROM sqlite_master WHERE type='table';\"\"\")\n", "tweet_schema = cursor.fetchall()\n", "for table in tweet_schema:\n", " print(\"Table:\",table[0])\n", " # Get Column Names\n", " cursor.execute(f\"\"\"SELECT * FROM {table[0]} LIMIT 1\"\"\")\n", " column_names = list(map(lambda x: x[0],cursor.description))\n", " print(\"Columns:\", column_names, \"\\n\\n\")\n", " \n", "# Close the cursor\n", "cursor.close()" ] }, { "cell_type": "markdown", "id": "7b367f12-20e4-4bd9-945c-803b74eff49f", "metadata": {}, "source": [ "The database is composed of 3 tables: `tweet`, `tweet_details`, and `tweet_traduction`.\n", "\n", "1. Table `tweet` includes only 2 columns the `tweet_id`, a string indicating the unique tweet_id, and `state`, an integer representing the processing stage of the tweet.\n", "\n", "2. Table `tweet_details` includes 6 columns. Bellow you can see the SQL command used to create the table.\n", "```\n", "CREATE TABLE tweet_detail (\n", " tweet_id TEXT,\n", " has_media INTEGER,\n", " description TEXT,\n", " is_meme INTEGER,\n", " language TEXT,\n", " has_slang INTEGER,\n", " PRIMARY KEY(\"tweet_id\"))\n", "```\n", "\n", "3. Table `tweet_traduction` includes 3 columns: `tweet_id`, `target_language_code` and `traduction`. This table works as a cache for storing google translation of a tweet's text in one or more languages." ] }, { "cell_type": "markdown", "id": "61d67768-209a-4e07-a266-092a8d3d3c70", "metadata": {}, "source": [ "## **Generate a Tweet State Report**\n", "\n", "It is possible to directly interact with the built-in database by accesing the SQLite3 database connection. Bellow the user interacts with the `tweet` table to get totals on the stage of processing using a 'Group By' SQL command." ] }, { "cell_type": "code", "execution_count": null, "id": "114179ba-8212-4d75-89a5-05f5c18dc49f", "metadata": {}, "outputs": [], "source": [ "# Connect to the database\n", "classifier.connect()\n", "\n", "# Extract database object and create a cursor\n", "database = classifier.db\n", "cursor = database.cursor()\n", "\n", "# Execute an sql command\n", "cursor.execute(\"\"\"\n", "SELECT state, count(*) FROM tweet GROUP BY state;\n", "\"\"\")\n", "rows = cursor.fetchall()\n", "cursor.close()\n", "print(\"{:>20} : {:<12}\".format(\"STATE\", \"TOTAL\"))\n", "for state, count in rows:\n", " state_name = PROCESSING_STAGES(state).name\n", " print(\"{:>20} : {:<12}\".format(state_name, count))\n", " " ] }, { "cell_type": "markdown", "id": "e2e5c55e-9f34-4927-8972-436c98e02f34", "metadata": {}, "source": [ "Although it would be posible to store the state of the tweets as text fields in the database, that would increment storage and reduce efficiency. For that reason our team used an Enumerator Class called `PROCESSING_STAGES` to identify each stage with an integer value and a name.\n", "\n", "The code bellow prints a table of the values and names used to represent each stage. In the database they are stored as numeric values." ] }, { "cell_type": "code", "execution_count": null, "id": "7cf4d2cc-2531-4886-af52-ba2191a93400", "metadata": {}, "outputs": [], "source": [ "print(\"{:>5} | {:20} | {:}\".format(\"VALUE\", \"NAME\", \"OBJECT\"))\n", "for stage in PROCESSING_STAGES:\n", " print(\"{:5} | {:20} | {:}\".format(stage.value, stage.name, stage))" ] }, { "cell_type": "markdown", "id": "27aefda0-7ce0-43d9-abac-790fc4b2fa6b", "metadata": {}, "source": [ "## **View some of the accepted tweets**\n", "\n", "The `display_accepted` method will display accepted tweets in pages. Pages can have arbitrary amounts of tweets and pages can be selected by changing the `page` parameter.\n", "\n", "Try it! Change the page number. We recommend (5, 12 and 22)" ] }, { "cell_type": "code", "execution_count": null, "id": "a187e99f-c1cf-4677-9505-df3ebe5d0c5d", "metadata": { "tags": [] }, "outputs": [], "source": [ "# Change the page number to see different tweets from the database\n", "PAGE=3\n", "classifier.display_accepted(page=PAGE, per_page=2)" ] }, { "cell_type": "markdown", "id": "928db40e-6426-4299-be70-9462245fb413", "metadata": {}, "source": [ "## **Evaluate Some Tweets**\n", "\n", "Now is your turn!\n", "\n", "Using the method `StartEvaluation` you can process some of the unprocessed tweets. This should display a tweet embeding into screeen and ask some questions that will store metadata on the tweets.\n", "\n", "By default the StartEvaluations only processess tweets from the `PREPROCESSED` stage. using the `preprocess_batch` some tweets can be preprocessed before evaluation begins." ] }, { "cell_type": "code", "execution_count": 7, "id": "29de6489-059f-483a-b6e4-dcf35c051ed9", "metadata": {}, "outputs": [ { "data": { "text/html": [ "

Preprocessed 2006

" ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "classifier.preprocess_batch(n=20)" ] }, { "cell_type": "code", "execution_count": null, "id": "aaeb8c63-3a30-4005-a359-fd71ab793e82", "metadata": { "tags": [] }, "outputs": [], "source": [ "classifier.StartEvaluations()" ] }, { "cell_type": "markdown", "id": "52662e7c-07ed-4f68-b0a7-bcaaccd91e1a", "metadata": {}, "source": [ "## **Verify if the totals have changed.**" ] }, { "cell_type": "code", "execution_count": null, "id": "874370eb-1bb4-4fd3-9126-ab7f58836b83", "metadata": {}, "outputs": [], "source": [ "# Connect to the database\n", "classifier.connect()\n", "\n", "# Extract database object and create a cursor\n", "database = classifier.db\n", "cursor = database.cursor()\n", "\n", "# Execute an sql command\n", "cursor.execute(\"\"\"\n", "SELECT state, count(*) FROM tweet GROUP BY state;\n", "\"\"\")\n", "rows = cursor.fetchall()\n", "cursor.close()\n", "print(\"{:>20} : {:<12}\".format(\"STATE\", \"TOTAL\"))\n", "for state, count in rows:\n", " state_name = PROCESSING_STAGES(state).name\n", " print(\"{:>20} : {:<12}\".format(state_name, count))\n", "\n", "# Close the cursor\n", "cursor.close()" ] }, { "cell_type": "markdown", "id": "8e93100b-0da4-4fac-ad04-f8c4a66fea4c", "metadata": {}, "source": [ "How many tweets are missing?... **TODO**" ] }, { "cell_type": "markdown", "id": "fdc37169-479c-41f1-be7b-5725d9c8d253", "metadata": {}, "source": [ "## **Exercise 1**\n", "\n", "Using the example above. Run 2 different queries to visualize:\n", "1. the amount of tweets that have slang among the processed tweets.\n", "2. the amount of tweets that have multimedia among the processed tweets.\n", "\n", "Observe the column names of the tables `tweet` and `tweet_details`. The code bellow iterates through all tables and shows the column names." ] }, { "cell_type": "markdown", "id": "60252c24-484d-472f-a556-25c579d53797", "metadata": {}, "source": [ "**Consider**\n", "\n", "- What two tables will you need to join together to get both the tweet state and the details about slang and media?\n", "- What table has the tweet `state`? \n", "- What table has the media and slang information?\n", "- Visit [SQL Inner Join Tutorial](https://www.sqlitetutorial.net/sqlite-inner-join/) to get a better notion about what an \"Inner Join\" is.\n", "\n", "The column `state` is integer and matches the PROCESSING_STAGE values seen before.\n", "The columns `has_media` and `has_slang` are integers but can be treated as boolean.\n", "\n", "Replace any values between `` to make the commands work:\n", "- \\\n", "- sdf\n" ] }, { "cell_type": "code", "execution_count": null, "id": "1d7d16b8-c924-4485-8a85-9e92434d7efe", "metadata": {}, "outputs": [], "source": [ "# Connect to the database\n", "classifier.connect()\n", "\n", "# Extract database object and create a cursor\n", "database = classifier.db\n", "cursor = database.cursor()" ] }, { "cell_type": "code", "execution_count": null, "id": "22fd32a2-ad69-444f-9326-3e3c2e4fdec7", "metadata": {}, "outputs": [], "source": [ "###### SQL Command 1\n", "# Total of tweets with slang among the processed.\n", "# cursor.execute(\"\"\"\n", "# SELECT , count(*) \n", "# FROM a\n", "# INNER JOIN b\n", "# ON a.tweet_id=b.tweet_id\n", "# WHERE a.state= \n", "# GROUP BY ;\n", "# \"\"\")\n", "cursor.execute(\"\"\"\n", "SELECT has_slang, count(*) \n", "FROM tweet a\n", "INNER JOIN tweet_user_detail b\n", "ON a.tweet_id=b.tweet_id\n", "WHERE a.state in (2,6)\n", "GROUP BY has_slang;\n", "\"\"\")\n", "\n", "slang_rows = cursor.fetchall()\n", "\n", "# Print Results to screen\n", "print(\"{:>15} : {:<12}\".format(\"Slang?\", \"TOTAL\"))\n", "for slang, count in slang_rows:\n", " if slang:\n", " state=\"Has Slang\"\n", " else:\n", " state=\"No Slang\"\n", " print(\"{:>15} : {:<12}\".format(state, count))\n", " " ] }, { "cell_type": "code", "execution_count": null, "id": "ca655ea0-ec37-4ac6-a465-147b1c719b73", "metadata": {}, "outputs": [], "source": [ "# SQL Command 2\n", "# Total of tweets with multimedia among the processed and preprocessed.\n", "# cursor.execute(\"\"\"\n", "# SELECT\n", "# ,\n", "# count(*) \n", "# FROM\n", "# a\n", "# INNER JOIN\n", "# b\n", "# ON\n", "# a.tweet_id=b.tweet_id\n", "# WHERE\n", "# a.state in (, )\n", "# GROUP BY\n", "# \n", "# ORDER BY\n", "# DESC;\n", "# \"\"\")\n", "cursor.execute(\"\"\"\n", "SELECT\n", " has_media,\n", " count(*) \n", "FROM\n", " tweet a\n", "INNER JOIN\n", " tweet_auto_detail b\n", " ON\n", " a.tweet_id=b.tweet_id\n", "WHERE\n", " a.state in (2,6) \n", "GROUP BY\n", " has_media\n", "ORDER BY\n", " has_media DESC;\n", "\"\"\")\n", "\n", "multimedia_rows = cursor.fetchall()\n", "\n", "# Print Results to screen\n", "print(\"{:>12} : {:<12}\".format(\"HAS MEDIA?\", \"TOTAL\"))\n", "for has_media, count in multimedia_rows:\n", " if has_media:\n", " state=\"Yes\"\n", " else:\n", " state=\"No\"\n", " print(\"{:>12} : {:<12}\".format(state, count))\n", " \n" ] }, { "cell_type": "markdown", "id": "506e0fa9-bdc7-47ca-9a0c-efbf72d1277f", "metadata": {}, "source": [ "## **Exercise 2**\n", "\n", "Inspired by the commands above, write a command that lets you visualize the languages of the tweets. \n", "\n", "To complete this exercise you will need to answer the next questions:\n", "- What **table** has language information?\n", "- What **column** holds the language information?\n", "\n", "Remember to output the results." ] }, { "cell_type": "code", "execution_count": null, "id": "1136597c-a611-4ed3-ab93-4e21f4928303", "metadata": {}, "outputs": [], "source": [ "# SQL Command 3\n", "# Total of tweets by language\n", "cursor.execute(\"\"\"\n", "SELECT , count(*) \n", "FROM \n", "GROUP BY ;\n", "\"\"\")\n", "\n", "language_rows = cursor.fetchall()\n", "\n", "# Print Results to screen\n" ] }, { "cell_type": "markdown", "id": "9c1b462b-08a9-48ee-b5f5-c42bc30aee36", "metadata": {}, "source": [ "## Where is information stored?" ] }, { "cell_type": "code", "execution_count": null, "id": "50e8a64e-bfed-4beb-a51a-b6de6d0c18a7", "metadata": {}, "outputs": [], "source": [ "print(type(classifier))" ] }, { "cell_type": "markdown", "id": "809ba412-18c6-4d4e-8187-d3ed790890b7", "metadata": {}, "source": [ "The `classifier` is of type `JsonLInteractiveClassifier`. This class has two attributes, `original_filename`, a list of Tweet IDs, and `sqlite_filename`, the file holding the SQLite database, " ] }, { "cell_type": "code", "execution_count": null, "id": "adda27f7-64f1-4a40-9b6a-63ae6e0c2883", "metadata": {}, "outputs": [], "source": [ "print(classifier.original_filename)\n", "print(classifier.sqlite_filename)" ] }, { "cell_type": "code", "execution_count": null, "id": "96347af7-df87-4608-b538-b01b5718e688", "metadata": {}, "outputs": [], "source": [ "# Close the cursor\n", "cursor.close()" ] }, { "cell_type": "markdown", "id": "91a98ea4-bd9f-4c89-9d3e-9489c962dddc", "metadata": {}, "source": [ "## What's next?\n", "\n", "Continue to [Media Rating](./4-Media_Rating.ipynb) to continue with the experience and learn how to use `pandas` to interact with an SQLite database." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.8.10" } }, "nbformat": 4, "nbformat_minor": 5 }