{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Find this project in [Github: UberHowley/MOOC-File-Processing](https://github.com/UberHowley/mooc-file-processing)\n",
"\n",
"---\n",
"\n",
"# Python Processing of the QuickHelper System Logfiles\n",
"In Massive Open Online Courses (MOOCs), it is common for students to use the discussion forums to seek help, but oftentimes their discussion thread gets buried under hundreds of other threads resulting in their question never being answered. Our QuickHelper system uses a social recommendation algorithm to recommend three qualified peers that the student can invite to their public discussion thread, but **how do we present the selected peers in such a way as to increase the number of helpers selected**? \n",
"\n",
"*I used Python to clean the logfile data, gensim to assign automated [LDA] topics to each message board post, and pandas to perform statistical analyses in order to answer this research question.*\n",
"\n",
"## Research Questions\n",
"Our QuickHelper system was designed to advise students on which peers they might want to help them, but also to answer theory-based questions about student motivation and decision-making in the face of common interactional archetypes employed in MOOCs. This yielded the following research questions:\n",
"1. Does the presence of Help Giver badges increase the number of helpers students invite to their question thread? (*isBadgeCondition --> numHelpersSelected*)\n",
" 1. Does the number of stars shown on the badge impact whether or not the peer is selected?\n",
"2. Does showing expertise information about potential helpers increase the number of helpers the student invites to her question thread? (*sentenceType --> numHelpersSelected*)\n",
" 1. Does the number of weeks joined or the topic knowledge match percentage increase the likelihood of a peer being selected?\n",
"3. Does knowing your selected helpers will be up or downvoting on the quality of the question negatively impact the number of helpers invited to the question thread? (*isVotingCondition --> numHelpersSelected*)\n",
"4. Do the badges or expertise sentences lessen the negative effects of the up and down voting?\n",
"I discuss these questions and the experimental set up more in-depth [here](http://www.irishowley.com/website/pMOOChelpers.html).\n",
"\n",
"## The QuickHelper System\n",
"The QuickHelper system process is shown in the figure below. (1) The student clicks the QuickHelper button and writes her question. (2) The question posts to the public discussion forum. (3) Simultaneously, the message board post is submitted to the QuickHelper system. (4) The QuickHelper system selects 3 qualified potential helpers based on information it knows about the students and the provided question. (5) The QuickHelper system shows the student the 3 selected potential helpers. (6) The student picks 0-3 helpers. (7) The system emails the selected helpers with a link to the discussion forum post. (8) [not shown] The helpers may or may not click the link from the email.\n",
"\n",
"When designing the QuickHelper system, we had to take into consideration our logging needs with respect to our research questions. There are three separate moments of timing in our system, yielding at least four logs: (1) At step 4 where the system selects helpers, (2) step 5 where the system presents the helpers to the student. (3) At step 6 where the students picks 0-3 helpers. (4) Step 8 where the helpers click the link they receive via email and potentially up/downvote the question. In this analysis, we're mostly concerned with the first three logs: `user.log, helper.log, and selection.log`. Each line in `user.log` represents one question/forum message (i.e., one usage of the QuickHelper system). One line in `user.log` has three corresponding lines in `helper.log`: one for each potential helper shown to the student. One line in `selection.log` represents a single helper that was selected.\n",
"\n",
"Somewhere between steps 4 and 5, before the QuickHelper system shows the student her potential helpers, the system assigns this particular QuickHelper instance to a series of conditions relating to ways of presenting the potential helpers. This include the badge/expectancy emphasis condition in which Help Giver badges with 0-3 stars are shown. There is also an expertise/value emphasis condition in which either an irrelevant sentence or a relevant sentence including the helpers' number of weeks active and topic match percentage is shown. The third condition is a up-downvoting/cost emphasis condition, in which the student is shown a preview of the message that the system will send to the selected helpers. The message either shows that the helper will vote on the quality of the question, or not.\n",
"\n",
"## Processing Logfiles\n",
"Processing the logfiles removes the metadata and major inconsistencies in the raw data in order to prepare it for analysis by external programs. There were (1) various errors in the logging software (i.e., mysterious duplicates whose instance IDs were different, extra columns on some lines with a URL), (2) general cleaning that was necessary (i.e., removing test entries, removing entries from our researchers, removing entries outside the course date range), and (3) more complicated post-processing that was necessary (i.e., cross-referencing and compiling values from one logfile into another). My `logfileMOOC.py` script compiles and resolves all the log inter-dependencies, as shown below.:\n",
"\n",
"\n",
"### QuickHelper Version\n",
"In the initial two weeks of the course, the QuickHelper system did not have enough information to be able to recommend helpers. This 'cold start' required the implementation of two versions of the system: the `TA` version, and the `student` version. In the TA version, all selected helpers were actually volunteer teaching assistants for the course. Assignment to the badge and voting conditions remained the same, but all helpers would be shown with 4 stars on their badge, if in the `isBadgeCondition`. The TA version did not have anyone in the anonymized `isUsernameCondition` nor did the `isIrrelevantSentence` condition exist: all TAs were shown with the same sentence.\n",
"\n",
"### Removing Duplicates ###\n",
"The 'mysterious duplicates' were an interesting side issue. Previously, we had been removing what appeared to be duplicate instances from the log manually. The issue was that these duplicates would have the same author, question title/body, and date, but everything else would be different. I created a duplicate key for each QuickHelper instance using the author, question title, and date. The duplicate key was used in a dictionary to map to a list of all the duplicates that shared that key. To determine which duplicate was the instance the user actually saw, I iterated through each list of duplicates and found the one with entries in the selection.log (i.e., the user only selected helpers from one instance, the real one). If there was no helper was selected, I removed all information that the duplicates did not share. By random chance, some of the duplicates would share conditions or helpers shown. This did result in some columns missing data for some entries in the user and helper logfiles, but there was no other way to determine which duplicate was the instance seen by the user."
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"# example code - do not execute\n",
"def remove_duplicates():\n",
" \"\"\"\n",
" Remove duplicates from our list of instances, based on whatever key was used in duplicate_instances\n",
" :return: A list of QHInstances with all duplicates removed\n",
" \"\"\"\n",
" for duplicate_key in instances_by_dupkey: # iterate through each key in our duplicate-arranged list\n",
" selected_dup = None # instance with a selection (the one shown)\n",
" for dup in instances_by_dupkey[duplicate_key]: # for each instance object in these duplicates\n",
" num_helpers = dict_num_helpers.get(getattr(dup, 'instance_id'), 0)\n",
" if num_helpers > 0: # If it has helpers selected, it's the one\n",
" selected_dup = dup\n",
" setattr(dup, 'num_helpers_selected', num_helpers)\n",
" if selected_dup is None:\n",
" selected_dup = create_new_duplicate(instances_by_dupkey[duplicate_key]) # Clear out non-matching condition variables\n",
" list_no_duplicates.append(selected_dup) # Record selected_dup as our correct one\n",
" # Store this sentence, too (for topic modeling)\n",
" list_sentences.append(ldat.clean_string(\" \".join([getattr(selected_dup, 'question_title', ''), ' '+getattr(selected_dup,'question_body','')])))\n",
"\n",
" if len(instances_by_dupkey[duplicate_key]) > 1: # counting our duplicates\n",
" global count_repeat\n",
" count_repeat += len(instances_by_dupkey[duplicate_key])-1\n",
" return list_no_duplicates"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"See a sample of two (simulated) lines from the outputted logfile below: the bottom line being produced from several duplicates that shared helper ID numbers, isVotingCondition, among other expected features (question title, question body, author, date, etc.). Condition assignments that did not match across all duplicates are removed.\n",
"\n",
"| instanceID | version | isBadgeCond | isIrrelevantSentence | isVotingCond | isNameCond | helper0 | helper1 | helper2 | numHelpersSelected |
\n",
"| 1234...7890 | student | n | n | y | y | 135790 | 246802 | 998877 | 0 |
\n",
"| 0987...4321 | student | | | y | | 235689 | 986532 | 555123 | 0 |
\n",
"
\n",
"\n",
"## Statistical Analysis\n",
"I used the pandas and statsmodels libraries to run descriptive statistics, generate plots to better understand the data, and answer our research questions (from above). The main effects of interest were the categorical condition variables (i.e., `isBadgeCondition, sentenceType, isVotingCondition`) and the scalar dependent variable, `number of helpers selected`.\n",
"\n",
"It's important to note that this analysis was done on an uncleaned version of the dataset (i.e., posts that are clearly not help requests have not been removed), and so the results discussed here vary slightly from those in the related publications."
]
},
{
"cell_type": "code",
"execution_count": 56,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"# necessary libraries/setup\n",
"%matplotlib inline\n",
"import utilsMOOC as utils # separate file that contains all the constants we need\n",
"import pandas as pd\n",
"import matplotlib.pyplot as plt\n",
"import seaborn as sns # importing seaborns for its factorplot\n",
"sns.set_style(\"darkgrid\")\n",
"\n",
"data = pd.io.parsers.read_csv(\"user.csv\", encoding=\"utf-8-sig\")\n",
"conditions = [utils.COL_VERSION, utils.COL_BADGE, utils.COL_SENTENCE_TYPE, utils.COL_VOTING] # all our categorical IVs of interest\n",
"outcome = utils.COL_NUMHELPERS"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Descriptive Statistics\n",
"Descriptive statistics showed that 75% of users selected zero helpers, with 3 and 1 helpers being seelcted the second and third most popular choice, respectively."
]
},
{
"cell_type": "code",
"execution_count": 13,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"
\n",
" \n",
" \n",
" | \n",
" numHelpersSelected | \n",
"
\n",
" \n",
" \n",
" \n",
" | count | \n",
" 159.000000 | \n",
"
\n",
" \n",
" | mean | \n",
" 0.779874 | \n",
"
\n",
" \n",
" | std | \n",
" 1.156423 | \n",
"
\n",
" \n",
" | min | \n",
" 0.000000 | \n",
"
\n",
" \n",
" | 25% | \n",
" 0.000000 | \n",
"
\n",
" \n",
" | 50% | \n",
" 0.000000 | \n",
"
\n",
" \n",
" | 75% | \n",
" 1.000000 | \n",
"
\n",
" \n",
" | max | \n",
" 3.000000 | \n",
"
\n",
" \n",
"
\n",
"