![](\"http://scikit-learn.org/stable/_static/ml_map.png\")
\n",
"\n",
"According to this, we should be starting out with Linear SVC."
]
},
{
"cell_type": "code",
"execution_count": 41,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.svm import LinearSVC\n",
"clf = LinearSVC()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 4: Train the model\n",
"\n",
"Use the classifier to fit a model based on the feature matrix of `X_train` and the label vector of `y_train`."
]
},
{
"cell_type": "code",
"execution_count": 42,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"model = clf.fit(X_train_feats, y_train)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 5: Predict the labels of the test set\n",
"\n",
"Now that we have our model, we can use it to predict labels on a fresh test set."
]
},
{
"cell_type": "code",
"execution_count": 43,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"y_predicted = model.predict(X_test_feats)"
]
},
{
"cell_type": "code",
"execution_count": 44,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"array(['Malay', 'Malay', 'British', 'Malay', 'British', 'British',\n",
" 'British', 'British', 'British', 'British', 'Malay', 'Chinese',\n",
" 'British', 'Chinese', 'British', 'Other', 'Generic', 'Malay',\n",
" 'Malay', 'Chinese', 'British', 'British', 'Malay', 'British',\n",
" 'British', 'Generic', 'Other', 'British', 'British', 'British',\n",
" 'British', 'British', 'Malay', 'Generic', 'Malay', 'Generic',\n",
" 'Malay', 'British', 'Malay', 'British', 'British', 'Malay', 'Malay',\n",
" 'Generic'], dtype=object)"
]
},
"execution_count": 44,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"y_predicted"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Step 6: select an evaluation metric\n",
"\n",
"scikit-learn comes with a bunch of evaluation metrics. Which one should be chosen depends on what we're trying to minimise/maximise. In this case, we want to make as few errors as possible, so it makes sense to use accuracy as our metric.\n",
"\n",
"$$ accuracy = \\frac{\\# correct}{\\# classified} $$"
]
},
{
"cell_type": "code",
"execution_count": 45,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"from sklearn.metrics import accuracy_score"
]
},
{
"cell_type": "code",
"execution_count": 46,
"metadata": {
"collapsed": false
},
"outputs": [
{
"data": {
"text/plain": [
"0.59090909090909094"
]
},
"execution_count": 46,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"accuracy_score(y_true, y_predicted)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"So we got 60% accuracy. Let's try it with a few more train/test splits to see whether this is typical."
]
},
{
"cell_type": "code",
"execution_count": 47,
"metadata": {
"collapsed": true
},
"outputs": [],
"source": [
"def classify(X, y):\n",
" # do the train-test split\n",
" X_train, X_test, y_train, y_true = train_test_split(X, y)\n",
"\n",
" # get our features\n",
" X_train_feats = vect.fit_transform(X_train)\n",
" X_test_feats = vect.transform(X_test) \n",
"\n",
" # train our model\n",
" model = clf.fit(X_train_feats, y_train)\n",
" \n",
" # predict labels on the test set\n",
" y_predicted = model.predict(X_test_feats)\n",
" \n",
" # return the accuracy score obtained\n",
" return accuracy_score(y_true, y_predicted)"
]
},
{
"cell_type": "code",
"execution_count": 50,
"metadata": {
"collapsed": false
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"0.551818181818\n"
]
}
],
"source": [
"scores = list()\n",
"num_expts = 100\n",
"for i in range(num_expts):\n",
" score = classify(X,y)\n",
" scores.append(score)\n",
" \n",
"print sum(scores) / num_expts"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Conclusion\n",
"\n",
"The accuracy we obtain with this set of features and this classifier is about 55%. This isn't completely terrible. With 6 categories, a completely random classifier should expect to get only 16.6% of them right. But 55% accuracy also means that I'd have to go through and correct every other label. How can we improve this?\n",
"\n",
"There are a few ways that spring to mind:\n",
"\n",
"* Increase the amount of data - easier said than done\n",
"* Try different classifiers - scikit-learn makes this dead easy\n",
"* Use more features - worth a try (and we will)\n",
"* Adjust the hyperparameters of the classifiers - more on this later"
]
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 2",
"language": "python",
"name": "python2"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
"version": 2
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython2",
"version": "2.7.6"
}
},
"nbformat": 4,
"nbformat_minor": 0
}