{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"+ title: Natural Language Processing of German texts - Part 1: Using machine-learning to predict ratings\n",
"+ date: 2020-04-10\n",
"+ tags: python, NLP, sklearn, classification, machine-learning, tf-idf\n",
"+ Slug: german-nlp-binary-text-classification-of-reviews-part1\n",
"+ Category: Python\n",
"+ Authors: MC\n",
"+ Summary: Using a unique German data set containing ratings and comments on doctors, we build a Binary Text Classifier. To do so, we implement a complete machine learning work flow that predicts ratings from comments. In this first part, we start with basic methods. We go through text pre processing, feature creation (TF-IDF), classification and model optimization. Finally, we evaluate our model's ability to predict the sentiment of comments."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Motivation\n",
"\n",
"The domain of Natural Language Processing (NLP) has seen a rapid advancement during the last few years. In particular, 2019 has been a remarkable year. New state of the art results on all relevant benchmarks have been established on a regular basis. In this multi part post we apply and compare several methods of Binary Text Classification, starting with traditional, basic methods and leading to state of the art models. For this, we use a novel German data set that contains hundreds of thousands of comments. They were submitted by patients on an online platform for reviewing doctors. The data also contains ratings so that it can be used to train supervised models in order to predict the rating / sentiment of the comment. \n",
"Most data sets for NLP are in English and the majority of learning resources also focus on English. Hence, the unique data at our hands is valuable for applying NLP methods to German text. Because of its many observations and the fact that it contains labels, it is ideal for applying machine-learning methods. \n",
"Let's take a look at a basic NLP workflow:\n",
"\n",
"
\n",
"\n",
"![\"Chart](\"./img/nlp/nlp_workflow.png\")
\n",
"\n",
"A typical NLP machine-learning workflow (own illustration)\n",
"
\n",
"\n",
"In the following notebook, we will go through this process step by step. In this first part, we apply frequency based methods for feature creation and compare several classification models. Moreover, we optimize the parameters of our process using a grid search. We'll see that these \"traditional\" methods already achieve adequate results on our data, that we'll use as a baseline. In follow up posts, we will apply more advanced methods and try to improve on our results in the search for the best performing model. During this journey, we will compare the different methods and discuss their pros and cons. \n",
"\n",
"You can download this notebook or follow along in an interactive version of it on ![\"Open](\"https://mybinder.org/badge_logo.svg\")
and \n",
" ![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\")
."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### The data set\n",
"\n",
"You can take a look at the data on [data.world](https://data.world/mc51/german-language-reviews-of-doctors-by-patients) or directly download it from [here](https://query.data.world/s/v5xl53bs2rgq476vqy7cg7xx2db55y):"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"scrolled": true
},
"outputs": [],
"source": [
"# store current path and download data there\n",
"CURR_PATH = !pwd\n",
"!wget -O reviews.zip https://query.data.world/s/v5xl53bs2rgq476vqy7cg7xx2db55y\n",
"!unzip reviews.zip"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" After unzipping you'll get a csv file. \n",
"Before we open it, let's setup our notebook by loading all relevant modules and setting some options:"
]
},
{
"cell_type": "code",
"execution_count": 91,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
" \n",
"
\n",
"
Loading BokehJS ...\n",
"
\\n\"+\n",
" \"
\\n\"+\n",
" \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n",
" \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n",
" \"
\\n\"+\n",
" \"
\\n\"+\n",
" \"- re-rerun `output_notebook()` to attempt to load from CDN again, or
\\n\"+\n",
" \"- use INLINE resources instead, as so:
\\n\"+\n",
" \"
\\n\"+\n",
" \"
\\n\"+\n",
" \"from bokeh.resources import INLINE\\n\"+\n",
" \"output_notebook(resources=INLINE)\\n\"+\n",
" \"\\n\"+\n",
" \"
\\n\"+\n \"
\\n\"+\n \"BokehJS does not appear to have successfully loaded. If loading BokehJS from CDN, this \\n\"+\n \"may be due to a slow or bad network connection. Possible fixes:\\n\"+\n \"
\\n\"+\n \"
\\n\"+\n \"- re-rerun `output_notebook()` to attempt to load from CDN again, or
\\n\"+\n \"- use INLINE resources instead, as so:
\\n\"+\n \"
\\n\"+\n \"
\\n\"+\n \"from bokeh.resources import INLINE\\n\"+\n \"output_notebook(resources=INLINE)\\n\"+\n \"\\n\"+\n \"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" comment | \n",
" rating | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Ich bin franzose und bin seit ein paar Wochen in muenchen. Ich hatte Zahn Schmerzen und mein Kol... | \n",
" 2.0 | \n",
"
\n",
" \n",
" | 1 | \n",
" Dieser Arzt ist das unmöglichste was mir in meinem Leben je begegnet ist,er ist unfreundlich ,se... | \n",
" 6.0 | \n",
"
\n",
" \n",
" | 2 | \n",
" Hatte akute Beschwerden am Rücken. Herr Magura war der erste Arzt der sich wirklich Zeit für ein... | \n",
" 1.0 | \n",
"
\n",
" \n",
"
\n",
"
`. Nonetheless, compared to data from Facebook or Twitter this is mostly harmless and probably won't pose too many issues for us. We will deal with all of that in the next step."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Cleaning and pre processing\n",
"\n",
"Having consistent and clean data is fundamental for good modeling results. No matter how sophisticated your model the basic principle is: trash in trash out. When dealing with NLP the cleaning and pre processing can differ depending on which model you intend to use. We will use frequency based representation methods for our text. Thus, we usually want to have a pretty thorough manipulation of the input data:"
]
},
{
"cell_type": "code",
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"[nltk_data] Downloading package punkt to /home/mc/nltk_data...\n",
"[nltk_data] Package punkt is already up-to-date!\n",
"[nltk_data] Downloading package stopwords to /home/mc/nltk_data...\n",
"[nltk_data] Unzipping corpora/stopwords.zip.\n"
]
}
],
"source": [
"nltk.download('punkt')\n",
"nltk.download('stopwords')\n",
"\n",
"stemmer = SnowballStemmer(\"german\")\n",
"stop_words = set(stopwords.words(\"german\"))\n",
"\n",
"\n",
"def clean_text(text, for_embedding=False):\n",
" \"\"\"\n",
" - remove any html tags (< /br> often found)\n",
" - Keep only ASCII + European Chars and whitespace, no digits\n",
" - remove single letter chars\n",
" - convert all whitespaces (tabs etc.) to single wspace\n",
" if not for embedding (but e.g. tdf-idf):\n",
" - all lowercase\n",
" - remove stopwords, punctuation and stemm\n",
" \"\"\"\n",
" RE_WSPACE = re.compile(r\"\\s+\", re.IGNORECASE)\n",
" RE_TAGS = re.compile(r\"<[^>]+>\")\n",
" RE_ASCII = re.compile(r\"[^A-Za-zÀ-ž ]\", re.IGNORECASE)\n",
" RE_SINGLECHAR = re.compile(r\"\\b[A-Za-zÀ-ž]\\b\", re.IGNORECASE)\n",
" if for_embedding:\n",
" # Keep punctuation\n",
" RE_ASCII = re.compile(r\"[^A-Za-zÀ-ž,.!? ]\", re.IGNORECASE)\n",
" RE_SINGLECHAR = re.compile(r\"\\b[A-Za-zÀ-ž,.!?]\\b\", re.IGNORECASE)\n",
"\n",
" text = re.sub(RE_TAGS, \" \", text)\n",
" text = re.sub(RE_ASCII, \" \", text)\n",
" text = re.sub(RE_SINGLECHAR, \" \", text)\n",
" text = re.sub(RE_WSPACE, \" \", text)\n",
"\n",
" word_tokens = word_tokenize(text)\n",
" words_tokens_lower = [word.lower() for word in word_tokens]\n",
"\n",
" if for_embedding:\n",
" # no stemming, lowering and punctuation / stop words removal\n",
" words_filtered = word_tokens\n",
" else:\n",
" words_filtered = [\n",
" stemmer.stem(word) for word in words_tokens_lower if word not in stop_words\n",
" ]\n",
"\n",
" text_clean = \" \".join(words_filtered)\n",
" return text_clean"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The `clean_text` function takes a string input and applies a bunch of manipulations to it (described in the code). Check out this example:"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'python best programmiersprach welt'"
]
},
"execution_count": 17,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"clean_text(\"Python ist die beste Programmiersprache der Welt.\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This transformation has a few benefits. Removing characters and words that don't hold much meaning reduces the size of our data. Moreover, it can improve prediction performance when modeling by lowering the noise in the data. This is because e.g. stop words like prepositions or punctuation won't allow our model to extract additional information / meaning (at least when using simple models). By stemming and lower casing words we make sure that similar words are treated identically. Thus, we can improve model performance again by increasing the number of relevant data points. \n",
"Let's apply this to our data:"
]
},
{
"cell_type": "code",
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"CPU times: user 3min 18s, sys: 148 ms, total: 3min 18s\n",
"Wall time: 3min 18s\n"
]
}
],
"source": [
"%%time\n",
"# Clean Comments\n",
"data[\"comment_clean\"] = data.loc[data[\"comment\"].str.len() > 20, \"comment\"]\n",
"data[\"comment_clean\"] = data[\"comment_clean\"].map(\n",
" lambda x: clean_text(x, for_embedding=False) if isinstance(x, str) else x\n",
")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"For our classification task, we want to be able to recognize whether a comment has a positive or negative sentiment. We make use of the ratings that come alongside with all comments. Naturally, we assume that good ratings (1-2) convey a positive message while low ratings (5-6) convey a negative one. We exclude neutral ratings, so that our task becomes a binary classification. Keeping them would turn the task into a multi label classification problem, requiring a slightly different modeling approach."
]
},
{
"cell_type": "code",
"execution_count": 19,
"metadata": {},
"outputs": [],
"source": [
"# Create binary grade, class 1-2 or 5-6 = good or bad\n",
"data[\"grade_bad\"] = 0\n",
"data.loc[data[\"rating\"] >= 3, \"grade_bad\"] = np.NaN\n",
"data.loc[data[\"rating\"] >= 5, \"grade_bad\"] = 1\n",
"\n",
"# Drop when any of x missing\n",
"data = data[(data[\"comment_clean\"] != \"\") & (data[\"comment_clean\"] != \"null\")]\n",
"\n",
"data = data.dropna(\n",
" axis=\"index\", subset=[\"grade_bad\", \"comment\", \"comment_clean\"]\n",
").reset_index(drop=True)"
]
},
{
"cell_type": "code",
"execution_count": 20,
"metadata": {
"tags": [
"hide"
]
},
"outputs": [],
"source": [
"# data.to_csv(\"../../data/processed/comments_clean.csv\", index=False)\n",
"data_clean = data.copy()\n",
"# data_clean = pd.read_csv(\"../../data/processed/comments_clean.csv\")"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"These steps conclude the cleaning and pre processing. In result, we get this:"
]
},
{
"cell_type": "code",
"execution_count": 21,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" comment | \n",
" rating | \n",
" comment_clean | \n",
" grade_bad | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Ich bin franzose und bin seit ein paar Wochen in muenchen. Ich hatte Zahn Schmerzen und mein Kol... | \n",
" 2.0 | \n",
" franzos seit paar woch muench zahn schmerz kollegu dr mainka empfohl schnell termin bekomm team ... | \n",
" 0.0 | \n",
"
\n",
" \n",
" | 1 | \n",
" Dieser Arzt ist das unmöglichste was mir in meinem Leben je begegnet ist,er ist unfreundlich ,se... | \n",
" 6.0 | \n",
" arzt unmog leb je begegnet unfreund herablass medizin unkompetent diagnos hautarzt gegang ordent... | \n",
" 1.0 | \n",
"
\n",
" \n",
" | 2 | \n",
" Hatte akute Beschwerden am Rücken. Herr Magura war der erste Arzt der sich wirklich Zeit für ein... | \n",
" 1.0 | \n",
" akut beschwerd ruck herr magura erst arzt wirklich zeit therapieplan genomm nachhalt schmerz beseit | \n",
" 0.0 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"
\n",
"
\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"
\n",
"
\n",
"\n",
"\n",
"\n",
"\n",
"\n",
"
\n",
"
\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" grade_bad | \n",
" kommentar | \n",
"
\n",
" \n",
" \n",
" \n",
" | 270076 | \n",
" 1.0 | \n",
" ... nimmt sich Zeit, erklärt und kümmert sich. | \n",
"
\n",
" \n",
" | 184797 | \n",
" 1.0 | \n",
" Ich bin Erika , ich hatte am 15 .12.2017 zahnimplanta Bai voll Narkose alles ist gut gelaufen:) ich bin super zufrieden mit Dr Zeides er ist voll nett und locker drauf, ich kann für jede Empfehlen er ist die beste.ich muss sagen ich habe tierisch Angst von Zahnarzt aber egal wie oft ich Frage hatte und ich hatte viele frage:) er hätt e mir immer zeit genomen und erklärt wie es laufen wird. | \n",
"
\n",
" \n",
" | 208551 | \n",
" 1.0 | \n",
" Super zufriedener Zahnarzt <br />\\nTop Behandlung.... nur weiter zu empfehlen <br />\\nBin ein Angst Patient und er nimmt dir die Angst weg<br />\\nUnd redet auf dich ein . Bester Zahnarzt den ich vertraue. | \n",
"
\n",
" \n",
" | 103690 | \n",
" 1.0 | \n",
" Trotz akuten Schmerzen und Fremdkörper im Ohr, wird man weggeschickt mit einer Arroganz der Sprechstundenhilfe die wohl einmalig ist und ich noch nie so gesehen habe. Bin jetzt im Heegberg bei einem anderen Arzt super Betreuung und super freundlich. | \n",
"
\n",
" \n",
" | 213631 | \n",
" 1.0 | \n",
" Bie Dr. Hamdosch alls prima, sind zehr nett und perfekt | \n",
"
\n",
" \n",
" | 246987 | \n",
" 1.0 | \n",
" Ich wollte telefonisch ein Termin vereinbaren, mir wurde nur Montags ein Termin angeboten, als ich um den Dienstag gebeten habe, hat die Artthelferin wieder mich auch den Montag verwiesen. Ich habe ihr erklärt das ich nur Dienstags oder Freitag kann. Daraufhin hat diese unverschämte Person einfach aufgelegt. | \n",
"
\n",
" \n",
"
\n",
"
\\nTop Behandlung.... nur weiter zu empfehlen
\\nBin ein Angst Patient und er nimmt dir die Angst weg
\\nUnd redet auf dich ein . Bester Zahnarzt den ich vertraue. \n",
"103690 Trotz akuten Schmerzen und Fremdkörper im Ohr, wird man weggeschickt mit einer Arroganz der Sprechstundenhilfe die wohl einmalig ist und ich noch nie so gesehen habe. Bin jetzt im Heegberg bei einem anderen Arzt super Betreuung und super freundlich. \n",
"213631 Bie Dr. Hamdosch alls prima, sind zehr nett und perfekt \n",
"246987 Ich wollte telefonisch ein Termin vereinbaren, mir wurde nur Montags ein Termin angeboten, als ich um den Dienstag gebeten habe, hat die Artthelferin wieder mich auch den Montag verwiesen. Ich habe ihr erklärt das ich nur Dienstags oder Freitag kann. Daraufhin hat diese unverschämte Person einfach aufgelegt. "
]
},
"execution_count": 172,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"pd.options.display.max_colwidth = 800\n",
"# Predicted good but rated bad\n",
"test[[\"grade_bad\", \"comment\"]][(Y_test != pipe_pred) & (conf_score <= score_pos)]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"We've learned before that our model is really good at classifying the positive class. Thus, it is not surprising that it's able to correctly recognize that most comments above are positive even though they have a negative rating. However, the model makes two mistakes in the sentences `103690` and `246987` which really have a negative sentiment. We can hypothesize why this is the case. The first sentence first talks about a negative experience with the actual doctor. Then, the patient switched doctors and talks very positive about his new doctor. Since our model is rather simple it won't be able to identify that only the first part of the review is addressing the rated doctor. Following, the positive sentiment is attributed to the doctor as well and seems to weigh more than the negative one. The second review is probably too complex and contains a lot of neutral information while the negative sentiment is not very strong. \n",
"Let's check the comments that our model predicts as negative with high confidence:"
]
},
{
"cell_type": "code",
"execution_count": 173,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" grade_bad | \n",
" kommentar | \n",
"
\n",
" \n",
" \n",
" \n",
" | 67996 | \n",
" 0.0 | \n",
" - nicht ernsthaft mit den Beschwerden befaßt<br />\\n- oberflächliche und dazu falsche Diagnose <br />\\n- ....schnell schnell<br />\\n<br />\\n<br />\\ndas hat gereicht - nie wieder | \n",
"
\n",
" \n",
" | 146189 | \n",
" 0.0 | \n",
" Wie kann man so unfreundlich sein?! | \n",
"
\n",
" \n",
" | 331631 | \n",
" 0.0 | \n",
" Überhaupt nicht gut, weder Zahnärzlich noch Zwischenmenschlich. Sehr schlecht, ewig wenig Zeit, immer unkonzentriert und lässt sich nicht auf dich ein. Er will immer nur was verkaufen und redet bei jedem Besuch über Geld (Vorschuss usw.) Auch seine eigenen Pläne hält er nicht ein und verändert bei Stress und Unzufriedenheit die Rechnungen. Wir haben dort 910€ für Implantate angezahlt, dieses Geld hat er einfach abkassiert für 2Kronen, welche die DAK auch mit 2500€ honoriert hat. Also wenn euch euere Gesundheit, Zähne und Geld wichtig sind- sucht euch jemand anders, wir haben genug GELD und Zeit dort verloren- deshalb will ich dass es andere Leute auch erfahren. | \n",
"
\n",
" \n",
" | 260897 | \n",
" 0.0 | \n",
" Unfreundlich, unsensibel...unmöglich! Keinerlei Einfühlungsvermögen und Verständnis für Angstpatienten. Direkt nach dem Umbinden der Serviette, noch bevor ich den Arzt überhaupt zu sehen bekam, wollte man mir mit Kältespray an meine temperaturempfindlichen Zähne (angeblich sei dies als Eingangsuntersuchung vorgeschrieben) Ich lehnte dies ab, sah für eine solche Untersuchung überhaupt keinen Anlass, zumal ich davor überhaupt nicht nach meinen Beschwerden gefragt wurde. Verständnis und Aufklärung? Fehlanzeige! Zu einem solchen Arzt kann man kein Vertrauen aufbauen. Ich habe mit einem sehr schlechten Gefühl und ohne Behandlung die Praxis verlassen. | \n",
"
\n",
" \n",
" | 140500 | \n",
" 0.0 | \n",
" Sehr schlechte Beziehung, sehr unfreundlicher Arzt, jetzt es ist mir klar geworden warum das Praxis so leer ist. Furchtbar! Schräklich! Nie wieder! | \n",
"
\n",
" \n",
" | 335001 | \n",
" 0.0 | \n",
" Unhöflich, gemein, unfreundlich. Nie wieder. Die Anamnese hat 5 Sekunden gedauert. Ohne Anamnese wollte er was er will machen und die Patienten dürfen dazu kein Wort aussprechen, sonst bekommen sie Ärger. Geld Industrie, kein Mensch zu vertrauen als Arzt. Ich würde diesen Arzt niemand empfehlen. | \n",
"
\n",
" \n",
"
\n",
"
\\n- oberflächliche und dazu falsche Diagnose
\\n- ....schnell schnell
\\n
\\n
\\ndas hat gereicht - nie wieder \n",
"146189 Wie kann man so unfreundlich sein?! \n",
"331631 Überhaupt nicht gut, weder Zahnärzlich noch Zwischenmenschlich. Sehr schlecht, ewig wenig Zeit, immer unkonzentriert und lässt sich nicht auf dich ein. Er will immer nur was verkaufen und redet bei jedem Besuch über Geld (Vorschuss usw.) Auch seine eigenen Pläne hält er nicht ein und verändert bei Stress und Unzufriedenheit die Rechnungen. Wir haben dort 910€ für Implantate angezahlt, dieses Geld hat er einfach abkassiert für 2Kronen, welche die DAK auch mit 2500€ honoriert hat. Also wenn euch euere Gesundheit, Zähne und Geld wichtig sind- sucht euch jemand anders, wir haben genug GELD und Zeit dort verloren- deshalb will ich dass es andere Leute auch erfahren. \n",
"260897 Unfreundlich, unsensibel...unmöglich! Keinerlei Einfühlungsvermögen und Verständnis für Angstpatienten. Direkt nach dem Umbinden der Serviette, noch bevor ich den Arzt überhaupt zu sehen bekam, wollte man mir mit Kältespray an meine temperaturempfindlichen Zähne (angeblich sei dies als Eingangsuntersuchung vorgeschrieben) Ich lehnte dies ab, sah für eine solche Untersuchung überhaupt keinen Anlass, zumal ich davor überhaupt nicht nach meinen Beschwerden gefragt wurde. Verständnis und Aufklärung? Fehlanzeige! Zu einem solchen Arzt kann man kein Vertrauen aufbauen. Ich habe mit einem sehr schlechten Gefühl und ohne Behandlung die Praxis verlassen. \n",
"140500 Sehr schlechte Beziehung, sehr unfreundlicher Arzt, jetzt es ist mir klar geworden warum das Praxis so leer ist. Furchtbar! Schräklich! Nie wieder! \n",
"335001 Unhöflich, gemein, unfreundlich. Nie wieder. Die Anamnese hat 5 Sekunden gedauert. Ohne Anamnese wollte er was er will machen und die Patienten dürfen dazu kein Wort aussprechen, sonst bekommen sie Ärger. Geld Industrie, kein Mensch zu vertrauen als Arzt. Ich würde diesen Arzt niemand empfehlen. "
]
},
"execution_count": 173,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# Predicted bad but rated good\n",
"test[[\"grade_bad\", \"comment\"]][(Y_test != pipe_pred) & ((conf_score >= score_neg))]"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Here, the model's rating prediction fits the comment's sentiment better than the rating given by the patients in all cases. Unlike before, all comments have a clear, direct and strong sentiment. This is why the model performs error free. \n",
"We conclude that for cases where our model shows high confidence in the prediction we are able to disclose cases were original comment and rating are mismatched. However, we've also seen that the prediction accuracy has its limit. Particularly, when dealing with more complex sentence structures or more indirect sentiment expressions."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"As a last test of our model's performance, let's see how it copes with data that was never seen before. For that, we use some new comments as input. Then, we apply the same pre processing and cleaning as in our model preparation. Finally, we convert the text to its numeric representations and feed it to our model for the binary prediction:"
]
},
{
"cell_type": "code",
"execution_count": 87,
"metadata": {},
"outputs": [],
"source": [
"# Get new comments from website that were not included in original data\n",
"INPUT = [\n",
" \"Super sympathische Ärztin, fühle mich bei ihr bestens aufgehoben.\"\n",
" \"Sprechstundenhilfe war super nett man fühlt sich wohl.\",\n",
" \"Frau Doktor Merz nimmt sich richtig Zeit für mich. Hilft wo sie kann.\"\n",
" \"Hört wirklich einen zu. Sehr nett und freundlich. Sie ist sehr kompetent,\"\n",
" \"zuverlässig und vertrauenswürdig.\",\n",
" \"Nach meiner Beobachtung hat diese Praxis eine schlechte Hygiene. \",\n",
" \"Mangels akriebischer Behandlung musste mehrmals nachgebessert werden.\",\n",
"]"
]
},
{
"cell_type": "code",
"execution_count": 88,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Input after pre-processing / cleaning:\n",
"\n",
"sup sympath arztin fuhl best aufgehob sprechstundenhilf sup nett fuhlt wohl \n",
"frau doktor merz nimmt richtig zeit hilft hort wirklich nett freundlich kompetent zuverlass vertrauenswurd \n",
"beobacht praxis schlecht hygi \n",
"mangel akrieb behandl mehrmal nachgebessert\n"
]
}
],
"source": [
"# Pre-Process comments as we did with train data\n",
"text = [clean_text(comment) for comment in INPUT]\n",
"text_out = \" \\n\".join(text)\n",
"print(f\"Input after pre-processing / cleaning:\\n\\n{text_out}\")\n",
"# run comments through pipe: predict using our best model from above\n",
"predictions = pipe.predict(text)"
]
},
{
"cell_type": "code",
"execution_count": 89,
"metadata": {},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" comment | \n",
" prediction | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" Super sympathische Ärztin, fühle mich bei ihr bestens aufgehoben.Sprechstundenhilfe war super nett man fühlt sich wohl. | \n",
" good | \n",
"
\n",
" \n",
" | 1 | \n",
" Frau Doktor Merz nimmt sich richtig Zeit für mich. Hilft wo sie kann.Hört wirklich einen zu. Sehr nett und freundlich. Sie ist sehr kompetent,zuverlässig und vertrauenswürdig. | \n",
" good | \n",
"
\n",
" \n",
" | 2 | \n",
" Nach meiner Beobachtung hat diese Praxis eine schlechte Hygiene. | \n",
" bad | \n",
"
\n",
" \n",
" | 3 | \n",
" Mangels akriebischer Behandlung musste mehrmals nachgebessert werden. | \n",
" bad | \n",
"
\n",
" \n",
"
\n",
"