{ "cells": [ { "cell_type": "markdown", "metadata": { "id": "Ss6WwQxZcyB3" }, "source": [ "# 머신 러닝 교과서 3판" ] }, { "cell_type": "markdown", "metadata": { "id": "lTv_uMjxcyB4" }, "source": [ "# 8장 - 감성 분석에 머신 러닝 적용\n" ] }, { "cell_type": "markdown", "metadata": { "id": "qeuUzWpZcyB5" }, "source": [ "**아래 링크를 통해 이 노트북을 주피터 노트북 뷰어(nbviewer.jupyter.org)로 보거나 구글 코랩(colab.research.google.com)에서 실행할 수 있습니다.**\n", "\n", "\n", " \n", " \n", "
\n", " 주피터 노트북 뷰어로 보기\n", " \n", " 구글 코랩(Colab)에서 실행하기\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "6Ss0AF_gcyB5" }, "source": [ "### 목차" ] }, { "cell_type": "markdown", "metadata": { "id": "kkpNAjXtcyB5" }, "source": [ "- 텍스트 처리용 IMDb 영화 리뷰 데이터 준비\n", " - 영화 리뷰 데이터셋 구하기\n", " -영화 리뷰 데이터셋을 더 간편한 형태로 전처리\n", "- BoW 모델 소개\n", " - 단어를 특성 벡터로 변환\n", " - tf-idf를 사용하여 단어 적합성 평가\n", " - 텍스트 데이터 정제\n", " - 문서를 토큰으로 나누기\n", "- 문서 분류를 위한 로지스틱 회귀 모델 훈련\n", "- 대용량 데이터 처리: 온라인 알고리즘과 외부 메모리 학습\n", "- 잠재 디리클레 할당을 사용한 토픽 모델링\n", " - LDA를 사용한 텍스트 문서 분해\n", " - 사이킷런의 LDA\n", "- 요약" ] }, { "cell_type": "markdown", "metadata": { "id": "WPm8CyFWcyB5" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "OqHmARyScyB6" }, "source": [ "# 8.1 텍스트 처리용 IMDb 영화 리뷰 데이터 준비" ] }, { "cell_type": "markdown", "metadata": { "id": "agsmeXazcyB6" }, "source": [ "## 8.1.1 영화 리뷰 데이터셋 구하기" ] }, { "cell_type": "markdown", "metadata": { "id": "qMsd8qjbcyB6" }, "source": [ "IMDB 영화 리뷰 데이터셋은 [http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz](http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz)에서 내려받을 수 있습니다. 다운로드된 후 파일 압축을 해제합니다.\n", "\n", "A) 리눅스(Linux)나 macOS를 사용하면 새로운 터미널(Terminal) 윈도우를 열고 `cd` 명령으로 다운로드 디렉터리로 이동하여 다음 명령을 실행하세요.\n", "\n", "`tar -zxf aclImdb_v1.tar.gz`\n", "\n", "B) 윈도(Windows)를 사용하면 7-Zip(http://www.7-zip.org) 같은 무료 압축 유틸리티를 설치하여 다운로드한 파일의 압축을 풀 수 있습니다." ] }, { "cell_type": "markdown", "metadata": { "id": "30Y2_kwrcyB6" }, "source": [ "**다음처럼 파이썬에서 다운로드하고 압축을 풀 수도 있습니다:**" ] }, { "cell_type": "code", "execution_count": 1, "metadata": { "id": "eOs3FJPTcyB7", "outputId": "adee16cc-dd23-4d18-f829-f70977bffd47", "colab": { "base_uri": "https://localhost:8080/" } }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "100% | 80 MB | 4.97 MB/s | 16 sec elapsed" ] } ], "source": [ "import os\n", "import sys\n", "import tarfile\n", "import time\n", "import urllib.request\n", "\n", "\n", "source = 'http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz'\n", "target = 'aclImdb_v1.tar.gz'\n", "\n", "\n", "def reporthook(count, block_size, total_size):\n", " global start_time\n", " if count == 0:\n", " start_time = time.time()\n", " return\n", " duration = time.time() - start_time\n", " progress_size = int(count * block_size)\n", " speed = progress_size / (1024.**2 * duration)\n", " percent = count * block_size * 100. / total_size\n", "\n", " sys.stdout.write(\"\\r%d%% | %d MB | %.2f MB/s | %d sec elapsed\" %\n", " (percent, progress_size / (1024.**2), speed, duration))\n", " sys.stdout.flush()\n", "\n", "\n", "if not os.path.isdir('aclImdb') and not os.path.isfile('aclImdb_v1.tar.gz'):\n", " urllib.request.urlretrieve(source, target, reporthook)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": { "id": "WB9vcQJ9cyB7", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "e35aeb53-d7ec-4f3b-d6c4-18fa32433081" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/tmp/ipython-input-23546269.py:4: DeprecationWarning: Python 3.14 will, by default, filter extracted tar archives and reject files or modify their metadata. Use the filter argument to control this behavior.\n", " tar.extractall()\n" ] } ], "source": [ "if not os.path.isdir('aclImdb'):\n", "\n", " with tarfile.open(target, 'r:gz') as tar:\n", " tar.extractall()" ] }, { "cell_type": "markdown", "metadata": { "id": "mhTLvtAdcyB7" }, "source": [ "## 8.1.2 영화 리뷰 데이터셋을 더 간편한 형태로 전처리" ] }, { "cell_type": "markdown", "metadata": { "id": "729l0colcyB7" }, "source": [ "`pyprind`는 주피터 노트북에서 진행바를 출력하기 위한 유틸리티입니다. `pyprind` 패키지를 설치하려면 다음 셀을 실행하세요." ] }, { "cell_type": "code", "execution_count": 3, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "4NUkjwFLcyB7", "outputId": "dd6128f4-979a-4127-97d0-dfaa9cc2bc4c", "scrolled": true }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Collecting pyprind\n", " Downloading PyPrind-2.11.3-py2.py3-none-any.whl.metadata (1.1 kB)\n", "Downloading PyPrind-2.11.3-py2.py3-none-any.whl (8.4 kB)\n", "Installing collected packages: pyprind\n", "Successfully installed pyprind-2.11.3\n" ] } ], "source": [ "!pip install pyprind" ] }, { "cell_type": "code", "execution_count": 4, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "xcfgFFkDcyB7", "outputId": "e09481b5-fde3-4989-ee15-6b5176555198" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "0% [##############################] 100% | ETA: 00:00:00\n", "Total time elapsed: 00:00:02\n" ] } ], "source": [ "import pyprind\n", "import pandas as pd\n", "import os\n", "\n", "# `basepath`를 압축 해제된 영화 리뷰 데이터셋이 있는\n", "# 디렉토리로 바꾸세요\n", "\n", "basepath = 'aclImdb'\n", "\n", "labels = {'pos': 1, 'neg': 0}\n", "pbar = pyprind.ProgBar(50000)\n", "# df = pd.DataFrame()\n", "rows = []\n", "for s in ('test', 'train'):\n", " for l in ('pos', 'neg'):\n", " path = os.path.join(basepath, s, l)\n", " for file in sorted(os.listdir(path)):\n", " with open(os.path.join(path, file),\n", " 'r', encoding='utf-8') as infile:\n", " txt = infile.read()\n", " rows.append([txt, labels[l]])\n", " # df = df.append([[txt, labels[l]]],\n", " # ignore_index=True)\n", " pbar.update()\n", "df = pd.DataFrame(rows, columns=['review', 'sentiment'])\n", "# df.columns = ['review', 'sentiment']" ] }, { "cell_type": "markdown", "metadata": { "id": "of8QKAPocyB8" }, "source": [ "데이터프레임을 섞습니다:" ] }, { "cell_type": "code", "execution_count": 5, "metadata": { "id": "1b7BhbYxcyB8" }, "outputs": [], "source": [ "import numpy as np\n", "\n", "np.random.seed(0)\n", "df = df.reindex(np.random.permutation(df.index))" ] }, { "cell_type": "markdown", "metadata": { "id": "djnDKKULcyB8" }, "source": [ "선택사항: 만들어진 데이터를 CSV 파일로 저장합니다:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": { "id": "kyV0QwoXcyB9" }, "outputs": [], "source": [ "df.to_csv('movie_data.csv', index=False, encoding='utf-8')" ] }, { "cell_type": "code", "execution_count": 7, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 148 }, "id": "7LNPVvAKcyB9", "outputId": "43abdb6b-72c1-4fa7-8855-5628799ae552" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " review sentiment\n", "0 In 1974, the teenager Martha Moxley (Maggie Gr... 1\n", "1 OK... so... I really like Kris Kristofferson a... 0\n", "2 ***SPOILER*** Do not read this, if you think a... 0" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
reviewsentiment
0In 1974, the teenager Martha Moxley (Maggie Gr...1
1OK... so... I really like Kris Kristofferson a...0
2***SPOILER*** Do not read this, if you think a...0
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "df", "summary": "{\n \"name\": \"df\",\n \"rows\": 50000,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 49582,\n \"samples\": [\n \"Every time I think about this film I feel physically ill. To read such a great book and later discover there's a film of it was a great feeling. Years later and imagine my joy at switching on the sci-fi channel and finding it starts in just 5mins!!! Up go the titles and then uggg. If just a couple of things had changed OK. Everything is changed. Numerous characters are removed entirely new rubbish ones are added. The main hero is shrunk and de-aged by about 30 years, and hilariously his girlfriend/wife is now his mother! Even the dog is reduced to sub-lassie capabilities. This is truly appalling cinema at its absolute worst. I would quite happily remove my own toenails with pliers rather than sit through another horrific viewing, and I urge anyone thinking of watching this - please don't. If you own a copy burn it now, right now and think how much better your life would have been had this celluloid insult never occurred.\",\n \"As with all environmentally aware films from the 1970s SOYLENT GREEN has a rather cheesy view of what ecological meltdown is . Overpopulation means there`s too many people to feed ? I was under the impression that famines were caused by either war or failed economic policies . Stalin`s policy in the Soviet Union in the 1930s left millions dead because of famine and to this day the greatest man made tragedy was Mao`s rural policy in China which led over 30 million starvation deaths in the 1950s . And let`s not forget the great famines in the horn of Africa in the 1980s and 90s which were to do with conflicts not overpopulation . You might like to also consider that two of the most heavily populated areas on Earth , Hong Kong and Macau , have never suffered a famine in modern times . Likewise the expansion of shanty towns around cities as seen here isn`t strictly down to overpopulation - it`s down to economic factors where people flock to cities to find better paid work than in the countryside ( It`s a symptom of industrial progress - not of too many births ) so the image of the streets of New York city being too congested to walk through and of having people sleep in stairwells is somewhat laughable

But don`t be fooled into thinking SOYLENT GREEN is a pile of corny tree hugging crap because I consider this to be the best ecological film of the

70s . It plays on the contempary audience`s knowledge of the world where Sol and Thorn are beside themselves with joy at finding fruit , brandy and fresh meat . Thorn gasps in amazement at having ice in his whisky , puffs on a cigarette and delivers the classic line \\\" If I could afford it I`d smoke two , maybe three of these a day \\\" . But it`s the visage of the euthanasia chamber that`s memorable as Thorn gazes at the images of wild animals , flowers , running water and snow covered mountains , a world Thorn`s generation has never known . This is a very haunting scene which makes SOYLENT GREEN a very memorable film , combined with the fact it features the final screen appearance of Edward G Robinson as the wise old Jew Sol Roth\",\n \"Yah. I know. It has the name \\\"Sinatra\\\" in the title, so how bad can it be? Well, it's bad, trust me! I rented this thinking it was some movie I missed in the theaters. It's not. It's some garbage \\\"movie\\\" made by the folks at Showtime (cable station). Geez, these cable stations make a few bucks they think they can make whatever garbage movies they want! It's not good. I am as big a Sinatra fan as any sane man, but this movie was just dumb. Boring. Dull. Unfunny. Uninteresting. The only redeeming quality is that (assuming they did stick to the facts) you do learn about what happened to the captors of Frank Jr. Otherwise it's just a stupid film.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 0,\n \"max\": 1,\n \"num_unique_values\": 2,\n \"samples\": [\n 0,\n 1\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {}, "execution_count": 7 } ], "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n", "df.head(3)" ] }, { "cell_type": "code", "execution_count": 8, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "7cR49jwpcyB9", "outputId": "550725bd-c2a1-4197-aac1-4e41cec74bac" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(50000, 2)" ] }, "metadata": {}, "execution_count": 8 } ], "source": [ "df.shape" ] }, { "cell_type": "markdown", "metadata": { "id": "phyGy5GScyB9" }, "source": [ "---\n", "\n", "### 노트\n", "\n", "`movie_data.csv` 파일을 만드는데 문제가 있다면 https://github.com/rickiepark/python-machine-learning-book-3rd-edition/tree/master/ch08/ 에서 zip 압축된 버전을 다운로드할 수 있습니다.\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": { "id": "WbiutklPcyB9" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "7OISRfu-cyB9" }, "source": [ "# 8.2 BoW 모델 소개" ] }, { "cell_type": "markdown", "metadata": { "id": "qL4sPT17cyB-" }, "source": [ "## 8.2.1 단어를 특성 벡터로 변환" ] }, { "cell_type": "markdown", "metadata": { "id": "bmEJ2BPtcyB-" }, "source": [ "CountVectorizer의 fit_transform 메서드를 호출하여 BoW 모델의 어휘사전을 만들고 다음 세 문장을 희소한 특성 벡터로 변환합니다:\n", "1. The sun is shining\n", "2. The weather is sweet\n", "3. The sun is shining, the weather is sweet, and one and one is two" ] }, { "cell_type": "code", "execution_count": 9, "metadata": { "id": "Q5rK2FRicyB-" }, "outputs": [], "source": [ "import numpy as np\n", "from sklearn.feature_extraction.text import CountVectorizer\n", "\n", "count = CountVectorizer()\n", "docs = np.array([\n", " 'The sun is shining',\n", " 'The weather is sweet',\n", " 'The sun is shining, the weather is sweet, and one and one is two'])\n", "bag = count.fit_transform(docs)" ] }, { "cell_type": "markdown", "metadata": { "id": "pgjoT97McyB-" }, "source": [ "어휘 사전의 내용을 출력해 보면 BoW 모델의 개념을 이해하는 데 도움이 됩니다:" ] }, { "cell_type": "code", "execution_count": 10, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ZgYsyO_WcyB-", "outputId": "f073d3c2-13ef-43ec-b442-a607ac0b6589" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "{'the': 6, 'sun': 4, 'is': 1, 'shining': 3, 'weather': 8, 'sweet': 5, 'and': 0, 'one': 2, 'two': 7}\n" ] } ], "source": [ "print(count.vocabulary_)" ] }, { "cell_type": "markdown", "metadata": { "id": "s4MkGIeccyB-" }, "source": [ "이전 결과에서 볼 수 있듯이 어휘 사전은 고유 단어와 정수 인덱스가 매핑된 파이썬 딕셔너리에 저장되어 있습니다. 그다음 만들어진 특성 벡터를 출력해 봅시다:" ] }, { "cell_type": "markdown", "metadata": { "id": "LqefbeRYcyB_" }, "source": [ "특성 벡터의 각 인덱스는 CountVectorizer의 어휘 사전 딕셔너리에 저장된 정수 값에 해당됩니다. 예를 들어 인덱스 0에 있는 첫 번째 특성은 ‘and’ 단어의 카운트를 의미합니다. 이 단어는 마지막 문서에만 나타나네요. 인덱스 1에 있는 (특성 벡터의 두 번째 열) 단어 ‘is’는 세 문장에 모두 등장합니다. 특성 벡터의 이런 값들을 단어 빈도(term frequency) 라고도 부릅니다. 문서 d에 등장한 단어 t의 횟수를 *tf (t,d)*와 같이 씁니다." ] }, { "cell_type": "code", "execution_count": 11, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "mKbtk17XcyB_", "outputId": "04a3713d-163e-4f7a-a030-ee2cb9ad78ad" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[[0 1 0 1 1 0 1 0 0]\n", " [0 1 0 0 0 1 1 0 1]\n", " [2 3 2 1 1 1 2 1 1]]\n" ] } ], "source": [ "print(bag.toarray())" ] }, { "cell_type": "markdown", "metadata": { "id": "7Gm1_TVecyB_" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "FWOx1Z7_cyB_" }, "source": [ "## 8.2.2 tf-idf를 사용하여 단어 적합성 평가" ] }, { "cell_type": "code", "execution_count": 12, "metadata": { "id": "6heXG7odcyB_" }, "outputs": [], "source": [ "np.set_printoptions(precision=2)" ] }, { "cell_type": "markdown", "metadata": { "id": "tnWdHBptcyB_" }, "source": [ "텍스트 데이터를 분석할 때 클래스 레이블이 다른 문서에 같은 단어들이 나타나는 경우를 종종 보게 됩니다. 일반적으로 자주 등장하는 단어는 유용하거나 판별에 필요한 정보를 가지고 있지 않습니다. 이 절에서 특성 벡터에서 자주 등장하는 단어의 가중치를 낮추는 기법인 tf-idf(term frequency-inverse document frequency)에 대해 배우겠습니다. tf-idf는 단어 빈도와 역문서 빈도(inverse document frequency)의 곱으로 정의됩니다:\n", "\n", "$$\\text{tf-idf}(t,d)=\\text{tf (t,d)}\\times \\text{idf}(t,d)$$\n", "\n", "여기에서 tf(t, d)는 이전 절에서 보았던 단어 빈도입니다. *idf(t, d)*는 역문서 빈도로 다음과 같이 계산합니다:\n", "\n", "$$\\text{idf}(t,d) = \\text{log}\\frac{n_d}{1+\\text{df}(d, t)},$$\n", "\n", "여기에서 $n_d$는 전체 문서 개수이고 *df(d, t)*는 단어 t가 포함된 문서 d의 개수입니다. 분모에 상수 1을 추가하는 것은 선택 사항입니다. 훈련 샘플에 한 번도 등장하지 않는 단어가 있는 경우 분모가 0이 되지 않게 만듭니다. log는 문서 빈도 *df(d, t)*가 낮을 때 역문서 빈도 값이 너무 커지지 않도록 만듭니다.\n", "\n", "사이킷런 라이브러리에는 `CountVectorizer` 클래스에서 만든 단어 빈도를 입력받아 tf-idf로 변환하는 `TfidfTransformer` 클래스가 구현되어 있습니다:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "397IiqLvcyB_", "outputId": "c1d0c9cd-ce51-492c-88ef-b4b02c5f5d39" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "[[0. 0.43 0. 0.56 0.56 0. 0.43 0. 0. ]\n", " [0. 0.43 0. 0. 0. 0.56 0.43 0. 0.56]\n", " [0.5 0.45 0.5 0.19 0.19 0.19 0.3 0.25 0.19]]\n" ] } ], "source": [ "from sklearn.feature_extraction.text import TfidfTransformer\n", "\n", "tfidf = TfidfTransformer(use_idf=True,\n", " norm='l2',\n", " smooth_idf=True)\n", "print(tfidf.fit_transform(count.fit_transform(docs))\n", " .toarray())" ] }, { "cell_type": "markdown", "metadata": { "id": "UyUxVvv-cyCA" }, "source": [ "이전 절에서 보았듯이 세 번째 문서에서 단어 ‘is’가 가장 많이 나타났기 때문에 단어 빈도가 가장 컸습니다. 동일한 특성 벡터를 tf-idf로 변환하면 단어 ‘is’는 비교적 작은 tf-idf를 가집니다(0.45). 이 단어는 첫 번째와 두 번째 문서에도 나타나므로 판별에 유용한 정보를 가지고 있지 않을 것입니다." ] }, { "cell_type": "markdown", "metadata": { "id": "U9h-50XUcyCA" }, "source": [ "수동으로 특성 벡터에 있는 각 단어의 tf-idf를 계산해 보면 `TfidfTransformer`가 앞서 정의한 표준 tf-idf 공식과 조금 다르게 계산한다는 것을 알 수 있습니다. 사이킷런에 구현된 역문서 빈도 공식은 다음과 같습니다." ] }, { "cell_type": "markdown", "metadata": { "id": "UuC9xxiTcyCA" }, "source": [ "$$\\text{idf} (t,d) = log\\frac{1 + n_d}{1 + \\text{df}(d, t)}$$\n", "\n", "비슷하게 사이킷런에서 계산하는 tf-idf는 앞서 정의한 공식과 조금 다릅니다:\n", "\n", "$$\\text{tf-idf}(t,d) = \\text{tf}(t,d) \\times (\\text{idf}(t,d)+1)$$\n", "\n", "일반적으로 tf-idf를 계산하기 전에 단어 빈도(tf)를 정규화하지만 `TfidfTransformer` 클래스는 tf-idf를 직접 정규화합니다. 사이킷런의 `TfidfTransformer`는 기본적으로 L2 정규화를 적용합니다(norm=’l2’). 정규화되지 않은 특성 벡터 v를 L2-노름으로 나누면 길이가 1인 벡터가 반환됩니다:\n", "\n", "$$v_{\\text{norm}} = \\frac{v}{||v||_2} = \\frac{v}{\\sqrt{v_{1}^{2} + v_{2}^{2} + \\dots + v_{n}^{2}}} = \\frac{v}{\\big (\\sum_{i=1}^{n} v_{i}^{2}\\big)^\\frac{1}{2}}$$\n", "\n", "TfidfTransformer의 작동 원리를 이해하기 위해 세 번째 문서에 있는 단어 ‘is'의 tf-idf를 예로 들어 계산해 보죠.\n", "\n", "세 번째 문서에서 단어 ‘is’의 단어 빈도는 3입니다(tf=3). 이 단어는 세 개 문서에 모두 나타나기 때문에 문서 빈도가 3입니다(df=3). 따라서 역문서 빈도는 다음과 같이 계산됩니다:\n", "\n", "$$\\text{idf}(\"is\", d3) = log \\frac{1+3}{1+3} = 0$$\n", "\n", "이제 tf-idf를 계산하기 위해 역문서 빈도에 1을 더하고 단어 빈도를 곱합니다:\n", "\n", "$$\\text{tf-idf}(\"is\",d3)= 3 \\times (0+1) = 3$$" ] }, { "cell_type": "code", "execution_count": 14, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "DuhA5F0mcyCA", "outputId": "5d638ce9-8aaf-4a55-c432-f1ea680eae71" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "tf-idf of term \"is\" = 3.00\n" ] } ], "source": [ "tf_is = 3\n", "n_docs = 3\n", "idf_is = np.log((n_docs+1) / (3+1))\n", "tfidf_is = tf_is * (idf_is + 1)\n", "print('tf-idf of term \"is\" = %.2f' % tfidf_is)" ] }, { "cell_type": "markdown", "metadata": { "id": "p3sEX98ocyCA" }, "source": [ "세 번째 문서에 있는 모든 단어에 대해 이런 계산을 반복하면 tf-idf 벡터 [3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0, 1.69, 1.29]를 얻습니다. 이 특성 벡터의 값은 앞서 사용했던 TfidfTransformer에서 얻은 값과 다릅니다. tf-idf 계산에서 빠트린 마지막 단계는 다음과 같은 L2-정규화입니다::" ] }, { "cell_type": "markdown", "metadata": { "id": "hrFAsgmdcyCA" }, "source": [ "$$\\text{tfi-df}_{norm} = \\frac{[3.39, 3.0, 3.39, 1.29, 1.29, 1.29, 2.0 , 1.69, 1.29]}{\\sqrt{[3.39^2, 3.0^2, 3.39^2, 1.29^2, 1.29^2, 1.29^2, 2.0^2 , 1.69^2, 1.29^2]}}$$\n", "\n", "$$=[0.5, 0.45, 0.5, 0.19, 0.19, 0.19, 0.3, 0.25, 0.19]$$\n", "\n", "$$\\Rightarrow \\text{tfi-df}_{norm}(\"is\", d3) = 0.45$$" ] }, { "cell_type": "markdown", "metadata": { "id": "WL6MVgwPcyCA" }, "source": [ "결과에서 보듯이 사이킷런의 `TfidfTransformer`에서 반환된 결과와 같아졌습니다. tf-idf 계산 방법을 이해했으므로 다음 절로 넘어가 이 개념을 영화 리뷰 데이터셋에 적용해 보죠." ] }, { "cell_type": "code", "execution_count": 15, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "0MJlrxXJcyCB", "outputId": "b22bb509-8a84-48a4-b080-a4f4fd56c82d" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([3.39, 3. , 3.39, 1.29, 1.29, 1.29, 2. , 1.69, 1.29])" ] }, "metadata": {}, "execution_count": 15 } ], "source": [ "tfidf = TfidfTransformer(use_idf=True, norm=None, smooth_idf=True)\n", "raw_tfidf = tfidf.fit_transform(count.fit_transform(docs)).toarray()[-1]\n", "raw_tfidf" ] }, { "cell_type": "code", "execution_count": 16, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "aIJOmCCTcyCB", "outputId": "c63383df-f91d-4da5-e479-48f7d58dcc7e" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "array([0.5 , 0.45, 0.5 , 0.19, 0.19, 0.19, 0.3 , 0.25, 0.19])" ] }, "metadata": {}, "execution_count": 16 } ], "source": [ "l2_tfidf = raw_tfidf / np.sqrt(np.sum(raw_tfidf**2))\n", "l2_tfidf" ] }, { "cell_type": "markdown", "metadata": { "id": "yREuvKk8cyCB" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "qY754xCtcyCB" }, "source": [ "## 8.2.3 텍스트 데이터 정제" ] }, { "cell_type": "code", "execution_count": 17, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "N6mpkSpEcyCB", "outputId": "0b08a24d-a0f3-4b8f-d2d7-3ed9b6139660" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'is seven.

Title (Brazil): Not Available'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 17 } ], "source": [ "df.loc[0, 'review'][-50:]" ] }, { "cell_type": "code", "execution_count": 18, "metadata": { "id": "COv5qDRwcyCB", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "860c880d-e2c4-4937-bc51-bb428f8987fb" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "<>:4: SyntaxWarning: invalid escape sequence '\\)'\n", "<>:6: SyntaxWarning: invalid escape sequence '\\W'\n", "<>:4: SyntaxWarning: invalid escape sequence '\\)'\n", "<>:6: SyntaxWarning: invalid escape sequence '\\W'\n", "/tmp/ipython-input-1939991203.py:4: SyntaxWarning: invalid escape sequence '\\)'\n", " emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)',\n", "/tmp/ipython-input-1939991203.py:6: SyntaxWarning: invalid escape sequence '\\W'\n", " text = (re.sub('[\\W]+', ' ', text.lower()) +\n" ] } ], "source": [ "import re\n", "def preprocessor(text):\n", " text = re.sub('<[^>]*>', '', text)\n", " emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)',\n", " text)\n", " text = (re.sub('[\\W]+', ' ', text.lower()) +\n", " ' '.join(emoticons).replace('-', ''))\n", " return text" ] }, { "cell_type": "code", "execution_count": 19, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "kvJb2z2wcyCC", "outputId": "a989eb4c-b904-4c15-e9ed-a0239de4f855" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'is seven title brazil not available'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 19 } ], "source": [ "preprocessor(df.loc[0, 'review'][-50:])" ] }, { "cell_type": "code", "execution_count": 20, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 36 }, "id": "92Bvy1TWcyCC", "outputId": "e1121722-bf14-4c6b-8aa4-ad4e9fd7b699" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "'this is a test :) :( :)'" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "string" } }, "metadata": {}, "execution_count": 20 } ], "source": [ "preprocessor(\"This :) is :( a test :-)!\")" ] }, { "cell_type": "code", "execution_count": 21, "metadata": { "id": "j_jzzuN8cyCC" }, "outputs": [], "source": [ "df['review'] = df['review'].apply(preprocessor)" ] }, { "cell_type": "markdown", "metadata": { "id": "6IG2tpmUcyCC" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "i_r0KbbWcyCC" }, "source": [ "## 8.2.4 문서를 토큰으로 나누기" ] }, { "cell_type": "code", "execution_count": 22, "metadata": { "id": "kLpd983UcyCC" }, "outputs": [], "source": [ "from nltk.stem.porter import PorterStemmer\n", "\n", "porter = PorterStemmer()\n", "\n", "def tokenizer(text):\n", " return text.split()\n", "\n", "\n", "def tokenizer_porter(text):\n", " return [porter.stem(word) for word in text.split()]" ] }, { "cell_type": "code", "execution_count": 23, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "Ol5jOif8cyCC", "outputId": "35c9b6c9-d924-4351-aa52-506f942d8c25" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['runners', 'like', 'running', 'and', 'thus', 'they', 'run']" ] }, "metadata": {}, "execution_count": 23 } ], "source": [ "tokenizer('runners like running and thus they run')" ] }, { "cell_type": "code", "execution_count": 24, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fF_2aZy4cyCD", "outputId": "4289bae3-9f67-40a8-c733-4b2b58d93da6" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['runner', 'like', 'run', 'and', 'thu', 'they', 'run']" ] }, "metadata": {}, "execution_count": 24 } ], "source": [ "tokenizer_porter('runners like running and thus they run')" ] }, { "cell_type": "code", "execution_count": 25, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "AokNRHlgcyCD", "outputId": "00f20a31-2176-4ffc-bc8c-42395590b122" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "[nltk_data] Downloading package stopwords to /root/nltk_data...\n", "[nltk_data] Unzipping corpora/stopwords.zip.\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "True" ] }, "metadata": {}, "execution_count": 25 } ], "source": [ "import nltk\n", "\n", "nltk.download('stopwords')" ] }, { "cell_type": "code", "execution_count": 26, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "9nnbYs76cyCD", "outputId": "b3d2a7f9-b804-42d6-820e-c911fff91c68" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "['runner', 'like', 'run', 'run', 'lot']" ] }, "metadata": {}, "execution_count": 26 } ], "source": [ "from nltk.corpus import stopwords\n", "\n", "stop = stopwords.words('english')\n", "[w for w in tokenizer_porter('a runner likes running and runs a lot')[-10:]\n", "if w not in stop]" ] }, { "cell_type": "markdown", "metadata": { "id": "TTNaG1HgcyCD" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "Z-9h1iZPcyCD" }, "source": [ "# 8.3 문서 분류를 위한 로지스틱 회귀 모델 훈련" ] }, { "cell_type": "markdown", "metadata": { "id": "LxZR-OXGcyCD" }, "source": [ "GridSearch 속도를 높이기 위해 HTML과 구두점을 삭제합니다:" ] }, { "cell_type": "code", "execution_count": 27, "metadata": { "collapsed": true, "id": "D82QePVRcyCD" }, "outputs": [], "source": [ "X_train = df.loc[:25000, 'review'].values\n", "y_train = df.loc[:25000, 'sentiment'].values\n", "X_test = df.loc[25000:, 'review'].values\n", "y_test = df.loc[25000:, 'sentiment'].values" ] }, { "cell_type": "code", "execution_count": 28, "metadata": { "id": "R-hUDykscyCE" }, "outputs": [], "source": [ "from sklearn.pipeline import Pipeline\n", "from sklearn.linear_model import LogisticRegression\n", "from sklearn.feature_extraction.text import TfidfVectorizer\n", "from sklearn.model_selection import GridSearchCV\n", "\n", "tfidf = TfidfVectorizer(strip_accents=None,\n", " lowercase=False,\n", " preprocessor=None)\n", "\n", "param_grid = [{'vect__ngram_range': [(1, 1)],\n", " 'vect__stop_words': [stop, None],\n", " 'vect__tokenizer': [tokenizer, tokenizer_porter],\n", " 'clf__penalty': ['l1', 'l2'],\n", " 'clf__C': [1.0, 10.0, 100.0]},\n", " {'vect__ngram_range': [(1, 1)],\n", " 'vect__stop_words': [stop, None],\n", " 'vect__tokenizer': [tokenizer, tokenizer_porter],\n", " 'vect__use_idf':[False],\n", " 'vect__norm':[None],\n", " 'clf__penalty': ['l1', 'l2'],\n", " 'clf__C': [1.0, 10.0, 100.0]},\n", " ]\n", "\n", "lr_tfidf = Pipeline([('vect', tfidf),\n", " ('clf', LogisticRegression(random_state=0, solver='liblinear'))])\n", "\n", "gs_lr_tfidf = GridSearchCV(lr_tfidf, param_grid,\n", " scoring='accuracy',\n", " cv=5,\n", " n_jobs=-1)" ] }, { "cell_type": "markdown", "metadata": { "id": "O6lOUGvwcyCE" }, "source": [ "**`n_jobs` 매개변수에 대하여**\n", "\n", "앞의 코드 예제에서 컴퓨터에 있는 모든 CPU 코어를 사용해 그리드 서치의 속도를 높이려면 (`n_jobs=1` 대신) `n_jobs=-1`로 지정하는 것이 좋습니다. 일부 시스템에서는 멀티프로세싱을 위해 `n_jobs=-1`로 지정할 때 `tokenizer` 와 `tokenizer_porter` 함수의 직렬화에 문제가 발생할 수 있습니다. 이런 경우 `[tokenizer, tokenizer_porter]`를 `[str.split]`로 바꾸어 문제를 해결할 수 있습니다. 다만 `str.split`로 바꾸면 어간 추출을 하지 못합니다." ] }, { "cell_type": "markdown", "metadata": { "id": "4urLkAyGcyCE" }, "source": [ "**코드 실행 시간에 대하여**\n", "\n", "다음 코드 셀을 실행하면 시스템에 따라 **30~60분 정도 걸릴 수 있습니다**. 매개변수 그리드에서 정의한 대로 2*2*2*3*5 + 2*2*2*3*5 = 240개의 모델을 훈련하기 때문입니다.\n", "\n", "**코랩을 사용할 경우에도 CPU 코어가 많지 않기 때문에 실행 시간이 오래 걸릴 수 있습니다.**\n", "\n", "너무 오래 기다리기 어렵다면 데이터셋의 훈련 샘플의 수를 다음처럼 줄일 수 있습니다:\n", "\n", " X_train = df.loc[:2500, 'review'].values\n", " y_train = df.loc[:2500, 'sentiment'].values\n", " \n", "훈련 세트 크기를 줄이는 것은 모델 성능을 감소시킵니다. 그리드에 지정한 매개변수를 삭제하면 훈련한 모델 수를 줄일 수 있습니다. 예를 들면 다음과 같습니다:\n", "\n", " param_grid = [{'vect__ngram_range': [(1, 1)],\n", " 'vect__stop_words': [stop, None],\n", " 'vect__tokenizer': [tokenizer],\n", " 'clf__penalty': ['l1', 'l2'],\n", " 'clf__C': [1.0, 10.0]},\n", " ]" ] }, { "cell_type": "code", "execution_count": 29, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 249 }, "id": "VUm_76rGcyCE", "outputId": "02725e16-e37a-49b6-8f74-cb2a5c3ddb3c" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.12/dist-packages/sklearn/feature_extraction/text.py:517: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'\n", " warnings.warn(\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "GridSearchCV(cv=5,\n", " estimator=Pipeline(steps=[('vect',\n", " TfidfVectorizer(lowercase=False)),\n", " ('clf',\n", " LogisticRegression(random_state=0,\n", " solver='liblinear'))]),\n", " n_jobs=-1,\n", " param_grid=[{'clf__C': [1.0, 10.0, 100.0],\n", " 'clf__penalty': ['l1', 'l2'],\n", " 'vect__ngram_range': [(1, 1)],\n", " 'vect__stop_words': [['a', 'about', 'above', 'after',\n", " 'again', 'against', 'ain',\n", " 'all', 'am', 'an', 'and', '...\n", " 'vect__stop_words': [['a', 'about', 'above', 'after',\n", " 'again', 'against', 'ain',\n", " 'all', 'am', 'an', 'and', 'any',\n", " 'are', 'aren', \"aren't\", 'as',\n", " 'at', 'be', 'because', 'been',\n", " 'before', 'being', 'below',\n", " 'between', 'both', 'but', 'by',\n", " 'can', 'couldn', \"couldn't\", ...],\n", " None],\n", " 'vect__tokenizer': [,\n", " ],\n", " 'vect__use_idf': [False]}],\n", " scoring='accuracy')" ], "text/html": [ "
GridSearchCV(cv=5,\n",
              "             estimator=Pipeline(steps=[('vect',\n",
              "                                        TfidfVectorizer(lowercase=False)),\n",
              "                                       ('clf',\n",
              "                                        LogisticRegression(random_state=0,\n",
              "                                                           solver='liblinear'))]),\n",
              "             n_jobs=-1,\n",
              "             param_grid=[{'clf__C': [1.0, 10.0, 100.0],\n",
              "                          'clf__penalty': ['l1', 'l2'],\n",
              "                          'vect__ngram_range': [(1, 1)],\n",
              "                          'vect__stop_words': [['a', 'about', 'above', 'after',\n",
              "                                                'again', 'against', 'ain',\n",
              "                                                'all', 'am', 'an', 'and', '...\n",
              "                          'vect__stop_words': [['a', 'about', 'above', 'after',\n",
              "                                                'again', 'against', 'ain',\n",
              "                                                'all', 'am', 'an', 'and', 'any',\n",
              "                                                'are', 'aren', "aren't", 'as',\n",
              "                                                'at', 'be', 'because', 'been',\n",
              "                                                'before', 'being', 'below',\n",
              "                                                'between', 'both', 'but', 'by',\n",
              "                                                'can', 'couldn', "couldn't", ...],\n",
              "                                               None],\n",
              "                          'vect__tokenizer': [<function tokenizer at 0x7d90d71d4400>,\n",
              "                                              <function tokenizer_porter at 0x7d90d515af20>],\n",
              "                          'vect__use_idf': [False]}],\n",
              "             scoring='accuracy')
In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook.
On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.
" ] }, "metadata": {}, "execution_count": 29 } ], "source": [ "gs_lr_tfidf.fit(X_train, y_train)" ] }, { "cell_type": "code", "execution_count": 30, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "jS5WP6zEcyCF", "outputId": "ce38a90b-c981-418a-c4c8-ce441996854c" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "최적의 매개변수 조합: {'clf__C': 10.0, 'clf__penalty': 'l2', 'vect__ngram_range': (1, 1), 'vect__stop_words': None, 'vect__tokenizer': } \n", "CV 정확도: 0.897\n" ] } ], "source": [ "print('최적의 매개변수 조합: %s ' % gs_lr_tfidf.best_params_)\n", "print('CV 정확도: %.3f' % gs_lr_tfidf.best_score_)" ] }, { "cell_type": "code", "execution_count": 31, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "L9UElDZpcyCF", "outputId": "52752c0a-fbed-44d4-cd59-ce4b6b7e05cd" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "테스트 정확도: 0.899\n" ] } ], "source": [ "clf = gs_lr_tfidf.best_estimator_\n", "print('테스트 정확도: %.3f' % clf.score(X_test, y_test))" ] }, { "cell_type": "markdown", "metadata": { "id": "LoTPCB5YcyCF" }, "source": [ "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "3PlVtolxcyCF" }, "source": [ "#### 주석 시작:\n", "\n", "`gs_lr_tfidf.best_score_`는 k-폴드 교차 검증의 평균 점수입니다. 즉 5-폴드 교차 검증으로 `GridSearchCV` 객체를 훈련하면 `best_score_` 속성은 최상의 모델에 대한 5-폴드 점수의 평균을 반환합니다. 예를 들면 다음과 같습니다:" ] }, { "cell_type": "code", "execution_count": 32, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "dqnaINsIcyCF", "outputId": "2a9ba454-1e3c-4ed7-eb69-7561ca34c512" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "array([0.4, 0.2, 0.6, 0.2, 0.4])" ] }, "metadata": {}, "execution_count": 32 } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "import numpy as np\n", "\n", "from sklearn.model_selection import StratifiedKFold\n", "from sklearn.model_selection import cross_val_score\n", "\n", "np.random.seed(0)\n", "np.set_printoptions(precision=6)\n", "y = [np.random.randint(3) for i in range(25)]\n", "X = (y + np.random.randn(25)).reshape(-1, 1)\n", "\n", "cv5_idx = list(StratifiedKFold(n_splits=5, shuffle=False).split(X, y))\n", "\n", "lr = LogisticRegression(random_state=123, multi_class='ovr', solver='lbfgs')\n", "cross_val_score(lr, X, y, cv=cv5_idx)" ] }, { "cell_type": "markdown", "metadata": { "id": "_Y0TYRZFcyCG" }, "source": [ "위 코드를 실행하면 클래스 레이블에 해당하는 랜덤한 정수 데이터셋을 만듭니다. 그다음 5-폴드의 인덱스(`cv5_idx`)를 `cross_val_score` 함수에 전달하여 5개의 정확도 점수를 받습니다. 이 점수가 다섯 개의 테스트 폴드에 대한 정확도 값입니다.\n", "\n", "그다음 `GridSearchCV` 객체를 사용해 동일한 5-폴드 인덱스(`cv5_idx`)를 전달해 보죠:" ] }, { "cell_type": "code", "execution_count": 33, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "ieGoMTv5cyCG", "outputId": "d01b44b4-b797-4af8-ad7b-41464b15ef11" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Fitting 5 folds for each of 1 candidates, totalling 5 fits\n", "[CV 1/5] END ..................................., score=0.400 total time= 0.0s\n", "[CV 2/5] END ..................................., score=0.200 total time= 0.0s\n", "[CV 3/5] END ..................................., score=0.600 total time= 0.0s\n", "[CV 4/5] END ..................................., score=0.200 total time= 0.0s\n", "[CV 5/5] END ..................................., score=0.400 total time= 0.0s\n" ] }, { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n" ] } ], "source": [ "from sklearn.model_selection import GridSearchCV\n", "\n", "lr = LogisticRegression(solver='lbfgs', multi_class='ovr', random_state=1)\n", "gs = GridSearchCV(lr, {}, cv=cv5_idx, verbose=3).fit(X, y)" ] }, { "cell_type": "markdown", "metadata": { "id": "7CSWJmkkcyCG" }, "source": [ "여기서 볼 수 있듯이 5-폴드에 대한 점수는 `cross_val_score`에서 얻은 것과 정확히 일치합니다." ] }, { "cell_type": "markdown", "metadata": { "id": "TWOgOEI7cyCG" }, "source": [ "이제 `fit` 메서드를 호출한 후 `GridSearchCV` 객체의 `best_score_` 속성은 최상의 모델의 평균 정확도를 반환합니다:" ] }, { "cell_type": "code", "execution_count": 34, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "KMQFOhwScyCG", "outputId": "dc699145-576e-40b7-f5a9-d589a94abc56" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "np.float64(0.36000000000000004)" ] }, "metadata": {}, "execution_count": 34 } ], "source": [ "gs.best_score_" ] }, { "cell_type": "markdown", "metadata": { "id": "uMEbcfN0cyCG" }, "source": [ "이 점수는 `cross_val_score`로 계산한 평균 정확도와 동일합니다." ] }, { "cell_type": "code", "execution_count": 35, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "UwWre_ZocyCH", "outputId": "4b086c20-6ccc-4ca1-e16d-137525f4db71" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n", "/usr/local/lib/python3.12/dist-packages/sklearn/linear_model/_logistic.py:1256: FutureWarning: 'multi_class' was deprecated in version 1.5 and will be removed in 1.7. Use OneVsRestClassifier(LogisticRegression(..)) instead. Leave it to its default value to avoid this warning.\n", " warnings.warn(\n" ] }, { "output_type": "execute_result", "data": { "text/plain": [ "np.float64(0.36000000000000004)" ] }, "metadata": {}, "execution_count": 35 } ], "source": [ "lr = LogisticRegression(solver='lbfgs', multi_class='ovr', random_state=1)\n", "cross_val_score(lr, X, y, cv=cv5_idx).mean()" ] }, { "cell_type": "markdown", "metadata": { "id": "TNKUcBuNcyCH" }, "source": [ "#### 주석 끝\n", "\n", "
\n", "
" ] }, { "cell_type": "markdown", "metadata": { "id": "ZCrR18tdcyCH" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "ZAaA7vnucyCH" }, "source": [ "# 8.4 대용량 데이터 처리: 온라인 알고리즘과 외부 메모리 학습" ] }, { "cell_type": "code", "execution_count": 36, "metadata": { "id": "IbhTytKWcyCH" }, "outputs": [], "source": [ "# 이 셀의 코드는 책에 포함되어 있지 않습니다.\n", "# 이전 코드를 실행하지 않고 바로 시작할 수 있도록 편의를 위해 추가했습니다.\n", "\n", "import os\n", "import gzip\n", "\n", "\n", "if not os.path.isfile('movie_data.csv'):\n", " if not os.path.isfile('movie_data.csv.gz'):\n", " print('Please place a copy of the movie_data.csv.gz'\n", " 'in this directory. You can obtain it by'\n", " 'a) executing the code in the beginning of this'\n", " 'notebook or b) by downloading it from GitHub:'\n", " 'https://github.com/rickiepark/python-machine-learning-'\n", " 'book-3rd-edition/blob/master/ch08/movie_data.csv.gz')\n", " else:\n", " with gzip.open('movie_data.csv.gz', 'rb') as in_f, \\\n", " open('movie_data.csv', 'wb') as out_f:\n", " out_f.write(in_f.read())" ] }, { "cell_type": "code", "execution_count": 37, "metadata": { "id": "b6ZDaEfJcyCH", "colab": { "base_uri": "https://localhost:8080/" }, "outputId": "b7634e5a-14da-435e-812b-18c05e9b6496" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "<>:13: SyntaxWarning: invalid escape sequence '\\)'\n", "<>:14: SyntaxWarning: invalid escape sequence '\\W'\n", "<>:13: SyntaxWarning: invalid escape sequence '\\)'\n", "<>:14: SyntaxWarning: invalid escape sequence '\\W'\n", "/tmp/ipython-input-2241221198.py:13: SyntaxWarning: invalid escape sequence '\\)'\n", " emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)', text.lower())\n", "/tmp/ipython-input-2241221198.py:14: SyntaxWarning: invalid escape sequence '\\W'\n", " text = re.sub('[\\W]+', ' ', text.lower()) +\\\n" ] } ], "source": [ "import numpy as np\n", "import re\n", "from nltk.corpus import stopwords\n", "\n", "\n", "# `stop` 객체를 앞에서 정의했지만 이전 코드를 실행하지 않고\n", "# 편의상 여기에서부터 코드를 실행하기 위해 다시 만듭니다.\n", "stop = stopwords.words('english')\n", "\n", "\n", "def tokenizer(text):\n", " text = re.sub('<[^>]*>', '', text)\n", " emoticons = re.findall('(?::|;|=)(?:-)?(?:\\)|\\(|D|P)', text.lower())\n", " text = re.sub('[\\W]+', ' ', text.lower()) +\\\n", " ' '.join(emoticons).replace('-', '')\n", " tokenized = [w for w in text.split() if w not in stop]\n", " return tokenized\n", "\n", "\n", "def stream_docs(path):\n", " with open(path, 'r', encoding='utf-8') as csv:\n", " next(csv) # 헤더 넘기기\n", " for line in csv:\n", " text, label = line[:-3], int(line[-2])\n", " yield text, label" ] }, { "cell_type": "code", "execution_count": 38, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "-oHQu4x-cyCH", "outputId": "30a96b64-8a4b-4ae3-c1d7-d066a717cf4c" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "('\"In 1974, the teenager Martha Moxley (Maggie Grace) moves to the high-class area of Belle Haven, Greenwich, Connecticut. On the Mischief Night, eve of Halloween, she was murdered in the backyard of her house and her murder remained unsolved. Twenty-two years later, the writer Mark Fuhrman (Christopher Meloni), who is a former LA detective that has fallen in disgrace for perjury in O.J. Simpson trial and moved to Idaho, decides to investigate the case with his partner Stephen Weeks (Andrew Mitchell) with the purpose of writing a book. The locals squirm and do not welcome them, but with the support of the retired detective Steve Carroll (Robert Forster) that was in charge of the investigation in the 70\\'s, they discover the criminal and a net of power and money to cover the murder.

\"\"Murder in Greenwich\"\" is a good TV movie, with the true story of a murder of a fifteen years old girl that was committed by a wealthy teenager whose mother was a Kennedy. The powerful and rich family used their influence to cover the murder for more than twenty years. However, a snoopy detective and convicted perjurer in disgrace was able to disclose how the hideous crime was committed. The screenplay shows the investigation of Mark and the last days of Martha in parallel, but there is a lack of the emotion in the dramatization. My vote is seven.

Title (Brazil): Not Available\"',\n", " 1)" ] }, "metadata": {}, "execution_count": 38 } ], "source": [ "next(stream_docs(path='movie_data.csv'))" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "id": "pDIeRNdWcyCH" }, "outputs": [], "source": [ "def get_minibatch(doc_stream, size):\n", " docs, y = [], []\n", " try:\n", " for _ in range(size):\n", " text, label = next(doc_stream)\n", " docs.append(text)\n", " y.append(label)\n", " except StopIteration:\n", " return None, None\n", " return docs, y" ] }, { "cell_type": "code", "execution_count": 40, "metadata": { "id": "Q1Nv4vyicyCI" }, "outputs": [], "source": [ "from sklearn.feature_extraction.text import HashingVectorizer\n", "from sklearn.linear_model import SGDClassifier\n", "\n", "\n", "vect = HashingVectorizer(decode_error='ignore',\n", " n_features=2**21,\n", " preprocessor=None,\n", " tokenizer=tokenizer)" ] }, { "cell_type": "code", "execution_count": 41, "metadata": { "id": "Y8mJyKD7cyCI" }, "outputs": [], "source": [ "from distutils.version import LooseVersion as Version\n", "\n", "clf = SGDClassifier(loss='log_loss', random_state=1)\n", "\n", "doc_stream = stream_docs(path='movie_data.csv')" ] }, { "cell_type": "code", "execution_count": 42, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "dNGiQGLjcyCI", "outputId": "ac6e4741-f35e-4c5e-8f7e-5b904efd6463" }, "outputs": [ { "output_type": "stream", "name": "stderr", "text": [ "0% [##############################] 100% | ETA: 00:00:00\n", "Total time elapsed: 00:00:34\n" ] } ], "source": [ "import pyprind\n", "pbar = pyprind.ProgBar(45)\n", "\n", "classes = np.array([0, 1])\n", "for _ in range(45):\n", " X_train, y_train = get_minibatch(doc_stream, size=1000)\n", " if not X_train:\n", " break\n", " X_train = vect.transform(X_train)\n", " clf.partial_fit(X_train, y_train, classes=classes)\n", " pbar.update()" ] }, { "cell_type": "code", "execution_count": 43, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "fMLgSGMHcyCI", "outputId": "9054a3ee-0f80-4201-bdc9-b3f420173dbe" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "정확도: 0.868\n" ] } ], "source": [ "X_test, y_test = get_minibatch(doc_stream, size=5000)\n", "X_test = vect.transform(X_test)\n", "print('정확도: %.3f' % clf.score(X_test, y_test))" ] }, { "cell_type": "code", "execution_count": 44, "metadata": { "id": "CjpZRaP0cyCI" }, "outputs": [], "source": [ "clf = clf.partial_fit(X_test, y_test)" ] }, { "cell_type": "markdown", "metadata": { "id": "5MztubRoL8tD" }, "source": [ "
" ] }, { "cell_type": "markdown", "metadata": { "id": "DUuim6rGcyCI" }, "source": [ "# 8.5 잠재 디리클레 할당을 사용한 토픽 모델링" ] }, { "cell_type": "markdown", "metadata": { "id": "VCoicyiUcyCI" }, "source": [ "## 8.5.1 LDA를 사용한 텍스트 문서 분해" ] }, { "cell_type": "markdown", "metadata": { "id": "BgKVwW-WcyCI" }, "source": [ "## 8.5.2 사이킷런의 LDA" ] }, { "cell_type": "code", "execution_count": 45, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 148 }, "id": "Xd8wHrIicyCJ", "outputId": "53dc913e-5b28-47a8-ceaa-fab09ea1ed37" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ " review sentiment\n", "0 In 1974, the teenager Martha Moxley (Maggie Gr... 1\n", "1 OK... so... I really like Kris Kristofferson a... 0\n", "2 ***SPOILER*** Do not read this, if you think a... 0" ], "text/html": [ "\n", "
\n", "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
reviewsentiment
0In 1974, the teenager Martha Moxley (Maggie Gr...1
1OK... so... I really like Kris Kristofferson a...0
2***SPOILER*** Do not read this, if you think a...0
\n", "
\n", "
\n", "\n", "
\n", " \n", "\n", " \n", "\n", " \n", "
\n", "\n", "\n", "
\n", " \n", "\n", "\n", "\n", " \n", "
\n", "\n", "
\n", "
\n" ], "application/vnd.google.colaboratory.intrinsic+json": { "type": "dataframe", "variable_name": "df", "summary": "{\n \"name\": \"df\",\n \"rows\": 50000,\n \"fields\": [\n {\n \"column\": \"review\",\n \"properties\": {\n \"dtype\": \"string\",\n \"num_unique_values\": 49582,\n \"samples\": [\n \"Every time I think about this film I feel physically ill. To read such a great book and later discover there's a film of it was a great feeling. Years later and imagine my joy at switching on the sci-fi channel and finding it starts in just 5mins!!! Up go the titles and then uggg. If just a couple of things had changed OK. Everything is changed. Numerous characters are removed entirely new rubbish ones are added. The main hero is shrunk and de-aged by about 30 years, and hilariously his girlfriend/wife is now his mother! Even the dog is reduced to sub-lassie capabilities. This is truly appalling cinema at its absolute worst. I would quite happily remove my own toenails with pliers rather than sit through another horrific viewing, and I urge anyone thinking of watching this - please don't. If you own a copy burn it now, right now and think how much better your life would have been had this celluloid insult never occurred.\",\n \"As with all environmentally aware films from the 1970s SOYLENT GREEN has a rather cheesy view of what ecological meltdown is . Overpopulation means there`s too many people to feed ? I was under the impression that famines were caused by either war or failed economic policies . Stalin`s policy in the Soviet Union in the 1930s left millions dead because of famine and to this day the greatest man made tragedy was Mao`s rural policy in China which led over 30 million starvation deaths in the 1950s . And let`s not forget the great famines in the horn of Africa in the 1980s and 90s which were to do with conflicts not overpopulation . You might like to also consider that two of the most heavily populated areas on Earth , Hong Kong and Macau , have never suffered a famine in modern times . Likewise the expansion of shanty towns around cities as seen here isn`t strictly down to overpopulation - it`s down to economic factors where people flock to cities to find better paid work than in the countryside ( It`s a symptom of industrial progress - not of too many births ) so the image of the streets of New York city being too congested to walk through and of having people sleep in stairwells is somewhat laughable

But don`t be fooled into thinking SOYLENT GREEN is a pile of corny tree hugging crap because I consider this to be the best ecological film of the

70s . It plays on the contempary audience`s knowledge of the world where Sol and Thorn are beside themselves with joy at finding fruit , brandy and fresh meat . Thorn gasps in amazement at having ice in his whisky , puffs on a cigarette and delivers the classic line \\\" If I could afford it I`d smoke two , maybe three of these a day \\\" . But it`s the visage of the euthanasia chamber that`s memorable as Thorn gazes at the images of wild animals , flowers , running water and snow covered mountains , a world Thorn`s generation has never known . This is a very haunting scene which makes SOYLENT GREEN a very memorable film , combined with the fact it features the final screen appearance of Edward G Robinson as the wise old Jew Sol Roth\",\n \"Yah. I know. It has the name \\\"Sinatra\\\" in the title, so how bad can it be? Well, it's bad, trust me! I rented this thinking it was some movie I missed in the theaters. It's not. It's some garbage \\\"movie\\\" made by the folks at Showtime (cable station). Geez, these cable stations make a few bucks they think they can make whatever garbage movies they want! It's not good. I am as big a Sinatra fan as any sane man, but this movie was just dumb. Boring. Dull. Unfunny. Uninteresting. The only redeeming quality is that (assuming they did stick to the facts) you do learn about what happened to the captors of Frank Jr. Otherwise it's just a stupid film.\"\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n },\n {\n \"column\": \"sentiment\",\n \"properties\": {\n \"dtype\": \"number\",\n \"std\": 0,\n \"min\": 0,\n \"max\": 1,\n \"num_unique_values\": 2,\n \"samples\": [\n 0,\n 1\n ],\n \"semantic_type\": \"\",\n \"description\": \"\"\n }\n }\n ]\n}" } }, "metadata": {}, "execution_count": 45 } ], "source": [ "import pandas as pd\n", "\n", "df = pd.read_csv('movie_data.csv', encoding='utf-8')\n", "df.head(3)" ] }, { "cell_type": "code", "execution_count": 46, "metadata": { "id": "x6iBTTXEcyCJ" }, "outputs": [], "source": [ "from sklearn.feature_extraction.text import CountVectorizer\n", "\n", "count = CountVectorizer(stop_words='english',\n", " max_df=.1,\n", " max_features=5000)\n", "X = count.fit_transform(df['review'].values)" ] }, { "cell_type": "code", "execution_count": 47, "metadata": { "id": "Tuj9PLJCcyCJ" }, "outputs": [], "source": [ "from sklearn.decomposition import LatentDirichletAllocation\n", "\n", "lda = LatentDirichletAllocation(n_components=10,\n", " random_state=123,\n", " learning_method='batch')\n", "X_topics = lda.fit_transform(X)" ] }, { "cell_type": "code", "execution_count": 48, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "3Ejgt5FScyCJ", "outputId": "3b9b568b-2a39-44fb-cef2-caae08141513" }, "outputs": [ { "output_type": "execute_result", "data": { "text/plain": [ "(10, 5000)" ] }, "metadata": {}, "execution_count": 48 } ], "source": [ "lda.components_.shape" ] }, { "cell_type": "code", "execution_count": 49, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "HEJ6hcpDcyCJ", "outputId": "edeb7ad6-f0e4-42ec-c179-70066456ead7" }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "Topic 1:\n", "horror worst script effects budget\n", "Topic 2:\n", "dvd watched video music guy\n", "Topic 3:\n", "war american series history documentary\n", "Topic 4:\n", "game killer murder thriller crime\n", "Topic 5:\n", "kids comedy episode series school\n", "Topic 6:\n", "family woman mother beautiful feel\n", "Topic 7:\n", "role performance comedy john plays\n", "Topic 8:\n", "action horror john effects dr\n", "Topic 9:\n", "book version original read music\n", "Topic 10:\n", "action wife father police james\n" ] } ], "source": [ "n_top_words = 5\n", "feature_names = count.get_feature_names_out()\n", "\n", "for topic_idx, topic in enumerate(lda.components_):\n", " print(\"Topic %d:\" % (topic_idx + 1))\n", " print(\" \".join([feature_names[i]\n", " for i in topic.argsort()\\\n", " [:-n_top_words - 1:-1]]))" ] }, { "cell_type": "markdown", "metadata": { "id": "Nkj0vL2WcyCJ" }, "source": [ "각 토픽에서 가장 중요한 단어 다섯 개를 기반으로 LDA가 다음 토픽을 구별했다고 추측할 수 있습니다.\n", "\n", "1.\t대체적으로 형편없는 영화(실제 토픽 카테고리가 되지 못함)\n", "2.\t가족 영화\n", "3.\t전쟁 영화\n", "4.\t예술 영화\n", "5.\t범죄 영화\n", "6.\t공포 영화\n", "7.\t코미디 영화\n", "8.\tTV 쇼와 관련된 영화\n", "9.\t소설을 원작으로 한 영화\n", "10.\t액션 영화" ] }, { "cell_type": "markdown", "metadata": { "id": "s4xvfj7EcyCJ" }, "source": [ "카테고리가 잘 선택됐는지 확인하기 위해 공포 영화 카테고리에서 3개 영화의 리뷰를 출력해 보죠(공포 영화는 카테고리 6이므로 인덱스는 5입니다):" ] }, { "cell_type": "code", "execution_count": 50, "metadata": { "colab": { "base_uri": "https://localhost:8080/" }, "id": "v6shgjZfcyCK", "outputId": "d856e6f1-9409-4e56-c4b1-21cbde844a10", "scrolled": true }, "outputs": [ { "output_type": "stream", "name": "stdout", "text": [ "\n", "공포 영화 #1:\n", "\"Come Undone\" appears to elicit a lot of opinions among the contributors to this forum. Granted, it's a film that promises a take on gay life, as most viewers expect and somehow, it gets away from that promise into an introspective view at a young man's soul. The film has a way of staying with us ev ...\n", "\n", "공포 영화 #2:\n", "The subject of children being terminally ill is difficult and saddening but 'The Cure' successfully portrays the idea that it doesn't have to be all doom and gloom and, if anything, children need to have hope and delight in their lives if they are to find peace before the end. It is also a film of r ...\n", "\n", "공포 영화 #3:\n", "I knew absolutely nothing about Chocolat before my viewing of it. I didn't know anything about the story, the cast, the director, or anything about the film's history. All I knew was it was a highly-acclaimed French film. Had I known more, I probably wouldn't have viewed the picture with an open min ...\n" ] } ], "source": [ "horror = X_topics[:, 5].argsort()[::-1]\n", "\n", "for iter_idx, movie_idx in enumerate(horror[:3]):\n", " print('\\n공포 영화 #%d:' % (iter_idx + 1))\n", " print(df['review'][movie_idx][:300], '...')" ] }, { "cell_type": "markdown", "metadata": { "id": "QXblpnHrcyCK" }, "source": [ "앞 코드에서 공포 영화 카테고리 중 최상위 3개의 리뷰에서 300자씩 출력했습니다. 정확히 어떤 영화에 속한 리뷰인지는 모르지만 공포 영화의 리뷰임을 알 수 있습니다(하지만 영화 #2는 1번 카테고리에 속하는 것처럼 보이기도 합니다)." ] }, { "cell_type": "markdown", "metadata": { "id": "Dv5Svl-8cyCK" }, "source": [ "
" ] } ], "metadata": { "anaconda-cloud": {}, "colab": { "name": "ch08.ipynb", "provenance": [] }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" }, "toc": { "nav_menu": {}, "number_sections": true, "sideBar": true, "skip_h1_title": false, "title_cell": "Table of Contents", "title_sidebar": "Contents", "toc_cell": false, "toc_position": {}, "toc_section_display": true, "toc_window_display": false } }, "nbformat": 4, "nbformat_minor": 0 }