{
"cells": [
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"***\n",
"***\n",
"# 文本挖掘简介\n",
"\n",
"***\n",
"***\n",
"\n",
"王成军\n",
"\n",
"wangchengjun@nju.edu.cn\n",
"\n",
"计算传播网 http://computational-communication.com"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# What can be learned from 5 million books\n",
"\n",
"http://v.youku.com/v_show/id_XMzA3OTA5MjUy.html\n",
"\n",
"This talk by Jean-Baptiste Michel and Erez Lieberman Aiden is phenomenal. \n",
"\n",
"Michel, J.-B., et al. (2011). Quantitative Analysis of Culture Using Millions of Digitized Books. Science, 331, 176–182."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
""
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"试一下谷歌图书的数据: https://books.google.com/ngrams/\n",
" \n",
"\n",
"数据下载： http://www.culturomics.org/home"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Bag-of-words model （BOW)\n",
"\n",
"Represent text as numerical feature vectors"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"- We create a vocabulary of unique tokens—for example, words—from the entire set of documents.\n",
"- We construct a feature vector from each document that contains the counts of how often each word occurs in the particular document."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"Since the unique words in each document represent only a small subset of all the\n",
"words in the bag-of-words vocabulary, the feature vectors will consist of mostly\n",
"zeros, which is why we call them sparse"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"
"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"Bag of words，也叫做“词袋”，在信息检索中，Bag of words model假定对于一个文本，忽略其词序和语法，句法，将其仅仅看做是一个词集合，或者说是词的一个组合，文本中每个词的出现都是独立的，不依赖于其他词是否出现，或者说当这篇文章的作者在任意一个位置选择一个词汇都不受前面句子的影响而独立选择的。这种假设虽然对自然语言进行了简化，便于模型化。\n",
"\n",
"假定在有些情况下是不合理的，例如在新闻个性化推荐中，采用Bag of words的模型就会出现问题。例如用户甲对“南京醉酒驾车事故”这个短语很感兴趣，采用bag of words忽略了顺序和句法，则认为用户甲对“南京”、“醉酒”、“驾车”和“事故”感兴趣，因此可能推荐出和“南京”，“公交车”，“事故”相关的新闻，这显然是不合理的。\n",
"\n",
"解决的方法可以采用SCPCD的方法抽取出整个短语，或者采用高阶（2阶以上）统计语言模型，例如bigram，trigram来将词序保留下来，相当于bag of bigram和bag of trigram，这样能在一定程度上解决这种问题。简言之，bag of words模型是否适用需要根据实际情况来确定。对于那些不可以忽视词序，语法和句法的场合均不能采用bag of words的方法。"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "slide"
}
},
"source": [
"# Transforming words into feature vectors"
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "fragment"
}
},
"source": [
"A document-term matrix or term-document matrix is a mathematical matrix that describes the frequency of terms that occur in a collection of documents. \n",
"\n",
"In a document-term matrix, rows correspond to documents in the collection and columns correspond to terms. \n",
"\n",
"There are various schemes for determining the value that each entry in the matrix should take. One such scheme is tf-idf. They are useful in the field of natural language processing."
]
},
{
"cell_type": "markdown",
"metadata": {
"slideshow": {
"slide_type": "subslide"
}
},
"source": [
"D1 = \"I like databases\"\n",
"\n",
"D2 = \"I hate databases\"\n",
"\n",
"| | I | like |hate | databases |\n",
"| -------------|:-------------:|:-------------:|:-------------:|-----:|\n",
"| D1| 1| 1 | 0 |1|\n",
"| D2| 1| 0 | 1 |1|"
]
},
{
"cell_type": "code",
"execution_count": 1,
"metadata": {
"ExecuteTime": {
"end_time": "2019-06-14T01:45:29.195050Z",
"start_time": "2019-06-14T01:45:28.326659Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [],
"source": [
"import numpy as np\n",
"from sklearn.feature_extraction.text import CountVectorizer\n",
"count = CountVectorizer()\n",
"docs = np.array([\n",
" 'The sun is shining',\n",
" 'The weather is sweet',\n",
" 'The sun is shining and the weather is sweet'])\n",
"bag = count.fit_transform(docs)"
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2018-04-30T02:37:14.609464Z",
"start_time": "2018-04-30T02:37:14.597741Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"'__class__ __delattr__ __dict__ __dir__ __doc__ __eq__ __format__ __ge__ __getattribute__ __gt__ __hash__ __init__ __le__ __lt__ __module__ __ne__ __new__ __reduce__ __reduce_ex__ __repr__ __setattr__ __sizeof__ __str__ __subclasshook__ __weakref__ _char_ngrams _char_wb_ngrams _check_vocabulary _count_vocab _get_param_names _limit_features _sort_features _validate_vocabulary _white_spaces _word_ngrams analyzer binary build_analyzer build_preprocessor build_tokenizer decode decode_error dtype encoding fit fit_transform fixed_vocabulary fixed_vocabulary_ get_feature_names get_params get_stop_words input inverse_transform lowercase max_df max_features min_df ngram_range preprocessor set_params stop_words stop_words_ strip_accents token_pattern tokenizer transform vocabulary vocabulary_'"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"' '.join(dir(count)) "
]
},
{
"cell_type": "code",
"execution_count": 2,
"metadata": {
"ExecuteTime": {
"end_time": "2019-06-14T01:46:37.149709Z",
"start_time": "2019-06-14T01:46:37.137530Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"['and', 'is', 'shining', 'sun', 'sweet', 'the', 'weather']"
]
},
"execution_count": 2,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"count.get_feature_names()"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {
"ExecuteTime": {
"end_time": "2019-06-14T01:46:53.244569Z",
"start_time": "2019-06-14T01:46:53.241008Z"
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"{'weather': 6, 'sweet': 4, 'the': 5, 'and': 0, 'is': 1, 'sun': 3, 'shining': 2}\n"
]
}
],
"source": [
"print(count.vocabulary_)"
]
},
{
"cell_type": "code",
"execution_count": 4,
"metadata": {
"ExecuteTime": {
"end_time": "2019-06-14T01:47:32.446539Z",
"start_time": "2019-06-14T01:47:32.442491Z"
},
"slideshow": {
"slide_type": "subslide"
}
},
"outputs": [
{
"data": {
"text/plain": [
"scipy.sparse.csr.csr_matrix"
]
},
"execution_count": 4,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"type(bag)"
]
},
{
"cell_type": "code",
"execution_count": 5,
"metadata": {
"ExecuteTime": {
"end_time": "2019-06-14T01:47:42.909814Z",
"start_time": "2019-06-14T01:47:42.905300Z"
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"[[0 1 1 1 0 1 0]\n",
" [0 1 0 0 1 1 1]\n",
" [1 2 1 1 1 2 1]]\n"
]
}
],
"source": [
"print(bag.toarray())"
]
},
{
"cell_type": "code",
"execution_count": 6,
"metadata": {
"ExecuteTime": {
"end_time": "2019-06-14T01:48:15.772203Z",
"start_time": "2019-06-14T01:48:15.251903Z"
},
"slideshow": {
"slide_type": "fragment"
}
},
"outputs": [
{
"data": {
"text/html": [
"\n",
"\n",
"
\n",
" \n",
" \n",
" | \n",
" and | \n",
" is | \n",
" shining | \n",
" sun | \n",
" sweet | \n",
" the | \n",
" weather | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0 | \n",
" 1 | \n",
" 1 | \n",
" 1 | \n",
" 0 | \n",
" 1 | \n",
" 0 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0 | \n",
" 1 | \n",
" 0 | \n",
" 0 | \n",
" 1 | \n",
" 1 | \n",
" 1 | \n",
"
\n",
" \n",
" | 2 | \n",
" 1 | \n",
" 2 | \n",
" 1 | \n",
" 1 | \n",
" 1 | \n",
" 2 | \n",
" 1 | \n",
"
\n",
" \n",
"
\n",
"
\n",
"
\n",
" \n",
" \n",
" | \n",
" and | \n",
" is | \n",
" shining | \n",
" sun | \n",
" sweet | \n",
" the | \n",
" weather | \n",
"
\n",
" \n",
" \n",
" \n",
" | 0 | \n",
" 0.000000 | \n",
" 1.0 | \n",
" 1.287682 | \n",
" 1.287682 | \n",
" 0.000000 | \n",
" 1.0 | \n",
" 0.000000 | \n",
"
\n",
" \n",
" | 1 | \n",
" 0.000000 | \n",
" 1.0 | \n",
" 0.000000 | \n",
" 0.000000 | \n",
" 1.287682 | \n",
" 1.0 | \n",
" 1.287682 | \n",
"
\n",
" \n",
" | 2 | \n",
" 1.693147 | \n",
" 2.0 | \n",
" 1.287682 | \n",
" 1.287682 | \n",
" 1.287682 | \n",
" 2.0 | \n",
" 1.287682 | \n",
"
\n",
" \n",
"
\n",
"