{ "cells": [ { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "***\n", "# 数据清洗：\n", "> # 对占中新闻进行数据清洗\n", "***\n", "***\n", "\n", "王成军\n", "\n", "wangchengjun@nju.edu.cn\n", "\n", "计算传播网 http://computational-communication.com" ] }, { "cell_type": "code", "execution_count": 70, "metadata": { "collapsed": false, "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "# 使用with open读取每一行数据\n", "with open(\"/Users/chengjun/github/cjc2016/data/occupycentral/zz-hk-2014-10.rtf\") as f:\n", " news = f.readlines()" ] }, { "cell_type": "code", "execution_count": 71, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "16541" ] }, "execution_count": 71, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 查看总共有多少行\n", "len(news)" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "~~~~~~~~~~~~~~~~~~~~~~~~~~ #1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n", "\n", "am730 | 2014-10-31 \\par\n", "\n", "A16| NEWS| C观点| By 施永青 \\par \n", "\n", "\n", "\n", "法治有整合社会功能 \\par\n", "\n", "\\par\n", "\n", "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n", "\n" ] } ], "source": [ "# 注意：标题和版面之间存在一个空行！所以title是block的第4个元素。\n", "for i in range(1, 8):\n", " print news[i].decode('gb18030')[:500]" ] }, { "cell_type": "code", "execution_count": 74, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\\par\\par 前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适应)与Goal Attainment(达标)的社会功能，今天续谈Integration(整合)的功能。人类社会是一个复杂系统，大系统内还有很多次系统，这些系统既独立，又牵连；既矛盾，又互适；有时互相促进，有时互相制约。正如一个人，既是社会的成员，又是公司的雇员；既是别人的儿子，又是朋辈中的「大哥」；其言行会同时受所处的不同系统所影响。\\par\\par 这些错综复杂的系统之间不可能没有冲突，为了避免社会因纠纷得不到恰当的处理而分崩离析，社会必须发展出一套整合矛盾的方式，这就需要有司法制度。\\par\\par 按帕森斯的说法，法治的基础是先要界定产权。这样才能避免因争夺资源而产生无休止的冲突。再者，交易亦需要在产权获得界定后才能进行。有交易才有市场，才能透过市场机制进行公平竞争，推动经济发展。\\par\\par 另一方面，社会亦需要为人权下定义，这样，政府才能在有认受性的情况下组成，才能有效地去处理公众事务。此外，社会还需要有一套合乎公义的会议程序，以决定如何汇聚众人的意愿。\\par\\par 有了这些基础之\n" ] } ], "source": [ "# 需要对中文编码的对象使用中文的方式进行解码\n", "print news[17].decode('gb18030')[:500]" ] }, { "cell_type": "code", "execution_count": 75, "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "# 定义一个函数：实现解码、编码、清洗效果\n", "def stringclean(s):\n", " s = s.decode('gb18030').encode('utf8')\n", " s = s.replace(r'\\loch\\af0\\hich\\af0\\dbch\\f15 \\b\\cf6 ', '')\n", " s = s.replace(r'\\loch\\af0\\hich\\af0\\dbch\\f15 \\b0\\cf0 ', '')\n", " s = s.replace('\\par', '').replace('\\n', '')\n", " return s" ] }, { "cell_type": "code", "execution_count": 87, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "'bbccdd ee'" ] }, "execution_count": 87, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'aabbccdd ee'.strip('a')" ] }, { "cell_type": "code", "execution_count": 94, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "u'ccdd ee'" ] }, "execution_count": 94, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'aabbccdd ee'.strip('ab')" ] }, { "cell_type": "code", "execution_count": 103, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "'abccdd ee'" ] }, "execution_count": 103, "metadata": {}, "output_type": "execute_result" } ], "source": [ "'aabbccdd ee'.replace('ab', '')" ] }, { "cell_type": "code", "execution_count": 76, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适应)与Goal Attainment(达标)的社会功能，今天续谈Integration(整合)的功能。人类社会是一个复杂系统，大系统内还有很多次系统，这些系统既独立，又牵连；既矛盾，又互适；有时互相促进，有时互相制约。正如一个人，既是社会的成员，又是公司的雇员；既是别人的儿子，又是朋辈中的「大哥」；其言行会同时受所处的不同系统所影响。 这些错综复杂的系统之间不可能没有冲突，为了避免社会因纠纷得不到恰当的处理而分崩离析，社会必须发展出一套整合矛盾的方式，这就需要有司法制度。 按帕森斯的说法，法治的基础是先要界定产权。这样才能避免因争夺资源而产生无休止的冲突。再者，交易亦需要在产权获得界定后才能进行。有交易才有市场，才能透过市场机制进行公平竞争，推动经济发展。 另一方面，社会亦需要为人权下定义，这样，政府才能在有认受性的情况下组成，才能有效地去处理公众事务。此外，社会还需要有一套合乎公义的会议程序，以决定如何汇聚众人的意愿。 有了这些基础之后，社会就可以发展出一整套司法制度，让成员知所行止，令社会的矛盾不会恶化。 英国人为香港留下的，可不只是一套可以依据的律例，而是一整套法治的理念与司法程序。香港的回归能进行得相对平稳，与特区政府基本上原封不动地承继了原有的司法系统有莫大的关系。 回归后，虽有人危言耸听，说香港的法治已死，但世人仍公认香港的法治达国际一流水准，而港人亦可以如常在香港生活与做生意，不觉有失去法治的实质威胁。直到占中运动的出现，香港人才真正感受到失去法治的害处。 占中运动挑战的可不只是个别「恶法」，而是侵犯了整个法治的根基——产权、人权与政府的执法权。 占领区的物业，地契上列明有Right of way，但现在占领者却不容停车场的车辆出入。这分明损害了这些物业的产权。现在政府却无法加以维护；法庭出了禁制令，占中者却一样藐视。这样发展下去，谁敢在香港置业？ 其实，损害产权等同损害人权，因为人权的一项重要内容，就是个人的财产应获保障。此外，人人都应有追求幸福的权利，但现在占领区生意难做，怎会不妨碍别人追求幸福？ 占中者把自己的行为说成是公民抗命，但公民抗命只是个人行为在道德上的解释，用来拒绝遵守某些个人不认同的法令还讲得通，但绝不可以借此损害他人的产权与人权。 再者，占中者现时在争取的是宪政改革，本应获社会上绝大多数人赞同才有条件实施，不宜用占领交通要津的方式去逼其他人就范。如果祭起公民抗命的旗帜就可以欲所欲为，只会天下大乱，令法治失去协调与整合作用。 \n" ] } ], "source": [ "# 调用stringclean函数\n", "print stringclean(news[17])" ] }, { "cell_type": "code", "execution_count": 77, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "16541" ] }, "execution_count": 77, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 列表内的for循环\n", "news_clean = [stringclean(n) for n in news]\n", "len(news_clean)" ] }, { "cell_type": "code", "execution_count": 85, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " 前文已介绍过塔尔科特·帕森斯的AGIL理论中的Adaptation(适应)与Goal Attainment(达标)的社会功能\n" ] } ], "source": [ "print news_clean[17][:120]" ] }, { "cell_type": "code", "execution_count": 79, "metadata": { "collapsed": true, "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "# 定义两个函数\n", "\n", "def deletetab(s):\n", " return s.replace('\\t', '')\n", "\n", "\n", "import sys\n", "def flushPrint(s):\n", " sys.stdout.write('\\r')\n", " sys.stdout.write('%s' % s)\n", " sys.stdout.flush() # 清洗掉 " ] }, { "cell_type": "code", "execution_count": 102, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Help on OutStream in module ipykernel.iostream object:\n", "\n", "class OutStream(__builtin__.object)\n", " | A file like object that publishes the stream to a 0MQ PUB socket.\n", " | \n", " | Output is handed off to an IO Thread\n", " | \n", " | Methods defined here:\n", " | \n", " | __init__(self, session, pub_thread, name, pipe=None)\n", " | \n", " | __next__(self)\n", " | \n", " | close(self)\n", " | \n", " | fileno(self)\n", " | \n", " | flush(self)\n", " | trigger actual zmq send\n", " | \n", " | send will happen in the background thread\n", " | \n", " | isatty(self)\n", " | \n", " | next = __next__(self)\n", " | \n", " | read(self, size=-1)\n", " | \n", " | readline(self, size=-1)\n", " | \n", " | set_parent(self, parent)\n", " | \n", " | write(self, string)\n", " | \n", " | writelines(self, sequence)\n", " | \n", " | ----------------------------------------------------------------------\n", " | Data descriptors defined here:\n", " | \n", " | __dict__\n", " | dictionary for instance variables (if defined)\n", " | \n", " | __weakref__\n", " | list of weak references to the object (if defined)\n", " | \n", " | closed\n", " | \n", " | ----------------------------------------------------------------------\n", " | Data and other attributes defined here:\n", " | \n", " | flush_interval = 0.2\n", " | \n", " | topic = None\n", "\n" ] } ], "source": [ "help(sys.stdout)" ] }, { "cell_type": "code", "execution_count": 95, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "data": { "text/plain": [ "'a'" ] }, "execution_count": 95, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 调用deletetab\n", "deletetab('\\ta')" ] }, { "cell_type": "code", "execution_count": 97, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "9" ] } ], "source": [ "# 演示：flushPrint\n", "import time, random\n", "for i in range(10):\n", " time.sleep(random.random())\n", " flushPrint(i) " ] }, { "cell_type": "code", "execution_count": 80, "metadata": { "collapsed": false, "slideshow": { "slide_type": "slide" } }, "outputs": [], "source": [ "from collections import defaultdict\n", "\n", "def readblocks(data):\n", " copy = False\n", " n = 0\n", " block = []\n", " chunk = defaultdict(lambda:[])\n", " for i in data:\n", " try:\n", " if \"~~~~~~~~~~~~~~~~~~~~~~~~~~ #\" in i:\n", " copy = True\n", " elif \"文章编号:\" in i:\n", " id = i.replace('文章编号: ', '')\n", " source = block[0].split('|')[0]\n", " info = block[1]\n", " title = deletetab(block[3]) # \n", " body = [j for j in block[6:] if j != '\\n']\n", " body = ' '.join(body)\n", " body = deletetab(body)\n", " body = '\"' + body + '\"'\n", " line = '\\t'.join([id, source, info, title, body])\n", " chunk[id] = line\n", " block = []\n", " n += 1\n", " if n%10 == 0:\n", " flushPrint(n)\n", " copy = False\n", " elif copy:\n", " block.append(i)\n", " except Exception, e:\n", " print i, e\n", " pass\n", " return chunk" ] }, { "cell_type": "code", "execution_count": 73, "metadata": { "collapsed": false, "slideshow": { "slide_type": "subslide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "~~~~~~~~~~~~~~~~~~~~~~~~~~ #1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n", "\n", "am730 | 2014-10-31 \\par\n", "\n", "A16| NEWS| C观点| By 施永青 \\par \n", "\n", "\n", "\n", "法治有整合社会功能 \\par\n", "\n", "\\par\n", "\n", "~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ \\par\n", "\n" ] } ], "source": [ "# 注意：标题和版面之间存在一个空行！所以title是block的第4个元素。\n", "for i in range(1, 8):\n", " print news[i].decode('gb18030')[:500]" ] }, { "cell_type": "code", "execution_count": 81, "metadata": { "collapsed": false, "slideshow": { "slide_type": "slide" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "710" ] } ], "source": [ "# 按block清洗新闻报道\n", "news_result = readblocks(news_clean)" ] }, { "cell_type": "code", "execution_count": 82, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "719" ] }, "execution_count": 82, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 新闻的数量\n", "len(news_result)" ] }, { "cell_type": "code", "execution_count": 83, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "data": { "text/plain": [ "['201410230290186 ',\n", " '201410080010030 ',\n", " '201410229900052 ',\n", " '201410060340162 ',\n", " '201410115313102 ']" ] }, "execution_count": 83, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 查看字典的keys\n", "news_result.keys()[:5]" ] }, { "cell_type": "code", "execution_count": 84, "metadata": { "collapsed": false }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "201410095304453 \t大公报 \tB01| 大公经济 \t马鞍山地皮招标估值16.7亿 \t\" 地皮密密推，马鞍山沙路地皮安排10月24日招标，12月5日截标，市场估值约15.5亿至16.7亿元，每呎4000至4300元。该地皮位于沙田市地段第601号，地盘面积约25.3万方呎，指定作私人住宅用途，可建总楼面38.75方呎。 中原测量师行执行董事张竞达表示，马鞍山耀沙街地皮属临海地段，尽揽海景，发展潜力优厚，将来可发展成豪宅出售，但与港铁站距离略远，此类型海景住宅供应于新界区属罕有，加上近日邻近之迎海．星湾御录理想销售成绩，将来亦会为区内提供商场等配套，预计发展商入标反应佳。惟近日新界区地皮均以较贴近市场下限价钱批出，加上市场对后市未感太乐观，加上近日「占中」事件持续等因素影响，预计多数发展商出价会相对保守，预期楼面呎价约4000元，总值约15.5亿元。美联测量师行董事林子彬表示，地皮虽则与马铁站距离较远，但料部分单位可坐拥海景，若以楼面呎价4300元计算，估值约16.7亿元。 \"\n" ] } ], "source": [ "# 查看字典的values\n", "print news_result.values()[10]" ] }, { "cell_type": "code", "execution_count": 39, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [], "source": [ "# 保存数据：将数据写入硬盘\n", "with open('/Users/chengjun/github/cjc2016/data/zz-hk-2014-9-clean.txt','a') as p:\n", " for record in news_result.values():\n", " p.write(record+\"\\n\")" ] }, { "cell_type": "code", "execution_count": 58, "metadata": { "collapsed": false, "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/html": [ "
\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
01234
0201409015325665南华早报EDT13| EDTPain of stasis没有文字档。
1201409055305251大公报A20| 专版坚决支持和拥护 全国人大常委会对香港政改的决定一、本会坚决支持全国人大常委会关于香港特别行政区行政长官普选问题和201...
2201409165304250大公报B02| 经济．航运粤投1.8亿入股中超【大公报记者毛丽娟深圳十五日电】中国水业（01129）与粤海投资（002...
3201409300050105文汇报B04| 地产新闻提早开标 信置高价夺粉岭地 中标价7.3亿 呎价区内新高香港文汇报讯 (记者 颜伦乐) 地政总署昨日突然提早开标，较平日三个工作...
4201409290320076东方日报B12| 产经星湾御累收逾2200票多家发展商表示，新盘部署暂未受到「占中」影响，恒地（00012）马鞍山迎...
5201409300010113香港商报A10| 投资分析| 股海追踪| By 蔺常念炒壳股秘诀九月份环球利率上升，内地经济数据转弱，加上9月28日开始占中，令港股寻底...
6201409160040016明报A22| 教育| By 刘锦辉时事通识教材文：通识科教师刘锦辉漫画创作：梁浩铨 时事焦点：黑布游行涉及单元：今日香...
7201409305305767am730A62| 娱乐王敏德担心香港会暴乱王敏德(Michael)与女儿王曼喜，昨日坐直升机现身东莞出席高尔夫球赛...
8201409264480062英文虎报P08| Top News| By Imogene WongOccupy fears trim benchmarkHong Kong shares skidded to their l...
9201409135304499大公报A08| 要闻谭耀宗鼓励港人发声撑普选【大公报讯】记者张媞报道：全国政协委员、民建联主席谭耀宗昨日在「闽港合作...
\n", "
" ], "text/plain": [ " 0 1 2 \\\n", "0 201409015325665 南华早报 EDT13| EDT \n", "1 201409055305251 大公报 A20| 专版 \n", "2 201409165304250 大公报 B02| 经济．航运 \n", "3 201409300050105 文汇报 B04| 地产新闻 \n", "4 201409290320076 东方日报 B12| 产经 \n", "5 201409300010113 香港商报 A10| 投资分析| 股海追踪| By 蔺常念 \n", "6 201409160040016 明报 A22| 教育| By 刘锦辉 \n", "7 201409305305767 am730 A62| 娱乐 \n", "8 201409264480062 英文虎报 P08| Top News| By Imogene Wong \n", "9 201409135304499 大公报 A08| 要闻 \n", "\n", " 3 \\\n", "0 Pain of stasis \n", "1 坚决支持和拥护 全国人大常委会对香港政改的决定 \n", "2 粤投1.8亿入股中超 \n", "3 提早开标 信置高价夺粉岭地 中标价7.3亿 呎价区内新高 \n", "4 星湾御累收逾2200票 \n", "5 炒壳股秘诀 \n", "6 时事通识教材 \n", "7 王敏德担心香港会暴乱 \n", "8 Occupy fears trim benchmark \n", "9 谭耀宗鼓励港人发声撑普选 \n", "\n", " 4 \n", "0 没有文字档。 \n", "1 一、本会坚决支持全国人大常委会关于香港特别行政区行政长官普选问题和201... \n", "2 【大公报记者毛丽娟深圳十五日电】中国水业（01129）与粤海投资（002... \n", "3 香港文汇报讯 (记者 颜伦乐) 地政总署昨日突然提早开标，较平日三个工作... \n", "4 多家发展商表示，新盘部署暂未受到「占中」影响，恒地（00012）马鞍山迎... \n", "5 九月份环球利率上升，内地经济数据转弱，加上9月28日开始占中，令港股寻底... \n", "6 文：通识科教师刘锦辉漫画创作：梁浩铨 时事焦点：黑布游行涉及单元：今日香... \n", "7 王敏德(Michael)与女儿王曼喜，昨日坐直升机现身东莞出席高尔夫球赛... \n", "8 Hong Kong shares skidded to their l... \n", "9 【大公报讯】记者张媞报道：全国政协委员、民建联主席谭耀宗昨日在「闽港合作... " ] }, "execution_count": 58, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 使用pandas读取数据，并查看。\n", "import pandas as pd\n", "file_path = '/Users/chengjun/github/cjc2016/data/zz-hk-2014-9-clean.txt'\n", "df = pd.read_csv(file_path, sep = \"\\t\", header=None)\n", "df[:10]" ] }, { "cell_type": "code", "execution_count": 59, "metadata": { "collapsed": false, "slideshow": { "slide_type": "slide" } }, "outputs": [ { "data": { "text/plain": [ "['zz-hk-2014-10.rtf', 'zz-hk-2014-9.rtf']" ] }, "execution_count": 59, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# 使用os改变默认的工作路径\n", "import os\n", "os.chdir('/Users/chengjun/github/cjc2016/data/occupycentral/')\n", "# 使用glob读取某一类文件的所有名称\n", "import glob\n", "filenames = glob.glob('*.rtf')\n", "filenames" ] }, { "cell_type": "code", "execution_count": 60, "metadata": { "collapsed": false, "slideshow": { "slide_type": "fragment" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "zz-hk-2014-10.rtf\n", "710zz-hk-2014-9.rtf\n", "410" ] } ], "source": [ "for i in filenames:\n", " print i\n", " with open(i) as f:\n", " news = f.readlines()\n", " news = [stringclean(n) for n in news]\n", " news_result = readblocks(news)\n", " with open('/Users/chengjun/github/cjc2016/data/zz-hk-all-clean2.txt','a') as p:\n", " for record in news_result.values():\n", " p.write(record+\"\\n\")" ] }, { "cell_type": "markdown", "metadata": { "slideshow": { "slide_type": "slide" } }, "source": [ "# This is the End.\n", "> Thank you for your attention." ] } ], "metadata": { "celltoolbar": "Slideshow", "kernelspec": { "display_name": "Python 2", "language": "python", "name": "python2" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 2 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython2", "version": "2.7.11" } }, "nbformat": 4, "nbformat_minor": 0 }