{ "cells": [ { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [], "source": [ "%reload_ext autoreload\n", "%autoreload 2\n", "%matplotlib inline\n", "import os\n", "import numpy as np\n", "import pandas as pd\n", "pd.set_option('display.max_colwidth', -1)" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import ktrain" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## STEP 1: Get Raw Document Data" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [], "source": [ "# 20newsgroups\n", "from sklearn.datasets import fetch_20newsgroups\n", "remove = ('headers', 'footers', 'quotes')\n", "newsgroups_train = fetch_20newsgroups(subset='train', remove=remove)\n", "newsgroups_test = fetch_20newsgroups(subset='test', remove=remove)\n", "texts = newsgroups_train.data + newsgroups_test.data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## STEP 2: Train an LDA Topic Model to Discover Topics\n", "\n", "The `get_topic_model` function learns a [topic model](https://en.wikipedia.org/wiki/Topic_model) using [Latent Dirichlet Allocation (LDA)](https://en.wikipedia.org/wiki/Latent_Dirichlet_allocation)." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "n_topics automatically set to 97\n", "preprocessing texts...\n", "fitting model...\n", "iteration: 1 of max_iter: 5\n", "iteration: 2 of max_iter: 5\n", "iteration: 3 of max_iter: 5\n", "iteration: 4 of max_iter: 5\n", "iteration: 5 of max_iter: 5\n", "done.\n", "CPU times: user 16min 43s, sys: 43min 27s, total: 1h 11s\n", "Wall time: 2min\n" ] } ], "source": [ "%%time\n", "tm = ktrain.text.get_topic_model(texts, n_features=10000)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Compute the topic-probability distribution for each document. We need this to compute document similarity." ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "done.\n", "CPU times: user 1min 33s, sys: 3min 37s, total: 5min 10s\n", "Wall time: 13.2 s\n" ] } ], "source": [ "%%time\n", "tm.build(texts, threshold=0.25)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ " We can examine the discovered topics using `print_topics`, `get_topics`, or `topics`. Here, we will use `print_topics`:" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "topic 0 | tape adam tim case moved bag quote mass marked zionism\n", "topic 1 | image jpeg images format programs tiff files jfif save lossless\n", "topic 2 | alternative movie film static cycles films philips dynamic hou phi\n", "topic 3 | hell humans poster frank reality kent gerard gant eternal bell\n", "topic 4 | air phd chz kit cbc ups w-s rus w47 mot\n", "topic 5 | dog math great figure poster couldn don trying rushdie fatwa\n", "topic 6 | collaboration nazi fact end expression germany philly world certified moore\n", "topic 7 | gif points scale postscript mirror plane rendering algorithm polygon rayshade\n", "topic 8 | fonts font shell converted iii characters slight composite breaks compress\n", "topic 9 | power station supply options option led light tank plastic wall\n", "topic 10 | transmission rider bmw driver automatic shift gear japanese stick highway\n", "topic 11 | tyre ezekiel ruler hernia appeared appointed supreme man land power\n", "topic 12 | space nasa earth data launch surface solar moon mission planet\n", "topic 13 | israel jews jewish israeli arab peace war arabs palestinian kuwait\n", "topic 14 | olvwm xremote animals kinds roughing toolkit close corp glenn imakefile\n", "topic 15 | medical health disease cancer patients drug treatment drugs aids study\n", "topic 16 | biden chip gear like information number automatic mode insurance know\n", "topic 17 | graphics zip amiga shareware formats ftp gif program sgi convert\n", "topic 18 | brilliant mail did god coming christianity people got ideas reading\n", "topic 19 | black red white blue green cross wires lines helmet mask\n", "topic 20 | car engine cars miles clutch new ford rear slip road\n", "topic 21 | list mailing service model small large lists radar available major\n", "topic 22 | key encryption chip keys clipper phone security use government privacy\n", "topic 23 | talking pit nyr stl phi edm mtl wsh hfd cgy\n", "topic 24 | signal input switch connected circuit audio noise output control voltage\n", "topic 25 | stuff deleted die posting beware fantastic motives authentic reluctant hope\n", "topic 26 | adams douglas dc-x garrett ingres tin sdio incremental mcdonnell guide\n", "topic 27 | men homosexual homosexuality women gay sexual homosexuals male kinsey pop\n", "topic 28 | usual leo rs-232 martian reading cooperative unmanned somalia decompress visited\n", "topic 29 | edu university information send new computer research mail internet address\n", "topic 30 | reserve naval marine ret commission one-way irgun prior closure facilities\n", "topic 31 | state intelligence militia units army zone georgia sam croats belongs\n", "topic 32 | says article pain known warning doctor stone bug kidney response\n", "topic 33 | faq rsa ripem lights yes patent nist management wax cipher\n", "topic 34 | wolverine comics hulk appearance special liefeld sabretooth incredible hobgoblin x-force\n", "topic 35 | software ram worth cycles controller available make dram dynamic situation\n", "topic 36 | religion people religious catalog bobby used driven involved long like\n", "topic 37 | intel sites experiment ftp does know family good like mrs\n", "topic 38 | armenian people army russian turkish genocide armenians ottoman turks jews\n", "topic 39 | theft geo available face couldn cover sony people number shop\n", "topic 40 | christianity did exists mail matter mind tool status god reading\n", "topic 41 | propane probe earth orbit orbiter titan cassini space atmosphere gravity\n", "topic 42 | people government right think rights law make public fbi don\n", "topic 43 | god people does say believe bible true think evidence religion\n", "topic 44 | mov phone south key war supply push left just registered\n", "topic 45 | period goal pts play chicago pittsburgh buffalo shots new blues\n", "topic 46 | game team games year hockey season players player baseball league\n", "topic 47 | speed dod student technician just hits right note giant light\n", "topic 48 | sex marriage relationship family married couple depression pregnancy childhood trademark\n", "topic 49 | protects rejecting com4 couple decides taking connect unc nearest richer\n", "topic 50 | president states united american national press april washington america white\n", "topic 51 | card memory windows board ram bus drivers driver cpu problem\n", "topic 52 | window application manager display button xterm path widget event resources\n", "topic 53 | cable win van det bos tor cal nyi chi buf\n", "topic 54 | americans baltimore rochester cape springfield moncton providence utica binghamton adirondack\n", "topic 55 | color monitor screen mouse video colors resolution vga colour monitors\n", "topic 56 | option power ssf flights capability module redesign missions station options\n", "topic 57 | body father son vitamin diet day cells cell form literature\n", "topic 58 | max g9v b8f a86 bhj giz bxn biz qax b4q\n", "topic 59 | bit fast chip ibm faster mode chips scsi-2 speeds quadra\n", "topic 60 | book books law adl islam islamic iran media bullock muslims\n", "topic 61 | armenian russian turkish ottoman people army armenians genocide war turks\n", "topic 62 | oscillator partition tune nun umumiye nezareti mecmuasi muharrerat-i evrak version\n", "topic 63 | tongues seat est didn raise copied lazy schemes adapter leap\n", "topic 64 | com object jim app function motorola heterosexual objects pointers encountered\n", "topic 65 | effective boy projects grow jason ain dump keyboards vastly grants\n", "topic 66 | armenian people russian armenians turks ottoman army turkish genocide muslim\n", "topic 67 | mac apple pin ground wire quicktime macs pins connector simms\n", "topic 68 | bastard turning likes hooks notions turks cited proud pointers chuck\n", "topic 69 | bought dealer cost channel replaced face sony stereo warranty tube\n", "topic 70 | myers food reaction msg writes loop eat dee effects taste\n", "topic 71 | lander contradiction reconcile apparent somebody supplement essential needs produce insulin\n", "topic 72 | re-boost systems virginia voice unix input ken easily summary developing\n", "topic 73 | block tests suck shadow dte screws macedonia sunlight fin message\n", "topic 74 | jesus church christ god lord holy spirit mary shall heaven\n", "topic 75 | gun number year guns rate insurance police years new firearms\n", "topic 76 | rule automatically characteristic wider thumb recommendation inline mr2 halfway width\n", "topic 77 | drive disk hard scsi drives controller floppy ide master transfer\n", "topic 78 | stephanopoulos water gas oil heat energy hot temperature cold nuclear\n", "topic 79 | like know does use don just good thanks need want\n", "topic 80 | starters mlb mov higher signing left accessible argument viola teams\n", "topic 81 | entry rules info define entries year int printf include contest\n", "topic 82 | price new sale offer sell condition shipping interested asking prices\n", "topic 83 | issue germany title magazine german cover race generation origin nazi\n", "topic 84 | armenian armenians people turkish war said killed children russian turkey\n", "topic 85 | dos windows software comp library os/2 version microsoft applications code\n", "topic 86 | probe space launch titan earth cassini orbiter orbit atmosphere mission\n", "topic 87 | housed throws fills daylight occurring activities adjacent presenting punish occuring\n", "topic 88 | statement folk raids thor disarmed anatolia polygon inria arrive smehlik\n", "topic 89 | sound steve pro convert ati ultra fahrenheit orchid hercules blaster\n", "topic 90 | joke tricky wearing golden trickle seen geneva csh course caesar\n", "topic 91 | moral objective values morality child defined bank definition wrong different\n", "topic 92 | files file edu ftp available version server data use sun\n", "topic 93 | catalog tons seal ordering kawasaki tools fax free ultraviolet packages\n", "topic 94 | file program error output use section line code command problem\n", "topic 95 | power ssf module capability option flights redesign missions human station\n", "topic 96 | just don think know like time did going didn people\n" ] } ], "source": [ "tm.print_topics()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## STEP 3: Select Topics of Interest" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's combine some technology-related documents into a set of positive examples of technology-focused posts. We can use these documents as seeds to find new documents about technology. To measure semantic similarity among documents, we will represent each document by its topic probability distribution that we computed above. We will also compile the document IDs for each document associated with these selected topics." ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "tech_topics = [51, 85, 94, 22]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Scoring Documents by Similarity\n", "\n", "We will score new documents based on how similar they are to the sample selected above using a [One-Class classifier](https://en.wikipedia.org/wiki/One-class_classification). " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [], "source": [ "tm.train_scorer(topic_ids=tech_topics)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can now invoke the `scorer` method to measure the degree to which new documents are similar to our technology-related topics. Let's use `scorer` to measure the similarity of the remaining documents in the corpus. Note that, although we are applying the scorer to documents within the set corpus used to train the topic model, this is not required. Our `scorer` can be applied to any arbitrary set of documents." ] }, { "cell_type": "code", "execution_count": 11, "metadata": {}, "outputs": [], "source": [ "other_topics = [i for i in range(tm.n_topics) if i not in tech_topics]\n", "other_texts = [d['text'] for d in tm.get_docs(topic_ids=other_topics)]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's score these documents and place into a Pandas dataframe." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [], "source": [ "# score documents based on similarity\n", "other_scores = tm.score(other_texts)" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Top Inliers (or Most Similar to Our Technology-Related Topics\n", "\t\tNumber of Predicted Inliners: 377\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PredictionScoreText
010.212587I'm looking for recommendations for a laser printer. It will\\nbe used mostly for text by a single user. It doesn't need to\\nbe a postscript printer. Any advice would be appreciated.\\n
110.211690I get the picture, I just find it humorous that Running Windows 3.1 apps ( 3.0 for 2.0 ) \\nis what makes os/2 more credible...
210.211690Two-part question:\\n\\n1) What is Windows NT - a 'real' windows OS?\\n\\n2) This past weekend, a local 'hacker' radio show metioned a new product\\n from Microsoft called 'Chicago' if I recall. Anyone know what this is?\\n\\nThat is it -\\n\\nThanks a heap.\\n\\n- Alan\\n
310.211690Is there any one know:\\n\\nWhat is the FTP tool for Windows and where to get the tool ?\\n\\nThanks for any help !!
410.205488Could someone point me toward a source (FTP/BBS/whatever) for development\\ntools for the 8051 microprocessor. I specifically am looking for a Macintosh\\ncross-assembler/disassembler. Also, is there a mailing-list dedicated to\\ndiscussing the 8051? Thanks.\\n
\n", "
" ], "text/plain": [ " Prediction Score \\\n", "0 1 0.212587 \n", "1 1 0.211690 \n", "2 1 0.211690 \n", "3 1 0.211690 \n", "4 1 0.205488 \n", "\n", " Text \n", "0 I'm looking for recommendations for a laser printer. It will\\nbe used mostly for text by a single user. It doesn't need to\\nbe a postscript printer. Any advice would be appreciated.\\n \n", "1 I get the picture, I just find it humorous that Running Windows 3.1 apps ( 3.0 for 2.0 ) \\nis what makes os/2 more credible... \n", "2 Two-part question:\\n\\n1) What is Windows NT - a 'real' windows OS?\\n\\n2) This past weekend, a local 'hacker' radio show metioned a new product\\n from Microsoft called 'Chicago' if I recall. Anyone know what this is?\\n\\nThat is it -\\n\\nThanks a heap.\\n\\n- Alan\\n \n", "3 Is there any one know:\\n\\nWhat is the FTP tool for Windows and where to get the tool ?\\n\\nThanks for any help !! \n", "4 Could someone point me toward a source (FTP/BBS/whatever) for development\\ntools for the 8051 microprocessor. I specifically am looking for a Macintosh\\ncross-assembler/disassembler. Also, is there a mailing-list dedicated to\\ndiscussing the 8051? Thanks.\\n " ] }, "execution_count": 13, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# display results in Pandas dataframe\n", "other_preds = [int(score > 0) for score in other_scores]\n", "data = sorted(list(zip(other_preds, other_scores, other_texts)), key=lambda item:item[1], reverse=True)\n", "print('Top Inliers (or Most Similar to Our Technology-Related Topics')\n", "print('\\t\\tNumber of Predicted Inliners: %s' % sum(other_preds))\n", "df = pd.DataFrame(data, columns=['Prediction', 'Score', 'Text'])\n", "df.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, we see we've found additional technology-related posts in the dataset.\n", "\n", "Our `scorer` assigns a score to each document, where higher scores indicate a higher degree of similarity to technology-related seed docments. The `scorer` implements a decision function to make binary decisions on similarity such that documents with positive scores are deemed as similar and negative scores are deemed dissimilar. We've used this to create a prediction of 1 for similar and 0 for dissimilar. This identifies 377 documents as similar. The `scorer`, however, employs a One-Class classifier, which tends to be more strict. That is, there are likely documents with negative scores close to zero that are also similar. Let's look at these." ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PredictionScoreText
3770-0.000711I was at avalon today and found texture maps in some \"tex\" and \"txc\"\\nformat, something I've never encountered before. These are obviously\\nnot tex or LaTeX files.\\n\\nIF you have a clue how I can convert these to something\\nreasonable, please let me know.
3780-0.002478I need to be able to cause a beep, but without using any interrupt\\nroutines, as I cannot use the BIOS. I believe that the PIC might have\\nsomething to do with it, but I'm having troubles deciphering the\\ninformation I have on it to figure out how to program it!\\n\\n\\tI'm programming all of this in Turbo C, if that makes any\\ndiference at all...\\n\\n\\tPlease can anyone help me??!\\n\\nThanks,
3790-0.003216\\nThe only things you'll be able to salvage from the junior are the floppy drives\\nand monitor. The floppies are 360k, and the monitor is CGA, but you will need\\nan adaptor cable to use it. The junior does not use standard cards. Unless \\nyou're really strapped for cash, you should just junk the thing and buy new \\nstuff.\\n\\nDan\\n
3800-0.003326\\nMacintosh II cx with 40 MB HD, 8 MB RAM and 19\" monochrome\\nmonitor (Ikegami) is for sale.\\nAsking $3,000, no reasonable (best) offer will be rejected.\\nContact Konrad at (416) 365-0564m Mon-Frii 9-5.\\n
3810-0.003883I edited a few newsgroup from that line (don't like to crosspost THAT\\nmuch). I can't compare the two, but I recently got an HP DeskJet 500.\\n\\nI'm very pleased with the output (remember that I'm used to imagens,\\nlaser and postscript printers at school -- looks very good. You have\\nto be careful to let it dry before touching it, as it will smudge.\\n\\nThe deskjet is SLOW. This is in comparison to the other printers I\\nmentioned. I have no idea how the bubblejet compares.\\n\\nThe interface between Win3.1 and the printer is just dandy, I've not\\nhad any problems with it.\\n\\nHope that helps some.\\n\\n--Cindy\\n\\n--\\nCindy Tittle Moore
\n", "
" ], "text/plain": [ " Prediction Score \\\n", "377 0 -0.000711 \n", "378 0 -0.002478 \n", "379 0 -0.003216 \n", "380 0 -0.003326 \n", "381 0 -0.003883 \n", "\n", " Text \n", "377 I was at avalon today and found texture maps in some \"tex\" and \"txc\"\\nformat, something I've never encountered before. These are obviously\\nnot tex or LaTeX files.\\n\\nIF you have a clue how I can convert these to something\\nreasonable, please let me know. \n", "378 I need to be able to cause a beep, but without using any interrupt\\nroutines, as I cannot use the BIOS. I believe that the PIC might have\\nsomething to do with it, but I'm having troubles deciphering the\\ninformation I have on it to figure out how to program it!\\n\\n\\tI'm programming all of this in Turbo C, if that makes any\\ndiference at all...\\n\\n\\tPlease can anyone help me??!\\n\\nThanks, \n", "379 \\nThe only things you'll be able to salvage from the junior are the floppy drives\\nand monitor. The floppies are 360k, and the monitor is CGA, but you will need\\nan adaptor cable to use it. The junior does not use standard cards. Unless \\nyou're really strapped for cash, you should just junk the thing and buy new \\nstuff.\\n\\nDan\\n \n", "380 \\nMacintosh II cx with 40 MB HD, 8 MB RAM and 19\" monochrome\\nmonitor (Ikegami) is for sale.\\nAsking $3,000, no reasonable (best) offer will be rejected.\\nContact Konrad at (416) 365-0564m Mon-Frii 9-5.\\n \n", "381 I edited a few newsgroup from that line (don't like to crosspost THAT\\nmuch). I can't compare the two, but I recently got an HP DeskJet 500.\\n\\nI'm very pleased with the output (remember that I'm used to imagens,\\nlaser and postscript printers at school -- looks very good. You have\\nto be careful to let it dry before touching it, as it will smudge.\\n\\nThe deskjet is SLOW. This is in comparison to the other printers I\\nmentioned. I have no idea how the bubblejet compares.\\n\\nThe interface between Win3.1 and the printer is just dandy, I've not\\nhad any problems with it.\\n\\nHope that helps some.\\n\\n--Cindy\\n\\n--\\nCindy Tittle Moore " ] }, "execution_count": 14, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[df.Score <=0].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, these documents are also similar and related to technology (albeit slightly different aspects of technology than that of our seed set of documents). Such negatively-scored documents are useful for identifying so-called informative examples. Since documents are sorted by score (descending order), we can start at the beginning of the dataframe containing negatively-scored documents and add documents to the positive class until we start seeing negative documents that are **not** related to technology. These informative negative examples can, then, be added to a negative class for training a traditional binary classsifier. This process is referred to as [active learning](https://en.wikipedia.org/wiki/Active_learning_(machine_learning)).\n", "\n", "For instance, in this example, scores below -0.5 start to become **unrelated** to the themes covered by our technical topics." ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PredictionScoreText
22170-0.510036\\nDon't forget Chemical Abstracts Service (which is pretty much the international\\nclearinghouse for all chemical information), whose former director (Ronald\\nWigington) and head of R&D (Nick Farmer) were openly former NSA employees.
22180-0.510492From article <1993Apr21.013846.1374@cx5.com>, by tlc@cx5.com:\\n\\nAccording to my ColoRIX manual .SCF files are 640x480x256\\n\\n\\nYou may try VPIC, I think it handles the 256 color RIX files OK..\\n
22190-0.510556What about disks? Won't it erase them if you're carrying them in the bag?
22200-0.510582was\\nYuppies\\nstarted\\nYep, that's when I noticed it too. I stopped replacing the hood badge \\nafter the second or third one (at $12.00 each).\\n\\n2002 drivers used to flash their headlight at each other in greeting. Try \\nflashing your headlights at a 318i driver and see what kind of look you \\nget. They usually check their radar detector...they think you're alerting \\nthem to a cop.
22210-0.510692refrettably you are mistaken. alt.drugs was used to recruit people for the\\nworldwide pot religion. I, however hve no problem being in both of them\\n\\n
\n", "
" ], "text/plain": [ " Prediction Score \\\n", "2217 0 -0.510036 \n", "2218 0 -0.510492 \n", "2219 0 -0.510556 \n", "2220 0 -0.510582 \n", "2221 0 -0.510692 \n", "\n", " Text \n", "2217 \\nDon't forget Chemical Abstracts Service (which is pretty much the international\\nclearinghouse for all chemical information), whose former director (Ronald\\nWigington) and head of R&D (Nick Farmer) were openly former NSA employees. \n", "2218 From article <1993Apr21.013846.1374@cx5.com>, by tlc@cx5.com:\\n\\nAccording to my ColoRIX manual .SCF files are 640x480x256\\n\\n\\nYou may try VPIC, I think it handles the 256 color RIX files OK..\\n \n", "2219 What about disks? Won't it erase them if you're carrying them in the bag? \n", "2220 was\\nYuppies\\nstarted\\nYep, that's when I noticed it too. I stopped replacing the hood badge \\nafter the second or third one (at $12.00 each).\\n\\n2002 drivers used to flash their headlight at each other in greeting. Try \\nflashing your headlights at a 318i driver and see what kind of look you \\nget. They usually check their radar detector...they think you're alerting \\nthem to a cop. \n", "2221 refrettably you are mistaken. alt.drugs was used to recruit people for the\\nworldwide pot religion. I, however hve no problem being in both of them\\n\\n " ] }, "execution_count": 15, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[(df.Score<-0.51)].head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Using Keyword Searches to Construct Seed Sets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's construct a set of seed documents from a keyword search instead of by LDA-discovered topics. Let's search all the documents for the word 'Christ':" ] }, { "cell_type": "code", "execution_count": 16, "metadata": {}, "outputs": [ { "data": { "text/html": [ "done." ], "text/plain": [ "" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "results = tm.search('Christ', case_sensitive=False)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "There are 313 of them." ] }, { "cell_type": "code", "execution_count": 17, "metadata": {}, "outputs": [ { "data": { "text/plain": [ "313" ] }, "execution_count": 17, "metadata": {}, "output_type": "execute_result" } ], "source": [ "len(results)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Most documents in this set are about Christianity, as expected:" ] }, { "cell_type": "code", "execution_count": 18, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\n", "Yep, that's pretty much it. I'm not a Jew but I understand that this is the\n", "Jewish way of thinking. However, the Jews believe that the Covenant between\n", "YHWH and the Patriarchs (Abraham and Moses, in this case) establishes a Moral\n", "Code to follow for mankind. Even the Jews could not decide where the boundaries\n", "fall, though.\n", "\n", "As I understand it, the Sadducees believed that the Torah was all that was\n", "required, whereas the Pharisees (the ancestors of modern Judaism) believed that\n", "the Torah was available for interpretation to lead to an understanding of\n", "the required Morality in all its nuances (->Talmud).\n", "\n", "The essence of all of this is that Biblical Morality is an interface between\n", "Man and YHWH (for a Jew or Christian) and does not necessarily indicate\n", "anything about YHWH outside of that relationship (although one can speculate).\n", "\n", "\n", "The trouble with all of this is that we don't really know what the \"created\n", "in His image\" means. I've heard a number of different opinions on this and\n", "have still not come to any conclusion. This rather upsets the Apple Cart if\n", "one wants to base a Life Script on this shaky foundation (to mix metaphors\n", "unashamedly!) As to living by Christ's example, we know very little about\n", "Jesus as a person. We only have his recorded utterances in a set of narratives\n", "by his followers, and some very small references from comtemporary historians.\n", "Revelation aside, one can only \"know\" Christ second-hand or worse.\n", "\n", "This is not an attempt to debunk Christianity (although it may seem that way\n", "initially), the point I`m trying to make is that we only really have the Bible\n", "to interpret, and that interpretation is by humanity. I guess this is where\n", "Faith or Relevation comes in with all its inherent subjectiveness.\n", "\n", "\n", "No. There may be an absolute moral code. There are undoubtably multiple\n", "moral codes. The multiple moral codes may be founded in the absolute moral\n", "code. As an example, a parent may tell a child never to swear, and the child\n", "may assume that the parent never swears simply because the parent has told\n", "the child that it is \"wrong\". Now, the parent may swear like a trooper in\n", "the pub or bar (where there are no children). The \"wrongness\" here is if\n", "the child disobeys the parent. The parent may feel that it is \"inappropriate\"\n", "to swear in front of children but may be quite happy to swear in front of\n", "animals. The analogy does not quite hold water because the child knows that\n", "he is of the same type as the parent (and may be a parent later in life) but\n", "you get the gist of it? Incidentally, the young child considers the directive\n", "as absolute until he gets older (see Piaget) and learns a morality of his own.\n", "\n", "David.\n", "\n", "---\n", "On religion:\n" ] } ], "source": [ "print(results[0]['text'])" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Let's construct a positive class from these 313 documents and use them to find other religious documents:" ] }, { "cell_type": "code", "execution_count": 19, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Top Inliers (or Most Similar to Our Technology-Related Topics\n", "\t\tNumber of Predicted Inliners: 4759\n" ] }, { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PredictionScoreText
010.418117\\nAnd does it not say in scripture that no man knows the hour of His coming, not\\neven the angels in Heaven but only the Father Himself? DK was trying to play\\nGod by breaking the seals himself. DK killed himself and as many of his\\nfollowers as he could. BTW, God did save the children. They are in Heaven,\\na far better place. How do I know? By faith.\\n\\nGod be with you,
110.409673\\n \\nFirst of all, the original poster misquoted. The reference is from 2 Tim 3:16.\\nThe author was Paul, and his revelations were anything but \"(at best) \\nsecond-hand\".\\n\\n\\t\"And is came about that as [Saul] journeyed, he was approaching\\n\\t Damascus, and suddenly a light from heaven flashed around him; and\\n\\t he fell to the ground, and heard a voice saying to him, \"Saul, Saul,\\n\\t why are you persecuting Me?\" And he said, \"Who art Thou, Lord?\" And\\n\\t He said, \"I am Jesus whom you are persecuting, . . .\"\\n\\t\\t(Acts 9:3-5, NAS)\\n\\nPaul received revelation directly from the risen Jesus! (Pretty cool, eh?) He\\nbecame closely involved with the early church, the leaders of which were \\nfollowers of Jesus throughout his ministry on earth.\\n\\n\\nI agree. I don't believe anyone but the Spirit would be able to convince you \\nthe Spirit exists. Please don't complain about this being circular. I know\\nit is, but really, can anything of the natural world explain the supernatural?\\n(This is why revelation is necessary to the authors of the Bible.)\\n\\n\\nThe Spirit is part of God. How much closer to the source can you get?\\nThe Greek in 2 Timothy which is sometimes translated as \"inspired by God\", \\nliterally means \"God-breathed\". In other words, God spoke the actual words \\ninto the scriptures. Many theologians and Bible scholars (Dr. James Boice is \\none that I can remember off-hand) get quite annoyed by the dryness and \\nincompleteness of \"inspired by God\".\\n\\n\\nThat's what the verse taken from 2 Timothy was all about. The continuity of a \\nbook written over a span of 1500 years by more than 40 authors from all walks \\nof life is a testimony to the single authorship of God.\\n\\n\\n\\nWhat source to you claim to have discovered which has information of superior\\nhistoricity to the Bible? Certainly not Josephus' writings, or the writings \\nof the Gnostics which were third century, at the earliest.\\n\\n\\nJesus was fully God as well. That's why I'd assert that he is wise.\\n\\n\\nPlease rethink this last paragraph. If there is no God, which seems to be your\\ncurrent belief, then Jesus was either a liar or a complete nut because not\\nonly did he assert that God exists, but he claimed to be God himself! (regards\\nto C.S. Lewis) How then could you have the least bit of respect for Jesus?\\n\\tIn conclusion, be careful about logically unfounded hypotheses based\\non gut feelings about the text and other scholars' unsubstantiated claims. \\nThe Bible pleads that we take it in its entirety or throw the whole book out.\\n\\tAbout your reading of the Bible, not only does the Spirit inspire the\\nwriters, but he guides the reader as well. We cannot understand it in the \\nleast without the Spirit's guidance:\\n\\n\\t\"For to us God revealed them through the Spirit; for the Spirit \\n\\tsearches all things, even the depths of God.\" (1 Cor 2:10, NAS)\\n \\nPeace and may God guide us in wisdom.\\n\\n+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=\\nCarter C. Page | Of happiness the crown and chiefest part is wisdom,\\nA Carpenter's Apprentice | and to hold God in awe. This is the law that,\\ncpage@seas.upenn.edu | seeing the stricken heart of pride brought down,\\n | we learn when we are old. -Adapted from Sophocles\\n+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=+-=-+-=+-=-+=-+-=-+-=-+=-+-=
210.405350I differ with our moderator on this. I thought the whole idea of God coming\\ndown to earth to live as one of us \"subject to sin and death\" (as one of\\nthe consecration prayers in the Book of Common Prayer (1979) puts it) was\\nthat Jesus was tempted, but did not succumb. If sin is not part of the\\nbasic definition of humanity, then Jesus \"fully human\" (Nicea) would not\\nbe \"subject to sin\", but then the Resurrection loses some of its meaning,\\nbecause we encounter our humanity most powerfully when we sin. To distinguish\\nbetween \"human\" and \"fallen human\" makes Jesus less like one of us at the\\ntime we need him most.\\n\\n\\nFirst, the Monophysites inherited none of Nestorius's version -- they \\nwere on the opposite end of the spectrum from him. Second, the historical\\nrecord suggests that the positions attributed to Nestorius were not as\\nextreme as his (successful) opponents (who wrote the conventional history)\\nclaimed. Mainly Nestorius opposed the term Theotokos for Mary, arguing\\n(I think correctly) that a human could not be called Mother of God. I mean,\\nin the Athanasian Creed we talk about the Son \"uncreate\" -- surely even \\nArians would concede that Jesus existed long before Mary. Anyway, Nestorius's\\nopponents claimed that by saying Mary was not Theotokos, that he claimed\\nthat she only gave birth to the human nature of Jesus, which would require\\ntwo seperate and distinct natures. The argument fails though, because\\nMary simply gave birth to Jesus, who preexisted her either divinely,\\nif you accept \"Nestorianism\" as commonly defined, or both natures intertwined,\\na la Chalcedon.\\n\\nSecond, I am not sure that \"Nestorianism\" is not a better alternative than\\nthe orthodox view. After all, I find it hard to believe that pre-Incarnation\\nthat Jesus's human nature was in heaven; likewise post-Ascension. I think\\nrather that God came to earth and took our nature upon him. It was a seperate\\nnature, capable of being tempted as in Gethsemane (since I believe the divine\\nnature could never be tempted) but in its moments of weakness the divine nature\\nprevailed.\\n\\nComments on the above warmly appreciated.\\n\\nJason Albert\\n\\n[There may be differences in what we mean by \"subject to sin\". The\\noriginal complaint was from someone who didn't see how we could call\\nJesus fully human, because he didn't sin. I completely agree that\\nJesus was subject to temptation. I simply object to the idea that by\\nnot succumbing, he is thereby not fully human. I believe that you do\\nnot have to sin in order to be human.\\n\\nI again apologize for confusing Nestorianism and monophysitism. I\\nagree with you, and have said elsewhere, that there's reason to think\\nthat not everyone who is associated with heretical positions was in\\nfact heretical. There are scholars who maintain that Nestorius was\\nnot Nestorian. I have to confess that the first time I read some of\\nthe correspondence between Nestorius and his opponents, I thought he\\ngot the better of them.\\n\\nHowever, most scholars do believe that the work that eventually led to\\nChalcedon was an advance, and that Nestorius was at the very least\\n\"rash and dogmatic\" (as the editor of \"The Christological Controversy\"\\nrefers to him) in rejecting all approaches other than his own. As\\nregular Usenet readers know, narrowness can be just as much an\\nimpediment as being wrong. Furthermore, he did say some things that I\\nthink are problematical. He responds to a rather mild letter from\\nCyril with a flame worthy of Usenet. In it he says \"To attribute also\\nto [the Logos], in the name of [the incarnation] the characteristics\\nof the flesh that has been conjoined with him ... is, my brother,\\neither the work of a mind which truly errs in the fashion of the\\nGreeks or that of a mind diseased with the insane heresy of Arius and\\nApollinaris and the others. Those who are thus carried away with the\\nidea of this association are bound, because of it, to make the divine\\nLogos have a part in being fed with milk and participate to some\\ndegree in growh and stand in need of angelic assistance because of his\\nfearfulness ... These things are taken falsely when they are put off\\non the deity and they become the occasion of just condemnation for us\\nwho perpetrate the falsehood.\"\\n\\nIt's all well and good to maintain a proper distinction between\\nhumanity and divinity. But the whole concept of incarnation is based\\non exactly the idea that the divine Logos does in fact have \"to some\\ndegree\" a part in being born, growing up, and dying. Of course it\\nmust be understood that there's a certain indirectness in the Logos'\\nparticipation in these things. But there must be some sort of\\nidentification between the divine and human, or we don't have an\\nincarnation at all. Nestorius seemed to think in black and white\\nterms, and missed the sorts of nuances one needs to deal with this\\narea.\\n\\nYou say \"I find it hard to believe that pre-Incarnation that Jesus's\\nhuman nature was in heaven.\" I don't think that's required by\\northodox doctrine. It's the divine Logos that is eternal.
\n", "
" ], "text/plain": [ " Prediction Score \\\n", "0 1 0.418117 \n", "1 1 0.409673 \n", "2 1 0.405350 \n", "\n", " Text \n", "0 \\nAnd does it not say in scripture that no man knows the hour of His coming, not\\neven the angels in Heaven but only the Father Himself? DK was trying to play\\nGod by breaking the seals himself. DK killed himself and as many of his\\nfollowers as he could. BTW, God did save the children. They are in Heaven,\\na far better place. How do I know? By faith.\\n\\nGod be with you, \n", "1 \\n \\nFirst of all, the original poster misquoted. The reference is from 2 Tim 3:16.\\nThe author was Paul, and his revelations were anything but \"(at best) \\nsecond-hand\".\\n\\n\\t\"And is came about that as [Saul] journeyed, he was approaching\\n\\t Damascus, and suddenly a light from heaven flashed around him; and\\n\\t he fell to the ground, and heard a voice saying to him, \"Saul, Saul,\\n\\t why are you persecuting Me?\" And he said, \"Who art Thou, Lord?\" And\\n\\t He said, \"I am Jesus whom you are persecuting, . . .\"\\n\\t\\t(Acts 9:3-5, NAS)\\n\\nPaul received revelation directly from the risen Jesus! (Pretty cool, eh?) He\\nbecame closely involved with the early church, the leaders of which were \\nfollowers of Jesus throughout his ministry on earth.\\n\\n\\nI agree. I don't believe anyone but the Spirit would be able to convince you \\nthe Spirit exists. Please don't complain about this being circular. I know\\nit is, but really, can anything of the natural world explain the supernatural?\\n(This is why revelation is necessary to the authors of the Bible.)\\n\\n\\nThe Spirit is part of God. How much closer to the source can you get?\\nThe Greek in 2 Timothy which is sometimes translated as \"inspired by God\", \\nliterally means \"God-breathed\". In other words, God spoke the actual words \\ninto the scriptures. Many theologians and Bible scholars (Dr. James Boice is \\none that I can remember off-hand) get quite annoyed by the dryness and \\nincompleteness of \"inspired by God\".\\n\\n\\nThat's what the verse taken from 2 Timothy was all about. The continuity of a \\nbook written over a span of 1500 years by more than 40 authors from all walks \\nof life is a testimony to the single authorship of God.\\n\\n\\n\\nWhat source to you claim to have discovered which has information of superior\\nhistoricity to the Bible? Certainly not Josephus' writings, or the writings \\nof the Gnostics which were third century, at the earliest.\\n\\n\\nJesus was fully God as well. That's why I'd assert that he is wise.\\n\\n\\nPlease rethink this last paragraph. If there is no God, which seems to be your\\ncurrent belief, then Jesus was either a liar or a complete nut because not\\nonly did he assert that God exists, but he claimed to be God himself! (regards\\nto C.S. Lewis) How then could you have the least bit of respect for Jesus?\\n\\tIn conclusion, be careful about logically unfounded hypotheses based\\non gut feelings about the text and other scholars' unsubstantiated claims. \\nThe Bible pleads that we take it in its entirety or throw the whole book out.\\n\\tAbout your reading of the Bible, not only does the Spirit inspire the\\nwriters, but he guides the reader as well. We cannot understand it in the \\nleast without the Spirit's guidance:\\n\\n\\t\"For to us God revealed them through the Spirit; for the Spirit \\n\\tsearches all things, even the depths of God.\" (1 Cor 2:10, NAS)\\n \\nPeace and may God guide us in wisdom.\\n\\n+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=\\nCarter C. Page | Of happiness the crown and chiefest part is wisdom,\\nA Carpenter's Apprentice | and to hold God in awe. This is the law that,\\ncpage@seas.upenn.edu | seeing the stricken heart of pride brought down,\\n | we learn when we are old. -Adapted from Sophocles\\n+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=-+-=+-=-+-=+-=-+=-+-=-+-=-+=-+-= \n", "2 I differ with our moderator on this. I thought the whole idea of God coming\\ndown to earth to live as one of us \"subject to sin and death\" (as one of\\nthe consecration prayers in the Book of Common Prayer (1979) puts it) was\\nthat Jesus was tempted, but did not succumb. If sin is not part of the\\nbasic definition of humanity, then Jesus \"fully human\" (Nicea) would not\\nbe \"subject to sin\", but then the Resurrection loses some of its meaning,\\nbecause we encounter our humanity most powerfully when we sin. To distinguish\\nbetween \"human\" and \"fallen human\" makes Jesus less like one of us at the\\ntime we need him most.\\n\\n\\nFirst, the Monophysites inherited none of Nestorius's version -- they \\nwere on the opposite end of the spectrum from him. Second, the historical\\nrecord suggests that the positions attributed to Nestorius were not as\\nextreme as his (successful) opponents (who wrote the conventional history)\\nclaimed. Mainly Nestorius opposed the term Theotokos for Mary, arguing\\n(I think correctly) that a human could not be called Mother of God. I mean,\\nin the Athanasian Creed we talk about the Son \"uncreate\" -- surely even \\nArians would concede that Jesus existed long before Mary. Anyway, Nestorius's\\nopponents claimed that by saying Mary was not Theotokos, that he claimed\\nthat she only gave birth to the human nature of Jesus, which would require\\ntwo seperate and distinct natures. The argument fails though, because\\nMary simply gave birth to Jesus, who preexisted her either divinely,\\nif you accept \"Nestorianism\" as commonly defined, or both natures intertwined,\\na la Chalcedon.\\n\\nSecond, I am not sure that \"Nestorianism\" is not a better alternative than\\nthe orthodox view. After all, I find it hard to believe that pre-Incarnation\\nthat Jesus's human nature was in heaven; likewise post-Ascension. I think\\nrather that God came to earth and took our nature upon him. It was a seperate\\nnature, capable of being tempted as in Gethsemane (since I believe the divine\\nnature could never be tempted) but in its moments of weakness the divine nature\\nprevailed.\\n\\nComments on the above warmly appreciated.\\n\\nJason Albert\\n\\n[There may be differences in what we mean by \"subject to sin\". The\\noriginal complaint was from someone who didn't see how we could call\\nJesus fully human, because he didn't sin. I completely agree that\\nJesus was subject to temptation. I simply object to the idea that by\\nnot succumbing, he is thereby not fully human. I believe that you do\\nnot have to sin in order to be human.\\n\\nI again apologize for confusing Nestorianism and monophysitism. I\\nagree with you, and have said elsewhere, that there's reason to think\\nthat not everyone who is associated with heretical positions was in\\nfact heretical. There are scholars who maintain that Nestorius was\\nnot Nestorian. I have to confess that the first time I read some of\\nthe correspondence between Nestorius and his opponents, I thought he\\ngot the better of them.\\n\\nHowever, most scholars do believe that the work that eventually led to\\nChalcedon was an advance, and that Nestorius was at the very least\\n\"rash and dogmatic\" (as the editor of \"The Christological Controversy\"\\nrefers to him) in rejecting all approaches other than his own. As\\nregular Usenet readers know, narrowness can be just as much an\\nimpediment as being wrong. Furthermore, he did say some things that I\\nthink are problematical. He responds to a rather mild letter from\\nCyril with a flame worthy of Usenet. In it he says \"To attribute also\\nto [the Logos], in the name of [the incarnation] the characteristics\\nof the flesh that has been conjoined with him ... is, my brother,\\neither the work of a mind which truly errs in the fashion of the\\nGreeks or that of a mind diseased with the insane heresy of Arius and\\nApollinaris and the others. Those who are thus carried away with the\\nidea of this association are bound, because of it, to make the divine\\nLogos have a part in being fed with milk and participate to some\\ndegree in growh and stand in need of angelic assistance because of his\\nfearfulness ... These things are taken falsely when they are put off\\non the deity and they become the occasion of just condemnation for us\\nwho perpetrate the falsehood.\"\\n\\nIt's all well and good to maintain a proper distinction between\\nhumanity and divinity. But the whole concept of incarnation is based\\non exactly the idea that the divine Logos does in fact have \"to some\\ndegree\" a part in being born, growing up, and dying. Of course it\\nmust be understood that there's a certain indirectness in the Logos'\\nparticipation in these things. But there must be some sort of\\nidentification between the divine and human, or we don't have an\\nincarnation at all. Nestorius seemed to think in black and white\\nterms, and missed the sorts of nuances one needs to deal with this\\narea.\\n\\nYou say \"I find it hard to believe that pre-Incarnation that Jesus's\\nhuman nature was in heaven.\" I don't think that's required by\\northodox doctrine. It's the divine Logos that is eternal. " ] }, "execution_count": 19, "metadata": {}, "output_type": "execute_result" } ], "source": [ "# train scorer from document IDs returned by keyword search\n", "doc_ids = [doc['doc_id'] for doc in results]\n", "tm.train_scorer(doc_ids=doc_ids)\n", "\n", "# get text and scores of remaining documents\n", "other_texts = [d['text'] for d in tm.get_docs() if d['doc_id'] not in doc_ids]\n", "other_scores = tm.score(other_texts)\n", "\n", "# display results in Pandas dataframe\n", "other_preds = [int(score > 0) for score in other_scores]\n", "data = sorted(list(zip(other_preds, other_scores, other_texts)), key=lambda item:item[1], reverse=True)\n", "print('Top Inliers (or Most Similar to Our Technology-Related Topics')\n", "print('\\t\\tNumber of Predicted Inliners: %s' % sum(other_preds))\n", "df = pd.DataFrame(data, columns=['Prediction', 'Score', 'Text'])\n", "df.head(3)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Here, we've easily found many other documents about religion-focused documents that **do not** explicitly mention Christ.\n", "\n", "Notice that our One Class classifier classified 4759 documents as within-class. This is because many documents mentioning \"Christ\" may not be about religion or Christianity. As a result, our One-Class classifier predicts 1 for non-relgious documents that are also similar to our positive set. See documents predicted as positive with low scores close to zero:" ] }, { "cell_type": "code", "execution_count": 23, "metadata": {}, "outputs": [ { "data": { "text/html": [ "
\n", "\n", "\n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", " \n", "
PredictionScoreText
475410.000214Ken:\\n\\nYour arguments are thoughtful but you are going up against the Big\\nBoys if you're tackling Henry. Allen Sherzer will doubtless chime in\\non the subject of staggering operational costs, too. Good luck, son.\\n\\n\\nAhem. The Russians are in the Free World now, or at least it would be\\nPolitically Correct to contend so.\\n\\n\\nIt will be tough to make DC-X succeed, and to turn it into an\\noperational orbital vehicle. Doubtless it will fail to meet some of\\nthe promised goals. The reason people are so fond of it is that it's\\nthe *only* chance we have now, or will have for a *long* time to come,\\nto develop a launch vehicle with radically lower costs. \\n\\nThere is no Shuttle successor in funded development, NASP is dwindling\\naway, and ALS/NLS/Spacelifter sure as hell aren't gonna knock any\\nzeroes off that $2000-$3000 per pound cost. Part of the blame for\\nthis must be placed on a Shuttle program that consumes many annual\\nbillions of the, er, Free World's available space cash. As you will\\nno doubt hear from many correspondents in the days to come. (-:\\n\\nDC-X is an attempt to break out of the vicious cycle by keeping\\ndevelopment costs low and flying incremental \"X-plane\" hardware.\\nIt's been, to my mind, incredibly successful already-- they've built a\\ncomplex prototype in under 600 days for under 60 megabucks. I would\\nhave been extremely skeptical that this could be accomplished in 1990s\\nAmerica, never mind flying the thing, getting a successor funded, or\\nbuilding the DC-Y.\\n\\nI'm sure you know well that launch costs are THE basic problem for any\\nexpansion of astronautics. I don't see a realistic prospect for\\nbeating down those costs, for multi-ton payloads, anywhere else. If\\nthe DC flops, it'll be business as usual in space. The Nineties and\\nthe Double-Oughts will look just like the Seventies and Eighties, a\\nprospect too depressing to bear.\\n\\n(Pegasus represents another assault on the problem from a different\\ndirection. It doesn't lower cost-per-pound but it offers an orbital\\nlaunch for under ten megabucks. It's creating its own market for\\nsmall payloads.)\\n\\nI read the magazines and I've attended the last two IAFs. There are\\nplenty of engineers with paper ideas for cheaper launch systems, some\\nof them as good as or better than SSTO. There is no sign in today's\\nworld that any of these designs will be allowed anywhere near an\\nassembly line.\\n\\n[...deleting some things I'm not going to prove tonight...]\\n\\nStrawman. Is anybody seriously proposing this? References, please. \\nThe DC must be developed in the real-world funding climate, which\\nincludes a NASA ferociously committed to continuing Shuttle\\noperations, as well as the \"bird in the hand\" argument your common\\nsense tells you. If DC-Y flies at all, it flies alongside the\\nShuttle, not instead of it.\\n\\nAlso, of course, DC-Y and its operational descendants will be useful\\nfor a wide variety of jobs even if they are *not* man-rated.\\n\\n\\nIf a DC-X successor can fly a 10,000-kg payload for $1M, or even $5M,\\nrather than the $40M it now costs, more people will be able to afford\\nmore payloads... for the same money, you can fly several satellites\\ninstead of one. Big outfits can fly multi-satellite series. Little\\noutfits will be able to fly spacecraft of their own, instead of\\nbegging a ride. This is just supply and demand. You should be able\\nto convince *yourself* that point 4 will be true, assuming DC makes a\\nbig difference in costs. Do you have some reason to think not?
475510.000159Wanted: Summer sublet in NW DC, on red Metro line. Have own bedroom, but can\\n share common areas with others. Apartment or room for $400 or less.\\n Move in Memorial Day weekend through end of August. No smokers.
475610.000159Wanted: Summer sublet in NW DC, on red Metro line. Have own bedroom, but can\\n share common areas with others. Apartment or room for $400 or less.\\n Move in Memorial Day weekend through end of August. No smokers.
475710.000103\\nFirstly, I would never consider trying to make a one-shot timer. Your 2F2 cap\\nwill have been designed as a RAM battery substitute, rather than for use in\\napplications where you wish to drain the cap with a constant current. Their\\nperformance, in some respects, is more akin to batteries than to a normal cap.\\nThe other point is that big electro's are usually rated with -5%+20% tolerances,\\nso calculating exactly what capacitance you have (particularly when considering\\nthe cold).\\n\\nYou should be looking at designing a micropower oscillator and divider chain,\\nthat \"rolls over\" (from zero to maximum count, or vice-versa) once every\\n1,209,600 seconds. If you were to use something like a 110Hz clock, you would\\nneed a divider chain of 2^27, to give an error of less than one percent in the\\nfiring times over the six week period. Of course, you could trim the 110Hz\\noscillator to give you exact time, but the likelyhood of the oscillator\\nremaining exactly constant at -40 is low anyway.\\n\\nI would suspect that there would be far more battrey drain in firing the\\nsolenoid than there would be in the timer circuit.\\n\\nCaveat - I'm not experienced in low temperature electronics, so I can't\\nguarantee that this (or any other) approach is the best for inhospitable\\nconditions.\\n\\nJohnH\\n\\n----------------------------------------------------------------------------\\n\\n | _ |_ _ |_| _ _| _| Electronics Department\\n |_| (_) | | | | | | (_| (_| (_| \\/ School of MPCE\\n ---------------------------------/- Macquarie University\\n Sydney, AUSTRALIA 2109\\n\\n Email: johnh@mpce.mq.edu.au, Ph: +61 2 805 8959, Fax: +61 2 805 8983\\n
475810.000075I am writing a paper on religion and how it reflects \\nand or affects modern music. This brief questionaire is summary of\\nthe questions I would like answered. A response is requested and \\ncan be mailed to me directly at: \\n \\n gtd259a@prism.gatech.edu \\n \\n *PLEASE MAIL - DO NOT POST*\\n\\nThanks in advance,\\nMatt Kressel\\n\\n\\n----------------------------------------------------------------------\\n\\n1.) How do you feel about groups like Diecide, Slayer, and Dio who\\nfreely admit to practicing satanism and preach it in their songs?\\n\\n\\n2.) How do you feel about groups like Petra, old Stryper, Whitecross,\\nand Holy Soldier who promote and sing about Cristianity?\\n\\n\\n3.) How do you feel about groups like Front 242, XTC, Revolting Cocks,\\nMinor Threat, and Ministry who condone and sing about atheism?\\n\\n\\n4.) How do you feel about bands like Shelter who preach the Hare\\nKrishna religion and other minority(but not unheard of) religions?\\n\\n\\n5.a) Do you feel there is any difference between promoting music that\\nsupports Cristianity and music that condones satanism?\\n\\n b) Why do you feel this way?\\n\\n\\n6.) What types of music do you listen to?\\n\\n\\n7.a) How often do you purchase music?\\n\\n b) How often does that music contain lyrics with undertones in\\nreligion?\\n\\n\\n8.a) Do you feel that music one listens to affects the way one views\\na particular religion? Religion in general?\\n\\n b) How does it affect the way you view your religion? All religions?\\n\\n\\n9.) FEEL FREE TO ADD ANY COMMENTS HERE\\n\\n\\n\\n\\n
\n", "
" ], "text/plain": [ " Prediction Score \\\n", "4754 1 0.000214 \n", "4755 1 0.000159 \n", "4756 1 0.000159 \n", "4757 1 0.000103 \n", "4758 1 0.000075 \n", "\n", " Text \n", "4754 Ken:\\n\\nYour arguments are thoughtful but you are going up against the Big\\nBoys if you're tackling Henry. Allen Sherzer will doubtless chime in\\non the subject of staggering operational costs, too. Good luck, son.\\n\\n\\nAhem. The Russians are in the Free World now, or at least it would be\\nPolitically Correct to contend so.\\n\\n\\nIt will be tough to make DC-X succeed, and to turn it into an\\noperational orbital vehicle. Doubtless it will fail to meet some of\\nthe promised goals. The reason people are so fond of it is that it's\\nthe *only* chance we have now, or will have for a *long* time to come,\\nto develop a launch vehicle with radically lower costs. \\n\\nThere is no Shuttle successor in funded development, NASP is dwindling\\naway, and ALS/NLS/Spacelifter sure as hell aren't gonna knock any\\nzeroes off that $2000-$3000 per pound cost. Part of the blame for\\nthis must be placed on a Shuttle program that consumes many annual\\nbillions of the, er, Free World's available space cash. As you will\\nno doubt hear from many correspondents in the days to come. (-:\\n\\nDC-X is an attempt to break out of the vicious cycle by keeping\\ndevelopment costs low and flying incremental \"X-plane\" hardware.\\nIt's been, to my mind, incredibly successful already-- they've built a\\ncomplex prototype in under 600 days for under 60 megabucks. I would\\nhave been extremely skeptical that this could be accomplished in 1990s\\nAmerica, never mind flying the thing, getting a successor funded, or\\nbuilding the DC-Y.\\n\\nI'm sure you know well that launch costs are THE basic problem for any\\nexpansion of astronautics. I don't see a realistic prospect for\\nbeating down those costs, for multi-ton payloads, anywhere else. If\\nthe DC flops, it'll be business as usual in space. The Nineties and\\nthe Double-Oughts will look just like the Seventies and Eighties, a\\nprospect too depressing to bear.\\n\\n(Pegasus represents another assault on the problem from a different\\ndirection. It doesn't lower cost-per-pound but it offers an orbital\\nlaunch for under ten megabucks. It's creating its own market for\\nsmall payloads.)\\n\\nI read the magazines and I've attended the last two IAFs. There are\\nplenty of engineers with paper ideas for cheaper launch systems, some\\nof them as good as or better than SSTO. There is no sign in today's\\nworld that any of these designs will be allowed anywhere near an\\nassembly line.\\n\\n[...deleting some things I'm not going to prove tonight...]\\n\\nStrawman. Is anybody seriously proposing this? References, please. \\nThe DC must be developed in the real-world funding climate, which\\nincludes a NASA ferociously committed to continuing Shuttle\\noperations, as well as the \"bird in the hand\" argument your common\\nsense tells you. If DC-Y flies at all, it flies alongside the\\nShuttle, not instead of it.\\n\\nAlso, of course, DC-Y and its operational descendants will be useful\\nfor a wide variety of jobs even if they are *not* man-rated.\\n\\n\\nIf a DC-X successor can fly a 10,000-kg payload for $1M, or even $5M,\\nrather than the $40M it now costs, more people will be able to afford\\nmore payloads... for the same money, you can fly several satellites\\ninstead of one. Big outfits can fly multi-satellite series. Little\\noutfits will be able to fly spacecraft of their own, instead of\\nbegging a ride. This is just supply and demand. You should be able\\nto convince *yourself* that point 4 will be true, assuming DC makes a\\nbig difference in costs. Do you have some reason to think not? \n", "4755 Wanted: Summer sublet in NW DC, on red Metro line. Have own bedroom, but can\\n share common areas with others. Apartment or room for $400 or less.\\n Move in Memorial Day weekend through end of August. No smokers. \n", "4756 Wanted: Summer sublet in NW DC, on red Metro line. Have own bedroom, but can\\n share common areas with others. Apartment or room for $400 or less.\\n Move in Memorial Day weekend through end of August. No smokers. \n", "4757 \\nFirstly, I would never consider trying to make a one-shot timer. Your 2F2 cap\\nwill have been designed as a RAM battery substitute, rather than for use in\\napplications where you wish to drain the cap with a constant current. Their\\nperformance, in some respects, is more akin to batteries than to a normal cap.\\nThe other point is that big electro's are usually rated with -5%+20% tolerances,\\nso calculating exactly what capacitance you have (particularly when considering\\nthe cold).\\n\\nYou should be looking at designing a micropower oscillator and divider chain,\\nthat \"rolls over\" (from zero to maximum count, or vice-versa) once every\\n1,209,600 seconds. If you were to use something like a 110Hz clock, you would\\nneed a divider chain of 2^27, to give an error of less than one percent in the\\nfiring times over the six week period. Of course, you could trim the 110Hz\\noscillator to give you exact time, but the likelyhood of the oscillator\\nremaining exactly constant at -40 is low anyway.\\n\\nI would suspect that there would be far more battrey drain in firing the\\nsolenoid than there would be in the timer circuit.\\n\\nCaveat - I'm not experienced in low temperature electronics, so I can't\\nguarantee that this (or any other) approach is the best for inhospitable\\nconditions.\\n\\nJohnH\\n\\n----------------------------------------------------------------------------\\n\\n | _ |_ _ |_| _ _| _| Electronics Department\\n |_| (_) | | | | | | (_| (_| (_| \\/ School of MPCE\\n ---------------------------------/- Macquarie University\\n Sydney, AUSTRALIA 2109\\n\\n Email: johnh@mpce.mq.edu.au, Ph: +61 2 805 8959, Fax: +61 2 805 8983\\n \n", "4758 I am writing a paper on religion and how it reflects \\nand or affects modern music. This brief questionaire is summary of\\nthe questions I would like answered. A response is requested and \\ncan be mailed to me directly at: \\n \\n gtd259a@prism.gatech.edu \\n \\n *PLEASE MAIL - DO NOT POST*\\n\\nThanks in advance,\\nMatt Kressel\\n\\n\\n----------------------------------------------------------------------\\n\\n1.) How do you feel about groups like Diecide, Slayer, and Dio who\\nfreely admit to practicing satanism and preach it in their songs?\\n\\n\\n2.) How do you feel about groups like Petra, old Stryper, Whitecross,\\nand Holy Soldier who promote and sing about Cristianity?\\n\\n\\n3.) How do you feel about groups like Front 242, XTC, Revolting Cocks,\\nMinor Threat, and Ministry who condone and sing about atheism?\\n\\n\\n4.) How do you feel about bands like Shelter who preach the Hare\\nKrishna religion and other minority(but not unheard of) religions?\\n\\n\\n5.a) Do you feel there is any difference between promoting music that\\nsupports Cristianity and music that condones satanism?\\n\\n b) Why do you feel this way?\\n\\n\\n6.) What types of music do you listen to?\\n\\n\\n7.a) How often do you purchase music?\\n\\n b) How often does that music contain lyrics with undertones in\\nreligion?\\n\\n\\n8.a) Do you feel that music one listens to affects the way one views\\na particular religion? Religion in general?\\n\\n b) How does it affect the way you view your religion? All religions?\\n\\n\\n9.) FEEL FREE TO ADD ANY COMMENTS HERE\\n\\n\\n\\n\\n " ] }, "execution_count": 23, "metadata": {}, "output_type": "execute_result" } ], "source": [ "df[df.Score > 0].tail()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.9" } }, "nbformat": 4, "nbformat_minor": 2 }