Obtain your text sources

Text data can come from lots of areas:

• Web sites
  • Twitter
• Databases
• PDF documents
• Digital scans of printed materials

The easier to convert your text data into digitally stored text, the cleaner your results and fewer transcription errors.

Extract documents and move into a corpus

A text corpus is a large and structured set of texts. It typically stores the text as a raw character string with meta data and details stored with the text.

Transformation

Examples of typical transformations include:

• Tagging segments of speech for part-of-speech (nouns, verbs, adjectives, etc.) or entity recognition (person, place, company, etc.)
• Standard text processing - we want to remove extraneous information from the text and standardize it into a uniform format. This typically involves:
  • Converting to lower case
  • Removing punctuation
  • Removing numbers
  • Removing stopwords - common parts of speech that are not informative such as a, an, be, of, etc.
  • Removing domain-specific stopwords
  • Stemming - reduce words to their word stem
    • “Fishing”, “fished”, and “fisher” -> “fish”

Extract features

Feature extraction involves converting the text string into some sort of quantifiable measures. The most common approach is the bag-of-words model, whereby each document is represented as a vector which counts the frequency of each term’s appearance in the document. You can combine all the vectors for each document together and you create a term-document matrix:

• Each row is a document
• Each column is a term
• Each cell represents the frequency of the term appearing in the document

However the bag-of-word model ignores context. You could randomly scramble the order of terms appearing in the document and still get the same term-document matrix.

Perform analysis

At this point you now have data assembled and ready for analysis. There are several approaches you may take when analyzing text depending on your research question. Basic approaches include:

• Word frequency - counting the frequency of words in the text
• Collocation - words commonly appearing near each other
• Dictionary tagging - locating a specific set of words in the texts

More advanced methods include document classification, or assigning documents to different categories. This can be supervised (the potential categories are defined in advance of the modeling) or unsupervised (the potential categories are unknown prior to analysis). You might also conduct corpora comparison, or comparing the content of different groups of text. This is the approach used in plagiarism detecting software such as Turn It In. Finally, you may attempt to detect clusters of document features, known as topic modeling.