source/Working with text.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Working with text\n",
      "=================\n",
      "\n",
      "This is a shortened version of the tutorial \"Working with text\" in the form of an interactive IPython notebook. This document is \"live\"; any code example can be edited and executed in the browser. To see this in action, change some part of the code in the *cell* below and then click on the play button above."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "list_of_strings = ['working', 'with', 'text']\n",
      "for s in list_of_strings:\n",
      "    print(s)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "working\n",
        "with\n",
        "text\n"
       ]
      }
     ],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Creating a document-term matrix\n",
      "-------------------------------\n",
      "\n",
      "Word frequencies and document-term matrices are typical units of\n",
      "analysis when working with text collections. It may come as a surprise\n",
      "that reducing a book to a list of word frequencies retains useful\n",
      "information, but practice has shown this to be the case. Treating texts\n",
      "as a list of word frequencies (a vector) also makes available a range of\n",
      "mathematical tools developed for [studying and manipulating\n",
      "vectors](http://en.wikipedia.org/wiki/Euclidean_vector#History).\n",
      "\n",
      "> **Note**: Turning texts into unordered lists (or \"bags\") of words is easy in\n",
      "> Python. [Python Programming for the\n",
      "> Humanities](http://fbkarsdorp.github.io/python-course/) includes a\n",
      "> chapter entitled [Text\n",
      "> Processing](http://nbviewer.ipython.org/urls/raw.github.com/fbkarsdorp/python-course/master/Chapter%203%20-%20Text%20Preprocessing.ipynb)\n",
      "> that describes the steps in detail.\n",
      "\n",
      "This document assumes some prior exposure to text analysis so we will\n",
      "gather word frequencies (or term frequencies) derived from the lists of\n",
      "words appearing in texts into a document-term matrix using the\n",
      "[CountVectorizer](http://scikit-learn.sourceforge.net/dev/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)\n",
      "class from the [scikit-learn](http://scikit-learn.sourceforge.net/)\n",
      "package. (For those familiar with R and the\n",
      "[tm](http://cran.r-project.org/web/packages/tm/) package, this function\n",
      "performs the same operation as `DocumentTermMatrix` and takes\n",
      "recognizably similar arguments.)\n",
      "\n",
      "First we need to import the functions and classes we intend to use,\n",
      "along with our customary abbreviation for functions in the `numpy`\n",
      "package."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import numpy as np  # a conventional alias\n",
      "from sklearn.feature_extraction.text import CountVectorizer"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 1
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we use the\n",
      "[CountVectorizer](http://scikit-learn.sourceforge.net/dev/modules/generated/sklearn.feature_extraction.text.CountVectorizer.html)\n",
      "class to create a document-term matrix. `CountVectorizer` is highly\n",
      "customizable. For example, a list of \"stop words\" can be specified with\n",
      "the ``stop_words`` parameter. Other important parameters include:\n",
      "\n",
      "-   `lowercase` (default `True`) convert all text to lowercase before\n",
      "    tokenizing\n",
      "-   `min_df` (default `1`) remove from the vocabulary terms that occur\n",
      "    in fewer than `min_df` documents\u2013with a large corpus this may be set\n",
      "    to `5` to eliminate rare words\n",
      "-   `vocabulary` ignore words that do not appear in the list (or\n",
      "    iterable) assigned to parameter `vocabulary`\n",
      "-   `strip_accents` remove accents\n",
      "-   `token_pattern` (default `u'(?u)\\b\\w\\w+\\b'`) regular expression\n",
      "    identifying tokens\u2013by default words that consist of a single\n",
      "    character (e.g., 'a', '2') are ignored, setting `token_pattern` to\n",
      "    `'(?u)\\b\\w+\\b'` will include these tokens\n",
      "-   `tokenizer` (default unused) use a custom function for tokenizing\n",
      "\n",
      "For this example we will use texts by Jane Austen and Charlotte Bront\u00eb. These\n",
      "texts are available in the *Datasets* section of the collected tutorials.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filenames = ['data/austen-bront\u00eb/Austen_Emma.txt',           \n",
      "             'data/austen-bront\u00eb/Austen_Pride.txt',          \n",
      "             'data/austen-bront\u00eb/Austen_Sense.txt',          \n",
      "             'data/austen-bront\u00eb/CBronte_Jane.txt',          \n",
      "             'data/austen-bront\u00eb/CBronte_Professor.txt',     \n",
      "             'data/austen-bront\u00eb/CBronte_Villette.txt']      \n",
      "                                                             \n",
      "vectorizer = CountVectorizer(input='filename')               \n",
      "dtm = vectorizer.fit_transform(filenames)  # a sparse matrix \n",
      "vocab = vectorizer.get_feature_names()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 4
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we have a document-term matrix and a vocabulary list. Before we can\n",
      "query the matrix and find out, for example, how many times the word\n",
      "'house' occurs in *Emma* (the first text in `filenames`), we need to\n",
      "convert this matrix from its current format, a [sparse\n",
      "matrix](http://docs.scipy.org/doc/scipy/reference/sparse.html), into a\n",
      "normal NumPy array. We will also convert `vocab`, a list of vocabulary,\n",
      "to an array of strings, as an array supports a greater variety of\n",
      "operations.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# for reference, note the current class of `dtm`  \n",
      "type(dtm)                                         \n",
      "dtm = dtm.toarray()  # convert to a regular array \n",
      "vocab = np.array(vocab_list)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 5
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "> **Note:** A sparse matrix is used to store matrices that contain a significant\n",
      "> number of entries that are zero. Typically, a sparse matrix only\n",
      "> records non-zero entries. To understand why this matters so much\n",
      "> that `CountVectorizer` returns a sparse matrix by default,\n",
      "> consider a 4000 by 50000 matrix that is 60% zeros. In Python an\n",
      "> integer takes up 4 bytes, so using a sparse matrix saves almost\n",
      "> 500M, which is a significant amount of computer memory. (Remember\n",
      "> that arrays are usually stored in memory, not on disk).\n",
      "\n",
      "Querying the document-term matrix and the vocabulary is straightforward.\n",
      "For example, here are two ways of finding how many times the word\n",
      "'house' occurs in the first text, *Emma*:\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# the first file, indexed by 0 in Python, is *Emma*                 \n",
      "filenames[0] == 'data/austen-bront\u00eb/Austen_Emma.txt'                \n",
      "                                                                    \n",
      "# use the standard Python list method index(...)                    \n",
      "house_idx = vocab_list.index('house')                               \n",
      "dtm[0, house_idx]                                                   \n",
      "                                                                    \n",
      "# alternatively, use NumPy indexing                                 \n",
      "# in R this would be essentially the same, dtm[1, vocab == 'house'] \n",
      "dtm[0, vocab == 'house']                                         \n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 6,
       "text": [
        "array([95], dtype=int64)"
       ]
      }
     ],
     "prompt_number": 6
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# verify that this is the result we anticipated\n",
      "vocab[house_idx]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 7,
       "text": [
        "'house'"
       ]
      }
     ],
     "prompt_number": 7
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Sandbox\n",
      "=======\n",
      "Feel free to experiment with the document-term matrix `dtm` in the code cells below."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(dtm.shape)\n",
      "for fn in filenames:\n",
      "    print(fn)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "(6, 22854)\n",
        "data/austen-bront\u00eb/Austen_Emma.txt\n",
        "data/austen-bront\u00eb/Austen_Pride.txt\n",
        "data/austen-bront\u00eb/Austen_Sense.txt\n",
        "data/austen-bront\u00eb/CBronte_Jane.txt\n",
        "data/austen-bront\u00eb/CBronte_Professor.txt\n",
        "data/austen-bront\u00eb/CBronte_Villette.txt\n"
       ]
      }
     ],
     "prompt_number": 8
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print(len(vocab))\n",
      "vocab[500:550]  # look at some of the vocabulary"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "22854\n"
       ]
      },
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "array(['abuse', 'abused', 'abuses', 'abusing', 'abusive', 'abyss',\n",
        "       'acacia', 'acacias', 'academician', 'academicians', 'accede',\n",
        "       'acceded', 'acceding', 'accelerate', 'accelerated', 'accent',\n",
        "       'accented', 'accents', 'accentuated', 'accept', 'acceptable',\n",
        "       'acceptably', 'acceptance', 'accepted', 'accepting', 'accepts',\n",
        "       'access', 'accessible', 'accession', 'accessory', 'accident',\n",
        "       'accidental', 'accidentally', 'accidently', 'accidents',\n",
        "       'accommodate', 'accommodated', 'accommodating', 'accommodation',\n",
        "       'accommodations', 'accompanied', 'accompanies', 'accompaniment',\n",
        "       'accompaniments', 'accompany', 'accompanying', 'accompli',\n",
        "       'accomplices', 'accomplish', 'accomplished'], \n",
        "      dtype='<U20')"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}