Coverage for nltk.chunk.regexp : 78%
Hot-keys on this page
r m x p toggle line displays
j k next/prev highlighted chunk
0 (zero) top of page
1 (one) first highlighted chunk
|
# Natural Language Toolkit: Regular Expression Chunkers # # Copyright (C) 2001-2012 NLTK Project # Author: Edward Loper <edloper@gradient.cis.upenn.edu> # Steven Bird <sb@csse.unimelb.edu.au> (minor additions) # URL: <http://www.nltk.org/> # For license information, see LICENSE.TXT
##////////////////////////////////////////////////////// ## ChunkString ##//////////////////////////////////////////////////////
""" A string-based encoding of a particular chunking of a text. Internally, the ``ChunkString`` class uses a single string to encode the chunking of the input text. This string contains a sequence of angle-bracket delimited tags, with chunking indicated by braces. An example of this encoding is::
{<DT><JJ><NN>}<VBN><IN>{<DT><NN>}<.>{<DT><NN>}<VBD><.>
``ChunkString`` are created from tagged texts (i.e., lists of ``tokens`` whose type is ``TaggedType``). Initially, nothing is chunked.
The chunking of a ``ChunkString`` can be modified with the ``xform()`` method, which uses a regular expression to transform the string representation. These transformations should only add and remove braces; they should *not* modify the sequence of angle-bracket delimited tags.
:type _str: str :ivar _str: The internal string representation of the text's encoding. This string representation contains a sequence of angle-bracket delimited tags, with chunking indicated by braces. An example of this encoding is::
{<DT><JJ><NN>}<VBN><IN>{<DT><NN>}<.>{<DT><NN>}<VBD><.>
:type _pieces: list(tagged tokens and chunks) :ivar _pieces: The tagged tokens and chunks encoded by this ``ChunkString``. :ivar _debug: The debug level. See the constructor docs.
:cvar IN_CHUNK_PATTERN: A zero-width regexp pattern string that will only match positions that are in chunks. :cvar IN_CHINK_PATTERN: A zero-width regexp pattern string that will only match positions that are in chinks. """
# These are used by _verify
""" Construct a new ``ChunkString`` that encodes the chunking of the text ``tagged_tokens``.
:type chunk_struct: Tree :param chunk_struct: The chunk structure to be further chunked. :type debug_level: int :param debug_level: The level of debugging which should be applied to transformations on the ``ChunkString``. The valid levels are: - 0: no checks - 1: full check on to_chunkstruct - 2: full check on to_chunkstruct and cursory check after each transformation. - 3: full check on to_chunkstruct and full check after each transformation. We recommend you use at least level 1. You should probably use level 3 if you use any non-standard subclasses of ``RegexpChunkRule``. """
else: 'tokens or trees')
""" Check to make sure that ``s`` still corresponds to some chunked version of ``_pieces``.
:type verify_tags: bool :param verify_tags: Whether the individual tags should be checked. If this is false, ``_verify`` will check to make sure that ``_str`` encodes a chunked version of *some* list of tokens. If this is true, then ``_verify`` will check to make sure that the tags in ``_str`` match those in ``_pieces``.
:raise ValueError: if the internal string representation of this ``ChunkString`` is invalid or not consistent with _pieces. """ # Check overall form 'chunkstring:\n %s' % s)
# Check that parens are balanced. If the string is long, we # have to do this in pieces, to avoid a maximum recursion # depth limit for regular expressions. 'chunkstring:\n %s' % s)
'chunkstring: tag changed')
""" Return the chunk structure encoded by this ``ChunkString``.
:rtype: Tree :raise ValueError: If a transformation has generated an invalid chunkstring. """
# Use this alternating list to create the chunkstruct.
# Find the list of tokens contained in this piece.
# Add this list of tokens to our pieces. else:
# Update index, piece_in_chunk
""" Apply the given transformation to the string encoding of this ``ChunkString``. In particular, find all occurrences that match ``regexp``, and replace them using ``repl`` (as done by ``re.sub``).
This transformation should only add and remove braces; it should *not* modify the sequence of angle-bracket delimited tags. Furthermore, this transformation may not result in improper bracketing. Note, in particular, that bracketing may not be nested.
:type regexp: str or regexp :param regexp: A regular expression matching the substring that should be replaced. This will typically include a named group, which can be used by ``repl``. :type repl: str :param repl: An expression specifying what should replace the matched substring. Typically, this will include a named replacement group, specified by ``regexp``. :rtype: None :raise ValueError: If this transformation generated an invalid chunkstring. """ # Do the actual substitution
# The substitution might have generated "empty chunks" # (substrings of the form "{}"). Remove them, so they don't # interfere with other transformations.
# Make sure that the transformation was legal.
# Commit the transformation.
""" Return a string representation of this ``ChunkString``. It has the form::
<ChunkString: '{<DT><JJ><NN>}<VBN><IN>{<DT><NN>}'>
:rtype: str """
""" Return a formatted representation of this ``ChunkString``. This representation will include extra spaces to ensure that tags will line up with the representation of other ``ChunkStrings`` for the same text, regardless of the chunking.
:rtype: str """ # Add spaces to make everything line up.
##////////////////////////////////////////////////////// ## Chunking Rules ##//////////////////////////////////////////////////////
""" A rule specifying how to modify the chunking in a ``ChunkString``, using a transformational regular expression. The ``RegexpChunkRule`` class itself can be used to implement any transformational rule based on regular expressions. There are also a number of subclasses, which can be used to implement simpler types of rules, based on matching regular expressions.
Each ``RegexpChunkRule`` has a regular expression and a replacement expression. When a ``RegexpChunkRule`` is "applied" to a ``ChunkString``, it searches the ``ChunkString`` for any substring that matches the regular expression, and replaces it using the replacement expression. This search/replace operation has the same semantics as ``re.sub``.
Each ``RegexpChunkRule`` also has a description string, which gives a short (typically less than 75 characters) description of the purpose of the rule.
This transformation defined by this ``RegexpChunkRule`` should only add and remove braces; it should *not* modify the sequence of angle-bracket delimited tags. Furthermore, this transformation may not result in nested or mismatched bracketing. """ """ Construct a new RegexpChunkRule.
:type regexp: regexp or str :param regexp: The regular expression for this ``RegexpChunkRule``. When this rule is applied to a ``ChunkString``, any substring that matches ``regexp`` will be replaced using the replacement string ``repl``. Note that this must be a normal regular expression, not a tag pattern. :type repl: str :param repl: The replacement expression for this ``RegexpChunkRule``. When this rule is applied to a ``ChunkString``, any substring that matches ``regexp`` will be replaced using ``repl``. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. """
# Keep docstring generic so we can inherit it. """ Apply this rule to the given ``ChunkString``. See the class reference documentation for a description of what it means to apply a rule.
:type chunkstr: ChunkString :param chunkstr: The chunkstring to which this rule is applied. :rtype: None :raise ValueError: If this transformation generated an invalid chunkstring. """
""" Return a short description of the purpose and/or effect of this rule.
:rtype: str """
""" Return a string representation of this rule. It has the form::
<RegexpChunkRule: '{<IN|VB.*>}'->'<IN>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """ '->'+repr(self._repl)+'>')
def parse(s): """ Create a RegexpChunkRule from a string description. Currently, the following formats are supported::
{regexp} # chunk rule }regexp{ # chink rule regexp}{regexp # split rule regexp{}regexp # merge rule
Where ``regexp`` is a regular expression for the rule. Any text following the comment marker (``#``) will be used as the rule's description:
>>> from nltk.chunk.regexp import RegexpChunkRule >>> RegexpChunkRule.parse('{<DT>?<NN.*>+}') <ChunkRule: '<DT>?<NN.*>+'> """ # Split off the comment (but don't split on '\#')
# Pattern bodies: chunk, chink, split, merge raise ValueError('Empty chunk pattern') else: raise ValueError('Illegal chunk pattern: %s' % rule)
""" A rule specifying how to add chunks to a ``ChunkString``, using a matching tag pattern. When applied to a ``ChunkString``, it will find any substring that matches this tag pattern and that is not already part of a chunk, and create a new chunk containing that substring. """
""" Construct a new ``ChunkRule``.
:type tag_pattern: str :param tag_pattern: This rule's tag pattern. When applied to a ``ChunkString``, this rule will chunk any substring that matches this tag pattern and that is not already part of a chunk. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ (tag_pattern2re_pattern(tag_pattern), ChunkString.IN_CHINK_PATTERN))
""" Return a string representation of this rule. It has the form::
<ChunkRule: '<IN|VB.*>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """
""" A rule specifying how to remove chinks to a ``ChunkString``, using a matching tag pattern. When applied to a ``ChunkString``, it will find any substring that matches this tag pattern and that is contained in a chunk, and remove it from that chunk, thus creating two new chunks. """ """ Construct a new ``ChinkRule``.
:type tag_pattern: str :param tag_pattern: This rule's tag pattern. When applied to a ``ChunkString``, this rule will find any substring that matches this tag pattern and that is contained in a chunk, and remove it from that chunk, thus creating two new chunks. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ (tag_pattern2re_pattern(tag_pattern), ChunkString.IN_CHUNK_PATTERN))
""" Return a string representation of this rule. It has the form::
<ChinkRule: '<IN|VB.*>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """
""" A rule specifying how to remove chunks to a ``ChunkString``, using a matching tag pattern. When applied to a ``ChunkString``, it will find any complete chunk that matches this tag pattern, and un-chunk it. """ """ Construct a new ``UnChunkRule``.
:type tag_pattern: str :param tag_pattern: This rule's tag pattern. When applied to a ``ChunkString``, this rule will find any complete chunk that matches this tag pattern, and un-chunk it. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ tag_pattern2re_pattern(tag_pattern))
""" Return a string representation of this rule. It has the form::
<UnChunkRule: '<IN|VB.*>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """
""" A rule specifying how to merge chunks in a ``ChunkString``, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk whose end matches left pattern, and immediately followed by a chunk whose beginning matches right pattern. It will then merge those two chunks into a single chunk. """ """ Construct a new ``MergeRule``.
:type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches ``left_tag_pattern``, and immediately followed by a chunk whose beginning matches this pattern. It will then merge those two chunks into a single chunk. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches this pattern, and immediately followed by a chunk whose beginning matches ``right_tag_pattern``. It will then merge those two chunks into a single chunk.
:type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ # Ensure that the individual patterns are coherent. E.g., if # left='(' and right=')', then this will raise an exception:
(tag_pattern2re_pattern(left_tag_pattern), tag_pattern2re_pattern(right_tag_pattern)))
""" Return a string representation of this rule. It has the form::
<MergeRule: '<NN|DT|JJ>', '<NN|JJ>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """ repr(self._right_tag_pattern)+'>')
""" A rule specifying how to split chunks in a ``ChunkString``, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk that matches the left pattern followed by the right pattern. It will then split the chunk into two new chunks, at the point between the two pattern matches. """ """ Construct a new ``SplitRule``.
:type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk containing a substring that matches ``left_tag_pattern`` followed by this pattern. It will then split the chunk into two new chunks at the point between these two matching patterns. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk containing a substring that matches this pattern followed by ``right_tag_pattern``. It will then split the chunk into two new chunks at the point between these two matching patterns. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ # Ensure that the individual patterns are coherent. E.g., if # left='(' and right=')', then this will raise an exception:
(tag_pattern2re_pattern(left_tag_pattern), tag_pattern2re_pattern(right_tag_pattern)))
""" Return a string representation of this rule. It has the form::
<SplitRule: '<NN>', '<DT>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """ repr(self._right_tag_pattern)+'>')
""" A rule specifying how to expand chunks in a ``ChunkString`` to the left, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk whose beginning matches right pattern, and immediately preceded by a chink whose end matches left pattern. It will then expand the chunk to incorporate the new material on the left. """ """ Construct a new ``ExpandRightRule``.
:type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose beginning matches ``right_tag_pattern``, and immediately preceded by a chink whose end matches this pattern. It will then merge those two chunks into a single chunk. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose beginning matches this pattern, and immediately preceded by a chink whose end matches ``left_tag_pattern``. It will then expand the chunk to incorporate the new material on the left.
:type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ # Ensure that the individual patterns are coherent. E.g., if # left='(' and right=')', then this will raise an exception:
(tag_pattern2re_pattern(left_tag_pattern), tag_pattern2re_pattern(right_tag_pattern)))
""" Return a string representation of this rule. It has the form::
<ExpandLeftRule: '<NN|DT|JJ>', '<NN|JJ>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """ repr(self._right_tag_pattern)+'>')
""" A rule specifying how to expand chunks in a ``ChunkString`` to the right, using two matching tag patterns: a left pattern, and a right pattern. When applied to a ``ChunkString``, it will find any chunk whose end matches left pattern, and immediately followed by a chink whose beginning matches right pattern. It will then expand the chunk to incorporate the new material on the right. """ """ Construct a new ``ExpandRightRule``.
:type right_tag_pattern: str :param right_tag_pattern: This rule's right tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches ``left_tag_pattern``, and immediately followed by a chink whose beginning matches this pattern. It will then merge those two chunks into a single chunk. :type left_tag_pattern: str :param left_tag_pattern: This rule's left tag pattern. When applied to a ``ChunkString``, this rule will find any chunk whose end matches this pattern, and immediately followed by a chink whose beginning matches ``right_tag_pattern``. It will then expand the chunk to incorporate the new material on the right.
:type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ # Ensure that the individual patterns are coherent. E.g., if # left='(' and right=')', then this will raise an exception:
(tag_pattern2re_pattern(left_tag_pattern), tag_pattern2re_pattern(right_tag_pattern)))
""" Return a string representation of this rule. It has the form::
<ExpandRightRule: '<NN|DT|JJ>', '<NN|JJ>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """ repr(self._right_tag_pattern)+'>')
""" A rule specifying how to add chunks to a ``ChunkString``, using three matching tag patterns: one for the left context, one for the chunk, and one for the right context. When applied to a ``ChunkString``, it will find any substring that matches the chunk tag pattern, is surrounded by substrings that match the two context patterns, and is not already part of a chunk; and create a new chunk containing the substring that matched the chunk tag pattern.
Caveat: Both the left and right context are consumed when this rule matches; therefore, if you need to find overlapping matches, you will need to apply your rule more than once. """ right_context_tag_pattern, descr): """ Construct a new ``ChunkRuleWithContext``.
:type left_context_tag_pattern: str :param left_context_tag_pattern: A tag pattern that must match the left context of ``chunk_tag_pattern`` for this rule to apply. :type chunk_tag_pattern: str :param chunk_tag_pattern: A tag pattern that must match for this rule to apply. If the rule does apply, then this pattern also identifies the substring that will be made into a chunk. :type right_context_tag_pattern: str :param right_context_tag_pattern: A tag pattern that must match the right context of ``chunk_tag_pattern`` for this rule to apply. :type descr: str :param descr: A short description of the purpose and/or effect of this rule. """ # Ensure that the individual patterns are coherent. E.g., if # left='(' and right=')', then this will raise an exception:
(tag_pattern2re_pattern(left_context_tag_pattern), tag_pattern2re_pattern(chunk_tag_pattern), tag_pattern2re_pattern(right_context_tag_pattern), ChunkString.IN_CHINK_PATTERN))
""" Return a string representation of this rule. It has the form::
<ChunkRuleWithContext: '<IN>', '<NN>', '<DT>'>
Note that this representation does not include the description string; that string can be accessed separately with the ``descr()`` method.
:rtype: str """ self._left_context_tag_pattern, self._chunk_tag_pattern, self._right_context_tag_pattern)
##////////////////////////////////////////////////////// ## Tag Pattern Format Conversion ##//////////////////////////////////////////////////////
# this should probably be made more strict than it is -- e.g., it # currently accepts 'foo'. ('[^\{\}<>]+', '[^\{\}<>]+'))
""" Convert a tag pattern to a regular expression pattern. A "tag pattern" is a modified version of a regular expression, designed for matching sequences of tags. The differences between regular expression patterns and tag patterns are:
- In tag patterns, ``'<'`` and ``'>'`` act as parentheses; so ``'<NN>+'`` matches one or more repetitions of ``'<NN>'``, not ``'<NN'`` followed by one or more repetitions of ``'>'``. - Whitespace in tag patterns is ignored. So ``'<DT> | <NN>'`` is equivalant to ``'<DT>|<NN>'`` - In tag patterns, ``'.'`` is equivalant to ``'[^{}<>]'``; so ``'<NN.*>'`` matches any single tag starting with ``'NN'``.
In particular, ``tag_pattern2re_pattern`` performs the following transformations on the given pattern:
- Replace '.' with '[^<>{}]' - Remove any whitespace - Add extra parens around '<' and '>', to make '<' and '>' act like parentheses. E.g., so that in '<NN>+', the '+' has scope over the entire '<NN>'; and so that in '<NN|IN>', the '|' has scope over 'NN' and 'IN', but not '<' or '>'. - Check to make sure the resulting pattern is valid.
:type tag_pattern: str :param tag_pattern: The tag pattern to convert to a regular expression pattern. :raise ValueError: If ``tag_pattern`` is not a valid tag pattern. In particular, ``tag_pattern`` should not include braces; and it should not contain nested or mismatched angle-brackets. :rtype: str :return: A regular expression pattern corresponding to ``tag_pattern``. """ # Clean up the regular expression
# Check the regular expression
# Replace "." with CHUNK_TAG_CHAR. # We have to do this after, since it adds {}[]<>s, which would # confuse CHUNK_TAG_PATTERN. # PRE doesn't have lookback assertions, so reverse twice, and do # the pattern backwards (with lookahead assertions). This can be # made much cleaner once we can switch back to SRE.
##////////////////////////////////////////////////////// ## RegexpChunkParser ##//////////////////////////////////////////////////////
""" A regular expression based chunk parser. ``RegexpChunkParser`` uses a sequence of "rules" to find chunks of a single type within a text. The chunking of the text is encoded using a ``ChunkString``, and each rule acts by modifying the chunking in the ``ChunkString``. The rules are all implemented using regular expression matching and substitution.
The ``RegexpChunkRule`` class and its subclasses (``ChunkRule``, ``ChinkRule``, ``UnChunkRule``, ``MergeRule``, and ``SplitRule``) define the rules that are used by ``RegexpChunkParser``. Each rule defines an ``apply()`` method, which modifies the chunking encoded by a given ``ChunkString``.
:type _rules: list(RegexpChunkRule) :ivar _rules: The list of rules that should be applied to a text. :type _trace: int :ivar _trace: The default level of tracing.
""" """ Construct a new ``RegexpChunkParser``.
:type rules: list(RegexpChunkRule) :param rules: The sequence of rules that should be used to generate the chunking for a tagged text. :type chunk_node: str :param chunk_node: The node value that should be used for chunk subtrees. This is typically a short string describing the type of information contained by the chunk, such as ``"NP"`` for base noun phrases. :type top_node: str :param top_node: The node value that should be used for the top node of the chunk structure. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. """
""" Apply each rule of this ``RegexpChunkParser`` to ``chunkstr``, in turn. Generate trace output between each rule. If ``verbose`` is true, then generate verbose output.
:type chunkstr: ChunkString :param chunkstr: The chunk string to which each rule should be applied. :type verbose: bool :param verbose: Whether output should be verbose. :rtype: None """ print('#', rule.descr()+' ('+repr(rule)+'):') else:
""" Apply each rule of this ``RegexpChunkParser`` to ``chunkstr``, in turn.
:param chunkstr: The chunk string to which each rule should be applied. :type chunkstr: ChunkString :rtype: None """
""" :type chunk_struct: Tree :param chunk_struct: the chunk structure to be (further) chunked :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or highter will generate verbose tracing output. This value overrides the trace level value that was given to the constructor. :rtype: Tree :return: a chunk structure that encodes the chunks in a given tagged sentence. A chunk is a non-overlapping linguistic group, such as a noun phrase. The set of chunks identified in the chunk structure depends on the rules used to define this ``RegexpChunkParser``. """
# Use the default trace value?
# Apply the sequence of rules to the chunkstring. else:
# Use the chunkstring to create a chunk structure.
""" :return: the sequence of rules used by ``RegexpChunkParser``. :rtype: list(RegexpChunkRule) """
""" :return: a concise string representation of this ``RegexpChunkParser``. :rtype: str """
""" :return: a verbose string representation of this ``RegexpChunkParser``. :rtype: str """ else:
##////////////////////////////////////////////////////// ## Chunk Grammar ##//////////////////////////////////////////////////////
""" A grammar based chunk parser. ``chunk.RegexpParser`` uses a set of regular expression patterns to specify the behavior of the parser. The chunking of the text is encoded using a ``ChunkString``, and each rule acts by modifying the chunking in the ``ChunkString``. The rules are all implemented using regular expression matching and substitution.
A grammar contains one or more clauses in the following form::
NP: {<DT|JJ>} # chunk determiners and adjectives }<[\.VI].*>+{ # chink any tag beginning with V, I, or . <.*>}{<DT> # split a chunk at a determiner <DT|JJ>{}<NN.*> # merge chunk ending with det/adj # with one starting with a noun
The patterns of a clause are executed in order. An earlier pattern may introduce a chunk boundary that prevents a later pattern from executing. Sometimes an individual pattern will match on multiple, overlapping extents of the input. As with regular expression substitution more generally, the chunker will identify the first match possible, then continue looking for matches after this one has ended.
The clauses of a grammar are also executed in order. A cascaded chunk parser is one having more than one clause. The maximum depth of a parse tree created by this chunk parser is the same as the number of clauses in the grammar.
When tracing is turned on, the comment portion of a line is displayed each time the corresponding pattern is applied.
:type _start: str :ivar _start: The start symbol of the grammar (the root node of resulting trees) :type _stages: int :ivar _stages: The list of parsing stages corresponding to the grammar
""" """ Create a new chunk parser, from the given start state and set of chunk patterns.
:param grammar: The grammar, or a list of RegexpChunkParser objects :type grammar: str or list(RegexpChunkParser) :param top_node: The top node of the tree being created :type top_node: str or Nonterminal :param loop: The number of times to run through the patterns :type loop: int :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or higher will generate verbose tracing output. """
else: # Make sur the grammar looks like it has the right type: type_err = ('Expected string or list of RegexpChunkParsers ' 'for the grammar.') try: grammar = list(grammar) except: raise TypeError(type_err) for elt in grammar: if not isinstance(elt, RegexpChunkParser): raise TypeError(type_err) self._stages = grammar
""" Helper function for __init__: parse the grammar if it is a string. """
# New stage begins if there's an unescaped ':' # Record the stage that we just completed. # Start a new stage.
# Skip blank & comment-only lines
# Add the rule
# Record the final stage
""" Helper function for __init__: add a new stage to the parser. """ raise ValueError('Expected stage marker (eg NP:)') top_node=top_node, trace=trace)
""" Apply the chunk parser to this input.
:type chunk_struct: Tree :param chunk_struct: the chunk structure to be (further) chunked (this tree is modified, and is also returned) :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; ``1`` will generate normal tracing output; and ``2`` or highter will generate verbose tracing output. This value overrides the trace level value that was given to the constructor. :return: the chunked output. :rtype: Tree """
""" :return: a concise string representation of this ``chunk.RegexpParser``. :rtype: str """
""" :return: a verbose string representation of this ``RegexpParser``. :rtype: str """
##////////////////////////////////////////////////////// ## Demonstration code ##//////////////////////////////////////////////////////
""" Demonstration code for evaluating a chunk parser, using a ``ChunkScore``. This function assumes that ``text`` contains one sentence per line, and that each sentence has the form expected by ``tree.chunk``. It runs the given chunk parser on each sentence in the text, and scores the result. It prints the final score (precision, recall, and f-measure); and reports the set of chunks that were missed and the set of chunks that were incorrect. (At most 10 missing chunks and 10 incorrect chunks are reported).
:param chunkparser: The chunkparser to be tested :type chunkparser: ChunkParserI :param text: The chunked tagged text that should be used for evaluation. :type text: str """ from nltk import chunk from nltk.tree import Tree
# Evaluate our chunk parser. chunkscore = chunk.ChunkScore()
for sentence in text.split('\n'): print(sentence) sentence = sentence.strip() if not sentence: continue gold = chunk.tagstr2tree(sentence) tokens = gold.leaves() test = chunkparser.parse(Tree('S', tokens), trace=1) chunkscore.score(gold, test) print()
print('/'+('='*75)+'\\') print('Scoring', chunkparser) print(('-'*77)) print('Precision: %5.1f%%' % (chunkscore.precision()*100), ' '*4, end=' ') print('Recall: %5.1f%%' % (chunkscore.recall()*100), ' '*6, end=' ') print('F-Measure: %5.1f%%' % (chunkscore.f_measure()*100))
# Missed chunks. if chunkscore.missed(): print('Missed:') missed = chunkscore.missed() for chunk in missed[:10]: print(' ', ' '.join(c.__str__() for c in chunk)) if len(chunkscore.missed()) > 10: print(' ...')
# Incorrect chunks. if chunkscore.incorrect(): print('Incorrect:') incorrect = chunkscore.incorrect() for chunk in incorrect[:10]: print(' ', ' '.join(c.__str__() for c in chunk)) if len(chunkscore.incorrect()) > 10: print(' ...')
print('\\'+('='*75)+'/') print()
""" A demonstration for the ``RegexpChunkParser`` class. A single text is parsed with four different chunk parsers, using a variety of rules and strategies. """
from nltk import chunk, Tree
text = """\ [ the/DT little/JJ cat/NN ] sat/VBD on/IN [ the/DT mat/NN ] ./. [ John/NNP ] saw/VBD [the/DT cats/NNS] [the/DT dog/NN] chased/VBD ./. [ John/NNP ] thinks/VBZ [ Mary/NN ] saw/VBD [ the/DT cat/NN ] sit/VB on/IN [ the/DT mat/NN ]./. """
print('*'*75) print('Evaluation text:') print(text) print('*'*75) print()
grammar = r""" NP: # NP stage {<DT>?<JJ>*<NN>} # chunk determiners, adjectives and nouns {<NNP>+} # chunk proper nouns """ cp = chunk.RegexpParser(grammar) chunk.demo_eval(cp, text)
grammar = r""" NP: {<.*>} # start by chunking each tag }<[\.VI].*>+{ # unchunk any verbs, prepositions or periods <DT|JJ>{}<NN.*> # merge det/adj with nouns """ cp = chunk.RegexpParser(grammar) chunk.demo_eval(cp, text)
grammar = r""" NP: {<DT>?<JJ>*<NN>} # chunk determiners, adjectives and nouns VP: {<TO>?<VB.*>} # VP = verb words """ cp = chunk.RegexpParser(grammar) chunk.demo_eval(cp, text)
grammar = r""" NP: {<.*>*} # start by chunking everything }<[\.VI].*>+{ # chink any verbs, prepositions or periods <.*>}{<DT> # separate on determiners PP: {<IN><NP>} # PP = preposition + noun phrase VP: {<VB.*><NP|PP>*} # VP = verb words + NPs and PPs """ cp = chunk.RegexpParser(grammar) chunk.demo_eval(cp, text)
# Evaluation
from nltk.corpus import conll2000
print() print("Demonstration of empty grammar:")
cp = chunk.RegexpParser("") print(chunk.accuracy(cp, conll2000.chunked_sents('test.txt', chunk_types=('NP',))))
print() print("Demonstration of accuracy evaluation using CoNLL tags:")
grammar = r""" NP: {<.*>} # start by chunking each tag }<[\.VI].*>+{ # unchunk any verbs, prepositions or periods <DT|JJ>{}<NN.*> # merge det/adj with nouns """ cp = chunk.RegexpParser(grammar) print(chunk.accuracy(cp, conll2000.chunked_sents('test.txt')[:5]))
print() print("Demonstration of tagged token input")
grammar = r""" NP: {<.*>*} # start by chunking everything }<[\.VI].*>+{ # chink any verbs, prepositions or periods <.*>}{<DT> # separate on determiners PP: {<IN><NP>} # PP = preposition + noun phrase VP: {<VB.*><NP|PP>*} # VP = verb words + NPs and PPs """ cp = chunk.RegexpParser(grammar) print(cp.parse([("the","DT"), ("little","JJ"), ("cat", "NN"), ("sat", "VBD"), ("on", "IN"), ("the", "DT"), ("mat", "NN"), (".", ".")]))
demo() |