## Breaking changes in v2.0!

Please see the [changelog](CHANGELOG.md) for details!

## Usage

```js
const Lexer = require('snapdragon-lexer');
const lexer = new Lexer();

lexer.capture('slash', /^\//);
lexer.capture('text', /^\w+/);
lexer.capture('star', /^\*/);

console.log(lexer.tokenize('foo/*'));
```

## API
{%= apidocs("index.js") %}


### .set

Register a handler function.

**Params**

* `type` **{String}**
* `fn` **{Function}**: The handler function to register.

**Example**

```js
lexer.set('star', function(token) {
  // do parser, lexer, or compiler stuff
});
```

As an alternative to `.set`, the [.capture](#capture) method will automatically register a handler when a function is passed as the last argument.

### .get

Get a registered handler function.

**Params**

* `type` **{String}**
* `fn` **{Function}**: The handler function to register.

**Example**

```js
lexer.set('star', function() {
  // do parser, lexer, or compiler stuff
});
const star = handlers.get('star');
```

## Properties

### lexer.isLexer

Type: **{boolean}** 

Default: `true` (contant)

This property is defined as a convenience, to make it easy for plugins to check for an instance of Lexer. 

### lexer.input

Type: **{string}** 

Default: `''`

The unmodified source string provided by the user.

### lexer.string

Type: **{string}** 

Default: `''`

The source string minus the part of the string that has already been [consumed](#consume).

### lexer.consumed

Type: **{string}** 

Default: `''`

The part of the source string that has been consumed.

### lexer.tokens

Type: **{array}** 

Default: `[]` (instance of [snapdragon-stack][])

Array of lexed tokens.

### lexer.stash

Type: **{array}** 

Default: `['']` (instance of [snapdragon-stack][])

Array of captured strings. Similar to the [lexer.tokens](#lexertokens) array, but stores strings instead of token objects.

### lexer.stack

Type: **{array}** 

Default: `[]` (instance of [snapdragon-stack][])

LIFO (last in, first out) array. A token is pushed onto the stack when an "opening" character or character sequence needs to be tracked. When the (matching) "closing" character or character sequence is encountered, the (opening) token is popped off of the stack.

The stack is not used by any lexer methods, it's reserved for the user. Stacks are necessary for creating Abstract Syntax Trees (ASTs), but if you require this functionality it would be better to use a parser such as [snapdragon-parser][snapdragon-parser], with methods and other conveniences for creating an AST.

### lexer.queue

Type: **{array}** 

Default: `[]` 

FIFO (first in, first out) array, for temporarily storing tokens that are created when [.lookahead()](#lookahead) is called (or a method that calls `.lookhead()`, such as [.peek()](#peek)). 

Tokens are [dequeued](#dequeue) when [.next()](#next) is called.

### lexer.loc

Type: **{Object}** 

Default: `{ index: 0, column: 0, line: 1 }`

The updated source string location with the following properties.

- `index` - 0-index
- `column` - 0-index
- `line` - 1-index

The following plugins are available for automatically updating tokens with the location:

- [snapdragon-location][] 
- [snapdragon-position][]

## Options

### options.source

Type: **{string}** 

Default: `undefined`

The source of the input string. This is typically a filename or file path, but can also be `'string'` if a string or buffer is provided directly.

If `lexer.input` is undefined, and `options.source` is a string, the lexer will attempt to set `lexer.input` by calling `fs.readFileSync()` on the value provided on `options.source`.

### options.mode

Type: **{string}** 

Default: `undefined`

If `options.mode` is `character`, instead of calling handlers (which match using regex) the [.advance()](advance) method will [consume](#consume) and return one character at a time.

### options.value

Type: **{string}** 

Default: `undefined`

Specify the token property to use when the [.push](#push) method pushes a value onto [lexer.stash](#lexerstash). The logic works something like this:

```js
lexer.append(token[lexer.options.value || 'value']);
```

## Tokens

See the [snapdragon-token][] documentation for more details.

## Plugins

Plugins are registered with the `lexer.use()` method and use the following conventions.

### Plugin Conventions

Plugins are functions that take an instance of snapdragon-lexer. 

However, it's recommended that you always wrap your plugin function in another function that takes an options object. This allow users to pass options when using the plugin. _Even if your plugin doesn't take options, it's a best practice for users to always be able to use the same signature_.

**Example**

```js
function plugin(options) {
  return function(lexer) {
    // do stuff 
  };
}

lexer.use(plugin());
```