Parsing Program Source
**********************

Emacs provides various ways to parse program source text and produce a syntax tree.  In a syntax tree, text is no longer a one-dimensional stream but a structured tree of nodes, where each node representing a piece of text.  Thus a syntax tree can enable interesting features like precise fontification, indentation, navigation, structured editing, etc.

Emacs has a simple facility for parsing balanced expressions (see Parsing Expressions).  There is also SMIE library for generic navigation and indentation (see SMIE).

Emacs also provides integration with tree-sitter library (https://tree-sitter.github.io/tree-sitter) if compiled with it.  The tree-sitter library implements an incremental parser and has support from a wide range of programming languages.

 -- Function: tree-sitter-available-p
     This function returns non-nil if tree-sitter features are available for this Emacs instance.

For using tree-sitter features in font-lock and indentation, see Parser-based Font Lock, see Parser-based Indentation.

To access the syntax tree of the text in a buffer, we need to first load a language definition and create a parser with it.  Next, we can query the parser for specific nodes in the syntax tree.  Then, we can access various information about the node, and we can pattern-match a node with a powerful syntax.  Finally, we explain how to work with source files that mixes multiple languages.  The following sections explain how to do each of the tasks in detail.

Tree-sitter Language Definitions
================================

Loading a language definition
=============================

Tree-sitter relies on language definitions to parse text in that language. In Emacs, A language definition is represented by a symbol tree-sitter-<language>.  For example, C language definition is represented as tree-sitter-c, and tree-sitter-c can be passed to tree-sitter functions as the LANGUAGE argument.

Tree-sitter language definitions are distributed as dynamic libraries. In order to use a language definition in Emacs, you need to make sure that the dynamic library is installed on the system, either in standard locations or in LD_LIBRARY_PATH (on some systems, it is DYLD_LIBRARY_PATH).  If Emacs cannot find the library or has problem loading it, Emacs signals TREE-SITTER-LOAD-LANGUAGE-ERROR.  The signal data is a list of specific error messages.

 -- Function: tree-sitter-language-available-p language
     This function checks whether the dynamic library for LANGUAGE is present on the system, and return non-nil if it is.

By convention, the dynamic library for tree-sitter-<language> is libtree-sitter-<language>.EXT, where EXT is the system-specific extension for dynamic libraries. Also by convention, the function provided by that library is named tree_sitter_<language>.  If a language definition doesn't follow this convention, you should add an entry

     (LANGUAGE-SYMBOL LIBRARY-BASE-NAME FUNCTION-NAME)

to TREE-SITTER-LOAD-NAME-OVERRIDE-LIST, where LIBRARY-BASE-NAME is the base filename for the dynamic library (conventionally libtree-sitter-<language>), and FUNCTION-NAME is the function provided by the library (conventionally tree_sitter_<language>). For example,

     (tree-sitter-cool-lang "libtree-sitter-coool" "tree_sitter_coool")

for a language too cool to abide by the rules.

Concrete syntax tree
====================

A syntax tree is what a language definition defines (more or less) and what a parser generates.  In a syntax tree, each node represents a piece of text, and is connected to each other by a parent-child relationship.  For example, if the source text is

     1 + 2

its syntax tree could be

                       +--------------+
                       | root "1 + 2" |
                       +--------------+
                              |
             +--------------------------------+
             |       expression "1 + 2"       |
             +--------------------------------+
                |             |            |
     +------------+   +--------------+   +------------+
     | number "1" |   | operator "+" |   | number "2" |
     +------------+   +--------------+   +------------+

We can also represent it in s-expression:

     (root (expression (number) (operator) (number)))

Node types
----------

Names like root, expression, number, operator are nodes' type.  However, not all nodes in a syntax tree have a type.  Nodes that don't are anonymous nodes, and nodes with a type are named nodes.  Anonymous nodes are tokens with fixed spellings, including punctuation characters like bracket ], and keywords like return.

Field names
-----------

 To make the syntax tree easier to analyze, many language definitions assign field names to child nodes.  For example, a function_definition node could have a declarator and a body:

     (function_definition
      declarator: (declaration)
      body: (compound_statement))

 -- Command: tree-sitter-inspect-mode
     This minor mode displays the node that starts at point in mode-line.  The mode-line will display

          PARENT FIELD-NAME: (CHILD (GRAND-CHILD (...)))

     CHILD, GRAND-CHILD, and GRAND-GRAND-CHILD, etc, are nodes that have their beginning at point.  And PARENT is the parent of CHILD.

     If there is no node that starts at point, i.e., point is in the middle of a node, then the mode-line only displays the smallest node that spans point, and its immediate parent.

     This minor mode doesn't create parsers on its own.  It simply uses the first parser in TREE-SITTER-PARSER-LIST (see Using Parser).

Reading the grammar definition
==============================

Authors of language definitions define the grammar of a language, and this grammar determines how does a parser construct a concrete syntax tree out of the text.  In order to used the syntax tree effectively, we need to read the grammar file.

The grammar file is usually grammar.js in a language definition’s project repository.  The link to a language definition’s home page can be found in tree-sitter’s homepage (https://tree-sitter.github.io/tree-sitter).

The grammar is written in JavaScript syntax.  For example, the rule matching a function_definition node looks like

     function_definition: $ => seq(
       $.declaration_specifiers,
       field('declarator', $.declaration),
       field('body', $.compound_statement)
     )

The rule is represented by a function that takes a single argument $, representing the whole grammar.  The function itself is constructed by other functions: the seq function puts together a sequence of children; the field function annotates a child with a field name.  If we write the above definition in BNF syntax, it would look like

     function_definition :=
       <declaration_specifiers> <declaration> <compound_statement>

and the node returned by the parser would look like

     (function_definition
       (declaration_specifier)
       declarator: (declaration)
       body: (compound_statement))

Below is a list of functions that one will see in a grammar definition.  Each function takes other rules as arguments and returns a new rule.

  * seq(rule1, rule2, ...) matches each rule one after another.

  * choice(rule1, rule2, ...) matches one of the rules in its arguments.

  * repeat(rule) matches RULE for zero or more times. This is like the * operator in regular expressions.

  * repeat1(rule) matches RULE for one or more times. This is like the + operator in regular expressions.

  * optional(rule) matches RULE for zero or one time. This is like the ? operator in regular expressions.

  * field(name, rule) assigns field name NAME to the child node matched by RULE.

  * alias(rule, alias) makes nodes matched by RULE appear as ALIAS in the syntax tree generated by the parser.  For example,

          alias(preprocessor_call_exp, call_expression)

     makes any node matched by preprocessor_call_exp to appear as call_expression.

Below are grammar functions less interesting for a reader of a language definition.

  * token(rule) marks RULE to produce a single leaf node. That is, instead of generating a parent node with individual child nodes under it, everything is combined into a single leaf node.

  * Normally, grammar rules ignore preceding whitespaces, token.immediate(rule) changes RULE to match only when there is no preceding whitespaces.

  * prec(n, rule) gives RULE a level N precedence.

  * prec.left([n,] rule) marks RULE as left-associative, optionally with level N.

  * prec.right([n,] rule) marks RULE as right-associative, optionally with level N.

  * prec.dynamic(n, rule) is like prec, but the precedence is applied at runtime instead.

The tree-sitter project talks about writing a grammar in more detail: https://tree-sitter.github.io/tree-sitter/creating-parsers. Read especially ``The Grammar DSL'' section.

Using Tree-sitter Parser
========================

This section described how to create and configure a tree-sitter parser.  In Emacs, each tree-sitter parser is associated with a buffer.  As we edit the buffer, the associated parser is automatically kept up-to-date.

 -- Variable: tree-sitter-disabled-modes
     Before creating a parser, it is perhaps good to check whether we should use tree-sitter at all.  Sometimes a user don't want to use tree-sitter features for a major mode.  To turn-off tree-sitter for a mode, they add that mode to this variable.

 -- Variable: tree-sitter-maximum-size
     If users want to turn off tree-sitter for buffers larger than a particular size (because tree-sitter consumes memory ~10 times the buffer size for storing the syntax tree), they set this variable to that size.

 -- Function: tree-sitter-should-enable-p &optional mode
     This function returns non-nil if MODE (default to the current major mode) should activate tree-sitter features.  The result depends on the value of TREE-SITTER-DISABLED-MODES and TREE-SITTER-MAXIMUM-SIZE described above.  The result also depends on, of course, the result of tree-sitter-avaliabe-p.

     Writer of major modes or other packages are responsible for calling this function and determine whether to activate tree-sitter features.

To create a parser, we provide a buffer to parse and the language to use (see Language Definitions).  Emacs provides several creation functions for different use cases.

 -- Function: tree-sitter-get-parser-create language
     This function is the most convenient one.  It gives you a parser that recognizes LANGUAGE for the current buffer.  The function checks if there already exists a parser suiting the need, and only creates a new one when it can't find one.

          ;; Create a parser for C programming language.
          (tree-sitter-get-parser-create 'tree-sitter-c)

 -- Function: tree-sitter-get-parser language
     This function is like tree-sitter-get-parser-create, but it always creates a new parser.

 -- Function: tree-sitter-parser-create buffer language
     This function is the most primitive, requiring both the buffer to associate to, and the language to use.  If BUFFER is nil, the current buffer is used.

Given a parser, we can query information about it:

 -- Function: tree-sitter-parser-buffer parser
     Returns the buffer associated with PARSER.

 -- Function: tree-sitter-parser-language parser
     Returns the language that PARSER uses.

 -- Function: tree-sitter-parser-p object
     Checks if OBJECT is a tree-sitter parser. Return non-nil if it is, return nil otherwise.

There is no need to explicitly parse a buffer, because parsing is done automatically and lazily.  A parser only parses when we query for a node in its syntax tree.  Therefore, when a parser is first created, it doesn't parse the buffer; instead, it waits until we query for a node for the first time.  Similarly, when some change is made in the buffer, a parser doesn't re-parse immediately and only records some necessary information to later re-parse when necessary.

When a parser do parse, it checks for the size of the buffer. Tree-sitter can only handle buffer no larger than about 4GB.  If the size exceeds that, Emacs signals TREE-SITTER-BUFFER-TOO-LARGE with signal data being the buffer size.

Once a parser is created, Emacs automatically adds it to the buffer-local variable TREE-SITTER-PARSER-LIST.  Every time a change is made to the buffer, Emacs updates parsers in this list so they can update their syntax tree incrementally.  Therefore, one must not remove parsers from this list and put the parser back in: if any change is made when that parser is absent, the parser will be permanently out-of-sync with the buffer content, and shouldn't be used anymore.

 Normally, a parser ``sees'' the whole buffer, but when the buffer is narrowed (see Narrowing), the parser will only see the visible region.  As far as the parser can tell, the hidden region is deleted.  And when the buffer is later widened, the parser thinks text is inserted in the beginning and in the end.  Although parsers respect narrowing, narrowing shouldn't be the mean to handle a multi-language buffer; instead, set the ranges in which a parser should operate in.  See Multiple Languages.

Because a parser parses lazily, when we narrow the buffer, the parser doesn't act immediately; as long as we don't query for a node while the buffer is narrowed, narrowing does not affect the parser.

 -- Function: tree-sitter-parse-string string language
     Besides creating a parser for a buffer, we can also just parse a string.  Unlike a buffer, parsing a string is a one-time deal, and there is no way to update the result.

     This function parses STRING with LANGUAGE, and returns the root node of the generated syntax tree.

Retrieving Node
===============

There are two ways to retrieve a node: directly from the syntax tree, or by traveling from other nodes.  But before we continue, lets go over some conventions of tree-sitter functions.

We talk about a node being ``smaller'' or ``larger'', and ``lower'' or ``higher''.  A smaller and lower node is lower in the syntax tree and therefore spans a smaller piece of text; a larger and higher node is higher up in the syntax tree, containing many smaller nodes as its children, and therefore spans a larger piece of text.

When a function cannot find a node, it returns nil.  And for the convenience for function chaining, all the functions that take a node as argument and returns a node accept the node to be nil; in that case, the function just returns nil.

Nodes are not automatically updated when the associated buffer is modified.  In fact, there is no way to update a node once it is retrieved.  It is best to use a node and throw it away and not save it.  A node is outdated if the buffer has changed since the node is retrieved.  Using an outdated node throws TREE-SITTER-NODE-OUTDATED error.

Retrieving node from syntax tree
================================

 -- Function: tree-sitter-node-at beg &optional end parser-or-lang named
     This function returns the smallest node that covers the span from BEG to END.  In other words, the start of the node <= BEG, and the end of the node >= END.  If END is omitted, it defaults to the value of BEG.

     When PARSER-OR-LANG is nil, this function uses the first parser in TREE-SITTER-PARSER-LIST in the current buffer.  If PARSER-OR-LANG is a parser object, it use that parser; if PARSER-OR-LANG is a language, it finds the first parser using that language in TREE-SITTER-PARSER-LIST and use that.

     If NAMED is non-nil, this function looks for a named node instead (see named node).

          ;; Find the node at point in a C parser's syntax tree.
          (tree-sitter-node-at (point) (point) 'tree-sitter-c)

 -- Function: tree-sitter-parser-root-node parser
     This function returns the root node of the syntax tree generated by PARSER.

 -- Function: tree-sitter-buffer-root-node &optional language
     This function finds the first parser that uses LANGUAGE in TREE-SITTER-PARSER-LIST in the current buffer, and returns the root node of that buffer.  If it cannot find an appropriate parser, it returns nil.

Once we have a node, we can retrieve other nodes from it, or query for information about this node.

Retrieving node from other nodes
================================

By kinship
----------

 -- Function: tree-sitter-node-parent node
     This function returns the immediate parent of NODE.

 -- Function: tree-sitter-node-child node n &optional named
     This function returns the N'th child of NODE.  If NAMED is non-nil, then it only counts named nodes (see named node).  For example, in a node that represents a string: "text", there are three children nodes: the opening quote ", the string content text, and the enclosing quote ".  Among these nodes, the first child is the opening quote ", the first named child is the string content text.

 -- Function: tree-sitter-node-children node &optional named
     This function returns all of NODE's children in a list.  If NAMED is non-nil, then it only retrieves named nodes (see named node).

 -- Function: tree-sitter-next-sibling node &optional named
     This function finds the next sibling of NODE.  If NAMED is non-nil, it finds the next named sibling (see named node).

 -- Function: tree-sitter-prev-sibling node &optional named
     This function finds the previous sibling of NODE.  If NAMED is non-nil, it finds the previous named sibling (see named node).

By field name
-------------

To make the syntax tree easier to analyze, many language definitions assign field names to child nodes (see field name).  For example, a function_definition node could have a declarator and a body.

 -- Function: tree-sitter-child-by-field-name node field-name
     This function finds the child of NODE that has FIELD-NAME as its field name.

          ;; Get the child that has "body" as its field name.
          (tree-sitter-child-by-field-name node "body")

By position
-----------

 -- Function: tree-sitter-first-child-for-pos node pos &optional named
     This function finds the first child of NODE that extends beyond POS.  ``Extend beyond'' means the end of the child node >= POS.  This function only looks for immediate children of NODE, and doesn't look in its grand children.  If NAMED is non-nil, it only looks for named child (see named node).

 -- Function: tree-sitter-node-descendant-for-range node beg end &optional named
     This function finds the smallest (grand)child of NODE that spans the range from BEG to END.  It is similar to tree-sitter-node-at.  If NAMED is non-nil, it only looks for named child (see named node).

More convenient functions
=========================

 -- Function: tree-sitter-filter-child node pred &optional named
     This function finds children of NODE that satisfies PRED.

     Function PRED takes the child node as the argument and should return non-nil to indicated keeping the child.  If NAMED non-nil, this function only searches for named nodes."

 -- Function: tree-sitter-parent-until node pred
     This function repeatedly finds the parent of NODE, and returns the parent if it satisfies PRED (which takes the parent as the argument).  If no parent satisfies PRED, this function returns nil.

 -- Function: tree-sitter-parent-while
     This function repeatedly finds the parent of NODE, and keeps doing so as long as the parent satisfies PRED (which takes the parent as the single argument).  I.e., this function returns the farthest parent that still satisfies PRED.

Accessing Node Information
==========================

Before going further, make sure you have read the basic conventions about tree-sitter nodes in the previous node.

Basic information
=================

Every node is associated with a parser, and that parser is associated with a buffer.  The following functions let you retrieve them.

 -- Function: tree-sitter-node-parser node
     This function returns NODE's associated parser.

 -- Function: tree-sitter-node-buffer node
     This function returns NODE's parser's associated buffer.

 -- Function: tree-sitter-node-language node
     This function returns NODE's parser's associated language.

Each node represents a piece of text in the buffer.  Functions below finds relevant information about that text.

 -- Function: tree-sitter-node-start node
     Return the start position of NODE.

 -- Function: tree-sitter-node-end node
     Return the end position of NODE.

 -- Function: tree-sitter-node-text node &optional object
     Returns the buffer text that NODE represents.  (If NODE is retrieved from parsing a string, it will be the text from that string.)

Here are some basic checks on tree-sitter nodes.

 -- Function: tree-sitter-node-p object
     Checks if OBJECT is a tree-sitter syntax node.

 -- Function: tree-sitter-node-eq node1 node2
     Checks if NODE1 and NODE2 are the same node in a syntax tree.

Property information
====================

In general, nodes in a concrete syntax tree fall into two categories: named nodes and anonymous nodes.  Whether a node is named or anonymous is determined by the language definition (see named node).

Apart from being named/anonymous, a node can have other properties.  A node can be ``missing'': missing nodes are inserted by the parser in order to recover from certain kinds of syntax errors, i.e., something should probably be there according to the grammar, but not there.

A node can be ``extra'': extra nodes represent things like comments, which can appear anywhere in the text.

A node ``has changes'' if the buffer changed since when the node is retrieved.  In this case, the node's start and end position would be off and we better throw it away and retrieve a new one.

A node ``has error'' if the text it spans contains a syntax error.  It can be the node itself has an error, or one of its (grand)children has an error.

 -- Function: tree-sitter-node-check node property
     This function checks if NODE has PROPERTY.  PROPERTY can be 'named, 'missing, 'extra, 'has-changes, or 'has-error.

Named nodes have ``types'' (see node type). For example, a named node can be a string_literal node, where string_literal is its type.

 -- Function: tree-sitter-node-type node
     Return NODE's type as a string.

Information as a child or parent
================================

 -- Function: tree-sitter-node-index node &optional named
     This function returns the index of NODE as a child node of its parent.  If NAMED is non-nil, it only count named nodes (see named node).

 -- Function: tree-sitter-node-field-name node
     A child of a parent node could have a field name (see field name).  This function returns the field name of NODE as a child of its parent.

 -- Function: tree-sitter-node-field-name-for-child node n
     This is a more primitive function that returns the field name of the N'th child of NODE.

 -- Function: tree-sitter-child-count node &optional named
     This function finds the number of children of NODE.  If NAMED is non-nil, it only counts named child (see named node).

Pattern Matching Tree-sitter Nodes
==================================

Tree-sitter let us pattern match with a small declarative language. Pattern matching consists of two steps: first tree-sitter matches a pattern against nodes in the syntax tree, then it captures specific nodes in that pattern and returns the captured nodes.

We describe first how to write the most basic query pattern and how to capture nodes in a pattern, then the pattern-match function, finally more advanced pattern syntax.

Basic query syntax
==================

A query consists of multiple patterns, each pattern is an s-expression that matches a certain node in the syntax node.  A pattern has the following shape:

     (TYPE CHILD...)

For example, a pattern that matches a binary_expression node that contains number_literal child nodes would look like

     (binary_expression (number_literal))

To capture a node in the query pattern above, append @capture-name after the node pattern you want to capture.  For example,

     (binary_expression (number_literal) @number-in-exp)

captures number_literal nodes that are inside a binary_expression node with capture name number-in-exp.

We can capture the binary_expression node too, with capture name biexp:

     (binary_expression
      (number_literal) @number-in-exp) @biexp

Query function
==============

Now we can introduce the query functions.

 -- Function: tree-sitter-query-capture node query &optional beg end
     This function matches patterns in QUERY in NODE. Argument QUERY can be a either string or a s-expression.  For now, we focus on the string syntax; s-expression syntax is described at the end of the section.

     The function returns all captured nodes in a list of (CAPTURE_NAME . NODE).  If BEG and END are both non-nil, it only pattern matches nodes in that range.

     This function raise a TREE-SITTER-QUERY-ERROR if QUERY is malformed.  The signal data contains a description of the specific error.

 -- Function: tree-sitter-query-in source query &optional beg end
     This function matches patterns in QUERY in SOURCE, and returns all captured nodes in a list of (CAPTURE_NAME . NODE).  If BEG and END are both non-nil, it only pattern match nodes in that range.

     Argument SOURCE designates a node, it can be a language symbol, a parser, or simply a node.  If a language symbol, SOURCE represents the root node of the first parser for that language in the current buffer; if a parser, SOURCE represents the root node of that parser.

     This function also raises TREE-SITTER-QUERY-ERROR.

For example, suppose NODE's content is 1 + 2, and QUERY is

     (setq query
           "(binary_expression
             (number_literal) @number-in-exp) @biexp")

Querying that query would return

     (tree-sitter-query-capture node query)
         => ((biexp . <NODE FOR "1 + 2">)
            (number-in-exp . <NODE FOR "1">)
            (number-in-exp . <NODE FOR "2">))

As we mentioned earlier, a QUERY could contain multiple patterns. For example, it could have two top-level patterns:

     (setq query
           "(binary_expression) @biexp
            (number_literal)  @number @biexp")

 -- Function: tree-sitter-query-string string query language
     This function parses STRING with LANGUAGE, pattern matches its root node with QUERY, and returns the result.

More query syntax
=================

Besides node type and capture, tree-sitter's query syntax can express anonymous node, field name, wildcard, quantification, grouping, alternation, anchor, and predicate.

Anonymous node
--------------

An anonymous node is written verbatim, surrounded by quotes.  A pattern matching (and capturing) keyword return would be

     "return" @keyword

Wild card
---------

In a query pattern, (_) matches any named node, and _ matches any named and anonymous node.  For example, to capture any named child of a binary_expression node, the pattern would be

     (binary_expression (_) @in_biexp)

Field name
----------

We can capture child nodes that has specific field names:

     (function_definition
       declarator: (_) @func-declarator
       body: (_) @func-body)

We can also capture a node that doesn't have certain field, say, a function_definition without a body field.

     (function_definition !body) @func-no-body

Quantify node
-------------

Tree-sitter recognizes quantification operators *, + and ?.  Their meanings are the same as in regular expressions: * matches the preceding pattern zero or more times, + matches one or more times, and ? matches zero or one time.

For example, this pattern matches type_declaration nodes that has zero or more long keyword.

     (type_declaration "long"* @long-in-type)

And this pattern matches a type declaration that has zero or one long keyword:

     (type_declaration "long"?) @type-decl

Grouping
--------

Similar to groups in regular expression, we can bundle patterns into a group and apply quantification operators to it.  For example, to express a comma separated list of identifiers, one could write

     (identifier) ("," (identifier))*

Alternation
-----------

Again, similar to regular expressions, we can express ``match anyone from this group of patterns'' in the query pattern.  The syntax is a list of patterns enclosed in square brackets.  For example, to capture some keywords in C, the query pattern would be

     [
       "return"
       "break"
       "if"
       "else"
     ] @keyword

Anchor
------

The anchor operator . can be used to enforce juxtaposition, i.e., to enforce two things to be directly next to each other.  The two ``things'' can be two nodes, or a child and the end of its parent. For example, to capture the first child, the last child, or two adjacent children:

     ;; Anchor the child with the end of its parent.
     (compound_expression (_) @last-child .)

     ;; Anchor the child with the beginning of its parent.
     (compound_expression . (_) @first-child)

     ;; Anchor two adjacent children.
     (compound_expression
      (_) @prev-child
      .
      (_) @next-child)

Note that the enforcement of juxtaposition ignores any anonymous nodes.

Predicate
---------

We can add predicate constraints to a pattern.  For example, if we use the following query pattern

     (
      (array . (_) @first (_) @last .)
      (#equal @first @last)
     )

Then tree-sitter only matches arrays where the first element equals to the last element.  To attach a predicate to a pattern, we need to group then together.  A predicate always starts with a #. Currently there are two predicates, #equal and #match.

 -- Predicate: equal arg1 arg2
     Matches if ARG1 equals to ARG2.  Arguments can be either a string or a capture name.  Capture names represent the text that the captured node spans in the buffer.

 -- Predicate: match regexp capture-name
     Matches if the text that CAPTURE-NAME’s node spans in the buffer matches regular expression REGEXP.  Matching is case-sensitive.

Note that a predicate can only refer to capture names appeared in the same pattern.  Indeed, it makes little sense to refer to capture names in other patterns anyway.

S-expression patterns
=====================

Besides strings, Emacs provides a s-expression based syntax for query patterns. It largely resembles the string-based syntax.  For example, the following pattern

     (tree-sitter-query-capture
      node "(addition_expression
             left: (_) @left
             \"+\" @plus-sign
             right: (_) @right) @addition

             [\"return\" \"break\"] @keyword")

is equivalent to

     (tree-sitter-query-capture
      node '((addition_expression
              left: (_) @left
              "+" @plus-sign
              right: (_) @right) @addition

              ["return" "break"] @keyword))

Most pattern syntax can be written directly as strange but never-the-less valid s-expressions.  Only a few of them needs modification:

  * Anchor . is written as :anchor.

  * ? is written as :?.

  * * is written as :*.

  * + is written as :+.

  * #equal is written as :equal.  In general, predicates change their # to :.

For example,

     "(
       (compound_expression . (_) @first (_)* @rest)
       (#match \"love\" @first)
       )"

is written in s-expression as

     '((
        (compound_expression :anchor (_) @first (_) :* @rest)
        (:match "love" @first)
        ))

 -- Function: tree-sitter-expand-query query
     This function expands the s-expression QUERY into a string query.  It is usually a good idea to expand the s-expression patterns into strings for font-lock queries since they are called repeatedly.

Tree-sitter project's documentation about pattern-matching can be found at https://tree-sitter.github.io/tree-sitter/using-parsers#pattern-matching-with-queries.

Parsing Text in Multiple Languages
==================================

Sometimes, the source of a programming language could contain sources of other languages, HTML + CSS + JavaScript is one example.  In that case, we need to assign individual parsers to text segments written in different languages.  Traditionally this is achieved by using narrowing.  While tree-sitter works with narrowing (see narrowing), the recommended way is to set ranges in which a parser will operate.

 -- Function: tree-sitter-parser-set-included-ranges parser ranges
     This function sets the range of PARSER to RANGES.  Then PARSER will only read the text covered in each range.  Each range in RANGES is a list of cons (BEG . END).

     Each range in RANGES must come in order and not overlap.  That is, in pseudo code:

          (cl-loop for idx from 1 to (1- (length ranges))
                   for prev = (nth (1- idx) ranges)
                   for next = (nth idx ranges)
                   should (<= (car prev) (cdr prev)
                              (car next) (cdr next)))

     If RANGES violates this constraint, or something else went wrong, this function signals a TREE-SITTER-RANGE-INVALID.  The signal data contains a specific error message and the ranges we are trying to set.

     This function can also be used for disabling ranges.  If RANGES is nil, the parser is set to parse the whole buffer.

     Example:

          (tree-sitter-parser-set-included-ranges
           parser '((1 . 9) (16 . 24) (24 . 25)))

 -- Function: tree-sitter-parser-included-ranges parser
     This function returns the ranges set for PARSER.  The return value is the same as the RANGES argument of tree-sitter-parser-included-ranges: a list of cons (BEG . END).  And if PARSER doesn't have any ranges, the return value is nil.

          (tree-sitter-parser-included-ranges parser)
              => ((1 . 9) (16 . 24) (24 . 25))

 -- Function: tree-sitter-set-ranges parser-or-lang ranges
     Like tree-sitter-parser-set-included-ranges, this function sets the ranges of PARSER-OR-LANG to RANGES.  Conveniently, PARSER-OR-LANG could be either a parser or a language.  If it is a language, this function looks for the first parser in TREE-SITTER-PARSER-LIST for that language in the current buffer, and set range for it.

 -- Function: tree-sitter-get-ranges parser-or-lang
     This function returns the ranges of PARSER-OR-LANG, like tree-sitter-parser-included-ranges.  And like tree-sitter-set-ranges, PARSER-OR-LANG can be a parser or a language symbol.

 -- Function: tree-sitter-query-range source pattern &optional beg end
     This function matches SOURCE with PATTERN and returns the ranges of captured nodes.  The return value has the same shape of other functions: a list of (BEG . END).

     For convenience, SOURCE can be a language symbol, a parser, or a node.  If a language symbol, this function matches in the root node of the first parser using that language; if a parser, this function matches in the root node of that parser; if a node, this function matches in that node.

     Parameter PATTERN is the query pattern used to capture nodes (see Pattern Matching). The capture names don't matter.  Parameter BEG and END, if both non-nil, limits the range in which this function queries.

     Like other query functions, this function raises an TREE-SITTER-QUERY-ERROR if PATTERN is malformed.

 -- Function: tree-sitter-language-at point
     This function tries to figure out which language is responsible for the text at POINT.  It goes over each parser in TREE-SITTER-PARSER-LIST and see if that parser's range covers POINT.

 -- Variable: tree-sitter-range-functions
     A list of range functions.  Font-locking and indenting code uses functions in this alist to set correct ranges for a language parser before using it.

     The signature of each function should be

          (START END &rest _)

     where START and END marks the region that is about to be used.  A range function only need to (but not limited to) update ranges in that region.

     Each function in the list is called in-order.

 -- Function: tree-sitter-update-ranges &optional start end
     This function is used by font-lock and indent to update ranges before using any parser.  Each range function in TREE-SITTER-RANGE-FUNCTIONS is called in-order.  Arguments START and END are passed to each range function.

An example
==========

Normally, in a set of languages that can be mixed together, there is a major language and several embedded languages. The major language parses the whole document, and skips the embedded languages. Then the parser for the major language knows the ranges of the embedded languages. So we first parse the whole document with the major language’s parser, set ranges for the embedded languages, then parse the embedded languages.

Suppose we want to parse a very simple document that mixes HTML, CSS and JavaScript:

     <html>
       <script>1 + 2</script>
       <style>body { color: "blue"; }</style>
     </html>

We first parse with HTML, then set ranges for CSS and JavaScript:

     ;; Create parsers.
     (setq html (tree-sitter-get-parser-create 'tree-sitter-html))
     (setq css (tree-sitter-get-parser-create 'tree-sitter-css))
     (setq js (tree-sitter-get-parser-create 'tree-sitter-javascript))

     ;; Set CSS ranges.
     (setq css-range
           (tree-sitter-query-range
            'tree-sitter-html
            "(style_element (raw_text) @capture)"))
     (tree-sitter-parser-set-included-ranges css css-range)

     ;; Set JavaScript ranges.
     (setq js-range
           (tree-sitter-query-range
            'tree-sitter-html
            "(script_element (raw_text) @capture)"))
     (tree-sitter-parser-set-included-ranges js js-range)

We use a query pattern (style_element (raw_text) @capture) to find CSS nodes in the HTML parse tree. For how to write query patterns, see Pattern Matching.

Tree-sitter C API Correspondence
================================

Emacs' tree-sitter integration doesn't expose every feature tree-sitter's C API provides.  Missing features include:

  * Creating a tree cursor and navigating the syntax tree with it.

  * Setting timeout and cancellation flag for a parser.

  * Setting the logger for a parser.

  * Printing a DOT graph of the syntax tree to a file.

  * Coping and modifying a syntax tree.  (Emacs doesn't expose a tree object.)

  * Using (row, column) coordinates as position.

  * Updating a node with changes. (In Emacs, retrieve a new node instead of updating the existing one.)

  * Querying statics of a language definition.

In addition, Emacs makes some changes to the C API to make the API more convenient and idiomatic:

  * Instead of using byte positions, the ELisp API uses character positions.

  * Null nodes are converted to nil.

Below is the correspondence between all C API functions and their ELisp counterparts.  Sometimes one ELisp function corresponds to multiple C functions, and many C functions don't have an ELisp counterpart.

     ts_parser_new                           tree-sitter-parser-create
     ts_parser_delete
     ts_parser_set_language
     ts_parser_language                      tree-sitter-parser-language
     ts_parser_set_included_ranges           tree-sitter-parser-set-included-ranges
     ts_parser_included_ranges               tree-sitter-parser-included-ranges
     ts_parser_parse
     ts_parser_parse_string                  tree-sitter-parse-string
     ts_parser_parse_string_encoding
     ts_parser_reset
     ts_parser_set_timeout_micros
     ts_parser_timeout_micros
     ts_parser_set_cancellation_flag
     ts_parser_cancellation_flag
     ts_parser_set_logger
     ts_parser_logger
     ts_parser_print_dot_graphs
     ts_tree_copy
     ts_tree_delete
     ts_tree_root_node
     ts_tree_language
     ts_tree_edit
     ts_tree_get_changed_ranges
     ts_tree_print_dot_graph
     ts_node_type                            tree-sitter-node-type
     ts_node_symbol
     ts_node_start_byte                      tree-sitter-node-start
     ts_node_start_point
     ts_node_end_byte                        tree-sitter-node-end
     ts_node_end_point
     ts_node_string                          tree-sitter-node-string
     ts_node_is_null
     ts_node_is_named                        tree-sitter-node-check
     ts_node_is_missing                      tree-sitter-node-check
     ts_node_is_extra                        tree-sitter-node-check
     ts_node_has_changes                     tree-sitter-node-check
     ts_node_has_error                       tree-sitter-node-check
     ts_node_parent                          tree-sitter-node-parent
     ts_node_child                           tree-sitter-node-child
     ts_node_field_name_for_child            tree-sitter-node-field-name-for-child
     ts_node_child_count                     tree-sitter-node-child-count
     ts_node_named_child                     tree-sitter-node-child
     ts_node_named_child_count               tree-sitter-node-child-count
     ts_node_child_by_field_name             tree-sitter-node-by-field-name
     ts_node_child_by_field_id
     ts_node_next_sibling                    tree-sitter-next-sibling
     ts_node_prev_sibling                    tree-sitter-prev-sibling
     ts_node_next_named_sibling              tree-sitter-next-sibling
     ts_node_prev_named_sibling              tree-sitter-prev-sibling
     ts_node_first_child_for_byte            tree-sitter-first-child-for-pos
     ts_node_first_named_child_for_byte      tree-sitter-first-child-for-pos
     ts_node_descendant_for_byte_range       tree-sitter-descendant-for-range
     ts_node_descendant_for_point_range
     ts_node_named_descendant_for_byte_range tree-sitter-descendant-for-range
     ts_node_named_descendant_for_point_range
     ts_node_edit
     ts_node_eq                              tree-sitter-node-eq
     ts_tree_cursor_new
     ts_tree_cursor_delete
     ts_tree_cursor_reset
     ts_tree_cursor_current_node
     ts_tree_cursor_current_field_name
     ts_tree_cursor_current_field_id
     ts_tree_cursor_goto_parent
     ts_tree_cursor_goto_next_sibling
     ts_tree_cursor_goto_first_child
     ts_tree_cursor_goto_first_child_for_byte
     ts_tree_cursor_goto_first_child_for_point
     ts_tree_cursor_copy
     ts_query_new
     ts_query_delete
     ts_query_pattern_count
     ts_query_capture_count
     ts_query_string_count
     ts_query_start_byte_for_pattern
     ts_query_predicates_for_pattern
     ts_query_step_is_definite
     ts_query_capture_name_for_id
     ts_query_string_value_for_id
     ts_query_disable_capture
     ts_query_disable_pattern
     ts_query_cursor_new
     ts_query_cursor_delete
     ts_query_cursor_exec                    tree-sitter-query-capture
     ts_query_cursor_did_exceed_match_limit
     ts_query_cursor_match_limit
     ts_query_cursor_set_match_limit
     ts_query_cursor_set_byte_range
     ts_query_cursor_set_point_range
     ts_query_cursor_next_match
     ts_query_cursor_remove_match
     ts_query_cursor_next_capture
     ts_language_symbol_count
     ts_language_symbol_name
     ts_language_symbol_for_name
     ts_language_field_count
     ts_language_field_name_for_id
     ts_language_field_id_for_name
     ts_language_symbol_type
     ts_language_version