<title>The Lemon Parser Generator</title>

<a id="main"></a>

<h1 align='center'>The Lemon Parser Generator</h1>

<p>Lemon is an LALR(1) parser generator for C.

It does the same job as "bison" and "yacc".

But Lemon is not a bison or yacc clone. Lemon

uses a different grammar syntax which is designed to

reduce the number of coding errors. Lemon also uses a

parsing engine that is faster than yacc and

bison and which is both reentrant and threadsafe.

(Update: Since the previous sentence was written, bison

has also been updated so that it too can generate a

reentrant and threadsafe parser.)

Lemon also implements features that can be used

to eliminate resource leaks, making it suitable for use

in long-running programs such as graphical user interfaces

or embedded controllers.</p>

<p>This document is an introduction to the Lemon

parser generator.</p>

<a id="toc"></a>

<h2>1.0 Table of Contents</h2>

<li><a href="#main">Introduction</a>

<li><a href="#toc">1.0 Table of Contents</a>

<li><a href="#secnot">2.0 Security Notes</a><br>

<li><a href="#optheory">3.0 Theory of Operation</a>

<ul>

<li><a href="#options">3.1 Command Line Options</a>

<li><a href="#interface">3.2 The Parser Interface</a>

<ul>

<li><a href="#onstack">3.2.1 Allocating The Parse Object On Stack</a>

<li><a href="#ifsum">3.2.2 Interface Summary</a>

</ul>

<li><a href="#yaccdiff">3.3 Differences With YACC and BISON</a>

<li><a href="#build">3.4 Building The "lemon" Or "lemon.exe" Executable</a>

</ul>

<li><a href="#syntax">4.0 Input File Syntax</a>

<ul>

<li><a href="#tnt">4.1 Terminals and Nonterminals</a>

<li><a href="#rules">4.2 Grammar Rules</a>

<li><a href="#precrules">4.3 Precedence Rules</a>

<li><a href="#special">4.4 Special Directives</a>

</ul>

<li><a href="#errors">5.0 Error Processing</a>

<li><a href="#history">6.0 History of Lemon</a>

<li><a href="#copyright">7.0 Copyright</a>

<a id="secnot"></a>

<h2>2.0 Security Note</h2>

<p>The language parser code created by Lemon is very robust and

is well-suited for use in internet-facing applications that need to

safely process maliciously crafted inputs.</p>

<p>The "lemon.exe" command-line tool itself works great when given a valid

input grammar file and almost always gives helpful

error messages for malformed inputs. However, it is possible for

a malicious user to craft a grammar file that will cause

lemon.exe to crash.

We do not see this as a problem, as lemon.exe is not intended to be used

with hostile inputs.

To summarize:</p>

<li>Parser code generated by lemon &rarr; Robust and secure

<li>The "lemon.exe" command line tool itself &rarr; Not so much

<a id="optheory"></a>

<h2>3.0 Theory of Operation</h2>

<p>Lemon is computer program that translates a context free grammar (CFG)

for a particular language into C code that implements a parser for

that language.

The Lemon program has two inputs:</p>

<li>The grammar specification.

<li>A parser template file.

<p>Typically, only the grammar specification is supplied by the programmer.

Lemon comes with a default parser template

("<a href="https://sqlite.org/src/file/tool/lempar.c">lempar.c</a>")

that works fine for most applications. But the user is free to substitute

a different parser template if desired.</p>

<p>Depending on command-line options, Lemon will generate up to

three output files.</p>

<li>C code to implement a parser for the input grammar.

<li>A header file defining an integer ID for each terminal symbol

(or "token").

<li>An information file that describes the states of the generated parser

automaton.

<p>By default, all three of these output files are generated.

The header file is suppressed if the "-m" command-line option is

used and the report file is omitted when "-q" is selected.</p>

<p>The grammar specification file uses a ".y" suffix, by convention.

In the examples used in this document, we'll assume the name of the

grammar file is "gram.y". A typical use of Lemon would be the

following command:</p>

lemon gram.y

<p>This command will generate three output files named "gram.c",

"gram.h" and "gram.out".

The first is C code to implement the parser. The second

is the header file that defines numerical values for all

terminal symbols, and the last is the report that explains

the states used by the parser automaton.</p>

<a id="options"></a>

<h3>3.1 Command Line Options</h3>

<p>The behavior of Lemon can be modified using command-line options.

You can obtain a list of the available command-line options together

with a brief explanation of what each does by typing</p>

lemon "-?"

<p>As of this writing, the following command-line options are supported:</p>

<li><b>-b</b>

Show only the basis for each parser state in the report file.

<li><b>-c</b>

Do not compress the generated action tables. The parser will be a

little larger and slower, but it will detect syntax errors sooner.

<li><b>-d</b><i>directory</i>

Write all output files into <i>directory</i>. Normally, output files

are written into the directory that contains the input grammar file.

<li><b>-D<i>name</i></b>

Define C preprocessor macro <i>name</i>. This macro is usable by

"<tt><a href='#pifdef'>%ifdef</a></tt>",

"<tt><a href='#pifdef'>%ifndef</a></tt>", and

"<tt><a href="#pifdef">%if</a></tt> lines

in the grammar file.

<li><b>-E</b>

Run the "%if" preprocessor step only and print the revised grammar

file.

<li><b>-g</b>

Do not generate a parser. Instead write the input grammar to standard

output with all comments, actions, and other extraneous text removed.

<li><b>-l</b>

Omit "#line" directives in the generated parser C code.

<li><b>-m</b>

Cause the output C source code to be compatible with the "makeheaders"

program.

<li><b>-p</b>

Display all conflicts that are resolved by

<a href='#precrules'>precedence rules</a>.

<li><b>-q</b>

Suppress generation of the report file.

<li><b>-r</b>

Do not sort or renumber the parser states as part of optimization.

<li><b>-s</b>

Show parser statistics before exiting.

<li><b>-T<i>file</i></b>

Use <i>file</i> as the template for the generated C-code parser implementation.

<li><b>-x</b>

Print the Lemon version number.

<a id="interface"></a>

<h3>3.2 The Parser Interface</h3>

<p>Lemon doesn't generate a complete, working program. It only generates

a few subroutines that implement a parser. This section describes

the interface to those subroutines. It is up to the programmer to

call these subroutines in an appropriate way in order to produce a

complete system.</p>

<p>Before a program begins using a Lemon-generated parser, the program

must first create the parser.

A new parser is created as follows:</p>

void *pParser = ParseAlloc( malloc );

<p>The ParseAlloc() routine allocates and initializes a new parser and

returns a pointer to it.

The actual data structure used to represent a parser is opaque &mdash;

its internal structure is not visible or usable by the calling routine.

For this reason, the ParseAlloc() routine returns a pointer to void

rather than a pointer to some particular structure.

The sole argument to the ParseAlloc() routine is a pointer to the

subroutine used to allocate memory. Typically this means malloc().</p>

<p>After a program is finished using a parser, it can reclaim all

memory allocated by that parser by calling</p>

ParseFree(pParser, free);

<p>The first argument is the same pointer returned by ParseAlloc(). The

second argument is a pointer to the function used to release bulk

memory back to the system.</p>

<p>After a parser has been allocated using ParseAlloc(), the programmer

must supply the parser with a sequence of tokens (terminal symbols) to

be parsed. This is accomplished by calling the following function

once for each token:<p>

Parse(pParser, hTokenID, sTokenData, pArg);

<p>The first argument to the Parse() routine is the pointer returned by

ParseAlloc().

The second argument is a small positive integer that tells the parser the

type of the next token in the data stream.

There is one token type for each terminal symbol in the grammar.

The gram.h file generated by Lemon contains #define statements that

map symbolic terminal symbol names into appropriate integer values.

A value of 0 for the second argument is a special flag to the

parser to indicate that the end of input has been reached.

The third argument is the value of the given token. By default,

the type of the third argument is "void*", but the grammar will

usually redefine this type to be some kind of structure.

Typically the second argument will be a broad category of tokens

such as "identifier" or "number" and the third argument will

be the name of the identifier or the value of the number.</p>

<p>The Parse() function may have either three or four arguments,

depending on the grammar. If the grammar specification file requests

it (via the <tt><a href='#extraarg'>%extra_argument</a></tt> directive),

the Parse() function will have a fourth parameter that can be

of any type chosen by the programmer. The parser doesn't do anything

with this argument except to pass it through to action routines.

This is a convenient mechanism for passing state information down

to the action routines without having to use global variables.</p>

<p>A typical use of a Lemon parser might look something like the

following:</p>

1 ParseTree *ParseFile(const char *zFilename){

2 Tokenizer *pTokenizer;

3 void *pParser;

4 Token sToken;

5 int hTokenId;

6 ParserState sState;

7

8 pTokenizer = TokenizerCreate(zFilename);

9 pParser = ParseAlloc( malloc );

10 InitParserState(&amp;sState);

11 while( GetNextToken(pTokenizer, &amp;hTokenId, &amp;sToken) ){

12 Parse(pParser, hTokenId, sToken, &amp;sState);

13 }

14 Parse(pParser, 0, sToken, &amp;sState);

15 ParseFree(pParser, free );

16 TokenizerFree(pTokenizer);

17 return sState.treeRoot;

18 }

<p>This example shows a user-written routine that parses a file of

text and returns a pointer to the parse tree.

(All error-handling code is omitted from this example to keep it

simple.)

We assume the existence of some kind of tokenizer which is created

using TokenizerCreate() on line 8 and deleted by TokenizerFree()

on line 16. The GetNextToken() function on line 11 retrieves the

next token from the input file and puts its type in the

integer variable hTokenId. The sToken variable is assumed to be

some kind of structure that contains details about each token,

such as its complete text, what line it occurs on, etc.</p>

<p>This example also assumes the existence of a structure of type

ParserState that holds state information about a particular parse.

An instance of such a structure is created on line 6 and initialized

on line 10. A pointer to this structure is passed into the Parse()

routine as the optional 4th argument.

The action routine specified by the grammar for the parser can use

the ParserState structure to hold whatever information is useful and

appropriate. In the example, we note that the treeRoot field of

the ParserState structure is left pointing to the root of the parse

tree.</p>

<p>The core of this example as it relates to Lemon is as follows:</p>

ParseFile(){

pParser = ParseAlloc( malloc );

while( GetNextToken(pTokenizer,&amp;hTokenId, &amp;sToken) ){

Parse(pParser, hTokenId, sToken);

}

Parse(pParser, 0, sToken);

ParseFree(pParser, free );

}

<p>Basically, what a program has to do to use a Lemon-generated parser

is first create the parser, then send it lots of tokens obtained by

tokenizing an input source. When the end of input is reached, the

Parse() routine should be called one last time with a token type

of 0. This step is necessary to inform the parser that the end of

input has been reached. Finally, we reclaim memory used by the

parser by calling ParseFree().</p>

<p>There is one other interface routine that should be mentioned

before we move on.

The ParseTrace() function can be used to generate debugging output

from the parser. A prototype for this routine is as follows:</p>

ParseTrace(FILE *stream, char *zPrefix);

<p>After this routine is called, a short (one-line) message is written

to the designated output stream every time the parser changes states

or calls an action routine. Each such message is prefaced using

the text given by zPrefix. This debugging output can be turned off

by calling ParseTrace() again with a first argument of NULL (0).</p>

<a id="onstack"></a>

<h4>3.2.1 Allocating The Parse Object On Stack</h4>

<p>If all calls to the Parse() interface are made from within

<a href="#pcode"><tt>%code</tt> directives</a>, then the parse

object can be allocated from the stack rather than from the heap.

These are the steps:

<li> Declare a local variable of type "yyParser"

<li> Initialize the variable using ParseInit()

<li> Pass a pointer to the variable in calls to Parse()

<li> Deallocate substructure in the parse variable using ParseFinalize().

<p>The following code illustrates how this is done:

ParseFile(){

yyParser x;

ParseInit( &x );

while( GetNextToken(pTokenizer,&amp;hTokenId, &amp;sToken) ){

Parse(&x, hTokenId, sToken);

}

Parse(&x, 0, sToken);

ParseFinalize( &x );

}

<a id="ifsum"></a>

<h4>3.2.2 Interface Summary</h4>

<p>Here is a quick overview of the C-language interface to a

Lemon-generated parser:</p>

void *ParseAlloc( (void*(*malloc)(size_t) );

void ParseFree(void *pParser, (void(*free)(void*) );

void Parse(void *pParser, int tokenCode, ParseTOKENTYPE token, ...);

void ParseTrace(FILE *stream, char *zPrefix);

<p>Notes:</p>

<li> Use the <a href="#pname"><tt>%name</tt> directive</a> to change

the "Parse" prefix names of the procedures in the interface.

<li> Use the <a href="#token_type"><tt>%token_type</tt> directive</a>

to define the "ParseTOKENTYPE" type.

<li> Use the <a href="#extraarg"><tt>%extra_argument</tt> directive</a>

to specify the type and name of the 4th parameter to the

Parse() function.

<a id="yaccdiff"></a>

<h3>3.3 Differences With YACC and BISON</h3>

<p>Programmers who have previously used the yacc or bison parser

generator will notice several important differences between yacc and/or

bison and Lemon.</p>

<li>In yacc and bison, the parser calls the tokenizer. In Lemon,

the tokenizer calls the parser.

<li>Lemon uses no global variables. Yacc and bison use global variables

to pass information between the tokenizer and parser.

<li>Lemon allows multiple parsers to be running simultaneously. Yacc

and bison do not.

<p>These differences may cause some initial confusion for programmers

with prior yacc and bison experience.

But after years of experience using Lemon, I firmly

believe that the Lemon way of doing things is better.</p>

<p><i>Updated as of 2016-02-16:</i>

The text above was written in the 1990s.

We are told that Bison has lately been enhanced to support the

tokenizer-calls-parser paradigm used by Lemon, eliminating the

need for global variables.</p>

<a id="build"><a>

<h3>3.4 Building The "lemon" or "lemon.exe" Executable</h3>

<p>The "lemon" or "lemon.exe" program is built from a single file

of C-code named

"<a href="https://sqlite.org/src/tool/lemon.c">lemon.c</a>".

The Lemon source code is generic C89 code that uses

no unusual or non-standard libraries. Any

reasonable C compiler should suffice to compile the lemon program.

A command-line like the following will usually work:</p>

cc -o lemon lemon.c

<p>On Windows machines with Visual C++ installed, bring up a

"VS20<i>NN</i> x64 Native Tools Command Prompt" window and enter:

cl lemon.c

<p>Compiling Lemon really is that simple.

Additional compiler options such as

"-O2" or "-g" or "-Wall" can be added if desired, but they are not

necessary.</p>

<a id="syntax"></a>

<h2>4.0 Input File Syntax</h2>

<p>The main purpose of the grammar specification file for Lemon is

to define the grammar for the parser. But the input file also

specifies additional information Lemon requires to do its job.

Most of the work in using Lemon is in writing an appropriate

grammar file.</p>

<p>The grammar file for Lemon is, for the most part, a free format.

It does not have sections or divisions like yacc or bison. Any

declaration can occur at any point in the file. Lemon ignores

whitespace (except where it is needed to separate tokens), and it

honors the same commenting conventions as C and C++.</p>

<a id="tnt"></a>

<h3>4.1 Terminals and Nonterminals</h3>

<p>A terminal symbol (token) is any string of alphanumeric

and/or underscore characters

that begins with an uppercase letter.

A terminal can contain lowercase letters after the first character,

but the usual convention is to make terminals all uppercase.

A nonterminal, on the other hand, is any string of alphanumeric

and underscore characters than begins with a lowercase letter.

Again, the usual convention is to make nonterminals use all lowercase

letters.</p>

<p>In Lemon, terminal and nonterminal symbols do not need to

be declared or identified in a separate section of the grammar file.

Lemon is able to generate a list of all terminals and nonterminals

by examining the grammar rules, and it can always distinguish a

terminal from a nonterminal by checking the case of the first

character of the name.</p>

<p>Yacc and bison allow terminal symbols to have either alphanumeric

names or to be individual characters included in single quotes, like

this: ')' or '$'. Lemon does not allow this alternative form for

terminal symbols. With Lemon, all symbols, terminals and nonterminals,

must have alphanumeric names.</p>

<a id="rules"></a>

<h3>4.2 Grammar Rules</h3>

<p>The main component of a Lemon grammar file is a sequence of grammar

rules.

Each grammar rule consists of a nonterminal symbol followed by

the special symbol "::=" and then a list of terminals and/or nonterminals.

The rule is terminated by a period.

The list of terminals and nonterminals on the right-hand side of the

rule can be empty.

Rules can occur in any order, except that the left-hand side of the

first rule is assumed to be the start symbol for the grammar (unless

specified otherwise using the <tt><a href='#start_symbol'>%start_symbol</a></tt>

directive described below.)

A typical sequence of grammar rules might look something like this:</p>

expr ::= expr PLUS expr.

expr ::= expr TIMES expr.

expr ::= LPAREN expr RPAREN.

expr ::= VALUE.

<p>There is one non-terminal in this example, "expr", and five

terminal symbols or tokens: "PLUS", "TIMES", "LPAREN",

"RPAREN" and "VALUE".</p>

<p>Like yacc and bison, Lemon allows the grammar to specify a block

of C code that will be executed whenever a grammar rule is reduced

by the parser.

In Lemon, this action is specified by putting the C code (contained

within curly braces <tt>{...}</tt>) immediately after the

period that closes the rule.

For example:</p>

expr ::= expr PLUS expr. { printf("Doing an addition...\n"); }

<p>In order to be useful, grammar actions must normally be linked to

their associated grammar rules.

In yacc and bison, this is accomplished by embedding a "$$" in the

action to stand for the value of the left-hand side of the rule and

symbols "$1", "$2", and so forth to stand for the value of

the terminal or nonterminal at position 1, 2 and so forth on the

right-hand side of the rule.

This idea is very powerful, but it is also very error-prone. The

single most common source of errors in a yacc or bison grammar is

to miscount the number of symbols on the right-hand side of a grammar

rule and say "$7" when you really mean "$8".</p>

<p>Lemon avoids the need to count grammar symbols by assigning symbolic

names to each symbol in a grammar rule and then using those symbolic

names in the action.

In yacc or bison, one would write this:</p>

expr -&gt; expr PLUS expr { $$ = $1 + $3; };

<p>But in Lemon, the same rule becomes the following:</p>

expr(A) ::= expr(B) PLUS expr(C). { A = B+C; }

<p>In the Lemon rule, any symbol in parentheses after a grammar rule

symbol becomes a place holder for that symbol in the grammar rule.

This place holder can then be used in the associated C action to

stand for the value of that symbol.</p>

<p>The Lemon notation for linking a grammar rule with its reduce

action is superior to yacc/bison on several counts.

First, as mentioned above, the Lemon method avoids the need to

count grammar symbols.

Secondly, if a terminal or nonterminal in a Lemon grammar rule

includes a linking symbol in parentheses but that linking symbol

is not actually used in the reduce action, then an error message

is generated.

For example, the rule</p>

expr(A) ::= expr(B) PLUS expr(C). { A = B; }

<p>will generate an error because the linking symbol "C" is used

in the grammar rule but not in the reduce action.</p>

<p>The Lemon notation for linking grammar rules to reduce actions

also facilitates the use of destructors for reclaiming memory

allocated by the values of terminals and nonterminals on the

right-hand side of a rule.</p>

<a id='precrules'></a>

<h3>4.3 Precedence Rules</h3>

<p>Lemon resolves parsing ambiguities in exactly the same way as

yacc and bison. A shift-reduce conflict is resolved in favor

of the shift, and a reduce-reduce conflict is resolved by reducing

whichever rule comes first in the grammar file.</p>

<p>Just like in

yacc and bison, Lemon allows a measure of control

over the resolution of parsing conflicts using precedence rules.

A precedence value can be assigned to any terminal symbol

using the

<tt><a href='#pleft'>%left</a></tt>,

<tt><a href='#pright'>%right</a></tt> or

<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives. Terminal symbols

mentioned in earlier directives have a lower precedence than

terminal symbols mentioned in later directives. For example:</p>

%left AND.

%left OR.

%nonassoc EQ NE GT GE LT LE.

%left PLUS MINUS.

%left TIMES DIVIDE MOD.

%right EXP NOT.

<p>In the preceding sequence of directives, the AND operator is

defined to have the lowest precedence. The OR operator is one

precedence level higher. And so forth. Hence, the grammar would

attempt to group the ambiguous expression</p>

a AND b OR c

<p>like this</p>

a AND (b OR c).

<p>The associativity (left, right or nonassoc) is used to determine

the grouping when the precedence is the same. AND is left-associative

in our example, so</p>

a AND b AND c

<p>is parsed like this</p>

(a AND b) AND c.

<p>The EXP operator is right-associative, though, so</p>

a EXP b EXP c

<p>is parsed like this</p>

a EXP (b EXP c).

<p>The nonassoc precedence is used for non-associative operators.

So</p>

a EQ b EQ c

<p>is an error.</p>

<p>The precedence of non-terminals is transferred to rules as follows:

The precedence of a grammar rule is equal to the precedence of the

left-most terminal symbol in the rule for which a precedence is

defined. This is normally what you want, but in those cases where

you want the precedence of a grammar rule to be something different,

you can specify an alternative precedence symbol by putting the

symbol in square braces after the period at the end of the rule and

before any C-code. For example:</p>

expr = MINUS expr. [NOT]

<p>This rule has a precedence equal to that of the NOT symbol, not the

MINUS symbol as would have been the case by default.</p>

<p>With the knowledge of how precedence is assigned to terminal

symbols and individual

grammar rules, we can now explain precisely how parsing conflicts

are resolved in Lemon. Shift-reduce conflicts are resolved

as follows:</p>

<li> If either the token to be shifted or the rule to be reduced

lacks precedence information, then resolve in favor of the

shift, but report a parsing conflict.

<li> If the precedence of the token to be shifted is greater than

the precedence of the rule to reduce, then resolve in favor

of the shift. No parsing conflict is reported.

<li> If the precedence of the token to be shifted is less than the

precedence of the rule to reduce, then resolve in favor of the

reduce action. No parsing conflict is reported.

<li> If the precedences are the same and the shift token is

right-associative, then resolve in favor of the shift.

No parsing conflict is reported.

<li> If the precedences are the same and the shift token is

left-associative, then resolve in favor of the reduce.

No parsing conflict is reported.

<li> Otherwise, resolve the conflict by doing the shift, and

report a parsing conflict.

<p>Reduce-reduce conflicts are resolved this way:</p>

<li> If either reduce rule

lacks precedence information, then resolve in favor of the

rule that appears first in the grammar, and report a parsing

conflict.

<li> If both rules have precedence and the precedence is different,

then resolve the dispute in favor of the rule with the highest

precedence, and do not report a conflict.

<li> Otherwise, resolve the conflict by reducing by the rule that

appears first in the grammar, and report a parsing conflict.

<a id="special"></a>

<h3>4.4 Special Directives</h3>

<p>The input grammar to Lemon consists of grammar rules and special

directives. We've described all the grammar rules, so now we'll

talk about the special directives.</p>

<p>Directives in Lemon can occur in any order. You can put them before

the grammar rules, or after the grammar rules, or in the midst of the

grammar rules. It doesn't matter. The relative order of

directives used to assign precedence to terminals is important, but

other than that, the order of directives in Lemon is arbitrary.</p>

<p>Lemon supports the following special directives:</p>

<li><tt><a href='#pcode'>%code</a></tt>

<li><tt><a href='#default_destructor'>%default_destructor</a></tt>

<li><tt><a href='#default_type'>%default_type</a></tt>

<li><tt><a href='#destructor'>%destructor</a></tt>

<li><tt><a href='#pifdef'>%else</a></tt>

<li><tt><a href='#pifdef'>%endif</a></tt>

<li><tt><a href='#extraarg'>%extra_argument</a></tt>

<li><tt><a href='#pfallback'>%fallback</a></tt>

<li><tt><a href='#reallc'>%free</a></tt>

<li><tt><a href='#pifdef'>%if</a></tt>

<li><tt><a href='#pifdef'>%ifdef</a></tt>

<li><tt><a href='#pifdef'>%ifndef</a></tt>

<li><tt><a href='#pinclude'>%include</a></tt>

<li><tt><a href='#pleft'>%left</a></tt>

<li><tt><a href='#pname'>%name</a></tt>

<li><tt><a href='#pnonassoc'>%nonassoc</a></tt>

<li><tt><a href='#parse_accept'>%parse_accept</a></tt>

<li><tt><a href='#parse_failure'>%parse_failure</a></tt>

<li><tt><a href='#pright'>%right</a></tt>

<li><tt><a href='#reallc'>%realloc</a></tt>

<li><tt><a href='#stack_overflow'>%stack_overflow</a></tt>

<li><tt><a href='#reallc'>%stack_size_limit</a></tt>

<li><tt><a href='#stack_size'>%stack_size</a></tt>

<li><tt><a href='#start_symbol'>%start_symbol</a></tt>

<li><tt><a href='#syntax_error'>%syntax_error</a></tt>

<li><tt><a href='#token'>%token</a></tt>

<li><tt><a href='#token_class'>%token_class</a></tt>

<li><tt><a href='#token_destructor'>%token_destructor</a></tt>

<li><tt><a href='#token_prefix'>%token_prefix</a></tt>

<li><tt><a href='#token_type'>%token_type</a></tt>

<li><tt><a href='#ptype'>%type</a></tt>

<li><tt><a href='#pwildcard'>%wildcard</a></tt>

<p>Each of these directives will be described separately in the

following sections:</p>

<a id='pcode'></a>

<h4>4.4.1 The <tt>%code</tt> directive</h4>

<p>The <tt>%code</tt> directive is used to specify additional C code that

is added to the end of the main output file. This is similar to

the <tt><a href='#pinclude'>%include</a></tt> directive except that

<tt>%include</tt> is inserted at the beginning of the main output file.</p>

<p><tt>%code</tt> is typically used to include some action routines or perhaps

a tokenizer or even the "main()" function

as part of the output file.</p>

<p>There can be multiple <tt>%code</tt> directives. The arguments of

all <tt>%code</tt> directives are concatenated.</p>

<a id='default_destructor'></a>

<h4>4.4.2 The <tt>%default_destructor</tt> directive</h4>

<p>The <tt>%default_destructor</tt> directive specifies a destructor to

use for non-terminals that do not have their own destructor

specified by a separate <tt>%destructor</tt> directive. See the documentation

on the <tt><a href='#destructor'>%destructor</a></tt> directive below for

additional information.</p>

<p>In some grammars, many different non-terminal symbols have the

same data type and hence the same destructor. This directive is

a convenient way to specify the same destructor for all those

non-terminals using a single statement.</p>

<a id='default_type'></a>

<h4>4.4.3 The <tt>%default_type</tt> directive</h4>

<p>The <tt>%default_type</tt> directive specifies the data type of non-terminal

symbols that do not have their own data type defined using a separate

<tt><a href='#ptype'>%type</a></tt> directive.</p>

<a id='destructor'></a>

<h4>4.4.4 The <tt>%destructor</tt> directive</h4>

<p>The <tt>%destructor</tt> directive is used to specify a destructor for

a non-terminal symbol.

(See also the <tt><a href='#token_destructor'>%token_destructor</a></tt>

directive which is used to specify a destructor for terminal symbols.)</p>

<p>A non-terminal's destructor is called to dispose of the

non-terminal's value whenever the non-terminal is popped from

the stack. This includes all of the following circumstances:</p>

<li> When a rule reduces and the value of a non-terminal on

the right-hand side is not linked to C code.

<li> When the stack is popped during error processing.

<li> When the ParseFree() function runs.

<p>The destructor can do whatever it wants with the value of

the non-terminal, but its design is to deallocate memory

or other resources held by that non-terminal.</p>

<p>Consider an example:</p>

%type nt {void*}

%destructor nt { free($$); }

nt(A) ::= ID NUM. { A = malloc( 100 ); }

<p>This example is a bit contrived, but it serves to illustrate how

destructors work. The example shows a non-terminal named

"nt" that holds values of type "void*". When the rule for

an "nt" reduces, it sets the value of the non-terminal to

space obtained from malloc(). Later, when the nt non-terminal

is popped from the stack, the destructor will fire and call

free() on this malloced space, thus avoiding a memory leak.

(Note that the symbol "$$" in the destructor code is replaced

by the value of the non-terminal.)</p>

<p>It is important to note that the value of a non-terminal is passed

to the destructor whenever the non-terminal is removed from the

stack, unless the non-terminal is used in a C-code action. If

the non-terminal is used by C-code, then it is assumed that the

C-code will take care of destroying it.

More commonly, the value is used to build some

larger structure, and we don't want to destroy it, which is why

the destructor is not called in this circumstance.</p>

<p>Destructors help avoid memory leaks by automatically freeing

allocated objects when they go out of scope.

To do the same using yacc or bison is much more difficult.</p>

<a id='extraarg'></a>

<h4>4.4.5 The <tt>%extra_argument</tt> directive</h4>

<p>The <tt>%extra_argument</tt> directive instructs Lemon to add a 4th parameter

to the parameter list of the Parse() function it generates. Lemon

doesn't do anything itself with this extra argument, but it does

make the argument available to C-code action routines, destructors,

and so forth. For example, if the grammar file contains:</p>

%extra_argument { MyStruct *pAbc }

<p>Then the Parse() function generated will have an 4th parameter

of type "MyStruct*" and all action routines will have access to

a variable named "pAbc" that is the value of the 4th parameter

in the most recent call to Parse().</p>

<p>The <tt>%extra_context</tt> directive works the same except that it

is passed in on the ParseAlloc() or ParseInit() routines instead of

on Parse().</p>

<a id='extractx'></a>

<h4>4.4.6 The <tt>%extra_context</tt> directive</h4>

<p>The <tt>%extra_context</tt> directive instructs Lemon to add a 2nd parameter

to the parameter list of the ParseAlloc() and ParseInit() functions. Lemon

doesn't do anything itself with these extra argument, but it does

store the value make it available to C-code action routines, destructors,

and so forth. For example, if the grammar file contains:</p>

%extra_context { MyStruct *pAbc }

<p>Then the ParseAlloc() and ParseInit() functions will have an 2nd parameter

of type "MyStruct*" and all action routines will have access to

a variable named "pAbc" that is the value of that 2nd parameter.</p>

<p>The <tt>%extra_argument</tt> directive works the same except that it

is passed in on the Parse() routine instead of on ParseAlloc()/ParseInit().</p>

<a id='pfallback'></a>

<h4>4.4.7 The <tt>%fallback</tt> directive</h4>

<p>The <tt>%fallback</tt> directive specifies an alternative meaning for one

or more tokens. The alternative meaning is tried if the original token

would have generated a syntax error.</p>

<p>The <tt>%fallback</tt> directive was added to support robust parsing of SQL

syntax in <a href='https://sqlite.org/'>SQLite</a>.

The SQL language contains a large assortment of keywords, each of which

appears as a different token to the language parser. SQL contains so

many keywords that it can be difficult for programmers to keep up with

them all. Programmers will, therefore, sometimes mistakenly use an

obscure language keyword for an identifier. The <tt>%fallback</tt> directive

provides a mechanism to tell the parser: "If you are unable to parse

this keyword, try treating it as an identifier instead."</p>

<p>The syntax of <tt>%fallback</tt> is as follows:</p>

<tt>%fallback</tt> <i>ID</i> <i>TOKEN...</i> <b>.</b>

<p>In words, the <tt>%fallback</tt> directive is followed by a list of token

names terminated by a period.

The first token name is the fallback token &mdash; the

token to which all the other tokens fall back to. The second and subsequent

arguments are tokens which fall back to the token identified by the first

argument.</p>

<a id='pifdef'></a>

<h4>4.4.8 The <tt>%if</tt> directive and its friends</h4>

<p>The <tt>%if</tt>, <tt>%ifdef</tt>, <tt>%ifndef</tt>, <tt>%else</tt>,

and <tt>%endif</tt> directives

are similar to #if, #ifdef, #ifndef, #else, and #endif in the C-preprocessor,

just not as general.

Each of these directives must begin at the left margin. No whitespace

is allowed between the "%" and the directive name.</p>

<p>Grammar text in between "<tt>%ifdef MACRO</tt>" and the next nested

"<tt>%endif</tt>" is

ignored unless the "-DMACRO" command-line option is used. Grammar text

between "<tt>%ifndef MACRO</tt>" and the next nested "<tt>%endif</tt>" is

included except when the "-DMACRO" command-line option is used.<p>

<p>The text in between "<tt>%if</tt> <i>CONDITIONAL</i>" and its

corresponding <tt>%endif</tt> is included only if <i>CONDITIONAL</i>

is true. The CONDITION is one or more macro names, optionally connected

using the "||" and "&amp;&amp;" binary operators, the "!" unary operator,

and grouped using balanced parentheses. Each term is true if the

corresponding macro exists, and false if it does not exist.</p>

<p>An optional "<tt>%else</tt>" directive can occur anywhere in between a

<tt>%ifdef</tt>, <tt>%ifndef</tt>, or <tt>%if</tt> directive and

its corresponding <tt>%endif</tt>.</p>

<p>Note that the argument to <tt>%ifdef</tt> and <tt>%ifndef</tt> is

intended to be a single preprocessor symbol name, not a general expression.

Use the "<tt>%if</tt>" directive for general expressions.</p>

<a id='pinclude'></a>

<h4>4.4.9 The <tt>%include</tt> directive</h4>

<p>The <tt>%include</tt> directive specifies C code that is included at the

top of the generated parser. You can include any text you want &mdash;

the Lemon parser generator copies it blindly. If you have multiple

<tt>%include</tt> directives in your grammar file, their values are concatenated

so that all <tt>%include</tt> code ultimately appears near the top of the

generated parser, in the same order as it appeared in the grammar.</p>

<p>The <tt>%include</tt> directive is very handy for getting some extra #include

preprocessor statements at the beginning of the generated parser.

For example:</p>

%include {#include &lt;unistd.h&gt;}

<p>This might be needed, for example, if some of the C actions in the

grammar call functions that are prototyped in unistd.h.</p>

<p>Use the <tt><a href="#pcode">%code</a></tt> directive to add code to

the end of the generated parser.</p>

<a id='pleft'></a>

<h4>4.4.10 The <tt>%left</tt> directive</h4>

The <tt>%left</tt> directive is used (along with the

<tt><a href='#pright'>%right</a></tt> and

<tt><a href='#pnonassoc'>%nonassoc</a></tt> directives) to declare

precedences of terminal symbols.

Every terminal symbol whose name appears after

a <tt>%left</tt> directive but before the next period (".") is

given the same left-associative precedence value. Subsequent

<tt>%left</tt> directives have higher precedence. For example:</p>

%left AND.

%left OR.

%nonassoc EQ NE GT GE LT LE.

%left PLUS MINUS.

%left TIMES DIVIDE MOD.

%right EXP NOT.

<p>Note the period that terminates each <tt>%left</tt>,

<tt>%right</tt> or <tt>%nonassoc</tt>

directive.</p>

<p>LALR(1) grammars can get into a situation where they require

a large amount of stack space if you make heavy use or right-associative

operators. For this reason, it is recommended that you use <tt>%left</tt>

rather than <tt>%right</tt> whenever possible.</p>

<a id='pname'></a>

<h4>4.4.11 The <tt>%name</tt> directive</h4>

<p>By default, the functions generated by Lemon all begin with the

five-character string "Parse". You can change this string to something

different using the <tt>%name</tt> directive. For instance:</p>

%name Abcde

<p>Putting this directive in the grammar file will cause Lemon to generate

functions named</p>

<li> AbcdeAlloc(),

<li> AbcdeFree(),

<li> AbcdeTrace(), and

<li> Abcde().

</p>The <tt>%name</tt> directive allows you to generate two or more different

parsers and link them all into the same executable.</p>

<a id='pnonassoc'></a>

<h4>4.4.12 The <tt>%nonassoc</tt> directive</h4>

<p>This directive is used to assign non-associative precedence to

one or more terminal symbols. See the section on

<a href='#precrules'>precedence rules</a>

or on the <tt><a href='#pleft'>%left</a></tt> directive

for additional information.</p>

<a id='parse_accept'></a>

<h4>4.4.13 The <tt>%parse_accept</tt> directive</h4>

<p>The <tt>%parse_accept</tt> directive specifies a block of C code that is

executed whenever the parser accepts its input string. To "accept"

an input string means that the parser was able to process all tokens

without error.</p>

<p>For example:</p>

%parse_accept {

printf("parsing complete!\n");