1. Introducing of Bison 天官 2011-09-15 1

4. Parsing methods Bison parsers can use either of two parsing methods, known as LALR(1) and GLR LALR(1) (Look Ahead Left to Right with a one-token lookahead), which is less powerful but considerably faster and easier to use than GLR. GLR (Generalized Left to Right). The most common kind of language that computer parsers handle is a context-free grammar(CFG) The standard form to write down a CFG is Baskus-Naur Form (BNF)

5. LR parser LR parser is a parser that reads input from Left to right and produces a Rightmost derivation. The term LR(k) parser is also used; where the k refers to the number of unconsumed "look ahead" input symbols that are used in making parsing decisions. Usually k is 1 and the term LR parser is often intended to refer to this case. (LALR(1))

6. look ahead LALR(1) cannot deal with grammars that need more than one token of lookahead to tell whether it has matched a rule. phrase: cart_animal AND CART | work_animal AND PLOW cart_animal: HORSE | GOAT work_animal: HORSE | OX phrase: cart_animalCART | work_animalPLOW cart_animal: HORSE | GOAT work_animal: HORSE | OX Not support! OR phrase: cart_animal AND CART | work_animal AND PLOW cart_animal: HORSE | GOAT work_animal: OX

7. Rightmost Derivation Rule 1 expr  expr – digit exprexpr – digit exprexpr + digit expr digit digit 0|1|2|…|9 Example input: 3 + 8 - 2 The rightmost non-terminal is replaced in each step Rule 4 expr – digit  expr – 2 Rule 2 expr – 2 expr + digit - 2 Rule 4 expr + digit - 2  expr + 8-2 Rule 3 expr + 8-2 digit + 8-2 Rule 4 digit + 8-23+8 -2

8. Leftmost Derivation Rule 1 expr  expr – digit The leftmost non-terminal is replaced in each step expr 1 1 Rule 2 expr – digit  expr + digit – digit 2 2 3 expr - digit Rule 3 expr + digit – digit  digit + digit – digit 3 5 4 expr digit + Rule 4 4 digit + digit – digit3 + digit – digit 2 Rule 4 3 + digit – digit 3 + 8 – digit 5 6 digit 8 Rule 4 3 + 8 – digit 3 + 8 – 2 6 3

9. Leftmost Derivation Rule 1 expr  expr – digit expr  expr – digit expr  expr + digit expr  digit digit 0|1|2|…|9 Example input: 3 + 8 - 2 The leftmost non-terminal is replaced in each step Rule 2 expr – digit  expr + digit – digit Rule 3 expr + digit – digit  digit + digit – digit Rule 4 digit + digit – digit3 + digit – digit Rule 4 3 + digit – digit 3 + 8 – digit Rule 4 3 + 8 – digit 3 + 8 – 2

10. Leftmost Derivation Rule 1 expr  expr – digit The leftmost non-terminal is replaced in each step expr 1 1 Rule 2 expr – digit  expr + digit – digit 6 2 2 expr - digit Rule 3 expr + digit – digit  digit + digit – digit 3 3 5 expr digit + Rule 4 4 digit + digit – digit3 + digit – digit 2 Rule 4 3 + digit – digit 3 + 8 – digit 5 4 digit 8 Rule 4 3 + 8 – digit 3 + 8 – 2 6 3

11. Context-Free Grammars A context-free grammar G is defined by the 4-tuple: G = (V, ∑, R, S) where V is a finite set; each element v ϵ V is called a non-terminal character or a variable. Each variable represents a different type of phrase or clause in the sentence. Variables are also sometimes called syntactic categories. Each variable defines a sub-language of the language defined by . ∑ is a finite set of terminals, disjoint from V, which make up the actual content of the sentence. The set of terminals is the alphabet of the language defined by the grammar G. R is a finite relation from V to (V U ∑)*. The members of R are called the (rewrite) rules or productions of the grammar. S is the start variable (or start symbol), used to represent the whole sentence (or program). It must be an element of V. The asterisk represents the Kleene star operation.

12. Context-free language The language of grammar G = (V, ∑, R, S) is the set L(G) = { ωϵ ∑* : S ωω } A language L is said to be context-free languange(CFL), if there exists a CFG G, such that L = L(G).

14. Non-terminals: expr, digit

16. Terms Symbols are strings of letters, digits, periods, and underscores that do not start with a digit. error is reserved for error recovery. Do not use C reserved words or bison's own symbols such as yyparse. Symbols produced by the lexer are called terminal symbols or tokens Those that are defined on the left-hand side of rules are called nonterminal symbols or nonterminals.

17. Structure of a Bison Specification ... definition section ... %% ... rules section ... %% ... user subroutines ...

18. Literal Block %{ ... C code and declarations ... %} The contents of the literal block are copied verbatim to the generated C source file near the beginning, before the beginning of yypare(). Usually contains declarations of variables and functions, as well as #include. Bison also provides an experimental %code POS { ... } where POS is a keyword to suggest where in the generated parser the code should go.

19. Delaration %parse-param %require "2.4“ declare the minimum version of bison needed to compile it %start identifies the top-level rule (Named the first rule.) %union %token %type %left %right %nonassoc %expect

20. Token Define the ternimators. Bison treats a character in single quotes as a token Bison also allows you to decalre strings as aliases for tokens This defines the token NE and lets you use NE and != interchangeably in the parser. The lexer must still return the internal token values for NE when the token is read, not a string. expr: '(' expr ')'; %token NE "!=" %% ... expr: expr "!=" exp;

21. Parse-param Normally, you call yyparse() with no arguments, if you need, youcan add parameters to its definition: %parse-param { char *modulename } %parse-param { int intensity } This allows you to call yyparse("mymodule", 42)

22. Type The %union declaration specifies the entire list of possible types %token is used for declaring token types %type is used for declaring nonterminal symbols %{ #include "calc.h“ /* Contains definition of `symrec' */ %} %union { double val; /* For returning numbers. */ symrec *tptr; /* For returning symbol-table pointers */ } %token <tptr> VAR FNCT /* Variable and Function */ %type <val> exp %%

23. Structure of a Bison Specification ... definition section ... %% ... rules section ... %% ... user subroutines ...

24. Actions An action is C code executed when bison matches a rule in the grammar. The action can refer to the values associated with the symbols in the rule by using a dollar sign followed by a number. The name $$ refers to the value for the left-hand side (LHS) symbol. For rules with no action, bison uses a default of the following date: month '/' day '/' year { printf("date %d-%d-%d found", $1, $3, $5); } ; { $$ = $1; }

25. Rules Recursive Rules The action can refer to the values associated with the symbols in the rule by using a dollar sign followed by a number. In most cases, Bison handles left recursion much more efficiently than right recursion. numberlist : /* empty */ | numberlist NUMBER ; exprlist: exprlist ',' expr; /* left recursion */ or exprlist: expr ',' exprlist; /* right recursion */

26. Special Characters % All of the declarations in the definition section start with %. $ In actions, a dollar sign introduces a value reference. @ In actions, an @ sign introduces a location reference, such as @2 for the location of the second symbol in the RHS. ' Literal tokens are enclosed in single quotes. " Bison lets you declare quoted string as parser alias for tokens. <> In a value reference in an action, you can override the value's default type by enclosing the type name in angle brackets. {} The C code in actions is enclosed in curly braces. ; Each rule in the rules section should end with a semicolon. | or syntax for multi-rules with same LHS. : separate left-hand side and right-hand side - Symbols may include underscores along with letters, digits, and periods. . Symbols may include periods along with letters, digits, and underscores.

27. Reserved YYABORT In an action makes the parser routine yyparse() return immediately with a nonzero value, indicating failure. YYACCEPT In an action makes the parser routine yyparse() return immediately with a value 0, indicating success. YYBACKUP The macro YYBACKUP lets you unshift the current token and replace it with something else. sym: TOKEN { YYBACKUP(newtok, newval); } It is extremely difficult to use YYBACKUP() correctly, so you're best off not using it.

28. Reserved yyclearin The macro yyclearin in an action discards a lookahead token if one has been read. It is most oftern useful in error recovery in an interactive parser to put the paarser into a known state after an error: YYDEBUG To include the trace code, either use the -t flag on the bison command line or else define the C preprocessor symbol YYDEBUG to be nonzero either on the C compiler command line or by inlcuding something like this in the definition section: stmtlist : stmt | stmtlist stmt; stmt : error { reset_input(); yyclearin; }; %{ #define YYDEBUG 1 %}

29. Ambiguity and Conflicts The grammar is truly ambiguous Shift/Reduce Conflicts Reduce/Reduce Conflicts The grammar is unambiguous, but the standard parsing technique that bison uses is not powerful enough to parse the grammar. (need to look more than one token ahead) We have already told about it of LALR(1).

30. Reduce/Reduce Conflicts A reduce/reduce conflict occurs when the same token could complete two different rules. %% prog: proga | progb; proga: 'X'; progb: 'X';

31. Shift/Reduce Conflicts %type <a> exp ... %% ... expr: expr '+' exp { $$ = newast('+', $1, $3); } | expr '-' exp { $$ = newast('-', $1, $3); } | expr '*' exp { $$ = newast('*', $1, $3); } | expr '/' exp { $$ = newast('/', $1, $3); } | '|' exp { $$ = newast('|', $2, NULL); } | '(' exp ')' { $$ = $2); } | '-' exp { $$ = newast('M', $2, NULL); } | NUMBER { $$ = newnum($1); } ; %% Example 2+3*4

32. Problem At this point, the parser looks at the * and could either reduce 2+3 using; to an expression or shift the *, expecting to be able to reduce: later on. 2 shift NUMBER E reduce E->NUMBER E + shift + E + 3 shift NUMBER E + E reduce E->NUMBER Example 2+3 * 4 expr: expr '+' exp expr: expr ‘*' exp

33. Analysis The problem is that we haven't told bison about the precedence and associativity of the operators. Precedence controls which operators execute first in an expression. In and expression grammar, operators are grouped into levels of precedence from lowest to highest.The total number of levels depends on the language. The C language is notorious for having too many precedence levels, a total of 15 levels. Associativity controls the grouping of operators at the same precedence level.

34. Implicitly Solution %type <a> exp exp1 exp2 ... %% ... expr : expr1 '+' exp1 { $$ = newast('+', $1, $3); } | expr1 '-' exp1 { $$ = newast('-', $1, $3); } | expr1 { $$ = $1; } expr1: expr2 '*' exp2 { $$ = newast('*', $1, $3); } | expr2 '/' exp2 { $$ = newast('/', $1, $3); } | expr2 { $$ = $1; } expr2: '|' exp { $$ = newast('|', $2, NULL); } | '(' exp ')' { $$ = $2); } | '-' exp { $$ = newast('M', $2, NULL); } | NUMBER { $$ = newnum($1); } ; %%

35. Explicitly Solution %left '+' '-’ %left '*' '/’ %nonassoc '|' NMINUS %type <a> exp exp1 exp2 ... %% ... expr: expr '+' exp { $$ = newast('+', $1, $3); } | expr '-' exp { $$ = newast('-', $1, $3); } | expr '*' exp { $$ = newast('*', $1, $3); } | expr '/' exp { $$ = newast('/', $1, $3); } | '|' exp { $$ = newast('|', $2, NULL); } | '(' exp ')' { $$ = $2); } | '-' exp %prec UMINUS { $$ = newast('M', $2, NULL); } | NUMBER { $$ = newnum($1); } ; %%

36. Explicitly Solution %left, %right, and %nonassoc declarations defining the order of precedence from lowest to highest. %left, left associative %right, right associative %nonaccoc, no associativity UMINUS, pseudo token standing fro unary minus %prec UMINUS, %prec tells bison to use the precedence of UMINUS for this rule.

37. IF/THEN/ELSE conflict When Not to Use Precedence Rules In expression grammars and to resolve the "dangling else" conflict in grammars for if/then/else language constructs, it is easy to understand. But in other situations, it can be extremely difficult to understand. stmt: IF '(' cond ')' stmt | IF '(' cond ')' stmt ELSE stmt | TERMINAL cond: TERMINAL Ambiguous!!! IF ( cond ) IF ( cond ) stmt ELSE stmt Which one? IF ( cond ) { IF ( cond ) stmt } ELSE stmt IF ( cond ) { IF ( cond ) stmt ELSE stmt }

38. Implicitly Solution stmt :matched | unmatched ; matched :other_stmt | IF expr THEN matched ELSE matched ; unmatched : IF expr THEN stmt | IF expr THEN matched ELSE unmatched ; other_stmt: /* rules for other kinds of statement */ ... IF ( cond ) { IF ( cond ) stmt ELSE stmt }

39. Explicitly Solution %nonassoc THEN %nonassoc ELSE %% stmt : IF expr THEN stmt | IF expr stmt ELSE stmt ; Equal to: %nonassoc LOWER_THAN_ELSE %nonassoc ELSE %% stmt : IF expr stmt %prec LOWER_THAN_ELSE | IF expr stmt ELSE stmt ; IF ( cond ) { IF ( cond ) stmt ELSE stmt }

40. expect Occasionally you may have a grammar that has a few conflicts, you are confident that bison will resolve them the way you want, and it's too much hassle to rewrite the grammar to get rid of them. %expect N tells bison that your parser should have N shift/reduce conflicts. %expect-rr N to tell it how many reduce/reduce conflicts to expect.

41. Common Bugs In Bison Programs Infinite Recursion %% xlist: xlist ‘X’ ; should be ==> %% xlist : 'X' | xlist 'X’ ;

42. Common Bugs In Bison Programs Interchanging Precedence %token NUMBER %left PLUS %left MUL %% expr : expr PLUS expr %prec MUL | expr MUL expr %prec PLUS | NUMBER ;

43. Lexical Feedback Parsers can sometimes feed information back to the lexer to handle otherwise difficult situations. E.g. syntax like this: message ( any characters ) /* parser */ %{ init parenstring = 0; }% ... %% statement: MESSAGE { parenstring = 1; } '(' STRING ')';

44. Lexical Feedback /* lexer */ %{ extern int parenstring; %} %s PSTRING %% "message" return MESSAGE; "(" { if(parenstring) BEGIN PSTRING; return '('; } <PSTRING>[^)]* { yylval.svalue = strdup(yytext); BEGIN INITIAL; return STRING; }

45. Structure of a Bison Specification ... definition section ... %% ... rules section ... %% ... user subroutines ...

46. User subroutines Section This section typically includes routines called from the actions. Nothing special.

47. Discuss Everything