Of Rats And Dragons

Of Rats And
Dragons
Achieving Parsing Sanity

Sean Cribbs

Web Consultant
Ruby and Erlang Hacker

Saturday, September 12, 2009

Quick Review

context-free
grammars


G = (V, Σ, R, S)


V→w


non-deterministic
pushdown
automata


stack machine
w/ backtracking


massage it


ε productions


produce nothing
never produced


cycles


ambiguities
“dangling else”


if A then if B then C else D




parsing expression
grammars


top-down parsing
language (70’s)


direct
representation of
parsing functions


Brian Ford 2002


focused on
recognizing


computer
languages


V←e


e1 e2


e1 / e2


e+


e*


&e


!e


e?


“string”
.


PEG > regexps


combined
lex+parse


no ambiguity


choice is ordered


dangling else
obviated


greedy repetition


unlimited
lookahead
with predicates


no left-recursion!
(use *,+)


Parsing
Techniques


Tabular
test every rule


Recursive-descent
call & consume


Predictive
yacc/yecc


Packrat
RD with memo


sacriﬁce memory
for speed


supports PEGs and
some CFGs


Treetop
Pappy
neotoma


neotoma
Behind the CodeTM


can:has(cukes) ->
false.


Cucumber uses
Treetop


PEG → leex/yecc
FAIL


parsec → eParSec


HOF protocol


% Implements "?" PEG operator
optional(P) ->
fun(Input, Index) ->
case P(Input, Index) of
{fail, _} -> {[], Input, Index};
{_,_,_} = Success -> Success
% {Parsed, RemainingInput, NewIndex}
end
end.


% PEG
optional_space <- space?;

% Erlang
optional_space(Input,Index) ->
optional(fun space/2)(Input, Index).


Yay! RD!
make it memo


ets
Erlang Term
Storage


{key, value}


key = Index


value = dict


% Memoization wrapper
p(Inp, StartIndex, Name, ParseFun, TransformFun) ->
% Grab the memo table from ets
Memo = get_memo(StartIndex),
% See if the current reduction is memoized
case dict:find(Name, Memo) of
% If it is, return the result
{ok, Result} -> Result;
% If not, attempt to parse
_ ->
case ParseFun(Inp, StartIndex) of
% If it fails, memoize the failure
{fail,_} = Failure ->
memoize(StartIndex, dict:store(Name, Failure, Memo)),
Failure;
% If it passes, transform and memoize the result.
{Result, InpRem, NewIndex} ->
Transformed = TransformFun(Result, StartIndex),
memoize(StartIndex, dict:store(Name, {Transformed,
InpRem, NewIndex}, Memo)),
{Transformed, InpRem, NewIndex}
end
end.


parse_transform


alternative(Input, Index) ->
peg:p(Input, Index, alternative, fun(I,P) ->
peg:choose([fun sequence/2, fun primary/2])(I,P)
end).

rule(alternative) ->
peg:choose([fun sequence/2, fun primary/2]);


rules <- space? declaration_sequence space?;
declaration_sequence <- head:declaration tail:(space declaration)*;
declaration <- nonterminal space '<-' space parsing_expression space? ';';
parsing_expression <- choice / sequence / primary;
choice <- head:alternative tail:(space '/' space alternative)+;
alternative <- sequence / primary;
primary <- prefix atomic / atomic suffix / atomic;
sequence <- head:labeled_sequence_primary tail:(space labeled_sequence_primary)+;
labeled_sequence_primary <- label? primary;
label <- alpha_char alphanumeric_char* ':';
suffix <- repetition_suffix / optional_suffix;
optional_suffix <- '?';
repetition_suffix <- '+' / '*';
prefix <- '&' / '!';
atomic <- terminal / nonterminal / parenthesized_expression;
parenthesized_expression <- '(' space? parsing_expression space? ')';
nonterminal <- alpha_char alphanumeric_char*;
terminal <- quoted_string / character_class / anything_symbol;
quoted_string <- single_quoted_string / double_quoted_string;
double_quoted_string <- '"' string:(!'"' ("" / '"' / .))* '"';
single_quoted_string <- "'" string:(!"'" ("" / "'" / .))* "'";
character_class <- '[' characters:(!']' ('' . / !'' .))+ ']
anything_symbol <- '.';
alpha_char <- [a-z_];
alphanumeric_char <- alpha_char / [0-9];
space <- (white / comment_to_eol)+;
comment_to_eol <- '%' (!"n" .)*;
white <- [ tnr];


self-hosting


Future directions


self-contained
parsers


inline code in PEG


Reia
retem
sedate


questions?


Of Rats And Dragons

Recomendados

Recomendados

Más contenido relacionado

Destacado

Destacado (16)

Similar a Of Rats And Dragons

Similar a Of Rats And Dragons (20)

Más de Sean Cribbs

Más de Sean Cribbs (19)

Último

Último (20)

Of Rats And Dragons