This document discusses building a Python-like language with Xtext that is indentation aware. It covers parsing indentation aware languages, tokenizing and splitting tokens, defining indentation tokens, adjusting the Xtext workflow, marking tokens as splittable, and formatting indentation in the language. Key challenges discussed are parsing, content assist, formatting, folding and outline for indentation aware languages. The document provides code examples and recap of defining block structures with open and close tokens.
6. I N D E N TAT I O N A WA R E
Block:
OPEN
stmt+=Statement+
CLOSE;
7. C H A L L E N G E S
I N D E N TA T I O N A WA R E L A N G U A G E S
8. PA R S I N G
C O N T E N T A S S I S T
F O R M AT T I N G
F O L D I N G
O U T L I N E
C H A L L E N G E S
I N D E N TA T I O N A WA R E L A N G U A G E S
9. PA R S I N G
I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S
10. PA R S I N G
I N D E N TA T I O N A WA R E L A N G U A G E S - C H A L L E N G E S
Stateless, Upfront Lexing
Reentrant Lexing
CommonToken Oddities
Token Types
Only Basic Logic in Terminal Rules
11. S O M E P Y T H O N E S Q U E C O D E
if name==“Sebastian”:
println “Hello”
12. S O M E P Y T H O N E S Q U E C O D E
if.name==“Sebastian”:n
....println.“Hello”n
13. PA R S I N G - T O K E N I Z I N G
i f . n a m e = = “ S e b a s t i a
n ” : n . . . . p r i n t l n . “ H
e l l o ” n
14. PA R S I N G - T O K E N I Z I N G
i f . n a m
e = = “ S e
b a s t i a
n ” : n . .
. . p r i n
t l n . “ H
e l l o ” n
CharStream
o.a.IntStream
AntlrStringStream
15. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
CharStream
o.a.IntStream
AntlrStringStream
16. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
CharStream
o.a.IntStream
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
17. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
CharStream
o.a.IntStream
TokenSource
Lexer
AntlrStringStream
1 2
9
1 0
7
8
6
9
1 0
9
9
8
- 1
18. PA R S I N G - T O K E N I Z I N G
if
.
name
==
“Sebastian”
:
n....
println
.
“Hello”
n
1 2
9
1 0
7
8
6
9
1 0
9
9
8
TokenStream
- 1
TokenSource
Lexer
o.a.IntStream
30. 1. D E F I N E I N D E N TAT I O N
T O K E N S
terminal OPEN: ‘synthetic:OPEN’;
terminal CLOSE: ‘synthetic:CLOSE’;
31. fragment = parser.antlr.ex.rt.AntlrGeneratorFragment {}
..
fragment =
parser.antlr.ex.ca.ContentAssistParserGeneratorFragment {}
2. A D J U S T M W E 2
W O R K F L O W
1. GENERATE SPLITTING TOKEN SOURCE
2. DISABLE PARTIAL PARSING
32. public class MyDslTokenSource extends AbstractIndentationTokenSource {
public MyDslTokenSource(TokenSource delegate) {
super(delegate);
}
@Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@Override
protected int getBeginTokenType() {
return InternalMyDslParser.RULE_OPEN;
}
@Override
protected int getEndTokenType() {
return InternalMyDslParser.RULE_CLOSE;
}
}
3. M A R K T O K E N S A S
S P L I T TA B L E
33. 3. M A R K T O K E N S A S
S P L I T TA B L E
protected Token createEndToken(int offset) {
CommonToken result = new CommonToken(getEndTokenType());
result.setText("");
result.setChannel(Token.DEFAULT_CHANNEL);
result.setStartIndex(offset);
result.setStopIndex(offset-1);
return result;
}
public class AbstractIndentationTokenSource
extends AbstractSplittingTokenSource {
…
…
}
34. @Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@Override
protected int getBeginTokenType() {
return InternalMyDslParser.RULE_OPEN;
}
@Override
protected int getEndTokenType() {
return InternalMyDslParser.RULE_CLOSE;
}
3. M A R K T O K E N S A S
S P L I T TA B L E
35. @Override
protected boolean shouldSplitTokenImpl(Token token) {
return token.getType() == InternalMyDslParser.RULE_WS;
}
@Override
protected int getBeginTokenType() {
return InternalMyDslParser.RULE_OPEN;
}
@Override
protected int getEndTokenType() {
return InternalMyDslParser.RULE_CLOSE;
}
3. M A R K T O K E N S A S
S P L I T TA B L E
38. O N E M O R E T H I N G
I N D E N TA T I O N A WA R E L A N G U A G E S
39. def void format(Block block,
extension IFormattableDocument document) {
val open = block.regionForRuleCallTo(OPENRule)
open.prepend[ newLineAndIncreaseIndentation ]
val close = block.regionForRuleCallTo(CLOSERule)
if (block.isLast)
close.append[ decreaseIndentation ]
else
close.append[ newLineAndDecreaseIndentation ]
}
def void format(Block block,
extension IFormattableDocument document) {
val open = block.regionForRuleCallTo(OPENRule)
open.prepend[ newLineAndIncreaseIndentation ]
val close = block.regionForRuleCallTo(CLOSERule)
if (block.isLast)
close.append[ decreaseIndentation ]
else
close.append[ newLineAndDecreaseIndentation ]
}
5. F O R M AT T I N G
1. Indent Blocks
2. Adjust NewLines
3. Remove superfluous spaces
40. PA R S I N G
C O N T E N T A S S I S T
F O R M AT T I N G
F O L D I N G
O U T L I N E
D O N E
I N D E N TA T I O N A WA R E L A N G U A G E S