1. Programming
Languages
Telefónica Digital DevCon 2013
Iván Montes - @drslump

2. Disclaimer
•
Talk is focused on mainstream languages
•
Many of the stuff we will talk about is already available
•
I know your favourite language probably does something
similar, or you know a hack to make it do it.
•
Scripting languages do have a compiler, they just happen to
include an evaluator too :-)
•
C++ is purposely left out of any comparison (Guys you need to
simplify the spec!)
•
If you don’t understand some terminology, ask!

3. What is the best?
•
Good news! You’re probably using it already
•
Check list:
•
You can focus on solving your business problem
•
You are proficient with it
•
You can deploy your product without fear
•
You have tooling available (editors, testing, packaging…)
•
You are not the only soul who knows how it works :-)

4. But…

5. Can it be improved?
•
Given:
•
•
•
Expected functionality is ever growing
Solutions complexity is ever growing
When:
•
•
•
We add layers and layers to keep it under control
Those layers are usually libraries and frameworks
Then:
•
Your really nice language is not that important anymore
•
You end up programming for the framework (does it pass the check list?)

6. The problem
•
We can separate a language in:
•
Specification, Compiler, Runtime and Libraries
•
Of those we are only in control of the Libraries
•
So we abuse them to solve our problems
•
But we can end up creating others:
•
Support
•
Versioning
•
Troubleshooting

7. What’s wrong with Libraries?
•
They are meant for code reuse
•
We (ab)use them as language
customisation and extensibility
tools
•
•
Cycle ends with lots of
abandonware even if you don’t
notice it:
•
Check SourceForge, GitHub
and similar sites
•
The guy now maintaining
your 2009 project has
megabytes of legacy source
code, with no expertise to
hack it and nobody to offer
support.
The cycle of doom then begins:
•
Our cool language toolchain
breaks
•
We expend time modifying
editors, build tools, …
•
The library gets updated or
we want to support a new
tool

8. Language design
•
Disclaimer: It’s a hard discipline, requiring lots of talent
and resources.
•
Basically the current situation in the mainstream market
sucks.
•
It’s slowly changing though (guys at Microsoft Research
and Universities)
•
Designers are realising (again) that we are not totally
stupid!
•
What we need is a very old tech, we just need it refined.

9. Taking control
•
Premise: Any non-tech-savvy person assumes
that a developer is someone intelligent.
•
Think highly of yourself!
•
We want/need more control over our primary tool.
•
Make yourself this question:
What do you think has evolved more in the last
25 years: compilers or cad software?

10. but I already can!
•
No, you don’t.
•
Some compilers support writing add ons for them.
•
The problem is neither being able to write an extension in C to
be called from the host language.
•
The Language itself must define how this extension mechanisms
work. It must be part of its specification.
•
This way we get extensibility without sacrificing portability and
risking vendor lock-in.
•
It helps with versioning, you extend a version of the language not
a version of a given compiler.

11. Extensibility
•
We should be able to extend pretty much anything in our
language.
•
Extending the syntax and semantics (specification) is doable
but not very mature yet.
•
Extending the runtime is expensive. You wan’t something
tested and battle tested and then tested again.
•
However extending the compiler is ready for prime time!
•
Preprocessors and lispy macros are cool, don’t resent me.
•
But we need something more refined, almost fool-proof.

12. A Compiler
•
A compiler basically does three tasks:
•
•
Resolve (name resolution, type binding)
•
•
Parse (Ast construction)
Emit (machine code, bytecode).
Keep in mind that a bit is a measure for information!
Memory usage (Kb)
Code
Parse
Resolve
Emit
0
300
600
900
1200

13. Info!
Code
Parse
Resolve
Emit
•
Compilers produce a lot of information because they need
to understand our code.
•
Unfortunately this pretty cool information is trashed once the
compilation is done.
•
Some languages support reflection, at least not everything is
lost (re Java fan boys, type erasure is not cool).
•
Given that a lot of very clever people put a lot of effort over
several years creating a compiler.
•
It seems like a huge waste of resources to not use it more.

14. Compiler Service
•
Almost a reality (llvm, mono mcs, roslyn, …).
•
Note: You know that an IDE basically implements an incremental
compiler nowadays, right? Wonder why they feel slow now?
•
Use cases:
•
Better editor support
•
Optimised/Incremental builds
•
Documentation generation
•
Quality/Security audits

15. Compiler Service (II)
•
But we want more!
•
If only we could hook our logic into the compiler like we
do with build systems…
•
The following would be dirty cheap to have:
•
Injection of cross-concerns (logging, aspects, …)
•
Statically checked/expanded DSLs
•
Quality, Security and Business specific enforcements

16. Compiler Service (III)
•
All of the sudden, half our
libraries functionality is now
managed by the compiler!
•
That means for example:
•
Build systems will notify
errors generated by our
custom enforcements
Wait! won’t we suffer the “cycle
of doom” too?
•
•
Editors will autocomplete
and report errors for
expanded DSLs
•
•
•
No! :-)
The logic is encapsulated
under the compiler public
interface.
All the tooling using the
compiler will automatically
support our use cases!
•
Besides, we reduced some
heavy weight from the runtime,
pretty cool for restrained mobile
apps for example.

17. Macros
•
Extending the compiler is nice but could be made
more user friendly.
•
When you have a pattern coming again and again in
your code, it will probably be better solved with a
macro.
•
Macros are a pretty old concept but they mostly refer
to “text macros”. What we need are AST Macros.
•
Instead of replacing chunks of text we operate on the
syntax tree of the compiler. Much safer and powerful!

18. Example
macro using(expr as Expression):
yield [|
try:
$(using.Body)
ensure:
$(expr).close()
|]
fp = open(‘path/to/file.ext’)
using fp:
fp.write(‘foobar')
# Note that the file resource is closed automatically

19. Quasi-Quotations
•
Normal language syntax constructs that are
somehow quoted (`[| .. |]` in the example).
•
Their purpose is to make creating complex AST
structures a breeze.
•
The compiler generates a separate AST for
them. It’s like a template.
•
When we inject them somewhere the compiler
will apply that template in that AST node.

20. Splicing
•
Injection points in a quasi-quotation (`$(..)` in the
example).
•
Their purpose is to parametrize quasi-quotations
to dynamically build arbitrary AST structures.
•
Resolves to a syntax node, the compiler will
insert that node in the template tree.
•
They keep their lexical information once
inserted.

21. Why are they better?
•
Syntax is integrated and validated as normal code.
•
Most of the lexical information is kept, thus reported errors
after expansion contain valid information.
•
Debuggable!
•
Toolchain friendly. They are transparent to consuming
tools.
•
DSL friendly. Capture common patterns to solve your
problems and offer a statically checked interface to fit
your needs!

22. Can’t libraries do this?
•
Yes, but they are not the right tool for the job.
•
Lots of stuff can be done at compile time, no need to bloat the
runtime with stuff that just makes developing easier.
•
Unified error reporting. Find out incorrect usages at compile
time instead of uncontrolled runtime exceptions.
•
The `assert` example shows pretty well the difference:
#!/bin/env python
!
foo = 10
assert foo != 10
# AssertionError:
#!/bin/env booi
!
foo = 10
assert foo != 10
# AssertionError: foo != 10

23. Syntax
A very quick overview

24. Parsers
•
Disclaimer: Not saying it’s a good
idea for a general language
•
The technology allows it and
current systems have enough
memory to pull it.
•
Traditionally parsing involved a
lexer generating tokens and a
parser (either a hand-coded
recursive descent or a generated
table based).
•
PEG style parsers (and other
pattern matching techniques)
operate the same on a stream of
chars (source code in a file), a tree
of XML nodes or any other object
structure.
•
They can be made to have
constant parsing time using
memoization.
•
OMeta is an example of PEG
parser with a built in extension
model.
•
Parsing rules (productions) can be
inherited and modified at will using
a pretty simple syntax.

25. Making the language ours
•
Use cases:
•
•
Units: delay = 10h + 30m, weight = 20Kg
•
•
Any sort of DSL would benefit from having its specific syntax
Reduce verbosity: for i in 1..10
Short comings:
•
Editors won’t know how to colorise the new syntax
•
It’s tempting to abuse it
•
Makes your code less portable
•
Tightly coupled to the standard grammar

26. User eXperience
Have you checked clang recently?

27. Language design
•
Syntax of a language is the first and
most important point of interaction.
•
Language design is still mainly driven
by the target audience expectations
(oh those curly braces everywhere).
•
•
The syntax must be very carefully
drafted to avoid ambiguities. Even if
designers still care very much about
the aesthetics.
When asked if they performed user
tests when designing C#, Hejlsberg
answered "only for the integration in
Visual Studio”. More generally, check
how it integrates with the tools you’re
going to use.
•
Very terse syntaxes tend to be good
for DSLs and small and targeted
projects. Code is read and reasoned
about much more often than written!
•
Semantics are as important as the
syntax, don’t overlook its implications.
•
Stay away from languages that
release new keywords and constructs
every other minor version. A syntax
either works or doesn’t but can target
every use case.

28. Reporting
•
•
The compiler is our main tool
but still, it’s generally
speaking, a bitch reporting
errors.
Error reporting is the main
interface, besides the
language syntax, between
the compiler and the
developer.
IDEs partially solve this by
implementing their own
heuristics on the code.
•
But it should be the compiler!
I don’t want my IDE bloated
with these things.
•
•
clang has done a great job
with this. Other compilers
should follow their lead.
•
Compilers ought to be
extensible in this aspect too,
if we can extend them with
our own logic we must have
a clear way to give feedback
to the user.

29. Parsing errors
•
Language newcomers tend to have syntax errors until they get fluent.
•
The common approach is to extend the grammar to include invalid productions with
custom errors.
•
Grammars tend to get out of hand and difficult to refactor.
•
A pretty cool way to improve the errors is to use example snippets of incorrect syntax.
•
The snippets are fed to the parser and the state of it when it errors is recorded.
•
Recorded states are then used to build the final parser, so when hits an error that
matches a custom message can be displayed.
class Foo:
def foo() # Note: I forgot the colon
!
pass
Before:
<stdin>(3,13): BCE0043: Unexpected token: <INDENT>.
<stdin>(3,13): BCE0044: expecting "DEDENT", found '<EOL>'.
<stdin>(3,13): BCE0044: expecting "EOF", found '<DEDENT>'.
After:
<stdin>(3,13): BCE0043: Methods must use a ‘:’ for their body.

30. Type Safety
Wishing it could be extended

31. Type Safety
•
Major families:
•
Structural (Go, TypeScript, …)
•
Pattern matching solves a lot of
type safety derived issues (bye
bye Visitor)
•
The problem with extending them
is that the semantics of the
language change.
•
People can easily reason and
adapt to syntax changes but for
semantic ones…
•
In the current state of the art is
probably better switching
languages completely if the
problem at hand requires a
different model of type safety.
Nominal (Java, C#, …)
•
•
Duck typing (Python, JavaScript,
…)
•
Some languages offer a mix of
them (ie. Scala, ObjC, Boo, …)
•
Many allow for runtime duck
typing.

32. API Versioning
•
If your language uses nominal typing you actually have to embrace
it. Carefully plan your Interfaces to get the job done.
•
C# does it right. If you want to be confident about exposing an API
you can’t make every method virtual (*cough* Java *cough*).
•
public, protected and private are visibility modifiers, not extension
point indicators.
•
Structural typing really shines in this regard. But it must allow it from
the call site not only the definition site (Go vs Scala). The library
author doesn’t have a clue if his API works as designed or not :-)
•
Dynamic languages are flawed on this. Rapid prototyping but zero
confidence when upgrading.

33. Cool stuff
Old tech sold as a bright future

34. Live Coding
•
Imagine you could tap into the
runtime environment and modify
the executing instructions on
realtime.
•
We ask the compiler to
regenerate the changes in our
function, the result is a chunk of
byte code to fed into the VMs.
•
Many projects deploy nowadays
into a Virtual Machine, which
technically simplify it.
•
No need to run an interpreter and
very few limitations.
•
Still, this is the kind of thing that
would flourish with an extensible
compiler design. No big vendor is
going to release something like
this.
•
Still, the VM expects byte code,
we want to write in a higher level
syntax.
•
Extensible compilers would allow
bringing Live Coding to
languages that are not
interpreted.

35. Type Providers
•
Popularised recently by F#
•
They are basically specialised macros, whose purpose is to
generate new types in the program.
•
Remember when you had to preprocess some XML schema files to
generate Java classes? What if the compiler could do that for you
automatically?
•
Using Json? Just feed an example message file to the type
provider macro and it will generate a typed interface for it.
•
This is available in many languages at runtime via Reflection.
What’s cool is that now you have it at compile time. Type checking
and auto completion as obvious benefits.

36. What now?

37. Is your responsibility
To get better tools for our job
we have to earn it. Advocate
these concepts to gain traction
in the profession.
•
If you have to choose between
an extensible language and
one that’s not, weight the
former appropriately in the
comparison.
Don’t be afraid of moulding a
language to your needs. Know
your tool and help it help you!
If deploying into a VM (JVM,
CLR, JavaScript) there is little
risk in mixing languages in a
project that boost our
productivity.
•
Forget the idea that only
functional languages are
extensible. Imperative
language can do it too!
Language designers do listen
to its users even if for the
mainstream languages they
are slow reacting to it.
•
•
•
•

38. Conclusions

39. Conclusions
•
Languages should be
designed for extensibility.
•
Anyone that can create a
library should be also able to
extend the compiler/language.
Not everything must be done at
runtime!
•
Statically typed languages are
closing the gap with dynamic
ones in ease of use and user
experience. We have to review
our current stand points on this.
If you have the chance to learn
a new framework or a new
language, chose the later, it’s
much more rewarding.
•
Extensible compilers will foster
innovation, no need to wait for
Oracle or Microsoft to come up
with new ideas every 3 years.
People will be able to shape
the future trying new things and
introducing those that work as
part of the standard languages.
•
Good drivers don’t need to
know about the internals of the
car, but it certainly doesn’t
hurt :-)
We need better compilers in
every aspect.
•
•

40. “The limits of my language are
the limits of my world.”
–Ludwig Wittgenstein