ffffuuuuconf 2010 presentation

1. You Shall NotGet
Excited
Ivan Ribeiro Rocha
ivan.ribeiro@gmail.com
@irr

2. Agenda
● Why not?
●
”History” about getting excited...
●
What should I use...?
●
My choice: Erlang
●
Learning from
experience...

3. Why not?

4. ”Once upon a time...”
”History” about getting excited...

5. (what is a ”technical”
manager?)

6. …a ”technical”
manager meets...

7. … a ”smart guy”
(a.k.a salesman)

8. Deal! You deserveit!

9. ”no problem... we can fixit!”

10. ”relax... the guys will change
it in the upcoming version...”

11. sometimes is not so easy to
fix all problems

12. ”Ignorance more frequently begets
confidence than does
knowledge.”
Charles Darwin

13. What should I use...?

14. Programming Languages
and tools...
● Languages
– Java, Ruby, Python, Perl, ...
– C/C++, Lisp, Haskell, Ocaml, Erlang, ...
● Web Servers
– Apache, nginx, ...
– Jetty/Tomcat, ...
– Yaws, inets, ...

15. Anyone using it?

16. Is it stable?

17. Does it scale?

18. (think about...)
You're not:

19. what means
scalable?
- don’t design to scale infinitely
- consider 5X - 50X growth
- but > 100X requires redesign
- break large systems into smaller services
Jeff Dean
http://bit.ly/clIJfL

23. no documentation...?
● Good APIs?
● Available books?
● % Comments...?
● Source code?
● Please! No
”magazines”...

24. still in doubt...?

26. Joe Armstrong
Robert Virding
Mike Williams
Claes “Klacke” Wikström
and more...
My choice: Erlang

28. Learning from experience...

29. ● > 3 years and learning...
● back end development
– REST API
– memcached API
– syslog API
– MySQL and Mnesia (data storage)

30. ● functional + concurrent programming
● fault-tolerant and highly scalable
● very light-weight concurrency (processes)
●
running massive systems for 20 years
● bit syntax and binaries (for protocol programming)
● links (encourages “let it crash” programming)
●
”concurrency belongs to the language and not the
operating system”

31. Erlang
(Why you should get excited...)
Concurrent Programming
–lots of processes
●
the only way for processes to interact is
through message passing
– event
based (epoll)

32. Erlang
(Why you should get excited...)
No shared memory
– ”sharing is the property that prevents
fault tolerance”
– destructive shared data modifications
do not occur!

33. Erlang
(Why you should get excited...)
It's easyto test
and update your code

34. Erlang
(Why you should get excited...)
HTTP/JSON support
– yaws (embedded)
– mochiweb (json)
– misultin
–inets
– httpc

35. Erlang
(Why you should get excited...)
many Libraries/tools for distributed
programs...
– net_[adm|kernel]
net_kernel:connect_node(N).
net_adm:ping(N).
– rpc
{R, E} = rpc:multicall(nodes(), Mod, Fun, [N], T),
lists:foldl(fun(L, A) ->
merge(A, [X || X <- L, not member(X, A)], N)
end, C, R),
– epmd...

36. Erlang
(Why you should get excited...)
”Mnesia is a distributed DataBase Management System
(DBMS), appropriate for telecommunications applications and
other Erlang applications which require continuous operation
and exhibit soft real-time properties”
Mnesia is great
but you should really know how to use it
and you MUST architecture your application
to get the best results...

37. Erlang
(Why you should get excited...)
OTP (Unix <-> C <==> OTP <-> Erlang)
”It’s an application operating system and a set of
libraries and procedures used for building
large-scale, fault-tolerant, distributed applications”
– supervisor
– applications
– gen_server
– gen_fsm
– gen_event

38. ”We should forget about small efficiencies, say
about 97% of the time: premature
optimization is the root of all evil.”
Donald Knuth

39. Erlang
(caveats: You must be careful...)
Mnesia
– can't handle very large data
– 2 Gb
● ETS
● DETS
● MNESIA

40. Erlang
(caveats: You must be careful...)
Mnesia
–2 Gb fragments (mod (2^X))
– avoid rehash: create more fragments...
● add
● move
● del*

41. Erlang
(caveats: You must be careful...)
Mnesia
–CAP theorem
–replicax partitioned
networks

42. Erlang
(caveats: You must be careful...)
Mnesia
– need to check replica consistency?
●
vector clocks... (avoid dependencies)
●
should try timestamps

43. Erlang
(caveats: You must be careful...)
Mnesia
– event "running partitioned network"
● mnesia:set_master_nodes(L).
●
must restart other nodes
– if some node will not recover
soon...
● mnesia:force_load_table(T).

44. Erlang
(caveats: You must be careful...)
Mnesia (QLC)
●
avoid retrieve large data sets
●
use cursors inside
transactions
mnesia:activity(sync_transaction,
fun(X) ->
QC = qlc:cursor(X),
QR = qlc:next_answers(QC, N),
qlc:delete_cursor(QC),
QR
end,
[qlc:q([E || E <- mnesia:table(Tab)])],
mnesia_frag).

45. Erlang
(caveats: You must be careful...)
Mnesia
–load balance (read)
– avoid ”overload” one instance
mnesia_lib:set({T, where_to_read}, Node)

46. Erlang
(caveats: You must be careful...)
Logging
– avoid overhead
● yaws
● error_logger
– syslog
● udp
● tcp*
● gen_server
+ handle_cast

47. Erlang
(caveats: You must be careful...)
Messages
– like a mailbox
– per process (don't forget to read your messages)

48. Erlang
(caveats: You must be careful...)
OTP
–only API
– poor docs,
books,
articles...

49. Erlang
(caveats: You must be careful...)
OTP
– avoid ”synchronize” long calls
test(X) ->
gen_server:call(?MODULE, {test, X}, 5000).
handle_call({test, X}, From, State) →
spawn(fun() →
%% do some long task
gen_server:reply(From, Reply)
end),
{noreply, State}.

50. Erlang
(caveats: You must be careful...)
Security
– must be treated externally
● firewall
● private networks
– cookie based
-kernel inet_dist_use_interface {127,0,0,1}
-kernel inet_dist_listen_min <min>
-kernel inet_dist_listen_max <max>
> erl -kernel inet_dist_listen_min 9001 inet_dist_listen_max 9005
(4369 TCP port, as it is used by epmd, ERL_EPMD_PORT environment variable)

51. Erlang
(caveats: You must be careful...)
Code swapping/replacement
– Be careful with spawn inside old (replaced) module... it will die!
– The code of a module can exist in two variants in a system: current
and old (fully qualified function calls always refer to current code)
– If a third instance of the module is loaded, the code server will remove
(purge) the old code and any processes
lingering in it will be terminated
-module(m).
-export([loop/0]).
loop() ->
receive
code_switch ->
m:loop();
Msg ->
% ...
loop()
end.

52. Undocumented features...
you must be careful!

53. Erlang
(caveats: You must be careful...)
Using async acceptors (prim_inet)
– gen_tcp:accept(Listen)
Usages: https://github.com/irr/erl-tutorials/blob/master/ts/ts1/src/ts.erl
– prim_inet:async_accept(Socket, -1)
Usages:
https://github.com/irr/erl-tutorials/blob/master/ts/ts2/src/ts.erl
http://svn.apache.org/viewvc/incubator/thrift/trunk/thrift_server.erl
http://www.trapexit.org/Building_a_Non-blocking_TCP_server_using_OTP_principles
”It is undocumented because it is an internal module
that is not ment to be called from applications. Its interface may
change without warning in even the smallest patch.”

54. Erlang
(caveats: You must be careful...)
Using socket in http mode
case gen_tcp:listen(Port, [binary,
{packet, http},
{reuseaddr, true},
{active, false},
{backlog, 30}]) of ...
case gen_tcp:recv(C#c.sock, 0, ?server_idle_timeout) of
{ok, {http_header, _, 'Content-Length', _, Val}} ->
...
{error, {http_error, "rn"}} ->
...
{ok, http_eoh} ->
...
Usage:
http://www.trapexit.org/A_fast_web_server_demonstrating_some_undocumented_Erlang_features

55. Erlang
(caveats: You must be careful...)
1> inet:ifget("eth0", [addr]).
{ok,[{addr,{192,168,1,101}}]}
2> inet:getiflist().
{ok,["lo","eth0"]}
3> inet_parse:ntoa({192,168,1,101}).
"192.168.1.101"
4> inet_parse:address("192.168.1.101").
{ok,{192,168,1,101}}

57. there is no
silver
bullet!

58. Any
doubts?