8. CoTweet Robot: Task
• Pull updates from Twitter API:
• n * 10,000 channels ( n is going up -- o/ )
9. CoTweet Robot: Task
• Pull updates from Twitter API:
• n * 10,000 channels ( n is going up -- o/ )
channel update req. 4 http requests on average
= n million http requests / hr
10. CoTweet Robot: Task
• Pull updates from Twitter API:
• n * 10,000 channels ( n is going up -- o/ )
• Minimize latency:
• Ideally < 300s between updates
11. CoTweet Robot: Task
• Pull updates from Twitter API:
• n * 10,000 channels ( n is going up -- o/ )
• Minimize latency (< 300s between updates)
• <blink>Survivable</blink> !
17. Robot Evolution: I
• Main process: receive/after loop:
service_loop() ->
spawn(?MODULE, do_work, []),
receive after SleepyTimeMS * 1000 -> ok end,
service_loop().
18. Robot Evolution: I
• Main process: receive/after loop
• Reads user info from database
• For every N records ...
19. Robot Evolution: I
• Main process: receive/after loop
• Reads user info from database
• For every N records:
• spawn() a worker to process bucket:
• Retrieve updates from Twitter API
• Insert into database (returning ID)
• Insert (using ID) into memcached
22. Single Assignment != Stateless
• gen_server, gen_event, gen_fsm: State
-record(State, {foo,bar}).
handle_call({set_foo, V}, _From, State) ->
{reply, State#state{ foo = V };
23. Single Assignment != Stateless
• ets
• Many concurrent reads? No problem.
start() ->
ets:new(state, [named_table, public]).
set_foo(V) ->
ets:insert(state, {foo,V}).
24. Single Assignment != Stateless
• ets
• Many concurrent reads? No problem.
• Many concurrent writes?
• May need to Serialize through a gen_server.
handle_call({set_foo, V}, _From, State) ->
ets:insert(state, {foo, V}),
{reply, State};
25. Single Assignment != Stateless
• gen_server, gen_event, gen_fsm: State
• ets: app-wide
• serialize writes through a gen_server
• mnesia: persistent
• process dictionary: put(K,V) / get(K)
• Tiny, transient, process-level
• (think: private):
27. Robot Evolution: II
• All state held in ets
• Robot comprises many ‘services:’
• Feeder: poll db as needed, update ets
• Scheduler: add tasks to work queue
• Business logic:
• even work load, limit resource(s)
• Dispatcher: service queue for workers
28. Hm, our “workers” may block waiting for
result(i.e., ID) of an insert from DB . . .
29. Hm, our “workers” may block waiting for
result(i.e., ID) of an insert from DB . . .
Needs more OTP.
30. Robot Evolution: III
• gen_event as simplest MUX ever
• no time for downtime: hot deployment
31. gen_event
• Extraordinarily easy to multiplex:
• Create an event manager: gen_event
• Add handlers
• Fire an event
• Not just for logging . . .
Christian Westbrook2010.1.12 ErLounge SF/Bay
cw@props.org meetup.com/erlounge
Thanks for coming out
Nothing technically wrong with this: did, in fact, get working w/ plists
- parallel processing, but not concurrent services
- stateless! robot has no idea what&#x2019;s going on, ever.
... one of the misconceptions some beginners have about erlang ->
all state: robot config, list of channels to service, perceived state of: { twitter api, db, cache }
Benefits: smooth workload rather than bursts tapering off each iteration of service loop
can mix in business logic to govern usage
But: worst case, worker procs can still block waiting for database -- precluding cache insert
