Overview of current high-performance Common Lisp web servers and implementation techniques, and description of a new hybrid approach to asynchronous I/O based on separate racing accept() and epoll() thread pools.

1. Developing high-performance network servers in Lisp Vladimir Sedach <vsedach@gmail.com>

3. Strings, character encoding, and memory

4. Concurrency and asynchrony

5. Overview of Common Lisp HTTP servers employing these techniques

7. http://pl.atyp.us/content/tech/servers.html

9. Process input

10. Write output

12. High-latency connections

13. Need to serve each client quickly

14. Need to serve many clients simultaneously

16. Server is focused on I/O

17. I/O is protocol-specific

19. 1.1: Persistent connection, possibly multiple (pipelined) request-responses per connection

20. Comet: Server waits to respond until some event occurs (faking two-way communication)

22. Memory copying is expensive

24. SBCL: 32 bits per character

25. read() returns vector of bytes

27. Do everything with byte vectors (DIY character encoding)

28. No need to copy/re-encode result of read()

29. Do encoding of output at compile-time with (compiler) macros

31. System calls also slow

32. Use writev() to output a list of byte vectors in one system call

34. Reuse input buffers!

35. In some Common Lisp implementations (SBCL), think about declaring inline and ftype for FFI read() etc input functions to cut down on consing (small overhead but adds up)

37. Asynchronous IO

39. Thread blocks and sleeps when waiting for input

40. Threads have a lot more memory overhead than continuations

41. Passing information between threads expensive – synchronization and context switch

42. But can (almost) pretend we're servicing one client at a time

44. When cannot read from one client connection, try reading from another

45. Basically coroutines

46. epoll/kqueue interfaces in Linux/BSD provide efficient mechanism that does readiness notification for many fds

48. Just loop polling/reading fds until a complete HTTP request has been received, then hand it off to a monolithic process_request() function

49. What happens when process_request() blocks on DB/file access?

50. What happens with more complicated protocols where input and processing are intertwined?

52. For true AIO, need to involve state handling code with application code (even for HTTP for things like DB access)

53. Not feasible with state machines

54. Trivial with first-class continuations

56. Context switch before any work can get done

57. Synchronization costs on task queue

59. (let (fd)

60. (loop (with-lock-held (server-lock)

61. (setf fd (accept server-socket)))

62. (process-connection fd)))

63. On Linux and Windows, accept() is thread-safe – no need for taking server-lock

65. http://paultyma.blogspot.com/2008/03/writing-java-multithreaded-servers.html

66. For many simultaneous connections, AIO consumes much less memory while making progress on input

67. In SBCL, AIO seems to win

69. Async (predates both nginx and lighttpd)

70. Connections managed by state machine

71. Multi-process, single-threaded

72. Doesn't use Lisp strings (UTF-8 in byte vectors)

73. Vectored I/O (incl support for HTTP pipelining – send multiple responses in one writev())

75. Async

76. Connections managed by continuations (code transformed with cl-cont)

77. Single-process, single-threaded

78. Doesn't use Lisp strings (UTF-8 in byte vectors)

79. Vectored I/O

81. Comes with macros for UTF-8 encoding at compile-time (incl. HTML generation macros)

82. Really fast on John Fremlin's benchmark – 11k requests/sec vs 8k r/s for nginx and embedded Perl (http://john.freml.in/teepeedee2-c10k)

84. Want to respond to initial request quickly

85. For persistent connections, there may or may not be another request coming, but the connections themselves pile up because of long keepalive times

86. Comet and long polling – assume huge number of connections sitting idle waiting for something to happen (ex - Web2.0 chat)

88. (Needs repository version of IOLib with my patches, available on IOLib mailing list)

89. Hybrid TPC/AIO architecture (new)

90. Single process, multi-threaded

91. Doesn't use Lisp strings (UTF-8 in byte vectors)

92. Not finished yet

93. Not as fast as TPD2, faster than nginx

95. If HTTP connection is persistent, hand it off to second LF thread pool demultiplexing epoll/kqueue to handle (possible) second and subsequent requests

96. For now the second thread pool still does TPC. Can make it do AIO WOLOG

98. Planning to look at SEDA for ideas on moving threads around between pools to handle load

100. Planned third LF thread pool to demultiplex synthetic server-side events to handle Comet requests

101. File handling

102. SSL

103. Gzip encoding