My talk from the Megacomm 2012 conference in Jerusalem, on February 16th, 2012. I describe the fundamental underpinnings of the Web, how things have changed on both the browser and server sides, and what these technologies mean for users..
Boost PC performance: How more available memory can improve productivity
Modern Web technologies (and why you should care): Megacomm, Jerusalem, February 2012
1. Modern Web
Technologies (and why
you should care)
Reuven M. Lerner • reuven@lerner.co.il
Megacomm
February 15th, 2012
2. Who am I?
• Web developer since 1993
• Software architect/consultant/trainer
• Linux Journal columnist since 1996
• Mostly Ruby on Rails + PostgreSQL, but
also Python, jQuery, MySQL, and more...
• PhD candidate at Northwestern University
3. How does the Internet
(TCP/IP) work?
“Socket”
Client Server
Port Port
4. How does the Internet
(TCP/IP) work?
“Socket”
Client Server
Port Port
Client opens
connection
5. How does the Internet
(TCP/IP) work?
“Socket”
Client Server
Port Port
Client opens Server accepts
connection connection
6. Protocols
• Communication standards built on top of
TCP/IP, typically text-based
• SMTP (e-mail)
• FTP (file transfer)
• NNTP (transfer of “news” articles)
20. Submitting forms
POST /login
name=reuven&password=secret
Browser Server
21. Submitting forms
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
22. Document = request
• If an HTML page contains images, then each
is retrieved in a separate HTTP request
• Page with 30 images = 31 HTTP requests
• Page with 20 images, 10 JavaScript files,
and 5 CSS files = 36 HTTP requests
23.
24. Idea: Lie to the browser
• Don’t return a document to the user
• Rather, run a program when the user makes
a request, and send the program’s output
• If the output is in HTML, then the browser
will show it no differently than a static doc
25. Just in time production
• Wait as long as
possible to create
pages for the user
• Ideally, create them
when the user
requests them
• “Mass customization”
28. Dynamic document
GET /
Browser Server
200 OK
<html>
<head>...</head>
<body>...</body>
</html>
29. Dynamic document
GET /
Browser Server
200 OK
<html>
<head>...</head>
Program output
<body>...</body>
</html>
30.
31.
32. What we return
• Usually HTML
• Image (e.g., stock graphs, Google Analytics)
• Word/Excel doc (e.g., from Google docs)
• PDF (e.g., PDF reports)
• XML, JSON (for computers, not people)
• Basically, any defined MIME type
33. APIs and mobile apps
Computer A Computer B
(browser) (server)
34. APIs and mobile apps
GET /
Computer A Computer B
(browser) (server)
35. APIs and mobile apps
GET /
Computer A Computer B
(browser) (server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
36. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
37. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
38. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
39. APIs and mobile apps
GET /
Computer B
(server)
200 OK
<some-xml>
<talk>JWP</talk>
</some-xml>
40. What is a Web app?
• Receives its inputs via HTTP
• Sends its output via HTTP
• That’s it! A Web application can do
anything you want, within these limits
41. That’s it!
• Now you understand how the Web works.
• Really, that’s it.
• Go home.
42. OK, perhaps not...
• How do we write these programs?
• Where (and how) do we store user data?
• How have the underlying technologies
(URLs, HTTP, and HTML) advanced?
43. Early Web applications
• First server-side programs were in C
• They were compiled into binaries
• So you had the CGI source (cgi-src)
directory...
• ... and the CGI binary (cgi-bin) directory
44. “Scripting” languages
• No explicit compilation
• Cross platform
• Built-in, powerful text functions
• Do a lot in a little bit of code
• Perl, Python, PHP, Ruby
• Typically open source
45. Frameworks
• DRY (Don’t repeat yourself)
• Get rid of the drudgery
• Concentrate on your business, rather than
worrying about common Web issues
46. MVC paradigm
• Most modern frameworks use MVC
• From Smalltalk in the 1980s!
• Model — communicates with database
• View — HTML/JavaScript/CSS for user
• Controller — handles requests
• Clear division of labor
47. Web frameworks in
dynamic languages
• Programmer speed trumps execution
speed
• Community support
• Plugins for commonly requested features
• Create a domain-specific language (DSL)
for your specific needs
48. Ruby on Rails
• Ruby language
• Rails Web app framework (MVC)
• Designed for writing Web DSLs
• “Convention over configuration”
• Thousands of little improvements
• ActiveRecord — ORM
50. Where’s the definition?
• The computer takes care of it automatically
• ActiveRecord knows what columns you
have defined, and what their types are
• (More on columns later)
• Only write things that cannot be
understood automatically
51. Not only Rails
• Python (Django)
• PHP (Symfony)
• Perl (Catalyst)
• Groovy (Grails)
• Even if you don’t use Ruby on Rails, you
have benefitted from its “opinions”
52. Plugins
• Rails (and other systems) have open-source
plugins to handle common issues
• Authentication
• E-commerce
• Social networking
• Don’t write these yourself! Customize
existing code that has proved itself
53. Storage
• Applications are great!
• But what if we want to store information
about our users?
• Name, e-mail address, account balance
• We could use text files, but most people
will use a database
54. What is a database?
Store data
confidently
Database
Retrieve data
flexibly
55. Relational databases
Define tables,
store data in them
Database
Retrieve data from
related tables
57. Relational databases
• Everything is stored in 2-dimensional tables
• Data should appear only once (normalized)
• “Join” tables to connect tables
• Technology is extremely robust, fail-safe
• Not all data is a good fit for this paradigm
58. id first_name last_name phone
1 Reuven Lerner 054-496-8405
2 Charlie Kalech 02-671-9918
60. id first_name last_name email
1 Reuven Lerner reuven@lerner.co.il
2 Charlie Kalech charlie@j-town.com
person_id phone_number_type_id phone_number id type
1 2 054-496-8405
1 1 847-230-9795 1 work
2 1 02-671-9918
2 mobile
2 2 054-803-3356
2 3 501-629-8620 3 fax
4 home
61. id first_name last_name email
1 Reuven Lerner reuven@lerner.co.il
2 Charlie Kalech charlie@j-town.com
person_id phone_number_type_id phone_number id type
1 2 054-496-8405
1 1 847-230-9795 1 work
2 1 02-671-9918
2 mobile
2 2 054-803-3356
2 3 501-629-8620 3 fax
4 home
SELECT P.first_name, P.last_name, P.email,
PN.phone_number, PNT.type
FROM People P, Phone_Numbers PN,
Phone_Number_Types PNT
WHERE PN.person_id = P.id
AND PNT.id = PN.phone_number_type_id
62. Another language!
• SQL is the language of relational databases
• So a Web app will use a language (e.g.,
Ruby, Python, or PHP) + SQL
• Or use an ORM, which automatically
translates your language into SQL
Person.first.phone_number
73. Scaling problems
• Lots of requests?
• Optimize
• Even more requests?
• Buy a bigger server
• What next?
• Panic! (Or spend lots of money)
74. Sharding
• Split your data across multiple databases
• This works, but...
• Requires rewriting a lot of code
• Maintenance is a big issue
• Re-sharding as each server gets
overwhelmed can be expensive, time-
consuming
75. Non-relational
databases
• Don’t store things in tables!
• Don’t pre-define a schema
• No SQL!
• Indeed, known as “NoSQL” databases
• Only common factor: No SQL, non-
relational
77. MapReduce / Hadoop
• Google and Yahoo do it like this:
• Split data across many servers
• Run a function on all of those servers
• Collect the results
• Display to the user!
• (Too slow? Add more servers!)
78. NoSQL: Good news
• Often easier to administer, configure
• Scaling to multiple servers is a no-brainer
• No new programming language (SQL)!
• Better fit for certain kinds of data
• Better performance than a relational DB
• Lots of options to choose from!
79. NoSQL: Bad news
• Speed is in the eye of the beholder
• Is “eventually consistent” good enough?
• Non-normalized data — ugh!
• Reporting can be harder
• Less tested than relational databases
• Can you trust your data to them?
80. Meanwhile...
• Our browsers are displaying HTML
• HTML had several problems:
• Standardized set of tags
• Making it easy for programs to parse
• Semantic, display content were mixed
81. HTML standards
• HTML — several versions, several
standards, none universally accepted
• XML — lets us create HTML-like
languages, for computer conversations
• XHTML — HTML for pedantic people
• It was a big mess!
82. HTML5
• Relaxes much of the formality of XHTML,
while remaining easy for computers to read
• Backward compatible to a large degree
• Adds a number of tags for better semantics
• Best of all: Lots of new JavaScript goodies
• More on this in a moment
94. And more
• Validation — built-in validation of form
element inputs, via regular expressions
• No more JavaScript plugins!
• Sliders — more natural numeric inputs
• Canvas — draw any picture you might like,
and detect/change it with software
• Hints in text fields (e.g., “search”)
95. My favorite
• Private data in attributes!
• Any attribute starting with “data-” is
considered valid
• A great way to stash information inside of
HTML elements without violating standards
96. Oh, yeah
• These don’t all work.
• Many of them don’t work on any browser
• Most work on only some browsers.
• What to do? Wait. Or use Modernizr,
which uses JavaScript to detect features.
• If a feature isn’t there, you can fall back
97. CSS
• Cascading Style Sheets
• Split semantic markup from presentation
• One CSS file can apply to an entire site
• No more “style” tags in your text!
• Easy to move place things, create effects
100. It gets better
• You can set styles for when the user’s
mouse hovers over or clicks on an element
• In other words: Cheap animation!
• Many uses of JavaScript (e.g., some menus)
can now be done with simple CSS
• You can make beautiful sites with CSS
101. CSS3: Cool effects
• Rounded corners
• Transparency
• Text shadows and drop shadows
• Gradients
102. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
103. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
Equals
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
104. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
Starts with
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
105. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
p[id^=”foo”] { background: green}
Ends with
p[id$=”foo”] { background: green}
p[id*=”foo”] { background: green}
106. CSS3: Cool selectors
• If you love regular expressions, then these
selectors will be second nature to you:
p[id=”foo”] { background: green}
p[id^=”foo”] { background: green}
p[id$=”foo”] { background: green}
Contains
p[id*=”foo”] { background: green}
107. OK, that’s nice.
• But really, the big news with HTML5
doesn’t have to do with HTML at all.
• Instead, it has to do with JavaScript.
• Remember JavaScript?
• It’s the programming language that we love
to hate. (At least, I used to.)
108. JavaScript
• Originally “LiveScript,” a language that
executes programs in the browser
• Renamed “JavaScript” when an unrelated
language (“Java”) stole the thunder
• Renamed (officially) ECMAScript for
standardization purposes
• No one actually calls it this
109. HTML5 turns the
browser into a smart,
powerful, networked
application platform.
JavaScript makes it
possible.
110. Powerful? Huh?
• Didn’t I just say that I love to hate
JavaScript?
• And then I said that it was powerful?
• What gives?
111. JavaScript is
the new hotness
• Browsers are competing for fastest, best
• Google’s V8
• Mozilla’s TraceMonkey (and JägerMonkey)
• Safari’s Nitro
• IE’s Chakra
• JavaScript is faster, more stable than ever!
112. Server-side JavaScript!
• The latest JavaScript development
• Run it on your server!
• Why? Because it’s super-fast
113. Also: frameworks
• Remove the drudgery of JavaScript
• Handle differences between browsers
• Make it easy to perform common tasks
• Lots of plugins available
• For me, it’s the difference between pain and
pleasure when working with JavaScript
115. jQuery
• jQuery has taken the world by storm
• Super-easy to use
• Extremely fast
• Open source (of course!)
• Easy to write plugins
• Lots of plugins are available
116. Ajax
• One reason for JavaScript libraries: Ajax!
• Make HTTP requests from within the page
• No refresh! Just get results from the
server, and modify the page accordingly
• This has revolutionized our view of Web
pages
118. Ajax
POST /login
name=reuven&password=secret
Browser Server
119. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
120. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
121. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
122. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
123. Ajax
POST /login
name=reuven&password=secret
Browser Server
200 OK
<html>...
<p>Thank you!</p>
...</html>
124. Ajax isn’t everything
• What if I want a chat application, or
something else that stays open?
• What if I want to execute more than one
JavaScript function at a time?
• What if I want to store things locally?
• HTML5 provides all this — and more!
125. Canvas
• A complete drawing area, in your browser!
• Use JavaScript to:
• Draw arbitrary shapes
• Detect the mouse
• Detect the drawing
• The end of Flash? Maybe...
126. Geolocation
• Your browser can know where you are!
• It can send this info to the server
• It’s not perfect, but still pretty good
• To avoid privacy issues, users are always
asked if their location should be shared
127. Inter-page
communication
• Modern Web apps can span multiple pages
• HTML5 makes it easy for two pages (from
the same server) to communicate
• The receiver knows which server sent the
data — so it can filter incoming messages,
as well as screen them for security
128. Web sockets
• This is potentially the biggest deal of all
• Ajax allows for server connections. But:
• High overhead
• Stateless
• Web sockets have low overhead, and they
stay open as long as you need!
131. Using Web sockets
var weatherSocket = new
WebSocket("ws://localhost:8080");
weatherSocket.onopen = function(e)
{ alert("Opened weather socket");};
132. Using Web sockets
var weatherSocket = new
WebSocket("ws://localhost:8080");
weatherSocket.onopen = function(e)
{ alert("Opened weather socket");};
weatherSocket.onmessage =
function(e) { alert("Got message: "
+ e.data);};
133. Using Web sockets
var weatherSocket = new
WebSocket("ws://localhost:8080");
weatherSocket.onopen = function(e)
{ alert("Opened weather socket");};
weatherSocket.onmessage =
function(e) { alert("Got message: "
+ e.data);};
weatherSocket.onclose = function(e)
{ alert("Closing socket..."); };
134. What can sockets do?
• Chat servers
• Stock feeds
• Teleconferencing
• Who knows?
• Remember, HTTP was only invented after
sockets had been around for 15 years
135. Web workers
• Execute more than one thing at a time
• In other words:You can run JavaScript
functions in the background
• Process text
• Open Web sockets
• Perform calculations
136. Local storage
• Now Web apps can store data
• A little database of name-value pairs
var foo = localStorage.getItem("bar");
localStorage.setItem("bar", foo);
137. Local storage
• Now Web apps can store data
• A little database of name-value pairs
var foo = localStorage.getItem("bar");
localStorage.setItem("bar", foo);
var foo = localStorage["bar"];
138. Local storage
• Now Web apps can store data
• A little database of name-value pairs
var foo = localStorage.getItem("bar");
localStorage.setItem("bar", foo);
var foo = localStorage["bar"];
localStorage["bar"] = foo;
139. Media
• Standard (well, sort of) ways to play audio
and video
• No more Flash!
• Problem: No one format is supported by all
browsers
142. Don’t forget mobile!
• iOS and Android are growing massively
• Web site vs. native app?
• (Ask Jakob Nielsen — for now, apps are
better, but that won’t last for long)
• Ignore these users at your peril
143. Want a Web app?
• It used to be:
• “What operating system, language, and
database will I use?”
• Or:
• “How can I produce an interesting Web
site?”
144. Now it’s:
• What experience do we want to give
people?
• What will they be using to access our
system?
145. Those lead to a wide
variety of questions:
• What server language and framework?
JavaScript framework? Hosted or cloud?
• What type of database (relational, NoSQL)?
Which one? Hosted or cloud?
• Do we offer an API? A mobile app? Both?
• Which HTML5 features do we want to
use, and how do we gracefully degrade?
146. The key takeaway
• Web sites are far more than just static
text, blogs, or forums
• They’re full-fledged software applications
• Take advantage of these technologies, and
you can create a fabulous experience
• (Don’t take advantage of them, and your
competitors will!)
147. Oh, yeah: Testing
• Automated testing is amazing
• Your Web site should use it
• Most modern frameworks support some
sort of testing — if not, change framework
• Catch dumb bugs and issues before your
customers do!
• Faster and cheaper than people
148. My brain is too small!
• Yes, there’s a lot to learn
• Most of it can wait a little bit
• There are oodles of tutorials
and books that can help you
• Besides a lot of this is still
highly transitional
149. Whew!
• There’s a lot to the modern Web
• It’s fun and exciting, and continues to move
forward at breakneck speed
• Understanding as many of these parts as
possible will help you make better
decisions, and better applications!