3. The Law of Accelerating Returns
The price-performance, capacity & bandwidth of information
technologies progresses exponentially through multiple paradigm
shifts
Specific to information technology
o not to arbitrary exponential trends (like population)
o Still need to test viability of the next paradigm
A scientific theory
o 25 years of research
o Part of a broader theory of evolution
o Inventing: science and engineering
Moore’s law just one example of many
Yes there are limits
o But they’re not very limiting
o Based on the physics of computation and communication
o and on working paradigms (such as nanotubes)
3
15. Information Technologies (of all kinds)
double their power
(price performance, capacity, bandwidth)
every year
15
16. A Personal Experience
Measure MIT’s IBM 7094 Notebook Circa 2003
Year 1967 2003
Processor Speed (MIPS) 0.25 1,000
Main Memory (K Bytes) 144 256,000
Approximate Cost (2003 $) $11,000,000 $2,000
24 Doublings of Price-Performance in 36 years, doubling time: 18
months not including vastly greater RAM memory, disk storage,
instruction set, etc.
16
32. Doubling (or Halving) times
• Dynamic RAM Memory “Half Pitch” Feature Size 5.4 years
• Dynamic RAM Memory (bits per dollar) 1.5 years
• Average Transistor Price 1.6 years
• Microprocessor Cost per Transistor Cycle 1.1 years
• Total Bits Shipped 1.1 years
• Processor Performance in MIPS 1.8 years
• Transistors in Intel Microprocessors 2.0 years
• Microprocessor Clock Speed 2.7 years
32
57. The (converging) Sources of the Templates
of Intelligence
AI research
Reverse Engineering the Brain
Research into performance of the brain
(human thought)
Language: an ideal laboratory for studying human
ability for hierarchical, symbolic, recursive
thinking
All of these expand the AI tool kit
57
61. “Now, for the first time, we are
observing the brain at work in a global
manner with such clarity that we should
be able to discover the overall programs
behind its magnificent powers.”
-- J.G. Taylor, B. Horwitz, K.J. Friston
61
62. Ways that the brain differs from a
conventional computer:
Very few cycles available to make decisions
Massively parallel: 100 trillion interneuronal
connections
Combines digital & analog phenomena at
every level
Nonlinear dynamics can be modeled using digital
computation to any desired degree of accuracy
Benefits of modeling using transistors in their
analog native mode
62
63. Ways that the brain differs from a
conventional computer:
The brain is self-organizing at every level
Great deal of stochastic (random within
controlled constraints) process in every aspect
Self-organizing, stochastic techniques are routinely
used in pattern recognition
Information storage is holographic in its
properties
63
64. The brain’s design is a level of
complexity we can manage
Only about 20 megabytes of compressed design
information about the brain in the genome
A brain has ~ billion times more information than the genome
that describes its design
The brain’s design is a probabilistic fractal
We’ve already created simulations of ~ 20 regions (out
of several hundred) of the brain
64
66. Models often get simpler at a
higher level, not more complex
Consider an analogy with a computer
We do need to understand the detailed physics of
semiconductors to model a transistor, and the
equations underlying a single real transistor are
complex.
A digital circuit that multiplies two numbers,
however, although involving hundreds of
transistors, can be modeled far more simply.
66
67. Modeling Systems at the Right Level
Although chemistry is theoretically based on physics,
and could be derived entirely from physics, this would
be unwieldy and infeasible in practice.
So chemistry uses its own rules and models.
We should be able to deduce the laws of
thermodynamics from physics, but this is far from
straightforward.
Once we have a sufficient number of particles to call it a gas
rather than a bunch of particles, solving equations for each
particle interaction becomes hopeless, whereas the laws of
thermodynamics work quite well.
67
68. Modeling Systems at the Right Level
The same issue applies to the levels of modeling and
understanding in the brain – from the physics of
synaptic reactions up to the transformations of
information by neural clusters.
Often, the lower level is more complex.
A pancreatic islet cell is enormously complicated. Yet
modeling what a pancreas does (in terms of regulating
levels of insulin and digestive enzymes) is
considerably less complex than a detailed model of a
single islet cell.
68
70. The Cerebellum
The basic wiring method of the
cerebellum is repeated billions of times.
It is clear that the genome does not
provide specific information about each
repetition of this cerebellar structure
but rather specifies certain constraints as to
how this structure is repeated
o just as the genome does not specify the exact
location of cells in other organs, such the location
of each pancreatic Islet cell in the pancreas
70
71. The Cerebellum
Gathering data from
multiple studies, Javier F.
Medina, Michael D. Mauk,
and their colleagues at the
University of Texas Medical
School devised a detailed
bottom-up simulation of
the cerebellum.
Their simulation includes
over 10,000 simulated
neurons and 300,000
synapses, and includes all
of the principal types of
cerebellum cells.
71
72. The Law of Accelerating Returns is
driving economic growth
The portion of a product or service’s value comprised
of information is asymptoting to 100%
The cost of information at every level incurs deflation
at ~ 50% per year
This is a powerful deflationary force
Completely different from the deflation in the 1929
Depression (collapse of consumer confidence & money
supply)
72
81. Solar Energy
Emerging nanotechnology will accelerate progress of
cost of solar panels and storage – fuel cells
Tipping point (cost per watt less than oil and coal)
expected within 5 years
Progress on thermo-solar
Doubling time for watts from solar < 2 years
We are less than 10 doublings from meeting 100% of the
world’s energy needs
81
91. First All Solar City
Masdar City
- 50,000 people
- People will move in by 2010
-Carbon Neutral
-In Abu Dhabi
91
92. Contemporary examples of
self-organizing systems
The bulk of human intelligence is based on
pattern recognition: the quintessential
example of self-organization
92
93. Contemporary examples of
self-organizing systems
Machines are rapidly improving in pattern recognition
Progress will be accelerated now that we have the
tools to reverse engineer the brain
Human pattern recognition is limited to certain types
of patterns (faces, speech sounds, etc.)
Machines can apply pattern recognition to any type of
pattern
Humans are limited to a couple dozen variables,
machines can consider thousands simultaneously
93
96. 2010: Computers disappear
Images written directly to our retinas
Ubiquitous high bandwidth connection to the Internet
at all times
Electronics so tiny it’s embedded in the environment,
our clothing, our eyeglasses
Full immersion visual-auditory virtual reality
Augmented real reality
Interaction with virtual personalities as a primary
interface
Effective language technologies
96
98. 2029: An intimate merger
$1,000 of computation = 1,000 times the human brain
Reverse engineering of the human brain completed
Computers pass the Turing test
Nonbiological intelligence combines
the subtlety and pattern recognition strength of human
intelligence, with
the speed, memory, and knowledge sharing of machine
intelligence
Nonbiological intelligence will continue to grow
exponentially whereas biological intelligence is
effectively fixed
98
99. Nanobots provide…
Neural implants that are:
Noninvasive, surgery-free
Distributed to millions or billions of points in the brain
Full-immersion virtual reality incorporating all of the
senses
You can be someone else
“Experience Beamers”
Expansion of human intelligence
Multiply our 100 trillion connections many fold
Intimate connection to diverse forms of nonbiological
intelligence
99
100. Average Life Expectancy (Years)
Cro Magnon 18
Ancient Egypt 25
1400 Europe 30
1800 Europe & U.S. 37
1900 U.S. 48
2002 U.S. 78
100
101. Reference URLs:
Graphs available at:
www.KurzweilAI.net/pps/Google/
Home of the Big Thinkers:
www.KurzweilAI.net
101
103. The Criticism from Malthus
“Exponential trends can’t go on forever” (rabbits in Australia…)
Law of accelerating returns applies to information technologies
There are limits
o But they’re not very limiting
One paradigm leads to another….but
o Need to verify the viability of a new paradigm
o Molecular computing is already working
o Nanotube system with self-organizing features due to hit
the market next year
o Molecular computing not even needed: strong…cheap… AI
feasible with conventional chips according to ITRS
o Exotic technologies not needed
103
104. The Criticism from software
“Software / AI is stuck in the mud”
Computers still can’t do…..(fill in the blank)
The history of AI is the opposite of human maturation
o CMU’s GPS in the 1950’s solved hard adult math problems (that
stumped Russell & Whitehead)
o But computers could not match a young child in basic pattern
recognition
o This is the heart of human intelligence
o Tell the difference between a dog and a cat?
104
105. The Criticism from software cont.
Hundreds of AI applications deeply embedded in our
economic infrastructure
CAD, just in time, robotic assembly, billions of $ of daily
financial transactions, automated ECG, blood cell image
analysis, email routing, cell connections, landing airplanes,
autonomous weapons…..
If all the AI programs stopped….
These were all research projects when we had the last
summit in 1999
105
106. The Criticism from software cont.
“AI is the study of how to make computers do things at which, at
the moment, people are better.” - Elaine Rich
Unsolved Problems have a mystery
Intelligence also has a mystery about it…
As soon we know how to solve a problem, we no longer consider it
“intelligence”
“At first I thought that you had done something clever, but I see
that there was nothing in it, after all” – said to Sherlock Holmes
“I begin to think that I make a mistake in explaining.” – Sherlock
Holmes
106
107. The Criticism from software cont.
Software complexity and performance is improving
Especially in the key area of pattern recognition
o Only recently that brain reverse-engineering has been helpful
Take chess, for example
The saga of Deep Fritz
With only 1% of the computes of Deep Blue, it was equal in
performance
o Equal in computes to Deep Thought yet it rated 400 points
higher on chess rating (a log scale)
o How was this possible: Smarter pattern recognition software
applied to terminal leaf pruning in minimax algorithm
Or autonomous vehicles….and weapons
107
108. The Criticism from software cont.
Genetic Algorithms
Good laboratory for studying evolution
More intelligence from less
GA’s have become more complex, more capable
o Evolving the means of evolving
o Not just evolving the content of the genetic code but
adding new genes
o Reassigning the interpretation of genes
o Using codes to control gene expression
o Means to overcome over fitting to spurious data
o Larger genomes
But GA’s are not a silver bullet
o One self-organizing technique of many
108
109. The Criticism from software cont.
Military technology: steady increase of sophisticated
autonomous weapons
Software productivity exponentially increasing
Algorithms getting more sophisticated (e.g., search,
autocorrelation, compression, wavelets)
Measures of software complexity (log scale) increasing
steadily
Combined impact of:
Increasingly complex pattern recognition methods
o Starting to be influenced by biologically inspired paradigms
Vast data mining not feasible just 7 years ago
109
110. The criticism from reliability
“Software is too brittle, too crash prone”
(Jaron Lanier, Thomas Ray)
We CAN (and do) create reliable software
o Intensive care, 911, landing airplanes
o No airplane has crashed due to software crashes despite
software being responsible for most landings
Decentralized self-organizing systems are
inherently stable
o The downtime for the Internet over the last decade is
zero seconds
110
111. The criticism from the
complexity of brain processing
The complexity of all the nonlinearities (ion channels,
etc) in the brain is too complex for our technology to
model (according to Anthony Bell, Thomas Ray)
According to Thomas Ray, strong AI will need “billions
of lines of code”
But the genome has only 30-100 million bytes of compressed
code
The Brain is a recursive probabilistic fractal
Example: The Cerebellum
111
112. The criticism from micro tubules and
quantum computing
Human thinking requires quantum computing and that
is only possible in biological structures (i.e., tubules)
(according to Roger Penrose)
No evidence that quantum computing takes places in the
tubules
Human thinking does not show quantum computing
capabilities
Even if it were true, it would not be a barrier
Would just show that quantum computing is feasible
Nothing to restrict it to biological structures
112
113. The criticism from Ontology
John Searle’s Chinese Room:
“Because the program is purely formal or
syntactical and because minds have mental or
semantic contents, any attempt to produce a mind
purely with computers programs leaves out the
essential features of the mind.” – John Searle
o Searle ignores the emergent features of a complex,
dynamic system
o Can apply Searle’s argument to show that the human
brain “has no understanding”
113
114. Promise versus Peril
GNR enables our creativity
and our destructiveness
Ethical guidelines do work to protect
against inadvertent problems
30 year success of Asilomar Guidelines
114
115. Promise versus Peril cont.
So what about advertent problems
(asymmetric warfare)?
Designer pathogens, self-replicating nanotech,
unfriendly AI (Yudkowsky)….
So maybe we should relinquish these dangerous
technologies?
3 problems with that:
o Would require a totalitarian system
o Would deprive the world of profound benefits
o Wouldn’t work
o Would drive dangerous technologies underground
o Would deprive responsible scientists of the tools
needed for defense
115
116. Promise versus Peril cont.
So how do we protect ourselves?
Narrow relinquishment of dangerous
information
Invest in the defenses….
116
117. New York Times Op-Ed "Recipe for Destruction,"
by Ray Kurzweil and Bill Joy, October 17, 2005
117
118. “Enough”
“Is it possible that our technological reach is very
nearly sufficient now? That our lives, at least in the
West, are sufficiently comfortable.” (Bill McKibben)
My view: not until we…
can meet our energy needs through clean, renewable
methods (which nanotech can provide)
overcome disease….
o …and death
overcome poverty, etc.
Only technology – advanced, nanoscale, distributed,
decentralized, self-organizing, increasingly
intelligent technology – has the scale to overcome
these problems.
118
119. Okay, let’s say that overcoming disease is a
good thing, but perhaps we should stop before
transcending normal human abilities….
So just what is normal?
Going beyond “normal” is not a new story.
o Most of the audience wouldn’t be here if life expectancy
hadn’t increased (the rest of you would be senior
citizens)
We are the species that goes beyond our
limitations
o We need not define human by our limitations
“Death gives meaning to life…and to time”
o But we get true meaning from knowledge: art, music,
science, technology
119
120. Scientists: “We are not unique”
Universe doesn’t revolve around the Earth
We are not descended from the Gods
o But from apes….worms….bacteria…dust
But we are unique after all
We are the only species that creates
knowledge….art, music, science,
technology…
o Which is expanding exponentially
120
121. So is the take-off hard or soft?
Exponential growth is soft…
Gradual…
Incremental…
Smooth…
Mathematically identical at each point…
But ultimately, profoundly
transformative
121
122. Reference URLs:
Graphs available at:
www.KurzweilAI.net/pps/Google/
Home of the Big Thinkers:
www.KurzweilAI.net
122
Notas del editor
Total U.S. cellphone subscribers has doubled about every 2.3 years. Of course, as the US market becomes saturated, we're seeing the domestic growth rate decline (as noted in logarithmic plot), but even as we approach 90% penetration (255 million subscribers/300 million population = 85%), the U.S. is still adding about 20 million subscribers per year (the country didn't have 20 million total subscribers until 1994).
Calculations per second per $1000 have been following a double exponential trend in the past century, and over that period have been doubling every 2.1 years.
From 1991 through 2008, peak performance of supercomputers has doubled about every year. IBM's Roadrunner in 2008 is more than 250,000 times more powerful than the top supercomputer in 1991 and more than 83 million times more powerful than the Cray-1 in 1983. Roadrunner is equivalent to 10% of my 10^16 flops estimate of a human brain. Various projects aim for a computer within the decade that will exceed the human brain's processing capabilities.
The number of transistors per microprocessor chip has been doubling every 23 months since 1971. The Itanium "Tukwila" ("The next Itanium chip"), due in late 2008 or early 2009, will have about one million times as many transistors as the 4004 chip had in 1971.
Since 1971's 4004 chip, processor performance has been doubling about every 21 months, rising 1.7 million-fold through 2007.
Since 1971, Dynamic RAM Memory cost per bit has halved every 1.9 years, dropping by a factor of about a half-million between 1971 and 2008--a double exponential. So for the cost of 200 bits in 1971, you can get around 112 million bits now in 2008.
Since 1950, the cost of RAM has halved every 20 months -- a 25-billion-fold drop. Since 1975, the pace of decline has accelerated, halving every year-and-a-half.
Average Transistor price has halved about every year-and-a-half since 1968, falling some 60-million-fold over the 40-year period from 1968 to 2008
DRAM "half pitch" feature size (the distance between cells on a DRAM chip) has halved every 5 years since the late 1960's, dropping by a factor of 200 over 39 years. ITRS projects this feature size will continue dropping at about this rate through the early 2020's.
Dynamic RAM Memory "Half Pitch" Feature Size shows a consistent halving time of 5.4 years in feature size over 55 years (16 of which are based on forecast data). The halving time is 5.2 years using only 1967-2004. History and big-picture from ITRS 2007, Page 63: "Historically, DRAM products have been recognized as the technology drivers for the entire semiconductor industry. Prior to the late-1990s, logic (as exemplified by MPU) technology moved at the same pace as DRAM technology, but after 2000/180 nm began moving at a slower 2.5-year technology cycle pace, while DRAM technology continued on the accelerated two-year pace. During the last few years, the development rate of new technologies used to manufacture microprocessors has continued on the 2.5-year pace, while DRAMs are now forecast to slow to a three-year cycle pace through the 2020 Roadmap horizon. By moving on the faster 2.5-year cycle pace, microprocessor products are closing the half-pitch technology gap with DRAM,"
Total bits of memory shipped has doubled about every year since 1971. In 2007, about 6 billion times more bits were shipped than in 1971; the number shipped in 2007 is about equal to all the shipments between 1971 and 2005. We're now shipping what was a generation's worth--20 years of bits--in one year.
As noted in SIN, growth in the price-performance of magnetic data storage is not a result of Moore's Law. This exponential trend reflects the squeezing of data onto a magnetic substrate, rather than transistors onto an integrated circuit, a completely different technical challenge pursued by different engineering and different companies. The price-performance has followed a Moore's-Law-like trend, doubling every 1.8 years since 1956. In that year, a dollar could buy 200 bits. In 2008, that same dollar could buy 66 billion bits, or over 300 million times more storage per dollar.
Sequencing cost per base pair has been halving every 1.3 years since 1971, falling by a factor of over 100 million between the 1970's and 2008. The pace of price decline has displayed a double-exponential trend, so that since 1998 the halving rate has accelerated to about every 8 months. What would have cost $100 in the mid-1970's, and about $10 in 1990 fell to a dime in 2000 and 2 thousandths of a penny in 2008.
From 1982 to mid-2008, the number of base pairs and the number of sequences in GenBank's database have grown by about 60% per year, doubling every 18 months.
Doubling time for U.S. Internet data traffic has decreased from 1 year in the last chart to 10 months, exponentially doubling every 10 months since 1990. It has increased about 1.25 million-fold since 1990.
The capacity of the internet backbone is experiencing double-exponential growth. Since 1979 it's grown nearly 1-million-fold, to 40 Gigabytes per Second as of 2006 [using SONET OC-768 equipment]. The growth rate itself is accelerating. It took 5 years to go from 2.5 gigabytes/second to 10 gigabytes/second. It only took 3 to quadruple again, to 40 gigabytes/second. So we're already at a doubling rate of every year-and-a-half, and accelerating.
Calculations per second per $1000 have been following a double exponential trend in the past century, and over that period have been doubling every 2.1 years.
Over the past 80 years, the US economy has been growing at 3.1% per year. Since that growth is compounding, it now takes us about 8 (optional: 3.5) years to grow what the entire economy was in 1946 (optional: 1929), when the US was considered a very rich country. You can see economic growth, too, is a series of s-curves, punctuated by semi-regular recessions, as the economy retools and changes direction. Like a runner who has to stop every so often to check the map. The dot-com or railroad bubbles might have been bad for many investors but bubbles tend to rapidly commercialize new technologies. So even the bad times are important for technological progress.
Here’s a chart of per-capita growth over the past 80 years. Income has been doubling about every generation, an unprecedented rate in history. What's driving this acceleration is productivity growth. New technologies and innovations allow us to use our hours more productively, as we trade our shovels for harvesters, our secretary pools for gmail, and our typewriters for word processors.
Output per hour in manufacturing, one of the critical factor that makes us richer, has been growing at a rate of 2.9% since 1949, a 24-year doubling rate. Since the mid-1990's, productivity growth has sped up, to nearly 4.2%, which would be a doubling rate of about 17 years.
Here's a chart showing patent applications to the USPTO (or, "US Patent and Trademark Office"). You can see the exponential growth.There was a brief dip during both world wars, since lots of R&D went into the military, and that doesn't get patented. There was also a prolonged dip during the depression, since corporate R&D is sensitive to economic conditions. You can see that since the 1980's, we've had a real take-off in applications, suggesting a speed-up in innovation (this is clear with either linear and semi-log).
Annual PV production has been doubling every 2 years since 1996, a 40% CAGR. The growth rate has been double-exponential growth, itself doubling every 9 years.
Annual PV production has been doubling every 2 years since 1996, a 40% CAGR. The growth rate has been double-exponential growth, itself doubling every 9 years.
Since 1995, worldwide growth in PV production has been 37% per year. China and Taiwan have seen growth of over 100% per year, going from 1.7% of world production to nearly 22%. The US has actually been the slowest major economy, with 17% CAGR over the period, unsurprising since so much technology manufacturing is basing in Asia. (sidenote: US growth has NOT been double-exponential, though worldwide growth has been double-exponential.)
Since 2000, the main solar markets have seen exponential growth. Japan, the most mature market in 2000, has seen the lowest growth, a still-healthy 27% CAGR (doubling rate of under 3 years). Germany, and Europe in general, has seen annual growth rates in excess of 60%, a doubling rate of around 11-17 months, spurred by government subsidies as well as by high utility rates. The US, at a 48% CAGR, and a doubling rate of just under 2 years, comes in between the two. (sidenote: none are double-exponential; perhaps we're at an S-curve on penetration)
In the past 30 years, there has been a 26-fold reduction in the price per watt of solar modules, and this reduction in price has displayed an exponential trend downwards, halving every 6-and-a-half years (6.6y).