SlideShare una empresa de Scribd logo
1 de 48
Descargar para leer sin conexión
1
How Software is Transforming the U.S. Economy
Software Innovations and the Perpetual Development of New Software
Robert B. Cohen, Economic Strategy Institute, July 5, 2017
Final Report submitted to the Ewing Marion Kauffman Foundation for a Grant to Support “Research on
the Technology Changes that Constitute the “New Internet Protocol” and What it can do for
Entrepreneurial Growth and Job Creation”1
Executive Summary
This report builds upon ESI’s earlier efforts2
to forecast the potential impact of digital transformation on
the U.S. economy. In this earlier study, we employed a matrix-based approach to estimate that cloud
services and data analysis will add $1.2 trillion and $1.8 trillion to US GDP from 2015 to 2025.The shift to
software-defined cloud services, as we broadly defined the emerging digital infrastructure, will create
between 514,000 and 926,000 net new jobs between 2015 and 2025. Including multiplier effects, cloud
services could create 1.5 million to nearly 3 million new jobs.
This report examines the software innovations that have occurred in recent years. In doing so, it hopes to
disprove the contentious hypothesis some economists have proposed, that the Internet Revolution ended
in 2004. Based upon this hypothesis, these economists argue that there is little chance of Internet-related
innovation in the future. This is especially true, they argue, for industries that are not part of the
information and communications technology sectors in the economy.
In this essay, we describe the fundamental software innovations that support a range of new software
processes and techniques. Many of these are derived from Internet technologies and act as the
foundation for innovations that have disrupted the way software is developed and deployed. Our
examination of the core innovations focuses on new software processes and techniques such as the
virtualization of computing, storage and networking resources and the adoption of software-defined
infrastructure, such as software-defined data centers. These changes have been complemented by
improvements in software development processes; the most important of which are the widespread use
of Open Source software and software enhancements through GitHub. The latter serves as a prominent
location to exchange Open Source software and to refine and extend existing programs into new
applications. GitHub has changed the dynamics of software innovation, making it possible for many firms
like Facebook, JPMorganChase, PayPal, eBay, Comcast and Yahoo to develop new software quickly and at
low cost. After this we identify software innovations that include: the perpetual software development
model and continuous delivery of software; DevOps; microservices; serverless computing and application
protocol interfaces (APIs); containers and Docker; and software-defined data centers, Big Data and data
analytics.
1
This research was funded by the Ewing Marion Kauffman Foundation. The contents of this publication are solely
the responsibility of the Grantee.
2
Robert B. Cohen, “How Digital Transformation, the Move to the New IP, will Impact the US Economy
and Employment and Broad Implications for the Vendor Community: The Emergence of a “Megadigital”
Economy,” Economic Strategy Institute, October 13, 2016.
.
2
Economists resist recognizing that the software innovations we describe might improve productivity and
GDP growth. If the benefits of the software development and new processes were measured more
accurately in U.S. economic statistics, it could promote efforts to improve how we measure innovation in
the digital economy. Today, economists rely upon price and performance data for cellphones, laptops and
tablets. They assume this is a reliable yardstick to gauge the impact of new software.
Businesses adopt software innovations to accelerate their growth. This implementation counters thinking
by some economists that the Internet and Internet technologies are unlikely to have much impact on
productivity and GDP growth. It also counters expectations that productivity gains are likely to be meager
for the foreseeable future. Another assumption held by economists is that co-innovation, the creation of
economic value, especially new software, requires brainpower and experimentation and is a slow and
difficult process. We assert that processes such as the enhancement of software creation through GitHub,
the use of Open Source software, APIs and the new, perpetual software development model undercut
this contention. We argue that firms outside of the information and communications technology
industries operate in a world where software innovations are adopted quickly, suggesting a shift in the
importance of software in their operations. We also draw upon thinking by Paul David who pointed to the
parallels between the decades-long adoption of electricity and the long gestation period required for
software and computing innovations to have a large-scale impact.
The first part of this essay describes software innovations and the perpetual software development
model. Then, we address whether software will have an impact on productivity as well as why it has not
had a measurable impact on productivity. After this, we propose that economists should consider that
software – in a broad sense, cloud computing, cloud services and software-defined infrastructure – is a
General-Purpose Technology. By conceiving of software in this way, we underscore its potential to be a
ubiquitous technology. The last part of the essay explores the work of economist Robert Gordon who
asserts that innovations based on Internet-related technologies are unlikely to have much effect after the
Internet boom from the mid-1990s to 2004. It also analyzes a recent paper by Tim Bresnahan and Pai‐Ling
Yin.3
This paper argues that innovation in information and communications technologies is extremely
difficult to spread into other industries because of the difficulty of applying new innovations to these
sectors. As a result, its authors argue that the overall impact is to retard innovation and the growth of
productivity in the U.S. economy.
A. Software and the Digital Transformation
Software has become a prominent factor in firms’ operations. Marc Andreessen’s has asserted that
“software is eating the world” or “software is programming the world.” These expressions encapsulate
his belief that “six decades into the computer revolution, four decades since the invention of the
microprocessor, and two decades into the rise of the modern Internet, all of the technology required to
3
Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the
Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf to appear in Shane
Greenstein, Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17.
3
transform industries through software finally works and can be widely delivered at global scale.”4
(my
emphasis)
We explore Andreesen’s thinking by describing how software innovations have changed how firms use
information technology. We cite the great value firms obtain from new software tools and how software-
defined infrastructure facilitates this success.
This leads to a review of new approaches to software development. The innovations we describe below,
in our opinion, strongly contradict Robert Gordon’s contention that Internet innovations have been
“focused on the entertainment and information and communications technology.”5
In fact, software is
transforming a broad swath of industries. It is having a far greater impact than many economists believe.6
In 2017, software development is very different from what was common from 2005 to 2010.
1. Virtualization and Software-Defined Infrastructure: Innovations that transformed the
Way Software is Developed and Used, and How Data Centers and Networks Operate
Software benefits from two innovative abstractions, virtualization and software defined infrastructure.
These have heightened the role software plays in business. Software has disrupted the computing and
storage hardware world and turned proprietary products into commodities.
Virtualization is a way to create virtual rather than actual versions of parts within the computing, storage,
and networking ecosystem. Through virtualization, software creates virtual machines on a single physical
server or computer that hosts the virtual machines; the software in this case is called a hypervisor.
Virtualization can also take place with storage devices and with network resources.
When hardware is virtualized, software creates a virtual machine (Figure 1) that operates like a real
computer with an operating system. Before virtualization became widespread, many firms used physical
computers to run a single application. As a result, many tasks or applications drew upon 10 percent or 20
percent of a computer’s capacity. With virtualization, the entire resources of a computer can be applied
to different tasks at the same time, so virtualized machines operate at full capacity.
4
Marc Andreessen, “Why Software Is Eating the World,” Wall Street Journal, August 20, 2011.
http://www.wsj.com/articles/SB10001424053111903480904576512250915629460. Also see and Marc
Andreessen, Ben Horowitz, Scott Kupor, and Sonal Chokshi, “a16z Podcast: Software Programs the World,” July 10,
2016. http://a16z.com/2016/07/10/software-programs-the-world/
5
Robert Gordon, The Rise and Fall of American Growth, Princeton University Press, 2016, p. 567.
6
Erich H. Strassner, “Measuring the Digital Economy,” BEA Advisory Committee, November 16, 2016.
https://bea.gov/about/pdf/Measuring%20the%20Digital%20Economy.pdf
4
Figure 1. Traditional and Virtual Architecture
Source: Margaret Rouse and Brian Kirsch, “Definition: Virtualization,” TechTarget,
http://searchservervirtualization.techtarget.com/definition/virtualization
Virtualization also helps computing resources to scale. One example is Amazon Web Services’ virtualized
computing infrastructure. By using the public clouds Amazon supports, developers can rent tens of
thousands of servers for a short period of time to evaluate new versions of software at a realistic scale.
Firms have changed the process of software development. The innovative processes they employ build
upon enhancements in software-defined infrastructure.7
They also exploit additional innovations such as
containers.8
7
“Key Attributes of a Software-Defined Infrastructure,” SDx central, July7, 2015.
https://www.sdxcentral.com/articles/editorial/key-attributes-software-defined-infrastructure/2015/07/
8
Reza Roodsari, “Docker, Microservices And Kubernetes,” Mirantis Open Stack Training, December 22, 2016, p. 8.
https://content.mirantis.com/rs/451-RBY-185/images/mirantis-kubernetes-docker-mini-bootcamp_slides.pdf
5
Figure 2. Software Defined Infrastructure
Source: Anjanyea “Ruddy” Chagam and Shayne Huddleston, “Software Defined Storage – Open Framework and Intel Architecture
Technologies.” Intel talk at IDF 14, September 9, 2014. https://www.slideshare.net/LarryCover/software-defined-storage-open-
framework-and-intel-architecture-technologies
A Software Defined Data Center (SSDDC) (Figure 3) builds on virtualization to improve the operation of
data storage. An SDDC is “a data storage facility in which all infrastructure elements -- networking, storage,
CPU and security -- are virtualized and delivered as a service. Deployment, operation, provisioning and
configuration are abstracted from hardware. Those tasks are implemented through software
intelligence.”9
9
Margaret Rouse, “Definition: SDDC (software-defined data center),” TechTarget,
http://searchconvergedinfrastructure.techtarget.com/definition/software-defined-data-center-SDDC
6
Figure 3. Software-Defined Data Center
Source: Enrico Boverino, “Navigating Your Approach to an SDDC,” VMware Advisory Services blog.
https://blogs.vmware.com/accelerate/tag/software-defined-data-center
When software operates and manages infrastructure, high hardware costs are reduced. Virtualized data
centers use “commodity” hardware that is much lower cost than vendor-provided equipment. These data
centers also rely on overlays that transmit data entirely via software. Software-defined data centers
operate with virtualized firewalls and load balancers that reduce their physical infrastructure. This speeds
data from one part of a data center to another (Figure 4). Software for firewalls and load balancing is
usually included in the overlays that are part of the logical switching in data centers
Figure 4. Logical Switching via the Use of Overlays.
Source: Mora Gozani, Network Virtualization for Dummies, VMware Special Edition. John Wiley & Sons, Inc., 2016, p.
26. https://www.linkedin.com/pulse/ready-crash-course-network-virtualization-mora-gozani
7
Running several operating systems on a single central processing unit (CPU) can also reduce overhead
costs. With a virtualized enterprise, firms can perform updates to the operating system and applications
without disrupting a user.
2. The Importance of GitHub to Software Development and the Role of Open Source
Software
GitHub is an innovation that disrupts the traditional pattern of software development. It challenges the
usefulness of economists’10
co-invention model to conceive of the development of new software and how
rapidly it will contribute to higher productivity outside of the ICT industries, i.e., in non-vendor firms.
GitHub creates a way to share software that has been developed largely by relying upon Open Source
software.
Git is a distributed version control system. It lets developers have a complete history of a project or a
software program. This log of how the software was developed includes all the information provided on
the server it was taken from. As a result, developers working on software enhancements don’t have a
single point of failure.
Most importantly, GitHub also serves as a place (a “hub”) where developers can find new software or a
project to prepare a specific service or function, largely using Open Source software placed in a repository
or “repo.” Developers first “clone” or “pull,” i.e., copy, an earlier version of software from a “repo” to a
development server, or Linode. Then, they make changes to the code or project, and “push,” or forward,
these “forked,” or modified, projects to a developer’s GitHub user’s account. The final stage in the process
is the “pull request.” This asks the original “repo” project, or developer, to accept the revisions or sample
files the new developer created.11
“Pull requests” are a way to estimate the number of times developers
have created new versions of software.
10
Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the
Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf in Shane Greenstein,
Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17.
11
“Why You Should Switch from Subversion to Git,” Team Treehouse blog, August 7, 2012.
http://blog.teamtreehouse.com/why-you-should-switch-from-subversion-to-git
8
Figure 5. Git with GitHub Workflows.
Source: “Why You Should Switch from Subversion to Git,” Team Treehouse blog, August 7, 2012.
http://blog.teamtreehouse.com/why-you-should-switch-from-subversion-to-git
GitHub serves largely as a business software modification location, where businesses to improve their
own software. In 2016, 44 percent of the Fortune 5000 companies and half of the Fortune 10 firms used
GitHub. Facebook had the second largest number of contributors to Open Source after Microsoft.12
In
2016, nearly 85 percent13
of all “pull requests,” or requests to approve changes (Figure 5) sent to the
original programmer, came from within an organization (Figure 6); the rest came from user-owned
repositories.
Figure 6. GitHub Pull Requests
Source: “The State of the Octoverse 2016.” https://octoverse.github.com/
12
Matt Weinberger ,“Microsoft just edged out Facebook and proved that it's changed in an important way,”
Business Insider, September 14, 2016. http://www.businessinsider.com/microsoft-github-open-source-2016-9
13
9
In 2016, enterprises in industries outside of the software and the Internet plus communications sectors
accounted for more than two-thirds of GitHub’s enterprise customers (Table 1).
Table 1. Industrial Distribution of GitHub Enterprise Users
Source: “The State of the Octoverse 2016.” https://octoverse.github.com/
Open Source “is the new norm for software development.”14
The availability of Open Source software has
helped many industries and businesses benefit from the shared innovation it empowers. Open Source
software’s original source code is freely available on web-accessible locations such as GitHub. Free
distribution and modification relies upon having a permissive license in place, such as Apache 2.0. By using
these components, software developers, core developers, write Open Source code and other
programmers refine the code that is written and identify the flaws in it. The availability of Open Source
software lets businesses avoid paying licensing costs that vendors usually apply to their software products.
In this way, Open Source lowers development costs.
Open Source software is often more reliable than vendor-developed software. Large numbers of
programmers are involved in its creation. Open Source software frees firms from their previous
dependence on vendors that built applications. Firms, such as JPMorganChase use a great deal of Open
Source software. They also contribute software to GitHub.15
Microservices also contribute to changing the
dependence on vendors for important applications. They simplify the work that firms need to do to create
new software in the public cloud and add to Open Source software’s benefits. This reduces costs.16
14
Michael Dolan, “Commercial Dependencies and Sustainable Open Source Ecosystems,” CapitalOne DevExchange
blog, June 16, 2017. https://developer.capitalone.com/blog-post/commercial-dependencies-and-sustainable-
open-source-ecosystems/
15
Lori Beer, CIO of JPMorganChase, presentation on “Innovation at Scale in the Corporate and Investment Bank,”
WatersUSA2016, New York, December 5, 2016.
16
“7 Main Advantages and Disadvantages of Open Source Software,” ConnectUS, http://connectusfund.org/7-
main-advantages-and-disadvantages-of-open-source-software
Industrial Distribution of GitHub Enterprise Customers
Industry Percent of GitHub Users
Software and Internet 26
Business Services 15
Education 8
Manufacturing 8
Healthcare 6
Media and Entertainment 6
Retail 6
Telecommunications 6
Consumer Services 5
10
Table 2. The Differences between Proprietary and Open Source Software
Source: Kak Yong, “Chapter 2: Computer Systems and Open Source Software,” April 16, 2012.
https://www.slideshare.net/makyong1/proprietary-and-open-source-software
3. The Perpetual Software Development Model
Software development now proceeds in ways that are quite different what was common from 2005 to
2010. Developers working on new software, such as Linux, have made continuous refinements to the
original code. Figure 7 illustrates how changes, in “commits per month,” have been added to the Linux
operating system over a twenty-year period.
Figure 7. Commits per month in the Linux Source Code Management Repository, 1991-2011
11
Source: Jesus M. Gonzalez-Baharona et. al. “Studying the laws of software evolution in a long-lived FLOSS project,”
Journal of Software, July 26, 2014, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4375964/
At Facebook, as at many other “Internet Firms,” engineers write code in a “perpetual development mode,
in which engineers continuously develop new features and make them available to users. Consequently,
the [coding] system grows continuously, possibly at a super-linear rate.”17
Facebook has adopted a new model to create software. It dramatically reduces the time to deploy new
applications. Figure 8 illustrates the rapid progression to the Facebook model of delivering new software.
The old “waterfall” that required one year has been superseded by Facebook’s model of making new
software available once a day, before 2012. This has fallen to less than one hour, during the four years
from 2012 to 2016.
This new model is followed by at least 200 firms18
today. We have estimated that these firms are in
industries that accounted for about a third of U.S. GDP in 2016.19
Figure 8. Timescales in Making new software available. The Waterfall Process required one full year. Today
continuous deployment takes much less than an hour.
Source: Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at
Facebook,” p. 3. https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf
During Facebook’s growth (Figure 9), the number of software developers grew rapidly. The number of
“commits,” committing code to a versioning system, expanded about 10-fold over five years and the size
of Facebook’s codebase. In 2012, At Facebook’s “engineers commit[ted] code to the version control
system up to 500 times a day, recording changes in some 3,000 files.”20
In addition, Facebook’s lines of
code grew rapidly from 2005 to 2012, reaching an estimated 10.5 million lines of code.21
In 2016, one
report showed 60,000 commits, about 5000 per month.22
An earlier 2014 discussion estimated commits
at “thousands per week,” probably closer to 10,000 to 15,000 a month. It also noted that in 2014,
Facebook’s main source repository – its codebase size -- was much larger than the Linux kernel “which
17
Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at Facebook,” p.2.
https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf
18
Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.”
https://puppet.com/resources/white-paper/2016-state-of-devops-report
19
Robert B. Cohen, “The Economic Impact of Cloud Services,” Economic Strategy Institute, June 12, 2017
20
Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at Facebook,”
https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf
21
Feitelson, Frachtenberg, and Beck, p. 3.
22
Christine Abernathy, “Facebook Open Source 2016 year in review,” December 19, 2016.
https://code.facebook.com/posts/1058188987642144/facebook-open-source-2016-year-in-review/
12
checked in at 17 million lines of code and 44,000 files in 2013;”23
this seems to mean that Facebook had a
codebase size of about 68 to 136 million lines of code.24
That indicates a six-fold to thirteen-fold increase
in the codebase size in just two years’ time.
Figure 9. Facebook’s Increase in Developers, Coding Commits Per Month and Codebase Size, 2005 to 2012
Source: Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at Facebook,” p. 2.
https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf
4. Continuous Service Delivery and Continuous Integration
Continuous delivery or continuous deployment (CD) describes how teams produce software in short
cycles. Since the software is tested continuously, it can be confidently released at any time in the cycle.
This accelerates the building, testing and release of new software.
CD or continuous service delivery relies upon cross-disciplinary teams to program, deploy and test new
software (see Adrian Cockcroft’s diagrams below, Figures 10 and 11). In continuous service delivery,25
integrated design teams replace siloed, or isolated, “skill areas,” that were widely employed to create
“monolithic” software. These areas included quality assurance, systems administration and development.
This reduced the number of steps required to write and test software. Continuous service delivery lets
firms respond to demands from markets and customers. By creating software in this way, firms not only
improve their knowledge of customers and markets, but also quickly exploit new opportunities for sales.
Lori Beer, the CIO of JPMorganChase, noted 26
that her bank achieved significant savings by
moving to the cloud and creating a more software-based infrastructure. Nevertheless, the real
benefits of software innovations came from the productivity gains when it expanded into new
business areas without needing additional resources. JPMorganChase expanded its revenues by
being able to analyze markets rapidly and create new software to address them.
23
Durham Goode and Siddharth P Agarwal, “Scaling Mercurial at Facebook,”
https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/
24
Goode and Agarwal.
25
Adrian Cockcroft, “Creating Business Value with Cloud Infrastructure,” Open Networking User Group, Columbia
University, May 13-14, 2015.
26
Lori Beer, discussion with author after presentation on “Innovation at Scale in the Corporate and Investment
Bank,” WatersUSA2016, New York, December 5, 2016.
13
When News Corp27
migrated to the cloud, its “business case was to migrate 75% of our
infrastructure to the cloud over 3 years to achieve $100M in annual savings.” These efforts relied
upon making applications “cloud-ready.”28
Continuous integration (CI) is the process developers employ to integrate code they are writing into a
shared repository many times a day. By doing this, developers can check whether new code they have
written is problem-free as it is being created. If they find the new code is error-free, their work can
proceed to an automated build of a new software application. Many organizations achieve CD by
connecting CI to an automated infrastructure, such as the cloud or software-defined infrastructure.
Consequently, when firms develop applications in this new way, they reduce the time to business value
(Figure 10). Today, firms create software applications and deploy them using different processes than the
highly-segregated steps they used earlier (see Cockcroft’s chart in Figure 11). The new process eliminates
time-consuming, step-by-step approvals by integrating the software development and testing cycles.
Figure 10. Business Value Delivery – Software Changes from Monoliths to Functions: Log Time in Hours
Shown with a Reversed Scale
Sources: Draws upon Adrian Cockcroft, "Evolution of Business Logic from Monoliths through Microservices to Functions," Cloud
Guru,https://read.acloud.guru/evolution-of-business-logic-from-monoliths-through-microservices-to-functions-
ff464b95a44d#.r8eel3vze and Stephen Orban, "Transitioning to DevOps and the Cloud," https://medium.com/aws-enterprise-
collection/transitioning-to-devops-and-the-cloud-9488ddaf862f#.r7y6krhq5
27
Stephen Orban, “Always Be Reinventing & The Cloud of Youth,” Medium, January 265, 2017.
https://medium.com/aws-enterprise-collection/always-be-reinventing-the-cloud-of-youth-
137990b9d229#.29kuffhln
28
“Cloud-ready” applications have been reprogrammed from when they were used in traditional computing
environments so they can run in virtualized environments using cloud computing.
14
More traditional, “low-performing”29
IT organizations continue to rely upon the more cumbersome and
time-consuming “siloed,” or monolithic, approach to developing software. When they take this approach
to software development, each stage in software development is isolated from every other one. Figure
11 describes the main stages in this process.
Figure 11. Continuous Service Delivery: The Product Delivery Process with Monolithic Software and
Microservices – with the Latter Supporting a Reorganization to DevOps
Source: Adrian Cockcroft, “Creating Business Value with Cloud Infrastructure,” Open Networking Users Group
Conference, Columbia University, New York City, May 13-14, 2015.
As noted above, the shift to CD is built upon changes in software-defined infrastructure and new process
innovations. This results in an enormous reduction in the time required to develop new software as well
as to test, evaluate and deploy it in “real-life” situations. It also means that there is continuous learning
29
This characterization is drawn from the studies of DevOps use in enterprises. See Puppet, Inc., and DevOps
Research and Assessment, “2016 State of DevOps Report.” https://puppet.com/resources/white-paper/2016-
state-of-devops-report
15
about the new code, or software, that is being written, so that mistakes are identified before they are
introduced to new designs.
The innovations below demonstrate the large gap between CD and the assumptions of the co-innovation
of new ICT discoveries. Some contemporary economists30
rely upon the co-invention framework to assess
changes in the ICT industries. We believe that this approach overlooks recent innovations in software
development and is mistaken in believing the software development process remains tedious and
complex. We base our conclusion on the recent changes in software development.
5. DevOps
DevOps is an integral part of continuous service delivery (see Figure 11). It was the innovative model for
software development that provided firms with a way to accelerate deployment by optimizing the way
that development and operations collaborated. In DevOps, Ops (operations) groups let developers
manage the “operational characteristics” of applications they are building. In DevOps, a “Shift Left” is how
developers describe these changes in software development. The focus moves to developers, but requires
them to consider how effectively new software contributes to an enterprise’s operations.
Figure 12. DevOps and the “Shift Left” in Building Applications-Aware Environments
Source: Sanjeev Sharma, “Adopting DevOps – Part III: Aligning the Dev and Ops Teams,” May 9, 2013,
https://sdarchitect.wordpress.com/2013/04/12/adopting-devops-part-iii-aligning-the-dev-and-ops-teams/
DevOps supersedes the old, “Waterfall Model,” of programming where every stage waited for the
previous one to be completed (Figures 13 and 14). This meant creating “flattened” product teams for
DevOps that included people with expertise in most of the stages of software development, testing and
distribution (Figure 11).
30
Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the
Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf to appear in Shane
Greenstein, Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17. We discuss this
analysis later in this essay.
16
Figure 13. Issues with the Waterfall Model of Software Development and How the Challenges
were overcome
Source: Vineet Chaturvedi, “DevOps Tutorial: Introduction to DevOps” edureka, Oct 18, 2016.
https://www.edureka.co/blog/devops-tutorial
Figure 14. Proposed Solutions to the Challenges of the Waterfall Model of Software Development
Source: Vineet Chaturvedi, “DevOps Tutorial: Introduction to DevOps” edureka, Oct 18, 2016.
https://www.edureka.co/blog/devops-tutorial
17
Figures 15 and 16. How DevOps Solutions address Developmental and Operations Challenges
Source: Vineet Chaturvedi, “DevOps Tutorial: Introduction to DevOps” edureka, Oct 18, 2016.
https://www.edureka.co/blog/devops-tutorial
Today, many enterprises depend on software and data analytics to operate and create new strategies. As
noted above, this represents a major change. Software is now a major factor, if not the most important
factor determining corporate competitiveness. Innovations in software, i.e., “achieving higher IT and
organizational performance is a team effort spanning development and operations.”31
Recent surveys have analyzed how firms are using DevOps. Over the past year, “high performing” firms32
improved their performance over “low performing” ones. The ability to deploy new code is one way to
measure difference.
Between 2015 and 2016, higher-performing firms increased their ability to deploy software from 200
deployments per year to 1460 deployments per year (Figure 17). This occurred while low-performing firms
maintained their level of about 12 deployments per year.33
31
Puppet Labs and DevOps Research and Assessment, “2016 State of DevOps Report,” p.4
32
For a definition of high performing and low performing, see the Appendix.
33
Puppet and Dora, p. 18.
18
Figure 17. Deployment Frequency for “High Performing” Firms in Number of Deploys per Year
Source: Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.”
https://puppet.com/resources/white-paper/2016-state-of-devops-report, p. 18.
Nicole Forsgren, one of the analysts34
who identified the gap between the two groups of firms (Table 3)
noted that they were due to high performers’ greater sophistication in developing and deploying
software. The “high performers” spent far more time on new work, delivering more value to the business.
This indicates that they had streamlined their software development, testing and deployment skills. In
fact, they got code, or software, out faster, at 200 times the rate of low performing firms. They also
deployed more stable applications, having 24 times faster recovery from failure.
We expect that this performance difference will pressure additional firms to improve their software
delivery, agility and reliability. They are likely to move in this direction to realize improvements in their
delivery of content, the value obtained from A/B testing, the value from speed to market, and to benefit
from the compliance and regulatory benefits.
34
Nicole Forsgren, “The Data on DevOps,” Devopsdays Minneapolis 2016,
https://www.youtube.com/watch?v=Z6IjVf2dcKM. The author thanks Dr. Forsgren for her comments on these
paragraphs.
19
Table 3. 2016 IT Performance by Cluster for DevOps Users
Source: Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.”
https://puppet.com/resources/white-paper/2016-state-of-devops-report
6. Microservices
Microservices applications have an architecture where each functional element is a separate service. As a
result, firms can use microservices (Figure 18) and reuse parts of an application that do not change. This
increases the speed35
at which firms can deliver applications and services; i.e., “high velocity software
development.” This makes microservices an important part of continuous service delivery.
Microservices divide applications into smaller, composable pieces, very much like Lego blocks. So
microservices’ components are “easier to build and maintain.” Each component, “is developed separately.
An application is the sum of its “constituent components.””36
Previously, programmers had to write a
software application as a single, unified, or “monolithic” product.
Microservices can be easily “glued together.” Each one contains “an [Application Protocol Interface or]
API endpoint.” APIs are tools, protocols and routines that are used to develop software. APIs “specify how
35
Richard Li, “Microservices Essentials for Executives: The Key to High Velocity Software Development,” for
Entrepreneurs from David Skok. http://www.forentrepreneurs.com/microservices/
36
What are microservices? Opensource.com, https://opensource.com/resources/what-are-microservices
20
different software components should interact.”37
The API interface can be accessed like a “standard
webpage,”38
making it easier for developers to use microservices.
Figure 18. The Difference between Microservices architecture and Monolithic Applications
Source: James Lewis and Martin Fowler, “Microservices,” March 25, 2014.
https://martinfowler.com/articles/microservices.html.
With developers focusing on code, software engineers and developers report “as we move towards
software-defined environments, we [can] build, version [or model] and manage complex environments,
all as code.”39
7. “Serverless” Computing and Applications Protocol Interfaces (APIs)
Serverless computing is called “serverless” because developers no longer need to manage the physical or
virtual servers and data they need40
to code. This improves developers’ efficiency. Through the public
cloud, they can access resources from a cloud service provider – Amazon Web Services, IBM or Google.
“Serverless systems allow developers to build complex systems much more quickly and ensures that they
are spending most of their time focusing on core business issues rather than infrastructure based and
37
“What is API - Application Program Interface?” Webopedia www.webopedia.com/TERM/A/API.html
38
“What are microservices?”
39
Sanjeev Sharma, “Adopting DevOps – Part III: Aligning the Dev and Ops Teams,” May 9, 2013,
https://sdarchitect.wordpress.com/2013/04/12/adopting-devops-part-iii-aligning-the-dev-and-ops-teams/
40
David Ward of Cisco has described a “Policy Engine” that will manage computing, storage and network
infrastructure in future virtualized environments. David Ward, “Networking: The Logical Micro-Service
Infrastructure,” Open Networking Summit, April 5, 2017. http://sched.co/9kxu
21
administrative duties.”41
(my emphasis) Serverless systems such as Amazon’s Lambda or IBM’s
OpenWhisk can scale, grow and evolve without developers or solution architects having to patch a web
server.
With serverless computing, developers submit functions42
for execution (Table 4). They provide a function
code to a cloud service provider offering serverless computing and the computing program, for instance,
Amazon’s Lambda, executes it. This is possible because an API gateway eliminates “traffic management,
authentication and authorization, monitoring and API versioning [by converting them] into easily
configurable steps.”43
41
Scott Maurice, “What does “serverless computing” really mean?” http://scottmaurice.com/what-does-
serverless-computing-really-mean/
42
Sam Kroonenberg, “The Next Layer of Abstraction in Cloud Computing is Serverless,” A Cloud Guru, May 19,
2016. https://read.acloud.guru/iaas-paas-serverless-the-next-big-deal-in-cloud-computing-
34b8198c98a2#.9877q5ouf
43
Amazon, “Job posting for Senior API Software Engineer-Amazon API Gateway,” October 20, 2016. https://us-
amazon.icims.com/jobs/452236/senior-api-software-engineer-amazon-api-
gateway/job?iis=Job+Posting&iisn=Indeed+%28Paid+Sponsored+Posting%29&mobile=false&width=1027&height=
1200&bga=true&needsRedirect=false&jan1offset=-300&jun1offset=-240
22
Table 4. Stages in the Evolution of Business Logic from Monoliths to Microservices to Functions
Source: Adrian Cockcroft, "Evolution of Business Logic from Monoliths through Microservices to Functions," Cloud Guru,
https://read.acloud.guru/evolution-of-business-logic-from-monoliths-through-microservices-to-functions-
f464b95a44d#.r8eel3vze and Stephen Orban, "Transitioning to DevOps and the Cloud," Medium.com, https://medium.com/aws-
enterprise-collection/transitioning-to-devops-and-the-cloud-9488ddaf862f#.r7y6krhq5
At the heart of this new process is the “API Gateway, … the front-door of the Serverless revolution, an
approach that lets customers turn business logic and application code into scalable, fault-tolerant
production systems without requiring every developer to become an expert in distributed systems,
deployment technologies, and infrastructure management.”44
44
Amazon, “Job posting.”
23
Figure 19. An API Gateway, Using Oracle’s API Gateway as an Example
Source: Aaron Dolan, “Your API’s First Line of Defense: Oracle API Gateway,” AVIO Consulting, October 29, 2014.
http://www.avioconsulting.com/blog/your-apis-first-line-defense-oracle-api-gateway
“Serverless computing” is a process change that simplifies developers’ work. It eliminates tasks that were
previously required when using public cloud services. Developers had to reserve virtual server time and
learn how to manage traffic, authentication and authorization. This reduces the time needed to deploy
new software.
24
Figure 20. A Brief History of Cloud: Serverless Computing and the Evolution of Cloud Services
Source: Sam Kroonenberg, “The Next Layer of Abstraction in Cloud Computing is Serverless,” Cloud Guru, May 19, 2016.
https://read.acloud.guru/iaas-paas-serverless-the-next-big-deal-in-cloud-computing-34b8198c98a2
8. Containers and Docker Facilitate the Creation and Deployment of New Software
Containers (Figures 21 and 22) build upon operating system-level virtualization. They are an innovative,
interoperable format for applications, i.e., software, that can be wrapped with a full system needed to
run the software. So “containers wrap-up an application in a self-contained filesystem … that includes
everything the [application] needs to run independently: binaries, runtime libraries, system tools, system
packages, etc. This level of simplification and compartmentalization allows applications to be spun up [or
launched] much faster than before.”45
while ensuring consistent and predictive up time.
45
Scott Willson, “Webcast: Containerology – DevOps, Docker and Microservices in a Continuous Delivery World,”
https://offers.automic.com/ppc/containerology-devops-docker-and-microservices-in-a-continuous-delivery-world-
webcast-
ppc?network=g&campaignid=646348797&adgroupid=27262905662&keyword=docker%20technology&matchtype
=p&creative=94949079182&gclid=CjwKEAiAj7TCBRCp2Z22ue-
zrj4SJACG7SBE5G2uN07hlUU24bMYRldAYm2tp5Yxrhqb1bG5XVNqwBoCSf7w_wcB
25
Figure 21. Comparing Containers to Virtual Machines (VMs)
Source: “Docker, Containers, and the Future of Application Delivery,” OSCON 2013.
http://www.slideshare.net/dotCloud/why-docker2bisv4
Figure 22. Why are Docker containers Lightweight? -- Applications on Virtual Machines (VMs) and
Containers
Source: “Docker, Containers, and the Future of Application Delivery,” OSCON 2013. http://www.slideshare.net/dotCloud/why-
docker2bisv4
Containers represent a fundamental change in how workloads and applications can be virtualized.
Containers can scale more efficiently, operate faster and offer greater portability than hardware
26
virtualization. Eventually, they are expected to replace most instances where virtual machines are
involved.46
An April-May 2016 DevOps.com and ClusterHQ survey47
found that 79 percent of respondents’
organizations were using container technologies. Of this group, 76 percent of the deployments were
running in production environments, not experimental ones. This was a significant increase over 2015,
when only 38 percent of respondents had containers in production ecosystems. The report concluded
that container adoption was driven by a desire to “increase developer efficiency (39 percent) and support
microservices (36 percent).” Over two thirds of the survey’s respondents said their firms are realizing the
results they expected from using containers.
Bloomberg Inc. has adopted containers and software-defined networking over the last four years
to add simplicity and high volume to its development of new applications and products. It has
assembled a staff of 2500 developers and embraced the use of Open Stack. It notes that modern
applications (software) have become ephemeral in nature, with developers using templated and
automated images to write software. This has moved Bloomberg away from a model where
applications development required complex policies. The move to software-defined networking
has also let developers use microservices and micro-segmentation of applications.48
Google’s container was “a kind of virtualized, simplified OS [Operating System] which we used to
power all of Google’s applications.” Initially, Google developed cgroups, 49
“a framework pattern
that provides encapsulation and separation of concerns for the components that use them …. the
container will provide mechanisms to address cross-cutting concerns like security or transaction
management.…a container wraps the component.”50
Based on these benefits,
• “a developer has in their laptop plenty of compute power to run multiple containers,
making for easier and faster development”
• “a single command” can push out a “new version of a container,”
• With containers, it is much easier to compose “applications using open source software.”
This means that developers can bring together many tools that might be complicated to
set up individually, such Hadoop and MongoDB. Developers can use containers to deploy
46
Reza Roodsari, “Docker, Microservices And Kubernetes,” Mirantis Open Stack Training, December 22, 2016, p.
13. https://content.mirantis.com/rs/451-RBY-185/images/mirantis-kubernetes-docker-mini-bootcamp_slides.pdf
47
DevOps.com and ClusterHQ, Container Market Adoption:2016,” https://clusterhq.com/assets/pdfs/state-of-
container-usage-jgune-2016.pdf. The survey queried 310 computer professionals regarding their firms’ container
adoption and usage patterns.
48
Truman Boyes, “Open Cloud Infrastructure at Bloomberg,” Open Network Users Group, Columbia University,
May 2015.
49
“cgroups,” Wikipedia. https://en.wikipedia.org/wiki/Cgroups
50
Edward Ost, “What Is a Container? (Container Architecture Series Part 1),” Talend blog, December 2, 2014.
http://www.talend.com/blog/2014/12/02/what-is-a-container-container-architecture-series-part-1
27
numerous tools on a single computer. They can use these tools to improve the quality of
the software that they program.51
Containers, in contrast to virtual machines, offer:
• “Simple deployment: By packaging your application as a singularly addressable, registry-
stored, one-command-line deployable component, a container radically simplifies the
deployment of your app no matter where you’re deploying it.
• Rapid availability: By abstracting just the OS [operating system] rather than the whole
physical computer, this package can “boot” in ~1/20th of a second compared to a minute or so
for a modern VM.
• Leverage microservices: Containers allow developers and operators to further subdivide
compute resources.”52
9. Software-Defined Data Centers, Big Data and Data Analytics
McKinsey describes Big Data as “large pools of data that can be captured, communicated, aggregated,
stored, and analyzed.” In 2011, McKinsey found that Big Data is “part of every sector and function of the
global economy.”53
In 2013, ABI Research estimated that spending on Big Data was $31 billion and that
this spending would increase to $114 billion in 2018, a compound growth rate of 29.6 percent.54
The
Eckerson Group has concluded that if firms are going to capitalize on Big Data, they need to
“fundamentally rethink the way they capture, store, govern, transform and analyze” it.55
As Figure 23 illustrates, firms upgraded their data center infrastructure through several stages in the time
from 2000 to 2013. Once they implemented software-defined data centers, the time it took to obtain
business value from data centers dropped by more than 1000 times. Firms that made this transition found
they had greatly improved the usefulness of the analytic tools they applied to Big Data.
51
Quotes and section summarized from Miles Ward, “An introduction to containers, Kubernetes, and the
trajectory of modern cloud computing,” Google Cloud Platform Blog, January 9, 2015.
https://cloudplatform.googleblog.com/2015/01/in-coming-weeks-we-will-be-publishing.html
52
Miles Ward, “An introduction to containers, Kubernetes, and the trajectory of modern cloud computing,” Google
Cloud Platform Blog, January 9, 2015. https://cloudplatform.googleblog.com/2015/01/in-coming-weeks-we-will-
be-publishing.html
53
James Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey
Global Institute, May 2011, p. iii. http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-
data-the-next-frontier-for-innovation
54
ABI Research, "Unlocking the Value of Big Data in Enterprises," cited in Joanne Herman, “Big Data-Related
Investment to Hit $114bn in 2018,” Misco.co.uk blog, September 12, 2013.
http://www.misco.co.uk/blog/news/01279/big-data-related-investment-to-hit-114-billion-dollars-in-2018
55
Phil Bowermaster and James W. Eckerson, “Selecting a Big Data Platform,” The Eckerson Group, December 2015.
http://www.eckerson.com/register?content=selecting-a-big-data-platform-building-a-data-foundation-for-the-
future
28
Figure 23. Business Value Delivery – Changes in the Time to Value in Data Centers from 2000 to 2013.
Source: VMware Accelerate Advisory Services, "Delivering on the Promise of the Software-Defined Data Center,"
VMware Accelerate Advisory Services blog, 2013. https://blogs.vmware.com/vmtn/author/aluciani/page/3
Two different McKinsey Global Institute studies illustrate how much firms depend on data analytics. In its
first Big Data study in 2011,56
McKinsey discovered that firms in nearly all U.S. sectors possessed at least
200 terabytes of stored data per company. Firms having more than 10,000 employees in 2009 attained a
level where McKinsey believed they could capture real value from data analytics.57
. This report also
estimated that there were 300,000 employees in data analytics (my own estimates58
are much greater;
with about 15 million job postings between 2011 and 2017, allowing for duplication of job skills –
repeating mention of data analysis in postings for other jobs – I would estimate there might be as many
as million to 2.5 million data analysts today). McKinsey forecast that by 2018, there would be almost 50
percent more demand for these jobs; that demand would rise to between 440,000 to 480,000.59
McKinsey’s 2016 Big Data report concluded that “data is now a critical corporate asset.”60
While it found
that data was doubling every three years, it also noted that since its earlier study, many firms had not
56
James Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey
Global Institute, May 2011, p. 18. http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big-
data-the-next-frontier-for-innovation
57
Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global
Institute, May 2011, p. 19.
58
I derive this from the figures on job postings in Robert B. Cohen, “Highly Stratified Occupations and the Digital
Economy,” Economic Strategy Institute, June 26, 2017. Report to the Berggruen Foundation.
59
Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global
Institute, May 2011, p. 104.
60
Nicolaus Henke, “The Age of Analytics: Competing in a data-driven world,” McKinsey Global Institute, December
2016, p. vi.
0.01
0.1
1
10
100
1000
10000
2000 2003 2013
Business Value Delivery - Changes in the Time to Value in Data
Centers from 2000 to 2013: Log of Time in Hours with Reversed
Scale
VIRTUAL MACHINES: 6
SOFTWARE
DEFINED DATA
HARDWARE
DEFINED DATA
29
taken advantage of the gains that it had forecast in 2011. Some observers61
believe that many firms had
a difficult time improving the flow of analysis. They failed to create a framework of continuous learning
by experimentation that refined the use of Big Data.
Nonetheless, McKinsey asserts that a group of “analytics leaders are changing the nature of competition
and consolidating big advantages,”62
by deploying and using Big Data. These firms include “Apple,
Alphabet/Google, Amazon, Facebook, Microsoft, GE, and Alibaba Group.”63
One reason for this shift is
that these firms are exploiting a difference between their level of performance after they have been able
to evaluate Big Data and the performance of other firms that don’t rely on Big Data. In many cases, the
more successful firms use data to provide better situational awareness. They also employ analytics to
improve the situations they experience.
McKinsey estimated that firms could obtain significant savings when they implemented data analytics. It
calculated that when retailing uses data analytics “marketing levers can affect 10 to 30 percent of [the]
operating margin; merchandising levers can affect 10 to 40 percent [of the operating margin]; and supply
chain levers can have a 5 to 35 percent impact [on operating margin].”64
Nearly four-fifths of firms in another survey reported that data analysis had helped them institute new
business processes, such as creating an Internet of Things.65
In the same survey, almost one-third or more
of firms in the energy and utilities, automotive and retailing industries had adopted machine-to-machine
(M2M) communications. These were part of the infrastructure to analyze Big Data. This included more
than a quarter of firms in healthcare and consumer electronics industries as well as one-sixth or more of
firms in manufacturing, transport and logistics.66
McKinsey noted that Big Data’s ability to contribute real value to a business depended upon the type of
retail sector that used it. The retailing sectors that were early adopters of data analytics usually obtained
the greatest benefits. General merchandise stores, building material and garden, electronics and
appliances and health and personal care stores were forecast to have the greatest big data value potential.
61
The author thanks Chris Swan of CSC for his comments on why more firms did not take advantage of Big Data.
62
Henke, p. 5.
63
Henke, p.
64
James Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey
Global Institute, May 2011, p. 71.
65
Erik Brenneis “Vodafone M2M Barometer 2015,” Vodafone Germany, p. 3.
https://www.vodafone.de/media/downloads/press-releases/150729-vf-m2m-report-2015.pdf This is a survey of
machine-to-machine (M2M) communications use.
66
Brenneis, p. 19.
30
Table 5. The Big Data Value Potential in Retail Varies in Different Subsectors
Source: Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey
Global Institute, May 2011, p. 72.
B. Software as a General-Purpose Technology (GPT)
Economists who have studied the ascent of the US economy in the 20th
Century have identified the rise in
US productivity as the main driver of growth. The productivity gains are linked to several technologies
that drove the upswing from the 1920s to 1970s. “GPT's are characterized by pervasiveness (they are used
as inputs by many downstream sectors), inherent potential for technical improvements, and innovational
complementarities', meaning that the productivity of R&D in downstream sectors increases because of
innovation in the GPT. Thus, as GPT's improve they spread throughout the economy, [they brought] …
about generalized productivity gains.”67
67
Timothy F. Bresnahan and Manuel Trajtenberg, "General Purpose Technologies: Engines of Growth," Journal of
Econometrics, January 1995, 65(1), pp. 83–108. Earlier version cited as National Bureau of Economic Research
Working Paper 4148, August 1992, p. iii. www.nber.org/papers/w4148
31
General Purpose Technologies68
include the internal combustion engine and electricity. Because firms in
a wide range of industries exploited these technologies to improve productivity, GPTs were defined as “an
invention that can lead to many sub-inventions.”69
We compare electricity and the internal combustion engine to software innovations. The first two are the
GPTs70
that unleashed the expansion of US manufacturing and, thereby, the US economy in the 1920s,
1930s and 1940s. They continued to have a positive impact on productivity until the mid-1970s. These
technologies had an outsized impact on productivity because as industries adopted them, they created
innovative ways to use these technologies more efficiently.
We believe that software, particularly in the form of cloud computing, cloud services and software-
defined data centers has contributed to major advances in data analytics. Software is a 21st
Century GPT.
Table 6 compares software to internal combustion engines and electricity.
68
Timothy F. Bresnahan and Manuel Trajtenberg, "General Purpose Technologies: Engines of Growth," National
Bureau of Economic Research Working Paper 4148, August 1992. Robert J. Gordon, The Rise and Fall of American
Growth. Princeton University Press, 2016, pp. 555-565. Timothy F. Bresnahan and Robert J. Gordon, eds., The
Economics of New Goods, Studies in Income and Wealth, vol.58, University of Chicago Press for National Bureau of
Economic Research, 1997, pp. 1-26. http://www.nber.org/chapters/c6063.pdf
69
Gordon, The Rise and Fall…, p. 555.
70
Some research by economic historians has questioned the validity of asserting that electricity is a GPT because it
shows lower generality scores than other technologies. Nicholas, Tom, and Petra Moser. "Was Electricity a General
Purpose Technology: Evidence from Historical Patent Citations." American Economic Review: Papers and
Proceedings 94, no. 2 (May 2004), pp. 388-394.
32
Table 6. General Purpose Technologies, Electricity, the Internal Combustion Engine and Software
Innovations
This table summarizes the similarities between today’s software innovations and earlier GPTs. The use of
big data as well as advanced analytics – components of the Internet of Things -- are fundamental to how
firms are making the transition to a digital world. As firms like Facebook, Netflix, ETSY, Ford, Boeing, UPS,
and John Deere have become more efficient producers of products and creators of services; they have
also enhanced their ability to manage their supply chains.71
71
Our conclusion draws upon a series of case studies on the economic impact of the Internet of Things that we
prepared for the Organization for Economic Cooperation and Development. These cases will be cited in a new
chapter on digital technologies and future production. The cases will be collected into a short book by this author,
The Economic Impacts of the Internet of Things.
General Purpose Technologies: Electricity, the Internal Combustion Engine and the Software Innovations related to the Third Wave of the Internet, Cloud Services plus Data Analytics
Extending the Definition of General Purpose Technologies to Software and Services
Electricity Internal Combustion Engine Innovative Software -- Fully Digital Infrastructure, Cloud Services and Data Analysis
Productivity Drivers
The price of electrically-generated
power declined steadily in the early
decades of the 20th Century. There
were "constant improvements in the
efficiency of electric motors. As a
consequence, electric motors diffused
rapidly throughout manufacturing
displacing the steam engine."
(Bresnahan and Trajtenberg, 1992).
The price of automobiles and the cost of
operating them declined rapidly during the early
years of the 20th Century. There were a series of
improvements such as closed tops that made
vehicles easier to use in inclement weather. Paved
roads made it easier to use vehicles in many areas
that may not have been accessible.
Innovations in developing software permit firms to create new software or applications for specific
industries. This has facilitated complex data analysis. It also permits firms to offer faster and cheaper
services to customers. Software development, data analytics and the Internet of Things rely upon
cloud computing to operate. They lower costs through the improved management of supply chains
and via new services, such as driverless cars and mobile purchases in retail stores. Within cloud
computing, software-based management of data storage and computing accelerates the creation
and deployment of new software and services. New tools that complement software development,
such as containers (Docker) are being adopted rapidly. They support productivity gains in many
industries, such as the financial, auto, pharmaceutical, aircraft, autos, logistics, retailing, information
technology, and communications industries.
Pervasiveness
Used in a wide range of industries such
as manufacturing, transportation,
consumer durables, communications
and healthcare.
Automobiles were used widely on farms. They
helped speed the delivery of supplies, support
supply chains, and to accelerate shipments to
consumers. They changed marketing and the
delivery of services. They made it possible to
reorganize retailing and other services.E11
The free distribution of programmed code via structures such as GitHub has reduced software costs
and improved access to new innovations. This has been complemented by the rise of Open Source
software that is often free of charge. Both of these changes have accelerated the distribution of
software. The result is improved efficiency and more widespread use of cloud computing and data
analytics, as well as lower costs for software and software tools. When cloud service providers use
this software, it improves firms' reliance on Software-as-a-Service. This reduces the separation
between innovators in services and consumer software.
Potential for
Technological
Improvement
Electrical power eq+D12uipment
became more powerful and efficient.
Greater economies of scale were
achieved. The price performance ratio
of products, systems, or components in
which electric power was embedded
improved. Costs in downstream sectors
declined.
As internal combustion engines matured, more
powerful vehicles shortened delivery times and
expanded deliveries to a wider market.
The Internet of Things and Cloud Computing have an inherent tendency for technical improvement
(Bresnahan and Trajtenberg, 1992). Firms such as Amazon, Google, Facebook and Netflix have used
Cloud Computing to introduce new processes to create and test software. This practice is becoming
common in many industries. Cloud Computing supports these changes by facilitating firms' adoption
of more efficient software development processes such as DevOps, continuous delivery and
containers. These processes speed the creation of new software and services and lower their cost.
New tools, such as containers (Docker) for software development and deployment, dramatically
lower the cost of creating new software or services and deploying it in different locations. In
addition, the Internet of Things has been the foundation for Boeing and other firms to redesign
production> It has helped Ford to improve the management of its supply chain and UPS to develop
predictive analytics to speed deliveries.
Innovational
Complementarities --
Productivity of
R&D in a downstream
sector increases as a
consequence of
innovation in the GPT
technology
Electrical motors in factories not only
lowered energy costs, but also enabled
factory floor redesign (a redesign of the
production process). Growth may really
depend on the structure of markets
where the technology is used.
Automobiles made it possible for farmers to
"bargain in the sale of farm products or the
purchase of supplies." (Gordon, Rise and Fall , p.
163). Tractors revolutionized agricultural
productivity. Vehicles expanded the size of the
market for many industries. Aircraft changed the
costs of supplies and expanded the market served
by many industries.
Software development benefits from new tools and processes such as containers, continuous service
delivery and DevOps. Innovative software lowers the cost of analyzing complex data in many
industries. Software developers can benefit from using exchanges like GitHub and Open Source
software. This permits a large number of industries to improve existing software applications,
software processes and services. In addition, new security services, such as Blockchain, are likely to
reduce problems with security breaches. They are also likely to reduce the cost of operations and of
software use. New market structures, such as GitHub, overcome the problem of asymmetrical
information and uncertainty in the creation of new knowledge (Arrow, 1962 as cited in Bresnahan
and Trajtenberg, 1992).
Examples of GPT use
in industry
Elevators, electronic hand tools and
machine tools, electronic streetcars and
subways, consumer appliances
(refrigerators, washing machines, and
air conditioners), telephones and
broadcasting, power plants and
refrigeration, hospital x-ray machines,
and ship geolocation.
Cars, buses, and taxis. These vehicles made it
possible to build supermarkets, suburbs, and to
have personal travel, motels, and roadside
restaurants, and air travel.
New software applications that run on cloud computing infrastructure simplify complex genome
analysis for new drugs. They support the development of driverless cars, the analysis of consumer
behavior to restructure how retailing operates, and banks' use of complex investment strategies and
more sophisticated risk analysis. Data analytics and the Internet of Things change business models so
that products can be offered "as-a-service;" i.e., Rolls-Royce offers engines as "power by the hour."
Other innovations include the management of driverless cars, the restructuring of aircraft
production lines, greater efficiencies in farming using GPS and data analytics and the use of
predictive analytics for more efficient deliveries at firms like UPS.
We refer to the "Third Phase" of the Internet as one characterized by digital businesses that use infrastructure managed by software, such as software-defined data centers.
This characterization comes from Steve Case, The Third Wave, New York: simon & Schuster, 2016, pp. 42-55.
Sources: Arrow, K.J. "Economic Welfare and the Allocation of Resources for Inventions," in R. Nelson (ed.) The Rate and Direction of Inventive Activity , Princeton University Press, 1962.
Timothy F. Bresnahan and Manuel Trajtenberg, "General Purpose Technologies: Engines of Growth," National Bureau of Economic Research Working Paper 4148, August 1992.
Robert J. Gordon, The Rise and Fall of American Growth . Princeton University Press, 2016.
33
Figure 24 provides a very general illustration of the scope of these benefits using recent changes in output
per employee for a few firms that are cloud computing users. We drew upon data from SEC filings to
develop this rough gauge of productivity. We found that Facebook improved its productivity by 46 percent
from 2010 to 2015 and ETSY improved its productivity by 17 percent from 2014 to 2015.
Figure 24. Estimates of Productivity Changes at Specific Firms, Selected Years
Source: 10-K reports filed with the Securities and Exchange Commission, various years.
C. Will software increase productivity?
In cloud environments, software development relies upon the rapid creation of new applications and
swift modification of existing applications. This requires software-defined storage and cloud computing
that supports continuous software delivery (drawing upon DevOps, i.e., shortening the software and
services development cycle; microservices where software is assembled in Lego-like fashion; and
containers, where developers can create a single application and easily run it using a wide range of
operating systems without significant modifications). Using such techniques, firms have been able to
deploy new applications or services in less than an hour.
There are four ways that new software and infrastructure technology will increase productivity:
1. Through lower costs for software development, i.e., through coding new software more efficiently
and designing more efficient processes to write and distribute new software.
a. DevOps increases the time “high performing” firms can spend on new work and reduces
the time spent on rework. This increases the efficiency of software developers and those
in DevOps teams since they provide more output per developer. This conclusion comes
from the “2016 State of DevOps report.”72
It discovered that “high performing” firms are
spending 29 percent more time on new work than low performing firms. In addition, the
high performing firms spend 22 percent less time on rework and unplanned work.
72
Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.”
https://puppet.com/resources/white-paper/2016-state-of-devops-report
34
b. The software development process is also more efficient because of the shift to
“serverless” computing. As we note above, with “serverless” computing, developers no
longer need to be concerned about the physical or virtual servers they need to access to
be able to code This means they no longer need to spend time setting up the
infrastructure for their coding work. So, this also improves developer efficiency.
c. Another example of improved efficiency is microservices (see Figure 18 above) where
applications adopt an architecture in which each functional element is a separate service.
By using microservices, firms can deliver applications and services faster.73
This supports
“high velocity software development” and makes microservices an important part of
continuous service delivery.
d. A fourth example promoting efficiency is containers. For software development,
containers wrap applications in the full system needed to run them. This reduces the time
that would ordinarily be required to set up this infrastructure. The operating system
resides in the container. This advance also hastens software development; it simplifies
how software is written and deployed.
2. By Employing Software-Defined Infrastructure
a. Software costs often decline in software-defined infrastructure. Software employed in
Software Defined Data Centers can be obtained from Open Source software sites such as
GitHub, a sharing hub on the Internet. This can reduce spending on programming. The use of
Open Source and GitHub also permits enterprises to pay only a small fee or no fee rather than
a somewhat expensive licensing fee for vendors’ software.
b. In addition, innovations in the software used to manage software-defined data centers can
provide large gains in costs. Google’s Senior Vice President for Technical Infrastructure, Urs
Hoelzle 74
has noted that implementing a new generation of controller software reduced data
center costs by 50-fold.
3. Through direct spending on new infrastructure that is often “infrastructure as code.”
a. As firms make the transition to software-defined infrastructure, they will invest
substantial amounts to take advantage of cost savings and the ability to do analyses more
rapidly. The new infrastructure, software-defined data centers, supports more refined
data analysis as well as more rapid development and testing of new software and
applications.
b. This renewal of infrastructure is essential for moving firms to a “New Production
Ecosystem” where products can be produced more cheaply and services can be created
more rapidly and at lower cost.
c. This new infrastructure also accelerates the use of analytics. This makes firms more
productive. This occurs because analysis has become so much more central to a firm’s
73
Richard Li, “Microservices Essentials for Executives: The Key to High Velocity Software Development,” for
Entrepreneurs from David Skok. http://www.forentrepreneurs.com/microservices/
74
Urs Hoelzle, “OpenFlow @ Google,” May 7, 2012. https://www.youtube.com/watch?v=VLHJUfgxEO4. Hoelzle
cites an efficiency gain of 50 times since the Controller in Google’s infrastructure “uses modern server hardware
50x (!) better performance.” See Hoelzle’ s slides from the talk, http://opennetsummit.org/archives/apr12/hoelzle-
tue-openflow.pdf
35
operations. Thus, firms have a need to move to real-time analytics to insure they can
analyze production, supply chain, marketing and competitive aspects of their operations.
4. Through the expansion of business opportunities.
a. New ways of developing software reduce the time to market for new applications and
services. This lets enterprises expand the services and products they offer with very short
development times. The result is an increase in firm revenues and productivity, the output
produced per employee. Earlier, we provided an example of this benefit from Lori Beer,
the CIO of JPMorganChase.
b. Big Data analysis, the analysis of large amounts of data (data lakes) gives enterprises
insights into markets. This type of analysis was not easy to perform with highly distributed
data bases. With consolidated data centers, firms can exploit new business opportunities
in ways that add new value.
Businesses can also refine the analysis of designs. This helps them develop products and
services that compete better in the marketplace. Big data analysis also provides better
understanding of the markets businesses are trying to serve.
The improvements in productivity noted above:
a. Reduce the cost of developing and refining new products and services.
b. Create software innovations that operate very much like Moore’s Law, sparking ongoing
expansion in the amount of work that firms can perform in a broad range of operations, not
merely in information technology viewed in a restricted context.
c. Enhance the ability of firms to analyze data at much lower cost than had been the case
previously. This adds considerable value to product and services development. It also
contributes to the Moore’s Law-like cost reductions in a range of business operations that can
exploit the software innovations mentioned above.
D. Why hasn’t software improved productivity in recent years?
We argue that software is driving innovation in the U.S. economy. This is taking place as software plays a
key role in cloud computing, data storage, network use, and changing the operations of communications
and cloud service providers. These changes, driven by enhancements in software, change how firms
operate. In addition, infrastructure as code or software-defined infrastructure has not only reduced costs,
but also provided new opportunities for businesses to expand operations and refine products.
Economists need a better understanding of how software is developed. They need to understand how
programmers have created new generations of software; i.e., that software developers have created
substantial innovation in the ICT sector.
36
The contribution of software and cloud services to the growth of U.S. output is not estimated directly in
national income and product accounts. “ICT capital continues to grow and penetrate the economy --
increasingly via cloud services which are not fully accounted for in the standard narrative on ICT's
contribution to economic growth -- the contribution of ICT to growth in output per hour going forward is
calibrated to be substantially larger than it has been in the past”75
[my emphasis]
These changes are not showing up in U.S. economic statistics. We believe that one reason is
measurement. Government economists who estimate the size of the digital economy do not measure
software innovations directly. They rely upon indirect measures to gauge the size of the digital economy.
These include the prices and performance of mobile phones, laptops, and tablets. As a result, these
economists have not captured how software has created more ubiquitous changes in the economy. These
issues remind us of the spread of electricity through the economy.
One problem is that economists collect data on software innovations indirectly. To estimate changes in
products that use innovative software, economists examine the prices and performance of hardware that
incorporates a good deal of software, such as laptops, mobile devices, and cell phones. These metrics,
largely from hardware, are the main yardstick employed to measure the digital economy. There are no
data sets that measure software innovation directly.
For “cloud and related ICT services, Byrne and Corrado 76
(2016) … imply… these prices should fall no
slower than the rate of decline in ICT asset prices.”77
[This assertion is in line with Daniel Sichel’s conclusion
that “Desktop PCs: hedonic price indexes falling about 15 percent [from] 2007 -2010, more slowly
thereafter.”]78
Yet, both findings are not in accord with the substantial cost changes that Google’s Senior
Vice President for Technical Infrastructure, Urs Hoelzle, reported above.
Press reports suggest that cloud computing and storage services are falling very fast (in the 20 to 30
percent per year range). These services usually are purchased along with software services, which would
substantially moderate overall declines. It should also be noted that the total cost of cloud services (from
a purchasers’ perspective) includes high-speed broadband (WAN and LAN) services.”79
Direct measurement of the digital economy could include:
75
Byrne and Corrado, “ICT Prices and ICT Services.”
76
David Byrne and Carol Corrado, “ICT Prices and ICT Services: What do they tell us about Productivity and
Technology?” The Conference Board, Economics Program Working Paper Series #16-05. May 2016 (revised July
2016). https://www.conference-board.org/pdf_free/workingpapers/EPWP1605.pdf
77
Carol Corrado, “Discussion of: Improving ICT Deflators in the National Accounts,” papers were prepared for the
meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016, p. 4.
http://bea.gov/about/advisory.htm
78
Daniel Sichel, “A New Look at Prices of Personal Computers, Tablets, and Cell Phones: A Progress Report,” papers
were prepared for the meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016.
http://bea.gov/about/advisory.htm
79
Carol Corrado, “Discussion of: Improving ICT Deflators in the National Accounts,” papers were prepared for the
meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016. p. 3.
http://bea.gov/about/advisory.htm
37
a. The impact of continuous service delivery, microservices, containers and DevOps in
accelerating software development.80
b. Delivery speed of software development. Many firms no longer optimize software
development for cost, as economists assume. As Cockcroft notes,81
“Nordstrom is no
longer optimizing for software cost but for delivery speed.”
c. Estimate the contributions of continuous service delivery, microservices, containers and
DevOps to the creation and deployment of software. These innovations have changed the
way that developers82
create, test and deploy software and applications.
Economists have not yet taken these new processes into account. They have not developed ways to
measure innovations in software delivery. This hampers our understanding of how to incorporate
software into the national income accounts.83
At the firm level, the move to more rapid creation of
software is having a clear impact. The 2016 State of DevOps Report,84
notes that high performing firms
have sped up the deployment of new software and attained a level of 200 times more frequent
deployments than comparable low performing firms. Nonetheless, many economists are continuing to
measure price changes in some of the key devices used in the digital economy, such as tablets, desktops
and laptops.85
E. Economists, Software Innovations and the Third Industrial Revolution
Several economists have described their vision of future of economic growth and how it will be shaped by
innovations in information and communications technologies (ICT). We believe that, as a group,
economists have not appreciated the vast potential size of the impact of software innovations. This
oversight is primarily because economists possess few accurate tools to measure the digital economy.
This section takes the previous section’s exploration of software innovations as a given. It proceeds to
examine writings by a few well-known economists that examine whether the Internet Revolution is likely
to have any impacts on GDP and jobs.
80
Adrian Cockcroft, “Creating Business Value with Cloud Infrastructure,” Open Networking Users Group meeting,
May 13-14, 2015, Columbia University.
81
Cockcroft, p. 12.
82
Cockcroft, pp. 52- 112. Robert Cohen, The Internet of Things, Productivity, and Employment,” presentation for
Internet of Things Summit, Boston, Sept. 8-9, 2015 offers a summary of the main points Cockcroft makes.
83
Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.”
https://puppet.com/resources/white-paper/2016-state-of-devops-report
84
Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.”
85
Ana Aizcorbe, “Improving ICT Deflators in the National Accounts.” Dan Sichel, Dave Byrne and Steve Oliner, “A
New Look at Prices of Personal Computers, Tablets, and Cell Phones: A Progress Report: paper for BEA Advisory
Committee.” Guilia McHenry, “Measuring the Digital Economy: Motivations and Initiatives.” These papers were
prepared for the meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016.
http://bea.gov/about/advisory.htm
38
1. Robert Gordon and The Rise and Fall of American Growth
Robert Gordon is the strongest advocate of the view that ICT innovations are likely to have a small impact
on future US growth and productivity. Gordon’s recent book86
discusses many important innovations from
the Second Industrial Revolution that occurred from 1870 to 1970. These include the internal combustion
engine, electrification, the airplane, and the refrigerator. After reviewing these innovations and recent
ones tied to ICTs, Gordon concludes that there are likely to be fewer innovations in the future. This
conclusion is linked to his conviction that the chances that a widely used, revolutionary technology will
emerge over the next decade or two is unlikely.
After carefully reviewing the impact of General Purpose Technologies – technologies that were adopted
widely and adapted to the requirements of a broad range of industries during the Second Industrial
Revolution (IR#2), Gordon considers whether the Internet and ICT, including software, might play a similar
role in the Third Industrial Revolution. After an examination of recent innovations, he finds that the US
economy is likely to experience lower productivity growth and lower levels of GDP growth.
Gordon’s reasoning follows a logic that flows from his detailed review of the US experience during the
special century, or Second Industrial Revolution, from 1870 to 1970. The key characteristic of the special
century was that firms adopted important new technologies and refined them so that they could achieve
significant increases in productivity growth. These productivity and GDP gains peaked during the 1940s
and continued through the early 1970s.
In discussing Gordon’s book below, we identify ways that the Next Industrial Revolution (IR#3) is beginning
to have real impacts in businesses beyond the ICT and entertainment industries. Our examples suggest
there are likely to be opportunities for improvements in productivity and cost savings in IR#3.
2. Why Gordon concludes that the Next Industrial Revolution (IR#3) will have Little Impact on
GDP Growth and Jobs
After a review of more recent growth and productivity trends and ICT innovations, Gordon concludes that
nearly all innovations based on the Internet ended in 2004. He dismisses the notion that the Third
Industrial Revolution (IR#3), the Internet Revolution, associated with computers and digitization, will be
like the Second Industrial Revolution. He explains that this is likely to be true because “business practices
in the office, the retail sector, and in the banking and financial sector … current methods of production
had been largely achieved by 2004.”87
Gordon finds that innovation has continued since 1970, but it has not grown at the same rate as it did
earlier. He argues that this is likely to result in slow productivity and economic growth. One reason for
Gordon’s conclusion is his belief that the recent innovations of IR#3 affect only a few industries, including
“entertainment and information and communications technology (ICT).”88
He estimates that the growth
of total factor productivity (TFP) in the US declined after 2004, with productivity growth only half as fast
86
Robert Gordon, The Rise and Fall of American Growth, Princeton University Press, 2016, p. 567.
87
Robert Gordon, The Rise and Fall of American Growth, Princeton University Press, 2016, p. 567.
88
Gordon, p. 567.
39
as it was from 2004 to 2014, as compared to 1994 and 2004. In part, he finds that this is due to the slow
transformation of business practices. Today, Gordon notes, office employees’ productivity resembles
what it was in about 2004 because they had all the equipment used in office work today.89
Gordon buttresses his argument by asserting that IR#3 has impacted only a few key industries. He
discounts the chances that industries besides entertainment, information and communications
technologies will benefit from IR#3. He writes that there has been a “stasis in retailing” and that “the main
impact on retail productivity growth of big-box [retailing] stores … largely occurred by a decade ago.”90
This ignores the rise of Amazon and its reshaping of retailing, as well as significant efforts by WalMart,
Kroger’s and Nordstrom’s to reshape their operations around digitized services and mobile software.
Gordon cites the decline in stock trading after the financial crisis as evidence of a “plateau of activity in
finance and banking.”91
In reviewing the home and consumer electronics industries, Gordon concludes
that “within the past decade … computer hardware, software, and business methods ossified into a slowly
changing set of routines.”92
Intel’s work on artificial intelligence (AI) contradicts Gordon’s claim that the business models of
the hardware, software and home and consumer electronics industries have “ossified.” Nidhi
Chappell, Director of AI Strategy, at Intel notes93
that Intel has built upon Moore’s Law, data
availability and innovation in algorithms to drive more widespread AI use. Intel broadens AI’s use
by compressing the innovation cycle (a major change in routines), democratizing access to AI and
guiding the development of AI in service of humankind (by solving cancer, decoding the function
of the human brain, etc.).
Intel has also used Big Data and software engineering to solve high-value problems94
. It has
assembled teams of 5 people who focus on solving production problems for 6 months. They use
historical data as well as unstructured data to predict business outcomes. Each team is expected
to save Intel at least $10 million in six months. The Intel teams harness new, Big Data skills plus
software-defined, cloud computing infrastructure to analyze large databases. They reduce
manufacturing costs by enhancing the initial testing of new semiconductor manufacturing
processes. When Intel saves I second of test time in production, it saves $5 million to $10 million.
This breakthrough solves defects and problems in production by identifying their root cause.
Again, this evidence runs counter to Gordon’s contention that the industry has ossified.
89
Gordon, p. 580. Gordon should examine what has happened to productivity on farms where new technology has
benefitted from Big Data analysis. See the case of John Deere in Robert B. Cohen, “Case Studies of the Internet of
Things,” IoT Slam ’17, Durham, North Carolina, June 20, 2017. https://es.slideshare.net/bcohen777/case-studies-
of-the-internet-of-things-062017
90
Gordon, p. 581.
91
Gordon, p. 582.
92
Gordon, p. 583. Note, that the previous section offers little evidence of this “ossification.”
93
Nidhi Chappell, Intel’s Director of AI Strategy, “Under the Hood: Intel Accelerating the Future of Artificial
Intelligence | Intel IT Center,” https://www.youtube.com/watch?v=MKFIvNTre2I
94
Moty Fania and Assaf Araki, “Solving High Value Problems with Big Data Analytics,” Big Data Analytics DMBI 2014
Second Annual International Conference, Sept. 14, 2014. https://www.youtube.com/watch?v=HmhCjYYAmz8
40
Gordon’s conclusions also overlook the sizable recent changes in business practices and business
structure.
The Wall Street Journal has cited Equifax Inc., insurer Liberty Mutual, and consumer-products
giant Procter & Gamble Co., as firms that have adopted a mobile, cloud and data technology
strategy from Silicon Valley. They are changing the way they operate, taking on many of the
characteristics of tech and Internet firms, such as Facebook, Google, and Amazon.95
Many large
firms such as JPMorganChase, Proctor and Gamble, Boeing, Ford and General Motors, are shifting
to shorter development cycles. This improves their agility, particularly their responsiveness to
changes in markets. Gordon overlooks these changes in changes in corporate behavior.
Gordon emphasizes that price declines for ICT equipment relative to performance has slowed. He cites
data showing that by 2014, there were almost no price declines at all, as compared to rapid price declines
in the late 1990s.96
Gordon expects that this slowdown to continue at the same pace, repeating the slow
rate of TFP growth from 2004-2014.
Gordon asks whether the next wave of innovations might prove to be as revolutionary as they were during
the dot-com revolution of the late 1990s. He finds that if the pace of innovations decelerates, it will result
in a productivity growing at the same rate as it did during 2004-2014.97
One note in rebuttal is that some
scholars have developed a performance indicator that they have merged with the producer price index
(PPI) for servers. With this adjusted index, price changes, i.e., the blended indicator and PPI, fall 11 percent
faster than the Bureau of Economic Analysis Investment Index.98
3. Gordon’s Evaluation of Future Advances in Technology and their Impact on Total Factor
Productivity Growth
To assess future technology-based advances that Eric Brynjolfsson and Andrew McAfee as well as others
forecast, Gordon examines four categories where technology may have big impacts: medical; small robots
and 3D printing; big data; and driverless cars.99
Gordon evaluates whether new technologies in these areas
might bring TFP growth back to the levels it attained in the late 1990s. We review Gordon’s conclusions
about these recent innovations. We find, contrary to Gordon, that there is a greater chance for increased
TFP growth in the coming years than Gordon believes.100
95
Angus Loten and John Simons, “Leadership Evolves Amid Tech Changes: Equifax, P&G, Liberty Mutual embrace
digital tools; managers shift toward shorter development cycles, Wall Street Journal, CIO Journal, Jan 3, 2017.
http://blogs.wsj.com/cio/2017/01/03/tech-is-transforming-how-businesses-are-run/
96
Gordon, p. 593.
97
Gordon, p. 593.
98
David Byrne and Carol Corrado, “ICT Asset Prices: Marshaling Evidence into New Measures,” The Conference
Board, July 10, 2016. https://www.conference-board.org/pdf_free/workingpapers/EPWP1606.pdf
https://www.conference-board.org/publications/publicationdetail.cfm?publicationid=7241&centerId=8, The
Conference Board, July 2016, p. 12.
99
Gordon, p. 593.
100
See the discussion in Sections D and E.
41
In examining medical and pharmaceutical advances, Gordon finds that medical technology has continued
to advance since 1980 but at a “slower and measured pace.” He finds that pharmaceutical research has
hit a “brick wall of rapidly increasing costs and declining benefits.”101
It is disconcerting that Gordon has
not examined a few of the recent major achievements of medical science. The cost of decoding a human
genome has fallen much faster than Moore’s Law.102
With such a dramatic change, innovative firms have
decoded the human genome inexpensively and developed entirely new ways to treat diseases. This is
happening despite claims by the largest pharmaceutical firms that it costs $2.6 billion to create a new
drug. Gordon overlooks the fact that rapidly declining costs of decoding the human genome are likely to
result in more opportunities to develop new drugs at much lower cost.
Some of the earliest drugs to take advantage of breakthroughs in decoding the genome are
“Pfizer's lung cancer treatment Xalkori, … approved in 2011; [it]… targets mutations in tumors
driving the disease. … Vertex Pharmaceuticals … changed the treatment of cystic fibrosis with
Kalydeco; [it] … targets the disease’s underlying genetic cause.”103
These discoveries have
benefitted from the rapid decline in the cost of decoding genomes. As shown in Figure 25, this
cost has fallen faster than Moore’s Law.
101
Gordon, p. 594. g
102
National Institutes of Health, National Human Genome Research Institute “The Cost of Sequencing a Human
Genome” https://www.genome.gov/sequencingcosts/ As this site notes, “The underlying costs associated with
different methods and strategies for sequencing genomes are of great interest because they influence the scope
and scale of almost all genomics research projects.”
103
Julie Steenhuysen “How DNA sequencing is transforming the hunt for new drugs,” Science News, May 13, 2015.
http://www.reuters.com/article/us-health-precisionmedicine-insight-idUSKBN0NY0AX20150513
42
Figure 25. National Human Genome Research Institute’s Estimates of The Cost of Decoding based
on Costs at its Human Genome Centers, as compared to Moore’s Law Price Changes
Source: National Institutes of Health, National Human Genome Research Institute, “The Cost of Sequencing a Human
Genome” https://www.genome.gov/sequencingcosts/
Gordon contends that pharmaceutical research has hit a wall of rapidly increasing costs. This is true if one
accepts the findings of the Tufts Center for the Study of Drug Development that it costs $2.6 billion to
create a new drug. A cogent criticism104
of these results is that they consider only new molecular entities,
drugs with chemical compounds that have never been approved for individual use or combination
therapies. This sample represents a small part of the population of new drugs developed each year. Thus,
the Tufts results are biased and provide much higher costs of developing new drugs than may be the case.
Tufts excludes many of the drugs developed with funding from the National Institutes of Health and other
government entities. In addition, successful case, such as Pfizer’s lung cancer treatment and other drugs
that use the body’s immune system to fight cancer, have benefitted from cloud computing.
104
Aaron Carroll, “$2.6 Billion to Develop a Drug? New Estimate Makes Questionable Assumptions” New York
Times, November 19, 2014. https://www.nytimes.com/2014/11/19/upshot/calculating-the-real-costs-of-
developing-a-new-drug.html
43
In discussing the limitations of innovations such as small robots, Gordon reasons that it is difficult for them
to “match a human’s dexterity and problem-solving abilities.”105
He notes that it is difficult for robots to
distinguish between picking up lace and crumpled jeans.
Research completed after the publication of Gordon’s book reported advances in machine
learning that have improved robots’ dexterity. By combining vision and touch,106
a lab at the
University of California, Berkeley enhanced robotic dexterity so a robot could fold clothes. This is
the challenge that Gordon did not believe could be solved.
Gordon notes that 3-D printing is “not expected to have much effect on mass production and thus on how
most U.S. consumer goods are produced.”107
Recently, engineers at Renault Truck108
have boosted performance of the company’s truck engines
by using 3-D printing to produce 25 percent of these engines’ parts. This greatly simplifies
production and the advance is likely to be copied by other automakers around the world. In
addition, using 3-D printing for engine parts improves performance and increases the workloads
trucks can haul because the 3-D parts are often lighter and more reliable. Thus, they can lower
fuel consumption. Success in this area might be repeated in creating consumer goods.
4. Tim Bresnahan and Pai-Ling Yin on innovation in ICT industries
Timothy Bresnahan and Pai‐Ling Yin109
offer another interpretation of the potential impact of IR#3. They
indicate that “the invention of new applications based on information and communications technologies
(ICTs) has had two economic effects up to now.” It has “transformed production” and shifted the demand
for skilled labor in the workforce. They also make a case that ICT innovation requires a time-consuming
and complex co-invention process for innovations such as new software.
One of the assertions that Bresnahan and Yin make is that ICTs are enablers of the invention of new
applications and that most of the innovation occurs in firms outside of the ICT industries. They focus on
co-invention, “the product and process improvements created by industries as they apply new ICT,” noting
that:
“ICT co‐invention is defined as the product and process improvements created by industries
as they apply new ICT. One driver of co‐invention is the ICT advances themselves (supply), such
as cheaper storage, faster networks, or more capable software. ICT advances produce a large
scope of feasible opportunities. The other driver is the industry circumstances (demand) of
105
Gordon, p. 596.
106
John Markoff, “New Approach Trains Robots to Match Human Dexterity and Speed,” New York Times, May 21,
2015. https://www.nytimes.com/2015/05/22/science/robots-that-can-match-human-dexterity.html?_r=0
107
Gordon, p. 597.
108
Sam Davies, “Renault Truck introducing metal additive manufacturing to engine production process,” TCT
Magazine, January 11, 2017. http://www.tctmagazine.com/3D-printing-news/renault-truck-introducing-metal-
additive-manufacturing-engine/
109
Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the
Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf to appear in Shane
Greenstein, Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17.
44
firms trying to use ICT: competition, customer demand, and the production processes already
in place.”110
Bresnahan and Yin argue that co-invention today is very much like it was during previous ICT advances.
They present this as their view of ICT innovations, but do not consider evidence about software
innovations that we have mentioned above.
As we have noted, many new processes have streamlined software development and accelerated the rate
at which they are deployed. Tremendous changes have taken place in software development, including
continuous service delivery, containers, microservices and “serverless” computing. These innovations
should have resulted in a reconsideration of how today’s ICT co-invention process works.
Bresnahan and Ying write that “while there has been terrific technical progress in ICT, there has been little
change in the ICT co‐invention process. Co-invention still requires considerable brainpower and
experimentation. Co‐invention still looks for ways to change whole organizations. Indeed, modern co‐
invention often looks for ways to change whole supply chains.”111
Figure 26. ICT Co-Invention of Applications
Source: Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the
Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf , p. 3.
We would disagree with the characterization of the invention of new applications, particularly new
software, that Bresnahan and Yin utilize. The perpetual development model takes place largely in firms
110
Bresnahan and Yin, p. 2.
111
Bresnahan and Yin, p. 3.
Software Innovations Transforming the US Economy
Software Innovations Transforming the US Economy
Software Innovations Transforming the US Economy
Software Innovations Transforming the US Economy

Más contenido relacionado

La actualidad más candente

Sogeti big data - no more secrets with big data analytics
Sogeti   big data - no more secrets with big data analyticsSogeti   big data - no more secrets with big data analytics
Sogeti big data - no more secrets with big data analyticsRick Bouter
 
Leveraging the-digital-future
Leveraging the-digital-futureLeveraging the-digital-future
Leveraging the-digital-futureExo Futures
 
Internet of things presentation rick bouter - 16-09-2013 - sogeti - Part 1 ...
Internet of things presentation   rick bouter - 16-09-2013 - sogeti - Part 1 ...Internet of things presentation   rick bouter - 16-09-2013 - sogeti - Part 1 ...
Internet of things presentation rick bouter - 16-09-2013 - sogeti - Part 1 ...Rick Bouter
 
Melbourne ICT Scorecard
Melbourne ICT ScorecardMelbourne ICT Scorecard
Melbourne ICT ScorecardVishal Sharma
 
[McKinsey] Disruptive technologies: Advances that will transform life, busine...
[McKinsey] Disruptive technologies: Advances that will transform life, busine...[McKinsey] Disruptive technologies: Advances that will transform life, busine...
[McKinsey] Disruptive technologies: Advances that will transform life, busine...Andrew Kaplan
 
Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...
Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...
Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...Jonathan Gray
 
Sai priya the time flies, and it has been two centuries that th
Sai priya the time flies, and it has been two centuries that thSai priya the time flies, and it has been two centuries that th
Sai priya the time flies, and it has been two centuries that thamrit47
 
Software is Eating Bio
Software is Eating BioSoftware is Eating Bio
Software is Eating Bioa16z
 
Adoption of Communication Tools in Agriculture
Adoption of Communication Tools in AgricultureAdoption of Communication Tools in Agriculture
Adoption of Communication Tools in AgriculturePaulsen
 
Cloud Computing: Big Data Technology
Cloud Computing: Big Data TechnologyCloud Computing: Big Data Technology
Cloud Computing: Big Data TechnologyBooz Allen Hamilton
 
School introduction to mis is the most important class in the
School introduction to mis is the most important class in the School introduction to mis is the most important class in the
School introduction to mis is the most important class in the amrit47
 
Disruptive technologies: Advances that will transform life, business, and the...
Disruptive technologies: Advances that will transform life, business, and the...Disruptive technologies: Advances that will transform life, business, and the...
Disruptive technologies: Advances that will transform life, business, and the...alexandre stopnicki
 
FUTURISTIC TECHNOLOGIES
FUTURISTIC TECHNOLOGIESFUTURISTIC TECHNOLOGIES
FUTURISTIC TECHNOLOGIESChuck Brooks
 
Big Data has Big Problems
Big Data has Big ProblemsBig Data has Big Problems
Big Data has Big ProblemsAhmed Banafa
 
Software development for the diversification of Nigeria Ecomony
Software development for the diversification of Nigeria EcomonySoftware development for the diversification of Nigeria Ecomony
Software development for the diversification of Nigeria EcomonyPatrick Ogbuitepu
 

La actualidad más candente (18)

Sogeti big data - no more secrets with big data analytics
Sogeti   big data - no more secrets with big data analyticsSogeti   big data - no more secrets with big data analytics
Sogeti big data - no more secrets with big data analytics
 
Leveraging the-digital-future
Leveraging the-digital-futureLeveraging the-digital-future
Leveraging the-digital-future
 
GITA April 2015 Newsletter
GITA April 2015 NewsletterGITA April 2015 Newsletter
GITA April 2015 Newsletter
 
Internet of things presentation rick bouter - 16-09-2013 - sogeti - Part 1 ...
Internet of things presentation   rick bouter - 16-09-2013 - sogeti - Part 1 ...Internet of things presentation   rick bouter - 16-09-2013 - sogeti - Part 1 ...
Internet of things presentation rick bouter - 16-09-2013 - sogeti - Part 1 ...
 
Melbourne ICT Scorecard
Melbourne ICT ScorecardMelbourne ICT Scorecard
Melbourne ICT Scorecard
 
Digital Csuite
Digital CsuiteDigital Csuite
Digital Csuite
 
[McKinsey] Disruptive technologies: Advances that will transform life, busine...
[McKinsey] Disruptive technologies: Advances that will transform life, busine...[McKinsey] Disruptive technologies: Advances that will transform life, busine...
[McKinsey] Disruptive technologies: Advances that will transform life, busine...
 
Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...
Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...
Ways of Seeing Data: Towards a Critical Literacy for Data Visualisations as R...
 
Sai priya the time flies, and it has been two centuries that th
Sai priya the time flies, and it has been two centuries that thSai priya the time flies, and it has been two centuries that th
Sai priya the time flies, and it has been two centuries that th
 
Software is Eating Bio
Software is Eating BioSoftware is Eating Bio
Software is Eating Bio
 
Adoption of Communication Tools in Agriculture
Adoption of Communication Tools in AgricultureAdoption of Communication Tools in Agriculture
Adoption of Communication Tools in Agriculture
 
MIT THE FUTURE OF WORK 201616-02-Work
MIT THE FUTURE OF WORK 201616-02-WorkMIT THE FUTURE OF WORK 201616-02-Work
MIT THE FUTURE OF WORK 201616-02-Work
 
Cloud Computing: Big Data Technology
Cloud Computing: Big Data TechnologyCloud Computing: Big Data Technology
Cloud Computing: Big Data Technology
 
School introduction to mis is the most important class in the
School introduction to mis is the most important class in the School introduction to mis is the most important class in the
School introduction to mis is the most important class in the
 
Disruptive technologies: Advances that will transform life, business, and the...
Disruptive technologies: Advances that will transform life, business, and the...Disruptive technologies: Advances that will transform life, business, and the...
Disruptive technologies: Advances that will transform life, business, and the...
 
FUTURISTIC TECHNOLOGIES
FUTURISTIC TECHNOLOGIESFUTURISTIC TECHNOLOGIES
FUTURISTIC TECHNOLOGIES
 
Big Data has Big Problems
Big Data has Big ProblemsBig Data has Big Problems
Big Data has Big Problems
 
Software development for the diversification of Nigeria Ecomony
Software development for the diversification of Nigeria EcomonySoftware development for the diversification of Nigeria Ecomony
Software development for the diversification of Nigeria Ecomony
 

Similar a Software Innovations Transforming the US Economy

Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...
Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...
Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...Economic Strategy Institute
 
New IP Initial Findings from a New Research Project
New IP Initial Findings from a New Research ProjectNew IP Initial Findings from a New Research Project
New IP Initial Findings from a New Research ProjectEconomic Strategy Institute
 
The future of software engineering: Visions of 2025 and beyond
The future of software engineering: Visions of 2025 and beyond The future of software engineering: Visions of 2025 and beyond
The future of software engineering: Visions of 2025 and beyond IJECEIAES
 
New ip initial findings of a Project on the Economic and Business Impacts of ...
New ip initial findings of a Project on the Economic and Business Impacts of ...New ip initial findings of a Project on the Economic and Business Impacts of ...
New ip initial findings of a Project on the Economic and Business Impacts of ...Economic Strategy Institute
 
How to build effective and cheaper m-payments with Open Source
How to build effective and cheaper m-payments with Open SourceHow to build effective and cheaper m-payments with Open Source
How to build effective and cheaper m-payments with Open SourceBMI Healthcare
 
Ibm global technology outlook 2013
Ibm   global technology outlook 2013Ibm   global technology outlook 2013
Ibm global technology outlook 2013Rick Bouter
 
Global-Technology-Outlook-2013
Global-Technology-Outlook-2013Global-Technology-Outlook-2013
Global-Technology-Outlook-2013IBM Switzerland
 
THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915
THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915
THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915Economic Strategy Institute
 
The Internet of Things, Productivity, and Employment
The Internet of Things, Productivity, and Employment The Internet of Things, Productivity, and Employment
The Internet of Things, Productivity, and Employment Alex Krause
 
A Literature Review On Internet Of Things (IoT)
A Literature Review On Internet Of Things (IoT)A Literature Review On Internet Of Things (IoT)
A Literature Review On Internet Of Things (IoT)April Smith
 
It Trends 2008 (Tin180 Com)
It Trends   2008 (Tin180 Com)It Trends   2008 (Tin180 Com)
It Trends 2008 (Tin180 Com)Tin180 VietNam
 
How economists should think about the revolutionary changes taking place in h...
How economists should think about the revolutionary changes taking place in h...How economists should think about the revolutionary changes taking place in h...
How economists should think about the revolutionary changes taking place in h...Economic Strategy Institute
 
IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013
IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013
IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013IBM Switzerland
 
IBM Research - Technology Outlook 2013
IBM Research - Technology Outlook 2013IBM Research - Technology Outlook 2013
IBM Research - Technology Outlook 2013IBM_CH
 
Global technology outlook_2013
Global technology outlook_2013Global technology outlook_2013
Global technology outlook_2013IBM Software India
 
ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013
ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013
ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013IBM Switzerland
 
Predicted! Top Software Development Trends for 2021
Predicted! Top Software Development Trends for 2021Predicted! Top Software Development Trends for 2021
Predicted! Top Software Development Trends for 2021Pixel Crayons
 
World Economic Forum Tipping Points Report
World Economic Forum Tipping Points ReportWorld Economic Forum Tipping Points Report
World Economic Forum Tipping Points ReportSergey Nazarov
 
Open Source Software Survivability Analysis Using Communication Pattern Valid...
Open Source Software Survivability Analysis Using Communication Pattern Valid...Open Source Software Survivability Analysis Using Communication Pattern Valid...
Open Source Software Survivability Analysis Using Communication Pattern Valid...IOSR Journals
 

Similar a Software Innovations Transforming the US Economy (20)

Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...
Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...
Pervasive, intelligent cloud ecosystems, spectacular firms and frontier firms...
 
New IP Initial Findings from a New Research Project
New IP Initial Findings from a New Research ProjectNew IP Initial Findings from a New Research Project
New IP Initial Findings from a New Research Project
 
The future of software engineering: Visions of 2025 and beyond
The future of software engineering: Visions of 2025 and beyond The future of software engineering: Visions of 2025 and beyond
The future of software engineering: Visions of 2025 and beyond
 
New ip initial findings of a Project on the Economic and Business Impacts of ...
New ip initial findings of a Project on the Economic and Business Impacts of ...New ip initial findings of a Project on the Economic and Business Impacts of ...
New ip initial findings of a Project on the Economic and Business Impacts of ...
 
The Evolution of Software Development.pdf
The Evolution of Software Development.pdfThe Evolution of Software Development.pdf
The Evolution of Software Development.pdf
 
How to build effective and cheaper m-payments with Open Source
How to build effective and cheaper m-payments with Open SourceHow to build effective and cheaper m-payments with Open Source
How to build effective and cheaper m-payments with Open Source
 
Ibm global technology outlook 2013
Ibm   global technology outlook 2013Ibm   global technology outlook 2013
Ibm global technology outlook 2013
 
Global-Technology-Outlook-2013
Global-Technology-Outlook-2013Global-Technology-Outlook-2013
Global-Technology-Outlook-2013
 
THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915
THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915
THE INTERNET OF THINGS, PRODUCTIVITY AND EMPLOYMENT Boston 0915
 
The Internet of Things, Productivity, and Employment
The Internet of Things, Productivity, and Employment The Internet of Things, Productivity, and Employment
The Internet of Things, Productivity, and Employment
 
A Literature Review On Internet Of Things (IoT)
A Literature Review On Internet Of Things (IoT)A Literature Review On Internet Of Things (IoT)
A Literature Review On Internet Of Things (IoT)
 
It Trends 2008 (Tin180 Com)
It Trends   2008 (Tin180 Com)It Trends   2008 (Tin180 Com)
It Trends 2008 (Tin180 Com)
 
How economists should think about the revolutionary changes taking place in h...
How economists should think about the revolutionary changes taking place in h...How economists should think about the revolutionary changes taking place in h...
How economists should think about the revolutionary changes taking place in h...
 
IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013
IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013
IBM Health Innovation Forum 2013 - IBM Research Technology Outlook 2013
 
IBM Research - Technology Outlook 2013
IBM Research - Technology Outlook 2013IBM Research - Technology Outlook 2013
IBM Research - Technology Outlook 2013
 
Global technology outlook_2013
Global technology outlook_2013Global technology outlook_2013
Global technology outlook_2013
 
ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013
ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013
ISDC 2013_Referat_Moshe Rappoport_IBM GTO 2013
 
Predicted! Top Software Development Trends for 2021
Predicted! Top Software Development Trends for 2021Predicted! Top Software Development Trends for 2021
Predicted! Top Software Development Trends for 2021
 
World Economic Forum Tipping Points Report
World Economic Forum Tipping Points ReportWorld Economic Forum Tipping Points Report
World Economic Forum Tipping Points Report
 
Open Source Software Survivability Analysis Using Communication Pattern Valid...
Open Source Software Survivability Analysis Using Communication Pattern Valid...Open Source Software Survivability Analysis Using Communication Pattern Valid...
Open Source Software Survivability Analysis Using Communication Pattern Valid...
 

Más de Economic Strategy Institute

Future of work machine learning and middle level jobs 112618
Future of work machine learning and middle level jobs 112618Future of work machine learning and middle level jobs 112618
Future of work machine learning and middle level jobs 112618Economic Strategy Institute
 
How cloud computing, open source, AI and ML will change the Economy jerusale...
How cloud computing, open source, AI and ML will change the Economy  jerusale...How cloud computing, open source, AI and ML will change the Economy  jerusale...
How cloud computing, open source, AI and ML will change the Economy jerusale...Economic Strategy Institute
 
Future of jobs and digital economy citi conference 090618
Future of jobs and digital economy citi conference 090618Future of jobs and digital economy citi conference 090618
Future of jobs and digital economy citi conference 090618Economic Strategy Institute
 
How software is transforming the u.s. economy 080517
How software is transforming the u.s. economy 080517How software is transforming the u.s. economy 080517
How software is transforming the u.s. economy 080517Economic Strategy Institute
 
Illustrations of software innovation and time to business value or speed of d...
Illustrations of software innovation and time to business value or speed of d...Illustrations of software innovation and time to business value or speed of d...
Illustrations of software innovation and time to business value or speed of d...Economic Strategy Institute
 
IoT, new business models and digital agility 102116
IoT, new business models and digital agility 102116IoT, new business models and digital agility 102116
IoT, new business models and digital agility 102116Economic Strategy Institute
 
How IoT helps Firms Optimize their Digital Performance. Economic Impact of IoT
How IoT helps Firms Optimize their Digital Performance. Economic Impact of IoTHow IoT helps Firms Optimize their Digital Performance. Economic Impact of IoT
How IoT helps Firms Optimize their Digital Performance. Economic Impact of IoTEconomic Strategy Institute
 

Más de Economic Strategy Institute (17)

Digitization and organizational change 121918
Digitization and organizational change 121918Digitization and organizational change 121918
Digitization and organizational change 121918
 
Future of work machine learning and middle level jobs 112618
Future of work machine learning and middle level jobs 112618Future of work machine learning and middle level jobs 112618
Future of work machine learning and middle level jobs 112618
 
How cloud computing, open source, AI and ML will change the Economy jerusale...
How cloud computing, open source, AI and ML will change the Economy  jerusale...How cloud computing, open source, AI and ML will change the Economy  jerusale...
How cloud computing, open source, AI and ML will change the Economy jerusale...
 
Future of jobs and digital economy citi conference 090618
Future of jobs and digital economy citi conference 090618Future of jobs and digital economy citi conference 090618
Future of jobs and digital economy citi conference 090618
 
How software is transforming the U.S. economy
How software is transforming the U.S. economy How software is transforming the U.S. economy
How software is transforming the U.S. economy
 
Global ai conference nyc - oct 23 - 24 2017
Global ai conference  nyc - oct 23 - 24  2017Global ai conference  nyc - oct 23 - 24  2017
Global ai conference nyc - oct 23 - 24 2017
 
How software is transforming the u.s. economy 080517
How software is transforming the u.s. economy 080517How software is transforming the u.s. economy 080517
How software is transforming the u.s. economy 080517
 
Illustrations of software innovation and time to business value or speed of d...
Illustrations of software innovation and time to business value or speed of d...Illustrations of software innovation and time to business value or speed of d...
Illustrations of software innovation and time to business value or speed of d...
 
Case studies of the internet of things 062017
Case studies of the internet of things 062017Case studies of the internet of things 062017
Case studies of the internet of things 062017
 
Case studies of the internet of things 062017
Case studies of the internet of things 062017Case studies of the internet of things 062017
Case studies of the internet of things 062017
 
IoT, new business models and digital agility 102116
IoT, new business models and digital agility 102116IoT, new business models and digital agility 102116
IoT, new business models and digital agility 102116
 
How IoT helps Firms Optimize their Digital Performance. Economic Impact of IoT
How IoT helps Firms Optimize their Digital Performance. Economic Impact of IoTHow IoT helps Firms Optimize their Digital Performance. Economic Impact of IoT
How IoT helps Firms Optimize their Digital Performance. Economic Impact of IoT
 
Columbia CITI Econonomics of Net 060515 final
Columbia CITI Econonomics of Net 060515 finalColumbia CITI Econonomics of Net 060515 final
Columbia CITI Econonomics of Net 060515 final
 
Cross Skilling Jobs and Move to SDN
Cross Skilling Jobs and Move to SDNCross Skilling Jobs and Move to SDN
Cross Skilling Jobs and Move to SDN
 
Cross Skilling Jobs and Move to SDN
Cross Skilling Jobs and Move to SDNCross Skilling Jobs and Move to SDN
Cross Skilling Jobs and Move to SDN
 
Columbia citi economics of net 060515 final
Columbia citi economics of net 060515 finalColumbia citi economics of net 060515 final
Columbia citi economics of net 060515 final
 
Economics of networks and data centers
Economics of networks and data centersEconomics of networks and data centers
Economics of networks and data centers
 

Último

Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...
Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...
Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...amilabibi1
 
Money Forward Integrated Report “Forward Map” 2024
Money Forward Integrated Report “Forward Map” 2024Money Forward Integrated Report “Forward Map” 2024
Money Forward Integrated Report “Forward Map” 2024Money Forward
 
NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...
NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...
NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...Amil baba
 
Global Economic Outlook, 2024 - Scholaride Consulting
Global Economic Outlook, 2024 - Scholaride ConsultingGlobal Economic Outlook, 2024 - Scholaride Consulting
Global Economic Outlook, 2024 - Scholaride Consultingswastiknandyofficial
 
Guard Your Investments- Corporate Defaults Alarm.pdf
Guard Your Investments- Corporate Defaults Alarm.pdfGuard Your Investments- Corporate Defaults Alarm.pdf
Guard Your Investments- Corporate Defaults Alarm.pdfJasper Colin
 
Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...
Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...
Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...Amil baba
 
Economic Risk Factor Update: April 2024 [SlideShare]
Economic Risk Factor Update: April 2024 [SlideShare]Economic Risk Factor Update: April 2024 [SlideShare]
Economic Risk Factor Update: April 2024 [SlideShare]Commonwealth
 
『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书
『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书
『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书rnrncn29
 
Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...
Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...
Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...Amil baba
 
Financial Preparation for Millennia.pptx
Financial Preparation for Millennia.pptxFinancial Preparation for Millennia.pptx
Financial Preparation for Millennia.pptxsimon978302
 
The Inspirational Story of Julio Herrera Velutini - Global Finance Leader
The Inspirational Story of Julio Herrera Velutini - Global Finance LeaderThe Inspirational Story of Julio Herrera Velutini - Global Finance Leader
The Inspirational Story of Julio Herrera Velutini - Global Finance LeaderArianna Varetto
 
AnyConv.com__FSS Advance Retail & Distribution - 15.06.17.ppt
AnyConv.com__FSS Advance Retail & Distribution - 15.06.17.pptAnyConv.com__FSS Advance Retail & Distribution - 15.06.17.ppt
AnyConv.com__FSS Advance Retail & Distribution - 15.06.17.pptPriyankaSharma89719
 
NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...
NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...
NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...Amil baba
 
NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...
NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...
NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...Amil baba
 
cost of capital questions financial management
cost of capital questions financial managementcost of capital questions financial management
cost of capital questions financial managementtanmayarora23
 
Overview of Inkel Unlisted Shares Price.
Overview of Inkel Unlisted Shares Price.Overview of Inkel Unlisted Shares Price.
Overview of Inkel Unlisted Shares Price.Precize Formely Leadoff
 
Banking: Commercial and Central Banking.pptx
Banking: Commercial and Central Banking.pptxBanking: Commercial and Central Banking.pptx
Banking: Commercial and Central Banking.pptxANTHONYAKINYOSOYE1
 
Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...
Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...
Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...Amil baba
 
PMFBY , Pradhan Mantri Fasal bima yojna
PMFBY , Pradhan Mantri  Fasal bima yojnaPMFBY , Pradhan Mantri  Fasal bima yojna
PMFBY , Pradhan Mantri Fasal bima yojnaDharmendra Kumar
 
Unit 4.1 financial markets operations .pdf
Unit 4.1 financial markets operations .pdfUnit 4.1 financial markets operations .pdf
Unit 4.1 financial markets operations .pdfSatyamSinghParihar2
 

Último (20)

Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...
Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...
Amil Baba In Pakistan amil baba in Lahore amil baba in Islamabad amil baba in...
 
Money Forward Integrated Report “Forward Map” 2024
Money Forward Integrated Report “Forward Map” 2024Money Forward Integrated Report “Forward Map” 2024
Money Forward Integrated Report “Forward Map” 2024
 
NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...
NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...
NO1 Certified Black Magic Removal in Uk kala jadu Specialist kala jadu for Lo...
 
Global Economic Outlook, 2024 - Scholaride Consulting
Global Economic Outlook, 2024 - Scholaride ConsultingGlobal Economic Outlook, 2024 - Scholaride Consulting
Global Economic Outlook, 2024 - Scholaride Consulting
 
Guard Your Investments- Corporate Defaults Alarm.pdf
Guard Your Investments- Corporate Defaults Alarm.pdfGuard Your Investments- Corporate Defaults Alarm.pdf
Guard Your Investments- Corporate Defaults Alarm.pdf
 
Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...
Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...
Uae-NO1 Pakistani Amil Baba Real Amil baba In Pakistan Najoomi Baba in Pakist...
 
Economic Risk Factor Update: April 2024 [SlideShare]
Economic Risk Factor Update: April 2024 [SlideShare]Economic Risk Factor Update: April 2024 [SlideShare]
Economic Risk Factor Update: April 2024 [SlideShare]
 
『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书
『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书
『澳洲文凭』买科廷大学毕业证书成绩单办理澳洲Curtin文凭学位证书
 
Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...
Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...
Uae-NO1 Black Magic Specialist In Lahore Black magic In Pakistan Kala Ilam Ex...
 
Financial Preparation for Millennia.pptx
Financial Preparation for Millennia.pptxFinancial Preparation for Millennia.pptx
Financial Preparation for Millennia.pptx
 
The Inspirational Story of Julio Herrera Velutini - Global Finance Leader
The Inspirational Story of Julio Herrera Velutini - Global Finance LeaderThe Inspirational Story of Julio Herrera Velutini - Global Finance Leader
The Inspirational Story of Julio Herrera Velutini - Global Finance Leader
 
AnyConv.com__FSS Advance Retail & Distribution - 15.06.17.ppt
AnyConv.com__FSS Advance Retail & Distribution - 15.06.17.pptAnyConv.com__FSS Advance Retail & Distribution - 15.06.17.ppt
AnyConv.com__FSS Advance Retail & Distribution - 15.06.17.ppt
 
NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...
NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...
NO1 Certified Black Magic Specialist Expert In Bahawalpur, Sargodha, Sialkot,...
 
NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...
NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...
NO1 Certified Amil Baba In Lahore Kala Jadu In Lahore Best Amil In Lahore Ami...
 
cost of capital questions financial management
cost of capital questions financial managementcost of capital questions financial management
cost of capital questions financial management
 
Overview of Inkel Unlisted Shares Price.
Overview of Inkel Unlisted Shares Price.Overview of Inkel Unlisted Shares Price.
Overview of Inkel Unlisted Shares Price.
 
Banking: Commercial and Central Banking.pptx
Banking: Commercial and Central Banking.pptxBanking: Commercial and Central Banking.pptx
Banking: Commercial and Central Banking.pptx
 
Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...
Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...
Uae-NO1 Kala Jadu specialist Expert in Pakistan kala ilam specialist Expert i...
 
PMFBY , Pradhan Mantri Fasal bima yojna
PMFBY , Pradhan Mantri  Fasal bima yojnaPMFBY , Pradhan Mantri  Fasal bima yojna
PMFBY , Pradhan Mantri Fasal bima yojna
 
Unit 4.1 financial markets operations .pdf
Unit 4.1 financial markets operations .pdfUnit 4.1 financial markets operations .pdf
Unit 4.1 financial markets operations .pdf
 

Software Innovations Transforming the US Economy

  • 1. 1 How Software is Transforming the U.S. Economy Software Innovations and the Perpetual Development of New Software Robert B. Cohen, Economic Strategy Institute, July 5, 2017 Final Report submitted to the Ewing Marion Kauffman Foundation for a Grant to Support “Research on the Technology Changes that Constitute the “New Internet Protocol” and What it can do for Entrepreneurial Growth and Job Creation”1 Executive Summary This report builds upon ESI’s earlier efforts2 to forecast the potential impact of digital transformation on the U.S. economy. In this earlier study, we employed a matrix-based approach to estimate that cloud services and data analysis will add $1.2 trillion and $1.8 trillion to US GDP from 2015 to 2025.The shift to software-defined cloud services, as we broadly defined the emerging digital infrastructure, will create between 514,000 and 926,000 net new jobs between 2015 and 2025. Including multiplier effects, cloud services could create 1.5 million to nearly 3 million new jobs. This report examines the software innovations that have occurred in recent years. In doing so, it hopes to disprove the contentious hypothesis some economists have proposed, that the Internet Revolution ended in 2004. Based upon this hypothesis, these economists argue that there is little chance of Internet-related innovation in the future. This is especially true, they argue, for industries that are not part of the information and communications technology sectors in the economy. In this essay, we describe the fundamental software innovations that support a range of new software processes and techniques. Many of these are derived from Internet technologies and act as the foundation for innovations that have disrupted the way software is developed and deployed. Our examination of the core innovations focuses on new software processes and techniques such as the virtualization of computing, storage and networking resources and the adoption of software-defined infrastructure, such as software-defined data centers. These changes have been complemented by improvements in software development processes; the most important of which are the widespread use of Open Source software and software enhancements through GitHub. The latter serves as a prominent location to exchange Open Source software and to refine and extend existing programs into new applications. GitHub has changed the dynamics of software innovation, making it possible for many firms like Facebook, JPMorganChase, PayPal, eBay, Comcast and Yahoo to develop new software quickly and at low cost. After this we identify software innovations that include: the perpetual software development model and continuous delivery of software; DevOps; microservices; serverless computing and application protocol interfaces (APIs); containers and Docker; and software-defined data centers, Big Data and data analytics. 1 This research was funded by the Ewing Marion Kauffman Foundation. The contents of this publication are solely the responsibility of the Grantee. 2 Robert B. Cohen, “How Digital Transformation, the Move to the New IP, will Impact the US Economy and Employment and Broad Implications for the Vendor Community: The Emergence of a “Megadigital” Economy,” Economic Strategy Institute, October 13, 2016. .
  • 2. 2 Economists resist recognizing that the software innovations we describe might improve productivity and GDP growth. If the benefits of the software development and new processes were measured more accurately in U.S. economic statistics, it could promote efforts to improve how we measure innovation in the digital economy. Today, economists rely upon price and performance data for cellphones, laptops and tablets. They assume this is a reliable yardstick to gauge the impact of new software. Businesses adopt software innovations to accelerate their growth. This implementation counters thinking by some economists that the Internet and Internet technologies are unlikely to have much impact on productivity and GDP growth. It also counters expectations that productivity gains are likely to be meager for the foreseeable future. Another assumption held by economists is that co-innovation, the creation of economic value, especially new software, requires brainpower and experimentation and is a slow and difficult process. We assert that processes such as the enhancement of software creation through GitHub, the use of Open Source software, APIs and the new, perpetual software development model undercut this contention. We argue that firms outside of the information and communications technology industries operate in a world where software innovations are adopted quickly, suggesting a shift in the importance of software in their operations. We also draw upon thinking by Paul David who pointed to the parallels between the decades-long adoption of electricity and the long gestation period required for software and computing innovations to have a large-scale impact. The first part of this essay describes software innovations and the perpetual software development model. Then, we address whether software will have an impact on productivity as well as why it has not had a measurable impact on productivity. After this, we propose that economists should consider that software – in a broad sense, cloud computing, cloud services and software-defined infrastructure – is a General-Purpose Technology. By conceiving of software in this way, we underscore its potential to be a ubiquitous technology. The last part of the essay explores the work of economist Robert Gordon who asserts that innovations based on Internet-related technologies are unlikely to have much effect after the Internet boom from the mid-1990s to 2004. It also analyzes a recent paper by Tim Bresnahan and Pai‐Ling Yin.3 This paper argues that innovation in information and communications technologies is extremely difficult to spread into other industries because of the difficulty of applying new innovations to these sectors. As a result, its authors argue that the overall impact is to retard innovation and the growth of productivity in the U.S. economy. A. Software and the Digital Transformation Software has become a prominent factor in firms’ operations. Marc Andreessen’s has asserted that “software is eating the world” or “software is programming the world.” These expressions encapsulate his belief that “six decades into the computer revolution, four decades since the invention of the microprocessor, and two decades into the rise of the modern Internet, all of the technology required to 3 Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf to appear in Shane Greenstein, Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17.
  • 3. 3 transform industries through software finally works and can be widely delivered at global scale.”4 (my emphasis) We explore Andreesen’s thinking by describing how software innovations have changed how firms use information technology. We cite the great value firms obtain from new software tools and how software- defined infrastructure facilitates this success. This leads to a review of new approaches to software development. The innovations we describe below, in our opinion, strongly contradict Robert Gordon’s contention that Internet innovations have been “focused on the entertainment and information and communications technology.”5 In fact, software is transforming a broad swath of industries. It is having a far greater impact than many economists believe.6 In 2017, software development is very different from what was common from 2005 to 2010. 1. Virtualization and Software-Defined Infrastructure: Innovations that transformed the Way Software is Developed and Used, and How Data Centers and Networks Operate Software benefits from two innovative abstractions, virtualization and software defined infrastructure. These have heightened the role software plays in business. Software has disrupted the computing and storage hardware world and turned proprietary products into commodities. Virtualization is a way to create virtual rather than actual versions of parts within the computing, storage, and networking ecosystem. Through virtualization, software creates virtual machines on a single physical server or computer that hosts the virtual machines; the software in this case is called a hypervisor. Virtualization can also take place with storage devices and with network resources. When hardware is virtualized, software creates a virtual machine (Figure 1) that operates like a real computer with an operating system. Before virtualization became widespread, many firms used physical computers to run a single application. As a result, many tasks or applications drew upon 10 percent or 20 percent of a computer’s capacity. With virtualization, the entire resources of a computer can be applied to different tasks at the same time, so virtualized machines operate at full capacity. 4 Marc Andreessen, “Why Software Is Eating the World,” Wall Street Journal, August 20, 2011. http://www.wsj.com/articles/SB10001424053111903480904576512250915629460. Also see and Marc Andreessen, Ben Horowitz, Scott Kupor, and Sonal Chokshi, “a16z Podcast: Software Programs the World,” July 10, 2016. http://a16z.com/2016/07/10/software-programs-the-world/ 5 Robert Gordon, The Rise and Fall of American Growth, Princeton University Press, 2016, p. 567. 6 Erich H. Strassner, “Measuring the Digital Economy,” BEA Advisory Committee, November 16, 2016. https://bea.gov/about/pdf/Measuring%20the%20Digital%20Economy.pdf
  • 4. 4 Figure 1. Traditional and Virtual Architecture Source: Margaret Rouse and Brian Kirsch, “Definition: Virtualization,” TechTarget, http://searchservervirtualization.techtarget.com/definition/virtualization Virtualization also helps computing resources to scale. One example is Amazon Web Services’ virtualized computing infrastructure. By using the public clouds Amazon supports, developers can rent tens of thousands of servers for a short period of time to evaluate new versions of software at a realistic scale. Firms have changed the process of software development. The innovative processes they employ build upon enhancements in software-defined infrastructure.7 They also exploit additional innovations such as containers.8 7 “Key Attributes of a Software-Defined Infrastructure,” SDx central, July7, 2015. https://www.sdxcentral.com/articles/editorial/key-attributes-software-defined-infrastructure/2015/07/ 8 Reza Roodsari, “Docker, Microservices And Kubernetes,” Mirantis Open Stack Training, December 22, 2016, p. 8. https://content.mirantis.com/rs/451-RBY-185/images/mirantis-kubernetes-docker-mini-bootcamp_slides.pdf
  • 5. 5 Figure 2. Software Defined Infrastructure Source: Anjanyea “Ruddy” Chagam and Shayne Huddleston, “Software Defined Storage – Open Framework and Intel Architecture Technologies.” Intel talk at IDF 14, September 9, 2014. https://www.slideshare.net/LarryCover/software-defined-storage-open- framework-and-intel-architecture-technologies A Software Defined Data Center (SSDDC) (Figure 3) builds on virtualization to improve the operation of data storage. An SDDC is “a data storage facility in which all infrastructure elements -- networking, storage, CPU and security -- are virtualized and delivered as a service. Deployment, operation, provisioning and configuration are abstracted from hardware. Those tasks are implemented through software intelligence.”9 9 Margaret Rouse, “Definition: SDDC (software-defined data center),” TechTarget, http://searchconvergedinfrastructure.techtarget.com/definition/software-defined-data-center-SDDC
  • 6. 6 Figure 3. Software-Defined Data Center Source: Enrico Boverino, “Navigating Your Approach to an SDDC,” VMware Advisory Services blog. https://blogs.vmware.com/accelerate/tag/software-defined-data-center When software operates and manages infrastructure, high hardware costs are reduced. Virtualized data centers use “commodity” hardware that is much lower cost than vendor-provided equipment. These data centers also rely on overlays that transmit data entirely via software. Software-defined data centers operate with virtualized firewalls and load balancers that reduce their physical infrastructure. This speeds data from one part of a data center to another (Figure 4). Software for firewalls and load balancing is usually included in the overlays that are part of the logical switching in data centers Figure 4. Logical Switching via the Use of Overlays. Source: Mora Gozani, Network Virtualization for Dummies, VMware Special Edition. John Wiley & Sons, Inc., 2016, p. 26. https://www.linkedin.com/pulse/ready-crash-course-network-virtualization-mora-gozani
  • 7. 7 Running several operating systems on a single central processing unit (CPU) can also reduce overhead costs. With a virtualized enterprise, firms can perform updates to the operating system and applications without disrupting a user. 2. The Importance of GitHub to Software Development and the Role of Open Source Software GitHub is an innovation that disrupts the traditional pattern of software development. It challenges the usefulness of economists’10 co-invention model to conceive of the development of new software and how rapidly it will contribute to higher productivity outside of the ICT industries, i.e., in non-vendor firms. GitHub creates a way to share software that has been developed largely by relying upon Open Source software. Git is a distributed version control system. It lets developers have a complete history of a project or a software program. This log of how the software was developed includes all the information provided on the server it was taken from. As a result, developers working on software enhancements don’t have a single point of failure. Most importantly, GitHub also serves as a place (a “hub”) where developers can find new software or a project to prepare a specific service or function, largely using Open Source software placed in a repository or “repo.” Developers first “clone” or “pull,” i.e., copy, an earlier version of software from a “repo” to a development server, or Linode. Then, they make changes to the code or project, and “push,” or forward, these “forked,” or modified, projects to a developer’s GitHub user’s account. The final stage in the process is the “pull request.” This asks the original “repo” project, or developer, to accept the revisions or sample files the new developer created.11 “Pull requests” are a way to estimate the number of times developers have created new versions of software. 10 Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf in Shane Greenstein, Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17. 11 “Why You Should Switch from Subversion to Git,” Team Treehouse blog, August 7, 2012. http://blog.teamtreehouse.com/why-you-should-switch-from-subversion-to-git
  • 8. 8 Figure 5. Git with GitHub Workflows. Source: “Why You Should Switch from Subversion to Git,” Team Treehouse blog, August 7, 2012. http://blog.teamtreehouse.com/why-you-should-switch-from-subversion-to-git GitHub serves largely as a business software modification location, where businesses to improve their own software. In 2016, 44 percent of the Fortune 5000 companies and half of the Fortune 10 firms used GitHub. Facebook had the second largest number of contributors to Open Source after Microsoft.12 In 2016, nearly 85 percent13 of all “pull requests,” or requests to approve changes (Figure 5) sent to the original programmer, came from within an organization (Figure 6); the rest came from user-owned repositories. Figure 6. GitHub Pull Requests Source: “The State of the Octoverse 2016.” https://octoverse.github.com/ 12 Matt Weinberger ,“Microsoft just edged out Facebook and proved that it's changed in an important way,” Business Insider, September 14, 2016. http://www.businessinsider.com/microsoft-github-open-source-2016-9 13
  • 9. 9 In 2016, enterprises in industries outside of the software and the Internet plus communications sectors accounted for more than two-thirds of GitHub’s enterprise customers (Table 1). Table 1. Industrial Distribution of GitHub Enterprise Users Source: “The State of the Octoverse 2016.” https://octoverse.github.com/ Open Source “is the new norm for software development.”14 The availability of Open Source software has helped many industries and businesses benefit from the shared innovation it empowers. Open Source software’s original source code is freely available on web-accessible locations such as GitHub. Free distribution and modification relies upon having a permissive license in place, such as Apache 2.0. By using these components, software developers, core developers, write Open Source code and other programmers refine the code that is written and identify the flaws in it. The availability of Open Source software lets businesses avoid paying licensing costs that vendors usually apply to their software products. In this way, Open Source lowers development costs. Open Source software is often more reliable than vendor-developed software. Large numbers of programmers are involved in its creation. Open Source software frees firms from their previous dependence on vendors that built applications. Firms, such as JPMorganChase use a great deal of Open Source software. They also contribute software to GitHub.15 Microservices also contribute to changing the dependence on vendors for important applications. They simplify the work that firms need to do to create new software in the public cloud and add to Open Source software’s benefits. This reduces costs.16 14 Michael Dolan, “Commercial Dependencies and Sustainable Open Source Ecosystems,” CapitalOne DevExchange blog, June 16, 2017. https://developer.capitalone.com/blog-post/commercial-dependencies-and-sustainable- open-source-ecosystems/ 15 Lori Beer, CIO of JPMorganChase, presentation on “Innovation at Scale in the Corporate and Investment Bank,” WatersUSA2016, New York, December 5, 2016. 16 “7 Main Advantages and Disadvantages of Open Source Software,” ConnectUS, http://connectusfund.org/7- main-advantages-and-disadvantages-of-open-source-software Industrial Distribution of GitHub Enterprise Customers Industry Percent of GitHub Users Software and Internet 26 Business Services 15 Education 8 Manufacturing 8 Healthcare 6 Media and Entertainment 6 Retail 6 Telecommunications 6 Consumer Services 5
  • 10. 10 Table 2. The Differences between Proprietary and Open Source Software Source: Kak Yong, “Chapter 2: Computer Systems and Open Source Software,” April 16, 2012. https://www.slideshare.net/makyong1/proprietary-and-open-source-software 3. The Perpetual Software Development Model Software development now proceeds in ways that are quite different what was common from 2005 to 2010. Developers working on new software, such as Linux, have made continuous refinements to the original code. Figure 7 illustrates how changes, in “commits per month,” have been added to the Linux operating system over a twenty-year period. Figure 7. Commits per month in the Linux Source Code Management Repository, 1991-2011
  • 11. 11 Source: Jesus M. Gonzalez-Baharona et. al. “Studying the laws of software evolution in a long-lived FLOSS project,” Journal of Software, July 26, 2014, https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4375964/ At Facebook, as at many other “Internet Firms,” engineers write code in a “perpetual development mode, in which engineers continuously develop new features and make them available to users. Consequently, the [coding] system grows continuously, possibly at a super-linear rate.”17 Facebook has adopted a new model to create software. It dramatically reduces the time to deploy new applications. Figure 8 illustrates the rapid progression to the Facebook model of delivering new software. The old “waterfall” that required one year has been superseded by Facebook’s model of making new software available once a day, before 2012. This has fallen to less than one hour, during the four years from 2012 to 2016. This new model is followed by at least 200 firms18 today. We have estimated that these firms are in industries that accounted for about a third of U.S. GDP in 2016.19 Figure 8. Timescales in Making new software available. The Waterfall Process required one full year. Today continuous deployment takes much less than an hour. Source: Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at Facebook,” p. 3. https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf During Facebook’s growth (Figure 9), the number of software developers grew rapidly. The number of “commits,” committing code to a versioning system, expanded about 10-fold over five years and the size of Facebook’s codebase. In 2012, At Facebook’s “engineers commit[ted] code to the version control system up to 500 times a day, recording changes in some 3,000 files.”20 In addition, Facebook’s lines of code grew rapidly from 2005 to 2012, reaching an estimated 10.5 million lines of code.21 In 2016, one report showed 60,000 commits, about 5000 per month.22 An earlier 2014 discussion estimated commits at “thousands per week,” probably closer to 10,000 to 15,000 a month. It also noted that in 2014, Facebook’s main source repository – its codebase size -- was much larger than the Linux kernel “which 17 Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at Facebook,” p.2. https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf 18 Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.” https://puppet.com/resources/white-paper/2016-state-of-devops-report 19 Robert B. Cohen, “The Economic Impact of Cloud Services,” Economic Strategy Institute, June 12, 2017 20 Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at Facebook,” https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf 21 Feitelson, Frachtenberg, and Beck, p. 3. 22 Christine Abernathy, “Facebook Open Source 2016 year in review,” December 19, 2016. https://code.facebook.com/posts/1058188987642144/facebook-open-source-2016-year-in-review/
  • 12. 12 checked in at 17 million lines of code and 44,000 files in 2013;”23 this seems to mean that Facebook had a codebase size of about 68 to 136 million lines of code.24 That indicates a six-fold to thirteen-fold increase in the codebase size in just two years’ time. Figure 9. Facebook’s Increase in Developers, Coding Commits Per Month and Codebase Size, 2005 to 2012 Source: Dror G. Feitelson, Eitan Frachtenberg, and Kent L. Beck, “Development and Deployment at Facebook,” p. 2. https://pdfs.semanticscholar.org/566c/3ad271fcea439a4dfcc5b7aa388f6021d110.pdf 4. Continuous Service Delivery and Continuous Integration Continuous delivery or continuous deployment (CD) describes how teams produce software in short cycles. Since the software is tested continuously, it can be confidently released at any time in the cycle. This accelerates the building, testing and release of new software. CD or continuous service delivery relies upon cross-disciplinary teams to program, deploy and test new software (see Adrian Cockcroft’s diagrams below, Figures 10 and 11). In continuous service delivery,25 integrated design teams replace siloed, or isolated, “skill areas,” that were widely employed to create “monolithic” software. These areas included quality assurance, systems administration and development. This reduced the number of steps required to write and test software. Continuous service delivery lets firms respond to demands from markets and customers. By creating software in this way, firms not only improve their knowledge of customers and markets, but also quickly exploit new opportunities for sales. Lori Beer, the CIO of JPMorganChase, noted 26 that her bank achieved significant savings by moving to the cloud and creating a more software-based infrastructure. Nevertheless, the real benefits of software innovations came from the productivity gains when it expanded into new business areas without needing additional resources. JPMorganChase expanded its revenues by being able to analyze markets rapidly and create new software to address them. 23 Durham Goode and Siddharth P Agarwal, “Scaling Mercurial at Facebook,” https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ 24 Goode and Agarwal. 25 Adrian Cockcroft, “Creating Business Value with Cloud Infrastructure,” Open Networking User Group, Columbia University, May 13-14, 2015. 26 Lori Beer, discussion with author after presentation on “Innovation at Scale in the Corporate and Investment Bank,” WatersUSA2016, New York, December 5, 2016.
  • 13. 13 When News Corp27 migrated to the cloud, its “business case was to migrate 75% of our infrastructure to the cloud over 3 years to achieve $100M in annual savings.” These efforts relied upon making applications “cloud-ready.”28 Continuous integration (CI) is the process developers employ to integrate code they are writing into a shared repository many times a day. By doing this, developers can check whether new code they have written is problem-free as it is being created. If they find the new code is error-free, their work can proceed to an automated build of a new software application. Many organizations achieve CD by connecting CI to an automated infrastructure, such as the cloud or software-defined infrastructure. Consequently, when firms develop applications in this new way, they reduce the time to business value (Figure 10). Today, firms create software applications and deploy them using different processes than the highly-segregated steps they used earlier (see Cockcroft’s chart in Figure 11). The new process eliminates time-consuming, step-by-step approvals by integrating the software development and testing cycles. Figure 10. Business Value Delivery – Software Changes from Monoliths to Functions: Log Time in Hours Shown with a Reversed Scale Sources: Draws upon Adrian Cockcroft, "Evolution of Business Logic from Monoliths through Microservices to Functions," Cloud Guru,https://read.acloud.guru/evolution-of-business-logic-from-monoliths-through-microservices-to-functions- ff464b95a44d#.r8eel3vze and Stephen Orban, "Transitioning to DevOps and the Cloud," https://medium.com/aws-enterprise- collection/transitioning-to-devops-and-the-cloud-9488ddaf862f#.r7y6krhq5 27 Stephen Orban, “Always Be Reinventing & The Cloud of Youth,” Medium, January 265, 2017. https://medium.com/aws-enterprise-collection/always-be-reinventing-the-cloud-of-youth- 137990b9d229#.29kuffhln 28 “Cloud-ready” applications have been reprogrammed from when they were used in traditional computing environments so they can run in virtualized environments using cloud computing.
  • 14. 14 More traditional, “low-performing”29 IT organizations continue to rely upon the more cumbersome and time-consuming “siloed,” or monolithic, approach to developing software. When they take this approach to software development, each stage in software development is isolated from every other one. Figure 11 describes the main stages in this process. Figure 11. Continuous Service Delivery: The Product Delivery Process with Monolithic Software and Microservices – with the Latter Supporting a Reorganization to DevOps Source: Adrian Cockcroft, “Creating Business Value with Cloud Infrastructure,” Open Networking Users Group Conference, Columbia University, New York City, May 13-14, 2015. As noted above, the shift to CD is built upon changes in software-defined infrastructure and new process innovations. This results in an enormous reduction in the time required to develop new software as well as to test, evaluate and deploy it in “real-life” situations. It also means that there is continuous learning 29 This characterization is drawn from the studies of DevOps use in enterprises. See Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.” https://puppet.com/resources/white-paper/2016- state-of-devops-report
  • 15. 15 about the new code, or software, that is being written, so that mistakes are identified before they are introduced to new designs. The innovations below demonstrate the large gap between CD and the assumptions of the co-innovation of new ICT discoveries. Some contemporary economists30 rely upon the co-invention framework to assess changes in the ICT industries. We believe that this approach overlooks recent innovations in software development and is mistaken in believing the software development process remains tedious and complex. We base our conclusion on the recent changes in software development. 5. DevOps DevOps is an integral part of continuous service delivery (see Figure 11). It was the innovative model for software development that provided firms with a way to accelerate deployment by optimizing the way that development and operations collaborated. In DevOps, Ops (operations) groups let developers manage the “operational characteristics” of applications they are building. In DevOps, a “Shift Left” is how developers describe these changes in software development. The focus moves to developers, but requires them to consider how effectively new software contributes to an enterprise’s operations. Figure 12. DevOps and the “Shift Left” in Building Applications-Aware Environments Source: Sanjeev Sharma, “Adopting DevOps – Part III: Aligning the Dev and Ops Teams,” May 9, 2013, https://sdarchitect.wordpress.com/2013/04/12/adopting-devops-part-iii-aligning-the-dev-and-ops-teams/ DevOps supersedes the old, “Waterfall Model,” of programming where every stage waited for the previous one to be completed (Figures 13 and 14). This meant creating “flattened” product teams for DevOps that included people with expertise in most of the stages of software development, testing and distribution (Figure 11). 30 Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf to appear in Shane Greenstein, Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17. We discuss this analysis later in this essay.
  • 16. 16 Figure 13. Issues with the Waterfall Model of Software Development and How the Challenges were overcome Source: Vineet Chaturvedi, “DevOps Tutorial: Introduction to DevOps” edureka, Oct 18, 2016. https://www.edureka.co/blog/devops-tutorial Figure 14. Proposed Solutions to the Challenges of the Waterfall Model of Software Development Source: Vineet Chaturvedi, “DevOps Tutorial: Introduction to DevOps” edureka, Oct 18, 2016. https://www.edureka.co/blog/devops-tutorial
  • 17. 17 Figures 15 and 16. How DevOps Solutions address Developmental and Operations Challenges Source: Vineet Chaturvedi, “DevOps Tutorial: Introduction to DevOps” edureka, Oct 18, 2016. https://www.edureka.co/blog/devops-tutorial Today, many enterprises depend on software and data analytics to operate and create new strategies. As noted above, this represents a major change. Software is now a major factor, if not the most important factor determining corporate competitiveness. Innovations in software, i.e., “achieving higher IT and organizational performance is a team effort spanning development and operations.”31 Recent surveys have analyzed how firms are using DevOps. Over the past year, “high performing” firms32 improved their performance over “low performing” ones. The ability to deploy new code is one way to measure difference. Between 2015 and 2016, higher-performing firms increased their ability to deploy software from 200 deployments per year to 1460 deployments per year (Figure 17). This occurred while low-performing firms maintained their level of about 12 deployments per year.33 31 Puppet Labs and DevOps Research and Assessment, “2016 State of DevOps Report,” p.4 32 For a definition of high performing and low performing, see the Appendix. 33 Puppet and Dora, p. 18.
  • 18. 18 Figure 17. Deployment Frequency for “High Performing” Firms in Number of Deploys per Year Source: Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.” https://puppet.com/resources/white-paper/2016-state-of-devops-report, p. 18. Nicole Forsgren, one of the analysts34 who identified the gap between the two groups of firms (Table 3) noted that they were due to high performers’ greater sophistication in developing and deploying software. The “high performers” spent far more time on new work, delivering more value to the business. This indicates that they had streamlined their software development, testing and deployment skills. In fact, they got code, or software, out faster, at 200 times the rate of low performing firms. They also deployed more stable applications, having 24 times faster recovery from failure. We expect that this performance difference will pressure additional firms to improve their software delivery, agility and reliability. They are likely to move in this direction to realize improvements in their delivery of content, the value obtained from A/B testing, the value from speed to market, and to benefit from the compliance and regulatory benefits. 34 Nicole Forsgren, “The Data on DevOps,” Devopsdays Minneapolis 2016, https://www.youtube.com/watch?v=Z6IjVf2dcKM. The author thanks Dr. Forsgren for her comments on these paragraphs.
  • 19. 19 Table 3. 2016 IT Performance by Cluster for DevOps Users Source: Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.” https://puppet.com/resources/white-paper/2016-state-of-devops-report 6. Microservices Microservices applications have an architecture where each functional element is a separate service. As a result, firms can use microservices (Figure 18) and reuse parts of an application that do not change. This increases the speed35 at which firms can deliver applications and services; i.e., “high velocity software development.” This makes microservices an important part of continuous service delivery. Microservices divide applications into smaller, composable pieces, very much like Lego blocks. So microservices’ components are “easier to build and maintain.” Each component, “is developed separately. An application is the sum of its “constituent components.””36 Previously, programmers had to write a software application as a single, unified, or “monolithic” product. Microservices can be easily “glued together.” Each one contains “an [Application Protocol Interface or] API endpoint.” APIs are tools, protocols and routines that are used to develop software. APIs “specify how 35 Richard Li, “Microservices Essentials for Executives: The Key to High Velocity Software Development,” for Entrepreneurs from David Skok. http://www.forentrepreneurs.com/microservices/ 36 What are microservices? Opensource.com, https://opensource.com/resources/what-are-microservices
  • 20. 20 different software components should interact.”37 The API interface can be accessed like a “standard webpage,”38 making it easier for developers to use microservices. Figure 18. The Difference between Microservices architecture and Monolithic Applications Source: James Lewis and Martin Fowler, “Microservices,” March 25, 2014. https://martinfowler.com/articles/microservices.html. With developers focusing on code, software engineers and developers report “as we move towards software-defined environments, we [can] build, version [or model] and manage complex environments, all as code.”39 7. “Serverless” Computing and Applications Protocol Interfaces (APIs) Serverless computing is called “serverless” because developers no longer need to manage the physical or virtual servers and data they need40 to code. This improves developers’ efficiency. Through the public cloud, they can access resources from a cloud service provider – Amazon Web Services, IBM or Google. “Serverless systems allow developers to build complex systems much more quickly and ensures that they are spending most of their time focusing on core business issues rather than infrastructure based and 37 “What is API - Application Program Interface?” Webopedia www.webopedia.com/TERM/A/API.html 38 “What are microservices?” 39 Sanjeev Sharma, “Adopting DevOps – Part III: Aligning the Dev and Ops Teams,” May 9, 2013, https://sdarchitect.wordpress.com/2013/04/12/adopting-devops-part-iii-aligning-the-dev-and-ops-teams/ 40 David Ward of Cisco has described a “Policy Engine” that will manage computing, storage and network infrastructure in future virtualized environments. David Ward, “Networking: The Logical Micro-Service Infrastructure,” Open Networking Summit, April 5, 2017. http://sched.co/9kxu
  • 21. 21 administrative duties.”41 (my emphasis) Serverless systems such as Amazon’s Lambda or IBM’s OpenWhisk can scale, grow and evolve without developers or solution architects having to patch a web server. With serverless computing, developers submit functions42 for execution (Table 4). They provide a function code to a cloud service provider offering serverless computing and the computing program, for instance, Amazon’s Lambda, executes it. This is possible because an API gateway eliminates “traffic management, authentication and authorization, monitoring and API versioning [by converting them] into easily configurable steps.”43 41 Scott Maurice, “What does “serverless computing” really mean?” http://scottmaurice.com/what-does- serverless-computing-really-mean/ 42 Sam Kroonenberg, “The Next Layer of Abstraction in Cloud Computing is Serverless,” A Cloud Guru, May 19, 2016. https://read.acloud.guru/iaas-paas-serverless-the-next-big-deal-in-cloud-computing- 34b8198c98a2#.9877q5ouf 43 Amazon, “Job posting for Senior API Software Engineer-Amazon API Gateway,” October 20, 2016. https://us- amazon.icims.com/jobs/452236/senior-api-software-engineer-amazon-api- gateway/job?iis=Job+Posting&iisn=Indeed+%28Paid+Sponsored+Posting%29&mobile=false&width=1027&height= 1200&bga=true&needsRedirect=false&jan1offset=-300&jun1offset=-240
  • 22. 22 Table 4. Stages in the Evolution of Business Logic from Monoliths to Microservices to Functions Source: Adrian Cockcroft, "Evolution of Business Logic from Monoliths through Microservices to Functions," Cloud Guru, https://read.acloud.guru/evolution-of-business-logic-from-monoliths-through-microservices-to-functions- f464b95a44d#.r8eel3vze and Stephen Orban, "Transitioning to DevOps and the Cloud," Medium.com, https://medium.com/aws- enterprise-collection/transitioning-to-devops-and-the-cloud-9488ddaf862f#.r7y6krhq5 At the heart of this new process is the “API Gateway, … the front-door of the Serverless revolution, an approach that lets customers turn business logic and application code into scalable, fault-tolerant production systems without requiring every developer to become an expert in distributed systems, deployment technologies, and infrastructure management.”44 44 Amazon, “Job posting.”
  • 23. 23 Figure 19. An API Gateway, Using Oracle’s API Gateway as an Example Source: Aaron Dolan, “Your API’s First Line of Defense: Oracle API Gateway,” AVIO Consulting, October 29, 2014. http://www.avioconsulting.com/blog/your-apis-first-line-defense-oracle-api-gateway “Serverless computing” is a process change that simplifies developers’ work. It eliminates tasks that were previously required when using public cloud services. Developers had to reserve virtual server time and learn how to manage traffic, authentication and authorization. This reduces the time needed to deploy new software.
  • 24. 24 Figure 20. A Brief History of Cloud: Serverless Computing and the Evolution of Cloud Services Source: Sam Kroonenberg, “The Next Layer of Abstraction in Cloud Computing is Serverless,” Cloud Guru, May 19, 2016. https://read.acloud.guru/iaas-paas-serverless-the-next-big-deal-in-cloud-computing-34b8198c98a2 8. Containers and Docker Facilitate the Creation and Deployment of New Software Containers (Figures 21 and 22) build upon operating system-level virtualization. They are an innovative, interoperable format for applications, i.e., software, that can be wrapped with a full system needed to run the software. So “containers wrap-up an application in a self-contained filesystem … that includes everything the [application] needs to run independently: binaries, runtime libraries, system tools, system packages, etc. This level of simplification and compartmentalization allows applications to be spun up [or launched] much faster than before.”45 while ensuring consistent and predictive up time. 45 Scott Willson, “Webcast: Containerology – DevOps, Docker and Microservices in a Continuous Delivery World,” https://offers.automic.com/ppc/containerology-devops-docker-and-microservices-in-a-continuous-delivery-world- webcast- ppc?network=g&campaignid=646348797&adgroupid=27262905662&keyword=docker%20technology&matchtype =p&creative=94949079182&gclid=CjwKEAiAj7TCBRCp2Z22ue- zrj4SJACG7SBE5G2uN07hlUU24bMYRldAYm2tp5Yxrhqb1bG5XVNqwBoCSf7w_wcB
  • 25. 25 Figure 21. Comparing Containers to Virtual Machines (VMs) Source: “Docker, Containers, and the Future of Application Delivery,” OSCON 2013. http://www.slideshare.net/dotCloud/why-docker2bisv4 Figure 22. Why are Docker containers Lightweight? -- Applications on Virtual Machines (VMs) and Containers Source: “Docker, Containers, and the Future of Application Delivery,” OSCON 2013. http://www.slideshare.net/dotCloud/why- docker2bisv4 Containers represent a fundamental change in how workloads and applications can be virtualized. Containers can scale more efficiently, operate faster and offer greater portability than hardware
  • 26. 26 virtualization. Eventually, they are expected to replace most instances where virtual machines are involved.46 An April-May 2016 DevOps.com and ClusterHQ survey47 found that 79 percent of respondents’ organizations were using container technologies. Of this group, 76 percent of the deployments were running in production environments, not experimental ones. This was a significant increase over 2015, when only 38 percent of respondents had containers in production ecosystems. The report concluded that container adoption was driven by a desire to “increase developer efficiency (39 percent) and support microservices (36 percent).” Over two thirds of the survey’s respondents said their firms are realizing the results they expected from using containers. Bloomberg Inc. has adopted containers and software-defined networking over the last four years to add simplicity and high volume to its development of new applications and products. It has assembled a staff of 2500 developers and embraced the use of Open Stack. It notes that modern applications (software) have become ephemeral in nature, with developers using templated and automated images to write software. This has moved Bloomberg away from a model where applications development required complex policies. The move to software-defined networking has also let developers use microservices and micro-segmentation of applications.48 Google’s container was “a kind of virtualized, simplified OS [Operating System] which we used to power all of Google’s applications.” Initially, Google developed cgroups, 49 “a framework pattern that provides encapsulation and separation of concerns for the components that use them …. the container will provide mechanisms to address cross-cutting concerns like security or transaction management.…a container wraps the component.”50 Based on these benefits, • “a developer has in their laptop plenty of compute power to run multiple containers, making for easier and faster development” • “a single command” can push out a “new version of a container,” • With containers, it is much easier to compose “applications using open source software.” This means that developers can bring together many tools that might be complicated to set up individually, such Hadoop and MongoDB. Developers can use containers to deploy 46 Reza Roodsari, “Docker, Microservices And Kubernetes,” Mirantis Open Stack Training, December 22, 2016, p. 13. https://content.mirantis.com/rs/451-RBY-185/images/mirantis-kubernetes-docker-mini-bootcamp_slides.pdf 47 DevOps.com and ClusterHQ, Container Market Adoption:2016,” https://clusterhq.com/assets/pdfs/state-of- container-usage-jgune-2016.pdf. The survey queried 310 computer professionals regarding their firms’ container adoption and usage patterns. 48 Truman Boyes, “Open Cloud Infrastructure at Bloomberg,” Open Network Users Group, Columbia University, May 2015. 49 “cgroups,” Wikipedia. https://en.wikipedia.org/wiki/Cgroups 50 Edward Ost, “What Is a Container? (Container Architecture Series Part 1),” Talend blog, December 2, 2014. http://www.talend.com/blog/2014/12/02/what-is-a-container-container-architecture-series-part-1
  • 27. 27 numerous tools on a single computer. They can use these tools to improve the quality of the software that they program.51 Containers, in contrast to virtual machines, offer: • “Simple deployment: By packaging your application as a singularly addressable, registry- stored, one-command-line deployable component, a container radically simplifies the deployment of your app no matter where you’re deploying it. • Rapid availability: By abstracting just the OS [operating system] rather than the whole physical computer, this package can “boot” in ~1/20th of a second compared to a minute or so for a modern VM. • Leverage microservices: Containers allow developers and operators to further subdivide compute resources.”52 9. Software-Defined Data Centers, Big Data and Data Analytics McKinsey describes Big Data as “large pools of data that can be captured, communicated, aggregated, stored, and analyzed.” In 2011, McKinsey found that Big Data is “part of every sector and function of the global economy.”53 In 2013, ABI Research estimated that spending on Big Data was $31 billion and that this spending would increase to $114 billion in 2018, a compound growth rate of 29.6 percent.54 The Eckerson Group has concluded that if firms are going to capitalize on Big Data, they need to “fundamentally rethink the way they capture, store, govern, transform and analyze” it.55 As Figure 23 illustrates, firms upgraded their data center infrastructure through several stages in the time from 2000 to 2013. Once they implemented software-defined data centers, the time it took to obtain business value from data centers dropped by more than 1000 times. Firms that made this transition found they had greatly improved the usefulness of the analytic tools they applied to Big Data. 51 Quotes and section summarized from Miles Ward, “An introduction to containers, Kubernetes, and the trajectory of modern cloud computing,” Google Cloud Platform Blog, January 9, 2015. https://cloudplatform.googleblog.com/2015/01/in-coming-weeks-we-will-be-publishing.html 52 Miles Ward, “An introduction to containers, Kubernetes, and the trajectory of modern cloud computing,” Google Cloud Platform Blog, January 9, 2015. https://cloudplatform.googleblog.com/2015/01/in-coming-weeks-we-will- be-publishing.html 53 James Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global Institute, May 2011, p. iii. http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big- data-the-next-frontier-for-innovation 54 ABI Research, "Unlocking the Value of Big Data in Enterprises," cited in Joanne Herman, “Big Data-Related Investment to Hit $114bn in 2018,” Misco.co.uk blog, September 12, 2013. http://www.misco.co.uk/blog/news/01279/big-data-related-investment-to-hit-114-billion-dollars-in-2018 55 Phil Bowermaster and James W. Eckerson, “Selecting a Big Data Platform,” The Eckerson Group, December 2015. http://www.eckerson.com/register?content=selecting-a-big-data-platform-building-a-data-foundation-for-the- future
  • 28. 28 Figure 23. Business Value Delivery – Changes in the Time to Value in Data Centers from 2000 to 2013. Source: VMware Accelerate Advisory Services, "Delivering on the Promise of the Software-Defined Data Center," VMware Accelerate Advisory Services blog, 2013. https://blogs.vmware.com/vmtn/author/aluciani/page/3 Two different McKinsey Global Institute studies illustrate how much firms depend on data analytics. In its first Big Data study in 2011,56 McKinsey discovered that firms in nearly all U.S. sectors possessed at least 200 terabytes of stored data per company. Firms having more than 10,000 employees in 2009 attained a level where McKinsey believed they could capture real value from data analytics.57 . This report also estimated that there were 300,000 employees in data analytics (my own estimates58 are much greater; with about 15 million job postings between 2011 and 2017, allowing for duplication of job skills – repeating mention of data analysis in postings for other jobs – I would estimate there might be as many as million to 2.5 million data analysts today). McKinsey forecast that by 2018, there would be almost 50 percent more demand for these jobs; that demand would rise to between 440,000 to 480,000.59 McKinsey’s 2016 Big Data report concluded that “data is now a critical corporate asset.”60 While it found that data was doubling every three years, it also noted that since its earlier study, many firms had not 56 James Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global Institute, May 2011, p. 18. http://www.mckinsey.com/business-functions/digital-mckinsey/our-insights/big- data-the-next-frontier-for-innovation 57 Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global Institute, May 2011, p. 19. 58 I derive this from the figures on job postings in Robert B. Cohen, “Highly Stratified Occupations and the Digital Economy,” Economic Strategy Institute, June 26, 2017. Report to the Berggruen Foundation. 59 Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global Institute, May 2011, p. 104. 60 Nicolaus Henke, “The Age of Analytics: Competing in a data-driven world,” McKinsey Global Institute, December 2016, p. vi. 0.01 0.1 1 10 100 1000 10000 2000 2003 2013 Business Value Delivery - Changes in the Time to Value in Data Centers from 2000 to 2013: Log of Time in Hours with Reversed Scale VIRTUAL MACHINES: 6 SOFTWARE DEFINED DATA HARDWARE DEFINED DATA
  • 29. 29 taken advantage of the gains that it had forecast in 2011. Some observers61 believe that many firms had a difficult time improving the flow of analysis. They failed to create a framework of continuous learning by experimentation that refined the use of Big Data. Nonetheless, McKinsey asserts that a group of “analytics leaders are changing the nature of competition and consolidating big advantages,”62 by deploying and using Big Data. These firms include “Apple, Alphabet/Google, Amazon, Facebook, Microsoft, GE, and Alibaba Group.”63 One reason for this shift is that these firms are exploiting a difference between their level of performance after they have been able to evaluate Big Data and the performance of other firms that don’t rely on Big Data. In many cases, the more successful firms use data to provide better situational awareness. They also employ analytics to improve the situations they experience. McKinsey estimated that firms could obtain significant savings when they implemented data analytics. It calculated that when retailing uses data analytics “marketing levers can affect 10 to 30 percent of [the] operating margin; merchandising levers can affect 10 to 40 percent [of the operating margin]; and supply chain levers can have a 5 to 35 percent impact [on operating margin].”64 Nearly four-fifths of firms in another survey reported that data analysis had helped them institute new business processes, such as creating an Internet of Things.65 In the same survey, almost one-third or more of firms in the energy and utilities, automotive and retailing industries had adopted machine-to-machine (M2M) communications. These were part of the infrastructure to analyze Big Data. This included more than a quarter of firms in healthcare and consumer electronics industries as well as one-sixth or more of firms in manufacturing, transport and logistics.66 McKinsey noted that Big Data’s ability to contribute real value to a business depended upon the type of retail sector that used it. The retailing sectors that were early adopters of data analytics usually obtained the greatest benefits. General merchandise stores, building material and garden, electronics and appliances and health and personal care stores were forecast to have the greatest big data value potential. 61 The author thanks Chris Swan of CSC for his comments on why more firms did not take advantage of Big Data. 62 Henke, p. 5. 63 Henke, p. 64 James Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global Institute, May 2011, p. 71. 65 Erik Brenneis “Vodafone M2M Barometer 2015,” Vodafone Germany, p. 3. https://www.vodafone.de/media/downloads/press-releases/150729-vf-m2m-report-2015.pdf This is a survey of machine-to-machine (M2M) communications use. 66 Brenneis, p. 19.
  • 30. 30 Table 5. The Big Data Value Potential in Retail Varies in Different Subsectors Source: Manyika and others, “Big data: The next frontier for innovation, competition, and productivity,” McKinsey Global Institute, May 2011, p. 72. B. Software as a General-Purpose Technology (GPT) Economists who have studied the ascent of the US economy in the 20th Century have identified the rise in US productivity as the main driver of growth. The productivity gains are linked to several technologies that drove the upswing from the 1920s to 1970s. “GPT's are characterized by pervasiveness (they are used as inputs by many downstream sectors), inherent potential for technical improvements, and innovational complementarities', meaning that the productivity of R&D in downstream sectors increases because of innovation in the GPT. Thus, as GPT's improve they spread throughout the economy, [they brought] … about generalized productivity gains.”67 67 Timothy F. Bresnahan and Manuel Trajtenberg, "General Purpose Technologies: Engines of Growth," Journal of Econometrics, January 1995, 65(1), pp. 83–108. Earlier version cited as National Bureau of Economic Research Working Paper 4148, August 1992, p. iii. www.nber.org/papers/w4148
  • 31. 31 General Purpose Technologies68 include the internal combustion engine and electricity. Because firms in a wide range of industries exploited these technologies to improve productivity, GPTs were defined as “an invention that can lead to many sub-inventions.”69 We compare electricity and the internal combustion engine to software innovations. The first two are the GPTs70 that unleashed the expansion of US manufacturing and, thereby, the US economy in the 1920s, 1930s and 1940s. They continued to have a positive impact on productivity until the mid-1970s. These technologies had an outsized impact on productivity because as industries adopted them, they created innovative ways to use these technologies more efficiently. We believe that software, particularly in the form of cloud computing, cloud services and software- defined data centers has contributed to major advances in data analytics. Software is a 21st Century GPT. Table 6 compares software to internal combustion engines and electricity. 68 Timothy F. Bresnahan and Manuel Trajtenberg, "General Purpose Technologies: Engines of Growth," National Bureau of Economic Research Working Paper 4148, August 1992. Robert J. Gordon, The Rise and Fall of American Growth. Princeton University Press, 2016, pp. 555-565. Timothy F. Bresnahan and Robert J. Gordon, eds., The Economics of New Goods, Studies in Income and Wealth, vol.58, University of Chicago Press for National Bureau of Economic Research, 1997, pp. 1-26. http://www.nber.org/chapters/c6063.pdf 69 Gordon, The Rise and Fall…, p. 555. 70 Some research by economic historians has questioned the validity of asserting that electricity is a GPT because it shows lower generality scores than other technologies. Nicholas, Tom, and Petra Moser. "Was Electricity a General Purpose Technology: Evidence from Historical Patent Citations." American Economic Review: Papers and Proceedings 94, no. 2 (May 2004), pp. 388-394.
  • 32. 32 Table 6. General Purpose Technologies, Electricity, the Internal Combustion Engine and Software Innovations This table summarizes the similarities between today’s software innovations and earlier GPTs. The use of big data as well as advanced analytics – components of the Internet of Things -- are fundamental to how firms are making the transition to a digital world. As firms like Facebook, Netflix, ETSY, Ford, Boeing, UPS, and John Deere have become more efficient producers of products and creators of services; they have also enhanced their ability to manage their supply chains.71 71 Our conclusion draws upon a series of case studies on the economic impact of the Internet of Things that we prepared for the Organization for Economic Cooperation and Development. These cases will be cited in a new chapter on digital technologies and future production. The cases will be collected into a short book by this author, The Economic Impacts of the Internet of Things. General Purpose Technologies: Electricity, the Internal Combustion Engine and the Software Innovations related to the Third Wave of the Internet, Cloud Services plus Data Analytics Extending the Definition of General Purpose Technologies to Software and Services Electricity Internal Combustion Engine Innovative Software -- Fully Digital Infrastructure, Cloud Services and Data Analysis Productivity Drivers The price of electrically-generated power declined steadily in the early decades of the 20th Century. There were "constant improvements in the efficiency of electric motors. As a consequence, electric motors diffused rapidly throughout manufacturing displacing the steam engine." (Bresnahan and Trajtenberg, 1992). The price of automobiles and the cost of operating them declined rapidly during the early years of the 20th Century. There were a series of improvements such as closed tops that made vehicles easier to use in inclement weather. Paved roads made it easier to use vehicles in many areas that may not have been accessible. Innovations in developing software permit firms to create new software or applications for specific industries. This has facilitated complex data analysis. It also permits firms to offer faster and cheaper services to customers. Software development, data analytics and the Internet of Things rely upon cloud computing to operate. They lower costs through the improved management of supply chains and via new services, such as driverless cars and mobile purchases in retail stores. Within cloud computing, software-based management of data storage and computing accelerates the creation and deployment of new software and services. New tools that complement software development, such as containers (Docker) are being adopted rapidly. They support productivity gains in many industries, such as the financial, auto, pharmaceutical, aircraft, autos, logistics, retailing, information technology, and communications industries. Pervasiveness Used in a wide range of industries such as manufacturing, transportation, consumer durables, communications and healthcare. Automobiles were used widely on farms. They helped speed the delivery of supplies, support supply chains, and to accelerate shipments to consumers. They changed marketing and the delivery of services. They made it possible to reorganize retailing and other services.E11 The free distribution of programmed code via structures such as GitHub has reduced software costs and improved access to new innovations. This has been complemented by the rise of Open Source software that is often free of charge. Both of these changes have accelerated the distribution of software. The result is improved efficiency and more widespread use of cloud computing and data analytics, as well as lower costs for software and software tools. When cloud service providers use this software, it improves firms' reliance on Software-as-a-Service. This reduces the separation between innovators in services and consumer software. Potential for Technological Improvement Electrical power eq+D12uipment became more powerful and efficient. Greater economies of scale were achieved. The price performance ratio of products, systems, or components in which electric power was embedded improved. Costs in downstream sectors declined. As internal combustion engines matured, more powerful vehicles shortened delivery times and expanded deliveries to a wider market. The Internet of Things and Cloud Computing have an inherent tendency for technical improvement (Bresnahan and Trajtenberg, 1992). Firms such as Amazon, Google, Facebook and Netflix have used Cloud Computing to introduce new processes to create and test software. This practice is becoming common in many industries. Cloud Computing supports these changes by facilitating firms' adoption of more efficient software development processes such as DevOps, continuous delivery and containers. These processes speed the creation of new software and services and lower their cost. New tools, such as containers (Docker) for software development and deployment, dramatically lower the cost of creating new software or services and deploying it in different locations. In addition, the Internet of Things has been the foundation for Boeing and other firms to redesign production> It has helped Ford to improve the management of its supply chain and UPS to develop predictive analytics to speed deliveries. Innovational Complementarities -- Productivity of R&D in a downstream sector increases as a consequence of innovation in the GPT technology Electrical motors in factories not only lowered energy costs, but also enabled factory floor redesign (a redesign of the production process). Growth may really depend on the structure of markets where the technology is used. Automobiles made it possible for farmers to "bargain in the sale of farm products or the purchase of supplies." (Gordon, Rise and Fall , p. 163). Tractors revolutionized agricultural productivity. Vehicles expanded the size of the market for many industries. Aircraft changed the costs of supplies and expanded the market served by many industries. Software development benefits from new tools and processes such as containers, continuous service delivery and DevOps. Innovative software lowers the cost of analyzing complex data in many industries. Software developers can benefit from using exchanges like GitHub and Open Source software. This permits a large number of industries to improve existing software applications, software processes and services. In addition, new security services, such as Blockchain, are likely to reduce problems with security breaches. They are also likely to reduce the cost of operations and of software use. New market structures, such as GitHub, overcome the problem of asymmetrical information and uncertainty in the creation of new knowledge (Arrow, 1962 as cited in Bresnahan and Trajtenberg, 1992). Examples of GPT use in industry Elevators, electronic hand tools and machine tools, electronic streetcars and subways, consumer appliances (refrigerators, washing machines, and air conditioners), telephones and broadcasting, power plants and refrigeration, hospital x-ray machines, and ship geolocation. Cars, buses, and taxis. These vehicles made it possible to build supermarkets, suburbs, and to have personal travel, motels, and roadside restaurants, and air travel. New software applications that run on cloud computing infrastructure simplify complex genome analysis for new drugs. They support the development of driverless cars, the analysis of consumer behavior to restructure how retailing operates, and banks' use of complex investment strategies and more sophisticated risk analysis. Data analytics and the Internet of Things change business models so that products can be offered "as-a-service;" i.e., Rolls-Royce offers engines as "power by the hour." Other innovations include the management of driverless cars, the restructuring of aircraft production lines, greater efficiencies in farming using GPS and data analytics and the use of predictive analytics for more efficient deliveries at firms like UPS. We refer to the "Third Phase" of the Internet as one characterized by digital businesses that use infrastructure managed by software, such as software-defined data centers. This characterization comes from Steve Case, The Third Wave, New York: simon & Schuster, 2016, pp. 42-55. Sources: Arrow, K.J. "Economic Welfare and the Allocation of Resources for Inventions," in R. Nelson (ed.) The Rate and Direction of Inventive Activity , Princeton University Press, 1962. Timothy F. Bresnahan and Manuel Trajtenberg, "General Purpose Technologies: Engines of Growth," National Bureau of Economic Research Working Paper 4148, August 1992. Robert J. Gordon, The Rise and Fall of American Growth . Princeton University Press, 2016.
  • 33. 33 Figure 24 provides a very general illustration of the scope of these benefits using recent changes in output per employee for a few firms that are cloud computing users. We drew upon data from SEC filings to develop this rough gauge of productivity. We found that Facebook improved its productivity by 46 percent from 2010 to 2015 and ETSY improved its productivity by 17 percent from 2014 to 2015. Figure 24. Estimates of Productivity Changes at Specific Firms, Selected Years Source: 10-K reports filed with the Securities and Exchange Commission, various years. C. Will software increase productivity? In cloud environments, software development relies upon the rapid creation of new applications and swift modification of existing applications. This requires software-defined storage and cloud computing that supports continuous software delivery (drawing upon DevOps, i.e., shortening the software and services development cycle; microservices where software is assembled in Lego-like fashion; and containers, where developers can create a single application and easily run it using a wide range of operating systems without significant modifications). Using such techniques, firms have been able to deploy new applications or services in less than an hour. There are four ways that new software and infrastructure technology will increase productivity: 1. Through lower costs for software development, i.e., through coding new software more efficiently and designing more efficient processes to write and distribute new software. a. DevOps increases the time “high performing” firms can spend on new work and reduces the time spent on rework. This increases the efficiency of software developers and those in DevOps teams since they provide more output per developer. This conclusion comes from the “2016 State of DevOps report.”72 It discovered that “high performing” firms are spending 29 percent more time on new work than low performing firms. In addition, the high performing firms spend 22 percent less time on rework and unplanned work. 72 Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.” https://puppet.com/resources/white-paper/2016-state-of-devops-report
  • 34. 34 b. The software development process is also more efficient because of the shift to “serverless” computing. As we note above, with “serverless” computing, developers no longer need to be concerned about the physical or virtual servers they need to access to be able to code This means they no longer need to spend time setting up the infrastructure for their coding work. So, this also improves developer efficiency. c. Another example of improved efficiency is microservices (see Figure 18 above) where applications adopt an architecture in which each functional element is a separate service. By using microservices, firms can deliver applications and services faster.73 This supports “high velocity software development” and makes microservices an important part of continuous service delivery. d. A fourth example promoting efficiency is containers. For software development, containers wrap applications in the full system needed to run them. This reduces the time that would ordinarily be required to set up this infrastructure. The operating system resides in the container. This advance also hastens software development; it simplifies how software is written and deployed. 2. By Employing Software-Defined Infrastructure a. Software costs often decline in software-defined infrastructure. Software employed in Software Defined Data Centers can be obtained from Open Source software sites such as GitHub, a sharing hub on the Internet. This can reduce spending on programming. The use of Open Source and GitHub also permits enterprises to pay only a small fee or no fee rather than a somewhat expensive licensing fee for vendors’ software. b. In addition, innovations in the software used to manage software-defined data centers can provide large gains in costs. Google’s Senior Vice President for Technical Infrastructure, Urs Hoelzle 74 has noted that implementing a new generation of controller software reduced data center costs by 50-fold. 3. Through direct spending on new infrastructure that is often “infrastructure as code.” a. As firms make the transition to software-defined infrastructure, they will invest substantial amounts to take advantage of cost savings and the ability to do analyses more rapidly. The new infrastructure, software-defined data centers, supports more refined data analysis as well as more rapid development and testing of new software and applications. b. This renewal of infrastructure is essential for moving firms to a “New Production Ecosystem” where products can be produced more cheaply and services can be created more rapidly and at lower cost. c. This new infrastructure also accelerates the use of analytics. This makes firms more productive. This occurs because analysis has become so much more central to a firm’s 73 Richard Li, “Microservices Essentials for Executives: The Key to High Velocity Software Development,” for Entrepreneurs from David Skok. http://www.forentrepreneurs.com/microservices/ 74 Urs Hoelzle, “OpenFlow @ Google,” May 7, 2012. https://www.youtube.com/watch?v=VLHJUfgxEO4. Hoelzle cites an efficiency gain of 50 times since the Controller in Google’s infrastructure “uses modern server hardware 50x (!) better performance.” See Hoelzle’ s slides from the talk, http://opennetsummit.org/archives/apr12/hoelzle- tue-openflow.pdf
  • 35. 35 operations. Thus, firms have a need to move to real-time analytics to insure they can analyze production, supply chain, marketing and competitive aspects of their operations. 4. Through the expansion of business opportunities. a. New ways of developing software reduce the time to market for new applications and services. This lets enterprises expand the services and products they offer with very short development times. The result is an increase in firm revenues and productivity, the output produced per employee. Earlier, we provided an example of this benefit from Lori Beer, the CIO of JPMorganChase. b. Big Data analysis, the analysis of large amounts of data (data lakes) gives enterprises insights into markets. This type of analysis was not easy to perform with highly distributed data bases. With consolidated data centers, firms can exploit new business opportunities in ways that add new value. Businesses can also refine the analysis of designs. This helps them develop products and services that compete better in the marketplace. Big data analysis also provides better understanding of the markets businesses are trying to serve. The improvements in productivity noted above: a. Reduce the cost of developing and refining new products and services. b. Create software innovations that operate very much like Moore’s Law, sparking ongoing expansion in the amount of work that firms can perform in a broad range of operations, not merely in information technology viewed in a restricted context. c. Enhance the ability of firms to analyze data at much lower cost than had been the case previously. This adds considerable value to product and services development. It also contributes to the Moore’s Law-like cost reductions in a range of business operations that can exploit the software innovations mentioned above. D. Why hasn’t software improved productivity in recent years? We argue that software is driving innovation in the U.S. economy. This is taking place as software plays a key role in cloud computing, data storage, network use, and changing the operations of communications and cloud service providers. These changes, driven by enhancements in software, change how firms operate. In addition, infrastructure as code or software-defined infrastructure has not only reduced costs, but also provided new opportunities for businesses to expand operations and refine products. Economists need a better understanding of how software is developed. They need to understand how programmers have created new generations of software; i.e., that software developers have created substantial innovation in the ICT sector.
  • 36. 36 The contribution of software and cloud services to the growth of U.S. output is not estimated directly in national income and product accounts. “ICT capital continues to grow and penetrate the economy -- increasingly via cloud services which are not fully accounted for in the standard narrative on ICT's contribution to economic growth -- the contribution of ICT to growth in output per hour going forward is calibrated to be substantially larger than it has been in the past”75 [my emphasis] These changes are not showing up in U.S. economic statistics. We believe that one reason is measurement. Government economists who estimate the size of the digital economy do not measure software innovations directly. They rely upon indirect measures to gauge the size of the digital economy. These include the prices and performance of mobile phones, laptops, and tablets. As a result, these economists have not captured how software has created more ubiquitous changes in the economy. These issues remind us of the spread of electricity through the economy. One problem is that economists collect data on software innovations indirectly. To estimate changes in products that use innovative software, economists examine the prices and performance of hardware that incorporates a good deal of software, such as laptops, mobile devices, and cell phones. These metrics, largely from hardware, are the main yardstick employed to measure the digital economy. There are no data sets that measure software innovation directly. For “cloud and related ICT services, Byrne and Corrado 76 (2016) … imply… these prices should fall no slower than the rate of decline in ICT asset prices.”77 [This assertion is in line with Daniel Sichel’s conclusion that “Desktop PCs: hedonic price indexes falling about 15 percent [from] 2007 -2010, more slowly thereafter.”]78 Yet, both findings are not in accord with the substantial cost changes that Google’s Senior Vice President for Technical Infrastructure, Urs Hoelzle, reported above. Press reports suggest that cloud computing and storage services are falling very fast (in the 20 to 30 percent per year range). These services usually are purchased along with software services, which would substantially moderate overall declines. It should also be noted that the total cost of cloud services (from a purchasers’ perspective) includes high-speed broadband (WAN and LAN) services.”79 Direct measurement of the digital economy could include: 75 Byrne and Corrado, “ICT Prices and ICT Services.” 76 David Byrne and Carol Corrado, “ICT Prices and ICT Services: What do they tell us about Productivity and Technology?” The Conference Board, Economics Program Working Paper Series #16-05. May 2016 (revised July 2016). https://www.conference-board.org/pdf_free/workingpapers/EPWP1605.pdf 77 Carol Corrado, “Discussion of: Improving ICT Deflators in the National Accounts,” papers were prepared for the meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016, p. 4. http://bea.gov/about/advisory.htm 78 Daniel Sichel, “A New Look at Prices of Personal Computers, Tablets, and Cell Phones: A Progress Report,” papers were prepared for the meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016. http://bea.gov/about/advisory.htm 79 Carol Corrado, “Discussion of: Improving ICT Deflators in the National Accounts,” papers were prepared for the meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016. p. 3. http://bea.gov/about/advisory.htm
  • 37. 37 a. The impact of continuous service delivery, microservices, containers and DevOps in accelerating software development.80 b. Delivery speed of software development. Many firms no longer optimize software development for cost, as economists assume. As Cockcroft notes,81 “Nordstrom is no longer optimizing for software cost but for delivery speed.” c. Estimate the contributions of continuous service delivery, microservices, containers and DevOps to the creation and deployment of software. These innovations have changed the way that developers82 create, test and deploy software and applications. Economists have not yet taken these new processes into account. They have not developed ways to measure innovations in software delivery. This hampers our understanding of how to incorporate software into the national income accounts.83 At the firm level, the move to more rapid creation of software is having a clear impact. The 2016 State of DevOps Report,84 notes that high performing firms have sped up the deployment of new software and attained a level of 200 times more frequent deployments than comparable low performing firms. Nonetheless, many economists are continuing to measure price changes in some of the key devices used in the digital economy, such as tablets, desktops and laptops.85 E. Economists, Software Innovations and the Third Industrial Revolution Several economists have described their vision of future of economic growth and how it will be shaped by innovations in information and communications technologies (ICT). We believe that, as a group, economists have not appreciated the vast potential size of the impact of software innovations. This oversight is primarily because economists possess few accurate tools to measure the digital economy. This section takes the previous section’s exploration of software innovations as a given. It proceeds to examine writings by a few well-known economists that examine whether the Internet Revolution is likely to have any impacts on GDP and jobs. 80 Adrian Cockcroft, “Creating Business Value with Cloud Infrastructure,” Open Networking Users Group meeting, May 13-14, 2015, Columbia University. 81 Cockcroft, p. 12. 82 Cockcroft, pp. 52- 112. Robert Cohen, The Internet of Things, Productivity, and Employment,” presentation for Internet of Things Summit, Boston, Sept. 8-9, 2015 offers a summary of the main points Cockcroft makes. 83 Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.” https://puppet.com/resources/white-paper/2016-state-of-devops-report 84 Puppet, Inc., and DevOps Research and Assessment, “2016 State of DevOps Report.” 85 Ana Aizcorbe, “Improving ICT Deflators in the National Accounts.” Dan Sichel, Dave Byrne and Steve Oliner, “A New Look at Prices of Personal Computers, Tablets, and Cell Phones: A Progress Report: paper for BEA Advisory Committee.” Guilia McHenry, “Measuring the Digital Economy: Motivations and Initiatives.” These papers were prepared for the meeting of the Bureau of Economic Analysis Advisory Committee, November 18, 2016. http://bea.gov/about/advisory.htm
  • 38. 38 1. Robert Gordon and The Rise and Fall of American Growth Robert Gordon is the strongest advocate of the view that ICT innovations are likely to have a small impact on future US growth and productivity. Gordon’s recent book86 discusses many important innovations from the Second Industrial Revolution that occurred from 1870 to 1970. These include the internal combustion engine, electrification, the airplane, and the refrigerator. After reviewing these innovations and recent ones tied to ICTs, Gordon concludes that there are likely to be fewer innovations in the future. This conclusion is linked to his conviction that the chances that a widely used, revolutionary technology will emerge over the next decade or two is unlikely. After carefully reviewing the impact of General Purpose Technologies – technologies that were adopted widely and adapted to the requirements of a broad range of industries during the Second Industrial Revolution (IR#2), Gordon considers whether the Internet and ICT, including software, might play a similar role in the Third Industrial Revolution. After an examination of recent innovations, he finds that the US economy is likely to experience lower productivity growth and lower levels of GDP growth. Gordon’s reasoning follows a logic that flows from his detailed review of the US experience during the special century, or Second Industrial Revolution, from 1870 to 1970. The key characteristic of the special century was that firms adopted important new technologies and refined them so that they could achieve significant increases in productivity growth. These productivity and GDP gains peaked during the 1940s and continued through the early 1970s. In discussing Gordon’s book below, we identify ways that the Next Industrial Revolution (IR#3) is beginning to have real impacts in businesses beyond the ICT and entertainment industries. Our examples suggest there are likely to be opportunities for improvements in productivity and cost savings in IR#3. 2. Why Gordon concludes that the Next Industrial Revolution (IR#3) will have Little Impact on GDP Growth and Jobs After a review of more recent growth and productivity trends and ICT innovations, Gordon concludes that nearly all innovations based on the Internet ended in 2004. He dismisses the notion that the Third Industrial Revolution (IR#3), the Internet Revolution, associated with computers and digitization, will be like the Second Industrial Revolution. He explains that this is likely to be true because “business practices in the office, the retail sector, and in the banking and financial sector … current methods of production had been largely achieved by 2004.”87 Gordon finds that innovation has continued since 1970, but it has not grown at the same rate as it did earlier. He argues that this is likely to result in slow productivity and economic growth. One reason for Gordon’s conclusion is his belief that the recent innovations of IR#3 affect only a few industries, including “entertainment and information and communications technology (ICT).”88 He estimates that the growth of total factor productivity (TFP) in the US declined after 2004, with productivity growth only half as fast 86 Robert Gordon, The Rise and Fall of American Growth, Princeton University Press, 2016, p. 567. 87 Robert Gordon, The Rise and Fall of American Growth, Princeton University Press, 2016, p. 567. 88 Gordon, p. 567.
  • 39. 39 as it was from 2004 to 2014, as compared to 1994 and 2004. In part, he finds that this is due to the slow transformation of business practices. Today, Gordon notes, office employees’ productivity resembles what it was in about 2004 because they had all the equipment used in office work today.89 Gordon buttresses his argument by asserting that IR#3 has impacted only a few key industries. He discounts the chances that industries besides entertainment, information and communications technologies will benefit from IR#3. He writes that there has been a “stasis in retailing” and that “the main impact on retail productivity growth of big-box [retailing] stores … largely occurred by a decade ago.”90 This ignores the rise of Amazon and its reshaping of retailing, as well as significant efforts by WalMart, Kroger’s and Nordstrom’s to reshape their operations around digitized services and mobile software. Gordon cites the decline in stock trading after the financial crisis as evidence of a “plateau of activity in finance and banking.”91 In reviewing the home and consumer electronics industries, Gordon concludes that “within the past decade … computer hardware, software, and business methods ossified into a slowly changing set of routines.”92 Intel’s work on artificial intelligence (AI) contradicts Gordon’s claim that the business models of the hardware, software and home and consumer electronics industries have “ossified.” Nidhi Chappell, Director of AI Strategy, at Intel notes93 that Intel has built upon Moore’s Law, data availability and innovation in algorithms to drive more widespread AI use. Intel broadens AI’s use by compressing the innovation cycle (a major change in routines), democratizing access to AI and guiding the development of AI in service of humankind (by solving cancer, decoding the function of the human brain, etc.). Intel has also used Big Data and software engineering to solve high-value problems94 . It has assembled teams of 5 people who focus on solving production problems for 6 months. They use historical data as well as unstructured data to predict business outcomes. Each team is expected to save Intel at least $10 million in six months. The Intel teams harness new, Big Data skills plus software-defined, cloud computing infrastructure to analyze large databases. They reduce manufacturing costs by enhancing the initial testing of new semiconductor manufacturing processes. When Intel saves I second of test time in production, it saves $5 million to $10 million. This breakthrough solves defects and problems in production by identifying their root cause. Again, this evidence runs counter to Gordon’s contention that the industry has ossified. 89 Gordon, p. 580. Gordon should examine what has happened to productivity on farms where new technology has benefitted from Big Data analysis. See the case of John Deere in Robert B. Cohen, “Case Studies of the Internet of Things,” IoT Slam ’17, Durham, North Carolina, June 20, 2017. https://es.slideshare.net/bcohen777/case-studies- of-the-internet-of-things-062017 90 Gordon, p. 581. 91 Gordon, p. 582. 92 Gordon, p. 583. Note, that the previous section offers little evidence of this “ossification.” 93 Nidhi Chappell, Intel’s Director of AI Strategy, “Under the Hood: Intel Accelerating the Future of Artificial Intelligence | Intel IT Center,” https://www.youtube.com/watch?v=MKFIvNTre2I 94 Moty Fania and Assaf Araki, “Solving High Value Problems with Big Data Analytics,” Big Data Analytics DMBI 2014 Second Annual International Conference, Sept. 14, 2014. https://www.youtube.com/watch?v=HmhCjYYAmz8
  • 40. 40 Gordon’s conclusions also overlook the sizable recent changes in business practices and business structure. The Wall Street Journal has cited Equifax Inc., insurer Liberty Mutual, and consumer-products giant Procter & Gamble Co., as firms that have adopted a mobile, cloud and data technology strategy from Silicon Valley. They are changing the way they operate, taking on many of the characteristics of tech and Internet firms, such as Facebook, Google, and Amazon.95 Many large firms such as JPMorganChase, Proctor and Gamble, Boeing, Ford and General Motors, are shifting to shorter development cycles. This improves their agility, particularly their responsiveness to changes in markets. Gordon overlooks these changes in changes in corporate behavior. Gordon emphasizes that price declines for ICT equipment relative to performance has slowed. He cites data showing that by 2014, there were almost no price declines at all, as compared to rapid price declines in the late 1990s.96 Gordon expects that this slowdown to continue at the same pace, repeating the slow rate of TFP growth from 2004-2014. Gordon asks whether the next wave of innovations might prove to be as revolutionary as they were during the dot-com revolution of the late 1990s. He finds that if the pace of innovations decelerates, it will result in a productivity growing at the same rate as it did during 2004-2014.97 One note in rebuttal is that some scholars have developed a performance indicator that they have merged with the producer price index (PPI) for servers. With this adjusted index, price changes, i.e., the blended indicator and PPI, fall 11 percent faster than the Bureau of Economic Analysis Investment Index.98 3. Gordon’s Evaluation of Future Advances in Technology and their Impact on Total Factor Productivity Growth To assess future technology-based advances that Eric Brynjolfsson and Andrew McAfee as well as others forecast, Gordon examines four categories where technology may have big impacts: medical; small robots and 3D printing; big data; and driverless cars.99 Gordon evaluates whether new technologies in these areas might bring TFP growth back to the levels it attained in the late 1990s. We review Gordon’s conclusions about these recent innovations. We find, contrary to Gordon, that there is a greater chance for increased TFP growth in the coming years than Gordon believes.100 95 Angus Loten and John Simons, “Leadership Evolves Amid Tech Changes: Equifax, P&G, Liberty Mutual embrace digital tools; managers shift toward shorter development cycles, Wall Street Journal, CIO Journal, Jan 3, 2017. http://blogs.wsj.com/cio/2017/01/03/tech-is-transforming-how-businesses-are-run/ 96 Gordon, p. 593. 97 Gordon, p. 593. 98 David Byrne and Carol Corrado, “ICT Asset Prices: Marshaling Evidence into New Measures,” The Conference Board, July 10, 2016. https://www.conference-board.org/pdf_free/workingpapers/EPWP1606.pdf https://www.conference-board.org/publications/publicationdetail.cfm?publicationid=7241&centerId=8, The Conference Board, July 2016, p. 12. 99 Gordon, p. 593. 100 See the discussion in Sections D and E.
  • 41. 41 In examining medical and pharmaceutical advances, Gordon finds that medical technology has continued to advance since 1980 but at a “slower and measured pace.” He finds that pharmaceutical research has hit a “brick wall of rapidly increasing costs and declining benefits.”101 It is disconcerting that Gordon has not examined a few of the recent major achievements of medical science. The cost of decoding a human genome has fallen much faster than Moore’s Law.102 With such a dramatic change, innovative firms have decoded the human genome inexpensively and developed entirely new ways to treat diseases. This is happening despite claims by the largest pharmaceutical firms that it costs $2.6 billion to create a new drug. Gordon overlooks the fact that rapidly declining costs of decoding the human genome are likely to result in more opportunities to develop new drugs at much lower cost. Some of the earliest drugs to take advantage of breakthroughs in decoding the genome are “Pfizer's lung cancer treatment Xalkori, … approved in 2011; [it]… targets mutations in tumors driving the disease. … Vertex Pharmaceuticals … changed the treatment of cystic fibrosis with Kalydeco; [it] … targets the disease’s underlying genetic cause.”103 These discoveries have benefitted from the rapid decline in the cost of decoding genomes. As shown in Figure 25, this cost has fallen faster than Moore’s Law. 101 Gordon, p. 594. g 102 National Institutes of Health, National Human Genome Research Institute “The Cost of Sequencing a Human Genome” https://www.genome.gov/sequencingcosts/ As this site notes, “The underlying costs associated with different methods and strategies for sequencing genomes are of great interest because they influence the scope and scale of almost all genomics research projects.” 103 Julie Steenhuysen “How DNA sequencing is transforming the hunt for new drugs,” Science News, May 13, 2015. http://www.reuters.com/article/us-health-precisionmedicine-insight-idUSKBN0NY0AX20150513
  • 42. 42 Figure 25. National Human Genome Research Institute’s Estimates of The Cost of Decoding based on Costs at its Human Genome Centers, as compared to Moore’s Law Price Changes Source: National Institutes of Health, National Human Genome Research Institute, “The Cost of Sequencing a Human Genome” https://www.genome.gov/sequencingcosts/ Gordon contends that pharmaceutical research has hit a wall of rapidly increasing costs. This is true if one accepts the findings of the Tufts Center for the Study of Drug Development that it costs $2.6 billion to create a new drug. A cogent criticism104 of these results is that they consider only new molecular entities, drugs with chemical compounds that have never been approved for individual use or combination therapies. This sample represents a small part of the population of new drugs developed each year. Thus, the Tufts results are biased and provide much higher costs of developing new drugs than may be the case. Tufts excludes many of the drugs developed with funding from the National Institutes of Health and other government entities. In addition, successful case, such as Pfizer’s lung cancer treatment and other drugs that use the body’s immune system to fight cancer, have benefitted from cloud computing. 104 Aaron Carroll, “$2.6 Billion to Develop a Drug? New Estimate Makes Questionable Assumptions” New York Times, November 19, 2014. https://www.nytimes.com/2014/11/19/upshot/calculating-the-real-costs-of- developing-a-new-drug.html
  • 43. 43 In discussing the limitations of innovations such as small robots, Gordon reasons that it is difficult for them to “match a human’s dexterity and problem-solving abilities.”105 He notes that it is difficult for robots to distinguish between picking up lace and crumpled jeans. Research completed after the publication of Gordon’s book reported advances in machine learning that have improved robots’ dexterity. By combining vision and touch,106 a lab at the University of California, Berkeley enhanced robotic dexterity so a robot could fold clothes. This is the challenge that Gordon did not believe could be solved. Gordon notes that 3-D printing is “not expected to have much effect on mass production and thus on how most U.S. consumer goods are produced.”107 Recently, engineers at Renault Truck108 have boosted performance of the company’s truck engines by using 3-D printing to produce 25 percent of these engines’ parts. This greatly simplifies production and the advance is likely to be copied by other automakers around the world. In addition, using 3-D printing for engine parts improves performance and increases the workloads trucks can haul because the 3-D parts are often lighter and more reliable. Thus, they can lower fuel consumption. Success in this area might be repeated in creating consumer goods. 4. Tim Bresnahan and Pai-Ling Yin on innovation in ICT industries Timothy Bresnahan and Pai‐Ling Yin109 offer another interpretation of the potential impact of IR#3. They indicate that “the invention of new applications based on information and communications technologies (ICTs) has had two economic effects up to now.” It has “transformed production” and shifted the demand for skilled labor in the workforce. They also make a case that ICT innovation requires a time-consuming and complex co-invention process for innovations such as new software. One of the assertions that Bresnahan and Yin make is that ICTs are enablers of the invention of new applications and that most of the innovation occurs in firms outside of the ICT industries. They focus on co-invention, “the product and process improvements created by industries as they apply new ICT,” noting that: “ICT co‐invention is defined as the product and process improvements created by industries as they apply new ICT. One driver of co‐invention is the ICT advances themselves (supply), such as cheaper storage, faster networks, or more capable software. ICT advances produce a large scope of feasible opportunities. The other driver is the industry circumstances (demand) of 105 Gordon, p. 596. 106 John Markoff, “New Approach Trains Robots to Match Human Dexterity and Speed,” New York Times, May 21, 2015. https://www.nytimes.com/2015/05/22/science/robots-that-can-match-human-dexterity.html?_r=0 107 Gordon, p. 597. 108 Sam Davies, “Renault Truck introducing metal additive manufacturing to engine production process,” TCT Magazine, January 11, 2017. http://www.tctmagazine.com/3D-printing-news/renault-truck-introducing-metal- additive-manufacturing-engine/ 109 Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf to appear in Shane Greenstein, Josh Lerner, and Scott Stern, editors, Innovation Policy and the Economy, Volume 17.
  • 44. 44 firms trying to use ICT: competition, customer demand, and the production processes already in place.”110 Bresnahan and Yin argue that co-invention today is very much like it was during previous ICT advances. They present this as their view of ICT innovations, but do not consider evidence about software innovations that we have mentioned above. As we have noted, many new processes have streamlined software development and accelerated the rate at which they are deployed. Tremendous changes have taken place in software development, including continuous service delivery, containers, microservices and “serverless” computing. These innovations should have resulted in a reconsideration of how today’s ICT co-invention process works. Bresnahan and Ying write that “while there has been terrific technical progress in ICT, there has been little change in the ICT co‐invention process. Co-invention still requires considerable brainpower and experimentation. Co‐invention still looks for ways to change whole organizations. Indeed, modern co‐ invention often looks for ways to change whole supply chains.”111 Figure 26. ICT Co-Invention of Applications Source: Timothy Bresnahan and Pai‐Ling Yin, “Adoption of New Information and Communications Technologies in the Workplace Today,” Stanford University, June 21, 2016. www.nber.org/chapters/c13762.pdf , p. 3. We would disagree with the characterization of the invention of new applications, particularly new software, that Bresnahan and Yin utilize. The perpetual development model takes place largely in firms 110 Bresnahan and Yin, p. 2. 111 Bresnahan and Yin, p. 3.