3. Assessing Your Business Analytics Initiatives
Introduction
It’s no secret that using analytics to uncover meaningful insights from data is crucial
for making fact-based decisions. Now considered mainstream, the business analytics
market worldwide is expected to exceed $50 billion by the year 2016.1 Yet when it
comes to making analytics work, not all organizations are equal. In fact, despite the
transformative power of big data and analytics, many organizations still struggle to
wring value from their information. The complexities of dealing with big data, integrating
technologies, finding analytical talent and challenging corporate culture are the main
pitfalls to the successful use of analytics within organizations.
The management of information – including the analytics used to transform it – is an
evolutionary process, and organizations are at various levels of this evolution. Those
wanting to advance analytics to a new level need to understand their analytics activities
across the organization, from both an IT and business perspective. Toward that end,
an assessment focusing on eight key analytics metrics can be used to identify strengths
and areas for improvement in the analytics life cycle.2
The Metrics
The evaluation of an organization’s proficiency across the following eight metrics
provides guidance for short- and long-term efforts needed to enhance analytical
effectiveness.
• Productivity
• Governance
• Timeliness
• ROI
• Accuracy
• Effectiveness
• Empowerment
• Maturity
Productivity is the efficiency of processes supporting the analytics life cycle across IT
and business functions. Companies with a high level of productivity in their analytics
activities are characterized primarily by their integration of information technology
and capacity, strong data management and continuity of business resources. These
organizations invest in the appropriate analytical training for their employees and are
able to garner and share insights from complex data sets. Importantly, IT and business
priorities in these companies are aligned.
1
IDC, Worldwide Business Analytics Software 2012-2016, Forecast and Vendor Shares, June 2012.
Analytics life cycle: An iterative process using data to solve business problems and make informed
decisions. The process comprises the following steps: prepare and explore data, develop and deploy
models, and monitor results.
2
1
4. SAS White Paper
Governance is the overall rigor placed around data and model stewardship.
Organizations proficient at governance are marked by consistent monitoring of data and
quick corrections to deviations. They adhere to specific and standard methodologies
and tools. Between IT and business functions, there are service-level agreements
and alignment.
Timeliness means more than just speed. It denotes whether analytical value is realized
within the available time window. Timeliness in the analytics life cycle is exhibited by
organizations that can handle large volumes and varieties of data, quickly get it into a
usable state, derive meaningful insights from it and put analytical models into production.
Data received from other areas within the organization needs to be assimilated in a
timely fashion to make the best decisions and to be able to react to changes in the
market quickly.
ROI is the value generated from analytics as compared to the cost of providing that
value. Organizations that are highly proficient in delivering ROI have analytics-driven
culture and operations. These organizations are able to quickly identify the variables that
predict outcomes, and they are able to customize their marketing approaches. Upper
management advocates the use of analytics, and the organization has the right amount
of analytical talent. Product offerings and services are up-to-date with the market. The
impact of inaccuracy in the organization is low. Costs in the organization are transparent
and well-understood. Cost/benefit analyses for new projects are documented, and there
is agreement on the benefits of new initiatives. Investments in IT translate to value.
Accuracy pertains to the accuracy of data in the analytics life cycle and the impact
it has on effective decision making. Organizations performing well on the accuracy
metric continuously focus on finding and rectifying inaccurate data as it gets modified
in each step of the analytic life cycle and reused by a variety of applications. They have
few costly mistakes in their history because of continuous data quality monitoring.
Information is precise, accurate and timely. They have implemented data accuracy
processes, and data quality and analytical results are consistent across the organization.
Effectiveness is the organization’s ability to overcome challenges and generate value
across people, processes, technology, data and culture. Organizations that have
mastered effectiveness in analytics are marked by their feedback mechanism for
systems and process improvement now and over time. They have reduced the reliance
on IT for ad hoc and one-off reports. These organizations receive a high level of value
from their technology and have adequate analytical talent to meet their needs. They
are using analytics to address key business issues. Importantly, IT and line-of-business
requirements are aligned.
2
5. Assessing Your Business Analytics Initiatives
Empowerment is the level of self-sufficiency for employees supporting the analytics
life cycle across IT and business functions. Organizations that have been successful
empowering their employees in the analytics life cycle provide decision makers with
access to necessary information. They invest in end-user training on analytics software
and provide the appropriate analytical resources. End users are self-sufficient in data
exploration and reporting and have the ability to explore data for patterns and insights.
A key differentiator for this metric is that employees are independent and empowered
to find and solve business problems. Decision making is timely.
Maturity is the organization’s analytical competency, consistency and alignment across
people, processes, technology, data and culture. Organizations that are mature in their
use of analytics have mastered many of the components of the other metrics. They
are characterized by the high level of sophistication in their daily analyses. Employees
have easy access to statistical consultants. Information is shared across departments.
These organizations leverage new and emerging technologies and use data to
improve processes. Finally, these mature analytics users have clear priorities and are
strategically focused.
Business Analytics Benchmark Study
These metrics were used in benchmark research with more than 400 US companies,
including 30 in-depth company assessments and 375 online surveys conducted among
organizations across all industries and company sizes.3 The metrics were derived
from a survey of 64 questions, eight for each category, pertaining to an organization’s
information management processes and activities. The results of the survey research
have been aggregated and the highlights are presented below.
Overall Metrics Scores
In general, our benchmark research revealed that organizations have not quite reached
a high level of proficiency for the key metrics. On a proficiency scale of 1 (low) to 5 (high),
the average scores of survey respondents ranged from a low of 2.56 on effectiveness
to a high of 3.17 on governance. This suggests that organizations are having the
most difficulty overcoming analytical challenges and generating value across people,
processes, technology, data and culture, but are somewhat more adept at managing
the overall rigor around their data and model management.
Timeliness is also an area that received a lower score compared to most other metrics
based on our survey results. As organizations continue to address the complexities of
more data in a variety of forms, processes in the analytics life cycle can require more
resources, which can create bottlenecks. (See Figure 1.)
Looking at the balance of these eight metric scores provides information that
organizations can use to advance their analytics initiatives. The specific questions
that can be used to determine your metric score are discussed below (The Essential
64 Questions).
3
For a breakdown of the survey respondent demographics, see the Appendix.
3
6. SAS White Paper
Figure 1: Average benchmark scores on analytics metrics.
Metrics by Industry
Our benchmark research indicates that industries are at various levels in their proficiency
across the eight metrics used to assess their business analytics initiatives. Compared
to the average metric scores across all industries, education, professional services,
financial services and health care are generally ahead of their counterparts. Interestingly,
the education sector lags others in governance, while professional services struggles
with analytics ROI, and financial services is below the average on timeliness. Health care,
on the other hand, is not as proficient as other industries in their analytics productivity
and accuracy.
Surprisingly, respondents from the IT and technology industry lag those in other sectors
on all but one metric. Manufacturing and retail organizations, according to the survey,
are less proficient than other industries on all metrics. (See Table 1.)
4
7. Governance
IT and technology
Manufacturing
Retail/Wholesale
All industries
Above Industry
Average
Effectiveness
Energy and natural resources
Timeliness
Communications
Maturity
Government/Public sector
Accuracy
Health care
Empowerment
Financial services
3.13
3.03
3.02
2.95
2.79
2.73
3.17
3.28
3.03
3.08
3.09
3.01
2.74
2.62
3.27
3.09
3.24
3.03
3.02
2.84
2.58
2.67
3.07
3.15
2.95
2.86
2.92
2.68
2.56
3.22
3.04
2.81
2.86
3.02
2.81
2.78
2.52
3.03
3.17
2.96
2.94
2.86
2.64
2.59
2.59
3.30
2.92
2.87
2.87
2.78
2.48
2.56
2.57
2.98
2.92
2.95
2.98
2.86
2.53
2.45
2.32
3.08
3.06
2.77
2.72
2.78
2.76
2.46
2.35
2.85
3.04
3.00
2.87
2.82
2.76
2.48
2.47
3.17
Professional services
3.26
3.33
Education
ROI
3.01
Industry
Productivity
Assessing Your Business Analytics Initiatives
3.09
3.04
2.94
2.94
2.81
2.61
2.56
Below Industry
Average
At Industry
Average
Table 1: Metrics by Industry
Metrics by Organization Size
According to responses provided by our survey participants, it appears that smaller
organizations (under $1 billion in revenue) are actually somewhat more proficient in
all of the key areas we measured, with the exception of productivity. Although larger
organizations have bigger budgets and generally more resources, it makes sense that
smaller entities are more agile than their larger counterparts. There are fewer silos, less
data and smaller infrastructures to deal with. The biggest difference in the metrics is
accuracy. Larger organizations continue to struggle with data management, particularly
as big data enters the scene. Getting to the single version of the truth becomes more
difficult when dealing with higher volumes of data and disparate sources. (See Figure 2.)
5
8. SAS White Paper
Governance
3.15
ROI
3.04
3.25
3.15
3.08
3.10
Productivity
Accuracy
2.88
3.06
2.98
2.95
Empowerment
2.85
2.78
Maturity
Timeliness
2.58
Effectiveness
2.52
Under $1B
2.69
2.68
$1B and over
Figure 2: Metrics by organization size.
The Essential 64 Questions
The questions asked of survey respondents to assess their business analytics
proficiency on these eight metrics address a variety of factors, including processes,
infrastructure, people, data and culture. Within a specific organization, gathering
responses from a representative group of employees from both the business and IT is
critical to developing an accurate picture of the proficiency of the organization on our
eight analytics metrics. The differences in perceptions (based on their responses) across
various groups within the business are also important when determining action steps to
address areas for improvement.
Below are the 64 survey questions in descending order based on the average survey
response for each question. Note that some questions are worded in the negative, so
the metric scores are not simply an average of the ratings for each question.
Interestingly, our benchmark research shows that organizations are looking to analytics
to improve the way they do business. The top three issues in the questions below relate
to the need for the organization to use technology, specifically analytics, to drive better
decisions. The details in many of the remaining questions provide guidance on what
areas should be addressed to improve the proficiency of analytics initiatives.
6
9. Assessing Your Business Analytics Initiatives
Please indicate the extent to which you agree with the following statements:
1 = Strongly Disagree
5 = Strongly Agree
Average
Level of
Agreement
There are a significant number of issues at my company that could
benefit from better use of analytics.
4.3
We need to get more value out of the current technology we have.
4.2
We need to react faster to market changes or opportunities.
4.1
We have dedicated resources to ensure the continuity of business.
3.8
Resource constraints (people and infrastructure) at my organization
make completing work harder than it needs to be.
3.9
My department systematically follows standard methodologies or
processes as a practice.
3.7
Employees at my organization are empowered to find and solve
problems.
3.7
My organization has analytical data sets that can be used to support
multiple initiatives.
3.7
It is time-consuming and difficult to get analytical models into
production.
3.8
We typically use the results of data analysis to improve our
processes.
3.7
Most of our marketing efforts focus on either large customer
segments or definitions of group membership rather than
customized approaches or microtargeting.
3.5
Any new reporting or changes to existing reports take a long time to
develop.
3.6
Work priorities change frequently in my job and are very tactically
focused.
3.6
Key stakeholders generally agree on the benefits of new information
technology initiatives.
3.5
Our computer systems (hardware, software, network) are able to
handle multiple peak periods of usage.
3.5
We are lacking feedback loops to ensure that internal processes
become more effective and efficient over time.
3.6
7
10. SAS White Paper
Any deviations from established operational norms are slow to be
corrected.
3.6
There are specific service-level agreements between IT and business 3.5
areas.
Analytics talent is too diluted across the company.
There is a disconnect between IT and business or line-of-business
requirements.
3.6
Imprecision negatively affects business outcomes and decisions I
have to make.
3.6
Upper management strongly advocates or promotes analytics.
3.6
There is a discrepancy in the priorities of the IT department and the
needs of the business.
3.6
Accurate information is delivered in a timely manner.
3.4
It takes too long to get data that is in a usable state.
3.5
We use standard processes and tools to compare actual results to
goals.
3.4
Our IT group suffers from lost productivity and time due to ad hoc
and one-off requests involving different data elements.
3.5
We could benefit from having easier access to statistical consultants
for some of the work we do.
3.4
The cost of inaccuracy in our organization is high and has a
significant impact on profitability, market share and the ability to
meet competitive pressures.
3.6
We often cannot get the internal data we need from other
departments, so we find and analyze it ourselves.
3.4
Data security and authorization issues inhibit productivity.
3.3
It takes too long to get meaningful insights from data.
3.4
We are generally able to make good decisions in a timely fashion.
3.4
We have difficulty analyzing data in a timely manner.
3.2
Data quality and data deviations are consistently monitored.
3.1
We typically count on one specific person within our department to
do analysis, interpretation and deliver results.
8
3.5
3.1
11. Assessing Your Business Analytics Initiatives
I am able to quickly identify key variables that influence or predict
business outcomes.
3.2
We have a formal feedback mechanism to help improve processes
and systems.
3.1
We are easily able to identify, understand and share insights from
complex sets of data.
3.2
There is little statistical sophistication in our daily data analysis.
3.1
A lot of costly business decision mistakes have been made in the
past at my organization.
3.4
There is little correlation between the cost of our information
technology and the value we receive from it.
3.2
The reasons for internal hardware/software changes are clear.
3.2
The majority of data quality inconsistencies are identified and
addressed.
3.0
We are stifled in our decision making due to the volume and variety
of data we have.
3.1
We often get inconsistent results when analyzing the same data
source.
3.1
Decision making is enabled in a timely fashion.
3.1
My organization does not have self-service capabilities for data
reporting or analytics.
3.1
It is difficult to explore our data for patterns or regularities.
2.9
Cost/benefit analysis for new initiatives is well-documented.
2.9
Data from other departments needed for reporting is received in a
timely fashion.
3.0
We have the right amount of analytical talent in our organization to
address critical business challenges.
2.9
I typically have all the information I need to make effective business
decisions.
2.9
We effectively leverage new and emerging technologies to address
business challenges.
3.0
End users can quickly explore data and create reports in an ad hoc
fashion without relying on experts or IT.
2.8
9
12. SAS White Paper
Our current computing infrastructure makes it easy to implement
new processes that are similar to existing processes.
2.8
When working on a major project or initiative, we do not usually
follow a specific methodology to complete the work.
2.7
End users are adequately trained in the software to analyze data.
2.8
Costs within my organization are transparent and well-understood.
2.8
Our employees get the training they need to leverage analytical
software.
2.8
Our product offerings and services have remained relatively
unchanged for a long period of time.
2.7
Most of our information technology is integrated well.
2.7
There are currently no processes in place to improve data accuracy.
2.7
We do not have specialized data sets for doing analysis.
2.4
Details Behind the Data
As explained above, the survey asked respondents how strongly they agreed with
statements related to their organizations’ information management processes and
practices. The detail of the questions for each metric is presented in the graphs below.
The percentages represent the level of agreement to statements related to each metric.
10
13. Assessing Your Business Analytics Initiatives
Governance
Overall Score: 3.17
My department systematically follows standard
methodologies or processes as a practice.
11%
Any deviations from established operational norms are
slow to be corrected.
19%
Data quality and data deviations are consistently
monitored.
Our current computing infrastructure makes it easy to
implement new processes that are similar to existing
processes.
When working on a major project or initiative, we do not
usually follow a specific methodology to complete the
work.
Strongly
disagree
Disagree
16%
59%
44%
18%
15%
59%
Neither agree nor disagree
42%
6%
33%
10%
32%
22%
16%
35%
17%
9%
27%
22%
35%
10%
48%
33%
27%
28%
55%
13%
35%
15%
25%
13%
42%
21%
19%
The reasons for internal hardware/software changes are
clear.
69%
43%
21%
13%
17%
52%
17%
There are specific service-level agreements between IT and
7% 16%
business areas.
We use standard processes and tools to compare actual
6%
results to goals.
% Agree
Agree
Strongly
agree
Figure 3: Metrics for governance.
Productivity
Overall Score: 3.09
We have dedicated resources to ensure the continuity of
business.
7%
Our computer systems (hardware, software, network) are
able to handle multiple peak periods of usage.
17%
Data security and authorization issues inhibit productivity. 5%
27%
7%
We do not have specialized data sets for doing analysis.
Strongly
disagree
Disagree
38%
14%
23%
Neither agree nor disagree
58%
20%
16%
34%
61%
12%
53%
46%
22%
37%
27%
4%
31%
24%
34%
24%
73%
10%
37%
25%
42%
13%
25%
51%
24%
Our employees get the training they need to leverage
6%
analytical software.
Most of our information technology is integrated well.
48%
18%
There is a discrepancy in the priorities of the IT department
and the needs of the business.
We are easily able to identify, understand and share
insights from complex sets of data.
% Agree
17%
24%
4%
28%
17%
Agree
13% 8%
21%
Strongly
agree
Figure 4: Metrics for productivity.
11
14. SAS White Paper
ROI
Overall Score: 3.04
% Agree
Key stakeholders generally agree on the benefits of new
4% 19%
information technology initiatives.
Upper management strongly advocates or promotes
5%
analytics.
The cost of inaccuracy in our organization is high and has a
significant impact on profitability, market share and the
ability to meet competitive pressures.
There is little correlation between the cost of our
4%
information technology and the value we receive from it.
Cost/benefit analysis for new initiatives is well
documented.
13%
We have the right amount of analytical talent in our
organization to address critical business challenges.
14%
32%
Costs within my organization are transparent and well
understood.
12%
34%
Our product offerings and services have remained
relatively unchanged for a long period of time.
11%
Strongly
disagree
Disagree
29%
7%
38%
26%
19%
9%
35%
5%
27%
22%
9%
21%
13%
Neither agree nor disagree
42%
31%
21%
46%
54%
15%
27%
21%
33%
25%
29%
25%
61%
69%
28%
31%
23%
19%
16%
45%
16%
Agree
32%
30%
Strongly
agree
Figure 5: Metrics for ROI.
Empowerment
Overall Score: 2.94
Resource constraints (people and infrastructure) at my
organization make completing work harder than it needs
to be.
11%
Employees at my organization are empowered to find and
solve problems.
My organization does not have self-service capabilities for
data reporting or analytics.
Decision making is enabled in a timely fashion.
10%
7%
35%
29%
33%
I typically have all the information I need to make effective
business decisions.
7%
End users are adequately trained in the software to
analyze data.
8%
Disagree
Figure 6: Metrics for empowerment.
12
33%
39%
Neither agree nor disagree
20%
15%
25%
21%
Strongly
agree
28%
21%
28%
28%
Strongly
agree
68%
40%
23%
17%
33%
28%
27%
End users can quickly explore data and create reports in an
ad hoc fashion without relying on experts or IT.
16%
71%
15%
53%
20%
It is difficult to explore our data for patterns or regularities.
Strongly
disagree
%
35%
36%
15%
6%
39%
10%
38%
13%
34%
6%
34%
4%
32%
15. Assessing Your Business Analytics Initiatives
Accuracy
Overall Score: 2.94
%
Most of our marketing efforts focus on either large
customer segments or definitions of group membership 5% 18%
rather than customized approaches or micro-targeting.
Imprecision negatively affects business outcomes and
decisions I have to make.
15%
10%
28%
Accurate information is delivered in a timely manner. 5% 15%
I am able to quickly identify key variables that influence or
6%
predict business outcomes.
A lot of costly business decision mistakes have been made
in the past at my organization.
The majority of data quality inconsistencies are identified
and addressed.
23%
There are currently no processes in place to improve data
accuracy.
Strongly
disagree
Disagree
17%
32%
57%
9%
29%
33%
11%
12%
46%
37%
39%
We often get inconsistent results when analyzing the same
6%
data source.
59%
45%
25%
16%
62%
19%
40%
24%
9%
19%
43%
36%
22%
43%
5%
26%
20%
Neither agree nor disagree
15%
13%
19%
Agree
8%
44%
41%
39%
27%
Strongly
agree
Figure 7: Metrics for accuracy.
Maturity
Overall Score: 2.81
We typically use the results of data analysis to improve our
processes.
16%
%
Work priorities change frequently in my job and are very
tactically focused.
We often cannot get the internal data we need from other
4%
departments, so we find and analyze it ourselves.
We could benefit from having easier access to statistical
consultants for some of the work we do.
Strongly
disagree
Disagree
64%
38%
22%
60%
21%
59%
18%
21%
10%
38%
16%
54%
43%
15%
29%
25%
Neither agree nor disagree
14%
31%
Agree
11%
54%
37%
24%
28%
13%
38%
21%
16%
We have difficulty analyzing data in a timely manner.
We effectively leverage new and emerging technologies to
address business challenges.
21%
20%
Analytics talent is too diluted across the company. 4% 19%
There is little statistical sophistication in our daily data
analysis.
43%
18%
14%
51%
25%
20%
28%
7%
45%
35%
Strongly
agree
Figure 8: Metrics for maturity.
13
16. SAS White Paper
Timeliness
Overall Score: 2.61
We need to react faster to market changes or
opportunities.
% Agree
It is time consuming and difficult to get analytical models
into production.
14%
Any new reporting or changes to existing reports take a
long time to develop.
16%
21%
It takes too long to get meaningful insights from data. 4%
23%
Strongly
disagree
Disagree
55%
20%
52%
9%
50%
12%
40%
8%
36%
41%
30%
28%
28%
27%
Data from other departments needed for reporting is
6%
received in a timely fashion.
20%
32%
21%
We are generally able to make good decisions in a timely
3% 17%
fashion.
62%
35%
18%
67%
22%
40%
20%
80%
26%
41%
17%
It takes too long to get data that is in a usable state. 5%
We are stifled in our decision making due to the volume
6%
and variety of data we have.
34%
46%
16%
28%
27%
31%
Neither agree nor disagree
Agree
Strongly
agree
Figure 9: Metrics for timeliness.
Effectiveness
Overall Score: 2.56
There are a significant number of issues at my company
6% 6%
that could benefit from better use of analytics.
We need to get more value out of the current technology
we have.
We are lacking feedback loops to ensure that internal
4% 13%
processes become more effective and efficient over time.
Our IT group suffers from lost productivity and time due to
ad hoc and one-off requests involving different data
elements.
We typically count on one specific person within our
department to do analysis, interpretation and deliver
results.
We have a formal feedback mechanism to help improve
processes and systems.
Strongly
disagree
Disagree
Figure 10: Metrics for effectiveness.
14
9%
25%
23%
Neither agree nor disagree
19%
35%
25%
18%
24%
35%
23%
31%
15%
36%
22%
Agree
67%
21%
40%
24%
84%
30%
37%
15%
88%
37%
47%
16%
13%
50%
38%
16%
My organization has analytical data sets that can be used to
5% 13%
support multiple initiatives.
There is a disconnect between IT and business or line-ofbusiness requirements.
% Agree
15%
10%
Strongly
agree
61%
59%
54%
46%
46%
17. Assessing Your Business Analytics Initiatives
Appendix
What was the approximate gross annual
revenue in US$ for your organization in 2012?
Less than $100
million
9%
$100 million to $499
million
32% SMBs
11%
(under
$1B)
$5 billion or more
44%
$500 million to $999
million
12%
68%
Enterprise
$1 billion to $4.9
billion
24%
What is your organization's primary industry?
Professional services
Manufacturing
Retail/Wholesale
2%
7%
9%
IT and technology
5%
Other industries
9%
Communications,
entertainment,
media and
publishing
5%
Health care,
pharmaceuticals and
biotechnology
18%
Education
10%
Government/Public
sector
9%
Financial services
22%
Energy and natural
resources
4%
15
