In a recent study by Dimensional Research of 1,011 development and QA professionals, almost every survey respondent cited that application quality is important, with 84% believing it is very or critically important. Despite this, findings revealed that 94% of teams still face challenges when it comes to conducting adequate QA. View the presentation to learn why organizations must prioritize automated testing and QA practices to deliver high-quality applications and increase customer satisfaction.
2. The State of Web Application Testing
A Survey of QA and Development Professionals
3. Research Goal The goal of this survey was to capture data on current
trends in web application quality.
Goals and Methodology
Methodology An online survey was sent to global databases of web
technology professionals. Questions were asked on a
wide range of subjects including current quality, impact of
quality issues, quality processes, future of quality
initiatives, and more. The incentive for participant was a
copy of the final report.
Participants 1,011 individuals with responsibility for QA or
development participated in the survey, representing a
wide range of roles, geographies, and company size.
4. Participant Demographics
Less than 10
employees
23%
10 - 100
employees
30%
100 - 1,000
employees
26%
More than 1,000
employees
21%
Company Size
North or South
America
41%
Europe, Middle
East, or Africa
36%
Asia Pacific
23%
Region
46%
47%
68%
0% 10% 20% 30% 40% 50% 60% 70% 80%
B2C (Consumer)
Internal (employee or partner use)
B2B (Business)
Type of Application End User
14%
34%
36%
40%
85%
0% 20% 40% 60% 80% 100%
Business stakeholder
Support
Vision and strategy
Quality assurance
Development
Responsibilities
5. Web application quality is critical
How important is the
quality of the apps you
are responsible for?
Critically important
34%
Very important
50%
Important
15%
Not important
1%
6. Quality problems can have dire consequences
What are the potential
business impacts of quality
problems with your apps?
1%
3%
30%
43%
51%
56%
59%
79%
0% 10% 20% 30% 40% 50% 60% 70% 80% 90%
Quality problems do not have a business impact
Other
Miss time to market advantage
Lost revenue
Impact employee efficiency
Incur costs to fix problems
Damage reputation
Impact customer satisfaction
7. Quality issues are the norm – not an exception
In general, how frequently do
business-impacting quality
problems occur with the apps
you have responsibility for?
13% 15% 25% 34% 4% 7%2%
0% 20% 40% 60% 80% 100%
It is an ongoing problem
Weekly
Monthly
A few times a year
Yearly
Less frequently than yearly
Never
8. Good QA is hard
What challenges does your
team face in doing adequate
quality assurance to prevent
critical issues?
6%
7%
21%
22%
22%
40%
45%
55%
0% 10% 20% 30% 40% 50% 60%
We don’t face any QA challenges
Other
Too easy to ignore defects found in testing
Development and QA teams are siloed and don’t
communicate well
Quality is not taken seriously
Testing is not integrated into the workflow
Aren’t able to sufficiently test across all browsers
and devices needed
Schedule pressures don’t permit adequate testing
9. When development schedules slip, testing suffers
Are planned testing
efforts ever compressed
because development
schedules slip?
Yes, always
30%
Sometimes, but
not always
56%
No, never
14%
10. Test automation practices lack maturity
How would you
characterize your team’s
current test automation
capabilities?
8% 17% 30% 25% 21%
0% 20% 40% 60% 80% 100%
We have a fully mature test
automation practice that has
automated everything possible
We do a great job with test
automation, but we could do a
bit more
We have automated some of
the most important tests but
have significant room to do
more automation
We do minimal automation
We do not do any test
automation
11. Hiring good test automation engineers is difficult
In your experience, which
type of professional is
harder to hire? Test automation
engineer
61%
Software
developer
39%
12. Cultural issues may impact quality
If a frontline development
or QA professional
discovered a potentially
business-impacting issue
the day before a major
release, is there a process
to flag an urgent issue
and stop the launch?
Yes, we have a
clear process
52%
In theory we
have a process,
but it’s not
clear how it
actually works
30%
No, there is no
process
18%
61%
56%
48%
22%
30%
34%
0%
10%
20%
30%
40%
50%
60%
70%
Executive Manager Individual
contributor
By Job Level
Yes, we have a clear
process
In theory we have a
process, but it’s not
clear how it actually
works
13. Employees are rarely held responsible for issues
In the past two years,
what has been the result
of business-impacting
quality issues on the
careers of team members?
44%
6%
15%
15%
18%
32%
0% 10% 20% 30% 40% 50%
No team members have had their careers
impacted because of quality issues
Other
Bonuses were not paid
Individuals responsible lost their jobs
Individuals responsible were demoted or not
advanced
Negative performance reviews were given
14. Compensation rarely tied to quality
Is your compensation
impacted based on the
quality of the applications
your team delivers (i.e.
bonus is paid or held
back)?
Yes
34%
No
66%
46%
33%
31%
54%
67%
69%
0% 20% 40% 60% 80% 100%
Executive
Manager
Individual contributor
By Job Level
Yes
No
15. Organizations are investing in web app quality
How do you expect your
team’s testing investment
will change in the coming
year?
29% 44% 23% 4%
0% 20% 40% 60% 80% 100%
Increase significantly
Increase somewhat
No change
Decrease
16. Many factors drive testing investment
Why will your team
increase investment in
testing in the coming
year?
n = will increase testing investment
29%
30%
35%
36%
38%
40%
41%
42%
0% 5% 10% 15% 20% 25% 30% 35% 40% 45%
Create competitive differentiation
Business stakeholders are emphasizing the
value of higher quality
Reduce overall costs
Response to a business-impacting quality issue
Need to address customer satisfaction issues
Just part of our overall development investment
increase
Need to improve time-to-market and better
meet release schedules
End users are demanding higher quality
22. Invest in Tools to Help Deliver High-Quality Web Apps
Sencha Test: Unit and Functional Testing Ext JS Apps
• Complete, cohesive, integrated testing
platform for Ext JS apps – incorporates
years of experience
• Deepest integration into Ext JS for
faster test creation & highest accuracy
• Execute tests on multiple browsers
simultaneously
• Works seamlessly with existing tests
• WebDriver integration enables faster
testing of multi-page apps – mimics
user behavior
22
23. Sencha Test Components
23
Test Studio
Test Authoring
Test Runner
Test Execution
Test CLI
Test Automation
Test Archiver
Results Reporting
And Archiving
24. Test Authoring: Sencha Test Studio
• Write unit and end-to-end functional tests in
JavaScript (Jasmine)
• Create tests with built-in test editor or in a
separate IDE – store them in SCM
• Quickly write & execute tests while coding
• Leverage pre-authored tests for faster
results
• Advanced APIs improve testing efficiency
by minimizing the lines of code in a test
24
25. Test Authoring: Targeting Elements
• Any testing tool must allow testers to select elements upon which to act
• Sencha Test helps you avoid brittle selectors and dynamic IDs
• These ”locator strategies” improve effectiveness and reduce maintenance
• Consistently target the right element through:
• Ext JS Component query
• DOM query (XPATH or element ID)
• A combination of the above
• Create powerful page objects that maintain locators in a central location for
consistency and easier maintenance
25
26. Test Authoring: Event Recorder and Inspect
• Automatically records events while
interacting with the system under
test
• Even as you navigate multi-page
apps
• Generate tests in Chrome and
Microsoft Edge, execute on any
browser
Event Recorder
26
• Helps you easily locate components
on the app under test
• Choose the best locator from a list
of all possible options
• Automatically insert API code into
your test or page object, using that
locator
Inspect
27. Test Authoring: External Libraries
• Leverage external libraries in test code
• Connect to databases
• Communicate with web services
• Consolidate common code
• Use like you would in any other JavaScript
code (e.g. require)
• Works for local and remote (WebDriver)
scenarios
27
NoSQL
Web
Services
28. Test Execution: Test Runner
28
• Run selected tests on any or all browsers,
local machine or browser farm
• Dramatically reduce testing time by
executing tests simultaneously on
multiple browsers through integration with
leading browser farms such as Sauce
Labs and BrowserStack
29. Application Lifecycle Management: JIRA
• Reduce the effort of tracking test failures
and defects in JIRA
• Manage tests, results and defects in Sencha
Studio
• Failed tests may be marked as defects,
easily exported to specified JIRA project
• Sencha Studio tracks status of bugs to
minimize duplicated reports
29
SenchaTest
30. Test Automation: Sencha Test CLI (Command Line Interface)
• Maximize the efficiency of your testing
through automated test runs
• Launch tests within the CI system as soon as
application changes and tests are checked
into the source control repository
• Out of the box integration with TeamCity and
Jenkins
30
code
testreview
31. Test Execution: Test Archiver
31
• Review results from automated and
manual test runs
• Summary level results & drill into the
detail of failed tests
• Out of the box integration with Istanbul
enables users to identify and rectify
code coverage gaps
• View % coverage at the Statements,
Branches, Functions and Lines levels
32. Visual Screen Comparison: Test Archiver
32
• Minimize the time spent
verifying images on hundreds
of screens
• Review images that are
captured from previous test
runs and compare to each
subsequent test run
• Specify screen sizes to
improve accuracy of results
33. Sencha Test System Overview
33
Sencha Test Studio
Sencha Test Archiver
Sencha Test CLI
Browser Farm
CI Server
Local Workspace
Local Browsers
App Server
PROXY
PROXY
MANAGES
USES
USES
USES
STORES
RETRIEVES
37. Final Recommendation: Drive Accountability
• What gets measured/rewarded gets done
• Ensure entire team is held accountable for app quality
• Quality is not just the job of the QA team
37
38. Q&A
• Try Sencha Test for Free: https://www.sencha.com/products/test/evaluate/
• Contact us for help with your test strategy and automation needs:
https://www.sencha.com/company/contact/
38
To help you address many challenges with web application testing, you need to understand where you are on the Test Automation Maturity model. We frequently use this as part of Web App Testing Healthc heck with our customers to understand where they are today, where they hope to be, and work together to create the milestones to achieve those goals.
As Diane mentioned, many organization’ test automation practices lack maturity. Only 8% stated that they had mature processes and had automated everything they could.
We call Fully Automated the highest level of Test Automation Maturity. At this level, organizations have mostly automated testing and fully integrated Unit tests. They run tests locally and on fully merged code. They have testing fully integrated into the workflow and use the SCM for everything and the build runs at every merge.. Unlike some organizations who said in the survey that one of their challenges is siloed communications between Dev & QA, these organizations have integrated Dev and QA teams.
The bulk of respondents, more than 70% were what we would call partially automated. They have some automation in place, perform basic unit testing which they run locally, use the SCM for application source code. Testing is not fully integrated into their lifecycle. Only the dev team uses CI.
Finally, 21% of respondents said their testing was mostly manual. At this stage we typically see just manual UI testing with no unit testing. Teams are siloed and testing is not deeply incorporated into the workflow.
Test automation can provide huge benefits to an organization. If you think about it from the impact of just one tester. The difference in scale and efficiency of automation is exponential. With the same amount of effort, automation enables the tester to have a far more dramatic impact on quality
If we drill into this a little deeper, you can see that the time taken is reduced drastically by automated testing
Big gap between manual and automated testing as the release gets bigger
Upfront investment is justified given the long term benefits
Even when a project is in maintenance mode, teams still have to run tests to make sure that changes elsewhere doesn't break this application. This is called regression testing
Automated testing can be a boon to reduce cost for this type of testing
Look at the difference noted by dotted line
As you heard from Diane, many companies are investing in web app quality and we’ve seen that drive demand for Sencha Test.
Sencha test is the most comprehensive unit and functional testing solution for Ext JS applications. By providing the deepest integration into Ext JS, Sencha Test ensures the highest test accuracy. It enables tests to execute on multiple browsers simultaneously which is key to getting tests executed fast – especially when teams are under schedule pressures.
Sencha Test works seamlessly with existing tests and WebDriver integration enables faster testing of multi-page apps and mimics user behavior.
Sencha Test has for main components that are all designed to work together seamlessly to provide a comprehensive and integrated test environment
Sencha Test Studio is the graphical user interface that allows users to write Jasmine tests in a built-in Sencha Test editor. Tests are written using JavaScript and are stored in the teams’s preferred source control management system. Developers can create tests directly in Studio or in a separate IDE and execute the tests immediately
The event recorder is a mechanism which enables users to create quick, high level tests. Users can automatically record events while interacting with the system under test = even as users navigate multi-page apps
Users can generate test in Chrome or Edge and execute them on any browser.
The Inspect feature in Sencha Test is the recommended solution for test authoring. It enables users to locate components on the application under test and construct meaningful locators. Users can generated the API code and add actions in the editor to complete the test creation process.
Another key capabilities with test authoring is that users can leverage external libraries in test code. Teams can use node modules to expand the scope of testing by connecting to databases. Users can perform tests by communicating to the web service APIs without invoking the application.
The external libraries functionality works for both local and remote, (WebDriver) scenarios.
The test runner allows users to run tests on any or all browsers, local machine or browser farm. Teams can run end-to-end WebDriver tests using the built-in Selenium server and embedded Chrome browser. You can dramatically reduce testing time by executing tests simultaneously using the built in Selenium grid or leading browser farms including SauceLabs. Again this can be a key capability as 56% stated that planned testing efforts can become compressed when development schedules slip.
As Diane mentioned, one of the challeges companie face in doing adequate QA is that the tools are not integrated into the worflow. Sencha Test comes pre integrated with Jira. This integration can also help eliminate siloed communication between Dev and QA as all test failures and defects are captured in Jira and visible to the entire team. Users can manage tests, results and defects in Sencha studio. Failed tests may be marked as defects and easily exported to a specified JIRA project. Sencha Studio tracks the status of bugs to minimize duplicate reports.
Sencha Test can help teams overcome some of the quality challenges because it easily integrates with the workflow. Tests can be automatically launched within the CI system as soon as application changes and tests are checked into the source control repository. It has built in integrations with TeamCity and Jenkins. Users can also leverage any build tool that can invoke a command-line utility to run tests via Sencha Test CLI.
Sencha Test Studio Client manages the local workspace
Browser proxies to Sencha Test Runner
Browser requests are routed to the configured app server
Test Runner uses Browser Farm, which can be configured in Studio
CI Server schedules tests and invokes Test Runner on a “build agent”
Test Runner stores results in Studio Server and reports them to the CI
Studio Client provides interface to manage test issues
Assessment: Evaluate current test, build and dev processes. Understand the web app.
Initial POC: Create Reference Test Environment, Implement Tests, Review test cases with dev team
Test Infrastructure Setup: Customized reference environment, Browser farm setup,
Connect CI to your build environment
Test Development: Implement test strategy, Continue Test Creation and deepen test coverage, Create datasets needed for complex automated testing, Work in parallel with app dev team
Long-term sustainment: Scale test automation team to match development velocity, Ensure production application continues to operate correctly with each release, Ensure new browser versions do not introduce regressions, Develop new test cases as new functionality is created