MTLM Visual Studio 2010 ALM workshop - day1

Testing Practices with ALM tools

GOAL general course:
Know the value of MTLM and know how to use its capabilities …

GOAL deep dive:
Tune the value of MTLM, its capabilities and the processes it supports…

Lot of discussions
Agenda Day 1 and hands-on lab’s
Start Topics
9:00 Introduction, environments setup
9:15 introduction by the students
9:45 Application Lifecycle Management and Visual Studio General
9:55 Test Planning
10:15 Managing, controlling the testing effort, test plans, test suites and test
configuration
10:45
11:00 Specify test cases and corresponding capabilities of MTM
11:45 Test execution, fast forward, reporting, olap cub
12:00
12:45 Bug workflow, infrastructure configuration for data diagnostic adapters
13:30 Test impact and recommended tests
14:15
14:30 Exploratory testing and create test from bug
14:45 Test Controllers and Agents
15:45 Test Environments in MTLM

Environments setup
TFS supported operating system

Installed:
• Team Foundation Server with Basic configuration
• Visual Studio 2010 with SP1
• Microsoft Test Manager 2010 (comes with VS10 Ultimate)

You get an USB with:
• Demo applications (we create a Team Project for them, see lab 1)
• Microsoft Test Agents.msi (we install this together)
• Hands-on lab documents
• This deck, talk through and useful information

Lab 1
Create one team projects for the demo applications

Timebox: 15 minuts

Clemens Reijnen
Sogeti Netherlands | Management Consultant

http://www.ClemensReijnen.nl
@ClemensReijnen

Start Topics
9:55 Test Planning
configuration
10:45
12:00
14:15

Application Lifecycle
Management

GOAL:
Understand the term ALM, know it’s value and Visual Studio’s ALM support

Suite of Application Lifecycle Management tools

http://msdn.microsoft.com/en-us/library/fda2bad5.aspx

application lifecycle management
http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx

work together
__with work packages
__on alm artifacts

Testing Lifecycle

See also on www.tmap.net:
- The Test Lifecycle Model
- The 4 essentials

http://msdn.microsoft.com/en-us/library/bb385832.aspx

Testing Lifecycle
Test Activities Planning…

See also on www.tmap.net: Planning Phase

VS2010 Process Support with Process Templates
Multiple process template

Artifact types

Work items
Reports
Workbooks
Dashboards
Metrics

Work Item Type Flow

User Story

Testing processes in the Lifecycle

Process template with work items, flows,
links, queries and more
support | drive | enforce
a way of work

Lab 2
Create user stories (2 or 3)
and development and test tasks (optional)
for the employee application.

Timebox: 5 minuts

Create your own – advanced –
Query Types

create your own
Report Wizard
Excel Pivot


See also on www.tmap.net: Planning Phase

Microsoft Test Manager 2010
The UX

Test Case Management

See also on www.tmap.net: Control Phase

Test Plans
Test Suites
Test Configurations
Organize the testing effort in a Test Execution Plan
Create dynamic test suites for changing selection criteria
Define various test configurations and share automation across those

Test Plan Organization
Naming of test plans for agile environments:
Iteration number and name and one extra test plan for end
to end tests.

Test Suites Organization
Naming of test suites for agile environments:
Suite per user story (requirement based) and one additional
test suite for regression tests.(query based?)

(test cases are copied to the regression test suite)

Iteration 1
Create test plan for iteration and a ‘end to end’
test plan.
Create requirement based test suites which
hold test cases for that suite.

Iteration 1
Create two additional suites, one for
regression sets, which will be taken along
during the project and one specific for
automation.

Copy Test Cases

Copy a Test Case Copy the reference
to a Test Case

ID 21 ID 21

ID 22 ID 21

End iteration 1, some important test case are
promoted to the ‘end to end ’ test plan.

Iteration 2, test suites are created for the
requirements and the regression test suit is
copied (link or hard?)

During the execution of iteration 2 the
regression test suit grows and an automation
suit is added.

Again important ones are promoted to the
‘end to end’ test plan

Lab 3
Create Test Plan, Test Configurations and make
a test suite organization.

Optional: use TCM.exe to list the lab results

Timebox: 10 minuts

Share MTM Artifacts

mtm://clemenes-pc:8080/tfs/defaultcollection/p:MSF/Testing/testplan/connect?id=19

mtm://<server name>:<port>/<tfs vdir>/<Collection name>/p:<project name>/<center
group>/<group specific>
Open Test results
Open Test Cases (or any work item)
Open Test Runs
Open Test Plans
Connect to test plans
mtm://server:8080/tfs/ProjectCollection/p:Project/testing/testcase/open?id=67

http://blogs.msdn.com/b/vstsqualitytools/archive/2010/02/09/sharing-content-in-microsoft-
test-manager.aspx

Start Topics
9:55 Test Planning
configuration
10:45 BREAK
12:00
12:45 BREAK TILL:
Bug workflow, infrastructure configuration for data diagnostic adapters
13:30
14:15
Test impact and recommended tests 11:00
14:45 Test Controllers and Agents (extra time to finish the labs :)

Start Topics
9:55 Test Planning
configuration
10:45 BREAK
12:00
14:15

Test Case Specification

See also on www.tmap.net: Specification Phase

Test Case
Shared Steps
Test Data Iterations
Link test cases and requirements to provide requirement traceability
Data drive test cases with different parameters
Refactor common shared steps in test cases

Test Case
Naming of test case:
Purpose:
• Finding the right test case
• Common understanding
• Intention from the title
• Scan test plans more effectively and efficiently

Proposed solution
#1 make use of linked work items
#2 maximal length of the title 20 characters
#3 format <functionality identifier> <action> <additional information>
#4 make useful queries

Test steps
Make them descriptive
Not every step is a validation step
Use the step attachment

Test Data Iterations
Data drive test cases with different parameters

Shared Steps
Refactor common shared steps in test cases
Reduce test maintenance by sharing test steps across test cases

Shared Steps
Shared Step candidates?

• REUSE – Many steps, reused for many test cases . Benefit: time, retyping, boring work.
• EXPECTED CHANGES – for example environment variables change [URL]. Benefit: Time, test cases
which need this step are updated automatically, also action recording.

• PRE- CONDITIONS – bring the system under test to a certain state and execute the different
verifications. Benefit: beside time, you are certain the application under test is in the same state for every test case

• BACK TO INITIAL STATE - after test execution bring the environment back to the initial state for
new test cases. Benefit: beside time, you are certain the application under test is always in a clean state for new test
cases.

Shared Steps
what about parameters
and expected result in
shared steps?

Lab 4
1. Specify multiple test cases.
2. Add some test data iterations.
3. Create shared step and reuse it.

Optional:
play with the parameters and expected result
steps in the shared steps…

Timebox: 15 minuts

Test Execution
Test Runner

A test that you will want to run multiple times.
Different manual tests that contain common steps.
Verifying a bug is fixed by using a manual test.

Test Case Execution

See also on www.tmap.net: Execution Phase

View Test Results Using Microsoft Test Manager
1: from the ‘Run Tests’

2: from the ‘Analyze Test Runs’

From Analyze Test Runs
Test Results for multiple Test Cases

From Run Tests
Test Result for a Test Case

From TCM
run /export id:239 /resultsfile:”c:runresult”

Share with mtm://

mtm://clemens-pc:8080/tfs/defaultcollection/p:MSF/Testing/testrun/open?id=292

Lab 5
Execute the created test cases, view the results

Optional: run them from TCM.exe

Timebox: 10 minuts

• POINT COUNT TREND - I want to see the results for test in a particular test run
• RESULT COUNT - I want to see trend report for my tests regardless of build or plan
• RESULT COUNT TREND - I want to see trend report for my tests regardless of build or plan
• TEST CASE COUNT - I want to see the number of test cases that have been planned for automation

Microsoft Test Manager Test Artifact
organization

Test Plans and TFS items
MTLM Client

Server
TCM Server TFS

Plan “Iter Plan “Iteration 2” User Story 2
1”  Suite US1
Suite US2
Test Case A
Test Case B Test Case A Task X
Plan “Iter Test Case C
3” Test Case D Test Case B
Task Y
Test Case C
Task Z
Test Case C

Lab 6
Create excel pivot table to list test points per
Tester.

Timebox: 10 minuts

Start Topics
9:55 Test Planning
configuration
10:45 BREAK
12:00 Lunch
12:45 LUNCH TILL:
13:30
14:15
Test impact and recommended tests
Break
12:45

Start Topics
9:55 Test Planning
configuration
10:45 BREAK
12:00 Lunch
14:15 Break

Controlling the testing effort
BUG’s

Diagnostic Data Adapters
File a Bug
File an Exploratory Bug
Verify a Bug

File a Bug
Submit a Bug using Test Runner

Ways to file a Bug

Submit a Bug from the Test Runner
Submit a Bug from the Verify Bugs View
Submit a Bug from Test Results for a Test

Verify a Bug
Verify a Bug Fix Using Microsoft Test Manager

The Bug Flow
Link build 2 test
plan. T Execute File Bug Verify fix Close or
Configure Data E Manual by using reopen
collectors. S Tests playback Bug
T

Build
definition is
the
intergration

Setup build and
deploy D Resolve Checkin
E Bug Build
V Deploy

Lab 7
Create a build

File a bug
Resolve a bug
Verify a bug

Timebox: 15 minuts

Bug Reports
Bug Status Report
Bug Trends Report
Reactivations Report

Test Impact
Assign build 2
test plan. T Execute Assign Evaluated and reset
Execute
Execute
Configure Data E ‘passed’ ‘NEW’ build tests impacted by
‘passed’
‘passed’
collectors. S Tests 2 Test Plan code churn
Tests
Tests
T

Build
definition is
the
intergration

Setup build and
deploy D Make Checkin
E code Build
V changes Deploy

Lab 8
Attach build to test plan
Execute tests

Make code change
Kickoff build

Attach new build to test plan
Evaluate impacted tests

Timebox: 15 minuts

Iteration N
Implement
D US2
Sprint
E plan Implement Fix bugs
V User Story ‘1’
Fix bugs
(US1)

Build 1 Build 2 Build 3 Build 4 Build 5 Build 6 Build 7

Regress
Write tests Test US1 &
T Verify fixes impacted
for US1 file bugs
E Sprint tests
S plan
T for US2 Verify fixes
file bugs

User Stories
Key Value: Requirements Planning
and corresponding
Implement
D
E
Sprint
plan Implement
US2
done dev tasks.
Fix bugs
Fix bugs
(US1)

Create test tasks
Build 1 Build 2 Build 3 Build 4 and 5
Build assign to 7
Build 6 Build

someone else.
Regress
for US1 file bugs
E Sprint tests
S plan
file bugs

Execute the Test
Specification Tasks
Implement
D
Sprint US2 and other test
(US1)
tasks
Fix bugs


Regress
for US1 file bugs
E Sprint tests
S plan
file bugs

Execute the Test
Implement
Execution Tasks
D
E
Sprint
Implement
US2 and file bugs (if
Fix bugs
plan
(US1) any)
Fix bugs


Regress
for US1 file bugs
E Sprint tests
S plan
file bugs

Key Value: Virtuous Bug Cycle
Dev Implement
D US2
solves
Sprint
the (US1)
Fix bugs

bugs.

Test
Regress
Verifies WriteUS1
T
for
tests Test US1 &
file bugs
Verify fixes impacted
E Sprint tests
tests
S plan
file bugs

Key Value: Virtuous Bug Cycle
Check Implement
D US2
Results
Sprint
Fix bugs
(US1)


Regress
for US1 file bugs
E Sprint tests
S plan
file bugs

Start Topics
9:55 Test Planning
configuration
10:45 BREAK
12:00 Lunch
12:45 BREAK TILL:
13:30
14:15
Test impact and recommended tests
Break
14:30

File an Exploratory Bug
Submit an Exploratory Bug using Test Runner

PRO:
- Reuse scenario knowledge business user
- No typing detailed test steps
- Easy to parameterize
1 Record scenario by business user with - Writing standards for test steps
MTM on application under test. Action - MTM capabilities in step 3. (bug, codedUI,
recommended test etc …)
recording is captured. Most benefit during step 2 (test case writing),
benefit in step 1: run once in common situation.
Benefit in step 3: common MTM benefits (record
playback, fast forward, parameters etc) and
benefit of well written steps.

2 Create test steps (with parameters) in
MTM by using the action recording.
CON:
- Manual actions and tests.
- Test steps text are technical names
- Action recording (step 1) must be clean.
- Unknown what is validated, during step 2, need
additional knowledge.
- The business user must think of the important
scenarios
3 Save test case and execute it by a tester
The action recording during step 1 can have
on application under test, by using MTM. unnecessary steps, these can be removed during the
execution (1) or during the specification (2).

Lab 9
Create an exploratory bug and create a test case from it

Timebox: 10 minuts

Test Infrastructure

Where and how to run the automated test?

Test Environments
See also on www.TMap.net:
- Checklist "Intake Test Environment“
- Checklist "Test Environment“
- Infrastructure Phase

Test lab
Visual Studio 2010 Lab Management

Where and how to run the automated test?
physical test environments
virtualized test environments

physical test virtualized test
environments environments
MTLM Client MTLM Client

Physical
Test Client TFS 2010 System Center Virtual
TFS 2010
Machine Manager
(SCVMM)

Hyper-V Library
Physical Host Shares
Test Client VM
VM
share

share

Build
application
physical test environments
versus
virtualized test environments

Provision
Run tests
‘clean’ env

Deploy
application

Lab for Physical
Environments, the
different flavors…
Visual Studio Team
Foundation
Server

Test Manager
Build Server Test Agent
Test Agent
Test Agent

Configurations for automated test execution:

A

Flavor A: Execution from VS2010…
Real developer test / test automation dry run
(Triggered by: right mouse click – run tests, in test project)
http://msdn.microsoft.com/en-us/library/dd286580.aspx

No additional configuration needed*
* Data Diagnostic Settings in the Local.Testsettings file are configurable.

Pro: Con:
Easy setup No collection of test results in TFS
Debug-able test automation Run on developer environment


B

Flavor B: Execution during build with Build controller…
not recommended, strange to run UI tests on build
Part of Build Verification Tests
(triggered by: build)

Set the Build Service to run in interactive mode
http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
Configure the build to run the tests
http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType
* Data Diagnostic Settings in the *.Testsettings file are configurable.

Pro: Con:
Easy setup No collection of test results in TFS
Test results in build result Build Controller needs to run in interactive mode
Tests are executed on build environment
Run on build environment


C

Flavor C: Execution during build with Test controller… B
Preferred configuration above flavor
Part of Build Verification Tests on multiple test agents
(triggered by: build)

Configure Test Controller (don’t register it with a project collection )
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers
Configure Test Agents on clients (interactive mode)
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents
Configure *.Testsettings file in solution to use Test Controller and Test Agents
Configure the build to run the tests (see B)

Pro: Con:
Test run on test environments No collection of test results in TFS
Tests run on multiple environments Harder to configure
Test Results in Build result Need for specific test client environments
Test Settings from VS


D

Flavor D: Execution from Microsoft Test Manager…than BVT
Other type of test
Part of Regression Tests
(triggered by: MTM user, right mouse click on test case, run)

Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
http://msdn.microsoft.com/en-us/library/ee390842.aspx
Associate CodedUI test with WI Test Case from VS.
http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html

Pro: Con:
Test run on test environments Manually triggered by Tester
Tests run on multiple environments Hard to configure
Test Result in MTM and TFS Need for specific test client (same as MTM?)
Test Settings from MTM


E

Flavor E: Execution from MTM duringPreferred configuration above flavor C
Build…
Part of BVT Flavor D and E can be configured together
(triggered by: Build)

Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
Associate CodedUI test with WI Test Case from VS.
Create Build task to run TCM or MSTEST task for Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx

Pro: Con:
Test run on test environments Hard to configure
Tests run on multiple environments Need for specific test client (same as MTM?)
Test Result in MTM and TFS
Triggered by build

Lab 10
Install and configure the test controller and test agent

http://www.microsoft.com/downloads/en/details.aspx?FamilyID=a3216d2a-0530-4f6c-a7c9-0df37c54a902

Timebox: 20 minuts

RECAP

Test Case Management
RECAP

Start Topics
DAY 2
9:00 Recap
9:15 Create Basic CodedUI, from Test Case and Manual
9:45 UIMap and customize the code for optimization
10:15 Data driven Tests and assertions
10:45 break
11:00 Troubleshoot CodedUI, Common Practices and questions
11:30 Configure the build to execute the CodedUI tests, test settings from VS2010, Execute from VS2010 with test agents and analyze
the results
11:30 Associate test automation with test case, execute automated test case from MTM, configure the build to run from the build within
MTM and analyze the results
12:30 break
13:15 MTLM, Scrum and other project Methodologies
13:45 Lab Management - Virtualized Test Environment and execution Demo
14:15 Test info in the OLAP Cube
14:30 Extensibility - test Scribe, TCMIMport
14:45 Create your own Data Diagnostic Adapter
15:15 MTLM usages with Azure projects
16:00 Closing

MTLM Visual Studio 2010 ALM workshop - day1

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (20)

Similar a MTLM Visual Studio 2010 ALM workshop - day1

Similar a MTLM Visual Studio 2010 ALM workshop - day1 (20)

Más de Clemens Reijnen

Más de Clemens Reijnen (20)

Último

Último (20)

MTLM Visual Studio 2010 ALM workshop - day1