MTLM Visual Studio 2010 ALM workshop

BETA  work in progress, I add every training new material and tune current
material.

Sogeti Netherlands

Microsoft
Test and
Lab
Manager
Training
Guide
A description and short reference guide covering Microsoft Test and Lab
Manager training course “Testing Practices with ALM tools”.

Clemens Reijnen -- VERSION 0.4
8/8/2011

The training course
This guide is an extract from the two and three day course provided by me. It spans the
complete testing lifecycle and the tool usages. It will look at the infrastructure
implications and testing practices in formal and in agile teams. But, the main focus stays
on the usages of the Microsoft Visual Studio testing tools, the knowledge you need to get
starting with it, the practices you must have to work with it in real live and how you can
bend the tools, with extensibility and normal use to your team needs.

The course and this guide is work in progress. It is not a testing training (I expect you
already have testing knowledge), if you need that test process information I refer to the
TMap website from Sogeti where you can find tons of information. This training guide
follows the TMap testing lifecycle.

Contents
The training course .......................................................... 1
Application Lifecycle Management ....................................... 4
Test Lifecycle - Planning Phase ........................................... 8
Work items ........................................................................................................ 9
Lab 1: Create team projects. ..............................................................................10
Lab 2: Create user stories and tasks for the employee application. ..........................20
Work Item Queries ............................................................................................21
Reports Basic ....................................................................................................24

Microsoft Test Manager .................................................... 28
Test Lifecycle – The Control Phase I ................................... 29
Test Plans.........................................................................................................29
Test Configurations ............................................................................................32
Test Suites .......................................................................................................32
Lab 3: Create and configure test plan. .................................................................34
TCM.EXE ..........................................................................................................36

Test Lifecycle – Specification Phase .................................... 37
Test Cases ........................................................................................................38
Shared Steps ....................................................................................................41
Test Data Iterations ...........................................................................................42
Lab 4: Create test cases. ....................................................................................43

Test Lifecycle – Execution Phase ........................................ 46
Test Run ..........................................................................................................47
Test Runner ......................................................................................................48
Lab 5: Execute the created test cases. .................................................................50
Test Case Execution Results................................................................................51

Test Lifecycle – Control Phase II ........................................ 56
File a Bug .........................................................................................................57
Verify a bug ......................................................................................................60
Lab 6: file a bug. ...............................................................................................61
Bug Reports ......................................................................................................67
Test Impact ......................................................................................................67
Lab 7: test impact. ............................................................................................69

Create test from bug ..........................................................................................70
Lab 8: Create an exploratory bug and create a test case from it. .............................71

Test Lifecycle - Infrastructure Phase .................................. 72
On Physical Environments with Microsoft Test Manager ..........................................74
Lab 9: Install and configure the test controller and test agent .................................79
Lab 10: Setup Flavor D ......................................................................................79

Test Lifecycle - Completion Phase ...................................... 80
Test automation with Visual Studio 2010. .............................................................81
When is test automation the investment worth? ....................................................83
The investment levels explained. ......................................................................84
Lab 13: Create CodedUI .....................................................................................88
The UIMap ........................................................................................................89
Data driven CodedUI tests ..................................................................................90
Lab 14: Execute the CodedUI during the build from Visual Studio ............................91
Setting up Flavor C .........................................................................................91
Lab 16: Execute the codedUI from MTM ...............................................................93
Setting up Flavor D .........................................................................................93
Lab 17: Execute the codedUI during the build from MTM ........................................95
Setting up Flavor E .........................................................................................95
On Virtualized Environments with Lab Management and SCVMM ..............................96

Extensibility ................................................................... 99
Tools ...............................................................................................................99
Custome Data Adapter ..................................................................................... 100

Azure and test automation ............................................. 101
Test Manager and an agile process .................................. 101
Intelitrace .................................................................... 101
Appendix ..................................................................... 106
proposed training agenda 2 day training............................................................. 106
Student and trainer Environment ....................................................................... 107

Application Lifecycle
Management

You must understand ALM otherwise you will interpreter the test tools and connected
tools completely wrong and don’t see and use its full value.

IT organizations spend billions of dollars a year on development life-cycle tools that don’t
play well together. The result? For most shops, application life-cycle management (ALM)
— the coordination of development life-cycle activities — is still largely a manual process.
Today’s ALM suites don’t oﬀer much support for ALM beyond what can be accomplished
through brittle tool-to-tool integrations. But tomorrow’s ALM platforms will do much
better by providing common services to practitioner tools. These solutions will be easier
to implement, maintain, and employ. And at the end of the day, they’ll enable
development organizations to build better software.
From Forrester:
http://i.i.com.com/cnwk.1d/html/itp/Serena_Life_Cycle_Management.pdf

(tool vendors ALM proposition)
http://www-01.ibm.com/software/rational/alm/
http://www.microsoft.com/visualstudio/en-us/solutions/management
https://h10078.www1.hp.com/cda/hpms/display/main/hpms_content.jsp?zn=bto&cp=1-
11^37618_4000_100__

Software development and all roles involved are supposed to strive for one common
goal: providing added value to the customer. It does not matter what kind of application
needs to be implemented. All tasks and roles need to reach for this one single goal. The
synergy between roles, processes, and tools is also commonly known as Application
Lifecycle Management [ALM]. Notions like accountability, governance, and compliance
are regularly used when talking about ALM. But all these notions refer to the simple
notion of cooperation.

There would be a better mutual understanding if developers and testers would have more
insight in each other’s work and challenges. Annoyances disappear and people work
together on a common cause. Visual Studio Team System 2010 will better support this
cooperation. By giving the tester aids for performing his/her tasks and making him/her a
first-class-citizen within the Team Foundation Server, he/she has and gives more insight
in tasks to be done, progress on these tasks and what steps still need to be performed.

Aside from this insight, which often means quite a change of culture, the tester has
better insight in and access to the ALM-Artifacts from other roles. For example, use-case
and activity diagrams from business and information analysts and other diagrams from
designers and architects are easily accessible through Visual Studio Team Architect
Edition. Cooperation not only occurs by giving insight in work activities, but also by
working on common products.

http://www.clemensreijnen.nl/post/2009/04/21/Testing-in-the-Application-Lifecycle-
with-Visual-Studio-2010-Test-Edition.aspx

http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx

When talking about Application Lifecycle Management [ALM] terms like accountability,
governance and compliance are used. All of them refer back to “working together”, how
do we work together during the application lifecycle. ALM is not about tools, it’s about
working together. Working together seamless and flexible while staying in control, being
measurable and responsible. All the roles in the application lifecycle have a part in this
collaboration effort. Tools can help but isn’t’ the core driver.

There are lots of examples of wrong interpreted business requirements, miss
communication between development en test [Rob describes a very real example],
applications which won’t run in production, operations who don’t understand the
applications. All of them resulting in more work, more faults, more costs or even only
costs and no application because the project was unplugged. Most of these projects
faults, these extra costs are a slip in communication.

Having a strategy how people have to collaborate within the lifecycle is one important
piece of Application Lifecycle Management. Most organizational parts have already some
kind of process / methodology in place. Having an approach how to share information
and share ideas from their own role point of view through the other roles is a key success
factor for lowering the costs and raise the business value of IT investments.

Tools can help with this goal. Having gear in place which supports and stimulates
collaboration is a driver for a successful Application Lifecycle Management. But without a
plan, how people should collaborate and communicate, tools are useless.

http://www.clemensreijnen.nl/post/2009/02/13/Visual-Studio-Team-System-2010-
e28093-Episode-3-The-Lifecycle.aspx

Application Lifecycle Management and a Collaborative Culture.

Creating new modes of collaboration supported by technology can only be done by
addressing the human aspect. More specifically, we need to address some of the worries
and obstacles people encounter when collaborating using technology.
The three most important concerns are:
Trust. Trust is a condition for social interaction. People will only work with people,
companies, tools and information they know they can trust. Before we can expect
collaboration to take off online, there must be a way for people to get this “trust.” And a
topic closely associated with trust when it refers to people is Identity.
Collaborative culture. If one individual is the greatest collaborator in the world, he or
she is probably not getting anywhere. Only when all people involved are part of the same
collaborative culture will new levels of creativity and productivity be reached. A
collaborative culture consists of many things, including:
 Collaborative leadership;
 Shared goals;

 Shared model of the truth; and
 Rules or norms.
Reward. Changing the way people work takes effort, so it must be clear for the parties
involved what they will gain, at a personal level, from collaborating in a new way.
Surprisingly, a “reward” for successful collaboration is most often of a non-financial
nature.
“The ways that people work together shift over time, which can affect your culture of
collaboration. More important, the introduction of collaboration technologies can also
change the culture of collaboration. If handled properly, the tools and the culture will
coevolve.”
Dennis Kennedy

http://www.clemensreijnen.nl/post/2009/02/20/Application-Lifecycle-Management-and-
a-Collaborative-Culture.aspx

The TMap life cycle model is used both in the creation of the test plan and in the
execution of the other activities in the test process.

Life cycle model The life cycle model is a generic model. It can be applied to all test
levels and test types and used in parallel with the life cycle models for system
development. In the TMap life cycle model, the test activities are divided across seven
phases: Planning, Control, Setting up and maintaining infrastructure, Preparation,
Specification, Execution and Completion. Each phase is split up into a number of
activities.

In the Planning phase, the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment. This is laid down in
the test plan. In the Control phase the activities in the test plan are executed, monitored,
and adjusted if necessary. The Setting up and maintaining infrastructure phase aims to
provide the required test infrastructure that is used in the various TMap phases and
activities. The Preparation phase aims to have access to a test basis, agreed with the
client of the test, of adequate quality to design the test cases. The tests are specified in
the Specification phase and executed in the Execution phase. This provides insight into
the quality of the test object. The test assignment is concluded in the Completion phase.
This phase offers the opportunity to learn lessons from experiences gained in the project.
Furthermore activities are executed to guarantee reuse of products.

http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance
_and_system_tests/index.jsp

Business Driven Test Management
http://www.clemensreijnen.nl/post/2010/06/28/Business-Driven-Test-Management-with-
Microsoft-Test-Manager.aspx

Microsoft Visual Studio 2010 Ultimate is the comprehensive suite of application lifecycle
management tools for teams to ensure quality results, from design to deployment.
Whether you're creating new solutions or enhancing existing applications, Visual Studio
2010 Ultimate lets you bring your vision to life targeting an increasing number of
platforms and technologies—including cloud and parallel computing.
(from http://www.microsoft.com/visualstudio/en-us/products/2010-editions/ultimate )

Test Lifecycle - Planning Phase

Planning phase of the Testing Lifecycle.

The life cycle model is a generic model. It can be applied to all test levels and test types
and used in parallel with the life cycle models for system development. In the TMap life
cycle model, the test activities are divided across seven phases: Planning, Control,
Setting up and maintaining infrastructure, Preparation, Specification, Execution and
Completion. Each phase is split up into a number of activities.

In the Planning phase, the test manager formulates a coherent approach that is
supported by the client to adequately execute the test assignment. This is laid down in
the test plan.

This is where the test organization is going to work together and on the same level as
the other ALM roles to plan the effort to realize the business needs with a proven (risk
driven) quality.

More reading: http://eng.tmap.net/Home/TMap/The_4_essentials/index.jsp

Work items

Planning activities for all the ALM roles are supported by Visual Studio and Team
Foundation Server. In the central repository is the work items collected, so everybody
knows each other’s progress. Every role, every task has its own specific optimized tool,
connected with this ‘single point of truth’.
The white circle, just below the tools is the process template, which supports, drives a
predefined way of working.

More reading: http://msdn.microsoft.com/en-us/library/bb385832.aspx

There are many different process templates, which can be found in the MSDN landing
page and every organization has probably made some customizations. In the box there
are the MSF Agile and the MSF CMMi.

MSDN page: http://msdn.microsoft.com/en-us/vstudio/aa718795.aspx

Read: VSTS 2010 TMap Process Template 1,2 and 3:
http://www.clemensreijnen.nl/post/2009/07/03/VSTS-2010-TMap-Process-Template-12-
and-3.aspx

Lab 1: Create team projects.
1: Open VS2010 and open the Team Explorer

2: check if the team explorer is connected to the Default Project Collection.

3: add new team project

4: give it the name << EmployeeApplication >>
5: select msf agile process template, next next finish (an empty source tree will be
created)

6: after the team project is created, add the sources to source control.

7: open the application, build and run it as a check.
Do the same for the Tailspin application, only after step 6 we have to set some things for
the database access. Another dependency is the availability of MVC template in VS2010,
see the USB.

Every process template has artifacts, artifacts which support the process, the work of the
different roles and provide the team with metrics covering the quality of the process.

See:
Artifacts (Agile): http://msdn.microsoft.com/en-us/library/dd997580.aspx
Artifacts (CMMI): http://msdn.microsoft.com/en-us/library/ee332487.aspx
Scrum 1.0: http://msdn.microsoft.com/en-us/library/ff731587.aspx

(from MSDN)
You can use work items to track and manage your work and information about your team
project. A work item is a database record that Team Foundation uses to track the
assignment and progress of work. You can use different types of work items to track
different types of work, such as customer requirements, product bugs, and development
tasks.

The work item type determines the workflow states and transitions. The work item types
that are available in a project depend on the project template that the project uses. Also,
you can customize the work item types that are available in your team project to track
your work in the way it makes sense for your development environment.

Each work item represents an object that is stored in the Team Foundation database.
Each work item is assigned a unique identifier, a work item ID, and type, the work item
type. Work item IDs are unique across all work item types in all team projects in a
project collection. The work item type determines the work item fields that are available
for tracking information, defaults defined for each field, and rules and constraints
positioned on these fields and other objects that specify the work item workflow. Every
change made to a work item field is stored in the work item log, which maintains an
historical record of changes.

You can create and modify work items by using Team Explorer, Team System Web
Access, Office Excel, or Office Project. When creating or modifying individual work items,
you can work in the work item form, by using Team Explorer or Team System Web
Access. You can make bulk updates to many work items at a time by using Team System
Web Access, Office Excel, or Office Project.

Work item types in the TMap for MSF Agile process template:
User Story, A team creates user stories to define the features, functions, and
requirements to be implemented.
Task
Test Task
Test Basis Finding (see: http://tmap.codeplex.com/wikipage?title=Test basis finding)
Test Case
Bug, You can track a code defect by creating a bug work item. By creating a bug, you
can accurately report the defect in a way that helps other members of the team to
understand the full impact of the problem.
Issue, You can define known or potential problems, impediments, or risks to your
project by creating issue work items.
Shared Step, Shared steps are specific for test cases and are mainly used only in
Microsoft Test and Lab Manager. In shared steps, you define a sequence of action and
validation steps to run as part of a test case.

A user story communicates functionality that is of value to the end user of the product or
system.

When you define a user story, you must define the Title in the top section of the work
item form. You can leave all other fields blank or accept their default values.

Link types: work items can be linked to each other; each link has some specific
characteristics. You can create work items for bugs, tasks, features, requirements, and
value propositions for your project and then create relationships among them that meet
your project's business goals.

Work item types can have a specific flow, in this example the flow of the user story WIT.
Teams can change the flow specific their needs. But be careful, it must support the
system under development; you can make it as crazy as you can image.

The flow can be visualized by using the Web Access Tool or by using the Process
Template Editor.

More reading: http://msdn.microsoft.com/en-us/library/dd380634.aspx

An example: the role of the tester and the usages of work items.

During the planning phase of the project, also called iteration 0 [first blue piece], user
stories are collected / brainstormed / defined /… in VSTS this information is collected in
the work item type ‘user story’

During the planning phase of the project, also called iteration 0 [first blue piece], user stories
are collected / brainstormed / defined /… in VS this information is collected in the new work
item type ‘user story’ [image below].

During the planning of the iteration developers the team start to breakdown the user stories
[which are selected for that iteration] in implementation tasks. Within VSTS this is done in the
implementation tab. The new 2010 functionality of hierarchies between work items is used for
this.

Another task also executed during this phase is the creation of test cases. Within the TMap
methodology this is described in the planning phase. Where you create the test plan for that
iteration, based on the risk class [see user story work item] and discussion with the customer
the necessary test techniques are allocated for the user story and functional area. See the

initial work items for iteration 1 in the image below, added during the unfolding of the TMap
process template.

So, not only the implementation tasks needs to be determined also the test tasks needs to be
allocated during the planning phase of an iteration. These test tasks are specific to testers like
‘create test cases for area …. based on the test technique <… decide based on risk
and customer contact>’

List of different test techniques…

 Data combination test (DCoT)
 Data cycle test (DCyT)
 Decision table test (DTT)
 Elementary comparison test (ECT)
 Error guessing (EG)
 Error testing (ET)

 Process cycle test (PCT)
 Real life test (RLT)
 Semantic test (SEM)
 Syntactic test (SYN)
 Use case test (UCT)

So, just like the implementation tasks also the test tasks are a link type, have hierarchy with,
the user story. [this is TMap process template specific]

We end up with a list of tasks for a user story, implementation tasks and test tasks… testers
and developers are going to execute these tasks together in parallel, during the iteration we
have: implemented sources for the user story and test cases which are ready for execution.
This user story is finished when: every implementation task is fulfilled, all test cases are
successful executed and … the tester hasn’t got any open tasks, so all test cases are created.

When giving the tester/ the team a place where they can record their testing tasks, testing is
really going to be a first class citizen of the lifecycle. beside this benefit the connection
between test activities, risk, user story and test cases gives a great opportunity for reports
based on test effort, risks and implementation, later on more on this…

With this this lightweight addition to the user story work item we got a closer to several Agile
Testing key characteristics, for example:

–> just like the developers, testers breakdown
there work discusing it with the customer
–> and all tests are created..!

During the planning of the iteration the team starts to breakdown the user stories [which
are selected for that iteration] in implementation tasks. Within VSTS this is done in the
implementation tab of the user story Work item. The new 2010 functionality of
hierarchies between work items is used for this.

More reading: http://www.clemensreijnen.nl/post/2009/09/03/Agile-Testing-with-VSTS-
2010-and-TMap-Part-01-User-stories.aspx

A team creates tasks to track the level of effort that is required to implement a user story
or other areas of work that are identified for the project. Tasks should represent a small
unit of work that can be accomplished within one to two days. You can break larger tasks
down into smaller subtasks.

You can create a task to track work to develop code, design and run tests, address bug,
and perform regression testing. In addition, you can create tasks to support generic work
that must be performed.

By tracking work hours for each task, the team can gain insight into the progress that it
has made on the project.

You can use the Remaining Work and Burndown and Burn Rate reports to monitor team
progress, identify problems in the flow of work, and determine the team burn rate.


In Visual Studio Team System 2010 all test roles are provided with clear and better
support within the Application Lifecycle. Testers do not use their own separate technical
tools anymore, but use integrated tools that are used by architects and developers.
Effectively tearing down the wall between developers and testers.

But good tools are not enough. Also, a clear separation of roles, tasks, and authorizations
are necessary. Finally and most importantly, a structured approach determines how
successful you are with your test strategy.

For test tasks look at the activities in the different TMap phases:
http://eng.tmap.net/Home/TMap/The_4_essentials/Structured_Test_Process/Acceptance
_and_system_tests/Planning_Phase.jsp

More reading: http://www.clemensreijnen.nl/post/2009/04/21/Testing-in-the-
Application-Lifecycle-with-Visual-Studio-2010-Test-Edition.aspx

A team uses test cases to define tests that will support testing of user stories. You can
define manual test cases that specify a sequence of action and validation steps to run, or
you can specify automated test cases that reference an automation file.

Areas and Iterations are typically artifacts used by development teams, but now test is
connected with them, we should have some knowledge about these topics.

Areas are used to breakdown the application in functional pieces, and iterations are used
to breakdown the effort of creating the application in time boxes.

Often the testing effort is also broken-down in these parts.

More reading:
http://msdn.microsoft.com/en-us/library/dd997576.aspx
http://msdn.microsoft.com/en-us/library/ms181692.aspx

Specific privileges are needed to create them.

Lab 2: Create user stories and tasks
for the employee application.
the user stories are used in the other labs.

1: open you favorite TFS client (MTM, VS2010, Web access , ... )
2: add a user story and fill the necessary fields

3: add linked tasks

4: repeat several times

Work Item Queries

With work items in place, several queries and reports can provide valuable information
about the state of the project.

Default there are already some powerful query’s available, to plan and find work
assigned to you.
To access work item queries, you must have the appropriate permissions


Show and run the queries for the previously executed exercise.

Beside the default queries it’s also possible to create your own queries. When a query is
saved in the ‘my queries’ folder they will show up in all other tools.

More reading: http://msdn.microsoft.com/en-us/library/ms181352.aspx

Work Item Query Language (Work Item Query Reference)
You can find bugs, tasks, other types of work items, and links between work items by
using one of the WorkItemStore..::.Query methods or a Query object. These queries use
the work item query language (wiql), which resembles T-SQL.

More reading:
http://msdn.microsoft.com/en-us/library/bb130198(VS.90).aspx .
http://msdn.microsoft.com/en-us/library/ms181352.aspx

Beside the creation of flat lists queries it’s also possible within VS2010 to create
hierarchical list to visualize the relationship of the work items. Two different types of
queries can be created; one based on link types and one as a tree view.

you can perform the following tasks:

Track and check features against value propositions.
Review active tasks grouped by feature area
Assess changes that were made to dependent work items
Assess changes that were made to dependent work items at a point in time

Save changes to work items: You can quickly save any changes that you make to
multiple work items within the list of results from the query.

More reading: http://msdn.microsoft.com/en-us/library/dd286632.aspx#Find

Type of query: You can perform complex queries and view link associations between
work items with the addition of these two new types of queries:

Work Items and Directed Links: Displays a two-tiered set of work items and their
dependent links based on work-item and link-filter criteria that you specify. You can
expand and collapse leaf nodes and, within Team Explorer, drag work items to change
link associations.

Tree of Work Items: Displays multi-tiered hierarchical relationships between work items
that are associated with parent-child links. You can expand and collapse leaf nodes and,
within Team Explorer, drag work items to change link associations.

Reports Basic

Now we can collect the information with queries we need to visualize them. Within TFS
this can be done with SQL Reports and Excel Reports. First the default SQL Reports.

Team Foundation Server provides you with many different and powerful ways to get the
answers to your questions. In the 2010 release, we’ve enhanced every SQL Server
Reporting Services report. You’ll find and richer experience with better data presentation
and detailed documentation to help you use the reports right out of the box. In 2010,
we’ve also added new Excel Reports. TFS feeds the reports via Work Item queries for fast
and flexible reporting. Finally, rich SharePoint dashboards enable quikc visualization of
project status. In addition, when used with SharePoint Server products, you can publish
reports in a familiar and capable environment.

Let’s look at the new Stories Overview report. <Build>
First, you see the report contains a description describing what this report is showing.
<Build>

Second, you get a list of related reports that might help expanding on the data shown
here. <Build>
Finally, you see the work progress on the user stories in context with data about test
results and bugs.

Next as you scroll down, you see additional information in the footer. <Build>
First, on the left you have questions this report answers with a hyperlink that takes you
to the documentation about this report, which has additional details including healthy and
unhealthy report examples. <Build>
Second, you get the parameter values used to generate the report, something that helps
when looking over printed copies in meetings for example. <Build>
And finally, you get information on how up-to-date is this report. Every report in the
2010 release is like this.

As you can see the Quality dashboard has four main graphs. <Build>
First you can see if your test team is making progress on running test plans. <Build>
Second, you can see how build are doing over time. What’s the trend like? Are you
having lots of success or failure? <Build>
Third, what’s your bug trend like. Are you closing out bugs or are you stagnate. Or is the
velocity of your bug filing far exceeding your team’s ability to fix, test, and close out
bugs? <Build>
Finally, are you seeing a bad trend related to bug reactivations—bugs that were closed
reopened by test as not fixed? All of this information is there for you in a quick, heads up
dashboard format.


While SQL Reports are hard to change are Excel reports much easier.

More reading : http://msdn.microsoft.com/en-us/library/dd997876.aspx

For example a report based on the results of a flat list imported in
Excel

More sophisticated the creation of excel reports based on a wizard within Excel to create
different types of reports.

Beside the default SQL reports there are also default Excel reports and workbooks.

The default excel reports are:

You can use workbooks to build your product backlog and plan work by assigning the
user stories, tasks, bugs, and issues to specific iterations, also referred to as sprints. By
using the workbooks that are provided with the process template for MSF for Agile
Software Development v5.0, you can quickly create work items and modify the rank,
priority, state, and assignments of multiple work items at the same time.

Microsoft Test Manager

The not so standard user interface experience of MTM, what to find where.

http://msdn.microsoft.com/en-us/library/ff458183.aspx

(from Improving Developer-Tester Collaboration with
Microsoft® Visual Studio ® 2010 http://channel9.msdn.com/shows/VS2010Launch/Doug-
Seven-Improving-Developer-Tester-Collaboration-with-Microsoft-Visual-Studio-2010/ )

Microsoft Test Manager 2010 is for testers what Visual Studio is for developers. That is to
say, where Visual Studio is an IDE – an integrated development environment, Test
Manager is an ITE – an integrated test environment. This is the interface that a tester will
use to create test cases, organize test plans, track test results, and file bugs when
defects are found. Test Manager is integrated with Team Foundation Server, and is
designed to improve the productivity of testers. While I am not going to do a deep-dive
of all that Test Manager can do, it is important to understand how it integrates with the
Visual Studio Agents to make the most of test case execution, and ensure that when a
tester files a bug, it is actionable with very little work on the tester’s part.

Test Lifecycle – The Control
Phase I
Test Plans

Controlling the testing effort, beside bug triage this also means test case management,
what to run when, what has been run, etc.

Aim
Providing the client with sufficient insight into, and the opportunity to influence, the
following:

The progress of the test process
The quality and risks of the test object
The quality of the test process
To this end, the test manager manages the test process optimally and reports on it.

Within MTM test case can be organized, within test plans, suites and environmental
configurations. How the test cases are organized depends on the type of organization.

A test plan organization can be as simple or as complex as you want, based on your
testing approach. You might want to create a test plan for each sprint if you are using an
agile methodology. Then you can add the user stories for each sprint to your test plan
and create test cases for these user stories. Or, you might create a test plan for each
specific milestone if you are using another approach. Another way is creating test plans
per test type.

In Microsoft test manager a test plan holds all the information needed to maintain,
organize and execute test cases. It doesn’t hold information about the testing effort, this
is the reason I talk about test execution plan.


Without a test plan you can’t access test testing center in Microsoft Test Manager.

After you created a test plan you can set its properties in

Setting the state property to inactive will make the test plan disappear from the start
screen. But, it still can be accessed from another test plan in the ‘organize’ section.

Test plans should have a clean and descriptive title.
More reading: http://blogs.msdn.com/b/anutthara/archive/2010/09/22/guidance-for-
creating-test-plans-and-test-suites.aspx
http://www.clemensreijnen.nl/post/2009/09/09/02-Management-preserving-and-
organization-of-manual-test-cases-within-VSTS-2010.aspx

Test Configurations

You can assign a set of default configurations to your test plan that you want to cover for
quality purposes. You will be able to view which tests have passed or failed for each
configuration and how many tests you have left to run.
Some proposed configurations are the different environments people work with, test,
development etc

More reading: http://blogs.msdn.com/b/vstsqualitytools/archive/2009/10/30/test-
configurations-in-a-test-plan.aspx

Test Suites

You can create test suites in your test plan to group your test cases into suites, based on
your needs. There are three different suite types with different characteristics.

1. Requirement bases suite: You can add a requirement to form a suite in your plan
that contains all the test cases that are linked to this requirement.
Test cases added to this suite are automatically linked to the requirement, and
test cases which are linked somewhere else (in visual studio, another suite, in
web acces) to the requirement are also visble in this suite.

2. A folder like suite with no additional functionality other than hold the test suites.
Folder Test suites can contain child suites (the other two not).

3. A query based suite, which holds a collection of test cases with properties conform
the selection criteria.

You can’t add test cases to this suite they are selected by the query.

Lab 3: Create and configure test
plan.
1: select add test plan in the black start screen and add a test plan.

2: open the iteration test plan.

3: open the test plan properties and edit the configurations.

3: add a new variable were we can but information about 64 bit and 32 bit machines.

6: delete the current configurations and add or own client configuration.
 Windows 7 64 bit
 Windows 2008 R2 64 bit

7: set a default configuration for new test cases
8: save and close the test plan properties dialog.

Add an additional end2end test plan.

1: go to the ‘organize’ section and add a new test plan

2: call it E2E Test plan and set several properties and configurations.

Add Test suites to the iteration test plan.

1: add requirement based test suite for the userstories

2: add query based test suite, for regression candidates… prio 1

3: add query based test suite, for automation … automated status == planned or
automated

4: add a ‘normal’ test suite for exploratory tests.

TCM.EXE

With the command line tool TCM.exe you can get lists of test plans, configurations and
more. Also you can execute test cases with this command line tool.


See http://msdn.microsoft.com/en-us/library/ff942472.aspx for listing test plans

With the collected ID you also can share links to test artifacts (like test plans). In the
‘test plan’ start screen you can find on the top the ‘copy URL for plan’ button.

Othere artifacts can be shared the same way.

Test Lifecycle – Specification
Phase

After test planning and test control we get to the test specification phase in the testing

lifecycle.

Aim
During the Specification phase, the required tests and starting points are specified. The
aim is to have as much as possible prepared, in order to be able to run the test as
quickly as possible when the developers deliver the test object.

Activities
Within the Specification phase, the following activities are distinguished:

Creating test specifications
Defining central starting point(s)
Specifying the test object intake.

During the test case specification we create test cases with corresponding test steps.
Shared steps and test data iterations are test specific items which need special attention.

Test Cases

Test cases are work items with a specific tab where test steps can be defined. These test
steps can only be edited from within MTM.

You can create test cases for your manual tests with both action and validation test steps
by using Visual Studio 2010 Ultimate or Visual Studio Test Professional. You can add test
cases to your test plan using Microsoft Test Manager.

More information:


Test case naming conventions discussion.
What is the best way to name your test cases…

For quick editing of test cases (only the logical / not the steps) you can use the team
foundation toolbar in excel to open a test query.

A test step within a test case in Test Manager (MTM) can be a normal action step or a
validation action step, and both of them can contain additional information
(attachments).
See image three steps:

1.normal action step
2.action step with validation

3.normal action step with attachment (validation action step can also contain an
attachment)

What kind of action steps do I need to use for my test case and how to write them… you
are probably not the one who executes the test case.

more reading: http://www.clemensreijnen.nl/post/2010/10/29/Microsoft-Test-Manager-
Test-Case-Specification-what-kind-of-action-to-use.aspx

An easy way to share or re-use test steps is by copy paste the steps. It works perfect to
and from excel.

Shared Steps

Your team can use shared steps to streamline definition and maintenance of manual test
cases. Many tests require the same sequence of steps to be performed for multiple test
cases. By creating shared steps, you can define a sequence of steps once and insert it
into many test cases. For example, if each test case requires a tester to log on to the
application, you can create a set of shared steps to perform these actions. You can then
add the shared steps to each test case and run the steps using Test Runner.

Candidates and reasons for shared steps
• Reuse
– Many steps, reused in many test cases. Benefit: time, retyping, boring
work.

• Expected changes
– for example environment variables change [URL]. Benefit: Time, test
cases which need this step are updated automatically, also action
recording.

• Pre- conditions
– bring the system under test to a certain state and execute the different
verifications. Benefit: beside time, you are certain the application under
test is in the same state for every test case

• Back to initial state
- after test execution bring the environment back to the initial state for
new test cases. Benefit: beside time, you are certain the application under
test is always in a clean state for new test cases.

There are somethings to think about and know before using shared steps.
- Parameters defined in shared steps must also be defined in the test case, for
every test data iteraiton.
- Expected result in a shared step limits the re-use of the shared step. Are you
testing the same thing over and over again?

Test Data Iterations

If you want to run a manual test case multiple times with different data, you do not have
to create multiple copies of the test case. You can add parameters to the actions or
expected results for any test step in your test case. Then you can add multiple sets of
values for the parameters that you want to use in your test. Each set of values for the
parameters is run as an individual iteration of the test using Test Runner. You can copy
the value of each parameter when you run the test and then paste the value into your
input field in your application under test.


If you have configured test settings to create an action log for the test being run,
selecting the Create action recording option will use the settings that you configured. If
you did not configure the actions diagnostic data adapter in your test settings, then the
default values are used to create your action recording

The same as with test steps, test data can be copied and re-used in or from excel.

Lab 4: Create test cases.
For the employee application we are going to create test cases with test data iterations
and shared steps.

Specify multiple test cases.

1: open the iteration test plan.

2: add a new test case to the requirements based test suite.

3: give the test case a useful name.

4: add the steps:
1. open Employee application
2. fill ‘Clemens’
3. fill ‘Reijnen’
4. push lookup ‘with expected results’ ‘6913’
5. close application

5: check if all the other fields are correct and save the test case.

6: create some more test cases, for the ‘show salary’ functionality and employee not
found.

Add test data iterations and parameters.

1: open the test case which has been created during the first steps.

2: change,
6. fill ‘Clemens’  fill @givename
7. fill ‘Reijnen’  fill @surname
8. push lookup with expected results ‘6913’  expected results @phonenumber

3: add the values:
1. Clemens 6913 Reijnen
2. andre andre 1000
3. alf alf 2000

Create shared step and reuse it.

0: not necessary but will make life easier, create shortcut on the desktop for the
employee application executable. (the executable can be found after compiling the
solution within VS2010 in the bin directory.)

1: open the test case which has been created during the first steps.

2: select the first step ‘open Employee application’ right click and select the context
menu item ‘create shared step’.

3: give it a name and save the shared step. See that the step is now bold. (optional:
save and close the test case)

4: go to ‘organize’ – ‘shared steps manager’, select the shared step and click ‘create
action recording’.

5: Test runner will open, select ‘create action recording’ and start the application.

6: when finished save and close the Test Runner.

7: repeat these steps for the ‘close application’ step.

8: use the ‘open’ and ‘close’ shared steps in the other test cases.

Test Lifecycle – Execution
Phase

Aim

To obtain insight into the quality of the test object through the execution of the agreed
tests.

Activities
Within the Execution phase, the following activities are distinguished:

Intake of the test object
Preparing the starting points
Executing the (re)tests
Checking and assessing the test results.

Test Run

You can start as manual test run by selecting one or more test cases, this can be done
either by multiple selecting test cases in the test section or by selecting a complete test
suite, and click run. When the test run starts the test runner will guide the tester to the
test process.

From msdn.
Running manual tests using Test Runner can help you identify problems in your
application. Problems that are typically identified when you run manual tests include
incorrect functionality or bugs, poor performance, and security and connectivity issues.
The outcome, resolutions, and comments that you can enter during a manual test are
saved as test results.


Selecting the ‘run with options’ menu will give the testers some more control over the
test settings and environments. Also when the test case is an automated test case you
can use this menu item to run in manually.

More info: http://msdn.microsoft.com/en-us/library/dd286725.aspx

Test Runner

The test runner has a minimalistic user interface, but with a lot of options.
- The return to testing center button, opens microsoft test manager again but won’t
stop the test. This is great for small edit’s.
- In the drop down you can see the ammount of test cases and test data iteration in
this test run.
- On the bottom you can show hide the recorded steps, will executing the test case.

- On the bottom, in the recorded actions pane, you can remove unnecessary
recordings for cleaning the action recording.
- Steps with action recordings, recognized by the orange bar, can be re-played.

Besides using the test runner it is also possible to execute test runs with the TCM.exe
command line tool. Making it easy to automated test execution during builds.


Lab 5: Execute the created test
cases.
For the employee application we are executing the create test cases.

1: go to ‘’Test” – ‘run tests’ and select either a test suite or a single test case and click
‘run’.

2: test runner will open, check the ‘create action recording’ and start the test. Follow the
steps and mark the steps passed (if succesful).

Optional: run them from TCM.exe
1: open Visual Studio command prompt, type ‘tcm.exe run /help’ and follow the
guidance.

Test Case Execution Results

The most important thing after execution of test cases is reporting about the results,
what is the quality of my system.
In Team Foundation Server there are many ways to get the needed information. All the
data is stored in a data warehouse available for all the different client tools.

Test run results in MTM.

In Microsoft test manager the reporting about the test runs is minimal, you can see the
result of the test runs if they failed or passed.

After you run tests, you can review your test results to see which tests passed and which
tests failed. For tests that you run from a test plan by using Microsoft Test Manager you
can review your test results, assign a reason for a failed test, and assign a resolution.

You can view all these test results for a test plan using pre-defined test reports or by
creating your own custom reports.

As you analyze the test results for each test, you can select a failure type and resolution
for each failed test, or add a comment about your analysis to this point. To help with
your analysis, you can open the test result for each test to see all previous runs of that
test.

If you determine that there is an issue, you can create a bug and associate the data from
the test result for this test with the bug. If you determine that the failed test is caused by
a known issue, you can link that test to an existing bug. When this bug is fixed, the
tester who verifies the bug can then see all the test cases that have to be rerun to verify
that the bug is fixed.

If you want to add data to the test run, you can add attachments or comments to the
test run itself.

More readings: http://msdn.microsoft.com/en-us/library/dd553274.aspx

You can view the details of the results of your tests using Microsoft Test Manager based
on each test point in your test suite. A test point is the pairing of a test case with a test
configuration in a specific test suite. If you have multiple configurations that you want to
use when you run your test, you have a test point for each pairing of the test with a
configuration.

More readings: http://msdn.microsoft.com/en-us/library/ee843861.aspx

Also in the test plan properties window you can view the results. But for most
organizations all this test result information isn’t enough… to say something about the
quality of the system.

Share the test run with peers by using a URL and the runID

Because Test cases are Work item we can create queries to analyze the work. The
defaults are…

Also default reports and excel reports are available.
(show)

Browse through the created reports and discus.

There are also pivot capabilities for the test artifacts. Selecting the test fields show them.
One important thing to notice is are the fields which don’t belong to a work item, these
are MTM specific artifacts save to the datawarehouse.

The organization looks like this. Some pieces for the used test artifacts are stored in the
TFS database and some in the TCM server. Not accessible together, only in de olap
cubes.

Some specific useful fields in the warehouse for reportings.

(from msdn)
By using the Test perspective in the SQL Server Analysis Services cube for Visual Studio
Team Foundation Server, you can view just the measures, dimensions, and attributes
that pertain to reporting on tests results and test runs. For example, you can use these
measures to determine the overall quality of each build, the tests that a particular build
affected, and the number of test cases that were run. You can also answer questions

about changes to the result outcomes.

The Test measure group is based on the Test Results relational table, which enables
reporting on test results as either a property of the tests or an independent outcome. For
more information, see Test Result Tables.

More reading: http://msdn.microsoft.com/en-us/library/ms244708.aspx

It is still a bit complicated, or better a bit confusing, how the different parts within
Microsoft Test Manager are organized. This image is a try to clear things a bit.

More reading: http://www.clemensreijnen.nl/post/2010/07/19/Microsoft-Test-Manager-
Test-Artifact-organizatione280a6.aspx

Test Lifecycle – Control Phase
II

Actions a test organization does during the control is not only the management of the
testing effort, but also the management of bugs.

Aim
Providing the client with sufficient insight into, and the opportunity to influence, the

following:

The progress of the test process
The quality and risks of the test object
The quality of the test process

File a Bug

Submitting bugs and tracking the status of bugs is an important task that helps you
ensure that bugs are fixed. This improves the quality of the applications that you are
testing. The more information that you can provide with a bug, the more quickly it can be
recreated and fixed. Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate
provide functionality to collect data that can automatically be added to a bug that you
create, when you run your tests. You can submit bugs in the following ways by using
Visual Studio Test Professional 2010 or Visual Studio 2010 Ultimate:

When you run a test using the Test Runner

From Microsoft Test Manager when you view a test result

From Microsoft Test Manager when you view your bugs

You can also submit bugs from Microsoft Visual Studio 2010.

All these bugs can be viewed, updated and tracked using both Microsoft Test Manager
and Microsoft Visual Studio 2010.

Important topics for understanding and fully use the capabilities of MTM with bug filling
are:
 Diagnostic Data Adapters

 File a Bug
 File an Exploratory Bug
 Verify a Bug

Collect Data on a Local Machine by Using Default Test Settings

If you just want to run manual tests on a local machine and do not want to collect data
on another tier of your application, you can use the default test settings

More reading:
http://msdn.microsoft.com/en-us/library/ee231892.aspx
http://blogs.msdn.com/b/amit_chatterjee/archive/2009/04/11/diagnostic-data-adapters-
changing-how-developers-and-testers-work-together-part-1-of-2-the-test-impact-
collector.aspx

File a bug from Test Runner.
When you run a manual test and discover a defect in the application under test, you can
submit a bug directly from Test Runner without having to switch to Visual Studio.
Submitting a bug in this manner automatically adds information about your test and
testing environment to the bug. You can also update an existing bug if you discover
additional defects while you are running a test.

More reading:

Submitting Bugs When You Run Your Manual Tests: You can submit a bug immediately
when you run a test from Test Runner. Your test steps and any information that you
have collected are automatically added to the new bug that you create.

Submitting Bugs from Your Test Result: You can submit a bug from Microsoft Test
Manager when you view a test result. For example, when you analyze a test result and
you determine that the test failed because of a product issue. You can also submit a bug
from Microsoft Test Manager when you are viewing your bugs.

Submitting Bugs from Microsoft Visual Studio: You can submit bugs using Microsoft
Visual Studio 2010 at any time.

Verify a bug

Using Microsoft Test Manager, you can track the status of both bugs that are assigned to
you and bugs that you created. You can also track the bugs for your whole team project
to determine the overall quality of areas of the application under test. You can triage
bugs by viewing the details of the bug and determining whether the bug should be fixed
or not. You can assign the bug to another user to provide more information, decide to fix
the bug, or determine whether it is actually a bug.

When a bug is fixed, it can be assigned to the tester to verify that the bug has been
fixed. In the Verify Bugs view, you can view bugs that are assigned to you and have to
be retested, as shown in the following illustration. Or you can create a custom query to
track all bugs that have to be retested.

Lab 6: file a bug.
Create a build
We need a build for later use.
1: open Visual Studio.
2: open ‘Team Explorer’ and select ‘buils’ right mouse click and select the context menu
‘new build definition’

’

3: trigger ‘manual’

4: build defaults: set the drop folder.

5: process ‘items to build’ the employee app solution.

6: try the build, by selecting ‘queue new build’

7: check if successful. (Green bar)

When the build doesn’t run, check the build service in the team foundation console.

File a bug
1: set in the test settings, local test run the wanted data collectors on.

2: add to the first create test case (in lab 4) the test data iteration values 1,1,1.
3: run the test case (it will crash at 1.1.1 :-)
4: file a bug by selecting ‘create bug’.

5: give the bug work item a useful name – assign it to yourself, save and close it.

Resolve a bug
1: open Visual Studio and the employee application.
2: open the Team Explorer and open the team query, ‘my Bugs’, look for the previously
filed bug.

3: go to the solution explorer, open the ‘employees.cs’ file and look for the sentence with
‘else if (firstname.Equals("1") && lastname.Equals("1"))’

Remove the lines:
Exception ex = new Exception("Went something wrong");
throw ex;
and the // before the two below.
Should look like this afterwards:
else if (firstname.Equals("1") && lastname.Equals("1"))
{
Employee emp = new Employee("1", "1", "1@sogeti.nl", "1", 1, "1", "1");
return emp;
}

4: select the ‘employees.cs’ file, right click and select ‘check in’.

5: in the check in dialog, select ‘work items’ and check the bug, select for action
‘resolve’.

6: queue a new build and wait till it finishes successful.

Verify a bug
1: open Microsoft test manager.
2: go to ‘test’ – ‘verify bugs’, select the resolved bug and click verify.

3: test runner will open. Run the test case… and check if the bug is solved.
4: when all steps are finished and marked passed (where necessary) close the test
runner. MTM will ask if you want to close the bug.

Bug Reports

Monitor bug activity, reactivations, and trends. You can use bug reports to track the bugs
that the team is finding and the progress that the team is making toward fixing them.

Test Impact

By using Test Impact Analysis during code development, you can identify the methods in
a test project that have been affected by code changes in your managed code solution.
At each build of the solution on the local computer, Test Impact Analysis identifies the
methods in the code project that have changed and lists the test methods that directly or
indirectly call those methods. You can then run the tests from the Test Impact View
window. You can also use the Test Impact View window to identify and run any test
method that affects a particular code method.

demo

By using test impact analysis, you can identify the tests that you should run, based on
coding changes that were made to the application that you are testing between builds.

When you use test impact analysis with Microsoft Test Manager, you are required to use
Team Foundation Build to build your application and Team Foundation version control for
your source code for the application being tested. Test impact information is gathered
only on tests with a status of passed. It is not collected when you file a bug or when a
test that is marked as failed is complete.

The requirement for using Team Foundation Build is only applicable to collecting data
from Microsoft Test Manager, as the test plan must be associated with a build produced
by Team Foundation Build for the test impact analysis feature to work properly. To use
the Test Impact View in Microsoft Visual Studio 2010, you do not need to use Team
Foundation Build and the solution does not have to be under any source code control
system.

Lab 7: test impact.
Attach build to test plan
1: open MTM.
2: go to ‘plan’ – ‘properties’ and select the build filter created in the previous lab.
(no need to set the quality)

3: select ‘build in use’, select the latest build and assign it to the plan.

Execute tests
1: execute several test cases.

Make code change and kickoff a build
1: as in lab 7 ‘Resolve a bug’ … open the ‘employees.cs’ file/
2: remove the line:
Employee emp = new Employee("Clemens", "Reijnen", "Clemens.Reijnen@Sogeti.nl", "69
13", 12000, "Amstelveen", "Noord Holland");

And remove the // from this line.
//Employee emp = new Employee("1234", "4567", "1234.4567@Sogeti.nl", "6913", 1200
0, "Amstelveen", "Noord Holland");

3: as in lab 7 ‘Resolve a bug’ … check in the changes and kickoff a build.

Attach new build to test plan and evaluate impacted tests
1: go to ‘plan’ – ‘properties’ and assign the new build number to the plan.
2: review the ‘Recommended Tests’ and set them back to ‘active’

Create test from bug

An often overlooked feature in Microsoft Test Manager is the ‘create test from bug’ option
in the ‘verify bugs’ section. It sounds indeed a bit strange, probably there is already a
test case in place which found this bug. But, this create test case feature has some
interesing scenario’s.

When use the create test case functionality you will see that it uses the action recording
to define the steps for the tests case.
A scenario for which you can use this functionality is when the tester filled an exploratory
test with no predefined steps, you can very easily create a well written tests case from
such a action recording.

When you run an exploratory test and find a bug in the application, you can submit an
exploratory bug directly from Test Runner. Submitting an exploratory bug in this manner
lets you specify the time range for the actions to include in the bug. The actions are
recorded in an action log which can be used to create a manual test case.

More reading:
http://msdn.microsoft.com/en-us/library/ee803840.aspx

Other scenario’s for this feature is for example an application with no tests defined in
MTM, you can let the business guy run his most important process while the test runner
is recording. You can use this recoriding to create well written test cases. Another
interesting scenario are migration projects, when you want to migrate an application
from VB6 to .NET for example, record the most important test cases and create the test
cases.

Lab 8: Create an exploratory bug
and create a test case from it.
1: open MTM.
2: create a test case with the name expolritory test.
3: add the step ‘start exploratory testing for 5 minutes’, save and close it.
4: run the test and select after some clicking around ‘create exploratory bug.’
5: go to ‘verify bug’ select the bug, right click and select ‘create test case from bug’ in
the context menu.
6: look at the test steps

Test Lifecycle - Infrastructure
Phase

Before moving on to the completion fase we need to setup or infrastrcuture first.

Aim
To provide the required test infrastructure, which is used in the various TMap phases and
activities.

Activities
The Setting up and maintaining infrastructure phase consists of the following activities:

Specifying the infrastructure
Realizing the infrastructure
Specifying the infrastructure intake
Intake of the infrastructure
Maintaining the infrastructure
Preserving the infrastructure

The ‘green’ Lab Center is the place to manage your test infrastructure. you can manage
physical and vertualized test environments from this center.

Where can I run the tests?
There are different ways to execute tests, the main difference is between the
environment types and technology used for it within Visual Studio 2010

One major difference is the provisioning of clean environments.

On Physical Environments with
There are multiple ways to execute your test automation effort, you can run them from and
on Microsoft Test Manager, from and on Visual Studio, during the build and on the Build
Server or during the build on Test Agents configured with Test Controllers or … many flavors.
so, where should you execute your automated tests, and how should you configure your test
infrastructure.

Where to run your automated tests, and how to configure the test infrastructure?

The different flavors
Flavor A: Execution from
VS2010…

Purpose: Real developer test / test automation dry
run.
A valid scenario is when Developers create a
codedUI test from a test case which found a bug.
To reproduce the bug, developers generate the
CodedUI add the assertion and verify the result
when they fixed the bug.

Triggered by: right mouse click – run tests, in test
project, or Test List Editor, run checked tests.

Information: http://msdn.microsoft.com/en-
us/library/dd286580.aspx

Pro
Easy setup
Debug-able test automation
No additional configuration needed*

Con
No collection of test results in TFS/ Test Manager
Run on developer environment

* Data Diagnostic Settings in the Local.Testsettings
file are configurable.

Flavor B: Execution during
build with Build controller
Purpose: Part of Build Verification Tests (its
strange to run UI tests on a build and have a build
service running in interactive mode)

Triggered by: build

Information:
Set the Build Service to run in interactive mode.
http://blogs.msdn.com/b/mathew_aniyan/archive/
2009/05/26/coded-ui-test-in-a-team-build.aspx
Configure the build to run the tests
http://msdn.microsoft.com/en-
us/library/ms182465.aspx#CreateBuildType

Pro
Easy setup
Test results in build result

Con
No collection of test results in Test Manager (no
reporting on test points)
Build Service needs to run in interactive mode
(what happens after a restart)
Tests are executed on build environment

Flavor C: Execution during
build with Test controller

Purpose: Part of Build Verification Tests, distribute
tests over multiple test agents. (Preferred
configuration above flavor B)

Triggered by: build

Information:
Configure Test Controller (don’t register it with a
project collection )
us/library/dd648127.aspx#TestControllers
Configure
Test Agents on clients (interactive mode)
us/library/dd648127.aspx#TestAgents
Configure *.Testsettings file in solution to use Test
Controller and Test Agents
Configure the build to run the tests (see B)

Pro
Tests run distributed over multiple environments
Test Results in Build result

Con
No collection of test results in Test Manager
Harder to configure
Need for specific test client environments
Test Settings from VS

Flavor D: Execution from
Purpose: Part of Regression Tests (other type of
test than BVT).

Triggered by: MTM user, right mouse click on test
case, run

Information:
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode)
Configure Lab Center in MTM to use test controller
and create test ‘agent’ physical environment.
us/library/ee390842.aspx
Associate CodedUI test with WI Test Case from VS.
http://www.richard-banks.org/2010/11/how-to-
use-codedui-tests-watin-and-mtm.html

Pro
Test run distributed over test environments
Test Result in MTM
Test Settings from MTM
Full control by the tester

Con
Test Controller needs to be configured with a
Project Collection (one controller per collection)
Manually triggered by Tester (or pro)
Hard to configure
Hard to see which test case is automated

Flavor E: Execution from
MTM during Build…
Purpose: Part of BVT. Preferred configuration
above flavor C. Flavor D and E can be configured
together.

Triggered by: Build

Information:
Configure Test Controller (register it with a project
collection )
Configure Test Agents on clients (interactive mode,
can be the same as MTM)
Configure Lab Center in MTM to use test controller
and create test ‘agent’ environment.
Associate CodedUI test with WI Test Case from VS.
Create Build task to run TCM or MSTEST task for
Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/201
0/10/30/how-to-run-coded-ui-tests-from-
command-line.aspx
How to: Run Test Cases with Automation from the
Command Line Using Tcm

Pro
Test run distributed over test environments.
Tests can be configured to run on different
configured environments
Test Result in MTM and TFS
Triggered by build
Test Settings from MTM

Con
Hard to configure
maintenance of TCM commands in build

Enough flavors to choose from… the pro and cons I wrote down can differ per situation. My
preferred flavors are E and D configured together. The reason, test settings and test results
are where the belong, in the testers hands. And, the TCM command line gives much more
flexibility what to run where during the build.

Lab Management for virtual environments has different configurations, pros and cons…
will make a post for this later…
An update for my favorite flavor E is this Notions Physical Build-Deploy-Test Solution
from Colin Dembovsky it’s a XAML build template which lets you easily configure the
Build-Deploy-Test workflow for physical environments. one drawback from flavor E will be

minimized when using this template, the maintenance of the TCM commands.

Lab 9: Install and configure the test
controller and test agent

http://msdn.microsoft.com/en-us/library/ff469838.aspx

Lab 10: Setup Flavor D
Automated Test Execution from Microsoft Test Manager

Configure the test controller that it is associated
with the team project collection.

Create a physical environment. Verify in the
controllers menu that there is an environment
connected with a test agent.

In the test plan properties configure the
automated test settings.

To add an automation script to a test case we
must open visual studio and add a test script to
the test case. (see CodedUI test in this
document for the details)

Now your test infrastructure is ready to execute
automated tests from MTM. Select run on a test
case with an automation script.

Test Lifecycle - Completion
Phase

Aim
To learn from experience gained during this test and to preserve test ware for reuse in a
future test.

Activities
The Completion phase consists of the following activities:

Evaluating the test process
Preserving the testware.

Test automation with Visual Studio
2010.

Different test levels, in which phase should we use test automation?
What is what level and what do you do at that level

VS2010 has many technologies and capabilities for test automation.
What is what:
….
…
…
…

Within the different levels different technologies are interesting.
Why? …

In what phase of the project and by whom is the technology used?

Who en when…
…
…

Why do you want to automate it with that technology at that phase, what do you want to
accomplish.

In this guid we focus on the UI automation capabilities of VS2010. How to use it, why by
whom and how does the infrastructure looks like.

When is test automation the
investment worth?
With VS2010 ALM you can very easily take a manual test case and turn it in an automated
test case. But this isn’t without investment. There are still some additional things to do which
take time and money before we get the benefit of automated execution of a test case.

These are some very basic test automation investment levels. Just to keep in mind when
working with Microsoft Test Manager and CodedUI.

VS2010 ALM has several automation capabilities. The most noticeable is CodedUI, C# code
generated from a MTM action recording or recorded within VS2010 another automation
capability is the Fast Forward functionality in Microsoft Test Runner, Fast Forward till a
validation step for manual validation. This Fast Forward functionality is also valuable in the
shared steps and for test data iterations.

The investment levels explained.

Zero time investment: completely no investment in any kind of automation,
no Fast Forward no CodedUI. Although the FF is recorded for every test it isn’t used. MTM is
used in this situation not for its automation capabilities, but more for its test case
management, bug filling and reporting capabilities.

Which test cases don’t need any kind of automation depends. The most logic reason would be
that the test case only is going to be executed once. the other reason can be that automating
the test case would be a to big investment and to complex, and there are more reasons.

This post is a useful source “effort-estimation-for-test-automation” to make your decision to
automate or not, other good reading is this pdf “TEST AUTOMATION EFFORT ESTIMATION -
Best practices”.

Little time investment: executing the same test case on multiple
environments, or test cases executed several times during the sprint are candidates for
automation, but sometimes the investment of making it CodedUI is too big. Making use of the

Microsoft Test Runner’s Fast Forward capabilities will speed up test execution. but, getting
that additional benefit of it you have to tune the action recording to make the Fast Forward
functionality in the Test Runner more smooth and less error prone.

When executing a manual test case, you never do it correct the first time. You have to search
a bit, maybe make a wrong step, go back in the application etc… execute a test script
complete correct with the optimal amount of clicks the first time is hard. And when you do
click everything correct the first time the action recording will collect a bit too much
information, in some situation (see image, opening a browser window with Bing as homepage
and than move your mouse around to browse for the website under test). Cleaning the action
recording will make the FF more efficient and less error prone. this cleaning of the action
recording needs to be done during the execution of the test case in Microsoft Test Runner.

Note: The little investment of cleaning the action recording, also is for the shared step and
test cases with test data iteration. Test case which use test data iterations are candidate for a
bit more investment for a clean action recording, shared steps are for sure candidates.

Some more investment with Basic CodedUI functionality: Although it is
very easy to create CodedUI C# files for already run manual test cases by using their action
recordings (hopeful a clean action recording from the little investment step). It is still a bigger
in time investment to create them. Beside the time it takes to create the CodedUI C# it also
asks an investment in test infrastructure to execute them, see this post “Running Automated
Tests on Physical Environments, the different flavors…”.

See image, you also have to create the assertion for the automated manual test case. 1) the
manual validation in the Test Case. 2) Add the assertion by using the UI Test Builder 3) drag
the crosshair on the application under test and select the property to validate 4) the assertion
is added to the CodedUI test.

The main idea for this investment level is that only the basic generated code is used (not
customized) and an assertion is manual added to it.

Serious investment in time with advanced CodedUI, customization of
the CodedUI and UIMap: Customizing the generated CodedUI is a real trap. You can make
the most sophisticated UI tests in the world. Resulting in C# code which is even more
complicated as the functionality it tests with probably even as many or more bugs, who tests
the tests trap. So, this investment should be done careful, but sometimes it’s a worthy
investment. For example for test cases which are going to be run the whole life of the system,
or maybe are boring and have many steps. (see the links in the zero investment level.)

Customizations of this level can be started from the ‘UIMap builder’ (see image), change the
search properties, move steps out of the code generation and customize them, add fancy test
data iteration to the steps, etc etc…

Lab 13: Create CodedUI
Create CodedUI from Action Recording.

Add a CodedUI test to the test project in the
solution. Select ‘use excisting action recording’.

In the work item picker dialog select a test case
with a valid action recoding. Generate the C#.

Look at the generated methods and compare
them with the test steps in the test case. Run
the CodedUI test.

Add Assertion.
All steps are generated except the assertion, we
need to create / generate this to make the
CodedUI test complete.

Right click in the CodedUI code where you want
to add the assertion and select ‘Use CodedUI test
Builder’. In the lower right corner of your screen
a little dialog appears, you can use the crosshair
to select the control you want to validate.

Generate the assertion and run the CodedUI
again.

Create a CodedUI from scratch by using the recording option.

Record action for CodedUI

The UIMap
The UIMap file in created and used during the
generation of the CodedUI can be edited for
optimization purpose.

Methods can be moved, controls renamed and its
search criteria tuned.

The search configuration can be edited from the
properties menu.

us/library/ff398062.aspx

Data driven CodedUI tests
Just as unit tests can CodedUI test be data
craven, run them more as once with different
datasets. These datasets can come from a test
case or other data source like excel and
databases.


Lab 14: Execute the CodedUI during
the build from Visual Studio

Execute codedUI tests from Visual Studio within a build.

Setting up Flavor C
For this lab the test controller isn’t connected to
a team project collection, we are going to
execute the test case from Visual Studio.

In the test settings file we must configure the
roles where the test will be execute.


there has to be a test controller agent
infrastructure in pace.

Define a build and set the test properties to use
the test settings file.

Queue the build and watch the test agent.

Lab 16: Execute the codedUI from
MTM

Setting up Flavor D

The test controller must be associated with the
team project collection.

Create in the Lab Center in MTM a physical
environment from the environment with the test
agent installed and configured.

Lab 17: Execute the codedUI during
the build from MTM

Setting up Flavor E
You need two environments with test agents for this lab, I use a vpc and its host for it.

With the test configuration property in the test
plan properties we can configure for which
configurations the test cases need to be
executed. In this lab we will configure the build
so test cases are executed on these specific
environments.
Default the configuration of a test controller is
for load, run automated test cases on all agents
so we execute test cases as fast as possible.
When you want to run a test case on a specific
environment we need to do some additional
things.

We will execute the test cases from the
command line tool tcm.exe.
Get the necessary information. And compose the
command lines with parameters. Look at the
query in step 2 where we select the test cases.

Open the build defeinition workflow xaml and
add two ‘invokeprocess’ actions where we call
the tcm.exe tool with its parameters (note: the
build workflow definitely need more tweaks as
mentioned in this step when you want to use this
in production, this is only for demo purpose.)

Queue the build and watch the test agents

On Virtualized Environments with
Lab Management and SCVMM

For a complete guid about Virtulized Test environments see the Rangers Lab Guide
http://ralabman.codeplex.com/

From http://ralabman.codeplex.com/

Custome Data Adapter

See lab guid rangers

Azure and test automation
http://www.clemensreijnen.nl/post/2011/03/29/VS2010-ALM-MTLM-usages-patterns-for-
Windows-Azure-hosted-application-development.aspx

Test Manager and an agile
process
http://www.clemensreijnen.nl/post/2010/09/21/Agile-Test-practices-with-Microsoft-
Visual-Studio-2010.aspx

Intelitrace

Appendix
Proposed training agenda 2 day
training

Student and trainer Environment

MTLM Visual Studio 2010 ALM workshop

Recomendados

Recomendados

Más contenido relacionado

La actualidad más candente

La actualidad más candente (8)

Similar a MTLM Visual Studio 2010 ALM workshop

Similar a MTLM Visual Studio 2010 ALM workshop (20)

Más de Clemens Reijnen

Más de Clemens Reijnen (20)

Último

Último (20)

MTLM Visual Studio 2010 ALM workshop