VS2010 ALM MTLM usages patterns for Windows Azure hosted application development

1. ALM 4 Azure
Automated ALM for Azure Cloud Development
1

2. AGENDA
• ALM
• Azure
• ALM 4 Azure
2

3. Clemens Reijnen
• Sogeti | Management Consultant
www.ClemensReijnen.nl
@ClemensReijnen
Clemens.Reijnen@Sogeti.nl
3

4. application lifecycle management
4
http://www.clemensreijnen.nl/post/2008/02/18/ALM-Definitions.aspx

5. ALM - Agile | Lean
Application Lifecycle Management
Collaborate over Processes and Products with tools.
Agile, Lean and also TMap, DyA, ITIL
Methodologies, principles customized, adopted
and merged from all ALM roles.
optimized environments for supporting
the ALM team goals
5

6. work together
__with work packages
__on alm artifacts
6

7. 7
http://msdn.microsoft.com/en-us/library/bb385832.aspx

8. Guest OS Versioning
identical – independent environments 8

9. Development & Deployment
SDK
*.cspkg
Config
*.cscfg
9

10. Start
10

11. ALM 4 Azure
Cloud System
Goals: Collaborative
High Quality
Flexible
Automated
Repeatable
Efficient
11

12. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
12

13. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
13

14. Scenario 1:
Developer only
1 Source Production
Repository Staging
C# 3
TEAM (implementatie + unittests) (TFS)
Developers implementeren het systeem
 Local unittesting
‘SprintReview’Build
 Compile
 Run Unit Tests
2
14

15. Stages of Service Deployment
Taken from:

16. DEMO
Deploying a
Cloud Service
from Visual
Studio
16

17. Scenario 1:
Developer only
Pro: Con:
Easy installation and configuration No collaboration
Single click deployment from VS2010 Easy deployment errors (configuration)
What about test and ops
17

18. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
18

19. Scenario 2:
Developer with
manual tester
Source Production
Repository Staging
C# 3
TEAM (implementatie + unittests) (TFS)
1 Developers implementeren het systeem
 Local unittesting
Testers specificeren testcases ‘SprintReview’Build 4
 “Dry” run van tests tegen Development Fabric  Compile
 Run Unit Tests
2
19

21. Microsoft Test
Manager
21

22. Source Production
Repository Staging
TEAM (TFS) 3
1
4
2
1 2 4
Functional, system, Build Verification Platform accepatance
manual tests executed Tests (unit tests) and tests executed on
in compute emulator other quality gates, Azure stating with
with MTM. executed in the build. MTM.

23. 1
Get the CSX folder and CSFG file
2
3
4

24. DEMO
Test a Cloud
Service with
Microsoft Test
Manager
24

25. Diagnostic adapters

26. DEMO
Azure Logs
Diagnostic Data
Adapter
26

27. Capabilities
• Link test cases to requirements
• Fast Forward
• Test data Iterations
• Test case management
• Test configurations
Interesting capabilities 4 Azure
• Shared steps ( 4 VISP Swap )
• Diagnostic adapters ( Rich bug reports )
Not working capabilities in Azure
• Test Impact 4 manual tests
• Intelli Trace 4 manual tests (although possible custom
adapter)
27

28. Scenario 2:
Developer with
manual tester
Pro: Con:
Easy installation and configuration Easy deployment errors (configuration)
Single click deployment from VS2010 Time consuming (deploy and test)
Testers connected, same heartbeat as dev Not repeatable (annoyed testers)
Proven quality Testers connected 
28

29. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
29

30. Scenario 3:
Developer with manual tester
and deployment build
3
Source Production
Repository Staging
C#
TEAM (implementatie + unittests) (TFS)
1 Developers implementeren het systeem
 Local unittesting
Testers specificeren testcases ‘SprintReview’Build 4
 Dry run van tests tegen Development Fabric  Compile
 Run Unit Tests
2
 Package (CSRun.exe, CSPack,exe, CSManage.exe)
 Deploy 2 Staging (CMdLets, Powershell)
30

31. Automatic Deployment
Management API
CSPack
CSRun
CSManage
PowerShell CmdLets
Build Targets
http://scottdensmore.typepad.com/blog/2010/03/azure-deployment-for-your-build-server.html
http://blogs.msdn.com/b/tomholl/archive/2011/02/23/using-msbuild-to-deploy-to-multiple-windows-
azure-environments.aspx
31

32. Deploy during build
1. Change .ccproj project
• New build target which uses the built-in CorePublish target
2. Create PowerShell script
• Called by build target and uses Azure CMDLets
3. Edit Build Config
• Add the necessary parameters
4. Do something with certificates
• Put the Azure certificate on the build server
32

33. DEMO
Deploy a Cloud
Solution within
the Build
33

34. Scenario 3:
Developer with manual tester
and deployment build
Pro: Con:
Easy installation and configuration Time consuming testing
No click deployment from build Application can contain ‘annoying’ bugs
Repeatable ‘proven’ deployments* Build workflow knowledge necessary
Testers connected, same heartbeat as dev Powershell, ccproj tweaks, target files,
Proven quality certificates
*demo needs some additional environment tweaks for different csfg files
per environment
34

35. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
35

36. Scenario 4:
Developer with automated regresion
tests, manual tests and deployment build
3
Source Production
Repository Staging
C#
TEAM (implementatie + unittests) (TFS)
Test cases
(automated UI tests)
1 Developers implementeren het systeem
 Local unittesting
4
Testers specificeren testcases ‘SprintReview’Build
 Dry run van tests tegen Development Fabric  Compile
 Automation van testcases 5  Run Unit Tests
(moeten gedraaide testcases zijn voor CodedUI) 2
 UI automation tests run tegen Development Fabric  Package (CSRun.exe, CSPack,exe, CSManage.exe)
en associëren met test plan  Deploy 2 Staging (CMdLets, Powershell)
36

37. CodedUI

38. Where? When? How? Who? Why? What?

39. Where? When? How? Who? Why? What?

40. Different URL’s
1
2
40

41. UIMap Editor
41

42. DEMO
Create automatic
Tests for the Cloud
Application with
CodedUI
42

43. Where to run the auto test?
a) At the Build server?
b) In the Development environment?
c) In the Test environment?
d) In the Azure environment?
43

44. Visual Studio Team
Foundation
Server
Test Manager
Build Server Test Agent
Test Agent
Test Agent
44

45. Configurations for automated test execution:
On physical environments (no lab)
TFS 2010
VS 2010 Build Controller
Build Agent
MTM 2010 Test Controller
Test Agent
45

46. Automated Test Settings
46

47. Configurations for automated test execution:
A
Flavor A: Execution from VS2010…
Real developer test / test automation dry run
(Triggered by: right mouse click – run tests, in test project)
http://msdn.microsoft.com/en-us/library/dd286580.aspx
No additional configuration needed*
* Data Diagnostic Settings in the Local.Testsettings file are configurable.
Pro: Con:
Easy setup No collection of test results in TFS
Debug-able test automation Run on developer environment 47

48. Configurations for automated test execution:
B
Flavor B: Execution during build with Build controller…
not recommended, strange to run UI tests on build
Part of Build Verification Tests
(triggered by: build)
Set the Build Service to run in interactive mode
http://blogs.msdn.com/b/mathew_aniyan/archive/2009/05/26/coded-ui-test-in-a-team-build.aspx
Configure the build to run the tests
http://msdn.microsoft.com/en-us/library/ms182465.aspx#CreateBuildType
* Data Diagnostic Settings in the *.Testsettings file are configurable.
Pro: Con:
Easy setup No collection of test results in TFS
Test results in build result Build Controller needs to run in interactive mode
Tests are executed on build environment 48
Run on build environment

49. Configurations for automated test execution:
C
Flavor C: Execution during build with Test controller… B
Preferred configuration above flavor
Part of Build Verification Tests on multiple test agents
(triggered by: build)
Configure Test Controller (don’t register it with a project collection )
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestControllers
Configure Test Agents on clients (interactive mode)
http://msdn.microsoft.com/en-us/library/dd648127.aspx#TestAgents
Configure *.Testsettings file in solution to use Test Controller and Test Agents
Configure the build to run the tests (see B)
Pro: Con:
Test run on test environments No collection of test results in TFS
Tests run on multiple environments Harder to configure
Test Results in Build result Need for specific test client environments 49
Test Settings from VS

50. Configurations for automated test execution:
D
Flavor D: Execution from Microsoft Test Manager…than BVT
Other type of test
Part of Regression Tests
(triggered by: MTM user, right mouse click on test case, run)
Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
http://msdn.microsoft.com/en-us/library/ee390842.aspx
http://msdn.microsoft.com/en-us/library/dd293551.aspx
Associate CodedUI test with WI Test Case from VS.
http://www.richard-banks.org/2010/11/how-to-use-codedui-tests-watin-and-mtm.html
Pro: Con:
Test run on test environments Manually triggered by Tester
Tests run on multiple environments Hard to configure
Test Result in MTM and TFS Need for specific test client (same as MTM?)
50
Test Settings from MTM

51. Configurations for automated test execution:
E
Flavor E: Execution from MTM duringPreferred configuration above flavor C
Build…
Part of BVT Flavor D and E can be configured together
(triggered by: Build)
Configure Test Controller (register it with a project collection )
Configure Test Agents on clients (interactive mode, can be the same as MTM)
Configure Lab Center in MTM to use test controller and create test ‘agent’ environment.
Associate CodedUI test with WI Test Case from VS.
Create Build task to run MTC or MSTEST task for Test Plan
http://blogs.microsoft.co.il/blogs/shair/archive/2010/10/30/how-to-run-coded-ui-tests-from-command-line.aspx
Pro: Con:
Test run on test environments Hard to configure
Tests run on multiple environments Need for specific test client (same as MTM?)
Test Result in MTM and TFS
Triggered by build 51

52. DEMO
Automatic test a
Cloud Service with
Microsoft Test and
Lab Manager
infrastructure 52

53. Scenario 4:
Developer with automated regresion
tests, manual tests and deployment build
Pro: Con:
No click deployment from build Build workflow knowledge necessary
Repeatable ‘proven’ deployments* Powershell, ccproj tweaks, target files,
Testers connected, same heartbeat as dev Certificates
Proven quality Test Infrastructure knowledge necessary
Automated BVT on different Environments A balanced thinking of test automation
Comfortable Acc Testing
Done Done
*demo needs some additional environment tweaks for different csfg files
per environment 53

54. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
54

55. Automated ALM for Azure Cloud development
Efficient, repeatable with proven quality fast
flexible delivering of useful functionality … 4
Acceptation
Making optimal use of the characteristics of Azure:
- Identical environments (Guest OS versioning).
- Separate in depended staging production environments. ‘Release ’Drop
- Easy up and downgrade of applications.  Package (same as test env
- Unambiguous deployment mechanism 3 package)
- rich diagnostic capabilities  Acc testing met MTM en
- Simplified Application Deployment & Management Diagnostic Data Collectors
- …
BUILD Production
(TFS) Staging
C#
TEAM (implementation + unittests)
Test cases
(automated UI tests)
Developers implement the system
 Local unit testing
Testers specify test cases ‘SprintReview’Build
 Dry run tests cases against Development Fabric  Compile
 Automation of test cases  Run Unit Tests
1 
2 Package (CSRun.exe, CSPack,exe, CSManage.exe)
 UI automation tests execute against Development  Deploy 2 Staging (CMdLets, Powershell)
Fabric and associate with a test case  Run automated Test Suites (MTC.exe) with different
environments (test controller – agents infrastructure)
Operations determine instrumentation data  Collect and publish test results
 DFO, SLA  “Build Verification Tests” green -> deployment 2 ‘test
 Usages Metrics, payment 'production 55

56. Scenario 1: Dev only
Scenario 2: Dev and Test
Scenario 3: Dev, Test and Build, Deploy
Scenario 4: Dev, Auto Tests and Build, Deploy
Scenario 5: Dev, Auto Tests, Build, Deploy, Acc test
and Operations
Scenario 6: TFS on Azure
Scenario 7: Other Infrastructure on Azure
56

57. 57