Learn about how the core NuGet team the NuGet Client project. The project has challenges related to: - Open Source - Hard Deadlines - Very Large Backlog - Many Conflicting Requirements - Git We'll talk about our strategies for: - Issue tracking - Branching - Continuous integration - Testing - Release sign-off

1. Developing NuGet
JEFF HANDLEY | DEVELOPMENT LEAD | NUGET | MICROSOFT
@JEFFHANDLEY
JEFFHANDLEY.COM
1

2. Airborne (1993)
2SOURCE: WWW.IMDB.COM

3. 3

4. 4

5. 5

6. NuGet Landscape
WHAT MAKES UP THE NUGET PROJECT
6

7. NuGet Project
Open Source (owned by Outercurve Foundation)
◦ Releases shipped by Microsoft
Included in all Visual Studio installations
Millions of users (some knowingly, some not) around the world
Dozens of high-profile dependent projects at Microsoft
International contributors
◦ Only a few external contributors are in the US
Very large backlog of issues
Conflicting requirements
Backward compatibility
Mixture of hard and soft deadlines
Extremely costly test passes
7

8. Outercurve and Microsoft
NUGET CLIENT
Outercurve’s
NuGet Project
Microsoft’s
NuGet-Based
Package Manager
In the Box with
Visual Studio
NUGET GALLERY
Outercurve’s NuGet
Gallery Project
www.nuget.org
Azure Hosting
provided by
Microsoft
Operated by
Microsoft on behalf
of Outercurve
8

9. NuGet’s Core Team
NUGET CLIENT
Visual Studio Extension
WebMatrix Extension
NuGet.exe Command-Line
NuGet Packages
3 Developers
2 Testers
NUGET GALLERY
www.nuget.org
staging.nuget.org
docs.nuget.org
build.nuget.org
3 Developers
2 Testers
Always 1 member on ops/support rotation
9

10. NuGet’s Core Team
NUGET CLIENT
Visual Studio Extension
WebMatrix Extension
NuGet.exe Command-Line
NuGet Packages
3 Developers
2 Testers
NUGET GALLERY
www.nuget.org
staging.nuget.org
docs.nuget.org
build.nuget.org
3 Developers
2 Testers
Always 1 member on ops/support rotation
10

11. Release Management
NUGET CLIENT
11

12. NuGet Releases
New release every 10 weeks
since NuGet 1.5
12
2/ 12/ 2011
1.1
3/ 30/ 2011
1.2
4/ 25/ 2011
1.3
6/ 17/ 2011
1.4 8/ 30/ 2011
1.5
12/ 13/ 2011
1.6 4/ 4/ 2012
1.7
5/ 23/ 2012
1.8 6/ 19/ 2012
2.0
10/ 4/ 2012
2.1 12/ 12/ 2012
2.2
2/ 15/ 2013
2.2.1 4/ 25/ 2013
2.5
6/ 26/ 2013
2.6 8/ 22/ 2013
2.7
10/ 7/ 2013
2.7.1
1/ 14/ 2011
1.0

13. NuGet in Visual Studio Releases
13
2/ 12/ 2011
1.1
3/ 30/ 2011
1.2
4/ 25/ 2011
1.3
6/ 17/ 2011
1.4 8/ 30/ 2011
1.5
12/ 13/ 2011
1.6
Introduced
with
ASP.NET
MVC3
and
WebPages v1
4/ 4/ 2012
1.7
5/ 23/ 2012
1.8 6/ 19/ 2012
2.0
10/ 4/ 2012
2.1 12/ 12/ 2012
2.2
2/ 15/ 2013
2.2.1 4/ 25/ 2013
2.5
6/ 26/ 2013
2.6
8/ 22/ 2012
Visual Studio
2012 RTM
4/ 4/ 2013
Visual Studio
2012 Update 2
6/ 26/ 2013
Visual Studio
2013 Preview
9/ 9/ 2013
Visual Studio
2013 RC
1/ 14/ 2011
1.0
5/ 31/ 2012
Visual Studio
2012 RC
2/ 28/ 2012
Visual Studio
2012 Beta
8/ 22/ 2013
2.7
10/ 7/ 2013
2.7.1
Visual Studio
2013 RTM

14. Release Tiers
CI build from Outercurve Foundation always available at http://build.nuget.org
◦ Usually 100 or fewer users
◦ Build created from every commit
Beta/RC builds from Microsoft occasionally available at http://nuget.codeplex.com
◦ 1,000 – 2,500 users
◦ Published after we hit code-complete
◦ Real-world testing of newly introduced features or significant design changes
Final builds from Microsoft published to the Visual Studio Extension Gallery
◦ 500,000 – 750,000 users
◦ Released after our test team signs off, a few weeks before Visual Studio locks down for its next release
◦ Last stage real-world testing from users that are knowingly using NuGet and are a bit more forgiving
In the box with Visual Studio
◦ Millions of users
◦ Many of these users don’t know what NuGet is and will file Visual Studio bugs if we break them
14

15. Backlog
CodePlex: 858 open issues as of 10/4/2013
Microsoft Internal Issues
◦ TFS: Fluctuates from 0 to 25
◦ Dr. Watson reports
◦ Microsoft Connect
◦ Microsoft shipping requirements
◦ Future Visual Studio release issues
Strategic Product Features
◦ Schedule alignment for release within Visual Studio and Visual Studio Update
15

16. Strategic Product Features
Package Manager for the Microsoft Development Platform
◦ Not just ASP.NET
◦ Not just .NET
◦ Includes C++, Windows Store, and Windows Phone
◦ Cross-platform support with Command-Line, Xamarin Studio, MonoDevelop, and SharpDevelop
Improving Package Discovery (through collaboration with the NuGet Gallery crew)
Avoiding Package Version Hell
Platform Multi-Targeting and Re-Targeting
New Usage Workflows for Improved Application Maintenance and Ecosystem Growth
Helping Improve Gallery Reliability
16

17. Triaging Issues
1. Is it a high-priority issue (crash, hang, regression)?
◦ Include it in the current release
2. Is the bug fix or feature needed for the current release?
◦ Include it in the current release
3. Is it a high value item that fits in the next release?
◦ Include it in the next release
4. Do we agree with the concept?
◦ Put it in the “Up For Grabs” release
◦ Ideally picked up by the community
◦ High-voted items are reviewed when planning releases and some items are pulled in
5. Do we disagree with the concept?
◦ Close it
17

18. Priorities
1. Strategic roadmap that integrates with the Microsoft development platform
◦ Features planned for 2-3 releases at a time
◦ Align with Visual Studio releases: Preview, Beta, RC, RTM, Visual Studio Update
2. Community-reported high-value bug fixes or features
◦ Including those that come with external pull requests
3. Highly-voted community feature requests
4. Community-reported high-value bug fixes
5. Microsoft partner team bug reports
6. Microsoft partner team feature requests
7. Community-reported low-value bug fixes
18

19. Philosophy
Small and easy bug fixes are low priority
◦ Otherwise, we’d spend 100% of our time making small changes that affect only a handful of users
Don’t prematurely act upon cool ideas
◦ Does the idea spark new strategic concepts?
◦ Do other new ideas relate?
◦ Can the feature be implemented with backwards compatibility?
◦ Could this feature become obsolete based on other potential strategic directions?
Favor community-requested features over partner team features
◦ Avoids scenario-specific features that conflict with broad usage
◦ Encourages partner teams to find workarounds with existing features
◦ Those workarounds often surface pain points others might also be experiencing
We’d rather delight 1,000,000 external users than gain 1 more internal partner team
19

20. Conflicting Requirements
Microsoft’s Customers vs. NuGet’s Ecosystem
◦ Group Policy control over package sources
Bold New Features vs. Backwards
Compatibility
◦ Package Restore
◦ Build-time reference resolution
NuGet User Experience vs. Microsoft’s
shipping requirements
◦ Online consent
◦ Loading only Microsoft-signed assemblies
◦ Modified content files
Favor user desires as much as possible
Work within the constraints of Microsoft’s
shipping requirements
Negotiate and compromise with
privacy, security, and legal representatives to
arrive at acceptable implementations
20

21. Source Control
NUGET CLIENT
21

22. master
merge
Feature
A
Feature
B
Branch: 2.7
Branch: 2.7.1
mergemerge
Branch: 2.8
Tag: 2.7
Tag: 2.7.1
Git
Branch per feature
Branch per version
Tag for release
22

23. Continuous Integration
Outercurve Builds
◦ TeamCity – running on Virtual Machines in Windows Azure
◦ 1 VM build agent with .NET 4.0 for building the VS Extension for VS 2010 and VS 2012
◦ 1 VM build agent with .NET 4.5 for building the VS Extension for VS 2013
◦ Hosted at http://build.nuget.org
◦ Produces non-localized, unsigned builds with Outercurve branding and licensing
◦ “Developer Branches” build configuration builds all “dev-” prefixed branches
Microsoft Builds
◦ TeamCity – running on Virtual Machines in a group lab
◦ 1 VM build agent with .NET 4.0 for building the VS Extension for VS 2010 and VS 2012
◦ 1 VM build agent with .NET 4.5 for building the VS Extension for VS 2013
◦ Only accessible on the corporate network
◦ Produces localized, signed builds with Microsoft branding and licensing
23

24. Going Dark
Microsoft-confidential work cannot go to CodePlex
◦ Integration with new versions of Visual Studio
◦ Support for new platforms (e.g. Windows Phone 8)
Team Foundation Service private project
◦ Fork the NuGet Git repo into that project
◦ Move confidential development into that fork
◦ Keep all other development on CodePlex
Merge commits into TFS daily
Configure our Internal Builds to use the TFS fork instead of CodePlex
24

25. Testing
NUGET CLIENT
25

26. Release Timing
Release sign-off takes about 4 weeks after code-complete
◦ 40% of our 10-week release cadence is spent testing and fixing bugs
Snap releases to the Visual Studio schedule
Release to the gallery first, and then integrate into Visual Studio’s build
Between Visual Studio Beta and RTM, we have 2 or 3 tight quality-driven releases
◦ Meet Microsoft’s release requirements
◦ Address any partner team blocking issues
◦ Focus on extremely broad use of the product
After Visual Studio hits RTM, we go big with features and aim for a 3-month release
26

27. Test Matrix Dimensions
Visual Studio SKUS
◦ VS 2010
◦ Express for Web
◦ Express for Phone
◦ Pro+
◦ VS 2012+ - All SKUs
Project Types
◦ Virtually all Microsoft Project Types
◦ Lightswitch
◦ Windows Store
◦ Windows Phone
◦ Portable Class Libraries
Languages: C#, VB, F#, JavaScript
Operating Systems
◦ Windows XP
◦ Windows Server 2003+
◦ Windows Vista and Windows 7
◦ Windows 8
◦ Windows 8.1
Different types of NuGet Packages
Different types of NuGet Package Sources
◦ Gallery
◦ NuGet.Server
◦ Local Disk
◦ UNC Share
Source Control Integration
◦ TFS
◦ Git, Mercurial
◦ None 
NuGet Clients
◦ VS Extension
◦ WebMatrix Extension
◦ NuGet.exe Command-Line
Plus, all of the combinations of actual features
27

28. Achieving Release Sign-Off
Unit Tests managed and run by developers
End-to-end Functional tests managed and run by developers and through automation
◦ Based on a PowerShell test system included within our project
◦ Runs within Visual Studio, automating VS through the DTE
QA Tests for interactive testing
◦ Mostly automated, but exploratory testing always finds more bugs
◦ Run across a farm of machines in our group’s test lab, covering the machine configuration matrix
◦ A representative set of combinations give us confidence
◦ Uses an internal VS automation library to simulate user actions in VS
Group-wide bug bashes
Company-wide partner team bug bashes
Public Beta testing from CodePlex builds
Reach 0 bugs in CodePlex for the current release
28

29. Throughput
109 Weeks
• Total time working
on those 10 releases
10 Releases
• 816 work items have
been completed
3 Developers
• Averaging 27 work
items per release
2 Testers
• Each testing 3.7 bug
fixes or features per
week
29

30. Success
Never defined by throughput
No regressions
No minor updates necessary
Users delighted by new features
Took steps toward our strategic roadmap
Addressed highly-voted community bugs and features
Accepted a level of failure in smaller user base releases
30

31. Success Rate
NuGet 1.6
◦ Required a 1.6.1 update
◦ Regressions
◦ Missed bugs in new features
NuGet 1.7
◦ Success
NuGet 1.8
◦ Success
NuGet 2.0
◦ Success
NuGet 2.1
◦ Success
NuGet 2.2
◦ Success
NuGet 2.5
◦ Success
NuGet 2.6
◦ Success
NuGet 2.7
◦ Required a 2.7.1 update
◦ Regression
◦ Missed bugs in new features
31

32. Tiered Releases are Key
Our tiered release approach worked for 2.7
2.7.1 update in place in time for integrating
into Visual Studio
Thousands of users affected instead of millions
As Visual Studio’s broad user base jumps from
NuGet 2.2 to NuGet 2.7, they’ll have no idea
we ever introduced regressions
You wouldn’t have noticed the camera in
Airborne if I hadn’t pointed it out to you
32

33. Q & A
Get a NuGet Sticker!
Twitter: @jeffhandley
Website: jeffhandley.com
33