A super enjoyable and entertaining walk through API memory lane, then a primer on how to test API's from a unit, integration, and monitoring standpoint. Then a demo on a CI/CD implementation I created at Snagajob.
29. What • Why • When • Who • Where • How
● API testing…What’s the point, man?
29
● Quick feedback loop…like 200 ms quick
● Easy to create/run/maintain/extend
30. What • Why • When • Who • Where • How
● Where in the SDLC should we start validating API’s?
3
0
Before • During • Interval •After • Trigger • Manually
31. What • Why • When • Who • Where • How
● Where in our tech stack should we test?
3
1
Local • Sandbox • Staging • Prod
32. What • Why • When • Who • Where • How
● Who should own testing?
3
2
● Define “testing”, change culture
● PO, QA, Dev, robots
33. What • Why • When • Who • Where • How
● What approaches and tools should we use?
3
3
exploratory • regression • performance
34. Popular tools (free)
Name Summary Pros Cons*
Swagger UI Dynamically-generated
documentation for API’s
• Free. Beautiful, easy, & everyone likes it (dev, QA, business).
• “Try it out!” button.
• Not built for testing.
Postman
(free)
SaaS-based desktop solution
for REST & SOAP
• Awesome UI
• Super easy to test single calls or huge collections.
• Environment & global variables + JavaScript libraries
• Open-sourced CLI package that can run exported
collections + Active community
• No team collaboration
SoapUI
(free)
• Free
• Best solution for SOAP (I think)
• Less easy to use
Browser
console
Capture incoming HTTP
traffic from any web UI
• Free
Fiddler Capture incoming HTTP
traffic from any web UI
• Free
• Allows filtering and in-depth analysis of calls
• Sometimes doesn’t work on certain
browsers or certain sites.
35. Popular tools (not free)
Name Summary Pros Cons*
Postman
Pro
Free Postman + more… • Postman Pro API (grab collections w/ a token from any
environment)
• Team collaboration
• Prod monitoring
• Mock servers
• $8/mo per user
Runscope SaaS web solution for REST • Awesome UI
• Does pretty much everything (smoke, functional,
performance).
• Environment & global variables.
• Reporting & dashboard
• Great support
• Not free ($90/mo for 5 users).
• Gets slow when you start editing
large collections of tests.
• Must install agent to trigger.
Custom
framework
• Infinite possibilities
• More secure (in theory)
• Huge upfront commitment of time
and expertise. Not free.
• Enough rope to hang yourself
44. The problem
1. Greenfield development. Developing
lots of new stuff.
2. Regression issues. New features are fine,
but we’re breaking older stuff with each deploy.
3. UI tests suck. They’re (1) inconsistent; (2)
difficult to maintain; and (3) time-consuming to
create.
45. Additional hurdles
1. Aggressive timeline. We’re trying to
stay ahead of the competition.
2. Evolving product. Ongoing user testing
means changing front-end experience,
sometimes even underlying architecture.
3. Dev mentality. Developers do not
write/maintain automated tests or vet releases.
Spotify model (embedded QA)
46. Our solution
1. Automated testing on API’s against any
integration environment.
Bonus: Developers can maintain
Extra bonus: PO’s can, too.
47. The steps
1. Explore API testing. Make a list of options & rank them.
Prototype top 3 & score them.
Build consensus as a QA chapter.
2. Implement it on 1 product team. Find a willing
collaborator and start using it.
3. Roll out org-wide. Bring in Engineering leadership,
including app dev & devops. <???>. Profit.
48. Proof-of-concept demo
The proof of concept we ended up trashing, but that works!
GitHub push
(QA to UAT)
1
CircleCI build begins
(remote servers)
2
Trigger Runscope tests
in QA environment
3
App deployed in full to
UAT environment
4
Slack + Results display5
49. v2
This is what’s cooking at the moment…but doesn’t work quite yet ;)
CircleCI build begins
(remote servers)
2
Postman rests run in
QA environment
3
Deploy triggered (SNS +
Lambda)
4
Results indexed
with ElasticSearch
6
Slack +
Results display
7
GitHub push
(QA to UAT)
1
App deployed in
full to UAT
5
50.
51.
52. Challenges
1. Environment. Where exactly do we test the code?
Intermediary environment problem.
2. Triggering. How can we make CircleCI (hosted) talk
directly to our custom-built internal services?
3. Extensibility. How can we build this in a way that (a) acts
as a true service; (b) does not require regular maintenance; and
(c) does not require hand-holding?
API testing 101
About Me
About Snagajob
HTTP
Is how the web communicates. Super quick over the wire, 234 ms.
API's
An API isn’t the same as the remote server — rather it is the part of the server that receives requests and sends responses.
The code that takes the HTTP requests and turns it into business vaue.
Stores in database so another app can access it,
Applies some logic to transform it into a number someone really cares about
Feeds it into some proprietary algo to drive insight
All of the above
Public & Private (amazon button story)
Microservices are new hottness
HTTP basics
this is a protocol, just like a language.
1) headers (includes auth, tells service what type of content it's delivering, and what to accept in terms of a response)
2) querystrings (optional) which are on the actual route and specify what to ask the application for (e.g. `http://gateway.api.snagQA.corp/customers?isActive=true` is asking for customers where isActive=true)
3) body (also optional) which is giving some detailed data to the service
so, those 3 things
the BODY is usually part of anything related to creation, but it can also be passed to an existing entity to overwrite some existing data.
so new customers will have large bodies
Apple introduces the iMac in August of 1998. iBook comes in 1999.
XML 1.0 becomes a W3C recommendation on February 8, 1998.
IE just starting to eclipse NetScape
Jeff Bezos of Amazon.com is named Time Person of the Year December 27, 1999.
So, now the internet is like super OPEN, all thanks to HTTP among other things
Before, the internet was this hard-wired rigid proprietary place. Now anyone can sell stuff there.
All of the sudden, to do business on a flattened global marketplace, business apps had to get their shit together so they could talk to all of the services required to do business.
Before, you’d have to call a sales person who sat a desktop behind a firewall in a building somewhere, and they’d key in your order into some crusty old legacy app.
Now, with all of these people able to submit online orders, and all this demand to talk to other services quickly to make sure those orders got through and were fulfilled, CEO’s were facing a dilemma:
Rewrite our apps so they’re client-server ready (instead of being mainframe) - $10 million!
Figure out some way to open up those old closed systems so they could talk to eachother and the internet in general.
Solution? The message bus. SOA. Just make sure mainframe can throw stuff onto a message bus. So far, this is all internal. All of the sudden, people are having the clever idea of opening up their API’s to the e-commerce world (other systems or business partners). This is where SOAP comes in.
2000 - SOA & the word “loosely-coupled” is now a thing
Time to market I snow like ZERO
Salesforce talks to your apps for you so they don’t have to talk to each other (selling their API)
Salesforce.com launched its enterprise-class, web-based, sales force automation as a "Internet as a service". XML APIs were part of Salesforce.com from day one. Salesforce.com identified that customers needed to share data across their different business applications, and APIs were the way to do this.
Dave winer invents XML-RPC, the idea of wrapping HTTP requests in XML to standardize them
MEANWHILE, SOAP gets W3C endorsement thanks to push from IBM & Microsoft, and Roy Fielding goes off to create REST and publishes a huge treatise as to why it’s better. Soon, all the big guys are using it (IBM, Microsoft, Sun, Oracle, HP).
Fielding wanted to make more of the fact that _____ had built-in HTTP methods (GET, POST, DELETE, PUT). Another important difference between REST and SOAP is that it's resource-based. So the API accesses nouns (aka URIs), instead of verbs. Then, HTTP verbs are used to access those resources.
At the time, this is how SOAP worked.
XML was the new hotness, Before JSON became the go-to hypermedia.
JSON is way more succint, compact, easy-to-read, and is uniform so everyone knows what it does after 3 seconds of looking at it
This simplicity, while it has limitations, is good and lightweight, when you start dealing in complex systems. It forces simple code.
FIRST ATTEMPT AT STANDARDIZING web services
CORBA, DCOM, to share data or objects across network boundaries.
SOAP exposes operations that implement some business logic
The goal: designed to loosely couple systems for integration of diverse systems
STRONG typing = stability and governance
Had already been helping Tim Berners-Lee in forming the World Wide Web Consortium (to start defining standards), specifically in writing the first standards list for HTTP.
Architectural Styles and the Design of Network-based Software Architectures
Instead of SOAP (operations that represent the implementation of some business logic), REST is all about resources which represent data.
Instead of storing server-side
http://spf13.com/post/soap-vs-rest/
2000- The launch of the eBay API was a response to the growing number of applications that were already relying on its site either legitimately or illegitimately. The API aimed to standardize how applications integrated with eBay, and make it easier for partners and developers to build a business around the eBay ecosystem. eBay is considered the leading pioneer in the current era of web-based APIs and web services and still leads with one of the most successful developer ecosystem today.
2002- they actually start offering an accessible API for anyone to use. And it’s in REST, so it’s easy to use, too.
2004
Flickr launched their own REST API in August of 2004, just in time for the rise of social networking and blogging. They quickly became the go-to platform for images, which bloggers were finally able to easily embed on their sites and social media feeds.
They really led the way.
Local – you can run them locally and stub out services
Sandbox– you can run them in an integration environment to see if their “shippable” code passes
Integration – you can run them on a staging environment to see if a final PR is ready for release
Prod – you can run them on an ongoing basis as a monitor