There is so much we don’t talk about in software development.
I’m going to tackle one of these gaps. Testability. By that I mean, the ability to observe, control and understand a system to some degree. All of which happens to be useful when testing.
I’ve lost count of the number of times testers have got really upset because a system was hard to control and was utterly opaque. Or vomits vast amounts of information, sets databases into mysterious, unreproducible states. Left scratching heads, and then deciding to just go with it and see what happens.
Then the really upsetting part, when weird, wonderful issues start appearing in Production because of a hard to test system that overflows with poorly understood complexity.
I hate to say it, but sometimes testers just don’t get it. Or even think about it that much. Tend to plough on. Or say ‘its hard to test, so lets do more testing’
First and foremost if you don’t have a paradigm of what testability is and why it is beneficial, then like many things in life you might struggle.
Even with if the opportunity presents itself, knowing what to ask for can be a problem. If you don’t know what good looks like, how can one ask for it.
After all, one tends to look at problems from within the prism of your role, which I once did. Meaning I thought testability was about testing alone, turns out its much more than that.
I’ll tell you story of how I once asked for testability improvements without really knowing what to ask for and how, and how it turned out different to what I imagined. With the hope that one day you guys will be able to take these lessons forward, and enhance your testing lives.
So we’ve all had a project or two that really moulds what we believe. Really challenges your core assumptions.
Sometimes the hard way. It was a dreaded ‘rewrite’ but with a welcome clean slate and a sympathetic product person with technical awareness, rare beasts that they are. The system was a hub for orchestrating other internal and 3rd party services, with many integrations to consider. The API was the key interface for business volume, so we focused our efforts there.
Problem is we were rewriting something that wasn’t really all that testable. And the brief was to ‘replicate what was already there.’ Sigh, as if that is even a thing. And no really liked it as it was.
Things were really getting started so I felt I need to really stake a claim. Trying to cram it in. Like the product people who only get a release every year. Really cram it in.
That was my first mistake.
I pressed on with my shopping list:
Build something that’s like live. You’ve got to have a like live environment right? Otherwise your testing won’t tell you enough about the risks you identify? Especially performance.
Years of no unit tests? Lets go back and sort that. We’ll have a team that understand the code intimately and bridge the knowledge gap we fall into so often.
I’m a firm believer in logging being fundamental to testing. Its still one of my first questions to many testers. Can you see the logs when you test? Then your isolated test through the interface may be swanlike – serene above the water, flapping furiously below.
And learn php, the language of the application. I’ve always believed in being able to build rudimentary versions of the systems I test. RESTful API’s, Android Apps many other things. Sensible, promotes technical awareness. Strengths and weaknesses.
We were ready to get started, with what I thought was testability in the bag!
So what actually happened?
Despite my shopping list a different set of testability needs emerged.
In order to build the large we actually needed to start small. Surprise surprise, the current low testability system we were rewriting was also hard to deploy.
We decided to build a local environment for every team member regardless of discipline. How we learnt together, testers picked up bash skills, devs felt the pain of the ops guys who had to deploy this thing and hope, meaningful smoke tests emerged. Nice!
With a local environment in hand, we, as team members had a powerful means of observing, controlling and understanding a system. Our flow as a team improved, much less time spent in queues and waiting.
The like live, while still of value, became less important and was built later. In fact we rationalised environments, as most meaningful testing was done locally.
I’ve never looked back, testing locally is a massive part of my testing life and I advocate it greatly.
One of our key omissions was establishing how our dependencies affected our testability.
The system under test was hub for many other systems, the level of testability in those systems directly impacts your own system. We integrated with some opaque, ancient systems, with low remaining understanding. We spent lots of time understanding these systems. One dependency had a crazy, invisible naming convention for accounts, which was hard coded (7_password_7). Nuts.
If you can’t observe, control and understand your dependencies meaningfully, you are struggling. After all you will need to test your system in its totality eventually. Narrowness of thought costs you in massively integrated systems.
Relationships was the key here. With the people who build the systems that affect your system primarily.
Clean, understandable programmatic interfaces too, which are perhaps versioned, and change won’t break everything of everyone elses without telling them. We ended up mocking the majority of these out, wouldn’t it have been lovely if those who knew the systems had built those mocks. Sets the tone!
Devops. There I said it. I can almost hear the rolling eyes.
Testers who don’t know their ops pain struggle to influence testability. The power to observe and control helps to salve ops pain. As ops can become your biggest advocates.
During the project it became evident that it wasn’t the api traffic which caused pain, it was the obscure batch system, using ftp to move files around.
We built small tools to manage ftp accounts, build test files, manage cron schedules. Then we shared them with operations and they loved them.
On further investigation, the batch system was great at reporting success but terrible at failure. Things are bad messages. Focus on failure modes, they increasing meaningful info and testability.
Our desire to log more (driven by paucity previously), had unexpected side effects:
Bloatiness – We got a bit bloated. We had added more bloat on top of existing bloat.
Fix the Spam First – Before adding more junk, remove current junk.
Include externals – Calls to external services are inherently risky, when done asynchronously. Log them as a priority, they expose risks that will slow you down, as in beyond your sphere of influence.
Log levels – only warn and error when you need to. If its normal service, don’t WARN. Have a policy. Have the ability to control verbosity.
For something to be testable, it needs to be understandable. We tried to fill in the bottom of the pyramid which gives you a lot of tests but alas not a lot of knowledge.
The gap between code creation (by someone else, who’s no longer around) and retrofitting tests was too wide. We check that it does what it does, it returns X. So what? You learn what it does, not why it does it, nor the original intent.
It also exposed masses and masses of other testability crimes with poor abstraction and inheritance. With an enormous library of common functions upon which everything depended.
It’s a big endeavour too, which stakeholders were at first happy to invest the time, the law of diminishing returns kicked in. Key concept here don’t spend it all at once/bet the farm, especially on something which might not yield at the same scale.
But seriously, technical awareness is massive for testers to enhance testability. Knowing the strengths and weaknesses or a given technology potentially gives you insight into where enhanced testability will make the biggest difference.
Not lists - again
So, as testers we can talk less about
Environments do not equal testability – if you can’t administer one, having more will probably break you. And by the way, banging on about environments alienates your stakeholders. They don’t care.
Things are the way they are because they got that way. You cannot build a time machine. Be informed by the past but not a slave to it. Testability for me looks to the future, its about opportunity.
Big testability scares those with cash and the team too, start small, ask small, but get a commitment to iterate. A question in your defintion of done for example.
So lets talk in these terms to enhance our testability outlook:
Flow of product, easier to observe and control means more predictable results, which can feed into better business outcomes. Testers having a local environment was a great example, shortened queueing time for shared environments, able to do different type of test earlier.
Feedback – if we talk about meaningful information that everyone can use, from development to operations to product, we enhance our chances of gaining transparency. Logging and monitoring are closely associated with testing, and provide very useful oracles.
Every product has its ups and downs, but enhanced testability gives you better information about your state, and with that, a better chance of recovery from those awkward moments.
Its part of the product. Everyone who is responsible for the product is responsible for it.
To finish lets talk about simple things you guys can do, like nowish. 3 quick ideas to leave you with!
Set the tone – build and maintain a mock of your API for example, give others adjacent testability
Light the Forest – illuminate parts of an hard to test system with targeted tests, then have them run often to expose information over time – add the time dimension to alleviate shallowness
Walk a Mile – spend some time with that cranky DBA – if you can nail their crankiness, you are winning.
Gwen (please wave) and I are distilling some of this knowledge and experience (fails) into the ‘team guide to software testability’
If anyone has a tale to tell, please give Gwen or myself a holler, we’d love to hear from you
