The previous article in this series explained why it takes so many techniques to ensure software quality. This article explains a couple of these techniques.
Test methods in a nutshell
When I started investigating for this article, I stumbled upon a post resonating many of my thoughts on testing. In the meantime, the post has been deleted. It has been replaced by a Youtube video that’s slightly different (probably better). Be that as it may, the original post of Itamar Turner-Trauring inspired me to draw this picture, covering most of the popular test approaches and summarizing the post.
Let’s start with the quadrant you probably don’t associate with testing. The green quadrant focuses on the user and on our customer.
The user’s perspective (pass 1)
The earlier you begin quality assurance, the better. Traditionally, the developer writes a piece of software and passes it to the test department. That’s a lot better than letting the customer test the software. That, in turn, is a popular strategy, too. Just think of the countless software updates you have to install each month. But you need to work in a huge and powerful company to get away with what the Germans call “banana principle”: the software ripens at the customers’. Funny thing is that the English definition of the banana principle is exactly the other way round: testing endlessly without ever shipping to the customer.
Be that as it may, Kent Beck and his book on extreme programming told us there’s a better way. The programmers themselves can put their program under test. That’s unit testing. They also have other options. We’ll cover tests conducted by programmers in a minute. First, let’s think about the early stages of the project.
Defining acceptance criteria
Projects that are really successful think about testing when writing the specification of the program. They add acceptance criteria to the specification of the program. That may sound very bureaucratic, but it doesn’t have to be that way. If you’re using SCRUM, it suffices to define the acceptance criteria one week before the user story is implemented. The bottom line is that it’s defined before the customer (or tester) sees the final program. Otherwise, they are biased by what the development team has delivered.
Finding good acceptance criteria is an art of its own. A good hint is to define what you expect instead of describing how it’s done (see this blog). Equivalence class partitioning protects you from defining multiple similar test cases while missing other important cases.
Actually, we can test even earlier. Acceptance criteria are put under test after writing the software. But who says we’ve chosen the right acceptance criteria? At the end of the day, the users of the software accept or reject the software, often for reasons we couldn’t even imagine. So let’s ask them before writing the actual code.
The idea of creating an MVP does just that. An MVP is a minimum viable product. Instead of creating the final product offering every feature, we just manufacture a prototype. It should do something useful – that’s the “V” in MVP. But it doesn’t have to beautiful or perfect. It should be just good enough to show it to the user and watch their reaction. Maybe they need a completely different product. We want to know this as early as possible. That’s the “fail fast” philosophy. Of course, we don’t want to fail, but if we do, we want to fail as fast as possible so we can try something else.
Such an MVP doesn’t have to be a piece of software. It can also be a simple sheet of paper. In the early 90s, I had a hand in a paper-and-pen test conducted with hundreds, maybe even thousands of employees. They simulated the effect of a new software to be written. So we could find bugs, misconceptions, and weird ideas long before the first line of code was written. Most paper prototypes are a lot smaller. Draw a simple sketch of the application on a few sheets of paper and ask the users to pretend they’re working with a real application.
By the way, paper prototypes are also a good tool for UX design. Showing a user a sketch of the UI design and asking them to use this “software” is a surprisingly good tool to improve the user experience of the draft.
Sometimes you can run an even cheaper test. If you’ve got an idea for a great product, don’t develop it. Just create a landing page with a description and a couple of keywords and put it on the internet. If many people access it, you know it’s an idea worth pursuing. If not, you know it’s time to think over. The nice thing is that this approach only costs a few bucks.
Summarizing the user quadrant
So far, we’ve spent a lot of time learning the needs and requirements of the user. Contrary to public belief, that’s the most important part of testing. The other stages only assume we know what our customer needs. Most projects – including my projects – magically appear out of nothing, typically starting with a detailed three-hundred-page requirements document. So we developers don’t have any opportunity to put these assumptions to test. As a result, we may develop a software nobody needs. A long time ago, my father reported such a story. The software developers proudly presented their software. My father’s team wasn’t impressed. The software solved problems they didn’t have. “But we’ve developed the program specifically for you”, the developers said. Too bad they’d never talked to the real users.
More often than not, your customer is not the person who’s going to use your software. Getting in touch with the actual users isn’t easy. Plus, you have to learn to talk their language. They’re experts in their domains. As a software developer, it’s just a project to you, so you’re probably not an expert in this particular domain. However, if you really want to write good software, it pays to become a domain expert.
Cutting a long story short, the first pass of the user quadrant is about the users’ expectations. Every test below verifies that we write the program correctly. The tests above verify that we write the right program.
Until now we’ve challenged and tested the requirements document. At some point in time, actual software development starts. We assume the customer knows what they want, and we take the requirements document for granted. So the majority of tests during or after the development phase verifies whether the program fulfills the contract. In other words, these tests verify the correctness of the software.
Manual UI tests
If your software has got a UI, the simplest way to test it is to run a UI test. You use the software pretty much the way a user does. Only you concentrate much more on corner cases. What happens if I enter bad data? Does the program what we promised in the “good case”? This kind of tests may be assisted by check lists and equivalence class partitioning.
The nice thing about manual UI tests is that they can be run without a lot of preparation. Everyone can do it. You don’t need a programmer.
If you observe you’re running the same UI tests over and over again, think about test automation. We’ll cover this in a minute. But I don’t recommend to replace manual UI tests by unit tests or automated UI tests. Nothing beats human intuition. Plus, only humans may find weaknesses at the conceptual level.
Code reviews and me – that’s a love-hate-relationship. The idea of a code review is that a coworker spends an hour of their time to look at your code and to give you feedback. That’s generous, so there’s little to say against it. Of course, conducting such a code review requires some practice. If you’re used to pair programming, you already know the drill. Someone else reads your code and criticizes it. Developers who’ve never done pair programming need to learn not to be insulted. As to my experience, that’s not easy.
When I learned about code reviews many years ago, it was meant to be a formal process requiring a meeting room and one or two hours of time. Most teams feel this takes too long, so I’ve never seen this approach in the wild. We tried it a couple of months, but as far as I remember, we gave up before my second review.
Another option is two-phase committing. If you’re using Git, you might be interested in installing Gerrit. You can also achieve the same goal with pull requests, as most open-source projects on GitHub do. A contributor publishes their changes. However, they are only integrated into the main repository after someone reviewing it.
Code reviews sometimes result in an endless list of pedantic changes. They are useful if your team has a benevolent dictator knowing much more about the project than everybody else. For example, we appreciate the pull request system in our BootsFaces project because we want to make sure to get high-quality contributions. First-time committers are new to the project by definition, so they often miss one side effect or another.
However, in general, the quality of the pull requests is surprisingly good. That’s a side effect of code reviews: you know your code is going to be inspected closely, so you write it for a public audience. Sloppy shortcuts are likely to backfire, so you avoid them.
Code reviews are also useful if your software is mission-critical. Reviews are much more popular in avionics than in companies writing web shops.
Legal or business requirements
That’s probably a test you didn’t think of. However, it’s an important test. There are many legal pitfalls for software developers. If you’re doing in-house development, you’re well off. But if you’re selling a product or if you’re writing an open-source project, you have to care about things like software licenses. Some companies, such as Liferay, employ people specifically to verify the license compliance of your software. There are also tools to scan your code and check the licenses, such as Fossology.
Another important legal requirement is the protection of privacy. Where does the software store data? For instance, if you’re a European country, storing data in the USA is a usually no-go. That’s quite a challenge for cloud providers, and it’s the reason why this page doesn’t use Google Analytics or Akismet. Both tools are simply illegal in Germany. Granted, there are ways to use them legally. But you have to do something to legalize these tools.
As you can imagine, few people love reading legal textbooks. The average programmer tends to implement the requirements without asking twice. It’s a good idea to pay extra attention to the legal issues. I’ve picked only two legal requirements. In reality, there’s a lot more to consider. Just think of the current Diesel emissions scandal. Someone thought they’d found a clever loophole in the law, but on the long run, they didn’t get away with it. Asking a legal expert before implementing and shipping code may save you a lot of trouble.
Wrapping it up
It’s been a long article, but even so, we only managed to cover the left-hand side of the sketch of Itamar Turner-Trauring. In my eyes, that’s the most important part of the image. At the end of the day, humans pay your salary, so it feels natural to involve humans in all stages of software development. Nonetheless, a lot of software testing can be automated. That’s the topic of the next article.