Heuristics in testing

Understanding Heuristics

I have discussed about heuristics and how they help us in solving a problem in my previous post. You can go through it.

We understood that heuristics helps us in studying the problem well and aids us in figuring out a solution for it.

Why heuristics in testing?

Testing aims at figuring out many problems of the system. This includes a very exhaustive list of problems like - problems the system might face while functioning, while the user is using it, while competing with similar systems in the market etc.,

Tester while aiming to figure out these problems, has got many problems to solve for himself. This again is a very exhaustive list like - understanding the system, studying its behavior, planning the testing strategy, coming up with test ideas, finding the bugs, having a good test coverage etc.,

So here comes the role of heuristics to figure out solutions for these problems. I shall discuss here, how heuristics help tester to solve the following the following problems.

• Solving the problem of understanding the context of testing
• Solving the problem of building the test strategy
• Solving the problem of generating test ideas
• Solving the problem of buildling test coverage
• Solving the problem of confirming some behavior as a bug
• Solving the problem of finding bugs

Repeating again - Heuristic is an approach that could help us in solving a problem, which is fallible

Solving the problem of understanding the context of testing

Understanding the context of 'what to test?' is the first and foremost challenge a tester would face. The better understanding of the context will help the tester to test the application better and strengthen him to add the required value.

A tester could understand the context better when he is able to ask good number of questions to the stake holders. Now the problem here is 'What questions to ask?', 'How to cover all the possible questions?' etc., etc.,

Heuristics would solve the problem here. The tester could come up with various heuristics based on which he could list down the possible questions to ask. The better the coverage of heuristics, better the quality of questions, hence better is the understanding of context.

For example, 'Targeted audience', is an heuristic, around which a tester could frame quetsions. He could ask the stake holder about the type of targeted audience, their expectations, the previous feedback from them if any etc., to understand the user based context of the product.

Also, he can come up with more heuristics like 'Competitors of product', 'Business importance of the release' etc., to frame questions around them and understand the business context of the product. Similarly, he can apply this heuristic approach to get to know different other things about the product.

Solving the problem of building a test strategy

Test strategy contains various aspects of the testing activity which tester might want to consider while testing the system. It may include many things ranging from the mission of the project to the deliverables from the testing team.

The more relevant aspects you bring into your test strategy, the well equipped your testing activity would be. Here we can think of various heuristics to come up with different components of the strategy. Bringing in more and more heuristics will enrich the strategy with more possible things for consideration.

For example, we can think of heuristic 'What Quality criteria to focus on?'. This approach will help us to include all the possible quality criteria like functionality, security, performance, usability etc., that a tester might be interested to consider while testing.

Also we can think of other heuristic 'Project schedule', which would help the tester to consider things like - What is the duration of project? What are the scheduled release dates? When would the build be available for testing? What are the timelines of deliverables etc.,

You can also refer to the 'Heuristic test strategy model' designed by James Bach, for more insight on the heuristic approach in building a test strategy.

Solving the problem of generating test ideas

Test idea tells us how to perform a test on the application. Coming up with a test idea and testing the system is the major task of a tester. We try to have a good list of test ideas to test the system better. How we generate the test idea depends upon the approach or the way we are choosing to create these ideas. So, here comes again the role of heuristic methods or heuristics to help us in figuring this out.

Tester could think of heuristic ways to generate test ideas. Depending upon the context, we can come up with list of possible heuristics, considering which we could test the system. Each heuristic will help us build a bunch of test ideas around it and thus helping us to create more and more ideas from every new heuristic.

For example you can look refer this link Heuristics - User Experience Testing, which lists down few heuristics that I had come up with, to test for 'User Experience' of an application. Using these heuristics, you can generate test ideas and start testing the system.

You can also refer to the Test heuristics cheat sheet by Elisabeth Hendrickson for exhaustive list of heuristics.

Solving the problem of building test coverage

Test coverage is a track of various aspects against which we have tested our application. Having a good test coverage means that we have tested our application to a larger extent. So our next challenge as a tester lies in building a good test coverage.

The essence of test coverage lies in variety of test ideas we have. We can have varied test ideas by coming up with different approaches to build test ideas. So here again we see the need of heuristics. We discussed before that heuristics help us in building test ideas, so having diverse heuristics would help us in building varied test ideas, which in turn helps in building a good test coverage.

For example, there is a mnemonic of heuristics - 'SFDIPOT' quoted by James Bach in Rapid software testing slides, which lists down the heuristics like - Structure, Function, Data, Interface, Platform, Operations and Time to achieve the test coverage respectively.

Solving the problem of confirming some behavior as a bug

While we apply test ideas to test the application, we figure out some behavior of the system as a bug. We confirm something as a bug only when we acknowledge that the behavior is not as expected. Here the challenge is how do we confirm if some behavior is expected or not expected.

We have heuristic methods to decide the expected behavior and confirm something as a bug. These heuristics those help in confirming something as bug are called as Oracles.

For example, 'comparing the output with the requirement' is an Oracle. We can confirm something as a bug using this approach. Also our 'common sense' is an Oracle, we tend to declare some behavior as a bug using our common sense.

So better list of oracles you have got, there are more chances of solving your problem of confirming some behavior as a bug.

Of course, solving the problem of finding bugs

Keeping it short, if the heuristics you have applied so far has helped you to understand the context of testing better, build a efficient test strategy, generate good number of test ideas, have a wide test coverage, and have good set of oracles, then you are well armed and are sure to find good quality and good quantity of bugs too.

So finally solving the problem of 'solving the problem' while testing

So be it whatever problem we face while testing the system, we have got heuristic approach or heuristics to solve our problem of 'solving the encountered problem'.

More you think, more heuristics you have, and more are the chances to solve the problem.

Not to forget, heuristics are fallible

Yeah! You got to keep in mind, heuristics are fallible. Every heuristic you apply might not help you to solve the problem. Your heuristics may fail to figure out a solution.

So, every time you fail, try figuring out more relevant heuristics to solve the problem. At the end its context of problem you are going to solve matters. So, make sure your heuristics are always aligned with the context.

Thanks for reading, hope this helped you :)

Happy problem solving! Happy testing!