Heuristics in testing

Understanding Heuristics

I have discussed about heuristics and how they help us in solving a problem in my previous post. You can go through it.

We understood that heuristics helps us in studying the problem well and aids us in figuring out a solution for it.

Why heuristics in testing?

Testing aims at figuring out many problems of the system. This includes a very exhaustive list of problems like - problems the system might face while functioning, while the user is using it, while competing with similar systems in the market etc.,

Tester while aiming to figure out these problems, has got many problems to solve for himself. This again is a very exhaustive list like - understanding the system, studying its behavior, planning the testing strategy, coming up with test ideas, finding the bugs, having a good test coverage etc.,

So here comes the role of heuristics to figure out solutions for these problems. I shall discuss here, how heuristics help tester to solve the following the following problems.

• Solving the problem of understanding the context of testing
• Solving the problem of building the test strategy
• Solving the problem of generating test ideas
• Solving the problem of buildling test coverage
• Solving the problem of confirming some behavior as a bug
• Solving the problem of finding bugs

Repeating again - Heuristic is an approach that could help us in solving a problem, which is fallible

Solving the problem of understanding the context of testing

Understanding the context of 'what to test?' is the first and foremost challenge a tester would face. The better understanding of the context will help the tester to test the application better and strengthen him to add the required value.

A tester could understand the context better when he is able to ask good number of questions to the stake holders. Now the problem here is 'What questions to ask?', 'How to cover all the possible questions?' etc., etc.,

Heuristics would solve the problem here. The tester could come up with various heuristics based on which he could list down the possible questions to ask. The better the coverage of heuristics, better the quality of questions, hence better is the understanding of context.

For example, 'Targeted audience', is an heuristic, around which a tester could frame quetsions. He could ask the stake holder about the type of targeted audience, their expectations, the previous feedback from them if any etc., to understand the user based context of the product.

Also, he can come up with more heuristics like 'Competitors of product', 'Business importance of the release' etc., to frame questions around them and understand the business context of the product. Similarly, he can apply this heuristic approach to get to know different other things about the product.

Solving the problem of building a test strategy

Test strategy contains various aspects of the testing activity which tester might want to consider while testing the system. It may include many things ranging from the mission of the project to the deliverables from the testing team.

The more relevant aspects you bring into your test strategy, the well equipped your testing activity would be. Here we can think of various heuristics to come up with different components of the strategy. Bringing in more and more heuristics will enrich the strategy with more possible things for consideration.

For example, we can think of heuristic 'What Quality criteria to focus on?'. This approach will help us to include all the possible quality criteria like functionality, security, performance, usability etc., that a tester might be interested to consider while testing.

Also we can think of other heuristic 'Project schedule', which would help the tester to consider things like - What is the duration of project? What are the scheduled release dates? When would the build be available for testing? What are the timelines of deliverables etc.,

You can also refer to the 'Heuristic test strategy model' designed by James Bach, for more insight on the heuristic approach in building a test strategy.

Solving the problem of generating test ideas

Test idea tells us how to perform a test on the application. Coming up with a test idea and testing the system is the major task of a tester. We try to have a good list of test ideas to test the system better. How we generate the test idea depends upon the approach or the way we are choosing to create these ideas. So, here comes again the role of heuristic methods or heuristics to help us in figuring this out.

Tester could think of heuristic ways to generate test ideas. Depending upon the context, we can come up with list of possible heuristics, considering which we could test the system. Each heuristic will help us build a bunch of test ideas around it and thus helping us to create more and more ideas from every new heuristic.

For example you can look refer this link Heuristics - User Experience Testing, which lists down few heuristics that I had come up with, to test for 'User Experience' of an application. Using these heuristics, you can generate test ideas and start testing the system.

You can also refer to the Test heuristics cheat sheet by Elisabeth Hendrickson for exhaustive list of heuristics.

Solving the problem of building test coverage

Test coverage is a track of various aspects against which we have tested our application. Having a good test coverage means that we have tested our application to a larger extent. So our next challenge as a tester lies in building a good test coverage.

The essence of test coverage lies in variety of test ideas we have. We can have varied test ideas by coming up with different approaches to build test ideas. So here again we see the need of heuristics. We discussed before that heuristics help us in building test ideas, so having diverse heuristics would help us in building varied test ideas, which in turn helps in building a good test coverage.

For example, there is a mnemonic of heuristics - 'SFDIPOT' quoted by James Bach in Rapid software testing slides, which lists down the heuristics like - Structure, Function, Data, Interface, Platform, Operations and Time to achieve the test coverage respectively.

Solving the problem of confirming some behavior as a bug

While we apply test ideas to test the application, we figure out some behavior of the system as a bug. We confirm something as a bug only when we acknowledge that the behavior is not as expected. Here the challenge is how do we confirm if some behavior is expected or not expected.

We have heuristic methods to decide the expected behavior and confirm something as a bug. These heuristics those help in confirming something as bug are called as Oracles.

For example, 'comparing the output with the requirement' is an Oracle. We can confirm something as a bug using this approach. Also our 'common sense' is an Oracle, we tend to declare some behavior as a bug using our common sense.

So better list of oracles you have got, there are more chances of solving your problem of confirming some behavior as a bug.

Of course, solving the problem of finding bugs

Keeping it short, if the heuristics you have applied so far has helped you to understand the context of testing better, build a efficient test strategy, generate good number of test ideas, have a wide test coverage, and have good set of oracles, then you are well armed and are sure to find good quality and good quantity of bugs too.

So finally solving the problem of 'solving the problem' while testing

So be it whatever problem we face while testing the system, we have got heuristic approach or heuristics to solve our problem of 'solving the encountered problem'.

More you think, more heuristics you have, and more are the chances to solve the problem.

Not to forget, heuristics are fallible

Yeah! You got to keep in mind, heuristics are fallible. Every heuristic you apply might not help you to solve the problem. Your heuristics may fail to figure out a solution.

So, every time you fail, try figuring out more relevant heuristics to solve the problem. At the end its context of problem you are going to solve matters. So, make sure your heuristics are always aligned with the context.

Thanks for reading, hope this helped you :)

Happy problem solving! Happy testing!

Understanding Heuristics

Hello!

I have been listening to the word 'heuristic' and the role of heuristics in solving the problems,  in many conversations with my testing friends, and in many blogs that I read. I have been giving lot of thought about it and exercising it in my work.

I have also noticed that many of my friends had difficulty in understanding what heuristics are exactly and how are they helpful in solving problems.

So here I come up with a small article that might help to understand heuristics and know if they help in solving problems.

Let us first look at the google meaning of this word Heuristic

From the above definition we get that, Heuristic process - enables us to discover or learn something - by ourselves. So, by making use of heuristics we will be able to discover a problem or learn about a problem ourselves.

Let us look at another definition and understand this better

So heuristic is a - rule of thumb - educated guess (a guess with some sense behind it) - that helps us in searching a solution to a problem - which is difficult or poorly understood

Now lets look at this wiki definition

Heuristic (/hjʉˈrɪstɨk/; Greek: "Εὑρίσκω", "find" or "discover") refers to experience-based techniques for problem solving, learning, and discovery that give a solution which is not guaranteed to be optimal

So heuristic is a - experience based technique - to solve a problem - to learn a problem - to discover a problem - to give a solution - which is not guaranteed to be optimal


Combining all these, let us say

Heuristic is a - technique, approach, guess - to solve, learn, discover - a difficult or poorly understood problem - which might not be guaranteed to be optimal

Putting it in the simple language, we can understand heuristic as - any thought, idea or approach that helps us to find out or figure out something.

Let us take some daily life examples and understand heuristics better.

'Trial and error' - Heuristic

We all must be familiar with the 'Trial and error' method.  While applying this method to solve a problem, we assume a possible solution and check if it is feasible. If it fails, we come up with another trial and check again. We keep making trials every time we fail, and we may or may not find a solution.

So here 'trial and error' is an heuristic method, though every attempt of it din't promise us a solution, it has helped us in figuring out some or other method to try for a solution.

Turn on the tap - exercise

Let us take an exercise where you are asked to turn on the tap to wash your hands. Also let us say this tap is of new design which you haven't seen so far. By looking at it you could analyze that it has a valve that is stopping the water flow and all you need to do is move the valve in some direction so that water flows out.

Now the possible actions you might do is - turn it to right (may be most common practice), turn it to left (immediate alternate guess), lift it up, push it down etc., While you are doing these, the tap might turn on at a particular action or it might not.

So you have been trying to open it with some possible guess you could make out of whatever logic or whatever knowledge you had on it. Every trial of yours was again a heuristic to open the tap, which failed sometimes and which might have passed another time.

So, heuristics help us to solve a problem. How?

Looking at these definitions and examples, we can say that heuristics help us to solve a problem. Let us analyze how are they helping us to solve a problem.

By looking at the behavior of the system each time a heuristic is applied - we get a chance to study the system, we get a chance to gather more information about the system and get to think about more possible ways of solving the problem. So every time a new heuristic is applied, we get to dive more deeper to solve the problem.

So as we progress, every heuristic might help us in building new heuristics, and might help us in solving or studying the problem much better.

So, heuristics are fallible. How?

After reading all this, we understand that heuristic is not a formula that would guarantee us a solution. Heuristic is an approach that we build to try solving a problem. Heuristic is an idea that we have, that we think, that we assume that, that could solve a problem.

As - assumptions, ideas, approaches - have a probability of failing, so are the heuristics. They might fail at times and they might pass at times.

So heuristics are fallible.

So, my article is an heuristic. How?

I have tried my approach to explain you about heuristics and make you understand. I am not sure if this has helped you or not, but surely it might have helped you to solve your problem of understanding heuristics.

I am sure if you can find out more heuristic ways to learn heuristics yourselves, you will surely understand what heuristics are.

I would be happy if my heuristic article passed to help you, else please come up with some heuristics to help me understand heuristics better.

Thanks for reading :)