While I was interviewed for a new position I was usually asked to elaborate and write some test cases referred to a certain given situation.
More or less the same happened when I had to interview candidates for a new QA job position on my team.
As a QA Lead I had a set of technical QA questions and cases to be answered, developed and explained by the candidates during the interview.
As time passed by I felt that the candidate selection evaluation system that we were using did not reflect the way of thinking of the candidate, or at least it did not give us a sparkle of the needed ways of critical thinking to use as a guide to decide which candidate would best fit the position.
So I gathered my team in a brainstorming session in order to get some new technical questions or study cases set that could reveal the candidate’s ability to approach a desired performance.
We were not trying to find a very complex case nor so close to the theoretical one.
We started thinking about the usual interview cases like logins testing, payments options, multiple selection pages etc. but nothing seemed to fit with the idea.
Then a brand new idea appeared. It was a different one and gave us that piece of “out of the box and critical thought” clue needed to elaborate effective test cases.
As we were drinking some argentinian mate, we got the thermos at hand so we figured out the technical interview case study as follows:
“In the next exercise you are asked to elaborate some test cases for a test suite development of a thermos. Describe the cases in detail, all your considerations and needs.”
This exercise requires the candidate to think about testing but in a not computer based system
So as part of the game, I asked my team to solve the exercise, and here is a summary of the answers I received:
The previous table shows some of the test cases that were elaborated during the brainstorming session.
The case study is really interesting as we said before, it is about a physical element and not about a software piece. But there is still an error.
An error that is common to all the table’s test cases shown here. This error appears more often than desired on the candidate’s technical evaluations.
Are you able to find it?
Take your time. Take a look again at the test cases table, and try to find the error and later you can continue reading this article.
Let’s try to get some thoughts about the “hidden” error.
Regarding the table’s test cases 1 and 2, What if the thermos can only preserve the water temperature for 15 minutes? By doing those tests we can assure that the thermos will not pass those test cases.
Consider now the 3rd test case, What if we use boiling water? Boiling water is a particular type of hot water so: Is the thermos cap or seal able to resist the pressure generated by the boiling water steam?
On the 4th test case we can see that the external thermos temperature is checked for a variation. That is ok, but which criteria is “acceptable” for this case? What happens if the external thermos temperature is raised to a very hot stage when the thermos is filled with hot water?
And there are more questions to come with the next cases. Although the 5th and 6th are correctly written using a very accurate and analytic perspective, they share the same error with the previous cases.
If we give a job candidate this exercise there is a high chance that he or she can fall in the same error. This could be due to the nervousness of the technical interview added to a short time to solve it.
But What all these test cases have in common? Which is the error? Were you able to find it?
Well, as you can see, all the tests are suitable for a test suite as requested. All of them were done using the same thing: common sense and experience, and fortunately that is a really great thing but there is still something that does not fulfil our expectations.
In none of the test cases the thermos technical specifications were considered!
So what if the thermos was originally designed and built just to conserve only hot water and for just 15 minutes? Yes I know, this sounds ridiculous but let’s continue to complete the idea.
Some of these tests would fail when they should be successful.
The same principle applies to the rest of the cases; for example, What happens if the thermos wasn’t designed and built to have an external surface that is temperature isolated?
And in the last cases (5th and 6th) what does “an appreciable variation” mean?
Consider now the mental process needed to elaborate these test cases and how the results can change if the same cases are elaborated with the technical documentation at sight.
This experience showed us that several times, in fact more often than desired, we have no documentation available for what we have to test, so in those cases we use our experience and common sense.
Conclusion
As I said before, using common sense and experience is “not so wrong” but reflects a serious matter that is, again, more often than desired, that the QA division does not always have access to the technical documentation and specifications. This is clearly an organizational serious communication mistake
So do not rely just on common sense and experience, a successful QA division must have (and must claim for, if not) access to the technical documentation.
Otherwise the QA division could elaborate really great test suites but at the end those would be untrustable or even useless, wasting its effort and the most valuable resource: time.
As it is shown, it is not a bad idea to do from time to time, “QA over QA”. That means to review if the proper documentation is accessible, clear and updated.
Once this can be assured, the product quality also can be assured and improved via good testing practices. Otherwise the company can fall in a false sense of security about the product quality assurance.