QA Quarters

Ordered vs. Unordered Test Cases

Rob Baccus — Wed, 20 Jun 2007 18:02:23 +0000

There are two types of test cases that most testers like to write, ordered and unordered. Ordered test cases require that the test cases are executed in a specific order every single time and they build upon each other. Unordered test cases are executed as single entities and require that all setup and tear down functionality is contained in the single test case. Let us take a look at some pros and cons to Ordered and Unordered test cases.

Ordered Test Cases

Pros:

A single test case can focus on specific functionality and have short reproduction steps, which is easy to write and execute. This saves time and money when you don’t have a large team or a team that is over worked.
Test cases build on each other so they can be execute in less time than if each test case had to do a setup and tear down individually. Typically a tester that is manually testing will run through ordered test cases which saves them time because the setup will be a group of test cases, the functional testing will be a group of test cases, and the tear down will be a group of test cases.
They mimic more closely manual testing when automated. Automated ordered test cases mimic more closely what a tester or customer will do when using a system or product. This provides the basic level of test coverage on the system which is usually called Acceptance Testing based on customer usage.

Cons:

A failure in an early test case results in all following test cases to fail or be skipped. This can cause the testers a lot of wasted time when these failures are caused by simple problems or random system errors like network outages. This is especially the case when automating ordered test cases.
Building tools to support automated ordered test cases can be cumbersome and complicated. Tools that can support automated execution and results gathering become more complicated with ordered test cases because the tool requires a lot more logic to run the test cases in order and gather the results broken out per test case.
To find specific bugs caused by specific ordered operations the same set of test cases may have to be executed in many different orders. To represent this type of ordering test suites are most often used, which again adds a level of complexity to support tools for automated execution, and results gathering.

Unordered Test Cases

Pros:

Failures in a single test case may only effect that test case, all other test cases may pass. This is very useful when testing a system that may have seemly random failures when executing many similar test cases. Unordered test cases can help narrow down environmental, data or other causes of these types of errors.
When failures only happen in a single test case it saves the tester a lot of time by focusing only on that test case and rerunning just that one. They don’t have to rerun all following ordered test cases also.
Ordered test suites can be test cases that execute specific ordered steps in a single test case. This means a Test Suite in an Ordered approach now becomes a single test case in the Unordered approach. You still have the ordered test case steps wrapped in a single test case but when there are failures you can focus on the exact reproduction steps and they are not confused by other ordered steps. The trade off is you still have to write the ordered test suite in both approaches but with unordered the tester can focus on the exact steps the errors happened in and they may not effect other test cases.
Tools for unordered test cases abundantly available and are easier to develop, use, and maintain since they do not have the extra logic to execute and handle results for ordered tests.
Test suites can still be used by combining a group of unordered tests into a set that can be defined for features, components, system, or BVT’s.

Cons:

Each test case has to do a setup which loads data, creates connections, etc., or tear down which resets data, closes connections, etc. that has to be done for every test case.
Setup and tear down operations can takes a lot more time to execute per test case. Because of the setup and tear down operations unordered test cases are better suited for automated testing.
Test cases that are not ordered really don’t mimic true manual testing or customer usage. They focus specifically on a set of well defined steps, which can miss bugs that are found through testing using ordered, ad hoc or other testing approaches.

Conclusion

When trying to determine weather to use ordered or unordered test cases weigh the requirements of your organizational needs and resources. We can see that Ordered test cases are most beneficial when writing automated test cases because the tools that support this approach are abundant and easier to use and maintain, saving you time and money. When doing manual testing unordered test cases are easier to write, execute, and maintain for testers again saving time and money.

Don’t forget the third type of testing to provide even more test coverage, Ad Hoc. Ad hoc testing can find bugs that neither ordered nor unordered testing can find. When testers approach ad hoc testing like a user, they often find bugs that no one else has even thought about.

If you find that this was useful, have question, or ideas please add a comment. You support and feed back is much desired and appreciated.

Thank You.

QA Terms

Rob Baccus — Mon, 04 Jun 2007 20:40:17 +0000

Over the years we have worked at many different places that have many sublte differences in definitions of QA terms. A test case maybe basically the same from place to place but may have a subtle change in weather it is data depenedent or not, ordered or unordered, and differences in how they are categorized. One of the problems in the software QA industry are these subtle inconsistancies causing confusion for people on other teams. Program Managers and Developers have a hard time communicating with QA because there are no difinitive defintions and processes that are standard. They also are not taught at colleges or else where except most often on the job. When on the job new QA members get the flavor of the QA manager or team that may have some standards but contain the subtle differences that the team throws in from their own knowledge and experiences.

In order to combat this basic issue with communication we would like to start a central location for definitions to some common QA Terms. Please feel free to add comments and questions to what is here so that we can refine these definitions for everyone. Please add references if you find good definitions or suplemental information on else where.

Introduction

Rob Baccus — Thu, 24 May 2007 18:16:55 +0000

Let me introduce myself, my name is Rob Baccus. I have over 14 years experience working with computers, and over 10 years as a professional software developer and QA person. My work experience is varied and wide in the Seattle, WA area, which includes places like Microsoft, Attachmate, BEA Systems, T-Mobile, Real Networks, Amazon.com, and currently Revenue Science.

The goal of this site is to share knowledge about Software Quality practices and technology. To do this we would like to provide a place for the software community to come share what they have learned through their own experiences testing software on the Internet and desktop applications.

My personal focus will be to share knowledge about QA practices applied to Java, Linux, and Internet technologies. The challenges that I face on a daily basis, tools that help me over come those challenges, and the techniques that help me interface with testers, developers, and managers to get the products shipped in a reasonable time frame.

The first set of articles I will write will be to define some terminology that will allow new comers and experienced QA people to have a firm foundation in the topics to follow. These articles will hopefully spark great debates and help clarify areas of Software QA that currently vary from place to place.

If you would like to become a contributor please add a comment to this post letting me know what topics you would like to write about.

My ultimate goal for this venue is to make the Internet and software world better through Software QA.

Thank you!

Rob Baccus