|
 Today,
‘Test automation’ is a much talked about topic in the world of software
testing and quality. In this article Saket Godase discusses the basics
of test automation. What is test automation? He looks at the types of
test automation and also discusses the various myths about test
automation.
An Introduction to Software Test Automation
Abstract
This
article is an introduction to software test automation and will serve
as a tutorial to the testers who are new to Test Automation. It will be
a gentle reminder to experienced professionals. Topics covered include
an introduction to testing (manual and automation), different types of
automation testing and the myths and realities of Test Automation.
Software Testing
Though
there are many formal definitions of software testing it basically
boils down to “Software testing is a process used to identify the
correctness, completeness and quality of developed computer software.”
In reality, testing can never establish the correctness of the
software. It can only find defects, not prove that there are none. The
testing process is carried out either manually or using automation
tools or an effective combination of both.
Manual Testing –
is defined as developing and executing tests that rely primarily on
direct human interaction throughout the entire test case, especially in
terms of evaluating correctness and ascertaining test status.
Automation Testing – is
defined as developing and executing tests that can run unattended,
comparing the actual to expected results and logging status. The rest
of the paper will concentrate primarily on automation testing.
Simply put, "Automated Testing" is automating the manual testing
process currently in use. Minimal setup includes:
- Detailed Test Cases
- Predictable “Expected Results”
- Dedicated Test Environment
- And most importantly dedicated and skilled resources.
Automated
testing is expensive in terms of cost. It does not replace the need for
manual testing or enable you to "down-size" your testing department.
Automated testing is an addition to your testing process.
{mospagebreak title=Some time-tested techniques}
However, test automation can be cost-effective by following some of the
time-tested techniques given below.
- Choose
a test tool that best fits the testing requirements of your
organisation. A good place to start is "Automated Testing
Handbook" available from the Software Testing
Institute, which covers all of the major considerations involved in
choosing the right test tool for your purposes.
- One-time tests are not worth automating. Only automate those
tests that are going to be repeated.
- Before starting off with Test Automation, identify the test cases
that can be automated.
- Concentrate on automating the majority of your tests, which are
fairly straightforward.
- Leave the complex tests for manual testing.
- Never
make the mistake of using Record and Playback as a method of automation
testing. This method is fraught with problems, and such scripts are
very costly to maintain. At best it can be used to perform a Proof of
Concept. In the next section, I will elaborate on why I feel so
strongly against using this approach.
- As far as possible use a
data-driven automated testing methodology. This allows you to develop
automated test scripts that are more "generic", requiring only that the
input and expected results be updated. This methodology is ideal for
testing systems that are huge, complex and have varied test
environments. The different data-driven methodologies will be discussed
in detail later in this article.
- Most importantly, have a set
of resources (hardware, software and people) dedicated to test
automation. Resources that share their time across manual and
automation testing rarely achieve effective test automation.
Record and Playback (A Test Automation Nightmare)
This
methodology alone is probably most responsible for software companies
all over the world to give up on test automation out of sheer
frustration and return to manual testing. Ironically, this is the tag
line that most vendors use to sell their products.
Record/Playback is something that doesn’t work in a real-world
scenario and here are the reasons:
- Such
scripts contain hard coded values, which must change if anything at all
changes in the application. This creates an additional overhead, as the
testing team needs to update the scripts each time the data changes.
- These
scripts contain very little error handling and recovery scenarios.
Consequently, such scripts are not reliable (even in cases where the
application doesn’t change) and often fail during playback for a
variety of reasons such as pop-up windows, messages etc.
- The cost of maintaining these scripts is huge and quite
unacceptable.
- If the application under test changes, the tests must be
re-recorded.
- Record/Playback
scripts work well only if the application under test is already working
as per the expectations. In essence, this defeats the purpose of test
automation.
- Record/Playback might work in some cases; if the
tools used are intelligent enough to analyze the application under test
and auto generate meaningful test data.
{mospagebreak title=Test Automation Strategies}
Having
discarded Record/Playback as a feasible test automation strategy, we
will now move on to some of the more reasonable and widely used test
automation strategies.
Figure 1 will explain the different types of test
automation.
As Figure 1 indicates, Test Automation is broadly divided into
2 main methodologies.
- Functional Decomposition Method:
A system broken down into its components in such a manner that the
components correspond directly to system functions and sub functions.
- Key Word Driven Method:
A system broken down into certain predefined keywords in such a manner
that the keywords correspond directly to system functions and sub
functions.
Lets have a look at different approaches to test automation in detail.
Functional Decomposition Method
The
main and most important concept behind functional decomposition is to
separate the data from the functions. This is achieved by employing a
hierarchical architecture with a modular design.
The highest
script in the hierarchy is the controller / controlling script. This is
the engine of the test automation suite. Typically the Driver script
internally calls one or more scripts. These verification scripts are
the one’s, which perform the application testing. Apart from this,
usually there are a few utility scripts that are used by the driver and
verification scripts to perform a number of tasks.
- Controller
/ Driver Scripts: perform initialization (if required), and then call
the business logic verification scripts in the desired order.
- Verification
/ Test Case Scripts: perform the Business Functions. Usually, the
entire business logic is embedded within these scripts.
- Subroutine
/ Utility Scripts: perform application specific tasks required by two
or more Business scripts. The generic nature of these scripts can
contribute heavily to rapid script development.
- User-Defined Functions: general and/or application-specific
functions. Such scripts can be shared across test automation suites.
{mospagebreak title=Functional Decomposition Explained}
Listing 1 will explain the above steps in much better way.
Listing 1:"Login" Test Case for any web based email
application
1. Open a web browser.
2. Navigate to the appropriate URL.
3. Verify that the login page is displayed correctly.
4. Write any errors to an Error Log.
5. Retrieve the login details from a data file / database.
6. Enter the user name and password and click on the login button.
7. Verify that the login is correct.
8. Write any errors to an Error Log.
9. Verify the main menu displayed.
10. Write any errors to an Error Log.
11. Logout from the application and close the web browser.
The
above steps can be split up effectively into a verification script and
couple of subroutines, which can be placed in a library file. For
example, the step #5 (retrieves the login details from a data file /
database) can be written as a library subroutine, which will have the
file / database details as input parameter and return the retrieved
values, back to the calling script.
Such
a generic routine can be used for data retrieval in numerous other
verification scripts. Thus, if it were needed to retrieve test data for
10 different screens, we would need to call only a single subroutine
with different input parameters. Having the test data in text / flat
files gives an added advantage as updating these files does not require
knowledge of any tool, scripting or programming, meaning that the
non-technical testers can easily run the tests, whereas the technical
resources can create and maintain the test scripts. Hence, maintaining
such scripts is very cost effective.
Advantages of using the Functional Decomposition Method:
- A
modular design and using files or records to both input and verify
data, reduces redundancy and amount of effort in creating automated
test scripts.
- Scripts can be developed even when the
application development is in progress. If the functionality changes,
only the specific verification scripts containing the business logic
need to be updated.
- Script reusability is very high. As the
scripts are written to test various business functions independently,
they can easily be combined in a controlling script in order to
accommodate complex test scenarios.
- Maintaining the expected results for such scripts is very easy.
- Error handling is much more robust in these scripts. This allows
unattended execution of the test scripts.
- Since such scripts have very little interdependency they can be
used in a plug and play manner.
Disadvantages of using the Functional Decomposition Method:
- Technical and skilled personnel are required to create and
maintain the scripts.If the test automation suite is very large,
then maintaining the expected results can be quite time consuming.
- The testing team needs to maintain both the scripts as well as
the test data.
- Proper
care should be taken that the test data is maintained in appropriate
file formats. If the test data is updated using some non-standard tool,
it can cause problems during test execution.
{mospagebreak title=Key Word Driven Approach}
Lets now look at the other important type of test automation, the
key word driven method.
Key Word Driven Method:
Totally
based on keywords, this method is assists the non-technical resources
of the testing team in script development and maintenance. The method
uses test case document(s) developed by testing team using a
spreadsheet containing special key words. This method preserves most of
the advantages of the "Functional Decomposition" method, while
eliminating most of the disadvantages.
Listing 2
Consider the above test case of logging.
| Key_Word |
Field/Screen Name |
Input/Verification Data |
Comment |
Pass/Fail |
|
|
|
|
|
| Login_Page |
Login |
Login |
Verify the active screen |
|
| Input |
Username and
Password |
Test/Test |
Verify that the current
username and password
is authentic. |
|
| Action |
Press Login button |
Login successful |
The user is logged into
the application successfully. |
|
Each
of the "Key Words" in Column No. 1 corresponds to a utility script. The
utility script will in turn call functions that will take in the
remaining columns as input parameters in order to perform specific
functions. If the excel document is prepared correctly, it could also
double up as a manual test cases. To create additional tests the tester
would be required to modify the test data only.
Advantages of Key Word Driven Method:
- Easier to construct and maintain than the functional
decomposition scripts.
- Due to the diminished learning curve, even a non-technical
resource can be much more productive in a short period of time.
- The test automation team can reuse the existing (manual) test
cases.
- Maintaining the expected results for such scripts is very easy.
- Once the generic utility scripts are in place, building the
verification scripts take a much shorter period of time.
Disadvantages of Key Word Driven Method:
- Building the core utility scripts requires a very degree of
proficiency in the testing tool being used.
- Constructing scripts for complex scenarios could be time
consuming.
- Without a good architecture in place these scripts have a much
higher risk of failure.
A Gentle Reminder
Irrespective
of which methodology is being implemented, the essential thing is
preparation. Before plunging headlong into a test automation project
there are a few things that one has to keep in mind.
- A
good test environment should exist for the test automation team.
Preferably it should be a replica of the production environment.
- A
good hardware setup is equally important in any test automation
project. It is essential to have dedicated for script execution.
- The expected data against which the checks are to be performed
should be a known baseline.
- Skilled and experienced resources should be hired.
- Test cases to be automated should be comprehensive and up to date.
{mospagebreak title=Myths and Realities}
Let's now have a quick look at some of the popular myths and realities
in the test automation world. Myths and Realities
Myth: The tool is going to replace the testers.
Reality:
This is hardly the case. One has to remember that it is indeed very
difficult to eliminate manual testing. In reality the test tool helps
the testers to do their job in a much better manner. They relieve the
testers from performing mundane and boring tasks and free up their time
so that they can concentrate on the complex tasks.
Myth: The time span required to train all of the testers to use
the tool is too long.
Reality:
If the correct methodology is followed then the learning curve can be
reduced to a large extent and the testers can become productive within
a short span of time.
Myth: The tool will be too difficult for testers to use.
Reality: With adequate training the testers can be easily
migrated to any new testing tool.
Myth: A single tester can easily perform a dual role of
manual and automation.
Reality:
This rarely works and usually is one of the biggest blunders an
organization makes. Ultimately, the tester gets burned out and both,
the manual as well as automation testing suffers. For any test
automation project to succeed, it is imperative to have a set of
resources dedicated only to test automation.
Myth: Test Automation is very easy. Just record the scripts
at any given point of time and replay them whenever you want to.
Reality:
This kind of attitude is probably most responsible for companies
discarding test automation projects and switching back to manual
testing. Automation testing requires a lot of planning, preparation and
skilled resources.
Conclusion
In
this article we saw a brief introduction to test automation. In the
next few articles, we will explore some of the widely used Test
Automation Tools and have a look at their advantages, disadvantages and
usage. The bottom line is, with skilled resources, careful planning and
sufficient preparation there is a very high probability that a test
automation project can turn out to be a huge success. Compromising on
any of these can easily transform the Test Automation software into a
"shelf-ware".
Resources
Author
 Saket
has a master's degree in computer management. He is having extensive
experience in software testing, specifically in Test Automation. His
areas of interest include Astronomy, Windows Programming, Intelligent
Test Automation, Storage Management and Operating Systems. He can be
reached at (saket.godase AT gmail).
|
Home | Content | Resources | Contact | 
