How do you determine if a test case should be automated or only tested manually?

Nick:

What criteria would you use to determine if a test case should be automated or only tested manually?

Here I would think about - how critical the feature is, - how complex the feature is, - time taken to automate, - time spent manual testing and frequency, - how repetitive the test is


Alex:

Usually, test automation should be done only for functional tests.

Since test automation means creating code and since it takes time, you can only automate a small number of test cases.

Test automation should be the top of the pyramid where unit tests are at the bottom.

You should not automate negative tests as unit tests can do this better.

You should not automate GUI testing.

You should not automate low priority user scenarios.

So the answer is:

Automate test cases that are functional, have high priority (critical for the application), are on the user's happy path and corresponds to user scenarios that are very popular

or

Automate test cases that are used for a build's smoke test.



Nick:

You mention only creating automated tests for the happy path, I've seen this stated a lot but rarely with reasons why.  

Is yours simply that the negative tests are easier to do in unit testing?  

You also mention to restrict it to functional tests,  by this I'm assuming you mean its covering a large path, such an end-to-end test for a user placing an order, whereas something testing only a small piece of code/isolated function should be covered only by unit tests?


Alex:

The difference between positive and negative tests is that

- positive tests prove that the system works (if you use the happy path)

- the negative tests do not prove this

- there is 1 positive test for everything

- there are many negative tests

The difference between unit tests and the UI automated tests (Selenium, QTP, etc) is that

- unit tests are extremely fast; you can execute thousands in 1 hour

- UI automated tests are very slow as they go through the application (for a site, the browser needs to be opened, then the site, etc)

- the unit tests use the application framework components and are stable

- the UI automated tests use are brittle, break easily, use the application where things can change (change position of elements, ids of elements, etc)

So if the purpose of unit tests are to confirm that the components of the app are working correctly, then the unit test should cover all tests at the component level, including the negative ones.

The purpose of the test automation scripts is to confirm that the components fit well together and that the app works (for the happy path mostly).

Since these UI tests are slow and brittle and since the unit tests covered all component testing, you dont need to repeat yourself so just focus on functional, positive tests.

Share this post