This page can't be edited.

Types of automated tests

From $1

Table of contents
to the older version or return to version archive.

Combined revision comparison

Comparing version 23:14, 20 Jan 2016 by gfink with version 23:20, 20 Jan 2016 by gfink.

Summary**Summary** : Unit tests for logic, Integration tests to ensure units line up, Acceptance Tests for business requirements. **

Unit tests** are implemented by developers to check the behavior of the smallest level of the program.   Units generally are taken to mean objects in a language like Java.   There is a heretical minority that takes a unit to be a method.   Unit tests are entirely in the domain of the developer.   Many, if not most, units are implementation details below the concern of QA or the business.   An important aspect of a unit is that it is loosely coupled with the other code it interacts with and so that other code could be replaced with versions enabling more thorough testing. For example, if a unit is supposed to interact with a database, being able to swap out a real database for something that could throw an exception on demand makes it easier to test the
unit's error handling.   Unit tests are well suited to testing the logic internal to the unit.   By definition, unit tests are inappropriate for testing that two distinct units interact together
correctly.   Instead, the unit tests for unit A should test that unit B is correctly invoked or assuming that unit B returned a particular response; but unit A's tests should not be checking that unit B does the right thing.   Unit B should have its own tests for that. **

Integration tests** are implemented by developers to ensure components in the system work together.   Essentially integration tests are checking the flows and boundaries between components.   Integration tests should not be checking logic.   Integration tests are below the level of concern to QA and the business.   Integration tests are programmatic tests that generally run in the context of the real processes i.e., a real server running against a real database.

Unit and integration tests have distinct strengths.   Unit tests are fast and can explore a lot of little corner cases and exceptional situations which might be difficult or expensive to put the entire
system into.   For example, if a REST request comes into a server resource, which does argument checking, then invokes a command, which also does unit testing, it may be difficult to check the command's logic in an integration test because the REST layer will block the bad arguments.   The command needs its own tests because some future developer may reuse a command in a new context where it is no longer being shielded by the REST layer.   Unit tests cannot catch when some actor's contract was unilaterally changed.   Integration tests can.
Unit tests are considerably faster than integration tests.   Having the proper mix is necessary in order to have a complete, performant test suite. In a statically typed language like Java, the type system might check many of the things that would require an integration test in a dynamic language like Javascript.**

Acceptance tests** check the system against agreed upon specifications.
An "acceptance test" usually means a test that:

a) Can drive and control all the components of an application
b) Exists because of a user-visible feature requirement
c) Closely replicates a series of actions that a user or QA person would execute

Acceptance tests are of particular interest to the Continuous Delivery, QA and business teams, so they need to be easily written, reviewed, executed and analyzed by these teams. ** 

Development Acceptance Tests..** In applications with several dissimilar components, development teams might write acceptance tests for individual components of a release (e.g., server core, web application, data transport engine, native endpoints.) Developers can run these tests against code in development branches in addition to the unit and integration tests. The main challenge for such a development team is integrating components owned by other teams into a running test system.   In a Meteor environment, where component integration is handled by the framework, development acceptance tests are not especially important. **

Build Acceptance Tests** are run against code merged to the develop or release branches, prior to installation in a QA or Staging environment. Once the code is merged, the CD process builds a test system and executes the build acceptance test suites prior to advancing the build.   Our acceptance test suites should include basic edSpring operations, including punchouts to various assessment systems, and should use artificial but well-crafted data. **

System Acceptance Tests** are run against production builds installed on systems modeling customer usage, including production-level data and transaction volume.   System Acceptance Tests include most, if not all, of the tests in the Development and Build test runs, but also include punchouts to all of the supported assessment systems and ingestion of data from all supported sources. **

Here's an example.**. Suppose there's a small app exposing a REST API for voting on
different topics.   The app has the following files: \*

* domain/voter.js - a domain class representing a voter, correlates to a table in a database \*
* domain/election.js - a domain class representing an election.   Will have a topic, a collection of voters with their votes.   Correlates to a collection in the database.

The test suite might include the following kinds of tests:

Unit Tests: \*
* Check that voter objects are valid and invalid ones raise the appropriate exceptions \*
* Check that elections objects are valid and invalid ones raise the appropriate exceptions

Integration Tests: \*
* Check that the voter class is in sync with the db collection \*
* Check that the election class is in sync with the db collection \*
* Check that the contracts used between the election and voter class are in sync \*
* ("in sync" means the class object matches the database collection, or the method signatures match)

Acceptance Tests: \*
* Check that an administrator can set up a vote \*
* Check that a voter can register and vote \*
* Check that the results are correctly tallied and reported

In practice, unit tests should heavily outnumber integration tests and acceptance tests. In a statically typed language like Java, the type system might check many of the things that would require an integration test in a dynamic language like Javascript.

 

Version from 23:14, 20 Jan 2016

This revision modified by gfink (Ban)
**Summary** : Unit tests for logic, Integration tests to ensure units line up, Acceptance Tests for business requirements.

**Unit tests** are implemented by developers to check the behavior of the smallest level of the program.  Units generally are taken to mean objects in a language like Java.  There is a heretical minority that takes a unit to be a method.  Unit tests are entirely in the domain of the developer.  Many, if not most, units are implementation details below the concern of QA or the business.  An important aspect of a unit is that it is loosely coupled with the other code it interacts with and so that other code could be replaced with versions enabling more thorough testing. For example, if a unit is supposed to interact with a database, being able to swap out a real database for something that could throw an exception on demand makes it easier to test the

unit's error handling.  Unit tests are well suited to testing the logic internal to the unit.  By definition, unit tests are inappropriate for testing that two distinct units interact together

correctly.  Instead, the unit tests for unit A should test that unit B is correctly invoked or assuming that unit B returned a particular response; but unit A's tests should not be checking that unit B does the right thing.  Unit B should have its own tests for that.

**Integration tests** are implemented by developers to ensure components in the system work together.  Essentially integration tests are checking the flows and boundaries between components.  Integration tests should not be checking logic.  Integration tests are below the level of concern to QA and the business.  Integration tests are programmatic tests that generally run in the context of the real processes i.e., a real server running against a real database.

Unit and integration tests have distinct strengths.  Unit tests are fast and can explore a lot of little corner cases and exceptional situations which might be difficult or expensive to put the entire

system into.  For example, if a REST request comes into a server resource, which does argument checking, then invokes a command, which also does unit testing, it may be difficult to check the command's logic in an integration test because the REST layer will block the bad arguments.  The command needs its own tests because some future developer may reuse a command in a new context where it is no longer being shielded by the REST layer.  Unit tests cannot catch when some actor's contract was unilaterally changed.  Integration tests can.

Unit tests are considerably faster than integration tests.  Having the proper mix is necessary in order to have a complete, performant test suite.

**Acceptance tests** check the system against agreed upon specifications.

An "acceptance test" usually means a test that:

a) Can drive and control all the components of an application

b) Exists because of a user-visible feature requirement

c) Closely replicates a series of actions that a user or QA person would execute

Acceptance tests are of particular interest to the Continuous Delivery, QA and business teams, so they need to be easily written, reviewed, executed and analyzed by these teams.

**Development Acceptance Tests.** In applications with several dissimilar components, development teams might write acceptance tests for individual components of a release (e.g., server core, web application, data transport engine, native endpoints.) Developers can run these tests against code in development branches in addition to the unit and integration tests. The main challenge for such a development team is integrating components owned by other teams into a running test system.  In a Meteor environment, where component integration is handled by the framework, development acceptance tests are not especially important.

**Build Acceptance Tests** are run against code merged to the develop or release branches, prior to installation in a QA or Staging environment.Once the code is merged, the CD process builds a test system and executes the build acceptance test suites prior to advancing the build.  Our acceptance test suites should include basic edSpring operations, including punchouts to various assessment systems, and should use artificial but well-crafted data.

**System Acceptance Tests** are run against production builds installed on systems modeling customer usage, including production-level data and transaction volume.  System Acceptance Tests include most, if not all, of the tests in the Development and Build test runs, but also include punchouts to all of the supported assessment systems and ingestion of data from all supported sources.

**Here's an example**. Suppose there's a small app exposing a REST API for voting on

different topics.  The app has the following files:

\* domain/voter.js - a domain class representing a voter, correlates to a table in a database

\* domain/election.js - a domain class representing an election.  Will have a topic, a collection of voters with their votes.  Correlates to a collection in the database.

The test suite might include the following kinds of tests:

Unit Tests:

\* Check that voter objects are valid and invalid ones raise the appropriate exceptions

\* Check that elections objects are valid and invalid ones raise the appropriate exceptions

Integration Tests:

\* Check that the voter class is in sync with the db collection

\* Check that the election class is in sync with the db collection

\* Check that the contracts used between the election and voter class are in sync

\* ("in sync" means the class object matches the database collection, or the method signatures match)

Acceptance Tests:

\* Check that an administrator can set up a vote

\* Check that a voter can register and vote

\* Check that the results are correctly tallied and reported

In practice, unit tests should heavily outnumber integration tests. In a statically typed language like Java, the type system might check many of the things that would require an integration test in a dynamic language like Javascript.

Current version

This revision modified by gfink (Ban)

...

Unit tests are implemented by developers to check the behavior of the smallest level of the program.  Units generally are taken to mean objects in a language like Java.  There is a heretical minority that takes a unit to be a method.  Unit tests are entirely in the domain of the developer.  Many, if not most, units are implementation details below the concern of QA or the business.  An important aspect of a unit is that it is loosely coupled with the other code it interacts with and so that other code could be replaced with versions enabling more thorough testing. For example, if a unit is supposed to interact with a database, being able to swap out a real database for something that could throw an exception on demand makes it easier to test the
unit's error handling.  Unit tests are well suited to testing the logic internal to the unit.  By definition, unit tests are inappropriate for testing that two distinct units interact together
correctly.  Instead, the unit tests for unit A should test that unit B is correctly invoked or assuming that unit B returned a particular response; but unit A's tests should not be checking that unit B does the right thing.  Unit B should have its own tests for that.

Integration tests are implemented by developers to ensure components in the system work together.  Essentially integration tests are checking the flows and boundaries between components.  Integration tests should not be checking logic.  Integration tests are below the level of concern to QA and the business.  Integration tests are programmatic tests that generally run in the context of the real processes i.e., a real server running against a real database.

Unit and integration tests have distinct strengths.  Unit tests are fast and can explore a lot of little corner cases and exceptional situations which might be difficult or expensive to put the entire
system into.  For example, if a REST request comes into a server resource, which does argument checking, then invokes a command, which also does unit testing, it may be difficult to check the command's logic in an integration test because the REST layer will block the bad arguments.  The command needs its own tests because some future developer may reuse a command in a new context where it is no longer being shielded by the REST layer.  Unit tests cannot catch when some actor's contract was unilaterally changed.  Integration tests can.
Unit tests are considerably faster than integration tests.  Having the proper mix is necessary in order to have a complete, performant test suite.In a statically typed language like Java, the type system might check many of the things that would require an integration test in a dynamic language like Javascript.

Acceptance tests check the system against agreed upon specifications.
An "acceptance test" usually means a test that:

a) Can drive and control all the components of an application
b) Exists because of a user-visible feature requirement
c) Closely replicates a series of actions that a user or QA person would execute

Acceptance tests are of particular interest to the Continuous Delivery, QA and business teams, so they need to be easily written, reviewed, executed and analyzed by these teams. 

Development Acceptance Tests. In applications with several dissimilar components, development teams might write acceptance tests for individual components of a release (e.g., server core, web application, data transport engine, native endpoints.) Developers can run these tests against code in development branches in addition to the unit and integration tests. The main challenge for such a development team is integrating components owned by other teams into a running test system.  In a Meteor environment, where component integration is handled by the framework, development acceptance tests are not especially important.

Build Acceptance Tests are run against code merged to the develop or release branches, prior to installation in a QA or Staging environment. Once the code is merged, the CD process builds a test system and executes the build acceptance test suites prior to advancing the build.  Our acceptance test suites should include basic edSpring operations, including punchouts to various assessment systems, and should use artificial but well-crafted data.

System Acceptance Tests are run against production builds installed on systems modeling customer usage, including production-level data and transaction volume.  System Acceptance Tests include most, if not all, of the tests in the Development and Build test runs, but also include punchouts to all of the supported assessment systems and ingestion of data from all supported sources.

Here's an example. Suppose there's a small app exposing a REST API for voting on
different topics.  The app has the following files:

* domain/voter.js - a domain class representing a voter, correlates to a table in a database
* domain/election.js - a domain class representing an election.  Will have a topic, a collection of voters with their votes.  Correlates to a collection in the database.

The test suite might include the following kinds of tests:

Unit Tests:
* Check that voter objects are valid and invalid ones raise the appropriate exceptions
* Check that elections objects are valid and invalid ones raise the appropriate exceptions

Integration Tests:
* Check that the voter class is in sync with the db collection
* Check that the election class is in sync with the db collection
* Check that the contracts used between the election and voter class are in sync
* ("in sync" means the class object matches the database collection, or the method signatures match)

Acceptance Tests:
* Check that an administrator can set up a vote
* Check that a voter can register and vote
* Check that the results are correctly tallied and reported

In practice, unit tests should heavily outnumber integration tests and acceptance tests.