Stages

Smoke Test

• Smoke tests are functional checks against the application UI or API to verify core functionality are working as expected.

• Smoke tests do not require a reset of the database.

• Smoke tests do not perform transactions that may affect the state of other smoke tests.

• Automated smoke tests will be ran automatically after every deployment to any environment.

• Manual smoke tests will be ran after every deployment to PR and production.

• A deployment will be certified only after all automated and manual smoke tests have passed.

Release Test

• Release tests are functional tests against the UI or API that confirm changes made as part of a release meet the agreed user acceptance tests.

• Agree on the scenarios that must pass to satisfy user acceptance test (UAT).

• Write UAT scenario specifications.

  • Tag the scenario with the release ID (e.g. the version @5.5.0).

  • Tag the scenario as manual if it will not be automated.

• Write release test specifications. Until we have adequate unit test coverage release testing must attempt, where possible, to exercise underlying logic. These tests are difficult to maintain and write. We must push for and confirm that proper unit test coverage is achieved before we attempt to cover hundreds of code paths from the UI or service layer.

  • Negative conditions.

  • Exceptional conditions.

  • Boundary conditions.

• Manual exploration of the update will be done to confirm that the change meets the definition of done.

• Manual exploration of additional functionality surrounding the application update will be done to detect if there are any functional regressions as a result of the update.

• Automated release tests will be ran N times per day.

• A release will be certified only after all automated and manual release tests have passed.

Regression Test

• Regression tests will be ran nightly.

• A release will be certified only after all automated and manual regression tests have passed.

Performance Test

• Performance test are to assess the speed at which operations are performed by the application. The tests results are collected and subjected to trend analysis to help identify issues in the performance of the application.

• Currently, we don’t do any performance testing.

  Scalability Test

• Scalability test are to assess the application’s ability to operate under increased load. Various operations are performed at increasing load to find the applications buckling and breaking point. Buckling point is the point where the application begins to show signs that the stress on the application is pushing towards the breaking point. The breaking point is the point when the load causes the application to stop operating. Finding stress points in the application provides information that can be configured in our application monitoring that will allow us to identify when we need to address load issues.

• Currently, we don’t do any stress testing.

  Security Test

• Security tests are to assess the application ability to prevent security breaches.

• We currently only do manual third party penetration testing.