An Introduction to
Software Testing

A talk by @thorstenfrommen


Testing is an incredibly complex subject.

You won't become full-fledged testers by sitting here.

This is no tutorial for writing tests
for your individual WordPress plugin/theme.

Instead, you will be briefly introduced to
several relevant topics of software testing.

If you have more questions after this talk than you have now,
I reached my goal. Sorry. (ツ)




Why Test?

Reasons for Testing

  • Software contains defects. They hide.
  • Defects can cause software failures.
  • Failures can cost money, and even be mortal!
  • Testing assures the quality of the software.
  • Testing accelerates software development.
  • And much, much more.

“But we debug anyway!?„

Debugging is NOT testing!

Software Failures


  • Inertial navigation software taken from Ariane-4. Untested.
  • All other systems thoroughly tested component-by-component.
  • Ariane-5 had a different trajectory than Ariane 4.
  • Converting 64-bit floating-point data into 16-bit unsigned integer values.
    → Arithmetic overflow.
  • There was an exception handler for that. It had been disabled.
  • Not even 40 seconds after launch, Ariane-5 literally self-destructed. Successfully.

Inverted Flight

  • A developer of the US Air Force improved a program for an unmanned rocket:
    • when crossing the equator, flip the coordinates's leading sign.
  • The program therefore needed less memory.
  • The rocket also made a 180-degrees roll. So what?
  • The program later was used for the autopilot of an F-18 fighter jet.
  • When crossing the equator, the pilot certainly was somewhat suprised.


  • Heartbeat Extension of TLS is used to keep Datagram TLS sessions open.
  • Simple request and response scheme:
    • “Send me back the following (padded) string which is n bytes long.„
  • An attacker just had to request a long string, while telling it is short.
  • Other party responded with short string, then leaked potentially confidential data.

Do Software Testing. Right.

  • Testing has to be planned.
  • Testing costs easily up to 40 % of a project's budget.
  • Testing has to be performed in a reasonable way.
  • Testing shall be independent and objective.
  • Testing has to be managed.

Software testing is a fundamental part
of professional software development!

What is Testing?


The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. ISTQB®

This is a Test

Different Perspectives,
Different Objectives.

Perspective Objective
Developer Find defects.
QA Evaluate quality.
Tester Check functionality against requirements.
Customer Verify usability and applicability.

Requirements Specification?

Requirements and Specification

The requirements document is the input to a development phase.

The specification document is the ouput of a development phase.

The specification document resulting from development phase X
serves as requirements document for development phase X+1.

Requirements and specification are the same,
depending on the usage and point of view.

The product is 100 % specification.

Validation and Verification


The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements. IEEE Std 610.12-1990

“Are we building the right product?„


The process of evaluating a system or component to determine whether the system of a given development phase satisfies the conditions imposed at the start of that phase. IEEE Std 610.12-1990

“Are we building the product right?„

Static Testing

Static Testing

  • Verification only.
  • No execution of code.
  • Find defects.
  • Static analysis (automated).
  • Reviews (manual examination, tool support).

Static Analysis

  • Analyze code (e.g., control flow, data flow), and generated output.
  • Typical use cases for static analysis:
    • syntax checking;
    • code smell detection;
    • coding standards compliance.
  • Typical defects found by static analysis:
    • syntax violations;
    • unreachable code;
    • overly complicated constructs.


  • Different types of reviews:
    • informal review (no formal process; inexpensive);
    • walkthrough (train colleagues and users; gain understanding);
    • technical review (documented, defined process; discuss);
    • inspection (formal process; gain metrics).
  • Success factors:
    • clear predefined objectives;
    • defects found are welcomed and expressed objectively;
    • application of suitable review techniques;
    • management supports review process;
    • emphasis on learning and process improvement.

Dynamic Testing

Dynamic Testing

  • Primarily verification.
  • Execution of code.
  • Find failures.
  • Different testing methods:
    • black-box testing;
    • white-box testing.
  • Different test levels:
    • unit (component, module) testing;
    • integration testing;
    • system testing;
    • acceptance testing.

Dynamic Testing Methods

Black-box Testing

  • Specification-based testing.
  • Test functionality by observing external behavior.
  • No knowledge of internals (required).
  • Different black-box testing techniques:
    • equivalence partitioning;
    • boundary value analysis;
    • decision table testing;
    • ...

White-box Testing

  • Structure-based testing.
  • Close examination of procedural level of detail.
  • Knowledge of internals required.
  • Different white-box testing techniques:
    • statement testing;
    • decision testing;
    • (multiple) condition testing;
    • ...

Test Levels

Unit Testing

  • Test individual units in isolation.
  • Find defects in software components (e.g., functions, classes).
  • Done by developers.
  • In general, white-box tests.

Integration Testing

  • Test communication/interaction of units (i.e., their interfaces).
  • Maybe separate unit integration and system integration tests.
  • Done by developers.
  • Primarily white-box tests, but also black-box tests.

System Testing

  • Test complete, integrated system.
  • Evaluate system compliance with specified requirements.
  • Stress, performance, usability etc. testing.
  • Done by (external) testers.
  • In general, black-box tests. Additional white-box tests possible.

Acceptance Testing

  • Test complete, integrated system.
  • Evaluate system compliance with specified acceptance criteria.
  • May be performed at various times during development.
  • Done by customers/users.
  • Only black-box tests.

The Common V-model

It's just a model that provides an easy-to-grasp overview.

Development Phases

Requirements and Specification

Requirements and Specification

Requirements and Specification

Requirements and Specification

100 % Specification

Static Testing

Dynamic Testing

Software Testing
in the Context of

Static Testing

  • PHP_CodeSniffer:
    • detects violations of coding standards in PHP, JavaScript and CSS files.
  • PHP Mess Detector:
    • detects potential problems (e.g., possible bugs, unused stuff) in PHP code.
  • JSHint:
    • detects errors and potential problems in JavaScript code.
  • JSCS:
    • code style linter for programmatically enforcing your JavaScript style guide.
  • CSSLint:
    • detects syntax violations and problematic constructs in CSS files.

Pretty much all that comes with a decent IDE (e.g., PhpStorm).

Dynamic Testing

  • PHPUnit:
    • unit testing framework for PHP.
  • phpspec:
    • spec-level BDD framework for PHP.
  • Behat:
    • story-level BDD framework for PHP.
  • QUnit:
    • unit testing framework for JavaScript.
  • Jasmine:
    • BDD testing framework for JavaScript.

Dynamic Testing

  • Mockery:
    • PHP mock object framework for use in unit testing.
  • Prophecy:
    • object mocking framework for PHP.
  • Phake:
    • PHP mocking framework.
  • WP_Mock:
    • API mocking framework for unit testing within WordPress.
  • Brain Monkey:
    • mocking utility for PHP functions and WordPress plugin API.

Continuous Integration

Continuous Integration

  • Automated activities:
    • execute static code analysis;
    • execute unit tests, and check code coverage;
    • deploy to test environment;
    • execute integration tests;
    • ...
  • Benefits:
    • earlier detection and analysis of conflicting changes;
    • regular feedback on whether the code is working;
    • no big-bang integration;
    • reduces repetitive manual testing activities;
    • ...

Travis CI

  • Hosted continuous integration server.
  • Single requirement: GitHub account.
  • Getting started:
    • sign in with your GitHub account;
    • accept GitHub access permissions confirmation;
    • set up .travis.yml file for your repositories;
    • enable Travis CI builds for individual repositories;
    • develop continuously integrated software...

ISTQB® Certified Tester

Most widespread qualification scheme in the world. Syllabi contents and glossary de facto industry reference.

Lessons Learned

  • Software testing is important! So make it a topic.
  • Use (the right) tools, such as a decent IDE.
  • Do code reviews.
  • Write unit tests. Continue with integration tests.
  • Automate!