Who’s That Guy?

Foreword

Testing is an incredibly complex subject.

You won’t become full-fledged testers by sitting here.

This is no tutorial for writing tests
for your individual WordPress plugin/theme.

Instead, you will be briefly introduced to
several relevant topics of software testing.

If you have more questions after this talk than you have now,
I reached my goal. Sorry. (ツ)

Why Test?

Reasons for Testing

• Software contains defects. They hide.
• Defects can cause software failures.
• Failures can cost money, and even be mortal!
• Testing assures the quality of the software.
• Testing accelerates software development.
• And much, much more.

“But we debug anyway!?”

Debugging is NOT testing!

Infamous
Software Failures

Do Software Testing. Right.

• Testing has to be planned.
• Testing costs easily up to 40% of a project’s budget.
• Testing has to be performed in a reasonable way.
• Testing shall be independent and objective.
• Testing has to be managed.

Software testing is a fundamental part
of professional software development!

What is Testing?

Testing

The process consisting of all life cycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects. ISTQB®

This is a Test

Different Perspectives,
Different Objectives.

Requirements?
Specification?
Requirements Specification?

Requirements and Specification

The requirements document is the input to a development phase.

The specification document is the ouput of a development phase.

The specification document resulting from development phase X
serves as requirements document for development phase X+1.

Requirements and specification are the same,
depending on the usage and point of view.

The product is 100% specification.

Validation and Verification

Validation

The process of evaluating a system or component during or at the end of the development process to determine whether it satisfies specified requirements. IEEE Std 610.12-1990

“Are we building the right product?”

Verification

The process of evaluating a system or component to determine whether the system of a given development phase satisfies the conditions imposed at the start of that phase. IEEE Std 610.12-1990

“Are we building the product right?”

Static Testing

Static Testing

• Verification only.
• No execution of code.
• Find defects.
• Static analysis (automated).
• Reviews (manual examination, tool support).

Static Analysis

• Analyze code (e.g., control flow, data flow), and generated output.
• Typical use cases for static analysis:
  • syntax checking;
  • code smell detection;
  • coding standards compliance.
• Typical defects found by static analysis:
  • syntax violations;
  • unreachable code;
  • overly complicated constructs.

Reviews

• Different types of reviews:
  • informal review (no formal process; inexpensive);
  • walkthrough (train colleagues and users; gain understanding);
  • technical review (documented, defined process; discuss);
  • inspection (formal process; gain metrics).
• Success factors:
  • clear predefined objectives;
  • defects found are welcomed and expressed objectively;
  • application of suitable review techniques;
  • management supports review process;
  • emphasis on learning and process improvement.

Dynamic Testing

Dynamic Testing

• Primarily verification.
• Execution of code.
• Find failures.
• Different testing methods:
  • black-box testing;
  • white-box testing.
• Different test levels:
  • unit (component, module) testing;
  • integration testing;
  • system testing;
  • acceptance testing.

Dynamic Testing Methods

Black-box Testing

• Specification-based testing.
• Test functionality by observing external behavior.
• No knowledge of internals (required).
• Different black-box testing techniques:
  • equivalence partitioning;
  • boundary value analysis;
  • decision table testing;
  • ...

White-box Testing

• Structure-based testing.
• Close examination of procedural level of detail.
• Knowledge of internals required.
• Different white-box testing techniques:
  • statement testing;
  • decision testing;
  • (multiple) condition testing;
  • ...

Test Levels

Unit Testing

• Test individual units in isolation.
• Find defects in software components (e.g., functions, classes).
• Done by developers.
• In general, white-box tests.

Integration Testing

• Test communication/interaction of units (i.e., their interfaces).
• Maybe separate unit integration and system integration tests.
• Done by developers.
• Primarily white-box tests, but also black-box tests.

System Testing

• Test complete, integrated system.
• Evaluate system compliance with specified requirements.
• Stress, performance, usability etc. testing.
• Done by (external) testers.
• In general, black-box tests. Additional white-box tests possible.

Acceptance Testing

• Test complete, integrated system.
• Evaluate system compliance with specified acceptance criteria.
• May be performed at various times during development.
• Done by customers/users.
• Only black-box tests.

The “Common” V-model

It’s just a model that provides an easy-to-grasp overview.

Development Phases

Requirements and Specification

Requirements and Specification

Requirements and Specification

Requirements and Specification

100% Specification

Static Testing

Dynamic Testing

Software Testing
in the Context of WordPress

Static Testing

• PHP_CodeSniffer:
  • detects violations of coding standards in PHP, JavaScript and CSS files.
• PHP Mess Detector:
  • detects potential problems (e.g., possible bugs, unused stuff) in PHP code.
• ESLint:
  • finds problematic patterns or style guide violations in JavaScript code.
• CSSLint:
  • detects syntax violations and problematic constructs in CSS files.

Pretty much all that comes with a decent IDE (e.g., PhpStorm).

Dynamic Testing

• PHPUnit:
  • unit testing framework for PHP.
• phpspec:
  • spec-level BDD framework for PHP.
• Behat:
  • story-level BDD framework for PHP.
• QUnit:
  • unit testing framework for JavaScript.
• tape:
  • JavaScript test harness for node and browsers.
• Jasmine:
  • BDD testing framework for JavaScript.

Dynamic Testing

• Mockery:
  • PHP mock object framework for use in unit testing.
• Prophecy:
  • object mocking framework for PHP.
• Sinon.JS:
  • standalone test spies, stubs and mocks for JavaScript.
• Brain Monkey:
  • mocking utility for PHP functions and WordPress plugin API.
• WP_Mock:
  • API mocking framework for unit testing within WordPress.

Continuous Integration

Continuous Integration

• Automated activities:
  • execute static code analysis;
  • execute unit tests, and check code coverage;
  • deploy to test environment;
  • execute integration tests;
  • ...
• Benefits:
  • earlier detection and analysis of conflicting changes;
  • regular feedback on whether the code is working;
  • no big-bang integration;
  • reduces repetitive manual testing activities;
  • ...

Travis CI

• Hosted continuous integration server.
• Single requirement: GitHub account.
• Getting started:
  • sign in with your GitHub account;
  • accept GitHub access permissions confirmation;
  • set up .travis.yml file for your repositories;
  • enable Travis CI builds for individual repositories;
  • develop continuously integrated software...