Tomato documentation

Tomato documentation

Tomato documentation

Welcome to the Tomato documentation!

Introduction

Testomaton is a suite of tools for combinatoric testing. It consists of tomato - a combinatoric test generator and two tools used for postprocessing tomato's output - beaver and jigsaw.

Installing testomaton

Testomaton is hosted in the pypi package index, so installing it is as simple as:

Terminal

$ pip3 install testomaton

Testomaton is hosted in the pypi package index, so installing it is as simple as:

Terminal

$ tomato --version
tomato 0.3.0

This document will explain main features and options of tools from testomaton suite. The full and up-to-date description is available in the tool's help text. Type

Terminal

$ tomato --help
$ beaver --help
$ jigsaw --help

to know more.

Workflow

Workflow

                ╔═════════╗         ╔══════════╗
┌───────┐       ║         ║  .csv   ║ beaver   ║        ┌───────┐
│ model │╶─────►║ tomato  ║╶───────►║   +      ║╶──────►│ tests │
| (yaml)|       ║         ║         ║ jigsaw   ║        | (.csv)|
└───────┘       ╚═════════╝         ╚════════╤═╝        └───────┘
                                      ▲      │
                                      │      │
                                      └──────┘
                                      intermediate
                                      .csv

The tools are organized so that they can be used in a pipeline. Tomato's input is a yaml file that describes a model of a test function. The function contains parameters with the values that they can take and constraints that describe dependencies between the parameters. Tomato parses the model and generates an .csv file that contains combinations of the values of the input parameters, so that they validate the constraints and provide requested coverage (for example pairwise). The output of tomato is in the .csv (comma separated values) format. The output of tomato can be provided to beaver and jigsaw. Both accept .csv format on their input and both produce .csv on the output, so they can be used (multiple times at once) in a pipeline. Beaver is a simple tool that replaces elements in a .csv line that have format of @python EXPR with the result of the expression evaluation. Jigsaw is a simple .csv manipulation util that can be used to add, remove, replace or swap columns.

Combinatoric test generation with tomato

Tomato is the core of the testomaton suite. Simplifying, it reads a model of a test function and generates rows of tests. The model is defined in a yaml format and provides the description of what values can be assigned to individual parameters of the function.

Workflow

                              ┌───────────────┐
                              │ test function │
                              └───────┬───────┘
          ┌───────────────────┬───────┴──────┬────────────────┐
          X1                  X2           [...]             Xn                     
  ┌────┬──┴─┬────┐    ┌────┬──┴─┬────┐                ┌────┬──┴─┬────┐  
 x11  x12 [...] x1m  x21  x22 [...] x2m              xn1  xn2 [...] xnm

Additionaly for the definition of possible function's input, the model describes the relationships between the parameters in the form of constraints that are logical expressions defining invariants that must be always fulfilled in the generated tests. For example the expression

Expresion

"IF 'X1' IS 'x11' THEN 'X2' NOT IN ['x21', 'x23']"

indicates, that in tests where the value of the parameter X1 is x11, the value of the parameter X2 mustn't be x21 or x23.
The language of the model and the constraints allows for easily defining more complex relationships than this.

The simplest form of a function model would be:

YAML

functions:
- function: duel
  parameters:
  - parameter: good guy
    choices: [Peter, Susan, Edmund, Lucy]
  - parameter: bad guy
    choices: [Jadis, Maugrim]
  - parameter: location
    choices: [White Castle, Cair Paravel]

It defines a function of three parameters (good guy, bad guy, location) that can take values from the sets defined as their choices ([Peter, Susan, Edmund, Lucy], [Jadis, Maugrim] and [White Castle, Cair Paravel] respectively).

Using tomato with examples in this document

If called without a filename, tomato will read the model from the standard input. Using tomato without any additonal arguments will make it generate pairwise combinations from the first function defined in the model. So, to test the examples from this document without having to write them to a file, start tomato:

Terminal

$ tomato
Reading model from stdin

And then copy&paste the model to stdin:

YAML

functions:
- function: duel
  parameters:
  - parameter: good guy
    choices: [Peter, Susan, Edmund, Lucy]
  - parameter: bad guy
    choices: [Jadis, Maugrim]
  - parameter: location
    choices: [White Castle, Cair Paravel]

After pasting the model an EOF character must be sent (Ctrl+D on Linux and Mac, Ctrl+Z on Windows). An additional newline may also be required in some cases (press Enter).

Output

good guy,bad guy,location
Peter,Maugrim,White Castle
Peter,Jadis,Cair Paravel
Lucy,Maugrim,Cair Paravel
Lucy,Jadis,White Castle
Susan,Jadis,White Castle
Susan,Maugrim,Cair Paravel
Edmund,Maugrim,White Castle
Edmund,Jadis,Cair Paravel

Since the pairwise generation algorithm has a partially random nature, the output you see may be slightly different from the one above.

Pairwise generation is possible for functions with at least 3 paranmeters, so all the examples below will have at least 3 parameters, even if that is not required to illustrate something.

Model syntax

The format of tomato's input is a yaml file that consists an arbitrary number of functions. Tomato processes only one function at a time, but it may be useful to group many functions in the same file for organizational reasons or if they reuse some of the same parameters. The top elements of the yaml file may only be functions and global parameters, like described below:

YAML

global parameters:
# List of parameters that can be referred to from the functions defined below. Using global parameters enhances maintenance and keeps the file size smaller.

functions:
# the 'functions' element must contain a list of function elements that describe individual functions in the model. The name of the function is the defined by the value of the 'function' tag.
- function: F1
# definition of F1
- function: F2
# [...]

Only the functions element is required. It must contain a list with at least one function element.

General rules

There are very few restrictions about naming of model elements:

  • names cannot contain double colons (::), and they can't start or end with a colon :
  • a name cannot be an empty string
  • a name cannot start or end with a whitespace character (e.g. name will not be allowed)
  • a name must not contain line breaks
  • names must be unique on the same level of model hierarchy (for example all global parameters must have different names)
  • names on different levels may have the same names (for example nested choices)
  • be careful with yaml constants that will be evaluate by python yaml package on the parsing level. For example in choice: no, the choice name will be evaluated to False. To ensure correct interpretation, it is recommended to surround names with quotes.

Other rules are:

  • values of all elements in the model must be one line. The only exception is a value of an expression of elements that define function or structure logic.

Labels

Every element of the model (ie. function, parameter, linked parameter, output parameter, choice, assignment or constraint) may optionally contain a labels element. Labels are defined by a flow of strings. General usage of labels is to filter elements of the model that are parsed.

Whenever allowed types of elements are mentioned in this document, they may always be extended by a labels tag, unless explicitly forbidden.

YAML

labels: [label, other label]

Function

A function is the only allowed element of functions list. A function definition contains two parts: parameters and logic. The parameters element enumerates the function's parameters while logic describes relations between them. For example:

YAML

functions:
- function: character
  parameters:
  - parameter: name
    choices: [Peter, Susan, Edmund, Lucy]
  - parameter: gender
    choices: [M, F]
  - parameter: weapon
    choices: [sword, bow, dagger]

  logic:
  - alias: male
    expression: "'gender' IS 'M'"
  - constraint: naming males
    expression: "IF 'male' THEN 'name' IN ['Peter', 'Edmund']"
  - constraint: naming females
    expression: "IF NOT 'male' THEN 'name' IN ['Susan', 'Lucy']"

Here we have a definition of a function 'character' that has three parameters: name, gender and weapon plus logic that define dependencies between the parameters name and gender.