Welcome to the Tomato Quick Start guide! This streamlined manual focuses on the essentials to get you up and running in minutes. For a deep dive into advanced features and comprehensive configuration options, please refer to the:

Tomato is hosted on PyPI, making installation quick and easy:

pip install testomaton

Prefer a visual interface? Check out our:

Tomato GUI

Tomato is the core of the testomaton suite. Simplifying, it reads a model of a test function and generates rows of tests. The model is defined in a YAML format and provides the description of what values can be assigned to individual parameters of the function.

                                  ┌───────────────┐
                                  │ test function │
                                  └───────┬───────┘
              ┌───────────────────┬───────┴──────┬────────────────┐
              X1                  X2           [...]             Xn                      
      ┌────┬──┴─┬────┐    ┌────┬──┴─┬────┐                ┌────┬──┴─┬────┐  
     x11  x12 [...] x1m  x21  x22 [...] x2m              xn1  xn2 [...] xnm

Additionally, for the definition of a possible function’s input, the model describes the relationships between the parameters in the form of constraints that are logical expressions defining invariants that must be always fulfilled in the generated tests. For example, the expression:

IF 'X1' IS 'x11' THEN 'X2' NOT IN ['x21', 'x23']

indicates that in tests where the value of the parameter X1 is x11, the value of the parameter X2 mustn’t be x21 or x23. The language of the model and the constraints allows for easily defining more complex relationships than this.

The simplest form of a function model would be:

functions:
- function: duel
  parameters:
  - parameter: good guy
    choices: [Peter, Susan, Edmund, Lucy]
  - parameter: bad guy
    choices: [Jadis, Maugrim]
  - parameter: location
    choices: [White Castle, Cair Paravel]

It defines a function of three parameters (good guy, bad guy, location) that can take values from the sets defined as their choices ([Peter, Susan, Edmund, Lucy], [Jadis, Maugrim] and [White Castle, Cair Paravel] respectively).

The format of tomato’s input is a YAML file that consists of an arbitrary number of functions. Tomato processes only one function at a time, but it may be useful to group many functions in the same file for organizational reasons or if they reuse some of the same parameters.

The top elements of the YAML file may only be functions and global parameters, as described below:

global parameters:
# Using global parameters enhances maintenance and keeps the file size smaller. To learn more about them, see the full manual.

functions:
# The 'functions' element must contain a list of function elements that describe individual functions in the model. The name of the function is defined by the value of the 'function' tag.
- function: F1
# definition of F1
- function: F2
# [...]

Only the functions element is required. It must contain a list with at least one function element.

There are very few restrictions about naming of model elements:

• names cannot contain double colons (::), and they can’t start or end with a colon :,
• a name cannot be an empty string,
• a name cannot start or end with a whitespace character (e.g. name will not be allowed),
• a name must not contain line breaks,
• names must be unique on the same level of model hierarchy (for example all global parameters must have different names),
• names on different levels may have the same names (for example nested choices).

Other rules are:

• values of all elements in the model must be on one line. The only exception is a value of an expression of elements that define function or structure logic.

A function is the only allowed element of the functions list. A function definition contains two parts: parameters and logic. The parameters element enumerates the function’s parameters while logic describes relations between them. For example:

functions:
- function: character
  parameters:
  - parameter: name
    choices: [Peter, Susan, Edmund, Lucy]
  - parameter: gender
    choices: [M, F]
  - parameter: weapon
    choices: [sword, bow, dagger]
  logic:
  - alias: male
    expression: "'gender' IS 'M'"
  - constraint: naming males
    expression: "IF 'male' THEN 'name' IN ['Peter', 'Edmund']"
  - constraint: naming females
    expression: "IF NOT 'male' THEN 'name' IN ['Susan', 'Lucy']"

Here we have a definition of a function ‘character’ that has three parameters: name, gender and weapon, plus logic that defines dependencies between the parameters name and gender.

A parameter may contain a choices element that defines values taken by the parameter. There are two ways to define choices of a parameter. One way is using YAML’s flow notation, the other is with a list and an explicit choice definition.

functions:
- function: character
  parameters:
  - parameter: name
    choices: [Peter, Susan, Lucy, Edmund]
  - parameter: location
    choices: [Cair Paravel, Stone Table, Lantern Waste]
  - parameter: weapon
    choices: 
    - choice: sword
    - choice: bow
    - choice: dagger

Output parameters are parameters that do not take part in generating tests. Technically, they have only one choice that is always selected, so they do not have an impact on the size of the generated suite.

Their value, however, may be changed to an arbitrary value depending on the values of input parameters. This can be defined by assignments in the function logic.

Output parameters have one required element that is default value. This element defines the value that is assigned to the parameter in case that no assignment can be applied.

functions:
- function: duel
  parameters:
  - parameter: Good guy
    choices: [Peter, Susan, Edmund, Lucy]
  - parameter: Bad guy
    choices: [Jadis, Maugrim]
  - parameter: location
    choices: [White Castle, Cair Paravel]
  - output parameter: result
    default value: Good guy wins
  logic:
  - assignment: Jadis wins in her castle
    expression: "IF 'Bad guy' IS 'Jadis' AND 'location' IS 'White Castle' THEN 'result'= 'Jadis wins'"

Choices represent values that can be taken by parameters. Choices can be defined simply as a flow (choices: [sword, bow, dagger]) or as a list of choice elements.

Choices can be nested. Instead of providing a value of a choice, subchoices can be defined using the choices element. A choice that has a choices element is called an abstract choice, otherwise the choice is a leaf choice. Using abstract choices is a handy way to group choices together and simplify constraints. There is no limit for the nesting levels for choices.

functions:
- function: character
  parameters:
  - parameter: name
    choices:
    - choice: male
      choices: [Peter, Edmund]
    - choice: female
      choices: [Susan, Lucy]
  - parameter: weapon
    choices: [sword, bow, dagger]

The logic element of a function is a list of elements that can be alias, constraint, and assignment. However, this is out of scope of this short introduction. To read more about them and their syntax, please see the full manual.

The main function of tomato is generating tests. Tomato will read the model from a file or, if the file is not provided, from standard input. The input file is the only positional argument of tomato. The two following commands will have the same effect:

tomato model.yaml

and

cat model.yaml | tomato

Tomato reads the model and generates lines of test that are rows of a CSV file with individual tests. The output is sent to standard output. Any errors or other text that is not a test is sent to the error output. The Reading model from stdin text that is printed when tomato is started without defining the input file, is an example of this.

If more than one function is defined in the model, tomato will generate tests for the first of them. The function may be selected using the -f|--function argument. If your function has whitespace characters in the name, use quotes (e.g. tomato -f 'my function').

There are three main types of generation algorithms used by tomato: cartesian, random, and nwise that may be optionally customized with some additional options.

By default, the nwise algorithm is used with the parameter N set to 2, which means that the tool will generate tests with pairwise coverage.

The cartesian generator is the simplest generator that will output all possible combinations of parameter values that are valid according to defined constraints. This parameter does not take any additional options.

The random generator will generate rows with parameter values selected randomly. The --length switch will define the number of generated tests. The default value 0 is used for length by default and makes tomato generate tests until all valid combinations were generated.

Using the --duplicates switch will allow identical tests to be generated. Therefore, using --duplicates without limiting the length will make tomato generate tests forever (which may be useful in some scenarios).

The --adaptive switch will make tomato generate tests that are as different from tests already generated as possible. The metric of how different two tests are is the Hamming distance (number of elements that differ). For each step, tomato will look up to a maximum of 100 tests back and calculate a test that differs the most from all of them.

The N-Wise generator generates tests that cover all n-tuples of the space of all possible tests. For example, for the default value n=2 (pairwise coverage), it will cover all possible pairs of values of the input parameters.

Take this model as an example:

functions:
- function: duel
  parameters: 
  - parameter: good guy
    choices: [Peter, Susan, Edmund, Lucy]
  - parameter: weapon
    choices: [sword, bow, dagger]
  - parameter: bad guy
    choices: [Jadis, Maugrim]

Tomato will generate the following tests for it (using default arguments):

good guy,weapon,bad guy
Susan,bow,Maugrim
Edmund,sword,Jadis
Lucy,sword,Maugrim
Peter,dagger,Maugrim
Peter,bow,Jadis
Edmund,bow,Maugrim
Susan,sword,Jadis
Edmund,dagger,Jadis
Susan,dagger,Maugrim
Lucy,dagger,Jadis
Peter,sword,Jadis
Lucy,bow,Jadis

The nwise algorithm can be customized with the parameter -n that defines the size of tuples that must be covered (e.g., n=3 will cover all possible triplets, etc.).

The parameter --coverage defines the percentage of tuples that must be covered.