============================
fMBT test configuration file
============================

Configuration file tells fMBT

- how to load model and adapter,

- which test generation heuristics will be used,

- how to measure coverage,

- when to end test generation and execution, and

- what to do on different verdicts.


Configuration syntax
--------------------

All configuration options are of the form

key = single-line-definition

or

key = "multi-
       line-
       defintion"

Lines starting with "#" are comments.

For example, filesystemtest.conf defines:

# Regression test for filesystems.
model     = aal_remote(remote_pyaal -l aal.log filesystemtest.aal)
adapter   = aal
heuristic = lookahead(6)
coverage  = perm(2)
pass      = no_progress(6)
on_pass   = exit(0)
on_fail   = exit(1)

That is, fMBT will use filesystemtest.aal, an AAL/Python file, as both
model and adapter. 6-steps-long "lookahead" will be used as test
generation heuristic. It will try to cover permutations of any two
test steps (perm(2)), that is the measured coverage. Test ends with
"pass" verdict when six test steps have been executed without making
progress in the coverage. If the verdict is "pass", fMBT will exit
with exit status 0. On "fail" exit status will be 1.


Model
-----

Defines test model type or format and how to load the model.

Model must be defined in the configuration.

Syntax:

        model = type(params)

Available types & parameters:

        aal_remote(<command>)

                uses a child process that executes <command> as a
                model. Communicates with the child using fMBT AAL
                protocol.

        lib(<modelname> [, filename])

                uses model <modelname>. If the model is not already
                loaded (statically linked or LD_PRELOADed, for
                instance), shared library filename is loaded. If
                filename is not given, fMBT tries to load
                "modelname.so" and "libmodelname.so".

        lsts(<filename>)

                loads state machine in lsts from the given file.

        lsts_remote(<command>)

                loads state machine in lsts format from the standard
                output of <command>.

Examples:

        model = aal_remote(remote_pyaal -l aal.log mytest.aal)

        model = lib(myaaltest1(foo),myaaltests.so)

        model = lsts(models/mymodel.lsts)

        model = lsts_remote(fmbt-gt -f mymodel.gt)

        model = lsts_remote(wget -O- http://modelhost/mymodel.lsts)

        model = lsts_remote(./create_mymodel.sh --exclude negativecases)


Adapter
-------

Defines the top level adapter.

If not given, adapter defaults to "dummy".

Syntax:

        adapter = type(params)

Available types & parameters:

        aal

                the adapter is implemented by the same AAL instance
                that is configured as a model.

        aal_remote(<command>)

                executes <command> in a child process to invoke an AAL
                adapter. If exactly the same command is used for
                invoking an aal_remote model, too, the same child
                process will implement both the model and the adapter.

        dummy

                only logs actions that it should have executed and
                reports successful executions. Can be used for offline
                test generation.

        mapper(<filename>)

                redirects and possibly renames actions in test model
                to subadapters according to rules in mapper adapter
                configuration file <filename>.

        lib(<adaptername> [, filename])

                Uses adapter <adaptername>. If the adapter is not
                already loaded (statically linked or LD_PRELOADed, for
                instance), shared library filename is loaded. If
                filename is not given, fMBT tries to load
                "adaptername.so" and "libadaptername.so".

        remote(<command>)

                executes <command> in a child process and forwards
                actions to it for execution. <command> is executed
                only once, the same child process will execute
                multiple actions. fMBT comes with remote adapters
                remote_python and remote_exec.sh (see adapters.txt).

Examples:

        adapter = aal

        adapter = aal_remote(remote_pyaal -l aal.log mytest.aal)

        adapter = remote(remote_exec.sh)

        adapter = remote(ssh user@host ./remote_python -l adapter.log)

        adapter = mapper(adapters.mrules)

        adapter = lib(myaaltest1,myaaltests.so)


Heuristic
---------

Defines test generation heuristic that is used for choosing "the next
action" to be tested.

If not given, defaults to "random".

Syntax:

        heuristic = type(params)

Available types & parameters:

        exclude(action-regexp [, action-regexp...], heuristic-spec)

                limits test generation by excluding given actions and
                actions matching action-regexps. Uses heuristic-spec
                to choose the next action but hides it the possibility
                to execute excluded actions. Excluded actions are
                suggested by exclude(...) heuristics only if no other
                input actions are available. If action-regexp is not a
                name of action, it is handled as a regular expression.

        include(action-regexp [, action-regexp...], heuristic-spec)

                limits test generation to given actions or actions
                matching action-regexps. Uses heuristic-spec to choose
                the next action but hides it the possibility to
                execute other than included actions. Excluded actions
                are suggested by include(...) heuristics only if no
                other input actions are available. If action-regexp is
                not a name of action, it is handled as a regular
                expression.

        lookahead[(depth[, offset])]

                If depth is not given, lookahead will make the
                decision based on names of enabled actions in the
                current state without simulation.

                If depth is a positive number, lookahead will simulate
                execution of all possible paths of length <depth>. It
                will choose the first action of a path that results in
                the best coverage. Suffix "b" enables "burst mode" in
                which the found path will be tested before searching a
                new path. Without the burst mode new path is
                calculated on every step (slower to generate but
                resulting test has less steps).

                If depth is of the form MIN:MAX, lookahead will first
                search from all paths of MIN steps, and if that is not
                enough to increase coverage, then search from paths of
                MIN+1 up to MAX steps.

                If offset is not given, lookahead will choose the same
                action on every test run. It is likely to choose
                another, equally good action every time if the offset
                is a natural number. With "random" offset the choice
                is likely to be different on every test run.

        mrandom(weight, heuristic[, weight, heuristic...])

                eg. mrandom(20,random(1343649162),80,lookahead(5)) will
                use random heuristic with 20% propability and lookahead with
                80% propability.

        random[(seed)]

                evenly distributed random choice among all enabled
                input actions. If seed is given, that is used for
                initialising the random number generator. Otherwise a
                new seed is generated for each run (the generated seed
                is logged).

        weight(weightfile)

                weighted random choice. Weights of input actions are
                defined in weightfile. The syntax for weightfile lines
                is:

                    ["tag-regexp"...,] "action-regexp"... : wdelta

                    ["tag-regexp"...,] "action-regexp"... = abs_weight

                wdelta (an integer) is added to the weight of actions
                that match any of action-regexps. This has an effect
                in states where a tag matches any of tag-regexps. If
                tag-regexps are not defined, wdelta affects matching
                actions in all states. abs_weight is the absolute
                weight value of actions.

                By default, weight of every action is 0. If all
                enabled actions have weight 0 or below, the choice is
                an evenly distributed random choice. Otherwise, choice
                is made according to the weights, in which case
                actions with 0 and below weights cannot be chosen.

                For example, weightfile "actions.w":

                    "iUseCash":   35
                    "iUseCredit": 60
                    "paying a car",   "iUseBank":    200
                    "paying a house", "iUseBank":    1000
                    "paying a house", "iUseCredit": -1000

                is loaded by configuration:

                    heuristic = weight(actions.w)

Examples:

        heuristic = exclude('.*Error.*', lookahead(3))

                chooses the first action of a three-step-path that
                gives the best coverage and does not include actions
                containing string "Error". Suggests an action
                containing "Error" only if nothing else can be
                suggested.

        heuristic = lookahead(5)

                chooses the first action of a five-step-path that gives
                the best coverage.

        heuristic = random(42)

                chooses a random enabled action with a random number
                generator initialised with seed 42.


Coverage
--------

Defines how coverage of test run is measured.

Defaults to "perm".

Syntax:

        coverage = type(params)

Available types & parameters:

        exclude(action-regexp [, action-regexp...], type(params))

                uses coverage type(params) to measure coverage, but
                execution of excluded actions will not affect the
                result. If action-regexp is not a name of action, it
                is handled as a regular expression.

        include(action-regexp [, action-regexp...], type(params))

                uses coverage type(params) to measure coverage, but
                execution of included actions only will affect the
                result. If action-regexp is not a name of action, it
                is handled as a regular expression.

        max(type(params) [, type(params)...])
        min(type(params) [, type(params)...])

                measure the maximum and minimum value of all coverages
                given as parameters.

        perm[(count [, action-regexp...])]

                measures the percentage of coverage of all
                permutations of any <count> actions in the test
                model. If count is not given, it defaults to
                2. action-regexps limit counting only permutations of
                certain actions.

	shared(<command>, type(params))

		uses a child process, that executes <command>, for
                sharing tested action sequences with other fMBT test
                runs in the same session. The other fMBT test runs may
                include already finished and currently running tests,
                and new fMBTs may join the session while this test is
                running. All traces covered by this and other fMBT
                instances count as covered in the measured coverage,
                defined by type(params).

                Examples:

                    shared(fmbt-trace-share --session 42, perm(1))
                    shared(ssh user@host fmbt-trace-share, perm(2))

        sum(type(params) [, type(params)...])

                measures the sum of all coverages given as parameters.

        uexecs(from regexp_1 to regexp_2)
        uinputs(from regexp_1 to regexp_2)
        upaths(from regexp_1 to regexp_2)
        uwalks(from regexp_1 to regexp_2)

                measure the number of minimal unique subtests that
                start from an action or a tag matching regexp_1, and
                end to an action or a tag matching
                regexp_2.

                - Executions are unique if they differ by at least one
                  action.

                - Inputs are unique if they differ by at least one
                  input action.

                - Paths are unique if they differ by at least one
                  state tag at any state.

                - Walks are unique if they differ by at least one
                  different action or state tag at any state.

                Otherwise they are not considered unique.

        usecase(<expression>)

                measures percentage of coverage of the <expression>
                that is given in coverage language. Valid expressions
                are:

                - quoted action-regexps, optionally prefixed by "all",
                  "any" or "random" quantifier. If no quantifier is
                  given, the default is to cover "any" action matching
                  the action-regexp.

                  Example:

                    usecase(all 'iUse(Cash|Credit)')

                    coverage reaches 1.0 when generated test includes
                    actions iUseCash and iUseCredit.

                  Both "any" and "random" quantifiers require that one
                  matching action is covered. "Random" means that the
                  action is randomly selected before test generation,
                  while "any" counts any matching action during the
                  generation.

                - two expressions combined with "and", "or" or "then"
                  operator.

                  Example:

                    usecase('iBuyCar' then ('iUseCash' or 'iUseCredit'))

                    coverage reaches 1.0 if "iBuyCar" has been tested
                    first, and "iUseCash" or "iUseCredit" gets tested
                    after it.

                - "not" <expression>, meaning that <expression> must
                  not be covered.

                - "[" source-tagexpr "]" followed by "(" <expression>
                  ")", and "(" <expression> ")" followed by "["
                  dest-tagexpr "]" filters. Expression is covered only
                  if source-tagexpr holds before, and dest-tagexpr
                  holds after testing the expression.

                  Example:

                    usecase(['paying a car'](all 'iUse.*|iCancel.*'))

                    coverage reaches 1.0 when all actions starting
                    with 'iUse' and 'iCancel' have been tested starting
                    from states with the 'buying a car' tag.

                  If "[tagexpr]" is prefixed with "@", then tagexpr
                  must hold in every state during executing the
                  expression.

                  Example:

                    usecase(@['paying a car'](all 'iUse.*'))

                    coverage reaches 1.0 when all iUse actions have
                    been tested without leaving states tagged "paying
                    a car" between actions.

                  "every n" and "exactly every n" shorthands forms
                  combinations of n matching tags.

                  Example:

                    usecase([exactly every 2 "music|alarm|phone"]
                            ("i:connect headset"))

                  coverage reaches 1.0 when "i:connect headset" has
                  been tested in every combination of exactly two tags
                  from "music", "alarm" and "phone" (representing
                  sound types being played in the background).

                  Cross product of tag sets defines tag combinations
                  that should be covered.

                  Example:

                    usecase("i:connect headset" ["music" x "alarm|phone"])

                  coverage reaches 1.0 when "i:connect headset" has
                  been executed so that tag pairs ("music", "alarm") and
                  ("music", "phone") are enabled at the target state.

                - expressions in parenthesis.

                - <number> * <expression>, meaning that <expression>
                  needs to be covered <number> of times.

                  Example:

                    usecase((5 * 'iUseCash') and (10 * 'iUseCredit'))

                    coverage reaches 1.0 after testing iUseCase five
                    times and iUseCredit ten times. These actions can
                    be tested in any order.

                - perm(n)

                  Example:

                    usecase(["paying a car"](perm(2)))

                    cover permutations of any two actions in states
                    with tag "paying a car".

                - tag(tag-regexp) is covered when every matching tag
                  has been seen in at least one state.

                  Example:

                    usecase(all "i:connect.*" then tag("playing music"))

                    cover all "connect" inputs and finally a state
                    with the "playing music" tag. Note that if the
                    current state after the last "connect" input is
                    tagged with "playing music", coverage is reached
                    without executing anything else.

        walks(from regexp_1 to regexp_2)

                like uwalks, but not restricted to unique walks only.
                That is, testing the same walk again increases the
                count.

        tag[(tag-regexp)]

                measures the percentage of covered versus matching
                tags. If tag-regexp is not given, measures covered
                versus all tags in the model.


        Following EXPERIMENTAL coverages may not pass all their tests
        and may still be changed before getting their final shape.

        number EXPERIMENTAL

                decimal numbers (like 1, 1.0, 0.5) can be used as
                constant coverage functions.

        steps[([count [, restart]])] EXPERIMENTAL

                without parameters: measures the number of executed steps.

                with count parameter: returns 0 until number of steps
                reaches count, then 1.

                with count and restart parameters: as above, but step
                counter is reset after restart steps. After restarting
                steps returns 0 again until count is reached. In
                conjunction with the if coverage, the steps coverage
                enables toggling between different ways to measure
                coverage.

                Example:

                if(steps(10, 12), 0, perm(2))

                measure perm(2) coverage for ten steps, then
                measurement results in 0 for any next two steps, then
                again measure perm(2) for ten steps, and so on. This
                will cause the lookahead heuristics to take two
                consecutive random test steps in the end of each dozen
                test steps.

        skip(type1(params1), type2(params2)) EXPERIMENTAL

                measures type2(params2), but it will not grow until
                type1(params1) has reached or exceeded 1.0.

                Example:

                skip(steps(100), perm(1))

                will return 0 until 100 steps have been
                executed. After that, perm(1) starts growing as usual.

        lt(type(params), type(params)) EXPERIMENTAL
        le(type(params), type(params)) EXPERIMENTAL
        gt(type(params), type(params)) EXPERIMENTAL
        ge(type(params), type(params)) EXPERIMENTAL

                return 1.0 (or more) when the left parameter is less
                than (lt), less or equal than (le), greater than (gt),
                or greater or equal than (ge) the right one. Otherwise
                measured value is less than 1.0.

        if(type1(params1), type2(params2)) EXPERIMENTAL

                returns the value of type1(params1) if it is less than
                1.0. Otherwise returns the sum of type1(params1) and
                type2(params2).

        if(type1(params1), type2(params2), type3(params3)) EXPERIMENTAL

                returns the value of type3(params3) if type1(params1)
                is less than 1.0. Otherwise returns the value of
                type2(params2).

Examples:

        coverage = exclude('.*Choose(Cash|Credit|Bank).*', perm(1))

                measures coverage of all other actions but those which
                include ChooseCash, ChooseCredit or ChooseBank in
                their names.

        coverage = include('iChoose(Cash|Credit|Bank)', 'iPay', perm(2))

                measures coverage of action pairs from the set of four
                actions: iChooseCash, iChooseCredit, iChooseBank and
                iPay. Because of measuring perm(2), reaching full
                coverage requires, for instance, testing action pairs

                iChooseCash, iPay
                iChooseCredit, iPay
                iChooseBank, iPay

        coverage = perm(1)

                good for a smoke test, coverage reaches 1.0 when
                every action in the model has been executed at least
                once.

        coverage = perm(2)

                if there are actions A, B and C in the model, coverage
                of perm(2) reaches 1.0 when AA, AB, AC, BA, BB, BC,
                CA, CB and CC have been executed. Suitable for
                regression tests. Notice that not all permutations can
                be covered because of test model. See End conditions.

        coverage = perm(2, ".*", "iReceivePhoneCall")

                coverage reaches 1.0 when all action pairs with the
                last action being "iReceivePhoneCall" have been
                covered.

        coverage = uwalks(from 'iOrderCoffee' to 'iCancelOrder')

                measures how many different minimal sequences from
                iOrderCoffee to iCancelOrder has been tested.

        coverage = "sum(uwalks(from 'iPause' to 'iContinue'),
                        uwalks(from 'iContinue' to 'iPause'),
                        perm(2))"

                measures how many different sequences have been tested
                from pausing playback to continuing it, and vice
                versa. Percentage of covered permutations of action
                pairs is added to the measure.

        coverage = usecase(a A then (a B or a C))

                coverage reaches 1.0 when action A has been executed
                and at least either of actions B and C has been
                executed after that.


History
-------

Fill coverage by reading test history from given log files before
starting test generation. This can be used, for instance, to generate
tests that have not been generated in previous test runs. There can be
several history lines in the test configuration.

Syntax:

        history = type(params)

Available types & parameters:

        log(filename[, filters])

                reads log from filename and feed it to configured
                modules. By default, coverage and learning get the log
                data. This can be changed with the filters
                (comma-separated strings). Supported filters:

                  C   - skip coverage (see Learning example)


Examples:

        coverage = perm(2)
        history  = log(smoketest.log)

                Assume that an already executed smoke test produced
                smoketest.log. Measure now coverage perm(2) and,
                before starting test generation, consider all
                permutions of two actions covered that were already
                tested in the smoke test.


End conditions
--------------

Define under which conditions testing must be stopped with a given
verdict. There can be many end conditions that result in the same
verdict.

If end conditions of the form "pass = coverage(...)" are not defined,
then the default end condition "pass = coverage(1.0)" is used. This
can be disabled by using "pass = coverage(inf)".

Syntax:

        <verdict> = measure(value)

        <verdict> = condition

Available verdicts:

        pass    test did not find any errors and tested what was wanted.

        fail    test found an error in the implementation under test.

        inconc  test did not find an error but was not able to test
                what was wanted.

Available measures, values and conditions:

        coverage(<decimal>)

                finish with the verdict if coverage reaches the given
                level. perm coverages return values from 0.0 to 1.0
                (including 0.0 and 1.0), some other coverages like
                path counters can return values greater than 1.0, too.

        coverage(<type(params)>)

		finish with the verfict if coverage type(params)
		reaches 1.0.

        deadlock

                finish with the verdict if model is in a deadlock
                state, that is, there are no enabled actions.

        duration(<timespec>)

                finish with the verdict if the time limit is reached.
                <timespec> is anything accepted by

                date --date='<timespec>'

                for instance, "1.5 secs", "1 hour 30 minutes", "2
                days", "1 week", or "2012-06-24 15:30".

        failing_tag[(<include-exclude>)]

                finish with the verdict if a tag check fails.
                include('tagname'[, 'tagname'...]) and
                exclude('tagname'[, 'tagname'...]) can be used
                to scope the behaviour only to certain tags.
                Include/exclude behaviour is EXPERIMENTAL.

        lookahead_noprogress

                finish with the verdict when the lookahead heuristic
                cannot find any way to grow measured coverage. Test
                generation must use the lookahead heuristic.

        noprogress(<integer>)

                finish with the verdict if <integer> test steps have
                been executed in a row without progress in measured
                coverage.

        tag(<name>)

                finish with the verdict when the current state of the
                test model is tagged with <name>.

        steps(<integer>)

                finish with the verdict when the number of test steps
                reaches <integer>.

Examples:

        The following end conditions define that test is passed if 50
        percent coverage is achieved, or if 1000 test steps have been
        executed. Test ends with "inconclusive" verdict if test run
        has lasted longer than 7 seconds. Test fails immediately if a
        state with tag this_should_never_happen is reached or model is
        in deadlock.

        pass   = coverage(0.5)

        pass   = steps(1000)

        inconc = duration(7 seconds)

        fail   = tag(this_should_never_happen)

        fail   = coverage(usecase('o:.*Error.*'))

        fail   = deadlock


Exit hooks
----------

An exit hook defines how fmbt process behaves when test run is
finished with given verdict.

Syntax:

        on_<verdict> = type(params)

or

        on_error = type(params)

        on_error hook is entered if an unexpected error prevents
        continuing test run. For instance, if the connection to a
        remote adapter is lost.

Available types & parameters:

        exit(status)

                exit with the given exit status. status can be a
                number or a coverage expression whose value (floor)
                will be returned as exit status.

        interactive

                enter interactive mode.

Examples:

        on_fail   = exit(1)

        on_pass   = exit(coverage(sum(16,usecase('iInteresting'))))

                if test is passed, exit status will be
                - 16, if iInteresting step was not executed during the test.
                - 17, if iInteresting was executed during the test.

        on_inconc = interactive

        on_error  = exit(42)


Learning
--------

Learning modules define how various observations during test execution
affect test generation.

Syntax:

        learn = type(params)

Available types & parameters:

        time([f])

                learn adapter block execution times. The lookahead
                heuristic will generate tests that reach the best
                coverage in the shortest wall clock time. By default
                it optimizes tests for the minimal number of test
                steps. The multiplier f (from 0.0 to 1.0) defines how
                quickly action's learned execution time follows new
                measurements. The equation is:

                  new_time = f * latest_measured_time + (1.0-f) * old_time

                The default f is 0.5. (That is, 1.0 uses always the
                latest measured time as is.)

                Example: learn adapter execution times from test.log
                before test generation, but do not cover previously
                executed test steps.

                learn = time
                history = log(test.log, C)


Other
-----

disable_tag_checking

        never execute adapter() blocks of tags.