Multiple Mutation Testing for Timed Finite State Machine with Timed Guards and Timeouts

. The problem of generating tests detecting all logical and timing faults which can occur in real-time systems is challenging; this is because the number of (timing) faults is potentially too big or in(cid:28)nite. As a result, it might be time consuming to generate an important number of adequate tests. The traditional model based testing approach considers a fault domain as the universe of all machines with a given number of states and input-output alphabet while mutation based approaches de-(cid:28)ne a list of mutants to kill with a test suite. In this paper, we combine the two approaches by developing a mutation testing technique for real-time systems represented with deterministic timed (cid:28)nite state machines with timed guards and timeouts(TFSM-TG). In this approach, fault domains consisting of fault-seeded versions of the speci(cid:28)cation (mutants) are represented with non-deterministic TFSM-TG. The test generation avoids the one-by-one enumeration of the mutants and is based on constraint solving. We present the results of an empirical proof-of-concept implementation of the proposed approach.


Introduction
This paper deals with mutation testing of timed systems. Traditional modelbased testing approaches consist of using the model of the specication as a base for the generation of a test suite, then applying it on the implementation of the system under test. This is a black-box testing, meaning that the implementation is considered as unknown, but with the assumption that it may be described by a formal model. It is the so-called test hypothesis. Since it is generally impossible to generate an exhaustive test suite, several approaches have been proposed to limit the size of test cases while providing enough condence.
This work was partially supported by MEI (Ministère de l'Économie et Innovation) of Gouvernement du Québec.
Concerning FSM model-based testing, one may consider coverage criteria of the specication: the test may be satisfying if for instance all the transitions or all the states are covered by the test suite. Another approach consists of obtaining a sucient fault coverage: the test suite is guaranteed to detect all faults of a specic nature, e.g. output faults (the output provided by a transition is not correct) or transfer faults (the arrival state of a transition is not correct). As an example, the so called W-method [6] is known to detect both transfer and output faults whereas TT-method [16] 4 focuses only on output faults. Generally, fault coverage based methods consider the traditional fault domain as the universe of all machines with a specic alphabet and a given number of states.
On the other side, mutation testing is a technique originally developed to verify if a given test suite has a satisfying detecting power. In model-based testing, the principle consists of applying a certain number of small variations, called mutations, on the specication, then to check if these mutations are detected by the test suite. It is generally called killing the mutants. Note that mutation testing techniques are also used in code based testing, but in this case, the mutations are applied directly in the source code. Dealing with real-time systems increases the diculty of testing. For instance, faults related to timing errors should also be detected by a test method. For timed model based testing, many dierent models and conformance relations have already been proposed in the past. Some of them use a discrete representation of the time in the specication model, such as synchronous models [4,23,13], some other methods extend the so-called ioco theory [25] by dening a new conformance relation adapted to timed automata [1] with inputs and outputs [12,10,15,2,19,3]. Another category extends the FSM based testing theory using timed extensions of the FSM model and an adapted fault model [11,14,7,28].
The work in [9] proposes a method to generate a test suite with guarantee fault coverage for partial deterministic TFSM extended with timed guards only. TFSM with timed guards and timeouts (TFSM-TG) can express timed behaviors which cannot be expressed with TFSM with timeouts only [5]. More recently [26] proposes a method to generate tests detecting all the nonconforming implementations of an initialized TFSM with timeouts and TFSM-TG. The implementations belong to the traditional fault domain and they can have more states than the specication, up to a given number. The tests are generated by applying the W-method on an (possible huge) abstraction of TFSM-TG with classical FSM and transforming the resulting abstract tests into timed tests.
In this paper, we combine fault coverage based approaches and mutation testing applied to real-time systems. We propose a mutation testing technique for real-time systems represented with complete and deterministic TFSM-TG.
We derive complete test suites from mutation machines. A mutation machine can be designed to represent the traditional fault model; but it can also represent customized fault models as well. Customized fault models could be parts of the traditional fault model and they can be covered with a reduced number of tests. The tests generated for the traditional fault model remain valid to test 4 without special status input customized fault models. However they could suer from redundancy and could include useless tests. Running implementations with tests is time consuming.
Then, reducing the number of tests to apply is really useful. This is the main motivation of our work. Our test generation approach is based on constraint resolution. It uses the distinguishing automaton of the specication and the mutation machine to determine revealing combs. They characterized the undetected mutants and serve to encode them with Boolean formula. We use a solver to check the satisability of the formula and conclude about the completeness of test and generate new tests if needed. We propose an abstraction of timed states of TFSM-TG and we use it to build the distinguishing automaton. This paper extends the approach proposed in [21,17] in a timed context. This is the main contribution of our paper. It is adapted to be applied on the TFSM-TG model. This contribution includes the abstraction of timed states, the denition of the distinguishing automaton for TFSM-TG, and the representation of the transitions passed during executions of TFSM-TG with the so-called combs.
The paper is organized as follows. The next section introduces the general theoretical background, a fault model for TFSMs-TG and the coverage of fault models with complete test suites. Section 3 characterizes detected mutants with revealing combs. In Section 4 we introduce an abstraction for executions of TFSM-TG and we dene the distinguishing automaton. Section 5 presents an encoding of undetected mutants with Boolean formulas. We present a method for the test analysis and a method for the complete test suite generation in Section 6; we also present the results of an empirical proof-of-concept tool. We conclude the paper in Section 7.

Preliminaries
A timed guard π is an interval π = l π , u π where l π is a non-negative integer, while u π is either a non-negative integer or ∞, l π ≤ u π , and the symbols and represent ] or [. For example, the interval [0, 5[ contains all the non-negative real numbers smaller than 5. We let Π denote the set of timed guards. Given a timed guard π = l π , u π and a real number x, we dene π −x = max(0, l π − x), max(0, u π − x) .

Timed FSM with timed guards and timeouts
Denition 1. A timed nite state machine with timeouts and timed guards [5] (TFSM-TG) is a 6-tuple S = (S, s 0 , I, O, λ S , ∆ S ) where S, I and O arenite non-empty set of states, inputs and outputs, respectively, s 0 is the initial state, λ S ⊆ S × I × Π × O × S is an input/output transition relation and ∆ S ⊆ S × N ≥1 ∪ {∞} × S is a timeout transition relation.
Remark that we allow dening multiple timeout transitions in states of TFSM-TG, in the opposite of [5]. Later this will be used to compactly represent sets of implementations of a TFSM-TG specication. S in state s res an input/output transition (s, i, π, o, s ) to reach the transition's target state s and produces output o if input i is applied in transition's starting state s when timed guard π is respected; π expresses the minimal and the maximal delays in s to re the transition. It res a timeout transition (s, δ, s ) ∈ ∆ S dening timeout δ in s and reaches s if no input is applied in s before δ expires. Let δ max (s) denote the maximal (possibly innite) timeout dened in state s. We require that the maximal nite constant in guards of the transitions dened in every state s should be smaller than δ max (s). Because multiple timeouts can be dened in the same state, S necessarily res a timeout transition dening δ max (s) if no input is applied at s before δ max (s) expires. A clock measuring the amount of the time elapsed in every state is implicitly reset when transitions are red.
A timed state of TFSM-TG S is a pair (s, x) ∈ S × R ≥0 where s ∈ S is a state of S and x ∈ R ≥0 is the current value of the clock and x < δ for some δ ∈ N ≥1 ∪ {∞} such that (s, δ, s ) ∈ ∆ S . The initial timed state of S is (s 0 , 0). An execution step of S in timed state (s, x) corresponds either to the time elapsing or the ring of an input/output or timeout transition; it is permitted by is an execution step if it satises one of the following conditions: In the time-elapsing step, the target state s of t and s can be dierent. This is because the timeout δ is not expired and t is not red; δ max (s) is never exceeded.
An execution of S in timed state (s 0 , x 0 ) is a nite sequence of steps e = stp 1 stp 2 . . . stp n with stp k = ((s k−1 , x k−1 ), a k , t k , (s k , x k )), k ∈ [1, n] such that stp 1 is not an input/output step and stp k is an input/output step implies that stp k−1 is a time-elapsing step for every k ∈ [1..n]. If needed, the elapsing of zero time unit can be inserted before input/output steps which are not immediately preceded with a time-elapsing step. The timed input/output sequence of execution . . o l d l , respectively; they are said to be applicable and produced in (s, x), respectively. We denote by inp(s, x) the set of timed input sequences applicable in (s, x). Given a timed input sequence α, let out S ((s, x), α) denote the set of all timed output sequences which can be produced by S when α is applied in s, i.e., out((s, x), α) = {out(e) | e is an execution of S in (s, x) and inp(e) = α}. We let Exec S and Exec S (α) denote the set of executions of S and the set of executions with the timed input sequence α.
Transitions starting in the same state are called compatible if they are timeout transitions or have the same input and the intersection of their timed guards is non-empty. A TFSM-TG S is deterministic (DTFSM-TG) if it has no compatible transition; otherwise, it is non-deterministic. S is initially connected if every state of S is part of a reachable timed state. Let λ S (s, i) denote the set of input/output transitions dened in state s with input i. S is complete if the union of the timed guards of the transitions in λ S(s,i) equals [0, ∞[ for every (s, i) ∈ S × I; it implies that every i ∈ I is the input of at least one input/output step from every reachable timed state of S. Note that inp(s, x) = (I × R ≥0 ) * for every timed state (s, x) of a complete machine S.
We dene distinguishability and equivalence relations between timed states of complete TFSMs-TG. Then, we extend these relations to TFSM-TG. Similar notions were introduced in [28]. Intuitively, timed states producing different timed output sequences in response to the same timed input sequence are distinguishable. Formally, let (s, x s ) and (m, x m ) be the timed states of two complete TFSMs-TG dened on the same input and output sets. Given a timed input sequence α, (s,  [27,20] for testing other types of timed machines. Our operations are adapted to TFSM-TG. We compactly represent mutants with a mutation machine. Intuitively, mutants and the specication are all sub-machines of a global mutation machine describing the fault model.    Figure 2b has a dierent behavior in state s 3 for input a. The mutants and the specication have the same behavior in s 1 for input a; this is because t 1 is trusted.
Let P be a mutant with an initial state p 0 of the mutation machine M of S.
We use the equivalence relation to dene conforming mutants.  where the specication machine S is deterministic and complete and the mutation machine can have more states than S. Our approach consists in eliminating, from the fault model, the mutants detected by tests meanwhile avoiding their one-by-one enumeration. Undetected mutants will serve to generate new tests.

Revealing combs for characterizing detected mutants
We introduce combs to characterize mutants having common executions of the mutation machine since mutation machine includes all the mutants. The mutants dening all the transitions in a comb have common executions and can be detected with the same test. The comb for an execution e is the sequence of transitions permitting the steps in e; it is denoted by π e . The transitions in combs do not necessarily form paths in the state-transition diagram of M. This is because they contain non-red time-out transitions permitting time-elapsing steps when

Lemma 2. Test α detects mutant P if and only if there exists e ∈ Exec M (α)
such that π e ∈ Rev M (α) and Susp πe ⊆ λ P ∪ ∆ P . Corollary 1. Mutant P survives α if and only if for every π ∈ Rev M (α), some transitions in Susp π do not belong to λ P ∪ ∆ P . Figure 1b; it is also an execution of S 1 . The transitions permitting the steps appear in brackets and also serve to identify the comb. The rst, third and fourth steps in e 1 are time-elapsing. t 8 permits the fourth step. A distinct execution, let us call it e 2 , could involve t 11 at the fourth step. Another distinct execution is possible by making a timeout step at the third step, i.e., by ring t 11 . The comb π e1 = t 2 t 3 t 8 t 8 t 7 is for e 1 does not form a path in M 1 ; the symbol " " is inserted between timeout transitions permitting time-elapsing steps which just precede input/output steps. π e1 is deterministic and a mutant can dene all the suspicious transitions in it. π e1 is not revealing because the timed output sequence of e 1 , x0.5x7.5 is expected. The mutants dening only the suspicious transitions occurring in π e1 and all the trusted transitions cannot be detected by b0.5a7.5. π e2 = t 2 t 3 t 8 t 11 t 7 is not deterministic because t 8 and t 11 are compatible and suspicious; a mutant cannot dene these two transitions. π e2 is not revealing since it is not deterministic.

Distinguishing Automaton and Revealing Combs
The distinguishing automaton for a specication S and a mutation machine M is aimed to represent synchronous executions of S and the mutants in M. It will be used to identify unexpected executions of the mutants and the corresponding combs without enumerating the mutants one-by-one. Each of its states is com-   The deterministic revealing combs for a test α can be computed from the distinguishing automaton for the specication and mutation machines. Their computation works as follows. First we compute the accepted executions of the distinguishing automaton D with α; this can be done by dening a product an automaton for α and D, which we do not formalize for the sake of simplicity.
Each accepted execution corresponds to an execution of the specication and an unexpected execution of the mutation machine; the deterministic combs for the unexpected execution of the mutation machine belong to Rev M (α).
The following corollary is a consequence of Lemma 4 and Lemma 1. The encoding of the detected mutants uses the suspicious transitions in revealing combs corresponding to the tests in a test suites. This encoding is inspired by Lemma 2. Given a set of revealing combs Rev M (α) for test α, we let ϕ α = π∈Rev M (α) t∈Suspπ t be a Boolean formula over the variables for the suspicious transitions in revealing combs of Rev M (α). ϕ α encodes all the mutants and also non-deterministic sub-machines involved in revealing combs; this means that every solution of ϕ α determines a sub-machine of M that denes all the transitions in a comb π ∈ Rev M (α). So, ϕ α determines all the sub-machines with unexpected outputs for input α. Let ¬ϕ denote the negation of ϕ. Every solution of ¬ϕ α sets variables for some suspicious transitions in every revealing comb to False; this indicates that every mutant determined by ¬ϕ α does not dene some suspicious transitions from each comb in Rev M (α) . Lemma 5. For every P determined by ¬ϕ α and every π ∈ Rev M (α), there exists t ∈ Susp π which does not belong to λ P ∪ ∆ P .
The following lemma is a consequence of Lemma 5 and Corollary 1. Lemma 6. Every mutant determined by ¬ϕ α survives α.
Let T S = {α 1 , α 2 , . . . , α n } be a test suite. We dene ϕ T S = αi∈T S ϕ αi . The formula ϕ T S determines all the sub-machines which produce unexpected outputs for an input α i ∈ T S, i.e., the sub-machines detected by at least one test in T S. A determined sub-machine is not necessarily a mutant; so we need to encode all the mutants in M. We can proof the following lemma by using Lemma 6.

Lemma 7. Every mutant determined by ¬ϕ T S survives the test suite T S.
Note that ¬ϕ T S determines not only mutants but also non-deterministic sub-machines of the mutation machine M = (M, m 0 , I, O, λ M , ∆ M ). In other to exclude these non-deterministic sub-machines of M, we encode the set of mutants with a Boolean formula ϕ M . Each of its solutions determines exactly one timeout transition in every state (which is expressed by (Eq 2)) and a subset z mi ∈ Z mi , where Z mi is the maximal set of subsets of λ M (m, i) satisfying the cluster conditions introduced in Section 2.2. Let us dene: We can prove the following theorem by using Lemma 8 and Lemma 7.
Theorem 1. Formula ϕ M ∧ ¬ϕ T S determines exactly the mutants undetected by the test suite T S.
The test α 1 = b0.5a7.5 triggers executions in the distinguishing automaton for M 1 . These executions correspond to the expected execution e 1 , e 2 and : s 1 , 0). The comb for e 3 and e 4 are π e3 = t 2 t 3 t 11 t 15 t 12 and π e4 = t 2 t 3 t 11 t 15 t 13 ; they are revealing because the produced timed output sequence x0.5y7.5 is unexpected; so they characterize the mutants detected by test α 1 . We recall that π e1 and π e2 are not revealing. The formula ¬ϕ α1 = ¬((t 11 ∧ t 12 ) ∨ (t 11 ∧ t 13 )) encodes the mutants surviving α 1 , e,g., the mutant P 2 in Figure 2b In order to generate a complete test suite, our procedure analyzes the current test suite T S. If it is complete the procedure stops and return T S; otherwise we generate a test α detecting nonconforming mutants from a mutant surviving T S. α can be the timed input sequence of an execution of the distinguishing automaton for the specication and a surviving mutant (which is a part of the mutation machine), according to Corollary 4. Then we compute ϕ α and we analyze the new test suite T S = T S ∪ {α} by solving the formula ϕ M ∧ ¬ϕ T S ∧ ¬ϕ α ; this may trigger the generation of a new test. We generate iteratively new tests until all the procedure stops and a complete test suite is returned.
Let us compare our complete test suite generation approach with the one proposed in [26]. We focus on detecting specic faults represented in the mutation machine which can have more states than the specication, so do the mutants.
The specication machine is not necessarily "minimal". We believe that for the traditional fault model, the method in [26] could be faster than ours (especially if the time required to minimize the specication is not considered), but the method in [26] could generate redundant tests. The reason is that the method in [26] is an application of the W-method over a classical TFSM which represents an abstraction of TFSM-TG and the W-method was developed for minimal classical FSM. The size of the abstract TFSM-TG [26] could contribute to decrease the eciency of the test generation [8,26]. The W-method generates tests by combining input sequences (namely, state cover sets, all input sequences of a certain length and state identication input sequences or characterization sets); can be bigger and less adequate to focus on faults specied in customized fault models represented with mutation machines. Where [26] will generate a huge test suite to detect all possible faults, our method will generate test suites of reduced sizes for customized fault models.
We performed an empirical evaluation of our approach with a proof-ofconcept tool developed in C++ and randomly generated TFSM-TG with two inputs and outputs, with a maximal timeout of 5 (resp. 10) for the specication (resp. the mutations machines). The tool uses cryptoSAT [24]. Preliminary results appear in Table 1 which presents sizes and generating times for test suites from mutation machines dening multiple of mutants with few states. For each number of states and number of mutants, we generated tests for several mutation machines. The tool can take a long time in encoding the fault domain; this may happen when there are too many compatible (not necessarily equal) timed guards and timeouts dened in the same state. Since this operation is done for every state, it could be distributed. In most of the situations, the tool can rapidly encode the fault domain and the test generation becomes faster.

Conclusion
We have proposed a multiple mutation testing theory to testing real-time systems represented with nite state machine extended with timed guards and timeouts (TFSM-TG). We developed an approach to generate complete test suites for fault models represented with mutation machines. The approach relies on the denition of distinguishing automaton for mutation machine and the construction of Boolean formulas encoding the (faulty) implementations undetected by tests. We implemented the approach in a proof-of-concept tool which we used to evaluate the eciency of the approach. The experimental results show that the approach can be used to derive tests for non-trivial TFSM-TG fault models representing an important number of faults.
Ongoing work includes developing open access benchmarks and use them to compare the existing test generation methods for TFSM-TG. We also plan to generate symbolic tests, i.e., timed input sequences with delay intervals instead of simple delays, and complete test suites consisting of a single test.