Multiple Mutation Testing from Finite State Machines with Symbolic Inputs

. Recently, we proposed a mutation-testing approach from a classical ﬁnite state machine (FSM) for detecting nonconforming mutants in a given fault domain speciﬁed with a so-called mutation machine. In this paper, we lift this approach to a particular type of extended ﬁnite state machines called symbolic input ﬁnite state machine (SIFSM), where transitions are labeled with symbolic inputs, which are predicates on input variables possibly having inﬁnite domains. We deﬁne a well-formed mutation SIFSM for describing various types of faults. Given a mutation SIFSM, we develop a method for evaluating the adequacy of a test suite and a method for generating tests detecting all nonconforming mutants. Experimental results with the prototype tool we have developed indicate that the approach is applicable to industrial-like systems.


Introduction
Detecting nonconforming implementations is a major challenge during the design and the maintenance of systems, which motivates the elaboration of innovative and efficient testing [16,23], model-checking [6] and runtime verification techniques [12].Testing techniques [23] not only aim at exercising a system with adequate test cases to reveal failures and ideally to identify and to repair faults causing the failures.They may also target evaluating the adequacy of test cases and the generation of test cases to cover artefacts that can conceal faults [5,11,4,2], e.g., statements, branches, interfaces, requirements, mutants.Mutants which are versions of a specification of a system seeded with undesired faults can be used to generate test cases or to determine the adequacy of given test cases to reveal the faults.A fault domain can be specified with a set of mutants and test cases detecting the mutants which do not conform to the specification can be applied to detect faulty implementations of a system.Classical FSM model is often used in developing fault model based testing approaches for detecting nonconforming implementations.Recently we proposed an approach for this model to evaluate the adequacy of test cases in a given fault domain [19] and to generate test cases detecting all nonconforming mutants [20].
In case the testers need to deal with inputs with infinite domains, the finite input alphabets which is used in classical FSM to represent the inputs of a system becomes ineffective along with FSM-based testing approaches.In the automotive applications, the behaviors of some controllers [18] depend on the truth values of predicates defined over input variables with infinite domains.Extensions of FSMs with symbolic inputs and arithmetic operations on variables have been proposed [3,21,14] to relax limitations of the classical FSM and used in developing testing methods [9,21,14].Following the same trend, our test generation method by constraint solving from FSM in [19,20] could be enhanced to extended FSM.The work of [8] also uses an EFSM model and a mutation machine to model transition and output faults.Test generation requires (partial) unfolding of the specification, which we completely avoid.A test suite complete for used defined faults can only be generated if they satisfy certain sufficient conditions, which severely restrict types of detectable transition and output faults.Moreover, faults in transition predicates are not considered, as opposed to our approach.
In this paper, we lift the mutation testing approach from classical FSM in [19,20] to symbolic input finite state machine (SIFSM).SIFSM [21] is an extension of FSM with inputs specified with predicates on input variables possibly having infinite domains, which permits a more compact representation of data, data-flow relations and control-flow for determining outputs depending on the values of the predicates and states.Examples of realistic systems which can be specified with SIFSM can be found in [18,10].The contribution is three-fold.First, we define mutation operations for building well-formed mutation machines specifying mutants in fault domains.New mutation operations may change predicates used in the specification or introduce new predicates.Secondly, we propose a method for evaluating the completeness of a test suite, i.e., the adequacy of a test suite to detect all nonconforming mutants.Finally we propose a method for generating complete test suites.Following the ideas in our previous work [19,20], the methods rely on building and resolving constraints specifying the mutants undetected by given test cases.However, in this work the constraints differ from those in our previous work; they are represented with Boolean expressions for expressing both undetected mutants and the input-completeness property of the mutants.The latter property is formalized with a notion of cluster for state.This is needed because predicates cannot be mutated independently.We evaluate the methods with a prototype tool applied to a SIFSM model of a component from the automotive domain.
The remaining of the paper is organized as follows.Section 2 introduces mutation SIFSM and mutation operations used for its creation.In Section 3 we present an approach for determining the mutants undetected by a test, which leads to a method for completeness checking of a given test suite in Section 4. In Section 5 we develop a method for complete test suite generation.Section 6 reports some experimental evaluation of the approach.We summarize our contributions in Section 7.

Preliminaries
Let G denote the universe of inputs that are predicates over variables in a fixed set V for which a decision theory, e.g., an SMT solver, exists, excluding the predicates that are always false.G * denotes the universe of input sequences and ε denotes the empty sequence.Later in the paper, a test is just an input sequence.Let I V denote the set of all the valuations of the input variables in the set V , called concrete inputs.A set of concrete inputs is called a symbolic input; both, concrete and symbolic inputs are represented by predicates in G. Henceforth, we use set-theoretical operations on inputs.In particular, we say that concrete input x satisfies symbolic input g if x ∈ g.We also have that We define some relations between input sequences in G * .Given two input sequences α, β ∈ G * of the same length k, α = g 1 g 2 . . .g k , β = g 1 g 2 . . .g k , we let α ∩ β = g 1 ∩ g 1 . . .g k ∩ g k denote the sequence of intersections of inputs in sequences α and β; α and β are compatible, if for all i = 1, . . ., k, g i ∩ g i = ∅.We say that α is a reduction of β, denoted α ⊆ β, if α = α ∩ β.If α is a sequence of concrete inputs as well as a reduction of β then it is called an instance of β; given a finite set of input sequences E ⊆ G * , a set of concrete input sequences is called an instance of the set E, if it contains at least one instance for each input sequence in E.
Given a finite set of outputs O, a trace is a sequence of input-output pairs in We consider an extension of FSM called symbolic input finite state machine (SIFSM) [21], which operates in discrete time as a synchronous machine reading values of input variables and setting up the values of output variables.Output variables are assumed to have a finite number of valuations and form a finite output alphabet.On the other hand, the set of input valuations can be infinite.Definition 1.A symbolic input finite state machine S (or machine, for short) is a 5tuple (S, s 0 , V, O, T ), where -S is a finite set of states with the initial state s 0 , -V is a finite set of input variables over which inputs in G are defined, -O is a finite set of outputs, The semantics of SIFSM is defined by a Mealy state machine with a possibly infinite input set, where the state and output sets remain finite.The set of transitions outgoing from state s is denoted by T (s).We say that input g is defined in state s if g is the input of a transition in T (s).Then, G(s) denotes the sets of all the inputs defined in s.We say that transition (s, g, o, s ) is triggered by input g if g is a reduction of g.Several transitions in T (s) are nondeterministic if they can be triggered by the same input.If a set of transitions T (s) includes nondeterministic transitions, the set is said to be nondeterministic; otherwise it is deterministic.
An execution of S from state s is a sequence of transitions t 1 t 2 . . .t n forming a path from s in the state transition diagram of S. A deterministic execution is an execution such that its set of transitions is deterministic; otherwise, i.e., if for some state s and some transition in the execution there exists another transition such that both transitions belong to T (s) and are triggered by an identical input, the execution is nondeterministic.
A symbolic trace of S in state s is the projection of an execution from s on the inputoutput pairs in (G × O).A trace obtained from a symbolic trace in s by substituting every inputs by an instance of it is called a concrete trace of S in s.Let Tr S (s) (resp.STr S (s)) denote the set of all concrete (resp.symbolic) traces of S in state s and Tr S (resp.STr S ) denote the set of concrete (resp.symbolic) traces of S in the initial state.
We say that an input sequence triggers an execution of S (in state s) if it is a reduction of the input projection of a trace of the execution of S (in state s).Given an input sequence α, let out S (s, α) denote the set of all output sequences which can be produced by S in response to α at state s, that is out S (s, α) = {β ↓O | β ∈ STr S (s) and α ⊆ β ↓G }.We observe that out S (s, α) = out S (s, γ) whenever γ is a reduction of input α.
The machine S is deterministic (DSIFSM), if for every state s, T (s) is deterministic; otherwise S is a nondeterministic SIFSM (NSIFSM).Clearly, a DSIFSM has only deterministic executions, while an NSIFSM can have both.State s of S is completely specified, if G(s) is a tautology, i.e., each concrete input x ∈ I V satisfies at least one input defined at s.The machine S is completely specified, if each state is completely specified.The machine S is initially connected, if for any state s ∈ S there exists an execution from s 0 to s. Henceforth, we assume that all SIFSM are initially connected and completely specified.
We adapt several relations introduced in [19,20] for FSM to SIFSM and use tracebased definitions of the relations introduced in [21 . S is reduced if any pair of its states is distinguishable.Given two distinguishable states s 1 and s 2 , there exists a sequence α ∈ G * such that out S (s 1 , α) = out S (s 2 , α); α is called a distinguishing input sequence for states s 1 and s 2 , this is denoted s 1 α s 2 .
We also use relations between machines.Given SIFSM S = (S, s 0 , V, O, T ) and P = (P, p 0 , V, O, N ), P ≤ S if p 0 ≤ s 0 ; P S if p 0 s 0 ; P α S if p 0 α s 0 with α ∈ G * ; and P S if P α S for some distinguishing input sequence α for p 0 and s 0 .Later, we use equivalence relation between machines as a conformance relation between implementation and specification machines.
Given a NSIFSM S = (S, s 0 , V, O, T ), a machine P = (P, p 0 , V, O, N ) is a submachine of S if p 0 = s 0 , P ⊆ S and N ⊆ T .

Mutation machine
Let S = (S, s 0 , V, O, N ) be a DSIFSM, called the specification machine.
Transitions of M that are also transitions of S are called unaltered, while the others, in the set T \N , are mutated transitions.A transition of M is suspicious if it belongs to a nondeterministic set of transitions, let Susp(s) denote the set of all suspicious transitions in state s and Susp(M) denote the set of all suspicious transitions of M.An unaltered transition is trusted if it is not suspicious; otherwise it is untrusted and belongs to the set Untr(S) = Susp(M) ∩ N .Given state s, a subset of T (s) is called a cluster of s if Fig. 1: A mutation SIFSM, state 1 is initial.
it is deterministic and the inputs of its transitions constitute a tautology, in other words, the transitions of the cluster have a complete system of guards so that each concrete input enables a transition.Let Z(s) denote the set of all clusters of s.State s is said to be suspicious if |Z(s)| > 1.We use S susp to denote the set of all suspicious states of M.
In a mutation machine, untrusted transitions can be seen as the result of applying mutation operations transforming the specification into mutants.Mutation operations may also be considered as fault seeding in the specification.For an untrusted transition to belong to a mutant, it must participate in clusters.We say that a mutation machine is well-formed if each of its suspicious transitions belongs to a cluster.In what follows, we consider only well-formed mutation machines.
We assume that only completely specified deterministic submachines of M are possible implementation machines for the specification machine S. The set of all such submachines is called a fault domain for S, denoted Sub(M).If M is deterministic then Sub(M) contains just S. Since each implementation machine in Sub(M) is deterministic, each state of an implementation machine has only one cluster.The size of Sub(M) is the product of the sizes of the clusters of the states, i.e., |Sub(M)| = s∈S |Z(s)|.A DSIFSM P ∈ Sub(M), such that P = S, is called a mutant.Each mutant P has all the trusted transitions of M and the set of suspicious transitions Susp(P).It holds that for all P, P ∈ Sub(M), if P = P then Susp(P) = Susp(P ).
Fig. 1 presents an example of an NSIFSM which is a well-formed mutation machine with three Boolean input variables v 1 , v 2 and v 3 and two outputs in {0, 1}.The mutation machine has five mutated transitions depicted with dashed lines.The solid lines represent the unaltered transitions of the specification machine.Identifiers of transitions are presented in brackets and parentheses for mutated and unaltered transition, respectively.There are eight suspicious transitions t Let e be an execution of M and Susp(e) denote the set of suspicious transitions in e.We say that a (possibly nondeterministic or partially specified) submachine P is involved in e if Susp(e) ⊆ Susp(P).An execution of any submachine of M is an execution of M, but only deterministic executions of M are executions of submachines in Sub(M).P ∈ Sub(M) is the only mutant involved in e if Susp(e) = Susp(P).
Since the specification and mutants are completely specified and deterministic SIFSM, we use equivalence as a conformance relation for testing.A mutant P is nonconforming (faulty) if P S, otherwise, it is called a conforming mutant.We say that a distinguishing input sequence α ∈ G * such that P α S detects or kills the mutant P.
The tuple S, , Sub(M) is a fault model following [19,20,21].For a given specification machine S the equivalence partitions the set Sub(M) into conforming implementations and nonconforming ones.In this paper, we do not require the DSIFSM S to be reduced, this implies that a conforming mutant may have fewer states than the specification S; on the other hand, we assume that no fault creates new states in implementations, hence mutants with more states than the specification are not in Sub(M).

Mutation Operations for Building Well-formed Mutation Machines
Mutation operations permit seeding different types of faults including output, transition and other types of faults which cannot be represented with classical FSM.Considering for instance nondeterministic Simulink/Stateflow models, priorities which are automatically assigned to transitions based on the graphical layout may vary upon changes in the layout [22].The variation of the priorities causes transition faults which can be represented with mutated transitions.We consider mutation operations adding mutated transitions to well-formed mutation machines to build new well-formed mutation machines.Every mutated transition introduced by a mutation operation must belong to a cluster of a state.Let M = (S, s 0 , V, O, T ), M = (S, s 0 , V, O, T ) be two well-formed mutation machines, s ∈ S be a state, A ⊆ T (s) be a subset of unaltered transitions from state s in M and B ⊆ T (s) be a subset of mutated transitions from state s in M .We say that M is a mutation of M w.r.tA and B if the following four conditions hold: A ∩ B = ∅, T = T ∪ B, the union of the inputs of the transitions in A is equivalent to the union of the inputs of the transitions in B and there are t ∈ A and t ∈ B having compatible guards but different outputs or target states.We specify a mutation operation with a tuple (M, A, B) such that there exists a mutation of M w.r.t.A and B. The set B can be obtained from the transitions in A by changing target states or outputs, merging/splitting inputs of transitions, replacing variables with default values, swapping occurrences of variables in inputs, substituting a variable for another, modifying arithmetic/logical operations in guards.These operations introduce faults which cannot be represented in classical FSM; some of these faults are considered in [11,4,2].Any well-formed mutation machine for a specification can be obtained by iterative application of mutation operations on the specification.
we introduce |Susp(M)| Boolean variables each of which represents a suspicious transition of the mutation machine.From now on we will use t to refer to both a suspicious transition and the variable which represents it.Then the conjunction c e def = t∈Susp(e) t of variables of transitions in Susp(e) specifies the submachines involved in the revealing execution e.Moreover, the disjunction of conjunctions of all executions in E α gives Boolean expression c α def = e∈Eα c e specifying all the submachines which are involved in all executions in E α and killed by the test α.As usual, the disjunction over the empty set is False and the conjunction over the empty set is True.Boolean expression c α is satisfiable whenever E α = ∅, since an α-revealing execution of M is a projection of the execution of the distinguishing machine.A witness solution of c α provides all the variables evaluated to True and defines a corresponding subset of Susp(M) which together with the trusted transitions of M determines (the transition relation of) a submachine of M involved in α-revealing executions. Let 3 ) be a test case.It triggers four executions in the distinguishing automaton in Fig. 2.These executions are defined by four executions of mutation machine in Fig. 1 including e 1 = t 1 t 3 t 5 t 10 t 1 t 3 , e 2 = t 1 t 3 t 5 t 11 t 1 t 3 , e 3 = t 1 t 3 t 7 t 5 and e 4 = t 1 t 3 t 7 t 6 .The executions e 3 and e 4 are defined by the two executions to the sink state of the distinguishing automaton.Execution e 3 is not α-revealing because it is nondeterministic.Only execution e 4 is α-revealing and includes the two suspicious transitions in Susp(e 4 ) = {t 7 , t 6 }.Thus c e4 = t 7 t 6 and c α = c e4 .The solutions of c α determine the submachines involved in e 4 .
We denote by Generate_a_submachine(c) a function which either determines such a submachine from a solution of c it obtained after calling a solver or returns null if c has no solution.Nondeterministic and partially specified submachines are not mutants.To exclude such submachines as well as the specification from any solution, clusters in suspicious states has to be considered.
Let s be a suspicious state, Z(s) = {Z 1 , Z 2 , . . ., Z n } be the set of its clusters.Then the conjunction of variables of a cluster Z i expresses the requirement that all these transitions must be present together to ensure that a submachine with the cluster Z i is completely specified in state s.Moreover, since all mutants are deterministic, only one cluster in Z(s) can be chosen, therefore, the transitions are restricted by the expressions determining clusters.Each cluster Z i is exclusively determined by Boolean expression z i def = ( t∈Zi t) ∧ ( t∈Susp(s)\Zi t) which permits the selection of all the suspicious transitions in Z i and the exclusion of the remaining suspicious transitions leaving s, i.e., the exclusion of the other clusters.Lemma 1.Let Z i , Z j ∈ Z(s) be two clusters of state s.Every solution of z i is not a solution of z j .
Then each state s in S susp yields the expression c s def = n i=1 z i of which all the solutions determine all the clusters in Z(s).
Lemma 2. Every solution of c s determines a cluster in Z(s) and every cluster in Z(s) is determined by a solution of c s .
Each solution of s∈Ssusp c s determines the set of clusters of suspicious states either in the specification or in a mutant.Each such cluster in the specification has at least one untrusted transition in Untr(S).Excluding the specification can be expressed with the negation of the conjunction of the variables of all the untrusted transitions t∈Untr(S) t.Any of its solutions excludes at least one cluster in the specification and therefore cannot determine the specification.The Boolean expression c clstr Considering the example mutation machine, we determine the Boolean expressions for the suspicious states 3 and 4. For the four clusters of state 3 Z 31 = {t 5 , t 9 }, Z 32 = {t 5 , t 8 }, Z 33 = {t 6 , t 7 , t 9 } and Z 34 = {t 6 , t 7 , t 8 } we build Boolean expressions z 31 = t 5 t 9 (t 6 t 7 t 8 ), z 32 = t 5 t 8 (t 6 t 7 t 9 ), z 33 = t 6 t 7 t 9 (t 5 t 8 ) and z 34 = t 6 t 7 t 8 (t 5 t 9 ).Then c 3 = (z 31 ∨ z 32 ∨ z 33 ∨ z 34 ).Similarly for state 4, we build c 4 = (z 41 ∨ z 42 ) where z 41 = t 10 t 13 (t 11 t 12 ) and z 42 = t 11 t 12 t 10 t 13 .Finally, c clstr = c 3 ∧ c 4 ∧ (t 5 ∨ t 9 ∨ t 10 ).
A solution of c α ∧ c clstr defines a subset of Susp(M) which together with the trusted transitions of M determines (a transition relation of) a mutant detected by α.All solutions thus determine all mutants detected by the test α.For a non-trivial mutation machine, a sheer number of killed mutants makes their enumeration impracticable.Hence, instead of determining killed mutants, we determine a (conforming or nonconforming) mutant which survives the test α.The negation of c α , c α determines the transition relations of not only all mutants which survive α but also other submachines which are not mutants.Considering the running example, a partially specified submachine having the suspicious transitions t 9 and t 8 is determined by the solution of c α which assigns True t 9 and t 8 ; such a submachine is not a mutant and it does not belong to Sub(M).To eliminate them as well as the specification, we use c clstr as before.Finally, each mutant which survives test α is determined by a solution of the expression c α ∧ c clstr .
Theorem 2. Test α ∈ G * does not detect a mutant P if and only if there is a solution of c α ∧ c clstr which determines P.

Checking Completeness of a Test Suite
Given a fault model S, , M , a fault subdomain for S, FD is a subset of Sub(M).A test suite, TS is a set of tests.TS is complete for fault subdomain FD if it detects all the nonconforming mutants in FD.Let us define c TS def = α∈TS c α , a Boolean expression which determines the submachines involved in revealing executions for the tests in TS.Procedure Build_expression for building c TS is presented in Algorithm 1.
Let c fd be a Boolean expression specifying only all mutants in a fault subdomain FD.It can be formulated as the conjunction c clstr with another (possibly always True) Boolean expression over the variables of suspicious transitions, which excludes mutants from Sub(M) to obtain FD.A fault subdomain can always be refined with an expression specifying the mutants to be excluded.Later, in checking the completeness of a test suite for a given FD, we will be excluding conforming mutants.Theorem 3. Test suite TS is complete for fault subdomain FD if and only if c TS ∧ c fd has no solution or each of its solutions determines a conforming mutant.
The fault domain Sub(M) is specified with c clstr , which leads to Corollary 1.
Corollary 1. Test suite TS is complete for Sub(M) if and only if c TS ∧ c clstr has no solution or each of its solutions determines a conforming mutant.
Based on Theorem 3, checking the completeness of a test suite for a fault subdomain FD amounts to its iterative refinement by excluding conforming mutants as solutions to c TS ∧ c fd while no nonconforming mutant is found.In particular, the negation of the conjunction of variables of all suspicious transitions of a conforming mutant added to c fd excludes it from FD.This method is formalized in Algorithm 2 which presents Procedure Check_completeness for checking the completeness of a test suite TS for a fault subdomain specified by the input parameter c fd which is refined each time a conforming mutant is generated.The procedure Check_completeness also takes as inputs a test suite TS and the distinguishing automaton for the mutation and specification machines.It returns a witness test detecting a mutant surviving TS in case TS is not complete; otherwise the witness test is empty, which indicates that TS is complete.It also returns an updated expression of c fd specifying a reduced fault domain which is used to generate tests that make TS a complete test suite in Section 5. Procedure Check_completeness proceeds as follows.It calls Build_expression for building c TS , the Boolean expression which determines the submachines involved in revealing executions for tests in TS.Initialy, the fault domain is specified with the conjunction of c fd with the negation of c TS which determines all mutants surviving TS.The execution is iterative and each step consists in generating a mutant surviving TS, checking the conformance of the mutant and removing from the current fault domain the mutant in case it is conforming.
Procedure Check_completeness makes calls to Generate_a_submachine to select a mutant in a fault domain specified with Boolean expression c fd .Generate_a_submachine returns null in case the fault domain is empty.The execution of Check_completeness stops when Generate_a_submachine returns a nonconforming mutant or null.In case null is returned, the test suite is declared complete and Check_completeness returns the empty test; otherwise the test suite is declared incomplete and Check_completeness returns a non empty witness test detecting a nonconforming mutant.In both cases Check_completeness returns an expression specifying the reduced fault domain at the end of the execution.In the next section, we will check the completeness of generated tests (e.g., the witness tests) for the reduced fault domains in determining complete test suites for fault domains specified with mutation machines.Algorithm 2: Checking the completeness of a test suite for a fault domain In checking the completeness of the initial test suite {α} for the example mutation machine and test α, Check_completeness takes as input c fd = c clstr , TS = {α} and the distinguishing automaton in Fig. 2.Then, it determines c TS = c α = t 7 t 6 , sets c fd = c TS ∧ c clstr , c P = False and α = ε and starts executing the loop.In the first iteration, the call of Generate_a_submachine with input c fd has generated the mutant with the suspicious transitions t 8 , t 11 , t 12 .The mutant is nonconforming and killed by the test ) labeling a path to the sink state in the distinguishing automaton for the mutant.Then the execution of Check_completeness terminates with outputs c fd and non empty test β, which indicates that the test suite {α} is not complete.

Complete Test Suite Generation
In case an initial (possibly empty) test suite does not detect all the nonconforming mutants in a fault domain, we want to generate tests which together with the initial tests constitute a complete test suite for the fault domain.This can be done iteratively by adding a new test detecting a nonconforming mutant surviving the incomplete test suite, obtaining a new test suite which in turn can be augmented in case it is not complete.This complete test suite generation method is formalized in Algorithm 3 with procedure Complete_test_gen which takes as inputs an initial test suite TS init and a fault domain represented with a mutation machine.At every step, the procedure adds a current test  6 Prototype Tool and Experimental Results We implemented in JAVA a prototype tool consisting of three main modules.The first module for parsing mutation machines in text format was developed using ANTLR 4.1 [15].The second module is concerned with building clusters, distinguishing automata and Boolean expressions for undetected mutants; it uses as a back-end the solver Z3 [13] for solving of non Boolean expressions obtained by combining predicates in building clusters and automata.We integrated the solver in the tool using a Z3 API.The third module is responsible of solving Boolean expressions for mutants, extracting mutants and generating new tests.The module also uses solver Z3 though it may also use a SAT solver [7] since it deals with the resolution of Boolean expressions only.
In our experiments, we use a desktop computer with the following settings: 3.4Ghz Intel Core i7-3770 CPU, 16.0 GB of memory (RAM), Windows 7 (64 bits).
We use the prototype on an industrial-like SIFSM model obtained by transforming a Simulink/Stateflow model [18] of an automotive controller.To regulate the air quality in a vehicle, the controller sets an air source position to 0 or 1 depending on its current state and truth values of predicates on integer and Boolean input variables.The transformation required flattening and determinizing the original hierarchical Simulink/Stateflow model.The determinization is based on priorities assigned to nondeterministic transitions as it is done by Simulink [22].We obtained a SIFSM with 13 states, 62 transitions and 22 input variables.Then we have manually introduced faults (transition faults, output faults, swapping of variables, replacing variables with constants), obtaining a mutation machine with 2 13 − 1 = 8191 mutants.Our tool generates, within 30 seconds, a complete test suite with 14 tests detecting the mutants.Finally, we generate complete tests from automatically generated mutation machines with a generator executing randomly selected mutation operations.Table 2 presents the numbers of mutants in the mutation machines, the number of tests in the generated complete test suites.The maximal length of the tests is 8.We observed that the test generation is fast when the mutation operations introduce a small number of nonconforming mutants, which is a realistic assumption [11] for applying our method.

Conclusion
We lifted the multiple mutation testing approach developed for classical (Mealy) FSM to symbolic input finite state machine (SIFSM).SIFSM extends classical FSM with predicates defined over input variables with possibly infinite domains.
We defined well-formed mutation machines for SIFSM as a fault model for compact representation of a fault domain consisting of several faulty implementations (mutants) of a specification machine.Then we defined mutation operations for building wellformed mutation machines.Based on the machine equivalence and distinguishability relations, we have defined tests detecting nonconforming mutants and developed a multiple mutation testing approach from SIFSM.The proposed approach leveraging on that developed for classical FSM includes a method for checking the completeness of test suites, i.e., their adequacy to detect all nonconforming mutants in a fault domain, and a method for complete test suite generation avoiding mutant enumeration.The novelty of the proposed approach is that it can analyze and enhance completeness of symbolic tests w.r.t.user defined fault models for a specification with infinite input domains.
The experiments with a prototype tool we have developed indicate that our methods can be applied to industrial-like models of systems.
Our current work focuses on extending the approach to FSM with outputs determined by arithmetic operations over input and output variables [17], to FSM extended with timing predicates [1,14] and to C program.

Definition 3 .
Given a DSIFSM S = (S, s 0 , V, O, N ) and a mutation machine M = (S, s 0 , V, O, T ) of S, a finite automaton D = (D ∪ {∇}, d 0 , G, Θ, ∇), where D ⊆ S × S, ∇ is an accepting (sink) state and Θ ⊆ D × G × D is the transition relation is the distinguishing automaton for S and M, if it holds that

Fig. 2 Theorem 1 .
Fig.2presents the distinguishing automaton for the mutation and specification machines in Fig.1.Multiple transitions are represented with a single arc labeled with multiple inputs.An execution of D starting at the initial state d 0 and ending at the sink state ∇ is said to be accepted.The language of D, L D is the set of tests labeling accepted executions of D. Any nonconforming mutant in Sub(M) can be detected by a test in L D .Theorem 1.Given the distinguishing automaton D for S and M, P S for some P ∈ Sub(M) if and only if P α S for some α ∈ L D .

def=
s∈Ssusp c s ∧ t∈Untr(S) t Procedure Build_expression (TS, D); Input : TS, a test suite Input : D, the distinguishing automaton of mutation machine M and specification S Output : cTS, a Boolean expression defining submachines of M involved in revealing executions for tests in TS cTS := False; for each α ∈ TS do Using D, determine Eα, the set of α-revealing executions of M; cα := False; for each e ∈ Eα do ce := t∈Susp(e) t ; cα := cα ∨ ce; end cTS := cTS ∨ cα; end Return cTS; Algorithm 1: Building c TS excludes nondeterministic and partially specified submachines and the specification, which means that c clstr specifies only all mutants in the fault domain Sub(M).

Table 1 :
Execution of Procedure Complete_test_gen with the initial test α.