Inka: Ten years after the first ideas

INKA: Ten years after the first ideas Arnaud Gotlieb Lande project team IRISA-INRIA (In collaboration with Bernard Botella from Thales Aerospace) INKA: A constraint-based white-box testing tool • Automated white-box testing: Test data selection based on a code-based objective (reach a statement, a branch, a path, a def/use pair, etc.) or a coverage criterion (all_statements, all_decisions, MC/DC, etc.) • A constraint-based approach: Converts the test objective and the program under test into a constraint model and exploits constraint programming techniques to find the test data 1 A trivial example int f( int i ) { 1a. j=2 1b. if( i ≤ 16 ) 2. j=j*i if( j > 8) j=0 3. 4. 1 2 f Find a value of i such as statement 4 is executed ? 3 t f 4 5 return j 5. t } Intuition of our approach int f( int i ) { j=2 if( i ≤ 16 ) j=j*i i ≤ 16 ⇒ j = 2 * i if( j > 8) j=0 i ≤ 16, 2 * i > 8 return j } j>8 i > 16 ⇒ j = 2 5 ≤ i ≤ 16 2 The Automatic Test Data Generation problem (Undecidable in the general case) Difficulties for classical ad-hoc methods : 9 Non-feasible paths 9 Highly combinatorial f (int x1, int x2, int x3) { ... } 232 possibilities × 232 possibilities × 232 possibilities = 296 possibilities 9 Floating-point computations 9 Pointers, dynamic structures, function calls, etc. INKA: history 1995 -- Start of the Research works (Start of my PhD thesis) in Thales 1996– “Automatic Test Data Generation using CLP” ICSSEA’96 -1st publication 1998-- Prototype tool for C-- , first experimental results over a (small) realtime embedded program (parts of the BCE – Avion Banc d’Essai Rafale) 2000–02 RNTL Inka (Thales, Axlog, 3 academics labs I3S, LIFC, LSR) 2002-- INKA V1 (Principal investigator: me) – (Validation: SA RTOS Rafale) 2003–06 RNTL Danocops (Thales, Axlog, 3 academics labs I3S, LIFC, LSR) ACI V3F project (floating-point numbers) ( INRIA Cassis, Coprin, Lande/Vertecs, CEA, Thales) 2005 – INKA V2 in progress (Principal investigator: B. Botella from Thales) 2006 – (Re-)start of Research works in the INRIA’s Lande team 3 INKA V2: current scope LANGUAGE: a small subset of the C/C++ language All ANSI Integer data types (bool,char,short,long,unsigned short, ...) All control structures and almost all operators Some IEEE-754 Floating-point data type (float,double) Structures and «arrays/strings» with restrictions Function calls and some method calls (including static operator/method overloading) Pointers and references (including dynamic allocation and deallocation) Inheritance and some static type casting (implicit and explicit) Namespace But: No break/continue/goto, No exceptions No unconstrained pointer arithmetic No function pointers, no « const », no (void *) (2nd order programming), No dynamic C++ features (virtual method, templates, dyn. op. overloading, typeid,..) No library function/method calls, no external calls No function with an unknown number of parameters No memory type casting (ex: int * Æ float *): requires a bit-to-bit reinterpretation of the physical memory, no fields of bits INKA V2: features MAIN: - Coverage monitoring (on the constraint model) = simulated execution - Automatic test data generation for a given block of statements or decision Automatic test data generation for All_statements, All_decisions, and MC/DC ADDITIONAL: - Automatic detection of (some) non-feasible block or decision - Partial verification via automatic refutation of (numeric) properties 4 INKA V2: Architecture C/C++ program GUI : Test case management Test objectives design Pre/Postconditions design Makefile generator Test monitoring Symbols table ASA Outline of the talk 1. Interpretation and solving mechanisms 2. Control combinators 3. Introducing constraints on memories Normalisation 4. Further work FINOP Test objective Test data Control structures (ite, w,..) Pilot Interpreter Test data Path covered Solver Memory Structures Floats Tableaux Pointers Integers FINOP : Prolog Intermediate Normal Form 5 INKA V2: FINOP • FINOP (Prolog Intermediate Normal Form) • Small instructions set (i.e. arrays Æ pointers) • New temporary variables (complex statements are broken) • Reification of control statements (decision variables only) • Explicit allocations and casting (définition/undéfinition) • Numbering of control statements, allocations, function calls, etc. • INTERPRETER • Adds memory references to each interpreted statement • Adds a test objective to each interpreted statement • Computes the path followed within a control structure NB: if_then_else, while_do, … contain FINOP statements that are interpreted during statement evaluation Parts of the C FINOP’s grammar: Structure ::= defstruct(Ident, [ { Descr_type }* ] ) Program ::= { Function | Method }* Function ::= fct(Ident, { Formal_parameters }*, Formal_return, Body) Body ::= [{ Statement }*] Statement ::= Declaration | Control | Alloc | Dealloc | Assign | Assert | Sp_call Declaration ::= def(UID, Var, VName, Type, { Expression }*) Control ::= ite(UID, Var, Body, Body) | w(UID, Var, Body) | dow(UID, Var, Body) | sw(UID, Var, (Label)*, (Body)*, Body) Alloc ::= new(UID, Ident, BlocId, Type, Expression) ::= delete(UID, Ident) | undef(UID, Var) Dealloc Assign ::= assign(UID, Ident, Expression) Assert ::= test(UID, LogicalExp) Sp_call ::= fct_call(UID, Ident, { Ident }*, Ident) LogicalExp ::= Var | LogicalExp && LogicalExp, | LogicalExp || LogicalExp Expression ::= Ident | &(Ident) | Un_op(SimpleExpr) | Bin_op(SimpleExpr, SimpleExpr) Type ::= entier(Tint(size)) | flottant(Tfloat(size)) | structure(TName) | pointer(Type) Tint ::= char | int | short | long | uint | ushort | ulong | uchar | bool Tfloat ::= float | double Ident ::= Var | *Var | Var->Nfield | Var.Nfield Un_op::= ! | - | ~ Bin_op ::= == | != | <= | < | >= | > | & | '|' | ^ | && | || | + | - | / | % | * 6 INKA V2: several collaborating solvers • Constraint library over the pointers domain • Constraint library over the floats domain • Constraint library over structures accesses and updates (struct + class) -------------------------------------------------------------------------------------------------• Constraint library over memories, over tableaux from the memory -------------------------------------------------------------------------------------------------All these constraints are managed through a common propagation queue (called Agenda) that propagates the constraints ------------------------------------------------------------------------------------------------• Collaborates with clp(fd) for constraints on integers ------------------------------------------------------------------------------------------------Implementation: • Environment (agenda, ctrs_network, timeout, K_flag, epsilon, …) • Variables (sort, associated_ctrs, variables_of_the_neighborhood, …) • Constraints (relation, in_agenda, ignore, associated_vars, …) INKA V2: Architecture C/C++ program GUI : Test case management Test objectives design Pre/Postconditions design Makefile generator Test monitoring Symbols table ASA Outline of the talk 1. Interpretation and solving mechanisms 2. Control combinators 3. Introducing constraints on memories Normalisation 4. Further work FINOP Test objective Test data Control structures (ite, w,..) Pilot Interpreter Test data Path covered Solver Memory Structures Floats Tableaux Pointers Integers FINOP : Prolog Intermediate Normal Form 7 2. Control combinators ) ) Set of guarded-constraints with « don’t care » non-determinism { C1 Æ C’1 , ..., CnÆ C’n } Operational semantic: (with store σ: conjunction of domain constraints) -If Ci is entailed by σ then C’i is pushed on the propagation queue and {Cj Æ C’j}∀j are all removed from the queue -If Ci is disentailed by σ then only Ci → C’i is removed -Else Ci → C’i is suspended and would awaked whenever at least one of its variable domains is modified ) Detection of entailment: Ci is entailed by σ if σ ∧¬Ci is inconsistent Conditional: the combinator ite V := Decision ; 0 Then_part if(V) 2 Else_part 1 3 ite( V, CTHEN, CELSE, MIN, MOUT) :• V=1 → CTHEN ∧ MOUT = MTHEN • V=0 → CELSE ∧ MOUT = MELSE • ¬(V=1∧ CTHEN∧ MOUT = MTHEN) → V=0 ∧ CELSE∧ MOUT = MELSE • ¬(V=0∧CELSE∧ MOUT = MELSE) → V=1 ∧ CTHEN∧ MOUT = MTHEN • MOUT := Proj(OUT, MTHEN ∪ MELSE) MIN := Proj(IN, MTHEN ∪ MELSE) 8 Iteration: the combinator w V := Decision ; 2 while( V ) 1 Body 3 w(V, CBODY, MIN,MOUT) :• V=1 → CBODY ∧ w(V, CBODY,MBODY,MOUT) • V=0 → MOUT = MIN • ¬(V=1 ∧ CBODY ) → V=0 ∧ MOUT=MIN • ¬(V=0 ∧ MOUT=MIN)→ V=1 ∧CBODY ∧ w(V,CBODY,MBODY,MOUT) INKA V2: Architecture C/C++ program GUI : Test case management Test objectives design Pre/Postconditions design Makefile generator Test monitoring Symbols table ASA Outline of the talk 1. Interpretation and solving mechanisms 2. Control combinators 3. Introducing constraints on memories Normalisation 4. Further work FINOP Test objective Test data Control structures (ite, w,..) Pilot Interpreter Test data Path covered Solver Memory Structures Floats Tableaux Pointers Integers FINOP : Prolog Intermediate Normal Form 9 An abstract model of the physical memory TAB : tableau status: closed or not cont. : { @i – Vi, …} M : memory Integers : TABi Floats : TABf Pointers : TABp Structures : [S1,S2,..] V : integer within a finite domain Type : 16,32,64 bits, signed, unsigned dom : {possible values} Min .. Max V : float within an interval Type : float (32), double (64) dom. : Min .. Max V:pointer possibly_null : yes, no dom : {possibles values} nondom : {non-possible values} S : structure status : closed or not cont. : {@i} INKA V2: Introducing constraints on memories • Memories = unknowns representing states (sets of pairs Adress-Value) • Relations on these unknowns, constraint reasonning on these unknowns C program i=i+1 *p = 3 Constraints store -----------------------------------> load_elt(@i, I1, M1) I2 = I1 + 1 store_elt(@i, I2, M1,M2) ------------------------------------> load_elt(@p, P1, M2) DP1 = 3 store_elt(P1,DP1,M2,M3) j = i + 2 ------------------------------------> load_elt(@i,I3,M3) J1 = I3 + 2 store_elt(@j,J1, M3,M4) 10 Constraints on memories ƒ new_elt(TYPE, X, V_INIT, M0, M1, ENV) ƒ delete_elt(TYPE, X, M0, M1, ENV) ƒ load_elt(TYPE, X, VALUE, M, ENV) ƒ store_elt(TYPE, X, VALUE, M0, M1, ENV) ƒ M1 = M2 /* Useful in control structures */ ƒ closed(M) /* Useful to closed the memory during final search */ store_elt(P,V,M1,M2) M1 : Status : not closed Includes : i – Vi j – Vj k – Vk … P: Domain pointer {i,j} Store_elt M2 : Status : not closed Includes : i – Vi’ j – Vj’ k – Vk’ … V: Domain Integer 1.. 5 11 store_elt(P,V,M1,M2) M1 : Status : not closed Includes : i – Vi Æ 1.. 2 j – Vj Æ 5.. 9 k – VkÆ 2 … M2 : Status : not closed Includes : i – Vi’ Æ 3..6 j – Vj’ Æ 7..18 k – Vk’Æ ? … Store_elt P: Domain pointer {i, j} V: Domain Integer 1.. 5 Automatic deductions after the constraint propagation step : P = i, V = Vi’ in 3..5, Vj = Vj’ in 7..9, Vk = Vk’ =2 Metamodel for the definition of a new constraint success exit Constraints Store Awake SVAR Suspend fail failure reduce 12 store_elt(P,V,M1,M2) .. V ) and M2[p] = ( d d a d .. ) then ]=V) an != Null nd M2[i } a nd p a i {p = = ) P (P n add ( If( dom [i] ) the ] != M1 [i 2 M ( If success Constraints Store Awake SVAR Suspend If( d om If( d om (V) = (P) = ∅o {p} a reduce r do nd p m(P )= =N ull) ∅) the n the n fail fail dom(P) Å {i / dom(M2[i] ∩ dom(V) != ∅} dom(V) Å ∪i∈dom(P) dom(M2[i]) dom(M1[i]) Å dom(M2[i]) ∩ dom(M1[i]) if( i ∉ dom(P) ) dom(M2[i]) Å dom(M1[i]) ∩ dom(M2[i]) if( i ∉ dom(P) ) dom(M2[i]) Å dom(M1[i] ∪ dom(V)) otherwise failure Constraints over the integers domain • new(X, TYPE, ENV) • affiche(X) • get_domaine(X, DOM) • set_domaine(X, DOM) • empty_domaine(DOM) • intersection_domaine(D1, D2, DI) • union_domaine(X, LY, DX) ------------------------------------------------------• affect(X ,Y) • affect(const(ATOME),X) • affect(in(Min,Max) ,X) • affect('-' ,X,Y) • affect('~' ,X,Y) • affect('!' ,X,Y) • affect(conv(double,long) ,A,R) • affect(conv(float,long) ,A,R) • affect(conv(signed,unsigned),A,R) • • • • • • • • • • • • • • • • • • affect('==', X,Y,B) affect('!=' , X,Y,B) affect('<=', X,Y,B) affect('<' , X,Y,B) affect('>=', X,Y,B) affect('>' , X,Y,B) affect('&' , X,Y,Z) affect('|' , X,Y,Z) affect('^' , X,Y,Z) affect('&&', X,Y,Z) affect('||', X,Y,Z) affect('+', X,Y,Z) affect('-', X,Y,Z) affect('/', X,Y,Z) affect('%’ , X,Y,Z) affect('*', X,Y,Z) affect( '>>', X,Y,Z) affect('<<', X,Y,Z) 13 Constraints over the floats domain • new(X,T,ENV) • get_domaine(X,I..S) • set_domaine(X,I..S) • empty_domaine(DOM) • intersection_domaine(D1, D2, DI) • union_domaine(X, LY, DX) -----------------------------------------------------• affect(A ,R) • affect(const(ATOM) ,R) • affect(in(Binf,Bsup) ,R) • affect('-' ,A,R) • affect(conv(float,double) ,A,R) • affect(conv(double,float) ,A,R) • affect(conv(double,long) ,A,R) • affect(conv(long,double) ,A,R) • affect(conv(float,long) ,A,R) • affect(conv(long,float) ,A,R) • • • • • • • • • • affect('+' ,A,B,R) affect('-' ,A,B,R) affect('*' ,A,B,R) affect('/' ,A,B,R) affect('=<',A,B,R) affect('<' ,A,B,R) affect('>=',A,B,R) affect('>' ,A,B,R) affect('==',A,B,R) affect('!=' ,A,B,R) Constraints over the pointers domain • new(X, TYPE, ENV) • affiche(X) • get_domaine(X, DOM) • set_domaine(X, DOM) • empty_domaine(DOM) • intersection_domaine(D1, D2, DI) • union_domaine(X, LY, DX) ----------------------------------------------------• affect(X ,Y) • affect('>', X,Y,B) • affect(in(DOM), X) • affect('+', X,Y,Z) X: pointer, Y:integer, Z:pointer • affect(const(‘0’),X) • affect('-‘ , X,Y,Z) X: pointer, Y:integer, Z: pointer or • affect('==', X,Y,B) X pointer, Y pointer, Z: integer • affect('!=‘ , X,Y,B) • affect('=<' , X,Y,B) • affect('<' , X,Y,B) • affect('>=‘ , X,Y,B) 14 Constraints on structures • new(S,TYPE_S,ENV) Declaration of a structure variable S with void contents and not-closed status • new_s(TYPE_S, X, S0, S1, ENV) Definition (allocation) of a reference X over an object of type structure or class TYPE_S. • delete_s(TYPE, X, S0, S1, ENV) Deallocation of the object referenced by X • access_s(TYPE,X, Champ, VALUE, S, ENV) Access to the field « Champ » of the object referenced by X of type structure or class S. • S0 = S1 Equality between structures • closed(S) INKA V2: Labelling • Labelling on the contents of the input memory - Giving a « shape » to the input memory (pointed objects, structures) - Giving values to basic variables and valid references to pointers • Labelling on the test objective (reaching a selected element within a loop) • Labelling on paths of the control flow graph 15 Why giving a « shape » to the input memory ? • The function under test contains pointers or objects including references as inputs : int f( int *p, struct cell *t) • Reaching the selected element requires objects or structures to be created first: x = t->next->next ; ----------------------------------------------------labelling on the possible « shapes » of the input memory Tries first to avoid creating anything else in the memory and then instantiates all the tableaux p t p ? ? t null null The input memory is then closed, propagating so its shape to any other memories of the program Why giving a « shape » to the input memory Æ labelling on the possible « shapes » of the input memory • Creates a single object for each pointer and instantiates all the tableaux t p p t t.key t.key ? t.next ? 3 t.next 5 null ? t p t.key 3 t.next 5 16 Labelling on the shapes • Unbounded, requires strong stopping criteria • Strategy that can produce more objects than strictly required: p t p t.key t.next 5 t t.key null t.next 5 haven’t been tried ! DEMO ! 17 Current work • Function calls modeled as constraint combinators Æ to avoid introducing tones of constraints within the agenda (current work of Florence Charreteur) • Improving constraint refutation by building a solver that combines the advantages of finite domain constraint solving and a linear constraints solver (PPL, clp(Q)) (PhD thesis of Tristan Denmat) • Integrating static analyses during constraint resolution. In particular, points-to analyses will reduce the negative effects of defining a statement as a relation between two memories Perspectives • Dealing with (unconstrained) pointer arithmetic. Exploiting dynamic analyses to define a view of the physical memory • INKA for Java bytecode: constraints over the stack of operands (accesses and updates), virtual methods calls (bytecode invokevirtual) • Foundations of constraint-based testing: relational semantics 18 View publication stats