Review of paper

Integrated Language Definition Testing

GROUP- 13
Arora,Kashish (1049490)
Potu,Raja Sathvik (1056684)
Yenkammagari,Pranay Reddy (1023479)

Abstract
This paper is all about the configuration of a non specific, explanatory
test particular dialect for dialect definition testing. A completely dialect
rationalist way to deal with dialect installing that fuses syntactic,
semantic, and manager administration parts of a dialect under test. The
usage of such a testing dialect as the Spoofax testing dialect and a
portrayal of its execution building design.

Outlines
Introduction :- background on language definitions.
Section 2 :- the design of a language parametric testing language from
three angles
Section 3 :- purely linguistic perspective
Section 4 :- tool support perspective
Section 5 :- illustrating use cases with examples.
Section 6 :- implementation architecture is described.
Section 7 :- We conclude with related work on language testing
approaches and directions for future work.

1. Introduction
Programming dialects give phonetic deliberations to a space of
processing. Device backing gave by compilers, mediators, and
coordinated advancement situations (IDEs), permits engineers to
reason at a certain level of deliberation, decreasing the incidental
intricacy included in programming improvement (e.g., machineparticular calling traditions and express memory administration).
Space particular dialects (DSLs) further expand expressivity by
confining the extension to a specific application area. They expand
engineer benefit by giving area particular documentation,
examination, check, and enhancement.
Slips in compilers, translators, and IDEs for a dialect can prompt
off base execution of right projects, lapse messages about right
projects, or an absence of mistake messages for wrong projects

 Testing is one of the most important tools for software quality control
and inspires confidence in software
Scripts for automated testing and general-purpose testing tools such
as the xUnit family of frameworks have been successfully applied to
implementations of general-purpose languages and DSLs.
Universally useful testing procedures, as bolstered with xUnit and
testing scripts, require noteworthy interest in base to cover
experiments identified with punctuation, static semantics, and
supervisor administrations, particular for the tried dialect.
In this paper, we exhibit a novel way to deal with dialect definition
testing by presenting the idea of a dialect parametric testing dialect
(LPTL). This dialect gives a reusable, bland premise for definitively
determining dialect definition tests.

 Utilizing a LPTL fundamentally decreases the limit for dialect testing,

which is vital in light of the fact that such an edge is regularly an
explanation behind engineers to do without testing
There are two purposes behind this: To begin with, by giving a reusable framework to dialect test particulars
that encourages test execution, investigation, support, and
comprehension.
Second, by giving full dialect particular IDE support for composing tests.

2. Background: Language Definitions

The advancement of a compiler for a DSL for an area includes
numerous assignments, running from development of a parser to a
semantic analyzer and code generator.
Customarily, a considerable measure of exertion was needed for each
of these undertakings. Parsers, information structures for theoretical
sentence structure trees, traversals, changes, thus on would be coded
by hand for every dialect.
This implied that a noteworthy interest in time and exertion was
needed for the improvement of another dialect

2.1 Language engineering tools

Language workbenches are a new breed of language development
tools that integrate tools for most aspects of language engineering into
a single environment.
Language workbenches make the development of new languages and
their IDEs much more efficient in two ways: providing full IDE support for language development tasks and
integrating the development of the language compiler/interpreter and
its IDE..
Examples of language workbenches include MPS, MontiCore, Xtext,
and our own Spoofax.

Figure 1. Language workbenches can combine language

definition (left) and language use (right)

Language workbenches allow for an agile development model, as they allow

developers to use an IDE and to play with the language while it is still under
development. Figure 1 shows a screenshot illustrating how they can combine
the development of a language with the use of generated editors for that
language.
Once a syntax definition has been developed (at the left), they can generate an
editor with basic editor services such as syntax highlighting and syntax error
marking. From there, language engineers can incrementally and iteratively
implement new language components and editor services.

3. Test Specification Language Design

In this area we portray the configuration of a dialect parametric testing
dialect and show how it can be utilized to test diverse parts of dialect
definitions.

The focal objective set out for outline of a LPTL is to give a low-limit
test particular dialect that structures the premise for a reusable
foundation for testing diverse dialects

 The design principles of this language are as follows:

P1 Provide a language-agnostic framework. The language should provide a generic,
language-agnostic basis that caters for a wide spectrum of different types of tests.
P2 Maintain implementation independence. The language should emphasize blackbox testing , allowing tests to be written early in the design process, and abstracting
over implementation specifics.
P3 Support language-specific instantiation. It should be possible to instantiate the
language for a specific language under test, thereby integrating the two languages and
the tool support provided for the two.
P4 Facilitate series of tests with test fixtures. The language should support test
fixtures to specify series of tests with common boilerplate code such as import
headers.

3.1 A language-agnostic framework (P1)

Key to providing a reusable, language agnostic framework is
providing a generic language that can quote test fragments and can
specify conditions to validate for those test.
We realize this using the following syntax to specify tests:
test description [[
fragment
]] condition*
where description is a string that describes the current test case in
human readable form and fragment is an embedded program or
program fragment in the language under test. The condition* elements
specify the expectations of the test case, and control what test is
performed for the input fragment.

 Figure 2 shows an example test case where we test the mobl language, a
domain-specific language for mobile applications. In this example we
declare a local variable s of type String and assign an integer literal value
to it. This is a negative test case: a value of type String would be
expected here. The conditions clause of this test case indicates that
exactly one error was expected here, which means that the test case
passes.

3.2 Implementation independence via blackbox testing (P2)

Black-box tests test the functionality rather than the internal workings of
a software artifact.
As inputs of a black-box language test we use 1) the quoted fragment of
code, 2) the conditions clause, and 3) selections of code within the
quoted fragment. The first two were illustrated in the example of Figure
2.
Figure 3, which shows a content completion test. The double brackets
inside the quotation indicate a selected part of the program where content
completion would be applied. Selections can indicate identifiers or larger
program fragments for tests of features such as reference resolving,
content completion, and refactorings.

3.3 Language-specific instantiation (P3)

Instantiation of the testing language for a specific language under test
requires that the test suite specifies which language to use for its test cases.
We organize suites into one or more modules (i.e., files), where each
module has a series of test cases and its own configuration. For each
module we use headers that indicate their name, what language to use, and
what start symbol to use:
module test-assignments
language Mobl
start symbol Expression
Once the language under test is specified, the LPTL and the language
under test are composed together, and quoted test fragments are no longer
treated as mere strings but as structured part of test specifications.

3.4 Test fixtures (P4)

common technique in testing frameworks , is to use setup() and
tearDown() methods to create test fixtures.
The contents of the setup block serves as a template for the test cases,
where the [[...]] placeholder is filled in with the contents of each test
block. The placeholder is optional: if none is specified, we assume it
occurs at the end of the setup block.
Setup pieces are basically a simply syntactic, dialect skeptic highlight,
however they are very adaptable. They can be utilized to element out
standard code from individual tests, for example, module and import
presentations. They can likewise be utilized to pronounce sorts,
capacities and qualities utilized as a part of experiments. Much like
with the setup() and tearDown() routines for xUnit, they can likewise
be utilized to perform errands, for example, database introduction for
experiments that execute tried projects

 The table shows the possible condition clauses for syntactic, static
semantic, and dynamic semantics tests, and the patterns that can be used
with some condition clauses

4. Test Specification Interaction Design

Tool support is an important factor for productivity with
programming languages. For the domain of testing in
particular, good tool support is important to lower the
threshold of testing
a combination of four forms of tool support for language
definition testing.
the generic test specification language.
editor services of the language under test in test fragments.
live evaluation of test cases as they are edited.
a batch test runner for testing larger test suites.

4.1 Editor Services for Test Specification

Integrated development environments are a crucial factor in
programmer productivity, these incorporate many different
kinds of editor services , assisting developers in code
understanding and navigation, directing developers to
inconsistent or incomplete areas of code, and even helping
them with editing code by providing automatic indentation,
bracket insertion, and content completion.
o Editor services for the generic testing host language:
These are the meat and potatoes for making
language engineers more productive with testing. Our
implementation provides the full range of syntactic and
semantic editor services for working with the testing

o Editor services for the generic testing language

we use editor services of the language under test to
support them as first-class parts of a test specification. Our
implementation provides services such as syntax
highlighting, syntax error marking, semantic error marking,
and content completion, as shown in the screenshots of
Figure 6 (a) and (b). Note that error markers are only shown
for failing test cases, not for negative test cases where errors
are expected (Figure 6 (c)).

Fig 6 : IDE support for test specification.

4.2 RunningLanguageDenitionTests
Live evaluation of test cases : These test cases evaluates tests in the
background and shows which tests fail through error and warning
markers in the editor. With this feedback, developers can quickly
determine the status of tests in a testing module.
Batch Execution: To support long-running test cases and larger test
suites, a batch test runner is introduced as a test runner is particularly
important as a language project evolves and the number of tests grows
substantially and tests are divided across multiple test modules. The
test runner gives a quick overview of passing and failing tests in
different modules and allows developers to navigate to tests in a
language project.

4.3 Using Integrated Language Definition Testing

using LPTL each new feature can be tested at any stage of the
development process. This makes it possible to develop
languages in a test-driven fashion, following the rhythm
described:
Write a test case.
Watch it fail.
Implement the tested feature.
Watch all tests succeed.
Refactor when necessary and repeat.
This approach facilitates the process for language
engineering by providing a specialized language testing
infrastructure that gives direct feedback at any stage in this
development cycle.

5. Language definition testing by

example
Mobl : Mobl is a statically typed language and compiles to a
combination of HTML, Javascript and CSS. Mobl integrates
sub-languages for user interface design, data modeling and
querying, scripting and webservices into a single language
which is focused on the data modeling language.
In mobl, most les starts with a module header, followed by
a list of entity type denitions, functions, and possibly
statements. A subset of mobls syntax is as follows:

A subset of mobls syntax

Start ::= "module" QId Def*
Def ::= "entity" ID "{" EBD* "}"
| Function
| Stm
EBD ::= ID ":" Type
| Function
Function ::= "function" ID
"(" (FArg ("," FArg)*)? ")"
":" Type "{" Stm* "}"
Stm ::= "var" ID ":" Type "=" Exp ";
| "var" ID "=" Exp ";"
| Exp "=" Exp ";"
| Exp ";"
| "{" Stm* "}"
| "if" "(" Exp ")" Stm ("else" Stm)?
| "foreach" "(" ID "in" Exp ")" Stm
| "foreach" "(" ID ":" Type "in" Exp ")" Stm
Exp ::= STRING
| NUMBER
| "null"
| "this"
| Exp "." ID
| ID "(" (NameExp ("," NameExp)*)? ")"
NameExp ::= ID "=" Exp | Exp
FArg ::= ID ":" Type
Type ::= ID | "Collection" "<" Type ">"
QId ::= ID | QId "::" ID

 Entities are persistent data types that are stored in a database and can
be retrieved using mobls querying API. i.e., tasks::datamodel.
An example of a mobl module that denes a single entity type is as
follows
module tasks::datamodel
entity Task {
name : String
date : DateTime
}

5.1 Syntactic Tests

Syntax tests can be used to test newly added language constructs. They
can include various non-trivial tests such as tests for operator precedence,
reserved keywords, language embeddings, or complex lexical syntax such as
the quotation construct. Two forms of tests, they are black-box test (which
test if a code fragment can be parsed or not) and tree patterns to match
against the abstract syntax produced by the parser for a given fragment.

5.2 Static Semantic checks

With tests we can have better condence in the static checks dened for
a language. For example, we use a setup block to import the
tasks::datamodel module,and to initialize a single Task for testing.
The rst test case is a positive test, checking that the built-in all() accessor
returns a collection of tasks. The other tests are negative tests. For such test
cases, it is generally wise to test for a specic error message. We use regular
expressions such as /type/ to catch specic error messages that are expected.

 Example
language mobl
setup [[
module tasks
import tasks::datamodel
var todo = Task(name="Create task list");
]]
test Entity types have an all() built-in [[
Collection<Task> = Task.all(); ]]
succeeds
test Assigning a property to a Num [[
var name : Num = todo.name;
]] 1

error /type/

test Local variable shadowing [[

function f() {
var a : A;
{
var a : A;
}
}
]] 1 error /already defined/

var all :

5.3 Navigation
Modern IDEs provide editor services for navigation and code understanding, such as reference
resolving and content completion.
rst test case tests variable shadowing, while the second one tests reference resolving for function calls.
For content completion we test completion for normal local variables, and for built-ins such as the
all() accessor.
test Resolve a shadowing variable [[
function getExample() : String {
var [[example]] = "Columbus";
return [[example]];
}
]] resolve #2 to #1
test Resolve a function call [[
function [[loop]](count:Num){
[[loop]](count+1);
}
]] resolve #2 to #1
test Content completion for globals [[
var example2 = [[e]];
]] complete to "example"
test Content completion for queries [[
var example2 = Task.[[a]];
]] complete to "all()"

5.4 Code Generation and Execution

Testing can be used to conrm that exactly the right output is generated for a
particular input, but those tests are often rather fragile: one small change in a
compiler can break a test case even if the program still compiles correctly. It is more
practical to use an external oracle for those tests, such as a compiler or lint-type
checker. Another strategy is to ensure that the program is executable and to simply
run it: execution tests can indirectly serve as tests of generated code correctness. For
execution tests we use the notion of a runner. Similar to a builder, runners are
operations that execute code, through interpretation or by running a generated
program.

5.5 Testing for end-programmers

End-programmers that use a language are generally not interested in testing the
syntax or static semantics of a language. They are, however, interested in the
dynamic semantics; writing unit tests for programs written in the language. An LPTL
can be used as a basis for maintaining and running such tests. End-programmers then
get the same language-specic feedback and tooling for writing tests as metaprogrammers, and can use the same testing language for testing multiple DSLs that
may be employed in a project.

5.6 Freeform Tests

The test specication language is open-ended: if there are
aspects of a language denition that need testing but are not
covered by the xed conditions, free form test expressions can be
used. Freeform expressions can directly interact with the
language implementation to express white-box test cases. For
example, they can test whether an internal function that retrieves
all the ancestors in the inheritance chain of a class works, or they
can test that generate-artifacts correctly writes a .js le to disk.

5.7 Self Application

Aninterestingcapabilityofthetestinglanguageisthatitcan be applied
to itself. it can be applied to any language designed in the
language workbench, including instantiations of the testing
language

Example of self application

As the testing language itself uses the normal double
brackets. For the outer test specication, any selections
or setup placeholders should then also be specied using
triple brackets. The inner test specication is free to use
double brackets.
language Spoofax-Testing
test Testing a mobl test specification [[[
module test-mobl
language mobl test
Testing mobl [[
module erroneous
// ...
]] 1 error
]]] succeeds

6. Implementation
In this section we describe our implementation of an LPTL and the
infrastructure that makes its implementation possible.
We implemented the Spoofax testing language as a language
definition plugin for the Spoofax language workbench.
Most Spoofax language definitions consist of a combination of a
declarative SDF syntax definition and Stratego transformation rules
for the semantic aspects of languages, but for this language we also
wrote parts of the testing infrastructure in Java.

6.1 Infrastructure
The Spoofax language workbench provides an environment for
developing and using language definitions. It provides a number of key
features that are essential for the implementation of an LPTL.
A central language registry :Spoofax is implemented as an extension of the IDE Metatooling Platform (IMP) [7] which provides the notions of languages and
a language registry. It also allows for runtime reflection over the
services they provide and any meta-data that is available for each
language, and can be used to instantiate editor services for them

Dynamic loading of editor services:Spoofax supports dynamic, headless loading of separate language
and editor services of the language under test. This is required for instantiation
of these services in the same program instance (Eclipse environment) but
without opening an actual editor for them
Functional interfaces for editor services :Instantiated editor services have a functional interface. This
decouples them from APIs that control an editor, and allows the LPTL to
inspect editor service results and filter the list of syntactic and semantic error
markers shown for negative test cases.
Support for a customized parsing stage :Most Spoofax plugins use a generated parser from an SDF
definition, but it is also possible to customize the parser used. This allows the
LPTL to dynamically embed a language under test.

6.2 Syntax and Parsing

Language engineers can instantiate the testing language for any Spoofax
language that is loaded in the Eclipse environment, either as an Eclipse
plugin, or as a language project in source form.
Once the developer specifies which language to test, the syntax of the
testing language is instantly specialized by integrating the syntax of the
language under test.
This makes it possible to provide syntactic editor services such as syntax
highlighting, and to parse the file to a single abstract syntax tree that is
used in the implementation of semantic editor services and tests
evaluation.

6.3 Tool Support

The language registry provided by Spoofax and IMP maintains a collection
of all languages supported in the environment, and provides access to factory
classes to instantiate language-specific editor services.
Editor service support in test fragments is provided by delegation to services
of the language under test.
The editor services for the testing language simply detect that they are
invoked inside a test fragment, and then delegate the work to a service of the
language under test.
These produce lists of errors that must be filtered according to the test
expectations (e.g., if an error is expected, the IDE should not add a red
marker for it).

Test evaluation
Tests are evaluated by instantiating the appropriate editor services for the
language under test and applying them to the abstract syntax tree that
corresponds to the test input fragment.
Execution tests often depend on some external executable that runs
outside the IDE and that may even be deployed on another machine.
Our implementation is not specific for a particular runtime system or
compiler backend.
Instead, language engineers can define a custom runner function that
controls how to execute a program in the language

Test evaluation performance : Editor services in Spoofax are cached with instantiation, and run in a
background thread, ensuring low overhead and near-instant
responsiveness for live test evaluation.
Most editor services are very fast, but long-running tests such as
builds or runs are better executed in a non-interactive fashion.
We only run those through the batch test runner and display
information markers in the editor if the test was changed after it last
ran.

7. Discussion and Related Work

Related work on testing of language implementations can be divided
into a number of categories: testing with general purpose tools, testing
with homogeneous and heterogeneous language embeddings, and test
case generation.
Testing with general-purpose tools : The use of these tools introduces a number of challenges when
applied to language definitions.
First, a major issue is that a significant investment in languagespecific testing infrastructure must be made to support tests for
different aspects of languages, ranging from syntax to semantics and
editor services.

 A second issue is that to make sure tests are considered in isolation, each
test case is generally put in a separate file.
Using separate files for test cases introduces boilerplate code such as
import headers.
Another limitation of these general testing tools is that they do not
provide specialized IDE support for writing tests.
Standard IDEs are only effective for writing valid programs and report
spurious errors for negative test cases where errors are expected.
Testing with homogeneous embeddings : Language embedding is a language composition technique where
separate languages are integrated.
The embedding technique applied in this paper is a heterogeneous
embedding.
Homogeneously embedded languages must always target the same host
language.

 This can be a weakness when execution on a different platform is desired

(e.g., JavaScript in the case of mobile development with mobl.
A restriction of the approach used in MPS is that it can only be used test
the dynamic semantics of a language, and only allow for positive test
cases.
Testing with heterogeneous embeddings : This tool formed a precursor to the present work.
Parse-unit would embed quoted program fragments in a test module,
much like in the Spoofax testing language, but only supported syntactic
test cases.
There was also no IDE support for parse-unit, and all quoted program
fragments were treated as strings rather than forming a firstclass part of
the language.

 Other testing tools that support embedding have similar limitations as

parse-unit, supporting only syntactic tests, and lacking IDE support.
Test case generation techniques: These techniques use grammars to generate test programs.
To control the theoretically infinite set of programs that can be generated
using most grammars, they use annotations in grammars, external control
mechanisms, or even imperative generators to constrain this set.
The set of test programs generated with these approaches can be used to
stress-test compiler implementations.
As such, they provide an excellent complementary approach to our test
specifications, possibly catching corner cases that a language engineer
did not think of.

Unit tests for domain-specific languages : As a side-effect of providing a language for testing language
implementations, our test specification language can also be used to test
programs written in that language.
This makes it particularly useful in the domain of testing DSL programs,
where testing tools and frameworks are scarce.
The combined scripting and DSL language can then be used to specify
tests.
Their framework ensures that the mapping between the DSL test line
numbers and the generated JUnit tests is maintained for reporting failing
tests.
They also provide a graphical batch test runner.

Summary
Good approach to language definition testing by
introducing the notion of a language parametric testing
language.
The LPTL provides a zero threshold, domain-specific
testing infrastructure based on a declarative test
specification language and extensive tool support for
writing and executing tests.
Implementation in the form of the Spoofax testing
language shows the practical feasibility of the
approach.
Tests inspire confidence in language implementations,
and can be used to guide an agile, test-driven language
development process.

Our own view

Strengths , Weakness and
Improvements

In this paper the main emphasis was testing of observable

behavior of languages, for example reported errors and name
analysis.There are various other analysis such as type or flow
analysis for them there are no test cases written in the paper.
A better test abstraction mechanism is needed rather then
depending on IDE and compiler for testing so that a multiple test
which can interact with language definition the same way the
compiler and IDE does.
The declarative basis provided by the LPTL can be used to integrate
generally applicable supportive techniques for testing, such as test
case prioritization, coverage metrics, coverage visualization ,
mutation testing , and mutation-based analyses for untested code.
That declarative language test suites should become standard
components of language definitions, just as BNF style grammars
are. Supported by an LPTL, tests are concise , implementationindependent , and require little to no effort to setup.

References
[1] M. Fowler. Language workbenches: The killer-app for domain
specic
languages?
http://martinfowler.
com/articles/languageWorkbench.html, 2005.
[2] L. C. L. Kats and E. Visser. The Spoofax language workbench: rules
for declarative specication of languages and IDEs. In W. R. Cook, S.
Clarke,
and
M.
C.
Rinard,
editors,
ObjectOrientedProgramming,Systems,Languages,andApplications, OOPSLA
2010, pages 444463. ACM, 2010.
[3] B. Beizer. Software testing techniques. Dreamtech Press, 2002.