The Converge programming language

The Converge programming language Technical report TR-05-01, Department of Computer Science, King’s College London Laurence Tratt [email protected] February 26, 2005 1 Contents 1 Introduction 2 Converge basics 2.1 Syntax, scoping and modules . . . . . . . 2.2 Functions . . . . . . . . . . . . . . . . . 2.3 Goal-directed evaluation . . . . . . . . . 2.3.1 While loops . . . . . . . . . . . . 2.4 Data model . . . . . . . . . . . . . . . . 2.4.1 Built-in data types . . . . . . . . 2.5 Comparisons and comparison overloading 2.6 Exceptions . . . . . . . . . . . . . . . . . 2.7 Meta-object protocol . . . . . . . . . . . 2.8 Differences from Python . . . . . . . . . 2.9 Differences from Icon . . . . . . . . . . . 2.10 Implementation . . . . . . . . . . . . . . 2.11 Parsing . . . . . . . . . . . . . . . . . . 2.12 Related work . . . . . . . . . . . . . . . 3 4 4 . . . . . . . . . . . . . . 4 5 6 6 9 9 11 11 12 12 12 13 13 14 16 . . . . . . . . . . . . . . . . . . . 16 16 17 18 19 19 20 20 22 22 22 23 24 24 26 26 28 30 30 32 Implications of Converge’s compile-time meta-programming for other languages and their implementations 4.1 Language design implications . . . . . . . . . . . . . . . . . . . . . 33 34 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Compile-time meta-programming 3.1 Background . . . . . . . . . . . . . . . . . . . . . 3.2 A first example . . . . . . . . . . . . . . . . . . . 3.3 Splicing . . . . . . . . . . . . . . . . . . . . . . . 3.3.1 Permissible splice locations . . . . . . . . 3.4 The quasi-quotes mechanism . . . . . . . . . . . . 3.4.1 Splicing within quasi-quotes . . . . . . . . 3.5 Basic scoping rules in the presence of quasi-quotes 3.6 The CEI interface . . . . . . . . . . . . . . . . . . 3.6.1 ITree functions . . . . . . . . . . . . . . . 3.6.2 Names . . . . . . . . . . . . . . . . . . . 3.7 Lifting values . . . . . . . . . . . . . . . . . . . . 3.8 Dynamic scoping . . . . . . . . . . . . . . . . . . 3.9 Forward references and splicing . . . . . . . . . . 3.10 Compile-time meta-programming in use . . . . . . 3.10.1 Conditional compilation . . . . . . . . . . 3.11 Run-time efficiency . . . . . . . . . . . . . . . . . 3.12 Compile-time meta-programming costs . . . . . . 3.13 Error reporting . . . . . . . . . . . . . . . . . . . 3.14 Related work . . . . . . . . . . . . . . . . . . . . 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.2 4.3 Compiler structure . . . . . . . . . . . . . . . . . . . . . . . . . . . Compiler interface . . . . . . . . . . . . . . . . . . . . . . . . . . . 4.3.1 Abstract syntax trees . . . . . . . . . . . . . . . . . . . . . . 34 36 36 5 Syntax extension for DSL’s 5.1 DSL implementation functions . . . . . . . . . . . . . . . . . . . . . 5.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 38 38 6 Modelling language DSL 6.1 Example of use . . . . . . . . . 6.2 Data model . . . . . . . . . . . 6.3 Pre-parsing and grammar . . . . 6.4 Traversing the parse tree . . . . 6.5 Translating . . . . . . . . . . . 6.5.1 OCL expressions . . . . 6.5.2 Forward references . . . 6.5.3 Model class translation . 6.5.4 Summary of translation . 6.6 Diagrammatic visualization . . . 39 39 41 43 44 45 45 46 46 47 47 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Future work 48 8 Acknowledgments 49 A Converge grammar 50 B The ‘Simple UML’ modelling language translation 53 3 1 Introduction This paper details the Converge programming language, a new dynamically typed imperative programming language capable of compile-time meta-programming, and with an extendable syntax. Although Converge has been designed with the aim of implementing different model transformation approaches as embedded DSL’s in mind, it is also a General Purpose Language (GPL), albeit one with unusually powerful features. The motivation for a new approach to implementing model transformation approaches is simple: existing languages, and their associated tool-chains, lead to long and costly implementation cycles for model transformation approaches. The justification for creating a new language, rather than altering an existing one, is far less obvious — it is reasonable to suggest that, given the vast number of programming languages already in existence, one of them should present itself as a likely candidate for modification. There are two reasons why a new language is necessary to meet the aims of this paper. Firstly, in order to meet its aims, Converge contains a blend of features unique amongst programming languages; some fundamental design choices have been necessary to make these features coalesce, and imposing such choices retrospectively on an existing language would almost certainly lead to untidy results and backwards compatibility issues. Secondly, my personal experience strongly suggests that the complexity of modern languages implementations (when such implementations are available) can make adding new features a significant challenge. In short, I assert that it is easier in the context of model transformations to start with a fresh canvass than to alter an existing language. This paper comes in three main parts. The first part documents the basics of the Converge language itself;. The second part details Converge’s compile-time metaprogramming and syntax extension facilities, including a section detailing suggestions for how some of Converge’s novel features could be added to similar languages. The third part of this paper explains Converge’s syntax extension facility, and documents a user extension which allows simple UML-esque modelling languages to be embedded within Converge. As well as being a practical demonstration of Converge’s features, this facility is used extensively throughout the remainder of the paper. 2 Converge basics This section gives a brief overview of the core Converge features that are relevant to the main subject of this paper. Since most of the basic features of Converge are similar to other similar programming language, this section is intentionally terse. However it should allow readers familiar with a few other programming languages the opportunity to quickly come to grips with the most important areas of Converge, and to determine the areas where it differs from other languages. 4 2.1 Syntax, scoping and modules Converge’s most obvious ancestor is Python [vR03] resulting in an indentation based syntax, a similar range and style of datatypes, and general sense of aesthetics. The most significant difference is that Converge is a slightly more static language: all namespaces (e.g. a modules classes and functions, and all variable references) are determined statically at compile-time whereas even modern Python allows namespaces to be altered at run-time1. Converge’s scoping rules are also different from Python’s and many other languages, and are intentionally very simple. Essentially Converge’s functions are synonymous with both closures and blocks. Converge is lexically scoped, and there is only one type of scope (as opposed to Python’s notion of local and global scopes). Variables do not need to be declared before their use: assigning to a variable anywhere in a block makes that variable local throughout the block (and accessible to inner blocks) unless the variable is declared via the nonlocal keyword to refer to a variable in an outer block. Variable references search in order from the innermost block outwards, ultimately resulting in a compile-time error if a suitable reference is not found. As in Python, fields within a class are not accessible via the default scoping mechanism: they must be referenced via the self variable which is automatically brought into scope in any bound function (functions declared within a class are automatically bound functions). Converge’s justification for this is subtly different than Python’s, which has this feature to aid comprehension; although this is equally true in Converge, without this feature, namespaces would not be statically calculable since an objects slots are not always known at compile-time. Converge programs are split into modules, which contain a series of definitions (imports, functions, classes and variable definitions). Unlike Python, each module is individually compiled into a bytecode file by the Converge compiler convergec and linked by convergel to produce a static bytecode executable which can be run by the Converge VM. If a module is the main module of a program (i.e. passed first to the linker), Converge calls its main function to start execution. The following module shows a caching Fibonacci generating class, and indirectly shows Converge’s scoping rules (the i and fib cache variables are local to the functions they are contained within), printing 8 when run: import Sys class Fib_Cache: func init(): self.cache := [0, 1] func fib(x): i := self.cache.len() while i <= x: self.cache.append(self.cache[i - 2] + self.cache[i - 1]) i += 1 return self.cache[x] func main(): fib_cache := Fib_Cache() Sys.println(fib_cache.fib(6)) 1 Prior to version 2.1, Python’s namespaces were determined almost wholly dynamically; this often lead to subtle bugs, and hampered the utility of nested functions. 5 Compiling and running this fragment looks as follows: $ converge convergec -o fib.cvb fib.cv $ converge convergel -o fib fib.cvb lib/libconverge.cvl $ converge fib 8 As in Python, Converge modules are executed from top to bottom when they are first imported. This is because functions, classes and so on are normal objects within a Converge system that need to be instantiated from the appropriate builtin classes – therefore the order of their creation can be significant e.g. a class must be declared before its use by a subsequent class as a superclass. Note that this only effects references made at the modules top-level – references e.g. inside functions are not restricted thus. 2.2 Functions Converge uses the term function both in its traditional programming sense of a standalone function (or ‘procedure’), and also for functions which reside in classes (often called methods). The reason for this is that ‘normal’ functions and ‘methods’ are not restricted in Converge to only their traditional rôles: ‘normal’ functions can reside in classes and ‘methods’ can reside outside of classes. When it is important to distinguish between the two, Converge has two distinct types: unbound functions (‘normal’ functions) and bound functions (‘methods’). Bound functions expect to have an implicit first argument of the self object2 ; however they can not have arguments applied to it directly. Extracting a bound function from an object creates a function binding which wraps a bound function and a reference to the self object into an object which can then have arguments applied to it. Function bindings can be manually created by instantiating the Func Binding class, which allows bound functions to be used with arbitrary self objects. In normal use, Converge automatically assumes that the keyword func introduces an unbound function if it is used outside class, and a bound function if used inside a class. Using the bound func or unbound func keywords overrides this behaviour. Functions, bound or unbound, can have zero or more parameters; prefixing the final parameter in a function with a * denotes the ‘var args’ parameter. An important feature of functions is their apply slot which applies a list of objects as parameters to the function. This allows argument lists of arbitrary size to be built and applied at run-time. 2.3 Goal-directed evaluation An important, if less obvious, influence to Converge is Icon [GG96a]. Since Icon is likely to be unfamiliar to be most readers, a brief overview of Icon is instructive in understanding why it possesses an unusual, and interesting, feature set. Icon’s chief designer was Ralph Griswold, and is a descendant of the SNOBOL series of programming languages – whose design team Griswold had been a part of – and SNOBOL’s 2 Note that unlike Python, Converge does not force the user to explicitly list self as a function parameter. 6 short-lived successor SL5. SNOBOL4 in particular was specifically designed for the task of string manipulation, but an unfortunate dichotomy between pattern matching and the rest of the language, and the more general problems encountered when trying to use it for more general programming issues ensured that, whilst successful, it never achieved mass acceptance; SL5 suffered from almost the opposite problem by having an over-generalized and unwieldy procedure mechanism. See Griswold and Griswold [GG93] for an insight into the process leading to Icon’s conception. Since programs rarely manipulate strings in isolation, post-SL5 Griswold had as his aim to build a language which whilst being aimed at non-numeric manipulation also was usable as a general programming language. The eventual result was Icon [GG96a, GG96b], a language still in use and being developed to this day. In order to fulfil the goal of practical string manipulation, the premises on which Icon is founded are not only fundamentally different from those normally associated with GPLs, but are also tightly coupled with one another. As Icon, Converge is an expression-based language, with similar notions of expression success and failure. In essence, expressions which succeed produce a value; expressions which fail do not produce a value and percolate the failure to their outer expression. For example the following fragment: func main(): x := 1 < 2 y := 2 < 1 Sys.println(x) Sys.println(y) leads to the following output: 2 Traceback (most recent call last): File "expr.cv", line 5, column 13, in main Unassigned_Var_Exception: Var has not yet been assigned to. This is because when the expression 2 < 1 is evaluated, it fails (since 2 is not less than 1); the failure percolates outwards and prevents the assignment of a value to the variable y. Note that failure does not percolate outwards to arbitrary points: failure can not cross bound expressions. A bound expression thus denotes a ‘stop point’ for backtracking. The most obvious point at which bound expressions occur is when expressions are separated by newlines in an Converge program although bound expressions occur in various other points. For example, each branch of an if expression is bound, which prevents the failure of a branch causing the entire if expression to be re-evaluated. Converge directly inherits Icon’s bound expression rules which largely preserve traditional imperative language evaluation strategies, even in the face of backtracking. Success and failure are the building blocks for goal-directed evaluation, which is essentially a limited form of backtracking suitable for imperative programming languages. Functions which contain the yield statement are generators and can produce more than one return value. The yield statement is an alternative type of function return which effectively freezes the current functions closure and stack, and returns a value to the caller; if backtracking occurs, the function is resumed from its previous point of execution and may return another value. Generators complete by using the 7 return statement. Since the return statement returns the null object if no expression is specified, generators typically use return fail to ensure that the completion of the generator does not cause one final loop of the caller — return’ing the fail object causes a function to percolate failure to its caller immediately. The most frequent use of generators is seemingly mundane, and occurs in the following idiom, which uses the iterate generator on a list to print each list element l on a newline: l := [3, 9, 27] for x := l.iterate(): Sys.println(x) In simple terms, the for construct evaluates its condition expression and after each iteration of the loop backtracks in an attempt to pump the condition for more values. This idiom therefore subsumes the clumsy encoding of iterators found in most OO languages. Generators can be used for much more sophisticated purposes. Consider first the following generator which generates all Fibonacci numbers from 1 to high: func fib(high): a, b := [0, 1] while b < high: yield b a, b := [b, a + b] return fail The for construct exhaustively evaluates its condition (via backtracking) until it can produce no more values. Therefore the following fragment prints all Fibonacci values from 1 to 100000: for f := fib(100000): Sys.println(f) The conjunction operator & conjoins two or more expressions; the failure of any part of the expression causes backtracking to occur. Backtracking resumes the most recent generator which is still capable of producing values, only resuming older generators when more recent ones are exhausted. Thus backtracking in Converge is entirely deterministic because the sequence in which alternatives are tried is explicitly specified by the programmer – this makes the evaluation strategy significantly different than that found in logic languages such as Prolog. If all expressions in a conjunction succeed, the value of the final expression is used as the value of the conjunction. If failure occurs, and there are no generators capable of producing more values to be resumed, then the conjunction itself fails. Combining the for construct with the & operator can lead to terse, expressive examples such as the following which prints all Fibonacci numbers wholly divisible by 3 between 1 and 100000: for Sys.println(f := fib(100000) & f % 3 == 0 & f) A brief explanation of this can be instructive. Firstly f := fib(100000) pumps the fib generator and assigns each value it returns to the variable f. Since it is contained within the first expression of the & operator, when the fib generator completes, 8 its failure causes the f := ... assignment to fail, which causes the entire & operator to fail thus causing the for construct to fail and complete. Secondly f % 3 == 0 checks whether f modulo 3 is equal to 0 or not; if it is not, failure occurs and backtracking occurs back to the fib generator. Since f % 3 == 0, if it succeeds, always evaluates to 0 (== evaluates to its right hand argument on success), the final expression of f produces the value of the variable which Sys.println then prints. Neither Icon or Converge possess standard boolean logic since equivalent functionality is available through other means. The conjunction operator acts as an ‘and’ operator. Although the disjunction operator | is generally used as ‘or’, it is in fact a generator that successively evaluates all its expressions, producing values for those expressions which succeed. Thus in most circumstances the | operator neatly preserves the normal expectation of ‘or’ – that it evaluates expressions in order only until it finds one which succeeds – whilst also providing useful extra functionality. This section has detailed the most important aspects of Converge’s Icon-esque features, but for a more thorough treatment of these features I recommend Icon’s manual [GG96a] — virtually all the material on goal-directed evaluation is trivially transferable from Icon to Converge. Gudeman [Gud92] presents a detailed explanation of goaldirected evaluation in general, with its main focus on Icon, and presents a denotational semantics for Icon’s goal-directed evaluation scheme. Proebsting [Pro97] and Danvy et al. [DGR01] both take subsets of Icon chosen for their relevance to goal-directed evaluation, compiling the fragments into various programming languages (Danvy et. al also specify their Icon subset with a monadic semantics); both papers provide solid further reading on the topic. 2.3.1 While loops Converge also contains a while construct. The difference between the for and while constructs is initially subtle, but is ultimately more pronounced than in most languages. In essence, each time a for loop completes, the construct backtracks to the condition expression and pumps it for a new value. In contrast, a while construct evaluates its expression anew after each iteration. This means that if the condition of a while construct is a generator it can only ever generate a maximum of one value before it is discarded. To emphasise this, the following code endlessly repeats, printing 1 on each iteration: while f := fib(100000): Sys.println(f) 2.4 Data model Converge’s OO features are reminiscent of Smalltalk’s [GR89] everything-is-an-object philosophy, but with a prototyping influence that was inspired by Abadi and Cardelli’s theoretical work [AC96]. The internal data model is derived from ObjVLisp [Coi87]. Classes are provided as a useful, and common, convenience but are not fundamental to the object system in the way they are to most OO languages. The system is bootstrapped with two base classes Object and Class, with the latter being a subclass of 9 Object slots : Dict{String : Object} conforms_to(Class) get_slot(String) : Object get_slots(String) : Dict{String : Object} has_slot(String) id() : Object init(*Object) set_slot(String, Object) to_str() : String instance_of supers * Class name : String fields : Dict{String : Object} is_subclass(Class) new(*Object) : Object Figure 1: Core Converge data model. the former and both being instances of Class itself: this provides a full metaclass ability whilst avoiding the class / metaclass dichotomy found in Smalltalk [BC89, DM95]. The core data model can be seen in figure 1. Note that the slots field in the Object class is conceptual and can only be accessed via the get slot and get slots functions. In the Smalltalk school of OO, objects consist of a slot for each attribute in the class; calling a method on an object looks for a suitable method in the object’s instantiating class (and possibly its superclasses). In contrast Converge, by default, creates objects with a slot for each field in a class, including methods. This therefore moves method overriding in classes to object creation time, rather than the more normal invocation time. This is possible since, as in Python, a functions name is the only factor to be taken into account when overriding. Object creation in Converge thus has a higher overhead than in most OO languages; this is offset by the fact that calling a function in an object is faster (since classes and super-classes do not need to be searched). The reason for this design decision is to ensure that all objects in a Converge system are ‘free objects’ in that they can be individually manipulated without directly affecting other objects, a feature which can prove useful when manipulating and transforming objects. This behaviour also mirrors the real world where, for example, changing a cars design on paper does not change actual cars on the road; it does not however reflect the behaviour of non-prototyping OO languages. For example, in Converge adding (or deleting) a method in a class does not automatically affect objects which are instances of that class, whereas in Python all of the classes instances would appear to grow (or lose) a method. From a practical point of view it is important to note that in normal use most users will be unaware of the difference between Converge’s object creation scheme and its more normal counterparts. Note that this entire area of behaviour can be overridden by using meta-classes and the meta-object protocol (section 2.7). In similar fashion to ObjVLisp, metaclasses are otherwise normal objects which possess a new slot. Class is the default metaclass; individual classes can instantiate 10 a different class via the metaclass keyword. Metaclasses typically subclass Class although this is not a requirement. A simple example of a useful metaclass is the following singleton metaclass [GHJV94] which allows classes to enforce that at most one instance of the class can exist in the system. Noting that exbi (EXtract and BInd) can be viewed as being broadly equivalent to other languages super keyword, the Singleton class is defined and used as follows: class Singleton(Class): func new(): if not self.has_slot("instance"): self.instance := exbi Class.new() return self.instance class M metaclass Singleton: pass Note that the new function in Class automatically calls the init function on the newly created object, passing it all the arguments that were passed to new. 2.4.1 Built-in data types Converge provides a similar set of built-in data types to Python: strings, integers, dictionaries (key / value pairs) and sets. Dictionary keys and set elements should be immutable (though this is not enforced, violating this expectation can lead to unpredictable results), and must define == and hash functions, the latter of which should return an integer representing the object. All built-in types are subclasses of Object, and can be sub-classed by user classes (although the current implementation restricts user classes to sub-classing a maximum of one built-in type). 2.5 Comparisons and comparison overloading Converge defines a largely standard set of binary operators. The lack of standard boolean logic in Converge means that the not operator is slightly unusual and is not classed as a comparison operator. Rather than not taking in a boolean value and returning its negated value, the not operator evaluates its expression and, if it fails, not succeeds and produces the null object. If the expression succeeds, the value produced is discarded and the not operator fails. Objects can control their involvement in comparisons by defining, or overriding, the functions which are called by the various comparison operators. Functions are passed an object for comparison, and should fail if the comparison does not hold, or return the object passed to them if it does. Comparison operators are essentially syntactic sugar for calling a function of the same name in the left hand side object (e.g. the == operator looks up the == slot in an object). Note that although the Converge grammar (section A) bundles the is operator into the comparison op production, it is unlike the other comparison operators in that it tests two objects for equality of their identities, and can not be overridden by user objects. 11 2.6 Exceptions Converge provides exception handling that is largely similar to Python. The raise expression raises an exception, printing a detailed stack-trace, the type of the exception and a message from the exception object itself. All exceptions must be instances of the Exception class in the Exceptions module. The try ... catch construct is used to test and capture exceptions. 2.7 Meta-object protocol Converge implements a simple but powerful Meta-Object Protocol (MOP) [KdRB91], which allows objects to control all behaviour relating to slots. The default MOP is contained within the Object class and comprises the get slot, get slots, has slot and set slot functions. These can be arbitrarily overridden to control which slots the object claims it has (or has not), and what values such slots contain. Note that all accesses go through these functions; if they are overridden in a subclass, the user must exercise caution to call the ‘master’ MOP functions in the Object class to prevent infinite loops. The following example shows a MOP which returns a default value of null for unknown slot names: class M: func get_slot(n): if not self.has_slot(n): return null return exbi Object.get_slot(n) 2.8 Differences from Python Converge deliberately presents a feature set which can be used in a highly similar fashion to Python. Programmers used to Python can easily use Converge in a Python-esque fashion although they will miss out on some of Converge’s more advanced features. The chief differences from Python are that Converge is a more static language, able to make stronger guarantees about namespaces, and that Converge is an expression based language rather than Python’s statement based approach. Converge has a more uniform object system, and less reliance on a battery of globally available builtin functions than Python. One small change from Python to Converge is a generalization of the somewhat confusingly named finally branch which can be attached to Python’s for and while loops. The finally branch is executed if the loop construct terminates naturally (i.e. break is not called). Converge renames the finally branch to exhausted and also allows a broken branch to be added which will be called if a break is encountered. A slightly contrived example of this feature is as follows: high := 10000 for x := fib(high): if x % 9 == 0: break exhausted: Sys.println("No Fibonacci numbers wholly divisible by 9 upto ", high) broken: Sys.println("Fibonacci number ", x, " wholly divisible by 9") 12 2.9 Differences from Icon Converge’s expression system is highly similar to Icon. Provided they can adjust to the Python-esque veneer, Icon programmers will have little difficulty exploiting Converge’s expression system and implementation of goal-directed evaluation. There are however two significant differences in Converge’s functions and generators. Firstly, whereas Icon functions which do not have a return expression at the end of a function have an implicit return fail added, Converge functions instead default to return null (as do Python functions). Icon takes its approach so that generators do not accidentally return an extra object when they should instead fail, and Converge originally took the same approach as Icon. However in practise it is quite common, when developing code, to write incomplete functions — often one part of the code not initially filled in is the functions final return expression. Such functions then cause seemingly bizarre errors since they do not return a value, causing assignments in calling functions to fail and so on (indeed, this happened surprisingly frequently in the early stages of Converge development). Since the proportion of generators to normal functions is small, it seems more sensible to optimise the safety of normal functions at the expense of the safety of generators. As can be seen from section 2.3, generators in Converge generally have return fail as their final action in order to emulate Icon’s behaviour. Secondly, Converge does not propagate generation across a return expression. In Icon, if f is a generator then return f() turns the function containing the return expression into a generator itself which produces all the values that f produces. Converge does not emulate this behaviour, which somewhat arbitrarily turns return into a sort of for construct in certain situations that can only be determined by knowing whether the expression contains a generator. The same behaviour can be obtained in Converge via the following idiom: for yield f() return fail 2.10 Implementation The current Converge implementation consists of a Virtual Machine (VM) written in C, and a compiler written in Converge itself (the current compiler was bootstrapped several generations ago from a much simpler Python version). The VM has a simplistic semi-conservative garbage collector which frees the user from memory management concerns. The VM uses a continuation passing technique at the C level to make the implementation of goal-directed evaluation reasonably simple and transparent from the point of view of extension modules. Its instruction set is largely based on Icon’s, although the VM implementation itself shares more in common with modern VM’s such as Python’s. This paper is not overly concerned with the implementation of the VM and compiler. Interested readers are encouraged to visit http://convergepl.org/ where the VM and compiler can be downloaded and inspected. 13 2.11 Parsing An aspect of Converge and its implementation that is particularly important throughout this paper is its ability to easily parse text. Converge implements a parser toolkit (the Converge Parser Kit or CPK) which contains a parsing algorithm based on that presented by Earley [Ear70]. Earley’s parsing algorithm is interesting since it accepts and parses any Context Free Grammar (CFG) — this means that grammars do not need to be written in a restricted form to suit the parsing algorithm, as is the case with traditional parsing algorithms such as LALR. Practical implementations of Earley parsers have traditionally been scarce, since the flexibility of the algorithm results in slower parsing times than traditional parsing algorithms. The CPK utilises some (though not all) of the additional techniques developed by Aycock and Horspool [AH02] to improve its parsing time, particularly those relating to the ε production. Even though the CPK contains an inefficient implementation of the algorithm, on a modern machine, and even with a complex grammar, it is capable of parsing in the low hundreds of lines per second which is sufficient for the purposes of this paper. The performance of more sophisticated Earley parsers such as Accent [Sch] suggest that the CPK’s performance could be raised by approximately an order of magnitude with relatively little effort. Parsing in Converge is preceded by a tokenization (also known as lexing) phase. The CPK provides no special support for tokenization, since the built-in regular expression library makes the creation of custom tokenizers trivial. Tokenizers are expected to return a list of objects, each of which has slots type, value, src file and src offset. The first two slots represent the type (i.e. ID) and value (i.e. height) of a token and must be strings; the latter two slots record both the file and character offset within the file that a particular token originated in. The tokenizer for Converge itself is somewhat unusual in that it needs to understand about indentation in order that the grammar can be expressed satisfactorily. Essentially each increase in the level of indentation results in a INDENT token being generated; each decrease results in a DEDENT followed by a NEWLINE token. Each newline on the same level of indentation results in a NEWLINE token. The CPK implements an EBNF style system – essentially a BNF system with the addition of the Kleene star. CPK production rules consist of a rule name, and one or more alternatives. Each alternative consists of tokens, references to other rules and groupings. Currently the only form of grouping accepted is the Kleene star. The CPK grammar itself is as follows: hgrammari ::= hrulei* hrulei ::= ‘ID’ hrule alternativei* hrule alternativei ::= ‘::=’ hrule elemi* | ‘::=’ hrule elemi* ‘%PRECEDENCE’ ‘INT’ hrule elemi ::= hatomi | hgroupingi 14 hgroupingi ::= ‘{’ hatomi ‘}*’ hatomi ::= ‘"’ ‘TOKEN’ ‘"’ | ‘ID’ Since Earley grammars can express any CFG, grammars can be ambiguous — that is, given inputs can satisfy the grammar in more than one way. In order to disambiguate between alternatives when building the parse tree, the CPK allows grammar rules to have a precedence attached to them; if more than one rule has been used to parse a given group of tokens, the rule with the highest precedence is used 3 . In order to use the CPK, the user must provide it with a grammar, the name of a start rule within the grammar, and a sequence of tokens. The result of a CPK parse is an automatically constructed parse tree, which is represented as a nested Converge list of the form [production name, token or list1 , ..., token or listn ]. The following program fragment shows a CPK grammar for a simple calculator: GRAMMAR := """ S ::= E E ::= E "+" E %precedence 10 ::= E "*" E %precedence 30 ::= "(" E ")" ::= N "INT" %precedence 10 N ::= "-" ::= """ Assuming the existence of a suitable tokenize function, an example program which uses this grammar to parse input is as follows: import CPK.Grammar, CPK.Parser func calc_parse(input): grammar := Grammar.Grammar(GRAMMAR, "S") tokens := tokenize(input) parser := Parser.Parser(grammar) tree := parser.parse(tokens) Sys.println(tree) The parse tree is printed out as: ["S", ["E", ["E", ["N"], <INT 5>], <+>, ["E", ["E", ["N"], <INT 2>], <*>, ["E", ["N"], <INT 3>]]]] This is somewhat easier to visualize when using the parse tree function in a Parser instance to format the list as a tree: S-> E -> E -> N -> INT <5> + E -> E -> 3 Note that there is another, much rarer, type of ambiguity involving alternatives which contain different number of tokens. These are currently always resolved in favour of the alternative containing the least number of tokens, no matter its precedence. This generally gives the expected behaviour, but can cause problems in some rare cases. This limitation is purely due to a naı̈ve implementation. 15 N -> INT <2> * E -> N -> INT <3> The full Converge grammar can be seen in appendix A. 2.12 Related work This section has made several comparisons between Converge, and Icon and Python in particular. These are not repeated in this subsection. The Unicon project [JMPP03] is in the reasonably advanced stages of extending Icon with object orientated features. It differs significantly from Converge in maintaining virtually 100% compatibility with Icon. Unicon’s extensions to Icon, effectively being a bolt-on to the original, mean the resulting language contains more visible seams than does Converge. Godiva Godiva [Jef02], which aims to be a ‘very high level dialect of Java’ incorporating goal-directed evaluation. In reality, Godiva’s claim to be a dialect of Java is slightly tenuous: whilst it shares some syntax, the semantics are substantially different. Neither Unicon nor Godiva are meta-circular, and both are less dynamic languages than Converge. 3 Compile-time meta-programming 3.1 Background Compile-time meta-programming allows the user of a programming language a mechanism to interact with the compiler to allow the construction of arbitrary program fragments by user code. As Steele argues, ‘a main goal in designing a language should be to plan for growth’ [Ste99] – compile-time meta-programming is a powerful mechanism for allowing a language to be grown in ways limited only by a users imagination. Compile-time meta-programming allows users to e.g. add new features to a language [SeABP99] or apply application specific optimizations [SCK03]. The LISP family of languages, such as Scheme [KCR98], have long had powerful macro facilities allowing program fragments to be built up at compile-time. Such macro schemes suffered for many years from the problem of variable capture; fortunately modern implementations of hygienic macros [DHB92] allow macros to be used safely. LISP and Scheme programs make frequent use of macros, which are an integral and vital feature of the language. Compile-time meta-programming is, at first glance, just a new name for an old concept – macros. However, LISP-esque macros are but one way of realizing compile-time meta-programming. Brabrand and Schwartzbach differentiate between two main categories of macros [BS00]: those which operate at the syntactic level and those which operate at the lexing level. Scheme’s macro system works at the syntactic level: it operates on Abstract 16 Syntax Trees (AST’s), which structure a programs representation in a way that facilitates making sophisticated decisions based on a nodes context within the tree. Macro systems operating at the lexing level are inherently less powerful, since they essentially operate on a text string, and have little to no sense of context. Despite this, of the relatively few mainstream programming languages which have macro systems, by far the most widely used is the C preprocessor (CPP), a lexing system which is well-known for causing bizarre programming headaches due to unexpected side effects of its use (see e.g. [CMA93, Baw99, EBN02]). Despite the power of syntactic macro systems, and the wide-spread usage of the CPP, relatively few programming languages other than LISP and C explicitly incorporate such systems (of course, a lexing system such as the CPP can be used with other text files that share the same lexing rules). One of the reasons for the lack of macro systems in programming languages is that whilst lexing systems are recognised as being inadequate, modern languages do not share LISP’s syntactic minimalism. This creates a significant barrier to creating a system which matches LISP’s power and seamless integration with the host language [BP99]. Relatively recently languages such as the multi-staged MetaML [Tah99] and Template Haskell (TH) [SJ02] have shown that statically typed functional languages can house powerful compile-time meta-programming facilities where the run-time and compiletime languages are one and the same. Whereas lexing macro systems typically introduce an entirely new language to proceedings, and LISP macro systems need the compiler to recognise that macro definitions are different from normal functions, languages such as TH move the macro burden from the point of definition to the macro call point. In so doing, macros suddenly become as any other function within the host language, making this form of compile-time meta-programming in some way distinct from more traditional macro systems. Importantly these languages also provide powerful, but usable, ways of coping with the syntactic richness of modern languages. Most of the languages which fall into this new category of compile-time metaprogramming languages are statically typed functional languages. In this section I detail an extension to the core Converge language which adds compile-time metaprogramming facilities similar to TH. Since this is the first time that facilities of this nature have been added to a dynamically-typed OO language such as Converge, section 4 details the implications of adding such a feature to similar languages. 3.2 A first example The following program is a simple example of compile-time meta-programming, trivially adopted from it’s TH cousin in [CJOT04]. expand power recursively creates an expression that multiplies n x times; mk power takes a parameter n and creates a function that takes a single argument x and calculates xn ; power3 is a specific power function which calculates n3 : func expand_power(n, x): if n == 0: return [| 1 |] else: 17 return [| $<<x>> * $<<expand_power(n - 1, x)>> |] func mk_power(n): return [| func (x): return $<<expand_power(n, [| x |])>> |] power3 := $<<mk_power(3)>> The user interface to compile-time meta-programming is inherited fairly directly from TH: quasi-quote expressions [| ... |] build abstract syntax trees - ITree’s in Converge’s terminology - that represent the program code contained within them, and the splice annotation $<<...>> evaluates its expression at compile-time (and before VM instruction generation), replacing the splice annotation itself with the ITree resulting from its evaluation. When the above example has been compiled into VM instructions, power3 essentially looks as follows: power3 := func (x): return x * x * x * 1 By using the quasi-quotes and splicing mechanisms, we have been able to synthesise at compile-time a function which can efficiently calculate powers without resorting to recursion, or even iteration. Note how apart from the quasi-quotes and splicing mechanisms no extra features have been added to the base language – unlike LISP style languages, all parts of a Converge program are first-class elements regardless of whether they are executed at compile-time or run-time. This terse explanation hides much of the necessary detail which can allow readers who are unfamiliar with similar systems to make sense of this synthesis. In the following sections, I explore the interface to compile-time meta-programming in more detail, building up the picture step by step. 3.3 Splicing The key part of the ‘powers’ program is the splice annotation in the line power3 := $<<mk power(3)>>. The top-level splice tells the compiler to evaluate the expression between the chevrons at compile-time, and to include the result of that evaluation in the module for ultimate bytecode generation. In order to perform this evaluation, the compiler creates a temporary or ‘dummy’ module which contains all definitions up to, but excluding, the definition the splice annotation is a part of; to this temporary module a new splice function (conventionally called $$splice$$) is added which contains a single expression return splice expr. This temporary module is compiled to bytecode and injected into the running VM, whereupon the splice function is called. Thus the splice function ‘sees’ all the definitions prior to it in the module, and can call them freely – there are no other limits on the splice expression. The splice function must return a valid ITree which the compiler uses in place of the splice annotation. Evaluating a splice expression leads to a new ‘stage’ in the compiler being executed. Converge’s rules about which references can cross the staging boundary are simple: only references to top-level module definitions can be carried across the staging boundary (see section 3.5). For example the following code is invalid since the 18 variable x will only have a value at run-time, and hence is unavailable to the splice expression which is evaluated at compile-time: func f(x): $<<g(x)>> Although the implementation of splicing in Converge is more flexible than in TH – where splice expressions can only refer to definitions in imported modules – it raises a new issue regarding forward references. This is tackled in section 3.9. Note that splice annotations within a file are executed strictly in order from top to bottom, and that splice annotations can not contain splice annotations. 3.3.1 Permissible splice locations Converge is more flexible than TH in where it allows splice annotations. A representative sample of permissible locations is: Top-level definitions. Splice annotations in place of top-level definitions must return an ITree, or a list of ITree’s, each of which must be an assignment. Function names. Splice annotations in place of function names must return a Name (see section 3.6.2). Expressions. Splice annotations as expressions can return any normal ITree. A simple example is $<<x>> + 2. We saw another example in the ‘powers’ program with power3 := $<<mk power(3)>>. Within a block body. Splice annotations in block bodies (e.g. a functions body) accept either a single ITree, or a list of ITree’s. Lists of ITree’s will be spliced in as if they were expressions separated by newlines. A contrived example that shows the last three of these splice locations (in order) in one piece of code is as follows: func $<<create_a_name()>>(): x := $<<f()>> + g() $<<list_of_exprs()>> At compile-time, this will result in a function named by the result of create a name and containing 1 or more expressions, depending on the number of expressions returned in the list by list of exprs. Note that the splice expressions must return a valid ITree for the location of a splice annotation. For example, attempting to splice in a sequence of expressions into an expression splice such as $<<x>> + 2 results in a compile-time error. 3.4 The quasi-quotes mechanism In the previous section we saw that splice annotations are replaced by ITree’s. In many systems the only way to create ITree’s is to use a verbose and tedious interface of ITree creating functions which results in a ‘style of code [which] plagues meta-programming systems’ [WC93]. LISP’s quasi-quote mechanism allows programmers to build up 19 LISP S-expressions (which, for our purposes, are analogous to be ITree’s) by writing normal code prepended by the backquote ‘ notation; the resulting S-expression can be easily manipulated by a LISP program. Unfortunately LISP’s syntactic minimalism is unrepresentative of modern languages, whose rich syntaxes are not as easily represented and manipulated. MetaML and, later TH, introduce a quasi-quotes mechanism suited to syntactically rich languages. Converge inherits TH’s Oxford quotes notation [| ...|] notation to represent a quasi-quoted piece of code. Essentially a quasi-quoted expression evaluates to the ITree which represents the expression inside it. For example, whilst the raw Converge expression 4 + 2 prints 6 when evaluated, [| 4 + 2 |] evaluates to an ITree which prints out as 4 + 2. Thus the quasi-quote mechanism constructs an ITree directly from the users input - the exact nature of the ITree is of immaterial to the casual ITree user, who need not know that the resulting ITree is structured along the lines of add(int(4), int(2)). To match the fact that splice annotations in blocks can accept sequences of expressions to splice in, the quasi-quotes mechanism allows multiple expressions to be expressed within it, split over newlines. The result of evaluating such an expression is, unsurprisingly, a list of ITree’s. Note that as in TH, Converge’s splicing and quasi-quote mechanisms cancel each other out: $<<[| x |]>> is equivalent to x (though not necessarily vice versa). 3.4.1 Splicing within quasi-quotes In the ‘powers’ program, we saw the splice annotation being used within quasi-quotes. The explanation of splicing in section 3.3 would seem to suggest that the splice inside the quasi-quoted expression in the expand power function should lead to a staging error since it refers to variables n and x which were defined outside of the splice annotation. In fact, splices within quasi-quotes work rather differently to splices outside quasi-quotes: most significantly the splice expression itself is not evaluated at compile-time. Instead the splice expression is essentially copied as-is into the code that the quasi-quotes transforms to. For example, the quasi-quoted expression [| $<<x>> + 2 |] leads to an ITree along the lines of add(x, int(2)) – the variable x in this case would need to contain a valid ITree. As this example shows, since splice annotations within quasi-quotes are executed at run-time they can access variables without staging concerns. This feature completes the cancelling out relationship between splicing and quasiquoting: [| $<<x>> |] is equivalent to x (though not necessarily vice versa). 3.5 Basic scoping rules in the presence of quasi-quotes The quasi-quote mechanism can be used to surround any Converge expression to allow the easy construction of ITree’s. Quasi-quoting an expression also has another important feature: it fully respects lexical scoping. Take the following contrived example of module A: 20 func x(): return 4 func y(): return [| x() * 2 |] and module B: import A, Sys func x(): return 2 func main(): Sys.println($<<A.y()>>) The quasi-quotes mechanisms ensures that since the reference to x in the quasi-quoted expression in A.y refers lexically to A.x, that running module B prints out 8. This example shows one of the reasons why Converge needs to be able to statically determine namespaces: since the reference of x in A.y is lexically resolved to the function A.x, the quasi-quotes mechanism can replace the simple reference with an original name 4 that always evaluates to the slot x within the specific module A wherever it is spliced into, even if A is not in scope (or a different A is in scope) in the splice location. Some other aspects of scoping and quasi-quoting require a more subtle approach. Consider the following (again contrived) example: func f(): return [| x := 4 |] func g(): x := 10 $<<f()>> y := x What might one expect the value of y in function g to be after the value of x is assigned to it? A naı̈ve splicing of f() into g would mean that the x within [| x := 4 |] would be captured by the x already in g – y would end with the value 4. If this was the case, using the quasi-quote mechanism could potentially cause all sorts of unexpected interactions and problems. This problem of variable capture is well known in the LISP community, and hampered LISP macro implementations for many years until the concept of hygienic macros was invented [KFFD86]. A new subtlety is now uncovered: not only is Converge able to statically determine namespaces, but variable names can be α-renamed without affecting the programs semantics. This is a significant deviation from the Python heritage. The quasi-quotes mechanism determines all bound variables in a quasi-quoted expression, and preemptively α-renames each bound variable to a guaranteed unique name that the user can not specify; all references to the variable are updated similarly. Thus the x within [| x := 4 |] will not cause variable capture to occur, and the variable y in function g will be set to 10. There is one potential catch: top-level definitions (all of which are assignments to a variable, although syntactic sugar generally obscures this fact) can not be α-renamed without affecting the programs semantics. This is because Converge’s dynamic typing means that referencing a slot within a module can not in all cases be statically checked at runtime. Thus renaming top-level definitions could lead to run-time ‘slot missing’ exceptions being raised. Although the current compiler does not catch this case, since 4 This terminology is borrowed from TH, but with a much different implementation. 21 the user is unlikely to have cause to quasi-quote top-level definitions, barring it should be of little practical consequence. Whilst the above rules explain the most important of Converge’s scoping rules in the presence of quasi-quotes, upcoming sections add extra detail to the basic scoping rules explained in this section. 3.6 The CEI interface At various points when compile-time meta-programming, one needs to interact with the Converge compiler. The Converge compiler is entirely contained within a package called Compiler which is available to every Converge program. The CEI module within the Compiler package is the officially sanctioned interface to the Compiler, and can be imported with import Compiler.CEI. 3.6.1 ITree functions Although the quasi-quotes mechanism allows the easy, and safe, creation of many required ITree’s, there are certain legal ITree’s which it can not express. Most such cases come under the heading of ‘create an arbitrary number of X’ e.g. a function with an arbitrary number of parameters, or an if expression with an arbitrary number of elif clauses. In such cases the CEI interface presents a more traditional meta-programming interface to the user that allows ITree’s that are not expressible via quasi-quotes to be built. The downside to this approach is that recourse to the manual is virtually guaranteed: the user needs to know the name of the ITree element(s) required (each element has a corresponding function with a lower case name and a prepended ‘i’ in the CEI interface e.g. ivar), what the functions requirements are etc. Fortunately this interface needs to be used relatively infrequently; all uses of it are explained explicitly in this paper. 3.6.2 Names Section 3.3 showed that the Converge compiler sometimes uses names for variables that the user can not specify using concrete syntax. The same technique is used by the quasi-quote mechanism to α-rename variables to ensure that variable capture does not occur. However one of the by-products of the arbitrary ITree creating interface provided by the CEI interface is that the user is not constrained by Converge’s concrete syntax; potentially they could create variable names which would clash with the ‘safe’ names used by the compiler. To ensure this does not occur, the CEI interface contains several functions – similar to those in recent versions of TH – related to names which the user is forced to use; these functions guarantee that there can be no inadvertent clashes between names used by the compiler and by the user. In order to do this, the CEI interface deals in terms of instances of the CEI.Name class. In order to create a variable, a slot reference etc, the user must pass an instance of this class to the relevant function in the CEI interface. New names can be created 22 by one of two functions. The name(x) function validates x, raising an exception if it is invalid, and returning a Name otherwise. The fresh name function guarantees to create a unique Name each time it is called (this is the interface used by the quasiquotes mechanism). This allows e.g. variable names to be created safely with the idiom var := CEI.ivar(CEI.name("var name")). fresh name takes an optional argument x which, if present, is incorporated into the generated name whilst still guaranteeing the uniqueness of the resulting name; this feature aids debugging by allowing the user to trace the origins of a fresh name. Note that the name interface opens the door for dynamic scoping (see section 3.8). 3.7 Lifting values When meta-programming, one often needs to take a normal Converge value (e.g. a string) and obtain its ITree equivalent: this is known as lifting a value. Consider a debugging function log which prints out the debug string passed to it; this function is called at compile-time so that if the global DEBUG BUILD variable is set to fail there is no run-time penalty for using its facility. The log function is thus a safe means of performing what is often termed ‘conditional compilation’. Noting that pass is the Converge no-op, a first attempt at such a function is as follows: func log(msg): if DEBUG_BUILD: return [| Sys.println(msg) |] else: return [| pass |] This function fails to compile: the reference to the msg variable causes the Converge compiler to raise the error Var ‘msg’ is not in scope when in quasi-quotes (consider using $<<CEI.lift(msg)>>. Rewriting the offending piece of code to the following gives the correct solution: return [| Sys.println($<<CEI.lift(x)>>) |] What has happened here is that the string value of x is transformed by the lift function into its abstract syntax equivalent. Constants are automatically lifted by the quasiquotes mechanism: the two expressions [| $<<CEI.lift("str")>> |] and [| "str" |] are therefore equivalent. Converge’s refusal to lift the raw reference to msg in the original definition of log is a significant difference from TH, whose scoping rules would have caused msg to be lifted without an explicit call to CEI.lift. To explain this difference, assume the log function is rewritten to include the following fragment: return [| msg := "Debug: " + $<<CEI.lift(msg)>> Sys.println(msg) |] In a sense, the quasi-quotes mechanism can be considered to introduce its own block: the assignment to the msg variable forces it to be local to the quasi-quote block. This needs to be the case since the alternative behaviour is nonsensical: if the assignment referenced to the msg variable outside the quasi-quotes then what would the effect 23 of splicing in the quasi-quoted expression to a different context be? The implication of this is that referencing a variable within quasi-quotes would have a significantly different meaning if the variable had been assigned to within the quasi-quotes or outside it. Whilst it is easy for the Converge compiler writer to determine that a given variable was defined outside the quasi-quotes and should be automatically lifted in (or vice versa), from a user perspective the behaviour can be unnecessarily confusing. In fact Converge’s quasi-quote mechanism originally did automatically lift variable references when possible, but this feature proved confusing in practise. To avoid this, Converge forces variables defined outside of quasi-quotes to be explicitly lifted into it. This also maintains a simple symmetry with Converge’s main scoping rules: assigning to a variable in a block makes it local to that block. 3.8 Dynamic scoping Sometimes the quasi-quote mechanisms automatic α-renaming of variables is not what is needed. For example consider a function swap(x, y) which should swap the values of the two variables passed as strings in its parameters. In such a case, we want the result of the splice to capture the variables in the spliced environment. Because the quasi-quotes mechanism only renames variables which it can determine statically at compile time, any variables created via the idiom CEI.ivar(CEI.name(x)) and spliced into the quasi-quotes will not be renamed. The following succinct definition of swap takes advantage of this fact: func swap(x, y): x_var := CEI.ivar(CEI.name(x)) y_var := CEI.ivar(CEI.name(y)) return [| temp := $<<x_var>> $<<x_var>> := $<<y_var>> $<<y_var>> := temp |] Note that the variable temp within the quasi-quotes will be α-renamed and thus will be effectively invisible to the code that it is spliced into, but that the two variables referred to by x and y will be scoped by their splice location. This function can be used thus: a := 10 b := 20 $<<swap("a", "b")>> Dynamic scoping also tends to be useful when a quasi-quoted function is created piecemeal with many separate quasi-quote expressions. In such a case, variable references can only be resolved successfully when all the resulting ITree’s are spliced together since references to the functions parameters and so on will not be determined until that point. Since it is highly tedious to continually write CEI.ivar(CEI.name("foo")), Converge provides the special syntax &foo which is equivalent. 3.9 Forward references and splicing In section 3.3 we saw that when a splice annotation outside quasi-quotes is encountered, a temporary module is created which contains all the definitions up to, but ex24 cluding, the definition holding the splice annotation. This is a very useful feature since compile-time functions used only in one module can be kept in that module. However this introduces a real problem involving forward references. A forward reference is defined to be a reference to a definition within a module, where the reference occurs at an earlier point in the source file than the definition. If a splice annotation is encountered and compiles a subset of the module, then some definitions involved in forward references may not be included: thus the temporary module will fail to compile, leading to the entire module not compiling. Worse still, the user is likely to be presented with a highly confusing error telling them that a particular reference is undefined when, as far as they are concerned, the definition is staring at them within their text editor! Consider the following contrived example: func f1(): return [| 7 |] func f2(): x := f4() func f3(): return $<<f1()>> func f4(): pass If f2 is included in the temporary module created when evaluating the splice annotation in f3, then the forward reference to f4 will be unresolvable. The solution taken by Converge ensures that, by including only a minimal subset of definitions in the temporary module, most forward references do not raise a compiletime error. We saw in section 3.5 that the quasi-quotes mechanism uses Converge’s statically determined namespaces to calculate bound variables. That same property is now used to determine an expressions free variables. When a splice annotation is encountered, the Converge compiler does not immediately create a temporary module. First it calculates the splice expressions free variables; any previously encountered definition which has a name in the set of free variables is added to a set of definitions to include. These definitions themselves then have their free variables calculated, and again any previously encountered definition which has a name in the set of free variables is added to the set of definitions to include. This last step is repeated until an iteration adds no new definitions to the set. At this point, Converge then goes back in order over all previously encountered definitions, and if the definition is in the list of definitions to include, it is added to the temporary module. Recall that the order of definitions in a Converge file can be significant (see section 2.4): this last stage ensures that definitions are not reordered in the temporary module. Note also that free variables which genuinely do not refer to any definitions (i.e. a mistake on the part of the programmer) will pass through this scheme unmolested and will raise an appropriate error when the temporary module is compiled. Using this method, the temporary module that is created and evaluated for the example looks as follows: func f1(): return [| 7 |] func $$splice$$(): return f1() There are thus no unresolvable forward references in this example. 25 There is a secondary, but significant, advantage to this method: since it reduces the number of definitions in temporary modules it can lead to an appreciable saving in compile time, especially in files containing multiple splice annotations. 3.10 Compile-time meta-programming in use In this paper thus far we have seen several uses of compile-time meta-programming. There are many potential uses for this feature, many of which are too involved to detail in the available space. For example, one of the most exciting uses of the feature has been in conjunction with Converge’s extendable syntax feature (see section 5), allowing powerful DSL’s to be expressed in an arbitrary concrete syntax. One can see similar work involving DSL’s in e.g. [SCK03, CJOT04]. In this section I show two seemingly mundane uses of compile-time meta-programming: conditional compilation and compile-time optimization. Although mundane in some senses, both examples open up potential avenues not currently available to other dynamically typed OO languages. 3.10.1 Conditional compilation Whereas languages such as Java attempt to insulate their users from the underlying platform an application is running on, languages such as Python and Ruby allow the user access to many of the lower-level features the platform provides. Many applications rely on such low-level features being available in some fashion. However for the developer who has to provide access to such features a significant problem arises: how does one sensibly provide access to such features when they are available, and to remove that access when they are unavailable? The log function on page 23 was a small example of conditional compilation. Let us consider a simple but realistic example that is more interesting from an OO perspective. The POSIX fcntl (File CoNTrol) feature provides low-level control of file descriptors, for example allowing file reads and writes to be set to be non-blocking; it is generally only available on UNIX-like platforms. Assume that we wish to provide some access to the fcntl feature via a method within file objects; this method will need to call the raw function within the provided fcntl module iff that module is available on the current platform. In Python for example, there are two chief ways of doing this. The first mechanism is for a File class to defer checking for the existence of the fcntl module until the fcntl method is called, raising an exception if the feature is not detected in the underlying platform. Callers who wish to avoid use of the fcntl method on platforms lacking this feature must use catch the appropriate exception. This rather heavy handed solution goes against the spirit of duck typing [TH00], a practise prevalent in languages such as Ruby and Python. In duck typing, one essentially checks for the presence of a method(s) which appear to satisfy a particular API without worrying about the type of the object in question. Whilst this is perhaps unappealing from a theoretical point of view, this approach is common in practise due to the low-cost flexibility it leads to. 26 To ensure that duck typing is possible in our fcntl example, we are forced to use exception handling and the dynamic selection of an appropriate sub-class: try: import fcntl _HAVE_FCNTL = True except exceptions.ImportError: _HAVE_FCNTL = False class Core_File: # ... if _HAVE_FCNTL: class File(Core_File): def fcntl(op, arg): return fcntl.fcntl(self.fileno(), op, arg) else: class File(Core_File): pass Whilst this allows for duck typing, this idiom is far from elegant. The splitting of the File class into a core component and sub-classes to cope with the presence of the fcntl functionality is somewhat distasteful. This example is also far from scalable: if one wishes to use the same approach for more features in the same class then the resultant code is likely to be highly fragile and complex. Although it appears that the above idiom can be encoded largely ‘as is’ in Converge, we immediately hit a problem due to the fact that module imports are statically determined. Thus a direct Converge analogue would compile correctly only on platforms with a fcntl module. However by using compile-time meta-programming one can create an equivalent which functions correctly on all platforms and which cuts out the ugly dynamic sub-class selection. The core feature here is that class fields are permissible splice locations (see section 3.3.1). A splice which returns an ITree that is a function will have that function incorporated into the class; if the splice returns pass as an ITree then the class is unaffected. So at compile-time we first detect for the presence of a fcntl module (the VM.loaded module names function returns a list containing the names of all loaded modules); if it is detected, we splice in an appropriate fcntl method otherwise we splice in the no-op. This example make use of two hitherto unencountered features. Firstly, using an if construct as an expression requires a different syntax (to work around parsing limitations associated with indentation based grammars); the construct evaluates to the value of the final expression in whichever branch is taken, failing if no branch is taken. Secondly the modified Oxford quotes [d| ...|] – declaration quasi-quotes – act like normal quasi-quotes except they do not α-rename variables; declaration quotes are typically most useful at the top-level of a module. The Converge example is as follows: $<<if VM.loaded_module_names().contains("FCntl") { [d| import FCntl _HAVE_FCNTL := 1 |] } else { [d| _HAVE_FCNTL := 0 |] }>> 27 class File: $<<if _HAVE_FCNTL { [| func fcntl(op, arg): return FCntl.fcntl(self.fileno(), op, arg) |] } else { [| pass |] }>> Although this example is simplistic in many ways, it shows that compile-time metaprogramming can provide a conceptually neater solution than any purely run-time alternative since it allows related code fragments to be kept together. It also provides a potential solution to related problems. For example portability related code in dynamically typed OO languages often consists of many if statements which perform different actions depending on a condition which relates to querying the platform in use. Such code can become a performance bottleneck if called frequently within a program. The use of compile-time meta-programming can lead to a zero-cost run-time overhead. Perhaps significantly, the ability to tune a program at compile-time for portability purposes is the largest single use of the C preprocessor [EBN02] – compile-time meta-programming of the sort found in Converge not only opens similar doors for dynamically typed OO languages, but allows the process to occur in a far safer, more consistent and more powerful environment than the C preprocessor. 3.11 Run-time efficiency In this section I present the Converge equivalent of the TH compile-time printf function given in [SJ02]. Such a function takes a format string such as "%s has %d %s" and returns a quasi-quoted function which takes an argument per ‘%’ specifier and intermingles that argument with the main text string. For our purposes, we deal with decimal numbers %d and strings %s. The motivation for a TH printf is that such a function is not expressible in base Haskell. Although Converge functions can take a variable number of arguments (as Python, but unlike Haskell), having a compile-time version still has two benefits over its run-time version: any errors in the format string are caught at compile-time; an efficiency boost. This example assumes the existence of a function split format which given a string such as "%s has %d %s" returns a list of the form [PRINTF STRING, " has ", PRINTF INT, " ", PRINTF STRING] where PRINTF STRING and PRINTF INT are constants. First we define the main printf function which creates the appropriate number of parameters for the format string (of the form p0, p1 etc.). Parameters must be created by the CEI interface. An iparam has two components: a variable, and a default value (the latter can be set to null to signify the parameter is mandatory and has no default value). printf then returns an anonymous quasi-quoted function which contains the parameters, and a spliced-in expression returned by printf expr: 28 func printf(format): split := split_format(format) params := [] i := 0 for part := split.iterate(): if part == PRINTF_INT | part == PRINTF_STRING: params.append(CEI.iparam(CEI.ivar(CEI.name("p" + i.to_str())), null)) i += 1 return [| func ($<<params>>): Sys.println($<<printf_expr(split, 0)>>) |] printf expr is a recursive function which takes two parameters: a list representing the parts of the format string yet to be processed; an integer which signifies which parameter of the quasi-quoted function has been reached. func printf_expr(split, param_i): if split.len() == 0: return [| "" |] param := CEI.ivar(CEI.name("p" + param_i.to_str())) if split[0].conforms_to(String): return [| $<<CEI.lift(split[0])>> + $<<printf_expr(split[1 : ], param_i)>> |] elif split[0] == PRINTF_INT: return [| $<<param>>.to_str() + $<<printf_expr(split[1 : ], param_i + 1)>> |] elif split[0] == PRINTF_STRING: return [| $<<param>> + $<<printf_expr(split[1 : ], param_i + 1)>> |] Essentially, printf expr recursively calls itself, each time removing the first element from the format string list, and incrementing the param i variable iff a parameter has been processed. This latter condition is invoked when a string or integer ‘%’ specifier is encountered; raw text in the input is included as is, and as it does not involve any of the functions parameters, does not increment param i. When the format string list is empty, the recursion starts to unwind. When the result of printf expr is spliced into the quasi-quoted function, the dynamically scoped references to parameter names in printf expr become bound to the quasi-quoted functions’ parameters. As an example of calling this function, $<<printf("%s has %d %s")>> generates the following function: func (p0, p1, p2): Sys.println(p0 + " has " + p1.to_str() + " " + p2 + "") so that evaluating the following: $<<printf("%s has %d %s")>>("England", 39, "traditional counties") results in England has 39 traditional counties being printed to screen. This definition of printf is simplistic and lacks error reporting, partly because it is intended to be written in a similar spirit to its TH equivalent. Converge comes with a more complete compile-time printf function as an example, which uses an iterative solution with more compile-time and run-time error-checking. Simple benchmarking of the latter function reveals that it runs nearly an order of magnitude faster than its run-time equivalent5 – a potentially significant gain when a tight loop repeatedly calls printf. 5 This large differential is in part due to the fact that the current Converge VM imposes a relatively high overhead on function application. 29 3.12 Compile-time meta-programming costs Although compile-time meta-programming has a number of benefits, it would be naı̈ve to assume that it has no costs associated with it. However although Converge’s features have been used to build several small programs, and two systems of several thousand lines of code each, it will require a wider range of experience from multiple people working in different domains to make truly informed comments in this area. One thing is clear from experience with LISP: compile-time meta-programming in its rawest form is not likely to be grasped by every potential developer [Que96]. To use it to its fullest potential requires a deeper understanding of the host language than many developers are traditionally used to; indeed, it is quite possible that it requires a greater degree of understanding than many developers are prepared to learn. Whilst features such as extendable syntax (see section 5) which are layered on top of compile-time meta-programming may smooth off many of the usability rough edges, fundamentally the power that compile-time meta-programming extends to the user comes at the cost of increased time to learn and master. In Converge one issue that arises is that code which continually dips in and out of the meta-programming constructs can become rather messy and difficult to read on screen if over-used in any one area of code. This is due in no small part to the syntactic considerations that necessitate a move away from the clean Python-esque syntax to something closer to the C family of languages. It is possible that the integration of similar features into other languages with a C-like syntax would lead to less obvious syntactic seams. 3.13 Error reporting Perhaps the most significant unresolved issue in compile-time meta-programming systems relates to error reporting [CJOT04]. Although Converge does not have complete solutions to all issues surrounding error reporting, it does contain some rudimentary features which may give insight into the form of more powerful error reporting features both in Converge and other compile-time meta-programming systems. The first aspect of Converge’s error reporting facilities relates to exceptions. When an exception is raised, detailed stack traces are printed out allowing the user to inspect the sequence of calls that led to the exception being raised. These stack traces differ from those found in e.g. Python in that each level in the stack trace displays the file name, line number and column number that led to the error. Displaying the column number allows users to make use of the fine-grained information to more quickly narrow down the precise source of an exception. Converge is able to display such detailed information because when it parses text, it stores the file name, line number and column number of each token. Tokens are ordered into parse trees; parse trees are converted into ASTs; ASTs are eventually converted into VM instructions. At each point in this conversion, information about the source code elements is retained. Thus every VM instruction in a binary Converge program has a corresponding debugging entry which records which file, line number and column number the VM instruction relates 30 to. Whilst this does require more storage space than simpler forms of error information, the amount of space required is insignificant when the vast storage resources of modern hardware are considered. Whilst the base language needs to record the related source offset of each VM instruction, the source file a VM instruction relates to is required only due to compiletime meta-programming. Consider a file A.cv: func f(): return [| 2 + "3" |] and a file B.cv: import A func main(): $<<A.f()>> When the quasi-quoted code in A.f is spliced in, and then executed an exception will be raised about the attempted addition of an integer and a string. The exception that results from running B is as follows: Traceback (most recent call last): File "A.cv", line 2, column 13, in main Type_Exception: Expected instance of Int, but got instance of String. The fact that the A module is pinpointed as the source of the exception may initially seem surprising, since the code raising the exception will have been spliced into the B module. This is however a deliberate design choice in Converge. Although the code from A.f has been spliced into B.main, when B is run the quasi-quoted code retains the information about its original source file, and not its splice location. To the best of my knowledge, this approach to error reporting in the face of compile-time meta-programming is unique. As points of comparison, TH is not able to produce any detailed information during a stack-trace and SCM Scheme [Jaf03] pinpoints the source file and line number of run-time errors as that of the macro call site. In SCM Scheme if the code that a macro produces contains an error, all the user can work out is which macro would have led to the problem — the user has no way of knowing which part of the macro may be at fault. Converge allows customization of the error-reporting information stored about a given ITree. Firstly Converge adds a feature not present in TH: nested quasi-quotes. Essentially an outer quasi-quote returns the ITree of the code which would create the ITree of the nested quasi-quote. For example the following nested code: Sys.println([| [| 2 + "3" |] |].pp()) results in the following output: CEI.ibinary_add(CEI.iint(2, "ct.cv", 484), CEI.istring("3", "ct.cv", 488), "ct.cv", 486) Nested quasi-quotes provide a facility which allows users to analyse the ITrees that plain quasi-quotes generate: one can see in the above that each ITree element contains a reference to the file it was contained within (ct.cv in this case) and to the offset within the file (484 and so on). The CEI module provides a function src info to var 31 which given an ITree representing quasi-quoted code essentially copies the ITree 6 replacing the source code file and offsets with variables src file and src offset. This new ITree is then embedded in a quasi-quoted function which takes two arguments src file and src offset. When the user splices in, and then calls, this function they update the ITree’s relation to source code files and offsets. Using this function in the following fashion: Sys.println(CEI.src_info_to_var([| [| 2 + "3" |] |]).pp()) results in the following output: unbound_func (src_file, src_offset){ return CEI.ibinary_add(CEI.iint(2, src_file, src_offset), CEI.istring("3", src_file, src_offset), src_file, src_offset) } In practice when one wishes to customise the claimed location of quasi-quoted code, the nested quasi-quotes need to be cancelled out by a splice. For example, to change source information to be offset 77 in the file nt.cv we would use the following code: return $<<CEI.src_info_to_var([| [| 2 + "3" |] |], "nt.cv", 77>> Whilst this appears somewhat clumsy, it is worth noting that by adding only the simple concept of nested quasi-quotes, complex manipulation of the meta-system is possible. Converge’s current approach is not without its limitations. Its chief problem is that it can only relate one source code location to any given VM instruction. There is thus an ‘either / or’ situation in that the user can choose to record either the definition point of the quasi-quoted code, or change it to elsewhere (e.g. to record the splice point). It would be of considerable benefit to the user if it is possible to record all locations which a given VM instruction relates to. Assuming the appropriate changes to the compiler and VM, then the only user-visible change would be that src info to var would append src file and src offset information within a given ITree, rather than overwriting the information it already possessed. 3.14 Related work Perhaps surprisingly, the template system in C++ has been found to be a fairly effective, if crude, mechanism for performing compile-time meta-programming [Vel95, CJOT04]. Essentially the template system can be seen as an ad-hoc functional language which is interpreted at compile-time. However this approach is inherently limited compared to the other approaches described in this section. The dynamic OO language Dylan – perhaps one of the closest languages in spirit to Converge – has a similar macro system [BP99] to Scheme. In both languages there is a dichotomy between macro code and normal code; this is particularly pronounced in Dylan, where the macro language is quite different from the main Dylan language. As explained in the introduction, languages such as Scheme need to be able to identify macros as distinct from normal functions (although Bawden has suggested a way 6 In the current implementation, the src info to var actually mutates ITrees, but for reasons explained in section 4.3.1 this will not be possible in the future. 32 to make macros first-class citizens [Baw00]). The advantage of explicitly identifying macros is that there is no added syntax for calling a macro: macro calls look like normal function calls. Of course, this could just as easily be considered a disadvantage: a macro call is in many senses rather different than a function call. In both schemes, macros are evaluated by a macro expander based on patterns – neither executes arbitrary code during macro expansion. This means that their facilities are limited in some respects – furthermore, overuse of Scheme’s macros can lead to complex and confusing ‘language towers’ [Que96]. Since it can execute arbitrary code at compiletime Converge does not suffer from the same macro expansion limitations, but whether moving the syntax burden from the point of macro definition to call site will prevent the comprehension problems associated with Scheme is an open question. Whilst there are several proposals to add macros of one sort or another to existing languages (e.g. Bachrach and Playford’s Java macro system [BP01]), the lack of integration with their target language thwarts practical take-up. Nemerle [SMO04] is a statically typed OO language, in the Java / C# vein, which includes a macro system mixing elements of Scheme and TH’s systems. Macros are not first-class citizens, but AST’s are built in a manner reminiscent of TH. The disadvantage of this approach is that calculations often need to be arbitrarily pushed into normal functions if they need to be performed at compile-time. Comparisons between Converge and TH have been made throughout this section – I do not repeat them here. MetaML is TH’s most obvious forebear and much of the terminology in Converge has come from MetaML via TH. MetaML differs from TH and Converge by being a multi-stage language. Using its ‘run’ operator, code can be constructed and run (via an interpreter) at run-time, whilst still benefiting from MetaML’s type guarantees that all generated programs are type-correct. The downside of MetaML is that new definitions can not be introduced into programs. The MacroML proposal [GST01] aims to provide such a facility but – in order to guarantee type-correctness – forbids inspection of code fragments which limits the features expressivity. Significantly, with the exception of Dylan, I know of no other dynamically typed OO language in the vein of Converge which supports any form of compile-time metaprogramming natively. 4 Implications of Converge’s compile-time meta-programming for other languages and their implementations I believe that Converge shows that compile-time meta-programming facilities can be added in a seamless fashion to a dynamically-typed OO language and that such facilities provide useful functionality not available previously in such languages. In this section I first pinpoint the relatively minimal requirements on language design necessary to allow the safe and practical integration of compile-time meta-programming facilities. Since the implementation of such a facility is quite different from a normal language compiler, I then outline the makeup of the Converge compiler to demonstrate 33 how an implementation of such features may look in practice. Finally I discuss the requirements on the interface between user code and the languages compiler. 4.1 Language design implications Although Converge’s compile-time meta-programming facilities have benefited slightly from being incorporated in the early stages of the language design, there is surprisingly little coupling between the base language and the compile-time meta-programming constructs. The implications on the design of similar languages can thus be boiled down to the following two main requirements: 1. It must be possible to determine all namespaces statically, and also to resolve variable references between namespaces statically. This requirement is vital for ensuring that scoping rules in the presence of compile-time meta-programming are safe and practical (see section 3.5). Slightly less importantly, this requirement also allows functions called at compile-time to be stored in the same module as splices which call them whilst avoiding the forward reference problem (see section 3.9). 2. Variables within namespaces other than the outermost module namespace must be α-renameable without affecting the programs semantics. This requirement is vital to avoid the problem of variable capture. Note that there is an important, but non-obvious, corollary to the second point: when variables and slot names overlap then α-renaming can not take place. In section 3.5 we saw that, in Converge, top-level module definitions can not be renamed because the variable names are also the slot names of the module object. Since Converge forces all accesses of class fields via the self variable, Converge neatly sidesteps another potential place where this problem may arise. Fortunately, whilst many statically typed languages allow class fields to be treated as normal variables (i.e. making the self. prefix optional) most dynamically typed languages take a similar approach to Converge and should be equally immune to this issue in that context. Only two constructs in Converge are dedicated to compile-time meta-programming. Practically speaking both constructs would need to be added to other languages: 1. A splicing mechanism. This is vital since it is the sole user mechanism for evaluating expressions at compile-time. 2. A quasi-quoting mechanism to build up AST’s. Although such a facility is not strictly necessary, experience suggests that systems without such a facility tend towards the unusable [WC93]. 4.2 Compiler structure Typical language compilers follow a predictable structure: a parser creates a parse tree; the parse tree may be converted into an AST; the parse tree or AST is used to 34 $ Bytecode Generation ITree Generation Parsing [| |] [| |] [| |] Quasiquotes Mode $ Splice Mode Figure 2: Converge compiler states. generate target code (be that VM bytecode, machine code or an intermediate language). Ignoring optional components such as optimizers, one can see that normal compilers need only two or three major components (depending on the inclusion or omission of an explicit AST generator). Importantly the process of compilation involves an entirely linear data flow from one component to the next. Compile-time meta-programming however necessitates a different compiler structure, with five major components and a non-linear data flow between its components. In this section I detail the structure of the Converge compiler, which hopefully serves as a practical example for compilers for other languages. Whether existing language compilers can be retro-fitted to conform to such a structure, or whether a new compiler would need to be written can only be determined on a case-by-case basis; however in either case this general structure serves as an example. Figure 2 shows a (slightly non-standard) state-machine representing the most important states of the Converge compiler. Large arrows indicate a transition between compiler states; small arrows indicate a corresponding return transition from one state to another (in such cases, the compiler transitions to a state to perform a particular action and, when complete, returns to its previous state to carry on as before). Each of these states also corresponds to a distinct component within the compiler. The stages of the Converge compiler can be described thus: 1. Parsing. The compiler parses an input file into a parse tree. Once complete, the compiler transitions to the next state. 2. ITree Generation. The compiler converts the parse tree into an ITree; this stage continues until the complete parse tree has been converted into an ITree. Since ITree’s are exposed directly to the user, it is vital that the parse tree is converted into a format that the user can manipulate in a practical manner 7. (a) Splice mode / bytecode generation. When it encounters a splice annotation in the parse tree, the compiler creates a temporary ITree representing a module. It then transitions temporarily to the bytecode generation state to compile. The compiled temporary module is injected into the running 7 An early, and naı̈ve, prototype of the Converge compiler exposed parse trees directly to the user. This quickly lead to spaghetti code. 35 VM and executed; the result of the splice is used in place of the annotation itself when creating the ITree. (b) Quasi-quotes mode / splice mode. As the ITree generator encounters quasi-quotes in the parse tree, it transitions to the quasi-quote mode. Quasiquote mode creates an ITree respecting the scoping rules and other features of section 3.5. If, whilst processing a quasi-quoted expression, a splice annotation is encountered, the compiler enters the splice mode state. In this state, the parse tree is converted to an ITree in a manner mostly similar to the normal ITree Generation state. If, whilst processing a splice annotation, a quasi-quoted expression is encountered, the compiler enters the quasi-quotes mode state again. If, whilst processing a quasi-quoted expression, a nested quasiquoted expression is encountered the compiler enters a new quasi-quotes mode. 3. Bytecode generation. The complete ITree is converted into bytecode and written to disk. 4.3 Compiler interface Converge provides the CEI module which user code can use to interact with the language compiler. Similar implementations will require a similar interface to allow two important activities: 1. The creation of fresh variable names (see section 3.6.2). This is vital to provide a mechanism for the user to generate unique names which will not clash with other names, and thus will prevent unintended variable capture. To ensure that all fresh names are unique, most practical implementations will probably choose to inspect and restrict the variable names that a user can use within ITree’s via an analogue to Converge’s name interface; this is purely to prevent the user inadvertently using a name which the compiler has guaranteed (or might in the future guarantee) to be unique. 2. The creation of arbitrary AST’s. Since it is extremely difficult to make a quasiquote mechanisms completely general without making it prohibitively complex to use, there are likely to be valid AST’s which are not completely expressible via the quasi-quotes mechanism. Therefore the user will require a mechanism to allow them to create arbitrary AST fragments via a more-or-less traditional meta-programming interface [WC93]. 4.3.1 Abstract syntax trees One aspect of Converge’s design that has proved to be more important than expected, is the issue of AST design. In typical languages, the particular AST used by the compiler 36 is never exposed in any way to the user. Even in Converge, for many users the particulars of the ITree’s they generate via the quasi-quotes mechanism are largely irrelevant. However those users who find themselves needing to generate arbitrary ITree’s via the CEI interface, and especially those (admittedly few) who perform computations based on ITree’s, find themselves disproportionately affected by decisions surrounding the ITree’s representation. At the highest level, there are two main choices surrounding AST’s. Firstly should it be represented as an homogeneous, or heterogeneous tree? Secondly should the AST be mutable or immutable? The first question is relatively easy to answer: experience suggests that homogeneous trees are not a practical representation of a rich AST. Whilst parse trees are naturally homogeneous, the conversion to an AST leads to a more structured and detailed tree that is naturally heterogeneous. Let us then consider the issue of AST mutability. Initially Converge supported mutable AST’s; whilst this feature has proved useful from time to time, it has also proved somewhat more dangerous than expected. This is because one often naturally creates references to a given AST fragment from more than one node. Changing a node which is referenced by more than one other node can then result in unexpected changes, which all too frequently manifest themselves in hard to debug ways. Since it is not possible to check for this problem in the general case, the user is ultimately responsible for ensuring it does not occur; in practise this has proved to be unrealistic, and gradually all ITree-mutating code has been banished from Converge code. Future versions of Converge will force ITree’s to be immutable, and I would recommend other languages consider this point carefully. 5 Syntax extension for DSL’s Converge has a simple but powerful facility allowing users to embed arbitrary sequences of tokens within Converge source files. At compile-time these tokens are passed to a designated user function, which is expected to return an AST. This allows the user to extend the languages syntax in an arbitrary fashion, meaning that DSLs can be embedded within normal Converge code. Although this feature is somewhat less developed than the other aspects of Converge, it is useful even its current state. Essentially a DSL fragment is an indented block containing an arbitrary sequence of tokens. The DSL block is introduced by a variant on the splice syntax $< expr > where expr should evaluate to a function (the DSL implementation function). The DSL function will be called at compile-time with a list of tokens, and is expected to return an AST which will replace the DSL block in the same way as a normal splice. Compile-time meta-programming is thus the mechanism which facilitates embedding DSLs. An example DSL fragment is as follows. Colloquially this block is referred to as ‘a TM.model class’: import TM.TM $<TM.model_class>: 37 abstract class ML1_Element { name : String; inv nonempty_name: name != null and name.len() > 0 } Note that the DSL fragment is written in an entirely different syntax than Converge itself. Currently DSL blocks are automatically tokenized by the Converge compiler using its default tokenization rules — this is not a fundamental requirement of the technique, but a peculiarity of the current implementation. More sophisticated implementations might choose to defer tokenization to the DSL implementation function. However using the Converge tokenizer has the advantage that normal Converge code can be embedded inside the DSL itself assuming an appropriate link from the DSL’s grammar to the Converge grammar. 5.1 DSL implementation functions DSL implementation functions follow a largely similar sequence of steps in order to translate the input tokens into an ITree: 1. Alter the input tokens as necessary. Since DSL’s often use keywords that are not part of the main Converge grammar, such alterations mostly take the form of replacing ID tokens with specific keyword tokens. 2. Parse the input tokens according to the DSL’s grammar. 3. Traverse the parse tree, translating it into an ITree. Section 6 explores these steps in greater detail via a concrete example. 5.2 Related work Real-world implementations of a similar concept are surprisingly rare. The Camlp4 pre-processor [dR03] allows the normal OCaml grammar to be arbitrarily extended, and is an example of a heterogeneous syntax extension system in that the system doing the extension is distinct from the system being extended. The MetaBorg system [BV04] is a heterogeneous system that can be applied to any language; more sophisticated than the Camlp4 pre-processor, from an external point of view it more closely resembles Converge’s functionality, although the implementations and underlying philosophies are still very different. I am currently aware of only two homogeneous syntax extension systems apart from Converge. Nemerle [SMO04] allows limited syntax extension via its macro scheme. The commercial XMF tool [CESW04] presents only a small core grammar, with many normal language concepts being grammar extensions on top of the core grammar. Grammar extensions are compiled down into XMF’s AST. XMF is thus much closer in spirit to Converge, although the example grammar extensions available suggest that XMF’s compile-time facilities may be less powerful than Converge’s, 38 type Classifier name : String parents *{ordered} PrimitiveDataType Class name : String Attribute is_primary : bool name : String attrs *{ordered} Figure 3: ‘Simple UML’ model. seemingly being based on a simplified version of TH’s features. If true, this may limit the complexity of the grammar extensions. 6 Modelling language DSL This section presents an example of a Converge DSL for expressing typed modelling languages; modelling languages can be instantiated create models. In its current simplistic form, the modelling DSL operates with a fixed number of meta-levels in that it defines modelling languages that can create models, but those models are terminal instances (in ObjVLisp’s terminology) — that is, they can not be used to create new objects. This section serves as an example of Converge’s syntax extension system, and fleshes out the method of section 5.1. 6.1 Example of use The typed modelling language DSL is housed within the package TM; the DSL implementation function model class is contained within the TM module within the package. The following fragment uses the DSL to express a model of a simplified UML-esque modelling language as shown in figure 3: import TM.TM $<TM.model_class>: abstract class Classifier { name : String; } class PrimitiveDataType extends Classifier { } class Class extends Classifier { parents : Seq(Class); attrs : Seq(Attribute); inv unique_names: attrs->forAll(a1 a2 | a1 != a2 implies a1.name != a2.name) } 39 class Attribute { name : String; type : Classifier; is_primary : bool; } Note that although this particular examples shows a model of a modelling language, the DSL is capable of expressing any type of model. The TM.model class DSL implementation function translates each class in the model into a function in Converge which creates model objects. As a useful convenience, each constructor function takes arguments which correspond to the order attributes are specified in the model class. If a model class has parents, their attributes come first, and so on recursively. Model objects can have their slots accessed by name. Note that since the modelling language is typed, setting attributes either via the constructor function or through assigning to a slot forces the value to be of the correct type. Types can be of any model class (the DSL allows forward references), int, String, bool (where true and false are represented by 1 and 0 respectively), or sequences or sets of the preceding types. Note that sequences and sets can be nested arbitrarily. Model classes can contain invariants which are written in OCL; invariants are checked after an object has been initialized with values, and on every subsequent slot update. Currently only a subset of OCL 1.x is implemented, but the subset covers several different areas of OCL; implementing full OCL 1.x would be a relatively simple extension. Assuming the above is held in a file Simple UML.cv, one can then use the use the Simple UML modelling language to create models. The following example creates model classes Dog and Person, with Dog having an attribute owner of type Person: person := Simple_UML.Class("Person") dog := Simple_UML.Class("Dog") dog.attrs.append(Simple_UML.Attribute("owner", person, 0)) One can arbitrarily manipulate models in a natural fashion: dog.name := "Doggy" Attempting to update a model in a way that would conflict with its type information results in an exception being raised. For example, attempting to assign an integer to the Dog model class’ name raises the following exception: Traceback (most recent call last): File "Ex1.cv", line 42, column 4, in main File "TM/TM.cv", line 162, column 5, in set_slot Exception: Instance of ’Class’ expected object of type ’String’ for slot ’name’. In similar fashion, if one violates the unique names constraint by adding two attributes called owner to the Dog model class, the following exception is raised: Traceback (most recent call last): File "Ex1.cv", line 45, column 17, in main File "TM/TM.cv", line 327, column 31, in append File "TM/TM.cv", line 407, column 3, in _class_class_check_invs Exception: Invariant ’unique_names’ violated. 40 As can be seen, the result of using the TM.model class DSL is a natural embedding of an arbitrary modelling language within Converge. Furthermore by recording, and enforcing, type information using the modelling language DSL provides guarantees about models that would not have been the case if they had been implemented as normal Converge class’s. The typed modelling language DSL neatly sidesteps the data representation problems found in typical GPL’s. In the following sections I outline how this DSL is implemented. 6.2 Data model The TM package provides its own ObjVLisp style data model which is similar to, but distinct from, the Converge data model of section 2.4. The TM package needs to provide a new data model since the default Converge data model is inherently untyped; whilst figure 1 showed the core data model with types, such type information is purely for the benefit of the reader. In contrast, the TM data model is inherently typed, and the type information is used to enforce the correctness of models. The only exception to this is that functions are currently untyped; it would be relatively simple to extend the implementation to record and enforce functions’ type information. Figure 4 shows the TM data model. As in Converge, a bootstrapping phase is needed to set up the meta-circular data model. MObject and MClass are so named to avoid clashing with the builtin classes Object and Class. Similarly, method and attribute names which might conflict, or be confused with, those found in normal Converge classes are named differently. For example init becomes initialize, to str becomes to string and instance of becomes of. For brevity, and for easy interaction with external code, the TM package does not directly replicate all builtin Converge types such as strings; builtin Converge types are treated internally as instances of MObject. Once cosmetic differences between the two are ignored, some important differences in the TM and Converge data models become apparent. Most importantly the TM data model has the standard statically typed OO languages notions of separate methods and attributes. TM mclasses are also different in that they can be abstract (i.e. can not be directly instantiated) and have at most one super class. MObject classes possess a mod id slot which is a unique identifier, and which is typed as String to allow flexibility over the format of identifiers. The mod id slot plays an important rôle in change propagating transformations. As all this might suggest, the TM data model is intended to closely match the data model found within statically typed OO languages such as Java than the. Since methods and attributes are housed separately within classes, model instances require only a slot per attribute; invoking a method on an object searches the objects of class (and its superclasses) for an appropriate method. This is achieved by making use of the Converge MOP (see section 2.7). Although the actual implementation is relatively complex, a simplified version demonstrates the salient points. All model objects are instances of the Converge class Raw Object which is initialized with a blank slot per attribute 41 MObject mod_id : String initialize(*MObject) : void to_string() : String of super_class 0, 1 MClass attrs : Dict{String : MObject} invariants : Seq{MObject} is_abstract : bool methods : Dict{String : MObject} name : String check_invs() : void initialize(*MObject) : void new_(*MObject) : MObject Figure 4: TM data model. of a model class. The Converge MOP is overridden via a custom get slot function in the Raw Object class. If a slot name matches an attribute slot, that value is returned. Otherwise the model objects of class and, if necessary its superclasses, are searched for a method of the appropriate name. Finally, if a method is not found then if the slot name matches that of a normal Converge slot in the Raw Object instance, the value is returned; otherwise an exception is raised. The following, much simplified, version of the code shows the skeleton of the Raw Object class and part of its MOP: 1 2 3 4 5 class _Raw_Object: func init(attr_names): self._attr_slots := Dict{} for attr_name := attr_names.iterate(): self._attr_slots[attr_name] := null 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 func get_slot(name): if self._attr_slots.contains(name): return self._attr_slots[name] else: class_ := self._attr_slots["of"] while 1: if class_.methods.contains(name): return Func_Binding(self, class_.methods[name]) if (class_ = class_.super_class) == null: break if exbi Object.has_slot(name): return exbi Object.get_slot(name) else: raise Exceptions.Slot_Exception(Strings.format( \ "No such model / Converge slot ’%s’", name)) A few notes are in order. Firstly the class variable on line 11 is so named since class is a reserved keyword in Converge; by convention variable names are suffixed by ‘ ’ if they would otherwise clash with a reserved word. Note that definitions prefixed by ‘ ’ are conventionally considered to be private to the module or class they are contained within. On line 14, the Func Binding class creates a binding which, when invoked, will call the Converge function class .methods[name] with its self variable bound to self (i.e. the Raw Object instance; see section 2.2 for more details about functions and function bindings). The ability to create function 42 bindings in this fashion is an important feature of Converge, allowing a large deal of control over the behaviour of objects. The TM packages data model can be considered to be a suitable template for suggesting how more powerful typed modelling languages – perhaps including packages and package inheritance [ACE+ 02], or allowing classes to inherit from more than one superclass) – might be naturally represented in a Converge DSL. 6.3 Pre-parsing and grammar Before the TM.model class DSL can parse its input, it first iterates through the input tokens searching for tokens which have type ID and value any of abstract, and, at, collect, extends, forAll, implies, inv, Seq, Set. Such tokens are replaced by a keyword token, whose type is the ID’s value. Furthermore since the TM.model class DSL is intended to emulate typed languages such as C and Java, it implements a white space insensitive grammar; thus all INDENT, DEDENT, and NEWLINE tokens are removed from the input. The modified token list is then parsed according to the following grammar: top_level ::= { class }* class ::= class_abstract "CLASS" "ID" class_super "{" { class_field }* "}" class_abstract ::= "ABSTRACT" ::= class_super ::= "EXTENDS" "ID" ::= class_field ::= field_type ::= invariant field_type ::= "ID" ":" type ";" type ::= "ID" ::= "SEQ" "(" type ")" ::= "SET" "(" type ")" invariant ::= "INV" "ID" ":" expr expr ::= ::= ::= ::= ::= ::= ::= ::= int string slot_lookup application binary seq set "ID" int ::= "INT" string ::= "STRING" slot_lookup ::= ::= ::= ::= ::= ::= ::= ::= forall at collect application %precedence 20 %precedence 15 %precedence 10 expr "." "ID" expr "-" ">" forall expr "-" ">" at expr "-" ">" collect "FORALL" "(" "ID" "|" expr ")" "FORALL" "(" "ID" "ID" "|" expr ")" "AT" "(" expr ")" "COLLECT" "(" "ID" "ID" "=" expr "|" expr ")" ::= expr "(" expr { "," expr }* ")" ::= expr "(" ")" 43 binary ::= ::= ::= ::= ::= ::= ::= ::= expr expr expr expr expr expr expr expr "+" expr "-" expr ">" expr "<" expr "==" expr "!=" expr "IMPLIES" expr "AND" expr %precedence %precedence %precedence %precedence %precedence %precedence %precedence %precedence seq ::= "SEQ" "{" expr ".." expr "}" ::= "SEQ" "{" expr { "," expr }* "}" ::= "SEQ" "{" "}" set ::= "SET{" expr { "," expr }* "}" ::= "SET{" "}" 30 30 20 20 20 20 10 10 Most of this grammar is straightforward, although it is worth noting a few peculiarities that result from the fact that tokenization is performed by the Converge tokenizer. For example, ‘Set{’ is a single token (since Set{...} builds up a set in normal Converge). The equivalent notation for sets is represented by two tokens: ‘Seq’ (a new keyword introduced by the DSL) followed by ‘{’. Fortunately in practise, such idiosyncrasies are largely hidden from, and irrelevant to, the DSL’s users. 6.4 Traversing the parse tree The main part of the DSL implementation function is concerned with traversing the parse tree, and translating it into an appropriate ITree. At a high-level, the translation is fairly simple: each model class is converted into an object capable of creating model instances. The CPK provides a simple traversal class (essentially a Converge equivalent of that found in the SPARK parser [AH02]) which provides the basis for most such translations. Users need only subclass the Traverser class and create a function prefixed by t name for each rule in the grammar. The Traverser class provides a preorder function will traverse an input parse tree in preorder fashion, calling the appropriate t name function for each node encountered in the tree. Note that each t name function can choose whether to invoke the preorder rule on sub-nodes, or whether it is capable of processing the sub-nodes itself. For example, the TM.model class function defines a traversal class Model Class Creator which translates the DSL’s parse tree. An idealized version of the beginning of this class looks as follows: import CPK.Traverser class Model_Class_Creator(Traverser.Traverser): func translate(): return self.preorder() func _t_top_level(node): // top_level ::= { class }* classes := [] for class_node := node[1 : ].iterate() classes.extend(self.preorder(class_node)) return classes func _t_class(node): // class ::= class_abstract "CLASS" "ID" class_super "{" // class_field }* "}" ... return [| 44 { class $<<CEI.name(node[3].value)>>: ... |] As this example shows, by automatically creating a parse tree and presenting a simple but powerful mechanism for traversing that parse tree, the CPK imposes a low burden on its users. 6.5 Translating The actual translation of the parse tree to a ITree involves much repetition, and contains implementation details which are irrelevant to this paper. The first point to note about the translation is that the resulting ITree largely follows the structure of the parse tree. Having the translation follow the structure of the parse tree is desirable because it significantly lowers the conceptual burden involved in creating and comprehending the translation. In this subsection I highlight some interesting aspects of the translation; interested readers can use this as a step to exploring the full translation in the TM package. 6.5.1 OCL expressions Translating the OCL subset into Converge is a simple place to start in the translation because it is mostly simple and repetitive. For example, converting binary expressions from OCL into Converge is mostly a direct translation as the elided t binary traversal function shows: func _t_binary(node): // binary ::= expr "+" expr // ::= expr "<" expr // ::= expr "==" expr lhs := self.preorder(node[1]) rhs := self.preorder(node[3]) if node[2].type == "+": return [| $<<lhs>> + $<<rhs>> |] elif node[2].type == ">": return [| $<<lhs>> > $<<rhs>> |] elif node[2].type == "==": return [| func ocl_equals() { lhs := $<<lhs>> if lhs.conforms_to(Int) | lhs.conforms_to(String): return lhs == $<<rhs>> else: return lhs is $<<rhs>> }() |] Note that there is a slight complexity in translating the == operator, since OCL defines equality between objects to be based on their value if they are a primitive type, and on their identity if they are a model element. Whilst this is simple to encode as a sequence of expressions, it slightly complicates the t class traversal function, which is expected to a return only a single quasi-quoted expression. In order to work around this limitation we combine the necessary sequence of instructions into a function; the quasi-quotes returns the invocation of this function which is thus a single expression. This idiom occurs frequently in the translation. 45 6.5.2 Forward references Forward references between model classes might appear to slightly muddy the structure of the translation. Consider the following example: $<TM.model_class>: class Dog { owner : Person; } class Person { name : String; } Assuming a simple translation, the result would be similar to the following converge code: class Dog: attributes := Dict{"owner" : Person} name := "Dog" class Person: attributes := Dict{"name" : String} name := "Person" Such code would compile correctly, but lead to an exception being raised at run-time since the Person class will not have assigned a value to the Person variable when it is accessed in the Dog class. A standard approach to this problem would be to make the attributes field a function; by placing the reference to Person a function, the variable access would be deferred until after the Person variable contained a value. The TM.model class DSL takes an alternative approach which is simplistic and, whilst not generally applicable, effective. Essentially the TM module keeps a record of all model classes encountered. When a new model class is created (i.e. when importing a model containing a TM.model class block), it registers itself with the TM module. Rather than directly referring to model classes, type references are strings of the target model class name – when a model class needs to be retrieved, its name is looked up in the TM registry, and the appropriate object returned. Thus forward references are a non-issue, since references are only resolved when necessary. The reason the TM.model class DSL takes this approach is that all model classes live in the same namespace; when we come to transforming model elements it aids brevity that model classes do not need to be prefixed by a package or module name. 6.5.3 Model class translation The suggestion up to this point has largely been that model classes have been directly translated to normal Converge classes. In fact, model classes are instances of the MClass class. Whilst this would suggest that we can use the metaclass keyword shown in section 2.4, this is not possible since MClass requires more information than a normal Converge class. Normal Converge classes take only three parameters name, supers, fields whereas a model class requires information about whether it is 46 abstract, its invariants and so on. It is thus necessary to create the class manually. Fortunately this is relatively simple largely because the bound func keyword allows bound functions to be expressed outside a class. A much elided, and slightly simplified, version of the t class traversal function is as follows: is abstract := bool class name := String super := String or null attrs := Dict{name : type} invariants := List of tuples [name, function] operations := List of tuples [name, function] init func var := CEI.ivar(CEI.fresh name()) return [d| bound func initialize(*args): super attrs := all attrs($<<super >>, 1) if args.len() > (super attrs.len() + \ $<<CEI.lift(attrs.len())>>): raise Exceptions.Parameters Exception("Too many args") super args pos := Maths.min(super attrs.len(), args.len()) Func Binding(self, $<<super >>.methods["initialize"]). \ apply(args[ : super args pos]) $<<init func body>> $<<CEI.ivar(CEI.name(class name))>> := MClass( \ $<<is abstract>>, $<<CEI.lift(class name)>>, \ $<<super >>, $<<CEI.lift(attrs)>>, \ $<<CEI.idict(operations + [[CEI.lift("initialize"), \ init func var]])>>, $<<CEI.ilist(invariants)>>) |] Essentially the t class traversal function first evaluates and transforms the details of the model class, placing information such as whether the class is abstract into appropriately named variables. Finally it returns quasi-quoted code which contains two things: a function to initialize model class instances, and finally the instantiation of MClass itself. The arguments that MClass itself requires are hopefully obvious due to the names of the variables passed to it in this example. 6.5.4 Summary of translation Whilst this section has tersely presented the translation of a TM.model class block, I hope that it shows enough detail to suggest that the bulk of the translation is simple work, with only one or two areas requiring the use of more esoteric Converge features. Section B shows the pretty printed ITree resulting from the translation of the example in section 6.1. 6.6 Diagrammatic visualization A useful additional feature of the TM package is its ability to visualize modelling languages and model languages as diagrams. Visualization makes use of the fact 47 that the TM data model is fully reflective, making the traversal and querying of objects trivial. The Visualizer module defines several visualization functions, all of which use the GraphViz package [GN00] to create diagrams. For example, the visualize modelling language function takes a list of model classes, and visualizes them as a standard class diagram. Figure 5 shows the automatic visualization of the Simple UML modelling language originally shown in figure 3. Note that the modelling language visualization function explicitly shows that all model classes are subclasses of MObject. MObject mod_id : String of to_string() initialize() Classifier name : String initialize() PrimitiveDataType Class parents * ordered initialize() type initialize() attrs * ordered Attribute is_primary : bool name : String initialize() Figure 5: ‘Simple UML’ modelling language visualized. The Visualizer is also able to visualize models as UML object diagrams. Figure 6 shows the visualization of the following module: person := Simple_UML.Class("Person") dog := Simple_UML.Class("Dog") dog.attrs.append(Simple_UML.Attribute("owner", person, 0)) 7 Future work Because the core of Converge is a mix of established languages, the core language is largely stable. The implementation of the compile-time meta-programming facilities is currently less than satisfactory in one or two areas, but is eminently usable. One feature in particular that appears to be confuse new users to the language relates to the very different effects of the splice annotation. The syntax, inherited from Template 48 :Class mod_id = "9" name = "Dog" attrs :Attribute mod_id = "10" is_primary = 0 name = "owner" type :Class mod_id = "8" name = "Person" Figure 6: A ‘Simple UML’ model language visualized. Haskell, means that $ behaves very differently when inside (a simple replacement of the splice annotation) and outside (cause compile-time evaluation) quasi-quotes. A simple change of syntax may suffice to solve this problem, or it may be considered to be an inevitable part of the learning curve that compile-time meta-programming presents. Although Converge’s error reporting facilities are at least as good as any comparable language, there is still room for considerable improvement from the users point of view. Although section 3.13 used nested quasi-quotes to customise error reports, it may be necessary to find a lighter weight technique if DSL authors are to be encouraged to provide high quality error reporting. The syntax extension feature presents the greatest opportunity for future work. The most obvious improvement would be to allow the user to provide their own tokenization facility. This may result in simply passing a single string to the DSL implementation function, or it may involve a more sophisticated interaction between the Converge tokenizer and parser and the DSL tokenizer and parser. For example parsing algorithms such as Pack Rat parsing [For02] allow the conflation of tokenizing and parsing in a way that might lend themselves to syntax extension. 8 Acknowledgments This work has been funded by a grant from Tata Consultancy Services. My thanks to Kelly Androutsopoulos for insightful comments on an early draft of part of this paper. 49 A Converge grammar This section lists the CPK grammar for Converge. This is extracted directly from the Converge compiler file Compiler/CV Parser.cv: top_level ::= definition { "NEWLINE" definition }* ::= definition ::= ::= ::= ::= ::= class_def func_def import var { "," var }* ":=" expr splice import ::= "IMPORT" dotted_name import_as { "," dotted_name import_as }* dotted_name ::= "ID" { "." "ID" }* import_as ::= "AS" "ID" ::= class_def ::= "CLASS" class_name class_supers class_metaclass ":" "INDENT" class_fields "DEDENT" class_name ::= "ID" ::= splice class_supers ::= "(" expr { "," expr }* ")" ::= class_metaclass ::= "METACLASS" expr ::= class_fields ::= class_field { "NEWLINE" class_field }* class_field ::= class_def ::= func_def ::= var ":=" expr ::= splice ::= "PASS" func_def ::= func_type func_name "(" func_params ")" ":" "INDENT" func_nonlocals expr_body "DEDENT" ::= func_type func_name "(" func_params ")" "{" "INDENT" func_nonlocals expr_body "DEDENT" "NEWLINE" "}" func_type ::= "FUNC" ::= "BOUND_FUNC" ::= "UNBOUND_FUNC" func_name ::= "ID" ::= "+" ::= "-" ::= "/" ::= "*" ::= "<" ::= ">" ::= "==" ::= "!=" ::= ">=" ::= "<=" ::= splice ::= func_params ::= func_params_elems "," func_varargs ::= func_params_elems ::= func_varargs ::= func_params_elems ::= var func_param_default { "," var func_param_default }* ::= splice func_param_default ::= ":=" expr ::= func_varargs ::= "*" var ::= splice func_nonlocals ::= "NONLOCAL" "ID" { "," "ID" }* "NEWLINE" ::= expr_body ::= expr { "NEWLINE" expr }* 50 expr ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= ::= class_def func_def while if for try number var dict set list dict string slot_lookup list application lookup slice exbi return yield raise assert break continue conjunction alternation assignment not neg binary comparison pass import splice quasi_quotes brackets if ::= "IF" expr if_else ::= "IF" expr { if_elif if_elif ::= "NEWLINE" ::= "NEWLINE" "NEWLINE" if_else ::= "NEWLINE" ::= "NEWLINE" "}" ::= %precedence 50 %precedence 40 %precedence 40 %precedence 40 %precedence %precedence %precedence %precedence %precedence %precedence %precedence 10 10 15 17 35 30 20 %precedence 100 ":" "INDENT" expr_body "DEDENT" { if_elif }* "{" "INDENT" expr_body "DEDENT" "NEWLINE" "}" }* if_else "ELIF" expr ":" "INDENT" expr_body "DEDENT" "ELIF" expr "{" "INDENT" expr_body "DEDENT" "}" "ELSE" ":" "INDENT" expr_body "DEDENT" "ELSE" "{" "INDENT" expr_body "DEDENT" "NEWLINE" while ::= "WHILE" expr ":" "INDENT" expr_body "DEDENT" exhausted broken ::= "WHILE" expr for ::= "FOR" expr ":" "INDENT" expr_body "DEDENT" exhausted broken ::= "FOR" expr try ::= "TRY" ":" "INDENT" expr_body "DEDENT" { try_catch }* try_else try_catch ::= "NEWLINE" "CATCH" expr try_catch_var ":" "INDENT" expr_body "DEDENT" try_catch_var ::= "INTO" var ::= try_else ::= "NEWLINE" "ELSE" ":" "INDENT" expr_body "DEDENT" ::= exhausted ::= "NEWLINE" "EXHAUSTED" ":" "INDENT" expr_body "DEDENT" ::= broken ::= "NEWLINE" "BROKEN" ":" "INDENT" expr_body "DEDENT" ::= number ::= "INT" var ::= "ID" 51 ::= "&" "ID" ::= splice string ::= "STRING" slot_lookup ::= expr "." "ID" ::= expr "." splice list ::= "[" expr { "," expr }* "]" ::= "[" "]" dict ::= "DICT{" expr ":" expr { "," expr ":" expr }* "}" ::= "DICT{" "}" set ::= "SET{" expr { "," expr }* "}" ::= "SET{" "}" application ::= expr "(" expr { "," expr }* ")" ::= expr "(" ")" lookup ::= expr "[" expr "]" slice ::= ::= ::= ::= expr expr expr expr "[" "[" "[" "[" expr ":" expr "]" ":" expr "]" expr ":" "]" ":" "]" exbi ::= "EXBI" expr "." "ID" return ::= "RETURN" expr ::= "RETURN" yield ::= "YIELD" expr raise ::= "RAISE" expr assert ::= "ASSERT" expr break ::= "BREAK" continue ::= "CONTINUE" conjunction ::= expr "&" expr { "&" expr }* alternation ::= expr "|" expr { "|" expr }* assignment ::= assignment_target { "," assignment_target }* assignment_type expr assignment_target ::= var ::= slot_lookup ::= lookup ::= slice assignment_type ::= ":=" ::= "*=" ::= "/=" ::= "+=" ::= "-=" not ::= "NOT" expr neg ::= "-" expr binary ::= expr binary_op expr binary_op ::= "*" %precedence ::= "/" %precedence ::= "%" %precedence ::= "+" %precedence ::= "-" %precedence 40 30 30 20 20 comparison ::= expr comparison_op expr comparison_op ::= "IS" ::= "==" ::= "!=" ::= "<=" 52 ::= ">=" ::= "<" ::= ">" pass ::= "PASS" splice ::= ::= expr_splice ::= block_splice ::= expr_splice block_splice "$" "<" "<" expr ">" ">" "$" "<" expr ">" ":" "INDENT" "JUMBO" "DEDENT" quasi_quotes ::= expr_quasi_quotes ::= defn_quasi_quotes expr_quasi_quotes ::= "[|" "INDENT" expr { "NEWLINE" expr }* "DEDENT" "NEWLINE" "|]" ::= "[|" expr { "NEWLINE" expr }* "|]" defn_quasi_quotes ::= "[D|" definition { "NEWLINE" definition }* "|]" ::= "[D|" "INDENT" definition { "NEWLINE" definition }* "DEDENT" "NEWLINE" "|]" brackets ::= "(" expr ")" B The ‘Simple UML’ modelling language translation The following is the pretty printed ITree that results from translating the Simple UML modelling language shown in section 6.1: Simple_UML.cvb Simple_UML.cv $$1$$ := bound_func initialize_Classifier(*args){ super_attrs := TM._all_attrs(TM.MObject, 1) if args.len() > super_attrs.len() + 1: raise TM.Exceptions.Parameters_Exception("Too many args") super_args_pos := TM.Maths.min(super_attrs.len(), args.len()) TM.Func_Binding(self, TM.MObject.methods["initialize"]).apply(args[0 : \ super_args_pos]) if 0 < args.len() - super_args_pos: self.name := args[super_args_pos + 0] } Classifier := TM.MClass(1, "Classifier", TM.MObject, Dict{"name" : [3]}, \ ["name"], Dict{"initialize" : $$1$$}, []) $$2$$ := bound_func initialize_PrimitiveDataType(*args){ super_attrs := TM._all_attrs(TM._CLASSES_REPOSITORY["Classifier"], 1) if args.len() > super_attrs.len() + 0: raise TM.Exceptions.Parameters_Exception("Too many args") super_args_pos := TM.Maths.min(super_attrs.len(), args.len()) TM.Func_Binding(self, TM._CLASSES_REPOSITORY["Classifier"]. \ methods["initialize"]).apply(args[0 : super_args_pos]) } PrimitiveDataType := TM.MClass(0, "PrimitiveDataType", \ TM._CLASSES_REPOSITORY["Classifier"], Dict{}, [], Dict{"initialize" : $$2$$}, []) $$3$$ := bound_func initialize_Class(*args){ super_attrs := TM._all_attrs(TM._CLASSES_REPOSITORY["Classifier"], 1) if args.len() > super_attrs.len() + 2: raise TM.Exceptions.Parameters_Exception("Too many args") super_args_pos := TM.Maths.min(super_attrs.len(), args.len()) TM.Func_Binding(self, TM._CLASSES_REPOSITORY["Classifier"]. \ methods["initialize"]).apply(args[0 : super_args_pos]) if 0 < args.len() - super_args_pos: self.parents := args[super_args_pos + 0] if 0 >= args.len() - super_args_pos: self.parents := TM.TM_List(self) if 1 < args.len() - super_args_pos: self.attrs := args[super_args_pos + 1] if 1 >= args.len() - super_args_pos: self.attrs := TM.TM_List(self) } 53 Class := TM.MClass(0, "Class", TM._CLASSES_REPOSITORY["Classifier"], \ Dict{"parents" : [0, "Class"], "attrs" : [0, "Attribute"]}, ["parents", \ "attrs"], Dict{"initialize" : $$3$$}, [["unique_names", \ unbound_func unique_names(self){ return unbound_func ocl_for(){ expr := self.attrs for a1 := expr.iterate(): for a2 := expr.iterate(): if not unbound_func ocl_implies(){ if not unbound_func ocl_not_equals(){ lhs := a1 if lhs.conforms_to(TM.Int) | lhs.conforms_to(TM.String): return lhs != a2 else: return not lhs is a2 }(): return 1 if unbound_func ocl_not_equals(){ lhs := a1.name if lhs.conforms_to(TM.Int) | lhs.conforms_to(TM.String): return lhs != a2.name else: return not lhs is a2.name }(): return 1 return TM.fail }(): return TM.fail return 1 }() }]]) $$4$$ := bound_func initialize_Attribute(*args){ super_attrs := TM._all_attrs(TM.MObject, 1) if args.len() > super_attrs.len() + 3: raise TM.Exceptions.Parameters_Exception("Too many args") super_args_pos := TM.Maths.min(super_attrs.len(), args.len()) TM.Func_Binding(self, TM.MObject.methods["initialize"]).apply(args[0 : \ super_args_pos]) if 0 < args.len() - super_args_pos: self.name := args[super_args_pos + 0] if 1 < args.len() - super_args_pos: self.type := args[super_args_pos + 1] if 2 < args.len() - super_args_pos: self.is_primary := args[super_args_pos + 2] } Attribute := TM.MClass(0, "Attribute", TM.MObject, Dict{"name" : [3], \ "type" : "Classifier", "is_primary" : [5]}, ["name", "type", "is_primary"], \ Dict{"initialize" : $$4$$}, []) 54 References [AC96] Martı́n Abadi and Luca Cardelli. A Theory of Objects. Springer, 1996. [ACE+ 02] Biju Appukuttan, Tony Clark, Andy Evans, Stuart Kent, Girish Maskeri, Paul Sammut, Laurence Tratt, and James S. Willans. Unambiguous uml submission to uml 2 infrastructure rfp, September 2002. OMG document ad/2002-06-14. [AH02] John Aycock and R. Nigel Horspool. Practical earley parsing. The Computer Journal, 45(6):620–630, 2002. [Baw99] Alan Bawden. Quasiquotation in lisp. Workshop on Partial Evaluation and Semantics-Based Program Manipulation, January 1999. [Baw00] Alan Bawden. First-class macros have types. In Proc. 27th ACM SIGPLAN-SIGACT symposium on Principles of programming languages, pages 133–141, January 2000. [BC89] Jean-Pierre Briot and Pierre Cointe. Programming with explicit metaclasses in Smalltalk-80. In Proc. OOPSLA ’89, October 1989. [BP99] Jonathan Bachrach and Keith expressions: Lisp power, Playford. dylan style, http://www.ai.mit.edu/people/jrb/Projects/dexprs.pdf D1999. Accessed Sep 22 2004. [BP01] Jonathan Bachrach and Keith Playford. The java syntactic extender (jse). In Proc. OOPSLA, pages 31–42, November 2001. [BS00] Claus Brabrand and Michael Schwartzbach. Growing languages with metamorphic syntax macros. In Workshop on Partial Evaluation and Semantics-Based Program Manipulation, SIGPLAN. ACM, 2000. [BV04] Martin Bravenboer and Eelco Visser. Concrete syntax for objects. Domain-specific language embedding and assimilation without restrictions. In Douglas C. Schmidt, editor, Proc. OOPSLA’04, Vancouver, Canada, October 2004. ACM SIGPLAN. [CESW04] Tony Clark, Andy Evans, Paul Sammut, and James Willans. Applied metamodelling: A foundation for language driven development, September 2004. Available from http://www.xactium.com/ Accessed Sep 22 2004. [CJOT04] Krzysztof Czarnecki, Jörg Striegnitz John O’Donnell, and Walid Taha. DSL implementation in MetaOCaml, Template Haskell, and C++. 3016:50–71, 2004. 55 [CMA93] Luca Cardelli, Florian Matthes, and Martı́n Abadi. Extensible grammars for language specialization. In Proc. Fourth International Workshop on Database Programming Languages - Object Models and Languages, pages 11–31, August 1993. [Coi87] Pierre Cointe. Metaclasses are first class: the ObjVLisp model. In Object Oriented Programming Systems Languages and Applications, pages 156– 162, October 1987. [DGR01] Olivier Danvy, Bernd Grobauer, and Morten Rhiger. A unifying approach to goal-directed evaluation. New Generation Computing, 20(1):53–73, Nov 2001. [DHB92] R. Kent Dybvig, Robert Hieb, and Carl Bruggeman. Syntactic abstraction in scheme. In Lisp and Symbolic Computation, volume 5, pages 295–326, December 1992. [DM95] François-Nicola Demers and Jacques Malenfant. Reflection in logic, functional and object-oriented programming: a short comparative study. In Proc. IJCAI’95 Workshop on Reflection and Metalevel Architectures and Their Applications in AI, pages 29–38, August 1995. [dR03] Daniel de Rauglaudre. Camlp4 - Reference Manual, September 2003. http://caml.inria.fr/camlp4/manual/ Accessed Sep 22 2004. [Ear70] Jay Earley. An efficient context-free parsing algorithm. Communications of the ACM, 13(2), February 1970. [EBN02] Michael D. Ernst, Greg J. Badros, and David Notkin. An empirical analysis of C preprocessor use. IEEE Transactions on Software Engineering, 2002. [For02] Bryan Ford. Packrat parsing: Simple, powerful, lazy, linear time. In International Conference on Functional Programming, pages 36–47, October 2002. [GG93] Ralph E. Griswold and Madge T. Griswold. History of the Icon programming language. j-SIGPLAN, 28(3):53–68, March 1993. [GG96a] Ralph E. Griswold and Madge T. Griswold. The Icon Programming Language. Peer-to-Peer Communications, third edition, 1996. [GG96b] Ralph E. Griswold and Madge T. Griswold. The Implementation of the Icon Programming Language. Peer-to-Peer Communications, third edition, 1996. [GHJV94] Erich Gamma, Richard Helm, Ralph Johnson, and John Vlissides. Design Patterns: Elements of Reusable Object-Orientated Software. Addison Wesley, 1994. 56 [GN00] Emden R. Gansner and Stephen C. North. An open graph visualization system and its applications to software engineering. Software – Practice and Experience, 30(11):1203–1233, 2000. [GR89] Adele Goldberg and David Robson. Addison-Wesley, January 1989. [GST01] Steven E. Ganz, Amr Sabry, and Walid Taha. Macros as multi-stage computations: Type-safe, generative, binding macros in macroml. In Proc. International Conference on Functional Programming (ICFP), volume 36 of SIGPLAN. ACM, September 2001. [Gud92] David A. Gudeman. Denotational semantics of a goal-directed language. ACM Transactions on Programming Languages and System, 14(1):107– 125, January 1992. [Jaf03] Aubrey Jaffer. Smalltalk-80: The Language. SCM Scheme Implementation, November 2003. Accessed Sep 16 http://www.swiss.ai.mit.edu/˜jaffer/scm toc 2004. [Jef02] Clinton L. Jeffery. Godiva Language Reference Manual, November 2002. http://www.cs.nmsu.edu/˜jeffery/godiva/godiva.html. [JMPP03] Clinton Jeffery, Robert Parlett. [KCR98] Richard Kelsey, William Clinger, and Jonathan Rees. Revised(5) report on the algorithmic language Scheme. Higher-Order and Symbolic Computation, 11(1):7–105, 1998. Shamim Mohamed, Programming with Ray Pereda, and Unicon, April 2003. http://unicon.sourceforge.net/book/ub.pdf. [KdRB91] Gregor Kiczales, Jim des Rivieres, and Daniel G. Bobrow. The Art of the Metaobject Protocol. MIT Press, 1991. [KFFD86] Eugene Kohlbecker, Daniel P. Friedman, Matthias Felleisen, and Bruce Duba. Hygienic macro expansion. In Symposium on Lisp and Functional Programming, pages 151–161. ACM, 1986. [Pro97] Todd A. Proebsting. Simple translation of goal-directed evaluation. In SIGPLAN Conference on Programming Language Design and Implementation, pages 1–6, 1997. [Que96] Christian Queinnec. Macroexpansion reflective tower. In Proc. Reflection’96, pages 93–104, April 1996. [Sch] Friedrich Wilhelm Schröer. The ACCENT Grammar Language. Accessed Jan 25 http://accent.compilertools.net/language.html 2005. 57 [SCK03] Sean Seefried, Manuel M. T. Chakravarty, and Gabriele Keller. Optimising embedded DSLs using Template Haskell. In Draft Proc. Implementation of Functional Languages, 2003. [SeABP99] Tim Sheard, Zine el Abidine Benaissa, and Emir Pasalic. DSL implementation using staging and monads. In Proc. 2nd conference on Domain Specific Languages, volume 35 of SIGPLAN, pages 81–94. ACM, October 1999. [SJ02] Tim Sheard and Simon Peyton Jones. Template meta-programming for Haskell. In Proceedings of the Haskell workshop 2002. ACM, 2002. [SMO04] Kamil Skalski, Michal Moskal, and Pawel Olszta. Meta-programming in Nemerle, 2004. http://nemerle.org/metaprogramming.pdf Accessed Oct 1 2004. [Ste99] Guy L. Steele, Jr. Growing a language. Higher-Order and Symbolic Computation, 12(3):221 – 236, October 1999. [Tah99] Walid Taha. Multi-Stage Programming: Its Theory and Applications. PhD thesis, Oregon Graduate Institute of Science and Technology, October 1999. [TH00] David Thomas and Andrew Hunt. Programming Ruby: A Pragmatic Programmer’s Guide. Addison-Wesley, 2000. [Vel95] Todd Veldhuizen. Using C++ template metaprograms. C++ Report, 7(4):36–43, May 1995. [vR03] Guido van Rossum. Python 2.3 reference manual, 2003. Accessed Sep 23 http://www.python.org/doc/2.3/ref/ref.html 2004. [WC93] Daniel Weise and Roger Crew. Programmable syntax macros. In Proc. SIGPLAN, pages 156–165, 1993. 58