Using weaves for software construction and analysis

zyxwvu Using Weaves for Software Construction and Analysis Michael M. Gorlick and Rami R. Razouk* The Aerospace Corporation P.O. Box 92957, Los Angeles, CA 90009 Abstract cessive version is a testbed for investigating questions of functionality, user response and acceptance, performance, and cost. Continuous, low-overhead monitoring, whether for debugging, performance analysis, or human factors engineering, is always available. One abstract way to view a software system (a view commonly used as a design aid) is as a network of components that communicate via messages. However, high communication costs and the overhead of heavyweight parallel processes make the implementation of this idealized view comparatively expensive. The goal of using weaves is to reduce these expenses to the point where the abstract view can be implemented directly. For example, one application area that is conveniently characterized in this manner is satellite telemetry processing. Most satellites transmit a continuous, nonstop stream of information to earth. This telemetry stream contains payload sensor readings, measurements of the power output of solar arrays, operating tempera tures, electrical load, and command responses. Figure 1 illustrates a typical functional data flow for processing satellite-payload sensor readings whose interpretation depends upon the orientation of the satellite. The labeled boxes denote active functional elements, and the labeled arrows denote medium- to large-grain data flows from one functional element to another. These systems are characterized by a rich variety of data and the fact that individual functional elements can be successively decomposed into primitive, well-defined actions on streams of data. The ground stations that process these telemetry streams can be large, complicated systems requiring substantial computational resources; not surprisingly, designing, implementing, debugging, and performancetuning these software systems is notoriously difficult. These systems are distinguished by several features: e They are extremely long-lived. It can take from 5 to 10 years to design, build, and launch a single satellite. The accompanying ground stations may be in place for decades and may need to accommodate multiple ver- Weaves are networks of concurrently executing tool fragments that communicate by passing objects. Intended as an engineering medium for systems characterized by streams of data, weaves occupy a computational niche midway between heavyweight parallel processes and fine-grain dataflow. Weaves are distinguished from other data flow styles by their emphasis on instrumentation, continuous observability, and dynamic rearrangement: (1) Basic low-overhead instrumentation is inserted automatically. (2 )Executing weaves can be observed at any time by means of sophisticated analysis agents, without degrading the performance of the weave. (3) Weaves can be dynamically snipped and spliced without interrupting the data flow; this permits novel forms of experimentation and analysis. Weaves execute efficiently on a broad spectrum of architectures and offer numerous opportunities for parallel execution. Also, it is possible to synthesize nontrivial weaves automatically from libraries of components. We discuss the architectural features of weaves, their implementation, and their use in a variety of applications. Keywords: data flow, incremental development, objectoriented, parallel execution, performance analysis, software instrumentation, streams, threads 1 Introduction One of the central problems of software engineering concerns the economical development of software systems that perform satisfactorily. To deal with this problem we propose to use weaves, software organizations based on the low-overhead transmission of objects; weaves combine rapid incremental development with ongoing performance analysis and debugging. Rapid incremental development provides a continuous development sequence in which each development step is a small, easily understood change to, or extension of, the previous version of a system. Each suc- zyxwvutsr zyxwvut zyxwvutsrqpon zyxwvutsrqpon *This work was supported by the Aerospace Sponsored Research Program. 23 CH2982-7/91/0000/0023$01.OO (9 1991 IEEE Recommended by: Richard Kemmerer zyxwvu T Izyxwvutsrqpo 1 TelemetrvSatellite 1-1 Sensor R e a d i W - P I Filtering Star Sightings Filtered Sensor Readincs, Satellite Orientation Orientation Oriented Sensor Readings / Estimation Figure 1: A Typical Functional Data Flow for Telemetry Processing tended (“grown”) without disturbing the behavior of the tool fragments already in place. A weave can be refined by replacing a sink with a subweave. More generally, one subweave replaces another transparently if it respects the connectivity of the original. By systematically replacing tool fragments and subweaves with more refined versions, one can grow a complete weave, beginning with trivial tool fragments that mimic the behaviors of the final components. Entire subweaves can be attached anywhere within a running weave without disturbing the integrity of the original weave. Continuous low-overhead monitoring and observation. All weave components contain inexpensive instrumentation that can be left in place for the life of the system. This basic instrumentation is provided automatically, whereas specialized instrumentation can be transparently introduced into a weave at any time. Instruments and monitors that are application-specific can be easily derived, making “performance debugging” as natural and commonplace as functional debugging. Weaves can be observed at any time, without the need to make special provisions beforehand, and can be monitored at varying levels of detail. Concurrent communication. Weaves are graphs assembled from tool fragments, where the arcs of the graph serialize communication and synchronization and the data objects flowing through the weave serialize access. Tool fragments are written in commonly available deterministic languages, without regard to the issues of communication and synchronization. Considerable attention has been devoted to reducing the overhead of object passing. Object copying is the exception rather than the rule, and in most cases data-object transfer is by reference. Numerous opportunities for parallelism. Weaves are designed to take advantage of commonly available parallel architectures such as shared-memory multipro- sions of the same satellite. T h e y are eztremely large. Even modest systems contain over 100,000 lines of code. The largest examples contain in excess of 1,000,000 lines of code. Furthermore, history and evolution usually leave most systems with a legacy of two or more distinctly different programming languages. They must process large volumes of data in real or near-real t i m e . The telemetry transmission rates alone can strain the processing capacity of the most powerful processors available. The telemetry data may require sophisticated and extensive conditioning or processing before they are suitable for use by applications. They are opaque. It is nearly impossible, given the structure of these systems and their operational constraints, to instrument them once they are in place. For the sake of performance, all of the debugging aids have been optimized away. Even if such aids were still in place, determining where to look to resolve a performance problem or track down a bug remains a significant challenge. They undergo constant change. For example, a satellite may spontaneously exhibit anomalous behavior as a consequence of unforeseen environmental conditions or hardware degradation. These behaviors often require that bizarre “special cases” be inserted into the software. Furthermore, older satellites are often adapted to new missions that require additional software support. In some cases, existing telemetry may be exploited in a new way. In an effort to improve radically the software engineering practices for these ground systems, we have turned to weaves - networks of tool fragments that communicate by passing objects. Weaves are structured to permit, throughout the entire system lifecycle, the following: Continuous incremental change. Weaves are ex- zyxwvutsrqp zyxwvutsrq zyxwv 24 zyxwvutsrq zyxwvutsrqpon zyxwvutsrqpo zyxwvu zyxwvuts Figure 2: A Trivial Weave cessors. When suitably split, weaves can execute efficiently on nonuniform-memorymachines or distributed architectures. 0 Continuous scaling. A tool fragment in a weave may itself be an encapsulation of an entire subweave. Weaves scale hierarchically in a manner consistent with modern software engineering practices. This paper is divided into four sections. Section 2 contains a discussion of weaves and their constituent components, and Section 3 describes how weaves are assembled. Section 4 examines the mechanisms for monitoring and analyzing weaves. Finally, Section 5 describes our initial experience and outlines our future plans. 2 Figure 3: Snapshot of a Weave Weaves are designed to exploit a computational niche that lies midway between the large-grain filters introduced by Unix processes and pipes [l]and the fine-grain computations of classical data flow [2]. The computational model employed by weaves is medium-grain, non-blocking data flow with finite buffering, first-infirst-out stream access, and multiple readers and writers. Each individual stream datum is an object with encapsulated state, class membership, inheritance, and exported methods. Weaves are intended for situations that are characterized by such streams of data, for example, message switching and telemetry processing. Although weaves share some of the features of Unix filters (composed of processes and pipes), they differ significantly in several respects: Weave tool granularity is finer. A Unix tool, appearing as a component in a filter chain, is usually much larger than a single procedure. While many Unix tools adhere to the maxim “do one thing very well,” they are often composed of multiple routines that cannot be easily decomposed into separate tools. 0 Weave process granularity is finer. Unix tools run as heavyweight processes, with significant setup and switching times. However, the tool fragments in weaves execute as threads, which have much lower setup and switching times. This reduction in process granularity increases the opportunities for parallelism on multiprocessor architectures, particularly on uniform sharedmemory machines. 0 Weaves permit multiway communication. Even though some versions of Unix implement “named pipes,” no shell in common use permits anything other than linear filter chains. A weave may be an arbitrary multigraph that permits “many-to-many” communica- Weaves Weaves are interconnected networks of tool fragments, wherein data objects serve as the coin of information exchange. A tool fragment is a small software component, on the order of a procedure, that performs a single, well-defined function. Tool fragments consume objects as inputs and produce objects as outputs. Each tool fragment executes as an independent thread, a lightweight concurrent process that shares memory with others of its kind. Objects are transmitted from one tool fragment to another via ports attached to queues. Ports connect tool fragments to queues, which in turn buffer and synchronize communication among tool fragments. Formally, weaves are bipartite, directed multigraphs. Figure 2 is a trivial example of such a graph. The objects flowing through a weave, and the ports and queues through which they travel, are passive. Only the tool fragments are active, accepting objects from ports, invoking the methods implemented by the objects, performing tool-specific computations, and passing objects downstream. We have implemented in C++ a prototype environment for constructing, executing, and observing weaves. The weave and its components, tool fragments, ports, and queues are all objects, as are the data passing through the weave. A “snapshot” of a small running weave is shown in Figure 3. zyxwvutsr 25 zyxwvuts zyxwvut zyxwvutsrq zyxwvutsrqp zyxwvutsr zyxwvutsrq work. Thus the overhead of an object moving through the weave is amortized to a greater degree than is the corresponding overhead of classical data flow. The same datum m a y be processed simultaneously at diflerent points an the weave. In classical data flow, a datum never experiences a side effect; instead, a copy is produced. Weaves permit side effects and the same object may be referenced simultaneously by multiple tool fragments. Because synchronization and conflict resolution are the responsibility of the object itself, the implementor of the tool fragment is freed from such concerns. Each tool fragment is written as though it had sole access to the object; this makes each fragment easier to implement and comprehend than an alternate formulation where concurrency must be managed by the programmer. Finally, since all objects (except those representing primitive data) are denoted by reference, moving an object through a weave is inexpensive. Each tool fragment m a y retain state. Since each tool fragment is an arbitrary deterministic program, it may retain state. Local state, combined with the conditional consumption or production of objects, permits tremendous flexibility in a fragment’s communication. Weaves and the Stream Machine [9] share the same level of processing granularity; however, they manage streams in significantly different ways. In the Stream Machine the streams retain complete history, all stream values are copied, writing is always nonblocking, and each stream is restricted to a single writer (with possibly multiple readers). Stream reads may block, and readers can request future values, blocking until they arrrive. The stream data values are restricted to simple fundamental data types, integer, real, character, and the like. Finally, the stream machine is modeled on multiple separate processes, each with its own distinct memory space. With weaves, streams do not retain history, stream data values (with the exception of simple primitive data types) are passed from place to place by reference (except where they cross memory space or machine boundaries), both reading and writing are nonblocking, and each stream is permitted multiple readers and writers. No reader may request future values, and data values are arbitrary objects. Finally, weaves are explicitly designed to exploit multiple threads that share a single memory space. In passing we note that TDFL, the Task-level Data Flow Language [lo], also supports coarse-grained data flow and a level of task granularity equivalent to that of weaves. However, TDFL adheres strictly to the semantics of classical data flow. tion among tool fragments. Weave streams are structured. Unix pipes reduce communication to the lowest common denominator streams of uninterpreted bytes. On the other hand, tool fragments in weaves communicate by using streams of objects, which in a shared-memory tasking model are as efficient as byte streams. Since each object can encapsulate a large amount of data and many procedural interpretations, the effective bandwidth of an object stream is much higher than that of a flat byte stream. In classical data flow the unit of execution is a single machine instruction, and the data flowing through the network consist of simple machine-level values. Instructions execute only when all of their operands have arrived. While this scheme is conceptually attractive, its overhead is prohibitive. Despite resembling classical data-flow graphs, weaves differ from them in many ways as follows: Weaves are indifferent t o ihe composition of the tool fragments. Tool fragments may be derived from various languages and programming models. In this respect, weaves are an integrative glue for disparate serial codes, in the spirit of the Program Composition Notation [3, 41 or Durra [5, 61. In fact, the Durra application-specification language could be easily adapted, with minor modifications, to describe weaves, which themselves can be regarded as a lowerlevel implementation medium. Weave tool granularity is larger. Weave tool fragments execute from hundreds t o thousands of instructions for each set of input objects. This level of execution granularity is well-suited to shared-memory multiprocessors and operating systems that support threads such as Mach [7] or Synthesis [SI. Weave tool-fragment scheduling i s not data driven. In the classical data-flow model, instructions are blocked until all of their operands have arrived. Weaves do not enforce this restriction, but rather encourage fragments to exploit their input latency to perform useful computation. Tool fragment scheduling can be optimized in various ways to take advantage of the processing demands of the fragment and the arrival patterns of its inputs. e Weave data granularity is larger. Data flow typically deals with either primitive elements such as integers and booleans or simple uniform structures (arrays) of these primitive elements. In contrast, weaves process objects whose internal structure may be quite complex and whose methods represent substantial computation. At each stage in a weave, more effective work is accomplished than in an equivalent classical data-flow net- 26 I Quew zyxwvutsrqponm I zyx zyxwvutsrq zyxw zyxwvutsrq zyxwvuts Figure 4: Port and Queue Transport Service Figure 5: A Specialized Transport Service Weaves are primarily designed to foster the smooth, incremental construction of observable systems. The basic instrumentation for weaves must be so inexpensive and ubiquitous that no one would go to the bother of removing it. Additionally, it must be possible to include, at one’s discretion, more powerful (but potentially more expensive) mechanisms for observation. Observation, besides being cheap, should also be effortless; that is, it should be automatic without requiring additional work. Below we discuss the techniques used to achieve these goals. not prevent A from being connected to B. In this case, specialized ports, subclasses of the general port class, implement the necessary type coercion as a “valueadded” layer over the basic transport service. This concept is illustrated in Figure 5 . The transport service can also be used to resolve differences in representation (for example, transforming arrays stored in column-major order into row-major order). More generally, issues involving data translation across machine boundaries can be resolved at the transport service level transparently, between tool fragments executing on different processor architectures. The same principles can be applied to object transmission across machine boundaries, where the transport service can send an object transparently by transmitting only its internal state and reconstituting it on the receiving end (similarly, Durra [5] uses queues to implement data translation across host boundaries). Ports are also responsible for the atomicity of transfers of data objects to and from queues. Once a tool fragment “sends” through a write port to a queue, delivery to the queue is guaranteed. Conversely, a data object remains in a queue until transfer to a tool fragment via a read port is guaranteed. A data object can be in only one of two places at any one time - inside a tool fragment or a queue. This atomicity makes it possible to rearrange the network dynamically without disturbing the flow of data objects. Figure 6 illustrates this procedure, where tool fragments A and C are connected via a singleproducer/single-consumer queue, & I . A’s write port and C’s read port are detached from & I . Any data objects in transit are either already in the queue or waiting in the ports. Tool fragment B is inserted between fragments A and C by reconnecting the write port leading from A to a queue Q2,which feeds B instead. B’s write port, in turn, is attached to the queue feeding C. This reweaving is transparent to A , B , and C, since they all communicate blindly. 2.1 Ports and Queues Ports and queues are a blind, type-indifferent, twolayer transport service. No tool fragment using this service can discover the source (or destination) of the data objects it receives (or sends). Ports implement the wrapping and unwrapping of data objects by means of envelopes. Queues are finite buffers of envelopes. Envelopes are wrappers that hide the type of the underlying data object. This arrangement is illustrated in Figure 4. Blindness and type indifference increase the flexibility of interconnectivity. Since with weaves all communication is blind, tool fragments exhibit location transparency; that is, the actions of a tool fragment are independent of its location in the network. Each of these properties helps to ensure that a weave remains malleable irrespective of its size. (Note that Unix tools also exhibit location transparency.) Since the transport mechanism is type-indifferent , it is easy to construct specialized transport services that reflect the type coercions common to many programming languages. This increases the degree of tool fragment “plug compatibility,’’ since intercommunicating fragments need not be strictly type-compatible. For example, the fact that tool fragment A produces only integers and tool fragment B consumes only floats does zyxwvut zyxwvutsrqp 27 zyxw zyxwvutsrqp signed to a fragment for the lifetime of that fragment. IAl- Figure 6: Splicing a Tool Fragment into a Weave All tool fragments implement a minimal set of control methods. S t a r t initiates the execution of the fragment. Suspend halts the fragment in a timely manner, leaving it in a safe state from which it may be transparently continued. Resume is the converse of Suspend. The functional behavior of a resumed tool fragment is identical to that of a fragment that was never suspended in the first place. Sleep suspends the tool fragment for a minimum delay (given in milliseconds). Abort immediately halts the fragment but does not leave it in a state from which it may be safely restarted; Abort is the control action of last resort. zyxwvutsrq The transport service also separates the specification of communication from that of synchronization. Queues are distinguished by the producer/consumer relationships they support: 1 / 1 , l / n , m / l , or m / n . Ports are blind to the producer/consumer relationship supported by the queue to which they are attached. Consequently, the synchronization of multiple readers or writers is the responsibility of the queue; this permits efficient implementations that cater to special cases. For each producer/consumer relationship, a queue subclass is defined that exploits optimistic synchronization [8] to implement efficiently those queue operations that minimize locking. All queues are finite and all sends or receives on ports are nonblocking. For example, if a queue is full then a send on a port connected to that queue will fail and an error code is returned to the tool fragment undertaking the send. Each port supplies a Wait method (an implementation of some port-specific waiting policy) that will temporarily suspend the fragment. The tool fragment may wait or exploit the stream latency to perform other computations. If the fragment chooses to wait, it is free to reattempt the transmission; however, there is no guarantee that the condition that caused the transmission failure has been resolved. For example, a tool fragment can achieve the effect of a blocking send by repeatedly Waiting after each unsuccessful send. Blocking receives are constructed in an analogous fashion. 2.2 Tool Fragments and Threads Tool fragments are the active agents in weaves. Each tool fragment is composed of a routine, a vector of arguments, and a thread that executes the tool fragment. The routine is the working part of the tool fragment; it is responsible for processing incoming data objects and producing outgoing data objects. The argument vector is divided, by convention, into three subvectors. The first i arguments are those arguments that are not ports. The next j arguments are read ports. Finally, the last k arguments are write ports. The thread is as There are two major families of tool fragments. The first is designed to support the incorporation of foreign routines (written in common sequential programming languages) into weaves. In this case the routine is insulated from all of the details of sending or receiving data objects through ports. When the tool fragment is created, a mapping is given that specifies the rela, tionship between the argument vector of the fragment and the parameters of the routine. When inserted into a weave, the instance waits for data objects to arrive at the read ports. After all of the inputs have arrived, the routine is invoked with its parameters, which are extracted as specified by the argument mapping. The return value and output parameters of the routine are transshipped by the tool fragment through its output ports. All of the weave-related activity is invisible to the routine in a manner similar to that of TDFL [lo]; that is, the routine is called whenever a full set of input data objects have arrived. All of the control methods are implemented by the fragment on behalf of the foreign routine. As a simple metric this family maintains a count of the number of times the foreign routine is invoked. This measure is treated as a rough indicator of the total work performed by the fragment. The second family is designed for tool fragments that were implemented from scratch as weave components. Here no effort is made to insulate the routine from the weave. In this family the routine is invoked just once upon tool fragment S t a r t . It is the routine’s responsibility to unpack the argument vector, collect objects from read ports, process the objects, and then pass them on to write ports. In addition, the routine must respond to control requests. For example, in this family a Suspend only sets a boolean flag indicating that a suspend request is outstanding. The routine is obligated to check the flag regularly, as well as to suspend itself in a clean manner suitable for resumption. zyxwvutsrq zyxwvutsr 3 Weaving zyxwv mediates all of the environment services, including the instrumentation of programs, data collection, analysis, and presentation. The weaves constructed by the Gauge weaver are composed of small tools interconnected by data streams. Although the Gauge d a t a flow networks resemble weaves in the abstract, they are substantially different in detail and execution. Every action within Gauge is eventually translated into a relationship between inputs and desired outputs. The weaver must stitch together a weave that consumes the inputs, produces the desired outputs, and satisfies the relationship. The individual tools comprising the weave are described in terms of their input/output behavior. Each tool is characterized according to the types of its inputs and outputs and the properties they have. Some tools can be used in a variety of ways. There is one description for each of the possible tool applications, so that the same tool might be mentioned repeatedly in a related series of descriptions. The weaver works backwards from the goal outputs, by locating tools that can produce the desired outputs and satisfy the required input/output relationships. The inputs of these tools become the new goal outputs, and the weaver recursively tries to satisfy the new requirements by connecting the outputs of each newly discovered tool to the corresponding inputs of the consuming tool. Weaving terminates when all of the original inputs are being used as inputs to some tool in the weave and all relationships are satisfied. As a heuristic, if there are several weaves that fully satisfy the constraints, then the smallest weave is selected. If there are still several satisfactory weaves remaining, then the first weave that was constructed is selected. The Gauge weaving was done “on the fly” for every significant action. Contrary to our intuition, the cost of the dynamic weaving was insignificant (< 1%) in comparison to the total cost of executing the resulting weave. Furthermore, the use of the weaver had an important beneficial effect on the internal organization of Gauge. It was possible to make significant changes to Gauge by simply modifying the tool descriptions to reflect the addition of new tools or changes to existing tools. We would typically breadboard services by adding new tool descriptions (without bothering to implement the tools) and then request the weaver to create a weave to satisfy the service but not execute the weave. To satisfy the input requirements of new tools, in many cases the weaver was able to exploit existing tools in unanticipated ways, even at times making the weave smaller than those we had constructed by hand. One can assemble weaves in a variety of ways: manually, using a specification language to specifying individual software components and their interconnections; interactively, by using a graphical editor to sketch the weave, selecting its components from libraries of components; or automatically, by leaving the details of component selection and interconnection to the discretion of an automated agent, a weaver. All of these different methods of assembly, ranging from “hand-coded” to fully automatic, could be intermixed. For example, during rapid prototyping a system architect could direct the weaver to assemble subweaves automatically, using the graphical weave editor afterwards to connect the pieces. During system implementation, critical portions of a weave could be manually optimized by means of the specification language and then stored in the weave library for later use. Finally, during system test and integration, the test team could use the graphical weave editor to insert a test instrument into a specific communication path, to verify that a system requirement is satisfied. Graphical dataflow editors have appeared in a number of other contexts, and we will not discuss these editors here. The IDeA design environment [ll], TDFL [lo], PegaSys [12], and VERDI [13] allow a user to sketch a data flow task interactively, while the environment performs the clerical tasks of maintaining network consistency, recording designs, and performing a variety of design checks. Several commercial products for scientific visualization, such as ConMan [14] or AVS [15], also have data flow editors for constructing largegrain data-flow graphs, where each node in the graph executes as a separate heavyweight process. In the current implementation, weaves are objects and their methods constitute a simple assembly language for specifying weaves. In contrast, a weaver frees users to state their requirements in terms of high-level goals. A weaver then selects and interconnects software components that satisfy these goals. A similar planning component has been implemented in the IDeA environment [ll]. We have been experimenting with weavers since early 1988 and have used one as the central orga nizing agent in a large, complex performance-analysis environment called Gauge [16]. We describe the role of the weaver in Gauge and then briefly outline how we intend to deal with the scaling issues that a large, robust weaver faces. Gauge is an environment for the performance analysis of logic programs. The weaver embedded in Gauge zy zyxwvuts zyxw zyx 29 zyxwvuts In other words, the weaver acted as an integrative glue for Gauge, so that new tools or services could be added without concern for the details of interconnection or integration. The implementor of a tool was largely relieved of the burden of worrying about how the tool fit into the rest of Gauge (while still adhering to the data protocol conventions of inter-tool communication). Instead, the problem of providing new services was narrowed to local concerns, the construction of a single tool for performing a well-defined processing task. Gauge’s tool descriptions covered from 50 to 80 tools and the largest weaves the Gauge weaver ever produced contained from 15 to 20 tools. Encouraged by the Gauge experience, we are now investigating techniques for scaling weavers to produce weaves containing 50 to 100 tool fragments selected from repositories containing thousands of tool fragment descriptions. We expect the weaver t o be used in concert with graphical weave editors and hand-assembled weaves. However, even in this limited role, weavers, backed by adequate tool-fragment libraries, may improve the productivity of software engineers. Finally, systems containing an embedded dynamic weaver will probably be the exception, and we expect that most systems will be woven statically and then assembled. Our second-generation weaver will be based on the paradigm of partial-order programming [17, 181. Partial-order programming expresses computa tion declaratively, as statements of ordering. Each partial-order program specifies a domain D of values with partial order E; a set B 2 D of objects; and a set of ordering constraints C of the form U C ‘U, where U , ’ U E B. A semantics for a partial-order program is an assignment of values in D to each object B such that all constraints in C are satisfied. A partialorder programming problem is a statement of the form “minimize g subject to P,” where g E B is a specific object and P is a partial-order program as described above. A solution of a partial-order programming problem is a semantics for P that minimizes g. There may be many, one, or no solutions. It is profitable to view specifications of the properties and functional behaviors of tool fragments as s y s tems of constraints over partial orders, and a weave as the solution to a partial-order programming problem. Partial orders are powerful organizing mechanisms in knowledge representation. Ordering underlies type hierarchies, generalization or abstraction, composition, aggregation, and causal relationships. Tool fragments will be described by constraints that delimit their lo- cation in a multidimensional universe of partial orders. This application of partial-order programming is a generalization of the domain characteristics of IDeA [ll] and of the concept of facets [19] for cataloging reusable software components. In addition, there is a duality between weaves and partial-order programs, where tool fragments are equivalent to systems of constraints over partial orders, and interconnections among tool fragments are equivalent to common variables shared by separate systems of constraints. In this view, constructing a weave is equivalent to growing a system of constraints from a fixed pool (library) of subsystems of constraints, so that the system (weave) always remains consistent. This approach will permit the concise description of tool fragments, and algorithms for resolving partial-order programming problems can be directly applied to generating weaves. zyxwvutsrqpo 4 Analysis of Weaves The structure of weaves aids us in understanding their dynamic behavior; this understanding is achieved by capturing and analyzing information produced by the weave itself. In this section we discuss the mechanisms for controlling weaves and capturing information, after which we turn to their use in analysis. The principal mechanisms for the analysis of weaves are the following: 0 Self-metric tool fragments. In weaves, permanent low-overhead instrumentation is the norm. The authors of tool fragments embed inexpensive instrumentation in their software [ZO] and provide access methods (exploited by observers, described below) with which to extract the needed information. 0 Instruments. Instruments are specialized tool fragments whose sole purpose is to collect, transparently, information about the objects flowing through them. Inserted into a weave, instruments monitor objects that appear at the interfaces between tool fragments. Instruments are therefore limited by the granularity of the tool fragments; the finer the granularity, the more information that can be captured. 0 Observers. Observers collect information from instruments, queues, and tool fragments and perform analyses. The separation of observation from data capture permits us to build low-overhead instruments that can be left in place permanently, while we are free to construct observers with varying precisions and costs. 0 Actors. Actors, the agents for regulating tool fragments and queues, provide the control needed to per- zyxwvut zyxwvutsrq zyxwvutsrqp zyxwvutsrqp zyxwvutsr zyxwvuts 30 zyxwvut zyxw zyxwvuts zyxwvutsrq zyxw the weave. form experiments or make observations. We explore each of these mechanisms in detail below. 4.2 4.1 Self-Metric Tool Fragments Instruments Instruments are additional tool fragments whose sole purpose is to monitor data object flow at some flow boundary. Instruments must adhere to flow transparency; that is, they may inspect, but not modify, the objects passing through them. In other words, they are side-effect-free with respect to these data objects. Consequently, instruments form a subclass of tool fragments whose behavior is quite simple: they receive objects from read ports, perform simple counting or classification functions, and then send the objects on to write ports. Figure 7 shows a simple weave composed of two tool fragments with an instrument interposed (transparently) between them. Often the author of a tool fragment can anticipate that some metric will be useful in analyzing weaves containing that fragment. Software that contains embedded instrumentation is called self-metric [20]. Typically only the simplest and cheapest of mechanisms are used, such as boolean flags or counters. Despite their obvious limitations, these measures can be sufficient to obtain sophisticated analytical results [21]. Each tool fragment supplies access methods for retrieving the data collected by these flags and counters. Since tool fragments are objects, the class hierarchy of tool fragments can be used to define families of tools that provide the same functionality but with varying degrees of instrumentation. For example, all tool fragments inherit a method that reports whether the tool fragment is active or suspended. Various tools elaborate this by reporting on intermediate transitions. This instrumentation is not limited to tool fragments. Queues are also self-metric. Functional families of queues provide information at varying levels of detail. The base queue class supplies a method that reports the number of objects enqueued in a queue. More-specialized classes, by calculating the size of each waiting object, report the total amount of data enqueued. Other queue classes supply methods that report on the types or average waiting time of the objects being transported. Since each tool fragment runs as an independent thread and concurrently updates its state or sends and receives data objects, access contention over embedded measures can arise. The contention is of the singlewriter, multiple-reader variety; a tool fragment updates the state information, but many external observers may read it. Because we tolerate imprecision, locking is unnecessary. For example, acquiring the length of a queue does not interfere with reading or writing that queue, since no observer requires that the exact length be known. Sacrificing precision is an important means of reducing the overhead associated with instrumentation. Observers must also tolerate inconsistency, since tool fragments, instruments and the like may update state variables asynchronously during their inspection by an observer. Consequently, conservative observers may be limited in the accuracy of their analyses. Users are free to substitute more accurate observers at the price of higher overheads and greater interference with @ @ myweave Figure 7: Example Weave The simplest example of an instrument is the genericobject flow meter, which counts the total number of objects that have passed through it. It receives encapsulated objects (envelopes), increments a counter, and sends the objects on again. By deliberately ignoring the contents of the envelopes, flow meters monitor streams without incurring the overhead caused by the repeated decapsulation and encapsulation of data objects. A more-sophisticated instrument classifies the objects according to their types. Within the instrument, each object type has an associated counter. As each data object is received, its type is examined, the appropriate counter is incremented, and the object is passed on. Processing is limited to reading the type informa tion written on an envelope, but without opening the envelope. These two examples illustrate how a hierarchy of increasingly specialized instruments can be built. The most general instruments are oblivious to object types; they can therefore be inserted in weaves at any level and remain unchanged as object types become more refined. This generality is particularly useful as weaves are successively refined by the replacement of tool fragments 31 zyxwvutsr zyxwvutsrqp zyxwvutsr with subweaves. Although more-specialized instruments provide more type-specific information about the objects they monitor, they are limited accordingly in their general utility. Tradeoffs among the generality, utility, and cost of instrumentation are readily supported by the mechanisms outlined above. 4.3 per second is sufficient for weave debugging and coarse performance analysis. 0bservers The instruments discussed above are only collection mechanisms- they do not analyze or display the information in any way. This partitioning of responsibility ensures that instrumentation overhead does not significantly interfere with the performance of a weave. Observers are agents whose sole purpose is to analyze and present the measures collected by instruments, queues, and self-metric tool fragments. Three considerations dictate the overhead imposed by an observer: the frequency of measure collection, the complexity of the analysis function, and the presentation of the analysis. A variety of mechanisms exist for triggering observations, with each mechanism supporting a different frequency of collection and therefore representing some tradeoff between cost and precision. Here precision means information that is consistent with the internal state of tool fragments and queues. An observation that is triggered every time the state of a tool fragment changes can introduce significant overhead but be capable of yielding very precise information. Conversely, an observation that is triggered externally by a sampler enjoys a lower overhead but may be less precise. The separation of the observers from instruments, and the provision of a triggering mechanism for observations, allow us to implement analysis tools that have a full range of precisions and overheads. Users choose the precision appropriate to their needs and pay the corresponding price in overhead. To illustrate the importance of this separation, consider an observer that monitors queue length. If updates are posted to an observer every time an item is added to or removed from the queue, the observer will always have information consistent with the state of the queue. However, with message-passing rates on the order of thousands of messages per second, the overhead is prohibitive. Instead, if the same observations are triggered only periodically, the overhead is small but inconsistencies may arise. Often we overestimate the amount of precision required; that is, we worry needlessly about the effects of inconsistencies. For example, contrary to intuition, in our experience a “queue-length observer” that samples queue lengths only a few times Figure 8: Counts and a Histogram The uppermost display in Figure 8 is the output of an observer that periodically samples the count maintained by a generic flow-meter instrument and updates a display window. The lower display shows a more sophisticated observer that displays counts of data object types and a simple histogram of their distribution. Figure 9: Animation of a Weave Our most sophisticated observer is one that animates weaves. The animator constructs a representation of a weave as a network of tool fragments and queues. Figure 9 is a snapshot of the output of the animator. Each tool fragment, watched by its own tool observer, is represented by a labeled rectangle. Each queue, watched by its own queue observer, is represented by an openended box. The numbers at the upper left and upper right corners of each queue icon are the queue’s capacity and occupancy, respectively. The shaded portion of a queue icon is a histogram of its relative occupancy. The weave animator periodically triggers observations 32 zy 5 by each tool and queue observer. After interrogating the status of the corresponding tool, each tool observer updates its display of tool status. Similarly, queue observers update their queue icons after interrogating their queues. Although the individual actions of the animator and its component observers are very simple, the resulting animation is a complex and very useful debugging tool. Our experience in building weaves indicates that simple visualization tools can help one quickly identify subtle flaws that might otherwise go undetected. 4.4 Experience and Future Work We constructed a prototype implementation in C++, running on a Sun 3 under Unix, of the facilities described in the previous sections. The threads were implemented as a C++ layering over the Sun Lightweight Process Library [22]. The graphics facilities used by observers and actors were built on top of Interviews [23], an object-oriented widget set implemented in C++ to run on X Windows. A number of tool fragments have been implemented for processing satellite telemetry. A variety of instruments and observers have also been implemented; these are roughly split equally between those generic to weaves and those specific to the satellite telemetry domain. The animator has been used extensively as a debugging and performance analysis tool. We have constructed several dozen weaves, each dealing with some aspect of telemetry processing. The performance of weave message passing is about 6,000 objects/second, and their throughput is high enough that they have been used for the nontrivial filtering of satellite telemetry, delivering data rates in excess of 100 kilobits/second. Our initial experience has been so favorable that we are attempting to engineer a ten-fold increase in the performance and size of weaves. We recently ported the thread package to Mach, whose kernel-level support for threads should dramatically improve the performance of weaves. Preliminary experiments show that weaves with over 100 threads perform adequately. In contrast, the prototypes described above have been limited to 10-20 threads. The full Mach implementation of weaves will be demonstrated in mid-1991. We plan to use weaves to build a collection of tools for the temporal analysis of extremely large (gigabyte) event-trace streams. Simultaneously, a separate project will be underway to use weaves as the foundation for an engineering workbench for the development and exploration of alternative satellite telemetry-processing algorithms. Other avenues of investigation include the re-engineering of aging software into weaves, and astrodynamical modeling using weaves. Our research will concentrate on the specification of tool fragments and weaves as partial-order programs, the development of a weaver based on partial-order programming, the automated performance analysis of weaves, and the improved visualization of weaves and instruments. In summary, weaves are interesting in several respects: they use the object as the fundamental unit of ex- zyxwvutsrq zyxwvutsrqp Actors The analysis and debugging of weaves requires some level of control over the execution of the weave. Actors regulate weaves by translating high-level control requests into low-level control commands to individual tool fragments and queues. Control is partitioned in a manner analogous to that of observers. Tool fragments and queues are responsible for implementing low-level control, while issues of presentation and interaction are the responsibility of the actor. For example, the base class of tool fragments defines methods that guarantee a minimal level of control, that is, suspension and reactivation. The weave animator discussed above (see Figure 9) contains one actor for every tool fragment in the weave. The state of each tool fragment is managed by its actor and is presented as an icon - the idle and running figures, denoting suspended and active states, respectively. Clicking on a running (or idle) figure causes the tool fragment to be suspended (or resumed) by the actor. Weaves employ specialized tool fragments, known as “border crossings,” for debugging, and specialized actors, known as “border guards,” for controlling the crossings. Each border-crossing tool fragment holds a data object for inspection until the user signals that it may pass on. A generic border crossing can determine the class and size of the data object, and, for a number of primitive classes, can even extract the value of the data object. The generic border guard can display each of the properties extracted by the crossing and control the passage of objects through the crossing. Specialized crossings and guards can be constructed for application-specificdata objects. Like instruments, border crossings can be left in place at critical junctions for the lifetime of the weave. This partitioning of responsibility and expense allows users to use sophisticated debugging tools without incurring excessive overhead. zyxwvuts zyxwvuts zyxwvuts 33 zyxwvutsr zyxwvutsrqp zyxwvutsrq change in a data flow network; 0 they extensively use lightweight processes to implement large networks of cooperating tools; 0 they systematically exploit tool fragment boundaries as the natural sites for measurement and instrumentation; 0 they ubiquitously distribute low-overhead instrumentation that is so inexpensive it can be left in place over the lifetime of the system; 0 they provide for the transparent, dynamic rearrangement of the network for the insertion of instruments or subweaves; 0 they permit debugging and analysis at all points during system development, up to and including the deployment of a production weave; 0 they can be partly synthesized automatically from high-level specifications; and 0 they have the potential for parallel execution, on either shared-memory multiprocessors or distributed architectures. To our knowledge the combined application of these advantages to the systematic construction of streamoriented systems is novel. Over the next several years we intend to make a substantial research investment in weaves and their supporting technologies. [8] H. Massalin and C. Pu, “Threads and input/output in the Synthesis kernel,” in Proceedings of the Twelfth ACM Symposium on Operating System Principles, pp. 191-201,Dec. 1989. SIGOS Operating Systems Review, 23(5). [9] P. Barth, S. Guthery, and D. Barstow, “The stream machine: A data flow architecture for real-time applications,” in Proceedings 8th International Conference on Software Engineering, pp. 103-110,IEEE Computer Society Press, Aug. 1985. [lo] P. A. Suhler, J. Biswas, K. M. Korner, and J. C. Browne, IITDFL: A task-level dataflow language,” Joumal of Distributed and Parallel PTogramming, vol. 9, pp. 103-115, June 1990. [ll] ings 11th International Conference on Software Engineering, pp. 23-32,IEEE Computer Society Press, May 1989. [12] M. Moriconi and D. F. Hare, “The PegaSys system: Three papers,” Computer Science Laboratory Report 145,SRI International, Menlo Park, CA 94025-3493,Sept. 1985. [13] V. Y. Shen, C. Richter, M. L. Graf, and J. A. Brumfield, “Verdi: A visual environment for designing distributed systems,” Joumal of Distributed and Parallel Programming, vol. 9,pp. 128-137,June 1990. [14]P. E. Haeberli, “ConMan: A visual programming language for interactive graphics,’’ Computer Graphics, vol. 2 2 , pp. 103-111,Aug. 1988. [15] C. Upson et al., “The application visualization system,” IEEE Computer Graphics and Applications, vol. 9,pp. 3042,July 1989. zyxwvutsrqpon zyxwvutsrqp zyxwvutsrqpo zyxwvutsrqpo zyxwvutsrqpo [16] M. M. Gorlick and C. F. Kesselman, “Gauge: A workbench for the performance analysis of logic programs,” in Logic Programming: Proceedings of the Fifth International Conference (R. A. Kowalski and K. A. Bowen, eds.), pp. 548561,MIT Press, Aug. 1988. References [l] D. Ritchie and K. Thompson, “The UNIX time-sharingsystem,” The Bell System Technical Joumal, vol. 57,pp. 19051930,July-August 1978. (21 M.D. Lubars, “The IDeA design environment,” in Proceed- [17] D. S. Parker, “Partial order programming,” Tech. Rep. CSD-870067,Computer Science Department, University of California, Los Angeles, Dec. 1987. R. M. Karp and R. E. Miller, “Properties of a model for parallel computation: Determinacy, termination, queuing;,” SIAM Journal of Applied Mathematics, vol. 14,pp. 13901411,Nov. 1966. [18] D. S. Parker, “Partial order programming,” in Proceedings of Sixteenth Annual A C M SIGACT-SIGPLAN Symposium on Principles of Programming Language#, pp. 1-7, ACM Press, Jan. 1989. K. M. Chandy and S. Taylor, “The composition of parallel programs,” in Proceedings of the 1989 International Conference on Supercomputing, Nov. 1989. [19] R. Preito-Diaz and P. Freeman, “Classifying software for reusability,” IEEE Software, vol. 4,pp. 6-16, Jan. 1987. K. M. Chandy and S. Taylor, “Examplesin program composition,” in Beauty is Our Business - A Birthday Salute t o E. W . Dijkatra (W. Feijen e t al., eds.), pp. 94-101,SpringerVerlag, 1990. [20] L. Stuck, “Automatic generation of self-metric software,”in Proceedings 1979 IEEE Symposium on Computer Software Reliability, pp. 94-100, 1973. M. Barbacci and J. Wing, “A language for distributed applications,” in Proceedings of the 1990 International Conference on Computer Languages, IEEE Computer Society, Mar. 1990. [21] M. M. Gorlick and C. F. Kesselman, “Timing Prolog p r s grams without clocks,“ in Proceedings 1987 Symposium on Logic Programming, pp. 426-432,IEEE Computer Society Press, 1987. M. Barbacci, D. Doubleday, and C. Weinstock, ”Application-levelprogramming,” in Proceedings of the 10th International Conference on Distributed Computing Systems, pp. 458-465, IEEE Computer Society, May 1990. [22] Sun Microsystems, Lightweight Processes Library, Nov. 1987. Sun Release 4.0. [23] M. A. Linton, P. R. Calder, and J. M. Vlissides, “The design and implementation of Interviews,” in Proceedings of the USENIX C++ Workshop, (Sante Fe, New Mexico), Nov. 1987. D. L. Black, “Scheduling support for concurrency and parallelism in the Mach operating system,” IEEE Computer, vol. 23,pp. 3543,May 1990. 34