From Turing and von Neumann to the Present

.—. HISTORICAL PERSPECTIVE From Turing and von Neumann to the Present by Necia G. Cooper automaton—a mechanism that is relatively self-operating; a device or machine designed to follow automatically a predetermined sequence of operations or respond to encoded instructions. The notion of automata in the sense of machines that operate on their own from encoded instructions is very ancient, and one might say that mechanical clocks and music boxes fall under this category. The idea of computing machines is also very old. For instance. Pascal and Leibnitz outlined various schematics for such machines. In the latter part of the 18th century Baron de Kempelen built what was alleged to be the first chess-playing machine. Remarkable as it appeared. alas, it was a fake operated by a person hidden within it! The modern theory of automata can be traced to two giants in the field of mathematics. Alan Turing and John von Neumann. These two men laid much of the logical foundation for the development of present-day electronic computers, and both were involved in the practical design of real computing machines. 22 Before World War II Turing had proved the logical limits of computability and on the basis of this work had designed in idealized terms a universal computer, a machine that could perform all possible numerical computations. This idealized machine is now known as a Turing machine. (All modern computers have capabilities equivalent to some of the universal Turing machines.) During World War H Turing successfully applied his logical talent to the real and urgent problem of breaking the Nazi intelligence code. a feat that played a crucial role in the Allied victory. Prior to World War II von Neumann was aware of Turing’s work on computing machines and realized how useful such machines would be for investigating nonlinear problems in mathematical physics, in particular. the fascinating problem of turbulence. Numerical calculations might, for example, elucidate the mysterious role of the Reynolds number in turbulent phenomena. (The Reynolds number gives roughly the ratio of the inertial forces to the viscous forces. A flow that is regular becomes turbulent when this number is about 2000.) He was convinced that the best mathematics proceeds from empirical science and that numerical calculation on electronic computers might provide a new kind of empirical data on the properties of nonlinear equations. Stan Ulam suggests that the final impetus for von Neumann to work energetically on computer methods and design came from wartime Los Alamos, where it became obvious that analytical work alone was often not sufficient to provide even qualitative answers about the behavior of an atomic bomb. The best way to construct a computing machine thus presented a practical as well as a theoretical problem. Starting in 1944 von Neumann formulated methods of translating a set of mathematical procedures into a language of instructions for a computing machine. Before von Neumann’s work on the logical design of computers, the few existing electronic machines had to be rewired for each new problem. Von Neumann developed the idea of a fixed “flow diagram” and a stored “code,” or program, that would enable a machine with a fixed set of connections to solve a great variety of problems. Von Neumann was also interested, as was Turing, in discovering the logical elements Fall 1983 LOS ALAMOS SCIENCE Cellular Automata HISTORICAL —. and organization required to perform some of the more general types of functions that human beings and other life forms carry out and in trying to construct, at least at an abstract level, machines that contained such capabilities. But whereas Turing was primarily interested in developing “intelligent” automata that would imitate the thinking and decision-making abilities of the human brain, von Neumann focused on the broader problem of developing a general theory of complicated automata, a theory that would encompass both natural automata (such as the human nervous system and living organisms) and artificial automata (such as digital computers and communication networks). What is meant by the term “complicated”? As von Neumann put it, it is not a question of how complicated an object is but rather of how involved or difficult its purposive operations are. In a series of lectures delivered at the University of Illinois in 1949, von Neumann explored ideas about what constitutes complexity and what kind of a theory might be needed to describe complicated automata. He suggested that a new theory of information would be needed for such systems, one that would bear a resemblance to both formal logic and thermodynamics. It was at these lectures that he explained the logical machinery necessary to construct an artificial automaton that could carry out one very specific complicated function, namely, self-reproduction. Such an automaton was also logically capable of constructing automata more complex than itself. Von Neumann actually began constructing several models of selfreproducing automata. Based on an inspired suggestion by Ulam, one of these models was in the form of a “cellular” automaton (see the preceding article in this issue by Stephen Wolfram for the definition of a cellular automaton). From the Illinois lectures it is clear that von Neumann was struggling to arrive at a correct definition of complexity. Although his thoughts were still admittedly vague, they LOS ALAMOS SCIENCE Fall 1983 do seem, at least in some respects, related to the present efforts of Wolfram to find universal features of cellular automaton behavior and from these to develop new laws, analogous to those of thermodynamics, to describe self-organizing systems. Von Neumann suggested that a theory of information appropriate to automata would build on and go beyond the results of Turing, Godel, Szilard, and Shannon. Turing had shown the limits of what can be done with certain types of information—namely, anything that can be described in rigorously logical terms can be done by an automaton. and, conversely. anything that can be done by an automaton can be described in logical terms. Turing constructed. on paper, a universal automaton that could perform anything that any other automaton could do. It consisted of a finite automaton, one that exists in a finite number of states, plus an indefinitely extendible tape containing instructions. “The importance of Turing’s research is just this:” said von Neumann, “that if you construct an automaton right, then any additional requirements about the automaton can be handled by sufficiently elaborate instructions. This is true only if [the automaton] is sufficiently complicated. if it reaches a certain minimum level of complexity” (John von Neumann, Theory of Self-Reproducing Automata, edited and completed by Arthur W. Burks, University of Illinois Press, 1966, p. 50). Turing also proved that there are some things an automaton cannot do. For example, “Y OU cannot construct an automaton which can predict in how many steps another automaton which can solve a certain problem will actually solve it. . . . In other words, you can build an organ which can do anything that can be done. but you cannot build an organ which tells you whether it can be done” (ibid., p. 51). This result of Turing’s is connected with Godel’s work on the hierarchy of types in formal logic. Von Neumann related this result to his notion of complexity. He suggested that for objects of PERSPECTIVE low complexity, it is easier to predict their properties than to build them, but for objects of high complexity, the opposite is true. Von Neumann stated that the new theory of information should include not only the strict and rigorous considerations of formal logic but also statistical considerations. The reason one needs statistical considerations is to include the possibility of failure. The actual structure of both manmade and artificial automata is dictated by the need to achieve a state in which a majority of all failures will not be lethal. To include failure, one must develop a probabilistic system of logic. Von Neumann felt that the theory of entropy and information in thermodynamics and Shannon’s information theory would be relevant. Szilard had shown in 1929 that entropy in a physical system measures the lack of information; it gives the total amount of missing information on the microscopic structure of the system. Entropy defined as a physical quantity measures the degree of degradation suffered by any form of energy. “There are strong indications that information is similar to entropy and that the degenerative processes of entropy are paralleled by degenerative processes in the processing of information” (ibid., p. 62). Shannon’s work focused on the problem of transmitting information. He had developed a quantitative theory of measuring the capacity of a communication channel, a theory that included the role of redundancy. Redundancy makes it possible to correct errors and “is the only thing which makes it possible to write a text which is longer than, say, ten pages. In other words, a language which has maximum compression would actually be completely unsuited to conveying information beyond a certain degree of complexity, because you could never find out whether a text is right or wrong” (ibid., p. 60). Von Neumann emphasized the ability of living organisms to operate across errors. Such a system “is sufficiently flexible and 23 HISTORICAL PERSPECTIVE well organized that as soon as an error shows up in any one part of it, the system automatically senses whether this error matters or not. If it doesn’t matter, the system continues to operate without paying any attention to it. If the error seems to be important, the system blocks that region out. by-passes it and proceeds along other channels. The system then analyzes the region separately at leisure and corrects what goes on there. and if correction is impossible the system just blocks the region off and bypasses it forever. . . . “To apply the philosophy underlying natural automata to artificial automata we must understand complicated mechanisms better than we do, we must have elaborate statistics about what goes wrong, and we must have much more perfect statistical information about the milieu in which a mechanism lives than we now have. An automaton cannot be separated from the milieu to which it responds” (ibid., p p . 71-72). From artificial automata “one gets a very strong impression that complication, or productive potentiality in an organization, is degenerative, that an organization which synthesizes something is necessarily more complicated. of a higher order, than the organization it synthesizes” (ibid., p. 79). But life defeats degeneracy. Although the complicated aggregation of many elementary parts necessary to form a living organism is thermodynamically highly improbable, once such a peculiar accident occurs. the rules of probability do not apply because the organism can reproduce itself provided the milieu is reasonable—and a reasonable milieu is thermodynamically much less improbable. Thus probability leaves a loophole that is pierced by self-reproduction. Is it possible for an artificial automaton to reproduce itself? Further, is it possible for a machine to produce something that is more complicated than itself in the sense that the offspring can perform more difficult and involved tasks than the progenitor? These 24 A three-dimensional object grown from a single cube to the thirtieth generation (dark cubes). The model shows only one octant of the three-dimensional structure. This figure and the two others illustrating this article are from R. G. Schrandt and S. M. Ulam, “On Recursively Defined Geometrical Objects and Patterns of Growth,” Los Alamos Scientific Laboratory report LA-3762, November 1967 and are also reprinted in Arthur W. Burks, editor, Essays on Cellular Automata, University of Illinois Press, 1970. questions arise from looking at natural automata. In what sense can a gene contain a description of the human being that will come from it? How can an organism at a low level in the phylogenetic order develop into a higher level organism? From his comparison of natural and artificial automata, von Neumann suggested that complexity has one decisive property, namely, a critical size below which the process of synthesis is degenerative and above which the process is explosive in the sense that an automaton can produce others that are more complex and of higher potentiality than itself. However, to get beyond the realm of vague statements and develop a correct formulation of complexity, he felt it was necessary to construct examples that exhibit the “critical and paradoxical properties of complication” (ibid., p. 80). To this end he set out to construct, in principle, self-reproducing automata, automata “which can have outputs something like themselves” (ibid., p. 75). (All artificial automata discussed up to that point. such as Turing machines, computing machines, and Fall 1983 LOS ALAMOS SCIENCE Cellular Automata HISTORICAL PERSPECTIVE the network of abstract neurons discussed by McCulloch and Pitts (“A Logical Calculus of the Ideas Immanent in Nervous Activity,” Bulletin of Mathematical Biophysics, 1943), (a) (b) A “contest’’ between two patterns, one of lines within squares (shaded) and one of dots within squares, growing in a 23 by 23 checkerboard. Both patterns grow by a recursive rule stating that the newest generation (represented by diagonal lines or by dots in an x shape) may occupy a square if that square is orthogonally contiguous to one and only one square occupied by the immediately preceding generation (represented by perpendicularly bisecting lines or by dots in a + shape). In addition, no piece of either pattern may survive more than two generations. Initially, the line pattern occupied only the lower left corner square, and the dot pattern occupied only the square immediately to the left of the upper right corner square. (a) At generation 16 the two patterns are still separate. (b) At generation 25 the two patterns engage. (c)At 32 generations the dot pattern has penetrated enemy territory. (d) At 33 generations the dot pattern has won the contest. LOS ALAMOS SCIENCE Fall 1983 had inputs and outputs of completely different media than the automata themselves.) "There is no question of producing matter out of nothing. Rather, one imagines automata which can modify objects similar to themselves, or effect syntheses by picking up parts and putting them together, or take synthesized entities apart” (ibid., p. 75). Von Neumann drew up a list of unambiguously defined parts for the kinematic model of a self-reproducing automaton. Although this model ignored mechanical and chemical questions of force and energy, it did involve problems of movement, contact, positioning, fusing, and cutting of elements. Von Neumann changed his initial approach after extensive discussions with Ulam. Ulam suggested that the proof of existence and construction of a selfreproducing automaton might be done in a simpler, neater way that retained the logical and combinatorial aspects of the problem but eliminated complicated aspects of geometry and motion. Ulam’s idea was to construct the automaton in an indefinitely large space composed of cells. In two dimensions such a cellular structure is equivalent to an infinite checkerboard. The elements of the automaton are a set of allowable states for each cell. including an empty, or quiescent, state, and a transition rule for transforming one state into another. The rule defines the state of a cell at time interval t+1 in terms of its own state and the states of certain neighboring cells at time interval t. Motion is replaced by transmitting information from cell to cell; that is, the transition rule can change a quiescent cell into an active cell. Von Neumann’s universal self-reproducing cellular automaton, begun in 1952, was a rather baroque construction in which each cell had twenty-nine allowable states and a 25 HISTORICAL PERSPECTIVE neighborhood consisting of the four cells orthogonal to it. Influenced by the work of McCulloch and Pitts, von Neumann used a physiological simile of idealized neurons to help define these states. The states and transition rules among them were designed to perform both logical and growth operations. He recognized. of course. that his construction might not be the minimal or optimal one, and it was later shown by Edwin Roger Banks that a universal selfreproducing automaton was possible with only four allowed states per cell. The logical trick employed to make the automaton universal was to make it capable of reading any axiomatic description of any other automaton, including itself, and to include its own axiomatic description in its memory. This trick was close to that used by Turing in his universal computing machine. The basic organs of the automaton included a tape unit that could store information on and read from an indefinitely extendible linear array of cells, or tape, and a constructing unit containing a finite control unit and an indefinitely long constructing arm that could construct any automaton whose description was stored in the tape unit. Realization of the 29-state self-reproducing cellular automaton required some 200,000 cells. Von Neumann died in 1957 and did not complete this construction (it was completed by Arthur Burks). Neither did he complete his plans for two other models of selfreproducing automata. In one, based on the 29-state cellular automaton, the basic element was to be neuron-like and have fatigue mechanisms as well as a threshold for excitation. The other was to be a continuous model of self-reproduction described by a system of nonlinear partial differential equations of the type that govern diffusion in a fluid. Von Neumann thus hoped to proceed from the discrete to the continuous. He was inspired by the abilities of natural automata and emphasized that the nervous system was not purely digital but was a mixed analog-digital system. 26 Much effort since von Neumann’s time has gone into investigating the simulation capabilities of cellular automata. Can one define appropriate sets of states and transition rules to simulate natural phenomena’? Ulam was among the first to use cellular automata in this way. He investigated growth patterns of simple finite systems, simple in that each cell had only two states and obeyed some simple transition rule. Even very simple growth rules may yield highly complex patterns, both periodic and aperiodic. “The main feature of cellular automata,” Ulam points out, “is that simple recipes repeated many times may lead to very complicated behavior. Information analysts might look at some final pattern and infer that it contains a large amount of information, when in fact the pattern is generated by a very simple process. Perhaps the behavior of an animal or even ourselves could be reduced to two or three pages of simple rules applied in turn many times!" (private conversation. October 1983). Ulam’s study of the growth patterns of cellular automata had as one of its aims “to throw a sidelight on the question of how much ‘information’ is necessary to describe the seemingly enormously elaborate structures of living objects” (ibid.). His work with Holladay and with Schrandt on an electronic computing machine at Los Alamos in 1967 produced a great number of such patterns. Properties of their morphology were surveyed in both space and time. Ulam and Schrandt experimented with “contests” in which two starting configurations were allowed to grow until they collided. Then a fight would ensue, and sometimes one configuration would annihilate the other. They also explored three-dimensional automata. Another early investigator of cellular automata was Ed Fredkin. Around 1960 he began to explore the possibility that all physical phenomena down to the quantum mechanical level could be simulated by cellular automata. Perhaps the physical world is a discrete space-time lattice of A pattern grown according to a recursive rule from three noncontiguous squares at the vertices of an approximately equilateral triangle. A square of the next generation is formed if (a) it is contiguous to one and only one square of the current generation, and (b) it touches no other previously occupied square except if the square should be its “grandparent. ” In addition, of this set of prospective squares of the (n+l)th generation satisfying condition (b), all squares that would touch each other are eliminated. However, squares that have the same parent are allowed to touch. information bits that evolve according to simple rules. In other words, perhaps the universe is one enormous cellular automaton. There have been many other workers in this field. Several important mathematical results on cellular automata were obtained by Moore and Holland (University of Michigan) in the 1960s. The “Game of Life,” an example of a two-dimensional cellular automaton with very complex behavior, was invented by Conway (Cambridge University) around 1970 and extensively investigated for several years thereafter. Fall 1983 LOS ALAMOS SCIENCE Cellular Automata HISTORICAL PERSPECTIVE Cellular automata have been used in biological studies (sometimes under the names of “tessellation automata” or “homogeneous structures”) to model several aspects of the growth and behavior of organisms. They have been analyzed as parallel-processing computers (often under the name of “iterative arrays”). They have also been applied to problems in number theory under the name “stunted trees” and have been considered in ergodic theory, as endomorphisms of the “dynamical” shift system. A workshop on cellular automata at Los Alamos in March 1983 was attended by researchers from many different fields. The proceedings of this workshop will be published in the journal Physica D and will also be issued as a book by North-Holland Publishing Co. In all this effort the work of Stephen Wolfram most closely approaches von Neumann’s dream of abstracting from examples of complicated automata new concepts rele- vant to information theory and analogous to the concepts of thermodynamics. Wolfram has made a systematic study of one-dimensional cellular automata and has identified four general classes of behavior, as described in the preceding article. Three of these classes exhibit behavior analogous to the limit points, limit cycles, and strange attractors found in studies of nonlinear ordinary differential equations and transformation iterations. Such equations characterize dissipative systems. systems in which structure may arise spontaneously even from a disordered initial state. Fluids and living organisms are examples of such systems. (Non-dissipative systems, in contrast, tend toward disordered states of maximal entropy and are described by the laws of thermodynamics.) The fourth class mimics the behavior of universal Turing machines. Wolfram speculates that his identification of universal classes of behavior in cellular automata may represent a first step in the formulation of general laws for complex self-organizing systems. He says that what he is looking for is a new concept—maybe it will be complexity or maybe something else—that like entropy will be always increasing (or decreasing) in such a system and will be manifest in both the microscopic laws governing evolution of the system and in its macroscopic behavior. It may be closest to what von Neumann had in mind as he sought a correct definition of’ complexity. We can never know. We can Acknowledgment I wish to thank Arthur W. Burks for permission to reprint quotations from Theory of Self-Reproducing Automata. We are indebted to him for editing and completing von Neumann’s manuscripts in a manner that retains the patterns of thought of a great mind. Further Reading John von Neumann. Theory of Self-Reproducing Automata. Edited and completed by Arthur W. Burks. Urbana: University of Illinois Press. 1966. Part I is an edited version of the lectures delivered at the University of Illinois. Part II is von Neumann’s manuscript describing the construction of his 29-state self-reproducing automaton. Arthur W. Burks, editor. Essays on Cellular Automata. Urbana: University of Illinois Press. 1970. This volume contains early papers on cellular automata including those of Ulam and his coworkers. Andrew Hodges, “Alan Turing: Mathematician and Computer Builder.” New Scientist. 15 September 1983, pp. 789-792. This contains a wonderful illustration, “A Turing Machine in Action.” Martin Gardner. "On Cellular Automata, Self-Reproduction, the Garden of Eden. and the Game ‘Life,’ “ Scientific American, October 1971. The following publications deal with complexity per se: W. A. Beyer, M. L. Stein, and S. M. Ulam. “The Notion of Complexity.” Los Alamos Scientific Laboratory report LA-4822. December 1971. S. Winograd. Arithmetic Complexity of Computations. Philadelphia: Society of Industrial and Applied Mathematics. 1980. LOS ALAMOS SCIENCE Fall 1983 27