Node:Top, Next:Preface, Previous:(dir), Up:(dir)

C Programming Tutorial (K&R version 4)

This is a C Programming Tutorial for people who have a little experience with an interpreted programming language, such as Emacs Lisp or a GNU shell. Edition 4.02 Copyright 1987,1999 Mark Burgess Permission is granted to make and distribute verbatim copies of this manual provided the copyright notice and this permission notice are preserved on all copies.

Preface: Introduction: Reserved words & example: Operating systems: Libraries: Programming style: Form of a C program: Comments: Functions: Variables: Parameters: Scope: Preprocessor: Pointers: Standard Output and Standard Input: Assignments Expressions and Operators: Decisions: Loops: Arrays: Strings: Putting together a program: Special Library Functions and Macros: Hidden Operators: More on Data Types: Machine Level Operations: Files and Devices: Structures and Unions: Data structures: Recursion: Example Programs chapter: Errors and debugging: Summary: reserved words list: Comparisons : Character Conversion Table:

Emacs style file: Answers to questions: Index:

Node:Preface, Next:Introduction, Previous:Top, Up:Top

Preface
Every program is limited by the language which is used to write it. C is a programmer's language. Unlike BASIC or Pascal, C was not written as a teaching aid, but as an implementation language. C is a computer language and a programming tool which has grown popular because programmers like it! It is a tricky language but a masterful one. Sceptics have said that it is a language in which everything which can go wrong does go wrong. True, it does not do much hand holding, but also it does not hold anything back. If you have come to C in the hope of finding a powerful language for writing everyday computer programs, then you will not be disappointed. C is ideally suited to modern computers and modern programming. This book is a tutorial. Its aim is to teach C to a beginner, but with enough of the details so as not be outgrown as the years go by. It presumes that you have some previous aquaintance with programming -- you need to know what a variable is and what a function is -- but you do not need much experience. It is not essential to follow the order of the chapters rigorously, but if you are a beginner to C it is recommended. When it comes down to it, most languages have basically the same kinds of features: variables, ways of making loops, ways of making decisions, ways of accessing files etc. If you want to plan your assault on C, think about what you already know about programming and what you expect to look for in C. You will most likely find all of those things and more, as you work though the chapters. The examples programs range from quick one-function programs, which do no more than illustrate the sole use of one simple feature, to complete application examples occupying several pages. In places these examples make use of features before they have properly been explained. These programs serve as a taster of what is to come.

Mark Burgess. 1987, 1999

This book was first written in 1987; this new edition was updated and rewritten in 1999. The book was originally published by Dabs Press. Since the book has gone out of print, David Atherton of Dabs and I agreed to release the manuscript, as per the original contract. This new edition is written in Texinfo, which is a documentation system that uses a single source file to produce both on-line information and printed output. You can read this tutorial online, using either the Emacs Info reader, the standalone Info reader, or a World Wide Web browser, or you can read this same text as a typeset, printed book.

Node:Introduction, Next:Reserved words & example, Previous:Preface, Up:Top

Introduction
What is C? What is it for? Why is it special?

Levels: Basic ideas: The compiler: Errors: Use of Upper and Lower Case: Questions 1:

Node:Levels, Next:Basic ideas, Previous:Introduction, Up:Introduction

High Levels and Low Levels

Any kind of object that is sufficiently complicated can be thought of as having levels of detail; the amount of detail we see depends upon how closely we scrutinize it. A computer falls definitely into the category of complex objects and it can be thought of as working at many different levels. The terms low level and high level are often used to describe these onion-layers of complexity in computers. Low level is perhaps the easiest to understand: it describes a level of detail which is buried down amongst the working parts of the machine: the low level is the level at which the computer seems most primitive and machine-like. A higher level describes the same object, but with the detail left out. Imagine stepping back from the complexity of the machine level pieces and grouping together parts which work together, then covering up all the details. (For instance, in a car, a group of nuts, bolts, pistons can be grouped together to make up a new basic object: an engine.) At a high level a computer becomes a group of black boxes which can then be thought of as the basic components of the computer. C is called a high level, compiler language. The aim of any high level computer language is to provide an easy and natural way of giving a programme of instructions to a computer (a computer program). The language of the raw computer is a stream of numbers called machine code. As you might expect, the action which results from a single machine code instruction is very primitive and many thousands of them are required to make a program which does anything substantial. It is therefore the job of a high level language to provide a new set of black box instructions, which can be given to the computer without us needing to see what happens inside them - and it is the job of a compiler to fill in the details of these "black boxes" so that the final product is a

sequence of instructions in the language of the computer.

C is one of a large number of high level languages which can be used for general purpose programming, that is, anything from writing small programs for personal amusement to writing complex applications. It is unusual in several ways. Before C, high level languages were criticized by machine code programmers because they shielded the user from the working details of the computer, with their black box approach, to such an extent that the languages become inflexible: in other words, they did not not allow programmers to use all the facilities which the machine has to offer. C, on the other hand, was designed to give access to any level of the machine down to raw machine code and because of this it is perhaps the most flexible of all high level

languages. Surprisingly, programming books often ignore an important role of high level languages: high level programs are not only a way to express instructions to the computer, they are also a means of communication among human beings. They are not

merely monologues to the machine, they are a way to express ideas and a way to solve problems. The C language has been equipped with features that allow programs to be organized in an easy and logical way. This is vitally important for writing lengthy programs because complex problems are only manageable with a clear organization and program structure. C allows meaningful variable names and meaningful function names to be used in programs without any loss of efficiency and it gives a complete freedom of style; it has a set of very flexible loop constructions (for, while, do) and neat ways of making decisions. These provide an excellent basis for controlling the

flow of programs. Another unusual feature of C is the way it can express ideas concisely. The richness of a language shapes what it can talk about. C gives us the apparatus to build neat and compact programs. This sounds, first of all, either like a great bonus or something a bit suspect. Its conciseness can be a mixed blessing: the aim is to try to seek a balance between the often conflicting interests of readability of programs and their conciseness. Because this side of programming is so often presumed to be understood, we shall try to develop a style which finds the right balance. C allows things which are disallowed in other languages: this is no defect, but a very powerful freedom which, when used with caution, opens up possibilities enormously.

It does mean however that there are aspects of C which can run away with themselves unless some care is taken. The programmer carries an extra responsibility to write a careful and thoughtful program. The reward for this care is that fast, efficient programs can be produced. C tries to make the best of a computer by linking as closely as possible to the local environment. It is no longer necessary to have to put up with hopelessly inadequate input/output facilities anymore (a legacy of the timesharing/mainframe computer era): one can use everything that a computer has to offer. Above all it is flexible. Clearly no language can guarantee intrinsically good programs: there is always a responsibility on the programmer, personally, to ensure that a program is neat, logical and well organized, but it can give a framework in which it is easy to do so. The aim of this book is to convey some of the C philosophy in a practical way and to provide a comprehensive introduction to the language by appealing to a number of examples and by sticking to a strict structuring scheme. It is hoped that this will give a flavour of the kind of programming which C encourages. Node:Basic ideas, Next:The compiler, Previous:Levels, Up:Introduction

Basic ideas about C

What to do with a compiler. What can go wrong. Using a compiler language is not the same as using an interpreted language like BASIC or a GNU shell. It differs in a number of ways. To begin with, a C program has to be created in two stages:

Firstly, the program is written in the form of a number of text files using a screen editor. This form of the program is called the source program. It is not possible to execute this file directly. Secondly, the completed source file is passed to a compiler--a program which generates a new file containing a machine code translation of the source text. This file is called an object file or executable file. The executable file is said to have been compiled from the source text.

Compiler languages do not usually contain their own editor, nor do they have words like RUN with which to execute a finished program. You use a screen editor to create the words of a program (program text) and run the final program in its compiled form usually by simply typing the name of the executable file.

The compiler: Errors:

Node:The compiler, Next:Errors, Previous:Basic ideas, Up:Introduction

The Compiler

A C program is made by running a compiler which takes the typed source program and converts it into an object file that the computer can execute. A compiler usually operates in two or more phases (and each phase may have stages within it). These phases must be executed one after the other. As we shall see later, this approach provides a flexible way of compiling programs which are split into many files.

A two-phase compiler works in the following way:

Phase 1 scans a source program, perhaps generating an intermediate code (quadruples or pcode) which helps to simplify the grammar of the language for subsequent processing. It then converts the intermediate code into a file of object code (though this is usually not executable yet). A separate object file is built for each separate source file. In the GNU C compiler, these two stages are run with the command gcc -c; the output is one or more .o files. Phase 2 is a Linker. This program appends standard library code to the object file so that the code is complete and can "stand alone". A C compiler linker suffers the slightly arduous task of linking together all the functions in the C program. Even at this stage, the compiler can fail, if it finds that it has a reference to a function which does not exist. With the GNU C compiler this stage is activated by the command gcc -o or ld.

To avoid the irritation of typing two or three separate commands (which are often cumbersome) you will normally find a simple interface for executing compiler. Traditionally this is an executable program called cc for C Compiler:

cc filename gcc filename

On GNU systems, this results in the creation of an executable program with the default name a.out. To tell the compiler what you would like the executable program to be called, use the -o option for setting the name of the object code:
gcc -o program-name filname

For example, to create a program called myprog from a file called myprog.c, write
gcc -o myprog myprog.c

Node:Errors, Next:Use of Upper and Lower Case, Previous:The compiler, Up:Introduction

Errors
Errors are mistakes which we the programmers make. There are different kinds of error: Syntax Errors in the syntax, or word structure of a program are caught before you run it, at compilation time by the compiler program. They are listed all in one go, with the line number, in the text file, at which the error occurred and a message to say what was wrong. For example, suppose you write sin (x) y = ; in a program instead of y = sin (x);, which assigns the value of the sin of x to y. Upon compilation, you would see this error message:
eg.c: In function `main': eg.c:12: parse error before `y'

(If you compile the program in Emacs, you can jump directly to the error.) A program with syntax errors will cause a compiler program to stop trying to generate machine code and will not create an executable. However, a compiler will usually not stop at the first error it encounters but will attempt to continue checking the syntax of a program right to the last line before aborting, and it is common to submit a program for compilation only to receive a long and ungratifying list of errors from the compiler. It is a shock to everyone using a compiler for the first time how a single error can throw the compiler off course and result in a huge and confusing list of non-existent errors, following a single true culprit. The situation thus looks much worse than it really is. You'll get used to this with experience, but it can be very disheartening. As a rule, look for the first error, fix that, and then recompile. Of course, after you have become experienced, you will recognize when subsequent error

messages are due to independent problems and when they are due to a cascade. But at the beginning, just look for and fix the first error. Intention Errors in goal or purpose (logical errors) occur when you write a program that works, but does not do what you intend it to do. You intend to send a letter to all drivers whose licenses will expire soon; instead, you send a letter to all drivers whose licenses will expire sometime. If the compilation of a program is successful, then a new file is created. This file will contain machine code which can be executed according to the rules of the computer's local operating system. When a programmer wants to make alterations and corrections to a C program, these have to be made in the source text file itself using an editor; the program, or the salient parts, must then be recompiled. Node:Use of Upper and Lower Case, Next:Questions 1, Previous:Errors, Up:Introduction

Use of Upper and Lower Case

One of the reasons why the compiler can fail to produce the executable file for a program is you have mistyped something, even through the careless use of upper and lower case characters. The C language is case dependent. Unlike languages such as Pascal and some versions of BASIC, the C compiler distinguishes between small letters and capital letters. This is a potential source of quite trivial errors which can be difficult to spot. If a letter is typed in the wrong case, the compiler will complain and it will not produce an executable program.

Declarations
Compiler languages require us to make a list of the names and types of all variables which are going to be used in a program and provide information about where they are going to be used. This is called declaring variables. It serves two purposes: firstly, it provides the compiler with a definitive list of the variables, enabling it to cross check for errors, and secondly, it informs the compiler how much space must be reserved for each variable when the program is run. C supports a variety of variable types (variables which hold different kinds of data) and allows one type to be converted into another. Consequently, the type of a variable is of great importance to the compiler. If you fail to declare a variable, or declare it to be the wrong type, you will see a compilation error. Node:Questions 1, Previous:Use of Upper and Lower Case, Up:Introduction

Questions
1. What is a compiler? 2. How is a C program run? 3. How is a C program compiled usually?

4. Are upper and lower case equivalent in C? 5. What the two different kinds of error which can be in a program? Node:Reserved words & example, Next:Operating systems, Previous:Introduction, Up:Top

Reserved words and an example

C programs are constructed from a set of reserved words which provide control and from libraries which perform special functions. The basic instructions are built up using a reserved set of words, such as main, for, if,while, default, double, extern, for, and int, to name just a few. These words may not be used in just any old way: C demands that they are used only for giving commands or making statements. You cannot use default, for example, as the name of a variable. An attempt to do so will result in a compilation error. See All the Reserved Words, for a complete list of the reserverd words. Words used in included libaries are also, effectively, reserved. If you use a word which has already been adopted in a library, there will be a conflict between your choice and the library. Libraries provide frequently used functionality and, in practice, at least one library must be included in every program: the so-called C library, of standard functions. For example, the stdio library, which is part of the C library, provides standard facilities for input to and output from a program. In fact, most of the facilities which C offers are provided as libraries that are included in programs as plug-in expansion units. While the features provided by libraries are not strictly a part of the C language itself, they are essential and you will never find a version of C without them. After a library has been included in a program, its functions are defined and you cannot use their names.

printf: Example 1: Output 1: Questions 2:

Node:printf, Next:Example 1, Previous:Reserved words & example, Up:Reserved words & example

The printf() function

One invaluable function provided by the standard input/output library is called printf or `print-formatted'. It provides an superbly versatile way of printing text. The simplest way to use it is to print out a literal string:
printf ("..some string...");

Text is easy, but we also want to be able to print out the contents of variables. These can be inserted into a text string by using a `control sequence' inside the quotes and listing the variables after the string which get inserted into the string in place of the control sequence. To print out an integer, the control sequence %d is used:
printf ("Integer = %d",someinteger);

The variable someinteger is printed instead of %d. The printf function is described in full detail in the relevant chapter, but we'll need it in many places before that. The example program below is a complete program. If you are reading this in Info, you can copy this to a file, compile and execute it. Node:Example 1, Next:Output 1, Previous:printf, Up:Reserved words & example

Example Listing
/***********************************************************/ /* Short Poem */ /***********************************************************/ #include <stdio.h> /***********************************************************/ main () { printf printf printf printf printf printf printf printf printf printf printf printf } /* Poem */ ("Astronomy is %dderful \n",1); ("And interesting %d \n",2); ("The ear%d volves around the sun \n",3); ("And makes a year %d you \n",4); ("The moon affects the sur %d heard \n",5); ("By law of phy%d great \n",6); ("It %d when the the stars so bright \n",7); ("Do nightly scintill%d \n",8); ("If watchful providence be%d \n",9); ("With good intentions fraught \n"); ("Should not keep up her watch divine \n"); ("We soon should come to %d \n",0);

Node:Output 1, Next:Questions 2, Previous:Example 1, Up:Reserved words & example

Output
Astronomy is 1derful \n" And interesting 2 The ear3 volves around the sun And makes a year 4 you The moon affects the sur 5 heard By law of phy6d great It 7 when the the stars so bright Do nightly scintill8 If watchful providence be9 With good intentions fraught Should not keep up her watch divine We soon should come to 0

Node:Questions 2, Previous:Output 1, Up:Reserved words & example

Questions
1. 2. 3. 4. Write a command to print out the message "Wow big deal". Write a command to print out the number 22? Write two commands to print out "The 3 Wise Men" two different ways. Why are there only a few reserved command words in C?

Node:Operating systems, Next:Libraries, Previous:Reserved words & example, Up:Top

Operating systems and environments

Where is a C program born? How is it created? The basic control of a computer rests with its operating system. This is a layer of software which drives the hardware and provides users with a comfortable environment in which to work. An operating system has two main components which are of interest to users: a user interface (often a command language) and a filing system. The operating system is the route to all input and output, whether it be to a screen or to files on a disk. A programming language has to get at this input and output easily so that programs can send out and receive messages from the user and it has to be in contact with the operating system in order to do this. In C the link between these two is very efficient. Operating systems vary widely but most have a command language or shell which can be used to type in commands. Recently the tendency has been to try to eliminate typing completely by providing graphical user interfaces (GUIs) for every purpose. GUIs are good for carrying out simple procedures like editing, but they are not well suited to giving complicated instructions to a computer. For that one needs a command language. In the network version of this book we shall concentrate on Unix shell commands since they are the most important to programmers. On microcomputers command languages are usually very similar in concept, though more primitive, with only slightly different words for essentially the same commands. (This is a slightly superficial view). When most compiler languages were developed, they were intended to be run on large mainframe computers which operated on a multi-user, time-sharing principle and were incapable of interactive communication with the user. Many compiler languages still have this inadequacy when carried over to modern computers, but C is an exception, because of its unique design. Input and output are not actually defined as a fixed, unchanging part of the C language. Instead there is a standard file which has to be included in programs and defines the input/output commands that are supported by the language for a particular computer and operating system. This file is called a standard C library. (See the next chapter for more information.) The library is standard in the sense that C has developed a set of functions which all computers and operating systems must implement, but which are specially adapted to your system.

Files devices: Filenames: Command languages: Questions 3:

Node:Files devices, Next:Filenames, Previous:Operating systems, Up:Operating systems

Files and Devices

The filing system is also a part of input/output. In many operating systems all routes in and out of the computer are treated by the operating system as though they were files or data streams (even the keyboard!). C does this implicitly (it comes from Unix). The file from which C normally gets its input from is called stdin or standard input file and it is usually the keyboard. The corresponding route for output is called "stdout" or standard output file and is usually a monitor screen. Both of these are parts of stdio or standard input output. The keyboard and the monitor screen are not really files, of course, they are `devices', (it is not possible to re-read what has been sent to the monitor", or write to the keyboard.), but devices are represented by files with special names, so that the keyboard is treated as a read-only file, the monitor as a write only file... The advantage of treating devices like this is that it is not necessary to know how a particular device works, only that it exists somewhere, connected to the computer, and can be written to or read from. In other words, it is exactly the same to read or write from a device as it is to read or write from a file. This is a great simplification of input/output! The filenames of devices (often given the lofty title `pseudo device names') depend upon your particular operating system. For instance, the printer might be called "PRN" or "PRT". You might have to open it explicitly as a file. When input is taken solely from the keyboard and output is always to the screen then these details can just be forgotten. Node:Filenames, Next:Command languages, Previous:Files devices, Up:Operating systems

Filenames
The compiler uses a special convention for the file names, so that we do not confuse their contents. The name of a source program (the code which you write) is filename.c. The compiler generates a file of object code from this called filename.o, as yet unlinked. The final program, when linked to libraries is called filename on Unix-like operating systems, and filename.EXE on Windows derived systems. The libraries themselves are also files of object code, typically called liblibraryname.a or liblibraryname.so. Header files are always called libname.h. The endings `dot something' (called file extensions) identify the contents of files for the compiler. The dotted endings mean that the compiler can generate an executable file with the same name as the original source - just a different ending. The quad file and the object file are only working files and should be deleted by the compiler at the end of compilation. The .c suffix is to tell the compiler that the file contains a C

source program and similarly the other letters indicate non-source files in a convenient way. To execute the compiler you type,
cc filename

For example,

cc foo.c

Node:Command languages, Next:Questions 3, Previous:Filenames, Up:Operating systems

Command Languages and Consoles

In order to do anything with a compiler or an editor you need to know a little about the command language of the operating system. This means the instructions which can be given to the system itself rather than the words which make up a C program. e.g.
ls -l less filename emacs filename

In a large operating system (or even a relatively small one) it can be a major feat of recollection to know all of the commands. Fortunately it is possible to get by with knowing just handful of the most common ones and having the system manual around to leaf through when necessary. Another important object is the `panic button' or program interruption key. Every system will have its own way of halting or terminating the operation of a program or the execution of a command. Commonly this will involve two simultaneous key presses, such as CTRL C, CTRL Z or CTRL-D etc. In GNU/Linux, CTRL-C is used. Node:Questions 3, Previous:Command languages, Up:Operating systems

Questions
1. What is an operating system for? 2. What is a pseudo-device name? 3. If you had a C source program which you wanted to call `accounts' what name would you save it under? 4. What would be the name of the file produced by the compiler of the program in 3? 5. How would this program be run? Node:Libraries, Next:Programming style, Previous:Operating systems, Up:Top

Libraries

Plug-in C expansions. Header files. The core of the C language is small and simple. Special functionality is provided in the form of libraries of ready-made functions. This is what makes C so portable. Some libraries are provided for you, giving you access to many special abilities without needing to reinvent the wheel. You can also make your own, but to do so you need to know how your operating system builds libraries. We shall return to this later. Libraries are files of ready-compiled code which we can merge with a C program at compilation time. Each library comes with a number of associated header files which make the functions easier to use. For example, there are libraries of mathematical functions, string handling functions and input/output functions and graphics libraries. It is up to every programmer to make sure that libraries are added at compilation time by typing an optional string to the compiler. For example, to merge with the math library libm.a you would type
cc -o program_name prog.c -lm

when you compile the program. The -lm means: add in libm. If we wanted to add in the socket library libsocket.a to do some network programming as well, we would type
cc -o program_name prog.c -lm -lsocket

and so on. Why are these libraries not just included automatically? Because it would be a waste for the compiler to add on lots of code for maths functions, say, if they weren't needed. When library functions are used in programs, the appropriate library code is included by the compiler, making the resulting object code often much longer. Libraries are supplemented by header files which define macros, data types and external data to be used in conjunction with the libraries. Once a header file has been included, it has effectively added to the list of reserved words and commands in the language. You cannot then use the names of functions or macros which have already been defined in libraries or header files to mean anything other than what the library specifies. The most commonly used header file is the standard input/output library which is called stdio.h. This belongs to a subset of the standard C library which deals with file handling. The math.h header file belongs to the mathematics library libm.a. Header files for libraries are included by adding to the source code:
#include header.h

at the top of a program file. For instance:

#include "myheader.h"

includes a personal header file which is in the current directory. Or

#include <stdio.h>

includes a file which lies in a standard directory like /usr/include. The #include directive is actually a command to the C preprocessor, which is dealt with more fully later, See Preprocessor. Some functions can be used without having to include library files or special libraries explicitly since every program is always merged with the standard C library, which is called libc.
#include <stdio.h> main () { printf ("C standard I/O file is included\n"); printf ("Hello world!"); }

A program wishing to use a mathematical function such as cos would need to include a mathematics library header file.
#include <stdio.h> #include <math.h> main () { double x,y; y = sin (x); printf ("Maths library ready"); }

A particular operating system might require its own special library for certain operations such as using a mouse or for opening windows in a GUI environment, for example. These details will be found in the local manual for a particular C compiler or operating system. Although there is no limit, in principle, to the number of libraries which can be included in a program, there may be a practical limit: namely memory, since every library adds to the size of both source and object code. Libraries also add to the time it takes to compile a program. Some operating systems are smarter than others when running programs and can load in only what they need of the large libraries. Others have to load in everything before they can run a program at all, so many libraries would slow them down. To know what names libraries have in a particular operating system you have to search through its documentation. Unix users are lucky in having an online manual which is better than most written ones.

Questions 4:

Node:Questions 4, Previous:Libraries, Up:Libraries

Questions
1. How is a library file incorporated into a C program? 2. Name the most common library file in C. 3. Is it possible to define new functions with the same names as standard library functions? 4. What is another name for a library file? Node:Programming style, Next:Form of a C program, Previous:Libraries, Up:Top

Programming style
The shape of programs to come. C is actually a free format language. This means that there are no rules about how it must be typed, when to start new lines, where to place brackets or whatever. This has both advantages and dangers. The advantage is that the user is free to choose a style which best suits him or her and there is freedom in the way in which a program can be structured. The disadvantage is that, unless a strict style is adopted, very sloppy programs can be the result. The reasons for choosing a well structured style are that:

Long programs are manageable only if programs are properly organized. Programs are only understandable if care is taken in choosing the names of variables and functions. It is much easier to find parts of a program if a strict ordering convention is maintained. Such a scheme becomes increasingly difficult to achieve with the size and complexity of the problem.

No simple set of rules can ever provide the ultimate solution to writing good programs. In the end, experience and good judgement are the factors which decide whether a program is written well or poorly written. The main goal of any style is to achieve clarity. Previously restrictions of memory size, power and of particular compilers often forced restrictions upon style, making programs clustered and difficult. All computers today are equipped with more than enough memory for their purposes, and have very good optimizers which can produce faster code than most programmers could write themselves without help, so there are few good reasons not to make programs as clear as possible. Node:Form of a C program, Next:Comments, Previous:Programming style, Up:Top

The form of a C program

What goes into a C program? What will it look like? C is made up entirely of building blocks which have a particular `shape' or form. The form is the same everywhere in a program, whether it is the form of the main program or of a subroutine. A program is made up of functions, functions are made up of statements and declarations surrounded by curly braces { }. The basic building block in a C program is the function. Every C program is a collection of one or more functions, written in some arbitrary order. One and only one of these functions in the program must have the name main(). This function is always the starting point of a C program, so the simplest C program would be just a single function definition:
main () { }

The parentheses () which follow the name of the function must be included even though they apparently serve no purpose at this stage. This is how C distinguishes functions from ordinary variables.

The function main() does not have to be at the top of a program so a C program does not necessarily start at line 1. It always starts where main() is. Also, the function main() cannot be called from any other function in the program. Only the operating system can call the function main(): this is how a C program is started. The next most simple C program is perhaps a program which calls a function do_nothing and then ends.

/******************************************************/ /* */ /* Program : do nothing */ /* */ /******************************************************/ main() { do_nothing(); } /******************************************************/ do_nothing() { } /* Function called */ /* Main program */

The program now consists of two functions, one of which is called by the other. There are several new things to notice about this program. Firstly the function do_nothing() is called by typing its name followed by the characteristic () brackets and a semicolon. This is all that is required to transfer control to the new function. In some languages, words like CALL or PROC are used, or even a symbol like &. No such thing is needed in C. The semi-colon is vital however. All instructions in C must end with a semi-colon. This is a signal to inform the compiler that the end of a statement has been reached and that anything which follows is meant to be a part of another statement. This helps the compiler diagnose errors. The `brace' characters { and } mark out a block into which instructions are written. When the program meets the closing brace } it then transfers back to main() where it meets another } brace and the program ends. This is the simplest way in which control flows between functions in C. All functions have the same status as far as a program is concerned. The function main() is treated just as any other function. When a program is compiled, each function is compiled as a separate entity and then at the end the linker phase in the compiler attempts to sew them all together. The examples above are obviously very simple but they illustrate how control flows in a C program. Here are some more basic elements which we shall cover.

comments preprocessor commands functions declarations variables statements

The skeleton plan of a program, shown below, helps to show how the elements of a C program relate. The following chapters will then expand upon this as a kind of basic plan.
/****************************************************/ /* */ /* Skeleton program plan */ /* */

/****************************************************/ #include <stdio.h> #include <myfile.c> #define SCREAM #define NUMBER_OF_BONES /* Preprocessor defns */ "arghhhhh" 123

/****************************************************/ main () { int a,b; a=random(); b=function1(); function2(a,b); } /****************************************************/ function1 () { .... } /****************************************************/ function2 (a,b) int a,b; { .... } /* Purpose */ /* Purpose */ /* Main program & start */ /* declaration */

Question 5:

Neither comments nor preprocessor commands have a special place in this list: they do not have to be in any one particular place within the program. Node:Question 5, Previous:Form of a C program, Up:Form of a C program

Questions
1. 2. 3. 4. What is a block? Name the six basic things which make up a C program. Does a C program start at the beginning? (Where is the beginning?) What happens when a program comes to a } character? What does this character signify? 5. What vital piece of punctuation goes at the end of every simple C statement? Node:Comments, Next:Functions, Previous:Form of a C program, Up:Top

Comments

Annotating programs. Comments are a way of inserting remarks and reminders into a program without affecting its content. Comments do not have a fixed place in a program: the compiler treats them as though they were white space or blank characters and they are consequently ignored. Programs can contain any number of comments without losing speed. This is because comments are stripped out of a source program by the compiler when it converts the source program into machine code. Comments are marked out or delimited by the following pairs of characters:
/* ...... comment ......*/

Because a comment is skipped over as though it were a single space, it can be placed anywhere where spaces are valid characters, even in the middle of a statement, though this is not to be encouraged. You should try to minimize the use of comments in a program while trying to maximize the readability of the program. If there are too many comments you obscure your code and it is the code which is the main message in a program.

Example comment: Example comment 2: Question 7:

Node:Example comment, Next:Example comment 2, Previous:Comments, Up:Comments

Example 1
main () { /* This little line /* This little line /* This little line to the next line /* And so on ... */ } has no effect */ has none */ went all the way down */ /* The almost trivial program */

Node:Example comment 2, Next:Question 7, Previous:Example comment, Up:Comments

Example 2
#include <stdio.h> #define NOTFINISHED /* header file */ 0

/**********************************************/

/* A bar like the one above can be used to */ /* separate functions visibly in a program */ main () { int i; do { /* Nothing !!! */ } while (NOTFINISHED); } /* declarations */

Node:Question 7, Previous:Example comment 2, Up:Comments

Question
1. What happens if a comment is not ended? That is if the programmer types /* .. to start but forgets the ..*/ to close? Node:Functions, Next:Variables, Previous:Comments, Up:Top

Functions
Making black boxes. Solving problems. Getting results. A function is a module or block of program code which deals with a particular task. Making functions is a way of isolating one block of code from other independent blocks of code. Functions serve two purposes. They allow a programmer to say: `this piece of code does a specific job which stands by itself and should not be mixed up with anyting else', and they make a block of code reusable since a function can be reused in many different contexts without repeating parts of the program text. Functions help us to organize a program in a simple way; in Kernighan & Ritchie C they are always written in the following form:
identifier (parameter1,parameter2,..) types of parameters { variable declarations statements.. ...... .... }

For example

Pythagoras(x,y,z) double x,y,z; { double d; d = sqrt(x*x+y*y+z*z); printf("The distance to your point was %f\n",d); }

In the newer ANSI standard, the same function is written slightly differently:
Pythagoras(double x, double y, double z) { double d; d = sqrt(x*x+y*y+z*z); printf("The distance to your point was %f\n",d); }

You will probably see both styles in C programs. Each function has a name or identifier by which is used to refer to it in a program. A function can accept a number of parameters or values which pass information from outside, and consists of a number of statements and declarations, enclosed by curly braces { }, which make up the doing part of the object. The declarations and `type of parameter' statements are formalities which will be described in good time. The name of a function in C can be anything from a single letter to a long word. The name of a function must begin with an alphabetic letter or the underscore _ character but the other characters in the name can be chosen from the following groups:
a .. z

(any letter from a to z)

A .. Z 0 .. 9 _

(any letter from A to Z) (any digit from 0 to 9) (the underscore character)

This means that sensible names can easily be chosen for functions making a program easy to read. Here is a real example function which adds together two integer numbers a and b and prints the result c. All the variables are chosen to be integers to keep things simple and the result is printed out using the print-formatted function printf, from the the standard library, with a "%d" to indicate that it is printing a integer.
Add_Two_Numbers (a,b) int a,b; { int c; /* Add a and b */

c = a + b; printf ("%d",c); }

Notice the position of the function name and where braces and semi-colons are placed: they are crucial. The details are quickly learned with practice and experience. This function is not much use standing alone. It has to be called from somewhere. A function is called (i.e. control is passed to the function) by using its name with the usual brackets () to follow it, along with the values which are to be passed to the function:
main () { int c,d; c = 1; d = 53; Add_Two_Numbers (c,d); Add_Two_Numbers (1,2); }

The result of this program would be to print out the number 54 and then the number 3 and then stop. Here is a simple program which makes use of some functions in a playful way. The structure diagram shows how this can be visualized and the significance of the program `levels'. The idea is to illustrate the way in which the functions connect together:

Structure diagram: Program listing: Functions with values: Breaking out early: The exit function: Functions and types: Questions 6:

Node:Structure diagram, Next:Program listing, Previous:Functions, Up:Functions

Structure diagram
Level 0: main () | Level 1: / Level 2: DownLeft() DownOne () / \ \ DownRight()

Note: not all functions fit into a tidy hierarchy like these. Some functions call themselves, while others can be called from anywhere in a program. Where would you place the printf function in this hierarchy? Node:Program listing, Next:Functions with values, Previous:Structure diagram, Up:Functions

Program Listing
/***********************************************/ /* */ /* Function Snakes & Ladders */ /* */ /***********************************************/ #include <stdio.h> /***********************************************/ /* Level 0 */ /***********************************************/ main () { printf ("This is level 0: the main program\n"); printf ("About to go down a level \n"); DownOne (); printf ("Back at the end of the start!!\n"); } /************************************************/ /* Level 1 */ /************************************************/ DownOne () /* Branch out! */

{ printf ("Down here at level 1, all is well\n"); DownLeft (2); printf ("Through level 1....\n"); DownRight (2); printf ("Going back up a level!\n"); } /************************************************/ /* Level 2 */ /************************************************/ DownLeft (a) int a; { printf ("This is deepest level %d\n",a); printf ("On the left branch of the picture\n"); printf ("Going up!!"); } /* Left branch */

/************************************************/ DownRight (a) int a; { printf ("And level %d again!\n",a); } /* Right branch */

Node:Functions with values, Next:Breaking out early, Previous:Program listing, Up:Functions

Functions with values

In other languages and in mathematics a function is understood to be something which produces a value or a number. That is, the whole function is thought of as having a value. In C it is possible to choose whether or not a function will have a value. It is possible to make a function hand back a value to the place at which it was called. Take the following example:
bill = CalculateBill(data...);

The variable bill is assigned to a function CalculateBill() and data are some data which are passed to the function. This statement makes it look as though CalculateBill() is a number. When this statement is executed in a program, control will be passed to the function CalculateBill() and, when it is done, this function will then hand control back. The value of the function is assigned to "bill" and the program continues. Functions which work in this way are said to return a value. In C, returning a value is a simple matter. Consider the function CalculateBill() from the statement above:
CalculateBill(starter,main,dessert) int starter,main,dessert; { int total; total = starter + main + dessert; return (total); } /* Adds up values */

As soon as the return statement is met CalculateBill() stops executing and assigns the value total to the function. If there were no return statement the program could not know which value it should associate with the name CalculateBill and so it would not be meaningful to speak of the function as having one value. Forgetting a return statement can ruin a program. For instance if CalculateBill had just been:
CalculateBill (starter,main,dessert) int starter,main,dessert; /* WRONG! */

{ int total; total = starter + main + dessert; }

then the value bill would just be garbage (no predictable value), presuming that the compiler allowed this to be written at all. On the other hand if the first version were used (the one which did use the return(total) statement) and furthermore no assignment were made:
main () { CalculateBill (1,2,3); }

then the value of the function would just be discarded, quite legitimately. This is usually what is done with the input output functions printf() and scanf() which actually return values. So a function in C can return a value but it does not have to be used; on the other hand, a value which has not been returned cannot be used safely. NOTE : Functions do not have to return integers: you can decide whether they should return a different data type, or even no value at all. (See next chapter) Node:Breaking out early, Next:The exit function, Previous:Functions with values, Up:Functions

Breaking out early

Suppose that a program is in the middle of some awkward process in a function which is not main(), perhaps two or three loops working together, for example, and suddenly the function finds its answer. This is where the beauty of the return statement becomes clear. The program can simply call return(value) anywhere in the function and control will jump out of any number of loops or whatever and pass the value back to the calling statement without having to finish the function up to the closing brace }.
myfunction (a,b) int a,b; { while (a < b) { if (a > b) { return (b); } a = a + 1; } } /* breaking out of functions early */

The example shows this. The function is entered with some values for a and b and, assuming that a is less than b, it starts to execute one of C's loops called while. In that

loop, is a single if statement and a statement which increases a by one on each loop. If a becomes bigger than b at any point the return(b) statement gets executed and the function myfunction quits, without having to arrive at the end brace }, and passes the value of b back to the place it was called. Node:The exit function, Next:Functions and types, Previous:Breaking out early, Up:Functions

The exit() function

The function called exit() can be used to terminate a program at any point, no matter how many levels of function calls have been made. This is called with a return code, like this:
#define CODE exit (CODE); 0

This function also calls a number of other functions which perform tidy-up duties such as closing open files etc. Node:Functions and types, Next:Questions 6, Previous:The exit function, Up:Functions

Functions and Types

All the variables and values used up to now have been integers. But what happens if a function is required to return a different kind of value such as a character? A statement like:
bill = CalculateBill (a,b,c);

can only make sense if the variable bill and the value of the function CalculateBill() are the same kind of object: in other words if CalculatBill() returns a floating point number, then bill cannot be a character! Both sides of an assignment must match. In fact this is done by declaring functions to return a particular type of data. So far no declarations have been needed because C assumes that all values are integers unless you specifically choose something different. Declarations are covered in the next section. Node:Questions 6, Previous:Functions and types, Up:Functions

Questions
1. Write a function which takes two values a and b and returns the value of (a*b). 2. Is there anything wrong with a function which returns no value? 3. What happens if a function returns a value but it is not assigned to anything?

4. What happens if a function is assigned to an object but that function returns no value? 5. How can a function be made to quit early? Node:Variables, Next:Parameters, Previous:Functions, Up:Top

Variables, Types and Declarations

Storing data. Descriminating types. Declaring data. A variable is a seqeuence of program code with a name (also called its identifier). A name or identifier in C can be anything from a single letter to a word. The name of a variable must begin with an alphabetic letter or the underscore _ character but the other characters in the name can be chosen from the following groups:
a .. z A .. Z 0 .. 9 _

(any letter from a to z) (any letter from A to Z) (any digit from 0 to 9) (the underscore character)

Some examples of valid variable names are:

a total Out_of_Memory VAR integer etc...

In C variables do not only have names: they also have types. The type of a variable conveys to the the compiler what sort of data will be stored in it. In BASIC and in some older, largely obsolete languages, like PL/1, a special naming convention is used to determine the sort of data which can be held in particular variables. e.g. the dollar symbol $ is commonly used in BASIC to mean that a variable is a string and the percentage % symbol is used to indicate an integer. No such convention exists in C. Instead we specify the types of variables in their declarations. This serves two purposes:

It gives a compiler precise information about the amount of memory that will have to be given over to a variable when a program is finally run and what sort of arithmetic will have to be used on it (e.g. integer only or floating point or none). It provides the compiler with a list of the variables in a convenient place so that it can cross check names and types for any errors.

There is a lot of different possible types in C. In fact it is possible for us to define our own, but there is no need to do this right away: there are some basic types which are provided by C ready for use. The names of these types are all reserved words in C and they are summarized as follows:

char

A single ASCII character

short short int int long

A short integer (usually 16-bits) A short integer A standard integer (usually 32-bits) A long integer

long int float

A long integer (usually 32-bits, but increasingly 64 bits) A floating point or real number (short)

long float double void enum

a long floating point number A long floating point number Discussed in a later chapter. Discussed in a later chapter.

volatile

Discussed in a later chapter. There is some repetition in these words. In addition to the above, the word unsigned can also be placed in front of any of these types. Unsigned means that only positive or zero values can be used. (i.e. there is no minus sign). The advantage of using this kind of variable is that storing a minus sign takes up some memory, so that if no minus sign is present, larger numbers can be stored in the same kind of variable. The ANSI standard also allows the word signed to be placed in front of any of these types, so indicate the opposite of unsigned. On some systems variables are signed by default, whereas on others they are not.

Declarations: Where to declare things: Declarations and Initialization: Types: Choosing Variables: Assigning variables to one another: Types and The Cast Operator: Storage class register static and extern: Functions types: Questionsdeclare:

Node:Declarations, Next:Where to declare things, Previous:Variables, Up:Variables

Declarations
To declare a variable in a C program one writes the type followed by a list of variable names which are to be treated as being that type:
typename variablename1,..,..,variablenameN;

For example:
int i,j; char ch; double x,y,z,fred; unsigned long int Name_of_Variable;

Failing to declare a variable is more risky than passing through customs and failing to declare your six tonnes of Swiss chocolate. A compiler is markedly more efficient than a customs officer: it will catch a missing declaration every time and will terminate a compiling session whilst complaining bitterly, often with a host of messages, one for each use of the undeclared variable. Node:Where to declare things, Next:Declarations and Initialization, Previous:Declarations, Up:Variables

Where to declare things

There are two kinds of place in which declarations can be made, See Scope. For now it will do to simply state what these places are. 1. One place is outside all of the functions. That is, in the space between function definitions. (After the #include lines, for example.) Variables declared here are called global variables. There are also called static and external variables in special cases.)
2. #include <stdio.h> 3. 4. int globalinteger; 5. 6. float global_floating_point; 7. 8. main () 9. 10. { 11. } 12. /* Here! outside {} */

13. The other place where declarations can be made is following the opening brace, {}, of a block. Any block will do, as long as the declaration follows immediately after the opening brace. Variables of this kind only work inside their braces {} and are often called local variables. Another name for them is automatic variables.
14. 15. 16. 17. 18. 19. 20. 21. 22. main () { int a; float x,y,z; /* statements */ }

or
function () { int i; /* .... */

while (i < 10) { char ch; int g; /* ... */ } }

Node:Declarations and Initialization, Next:Types, Previous:Where to declare things, Up:Variables

Declarations and Initialization

When a variable is declared in C, the language allows a neat piece of syntax which means that variables can be declared and assigned a value in one go. This is no more efficient than doing it in two stages, but it is sometimes tidier. The following:
int i = 0; char ch = 'a';

are equivalent to the more longwinded

int i; char ch; i = 0; ch = 'a';

This is called initialization of the variables. C always allows the programmer to write declarations/initializers in this way, but it is not always desirable to do so. If there are just one or two declarations then this initialization method can make a program neat and tidy. If there are many, then it is better to initialize separately, as in the second case. A lot means when it starts to look as though there are too many. It makes no odds to the compiler, nor (ideally) to the final code whether the first or second method is used. It is only for tidiness that this is allowed. Node:Types, Next:Choosing Variables, Previous:Declarations and Initialization, Up:Variables

Individual Types

char: Example special chars: integers: Float:

Node:char, Next:Example special chars, Previous:Types, Up:Types

char

A character type is a variable which can store a single ASCII character. Groups of char form strings. In C single characters are written enclosed by single quotes, e.g. 'c'! (This is in contrast to strings of many characters which use double quotes, e.g. "string") For instance, if ch is the name of a character:
char ch; ch = 'a';

would give ch the value of the character a. The same effect can also be achieved by writing:
char ch = 'a';

A character can be any ASCII character, printable or not printable from values -128 to 127. (But only 0 to 127 are used.) Control characters i.e. non printable characters are put into programs by using a backslash \ and a special character or number. The characters and their meanings are:
\b

backspace BS
\f \n \r \t \v \" \' \\ \ddd

form feed FF (also clear screen) new line NL (like pressing return) carriage return CR (cursor to start of line) horizontal tab HT vertical tab (not all versions) double quotes (not all versions) single quote character ' backslash character \ character ddd where ddd is an ASCII code given in octal or base 8, See Character Conversion Table.

\xddd

character ddd where ddd is an ASCII code given in hexadecimal or base 16, See Character Conversion Table. Node:Example special chars, Next:integers, Previous:char, Up:Types

Listing
/***************************************************/ /* */ /* Special Characters */ /* */ /***************************************************/ #include <stdio.h>

main () { printf ("Beep! \7 \n"); printf ("ch = \'a\' \n"); printf (" <- Start of this line!! \r"); }

The output of this program is:

Beep! (and the BELL sound ) ch = 'a' <- Start of this line!!

and the text cursor is left where the arrow points. It is also possible to have the type:
unsigned char

This admits ASCII values from 0 to 255, rather than -128 to 127. Node:integers, Next:Float, Previous:Example special chars, Up:Types

Integers

Whole numbers
There are five integer types in C and they are called char, int, long, long long and short. The difference between these is the size of the integer which either can hold and the amount of storage required for them. The sizes of these objects depend on the operating system of the computer. Even different flavours of Unix can have varying sizes for these objects. Usually, the two to remember are int and short. int means a `normal' integer and short means a `short' one, not that that tells us much. On a typical 32 bit microcomputer the size of these integers is the following:
Type Bits Possible Values -32768 to 32767 0 to 65535 -2147483648 to 2147483647 (ditto) 0 to 4294967295 -9e18 to + 8e18

short 16 unsigned short 16 int long unsigned int long long 32 32 32 64

Increasingly though, 64 bit operating systems are appearing and long integers are 64 bits long. You should always check these values. Some mainframe operating systems are completely 64 bit, e.g. Unicos has no 32 bit values. Variables are declared in the usual way:
int i,j;

i = j = 0;

or

short i=0,j=0;

Node:Float, Previous:integers, Up:Types

Floating Point
There are also long and short floating point numbers in C. All the mathematical functions which C can use require double or long float arguments so it is common to use the type float for storage only of small floating point numbers and to use double elsewhere. (This not always true since the C `cast' operator allows temporary conversions to be made.) On a typical 32 bit implementation the different types would be organized as follows:
Type float double long float long double Bits 32 64 32 ??? Possible Values +/- 10E-37 to +/- 10E38 +/- 10E-307 to +/- 10E308 (ditto)

Typical declarations:
float x,y,z; x = 0.1; y = 2.456E5 z = 0; double bignum,smallnum; bignum = 2.36E208; smallnum = 3.2E-300;

Node:Choosing Variables, Next:Assigning variables to one another, Previous:Types, Up:Variables

Choosing Variables
The sort of procedure that you would adopt when choosing variable names is something like the following:

Decide what a variable is for and what type it needs to be. Choose a sensible name for the variable. Decide where the variable is allowed to exist. Declare that name to be a variable of the chosen type.

Some local variables are only used temporarily, for controlling loops for instance. It is common to give these short names (single characters). A good habit to adopt is to keep to a consistent practice when using these variables. A common one, for instance is to use the letters:
int i,j,k;

to be integer type variables used for counting. (There is not particular reason why this should be; it is just common practice.) Other integer values should have more meaningful names. Similarly names like:
double x,y,z;

tend to make one think of floating point numbers. Node:Assigning variables to one another, Next:Types and The Cast Operator, Previous:Choosing Variables, Up:Variables

Assigning variables to one another

Variables can be assigned to numbers:
var = 10;

and assigned to each other:

var1 = var2;

In either case the objects on either side of the = symbol must be of the same type. It is possible (though not usually sensible) to assign a floating point number to a character for instance. So
int a, b = 1; a = b;

is a valid statement, and:

float x = 1.4; char ch; ch = x;

is a valid statement, since the truncated value 1 can be assigned to ch. This is a questionable practice though. It is unclear why anyone would choose to do this. Numerical values and characters will interconvert because characters are stored by their ASCII codes (which are integers!) Thus the following will work:
int i; char ch = 'A'; i = ch; printf ("The ASCII code of %c is %d",ch,i);

The result of this would be:

The ASCII code of A is 65

Node:Types and The Cast Operator, Next:Storage class register static and extern, Previous:Assigning variables to one another, Up:Variables

Types and The Cast Operator

It is worth mentioning briefly a very valuable operator in C: it is called the cast operator and its function is to convert one type of value into another. For instance it would convert a character into an integer:
int i; char ch = '\n'; i = (int) ch;

The value of the integer would be the ASCII code of the character. This is the only integer which it would make any sense to talk about in connection with the character. Similarly floating point and integer types can be interconverted:
float x = 3.3; int i; i = (int) x;

The value of i would be 3 because an integer cannot represent decimal points, so the cast operator rounds the number. There is no such problem the other way around.

float x; int i = 12; x = (float) i;

The general form of the cast operator is therefore:

(type) variable

It does not always make sense to convert types. This will be seen particularly with regard to structures and unions. Cast operators crop up in many areas of C. This is not the last time they will have to be explained.
/***************************************************/ /* */ /* Demo of Cast operator */ /* */ /***************************************************/ #include <stdio.h> main () { float x; /* Use int float and char */

int i; char ch; x = 2.345; i = (int) x; ch = (char) x; printf ("From float x =%f i =%d ch =%c\n",x,i,ch); i = 45; x = (float) i; ch = (char) i; printf ("From int i=%d x=%f ch=%c\n",i,x,ch); ch = '*'; i = (int) ch; x = (float) ch; printf ("From char ch=%c i=%d x=%f\n",ch,i,x); }

Node:Storage class register static and extern, Next:Functions types, Previous:Types and The Cast Operator, Up:Variables

Storage class static and extern

Sometimes C programs are written in more than one text file. If this is the case then, on occasion, it will be necessary to get at variables which were defined in another file. If the word extern is placed in front of a variable then it can be referenced across files:
File 1 File 2 int i; main () { extern int i; } { } function ()

In this example, the function main() in file 1 can use the variable i from the function main in file 2. Another class is called static. The name static is given to variables which can hold their values between calls of a function: they are allocated once and once only and their values are preserved between any number of function calls. Space is allocated for static variables in the program code itself and it is never disposed of unless the whole program is. NOTE: Every global variable, defined outside functions has the type static automatically. The opposite of static is auto. Node:Functions types, Next:Questionsdeclare, Previous:Storage class register static and extern, Up:Variables

Functions, Types and Declarations

Functions do not always have to return values which are integers despite the fact that this has been exclusively the case up to now. Unless something special is done to force a function to return a different kind of value C will always assume that the type of a function is int. If you want this to be different, then a function has to be declared to be a certain type, just as variables have to be. There are two places where this must be done:

The name of the function must be declared a certain type where the function is declared. e.g. a function which returns a float value must be declared as:
float function1 () { return (1.229); }

A function which returns a character:

char function2 () { return ('*'); }

As well as declaring a function's identifier to be a certain type in the function definition, it must (irritatingly) be declared in the function in which it is called too! The reasons for this are related to the way in which C is compiled. So, if the two functions above were called from main(), they would have to declared in the variables section as:
main () { char ch, function2 (); float x, function1 (); x = function1 (); ch = function2 (); }

If a function whose type is not integer is not declared like this, then compilation errors will result! Notice also that the function must be declared inside every function which calls it, not just main(). Node:Questionsdeclare, Previous:Functions types, Up:Variables

Questions
1. What is an identifier? 2. Say which of the following are valid C identifiers: 1. Ralph23 2. 80shillings 3. mission_control

4. A% 5. A$ 6. _off 3. Write a statement to declare two integers called i and j. 4. What is the difference between the types floa and double. 5. What is the difference between the types int and unsigned int? 6. Write a statement which assigns the value 67 to the integer variable "I". 7. What type does a C function return by default? 8. If we want to declare a function to return long float, it must be done in, at least, two places. Where are these? 9. Write a statement, using the cast operator, to print out the integer part of the number 23.1256. 10. Is it possible to have an automatic global variable? Node:Parameters, Next:Scope, Previous:Variables, Up:Top

Parameters and Functions

Ways in and out of functions. Not all functions will be as simple as the ones which have been given so far. Functions are most useful if they can be given information to work with and if they can reach variables and data which are defined outside of them. Examples of this have already been seen in a limited way. For instance the function CalculateBill accepted three values a,b and c.
CalculateBill (a,b,c) int a,b,c; { int total; total = a + b + c; return total; }

When variable values are handed to a function, by writing them inside a functions brackets like this, the function is said to accept parameters. In mathematics a parameter is a variable which controls the behaviour of something. In C it is a variable which carries some special information. In CalculateBill the "behaviour" is the addition process. In other words, the value of total depends upon the starting values of a,b and c. Parameters are about communication between different functions in a program. They are like messengers which pass information to and from different places. They provide a way of getting information into a function, but they can also be used to hand information back. Parameters are usually split into two categories: value parameters and variable parameters. Value parameters are one-way communication carrying

information into a function from somewhere outside. Variable parameters are twoway.

Declaring parameters: Value parameters: Functions as actual parameters: Example 2: Example 3: Variable parameters: Example 4: Qulakfj:

Node:Declaring parameters, Next:Value parameters, Previous:Parameters, Up:Parameters

Declaring Parameters
A function was defined by code which looks like this:
identifier (parameters...) types of parameters { }

Parameters, like variables and functions, also have types which must be declared. For instance:
function1 (i,j,x,y) int i,j; float x,y; { }

or
char function2 (x,ch) double x; char ch; { char ch2 = '*'; return (ch2); }

Notice that they are declared outside the block braces.

Node:Value parameters, Next:Functions as actual parameters, Previous:Declaring parameters, Up:Parameters

Value Parameters
A value parameter is the most common kind of parameter. All of the examples up to know have been examples of value parameters. When a value parameter is passes information to a function its value is copied to a new place which is completely isolated from the place that the information came from. An example helps to show this. Consider a function which is called from main() whose purpose is to add together two numbers and to print out the result.
#include <stdio.h> main () { add (1,4); } /*******************************************/ add (a,b) int a,b; { printf ("%d", a+b); }

When this program is run, two new variables are automatically created by the language, called a and b. The value 1 is copied into a and the value 4 is copied into b. Obviously if a and b were given new values in the function add() then this could not change the values 1 and 4 in main(), because 1 is always 1 and 4 is always 4. They are

constants. However if instead the program had been:

main () { int a = 1, b = 4; add (a,b); } /**************************************/ add (a,b) int a,b; { printf ("%d", a+b); }

then it is less clear what will happen. In fact exactly the same thing happens:

When add() is called from main() two new variables a and b are created by the language (which have nothing to do with the variables a and b in main() and are completely isolated from them). The value of a in main() is copied into the value of a in add(). The value of b in main() is copied into the value of b in add().

Now, any reference to a and b within the function add() refers only to the two parameters of add and not to the variables with the same names which appeared in main(). This means that if a and b are altered in add() they will not affect a and b in main(). More advanced computing texts have names for the old and they new a and b: Actual Parameters These are the original values which were handed over to a function. Another name for this is an argument. Formal Parameters These are the copies which work inside the function which was called. Here are some points about value parameters.

The names of formal parameters can be anything at all. They do not have to be the same as the actual parameters. So in the example above it would be equally valid to write:
#include <stdio.h> main () { int a = 1, b = 4; add (a,b); } /*******************************************/ add (i,j) int i,j; { printf ("%d", i+j); }

In this case the value of a in main() would be copied to the value of i in add() and the value of b in main() would be copied to the value of j in add().

The parameters ought to match by datatype when taken in an ordered sequence. It is possible to copy a floating point number into a character formal parameter, causing yourself problems which are hard to diagnose. Some compilers will spot this if it is done accidentally and will flag it as an error. e.g.
main () { function ('*',1.0);

} /********************************/ function (ch,i) char ch; int i; { }

is probably wrong because 1.0 is a floating point value, not an integer.

The parameters ought to, but need not match in number! This surprising fact is important because programs can go wrong if a formal parameter was missed out. ANSI C has a way of checking this by function `prototyping', but in Kernighan & Ritchie C there is no way to check this. If the number of actual parameters is more than the number of formal parameters and all of the parameters match in type then the extra values are just discarded. If the number of actual parameters is less than the number of formal parameters, then the compiler will assign some unknown value to the formal parameters. This will probably be garbage. Our use of variables as parameters should not leave you with the impression that we can only use variables as parameters. In fact, we can send any literal value, or expression with an appropriate type to a function. For example,
sin(3.41415); cos(a+b*2.0); strlen("The length of this string");

Node:Functions as actual parameters, Next:Example 2, Previous:Value parameters, Up:Parameters

Functions as actual parameters

The value returned by a function can be used directly as a value parameter. It does not have to be assigned to a variable first. For instance:
main () { PrintOut (SomeValue()); } /*********************************************/ PrintOut (a) int a; { printf ("%d",a); } /**********************************************/ /* Print the value */

SomeValue () { return (42); }

/* Return an arbitrary no */

This often gives a concise way of passing a value to a function. Node:Example 2, Next:Example 3, Previous:Functions as actual parameters, Up:Parameters

Example Listing
/**************************************************/ /* */ /* Value Parameters */ /* */ /**************************************************/ /* Toying with value parameters */ #include <stdio.h> /**************************************************/ /* Level 0 */ /**************************************************/ main () /* Example of value parameters */

{ int i,j; double x,x_plus_one(); char ch; i = 0; x = 0; printf (" %f", x_plus_one(x)); printf (" %f", x); j = resultof (i); printf (" %d",j); } /***************************************************/ /* level 1 */ /***************************************************/ double x_plus_one(x) double x; { x = x + 1; return (x); } /****************************************************/ /* Add one to x ! */

resultof (j) int j; { return (2*j + 3); }

/* Work out some result */

/* why not... */

Node:Example 3, Next:Variable parameters, Previous:Example 2, Up:Parameters

Example Listing
/******************************************************/ /* */ /* Program : More Value Parameters */ /* */ /******************************************************/ /* Print out mock exam results etc */ #include <stdio.h> /******************************************************/ main () /* Print out exam results */

{ int pupil1,pupil2,pupil3; int ppr1,ppr2,ppr3; float pen1,pen2,pen3; pupil1 = 87; pupil2 = 45; pupil3 = 12; ppr1 = 200; ppr2 = 230; ppr3 = 10; pen1 = 1; pen2 = 2; pen3 = 20; analyse (pupil1,pupil2,pupil3,ppr1,ppr2, ppr3,pen1,pen2,pen3); } /*******************************************************/ analyse (p1,p2,p3,w1,w2,w3,b1,b2,b3) int p1,p2,p3,w1,w2,w3; float b1,b2,b3; { printf ("Pupil 1 scored %d percent\n",p1); printf ("Pupil 2 scored %d percent\n",p2); printf ("Pupil 3 scored %d percent\n",p3); printf ("However: \n"); printf ("Pupil1 wrote %d sides of paper\n",w1);

printf ("Pupil2 wrote %d sides\n",w2); printf ("Pupil3 wrote %d sides\n",w3); if (w2 > w1) { printf ("Which just shows that quantity"); printf (" does not imply quality\n"); } printf ("Pupil1 used %f biros\n",b1); printf ("Pupil2 used %f \n",b2); printf ("Pupil3 used %f \n",b3); printf ("Total paper used = %d", total(w1,w2,w3)); } /*****************************************************/ total (a,b,c) int a,b,c; { return (a + b + c); } /* add up total */

Node:Variable parameters, Next:Example 4, Previous:Example 3, Up:Parameters

Variable Parameters
(As a first time reader you may wish to omit this section until you have read about Pointers and Operators.) One way to hand information back is to use the return statement. This function is slightly limited however in that it can only hand the value of one variable back at a time. There is another way of handing back values which is less restrictive, but more awkward than this. This is by using a special kind of parameter, often called a variable

parameter. It is most easily explained with the aid of an example:

#include <stdio.h> main () { int i,j; GetValues (&i,&j); printf ("i = %d and j = %d",i,j) } /************************************/ GetValues (p,q) int *p,*q; { *p = 10; *q = 20; }

To understand fully what is going on in this program requires a knowledge of pointers and operators, which are covered in later sections, but a brief explanation can be given here, so that the method can be used. There are two new things to notice about this program: the symbols & and *. The ampersand & symbol should be read as "the address of..". The star * symbol should be read as "the contents of the address...". This is easily confused with the multiplication symbol (which is identical). The difference is only in the context in which the symbol is used. Fortunately this is not ambiguous since multiplication always takes place between two numbers or variables, whereas the "contents of a pointer" applies only to a single variable and the star precedes the variable name. So, in the program above, it is not the variables themselves which are being passed to the procedure but the addresses of the the variables. In other words, information about where the variables are stored in the memory is passed to the function GetValues(). These addresses are copied into two new variables p and q, which are said to be pointers to i and j. So, with variable parameters, the function does not receive a copy of the variables themselves, but information about how to get at the original variable which was passed. This information can be used to alter the "actual parameters" directly and this is done with the * operator.
*p = 10;

means: Make the contents of the address held in p equal to 10. Recall that the address held in p is the address of the variable i, so this actually reads: make i equal to 10. Similarly:
*q = 20;

means make the contents of the address held in q equal to 20. Other operations are also possible (and these are detailed in the section on pointers) such as finding out the value of i and putting it into a new variable, say, a:
int a; a = *p; /* is equivalent to a = i */

Notice that the * symbol is required in the declaration of these parameters. Node:Example 4, Next:Qulakfj, Previous:Variable parameters, Up:Parameters

Example Listing
/**************************************************/ /* */ /* Program : Variable Parameters */ /* */ /**************************************************/ /* Scale some measurements on a drawing, say */

#include <stdio.h> /**************************************************/ main () { int height,width; height = 4; width = 5; ScaleDimensions (&height,&width); printf ("Scaled height = %d\n",height); printf ("Scaled width = %d\n",width); } /****************************************************/ ScaleDimensions (h,w) int *h, *w; { int hscale = 3; int wscale = 1; *h = *h * hscale; *w = *w * wscale; } /* scale factors */ /* return scaled values */ /* Scale measurements*/

Node:Qulakfj, Previous:Example 4, Up:Parameters

Questions
1. 2. 3. 4. 5. Name two ways that values and results can be handed back from a function. Where are parameters declared? Can a function be used directly as a value parameter? Does it mean anything to use a function directly as a variable parameter? What do the symbols * and & mean, when they are placed in front of an identifier? 6. Do actual and formal parameters need to have the same names? Node:Scope, Next:Preprocessor, Previous:Parameters, Up:Top

Scope : Local And Global

Where a program's fingers can't reach. From the computer's point of view, a C program is nothing more than a collection of functions and declarations. Functions can be thought of as sealed capsules of program code which float on a background of white space, and are connected together by means of function calls. White space is the name given to the white of an imaginary piece of paper upon which a program is written, in other words the spaces and new line

characters which are invisible to the eye. The global white space is only the gaps between functions, not the gaps inside functions. Thinking of functions as sealed capsules is a useful way of understanding the difference between local and global objects and the whole idea of scope in a program. Another analogy is to think of what goes on in a function as being like watching a reality on television. You cannot go in and change the TV reality, only observe the output, but the television show draws its information from the world around it. You can send a parameter (e.g. switch channels) to make some choices. A function called by a function, is like seeing someone watching a televsion, in a television show.

Global variables: Local variables: Parameters again: Example 5: Style note: Scope and style: Questions 11:

Node:Global variables, Next:Local variables, Previous:Scope, Up:Scope

Global Variables
Global variables are declared in the white space between functions. If every function is a ship floating in this sea of white space, then global variables (data storage areas which also float in this sea) can enter any ship and also enter anything inside any ship (See the diagram). Global variables are available everywhere;. they are created when a program is started and are not destroyed until a program is stopped. They can be used anywhere in a program: there is no restriction about where they can be used, in principle. Node:Local variables, Next:Parameters again, Previous:Global variables, Up:Scope

Local Variables
Local variables are more interesting. They can not enter just any region of the program because they are trapped inside blocks. To use the ship analogy: if it is imagined that on board every ship (which means inside every function) there is a large swimming pool with many toy ships floating inside, then local variables will work anywhere in the swimming pool (inside any of the toys ships, but can not get out of the large ship into the wide beyond. The swimming pool is just like a smaller sea, but one which is restricted to being inside a particular function. Every function has its own swimming pool! The idea can be taken further too. What about swimming pools onboard the toy ships? (Meaning functions or blocks inside the functions!
/* Global white space "sea" */ function () { /* On board ship */

{ /* On board a toy ship */ } }

The same rules apply for the toy ships. Variables can reach anywhere inside them but they cannot get out. They cannot escape their block braces {}. Whenever a pair of block braces is written into a program it is possible to make variable declarations inside the opening brace. Like this:
{ int locali; char localch; /* statements */ }

These variables do not exist outside the braces. They are only created when the opening brace is encountered and they are destroyed when the closing brace is executed, or when control jumps out of the block. Because they only work in this local area of a program, they are called local variables. It is a matter of style and efficiency to use local variables when it does not matter whether variables are preserved outside of a particular block, because the system automatically allocates and disposes of them. The programmer does not have to think about this. Where a variable is and is not defined is called the scope of that variable. It tells a programmer what a variables horizons are! Node:Parameters again, Next:Example 5, Previous:Local variables, Up:Scope

Communication : parameters
If functions were sealed capsules and no local variables could ever communicate with other parts of the program, then functions would not be very useful. This is why parameters are allowed. Parameters are a way of handing local variables to other functions without letting them out! Value parameters (see last section) make copies of local variables without actually using them. The copied parameter is then a local variable in another function. In other words, it can't get out of the function to which is it passed ... unless it is passed on as another parameter. Node:Example 5, Next:Style note, Previous:Parameters again, Up:Scope

Example Listing
Notice about the example that if there are two variables of the same name, which are both allowed to be in the same place (c in the example below) then the more local one wins. That is, the last variable to be defined takes priority. (Technically adept readers will realize that this is because it was the last one onto the variable stack.)
/***************************************************************/ /* */ /* SCOPE : THE CLLLED CAPSULES */ /* */ /***************************************************************/ #include <stdio.h> /***************************************************************/ main () { int a = 1, b = 2, c = 3; if (a == 1) { int c; c = a + b; printf ("%d",c); } handdown (a,b); printf ("%d",c); } /**************************************************************/ handdown (a,b) int a,b; { ... } /* Some function */

Node:Style note, Next:Scope and style, Previous:Example 5, Up:Scope

Style Note
Some programmers complain about the use of global variables in a program. One complaint is that it is difficult to see what information is being passed to a function unless all that information is passed as parameters. Sometimes global variables are very useful however, and this problem need not be crippling. A way to make this clear is to write global variables in capital letters only, while writing the rest of the variables in mainly small letters..
int GLOBALINTEGER; .... { int local integer; }

This allows global variables to be spotted easily. Another reason for restricting the use of global variables is that it is easier to debug a program if only local variables are used. The reason is that once a function capsule is tested and sealed it can be guaranteed to work in all cases, provided it is not affected by any other functions from outside. Global variables punch holes in the sealed function capsules because they allow bugs from other functions to creep into tried and tested ones. An alert and careful programmer can usually control this without difficulty. The following guidelines may help the reader to decide whether to use local or global data:

Always think of using a local variable first. Is it impractical? Yes, if it means passing dozens of parameters to functions, or reproducing a lot of variables. Global variables will sometimes tidy up a program. Local variables make the flow of data in a program clearer and they reduce the amount of memory used by the program when they are not in use. The preference in this book is to use local variables for all work, except where a program centres around a single data structure. If a data structure is the main reason for a program's existence, it is nearly always defined globally.

Node:Scope and style, Next:Questions 11, Previous:Style note, Up:Scope

Scope and Style

All the programs in this book, which are longer than a couple of lines, are written in an unusual way: with a levelled structure There are several good reasons for this. One is that the sealed capsules are shown to be sealed, by using a comment bar between each function.
/**************************************/

Another good reason is that any function hands parameters down by only one level at a time and that any return() statement hands values up a single level. The global

variables are kept to a single place at the head of each program so that they can be seen to reach into everything. The diagram shows how the splitting of levels implies something about the scope of variables and the handing of parameters. Node:Questions 11, Previous:Scope and style, Up:Scope

Questions
1. 2. 3. 4. What is a global variable? What is a local variable? What is meant by calling a block (enclosed by braces {} ) a "sealed capsule"? Do parameters make functions leaky? i.e. Do they spoil them by letting the variables leak out into other functions? 5. Write a program which declares 4 variables. Two integer variables called number_of_hats,counter which are GLOBAL and two float variables called x_coord,y_coord which are LOCAL inside the function main(). Then add another function called another() and pass x_coord,y_coord to this function. How many different storage spaces are used when this program runs? (Hint: are x_coord,y_coord and their copies the same?) Node:Preprocessor, Next:Pointers, Previous:Scope, Up:Top

Preprocessor Commands
Making programming versatile. C is unusual in that it has a pre-processor. This comes from its Unix origins. As its name might suggest, the preprocessor is a phase which occurs prior to compilation of a program. The preprocessor has two main uses: it allows external files, such as header files, to be included and it allows macros to be defined. This useful feature traditionally allowed constant values to be defined in Kernighan and Ritchie C, which had no constants in the language. Pre-processor commands are distinguished by the hash (number) symbol #. One example of this has already been encountered for the standard header file stdio.h.
#include <stdio.h>

is a command which tells the preprocessor to treat the file stdio.h as if it were the actually part of the program text, in other words to include it as part of the program to be compiled. Macros are words which can be defined to stand in place of something complicated: they are a way of reducing the amount of typing in a program and a way of making

long ungainly pieces of code into short words. For example, the simplest use of macros is to give constant values meaningful names: e.g.
#define TELEPHNUM 720663

This allows us to use the word TELEPHNUM in the program to mean the number 720663. In this particular case, the word is clearly not any shorter than the number it will replace, but it is more meaningful and would make a program read more naturally than if the raw number were used. For instance, a program which deals with several different fixed numbers like a telephone number, a postcode and a street number could write:
printf("%d %d %d",TELEPHNUM,postcode,streetnum);

instead of
printf("%d %d %d",720663,345,14);

Using the macros instead makes the actions much clearer and allows the programmer to forget about what the numbers actually are. It also means that a program is easy to alter because to change a telephone number, or whatever, it is only necessary to change the definition, not to retype the number in every single instance. The important feature of macros is that they are not merely numerical constants which are referenced at compile time, but are strings which are physically replaced before compilation by the preprocessor! This means that almost anything can be defined:
#define SUM 1 + 2 + 3 + 4

would allow SUM to be used instead of 1+2+3+4. Or

#define STRING "Mary had a little lamb..."

would allow a commonly used string to be called by the identifier "string" instead of typing it out afresh each time. The idea of a define statement then is:
#define macroname definition on rest of line

Macros cannot define more than a single line to be substituted into a program but they can be used anywhere, except inside strings. (Anything enclosed in string quotes is assumed to be complete and untouchable by the compiler.) Some macros are defined already in the file stdio.h such as:
EOF

The end of file character (= -1 for instance)

NULL

The null character (zero) = 0

Macro functions: Macros with parameters:

Example 6: Note about include: Other Preprocessor commands: Example 7: Questions 12:

Node:Macro functions, Next:Macros with parameters, Previous:Preprocessor, Up:Preprocessor

Macro Functions
A more advanced use of macros is also permitted by the preprocessor. This involves macros which accept parameters and hand back values. This works by defining a macro with some dummy parameter, say x. For example: a macro which is usually defined in one of the standard libraries is abs() which means the absolute or unsigned value of a number. It is defined below:
#define ABS(x) ((x) < 0) ? -(x) : (x)

The result of this is to give the positive (or unsigned) part of any number or variable. This would be no problem for a function which could accept parameters, and it is, in fact, no problem for macros. Macros can also be made to take parameters. Consider the ABS() example. If a programmer were to write ABS(4) then the preprocessor would substitute 4 for x. If a program read ABS(i) then the preprocessor would substitute i for x and so on. (There is no reason why macros can't take more than one parameter too. The programmer just includes two dummy parameters with different names. See the example listing below.) Notice that this definition uses a curious operator which belongs to C:
<test> ? <true result> : <false result>

This is like a compact way of writing an if..then..else statement, ideal for macros. But it is also slightly different: it is an expression which returns a value, where as an if..then..else is a statement with no value. Firstly the test is made. If the test is true then the first statement is carried out, otherwise the second is carried out. As a memory aid, it could be read as:
if <test> then <true result> else <false result>

(Do not be confused by the above statement which is meant to show what a programmer might think. It is not a valid C statement.) C can usually produce much more efficient code for this construction than for a corresponding if-else statement. Node:Macros with parameters, Next:Example 6, Previous:Macro functions, Up:Preprocessor

When and when not to use macros with parameters

It is tempting to forget about the distinction between macros and functions, thinking that it can be ignored. To some extent this is true for absolute beginners, but it is not a good idea to hold on to. It should always be remembered that macros are substituted whole at every place where they are used in a program: this is potentially a very large amount of repetition of code. The advantage of a macro, however, is speed. No time is taken up in passing control over to a new function, because control never leaves the home function when a macro is used: it just makes the function a bit longer. There is a limitation with macros though. Function calls cannot be used as their parameters, such as:
ABS(function())

has no meaning. Only variables or number constants will be substituted. Macros are also severely restricted in complexity by the limitations of the preprocessor. It is simply not viable to copy complicated sequences of code all over programs. Choosing between functions and macros is a matter of personal judgement. No simple rules can be given. In the end (as with all programming choices) it is experience which counts towards the final ends. Functions are easier to debug than macros, since they allow us to single step through the code. Errors in macros are very hard to find, and can be very confusing. Node:Example 6, Next:Note about include, Previous:Macros with parameters, Up:Preprocessor

Example Listing
/************************************************************/ /* */ /* MACRO DEMONSTRATION */ /* */ /************************************************************/ #include <stdio.h> #define #define #define #define #define #define #define STRING1 STRING2 EXPRESSION EXPR2 ABS(x) MAX(a,b) BIGGEST(a,b,c) "A macro definition\n" "must be all on one line!!\n" 1 + 2 + 3 + 4 EXPRESSION + 10 ((x) < 0) ? -(x) : (x) (a < b) ? (b) : (a) (MAX(a,b) < c) ? (c) : (MAX(a,b))

/************************************************************/ main () { printf printf printf printf printf printf } /* No #definitions inside functions! */ (STRING1); (STRING2); ("%d\n",EXPRESSION); ("%d\n",EXPR2); ("%d\n",ABS(-5)); ("Biggest of 1 2 and 3 is %d",BIGGEST(1,2,3));

Node:Note about include, Next:Other Preprocessor commands, Previous:Example 6, Up:Preprocessor

Note about #include

When an include statement is written into a program, it is a sign that a compiler should merge another file of C programming with the current one. However, the #include statement is itself valid C, so this means that a file which is included may contain #includes itself. The includes are then said to be "nested". This often makes includes simpler. Node:Other Preprocessor commands, Next:Example 7, Previous:Note about include, Up:Preprocessor

Other Preprocessor commands

This section lies somewhat outside the main development of the book. You might wish to omit it on a first reading. There are a handful more preprocessor commands which can largely be ignored by the beginner. They are commonly used in "include" files to make sure that things are not defined twice. NOTE : true has any non zero value in C. false is zero.
#undef #if

This undefines a macro, leaving the name free. This is followed by some expression on the same line. It allows conditional compilation. It is an advanced feature which can be used to say: only compile the code between #if and #endif if the value following #if is true, else leave out that code altogether. This is different from not executing code--the code will not even be compiled.

#ifdef #ifndef #else #endif #line #line constant filename

This is followed by a macro name. If that macro is defined then this is true. This is followed by a macro name. If that name is not defined then this is true. This is part of an #if, #ifdef, #ifndef preprocessor statement. This marks the end of a preprocessor statement. Has the form:

This is for debugging mainly. This statement causes the compiler to believe that the next line is line number (constant) and is part of the file (filename).
#error

This is a part of the proposed ANSI standard. It is intended for debugging. It forces the compiler to abort compilation.

Node:Example 7, Next:Questions 12, Previous:Other Preprocessor commands, Up:Preprocessor

Example
/***********************************************************/ /* To compile or not to compile */ /***********************************************************/ #define SOMEDEFINITION 6546 #define CHOICE 1 /* Choose this before compiling */ /***********************************************************/ #if (CHOICE == 1) #define OPTIONSTRING "The programmer selected this" #define DITTO "instead of .... " #else #define OPTIONSTRING "The alternative" #define DITTO "i.e. This! " #endif /***********************************************************/ #ifdef SOMEDEFINITION #define WHATEVER "Something was defined!" #else #define WHATEVER "Nothing was defined" #endif /************************************************************/ main () { printf (OPTIONSTRING); printf (DITTO); }

Node:Questions 12, Previous:Example 7, Up:Preprocessor

Questions
1. Define a macro called "birthday" which describes the day of the month upon which your birthday falls. 2. Write an instruction to the preprocessor to include to maths library math.h. 3. A macro is always a number. True or false? 4. A macro is always a constant. True or false?

Node:Pointers, Next:Standard Output and Standard Input, Previous:Preprocessor, Up:Top

Pointers
Making maps of data. You have a map (a plan) of the computer's memory. You need to find that essential piece of information which is stored at some unknown location. How will you find it? You need a pointer! A pointers is a special type of variable which holds the address or location of another variable. Pointers point to these locations by keeping a record of the spot at which they were stored. Pointers to variables are found by recording the address at which a variable is stored. It is always possible to find the address of a piece of storage in C using the special & operator. For instance: if location were a float type variable, it would be easy to find a pointer to it called location_ptr.
float location; float *location_ptr,*address; location_ptr = &(location);

or
address = &(location);

The declarations of pointers look a little strange at first. The star * symbol which stands in front of the variable name is C's way of declaring that variable to be a pointer. The four lines above make two identical pointers to a floating point variable called location, one of them is called location_ptr and the other is called address. The point is that a pointer is just a place to keep a record of the address of a variable, so they are really the same thing. A pointer is a bundle of information that has two parts. One part is the address of the beginning of the segment of memory that holds whatever is pointed to. The other part is the type of value that the pointer points to the beginning of. This tells the computer how much of the memory after the beginning to read and how to interpret it. Thus, if the pointer is of a type int, the segment of memory returned will be four bytes long (32 bits) and be interpreted as an integer. In the case of a function, the type is the type of value that the function will return, although the address is the address of the beginning of the function executable. If, like some modern day programmers, you believe in sanctity of high level languages, it is probably a source of wonder why anyone Would ever want to know the address of these variables. Having gone to the trouble to design a high level language, like C, in which variables can be given elegant and meaningful names: it seems like a step in the backward direction to want to be able to find out the exact number of the memory

location at which it is stored! The whole point of variables, after all, is that it is not necessary to know exactly where information is really stored. This is not quite fair though. It is certainly rare indeed when we should want to know the actual number of the memory location at which something is stored. That would really make the idea of a high level language a bit pointless. The idea behind pointers is that a high level programmer can now find out the exact location of a variable without ever having to know the actual number involved. Remember: A pointer is a variable which holds the address of the storage location for another given variable. C provides two operators & and * which allow pointers to be used in many versatile ways.

Pointer operators: Uses for pointers: Pointers and Initialization: Example 8: Types Casts and Pointers: Function pointers: Calling functions by pointer: Questions 13:

Node:Pointer operators, Next:Uses for pointers, Previous:Pointers, Up:Pointers

&

and *

The & and * operators have already been used once to hand back values to variable parameters, See Value parameters. They can be read in a program to have the following meanings:
& *

The address of... The contents of the address held in...

Another way of saying the second of these is:

*

The contents of the location pointed to by... This reinforces the idea that pointers reach out an imaginary hand and point to some location in the memory and it is more usual to speak of pointers in this way. The two operators * and & are always written in front of a variable, clinging on, so that they refer, without doubt, to that one variable. For instance:
&x

The address at which the variable x is stored.

*ptr

The contents of the variable which is pointed to by ptr. The following example might help to clarify the way in which they are used:

int somevar; int *ptr_to_somevar; somevar = 42; ptr_to_somevar = &(somevar); printf ("%d",*ptr_to_somevar); *ptr_to_somevar = 56;

/* 1 */ /* 2 */ /* 3 */ /* 4 */ /* 5 */ /* 6 */

The key to these statements is as follows: 1. Declare an int type variable called somevar. 2. Declare a pointer to an int type called ptr_to_somevar. The * which stands in front of ptr_to_somevar is the way C declares ptr_to_somevar as a pointer to an integer, rather than an integer. 3. Let somevar take the value 42. 4. This gives a value to ptr_to_somevar. The value is the address of the variable somevar. Notice that only at this stage does is become a pointer to the particular variable somevar. Before this, its fate is quite open. The declaration (2) merely makes it a pointer which can point to any integer variable which is around. 5. Print out "the contents of the location pointed to by ptr_to_somevar" in other words somevar itself. So this will be just 42. 6. Let the contents of the location pointed to by ptr_to_somevar be 56. This is the same as the more direct statement:
7. 8. somevar = 56;

Node:Uses for pointers, Next:Pointers and Initialization, Previous:Pointer operators, Up:Pointers

Uses for Pointers

It is possible to have pointers which point to any type of data whatsoever. They are always declared with the * symbol. Some examples are given below.
int i,*ip; char ch,*chp; short s,*sp; float x,*xp; double y,*yp;

Pointers are extremely important objects in C. They are far more important in C than in, say, Pascal or BASIC (PEEK,POKE are like pointers). In particular they are vital when using data structures like strings or arrays or linked lists. We shall meet these objects in later chapters. One example of the use of pointers is the C input function, which is called scanf(). It is looked at in detail in the next section. scanf() is for getting information from the keyboard. It is a bit like the reverse of printf(), except that it uses pointers to variables, not variables themselves. For example: to read an integer:

int i; scanf ("%d",&i);

or
int *i; scanf ("%d",i);

The & sign or the * sign is vital. If it is forgotten, scanf will probably corrupt a program. This is one reason why this important function has been ignored up to now. Assembly language programmers might argue that there are occasions on which it would be nice to know the actual address of a variable as a number. One reason why one might want to know this would be for debugging. It is not often a useful thing to do, but it is not inconceivable that in developing some program a programmer would want to know the actual address. The & operator is flexible enough to allow this to be found. It could be printed out as an integer:
type *ptr: printf ("Address = %d",(int) ptr);

Node:Pointers and Initialization, Next:Example 8, Previous:Uses for pointers, Up:Pointers

Pointers and Initialization

Something to be wary of with pointer variables is the way that they are initialized. It is incorrect, logically, to initialize pointers in a declaration. A compiler will probably not prevent this however because there is nothing incorrect about it as far as syntax is concerned. Think about what happens when the following statement is written. This statement is really talking about two different storage places in the memory:
int *a = 2;

First of all, what is declared is a pointer, so space for a `pointer to int' is allocated by the program and to start off with that space will contain garbage (random numbers), because no statement like
a = &someint;

has yet been encountered which would give it a value. It will then attempt to fill the contents of some variable, pointed to by a, with the value 2. This is doomed to faliure. a only contains garbage so the 2 could be stored anywhere. There may not even be a variable at the place in the memory which a points to. Nothing has been said about that yet. This kind of initialization cannot possibly work and will most likely crash the program or corrupt some other data.

Node:Example 8, Next:Types Casts and Pointers, Previous:Pointers and Initialization, Up:Pointers

Example Listing
/**********************************************/ /* */ /* Swapping Pointers */ /* */ /**********************************************/ /* Program swaps the variables which a,b */ /* point to. Not pointless really ! */ #include <stdio.h> main () { int *a,*b,*c; int A,B; A = 12; B = 9; a = &A; b = &B; /* Declr ptrs */ /* Declare storage */ /* Initialize storage */ /* Initialize pointers */

printf ("%d %d\n",*a,*b); c = a; a = b; b = c; printf ("%d %d\n",*a,*b); } /* swap pointers */

Node:Types Casts and Pointers, Next:Function pointers, Previous:Example 8, Up:Pointers

Types, Casts and Pointers

It is tempting but incorrect to think that a pointer to an integer is the same kind of object as a pointer to a floating point object or any other type for that matter. This is not necessarily the case. Compilers distinguish between pointers to different kinds of objects. There are occasions however when it is actually necessary to convert one kind of pointer into another. This might happen with a type of variable called "unions" or even functions which allocate storage for special uses. These objects are met later on in this book. When this situation comes about, the cast operator has to be used to make sure that pointers have compatible types when they are assigned to one another. The cast operator for variables, See The Cast Operator, is written in front of a variable to force it to be a particular type:
(type) variable

For pointers it is:

(type *) pointer

Look at the following statement:

char *ch; int *i; i = (int *) ch;

This copies the value of the pointer ch to the pointer i. The cast operator makes sure that the pointers are in step and not talking at cross purposes. The reason that pointers have to be `cast' into shape is a bit subtle and depends upon particular computers. In practice it may not actually do anything, but it is a necessary part of the syntax of C. Pointer casting is discussed in greater detail in the chapter on Structures and Unions. Node:Function pointers, Next:Calling functions by pointer, Previous:Types Casts and Pointers, Up:Pointers

Pointers to functions
This section is somewhat outside of the main development of the book. You might want to omit it on first reading. Let's now consider pointers to functions as opposed to variables. This is an advanced feature which should be used with more than a little care. The idea behind pointers to functions is that you can pass a function as a parameter to another function! This seems like a bizarre notion at first but in fact it makes perfect sense. Pointers to functions enable you to tell any function which sub-ordinate function it should use to do its job. That means that you can plug in a new function in place of an old one just by passing a different parameter value to the function. You do not have to rewrite any code. In machine code circles this is sometimes called indirection or vectoring. When we come to look at arrays, we'll find that a pointer to the start of an array can be found by using the name of the array itself without the square brackets []. For functions, the name of the function without the round brackets works as a pointer to the start of the function, as long as the compiler understands that the name represents the function and not a variable with the same name. So--to pass a function as a parameter to another function you would write
function1(function2);

If you try this as it stands, a stream of compilation errors will be the result. The reason is that you must declare function2() explicitly like this:
int function2();

If the function returns a different type then clearly the declaration will be different but the form will be the same. The declaration can be placed together with other declarations. It is not important whether the variable is declared locally or globally, since a function is a global object regardless. What is important is that we declare specifically a pointer to a function which returns a type (even if it is void). The function which accepts a function pointer as an argument looks like this:
function1 (a)

int (*a)(); { int i; i = (*a)(parameters); }

This declares the formal parameter a to be a pointer to a function returning a value of type int. Similarly if you want to declare a pointer to a function to a general type typename with the name fnptr, you would do it like this:
typename (*fnptr)();

Node:Calling functions by pointer, Next:Questions 13, Previous:Function pointers, Up:Pointers

Calling a function by pointer

Given a pointer to a function how do we call the function? The syntax is this:
variable = (*fnptr)(parameters);

An example let us look at a function which takes an integer and returns a character.
int i; char ch, function();

Normally this function is called using the statement:

ch = function(i);

but we can also do the same thing with a pointer to the function. First define
char function(); char (*fnptr)(); fnptr = function;

then call the function with

ch = (*fnptr)(i);

A pointer to a function can be used to provide a kind of plug-in interface to a logical device, i.e. a way of choosing the right function for the job.
void printer(),textscreen(),windows(); switch (choice) { case 1: fnptr = printer; break; case 2: fnptr = textscreen; break; case 3: fnptr = windows;

} Output(data,fnptr);

This is the basis of `polymorphism' found in object oriented languages: a choice of a logical (virtual) function based on some abstract label (the choice). The C++ language provides an abstract form of this with a more advanced syntax, but this is the essence of virtual function methods in object oriented languages. BEWARE! A pointer to a function is an automatic local variable. Local variables are never initialized by the compiler in C. If you inadvertently forget to initialize the pointer to a function, you will come quickly to grief. Make sure that your pointers are assigned before you use them! Node:Questions 13, Previous:Calling functions by pointer, Up:Pointers

Questions
1. 2. 3. 4. What is a pointer? How is a variable declared to be a pointer? What data types can pointers "point to"? Write a statement which converts a pointer to a character into a pointer to a double type. (This is not as pointless as it seems. It is useful in dealing with unions and memory allocation functions.) 5. Why is it incorrect to declare: float *number = 2.65; ? Node:Standard Output and Standard Input, Next:Assignments Expressions and Operators, Previous:Pointers, Up:Top

Standard Output and Standard Input

Talking to the user. Getting information in and out of a computer is the most important thing that a program can do. Without input and output computers would be quite useless. C treats all its output as though it were reading or writing to different files. A file is really just an abtraction: a place where information comes from or can be sent to. Some files can only be read, some can only be written to, others can be both read from

and written to. In other situations files are called I/O streams.

C has three files (also called streams) which are always open and ready for use. They are called stdin, stdout and stderr, meaning standard input and standard output and standard error file. Stdin is the input which usually arrives from the keyboard of a computer. stdout is usually the screen. stderr is the route by which all error messages pass: usually the screen. This is only `usually' because the situation can be altered. In fact what happens is that these files are just handed over to the local operating system to deal with and it chooses what to do with them. Usually this means the keyboard and the screen, but it can also be redirected to a printer or to a disk file or to a modem etc.. depending upon how the user ran the program. The keyboard and screen are referred to as the standard input/output files because this is what most people use, most of the time. Also the programmer never has to open or close these, because C does it automatically. The C library functions covered by stdio.h provides some methods for working with stdin and stdout. They are simplified versions of the functions that can be used on any kind of file, See Files and Devices. In order of importance, they are:
printf () scanf () getchar() putchar() gets () puts ()

printf again: Example 9: Output 9: Formatting with printf: Example 10: Output 10: Special Control Characters again: Questions 15: scanf: Conversion characters: How does scanf see the input: First account of scanf: The dangerous function: Keeping scanf under control: Example 11: Matching without assigning: Formal Definition of scanf: Summary of points about scanf: Questions 15b: Low Level Input/Output: Questions 15c:

Node:printf again, Next:Example 9, Previous:Standard Output and Standard Input, Up:Standard Output and Standard Input
printf

The printf function has been used widely up to now for output because it provides a neat and easy way of printing text and numbers to stdout (the screen). Its name is meant to signify formatted printing because it gives the user control over how text and numerical data are to be laid out on the screen. Making text look good on screen is important in programming. C makes this easy by allowing you to decide how the text will be printed in the available space. The printf function has general form:
printf ("string...",variables,numbers)

It contains a string (which is not optional) and it contains any number of parameters to follow: one for each blank field in the string. The blank fields are control sequences which one can put into the string to be filled in with numbers or the contents of variables before the final result is printed out. These fields are introduced by using a % character, followed by some coded information, which says something about the size of the blank space and the type of number or string which will be filled into that space. Often the string is called the control string because it contains these control characters. The simplest use of printf is to just print out a string with no blank fields to be filled:
printf ("A pretty ordinary string.."); printf ("Testing 1,2,3...");

The next simplest case that has been used before now is to print out a single integer number:
int number = 42; printf ("%d",number);

The two can be combined:

int number = 42; printf ("Some number = %d",number);

The result of this last example is to print out the following on the screen:
Some number = 42

The text cursor is left pointing to the character just after the 2. Notice the way that %d is swapped for the number 42. %d defines a field which is filled in with the value of the variable. There are other kinds of data than integers though. Any kind of variable can be printed out with printf. %d is called a conversion character for integers because it tells the compiler to treat the variable to be filled into it as an integer. So it better had be an integer or things will go wrong! Other characters are used for other kinds of data. Here is a list if the different letters for printf.
d

signed denary integer

u x o s c f e g

unsigned denary integer hexadecimal integer octal integer string single character fixed decimal floating point scientific notation floating point use f or e, whichever is shorter

The best way to learn these is to experiment with different conversion characters. The example program and its output below give some impression of how they work: Node:Example 9, Next:Output 9, Previous:printf again, Up:Standard Output and Standard Input

Example Listing
/*******************************************************/ /* */ /* printf Conversion Characters and Types */ /* */ /*******************************************************/ #include <stdio.h> main () { int i = -10; unsigned int ui = 10; float x = 3.56; double y = 3.52; char ch = 'z'; char *string_ptr = "any old string"; printf ("signed integer %d\n", i); printf ("unsigned integer %u\n",ui); printf ("This is wrong! %u",i); printf ("See what happens when you get the "); printf ("character wrong!"); printf ("Hexadecimal %x %x\n",i,ui); printf ("Octal %o %o\n",i,ui); printf ("Float and double %f %f\n",x,y); printf (" ditto %e %e\n",x,y); printf (" ditto %g %g\n",x,y); printf ("single character %c\n",ch); printf ("whole string -> %s",string_ptr); }

Node:Output 9, Next:Formatting with printf, Previous:Example 9, Up:Standard Output and Standard Input

Output
signed integer -10 unsigned integer 10 This is wrong! 10See what happens when you get the character wrong! Hexadecimal FFFFFFF6 A Octal 37777777766 12 Float and double 3.560000 3.520000 ditto 3.560000E+00 3.520000E+00 ditto 3.560000 3.520000 single character z whole string -> any old string

Node:Formatting with printf, Next:Example 10, Previous:Output 9, Up:Standard Output and Standard Input

Formatting with printf

The example program above does not produce a very neat layout on the screen. The conversion specifiers in the printf string can be extended to give more information. The % and the character type act like brackets around the extra information. e.g.
%-10.3f

is an extended version of %f, which carries some more information. That extra information takes the form:
% [-] [fwidth] [.p] X

where the each bracket is used to denote that the item is optional and the symbols inside them stand for the following.
[fwidth]

[-]

This is a number which specifies the field width of this "blank field". In other words, how wide a space will be made in the string for the object concerned? In fact it is the minimum field width because if data need more room than is written here they will spill out of their box of fixed size. If the size is bigger than the object to be printed, the rest of the field will be filled out with spaces. If this included the output will be left justified. This means it will be aligned with the left hand margin of the field created with [fwidth]. Normally all numbers are right justified, or aligned with the right hand margin of the field "box".

[.p]

This has different meanings depending on the object which is to be printed. For a floating point type (float or double) p specifies the number of decimal places after the point which are to be printed. For a string it specifies how many characters are to be printed. Some valid format specifiers are written below here.
%10d %2.2f %25.21s %2.6f

The table below helps to show the effect of changing these format controls. The width of a field is draw in by using the | bars.
Object to be printed 42 42 324 -1 -1 'z' 'z' 2.71828 2.71828 2.71828 2.71828 Control Spec. %6d %-6d %10d %-10d %1d %3c %-3c %10f %10.2f %-10.2f %2.4f Actual Output | 42| |42 | | 324| |-1 | |-1|(overspill) | z| |z | | 2.71828| | 2.71| |2.71 | |2.7182|(overspill)

2.718 2.718 2.71828 2.71828 2.71828 "printf" "printf" "printf" "printf" "printf" "printf"

%.4f %10.5f %10e %10.2e %10.2g %s %10s %2s %5.3s %-5.3s %.3s

|2.7180| | 2.71800| |2.71828e+00| | 2.17e+00| | 2.71| |printf| | printf| |printf|(overspill) | pri| |pri | |pri|

Node:Example 10, Next:Output 10, Previous:Formatting with printf, Up:Standard Output and Standard Input

Example Listing
/***********************************************/ /* */ /* Multiplication Table */ /* */ /***********************************************/ #include <stdio.h> main () { int i,j; for (i = 1; i <= 10; i++) { for (j = 1; j <= 10; j++) { printf ("%5d",i * j); } printf ("\n"); } } /* Printing in columns */

Node:Output 10, Next:Special Control Characters again, Previous:Example 10, Up:Standard Output and Standard Input

Output
1 2 3 4 5 6 7 8 9 10 2 4 6 8 10 12 14 16 18 20 3 6 9 12 15 18 21 24 27 30 4 8 12 16 20 24 28 32 36 40 5 10 15 20 25 30 35 40 45 50 6 12 18 24 30 36 42 48 54 60 7 14 21 28 35 42 49 56 63 70 8 16 24 32 40 48 56 64 72 80 9 18 27 36 45 54 63 72 81 90 10 20 30 40 50 60 70 80 90 100

Node:Special Control Characters again, Next:Questions 15, Previous:Output 10, Up:Standard Output and Standard Input

Special Control Characters

Control characters are invisible on the screen. They have special purposes usually to do with cursor movement. They are written into an ordinary string by typing a backslash character \ followed by some other character. These characters are listed below.
\b

backspace BS
\f \n \r \t \v \" \' \\ \ddd

form feed FF (also clear screen) new line NL (like pressing return) carriage return CR (cursor to start of line) horizontal tab HT vertical tab double quote single quote character ' backslash character \ character ddd where ddd is an ASCII code given in octal or base 8, See Character Conversion Table.

\xddd

character ddd where ddd is an ASCII code given in hexadecimal or base 16, See Character Conversion Table. Node:Questions 15, Next:scanf, Previous:Special Control Characters again, Up:Standard Output and Standard Input

Questions
1. Write a program which simply prints out: 6.23e+00 2. Investigate what happens when you type the wrong conversion specifier in a program. e.g. try printing an integer with %f or a floating point number with %c. This is bound to go wrong - but how will it go wrong? 3. What is wrong with the following statements? 1. printf (x); 2. printf ("%d"); 3. printf (); 4. printf ("Number = %d"); Hint: if you don't know, try them in a program!

Node:scanf, Next:Conversion characters, Previous:Questions 15, Up:Standard Output and Standard Input
scanf

is the input function which gets formatted input from the file stdin (the keyboard). This is a very versatile function but it is also very easy to go wrong with. In fact it is probably the most difficult to understand of all the C standard library functions.
scanf

Remember that C treats its keyboard input as a file. This makes quite a difference to the way that scanf works. The actual mechanics of scanf are very similar to those of printf in reverse
scanf ("string...",pointers);

with one important exception: namely that it is not variables which are listed after the control string, but pointers to variables. Here are some valid uses of scanf:
int i; char ch; float x; scanf ("%d %c %f", &i, &ch, &x);

Notice the & characters which make the arguments pointers. Also notice the conversion specifiers which tell scanf what types of data it is going to read. The other possibility is that a program might already have pointers to a particular set of variables in that case the & is not needed. For instance:
function (i,ch,x) int *i; char *ch; float *x; { scanf ("%d %c %f", i, ch, x); }

In this case it would actually be wrong to write the ampersand & symbol. Node:Conversion characters, Next:How does scanf see the input, Previous:scanf, Up:Standard Output and Standard Input

Conversion characters
The conversion characters for scanf are not identical to those for printf and it is much more important to be precise and totally correct with these than it is with printf.
d

ld x o h f lf e le c s

denary integer (int or long int) long decimal integer hexadecimal integer octal integer short integer float type long float or double float type double single character character string

The difference between short integer and long integer can make or break a program. If it is found that a program's input seems to be behaving strangely, check these carefully. (See the section on Errors and Debugging for more about this.) Node:How does scanf see the input, Next:First account of scanf, Previous:Conversion characters, Up:Standard Output and Standard Input

How does scanf see the input?

When scanf is called in a program it checks to see what is in the input file, that is, it checks to see what the user has typed in at the keyboard. Keyboard input is usually buffered. This means that the characters are held in a kind of waiting bay in the memory until they are read. The buffer can be thought of as a part of the input file stdin, holding some characters which can be scanned though. If the buffer has some characters in it, scanf will start to look through these; if not, it will wait for some characters to be put into the buffer. There is an important point here: although scanf will start scanning through characters as soon as they are in the buffer, the operating system often sees to it that scanf doesn't get to know about any of the characters until the user has pressed the RETURN or ENTER key on the computer or terminal. If the buffer is empty scanf will wait for some characters to be put into it. To understand how scanf works, it is useful to think of the input as coming in `lines'. A line is a bunch of characters ending in a newline character \n. This can be represented by a box like the one below:
-------------------------------------| some...chars.738/. |'\n'| --------------------------------------

As far as scanf is concerned, the input is entirely made out of a stream of characters. If the programmer says that an integer is to be expected by using the %d conversion specifier then scanf will try to make sense of the characters as an integer. In other words, it will look for some characters which make up a valid integer, such as a group of numbers all between 0 and 9. If the user says that floating point type is expected then it will look for a number which may or may not have a decimal point in it. If the user just wants a character then any character will do! Node:First account of scanf, Next:The dangerous function, Previous:How does scanf see the input, Up:Standard Output and Standard Input

First account of scanf

Consider the example which was give above.
int i; char ch; float x; scanf ("%d %c %f", &i, &ch, &x);

Here is a simplified, ideal view of what happens. scanf looks at the control string and finds that the first conversion specifier is %d which means an integer. It then tries to find some characters which fit the description of an integer in the input file. It skips over any white space characters (spaces, newlines) which do not constitute a valid integer until it matches one. Once it has matched the integer and placed its value in the variable i it carries on and looks at the next conversion specifier %c which means a character. It takes the next character and places it in ch. Finally it looks at the last conversion specifier %f which means a floating point number and finds some characters which fit the description of a floating point number. It passes the value onto the variable x and then quits. This brief account of scanf does not tell the whole story by a long way. It assumes that all the characters were successfully found and that everything went smoothly: something which seldom happens in practice! Node:The dangerous function, Next:Keeping scanf under control, Previous:First account of scanf, Up:Standard Output and Standard Input

The dangerous function

What happens if scanf doesn't find an integer or a float type? The answer is that it will quit at the first item it fails to match, leaving that character and the rest of the input line still to be read in the file. At the first character it meets which does not fit in with the conversion string's interpretation scanf aborts and control passes to the next C statement. This is why scanf is a `dangerous' function: because it can quit in the middle of a task and leave a lot of surplus data around in the input file. These surplus data simply wait in the input file until the next scanf is brought into operation, where they can also cause it to quit. It is not safe, therefore, to use scanf by itself: without some check that it is working successfully.

is also dangerous for the opposite reason: what happens if scanf doesn't use up all the characters in the input line before it satisfies its needs? Again the answer is that it quits and leaves the extra characters in the input file stdin for the next scanf to read, exactly where it left off. So if the program was meant to read data from the input and couldn't, it leaves a mess for something else to trip over. scanf can get out of step with its input if the user types something even slightly out of line. It should be used with caution...
scanf

Node:Keeping scanf under control, Next:Example 11, Previous:The dangerous function, Up:Standard Output and Standard Input

Keeping scanf under control

may be dangerous for sloppy programs which do not check their input carefully, but it is easily tamed by using it as just a part of a more sophisticated input routine and sometimes even more simply with the aid of a very short function which can be incorporated into any program:
scanf skipgarb() /* skip garbage corrupting scanf */

{ while (getchar() != '\n') { } }

The action of this function is simply to skip to the end of the input line so that there are no characters left in the input. It cannot stop scanf from getting out of step before the end of a line because no function can stop the user from typing in nonsense! So to get a single integer, for instance, a program could try:
int i; scanf("%d",&i); skipgarb();

The programmer must police user-garbage personally by using a loop to the effect of:
while (inputisnonsense) { printf ("Get your act together out there!!\n"); scanf (..) skipgarb(); }

It is usually as well to use skipgarb() every time. Node:Example 11, Next:Matching without assigning, Previous:Keeping scanf under control, Up:Standard Output and Standard Input

Examples

Here are some example programs with example runs to show how scanf either works or fails.
/****************************************/ /* Example 1 */ /****************************************/ #include <stdio.h> main () { int i = 0; char ch = '*'; float x = 0; scanf ("%d %c %f",&i,&ch,&x); printf ("%d %c %f\n",i,ch,x); }

This program just waits for a line from the user and prints out what it makes of that line. Things to notice about these examples are the way in which scanf `misunderstands' what the user has typed in and also the values which the variables had

before the scanf function.

Input : 1x2.3 Output: 1 x 2.300000

The input gets broken up in the following way:

-----------------| 1 |'x'| 2.3 |'\n'| ------------------

In this example everything works properly. There are no spaces to confuse matters. it is simple for scanf to see what the first number is because the next character is x which is not a valid number.
Input : 1 x 2.3 Output: 1 0.000000 ----------|1|' '| <break> |x 2.3| -----------

In this example the integer is correctly matched as 1. The character is now a space and the x is left in the stream. The x does not match the description of a float value so scanf terminates, leaving x 2.3 still in the input stream.
Input : . Output: 0 * 0.000000 --|'.'| <break> ---

A single full-stop (period). scanf quits straight away because it looks for an integer. It leaves the whole input line (which is just the period .) in the input stream.
/****************************************/ /* Example 2 */ /****************************************/

#include <stdio.h> main () { int i = 0; char ch = '*',ch2,ch3; float x = 0; scanf ("%d %c %f", &i,&ch,&x); scanf ("%c %c", &ch2,&ch3); printf ("%d %c %f\n %c %c"); }

The input for this program is:

6 x2.36 and the output is: 6 0.000000 x 2 --------------------| 6 | ' ' | <break> |'x'|'2'| .36 | ---------------------

Here the integer is successfully matched with 6. The character is matched with a space but the float character finds an x in the way, so the first scanf aborts leaving the value of x unchanged and the rest of the characters still in the file. The second scanf function then picks these up. It can be seen that the first two characters are the x which caused the previous scanf to fail and the first 2 of the intended floating point number.
/****************************************/ /* Example 3 */ /****************************************/ #include <stdio.h> main() { char ch1,ch2,ch3; scanf ("%c %c %c",&ch1,&ch2,&ch3); printf ("%c %c %c",ch1,ch2,ch3); }

Trials:
input : abc output: a b c input : a [return] b [return] c [return] output: a b c

input : 2.3 output: 2 . 3

Node:Matching without assigning, Next:Formal Definition of scanf, Previous:Example 11, Up:Standard Output and Standard Input

Matching without assigning

allows input types to be matched but then discarded without being assigned to any variable. It also allows whole sequences of characters to be matched and skipped. For example:
scanf scanf ("%*c");

would skip a single character. The * character means do not make an assignment. Note carefully that the following is wrong:
scanf ("%*c", &ch);

A pointer should not be given for a dummy conversion character. In this simple case above it probably does not matter, but in a string with several things to be matched, it would make the conversion characters out of step with the variables, since scanf does not return a value from a dummy conversion character. It might seem as though there would be no sense in writing:
scanf ("%*s %f %c",&x,&ch);

because the whole input file is one long string after all, but this is not true because, as far as scanf is concerned a string is terminated by any white space character, so the float type x and the character ch would receive values provided there were a space or newline character after any string. If any non-conversion characters are typed into the string scanf will match and skip over them in the input. For example:
scanf (" Number = %d",&i);

If the input were: Number = 256, scanf would skip over the Number = . As usual, if the string cannot be matched, scanf will abort, leaving the remaining characters in the input stream.
/****************************************/ /* Example 4 */ /****************************************/ #include <stdio.h> main() { float x = 0; int i = 0; char ch = '*'; scanf("Skipthis! %*f %d %*c",&i);

printf("%f %d %c",x,i,ch); } Input : Skipthis! 23 Output: 0.000000 23 * Input : 26 Output: 0.000000 0 *

In this last case scanf aborted before matching anything. Node:Formal Definition of scanf, Next:Summary of points about scanf, Previous:Matching without assigning, Up:Standard Output and Standard Input

Formal Definition of scanf

The general form of the scanf function is:
n = scanf ("string...", pointers);

The value n returned is the number of items matched or the end of file character EOF, or NULL if the first item did not match. This value is often discarded. The control string contains a number of conversion specifiers with the following general form:
%[*][n]X [*] [n]

the optional assignment suppression character. this is a number giving the maximum field width to be accepted by scanf for a particular item. That is, the maximum number of characters which are to be thought of as being part of one the current variable value.

is one of the characters listed above. Any white space characters in the scanf string are ignored. Any other characters are matched. The pointers must be pointers to variables of the correct type and they must match the conversion specifiers in the order in which they are written. There are two variations on the conversion specifiers for strings, though it is very likely that many compilers will not support this. Both of the following imply strings:
%[set of characters]

a string made up of the given characters only.

%[^set of characters]

a string which is delimited by the set of characters given. For example, to read the rest of a line of text, up to but not including the end of line, into a string array one would write:
scanf("%[^\n]",stringarray);

Node:Summary of points about scanf, Next:Questions 15b, Previous:Formal Definition of scanf, Up:Standard Output and Standard Input

Summary of points about scanf

Scanf works across input lines as though it were dealing with a file. Usually the user types in a line and hits return. The whole line is then thought of as being part of the input file pointer stdin. If scanf finds the end of a line early it will try to read past it until all its needs are satisfied. If scanf fails at any stage to match the correct type of string at the correct time, it will quit leaving the remaining input still in the file. If an element is not matched, no value will be assigned to the corresponding variable. White space characters are ignored for all conversion characters except %c. Only a %c type can contain a white space character. White space characters in

Node:Questions 15b, Next:Low Level Input/Output, Previous:Summary of points about scanf, Up:Standard Output and Standard Input

Questions
1. What is a white space character? 2. Write a program which fetches two integers from the user and multiplies them together. Print out the answer. Try to make the input as safe as possible. 3. Write a program which just echoes all the input to the output. 4. Write a program which strips spaces out of the input and replaces them with a single newline character. 5. scanf always takes pointer arguments. True or false? Node:Low Level Input/Output, Next:Questions 15c, Previous:Questions 15b, Up:Standard Output and Standard Input

Low Level Input/Output

getchar and putchar: gets and puts:

Node:getchar and putchar, Next:gets and puts, Previous:Low Level Input/Output, Up:Low Level Input/Output
getchar scanf()

and putchar

and printf() are relatively high level functions: this means that they are versatile and do a lot of hidden work for the user. C also provides some functions for dealing with input and output at a lower level: character by character. These functions are called getchar() and putchar() but, in fact, they might not be functions: they could be macros instead, See Preprocessor.
high level: printf() | | | scanf()

/ low level: putchar()

| | | |

\ getchar()

gets a single character from the input file stdin; putchar writes a single character to the output file stdout. getchar returns a character type: the next character on the input file. For example:
getchar char ch; ch = getchar();

This places the next character, what ever it might be, into the variable ch. Notice that no conversion to different data types can be performed by getchar() because it deals with single characters only. It is a low level function and does not `know' anything about data types other than characters. was used in the function skipgarb() to tame the scanf() function. This function was written in a very compact way. Another way of writing it would be as below:
getchar skipgarb () { char ch; ch = getchar(); while (ch != '\n') { ch = getchar(); } } /* skip garbage corrupting scanf */

The != symbol means "is not equal to" and the while statement is a loop. This function keeps on getchar-ing until it finds the newline character and then it quits. This function has many uses. One of these is to copy immediate keypress statements of languages like BASIC, where a program responds to keys as they are pressed without having to wait for return to be pressed. Without special library functions to give this kind of input (which are not universal) it is only possible to do this with the return key itself. For example:
printf("Press RETURN to continue\n"); skipgarb();

does not receive any input until the user presses RETURN, and then it simply skips over it in one go! The effect is that it waits for RETURN to be pressed.
skipgarb() putchar()

writes a character type and also returns a character type. For example:

char ch = '*';

putchar (ch); ch = putchar (ch);

These two alternatives have the same effect. The value returned by putchar() is the character which was written to the output. In other words it just hands the same value back again. This can simply be discarded, as in the first line. putchar() is not much use without loops to repeat it over and over again. An important point to remember is that putchar() and getchar() could well be implemented as macros, rather than functions. This means that it might not be possible to use functions as parameters inside them:
putchar( function() );

This depends entirely upon the compiler, but it is something to watch out for. Node:gets and puts, Previous:getchar and putchar, Up:Low Level Input/Output
gets

and puts

Two functions which are similar to putchar() and getchar() are puts() and gets() which mean putstring and getstring respectively. Their purpose is either to read a whole string from the input file stdin or write a whole string to the output stdout. Strings are groups or arrays of characters. For instance:
char *string[length]; string = gets(string); puts(string);

More information about these is given later, See Strings. Node:Questions 15c, Previous:Low Level Input/Output, Up:Standard Output and Standard Input

Questions
1. Is the following statement possible? (It could depend upon your compiler: try it!)
2. 3. putchar(getchar());

What might this do? (Hint: re-read the chapter about the pre-processor.) 4. Re write the statement in question 1, assuming that putchar() and getchar() are macros. Node:Assignments Expressions and Operators, Next:Decisions, Previous:Standard Output and Standard Input, Up:Top

Assignments, Expressions and Operators

Thinking in C. Working things out. An operator is something which takes one or more values and does something useful with those values to produce a result. It operates on them. The terminology of operators is the following: operator Something which operates on someting. operand Each thing which is operated upon by an operator is called an operand. operation The action which was carried out upon the operands by the operator! There are lots of operators in C. Some of them may already be familiar:
+ * / = & ==

Most operators can be thought of as belonging to one of three groups, divided up arbitrarily according to what they do with their operands. These rough groupings are thought of as follows:

Operators which produce new values from old ones. They make a result from their operands. e.g. +, the addition operator takes two numbers or two variables or a number and a variable and adds them together to give a new number. Operators which make comparisons. e.g. less than, equal to, greater than... Operators which produce new variable types: like the cast operator.

The majority of operators fall into the first group. In fact the second group is a subset of the first, in which the result of the operation is a boolean value of either true of false. C has no less than thirty nine different operators. This is more than, say, Pascal and BASIC put together! The operators serve a variety of purposes and they can be used very freely. The object of this chapter is to explain the basics of operators in C. The more abstruse operators are looked at in another chapter.

Expressions and values: Example 12: Output 12: Parentheses and Priority: Unary Operator Precedence: Special Assignment Operators ++ --: More Special Assignments: Example 13: Output 13: The Cast Operator:

Expressions and Types: Summary of Operators and Precedence: Questions 16:

Node:Expressions and values, Next:Example 12, Previous:Assignments Expressions and Operators, Up:Assignments Expressions and Operators

Expressions and values

The most common operators in any language are basic arithmetic operators. In C these are the following:
+ + * / / %

plus (unary) minus (force value to be negative) addition subtraction multiplication floating point division integer division "div" integer remainder "mod"

These operators would not be useful without a partner operator which could attach the values which they produce to variables. Perhaps the most important operator then is the assignment operator:
= assignment operator

This has been used extensively up to now. For example:

double x,y; x = 2.356; y = x; x = x + 2 + 3/5;

The assignment operator takes the value of whatever is on the right hand side of the = symbol and puts it into the variable on the left hand side. As usual there is some standard jargon for this, which is useful to know because compilers tend to use this when handing out error messages. The assignment operator can be summarized in the following way:
lvalue = expression;

This statement says no more than what has been said about assignments already: namely that it takes something on the right hand side and attaches it to whatever is on

the left hand side of the = symbol. An expression is simply the name for any string of operators, variables and numbers. All of the following could be called expressions:
1 + 2 + 3 a + somefunction() 32 * x/3 i % 4 x 1 (22 + 4*(function() + 2)) function () /* provided it returns a sensible value */

Lvalues on the other hand are simply names for memory locations: in other words variable names, or identifiers. The name comes from `left values' meaning anything which can legally be written on the left hand side of an assignment. Node:Example 12, Next:Output 12, Previous:Expressions and values, Up:Assignments Expressions and Operators

Example
/**************************************/ /* */ /* Operators Demo # 1 */ /* */ /**************************************/ #include <stdio.h> /**************************************/ main () { int i; printf ("Arithmetic Operators\n\n"); i = 6; printf ("i = 6, -i is : %d\n", -i); printf ("int 1 + 2 = %d\n", 1 + 2); printf ("int 5 - 1 = %d\n", 5 - 1); printf ("int 5 * 2 = %d\n", 5 * 2); printf ("\n9 div 4 = 2 remainder 1:\n"); printf ("int 9 / 4 = %d\n", 9 / 4); printf ("int 9 % 4 = %d\n", 9 % 4); printf ("double 9 / 4 = %f\n", 9.0 / 4.0); }

Node:Output 12, Next:Parentheses and Priority, Previous:Example 12, Up:Assignments Expressions and Operators

Output
Arithmetic Operators i = int int int 6, -i 1 + 2 5 - 1 5 * 2 is : -6 = 3 = 4 = 10

9 div 4 = 2 remainder 1: int 9 / 4 = 2 int 9 4 = 1 double 9 / 4 = 2.250000

Node:Parentheses and Priority, Next:Unary Operator Precedence, Previous:Output 12, Up:Assignments Expressions and Operators

Parentheses and Priority

Parentheses are classed as operators by the compiler, although their position is a bit unclear. They have a value in the sense that they assume the value of whatever expression is inside them. Parentheses are used for forcing a priority over operators. If an expression is written out in an ambiguous way, such as:
a + b / 4 * 2

it is not clear what is meant by this. It could be interpreted in several ways:

((a + b) / 4) * 2

or
(a + b)/ (4 * 2)

or
a + (b/4) * 2

and so on. By using parentheses, any doubt about what the expression means is removed. Parentheses are said to have a higher priority than + * or / because they are evaluated as "sealed capsules" before other operators can act on them. Putting parentheses in may remove the ambiguity of expressions, but it does not alter than fact that
a + b / 4 * 2

is ambiguous. What will happen in this case? The answer is that the C compiler has a convention about the way in which expressions are evaluated: it is called operator precedence. The convention is that some operators are stronger than others and that the stronger ones will always be evaluated first. Otherwise, expressions like the one above are evaluated from left to right: so an expression will be dealt with from left to right unless a strong operator overrides this rule. Use parentheses to be sure. A table of all operators and their priorities is given in the reference section. Node:Unary Operator Precedence, Next:Special Assignment Operators ++ --, Previous:Parentheses and Priority, Up:Assignments Expressions and Operators

Unary Operator Precedence

Unary operators are operators which have only a single operand: that is, they operate on only one object. For instance:
++ -+ &

The precedence of unary operators is from right to left so an expression like:

*ptr++;

would do ++ before *. Node:Special Assignment Operators ++ --, Next:More Special Assignments, Previous:Unary Operator Precedence, Up:Assignments Expressions and Operators

Special Assignment Operators ++ and -C has some special operators which cut down on the amount of typing involved in a program. This is a subject in which it becomes important to think in C and not in other languages. The simplest of these perhaps are the increment and decrement operators:
++

increment: add one to

--

decrement: subtract one from These attach to any variable of integer or floating point type. (character types too, with care.) They are used to simply add or subtract 1 from a variable. Normally, in other languages, this is accomplished by writing:
variable = variable + 1;

In C this would also be quite valid, but there is a much better way of doing this:
variable++; or ++variable;

would do the same thing more neatly. Similarly:

variable = variable - 1;

is equivalent to:
variable--;

or
--variable;

Notice particularly that these two operators can be placed in front or after the name of the variable. In some cases the two are identical, but in the more advanced uses of C operators, which appear later in this book, there is a subtle difference between the two. Node:More Special Assignments, Next:Example 13, Previous:Special Assignment Operators ++ --, Up:Assignments Expressions and Operators

More Special Assignments

Here are some of the nicest operators in C. Like ++ and -- these are short ways of writing longer expressions. Consider the statement:
variable = variable + 23;

In C this would be a long winded way of adding 23 to variable. It could be done more simply using the general increment operator: +=
variable += 23;

This performs exactly the same operation. Similarly one could write:
variable1 = variable1 + variable2;

as

variable1 += variable2;

and so on. There is a handful of these

<operation>=

operators: one for each of the major operations which can be performed. There is, naturally, one for subtraction too:

variable = variable - 42;

can be written:
variable -= 42;

More surprisingly, perhaps, the multiplicative assignment:

variable = variable * 2;

may be written:
variable *= 2;

and so on. The main arithmetic operators all follow this pattern:
+=

add assign
-= *= /= %=

subtract assign multiply assign divide (double) and (int) types

remainder (int) type only. and there are more exotic kinds, used for bit operations or machine level operations, which will be ignored at this stage:
>>= <<= ^= |= &=

Node:Example 13, Next:Output 13, Previous:More Special Assignments, Up:Assignments Expressions and Operators

Example Listing
/**************************************/ /* */ /* Operators Demo # 2 */ /* */ /**************************************/ #include <stdio.h> /**************************************/ main () { int i;

printf ("Assignment Operators\n\n"); i = 10; printf("i = 10 : %d\n",i); i++; printf ("i++ : %d\n",i); i += 5; printf ("i += 5 : %d\n",i); i--; printf ("i-- : %d\n",i); i -= 2; printf ("i -= 2 : %d\n",i); i *= 5; printf ("i *= 5 :%d\n",i); i /= 2; printf ("i /= 2 : %d\n",i); i %= 3; printf ("i %%= 3 : %d\n",i); } /* Assignment */ /* i = i + 1 */ /* i = i + 5 */ /* i = i = 1 */ /* i = i - 2 */ /* i = i * 5 */ /* i = i / 2 */ /* i = i % 3 */

Node:Output 13, Next:The Cast Operator, Previous:Example 13, Up:Assignments Expressions and Operators

Output
Assignment Operators i = 10 : 10 i++ : 11 i += 5 : 16 i-- : 15 i -= 2 : 13 i *= 5 :65 i /= 2 : 32 i %= 3 : 2

Node:The Cast Operator, Next:Expressions and Types, Previous:Output 13, Up:Assignments Expressions and Operators

The Cast Operator

The cast operator is an operator which forces a particular type mould or type cast onto a value, hence the name. For instance a character type variable could be forced to fit into an integer type box by the statement:
char ch; int i; i = (int) ch;

This operator was introduced earlier, See Variables. It will always produce some value, whatever the conversion: however remotely improbable it might seem. For instance it is quite possible to convert a character into a floating point number: the result will be a floating point representation of its ASCII code! Node:Expressions and Types, Next:Summary of Operators and Precedence, Previous:The Cast Operator, Up:Assignments Expressions and Operators

Expressions and Types

There is a rule in C that all arithmetic and mathematical operations must be carried out with long variables: that is, the types
double long float int long int

If the programmer tries to use other types like short or float in a mathematical expression they will be cast into long types automatically by the compiler. This can cause confusion because the compiler will spot an error in the following statement:
short i, j = 2; i = j * 2 + 1;

A compiler will claim that there is a type mismatch between i and the expression on the right hand side of the assignment. The compiler is perfectly correct of course, even though it appears to be wrong. The subtlety is that arithmetic cannot be done in short type variables, so that the expression is automatically converted into long type or int type. So the right hand side is int type and the left hand side is short type: hence there is indeed a type mismatch. The programmer can get around this by using the cast operator to write:
short i, j = 2; i = (short) j * 2 + 1;

A similar thing would happen with float:

float x, y = 2.3; x = y * 2.5;

would also be incorrect for the same reasons as above. Comparisons and Logic

Comparisons and Logic

Six operators in C are for making logical comparisons. The relevance of these operators will quickly become clear in the next chapter, which is about decisions and comparisons. The six operators which compare values are:
==

is equal to
!= > < >= <=

is not equal to is greater than is less than is greater than or equal to is less than or equal to

These operators belong to the second group according to the scheme above but they do actually result in values so that they could be thought of as being a part of the first group of operators too. The values which they produce are called true and false. As words, "true" and "false" are not defined normally in C, but it is easy to define them as macros and they may well be defined in a library file:
#define TRUE 1 #define FALSE 0

Falsity is assumed to have the value zero in C and truth is represented by any non-zero value. These comparison operators are used for making decisions, but they are themselves operators and expressions can be built up with them.
1 == 1

has the value "true" (which could be anything except zero). The statement:
int i; i = (1 == 2);

would be false, so i would be false. In other words, i would be zero. Comparisons are often made in pairs or even in groups and linked together with words like OR and AND. For instance, some test might want to find out whether:
(A is greater than B) AND (A is greater than C)

C does not have words for these operations but gives symbols instead. The logical operators, as they are called, are as follows:
&& ||

logical AND

logical OR inclusive logical NOT

The statement which was written in words above could be translated as:
(A > B) && (A > C)

The statement:
(A is greater than B) AND (A is not greater than C)

translates to:
(A > B) && !(A > C)

Shakespeare might have been disappointed to learn that, whatever the value of a variable tobe the result of
thequestion = tobe || !tobe

must always be true. The NOT operator always creates the logical opposite: !true is false and !false is true. On or the other of these must be true. thequestion is therefore always true. Fortunately this is not a matter of life or death! Node:Summary of Operators and Precedence, Next:Questions 16, Previous:Expressions and Types, Up:Assignments Expressions and Operators

Summary of Operators and Precedence

The highest priority operators are listed first.
Operator () [] ++ -(type) * & ~ ! * / % + Operation parentheses square brackets increment decrement cast operator the contents of the address of unary minus one's complement logical NOT multiply divide remainder (MOD) add subtract Evaluated. left to right left to right right right right right right right right right to to to to to to to to left left left left left left left left

left to right left to right left to right left to right left to right

>> << > >= <= < == != & ^ | && || = += -= *= /= %= >>= <<= &= ^= |=

shift right shift left is greater than greater than or equal to less than or equal to less than is equal to is not equal to bitwise bitwise bitwise logical logical AND exclusive OR inclusive OR AND OR

left to right left to right left left left left to to to to right right right right

left to right left to right left left left left left right right right right right right right right right right right to to to to to to to to to to to to to to to to right right right right right left left left left left left left left left left left

assign add assign subtract assign multiply assign divide assign remainder assign right shift assign left shift assign AND assign exclusive OR assign inclusive OR assign

Node:Questions 16, Previous:Summary of Operators and Precedence, Up:Assignments Expressions and Operators

Questions
1. What is an operand? 2. Write a statement which prints out the remainder of 5 divided by 2. 3. Write a short statement which assigns the remainder of 5 divided by 2 to a variable called "rem". 4. Write a statement which subtracts -5 from 10. 5. Write in C: if 1 is not equal to 23, print out "Thank goodness for mathematics!" Node:Decisions, Next:Loops, Previous:Assignments Expressions and Operators, Up:Top

Decisions
Testing and Branching. Making conditions. Suppose that a fictional traveller, some character in a book like this one, came to the end of a straight, unfinished road and waited there for the author to decide where the road would lead. The author might decide a number of things about this road and its traveller:

The road will carry on in a straight line. If the traveller is thirsty he will stop for a drink before continuing. The road will fork and the traveller will have to decide whether to take the left branch or the right branch. The road might have a crossroads or a meeting point where many roads come together. Again the traveller has to decide which way to go.

We are often faced with this dilemma: a situation in which a decision has to be made. Up to now the simple example programs in this book have not had any choice about the way in which they progressed. They have all followed narrow paths without any choice about which way they were going. This is a very limited way of expressing ideas though: the ability to make decisions and to choose different options is very useful in programming. For instance, one might want to implement the following ideas in different programs:

If the user hits the jackpot, write some message to say so. "You've won the game!" If a bank balance is positive then print C for credit otherwise print D for debit. If the user has typed in one of five things then do something special for each special case, otherwise do something else.

These choices are actually just the same choices that the traveller had to make on his undecided path, thinly disguised. In the first case there is a simple choice: a do of don't choice. The second case gives two choices: do thing 1 or thing 2. The final choice has several possibilities. C offers four ways of making decisions like the ones above. They are listed here below. The method which is numbered 2b was encountered in connection with the C preprocessor; its purpose is very similar to 2a.
1: if (something_is_true) { /* do something */ } if (something_is_true) { /* do one thing */ } else { /* do something else */ } ? (something_is_true) : /* do one thing */ : /* do something else */ 3: switch (choice) { case first_possibility : /* do something */ case second_possibility : /* do something */

2a:

2b:

.... }

if: example f1: if else: Nested ifs and logic: Example 14: Stringing together if..else: switch: Example 15: To try:

Node:if, Next:example f1, Previous:Decisions, Up:Decisions

if

The first form of the if statement is an all or nothing choice. if some condition is satisfied, do what is in the braces, otherwise just skip what is in the braces. Formally, this is written:
if (condition) statement;

or
if (condition) { compound statement }

Notice that, as well as a single statement, a whole block of statements can be written under the if statement. In fact, there is an unwritten rule of thumb in C that wherever a single statement will do, a compound statement will do instead. A compound statement is a block of single statements enclosed by curly braces. A condition is usually some kind of comparison, like the ones discussed in the previous chapter. It must have a value which is either true or false (1 or 0) and it must be enclosed by the parentheses ( and ). If the condition has the value `true' then the statement or compound statement following the condition will be carried out, otherwise it will be ignored. Some of the following examples help to show this:
int i; printf ("Type in an integer"); scanf ("%ld",&i); if (i == 0) { printf ("The number was zero"); } if (i > 0) { printf ("The number was positive");

} if (i < 0) { printf ("The number was negative"); }

The same code could be written more briefly, but perhaps less consistently in the following way:
int i; printf ("Type in an integer"); scanf ("%ld",&i); if (i == 0) printf ("The number was zero"); if (i > 0) printf ("The number was positive"); if (i < 0) printf ("The number was negative");

The preference in this book is to include the block braces, even when they are not strictly required. This does no harm. It is no more or less efficient, but very often you will find that some extra statements have to go into those braces, so it is as well to include them from the start. It also has the appeal that it makes if statements look the same as all other block statements and it makes them stand out clearly in the program text. This rule of thumb is only dropped in very simple examples like:
if (i == 0) i++;

The if statement alone allows only a very limited kind of decision: it makes do or don't decisions; it could not decide for the traveller whether to take the left fork or the right fork of his road, for instance, it could only tell him whether to get up and go at all. To do much more for programs it needs to be extended. This is the purpose of the else statement, described after some example listings.. Node:example f1, Next:if else, Previous:if, Up:Decisions

Example Listings
/*****************************************/ /* */ /* If... #1 */ /* */ /*****************************************/ #include <stdio.h> #define TRUE #define FALSE 1 0

/******************************************/ main ()

{ int i; if (TRUE) { printf ("This is always printed"); } if (FALSE) { printf ("This is never printed"); } } /*******************************************/ /* */ /* If demo #2 */ /* */ /*******************************************/ /* On board car computer. Works out the */ /* number of kilometers to the litre */ /* that the car is doing at present */ #include <stdio.h> /*******************************************/ /* Level 0 */ /*******************************************/ main () { double fuel,distance; FindValues (&fuel,&distance); Report (fuel,distance); } /********************************************/ /* Level 1 */ /********************************************/ FindValues (fuel,distance) /* from car */

/* These values would be changing in */ /* a real car, independently of the */ /* program. */ double *fuel,*distance; { /* how much fuel used since last check on values */ printf ("Enter fuel used"); scanf ("%lf",fuel); /* distance travelled since last check on values */ printf ("Enter distance travelled"); scanf ("%lf",distance); } /**********************************************/

Report (fuel,distance) double fuel,distance; { double kpl; kpl = distance/fuel;

/* on dashboard */

printf ("fuel consumption: %2.1lf",kpl); printf (" kilometers per litre\n"); if (kpl <= 1) { printf ("Predict fuel leak or car"); printf (" needs a service\n"); } if (distance > 500) { printf ("Remember to check tyres\n"); } if (fuel > 30) /* Tank holds 40 l */ { printf ("Fuel getting low: %s left\n",40-fuel); } }

Node:if else, Next:Nested ifs and logic, Previous:example f1, Up:Decisions

if

... else

The if .. else statement has the form:

if (condition) statement1; else statement2;

This is most often written in the compound statement form:

if (condition) { statements } else { statements }

The if..else statement is a two way branch: it means do one thing or the other. When it is executed, the condition is evaluated and if it has the value `true' (i.e. not zero) then statement1 is executed. If the condition is `false' (or zero) then statement2 is executed. The if..else construction often saves an unnecessary test from having to be made. For instance:
int i; scanf ("%ld",i);

if (i > 0) { printf ("That number was positive!"); } else { printf ("That number was negative or zero!"); }

It is not necessary to test whether i was negative in the second block because it was implied by the if..else structure. That is, that block would not have been executed unless i were NOT greater than zero. The weary traveller above might make a decision such as:
if (rightleg > leftleg) { take_left_branch(); } else { take_right_branch(); }

Node:Nested ifs and logic, Next:Example 14, Previous:if else, Up:Decisions

Nested ifs and logic

Consider the following statements which decide upon the value of some variable i. Their purposes are exactly the same.
if ((i > 2) && (i < 4)) { printf ("i is three"); }

or:
if (i > 2) { if (i < 4) { printf ("i is three"); } }

Both of these test i for the same information, but they do it in different ways. The first method might been born out of the following sequence of thought:

If i is greater than 2 and i is less than four, both at the same time, then i has to be 3. The second method is more complicated. Think carefully. It says: If i is greater than 2, do what is in the curly braces. Inside these curly braces i is always greater than 2 because otherwise the program would never have arrived inside them. Now, if i is also less than 4, then do what is inside the new curly braces. Inside these curly braces i is always less than 4. But wait! The whole of the second test is held inside the "i is greater than 2" braces, which is a sealed capsule: nothing else can get in, so, if the program gets into the "i is less than 4" braces as well, then both facts must be true at the same time. There

is only one integer which is bigger than 2 and less than 4 at the same time: it is 3. So i is 3. The aim of this demonstration is to show that there are two ways of making multiple decisions in C. Using the logical comparison operators &&, || (AND,OR) and so on.. several multiple tests can be made. In many cases though it is too difficult to think in terms of these operators and the sealed capsule idea begins to look attractive. This is another advantage of using the curly braces: it helps the programmer to see that if statements and if..else statements are made up of sealed capsule parts. Once inside a sealed capsule
if (i > 2) { /* i is greater than 2 in here! */ } else { /* i is not greater than 2 here! */ }

the programmer can rest assured that nothing illegal can get in. The block braces are like regions of grace: they cannot be penetrated by anything which does not satisfy the right conditions. This is an enourmous weight off the mind! The programmer can sit back and think: I have accepted that i is greater than 2 inside these braces, so I can stop worrying about that now. This is how programmers learn to think in a structured way. They learn to be satisfied that certain things have already been proven and thus save themselves from the onset of madness as the ideas become too complex to think of all in one go. Node:Example 14, Next:Stringing together if..else, Previous:Nested ifs and logic, Up:Decisions

Example Listing
/***********************************************/ /* */ /* If demo #3 */ /* */ /***********************************************/ #include <stdio.h> /***********************************************/ main () { int persnum,usernum,balance; persnum = 7462; balance = -12; printf ("The Plastic Bank Corporation\n"); printf ("Please enter your personal number :"); usernum = getnumber();

if (usernum == 7462) { printf ("\nThe current state of your account\n"); printf ("is %d\n",balance); if (balance < 0) { printf ("The account is overdrawn!\n"); } } else { printf ("This is not your account\n"); } printf ("Have a splendid day! Thank you.\n"); } /**************************************************/ getnumber () { int num = 0; scanf ("%d",&num); if ((num > 9999) || (num <= 0)) { printf ("That is not a valid number\n"); } return (num); } /* get a number from the user */

Node:Stringing together if..else, Next:switch, Previous:Example 14, Up:Decisions

Stringing together if..else

What is the difference between the following programs? They both interpret some imaginary exam result in the same way. They both look identical when compiled and run. Why then are they different?
/**************************************************/ /* Program 1 */ /**************************************************/ #include <stdio.h> main () { int result; printf("Type in exam result"); scanf ("%d",&result); if (result < 10) { printf ("That is poor"); } if (result > 20)

{ printf ("You have passed."); } if (result > 70) { printf ("You got an A!"); } } /* end */ /**************************************************/ /* Program 2 */ /**************************************************/ #include <stdio.h> main () { int result; printf("Type in exam result"); scanf ("%d",&result); if (result < 10) { printf ("That is poor"); } else { if (result > 20) { printf ("You have passed."); } else { if (result > 70) { printf ("You got an A!"); } }

} }

The answer is that the second of these programs can be more efficient. This because it uses the else form of the if statement which in turn means that few things have to be calculated. Program one makes every single test, because the program meets every if statement, one after the other. The second program does not necessarily do this however. The nested if statements make sure that the second two tests are only made if the first one failed. Similarly the third test is only performed if the first two failed. So the second program could end up doing a third of the work of the first program, in the best possible case. Nesting decisions like this can be an efficient way of controlling long lists of decisions like those above. Nested loops make a program branch into lots of possible paths, but choosing one path would preclude others.

Node:switch, Next:Example 15, Previous:Stringing together if..else, Up:Decisions

switch

: integers and characters

The switch construction is another way of making a program path branch into lots of different limbs. It can be used as a different way of writing a string of if .. else statements, but it is more versatile than that and it only works for integers and character type values. It works like a kind of multi-way switch. (See the diagram.) The switch statement has the following form:
switch (int or char expression) { case constant : statement; break; ... }

/* optional */

It has an expression which is evaluated and a number of constant `cases' which are to be chosen from, each of which is followed by a statement or compound statement. An extra statement called break can also be incorporated into the block at any point.

break

is a reserved word.

The switch statement can be written more specifically for integers:

switch (integer value) { case 1: case 2: .... default: default statement break; } /* optional line */ statement1; break; statement2; break;

/* optional line */ /* optional line */

When a switch statement is encountered, the expression in the parentheses is evaluated and the program checks to see whether the result of that expression matches any of the constants labelled with case. If a match is made (for instance, if the expression is evaluated to 23 and there is a statement beginning "case 23 : ...") execution will start just after that case statement and will carry on until either the closing brace } is encountered or a break statement is found. break is a handy way of jumping straight out of the switch block. One of the cases is called default. Statements which follow the default case are executed for all cases which are not specifically listed. switch is a way of choosing some action from a number of known instances. Look at the following example. Node:Example 15, Next:To try, Previous:switch, Up:Decisions

Example Listing
/************************************************/ /* */ /* switch .. case */ /* */ /************************************************/ /* Morse code program. Enter a number and */ /* find out what it is in Morse code */ #include <stdio.h> #define CODE 0 /*************************************************/ main () { short digit; printf ("Enter any digit in the range 0..9"); scanf ("%h",&digit); if ((digit < 0) || (digit > 9)) { printf ("Number was not in range 0..9"); return (CODE); } printf ("The Morse code of that digit is "); Morse (digit); } /************************************************/ Morse (digit) short digit; { switch (digit) { /* print out Morse code */

case 0 : printf break; case 1 : printf break; case 2 : printf break; case 3 : printf break; case 4 : printf break; case 5 : printf break; case 6 : printf break; case 7 : printf break; case 8 : printf break; case 9 : printf }

("-----"); (".----"); ("..---"); ("...--"); ("....-"); ("....."); ("-...."); ("--..."); ("---.."); ("----.");

The program selects one of the printf statements using a switch construction. At every case in the switch, a break statement is used. This causes control to jump straight out of the switch statement to its closing brace }. If break were not included it would go right on executing the statements to the end, testing the cases in turn. break this gives a way of jumping out of a switch quickly. There might be cases where it is not necessary or not desirable to jump out of the switch immediately. Think of a function yes() which gets a character from the user and tests whether it was 'y' or 'Y'.
yes () /* A sloppy but simple function */

{ switch (getchar()) { case 'y' : case 'Y' : return TRUE default : return FALSE } }

If the character is either 'y' or 'Y' then the function meets the statement return TRUE. If there had been a break statement after case 'y' then control would not have been able to reach case 'Y' as well. The return statement does more than break out of switch, it breaks out of the whole function, so in this case break was not required. The default option ensures that whatever else the character is, the function returns false. Node:To try, Previous:Example 15, Up:Decisions

Things to try
1. Write a program to get a lot of numbers from the user and print out the maximum and minimum of those.

2. Try to make a counter which is reset to zero when it reaches 9999. 3. Try to write a program incorporating the statement if (yes()) {...}. Node:Loops, Next:Arrays, Previous:Decisions, Up:Top

Loops
Controlling repetitive processes. Nesting loops Decisions can also be used to make up loops. Loops free a program from the straitjacket of doing things only once. They allow the programmer to build a sequence of instructions which can be executed again and again, with some condition deciding when they will stop. There are three kinds of loop in C. They are called:
while do ... while for

These three loops offer a great amount of flexibility to programmers and can be used in some surprising ways!

while: Example 16: Example 17: do while: Example 18: for: The flexible for loop: Quitting Loops and Hurrying Them Up!: Nested Loops: Questions 18:

Node:while, Next:Example 16, Previous:Loops, Up:Loops

while

The simplest of the three loops is the while loop. In common language while has a fairly obvious meaning: the while-loop has a condition:
while (condition) { statements; }

and the statements in the curly braces are executed while the condition has the value "true" ( 1 ). There are dialects of English, however, in which "while" does not have its

commonplace meaning, so it is worthwhile explaining the steps which take place in a

while loop. The first important thing about this loop is that has a conditional expression (something like (a > b) etc...) which is evaluated every time the loop is executed by the computer. If the value of the expression is true, then it will carry on with the instructions in the curly braces. If the expression evaluates to false (or 0) then the instructions in the braces are ignored and the entire while loop ends. The computer then moves onto the next statement in the program. The second thing to notice about this loop is that the conditional expression comes at the start of the loop: this means that the condition is tested at the start of every `pass', not at the end. The reason that this is important is this: if the condition has the value false before the loop has been executed even once, the statements inside the braces will not get executed at all - not even once. The best way to illustrate a loop is to give an example of its use. One example was sneaked into an earlier chapter before its time, in order to write the skipgarb() function which complemented scanf(). That was:
skipgarb () /* skip garbage corrupting scanf */

{ while (getchar() != '\n') { } }

This is a slightly odd use of the while loop which is pure C, through and through. It is one instance in which the programmer has to start thinking C and not any other language. Something which is immediately obvious from listing is that the while loop in skipgarb() is empty: it contains no statements. This is quite valid: the loop will merely do nothing a certain number of times... at least it would do nothing if it were not for the assignment in the conditional expression! It could also be written:
skipgarb () /* skip garbage corrupting scanf */

{ while (getchar() != '\n') { } }

The assignment inside the conditional expression makes this loop special. What happens is the following. When the loop is encountered, the computer attempts to evaluate the expression inside the parentheses. There, inside the parentheses, it finds a function call to getchar(), so it calls getchar() which fetches the next character from the input. getchar() then takes on the value of the character which it fetched from the input file. Next the computer finds the != "is not equal to" symbol and the newline character \n. This means that there is a comparison to be made. The computer compares the character fetched by getchar() with the newline character and if they are `not equal' the expression is true. If they are equal the expression is false. Now, if the expression is true, the while statement will loop and start again - and it will evaluate the expression on every pass of the loop to check whether or not it is true. When the expression eventually becomes false the loop will quit. The net result of this subtlety is that skipgarb() skips all the input characters up to and including the next newline \n character and that usually means the rest of the input. Node:Example 16, Next:Example 17, Previous:while, Up:Loops

Example Listing
Another use of while is to write a better function called yes(). The idea of this function was introduced in the previous section. It uses a while loop which is always true to repeat the process of getting a response from the user. When the response is either yes or no it quits using the return function to jump right out of the loop.
/***********************************************/ /* */ /* Give me your answer! */ /* */ /***********************************************/ #include <stdio.h> #define TRUE #define FALSE 1 0

/*************************************************/ /* Level 0 */ /*************************************************/

main () { printf ("Yes or no? (Y/N)\n"); if (yes()) { printf ("YES!"); } else { printf ("NO!"); } } /*************************************************/ /* Level 1 */ /*************************************************/ yes () { char getkey(); while (true) { switch (getkey()) { case 'y' : case 'Y' : return (TRUE); case 'n' : case 'N' : return (FALSE); } } } /*************************************************/ /* Toolkit */ /*************************************************/ char getkey () { char ch; ch = getchar(); skipgarb(); } /**************************************************/ skipgarb () { while (getchar() != '\n') { } } /* end */ /* get a character + RETURN */ /* get response Y/N query */

Node:Example 17, Next:do while, Previous:Example 16, Up:Loops

Example Listing

This example listing prompts the user to type in a line of text and it counts all the spaces in that line. It quits when there is no more input left and printf out the number of spaces.
/***********************************************/ /* */ /* while loop */ /* */ /***********************************************/ /* count all the spaces in an line of input */ #include <stdio.h> main () { char ch; short count = 0; printf ("Type in a line of text\n"); while ((ch = getchar()) != '\n') { if (ch == ' ') { count++; } } printf ("Number of space = %d\n",count); }

Node:do while, Next:Example 18, Previous:Example 17, Up:Loops

do while

..

The do..while loop resembles most closely the repeat..until loops of Pascal and BASIC except that it is the `logical opposite'. The do loop has the form:
do { statements; } while (condition)

Notice that the condition is at the end of this loop. This means that a do..while loop will always be executed at least once, before the test is made to determine whether it should continue. This is the only difference between while and do..while. A do..while loop is like the "repeat .. until" of other languages in the following sense: if the condition is NOTed using the ! operator, then the two are identical.
repeat == do

until(condition)

while (!condition)

This fact might be useful for programmers who have not yet learned to think in C! Node:Example 18, Next:for, Previous:do while, Up:Loops

Example Listing
Here is an example of the use of a do..while loop. This program gets a line of input from the user and checks whether it contains a string marked out with "" quote marks. If a string is found, the program prints out the contents of the string only. A typical input line might be:
Onceupon a time "Here we go round the..."what a terrible..

The output would then be:

Here we go round the...

If the string has only one quote mark then the error message `string was not closed before end of line' will be printed.

/**********************************************/ /* */ /* do .. while demo */ /* */ /**********************************************/ /* print a string enclosed by quotes " " */ /* gets input from stdin i.e. keyboard */ /* skips anything outside the quotes */ #include <stdio.h> /*************************************************/ /* Level 0 */ /*************************************************/ main () { char ch,skipstring(); do { if ((ch = getchar()) == '"') { printf ("The string was:\n"); ch = skipstring(); } }

while (ch != '\n') { } } /*************************************************/ /* Level 1 */ /*************************************************/ char skipstring () /* skip a string "..." */ { char ch; do { ch = getchar(); putchar(ch); if (ch == '\n') { printf ("\nString was not closed "); printf ("before end of line\n"); break; } } while (ch != '"') { } return (ch); }

Node:for, Next:The flexible for loop, Previous:Example 18, Up:Loops

for

The most interesting and also the most difficult of all the loops is the for loop. The name for is a hangover from earlier days and other languages. It is not altogether appropriate for C's version of for. The name comes from the typical description of a classic for loop: For all values of variable from value1 to value2 in steps of value3, repeat the following sequence of commands.... In BASIC this looks like:
FOR variable = value1 TO value2 STEP value3 NEXT variable

The C for loop is much more versatile than its BASIC counterpart; it is actually based upon the while construction. A for loop normally has the characteristic feature of controlling one particular variable, called the control variable. That variable is somehow associated with the loop. For example it might be a variable which is used to count "for values from 0 to 10" or whatever. The form of the for loop is:
for (statement1; condition; statement2) { }

For normal usage, these expressions have the following significance. statement1 This is some kind of expression which initializes the control variable. This statement is only carried out once before the start of the loop. e.g. i = 0; condition This is a condition which behaves like the while loop. The condition is evaluated at the beginning of every loop and the loop is only carried out while this expression is true. e.g. i < 20; statement2 This is some kind of expression for altering the value of the control variable. In languages such as Pascal this always means adding or subtracting 1 from the variable. In C it can be absolutely anything. e.g. i++ or i *= 20 or i /= 2.3 ... Compare a C for loop to the BASIC for loop. Here is an example in which the loop counts from 0 to 10 in steps of 0.5:
FOR X = 0 TO 10 STEP 0.5 NEXT X for (x = 0; x <= 10; x += 0.5)

{ }

The C translation looks peculiar in comparison because it works on a subtly different principle. It does not contain information about when it will stop, as the BASIC one does, instead it contains information about when it should be looping. The result is that a C for loop often has the <= symbol in it. The for loop has plenty of uses. It could be used to find the sum of the first n natural numbers very simply:
sum = 0; for (i = 0; i <= n; i++) { sum += i; }

It generally finds itself useful in applications where a single variable has to be controlled in a well determined way. g4

Example Listing
This example program prints out all the primes numbers between 1 and the macro value maxint. Prime numbers are numbers which cannot be divided by any number except 1 without leaving a remainder.
/************************************************/ /* */ /* Prime Number Generator #1 */ /* */ /************************************************/ /* /* /* /* Check for prime number by raw number crunching. Try dividing all numbers up to half the size of a given i, if remainder == 0 then not prime! */ */ */ */

#include <stdio.h> #define MAXINT #define TRUE #define FALSE 500 1 0

/*************************************************/ /* Level 0 */ /*************************************************/ main () { int i; for (i = 2; i <= MAXINT; i++) { if (prime(i)) { printf ("%5d",i); }

} } /*************************************************/ /* Level 1 */ /*************************************************/ prime (i) int i; { int j; for (j = 2; j <= i/2; j++) { if (i % j == 0) { return FALSE; } } return TRUE; } /* check for a prime number */

Node:The flexible for loop, Next:Quitting Loops and Hurrying Them Up!, Previous:for, Up:Loops

The flexible for loop

The word `statement' was chosen carefully, above, to describe what goes into a for loop. Look at the loop again:
for (statement1; condition; statement2) { }

Statement really means what it says. C will accept any statement in the place of those above, including the empty statement. The while loop could be written as a for loop!
for (; condition; ) { } /* while ?? */

Here there are two empty statements, which are just wasted. This flexibility can be put to better uses though. Consider the following loop:
for (x = 2; x <= 1000; x = x * x) { .... }

This loop begins from 2 and each time the statements in the braces are executed x squares itself! Another odd looking loop is the following one:
for (ch = '*'; ch != '\n'; ch = getchar()) { }

This could be used to make yet another different kind of skipgarb() function. The loop starts off by initializing ch with a star character. It checks that ch != '\n' (which it isn't, first time around) and proceeds with the loop. On each new pass, ch is reassigned by calling the function getchar(). It is also possible to combine several incremental commands in a loop:
for (i = 0, j=10; i < j; i++, j--) { printf("i = %d, j= %d\n",i,j); }

Statement2 can be any statement at all which the programmer would like to be executed on every pass of the loop. Why not put that statement in the curly braces? In most cases that would be the best thing to do, but in special instances it might keep a program tidier or more readable to put it in a for loop instead. There is no good rule for when to do this, except to say: make you code as clear as possible. It is not only the statements which are flexible. An unnerving feature of the for construction (according to some programmers) is that even the conditional expression in the for loop can be altered by the program from within the loop itself if is written as a variable.
int i, number = 20; for (i = 0; i <= number; i++) { if (i == 9) { number = 30; } }

This is so nerve shattering that many languages forbid it outright. To be sure, is not often a very good idea to use this facility, but in the right hands, it is a powerful one to have around. Node:Quitting Loops and Hurrying Them Up!, Next:Nested Loops, Previous:The flexible for loop, Up:Loops

Quitting Loops and Hurrying Them Up!

C provides a simple way of jumping out of any of the three loops above at any stage, whether it has finished or not. The statement which performs this action is the same statement which was used to jump out of switch statements in last section.
break;

If this statement is encountered a loop will quit where it stands. For instance, an expensive way of assigning i to be 12 would be:

for (i = 1; i <= 20; i++) { if (i == 12) { break; } }

Still another way of making skipgarb() would be to perform the following loop:
while (TRUE) { ch = getchar(); if (ch == '\n') { break; } }

Of course, another way to do this would be to use the return() statement, which jumps right out of a whole function. break only jumps out of the loop, so it is less drastic. As well as wanting to quit a loop, a programmer might want to hurry a loop on to the next pass: perhaps to avoid executing a lot of irrelevant statements, for instance. C gives a statement for this too, called:
continue;

When a continue statement is encountered, a loop will stop whatever it is doing and will go straight to the start of the next loop pass. This might be useful to avoid dividing by zero in a program:
for (i = -10; i <= 10; i++) { if (i == 0) { continue; } printf ("%d", 20/i); }

Node:Nested Loops, Next:Questions 18, Previous:Quitting Loops and Hurrying Them Up!, Up:Loops

Nested Loops
Like decisions, loops will also nest: that is, loops can be placed inside other loops. Although this feature will work with any loop at all, it is most commonly used with the for loop, because this is easiest to control. The idea of nested loops is important for multi-dimensional arrays which are examined in the next section. A for loop controls the number of times that a particular set of statements will be carried out. Another outer loop could be used to control the number of times that a whole loop is carried out. To see the benefit of nesting loops, the example below shows how a square could be printed out using two printf statements and two loops.

/*****************************************/ /* */ /* A "Square" */ /* */ /*****************************************/ #include <stdio.h> #define SIZE 10

/*****************************************/ main () { int i,j; for (i = 1; i <= SIZE; i++) { for (j = 1; j <= SIZE; j++) { printf("*"); } printf ("\n"); } }

The output of this program is a "kind of" square:

********** ********** ********** ********** ********** ********** ********** ********** ********** **********

Node:Questions 18, Previous:Nested Loops, Up:Loops

Questions
1. 2. 3. 4. 5. How many kinds of loop does C offer, and what are they? When is the condition tested in each of the loops? Which of the loops is always executed once? Write a program which copies all input to output line by line. Write a program to get 10 numbers from the user and add them together.

Node:Arrays, Next:Strings, Previous:Loops, Up:Top

Arrays
Rows and tables of storage.

Arrays are a convenient way of grouping a lot of variables under a single variable name. Arrays are like pigeon holes or chessboards, with each compartment or square acting as a storage place; they can be one dimensional, two dimensional or more dimensional! An array is defined using square brackets []. For example: an array of three integers called "triplet" would be declared like this:
int triplet[3];

Notice that there is no space between the square bracket [ and the name of the array. This statement would cause space for three integers type variables to be created in memory next to each other as in the diagram below.
int triplet: | -----------------------------------| | | ------------------------------------

The number in the square brackets of the declaration is referred to as the `index' (plural: indicies) or `subscript' of the array and it must be an integer number between 0 and (in this case) 2. The three integers are called elements of the array and they are referred to in a program by writing:
triplet[0] triplet[1] triplet[2]

Note that the indicies start at zero and run up to one less than the number which is placed in the declaration (which is called the dimension of the array.) The reason for this will become clear later. Also notice that every element in an array is of the same type as every other. It is not (at this stage) possible to have arrays which contain many different data types. When arrays are declared inside a function, storage is allocated for them, but that storage space is not initialized: that is, the memory space contains garbage (random values). It is usually necessary, therefore, to initialize the array before the program truly begins, to prepare it for use. This usually means that all the elements in the array will be set to zero.

Why use arrays?: Limits and The Dimension of an array: Arrays and for loops: Example 19: Arrays Of More Than One Dimension: Arrays and Nested Loops: Example 20: Output of Game of Life: Initializing Arrays: Arrays and Pointers: Arrays as Parameters: Questions 19:

Node:Why use arrays?, Next:Limits and The Dimension of an array, Previous:Arrays, Up:Arrays

Why use arrays?

Arrays are most useful when they have a large number of elements: that is, in cases where it would be completely impractical to have a different name for every storage space in the memory. It is then highly beneficial to move over to arrays for storing information for two reasons:

The storage spaces in arrays have indicies. These numbers can often be related to variables in a problem and so there is a logical connection to be made between an array an a program. In C, arrays can be initialized very easily indeed. It is far easier to initialize an array than it is to initialize twenty or so variables.

The first of these reasons is probably the most important one, as far as C is concerned, since information can be stored in other ways with equally simple initialization facilities in C. One example of the use of an array might be in taking a census of the types of car passing on a road. By defining macros for the names of the different cars, they could easily be linked to the elements in an array.
Type car auto bil Array Element 0 1 2

The array could then be used to store the number of cars of a given type which had driven past. e.g.
/***********************************************/ /* */ /* Census */ /* */ /***********************************************/ #include <stdio.h> #define #define #define #define NOTFINISHED CAR AUTO BIL 1 0 1 2

/************************************************/ main () { int type[3]; int index; for (index = 0; index < 3; index++) { type[index] = 0; } while (NOTFINISHED) { printf ("Enter type number 0,1, or 2"); scanf ("%d", &index);

skipgarb(); type[index] += 1; } /* See text below */

This program, first of all, initializes the elements of the array to be zero. It then enters a loop which repeatedly fetches a number from the user and increases the value stored in the array element, labelled by that number, by 1. The effect is to count the cars as they go past. This program is actually not a very good program for two reasons in particular:

Firstly, it does not check that the number which the user typed is actually one of the elements of the array. (See the section below about this.) The loop goes on for ever and the program never gives up the information which is stores. In short: it is not very useful.

Another example, which comes readily to mind, would be the use of a two dimensional array for storing the positions of chess pieces in a chess game. Two dimensional arrays have a chessboard-like structure already and they require two numbers (two indicies) to pinpoint a particular storage cell. This is just like the numbers on chess board, so there is an immediate and logical connection between an array and the problem of keeping track of the pieces on a chess board. Arrays play an important role in the handling of string variables. Strings are important enough to have a section of their own, See Strings. Node:Limits and The Dimension of an array, Next:Arrays and for loops, Previous:Why use arrays?, Up:Arrays

Limits and The Dimension of an array

C does not do much hand holding. It is invariably up to the programmer to make sure that programs are free from errors. This is especially true with arrays. C does not complain if you try to write to elements of an array which do not exist! For example:
char array[5];

is an array with 5 elements. If you wrote:

array[7] = '*';

C would happily try to write the character * at the location which would have corresponded to the seventh element, had it been declared that way. Unfortunately this would probably be memory taken up by some other variable or perhaps even by the operating system. The result would be either:

The value in the incorrect memory location would be corrupted with unpredictable consequences. The value would corrupt the memory and crash the program completely! On Unix systems this leads to a memory segmentation fault.

The second of these tends to be the result on operating systems with proper memory protection. Writing over the bounds of an array is a common source of error. Remember that the array limits run from zero to the size of the array minus one. Node:Arrays and for loops, Next:Example 19, Previous:Limits and The Dimension of an array, Up:Arrays

Arrays and for loops

Arrays have a natural partner in programs: the for loop. The for loop provides a simple way of counting through the numbers of an index in a controlled way. Consider a one dimensional array called array. A for loop can be used to initialize the array, so that all its elements contain zero:
#define SIZE main () { int i, array[SIZE]; for (i = 0; i < SIZE; i++) { array[i] = 0; } } 10;

It could equally well be used to fill the array with different values. Consider:
#define SIZE main () { int i, array[size]; for (i = 0; i < size; i++) { array[i] = i; } } 10;

This fills each successive space with the number of its index:
index element contents 0 1 2 3 4 5 6 7 8 9

--------------------------------------| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ---------------------------------------

The for loop can be used to work on an array sequentially at any time during a program, not only when it is being initialized. The example listing below shows an example of how this might work for a one dimensional array, called an Eratosthenes sieve. This sieve is an array which is used for weeding out prime numbers, that is: numbers which cannot be divided by any number except 1 without leaving a remainder

or a fraction. It works by filling an array with numbers from 0 to some maximum value in the same way that was shown above and then by going through the numbers in turn and deleting (setting equal to zero) every multiple of every number from the array. This eliminates all the numbers which could be divided by something exactly and leaves only the prime numbers at the end. Try to follow the listing below. Node:Example 19, Next:Arrays Of More Than One Dimension, Previous:Arrays and for loops, Up:Arrays

Example Listing
/******************************************************/ /* */ /* Prime Number Sieve */ /* */ /******************************************************/ #include <stdio.h> #define SIZE #define DELETED 5000 0

/*******************************************************/ /* Level 0 */ /*******************************************************/ main () { short sieve[SIZE]; printf ("Eratosthenes Sieve \n\n"); FillSeive(sieve); SortPrimes(sieve); PrintPrimes(sieve); } /*********************************************************/ /* Level 1 */ /*********************************************************/ FillSeive (sieve) short sieve[SIZE]; { short i; for (i = 2; i < SIZE; i++) { sieve[i] = i; } } /**********************************************************/ SortPrimes (sieve) short sieve[SIZE]; { short i; /* Delete non primes */ /* Fill with integers */

for (i = 2; i < SIZE; i++) { if (sieve[i] == DELETED) { continue; } DeleteMultiplesOf(i,sieve); } } /***********************************************************/ PrintPrimes (sieve) short sieve[SIZE]; { short i; for (i = 2; i < SIZE; i++) { if (sieve[i] == DELETED) { continue; } else { printf ("%5d",sieve[i]); } } } /***********************************************************/ /* Level 2 */ /***********************************************************/ DeleteMultiplesOf (i,sieve) short i,sieve[SIZE]; { short j, mult = 2; for (j = i*2; j < SIZE; j = i * (mult++)) { sieve[j] = DELETED; } } /* end */ /* Delete.. of an integer */ /* Print out array */

Node:Arrays Of More Than One Dimension, Next:Arrays and Nested Loops, Previous:Example 19, Up:Arrays

Arrays Of More Than One Dimension

There is no limit, in principle, to the number of indicies which an array can have. (Though there is a limit to the amount of memory available for their storage.) An array of two dimensions could be declared as follows:
float numbers[SIZE][SIZE];

is some constant. (The sizes of the two dimensions do not have to be the same.) This is called a two dimensional array because it has two indicies, or two labels in square brackets. It has (SIZE * SIZE) or size-squared elements in it, which form an imaginary grid, like a chess board, in which every square is a variable or storage area.
SIZE -----------------------------------| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | ... (up to SIZE) -----------------------------------| 1 | | | | | | | | | -----------------------------------| 2 | | | | | | | | | -----------------------------------| 3 | | | | | | | | | -----------------------------------| 4 | | | | | | | | | -----------------------------------| 5 | | | | | | | | | -----------------------------------| 6 | | | | | | | | | -----------------------------------| 7 | | | | | | | | | -----------------------------------. . (up to SIZE)

Every element in this grid needs two indicies to pin-point it. The elements are accessed by giving the coordinates of the element in the grid. For instance to set the element 2,3 to the value 12, one would write:
array[2][3] = 12;

The usual terminology for the two indicies is that the first gives the row number in the grid and that the second gives the column number in the grid. (Rows go along, columns hold up the ceiling.) An array cannot be stored in the memory as a grid: computer memory is a one dimensional thing. Arrays are therefore stored in rows. The following array:
-----------| 1 | 2 | 3 | -----------| 4 | 5 | 6 | -----------| 7 | 8 | 9 | ------------

would be stored:
-----------------------------------| 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | -----------------------------------* ROW # 1 * ROW # 2 * ROW #3 *

Another way of saying that arrays are stored row-wise is to say that the second index varies fastest, because a two-dimensional array is always thought of as...
array[row][column]

so for every row stored, there will be lots of columns inside that row. That means the column index goes from 0..SIZE inside every row, so it is changing faster as the line of storage is followed. A three dimensional array, like a cube or a cuboid, could also be defined in the same kind of way:
double cube[SIZE][SIZE][SIZE];

or with different limits on each dimension:

short notcubic[2][6][8];

Three dimensional arrays are stored according to the same pattern as two dimensional arrays. They are kept in computer memory as a linear sequence of variable stores and the last index is always the one which varies fastest. Node:Arrays and Nested Loops, Next:Example 20, Previous:Arrays Of More Than One Dimension, Up:Arrays

Arrays and Nested Loops

Arrays of more than one dimension are usually handled by nested for loops. A two dimensional array might be initialized in the following way:
main () { int i,j; float array[SIZE1][SIZE2]; for (i = 0; i < SIZE1; i++) { for (j = 0; j < SIZE2; j++) { array[i][j] = 0; } } }

In three dimensions, three nested loops would be needed:

main () { int i,j,k; float array[SIZE1][SIZE2][SIZE3]; for (i = 0; i < SIZE1; i++)

{ for (j = 0; j < SIZE2; j++) { for (k = 0; k < SIZE3; k++) { array[i][j][k] = 0; } } } }

An example program helps to show how this happens in practice. The example below demonstrates the so-called "Game of Life". The aim is to mimic something like cell reproduction by applying some rigid rules to a pattern of dots . and stars *. A dot is a place where there is no life (as we know it!) and a star is a place in which there is a living thing. The rules will be clear from the listing. Things to notice are the way the program traverses the arrays and the way in which it checks that it is not overstepping the boundaries of the arrays. Node:Example 20, Next:Output of Game of Life, Previous:Arrays and Nested Loops, Up:Arrays

Example Listing
/*********************************************************/ /* */ /* Game of Life */ /* */ /*********************************************************/ /* /* /* /* /* /* /* Based upon an article from Scientific American in 1970. Simulates the reproduction of cells which depend on one another. The rules are that cells will only survive if they have a certain number of neighbours to support them but not too many, or there won't be enough food! */ */ */ */ */ */ */

#include <stdio.h> #define #define #define #define SIZE MAXNUM INBOUNDS NORESPONSE 20 15 (a>=0)&&(a<SIZE)&&(b>=0)&&(b<SIZE) 1

/*********************************************************/ /* Level 0 */ /*********************************************************/ main () { int count[SIZE][SIZE]; char array[SIZE][SIZE]; int generation = 0; printf ("Game of Life\n\n\n"); InitializeArray(array);

while (NORESPONSE) { CountNeighbours(array,count); BuildNextGeneration(array,count); UpdateDisplay(array,++generation); printf ("\n\nQ for quit. RETURN to continue.\n"); if(quit()) break; } } /**********************************************************/ /* Level 1 */ /**********************************************************/ InitializeArray (array) char array[SIZE][SIZE]; { int i,j; char ch; printf printf printf printf ("\nEnter starting setup. Type '.' for empty"); ("\nand any other character for occupied.\n"); ("RETURN after each line.\n\n"); ("Array size guide:\n\n"); /* Get starting conditions */

for (i=0; i++ < SIZE; printf("%c",'^')); printf ("\n\n"); for (i = 0; i < SIZE; i++) { for (j = 0; j < SIZE; j++) { scanf ("%c",&ch); if (ch == '.') { array[i][j] = '.'; } else { array[i][j] = '*'; } } skipgarb(); } printf ("\n\nInput is complete. Press RETURN."); skipgarb(); } /********************************************************/ CountNeighbours (array,count) /* count all neighbours */ char array[SIZE][SIZE]; int count[SIZE][SIZE]; { int i,j; for (i = 0; i < SIZE; i++) { for (j = 0; j < SIZE; j++)

{ count[i][j] = numalive(array,i,j); } } }

/*******************************************************/ BuildNextGeneration (array,count) /* A cell will survive if it has two or three */ /* neighbours. New life will be born to a dead */ /* cell if there are exactly three neighbours */ char array[SIZE][SIZE]; int count[SIZE][SIZE]; { int i,j; for (i = 0; i < SIZE; i++) { for (j = 0; j < SIZE; j++) { if (array[i][j] == '*') { switch (count[i][j]) { case 2 : case 3 : continue; default: array[i][j] = '.'; break; } } else { switch (count[i][j]) { case 3 : array[i][j] = '*'; break; default: continue; } } }

} }

/*******************************************************/ UpdateDisplay (array,g) char array[SIZE][SIZE]; int g; { int i,j; printf ("\n\nGeneration %d\n\n",g); for (i = 0; i < SIZE; i++) { for (j = 0; j < SIZE; j++) { printf("%c",array[i][j]); /* print out life array */

} printf("\n"); } } /*******************************************************/ /* Level 2 */ /*******************************************************/ numalive (array,i,j) /* Don't count array[i,j] : only its neighbours */ /* Also check that haven't reached the boundary */ /* of the array */ char array[SIZE][SIZE]; int i,j; { int a,b,census; census = 0; for (a = (i-1); (a <= (i+1)); a++) { for (b = (j-1); (b <= (j+1)); b++) { if (INBOUNDS && (array[a][b] == '*')) { census++; } } } if (array[i][j] == '*') census--; return (census); } /********************************************************/ /* Toolkit input */ /********************************************************/ quit() { char ch; while (NORESPONSE) { scanf ("%c",&ch); if (ch != '\n') skipgarb(); switch (ch) { case 'q' : case 'Q' : return (1); default : return (0); } } } /********************************************************/ skipgarb () {

while (getchar() != '\n') { } }

Node:Output of Game of Life, Next:Initializing Arrays, Previous:Example 20, Up:Arrays

Output of Game of Life

Game of Life Enter starting setup. Type '.' for empty and any other character for occupied. RETURN after each line. Array SIZE guide: ^^^^^^^^^^^^^^^^^^^^ (user types in: .................... .................... ..................... ..................... ..................... ..........***........ ...........*......... ...................... ..................... ..................... ..................... ********************* ..................... ...................... .................... ..................... ...................... ...................... ...................... ...................... ) (It doesn't matter if the input spills over the SIZE guide, because "skipgarb()" discards it.)

Input is complete. Press RETURN. Generation 1 .................... .................... .................... .................... ...........*........ ..........***....... ..........***....... .................... .................... .................... .******************. .******************. .******************.

.................... .................... .................... .................... .................... .................... .................... Q for quit. RETURN to continue. Generation 2 .................... .................... .................... .................... ..........***....... .................... ..........*.*....... ...........*........ .................... ..****************.. .*................*. *..................* .*................*. ..****************.. .................... .................... .................... .................... .................... .................... Q for quit. RETURN to continue. Generation 3 .................... .................... .................... ...........*........ ...........*........ ..........*.*....... ...........*........ ...........*........ ...*******...****... ..****************.. .******************. **................** .******************. ..****************.. ...**************... .................... .................... .................... .................... ....................

Q for quit. RETURN to continue. Generation 4 .................... .................... .................... .................... ..........***....... ..........*.*....... ..........***....... ....*****.*.*.**.... ..*..............*.. .*................*. *..................* *..................* *..................* .*................*. ..*..............*.. ....************.... .................... .................... .................... .................... Q for quit. RETURN to continue.

etc... Try experimenting with different starting patterns. Node:Initializing Arrays, Next:Arrays and Pointers, Previous:Output of Game of Life, Up:Arrays

Initializing Arrays
Arrays can be initialized in two ways. The first way is by assigning every element to some value with a statement like:
array[2] = 42; array[3] = 12;

or perhaps with the aid of one or more for loops. Because it is tedious, to say the least, not to mention uneconomical, to initialize the values of each element to as different value, C provides another method, which employs a single assignment operator = and curly braces { }. This method only works for static variables and external variables. Recall that arrays are stored row-wise or with the last index varying fastest. A 3 by 3 array could be initialized in the following way:
static int array[3][3] = { {10,23,42}, {1,654,0}, {40652,22,0}

};

The internal braces are unnecessary, but help to distinguish the rows from the columns. The same thing could be written:
int array[3][3] = { 10,23,42, 1,654,0 40652,22,0 };

Take care to include the semicolon at the end of the curly brace which closes the assignment. Note that, if there are not enough elements in the curly braces to account for every single element in an array, the remaining elements will be filled out with zeros. Static variables are always guaranteed to be initialized to zero anyway, whereas auto or local variables are guaranteed to be garbage: this is because static storage is created by the compiler in the body of a program, whereas auto or local storage is created at run time. Node:Arrays and Pointers, Next:Arrays as Parameters, Previous:Initializing Arrays, Up:Arrays

Arrays and Pointers

The information about how arrays are stored was not included just for interest. There is another way of looking at arrays which follows the BCPL idea of an array as simply a block of memory. An array can be accessed with pointers as well as with [] square brackets. The name of an array variable, standing alone, is actually a pointer to the first element in the array. For example: if an array is declared
float numbers[34];

then numbers is a pointer to the first floating point number in the array; numbers is a pointer in its own right. (In this case it is type `pointer to float'.) So the first element of the array could be accessed by writing:
numbers[0] = 22.3;

or by writing
*numbers = 22.3;

For character arrays, which are dealt with in some depth in chapter 20, this gives an alternative way of getting at the elements in the array.

char arrayname[5]; char *ptr; for (ptr = arrayname; ptr <= arrayname+4; ptr++) { *ptr = 0; }

The code above sets the array arrayname to zero. This method of getting at array data is not recommended by this author except in very simple computer environments. If a program is running on a normal microcomputer, then there should be few problems with this alternative method of handling arrays. On the hand, if the microcomputer is multi-tasking, or the program is running on a larger system which has a limited manager, then memory ceases to be something which can be thought of as a sequence of boxes standing next to one another. A multi-tasking system shares memory with other programs and it takes what it can find, where it can find it. The upshot of this is that it is not possible to guarantee that arrays will be stored in one simple string of memory locations: it might be scattered around in different places. So
ptr = arrayname + 5;

might not be a pointer to the fifth character in a character array. This could be found instead using the & operator. A pointer to the fifth element can be reliably found with:
ptr = &(arrayname[5]);

Be warned! Node:Arrays as Parameters, Next:Questions 19, Previous:Arrays and Pointers, Up:Arrays

Arrays as Parameters
What happens if we want to pass an array as a parameter? Does the program copy the entire array into local storage? The answer is no because it would be a waste of time and memory. Arrays can be passed as parameters, but only as variable ones. This is a simple matter, because the name of the array is a pointer to the array. The Game of Life program above does this. Notice from that program how the declarations for the parameters are made.
main () { char array[23]; function (array); ..... } function (arrayformal) char arrayformal[23];

{ }

Any function which writes to the array, passed as a parameter, will affect the original copy. Array parameters are always variable parameters Node:Questions 19, Previous:Arrays as Parameters, Up:Arrays

Questions
1. Given any array, how would you find a pointer to the start of it? 2. How do you pass an array as a parameter? When the parameter is received by a function does C allocate space for a local variable and copy the whole array to the new location? 3. Write a statement which declares an array of type double which measures 4 by 5. What numbers can be written in the indicies of the array? Node:Strings, Next:Putting together a program, Previous:Arrays, Up:Top

Strings
Communication with arrays. Strings are pieces of text which can be treated as values for variables. In C a string is represented as some characters enclosed by double quotes.
"This is a string"

A string may contain any character, including special control characters, such as \n, \r, \7 etc...
"Beep! \7 Newline \n..."

Conventions and Declarations: Strings Arrays and Pointers: Arrays of Strings: Example 21: Strings from the user: Handling strings: Example 22: String Input/Output: Example 23: Questions 20:

Node:Conventions and Declarations, Next:Strings Arrays and Pointers, Previous:Strings, Up:Strings

Conventions and Declarations

There is an important distinction between a string and a single character in C. The convention is that single characters are enclosed by single quotes e.g. * and have the type char. Strings, on the hand, are enclosed by double quotes e.g. "string..." and have the type "pointer to char" (char *) or array of char. Here are some declarations for strings which are given without immediate explanations.
/**********************************************************/ /* */ /* String Declaration */ /* */ /**********************************************************/ #define SIZE 10

char *global_string1; char global_string2[SIZE]; main () { char *auto_string; char arraystr[SIZE]; static char *stat_strng; static char statarraystr[SIZE]; }

Node:Strings Arrays and Pointers, Next:Arrays of Strings, Previous:Conventions and Declarations, Up:Strings

Strings, Arrays and Pointers

A string is really an array of characters. It is stored at some place the memory and is given an end marker which standard library functions can recognize as being the end of the string. The end marker is called the zero (or NULL) byte because it is just a byte which contains the value zero: \0. Programs rarely gets to see this end marker as most functions which handle strings use it or add it automatically. Strings can be declared in two main ways; one of these is as an array of characters, the other is as a pointer to some pre-assigned array. Perhaps the simplest way of seeing how C stores arrays is to give an extreme example which would probably never be used in practice. Think of how a string called string might be used to to store the message "Tedious!". The fact that a string is an array of characters might lead you to write something like:
#define LENGTH 9; main () { char string[LENGTH];

string[0] string[1] string[2] string[3] string[4] string[5] string[6] string[7] string[8]

= = = = = = = = =

'T'; 'e'; 'd'; 'i'; 'o'; 'u'; 's'; '!'; '\0';

printf ("%s", string); }

This method of handling strings is perfectly acceptable, if there is time to waste, but it is so laborious that C provides a special initialization service for strings, which bypasses the need to assign every single character with a new assignment!. There are six ways of assigning constant strings to arrays. (A constant string is one which is actually typed into the program, not one which in typed in by the user.) They are written into a short compilable program below. The explanation follows.
/**********************************************************/ /* */ /* String Initialization */ /* */ /**********************************************************/ char *global_string1 = "A string declared as a pointer"; char global_string2[] = "Declared as an array";

main () { char *auto_string = "initializer..."; static char *stat_strng = "initializer..."; static char statarraystr[] = "initializer...."; /* char arraystr[] = "initializer...."; IS ILLEGAL! */ /* This is because the array is an "auto" type */ /* which cannot be preinitialized, but... */ char arraystr[20]; printf ("%s %s", global_string1, global_string2); printf ("%s %s %s", auto_string, stat_strng, statarraystr); } /* end */

The details of what goes on with strings can be difficult to get to grips with. It is a good idea to get revise pointers and arrays before reading the explanations below. Notice the diagrams too: they are probably more helpful than words.

The first of these assignments is a global, static variable. More correctly, it is a pointer to a global, static array. Static variables are assigned storage space in the body of a program when the compiler creates the executable code. This means that they are saved on disk along with the program code, so they can be initialized at compile time. That is the reason for the rule which says that only static arrays can be initialized with a constant expression in a declaration. The first statement allocates space for a pointer to an array. Notice that, because the string which is to be assigned to it, is typed into the program, the compiler can also allocate space for that in the executable file too. In fact the compiler stores the string, adds a zero byte to the end of it and assigns a pointer to its first character to the variable called global_string1. The second statement works almost identically, with the exception that, this time the compiler sees the declaration of a static array, which is to be initialized. Notice that there is no size declaration in the square brackets. This is quite legal in fact: the compiler counts the number of characters in the initialization string and allocates just the right amount of space, filling the string into that space, along with its end marker as it goes. Remember also that the name of the array is a pointer to the first character, so, in fact, the two methods are identical. The third expression is the same kind of thing, only this time, the declaration is inside the function main() so the type is not static but auto. The difference between this and the other two declarations is that this pointer variable is created every time the function main() is called. It is new each time and the same thing holds for any other function which it might have been defined in: when the function is called, the pointer is created and when it ends, it is destroyed. The string which initializes it is stored in the executable file of the program (because it is typed into the text). The compiler returns a value which is a pointer to the string's first character and uses that as a value to initialize the pointer with. This is a slightly round about way of defining the string constant. The normal thing to do would be to declare the string pointer as being static, but this is just a matter of style. In fact this is what is done in the fourth example. The fifth example is again identical, in practice to other static types, but is written as an `open' array with an unspecified size. The sixth example is forbidden! The reason for this might seem rather trivial, but it is made in the interests of efficiency. The array declared is of type auto: this means that the whole array is created when the function is called and destroyed afterwards. autoarrays cannot be initialized with a string because they would have to be re-initialized every time the array were created: that is, each time the function were called. The final example could be used to overcome this, if the programmer were inclined to do so. Here an auto array of characters is declared (with a size this time, because there is nothing for the compiler to count the size of). There is no single assignment which will fill this array with a string though: the programmer would have to do it character by character so that the inefficiency is made as plain as possible! Node:Arrays of Strings, Next:Example 21, Previous:Strings Arrays and Pointers, Up:Strings

Arrays of Strings

In the previous chapter we progressed from one dimensional arrays to two dimensional arrays, or arrays of arrays! The same thing works well for strings which are declared static. Programs can take advantage of C's easy assignment facilities to let the compiler count the size of the string arrays and define arrays of messages. For example here is a program which prints out a menu for an application program:
/*********************************************************/ /* */ /* MENU : program which prints out a menu */ /* */ /*********************************************************/ main () { int str_number; for (str_number = 0; str_number < 13; str_number++) { printf ("%s",menutext(str_number)); } } /*********************************************************/ char *menutext(n) int n; { static char *t[] = { " -------------------------------------- \n", " | ++ MENU ++ |\n", " | ~~~~~~~~~~~~ |\n", " | (1) Edit Defaults |\n", " | (2) Print Charge Sheet |\n", " | (3) Print Log Sheet |\n", " | (4) Bill Calculator |\n", " | (q) Quit |\n", " | |\n", " | |\n", " | Please Enter Choice |\n", " | |\n", " -------------------------------------- \n" }; return (t[n]); } /* return n-th string ptr */

Notice the way in which the static declaration works. It is initialized once at compile time, so there is effectively only one statement in this function and that is the return statement. This function retains the pointer information from call to call. The Morse coder program could be rewritten more economically using static strings, See Example 15. Node:Example 21, Next:Strings from the user, Previous:Arrays of Strings, Up:Strings

Example Listing

/************************************************/ /* */ /* static string array */ /* */ /************************************************/ /* Morse code program. Enter a number and */ /* find out what it is in Morse code */ #include <stdio.h> #define CODE 0 /*************************************************/ main () { short digit; printf ("Enter any digit in the range 0..9"); scanf ("%h",&digit); if ((digit < 0) || (digit > 9)) { printf ("Number was not in range 0..9"); return (CODE); } printf ("The Morse code of that digit is "); Morse (digit); } /************************************************/ Morse (digit) short digit; { static char *code[] = { "dummy", "-----", ".----", "..---", "...--", "....-", ".....", "-....", "--...", "---..", "----.", }; /* print out Morse code */

/* index starts at 0 */

printf ("%s\n",code[digit]); }

Node:Strings from the user, Next:Handling strings, Previous:Example 21, Up:Strings

Strings from the user

All the strings mentioned so far have been typed into a program by the programmer and stored in a program file, so it has not been necessary to worry about where they were stored. Often though we would like to fetch a string from the user and store it somewhere in the memory for later use. It might even be necessary to get a whole bunch of strings and store them all. But how will the program know in advance how much array space to allocate to these strings? The answer is that it won't, but that it doesn't matter at all! One way of getting a simple, single string from the user is to define an array and to read the characters one by one. An example of this was the Game of Life program the the previous chapter:

Define the array to be a certain size Check that the user does not type in too many characters. Use the string in that array.

Another way is to define a static string with an initializer as in the following example. The function filename() asks the user to type in a filename, for loading or saving by and return it to a calling function.
char *filename() { static char *filenm = "........................"; do { printf ("Enter filename :"); scanf ("%24s",filenm); skipgarb(); } while (strlen(filenm) == 0); return (filenm); }

The string is made static and given an initializing expression and this forces the compiler to make some space for the string. It makes exactly 24 characters plus a zero byte in the program file, which can be used by an application. Notice that the conversion string in scanf prevents the characters from spilling over the bounds of the string. The function strlen() is a standard library function which is described below; it returns the length of a string. skipgarb() is the function which was introduced in chapter 15. Neither of the methods above is any good if a program is going to be fetching a lot of strings from a user. It just isn't practical to define lots of static strings and expect the user to type into the right size boxes! The next step in string handling is therefore to allocate memory for strings personally: in other words to be able to say how much storage is needed for a string while a program is running. C has special memory allocation functions which can do this, not only for strings but for any kind of object. Suppose then that a program is going to get ten strings from the user. Here is one way in which it could be done: 1. Define one large, static string (or array) for getting one string at a time. Call this a string buffer, or waiting place.

2. Define an array of ten pointers to characters, so that the strings can be recalled easily. 3. Find out how long the string in the string buffer is. 4. Allocate memory for the string. 5. Copy the string from the buffer to the new storage and place a pointer to it in the array of pointers for reference. 6. Release the memory when it is finished with. The function which allocates memory in C is called malloc() and it works like this:
malloc()

should be declared as returning the type pointer to character, with the

statement:
char *malloc();

takes one argument which should be an unsigned integer value telling the function how many bytes of storage to allocate. It returns a pointer to the first memory location in that storage:
malloc() char *ptr; unsigned int size; ptr = malloc(size);

The pointer returned has the value NULL if there was no memory left to allocate. This should always be checked.

The fact that malloc() always returns a pointer to a character does not stop it from being used for other types of data too. The cast operator can force malloc() to give a pointer to any data type. This method is used for building data structures in C with "struct" types. has a complementary function which does precisely the opposite: deallocates memory. This function is called free(). free() returns an integer code, so it does not have to be declared as being any special type.
malloc()

takes one argument: a pointer to a block of memory which has previously been allocated by malloc().
free() int returncode; returncode = free (ptr);

The pointer should be declared:

char *ptr;

The return code is zero if the release was successful.

An example of how strings can be created using malloc() and free() is given below. First of all, some explanation of Standard Library Functions is useful to simplify the program. Node:Handling strings, Next:Example 22, Previous:Strings from the user, Up:Strings

Handling strings

The C Standard Library commonly provides a number of very useful functions which handle strings. Here is a short list of some common ones which are immediately relevant (more are listed in the following chapter). Chances are, a good compiler will support a lot more than those listed below, but, again, it really depends upon the compiler.
strlen()

This function returns a type int value, which gives the length or number of characters in a string, not including the NULL byte end marker. An example is:
int len; char *string; len = strlen (string);

strcpy()

This function copies a string from one place to another. Use this function in preference to custom routines: it is set up to handle any peculiarities in the way data are stored. An example is
char *to,*from; to = strcpy (to,from);

Where to is a pointer to the place to which the string is to be copied and from is the place where the string is to be copied from.
strcmp()

This function compares two strings and returns a value which indicates how they compared. An example:
int value; char *s1,*s2; value = strcmp(s1,s2);

strstr()

The value returned is 0 if the two strings were identical. If the strings were not the same, this function indicates the (ASCII) alphabetical order of the two. s1 > s2, alphabetically, then the value is > 0. If s1 < s2 then the value is < 0. Note that numbers come before letters in the ASCII code sequence and also that upper case comes before lower case. Tests whether a substring is present in a larger string
int n; char *s1,*s2; if (n = strstr(s1,s2)) { printf("s2 is a substring of s1, starting at %d",n); }

strncpy() strncmp()

This function is like strcpy, but limits the copy to no more than n characters. This function is like strcmp, but limits the comparison to no more than n characters.

More string functions are described in the next section along with a host of Standard Library Functions. Node:Example 22, Next:String Input/Output, Previous:Handling strings, Up:Strings

Example Listing
This program aims to get ten strings from the user. The strings may not contain any spaces or white space characters. It works as follows: The user is prompted for a string which he/she types into a buffer. The length of the string is tested with strlen() and a block of memory is allocated for it using malloc(). (Notice that this block of memory is one byte longer than the value returned by strlen(), because strlen() does not count the end of string marker \0.) malloc() returns a pointer to the space allocated, which is then stored in the array called array. Finally the strings is copied from the buffer to the new storage with the library function strcpy(). This process is repeated for each of the 10 strings. Notice that the program exits through a low level function called QuitSafely(). The reason for doing this is to exit from the program neatly, while at the same time remembering to perform all a programmer's duties, such as de-allocating the memory which is no longer needed. QuitSafely() uses the function exit() which should be provided as a standard library function. exit() allows a program to end at any point.
/******************************************************/ /* */ /* String storage allocation */ /* */ /******************************************************/ #include <stdio.h> /* #include another file for malloc() and */ /* strlen() ???. Check the compiler manual! */ #define NOOFSTR #define BUFSIZE #define CODE 10 255 0

/******************************************************/ /* Level 0 */ /******************************************************/ main () { char *array[NOOFSTR], *malloc(); char buffer[BUFSIZE]; int i; for (i = 0; i < NOOFSTR; i++) { printf ("Enter string %d :",i); scanf ("%255s",buffer); array[i] = malloc(strlen(buffer)+1); if (array[i] == NULL)

{ printf ("Can't allocate memory\n"); QuitSafely (array); } strcpy (array[i],buffer); } for (i = 0; i < NOOFSTR; i++) { printf ("%s\n",array[i]); } QuitSafely(array); } /******************************************************/ /* Snakes & Ladders! */ /******************************************************/ QuitSafely (array) char *array[NOOFSTR]; { int i, len; for (i = 0; i < NOOFSTR; i++) { len = strlen(array[i]) + 1; if (free (array[i]) != 0) { printf ("Debug: free failed\n"); } } exit (CODE); } /* end */ /* Quit & de-alloc memory */

Node:String Input/Output, Next:Example 23, Previous:Example 22, Up:Strings

String Input/Output
Because strings are recognized to be special objects in C, some special library functions for reading and writing are provided for them. These make it easier to deal with strings, without the need for special user-routines. There are four of these functions:
gets() puts() sprintf() sscanf()

gets(): puts(): sprintf(): sscanf():

Node:gets(), Next:puts(), Previous:String Input/Output, Up:String Input/Output

gets()

This function fetches a string from the standard input file stdin and places it into some buffer which the programmer must provide.
#define SIZE 255

char *sptr, buffer[SIZE]; strptr = gets(buffer);

If the routine is successful in getting a string, it returns the value buffer to the string pointer strptr. Otherwise it returns NULL (==0). The advantage of gets() over scanf("%s"..) is that it will read spaces in strings, whereas scanf() usually will not. gets() quits reading when it finds a newline character: that is, when the user presses RETURN. NOTE: there are valid concerns about using this function. Often it is implemented as a macro with poor bounds checking and can be exploited to produce memory corruption by system attackers. In order to write more secure code, use fgets() instead. Node:puts(), Next:sprintf(), Previous:gets(), Up:String Input/Output
puts()

sends a string to the output file stdout, until it finds a NULL end of string marker. The NULL byte is not written to stdout, instead a newline character is written.
puts() char *string; int returncode; returncode = puts(string);

puts() returns an integer value, whose value is only guaranteed if there is an returncode == EOF if an end of file was encountered or there was an error.

error.

Node:sprintf(), Next:sscanf(), Previous:puts(), Up:String Input/Output

sprintf()

This is an interesting function which works in almost the same way as printf(), the exception being that it prints to a string! In other words it treats a string as though it were an output file. This is useful for creating formatted strings in the memory. On most systems it works in the following way:
int n; char *sp; n = sprintf (sp, "control string", parameters, values);

is an integer which is the number of characters printed. sp is a pointer to the destination string or the string which is to be written to. Note carefully that this function does not perform any check on the output string to make sure that it is long enough to contain the formatted output. If the string is not large enough, then a crash could be in store! This can also be considered a potential security problem, since buffer overflows can be used to capture control of important programs. Note that on system V Unix systems the sprintf functionr returns a pointer to the start of the printed string, breaking the pattern of the other printf functions. To make such an implementation compatible with the usual form you would have to write:
n n = strlen(sprintf(parameters......));

Node:sscanf(), Previous:sprintf(), Up:String Input/Output

sscanf()

This function is the complement of sprintf(). It reads its input from a string, as though it were an input file.
int n; char *sp; n = sscanf (sp,"control string", pointers...);

is a pointer to the string which is to be read from. The string must be NULL terminated (it must have a zero-byte end marker '\0'). sscanf() returns an integer value which holds the number of items successfully matched or EOF if an end of file marker was read or an error occurred. The conversion specifiers are identical to those for scanf().
sp

Node:Example 23, Next:Questions 20, Previous:String Input/Output, Up:Strings

Example Listing
/************************************************/ /* */ /* Formatted strings */ /* */ /************************************************/ /* program rewrites s1 in reverse into s2 */ #include <stdio.h> #define SIZE #define CODE 20 0

/************************************************/ main () { static char *s1 = "string 2.3 55x"; static char *s2 = "...................."; char ch, *string[SIZE];

int i,n; float x; sscanf (s1,"%s %f %d %c", string, &x, &i, &ch); n = sprintf (s2,"%c %d %f %s", ch, i, x, string); if (n > SIZE) { printf ("Error: string overflowed!\n"); exit (CODE); } puts (s2); }

Node:Questions 20, Previous:Example 23, Up:Strings

Questions
1. What are the two main ways of declaring strings in a program? 2. How would you declare a static array of strings? 3. Write a program which gets a number between 0 and 9 and prints out a different message for each number. Use a pre-initialized array to store the strings. Node:Putting together a program, Next:Special Library Functions and Macros, Previous:Strings, Up:Top

Putting together a program

Putting it all together.

argc and argv: getopt: envp:

Node:argc and argv, Next:getopt, Previous:Putting together a program, Up:Putting together a program

The argument vector

C was written in order to implement Unix in a portable form. Unix was designed with a command language which was built up of independent programs. These could be passed arguments on the command line. For instance:
ls -l /etc cc -o program prog.c

In these examples, the first word is the command itself, while the subsequent words are options and arguments to the command. We need some way getting this information into a C program. Unix solved this problem by passing C programs an array of these arguments together with their number as parameters to the function main(). Since then most other operating systems have adopted the same model, since it has become a part of the C language.
main (argc,argv) int argc; char *argv[]; { }

The traditional names for the parameters are the argument count argc and the argument vector (array) argv. The operating system call which starts the C program breaks up the command line into an array, where the first element argv[0] is the name of the command itself and the last argument argv[argc-1] is the last argument. For example, in the case of
cc -o program prog.c

would result in the values

argv[0] argv[1] argv[2] argv[3]

cc -o program

prog.c The following program prints out the command line arguments:
main (argc,argv) int argc; char *argv[]; { int i; printf ("This program is called %s\n",argv[0]); if (argc > 1) { for (i = 1; i < argc; i++) { printf("argv[%d] = %s\n",i,argv[i]); } } else { printf("Command has no arguments\n"); } }

Node:getopt, Next:envp, Previous:argc and argv, Up:Putting together a program

Processing options
getopt Node:envp, Previous:getopt, Up:Putting together a program

Environment variables
When we write a C program which reads command line arguments, they are fed to us by the argument vector. Unix processes also a set of text variable associations called environment variables. Each child process inherits the environment of its parent. The static environment variables are stored in a special array which is also passed to main() and can be read if desired.
main (argc,argv,envp) int argc; char *argv[], *envp[]; { }

The array of strings envp[] is a list of values of the environment variables of the system, formatted by
NAME=value

This gives C programmers access to the shell's global environment. In addition to the envp vector, it is possible to access the environment variables through the call getenv(). This is used as follows; suppose we want to access the shell environment variable $HOME.
char *string; string = getenv("HOME");

is now a pointer to static but public data. You should not use string as if it were you're own property because it will be used again by the system. Copy it's contents to another string before using the data.
string char buffer[500]; strcpy (buffer,string);

Node:Special Library Functions and Macros, Next:Hidden Operators, Previous:Putting together a program, Up:Top

Special Library Functions and Macros

Checking character types. Handling strings. Doing maths. C provides a repertoire of standard library functions and macros for specialized purposes (and for the advanced user). These may be divided into various categories. For instance

Character identification (ctype.h) String manipulation (string.h) Mathematical functions (math.h)

A program generally has to #include special header files in order to use special functions in libraries. The names of the appropriate files can be found in particular compiler manuals. In the examples above the names of the header files are given in parentheses.

Character Identification: Example 24: Output 24: String Manipulation: Example 25: Mathematical Functions: Examples 26: Maths Errors: Example 27: Questions 21:

Node:Character Identification, Next:Example 24, Previous:Special Library Functions and Macros, Up:Special Library Functions and Macros

Character Identification
Some or all of the following functions/macros will be available for identifying and classifying single characters. The programmer ought to beware that it would be natural for many of these facilities to exist as macros rather than functions, so the usual remarks about macro parameters apply, See Preprocessor. An example of their use is given above. Assume that `true' has any non-zero, integer value and that `false' has the integer value zero. ch stands for some character, or char type variable.
isalpha(ch)

isupper(ch)

This returns true if ch is alphabetic and false otherwise. Alphabetic means a..z or A..Z. Returns true if the character was upper case. If ch was not an alphabetic character, this returns false.

islower(ch)

isdigit(ch)

Returns true if the character was lower case. If ch was not an alphabetic character, this returns false. Returns true if the character was a digit in the range 0..9.

isxdigit(ch)

isspace(ch)

Returns true if the character was a valid hexadecimal digit: that is, a number from 0..9 or a letter a..f or A..F. Returns true if the character was a white space character, that is: a space, a TAB character or a newline.

ispunct(ch) isalnum(ch) isprint(ch)

Returns true if ch is a punctuation character. Returns true if a character is alphanumeric: that is, alphabetic or digit. Returns true if the character is printable: that is, the character is not a control character. Returns true if the character is graphic. i.e. if the character is printable (excluding the space)

isgraph(ch)

iscntrl(ch)

isascii(ch)

Returns true if the character is a control character. i.e. ASCII values 0 to 31 and 127. Returns true if the character is a valid ASCII character: that is, it has a code in the range 0..127.

iscsym(ch)

toupper(ch)

Returns true if the character was a character which could be used in a C identifier. This converts the character ch into its upper case counterpart. This does not affect characters which are already upper case, or characters which do not have a particular case, such as digits.

tolower(ch)

toascii(ch)

This converts a character into its lower case counterpart. It does not affect characters which are already lower case.

This strips off bit 7 of a character so that it is in the range 0..127: that is, a valid ASCII character. Node:Example 24, Next:Output 24, Previous:Character Identification, Up:Special Library Functions and Macros

Examples
/********************************************************/ /* */ /* Demonstration of character utility functions */ /* */ /********************************************************/ /* prints out all the ASCII characters which give */ /* the value "true" for the listed character fns */ #include <stdio.h> #include <ctype.h> #define ALLCHARS /* contains character utilities */ ch = 0; isascii(ch); ch++

/********************************************************/ main () /* A criminally long main program! */

{ char ch; printf ("VALID CHARACTERS FROM isalpha()\n\n"); for (ALLCHARS) { if (isalpha(ch)) { printf ("%c ",ch); } } printf ("\n\nVALID CHARACTERS FROM isupper()\n\n"); for (ALLCHARS) { if (isupper(ch)) { printf ("%c ",ch); } } printf ("\n\nVALID CHARACTERS FROM islower()\n\n"); for (ALLCHARS) { if (islower(ch)) { printf ("%c ",ch); } } printf ("\n\nVALID CHARACTERS FROM isdigit()\n\n"); for (ALLCHARS) { if (isdigit(ch)) { printf ("%c ",ch); } } printf ("\n\nVALID CHARACTERS FROM isxdigit()\n\n"); for (ALLCHARS) { if (isxdigit(ch)) { printf ("%c ",ch); } } printf ("\n\nVALID CHARACTERS FROM ispunct()\n\n"); for (ALLCHARS) { if (ispunct(ch)) { printf ("%c ",ch); } }

printf ("\n\nVALID CHARACTERS FROM isalnum()\n\n"); for (ALLCHARS) { if (isalnum(ch)) { printf ("%c ",ch); } } printf ("\n\nVALID CHARACTERS FROM iscsym()\n\n"); for (ALLCHARS) { if (iscsym(ch)) { printf ("%c ",ch); } } }

Node:Output 24, Next:String Manipulation, Previous:Example 24, Up:Special Library Functions and Macros

Program Output
VALID CHARACTERS FROM isalpha() A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z VALID CHARACTERS FROM isupper() A B C D E F G H I J K L M N O P Q R S T U V W X Y Z VALID CHARACTERS FROM islower() a b c d e f g h i j k l m n o p q r s t u v w x y z VALID CHARACTERS FROM isdigit() 0 1 2 3 4 5 6 7 8 9 VALID CHARACTERS FROM isxdigit() 0 1 2 3 4 5 6 7 8 9 A B C D E F a b c d e f VALID CHARACTERS FROM ispunct() ! " # $ % & ' ( ) * + , - . / : ; < = > ? @ [ \ ] ^ _ ` { | } ~ VALID CHARACTERS FROM isalnum() 0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z VALID CHARACTERS FROM iscsym() 0 1 2 3 4 5 6 7 8 9 A B C D E F G H I J K L M N O P Q R S T U V W X Y Z _ a b c d e f g h i j k l m n o p q r s t u v w x y z

Node:String Manipulation, Next:Example 25, Previous:Output 24, Up:Special Library Functions and Macros

String Manipulation
The following functions perform useful functions for string handling, See Strings.
strcat()

This function "concatenates" two strings: that is, it joins them together into one string. The effect of:
char *new,*this, onto[255]; new = strcat(onto,this);

is to join the string this onto the string onto. new is a pointer to the complete string; it is identical to onto. Memory is assumed to have been allocated for the starting strings. The string which is to be copied to must be large enough to accept the new string, tagged onto the end. If it is not then unpredictable effects will result. (In some programs the user might get away without declaring enough space for the "onto" string, but in general the results will be garbage, or even a crashed machine.) To join two static strings together, the following code is required:
char *s1 = "string one"; char *s2 = "string two"; main () { char buffer[255]; strcat(buffer,s1); strcat(buffer,s2); }

buffer would then contain "string onestring two".

strlen()

This function returns a type int value, which gives the length or number of characters in a string, not including the NULL byte end marker. An example is:
int len; char *string; len = strlen (string);

strcpy()

This function copies a string from one place to another. Use this function in preference to custom routines: it is set up to handle any peculiarities in the way data are stored. An example is
char *to,*from; to = strcpy (to,from);

Where to is a pointer to the place to which the string is to be copied and from is the place where the string is to be copied from.
strcmp()

This function compares two strings and returns a value which indicates how they compared. An example:
int value; char *s1,*s2; value = strcmp(s1,s2);

The value returned is 0 if the two strings were identical. If the strings were not the same, this function indicates the (ASCII) alphabetical order of the two. s1 > s2, alphabetically, then the value is > 0. If s1 < s2 then the value is < 0. Note that numbers come before letters in the ASCII code sequence and also that upper case comes before lower case. There are also variations on the theme of the functions above which begin with strn instead of str. These enable the programmer to perform the same actions with the first n characters of a string:
strncat()

This function concatenates two strings by copying the first n characters of this to the end of the onto string.
char *onto,*new,*this; new = strncat(onto,this,n);

strncpy()

This function copies the first n characters of a string from one place to another
char *to,*from; int n;

to = strncpy (to,from,n);

strncmp()

This function compares the first n characters of two strings

int value; char *s1,*s2;

value = strcmp(s1,s2,n);

The following functions perform conversions between strings and floating point/integer types, without needing to use sscanf(). They take a pre-initialized string and work out the value represented by that string.
atof()

ASCII to floating point conversion.

double x; char *stringptr;

x = atof(stringptr);

atoi()

ASCII to integer conversion.

int i; char *stringptr; i = atoi(stringptr);

atol()

ASCII to long integer conversion.

long i; char *stringptr; i = atol(stringptr);

Node:Example 25, Next:Mathematical Functions, Previous:String Manipulation, Up:Special Library Functions and Macros

Examples
/********************************************************/ /* */ /* String comparison */ /* */ /********************************************************/ #include <stdio.h> #define TRUE 1 #define MAXLEN 30 /********************************************************/ main () { char string1[MAXLEN],string2[MAXLEN]; int result; while (TRUE) { printf ("Type in string 1:\n\n"); scanf ("%30s",string1); printf ("Type in string 2:\n\n"); scanf ("%30s",string2); result = strcmp (string1,string2); if (result == 0) { printf ("Those strings were the same!\n"); } if (result > 0) { printf ("string1 > string2\n"); } if (result < 0) {

} }

printf ("string1 < string 2\n"); }

Node:Mathematical Functions, Next:Examples 26, Previous:Example 25, Up:Special Library Functions and Macros

Mathematical Functions
C has a library of standard mathematical functions which can be accessed by #including the appropriate header files (math.h etc.). It should be noted that all of these functions work with double or long float type variables. All of C's mathematical capabilities are written for long variable types. Here is a list of the functions which can be expected in the standard library file. The variables used are all to be declared long
int i; double x,y,result; /* long int */ /* long float */

The functions themselves must be declared long float or double (which might be done automatically in the mathematics library file, or in a separate file) and any constants must be written in floating point form: for instance, write 7.0 instead of just 7.
ABS()

fabs()

MACRO. Returns the unsigned value of the value in parentheses. See fabs() for a function version. Find the absolute or unsigned value of the value in parentheses:
result = fabs(x);

ceil()

Find out what the ceiling integer is: that is, the integer which is just above the value in parentheses. This is like rounding up.
i = ceil(x); /* ceil (2.2) is 3 */

floor()

Find out what the floor integer is: that is, the integer which is just below the floating point value in parentheses
i = floor(x); /* floor(2.2) is 2 */

exp()

Find the exponential value.

result = exp(x); result = exp(2.7);

log()

Find the natural (Naperian) logarithm. The value used in the parentheses must be unsigned: that is, it must be greater than zero. It does not have to be declared specifically as unsigned. e.g.
result = log(x); result = log(2.71828);

log10()

Find the base 10 logarithm. The value used in the parentheses must be unsigned: that is, it must be greater than zero. It does not have to be declared specifically as unsigned.
result = log10(x); result = log10(10000);

pow()

Raise a number to the power.

result = pow(x,y); /*raise x to the power y */ result = pow(x,2); /*find x-squared */ result = pow(2.0,3.2); /* find 2 to the power 3.2 ...*/ sqrt()

Find the square root of a number.

result = sqrt(x); result = sqrt(2.0);

sin()

Find the sine of the angle in radians.

result = sin(x); result = sin(3.14);

cos()

Find the cosine of the angle in radians.

result = cos(x); result = cos(3.14);

tan()

Find the tangent of the angle in radians.

result = tan(x); result = tan(3.14);

asin()

Find the arcsine or inverse sine of the value which must lie between +1.0 and -1.0.
result = asin(x); result = asin(1.0);

acos()

Find the arccosine or inverse cosine of the value which must lie between +1.0 and -1.0.
result = acos(x);

result = acos(1.0);

atan()

Find the arctangent or inverse tangent of the value.

result = atan(x); result = atan(200.0);

atan2()

This is a special inverse tangent function for calculating the inverse tangent of x divided by y. This function is set up to find this result more accurately than atan().
result = atan2(x,y); result = atan2(x/3.14);

sinh()

Find the hyperbolic sine of the value. (Pronounced "shine" or "sinch")

result = sinh(x); result = sinh(5.0);

cosh()

Find the hyperbolic cosine of the value.

result = cosh(x); result = cosh(5.0);

tanh()

Find the hyperbolic tangent of the value.

result = tanh(x); result = tanh(5.0);

Node:Examples 26, Next:Maths Errors, Previous:Mathematical Functions, Up:Special Library Functions and Macros

Examples
/******************************************************/ /* */ /* Maths functions demo #1 */ /* */ /******************************************************/ /* use sin(x) to work out an animated model */ #include <stdio.h> #include <math.h> #include <limits.h> #define TRUE #define AMPLITUDE #define INC double pi; 1 30 0.02 /* this may already be defined */ /* in the math file */

/******************************************************/

/* Level 0 */ /******************************************************/ main () { pi = asin(1.0)*2; /* The simple pendulum program */ /* if PI is not defined */

printf ("\nTHE SIMPLE PENDULUM:\n\n\n"); Pendulum(); } /*****************************************************/ /* Level 1 */ /*****************************************************/ Pendulum () { double x, twopi = pi * 2; int i,position; while (true) { for (x = 0; x < twopi; x += INC) { position = (int)(AMPLITUDE * sin(x)); for (i = -AMPLITUDE; i <= AMPLITUDE; i++) { if (i == position) { putchar('*'); } else { putchar(' '); } } startofline(); }

} }

/*****************************************************/ /* Toolkit */ /*****************************************************/ startofline() { putchar('\r'); }

Node:Maths Errors, Next:Example 27, Previous:Examples 26, Up:Special Library Functions and Macros

Maths Errors
Mathematical functions can be delicate animals. There exist mathematical functions which simply cannot produce sensible answers in all possible cases. Mathematical functions are not "user friendly"! One example of an unfriendly function is the inverse

sine function asin(x) which only works for values of x in the range +1.0 to -1.0. The reason for this is a mathematical one: namely that the sine function (of which asin() is the opposite) only has values in this range. The statement
y = asin (25.3);

is nonsense and it cannot possibly produce a value for y, because none exists. Similarly, there is no simple number which is the square root of a negative value, so an expression such as:
x = sqrt(-2.0);

would also be nonsense. This doesn't stop the programmer from writing these statements though and it doesn't stop a faulty program from straying out of bounds. What happens then when an erroneous statement is executed? Some sort of error condition would certainly have to result. In many languages, errors, like the ones above, are terminal: they cause a program to stop without any option to recover the damage. In C, as the reader might have come to expect, this is not the case. It is possible (in principle) to recover from any error, whilst still maintaining firm control of a program. Errors like the ones above are called domain errors (the set of values which a function can accept is called the domain of the function). There are other errors which can occur too. For example, division by zero is illegal, because dividing by zero is "mathematical nonsense" - it can be done, but the answer can be all the numbers which exist at the same time! Obviously a program cannot work with any idea as vague as this. Finally, in addition to these "pathological" cases, mathematical operations can fail just because the numbers they deal with get too large for the computer to handle, or too small, as the case may be. Domain error Illegal value put into function Division by zero Dividing by zero is nonsense. Overflow Number became too large Underflow Number became too small. Loss of accuracy No meaningful answer could be calculated Errors are investigated by calling a function called matherr(). The mathematical functions, listed above, call this function automatically when an error is detected. The function responds by returning a value which gives information about the error. The exact details will depend upon a given compiler. For instance a hypothetical example: if the error could be recovered from, matherr() returns 0, otherwise it returns -1. matherr() uses a "struct" type variable called an "exception" to diagnose faults in mathematical functions, See Structures and Unions. This can be examined by programs

which trap their errors dutifully. Information about this structure must be found in a given compiler manual. Although it is not possible to generalize, the following remarks about the behaviour of mathematical functions may help to avoid any surprises about their behaviour in error conditions.

A function which fails to produce a sensible answer, for any of the reasons above, might simply return zero or it might return the maximum value of the computer. Be careful to check this. (Division by zero and underflow probably return zero, whereas overflow returns the maximum value which the computer can handle.) Some functions return the value NaN. Not a form of Indian unleavened bread, this stands for `Not a Number', i.e. no sensible result could be calculated. Some method of signalling errors must clearly be used. This is the exception structure (a special kind of C variable) which gives information about the last error which occurred. Find out what it is and trap errors! Obviously, wherever possible, the programmer should try to stop errors from occurring in the first place.

Node:Example 27, Next:Questions 21, Previous:Maths Errors, Up:Special Library Functions and Macros

Example
Here is an example for the mathematically minded. The program below performs numerical integration by the simplest possible method of adding up the area under small strips of a graph of the function f(y) = 2*y. The integral is found between the limits 0 and 5 and the exact answer is 25. (See diagram.) The particular compiler used for this program returns the largest number which can be represented by the computer when numbers overflow, although, in this simple case, it is impossible for the numbers to overflow.
/**********************************************************/ /* */ /* Numerical Estimation of Integral */ /* */ /**********************************************************/ #include <stdio.h> #include <math.h> #include <limits.h> #define LIMIT 5 double inc = 0.001; double twopi; /* Increment width - arbitrary */

/***********************************************************/ /** LEVEL 0 */ /***********************************************************/ main () { double y,integrand();

double integral = 0; twopi = 4 * asin(1.0); for ( y = inc/2; y < LIMIT; y += inc ) { integral += integrand (y) * inc; } printf ("Integral value = %.10f \n",integral); } /***************************************************************/ /** LEVEL 1 **/ /***************************************************************/ double integrand (y) double y; { double value; value = 2*y; if (value > 1e308) { printf ("Overflow error\n"); exit (0); } return (value); }

Node:Questions 21, Previous:Example 27, Up:Special Library Functions and Macros

Questions
1. 2. 3. 4. 5. What type of data is returned from mathematical functions? All calculations are performed using long variables. True or false? What information is returned by strlen()? What action is performed by strcat()? Name five kinds of error which can occur in a mathematical function.

Node:Hidden Operators, Next:More on Data Types, Previous:Special Library Functions and Macros, Up:Top

Hidden operators and values

Concise expressions Many operators in C are more versatile than they appear to be, at first glance. Take, for example, the following operators
= ++ -+= -= etc...

the assignment, increment and decrement operators... These innocent looking operators can be used in some surprising ways which make C source code very neat and compact. The first thing to notice is that ++ and -- are unary operators: that is, they are applied to a single variable and they affect that variable alone. They therefore produce one unique value each time they are used. The assignment operator, on the other hand, has the unusual position of being both unary, in the sense that it works out only one expression, and also binary or dyadic because it sits between two separate objects: an "lvalue" on the left hand side and an expression on the right hand side. Both kinds of operator have one thing in common however: both form statements which have values in their own right. What does this mean? It means that certain kinds of statement, in C, do not have to be thought of as being complete and sealed off from the rest of a program. To paraphrase a famous author: "In C, no statement is an island". A statement can be taken as a whole (as a "black box") and can be treated as a single value, which can be assigned and compared to things! The value of a statement is the result of the operation which was carried out in the statement. Increment/decrement operator statements, taken as a whole, have a value which is one greater / or one less than the value of the variable which they act upon. So:
c = 5; c++;

The second of these statement c++; has the value 6, and similarly:
c = 5; c--;

The second of these statements c--; has the value 4. Entire assignment statements have values too. A statement such as:
c = 5;

has the value which is the value of the assignment. So the example above has the value 5. This has some important implications.

Extended and Hidden =: Example 28: Hidden ++ --: Arrays Strings and Hidden Operators: Example 29: Cautions about Style: Example 30: Ques 21:

Node:Extended and Hidden =, Next:Example 28, Previous:Hidden Operators, Up:Hidden Operators

Extended and Hidden =

The idea that assignment statement has a value, can be used to make C programs neat and tidy for one simple reason: it means that a whole assignment statement can be used in place of a value. For instance, the value c = 0; could be assigned to a variable b:
b = (c = 0);

or simply:
b = c = 0;

These equivalent statements set b and c to the value zero, provided b and c are of the same type! It is equivalent to the more usual:
b = 0; c = 0;

Indeed, any number of these assignments can be strung together:

a = (b = (c = (d = (e = 5))))

or simply:
a = b = c = d = e = 5;

This very neat syntax compresses five lines of code into one single line! There are other uses for the valued assignment statement, of course: it can be used anywhere where a value can be used. For instance:

In other assignments (as above) As a parameter for functions Inside a comparison (== > < etc..) As an index for arrays....

The uses are manifold. Consider how an assignment statement might be used as a parameter to a function. The function below gets a character from the input stream stdin and passes it to a function called ProcessCharacter():
ProcessCharacter (ch = getchar());

This is a perfectly valid statement in C, because the hidden assignment statement passes on the value which it assigns. The actual order of events is that the assignment is carried out first and then the function is called. It would not make sense the other way around, because, then there would be no value to pass on as a parameter. So, in fact, this is a more compact way of writing:
ch = getchar(); ProcessCharacter (ch);

The two methods are entirely equivalent. If there is any doubt, examine a little more of this imaginary character processing program:
ProcessCharacter(ch = getchar()); if (ch == '*') { printf ("Starry, Starry Night..."); }

The purpose in adding the second statement is to impress the fact that ch has been assigned quite legitimately and it is still defined in the next statement and the one after...until it is re-assigned by a new assignment statement. The fact that the assignment was hidden inside another statement does not make it any less valid. All the same remarks apply about the specialized assignment operators +=, *=, /= etc.. Node:Example 28, Next:Hidden ++ --, Previous:Extended and Hidden =, Up:Hidden Operators

Example
/************************************************/ /* */ /* Hidden Assignment #1 */ /* */ /************************************************/ main () { do

{ switch (ch = getchar()) { default : putchar(ch); break; case 'Q' : /* Quit */ } } while (ch != 'Q'); } /* end */ /************************************************/ /* */ /* Hidden Assignment #2 */ /* */ /************************************************/ main () { double x = 0; while ((x += 0.2) < 20.0) { printf ("%lf",x);

} } /* end */

Node:Hidden ++ --, Next:Arrays Strings and Hidden Operators, Previous:Example 28, Up:Hidden Operators

Hidden ++ and -The increment and decrement operators also form statements which have intrinsic values and, like assignment expressions, they can be hidden away in inconspicuous places. These two operators are slightly more complicated than assignments because they exist in two forms: as a postfix and as a prefix:
Postfix var++ var-Prefix ++var --var

and these two forms have subtly different meanings. Look at the following example:
int i = 3; PrintNumber (i++);

The increment operator is hidden in the parameter list of the function PrintNumber(). This example is not as clear cut as the assignment statement examples however, because the variable i has, both a value before the ++ operator acts upon it, and a different value afterwards. The question is then: which value is passed to the function? Is i incremented before or after the function is called? The answer is that this is where the two forms of the operator come into play. If the operator is used as a prefix, the operation is performed before the function call. If the operator is used as a postfix, the operation is performed after the function call. In the example above, then, the value 3 is passed to the function and when the function returns, the value of i is incremented to 4. The alternative is to write:
int i = 3; PrintNumber (++i);

in which case the value 4 is passed to the function PrintNumber(). The same remarks apply to the decrement operator. Node:Arrays Strings and Hidden Operators, Next:Example 29, Previous:Hidden ++ --, Up:Hidden Operators

Arrays, Strings and Hidden Operators

Arrays and strings are one area of programming in which the increment and decrement operators are used a lot. Hiding operators inside array subscripts or hiding assignments inside loops can often make light work of tasks such as initialization of arrays. Consider the following example of a one dimensional array of integers.
#define SIZE 20

int i, array[SIZE]; for (i = 0; i < SIZE; array[i++] = 0) { }

This is a neat way of initializing an array to zero. Notice that the postfixed form of the increment operator is used. This prevents the element array[0] from assigning zero to memory which is out of the bounds of the array. Strings too can benefit from hidden operators. If the standard library function strlen() (which finds the length of a string) were not available, then it would be a simple matter to write the function
strlen (string) char *string; { char *ptr; int count = 0; for (ptr = string; *(ptr++) != '\0'; count++) { } return (count); } /* count the characters in a string */

This function increments count while the end of string marker \0 is not found. Node:Example 29, Next:Cautions about Style, Previous:Arrays Strings and Hidden Operators, Up:Hidden Operators

Example
/*********************************************************/ /* */ /* Hidden Operator Demo */ /* */ /*********************************************************/ /* Any assignment or increment operator has a value */ /* which can be handed straight to printf() ... */ /* Also compare the prefix / postfix forms of ++/-- */

#include <stdio.h> /*********************************************************/ main () { int a,b,c,d,e; a = (b = (c = (d = (e = 0)))); printf ("%d %d %d %d %d\n", a, b++, c--, d = 10, e += 3); a = b = c = d = e = 0; printf ("%d %d %d %d %d\n", a, ++b, --c, d = 10, e += 3); } /* end */ /*******************************************************/ /* */ /* Hidden Operator demo #2 */ /* */ /*******************************************************/ #include <stdio.h> /*******************************************************/ main () { printf ("%d",Value()); } /*******************************************************/ Value() { int value; if ((value = GetValue()) == 0) { printf ("Value was zero\n"); } return (value); } /********************************************************/ GetValue() { return (0); } /* end */ /* Some function to get a value */ /* Check for zero .... */ /* prints out zero! */

Node:Cautions about Style, Next:Example 30, Previous:Example 29, Up:Hidden Operators

Cautions about Style

Hiding operators away inside other statements can certainly make programs look very elegant and compact, but, as with all neat tricks, it can make programs harder to understand. Never forget that programming is communication to other programmers and be kind to the potential reader of a program. (It could be you in years or months to come!) Statements such as:
if ((i = (int)ch++) <= --comparison) { }

are not recommendable programming style and they are no more efficient than the more longwinded:
ch++; i = (int)ch; if (i <= comparison) { } comparison--;

There is always a happy medium in which to settle on a readable version of the code. The statement above might perhaps be written as:
i = (int) ch++; if (i <= --comparison) { }

Node:Example 30, Next:Ques 21, Previous:Cautions about Style, Up:Hidden Operators

Example
/******************************************************/ /* */ /* Arrays and Hidden Operators */ /* */ /******************************************************/ #include <stdio.h> #define SIZE 10 /******************************************************/ /* Level 0 */ /******************************************************/ main () /* Demo prefix and postfix ++ in arrays */

{ int i, array[SIZE];

Initialize(array); i = 4; array[i++] = 8; Print (array); Initialize(array); i = 4; array[++i] = 8; Print(array); } /*******************************************************/ /* Level 1 */ /*******************************************************/ Initialize (array) int array[SIZE]; { int i; for (i = 0; i < SIZE; array[i++] = 0) { } } /******************************************************/ Print (array) int array[SIZE]; { int i = 0; while (i < SIZE) { printf ("%2d",array[i++]); } putchar ('\n'); } /* end */ /****************************************************/ /* */ /* Hidden Operator */ /* */ /****************************************************/ #include <stdio.h> #define MAXNO 20 /* to stdout */ /* set to zero */

/*****************************************************/ main () { int i, ctr = 0; for (i = 1; ++ctr <= MAXNO; i = ctr*5) { /* Print out 5 x table */

printf ("%3d",i); }

Node:Ques 21, Previous:Example 30, Up:Hidden Operators

Questions
1. Which operators can be hidden inside other statements? 2. Give a reason why you would not want to do this in every possible case. 3. Hidden operators can be used in return statements .e.g
4. 5. return (++x);

Would there be any point in writing:

return (x++);

Node:More on Data Types, Next:Machine Level Operations, Previous:Hidden Operators, Up:Top

More on data types

This section is about the remaining data types which C has to offer programmers. Since C allows you to define new data types we shall not be able to cover all of the possiblities, only the most important examples. The most important of these are
FILE

The type which files are classified under

enum void

Enumerated type for abstract data The "empty" type

volatile const struct union

New ANSI standard type for memory mapped I/O New ANSI standard type for fixed data Groups of variables under a single name Multi-purpose storage areas for dynamical memory allocation

Special Constant Expressions: FILE: enum: Example 31: Example 32: Suggested uses for enum: void: volatile: const: struct again: union:

typedef: Questions 23:

Node:Special Constant Expressions, Next:FILE, Previous:More on Data Types, Up:More on Data Types

Special Constant Expressions

Constant expressions are often used without any thought, until a programmer needs to know how to do something special with them. It is worth making a brief remark about some special ways of writing integer constants, for the latter half of this book. Up to now the distinction between long and short integer types has largely been ignored. Constant values can be declared explicitly as long values, in fact, by placing the letter L after the constant.
long int variable = 23L; variable = 236526598L;

Advanced programmers, writing systems software, often find it convenient to work with hexadecimal or octal numbers since these number bases have a special relationship to binary. A constant in one of these types is declared by placing either 0 (zero) or 0x in front of the appropriate value. If ddd is a value, then:
Octal number Hexadecimal number 0ddd 0xddd

For example:
oct_value = 077; hex_value = 0xFFEF; /* 77 octal */ /* FFEF hex */

This kind of notation has already been applied to strings and single character constants with the backslash notation, instead of the leading zero character:
ch = '\ddd'; ch = '\xdd';

The values of character constants, like these, cannot be any greater than 255. Node:FILE, Next:enum, Previous:Special Constant Expressions, Up:More on Data Types
FILE

In all previous sections, the files stdin, stdout and stderr alone have been used in programs. These special files are always handled implicitly by functions like printf()

and scanf(): the programmer never gets to know that they are, in fact, files. Programs do not have to use these functions however: standard input/output files can be treated explicitly by general file handling functions just as well. Files are distinguished by filenames and by file pointers. File pointers are variables which pass the location of files to file handling functions; being variables, they have to be declared as being some data type. That type is called FILE and file pointers have to be declared "pointer to FILE". For example:
FILE *fp; FILE *fp = stdin; FILE *fopen();

File handling functions which return file pointers must also be declared as pointers to files. Notice that, in contrast to all the other reserved words FILE is written in upper case: the reason for this is that FILE is not a simple data type such as char or int, but a structure which is only defined by the header file stdio.h and so, strictly speaking, it is not a reserved word itself. We shall return to look more closely at files soon. Node:enum, Next:Example 31, Previous:FILE, Up:More on Data Types
enum

Abstract data are usually the realm of exclusively high level languages such as Pascal. enum is a way of incorporating limited "high level" data facilities into C. is short for enumerated data. The user defines a type of data which is made up of a fixed set of words, instead of numbers or characters. These words are given substitute integer numbers by the compiler which are used to identify and compare enum type data. For example:
enum enum countries { England, Scotland, Wales, Eire, Norge, Sverige, Danmark, Deutschland }; main () { enum countries variable; variable = England; }

Why go to all this trouble? The point about enumerated data is that they allow the programmer to forget about any numbers which the computer might need in order to

deal with a list of words, like the ones above, and simply concentrate on the logic of using them. Enumerated data are called abstract because the low level number form of the words is removed from the users attention. In fact, enumerated data are made up of integer constants, which the compiler generates itself. For this reason, they have a natural partner in programs: the switch statement. Here is an example, which uses the countries above to make a kind of airport "help computer" in age of electronic passports! Node:Example 31, Next:Example 32, Previous:enum, Up:More on Data Types

Example
/**********************************************************/ /* */ /* Enumerated Data */ /* */ /**********************************************************/ #include <stdio.h> enum countries { England, Ireland, Scotland, Wales, Danmark, Island, Norge, Sverige }; /**********************************************************/ main () /* Electronic Passport Program */

{ enum countries birthplace, getinfo(); printf ("Insert electronic passport\n"); birthplace = getinfo(); switch (birthplace) { case England : printf ("Welcome home!\n"); break; case Danmark : case Norge : printf ("Velkommen til England\n"); break; } } /************************************************************/ enum countries getinfo() { return (England); } /* interrogate passport */

/* end */

makes words into constant integer values for a programmer. Data which are declared enum are not the kind of data which it makes sense to do arithmetic with (even integer arithmetic), so in most cases it should not be necessary to know or even care about what numbers the compiler gives to the words in the list. However, some compilers allow the programmer to force particular values on words. The compiler then tries to give the values successive integer numbers unless the programmer states otherwise. For instance:
enum enum planets { Mercury, Venus, Earth = 12, Mars, Jupiter, Saturn, Uranus, Neptune, Pluto };

This would probably yield values Mercury = 0, Venus = 1, Earth = 12, Mars = 13, Jupiter = 14 ... etc. If the user tries to force a value which the compiler has already used then the compiler will complain. The following example program listing shows two points:

types can be local or global. The labels can be forced to have certain values
enum

Node:Example 32, Next:Suggested uses for enum, Previous:Example 31, Up:More on Data Types

Example
/**********************************************************/ /* */ /* Enumerated Data */ /* */ /**********************************************************/ /* The smallest adventure game in the world */ #include <stdio.h> #define TRUE 1 #define FALSE 0 enum treasures { rubies, sapphires, gold, silver, /* Adventure Treasures */

mask, scroll, lamp }; /***********************************************************/ /* Level 0 */ /***********************************************************/ main () { enum treasures object = gold; if (getobject(object)) { printf ("Congratulations you've found the gold!\n"); } else { printf ("Too bad -- you just missed your big chance"); } } /***********************************************************/ /* Level 1 */ /***********************************************************/ getobject (ob) enum treasures ob; { enum answer { no = false, yes = true }; if (ob == gold) { printf ("Pick up object? Y/N\n"); switch (getchar()) { case 'y' : case 'Y' : return ((int) yes); /* true and false */ default : return ((int) no); /* are integers */ } } else { printf ("You grapple with the dirt\n"); return (false); } } /* end */ /* yes or no ? */ /* Tiny Adventure! */

Node:Suggested uses for enum, Next:void, Previous:Example 32, Up:More on Data Types

Suggested uses for enum

Here are some suggested uses for enum.

enum numbers { zero, one, two, three }; enum animals { cat, dog, cow, sheep, }; enum plants { grass, roses, cabbages, oaktree }; enum diseases { heart, skin, malnutrition, circulatory }; enum quarks { up, down, charmed, strange, top, bottom, truth, beauty };

Other suggestions: colours, names of roads or types of train. Node:void, Next:volatile, Previous:Suggested uses for enum, Up:More on Data Types
void

is a peculiar data type which has some debatable uses. The void datatypes was introduced in order to make C syntactically consistent. The main idea of void is to be able to declare functions which have no return value. The word `void' is intended in the meaning `empty' rather than `invalid'. If you recall, the default is for C functions to return a value of type int. The value returned by a function did not have to be specified could always be discarded, so this was not a problem in practice. It did make
void

compiler checks more difficult however: how do you warn someone about inconsistent return values if it is legal to ignore return values? The ANSI solution was to introduce a new data type which was called void for functions with no value. The word void is perhaps an unfortunate choice, since it has several implicit meanings none of which really express what is intended. The words `novalue' or `notype' would have been better choices. A variable or function can be declared void in the following ways.
void function(); void variable; void *ptr; (void) returnvalue();

The following are true of void:

A variable which is declared void is useless: it cannot be used in an expression and it cannot be assigned to a value. The data type was introduced with functions in mind but the grammar of C allows us to define variables of this type also, even though there is no point. A function which is declared void has no return value and returns simply with:
return;

A function call can be cast (void) in order to explicitly discard a return value (though this is done by the compiler anyway). For instance, scanf() returns the number of items it matches in the control string, but this is usually discarded.
scanf ("%c",&ch);

or
(void) scanf("%c",&ch);

Few programmers would do this since it merely clutters up programs with irrelevant verbiage.

A void pointer can point to to any kind of object. This means that any pointer can be assigned to a void pointer, regardless of its type. This is also a highly questionable feature of the ANSI draft. It replaces the meaning of void from `no type or value' to `no particular type'. It allows assignments between incompatible pointer types without a cast operator. This is also rather dubious.

Node:volatile, Next:const, Previous:void, Up:More on Data Types

volatile

is a type which has been proposed in the ANSI standard. The idea behind this type is to allow memory mapped input/output to be held in C variables. Variables which are declared volatile will be able to have their values altered in ways which a
volatile

program does not explicitly define: that is, by external influences such as clocks, external ports, hardware, interrupts etc... The volatile datatype has found another use since the arrival of multiprocessor, multithreaded operating systems. Independent processes which share common memory could each change a variable independently. In other words, in a multithreaded environment the value of a variable set by one process in shared memory might be altered by another process without its knowledge. The keyword volatile servers as a warning to the compiler that any optimizing code it produces should not rely on caching the value of the variable, it should always reread its value. Node:const, Next:struct again, Previous:volatile, Up:More on Data Types
const

The reserved word const is used to declare data which can only be assigned once, either because they are in ROM (for example) or because they are data whose values must not be corrupted. Types declared const must be assigned when they are first initialized and they exist as stored values only at compile time:
const double pi = 3.14; const int one = 1;

Since a constant array only exists at compile time, it can be initialized by the compiler.
const int array[] = { 1, 2, 3, 4 };

then has the value 1, array[1] has the value 2 ... and so on. Any attempt to assign values to const types will result in compilation errors.
array[0]

It is worth comparing the const declaration to enumerated data, since they are connected in a very simple way. The following two sets of of statements are the same:
enum numbers { zero, one, two, three, four };

and
const zero = 0; const one = 1; const two = 2;

const three = 3; const four = 4;

Constant types and enumerated data are therefore just different aspects of the same thing. Enumerated data provide a convenient way of classifying constants, however, while the compiler keeps track of the values and types. With const you have to keep track of constant values personally. Node:struct again, Next:union, Previous:const, Up:More on Data Types
struct

Structures are called records in Pascal and many other languages. They are packages of variables which are all wrapped up under a single name. Structures are described in detail in chapter 25. Node:union, Next:typedef, Previous:struct again, Up:More on Data Types
union

Unions are often grouped together with structures, but they are quite unlike them in almost all respects. They are like general purpose storage containers, which can hold a variety of different variable types, at different times. The compiler makes a container which is large enough to take any of these, See Structures and Unions. Node:typedef, Next:Questions 23, Previous:union, Up:More on Data Types
typedef

C allows us to define our own data types or to rename existing ones by using a compiler directive called typedef. This statement is used as follows:
typedef type newtypename;

So, for example, we could define a type called byte, which was exactly one byte in size by redefining the word char:
typedef unsigned char byte;

The compiler type checking facilities then treat byte as a new type which can be used to declare variables:
byte variable, function();

The typedef statement may be written inside functions or in the global white space of a program.
/**************************************************/ /* Program */ /**************************************************/

typedef int newname1; main () { typedef char newname2; }

This program will compile and run (though it will not do very much). It is not very often that you want to rename existing types in the way shown above. The most important use for typedef is in conjunction with structures and unions. Structures and unions can, by their very definition, be all kinds of shape and size and their names can become long and tedious to declare. typedef makes dealing with these simple because it means that the user can define a structure or union with a simple typename. Node:Questions 23, Previous:typedef, Up:More on Data Types

Questions
1. Is FILE a reserved word? If so why is it in upper case? 2. Write a statement which declares a file pointer called fp. 3. Enumerated data are given values by the compiler so that it can do arithmetic with them. True or false? 4. Does void do anything which C cannot already do without this type? 5. What type might a timer device be declared if it were to be called by a variable name? 6. Write a statement which declares a new type "real" to be like the usual type "double". 7. Variables declared const can be of any type. True or false? Node:Machine Level Operations, Next:Files and Devices, Previous:More on Data Types, Up:Top

Machine Level Operations

Bits and Bytes. Flags/messages. Shifting. Down in the depths of your computer, below even the operating system are bits of memory. These days we are used to working at such a high level that it is easy to forget them. Bits (or binary digits) are the lowest level software objects in a computer: there is nothing more primitive. For precisely this reason, it is rare for high level languages to even acknowledge the existence of bits, let alone manipulate them. Manipulating bit patterns is usually the preserve of assembly language programmers. C, however, is quite different from most other high level languages in that it allows a programmer full access to bits and even provides high level operators for manipulating them.

Since this book is an introductory text, we shall treat bit operations only superficially. Many of the facilities which are available for bit operations need not concern the majority of programs at all. This section concerns the main uses of bit operations for high level programs and it assumes a certain amount of knowledge about programming at the low level. You may wish to consult a book on assembly language programming to learn about low level memory operations, in more detail.

Bit Patterns: Flags registers: Bit Operators and Assignments: Bit operators: Shift Operations: Truth Tables and Masking: Example 33: Output 33: Example 34: Example 35: Questions 24:

Node:Bit Patterns, Next:Flags registers, Previous:Machine Level Operations, Up:Machine Level Operations

Bit Patterns
All computer data, of any type, are bit patterns. The only difference between a string and a floating point variable is the way in which we choose to interpret the patterns of bits in a computer's memory. For the most part, it is quite unnecessary to think of computer data as bit patterns; systems programmers, on the other hand, frequently find that they need to handle bits directly in order to make efficient use of memory when using flags. A flag is a message which is either one thing or the other: in system terms, the flag is said to be `on' or `off' or alternatively set or cleared. The usual place to find flags is in a status register of a CPU (central processor unit) or in a pseudo-register (this is a status register for an imaginary processor, which is held in memory). A status register is a group of bits (a byte perhaps) in which each bit signifies something special. In an ordinary byte of data, bits are grouped together and are interpreted to have a collective meaning; in a status register they are thought of as being independent. Programmers are interested to know about the contents of bits in these registers, perhaps to find out what happened in a program after some special operation is carried out. Other uses for bit patterns are listed below here:

Messages sent between devices in a complex operating environment use bits for efficiency. Serially transmitted data. Handling bit-planes in screen memory. (Raster ports and devices) Performing fast arithmetic in simple cases.

Programmers who are interested in performing bit operations often work in hexadecimal because every hexadecimal digit conveniently handles four bits in one go (16 is 2 to the power 4).

Node:Flags registers, Next:Bit Operators and Assignments, Previous:Bit Patterns, Up:Machine Level Operations

Flags, Registers and Messages

A register is a place inside a computer processor chip, where data are worked upon in some way. A status register is a register which is used to return information to a programmer about the operations which took place in other registers. Status registers contain flags which give yes or no answers to questions concerning the other registers. In advanced programming, there may be call for "pseudo registers" in addition to "real" ones. A pseudo register is merely a register which is created by the programmer in computer memory (it does not exist inside a processor). Messages are just like pseudo status registers: they are collections of flags which signal special information between different devices and/or different programs in a computer system. Messages do not necessarily have fixed locations: they may be passed a parameters. Messages are a very compact way of passing information to low level functions in a program. Flags, registers, pseudo-registers and messages are all treated as bit patterns. A program which makes use of them must therefore be able to assign these objects to C variables for use. A bit pattern would normally be declared as a character or some kind of integer type in C, perhaps with the aid of a typedef statement.
typedef char byte; typedef int bitpattern; bitpattern variable; byte message;

The flags or bits in a register/message... have the values 1 or 0, depending upon whether they are on or off (set or cleared). A program can test for this by using combinations of the operators which C provides. Node:Bit Operators and Assignments, Next:Bit operators, Previous:Flags registers, Up:Machine Level Operations

Bit Operators and Assignments

C provides the following operators for handling bit patterns:
<<

Bit shift left (a specified number or bit positions)

>> | ^ &

Bit shift right(a specified number of bit positions) Bitwise Inclusive OR Bitwise Exclusive OR Bitwise AND

Bitwise one's complement

&= |= ^= >>= <<=

AND assign (variable = variable & value) Exclusive OR assign (variable = variable | value) Inclusive OR assign (variable = variable ^ value) Shift right assign (variable = variable >> value) Shift left assign (variable = variable << value)

The meaning and the syntax of these operators is given below. Node:Bit operators, Next:Shift Operations, Previous:Bit Operators and Assignments, Up:Machine Level Operations

The Meaning of Bit Operators

Bitwise operations are not to be confused with logical operations (&&, ||...) A bit pattern is made up of 0s and 1s and bitwise operators operate individually upon each bit in the operand. Every 0 or 1 undergoes the operations individually. Bitwise operators (AND, OR) can be used in place of logical operators (&&,||), but they are less efficient, because logical operators are designed to reduce the number of comparisons made, in an expression, to the optimum: as soon as the truth or falsity of an expression is known, a logical comparison operator quits. A bitwise operator would continue operating to the last before the final result were known. Below is a brief summary of the operations which are performed by the above operators on the bits of their operands. Node:Shift Operations, Next:Truth Tables and Masking, Previous:Bit operators, Up:Machine Level Operations

Shift Operations
Imagine a bit pattern as being represented by the following group of boxes. Every box represents a bit; the numbers inside represent their values. The values written over the top are the common integer values which the whole group of bits would have, if they were interpreted collectively as an integer.
128 64 32 16 8 4 2 1 = 1

------------------------------| 0 | 0 | 0 | 0 | 0 | 0 | 0 | 1 | -------------------------------

Shift operators move whole bit patterns left or right by shunting them between boxes. The syntax of this operation is:
value << number of positions

value >> number of positions

So for example, using the boxed value (1) above:

1 << 1

would have the value 2, because the bit pattern would have been moved one place the the left:
128 64 32 16 8 4 2 1 = 2

------------------------------| 0 | 0 | 0 | 0 | 0 | 0 | 1 | 0 | -------------------------------

Similarly:
1 << 4

has the value 16 because the original bit pattern is moved by four places:
128 64 32 16 8 4 2 1 = 16

------------------------------| 0 | 0 | 0 | 1 | 0 | 0 | 0 | 0 | -------------------------------

And:
6 << 2 == 12 128 64 32 16 8 4 2 1 = 6

------------------------------| 0 | 0 | 0 | 0 | 0 | 1 | 1 | 0 | -------------------------------

Shift left 2 places:

128 64 32 16 8 4 2 1 = 12

------------------------------| 0 | 0 | 0 | 0 | 1 | 1 | 0 | 0 | -------------------------------

Notice that every shift left multiplies by 2 and that every shift right would divide by two, integerwise. If a bit reaches the edge of the group of boxes then it falls out and is lost forever. So:
1 >> 1 == 0 2 >> 1 == 1 2 >> 2 == 0

n >> n == 0

A common use of shifting is to scan through the bits of a bitpattern one by one in a loop: this is done by using masks. Node:Truth Tables and Masking, Next:Example 33, Previous:Shift Operations, Up:Machine Level Operations

Truth Tables and Masking

The operations AND, OR (inclusive OR) and XOR/EOR (exclusive OR) perform comparisons or "masking" operations between two bits. They are binary or dyadic operators. Another operation called COMPLEMENT is a unary operator. The operations performed by these bitwise operators are best summarized by truth tables. Truth tables indicate what the results of all possible operations are between two single bits. The same operation is then carried out for all the bits in the variables which are operated upon.

Complement ~: AND: OR: XOR:

Node:Complement ~, Next:AND, Previous:Truth Tables and Masking, Up:Truth Tables and Masking

Complement ~
The complement of a number is the logical opposite of the number. C provides a "one's complement" operator which simply changes all 1s into 0s and all 0s into 1s.
~1 has the value 0 ~0 has the value 1 (for each bit)

As a truth table this would be summarized as follows:

~value 0 1 == 1 0 result

Node:AND, Next:OR, Previous:Complement ~, Up:Truth Tables and Masking

AND &
This works between two values. e.g. (1 & 0)
value 1 0 0 & value 2 0 1 == result 0 0

1 1

0 1

0 1

Both value 1 AND value 2 have to be 1 in order for the result or be 1. Node:OR, Next:XOR, Previous:AND, Up:Truth Tables and Masking

OR |
This works between two values. e.g. (1 | 0)
value 1 0 0 1 1 | value 2 0 1 0 1 == result 0 1 1 1

The result is 1 if one OR the other OR both of the values is 1. Node:XOR, Previous:OR, Up:Truth Tables and Masking

XOR/EOR ^
Operates on two values. e.g. (1 ^ 0)
value 1 0 0 1 1 ^ value 2 0 1 0 1 == result 0 1 1 0

The result is 1 if one OR the other (but not both) of the values is 1. Bit patterns and logic operators are often used to make masks. A mask is as a thing which fits over a bit pattern and modifies the result in order perhaps to single out particular bits, usually to cover up part of a bit pattern. This is particularly pertinent for handling flags, where a programmer wishes to know if one particular flag is set or not set and does not care about the values of the others. This is done by deliberately inventing a value which only allows the particular flag of interest to have a non-zero value and then ANDing that value with the flag register. For example: in symbolic language:
MASK = 00000001 VALUE1 = 10011011 VALUE2 = 10011100 MASK & VALUE1 == 00000001 MASK & VALUE2 == 00000000

The zeros in the mask masks off the first seven bits and leave only the last one to reveal its true value. Alternatively, masks can be built up by specifying several flags:

FLAG1 = 00000001 FLAG2 = 00000010 FLAG3 = 00000100 MESSAGE = FLAG1 | FLAG2 | FLAG3 MESSAGE == 00000111

It should be emphasized that these expressions are only written in symbolic language: it is not possible to use binary values in C. The programmer must convert to hexadecimal, octal or denary first. (See the appendices for conversion tables). Node:Example 33, Next:Output 33, Previous:Truth Tables and Masking, Up:Machine Level Operations

Example
A simple example helps to show how logical masks and shift operations can be combined. The first program gets a denary number from the user and converts it into binary. The second program gets a value from the user in binary and converts it into hexadecimal.
/***************************************************/ /* */ /* Bit Manipulation #1 */ /* */ /***************************************************/ /* /* /* /* Convert denary numbers into binary */ Keep shifting i by one to the left */ and test the highest bit. This does*/ NOT preserve the value of i */ 8

#include <stdio.h> #define NUMBEROFBITS

/****************************************************/ main () { short i,j,bit,; short MASK = 0x80; printf ("Enter any number less than 128: "); scanf ("%h", &i); if (i > 128) { printf ("Too big\n"); return (0); } printf ("Binary value = "); for (j = 0; j < NUMBEROFBITS; j++) { bit = i & MASK; printf ("%1d",bit/MASK);

i <<= 1; } printf ("\n"); } /* end */

Node:Output 33, Next:Example 34, Previous:Example 33, Up:Machine Level Operations

Output
Enter any number less than 128: 56 Binary value = 00111000 Enter any value less than 128: 3 Binary value = 00000011

Node:Example 34, Next:Example 35, Previous:Output 33, Up:Machine Level Operations

Example
/***************************************************/ /* */ /* Bit Manipulation #2 */ /* */ /***************************************************/ /* Convert binary numbers into hex #include <stdio.h> #define NUMBEROFBITS 8 /****************************************************/ main () { short j,hex = 0; short MASK; char binary[NUMBEROFBITS]; printf ("Enter an 8-bit binary number: "); for (j = 0; j < NUMBEROFBITS; j++) { binary[j] = getchar(); } for (j = 0; j < NUMBEROFBITS; j++) { hex <<= 1; switch (binary[j]) { case '1' : MASK = 1; break; case '0' : MASK = 0; break; default : printf("Not binary\n"); */

return(0); hex |= MASK; } printf ("Hex value = %1x\n",hex); } /* end */ }

Node:Example 35, Next:Questions 24, Previous:Example 34, Up:Machine Level Operations

Example
Enter any number less than 128: 56 Binary value = 00111000 Enter any value less than 128: 3 Binary value = 00000011

Node:Questions 24, Previous:Example 35, Up:Machine Level Operations

Questions
1. What distinguishes a bit pattern from an ordinary variable? Can any variable be a bit pattern? 2. What is the difference between an inclusive OR operation and an exclusive OR operation? 3. If you saw the following function call in a program, could you guess what its parameter was?
4. OpenWindow (BORDER | GADGETS | MOUSECONTROL | SIZING); 5.

6. Find out what the denary (decimal) values of the following operations are: 1. 7 & 2 2. 1 & 1 3. 15 & 3 4. 15 & 7 5. 15 & 7 & 3 Try to explain the results. (Hint: draw out the numbers as binary patterns, using the program listed.) 7. Find out what the denary (decimal) values of the following operations are: 1. 1 | 2 2. 1 | 2 | 3 8. Find out the values of: 1. 1 & (~1) 2. 23 & (~23) 3. 2012 & (~2012) (Hint: write a short program to work them out. Use short type variables for all the numbers).

Node:Files and Devices, Next:Structures and Unions, Previous:Machine Level Operations, Up:Top

Files and Devices

Files are places for reading data from or writing data to. This includes disk files and it includes devices such as the printer or the monitor of a computer. C treats all information which enters or leaves a program as though it were a stream of bytes: a file. The most commonly used file streams are stdin (the keyboard) and stdout (the screen), but more sophisticated programs need to be able to read or write to files which are found on a disk or to the printer etc. An operating system allows a program to see files in the outside world by providing a number of channels or `portals' (`inlets' and `outlets') to work through. In order to examine the contents of a file or to write information to a file, a program has to open one of these portals. The reason for this slightly indirect method of working is that channels/portals hide operating system dependent details of filing from the programmer. Think of it as a protocol. A program which writes information does no more than pass that information to one of these portals and the operating system's filing subsystem does the rest. A program which reads data simply reads values from its file portal and does not have to worry about how they got there. This is extremely simple to work in practice. To use a file then, a program has to go through the following routine:

Open a file for reading or writing. (Reserve a portal and locate the file on disk or whatever.) Read or write to the file using file handling functions provided by the standard library. Close the file to free the operating system "portal" for use by another program or file.

A program opens a file by calling a standard library function and is returned a file pointer, by the operating system, which allows a program to address that particular file and to distinguish it from all others.

Files Generally: File Positions: High Level File Handling Functions: Opening files: Closing a file: fprintf: fscanf: skipfilegarb?: Single Character I/O: getc and fgetc: ungetc: putc and fputc: fgets and fputs: feof: Printer Output: Example 36:

Output 36: Converting example: File Errors: Other Facilities for High Level Files: fread() and fwrite(): ftell and fseek: rewind: fflush: Low Level Filing Operations: File Handles: open: close: creat: read: write: lseek: unlink remove: Example 37: Questions 25:

Node:Files Generally, Next:File Positions, Previous:Files and Devices, Up:Files and Devices

Files Generally
C provides two levels of file handling; these can be called high level and low level. High level files are all treated as text files. In fact, the data which go into the files are exactly what would be seen on the screen, character by character, except that they are stored in a file instead. This is true whether a file is meant to store characters, integers, floating point types. Any file, which is written to by high level file handling functions,

ends up as a text file which could be edited by a text editor.

High level text files are also read back as character files, in the same way that input is acquired from the keyboard. This all means that high level file functions are identical in concept to keyboard/screen input/output. The alternative to these high level functions, is obviously low level functions. These are more efficient, in principle, at filing data as they can store data in large lumps, in raw memory format, without converting to text files first. Low level input/output functions have the disadvantage that they are less `programmer friendly' than the high level ones, but they are likely to work faster.

File Positions: High Level File Handling Functions: Opening files: Closing a file: fprintf: fscanf: skipfilegarb?: Single Character I/O: getc and fgetc: ungetc: putc and fputc: fgets and fputs: feof:

Converting example: File Errors: Other Facilities for High Level Files: fread() and fwrite(): ftell and fseek: rewind: fflush: Low Level Filing Operations: File Handles: open: close: creat: read: write: lseek: unlink remove:

Node:File Positions, Next:High Level File Handling Functions, Previous:Files Generally, Up:Files and Devices

File Positions
When data are read from a file, the operating system keeps track of the current position of a program within that file so that it only needs to make a standard library call to `read the next part of the file' and the operating system obliges by reading some more and advancing its position within the file, until it reaches the end. Each single character which is read causes the position in a file to be advanced by one. Although the operating system does a great deal of hand holding regarding file positions, a program can control the way in which that position changes with functions such as ungetc() if need be. In most cases it is not necessary and it should be avoided, since complex movements within a file can cause complex movements of a disk drive mechanism which in turn can lead to wear on disks and the occurrence of errors. Node:High Level File Handling Functions, Next:Opening files, Previous:File Positions, Up:Files and Devices

High Level File Handling Functions

Most of the high level input/output functions which deal with files are easily recognizable in that they start with the letter `f'. Some of these functions will appear strikingly familiar. For instance:
fprintf() fscanf() fgets() fputs()

These are all generalized file handling versions of the standard input/output library. They work with generalized files, as opposed to the specific files stdin and stdout

which printf() and scanf() use. The file versions differ only in that they need an extra piece of information: the file pointer to a particular portal. This is passed as an extra parameter to the functions. they process data in an identical way to their standard I/O counterparts. Other filing functions will not look so familiar. For example:
fopen() fclose() getc() ungetc(); putc() fgetc() fputc() feof()

Before any work can be done with high level files, these functions need to be explained in some detail. Node:Opening files, Next:Closing a file, Previous:High Level File Handling Functions, Up:Files and Devices

Opening files
A file is opened by a call to the library function fopen(): this is available automatically when the library file <stdio.h> is included. There are two stages to opening a file: firstly a file portal must be found so that a program can access information from a file at all. Secondly the file must be physically located on a disk or as a device or whatever. The fopen() function performs both of these services and, if, in fact, the file it attempts to open does not exist, that file is created anew. The syntax of the fopen() function is:
FILE *returnpointer; returnpointer = fopen("filename","mode");

or
FILE returnpointer; char *fname, *mode; returnpointer = fopen(fname,mode);

The filename is a string which provides the name of the file to be opened. Filenames are system dependent so the details of this must be sought from the local operating system manual. The operation mode is also a string, chosen from one of the following:
r w a rw

Open file for reading Open file for writing Open file for appending

Open file for reading and writing (some systems) This mode string specifies the way in which the file will be used. Finally, returnpointer is a pointer to a FILE structure which is the whole object of calling this function. If the file (which was named) opened successfully when fopen() was called, returnpointer is a pointer to the file portal. If the file could not be opened, this pointer is set to the value NULL. This should be tested for, because it would not make sense to attempt to write to a file which could not be opened or created, for whatever reason. A read only file is opened, for example, with some program code such as:
FILE *fp; if ((fp = fopen ("filename","r")) == NULL) { printf ("File could not be opened\n"); error_handler(); }

A question which springs to mind is: what happens if the user has to type in the name of a file while the program is running? The solution to this problem is quite simple. Recall the function filename() which was written in chapter 20.
char *filename() /* return filename */

{ static char *filenm = "........................"; do { printf ("Enter filename :"); scanf ("%24s",filenm); skipgarb(); } while (strlen(filenm) == 0); return (filenm); }

This function makes file opening simple. The programmer would now write something like:
FILE *fp; char *filename(); if ((fp = fopen (filename(),"r")) == NULL) { printf ("File could not be opened\n"); error_handler(); }

and then the user of the program would automatically be prompted for a filename. Once a file has been opened, it can be read from or written to using the other library

functions (such as fprintf() and fscanf()) and then finally the file has to be closed again. Node:Closing a file, Next:fprintf, Previous:Opening files, Up:Files and Devices

Closing a file
A file is closed by calling the function fclose(). fclose() has the syntax:
int returncode; FILE *fp; returncode = fclose (fp);

is a pointer to the file which is to be closed and returncode is an integer value which is 0 if the file was closed successfully. fclose() prompts the file manager to finish off its dealings with the named file and to close the portal which the operating system reserved for it. When closing a file, a program needs to do something like the following:
fp if (fclose(fp) != 0) { printf ("File did not exist.\n"); error_handler(); }

Node:fprintf, Next:fscanf, Previous:Closing a file, Up:Files and Devices

fprintf()

This is the highest level function which writes to files. Its name is meant to signify "file-print-formatted" and it is almost identical to its stdout counterpart printf(). The form of the fprintf() statement is as follows:
fprintf (fp,"string",variables);

where fp is a file pointer, string is a control string which is to be formatted and the variables are those which are to be substituted into the blank fields of the format string. For example, assume that there is an open file, pointed to by fp:
int i = 12; float x = 2.356; char ch = 's'; fprintf (fp, "%d %f %c", i, x, ch);

The conversion specifiers are identical to those for printf(). In fact fprintf() is related to printf() in a very simple way: the following two statements are identical.
printf ("Hello world %d", 1);

fprintf (stdout,"Hello world %d", 1);

Node:fscanf, Next:skipfilegarb?, Previous:fprintf, Up:Files and Devices

fscanf()

The analogue of scanf() is fscanf() and, as with fprintf(), this function differs from its standard I/O counterpart only in one extra parameter: a file pointer. The form of an fscanf() statement is:
FILE *fp; int n; n = fscanf (fp,"string",pointers);

where n is the number of items matched in the control string and fp is a pointer to the file which is to be read from. For example, assuming that fp is a pointer to an open file:
int i = 10; float x = -2.356; char ch = 'x'; fscanf (fp, "%d %f %c", &i, &x, &ch);

The remarks which were made about scanf() also apply to this function: fscanf() is a `dangerous' function in that it can easily get out of step with the input data unless the input is properly formatted. Node:skipfilegarb?, Next:Single Character I/O, Previous:fscanf, Up:Files and Devices

skipfilegarb() ?
Do programs need a function such as skipgarb() to deal with instances of badly formatted input data? A programmer can assume a bit more about files which are read into a program from disk file than it can assume about the user's typed input. A disk file will presumably have been produced by the same program which generated it, or will be in a format which the program expects. Is a function like skipgarb() necessary then? The answer is: probably not. This does not mean to say that a program does not need to check for "bad files", or files which do not contain the data they are alleged to contain. On the other hand, a programmer is at liberty to assume that any file which does not contain correctly formatted data is just nonsense: he/she does not have to try to make sense of it with a function like skipgarb(), the program could simply return an error message like "BAD FILE" or whatever and recover in a sensible way. It would probably not make sense to use a function like skipgarb() for files. For comparison alone, skipfilegarb() is written below.
skipfilegarb(fp) FILE *fp; {

while (getc(fp) != '\n') { } }

Node:Single Character I/O, Next:getc and fgetc, Previous:skipfilegarb?, Up:Files and Devices

Single Character I/O

There are commonly four functions/macros which perform single character input/output to or from files. They are analogous to the functions/macros
getchar() putchar()

for the standard I/O files and they are called:

getc() ungetc(); putc() fgetc() fputc()

Node:getc and fgetc, Next:ungetc, Previous:Single Character I/O, Up:Files and Devices
getc()

and fgetc()

The difference between getc() and fgetc() will depend upon a particular system. It might be that getc() is implemented as a macro, whereas fgetc() is implemented as a function or vice versa. One of these alternatives may not be present at all in a library. Check the manual, to be sure! Both getc() and fgetc() fetch a single character from a file:
FILE *fp; char ch; /* open file */ ch = getc (fp); ch = fgetc (fp);

These functions return a character from the specified file if they operated successfully, otherwise they return EOF to indicate the end of a file or some other error. Apart from this, these functions/macros are quite unremarkable. Node:ungetc, Next:putc and fputc, Previous:getc and fgetc, Up:Files and Devices
ungetc()

is a function which `un-gets' a character from a file. That is, it reverses the effect of the last get operation. This is not like writing to a file, but it is like stepping back one position within the file. The purpose of this function is to leave the input in the correct place for other functions in a program when other functions go too far in a file. An example of this would be a program which looks for a word in a text file and processes that word in some way.
ungetc() while (getc(fp) != ' ') { }

The program would skip over spaces until it found a character and then it would know that this was the start of a word. However, having used getc() to read the first character of that word, the position in the file would be the second character in the word! This means that, if another function wanted to read that word from the beginning, the position in the file would not be correct, because the first character would already have been read. The solution is to use ungetc() to move the file position back a character:
int returncode; returncode = ungetc(fp);

The returncode is EOF if the operation was unsuccessful. Node:putc and fputc, Next:fgets and fputs, Previous:ungetc, Up:Files and Devices
putc()

and fputc()

These two functions write a single character to the output file, pointed to by fp. As with getc(), one of these may be a macro. The form of these statements is:
FILE *fp; char ch; int returncode; returncode = fputc (ch,fp); returncode = putc (ch,fp);

The returncode is the ascii code of the character sent, if the operation was successful, otherwise it is EOF. Node:fgets and fputs, Next:feof, Previous:putc and fputc, Up:Files and Devices
fgets()

and fputs()

Just as gets() and puts() fetched and sent strings to standard input/output files stdin and stdout, so fgets() and fputs() send strings to generalized files. The form of an fgets() statement is as follows:
char *strbuff,*returnval; int n; FILE *fp;

returnval = fgets (strbuff,n,fp);

is a pointer to an input buffer for the string; fp is a pointer to an open file. returnval is a pointer to a string: if there was an error in fgets() this pointer is set to the value NULL, otherwise it is set to the value of "strbuff". No more than (n-1) characters are read by fgets() so the programmer has to be sure to set n equal to the size of the string buffer. (One byte is reserved for the NULL terminator.) The form of an fputs() statement is as follows:
strbuff char *str; int returnval; FILE *fp; returnval = fputs (str,fp);

Where str is the NULL terminated string which is to be sent to the file pointed to by fp. returnval is set to EOF if there was an error in writing to the file. Node:feof, Next:Printer Output, Previous:fgets and fputs, Up:Files and Devices

feof()
This function returns a true or false result. It tests whether or not the end of a file has been reached and if it has it returns `true' (which has any value except zero); otherwise the function returns `false' (which has the value zero). The form of a statement using this function is:
FILE *fp; int outcome; outcome = feof(fp);

Most often feof() will be used inside loops or conditional statements. For example: consider a loop which reads characters from an open file, pointed to by fp. A call to feof() is required in order to check for the end of the file.
while (!feof(fp)) { ch = getc(fp); }

Translated into pidgin English, this code reads: `while NOT end of file, ch equals get character from file'. In better(?) English the loop continues to fetch characters as long as the end of the file has not been reached. Notice the logical NOT operator ! which stands before feof(). Node:Printer Output, Next:Example 36, Previous:feof, Up:Files and Devices

Printer Output

Any serious application program will have to be in full control of the output of a program. For instance, it may need to redirect output to the printer so that data can be made into hard copies. To do this, one of three things must be undertaken:
stdout

must be redirected so that it sends data to the printer device.

A new "standard file" must be used (not all C compilers use this method.) A new file must be opened in order to write to the printer device The first method is not generally satisfactory for applications programs, because the standard files stdin and stdout can only easily be redirected from the operating system command line interpreter (when a program is run by typing its name). Examples of this are:
type file > PRN

which send a text file to the printer device. The second method is reserved for only a few implementations of C in which another `standard file' is opened by the local operating system and is available for sending data to the printer stream. This file might be called "stdprn" or "standard printer file" and data could be written to the printer by switching writing to the file like this:
fprintf (stdprn,"string %d...", integer);

The final method of writing to the printer is to open a file to the printer, personally. To do this, a program has to give the "filename" of the printer device. This could be something like "PRT:" or "PRN" or "LPRT" or whatever. The filename (actually called a pseudo device name) is used to open a file in precisely the same way as any other file is opened: by using a call to fopen(). fopen() then returns a pointer to file (which is effectively "stdprn") and this is used to write data to a computer's printer driver. The program code to do this should look something like the following:
FILE *stdprn; if ((stdprn = fopen("PRT:","w")) == NULL) { printf ("Printer busy or disconnected\n"); error_handler; }

Node:Example 36, Next:Output 36, Previous:Printer Output, Up:Files and Devices

Example
Here is an example program which reads a source file (for a program, written in C, Pascal or whatever...) and lists it, along with its line numbers. This kind of program is useful for debugging programs. The program provides the user with the option of sending the output to the printer. The printer device is assumed to have the filename "PRT:". Details of how to convert the program for other systems is given at the end.
/***************************************************************/ /* */

/* LIST : program file utility */ /* */ /***************************************************************/ /* List a source file with line numbers attached. Like */ /* TYPE only with lines numbers too. */ #include <stdio.h> #define #define #define #define #define #define CODE SIZE ON OFF TRUE FALSE 0 255 1 0 1 0 /* where output goes to */

FILE *fin; FILE *fout = stdout;

/***************************************************************/ /* Level 0 */ /***************************************************************/ main () { char strbuff[size],*filename(); int Pon = false; int line = 1; printf ("Source Program Lister V1.0\n\n"); if ((fin = fopen(filename(),"r")) == NULL) { printf ("\nFile not found\n"); exit (CODE); } printf ("Output to printer? Y/N"); if (yes()) { Pon = Printer(ON); } while (!feof(fin)) { if (fgets(strbuff,size,fin) != strbuff) { if (!feof(fin)) { printf ("Source file corrupted\n"); exit (CODE); } } fprintf (fout,"%4d %s",line++,strbuff); } CloseFiles(Pon); } /*************************************************************/ /* Level 1 */

/*************************************************************/ CloseFiles(Pon) int Pon; { if (Pon) { Printer(OFF); } if (fclose(fin) != 0) { printf ("Error closing input file\n"); } } /***********************************************************/ Printer (status) int status; { switch (status) { case on: while ((fout = fopen("PRT:","w")) == NULL) { printf ("Printer busy or disconnected\n"); printf ("\n\nRetry? Y/N\n"); if (!yes()) { exit(CODE); } } break; case off: while (fclose(fout) != 0) { printf ("Waiting to close printer stream\r"); } /* switch printer file */ /* close & tidy */

} }

/***********************************************************/ /* Toolkit */ /***********************************************************/ char *filename() /* return filename */

{ static char *filenm = "........................"; do { printf ("Enter filename :"); scanf ("%24s",filenm); skipgarb(); } while (strlen(filenm) == 0);

return (filenm); } /*************************************************************/ yes () { char ch; while (TRUE) { ch = getchar(); skipgarb(); switch (ch) { case 'y' : case 'Y' : return (TRUE); case 'n' : case 'N' : return (FALSE); } } /* Get a yes/no response from the user */

/*************************************************************/ skipgarb() { while (getchar() != '\n') { } } /* end */ /* skip garbage corrupting input */

Node:Output 36, Next:Converting example, Previous:Example 36, Up:Files and Devices

Output
Here is a sample portion of the output of this program as applied to one of the example programs in section 30.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 /********************************************************/ /* */ /* C programming utility : variable referencer */ /* */ /********************************************************/ /* See section 30 */ #include <stdio.h> #include <ctype.h> #define #define #define #define #define TRUE 1 FALSE 0 DUMMY 0 MAXSTR 512 MAXIDSIZE 32

... and more of the same.

Node:Converting example, Next:File Errors, Previous:Output 36, Up:Files and Devices

Converting the example

The example program could be altered to work with a standard printer file "stdprn" by changing the following function.
Printer (status) int status; { switch (status) { case on: fout = stdprn; break; case off: } fout = stdout; /* switch printer file */

Node:File Errors, Next:Other Facilities for High Level Files, Previous:Converting example, Up:Files and Devices

Filing Errors
The standard library provides an error function/macro which returns a true/false result according to whether or not the last filing function call returned an error condition. This is called ferror(). To check for an error in an open file, pointed to by fp:
FILE *fp; if (ferror(fp)) { error_handler(); }

This function/macro does not shed any light upon the cause of errors, only whether errors have occurred at all. A detailed diagnosis of what went wrong is only generally possible by means of a deeper level call to the disk operating system (DOS). Node:Other Facilities for High Level Files, Next:fread() and fwrite(), Previous:File Errors, Up:Files and Devices

Other Facilities for High Level Files

Files which have been opened by fopen() can also be handled with the following additional functions:
fread() fwrite() ftell() fseek() rewind()

fflush()

These functions provide facilities to read and write whole blocks of characters in one operation as well as further facilities to locate and alter the current focus of attention within a file. They offer, essentially, low level filing operations for files which have been opened for high level use! Node:fread() and fwrite(), Next:ftell and fseek, Previous:Other Facilities for High Level Files, Up:Files and Devices
fread()

and fwrite()

These functions read and write whole blocks of characters at a time. The form of fread() is as follows:
FILE *fp; int noread,n,size; char *ptr; noread = fread (ptr,size,n,fp);

The parameters in parentheses provide information about where the data will be stored once they have been read from a file. fp is a pointer to an open file; ptr is a pointer to the start of a block of memory which is to store the data when it is read; size is the size of a block of data in characters; n is the number of blocks of data to be read. Finally noread is a return value which indicates the number of blocks which was actually read during the operation. It is important to check that the number of blocks expected is the same as the number received because something could have gone wrong with the reading process. (The disk might be corrupted or the file might have been altered in some way.) fwrite() has an identical call structure to fread():
FILE *fp; int nowritten,n,size; char *ptr; nowritten = fread (ptr,size,n,fp);

This time the parameters in parentheses provide information about where the data, to be written to a file, will be found. fp is a pointer to an open file; ptr is a pointer to the start of a block of memory at which the data are stored; size is the size of a "block" of data in characters; n is the number of blocks of data to be read; nowritten is a return value which indicates the actual number of blocks which was written. Again, this should be checked. A caution about these functions: each of these block transfer routines makes an important assumption about the way in which data are stored in the computer system. It is assumed that the data are stored contiguously in the memory, that is, side by side, in sequential memory locations. In some systems this can be difficult to arrange (in multi-tasking systems in particular) and almost impossible to guarantee. Memory which is allocated in C programs by the function malloc() does not guarantee to find

contiguous portions of memory on successive calls. This should be noted carefully when developing programs which use these calls. Node:ftell and fseek, Next:rewind, Previous:fread() and fwrite(), Up:Files and Devices

File Positions: ftell() and fseek()

tells a program its position within a file, opened by fopen(). fseek() seeks a specified place within a file, opened by fopen(). Normally high level read/write functions perform as much management over positions inside files as the programmer wants, but in the event of their being insufficient, these two routines can be used. The form of the function calls is:
ftell() long int pos; FILE *fp; pos = ftell(fp);

is an open file, which is in some state of being read or written to. pos is a long integer value which describes the position in terms of the number of characters from the beginning of the file. Aligning a file portal with a particular place in a file is more sophisticated than simply taking note of the current position. The call to fseek() looks like this:
fp long int pos; int mode,returncode; FILE *fp;

returncode = fseek (fp,pos,mode);

The parameters have the following meanings. fp is a pointer to a file opened by fopen(). pos is some way of describing the position required within a file. mode is an integer which specifies the way in which pos is to be interpreted. Finally, returncode is an integer whose value is 0 if the operation was successful and -1 if there was an error.
0 pos 1 2 pos pos

is an offset measured relative to the beginning of the file. is an offset measured relative to the current position.

is an offset measured relative to the end of the file. Some examples help to show how this works in practice:
long int pos = 50; int mode = 0,returncode; FILE *fp; if (fseek (fp,pos,mode) != 0) /* find 50th character */ { printf("Error!\n"); } fseek(fp,0L,0); fseek(fp,2L,0); if (fseek (fp,10L,1) != 0) /* find beginning of file */ /* find the end of a file */ /* move 10 char's forward */

{ printf("Error!\n"); }

The L's indicate long constants. Node:rewind, Next:fflush, Previous:ftell and fseek, Up:Files and Devices
rewind()

is a macro, based upon fseek(), which resets a file position to the beginning of the file. e.g.
rewind() FILE *fp; rewind(fp); fseek(fp,0L,0); /* = rewind() */

Node:fflush, Next:Low Level Filing Operations, Previous:rewind, Up:Files and Devices

fflush()

This is a macro/function which can be used on files which have been opened for writing or appending. It flushes the output buffer which means that it forces the characters in the output buffer to be written to the file. If used on files which are open for reading, it causes the input buffer to be emptied (assuming that this is allowed at all). Example:
FILE *fp; fflush(fp);

Node:Low Level Filing Operations, Next:File Handles, Previous:fflush, Up:Files and Devices

Low Level Filing Operations

Normally a programmer can get away with using the high level input/output functions, but there may be times when C's predilection for handling all high level input/output as text files, becomes a nuisance. A program can then use a set of low level I/O functions which are provided by the standard library. These are:
open() close() creat() read() write() rename() unlink()/remove() lseek()

These low level routines work on the operating system's end of the file portals. They should be regarded as being advanced features of the language because they are

dangerous routines for bug ridden programs. The data which they deal with is untranslated: that is, no conversion from characters to floating point or integers or any type at all take place. Data are treated as a raw stream of bytes. Low level functions should not be used on any file at the same time as high level routines, since high level file handling functions often make calls to the low level functions. Working at the low level, programs can create, delete and rename files but they are restricted to the reading and writing of untranslated data: there are no functions such as fprintf() or fscanf() which make type conversions. As well as the functions listed above a local operating system will doubtless provide special function calls which enable a programmer to make the most of the facilities offered by the particular operating environment. These will be documented, either in a compiler manual, or in an operating system manual, depending upon the system concerned. (They might concern special graphics facilities or windowing systems or provide ways of writing special system dependent data to disk files, such as date/time stamps etc.) Node:File Handles, Next:open, Previous:Low Level Filing Operations, Up:Files and Devices

File descriptors
At the low level, files are not handled using file pointers, but with integers known as file handles or file descriptors. A file handle is essentially the number of a particular file portal in an array. In other words, for all the different terminology, they describe the same thing. For example:
int fd;

would declare a file handle or descriptor or portal or whatever it is to be called. Node:open, Next:close, Previous:File Handles, Up:Files and Devices
open() open()

is the low level file open function. The form of this function call is:

int fd, mode; char *filename; fd = open (filename,mode);

where filename is a string which holds the name of the file concerned, mode is a value which specifies what the file is to be opened for and fd is either a number used to distinguish the file from others, or -1 if an error occurred. A program can give more information to this function than it can to fopen() in order to define exactly what open() will do. The integer mode is a message or a pseudo register which passes the necessary information to open(), by using the following flags:

O_RDONLY O_WRONLY O_RDWR

Read access only Write access only Read/Write access

and on some compilers:

O_CREAT O_TRUNC O_APPEND O_EXCL Create the file if it does not exist Truncate the file if it does exist Find the end of the file before each write Exclude. Force create to fail if the file exists.

The macro definitions of these flags will be included in a library file: find out which one and #include it in the program. The normal procedure is to open a file using one of the first three modes. For example:
#define FAILED -1 main() { char *filename(); int fd; fd = open(filename(), O_RDONLY); if (fd == FAILED) { printf ("File not found\n"); error_handler (failed); } }

This opens up a read-only file for low level handling, with error checking. Some systems allow a more flexible way of opening files. The four appended modes are values which can be bitwise ORed with one of the first three in order to get more mileage out of open(). The bitwise OR operator is the vertical bar "|". For example, to emulate the fopen() function a program could opt to create a file if it did not already exist:
fd = open (filename(), O_RDONLY | O_CREAT);

open()

sets the file position to zero if the file is opened successfully.

Node:close, Next:creat, Previous:open, Up:Files and Devices

close()

releases a file portal for use by other files and brings a file completely up to date with regard to any changes that have been made to it. Like all other filing functions, it returns the value 0 if it performs successfully and the value -1 if it fails. e.g.
close() #define FAILED -1 if (close(fd) == FAILED) {

printf ("ERROR!"); }

Node:creat, Next:read, Previous:close, Up:Files and Devices

creat()

This function creates a new file and prepares it for access using the low level file handling functions. If a file which already exists is created, its contents are discarded. The form of this function call is:
int fd, pmode; char *filename; fd = creat(filename,pmode);

must be a valid filename; pmode is a flag which contains access-privilege mode bits (system specific information about allowed access) and fd is a returned file handle. In the absence of any information about pmode, this parameter can be set to zero. Note that, the action of creating a file opens it too. Thus after a call to creat, you should close the file descriptor. Node:read, Next:write, Previous:creat, Up:Files and Devices
filename

read()
This function gets a block of information from a file. The data are loaded directly into memory, as a sequence of bytes. The user must provide a place for them (either by making an array or by using malloc() to reserve space). read() keeps track of file positions automatically, so it actually reads the next block of bytes from the current file position. The following example reads n bytes from a file:
int returnvalue, fd, n; char *buffer; if ((buffer = malloc(size)) == NULL) { puts ("Out of memory\n"); error_handler (); } returnvalue = read (fd,buffer,n);

The return value should be checked. Its values are defined as follows:
0

End of file
-1 n

Error occurred

the number of bytes actually read. (If all went well this should be equal to n.) Node:write, Next:lseek, Previous:read, Up:Files and Devices
write()

This function is the opposite of read(). It writes a block of n bytes from a contiguous portion of memory to a file which was opened by open(). The form of this function is:
int returnvalue, fd, n; char *buffer; returnvalue = write (fd,buffer,n);

The return value should, again, be checked for errors:

-1 n

Error

Number of bytes written Node:lseek, Next:unlink remove, Previous:write, Up:Files and Devices
lseek()

Low level file handing functions have their equivalent of fseek() for finding a specific position within a file. This is almost identical to fseek() except that it uses the file handle rather than a file pointer as a parameter and has a different return value. The constants should be declared long int, or simply long.
#define FAILED -1L

long int pos,offset,fd; int mode,returncode; if ((pos = fseek (fd,offset,mode)) == FAILED) { printf("Error!\n"); }

gives the new file position if successful, and -1 (long) if an attempt was made to read past the end of the file. The values which mode can take are:
pos 0 1 2

Offset measured relative to the beginning of the file. Offset measured relative to the current position.

Offset measured relative to the end of the file. Node:unlink remove, Next:Example 37, Previous:lseek, Up:Files and Devices
unlink()

and remove()

These functions delete a file from disk storage. Once deleted, files are usually irretrievable. They return -1 if the action failed.
#define FAILED -1 int returnvalue; char *filename;

if (unlink (filename) == FAILED) { printf ("Can't delete %s\n",filename); } if (remove (filename) == FAILED) { printf ("Can't delete %s\n",filename); }

is a string containing the name of the file concerned. This function can fail if a file concerned is protected or if it is not found or if it is a device. (It is impossible to delete the printer!)
filename

rename() This function renames a file. The programmer specifies two filenames: the old filename and a new file name. As usual, it returns the value -1 if the action fails. An example illustrates the form of the rename() call:
#define FAILED -1 char *old,*new; if (rename(old,new) == FAILED) { printf ("Can't rename %s as %s\n",old,new); }

can fail because a file is protected or because it is in use, or because one of the filenames given was not valid.
rename()

Node:Example 37, Next:Questions 25, Previous:unlink remove, Up:Files and Devices

Example
This example strings together some low level filing actions so as to illustrate their use in a real program. The idea is to present a kind of file or "project" menu for creating, deleting, renaming files. A rather feeble text editor allows the user to enter 255 characters of text which can be saved.
/***************************************************************/ /* */ /* LOW LEVEL FILE HANDLING */ /* */ /***************************************************************/ #include <stdio.h> #include <ctype.h> #include <fcntl.h> #define CODE 0 #define SIZE 255 #define FNMSIZE 30

/* defines O_RDONLY etc.. */

/* Max size of filenames */

#define TRUE #define FALSE #define FAILED

1 0 -1

#define CLRSCRN() putchar('\f') #define NEWLINE() putchar('\n') int fd; /***************************************************************/ /* Level 0 */ /***************************************************************/ main () { char *data,getkey(),*malloc(); if ((data = malloc(SIZE)) == NULL) { puts ("Out of memory\n"); return (CODE); } while (TRUE) { menu(); switch (getkey()) { case 'l' : LoadFile(data); break; case 's' : SaveFile(data); break; case 'e' : Edit(data); break; case 'd' : DeleteFile(); break; case 'r' : RenameFile(); break; case 'q' : if (sure()) { return (CODE); } break; } } } /*************************************************************/ /* Level 1 */ /*************************************************************/ menu () { CLRSCRN(); printf (" ---------------------------------\n"); printf ("| MENU |\n"); printf ("| ~~~~~~ |\n"); printf ("| |\n"); printf ("| L) Load File |\n"); printf ("| S) Save File |\n"); printf ("| E) Edit File |\n"); printf ("| D) Delete File |\n");

printf ("| R) Rename File |\n"); printf ("| Q) Quit |\n"); printf ("| |\n"); printf ("| Select Option and RETURN |\n"); printf ("| |\n"); printf (" --------------------------------- \n"); NEWLINE(); } /*************************************************************/ LoadFile(data) char *data; { char *filename(),getkey(); int error; fd = open(filename(), O_RDONLY); if (fd == FAILED) { printf ("File not found\n"); return (FAILED); } error = read (fd,data,SIZE); if (error == FAILED) { printf ("Error loading file\n"); wait(); } else { if (error != SIZE) { printf ("File was corrupted\n"); wait(); } } close (fd,data,SIZE); return (error); } /*************************************************************/ SaveFile(data) char *data; { char *filename(),getkey(),*fname; int error,fd; fd = open ((fname = filename()), O_WRONLY); if (fd == FAILED) { printf ("File cannot be written to\n"); printf ("Try to create new file? Y/N\n"); if (yes()) { /* Low Level save */ /* Low level load */

if ((fd = CreateFile(fname)) == FAILED) { printf ("Cannot create file %s\n",fname); return (FAILED); } } else { return (FAILED); } } error = write (fd,data,SIZE); if (error < SIZE) { printf ("Error writing to file\n"); if (error != FAILED) { printf ("File only partially written\n"); } } close (fd,data,SIZE); wait(); return (error); } /*************************************************************/ Edit(data) char *data; { char *ptr; int ctr = 0; printf ("Contents of file:\n\n"); for (ptr = data; ptr < (data + SIZE); ptr++) { if (isprint(*ptr)) { putchar(*ptr); if ((ctr++ % 60) == 0) { NEWLINE(); } } } printf ("\n\nEnter %1d characters:\n",SIZE); for (ptr = data; ptr < (data + SIZE); ptr++) { *ptr = getchar(); } skipgarb(); } /*************************************************************/ /* primitive text editor */

DeleteFile()

/* Delete a file from current dir */

{ char *filename(),getkey(),*fname; printf ("Delete File\n\n"); fname = filename(); if (sure()) { if (remove(fname) == FAILED) { printf ("Can't delete %s\n",fname); } } else { printf ("File NOT deleted!\n"); } wait(); } /*************************************************************/ RenameFile() { char old[FNMSIZE],*new; printf ("Rename from OLD to NEW\n\nOLD: "); strcpy (old,filename()); printf ("\nNEW: "); new = filename(); if (rename(old,new) == FAILED) { printf ("Can't rename %s as %s\n",old,new); } wait(); } /*************************************************************/ /* Level 2 */ /*************************************************************/ CreateFile (fname) char *fname; { int fd; if ((fd = creat(fname,0)) == FAILED) { printf ("Can't create file %s\n",fname); return (FAILED); } return (fd); } /*************************************************************/ /* Toolkit */ /*************************************************************/

char *filename() { static char statfilenm[FNMSIZE]; do { printf ("Enter filename :"); scanf ("%24s",statfilenm); skipgarb(); } while (strlen(statfilenm) == 0); return (statfilenm); }

/* return

filename */

/**************************************************************/ sure () /* is the user sure ? */

{ printf ("Are you absolutely, unquestionably certain? Y/N\n"); return(yes()); } /**************************************************************/ yes() { char getkey(); while (TRUE) { switch(getkey()) { case 'y' : return (TRUE); case 'n' : return (FALSE); } } } /**************************************************************/ wait() { char getkey(); printf ("Press a key\n"); getkey(); } /**************************************************************/ char getkey() { char ch; ch = getchar(); skipgarb(); return((char)tolower(ch)); } /**************************************************************/ /* single key + RETURN response */

skipgarb() { while (getchar() != '\n') { } }

/* skip garbage corrupting input */

/* end */

Node:Questions 25, Previous:Example 37, Up:Files and Devices

Questions
1. What are the following? 1. File name 2. File pointer 3. File handle 2. What is the difference between high and low level filing? 3. Write a statement which opens a high level file for reading. 4. Write a statement which opens a low level file for writing. 5. Write a program which checks for illegal characters in text files. Valid characters are ASCII codes 10,13,and 32..126. Anything else is illegal for programs. 6. What statement performs formatted writing to text files? 7. Print out all the header files on your system so that you can see what is defined where! Node:Structures and Unions, Next:Data structures, Previous:Files and Devices, Up:Top

Structures and Unions

Grouping data. Tidying up programs. Tidy programs are a blessing to programmers. Tidy data are just as important. As programs become increasingly complex, their data also grow in complexity and single, independent variables or arrays are no longer enough. What one then needs is a data structure. This is where a new type of variable comes in: it is called a struct type, or in other languages, a record. struct types or structures are usually lumped together with another type of variable called a union. In fact their purposes are quite different.

Black Box Data: struct: Declarations again: Scope again: Using Structures: Arrays of Structures: Example 38: Structures of Structures: Pointers to Structures:

Example 39: Pre-initializing Static Structures: Creating Memory for Dynamical struct Types: Unions: Questions 26:

Node:Black Box Data, Next:struct, Previous:Structures and Unions, Up:Structures and Unions

Organization: Black Box Data

What is the relationship between a program and its data? Think of a program as an operator which operates on the memory of the computer. Local data are operated upon inside sealed function capsules, where they are protected from the reach of certain parts of a program. Global data are wide open to alteration by any part of a program. If a program were visualized schematically what would it look like? A traditional flow diagram? No: a computer program only looks like a flow diagram at the machine code level and that is too primitive for C programmers. One way of visualizing a program is illustrated by the diagram over the page.

This shows a program as a kind of society of sealed function capsules which work together like a beehive of activity upon a honeycomb of program data. This imaginative idea is not a bad picture of a computer program, but it is not complete

either. A program has to manipulate data: it has to look at them, move them around and copy them from place to place. All of these things would be very difficult if data were scattered about liberally, with no particular structure. For this reason C has the facility, within it, to make sealed capsules - not of program code - but of program data, so that all of these actions very simply by grouping variables together in convenient packages for handling. These capsules are called structures. Node:struct, Next:Declarations again, Previous:Black Box Data, Up:Structures and Unions

struct
A structure is a package of one or usually more variables which are grouped under a single name. Structures are not like arrays: a structure can hold any mixture of different types of data: it can even hold arrays of different types. A structure can be as simple or as complex as the programmer desires. The word struct is a reserved word in C and it represents a new data type, called an aggregate type. It is not any single type: the purpose of structures is to offer a tool for making whatever shape or form of variable package that a programmer wishes. Any particular structure type is given a name, called a structure-name and the variables (called members) within a structure type are also given names. Finally, every variable which is declared to be a particular structure type has a name of its own too. This plethora of names is not really as complicated as it sounds. Node:Declarations again, Next:Scope again, Previous:struct, Up:Structures and Unions

Declarations
A structure is declared by making a blank template for a variable package. This is most easily seen with the help of an example. The following statement is actually a declaration, so it belongs with other declarations, either at the head of a program or at the start of a block.
struct PersonalData { char name[namesize]; char address[addresssize]; int YearOfBirth; int MonthOfBirth; int DayOfBirth; };

This purpose of this statement is to create a model or template to define what a variable of type struct PersonalData will look like. It says: define a type of variable which collectively holds a string called name, a string called address and three integers called YearOfBirth, MonthOfBirth and DayOfBirth. Any variable which is declared to be of type struct PersonalData will be collectively made up of parts like these. The list of variable components which make up the structure are called the members of the structure: the names of the members are not the names of variables,

but are a way of naming the parts which make up a structure variable. (Note: a variable which has been declared to be of type struct something is usually called just a structure rather than a structure variable. The distinction is maintained here in places where confusion might arise.) The names of members are held separate from the names of other identifiers in C, so it is quite possible to have variable names and struct member names which are the same. Older compilers did not support this luxury. At this stage, no storage has been given over to a variable, nor has any variable been declared: only a type has been defined. Having defined this type of structure, however, the programmer can declare variables to be of this type. For example:
struct PersonalData x;

declares a variable called x to be of type struct PersonalData. x is certainly not a very good name for any variable which holds a person's personal data, but it contrasts well with all the other names which are abound and so it serves its purpose for now. Before moving on to consider how structures can be used, it is worth pausing to show the different ways in which structures can be declared. The method shown above is probably the most common one, however there are two equivalent methods of doing the same thing. A variable can be declared immediately after the template definition.
struct PersonalData { char name[namesize]; char address[addresssize]; int YearOfBirth; int MonthOfBirth; int DayOfBirth; } x; /* variable identifier follows type */

Alternatively, typedef can be used to cut down a bit on typing in the long term. This type definition is made once at the head of the program and then subsequent declarations are made by using the new name:
typedef struct { char name[namesize]; char address[addresssize]; int YearOfBirth; int MonthOfBirth; int DayOfBirth; } PersonalData;

then declare:
PersonalData x;

Any one of these methods will do. Node:Scope again, Next:Using Structures, Previous:Declarations again, Up:Structures and Unions

Scope
Both structure types and structure variables obey the rules of scope: that is to say, a structure type declaration can be local or global, depending upon where the declaration is made. Similarly if a structure type variable is declared locally it is only valid inside the block parentheses in which it was defined.
main () { struct ONE { int a; float b; }; struct ONE x; } function () { struct ONE x; } /* This line is illegal, since ONE */ /* is a local type definition */ /* Defined only in main() */

Node:Using Structures, Next:Arrays of Structures, Previous:Scope again, Up:Structures and Unions

Using Structures
How does a program use the variables which are locked inside structures? The whole point about structures is that they can be used to group data into sensible packages which can then be treated as single objects. Early C compilers, some of which still exist today, placed very severe restrictions upon what a program could do with structures. Essentially, the members of a structure could be assigned values and pointers to individual structures could be found. Although this sounds highly restrictive, it did account for the most frequent uses of structures. Modern compilers allow more flexible use of structures: programs can assign one structure variable to another structure variable (provided the structures match in type); structure variables can be passed, whole, as parameters to functions and functions can return structure values. This makes structures extremely powerful data objects to have in a program. A structure is assigned to another structure by the following statements.
struct Personal x,y; x = y;

The whole bundle of members is copied in one statement! Structures are passed as parameters in the usual way:
function (x,y);

The function then has to be declared:

function (x,y) struct PersonalData x,y; { }

Finally, a function which returns a structure variable such as:

{ struct PersonalData x,function(); x = function(); }

would be declared in the following way:

struct PersonalData function () { }

Notice that the return type of such a function must also be declared in the function which calls that it, in the usual way. The reader will begin to see that structure names account for a good deal of typing! The typedef statement is a very good way of reducing this burden. The members of a structure are accessed with the . dot character. This is a structure member operator. Consider the structure variable x, which has the type struct PersonalData. The members of x could be assigned by the following program:
main () { struct PersonalData x; FillArray ("Some name", x.name); FillArray ("Some address", x.address); x.YearOfBirth = 1987; x.MonthOfBirth = 2; x.DayOfBirth = 19; }

where FillArray() is a hypothetical function which copies the string in the first parameter to the array in the second parameter. The dot between the variable and the names which follow implies that the statements in this brief program are talking about the members in the structure variable x, rather than the whole collective bundle. Members of actual structure variables are always accessed with this dot operator. The general form of a member reference is:

structure variable.member name

This applies to any type of structure variable, including those accessed by pointers. Whenever a program needs to access the members of a structure, this dot operator can be used. C provides a special member operator for pointers, however, because they are used so often in connection with structures. This new operator is described below. Node:Arrays of Structures, Next:Example 38, Previous:Using Structures, Up:Structures and Unions

Arrays of Structures
Just as arrays of any basic type of variable are allowed, so are arrays of a given type of structure. Although a structure contains many different types, the compiler never gets to know this information because it is hidden away inside a sealed structure capsule, so it can believe that all the elements in the array have the same type, even though that type is itself made up of lots of different types. An array would be declared in the usual way:
int i; struct PersonalData x,array[size];

The members of the arrays would then be accessed by statements like the following examples:
array[i] = x; array[i] = array[j]; array[i].YearOfBirth = 1987; i = array[2].MonthOfBirth;

Node:Example 38, Next:Structures of Structures, Previous:Arrays of Structures, Up:Structures and Unions

Example
This listing uses a structure type which is slightly different to PersonalData in that string pointers are used instead of arrays. This allows more convenient handling of real-life strings.
/*********************************************************/ /* */ /* Structures Demo */ /* */ /*********************************************************/ /* Simple program to initialize some structures */ /* and to print them out again. Does no error */ /* checking, so be wary of string sizes etc.. */

#include <stdio.h> #define #define #define #define NAMESIZE ADDRSIZE NOOFPERSONS NEWLINE() 30 80 20 putchar('\n');

/*********************************************************/ typedef struct { char *Name; char *Address; int YearOfBirth; int MonthOfBirth; int DayOfBirth; } PersonDat; /*********************************************************/ main () { PersonDat record[NOOFPERSONS]; PersonDat PersonalDetails(); int person; printf ("Birth Records For Employees"); printf ("\n---------------------------"); printf ("\n\n"); printf ("Enter data\n"); for (person = 0; person < NOOFPERSONS; person++) { record[person] = PersonalDetails(); NEWLINE(); } DisplayRecords (record); } /*********************************************************/ PersonDat PersonalDetails() /* No error checking! */ /* Make some records */

{ PersonDat dat; char strbuff[ADDRSIZE], *malloc(); printf ("Name :"); dat.Name = malloc(NAMESIZE); strcpy (dat.Name,gets(strbuff)); printf ("Address :"); dat.Address = malloc(ADDRSIZE); strcpy (dat.Address,gets(strbuff)); printf ("Year of birth:"); dat.YearOfBirth = getint (1900,1987); printf ("Month of birth:"); dat.MonthOfBirth = getint (1,12);

printf ("Day of birth:"); dat.DayOfBirth = getint(1,31); return (dat); } /**********************************************************/ DisplayRecords (rec) PersonDat rec[NOOFPERSONS]; { int pers; for (pers = 0; pers < NOOFPERSONS; pers++) { printf ("Name : %s\n", rec[pers].Name); printf ("Address : %s\n", rec[pers].Address); printf("Date of Birth: %1d/%1d/%1d\n",rec[pers].DayOfBirth, rec[pers].MonthOfBirth,rec[pers].YearOfBirth); NEWLINE(); } } /**********************************************************/ /* Toolkit */ /**********************************************************/ getint (a,b) int a,b; { int p, i = a - 1; /* return int between a and b */

for (p=0; ((a > i) || (i > b)); p++) { printf ("? : "); scanf ("%d",&i); if (p > 2) { skipgarb(); p = 0; } } skipgarb(); return (i); } /**********************************************************/ skipgarb() /* Skip input garbage corrupting scanf */

{ while (getchar() != '\n') { } } /* end */

Node:Structures of Structures, Next:Pointers to Structures, Previous:Example 38, Up:Structures and Unions

Structures of Structures
Structures are said to nest. This means that structure templates can contain other structures as members. Consider two structure types:
struct first_structure { int value; float number; };

and
struct second_structure { int tag; struct first_structure fs; } x;

These two structures are of different types, yet the first of the two is included in the second! An instance of the second structure would be initialized by the following assignments. The structure variable name is x:
x.tag = 10; x.fs.value = 20; x.fs.number = 30.0;

Notice the way in which the member operator . can be used over and over again. Notice also that no parentheses are necessary, because the reference which is calculated by this operator is worked out from left to right. This nesting can, in principle, go on many times, though some compilers might place restrictions upon this nesting level. Statements such as:
variable.tag1.tag2.tag3.tag4 = something;

are probably okay (though they do not reflect good programming). Structures should nest safely a few times. A word of caution is in order here. There is a problem with the above scheme that has not yet been addressed. It is this: what happens if a structure contains an instance of itself? For example:
struct Regression { int i; struct Regression tag; }

There is simply no way that this kind of statement can make sense, unless the compiler's target computer has an infinite supply of memory! References to variables

of this type would go on for ever and an infinite amount of memory would be needed for every variable. For this one reason, it is forbidden for a structure to contain an instance of itself. What is not forbidden, however, is for a structure to contain an instance of a pointer to its own type (because a pointer is not the same type as a structure: it is merely a variable which holds the address of a structure). Pointers to structures are quite invaluable, in fact, for building data structures such as linked lists and trees. These extremely valuable devices are described below. Node:Pointers to Structures, Next:Example 39, Previous:Structures of Structures, Up:Structures and Unions

Pointers to Structures
A pointer to a structure type variable is declared by a statement like:
struct Name *ptr;

is then, formally, a pointer to a structure of type Name only. ptr can be assigned to any other pointer of similar type and it can be used to access the members of a structure. It is in the second of these actions that a new structure operator is revealed. According to the rules which have described so far, a structure member could be accessed by pointers with the following statements:
ptr struct PersonalData *ptr; (*ptr).YearOfBirth = 20;

This says let the member YearOfBirth of the structure pointed to by ptr, have the value 20. Notice that *ptr, by itself, means the contents of the address which is held in ptr and notice that the parentheses around this statement avoid any confusion about the precedence of these operators. There is a better way to write the above statement, however, using a new operator: ->. This is an arrow made out of a minus sign and a greater than symbol and it is used simply as follows:
struct PersonalData *ptr; ptr->YearOfBirth = 20;

This statement is identical in every way to the first version, but since this kind of access is required so frequently, when dealing with structures, C provides this special operator to make the operation clearer. In the statements above, it is assumed that ptr has been assigned to the address of some pre-assigned structure: for example, by means of a statement such as:
ptr = &x;

where x is a pre-assigned structure.

Node:Example 39, Next:Pre-initializing Static Structures, Previous:Pointers to Structures, Up:Structures and Unions

Example
/*********************************************************/ /* */ /* Structures Demo #2 */ /* */ /*********************************************************/ /* This is the same program, using pointer references */ /* instead of straight variable references. i.e. this */ /* uses variable parameters instead of value params */ #include <stdio.h> #define #define #define #define NAMESIZE ADDRSIZE NOOFPERSONS NEWLINE() 30 80 20 putchar('\n');

/*********************************************************/ typedef struct { char *Name; char *Address; int YearOfBirth; int MonthOfBirth; int DayOfBirth; } PersonDat; /*********************************************************/ main () { PersonDat record[NOOFPERSONS]; int person; printf ("Birth Records For Employees"); printf ("\n---------------------------"); printf ("\n\n"); printf ("Enter data\n"); for (person = 0; person < NOOFPERSONS; person++) { PersonalDetails(&(record[person])); NEWLINE(); } DisplayRecords (record); } /*********************************************************/ PersonalDetails(dat) PersonDat *dat; /* No error checking! */ /* Make some records */

{ char strbuff[ADDRSIZE], *malloc(); printf ("Name :"); dat->Name = malloc(NAMESIZE); strcpy (dat->Name,gets(strbuff)); printf ("Address :"); dat->Address = malloc(ADDRSIZE); strcpy (dat->Address,gets(strbuff)); printf ("Year of birth:"); dat->YearOfBirth = getint (1900,1987); printf ("Month of birth:"); dat->MonthOfBirth = getint (1,12); printf ("Day of birth:"); dat->DayOfBirth = getint(1,31); } /**********************************************************/ DisplayRecords (rec) PersonDat rec[NOOFPERSONS]; { int pers; for (pers = 0; pers < NOOFPERSONS; pers++) { printf ("Name : %s\n", rec[pers].Name); printf ("Address : %s\n", rec[pers].Address); printf("Date of Birth: %1d/%1d/%1d\n",rec[pers].DayOfBirth, rec[pers].MonthOfBirth,rec[pers].YearOfBirth); NEWLINE(); } } /**********************************************************/ /* Toolkit */ /**********************************************************/ /* As before */

Node:Pre-initializing Static Structures, Next:Creating Memory for Dynamical struct Types, Previous:Example 39, Up:Structures and Unions

Pre-initializing Static Structures

In the chapter on arrays it was shown how static and external type arrays could be initialized with values at compile time. Static and external structures can also be preassigned by the compiler so that programs can set up options and starting conditions in a convenient way. A static variable of type PersonDat (as in the example programs) could be declared and initialized in the same statement:
#define NAMESIZE 20 #define ADDRESSSIZE 22 struct PersonDat {

char *name; char *address; int YearOfBirth; int MonthOfBirth; int DayOfBirth; }; main () { static struct PersonalData variable = { "Alice Wonderment", "Somewhere in Paradise", 1965, 5, 12 }; /* rest of program */ }

The items in the curly braces are matched to the members of the structure variable and any items which are not initialized by items in the list are filled out with zeros. Node:Creating Memory for Dynamical struct Types, Next:Unions, Previous:Preinitializing Static Structures, Up:Structures and Unions

Creating Memory for Dynamical struct Types

Probably the single most frequent use of struct type variables is in the building of dynamical data structures. Dynamical data are data which are created explicitly by a program using a scheme of memory allocation and pointers. Normal program data, which are reserved space by the compiler, are, in fact, static data structures because they do not change during the course of a program: an integer is always an integer and an array is always an array: their sizes cannot change while the program is running. A dynamical structure is built using the memory allocation function:
malloc()

and pointers. The idea is to create the memory space for a new structure as and when it is needed and to use a pointer to access the members of that structure, using the -> operator. malloc() was described in connection with strings: it allocates a fixed number of bytes of memory and returns a pointer to that data. For instance, to allocate ten bytes, one would write something like this:
char *malloc(), *ptr; ptr = malloc(10);

is then a pointer to the start of that block of 10 bytes. When a program wants to create the space for a structure, it has a template for that structure, which was used to define it, but it does not generally know, in advance, how many bytes long a structure
ptr

is. In fact, it is seldom possible to know this information, since a structure may occupy more memory than the sum of its parts. How then does a program know how must space to allocate? The C compiler comes to the rescue here, by providing a compile time operator called
sizeof ()

which calculates the size of an object while a program is compiling. For example:
sizeof(int)

Works out the number of bytes occupied by the type int.

sizeof(char)

Works out the number of bytes occupied by a single character. This equals 1, in fact. works out the number of bytes needed to store a single structure variable. Obviously this tool is very useful for working with malloc(). The memory allocation statement becomes something like:
sizeof(struct PersonalData) ptr = malloc(sizeof(type name));

There is a problem with this statement though: malloc() is declared as a function which returns a type `pointer to character' whereas, here, the programmer is interested in pointers of type "pointer to struct Something". malloc() has to be forced to produce a pointer of the correct type then and this is done by using the cast operator to mould it into shape. The cast operator casts pointers with a general form:
(type *) value

Consider the following example of C source code which allocates space for a structure type called SomeStruct and creates a correctly aligned pointer to it, called ptr.
struct SomeStruct *ptr; char *malloc(); ptr = (struct SomeStruct *) malloc(sizeof(struct Somestruct));

This rather laboured statement provides both the memory and the location of that memory in a legal and type-sensical way. The next section of this book discusses what we can do with dynamically allocated structures. Node:Unions, Next:Questions 26, Previous:Creating Memory for Dynamical struct Types, Up:Structures and Unions

Unions
A union is like a structure in which all the `members' are stored at the same address. Clearly they cannot all be there at the same time. Only one member can be stored in such an object at any one time, or it would be overwritten by another. Unions behave

like specially sized storage containers which can hold many different types of data. A union can hold any one of its members but only at different times. The compiler arranges that a union type variable is big enough to handle the job. The real purpose of unions is to prevent memory fragmentation by arranging for a standard size for data in the memory. By having a standard data size we can guarantee that any hole left when dynamically allocated memory is freed will always be reusable by another instance of the same type of union. This is a natural strategy in system programming where many instances of different kinds of variables with a related purpose and stored dynamically.

Declaration of union: Using unions:

Node:Declaration of union, Next:Using unions, Previous:Unions, Up:Unions

Declaration
A union is declared in the same way as a structure. It has a list of members, which are used to mould the type of object concerned.
union IntOrFloat { int ordinal; float continuous; };

This declares a type template. Variables are then declared as:

union IntOrFloat x,y,z;

At different times the program is to treat x,y and z as being either integers or float types. When the variables are referred to as
x.ordinal = 1;

the program sees x as being an integer type. At other times (when x is referred to as x.continuous) it takes on another aspect: its alter ego, the float type. Notice that x by itself does not have a value: only its members have values, x is just a box for the different members to share. Node:Using unions, Previous:Declaration of union, Up:Unions

Using unions
Unions are coded with the same constructions as structures. The dot . operator selects the different members for variable and the arrow -> selects different values for pointers. The form of such statements is:

union_variable.member; union_pointer->member;

Unions are seldom very useful objects to have in programs, since a program has no automatic way of knowing what type of member is currently stored in the union type. One way to overcome this is to keep a variable which signals the type currently held in the variable. This is done very easily with the aid of enumerated data. Consider the following kind of union:
union WhichType { int ordinal; float continuous; char letter; };

This could be accompanied by an enumerate declaration such as:

enum Types { INT, FLOAT, CHAR };

Variables could then go in pairs:

union WhichType x; enum Types x_status;

which would make union type-handling straightforward:

switch (x_status) { case INT : x.ordinal = 12; break; case FLOAT : x.continuous = 12.23; break; case CHAR : x.letter = '*'; }

These variables could even be grouped into a structure:

struct Union_Handler { union WhichType x; enum Types x_status; } var;

which would then require statements such as:

var.x.ordinal = 2;

ptr->x.ordinal = 2; var.x_status = CHAR;

and so on... Node:Questions 26, Previous:Unions, Up:Structures and Unions

Questions
1. 2. 3. 4. What is the difference between a structure and a union? What is a member? If x is a variable, how would you find out the value of a member called mem. If ptr is a pointer to a structure, how would you find out the value of a member called mem. 5. A union is a group of variables in a single package. True or false? Node:Data structures, Next:Recursion, Previous:Structures and Unions, Up:Top

Data Structures
Uses for struct variables. Structure diagrams. Data structures are organized patterns of data. The purpose of building a data structure is to create a pattern of information which models a particular situation clearly and efficiently. Take the simplest kind of data structure: the array. Arrays are good for storing patterns of information which look like tables, or share a tabular structure. For example, a chess board looks like a two dimensional array, so a chess game would naturally use a two dimensional array to store the positions of pieces on the chess board. The aim of a data structure is to model real life patterns with program data. Most real application programs require a more complex data structure than C variables can offer; often arrays are not suitable structures for a given application. To see this, consider an application example in which a program stores a map of the local countryside. This program has to store information about individual towns and it has to be able to give directions to the user about how to get to particular towns from some reference point. In real life, all of this information is most easily conveyed by means of a map, with towns' vital statistics written on it. (See figure 1.) The diagram shows such a simplified map of the surrounding land. This sort of map is, ideally, just what a computer ought to be able to store. The handicap is that the map does not look very computerish. If the map is ever going to be stored in a computer it will need to look more mechanical. A transformation is needed. In order to make the map into a more

computer-like picture, it must be drawn as a structure diagram.

A structure diagram is a picture which shows how something is connected up. Most often a structure diagram shows how a problem is connected up by relating all the parts which go together to make it up. In this case, the structure diagram just shows how program data are related to each other.

Data Structure Diagrams: Tools: Programme For Building Data Structures: Setting Up A Data Structure: Example Structures: Questions 27:

Node:Data Structure Diagrams, Next:Tools, Previous:Data structures, Up:Data structures

Data Structure Diagrams

Now examine figure 2. This diagram is a data structure diagram: it is a diagram which shows how boxes of data must relate to one another in order to solve the problem of the towns map. It has been drawn, quite deliberately, in a way which is intended to conjure up some particular thoughts. The arrows tend to suggest that pointers will play a role in the data structure. The blocks tend to suggest that sealed capsules or struct type data will also play a role. Putting these two together creates the idea of a `town structure' containing pointers to neighouring villages which lie on roads to the North, South, East and West of the town, as well as the information about the town itself. This town structure might look something like this:
struct Town

{ struct struct struct struct struct };

Town *north; Town *south; Town *east; Town *west; LocalInfo help;

Assume for now that LocalInfo is a structure which contains all the information about a town required by the program. This part of the information is actually irrelevant to the structure of the data because it is hidden inside the sealed capsule. It is the pointers which are the main items of concern because it is pointers which contain information that enables a program to find its way around the map very quickly. If the user of this imaginary application program wished to know about the town to the north of one particular place, the program would only have to refocus its attention on the new structure which was pointed to by the struct member north and similarly for other directions. A data structure is built up, like a model, by connecting struct type variables together with pointers: these are the building blocks. By thinking of struct types and pointers in terms of pictures, one begins to see how structures can be fashioned, in computer memory, to look exactly like the problems which they represent. What is interesting about data structure diagrams is the way in which they resemble the structure diagrams of C programs, which were drawn in chapter 7. There is a simple reason for this similarity: computer programs are themselves just data structures in which the data are program instructions and the pointers and sealed boxes are function calls. The structure of a computer program is called a hierachy. Sometimes the shape of data structures and programs are identical; when this happens, a kind of optimum efficiency has been reached in conceptual terms. Programs which behave exactly like

their data operate very simply. This is the reason why structure diagrams are so useful in programming: a structure diagram is a diagram which solves a problem and does so in a pictorial way, which models the way we think. Node:Tools, Next:Programme For Building Data Structures, Previous:Data Structure Diagrams, Up:Data structures

The Tools: Structures, Pointers and Dynamic Memory

The tools of the data structure trade are struct types and pointers. Data structures are built out of dynamically allocated memory, so storage places do not need names: all a program needs to do is to keep a record of a pointer, to a particular storage space, and the computer will be able to find it at any time after that. Pointers are the keys which unlock a program's data. The reader might object to this by saying that a pointer has to be stored in some C variable somewhere, so does a program really gain anything from working with pointers? The answer is yes, because pointers in data structures are invariably chained together to make up the structure. To understand this, make a note of the following terms: Root This is a place where a data structure starts. Every chain has to start somewhere. The address of the root of a data structure has to be stored explicitly in a C variable. Links A link is a pointer to a new struct type. Links are used to chain structures together. The address of the next element in a chain structure is stored inside the previous structure. Data structures do not have to be linear chains and they are often not. Structures, after all, can hold any number of pointers to other structures, so there is the potential to branch out into any number of new structures. In the map example above, there were four pointers in each structure, so the chaining was not linear, but more like a latticework. We need to think about where and how data structures are going to be stored. Remember that pointers alone do not create any storage space: they are only a way of finding out the contents of storage space which already exists. In fact, a program must create its own space for data structures. The key phrase is dynamic storage: a program makes space for structures as new ones are required and deletes space which is does not require. The functions which perform this memory allocation and release are:
malloc() and free()

There are some advantages which go with the use of dynamic storage for data structures and they are summarized by the following points:

Since memory is allocated as it is needed, the only restriction on data size is the memory capacity of the computer. We don't need to declare how much we shall use in advance. Using pointers to connect structures means that they can be re-connected in different ways as the need arises. (Data structures can be sorted, for example.)

Data structures can be made up of lots of "lesser" data structures, each held inside struct type storage. The limitations are few.

The remaining parts of this section aim to provide a basic plan or formula for putting data structures together in C. This is done with recourse to two example structures, which become two example programs in the next chapter. Node:Programme For Building Data Structures, Next:Setting Up A Data Structure, Previous:Tools, Up:Data structures

Programme For Building Data Structures

In writing programs which centre around their data, such as word processors, accounts programs or database managers, it is extremely important to plan data structures before any program code is written: changes in program code do not affect a data structure, but alterations to a data structure imply drastic changes to program code. Only in some numerical applications does a data structure actually assist an algorithm rather than vice versa. The steps which a programmer would undertake in designing a data structure follow a basic pattern:

Group all the data, which must be stored, together and define a struct type to hold them. Think of a pattern which reflects the way in which the data are connected and add structure pointers to the struct definition, to connect them. Design the programming algorithms to handle the memory allocation, link pointers and data storage.

Node:Setting Up A Data Structure, Next:Example Structures, Previous:Programme For Building Data Structures, Up:Data structures

Setting Up A Data Structure

Once the basic mould has been cast for the building blocks, a program actually has to go through the motions of putting all the pieces together, by connecting structures together with pointers and filling them up with information. The data structure is set up by repeating the following actions as many times as is necessary.

Define a struct type. For example:

struct Town { struct Town *north; struct Town *south; struct Town *east; struct Town *west; struct LocalInfo help; };

Declare two pointers to this type:

struct Town *ptr,*root;

One of these is used to hold the root of the data structure and the other is used as a current pointer.

Allocate memory for one structure type:

root = (struct Town *) malloc(sizeof(struct Town));

Be careful to check for errors. root will be NULL if no memory could be allocated.

Initialize the members of the structure with statements such as:

root->north = NULL; root->south = NULL; root->help.age = 56; /* if age is a member */ /* of struct LocalInfo */

This sets the pointers north and south to the value NULL, which conventionally means that the pointer does not point anywhere.

When other structures have been created, the pointers can be assigned to them:
ptr = (struct Town *) malloc(sizeof(struct Town)); ptr->north = NULL; ptr->south = NULL; /* etc.. initialize members */ root->north = ptr;

This last statement connects the new structure onto the north branch of root.

pointer assignments tell the program handling the data structure when it has come to the edge of the structure: that is when it has found a pointer which doesn't lead anywhere.
NULL

Node:Example Structures, Next:Questions 27, Previous:Setting Up A Data Structure, Up:Data structures

Example Structures
Two data structures of thids kind are very common: the linked list and the binary tree and both work upon the principles outlined above (In fact they are just different manifestations of the same thing.) A linked list is a linear sequence of structures joined together by pointers. If a structure diagram were drawn of a linked list, all the storage blocks in it would lie in a straight line, without branching out.
struct list { double value; struct list *succ; };

A linked list has only a single pointer per structure, which points to the successor in the list. If the blocks were labelled A B C D E... then B would be the successor of A; C would be the successor of B and so on. Linked lists have two advantages over one dimensional arrays: they can be sorted easily (see diagram) and they can be made any length at all. A binary tree is a sequence of structures, each of which branches out into two new ones.
struct BinaryTree { /* other info */ struct BinaryTree *left; struct BinaryTree *right; } *tree = NULL;

A binary tree structure has two pointers per struct type. This is useful for classifying data on a greater than/less than basis.

Right and left branches are taken to mean `greater than' and `less than' respectively. The programs which handle these data structures are written in the form of complete, usable application programs. They are simple by professional standards, but they are long by book standards so they are contained in a section by themselves, along with their accompanying programmers' documentation, See Example Programs chapter. Node:Questions 27, Previous:Example Structures, Up:Data structures

Questions
1. What is a structure diagram? 2. How are data linked together to make a data structure? 3. Every separate struct type in a data structure has its own variable name. True or false? 4. How are the members of structures accessed in a data structure? 5. Write a statement which creates a new structure of type "struct BinaryTree" and finds its address. Store that address in a variable which is declared as follows:
6. 7. struct BinaryTree *ptr;

8. Write a small program which makes a linked list, three structures long and assigns all their data to be zero. Can you automate this program with a loop? Can you make it work for any number of structures? Node:Recursion, Next:Example Programs chapter, Previous:Data structures, Up:Top

Recursion
The daemon which swallowed its tail. This section is about program structures which can talk about themselves. What happens to a function which makes a call itself? Examine the function below:
Well_Function ()

{ /* ... other statements ... */ Well_Function (); }

is said to be a recursive function. It is defined in terms of itself: it contains itself and it calls itself. It swallows its own tail! The act of self-reference is called recursion. What happens to such a function when it is called in a C program? In the simple example above, something dramatic and fatal happens. The computer, naturally, begins executing the statements in the function, inside the curly braces. This much is only normal: programs are designed to do this and the computer could do no more and no less. Eventually the program comes upon the statement Well_Function(); and it makes a call to that function again. It then begins executing statements in Well_function(), from the beginning, as though it were a new function, until it comes upon the statement Well_Function() and then it calls the function again....
Well_Function()

This kind of function calling scenario is doomed to continue without end, as, each time the function is called, it is inevitably called again. The computer becomes totally consumed with the task of calling Well_Function() over and over. It is apparently doomed to repeat the same procedure for ever. Or is it?

Functions and The Stack: Levels and Wells: Tame Recursion and Self-Similarity: Simple Example without a Data Structure: Simple Example With a Data Structure: Advantages and Disadvantages of Recursion: Recursion and Global Variables: Questions 28:

Node:Functions and The Stack, Next:Levels and Wells, Previous:Recursion, Up:Recursion

Functions and The Stack

We should think about the exact sequence of events which takes place when a function is called in a program. This will help to cast some light on the mechanics of recursion and recursive functions. When a function is called, control passes from one place in a program to another place. The statements in this new region of the program are carried out and then control returns to a statement immediately following the one which made the function call. But how does the computer know where it must go back to, when it has finished with a function call? It is suddenly thrown into a wildly different region of the memory and finds itself executing statements there. How can it get back again? A diagram does not answer this question: program structure diagrams hide this detail

from view.

function1() / / \ \ function3() / \

function2() / \

The answer to this puzzle is that the computer keeps a record of the addresses of the places to which it must return, no matter how many times functions are called. It does this by building a special data structure called a stack.

A stack is quite literally a pile of data, organized in the memory. Information is placed on top of a stack and taken from the top. It is called a last in, first out (LIFO) structure because the last thing to go on the top of a stack is always the first thing to come off it. C organizes a stack structure when it runs a program and uses it for storing local variables and for keeping track of where it has to return to. When it calls a function, it leaves itself a reminder, on the top of its program stack, which tells it where it has to go to when it has finished executing that function. C management makes sure that it does not put anything else on top of that reminder to spoil the flow of control. When a function is finished, the program takes the first message from the top of the stack and carries on executing statements at the place specified by the message. Normally this method works perfectly, without any problems at all: functions are called and they return again; the stack grows and shrinks and all is well. What happens when a recursive function, like Well_Function() calls itself? The system works as normal. C makes a note of the place it has to return to and puts that

note on top of the stack. It then begins executing statements. When it comes to the call Well_Function() again, it makes a new note of where it has to come back to and deposits it on top of the stack. It then begins the function again and when it finds the function call, it makes a new note and puts on the top of the stack.... As this process continues, the memory gets filled up with the program's messages to itself: the stack of messages gets larger and larger. Since the function has no chance of returning control to its caller, the messages never get taken off the stack and it just builds up. Eventually the computer runs out of memory and the computer crashes or interrupts the program with a fatal error message. Node:Levels and Wells, Next:Tame Recursion and Self-Similarity, Previous:Functions and The Stack, Up:Recursion

Levels and Wells

A stack is made up of frames or levels. Each time a function is called, the program is said to drop down a level. This is the reason for structure comments like:
/****************************************************/ /* Level 1 */ /****************************************************/

in the programs in this book. The main() function is at level 0 because it is the root of the program. If main() calls any functions at all, control drops down to level one. When a level one function returns, it hands control back to level zero. These level numbers actually count the height of the program stack at any point in a program. The level number is the number of messages or reminders on the stack. A function like Well_Function() digs itself a well of infinite depth. It punches a great hole in a program; it has no place in a levelled structure diagram. The function is pathological because it causes the stack fill up the memory of the computer. A better name for this function would be:
StackOverflow() { StackOverflow(); } /* Causes stack to grow out of control */

Node:Tame Recursion and Self-Similarity, Next:Simple Example without a Data Structure, Previous:Levels and Wells, Up:Recursion

Tame Recursion and Self-Similarity

Recursion does not have to be so dramatically disastrous as the example given. If recursion is tamed, it provides perhaps the most powerful way of handling certain kinds of problem in programming, particularly concerning data structures. Earlier we remarked that programs and data structures aim to model the situation they deal with as closely as possible. Some problems are made up of many levels of detail (see the introduction to this tutorial) and the details are identical at all levels. Since

recursion is about functions which contain themselves at all levels, this tends to suggest that recursion would be useful for dealing with these self-similar problems. Data structures are prime candidates for this because they are made up of identical structure types, connected together in a way which make them look like programs connected up by function calls. Recursive functions can be tamed by making sure that there is a safe way exit them, so that recursion only happens under particular circumstances. The aim is to control the number of times that recursion takes place by making a decision about what happens in the function: the decision about whether a function calls itself or not. For example, it is easy to make Well_Function recurse four times only, by making a test:
Well_Function(nooftimes) int nooftimes; { if (nooftimes == 0) { return (0); } else { Well_Function(nooftimes-1); } }

A call of WellFunction(4) would make this function drop down four stack levels and then return. Notice the way in which the if..else statement shields the program from the recursion when nooftimes equals zero. It effectively acts as a safety net, stopping the programming from plunging down the level well infinitely. Node:Simple Example without a Data Structure, Next:Simple Example With a Data Structure, Previous:Tame Recursion and Self-Similarity, Up:Recursion

Simple Example without a Data Structure

A completely standard example of controlled recursion is the factorial (or Gamma) function. This is a mathematical function which is important in statistics. (Mathematicians also deal with recursive functions; computer programs are not alone in this.) The factorial function is defined to be the "product" (multiplication) of all the natural (unsigned integer) numbers from 1 to the parameter of the function. For example:
factorial(4) == 1 * 2 * 3 * 4 factorial(6) == 1 * 2 * 3 * 4 * 5 * 6 == 24 == 720

Formally, the factorial function is defined by two mathematical statements:

factorial (n) = n * factorial(n-1)

and
factorial (0) = 1

The first of these statements is recursive, because it defines the value of factorial(n) in terms of the factorial function of (n-1). This strange definition seems to want to lift itself by its very bootstraps! The second statement saves it, by giving it a reference value. The factorial function can be written down immediately, as a controlled recursive function:
factorial (n) unsigned int n; { if (n == 0) { return (1); } else { return (n * factorial(n-1)); } }

To see how this works, try following it through for n equals three. The statement:
factorial (3);

causes a call to be made to factorial(). The value of n is set to three. factorial() then tests whether n is zero (which it is not) so it takes the alternative branch of the if..else statement. This instructs it to return the value of:
3 * factorial(3-1)

In order to calculate that, the function has to call factorial recursively, passing the value (3-1) or 2 to the new call. The new call takes this value, checks whether it is zero (it is not) and tries to return the value 2 * factorial(1). In order to work this out, it needs to call factorial again, which checks that n is not 0 (it is not) and so tries to return 1 * factorial(0). Finally, it calls factorial(0) which does not call factorial any more, but starts unloading the stack and returning the values. The expression goes through the following steps before finally being evaluated:
factorial (3) == == == == 3 3 3 3 * * * * factorial(2) (2 * factorial(1)) (2 * (1 * factorial(0))) (2 * (1 * 1)))

== 3 * 2 * 1 * 1

Try to write this function without using recursion and compare the two. Node:Simple Example With a Data Structure, Next:Advantages and Disadvantages of Recursion, Previous:Simple Example without a Data Structure, Up:Recursion

Simple Example With a Data Structure

A data structure earns the name recursive if its structure looks identical at every point within it. The simplest recursive structure is the linked list. At every point in a linked list, there are some data of identical type and one pointer to the next structure. The next simplest structure is the binary tree: this structure splits into two at every point. It has two pointers, one which branches left and one which branches to the right. Neither of these structures goes on for ever, so it seems reasonable to suppose that they might be handled easily using controlled recursive functions. is a function which releases the dynamic memory allocated to a linked list in one go. The problem it faces is this: if it deletes the first structure in the list, it will lose information about where the rest of the list is, because the pointer to the successor of a structure is held in its predecessor. It must therefore make a note of the pointer to the next structure in the list, before it deletes that structure, or it will never be able to get beyond the first structure in the list. The solution is to delete the list backwards from last to first using the following recursive routine.
deletetoend() /* structure definition */ struct list { /* some other data members */ struct list *succ; }; /**************************************************************/ struct list *deletetoend (ptr) struct list *ptr; { if (ptr != NULL) { deletetoend (ptr->succ); releasestruct (ptr); } return (NULL); } /**************************************************************/ releasestruct (ptr) struct list *ptr; { if (free((char *) ptr) != 0) { printf ("DEBUG [Z0/TktDtStrct] memory release failure\n"); } } /* release memory back to pool */

We supply a pointer to the place we would like the list to end. This need not be the very beginning: it could be any place in the list. The function then eliminates all structures after that point, up to the end of the list. It does assume that the programmer

has been careful to ensure that the end of the list is marked by a NULL pointer. This is the conventional way of denoting a pointer which does not point anywhere. If the pointer supplied is already NULL then this function does nothing. If it is not NULL then it executes the statements enclosed by the if braces. Notice that deletetoend() calls itself immediately, passing its successor in the list as a parameter. (ptr->succ) The function keeps doing this until it finds the end on the list. The very last-called deletetoend() then reaches the statement releasestruct() which frees the memory taken up by the last structure and hands it back to the free memory pool. That function consequently returns and allows the second-last deletetoend() to reach the releasestruct() statement, releasing the second last structure (which is now on the end of the list). This, in turn, returns and the process continues until the entire list has been deleted. The function returns the value NULL at each stage, so that when called, deletetoend() offers a very elegant way of deleting part or all of a linked list:
struct list *newlast; newlast->succ = deletetoend (newlast->succ); ptr = deletetoend (ptr);

newlast then becomes the new end of the list, and its successor is NULLified in a single statement. Node:Advantages and Disadvantages of Recursion, Next:Recursion and Global Variables, Previous:Simple Example With a Data Structure, Up:Recursion

Advantages and Disadvantages of Recursion

Why should programmers want to clutter up programs with techniques as mind boggling as recursion at all? The great advantage of recursion is that it makes functions very simple and allows them to behave just like the thing they are attempting to model. Unfortunately there are few situations in which recursion can be employed in a practical way. The major disadvantage of recursion is the amount of memory required to make it work: do not forget that the program stack grows each time a function call is made. If a recursive function buried itself a thousand levels deep, a program would almost certainly run out of memory. There is also the slight danger that a recursive function will go out of control if a program contains bugs. Node:Recursion and Global Variables, Next:Questions 28, Previous:Advantages and Disadvantages of Recursion, Up:Recursion

Recursion and Global Variables

Global variables and recursion do not mix well. Most recursive routines only work because they are sealed capsules and what goes on inside them can never affect the outside world. The only time that recursive functions should attempt to alter global storage is when the function concerned operates on a global data structure, as in the example above. To appreciate the danger, consider a recursive function, in which a second function alterGLOBAL() accidentally alters the value of GLOBAL in the middle of the function:

int GLOBAL = -2; recursion () { if (++GLOBAL == 0) { return (0); } alterGLOBAL(); recursion(); } /* another function which alters GLOBAL */

This function is treading a fine line between safety and digging its own recursive grave. If alterGLOBAL() makes GLOBAL more negative, as fast as ++ can make it more positive then GLOBAL will never be able to satisfy the condition of being zero and it will go on making recursive calls, never returning. If alterGLOBAL() makes the mistake of setting GLOBAL to a positive value, then the ++ operator in recursion() can only make GLOBAL larger and it will never be able to satisfy the condition that GLOBAL == 0 and so again the function would never be able to return. The stack would fill up the memory and the program would plunge down an unending recursive well. If global variables and parameters are used instead, this difficulty can be controlled much more easily. alterGLOBAL() cannot alter a variable in recursion() by accident, if only local variables are used, because it only works with its own local copies of parameters and variables which are locked away in a sealed capsule, out of harm's way. Node:Questions 28, Previous:Recursion and Global Variables, Up:Recursion

Questions
1. What is a recursive function? 2. What is a program "stack" and what is it for. 3. State the major disadvantage of recursion. Node:Example Programs chapter, Next:Errors and debugging, Previous:Recursion, Up:Top

Example Programs
The aim of this section is to provide two substantial examples of C, which use the data structures described in section 28.

Statistical Data Handler: Listing stat: Variable Cross Referencer:

Node:Statistical Data Handler, Next:Listing stat, Previous:Example Programs chapter, Up:Example Programs chapter

Statistical Data Handler

The first program is a utility which allows the user to type sets of floating point data into an editor and to calculate the mean, standard deviation...and so on, of those data. The program is capable of loading and saving the data to disk, as well as being able to handle several sets of data at once. The editor works in insert or overwrite modes. The program is menu driven and its operation should be reasonably self explanatory, so it is presented with rather sparse documentation.

The editor: Insert Overwrite: Quitting section: Program listing stat:

Node:The editor, Next:Insert Overwrite, Previous:Statistical Data Handler, Up:Statistical Data Handler

The Editor
A simple machine independent editor is provided for entering data. The editor first asks the user whether the current number of sets of data is to be altered. The default value is zero so, when data are typed in for the first time, this should be set up, by responding Y for yes. Up to twenty independent sets of data can be used. This number is set at the start and it is held in the memory and saved to disk with data files. If the number of sets is reduced at any time, the top sets are cut off from the calculations, but they are not lost forever, provided the number is changed back to include them before they are saved to disk, since the number of sets is used as an upper bound in a for loop: it does not actually alter the memory. More sets can be added at any time by making this value larger. Node:Insert Overwrite, Next:Quitting section, Previous:The editor, Up:Statistical Data Handler

Insert/Overwrite
A project file can be edited in either insert mode or overwrite mode. Files which contain no data may only be edited insert mode. The editor senses this and selects the mode automatically. In insert mode the user is prompted for values. Type 0.0 in place of an entry to get out of this mode. In overwrite mode the user is offered each entry in turn. If a non digit character is typed in (such as a . (dot) or a - (dash) etc..) the value of an entry is not altered. However, if a new value is entered, the new value will replace the old one. By default, the values are offered in turn from 1 to the final value. However, on selecting overwrite mode, the user is prompted for a starting value, and the values are offered from the starting number to the end. This is to avoid the rather tedious process of working through all the entries which are not required in a system independent way. Node:Quitting section, Next:Program listing stat, Previous:Insert Overwrite, Up:Statistical Data Handler

Quitting Sections

When quitting sections in which the user is supposed to enter data, the convention is that typing a zero value (0.0 for a time, 0 in any other instance) is a signal to break out of a section. Typing 0.0 while editing in insert mode causes the editor to quit. Node:Program listing stat, Previous:Quitting section, Up:Statistical Data Handler

The Program Listing

The program includes three library files, which are used for the following purposes.
#include <stdio.h>

Standard IO eader file

#include <ctype.h> #include <math.h>

Contains character ID macros Includes math function declarations

The flow of program logic is most easily described by means of a program structure diagram. The diagram shows the structure of function calls within the program and this can be related to the listing. The general scheme of the program is this:

1. Various flags concerning the data structure are cleared. 2. A menu is printed and the program cycles through the menu options. 3. The editor determines the data group to be edited, updates the screen with the data in the current group and loops through insert or overtype editing until the user quits. 4. The analysis calls custom functions which scan through the data structure calculating the relevant quantities. 5. Various toolkits perform run of the mill activities.

The data structure of this program is an array of linked lists. The array provides the roots of several independent linked lists: one for each group of data. These linked lists are attended to by toolkit routines and by special functions such as over(). Node:Listing stat, Next:Variable Cross Referencer, Previous:Statistical Data Handler, Up:Example Programs chapter

Listing
/************************************************************/ /* */ /* Statistical Calculator */ /* */ /************************************************************/ #include <stdio.h> #include <ctype.h> #include <math.h> /***********************************************************/ /** Manifest Constants / Macros / Static Variables **/ /***********************************************************/ #define #define #define #define #define #define #define #define #define TRUE FALSE GRPS CAREFULLY FAST NOTZERO ENDMARK NOTENDMARK BIGNUM 1 0 20 /* No grps which can be handled */ 1 0 1 -1.1 0 1e300 /* list data */ /* project name */

int DATSETS = 0; short DATATHERE = FALSE; char *FSP = "..........................";

/**********************************************************/ /** STRUCTURES **/ /**********************************************************/ struct list { double value; struct list *succ; }; struct Vlist { struct list *datptr; int datathere; } Data[GRPS]; /***********************************************************/ /** LEVEL 0 : Main Program **/ /***********************************************************/ main () { char getkey();

clrflags(); while (TRUE) { Menu(); switch (getkey()) { case '1' : edit(noofgroups()); break; case '2' : LoadSave(); break; case '3' : Analyse(); break; case 'q' : if (wantout(CAREFULLY)) quit(); } } } /************************************************************/ /** LEVEL 1 **/ /************************************************************/ clrflags() { short i; for (i=1; i<=GRPS; i++); { Data[i].datathere = FALSE; Data[i].datptr = NULL; } } /***********************************************************/ Menu () { CLRSCRN(); printf ("\nStatistical Calculator V1.0\n\n\n"); printf printf printf printf printf } ("1 : Edit Data Files\n\n"); ("2 : Project Files\n\n"); ("3 : Analyse Files\n\n"); ("q : Quit\n\n"); ("\nEnter Choice and RETURN : "); /* Initialize a virtual list */

/*************************************************************/ edit (no_grps) int no_grps; { char s,status(),getkey(); int i,stop = FALSE,ctr; void saveproject(); double over(),t,correct,getfloat(); struct list *ptr,*here,*eolist(), *install(),*startfrom(); while (TRUE) /* Edit a linked list */

{ i = whichgroup(); switch (s = status(i)) { case 'i': for (here = eolist(i,&ctr); TRUE; ctr++) { updatescrn (i,s); printf("%d:",ctr); if ((t = getfloat ()) == 0) break; here = install (here,t,i); } printf ("\n\nFile closed\n\n"); break; case 'o': for (ptr=startfrom(&ctr,i); ptr != NULL; ptr = ptr>succ) { if (ctr % 4 == 1) updatescrn (i,s); correct = over(ctr++,ptr->value); ptr->value = correct; } break;

case 's': saveproject(); break; case 'l': loadproject(); break; case 'q': stop = wantout(FAST); } if (stop) break; } } /************************************************************/ noofgroups () { char ch,getkey(); printf ("Project currently holds %d groups\n\n",DATSETS); printf ("Alter groups or Edit? (A/E)"); ch = getkey(); switch (tolower(ch)) { case 'a' : printf ("\nHow many groups for this file? (0..%d)\n\n",GRPS); return (DATSETS = getint(0,GRPS)); case 'e' : return (DATSETS); } /* Check no. of data groups */

/*************************************************************/ LoadSave () { char ch,getkey(); CLRSCRN(); /* Project options */

printf ("\nCurrent Project %s\n\n\n", FSP); printf ("Load new project or Save current one (L/S/Quit) ?\n\n"); ch = getkey(); switch (tolower(ch)) { case 'l' : if (sure()) { DATATHERE = loadproject (); } break; case 's' : if (sure()) { saveproject (); } case 'q' : } } /************************************************************/ Analyse () /* Work out some typical quantities */

{ char getkey(); double mean(), mn, millikan(); int i; printf ("Analysis of Data\n\n"); for (i = 1; i <= DATSETS; i++) { mn = mean(i); printf ("Mean value of group %2d : %f\n",i,mn); stddevs(mn); printf ("Millikan value %d %lg:\n",i,millikan(i)); NEWLINE(); } getkey(); } /************************************************************/ quit () { short i; struct list *deletetoend(); for (i = 0; i <= DATSETS; i++) { deletetoend (Data[i].datptr); } exit(0); } /************************************************************/ /* LEVEL 2 */ /************************************************************/ void saveproject () /* Quit program & tidy */

{ FILE *dfx; char *filename(),ch,getkey(); struct list *ptr; int i; if ((dfx = fopen (filename(),"w")) == 0) { printf ("Cannot write to file\nPress a key\n"); ch = getkey(); return; } fprintf (dfx,"%ld\n",DATSETS); for (i=1; i <= DATSETS; i++) { for (ptr = Data[i].datptr; ptr != NULL; ptr = ptr->succ) { fprintf (dfx,"%lf \n",ptr->value); } fprintf (dfx,"%f\n",ENDMARK); fprintf (dfx,"%d\n",Data[i].datathere); } while (fclose (dfx) != 0) { printf ("Waiting to close "); } blankline (); return; } /************************************************************/ loadproject () /* Load new list & delete old */

{ FILE *dfx; char *filename(),ch,getkey(); int r,i; double t = 1.0; struct list *ptr,*install(),*deletetoend(); if ((dfx = fopen(filename(),"r")) == NULL) { printf ("File cannot be read\nPress any key\n"); ch = getkey(); return (0); } fscanf (dfx,"%ld",&DATSETS); for (i = 1; i <= DATSETS; i++) { t = NOTENDMARK; Data[i].datptr = deletetoend(Data[i].datptr); Data[i].datathere = FALSE; for (ptr = Data[i].datptr; t != ENDMARK;) { fscanf (dfx,"%lf",&t); if (t != ENDMARK)

{ ptr = install (ptr,t,i); } } fscanf (dfx,"%ld",&r); Data[i].datathere = r; } while (fclose(dfx) != 0) { printf ("Waiting to close file"); } blankline(); return (TRUE); } /**********************************************************/ whichgroup () { int n = 0; printf ("\n\nEdit account number: "); n = getint (0,DATSETS); if (n == 0) { printf ("Quit!\n"); } return (n); } /***********************************************************/ char status (i) int i; { char stat; if (i==0) { stat = 'q'; } else { if (Data[i].datathere) { printf ("Insert/Overwrite/Load/Save/Quit?"); stat = getkey(); stat = tolower(stat); } else { stat = 'i'; } } return (stat); } /************************************************************/

updatescrn (grp,status) int grp; char status; { int ctr=0; struct list *ptr;

/* Update editor screen */

CLRSCRN(); printf ("\nStatistical Editor V1.0\n\n"); printf ("\nThis project file contains %d groups.\n",DATSETS); for (ptr = Data[grp].datptr; (ptr != NULL); ptr=ptr->succ) { if ((ctr % 3) == 0) NEWLINE(); printf (" (%2d) %12g ",ctr+1,(ptr->value)); ctr++; } printf ("\n\nEditing Group %d. Contains %d entries switch (tolower(status)) { case 'i' : printf ("INSERT MODE **\n"); break; case 'o' : printf ("OVERWRITE MODE **\n"); } NEWLINE(); } /**********************************************************/ double over (n,old) int n; double old; { double correct = 0; printf ("Entry %-2d : ",n); scanf("%lf",&correct); skipgarb(); if (correct == 0) { return (old); } else { return(correct); } } /************************************************************/ double mean (i) int i; { struct list *ptr; double sum; int num; /* find mean average */ /* Edit overtype mode */ ** ",grp,ctr);

sum = num = 0; for (ptr = Data[i].datptr; ptr != NULL; ptr=ptr->succ) { sum += ptr->value; num ++; } return (sum/num); } /**************************************************************/ stddevs (mean,i) double mean; int i; { double sum,num,var; struct list *ptr; sum = num = 0; for (ptr = Data[i].datptr; ptr != NULL; ptr=ptr->succ) { sum += (ptr->value - mean) * (ptr->value - mean); num ++; } var = sum/num; /* "biased" value */ /* find variance/std deviation */

printf ("Variance %d = %f\n",i,var); printf ("Std deviation %d = %f\n",i,sqrt(var)); } /************************************************************/ double millikan (i) int i; { double temp,record = BIGNUM; struct list *ptr1,*ptr2; for (ptr1 = Data[i].datptr; ptr1 != NULL; ptr1 = ptr1->succ) { for (ptr2=Data[i].datptr; ptr2!=ptr1; ptr2=ptr2->succ) { temp = (ptr1->value) - (ptr2->value); if (ABS(temp) < record) { record = ABS(temp); } } } return(record); } /************************************************************/ /* LEVEL 3 */ /************************************************************/ /* smallest diffnce between 2 data */

char *filename () { do { printf ("Enter filename : "); scanf ("%s",FSP); skipgarb(); } while (strlen(FSP) == 0); return (FSP); } /************************************************************/ /* Toolkit data structure */ /************************************************************/ struct list *eolist(i,c) int i,*c; { struct list *ptr,*p = NULL; *c = 1; for (ptr = Data[i].datptr; ptr != NULL; ptr = ptr->succ) { ++(*c); p = ptr; } return (p); } /*************************************************************/ struct list *startfrom (ctr,i) int *ctr,i; { struct list *ptr,*p = NULL; int j = 0; printf ("Overtype starting from which entry"); *ctr = getint(1,99); for (ptr=Data[i].datptr; (ptr != NULL) && (j++ != *ctr); ptr=ptr>succ) { p = ptr; } return (p); } /*************************************************************/ struct list *install (ptr,t,i) struct list *ptr; double t; int i; /* install item at thispos */ /* Find ith node in list */ /* Seek end of a linked Vlist */

{ struct list *thispos, *newstruct(); if ((thispos = newstruct()) == NULL) { warning(); printf ("DEBUG **: Free memory pool is empty"); exit(0); } if (!Data[i].datathere) { Data[i].datptr = thispos; Data[i].datathere = TRUE; } else { ptr->succ = thispos; } thispos->value = t; thispos->succ = NULL; return (thispos); } /************************************************************/ struct list *deletetoend (ptr) struct list *ptr; { if (ptr != NULL) { deletetoend (ptr->succ); releasestruct (ptr); } return (NULL); } /************************************************************/ struct list *newstruct () /* Allocate space for new item */ { char *malloc(); return ((struct list *) malloc(sizeof(struct list))); } /***********************************************************/ releasestruct (ptr) struct list *ptr; { if (free((char *) ptr) != 0) { printf ("DEBUG [Z0/TktDtStrct] memory release faliure\n"); } } /* release memory back to pool */ /* RECURSIVE WELL - returns NULL for easy deletion of call ptr */

/********************************************************/ /* Toolkit CONSOLE Output */ /********************************************************/ CLRSCRN () { printf ("\f"); } /*********************************************************/ newline () { printf ("\n"); } /**********************************************************/ blankline () { printf (" } \r");

/**********************************************************/ warning () { putchar('\7'); } /***********************************************************/ /*** Toolkit CONSOLE Input **/ /***********************************************************/ wantout (becareful) int becareful; { if (becareful) { printf ("Really quit? (Y/N)\n"); if (yes()) return (TRUE); else return (FALSE); } return (TRUE); } /*************************************************************/ sure (becareful) int becareful; { if (becareful) { printf ("Are you sure? (Y/N)\n"); if (yes()) return (TRUE); else return (FALSE); } /* Are you sure : boolean */ /* Exit from a section */

return (TRUE); } /***********************************************************/ yes () /* boolean response Y/N query */

{ while (TRUE) { switch (getkey()) { case 'y' : case 'Y' : return (TRUE); case 'n' : case 'N' : return (FALSE); } } } /***********************************************************/ char getkey () { char ch; scanf ("%c",&ch); skipgarb(); return (ch); } /***********************************************************/ getint (a,b) int a,b; { int p, i = a - 1; /* return int between a and b */ /* get single character */

for (p=0; ((a > i) || (i > b)); p++) { printf ("?"); scanf ("%d",&i); if (p > 3) { skipgarb(); p = 0; } } skipgarb(); return (i); } /***********************************************************/ double getfloat () { double x = 0; printf ("? "); scanf ("%lf",&x); skipgarb(); return (x); /* return long float */

} /************************************************************/ skipgarb() /* Skip input garbage corrupting scanf */

{ while (getchar() != '\n'); } /* end */

Node:Variable Cross Referencer, Previous:Listing stat, Up:Example Programs chapter

Variable Cross Referencer

A variable cross referencer is a utility which produces a list of all the identifiers in a C program (variables, macros, functions...) and lists the line numbers of those identifiers within the source file. This is sometimes useful for finding errors and for spotting variables, functions and macros which are never used, since they show up clearly as identifiers which have only a single reference. The program is listed here, with its line numbers, and its output (applied to itself) is supplied afterwards for reference. The structure diagram illustrates the operation of the program.

Listing Cref.c: Output of Cross Referencer: Commonts on cref.c:

Node:Listing Cref.c, Next:Output of Cross Referencer, Previous:Variable Cross Referencer, Up:Variable Cross Referencer

Listing Cref.c
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 /********************************************************/ /* */ /* C programming utility : variable referencer */ /* */ /********************************************************/ /* See notes above */ #include <stdio.h> #include <ctype.h> #define #define #define #define #define #define int char char char TRUE FALSE DUMMY MAXSTR MAXIDSIZE WORDTABLE 1 0 0 512 32 33 /* /* /* /* Contains line no. in file Input BUFFER for IDs Current input character macro/pointer flag */ */ */ */

LINECOUNT = 1; BUFFER[MAXIDSIZE]; CH; SPECIALCHAR;

/**********************************************************/ /* TABLE */ /**********************************************************/ char *WORDTABLE [WORDTABLE] = { "auto" , "break" , "case" , "char" , "const", "continue", "default" , "do" , "double" , "else" , "entry" , "enum" , "extern" , "float" , "for" , "goto" , "if" , "int" , "long" , "register", "return" , "short" , /* Table of resvd words */

53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115

"signed" , "sizeof" , "static" , "struct" , "switch" , "typedef" , "union" , "unsigned", "void" , "volatile", "while" , }; /********************************************************/ /** STRUCTURES **/ /********************************************************/ struct heap { short num; char spec; struct heap *next; }; /**********************************************************/ struct BinaryTree { char *name; struct heap *line; struct BinaryTree *left; struct BinaryTree *right; } *tree = NULL; /**********************************************************/ /* LEVEL 0 : main program */ /**********************************************************/ main () { FILE *fp; char *filename(); struct BinaryTree *CloseDataStruct(); printf ("\nIdentifier Cross Reference V 1.0\n\n"); if ((fp = fopen (filename(),"r")) == NULL) { printf ("Can't read file .. Aborted!\n\n"); exit(0); } CH = getc(fp); while (!feof(fp)) { SkipBlanks (fp); RecordWord (fp); } listIDs (tree);

116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178

CloseDataStruct(tree); printf ("\n%d lines in source file\n",LINECOUNT); } /**********************************************************/ /* LEVEL 1 */ /**********************************************************/ SkipBlanks (fp) FILE *fp; { while (!feof(fp)) { if (iscsymf(CH)) { return(DUMMY); } else { ParticularSkip(fp); } } /* Skip irrelevant characters */

/**********************************************************/ RecordWord (fp) FILE *fp; { int tok; CopyNextID (fp); if ((tok = token()) == 0) /* if not resved word */ { RecordUserID(isfunction(fp)); } SPECIALCHAR = ' '; } /**********************************************************/ listIDs (p) struct BinaryTree *p; { struct heap *h; int i = 0; if (p != NULL) { listIDs (p->left); printf ("\n%-20s",p->name); for (h = p->line; (h != NULL); h = h->next) { printf ("%c%-5d",h->spec,h->num); /* List Binary Tree */ /* get ID in buffer & tube it to data */

179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241

if ((++i % 8) == 0) { printf ("\n } } printf ("\n"); listIDs (p->right); } }

");

/*********************************************************/ struct BinaryTree *CloseDataStruct (p) struct BinaryTree *p; { if (p->left != NULL) { CloseDataStruct(p->left); } else if (p->right != NULL) { CloseDataStruct(p->right); } deleteheap(p->line); releasetree(p); return (NULL); } /*********************************************************/ /* LEVEL 2 */ /*********************************************************/ ParticularSkip (fp) FILE *fp; { char c; switch (CH) { case '/' : if ((c = getc(fp)) == '*') { skipcomment (fp); } else { CH = c; return (DUMMY); } break; case '"' : if (skiptochar (fp,'"') > MAXSTR) { printf ("String too long or unterminated "); printf ("at line %d\n",LINECOUNT); exit (0); } break; /* handle particular characters */ /* Recursive! */

242 243 case '\'': if (skiptochar (fp,'\'') == 1) 244 { 245 if (CH=='\'') CH = getc(fp);; 246 } 247 break; 248 249 case '#' : skiptochar(fp,' '); 250 SPECIALCHAR = '#'; 251 break; 252 253 case '\n': ++LINECOUNT; 254 default : CH = getc(fp); 255 SPECIALCHAR = ' '; 256 } 257 } 258 259 /*********************************************************/ 260 261 CopyNextID (fp) /* Put next identifier into BUFFER */ 262 263 FILE *fp; 264 265 { int i = 0; 266 267 while (!feof(fp) && (iscsym (CH))) 268 { 269 BUFFER[i++] = CH; 270 CH = getc (fp); 271 } 272 273 BUFFER[i] = '\0'; 274 } 275 276 /**********************************************************/ 277 278 token () /* Token: pos in WORDTABLE */ 279 280 { int i; 281 282 for (i = 0; i < WORDTABLE; i++) 283 { 284 if (strcmp(&(BUFFER[0]),WORDTABLE[i]) == 0) 285 { 286 return(i); 287 } 288 } 289 return(0); 290 } 291 292 /*********************************************************/ 293 294 RecordUserID (fnflag) /* check ID type & install data */ 295 296 int fnflag; 297 298 { char *strcat(); 299 struct BinaryTree *install(); 300 301 if (fnflag) 302 { 303 strcat (BUFFER,"()"); 304 tree = install (tree);

305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 338 339 340 341 342 343 344 345 346 347 348 349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366 367

} else { tree = install (tree); } } /**********************************************************/ isfunction (fp) FILE *fp; { while(!feof(fp)) { if (!(CH == ' ' || CH == '\n')) { break; } else if (CH == '\n') { ++LINECOUNT; } CH = getc(fp); } if (CH == '(') { return (TRUE); } else { return (FALSE); } } /**********************************************************/ deleteheap (h) struct heap *h; { struct heap *temp = h; while (h!=NULL && temp!=NULL) { temp = h->next; releaseheap(h); h = temp; } } /**********************************************************/ /** LEVEL 3 **/ /**********************************************************/ skipcomment (fp) FILE *fp; { char cs = 'x'; /* skip to char after comment */ /* Release back to free memory pool */ /* returns TRUE if ID is a fn */

368 for (CH = getc(fp); !feof(fp); CH = getc(fp)) 369 { 370 switch (CH) 371 { 372 case '\n': ++LINECOUNT; 373 break; 374 case '/' : if (cs == '*') 375 { 376 CH = getc(fp); 377 return(DUMMY); 378 } 379 } 380 cs = CH; 381 } 382 } 383 384 /*********************************************************/ 385 386 skiptochar (fp,ch) /* skip to char after ch */ 387 388 FILE *fp; 389 char ch; 390 391 { int c=0; 392 393 while (((CH =getc(fp)) != ch) && !feof(fp)) 394 { 395 if (CH == '\n') 396 { 397 ++LINECOUNT; 398 } 399 c++; 400 } 401 402 CH = getc(fp); 403 return (c); 404 } 405 406 /*********************************************************/ 407 408 struct BinaryTree *install (p) /* install ID in tree */ 409 410 struct BinaryTree *p; 411 412 { struct heap *pushonheap(); 413 struct BinaryTree *newtree(); 414 char *stringin(); 415 int pos; 416 417 if (p == NULL) /* new word */ 418 { 419 p = newtree(); 420 p->name = stringin(BUFFER); 421 p->line = pushonheap (NULL); 422 p->left = NULL; 423 p->right = NULL; 424 return (p); 425 } 426 427 if ((pos = strcmp (BUFFER,p->name)) == 0) /* found word*/ 428 { 429 p->line = pushonheap(p->line); 430 return (p);

431 } 432 433 if (pos < 0) /* Trace down list */ 434 { 435 p->left = install(p->left); 436 } 437 else 438 { 439 p->right = install(p->right); 440 } 441 442 return (p); 443 } 444 445 /*********************************************************/ 446 /* LEVEL 4 */ 447 /*********************************************************/ 448 449 struct heap *pushonheap (h) /* push nxt ln no.to heap */ 450 451 struct heap *h; 452 453 { struct heap *hp,*newheap(); 454 455 hp = newheap(); 456 hp->num = LINECOUNT; 457 hp->spec = SPECIALCHAR; 458 hp->next = h; 459 460 return (hp); 461 } 462 463 /*********************************************************/ 464 /* TOOLKIT file input */ 465 /*********************************************************/ 466 467 backone (ch,fp) /* backspace one in file */ 468 469 char ch; 470 FILE *fp; 471 472 { 473 if (ungetc(ch,fp) != ch) 474 { 475 printf ("\nDebug: Toolkit file input: backone() FAILED\n"); 476 exit(0); 477 } 478 } 479 480 /**********************************************************/ 481 /* TOOLKIT stdin */ 482 /**********************************************************/ 483 484 char *filename () 485 486 { static char *fsp = "................................."; 487 488 do 489 { 490 printf ("Enter filename of source program: "); 491 scanf ("%33s",fsp); 492 skipgarb ();

493 } 494 while (strlen(fsp) == 0); 495 return (fsp); 496 } 497 498 /*********************************************************/ 499 500 skipgarb () /* skip garbage upto end of line */ 501 502 { 503 while (getchar() != '\n'); 504 } 505 506 /**********************************************************/ 507 /* TOOLKIT data structure */ 508 /**********************************************************/ 509 510 char *stringin (array) /* cpy str in arry to ptr loc*/ 511 512 char *array; 513 514 { char *malloc(),*ptr; 515 int i; 516 517 ptr = malloc (strlen(array)+1); 518 for (i = 0; array[i] != '\0'; ptr[i] = array[i++]); 519 ptr[i] = '\0'; 520 return(ptr); 521 } 522 523 /**********************************************************/ 524 525 struct heap *newheap () 526 527 { char *malloc (); 528 return ((struct heap *) malloc(sizeof(struct heap))); 529 } 530 531 /**********************************************************/ 532 533 struct BinaryTree *newtree () 534 535 { char *malloc (); 536 return ((struct BinaryTree *) malloc(sizeof(struct BinaryTree))); 537 } 538 539 /*********************************************************/ 540 541 releaseheap (ptr) 542 543 struct heap *ptr; 544 545 { 546 if (free((char *) ptr) != 0) 547 548 { 549 printf ("TOOLKIT datastruct: link release failed\n"); 550 } 551 } 552 553 /**********************************************************/ 554

555 556 557 558 559 560 561 562 563 564 565 566 567 568

releasetree (ptr) struct BinaryTree *ptr; { if (free((char *) ptr) != 0) { printf ("TOOLKIT datastruct: link release failed\n"); } /* end */

Node:Output of Cross Referencer, Next:Commonts on cref.c, Previous:Listing Cref.c, Up:Variable Cross Referencer

Output of Cross Referencer

Identifier Cross Reference V 1.0 Enter filename of source program: Cref.c 568 BUFFER BinaryTree 427 557 194 402 332 267 107 203 261 126 LINECOUNT NULL 201 ParticularSkip() RecordUserID() RecordWord() SPECIALCHAR SkipBlanks() 19 470 97 456 423 197 215 294 146 457 124 420 536 192 395 329 254 21 199 152 388 397 422 176 139 156 112 255 111 250 159 22 364 372 421 171 316 327 417 102 263 253 350 89 217 238 350 148 117 208 303 536 166 393 325 245 192 284 533 99 380 321 245 116 273 413 86 376 321 230 99 269 410 85 370 270 221 20 408

299 82 368 269 133

CH

368

CloseDataStruct() CopyNextID() FILE

WORDTABLE WORDTABLE array backone() c ch cs deleteheap() DUMMY exit() FALSE feof() filename() fnflag fopen() fp 386 329 261 224 139 107

284 282 518 467 403 473 380 344 377 476 338 393 484 301 102 473 376 319 254 217 130 102 560 495 254 402 245 503 348 176 458 346 176 543 348 460 284 265 519 282 179 439

28 28 518 #17 517 512 510

399 473 374 206 231 239 #13 368 102 296

391 469 366

230 467

224 393

219 389 386

135 105

#14

319 98 294

267

130

109

470 368 316 249 215 126 97 546 494 393 224

467 368 314 245 156 124

402 368 270 243 152 112

393 364 267 235 148 111

393 362 263 226 146 109

388

free() fsp getc() getchar() h

491 376 107

486 368 368 329 270

451 344 168 528 346 458 518 282 169 435

449 178 528 168 457 518 282 408

354 178 525 84 456 518 280 308

353 176 453 75 455 518 273 304

352 176 451 72 453 515 269 299

350

heap 412 hp i

449

286

install()

iscsym() iscsymf() isfunction() left line listIDs() main() malloc() MAXIDSIZE MAXSTR name newheap() newtree() next num p 429 419 201 174 pos printf() 181 ptr pushonheap() releaseheap() releasetree() right scanf() skipcomment() skipgarb() skiptochar() 519

267 133 314 435 429 186 95 536 20 235 427 525 533 458 456 442 427 417 199 173 433 563 178 560 518 449 541 555 439 491 362 500 386 226 492 249 243 235 535 #16 #15 420 455 419 352 178 439 424 410 197 171 427 549 174 557 517 429 353 207 439 423 203 201 186 86 174 453 413 176 73 439 423 408 194 166 415 490 117 555 514 421 475 104 546 412 238 101 543 237 541 185 520 435 422 207 192 164 435 421 206 186 430 420 203 176 429 75 83 528 527 517 514 156 435 429 173 422 421 164 199 206 115 197 176 173 84 85

spec strcat() strcmp() stringin() strlen() temp tok token() tree TRUE ungetc()

457 303 427 510 517 354 154 278 308 334 473

178 298 284 420 494 352 150 154 308 #12

74

414

350

348

304

304

116

115

89

568 lines in source file

Node:Commonts on cref.c, Previous:Output of Cross Referencer, Up:Variable Cross Referencer

Comments
This simplified program could be improved in a number of ways. Here are some suggestions for improvement:

The program could determine whether an identifier was of type pointer or not and, if so, label the line number with a *, e.g. *123 342 *1234 At present the program only marks macros with a # symbol on the line at which they are defined. It could be made to mark them at every line, so that #undefined symbols and variables were clearly distinguished.

Node:Errors and debugging, Next:Summary, Previous:Example Programs chapter, Up:Top

Errors and debugging

Mistakes! Debugging can be a difficult process. In many cases compiler errors are not generated because the actual error which was present but because the compiler got out of step. Often the error messages give a completely misleading impression of what has gone wrong. It is useful therefore to build a list of errors and probable causes personally. These few examples here should help beginners get started and perhaps give some insight into the way C works.

Compiler Trappable Errors: Run time errors: Tracing errors: Pathological Problems: Porting Programs between computers: Qu:

Node:Compiler Trappable Errors, Next:Run time errors, Previous:Errors and debugging, Up:Errors and debugging

Compiler Trappable Errors

Missing semicolon;: Missing brace: Mistyping Upper Lower Case: Missing inv comma: Variable not declared or scope wrong: Using a function or assignment inside a macro: Forgetting to declare a function which is not type int: Type mismatch in expressions:

Node:Missing semicolon;, Next:Missing brace, Previous:Compiler Trappable Errors, Up:Compiler Trappable Errors

Missing semicolon;
A missing semicolon is easily trapped by the compiler. Every statement must end with a semi colon. A compound statement which is held in curly braces seldom needs a semi colon to follow.
but: statement; { }; <-- This semi colon is only needed if the curly braces enclose a type declaration or an initializer for static array/structure etc.

Node:Missing brace, Next:Mistyping Upper Lower Case, Previous:Missing semicolon;, Up:Compiler Trappable Errors

Missing closing brace }

This error is harder to spot and may cause a whole host of irrelevant and incorrect errors after the missing brace. Count braces carefully. One way to avoid this is to always fill braces in before the statements are written inside them. So write
{ }

and fill in the statements afterwards. Often this error will generate a message like `unexpected end of file' because it is particularly difficult for a compiler to diagnose.

Node:Mistyping Upper Lower Case, Next:Missing inv comma, Previous:Missing brace, Up:Compiler Trappable Errors

Mistyping Upper/Lower Case

C distinguishes between small and capital letters. If a program fails at the linking stage because it has found a reference to a function which had not been defined, this is often the cause. Node:Missing inv comma, Next:Variable not declared or scope wrong, Previous:Mistyping Upper Lower Case, Up:Compiler Trappable Errors

Missing quote "

If a quote is missed out of a statement containing a string then the compiler will usually signal this with a message like:
String too long or unterminated.

Node:Variable not declared or scope wrong, Next:Using a function or assignment inside a macro, Previous:Missing inv comma, Up:Compiler Trappable Errors

Variable not declared or scope wrong

This means that a variable is used which has not first been declared, or that a variable is used outside of its sealed capsule. Node:Using a function or assignment inside a macro, Next:Forgetting to declare a function which is not type int, Previous:Variable not declared or scope wrong, Up:Compiler Trappable Errors

Using a function or assignment inside a macro

If abs (x) is a macro and not a function then the following are incorrect:
abs (function()); abs (x = function());

Only a single variable can be substituted into a macro. This error might generate something like "lvalue required". Node:Forgetting to declare a function which is not type int, Next:Type mismatch in expressions, Previous:Using a function or assignment inside a macro, Up:Compiler Trappable Errors

Forgetting to declare a function which is not type int

All functions return values of int by default. If it is required that they return another type of variable, this must by declared in two places: a) in the function which calls the new function, along with the other declarations:

CallFunction () { char ch, function1(), *function2(); }

The function1() is type char; function2() is type pointer to char. This must also be declared where the function is defined:
char function1 () { }

and
char *function2() { }

This error might result in the message "type mismatch" or "external variable/function type/attribute mismatch" Node:Type mismatch in expressions, Previous:Forgetting to declare a function which is not type int, Up:Compiler Trappable Errors

Type mismatch in expressions

There is a rule in C that all maths operations have to be performed with long variables. These are
int long int double long float

The result is also a long type. If the user forgets this and tries to use short C automatically converts it into long form. The result cannot therefore be assigned to a short type afterwards or the compiler will complain that there is a type mismatch. So the following is wrong:
short i,j = 2; i = j * 2;

If a short result is required, the cast operator has to be used to cast the long result to be a short one.
short i,j = 2; i = (short) j * 2;

Node:Run time errors, Next:Tracing errors, Previous:Compiler Trappable Errors, Up:Errors and debugging

Errors not trappable by a compiler (run time errors)

Confusion of = and ==: Missing & in scanf: Confusing C++ and ++C: Unwarranted assumptions about storage of arrays/structures: Number of actual and formal parameters does not match: Conversion string in scanf or printf wrong: Accidental confusion of int short and char: Arrays out of bounds: Mathematical Error: Uncoordinated Output using put/get I/O: Global Variables and Recursion:

Node:Confusion of = and ==, Next:Missing & in scanf, Previous:Run time errors, Up:Run time errors

Confusion of = and ==
A statement such as:
if (a = 0) { }

is valid C, but notice that = is the assignment operator and not the equality operator ==. It is legal to put an assignment inside the if statement (or any other function) and the value of the assignment is the value being assigned! So writing the above would always give the result zero (which is `FALSE' in C) so the contents of the braces {} would never be executed. To compare a to zero the correct syntax is:
if (a == 0) { }

Node:Missing & in scanf, Next:Confusing C++ and ++C, Previous:Confusion of = and ==, Up:Run time errors

Missing & in scanf

This error can often be trapped by a compiler, but not in all cases. The arguments of the scanf statement must be pointers or addresses of variables, not the contents of the variables themselves. Thus the following is wrong:
int i; char ch; scanf ("%c %d",ch,i);

and should read:

int i; char; scanf ("%c %d", &ch, &i);

Notice however that the & is not always needed if the identifier in the expression is already a pointer. The following is correct:
int *i; char *ch; scanf ("%c %d", ch, i);

Including the & now would be wrong. If this error is trappable then it will be something like "Variable is not a pointer". Node:Confusing C++ and ++C, Next:Unwarranted assumptions about storage of arrays/structures, Previous:Missing & in scanf, Up:Run time errors

Confusing C++ and ++C

In many cases these two forms are identical. However, if they are hidden inside another statement e.g.
array [C++] = 0;

then there is a subtle difference. ++C causes C to be incremented by 1 before the assignment takes place whereas C++ causes C to be incremented by 1 after the assignment has taken place. So if you find that a program is out of step by 1, this could be the cause. Node:Unwarranted assumptions about storage of arrays/structures, Next:Number of actual and formal parameters does not match, Previous:Confusing C++ and ++C, Up:Run time errors

Unwarranted assumptions about storage

C stores arrays in rows, and as far as the language is concerned the storage locations are next to one another in one place up to the end of the array. This might not be exactly true, in general. A program will be loaded into one or more areas (where ever the operating system can find space) and new variable space will be found wherever it is available, but this will not generally be in whole blocks `side by side' in the memory. The following sort of construction only works for simple data types:
char array[10]; *array = 0; *(array + 1) = 0; ...

*(array + 10) = 0;

While it is true that the variable "array" used without its square brackets is a pointer to the first element of the array, it is not necessarily true that the array will necessarily be stored in this way. Using:
char array[10]; array[0] = 0; array[1] = 0; ... array[10] = 0;

is safe. When finding a pointer to, say, the third element, you should not assume that
array + 3 * sizeof (datatype)

will be the location. Use:

&(array[3])

Do not assume that the size of a structure is the sum of the sizes of its parts! There may be extra data inside for operating system use or for implementation reasons, like aligning variables with particular addresses. Node:Number of actual and formal parameters does not match, Next:Conversion string in scanf or printf wrong, Previous:Unwarranted assumptions about storage of arrays/structures, Up:Run time errors

The number of actual and formal parameters does not match

This problem can be avoided in ANSI C and C++ but not in K&R C. When passing values to a function the compiler will not spot whether you have the wrong number of parameters in a statement, provided they are all of the correct type. The values which are assumed for missing parameters cannot be guaranteed. They are probably garbage and will most likely spoil a program. Node:Conversion string in scanf or printf wrong, Next:Accidental confusion of int short and char, Previous:Number of actual and formal parameters does not match, Up:Run time errors

The conversion string in scanf/printf is wrong

Incorrect I/O is can be the result of poorly matched conversion strings in I/O statements. These are wrong:
float x; scanf ("%d",&x); should be float x; scanf ("%f",&x);

or even:
double x; scanf ("%f",&x); should perhaps be float x; scanf("%ld",&x);

Another effect which can occur if the conversion specifier is selected as being long when it the variable is really short is that neighbouring variables can receive the scanf values instead! For instance if two variables of the same type happen to be stored next to each other in the memory:
short i,j;

which might look like:

-------------------------------------| | | -------------------------------------i j

and the user tries to read into one with a long int value, scanf will store a long int value, which is the size of two of these short variables. Suppose the left hand box were i and the right hand box were j and you wanted to input the value of i: instead of getting:
| -------------------002345 | | -------------------i scanf j

might store

0000000000000002345

as
-----------------------| 000000000 | 0000002345 | -----------------------i j

because the value was long, but this would mean that the number would over flow out of i into j and in fact j might get the correct value and i would be set to zero!! Check the conversion specifiers!! Node:Accidental confusion of int short and char, Next:Arrays out of bounds, Previous:Conversion string in scanf or printf wrong, Up:Run time errors

Accidental confusion of int, short and char

Often when working with characters one also wants to know their ASCII values. If characters/integers are passed as parameters it is easy to mistype char for int etc.. The compiler probably won't notice this because no conversion is needed between int and char. Characters are stored by their ASCII values. On the other hand if the declaration is wrong:
function (ch) int (ch); { }

but the character is continually assumed to be a character by the program, a crashworthy routine might be the result. Node:Arrays out of bounds, Next:Mathematical Error, Previous:Accidental confusion of int short and char, Up:Run time errors

Arrays out of bounds

C does not check the limits of arrays. If an array is sized:
type array[5];

and the you allow the program to write to array[6] or more, C will not complain. However the computer might! In the worst case this could cause the program to crash. Node:Mathematical Error, Next:Uncoordinated Output using put/get I/O, Previous:Arrays out of bounds, Up:Run time errors

Mathematical Error
C does not necessarily signal mathematical errors. A program might continue regardless of the fact that a mathematical function failed. Some mathematical errors (often subtle ones) can be caused by forgetting to include to file math.h at the start of the program. Node:Uncoordinated Output using put/get I/O, Next:Global Variables and Recursion, Previous:Mathematical Error, Up:Run time errors

Uncoordinated Output using buffered I/O

Output which is generated by functions like putchar(), puts() is buffered. This means that it is not written to the screen until the buffer is either full or is specifically emptied. This results in strange effects such as programs which produce no output until all the input is complete (short programs) or spontaneous bursts of output at uncoordinated intervals. One cure is to terminate with a newline \n character which

flushes the buffers on each write operation. Special functions on some systems such as getch() may also suffer from this problem. Again the cure is to write:
printf ("\n"); ch = getch();

Node:Global Variables and Recursion, Previous:Uncoordinated Output using put/get I/O, Up:Run time errors

Global Variables and Recursion

Global variables and recursion should not be mixed. Most recursive routines work only because they are sealed capsules and what goes on inside them can never affect the outside world. The only time that recursive functions should alter global storage is when the function concerned operates on a global data structure. Consider a recursive function:
int GLOBAL; recursion () { if (++GLOBAL == 0) { return (0); } alterGLOBAL(); recursion(); } /* another function which alters GLOBAL */

This function is treading a fine line between safety and digging its own recursive grave. All it would take to crash the program, would be the careless use of GLOBAL in the function alterGLOBAL() and the function would never be able to return. The stack would fill up the memory and the program would plunge down an unending recursive well. Node:Tracing errors, Next:Pathological Problems, Previous:Run time errors, Up:Errors and debugging

Tracing Errors

Locating a problem:

Node:Locating a problem, Previous:Tracing errors, Up:Tracing errors

Locating a problem
Complex bugs can be difficult to locate. Here are some tips for fault finding: 1. Try to use local variables, in preference to global ones for local duties. Never rely on global variables for passing messages between functions. 2. Check variable declarations and missing parameters.

3. Check that a program has not run out of private memory. (If it repeatedly crashes for no apparent reason, this could be a cause.) Make the program stack size bigger if that is possible. 4. Use statements like printf("program is now here") to map out the progress of a program and to check that all function calls are made correctly. 5. Use statements like ch = getchar() to halt a program in certain places and to find out the exact location at which things go wrong. 6. Try "commenting out" lines of suspect code. In other words: put comment markers around lines that you would like to eliminate temporarily and then recompile to pinpoint errors. 7. Check that the compiler disk has not been corrupted (make a new copy) getting desperate now! 8. Try retyping the program, or using a filter which strips out any illegal characters which might have found their way into a program. 9. Get some sleep! Hope the problem has gone away in the morning. Failing these measures, try to find someone who programs in C regularly on the computer system concerned. Node:Pathological Problems, Next:Porting Programs between computers, Previous:Tracing errors, Up:Errors and debugging

Pathological Problems
Problems which defy reasonable explanations are called pathological or `sick'. Sometimes these will be the result of misconceptions about C functions, but occasionally they may be the result of compiler bugs, or operating system design peculiarities. Consider the following example which was encountered while writing the simple example in the chapter on Files and Devices, subsection `Low Level File Handling': in that program a seemingly innocent macro defined by
#define CLRSCRN() putchar('\f');

caused the C library functions creat() and remove() to fail is remarkable ways on an early Amiga C compiler! The problem was that a single call to CLRSCRN() at the start of the function DelFile() caused both of the library functions (in very different parts of the program) above to make recursing function calls the function DelFile(). The deletion of CLRSCRN() cured the problem entirely! In general it is worth checking carefully the names of all functions within a program to be sure that they do not infringe upon library functions. For example, read() and write() are names which everyone wishes to use at some point, but they are the names of standard library functions, so they may not be used. Even capitalizing (Read() / Write()) might not work: beware that special operating system libraries have not already reserved these words as library commands. It is almost impossible to advise about these errors. A programmer can only hope to try to eliminate all possibilities in homing in on the problem. To misquote Sherlock Holmes: "At the end of the day, when all else fails and the manuals are in the waste paper basket, the last possibility, however improbable, has to be the truth."

Node:Porting Programs between computers, Next:Qu, Previous:Pathological Problems, Up:Errors and debugging

Porting Programs between computers

Programs written according to the style guidelines described in this book should be highly portable. Nevertheless, there are almost inevitably problems in porting programs from one computer to another. The most likely area of incompatibility betwee compilers regards filing operations, especially scanf(). Programmers attempting to transfer programs between machines are recommended to look at all the scanf() statements first and to check all the conversion specifiers with a local compiler manual. scanf() is capable of producing a full spectrum of weird effects which have nothing to do with I/O. Here are some more potential problems to look out for:

Assumptions about the size of data objects such as int and float can be risky. Check conversion characters in printf() and scanf() as some compilers choose slightly different conventions for these. The stack size for (memory available to) a program is likely to vary between systems. This can cause errors at run time if a program runs out of space, even though there is nothing wrong with the code. Check for functions which rely on the speed of a particular computer. For example, pause() or wait loops. Some computers may scarcely notice counting to 50000, whereas others may labour at it for some time! Check for assumptions made about filenames. e.g. limited/unlimited size, valid characters etc..

Node:Qu, Previous:Porting Programs between computers, Up:Errors and debugging

Questions
Spot the errors in the following:
1. 2. 3. 4. 5. 6. 7. 8. function (string,i) { char *string; int i; }

9.
10. while (a < b) 11. { 12. while (b == 0) 13. { 14. printf ("a is negative"); 15. } 16.

17.

18. struct Name 19. { 20. int member1; 21. int member2;

22. 23.

24. Node:Summary, Next:reserved words list, Previous:Errors and debugging, Up:Top

Summary of C

Reserved: Preprocessor Directives: Header Files and Libraries: Constants: Primitive Data Types: Storage Classes: Identifiers: Statements: Character Utilities: Special Control Characters: Input/Output Functions: print conversions: scanf conversions: Maths Library Summary: goto:

Node:Reserved, Next:Preprocessor Directives, Previous:Summary, Up:Summary

Reserved Words
auto

storage class specifier (declaration)

break case char continue default do double else entry extern float for

statement (escape from switch or loop) option prefix within switch statement typename statement (branch to start of next loop) option in switch statement statement typename statement (reserved for the future use) storage class specifier typename

goto if int long

statement goto label statement typename

register return short sizeof static struct switch

typename storage class specifier functional statement typename compile time operator storage class specifier partial typename statement

typedef union

statement partial typename

unsigned while enum void const signed

typename statement partial typename: ordinal types only typename storage class specifier(no storage allocated) typename

volatile

storage class specifier Node:Preprocessor Directives, Next:Header Files and Libraries, Previous:Reserved, Up:Summary

Preprocessor Directives
#include #define #undef #if #ifdef

include file for linking define a preprocessor symbol/macro un-define a previously defnined symbol test for conditional compilation (ditto)

#ifndef

(ditto)
#else #endif #line #error

(ditto) (ditto) debug tool

debug tool Node:Header Files and Libraries, Next:Constants, Previous:Preprocessor Directives, Up:Summary

Header Files and Libraries

Header files contain macro definitions, type definitions and variable/ function declarations which are used in connection with standard libraries. They supplement the object code libraries which are linked at compile time for standard library functions. Some library facilities are not available unless header files are included. Typical names for header files are:
stdio.h

Standard I/O (libc).

ctype.h math.h

Macro for character types.

Mathematical definitions (libm) Node:Constants, Next:Primitive Data Types, Previous:Header Files and Libraries, Up:Summary

Constants
Integer Characters 0..9 only Octal Prefix 0 (zero) chars 0..7 only Hexadecimal Prefix 0x (zero ex) chars a..f A..f 0..9 Explicit Long Integer/Octal or Hexadecimal types can be declared long by writing L immediately after the constant. Character Declared in single quotes e.g. 'x' '\n' Float Characters 0..0 and one "." May also use scientific notation exponents with e or E preceding them. e.g. 2.14E12 3.2e-2 Strings String constants are written in double quotes e.g. "This is a string" and have type pointer to character. Node:Primitive Data Types, Next:Storage Classes, Previous:Constants, Up:Summary

Primitive Data Types

char

Holds any character

int short int long int float

Integer type Integer no larger than int Integer no smaller than int Floating point (real number)

long float double void

Double precision float (ditto)

Holds no value, uses no storage (except as a pointer) Node:Storage Classes, Next:Identifiers, Previous:Primitive Data Types, Up:Summary

Storage Classes
auto const extern static

Local variable (redundant keyword) No variable allocated, value doesn't change Variable is defined in another file Value is preserved between function calls

register volatile

Stored in a register, if possible

Value can be changed by agents outside the program. Node:Identifiers, Next:Statements, Previous:Storage Classes, Up:Summary

Identifiers
Idenitifiers may contain the characters: 0..9, A..Z, a..z and _ (the underscore character). Identifiers may not begin with a number. (The compiler assumes that an object beginning with a number is a number.) Node:Statements, Next:Character Utilities, Previous:Identifiers, Up:Summary

Statements
A single statement is any valid string in C which ends with a semi colon. e.g.
a = 6; printf ("I love C because...");

A compound statement is any number of single statements groued together in curly braces. The curly braces do not end with a semi colon and stand in place of a single statement. Any pair of curly braces may contain local declarations after the opening brace. e.g.
{ a = 6; } { int a; a = 6; printf ("I love C because..."); }

Summary of Operators and Precedence The highest priority operators are listed first.
Operator () [] ++ -(type) * & ~ ! * / % + >> << > >= <= < == != & ^ | && || = += -= Operation parentheses square brackets increment decrement cast operator the contents of the address of unary minus one's complement logical NOT multiply divide remainder (MOD) add subtract shift right shift left is greater than greater than or equal to less than or equal to less than is equal to is not equal to bitwise bitwise bitwsie logical logical AND exclusive OR includive OR AND OR Evaluated left to right left to right right right right right right right right right to to to to to to to to left left left left left left left left

left to right left to right left to right left to right left to right left to right left to right left left left left to to to to right right right right

left to right left to right left left left left left to to to to to right right right right right

assign add assign subtract assign

right to left right to left right to left

*= /= %= >>= <<= &= ^= |=

multiply assign divide assign remainder assign right shift assign left shift assign AND assign exclusive OR assign inclusive OR assign

right right right right right right right right

to to to to to to to to

left left left left left left left left

Node:Character Utilities, Next:Special Control Characters, Previous:Statements, Up:Summary

Character Utilities
char ch; isalpha(ch)

Is alphabetic a..z A..Z

isupper(ch) islower(ch) isdigit(ch) isxdigit(ch) isspace(ch) ispunct(ch) isalnum(ch) isprint(ch) isgraph(ch) iscntrl(ch) isascii(ch) iscsym(ch)

Is upper case Is lower case Is in the range 0..9 Is 0..9 or a..f or A..F Is white space character (space/newline/tab) Is punctuation or symbolic Is alphanumeric (alphavetic or number) Is printable on the screen (and space) If the character is printable (not space) Is a control character (not printable) Is in the range 0..127 Is a valid character for a C identifier

toupper(ch) tolower(ch) toascii(ch)

Converts character to upper case Converts character to lower case

Converts character to ascii (masks off top bit) Node:Special Control Characters, Next:Input/Output Functions, Previous:Character Utilities, Up:Summary

Special Control Characters

Control characters are invisible on the screen. They have special purposes usually to do with cursor movement and are written into an ordinary string or character by typing

a backslash character \ followed by some other character. These characters are listed below.
\b

backspace BS
\f \n \r \t \v \" \' \\ \ddd

form feed FF (also clear screen) new line NL (like pressing return) carriage return CR (cursor to start of line) horizontal tab HT vertical tab (not all versions) double quotes (not all versions) single quote character ' backslash character \

character ddd where ddd is an ASCII code given in octal or base 8. (See Appendix C) Node:Input/Output Functions, Next:print conversions, Previous:Special Control Characters, Up:Summary

Input/Output Functions
printf () scanf ()

Formatted printing Formatted input analysis

getchar() putchar() gets () puts ()

Get one character from stdin file buffer Put one charcter in stdout file buffer Get a string from stdin Put a string in stdout

fprintf() fscanf() fgets() fputs() fopen() fclose() getc() ungetc();

Formatted printing to general files Formatted input from general files Get a string from a file Put a string in a file Open/create a file for high level access Close a file opened by fopen() Get one character from a file (macro?)

putc()

Undo last get operation Put a character to a file (macro?)

fgetc() fputc() feof() fread()

Get a character from a file (function) Put a character from a file (function) End of file . returns true or false Read a block of characters

fwrite() ftell() fseek() rewind() fflush() open()

Write a block of characters Returns file position Finds a file position Moves file position to the start of file Empties file buffers Open a file for low level use

close() creat() read() write()

Close a file opened with open() Create a new file Read a block of untranslated bytes Write a block of untranslated bytes

rename() unlink() remove() lseek()

Rename a file Delete a file Delete a file

Find file position Node:print conversions, Next:scanf conversions, Previous:Input/Output Functions, Up:Summary
printf d

conversion specifiers
signed denary integer

u x o s c

Unsigned denary integer Hexadecimal integer Octal integer String Single character

Fixed decimal floating point

e g

Scientific notation floating point Use f or e, whichever is shorter

The letter l (ell) can be prefixed before these for long types. Node:scanf conversions, Next:Maths Library Summary, Previous:print conversions, Up:Summary
scanf

conversion specifers

The conversion characters for scanf are not identical to those for printf and it is important to distinguish the long types here.
d

Denary integer
ld x o h f lf e le c s

Long int Hexadecimal integer Octal integer Short integer Float type Long float or double Float type Double Single character

Character string Node:Maths Library Summary, Next:goto, Previous:scanf conversions, Up:Summary

Maths Library
These functions require double parameters and return double values unless otherwise stated. It is important to include math.h.
ABS(x)

Return absolute (unsigned) value. (macro)

fabs(x) ceil(x)

Return absolute (unsigned) value. (Function) Rounds up a "double" variable

floor(x)

Rounds down (truncates) a "double" variable.

exp(x)

Find exponent
log(x) log10(x) pow(x,y) sqrt(x) sin(x) cos(x) tan(x) asin(x) acos(x) atan(x)

Find natural logarithm Find logarithm to base 10 Raise x to the power y Square root Sine of (x in radians) Cosine of (x in radians) Tangent of (x in radians) Inverse sine of x in radians Inverse cosine of x in radians Inverse tangent of x in radians

atan2(x,y) sinh(x) cosh(x) tanh(x)

Inverse tangent of x/y in radians Hyperbolic sine Hyperbolic cosine

Hyperbolic tangent Node:goto, Previous:Maths Library Summary, Up:Summary

goto

This word is redundant in C and encourages poor programming style. For this reason it has been ignored in this book. For completeness, and for those who insist on using it (may their programs recover gracefully) the form of the goto statement is as follows:
goto label;

is an identifier which occurs somewhere else in the given function and is defined as a label by using the colon:
label label : printf ("Ugh! You used a goto!");

Node:reserved words list, Next:Comparisons, Previous:Summary, Up:Top

All the Reserved Words

Here is a list of all the reserved words in C. The set of reserved words above is used to build up the basic instructions of C; you can not use them in programs your write

Please note that this list is somewhat misleading. Many more words are out of bounds. This is because most of the facilities which C offers are in libraries that are included in programs. Once a library has been included in a program, its functions are defined and you cannot use their names yourself. C requires all of these reserved words to be in lower case. (This does mean that, typed in upper case, the reserved words could be used as variable names, but this is not recommended.) (A "d" by the word implies that it is used as part of a declaration.)
auto d break case char d continue default do double d else entry extern d float d for goto if int d long d register d return short d sizeof static d struct switch typedef d union d unsigned d while

also in modern implementations: enum d void d const d signed d volatile d

Node:Comparisons, Next:Character Conversion Table, Previous:reserved words list, Up:Top

Three Languages: Words and Symbols Compared

If you are already familiar with Pascal (Algol..etc) or BBC BASIC, the following table will give you a rough and ready indication of how the main words and symbols of the three languages relate.
C = == *,/ /,% printf (".."); Pascal := = *,/ div, mod writeln ('..'); write ('..'); BASIC = = *,/ DIV, MOD PRINT ".."

scanf ("..",a); for (x = ..;...;) { } while (..) { } do { } while (..); N/A if (..) ..; else ...; switch (..) { case : } /* .... */ * struct union

readln (a); read (a); for x := ...to begin end; while ...do begin end; N/A

INPUT a FOR x = ... NEXT x N/A

N/A

repeat until (..) if ... then ... else ....; case .. of end; { ..... } ^ record N/A

REPEAT UNTIL .. IF .. THEN.. ELSE N/A

REM ..... ? ! $ N/A N/A

The conditional expressions if and switch are essentially identical to Pascal's own words if and case but there is no redundant "then". BASIC has no analogue of the switch construction. The loop constructions of C are far superior to those of either BASIC or Pascal however. Input and Output in C is more flexible than Pascal, though correspondingly less robust in terms of program crashability. Input and Output in C can match all of BASICs string operations and provide more, though string variables can be more awkward to deal with. Node:Character Conversion Table, Next:Emacs style file, Previous:Comparisons, Up:Top

Character Conversion Table

This table lists the decimal, octal, and hexadecimal numbers for characters 0 - 127.
Decimal 0 1 2 3 Octal 0 1 2 3 Hexadecimal 0 1 2 3 Character CTRL-@ CTRL-A CTRL-B CTRL-C

4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66

4 5 6 7 10 11 12 13 14 15 16 17 20 21 22 23 24 25 26 27 30 31 32 33 34 35 36 37 40 41 42 43 44 45 46 47 50 51 52 53 54 55 56 57 60 61 62 63 64 65 66 67 70 71 72 73 74 75 76 77 100 101 102

4 5 6 7 8 9 A B C D E F 10 11 12 13 14 15 16 17 18 19 1A 1B 1C 1D 1E 1F 20 21 22 23 24 25 26 27 28 29 2A 2B 2C 2D 2E 2F 30 31 32 33 34 35 36 37 38 39 3A 3B 3C 3D 3E 3F 40 41 42

CTRL-D CTRL-E CTRL-F CTRL-G CTRL-H CTRL-I CTRL-J CTRL-K CTRL-L CTRL-M CTRL-N CTRL-O CTRL-P CTRL-Q CTRL-R CTRL-S CTRL-T CTRL-U CTRL-V CTRL-W CTRL-X CTRL-Y CTRL-Z CTRL-[ CTRL-\ CTRL-] CTRL-^ CTRL-_ ! " # $ % & ' ( ) * + , . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B

67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127

103 104 105 106 107 110 111 112 113 114 115 116 117 120 121 122 123 124 125 126 127 130 131 132 133 134 135 136 137 140 141 142 143 144 145 146 147 150 151 152 153 154 155 156 157 160 161 162 163 164 165 166 167 170 171 172 173 174 175 176 177

43 44 45 46 47 48 49 4A 4B 4C 4D 4E 4F 50 51 52 53 54 55 56 57 58 59 5A 5B 5C 5D 5E 5F 60 61 62 63 64 65 66 67 68 69 6A 6B 6C 6D 6E 6F 70 71 72 73 74 75 76 77 78 79 7A 7B 7C 7D 7E 7F

C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r s t u v w x y z { | } ~ DEL

Node:Emacs style file, Next:Answers to questions, Previous:Character Conversion Table, Up:Top

Emacs style file

The programming style used in this book can be taught to Emacs with the following site-lisp file:
;;; ;;; C, perl and C++ indentation, Burgess style. (Thomas Sevaldrud) ;;; (defconst burgess-c-style '((c-tab-always-indent (c-hanging-braces-alist after) (c-hanging-colons-alist . t) . ((substatement-open before (brace-list-open))) . ((member-init-intro before) (inher-intro) (case-label after) (label after) (access-label after))) . (scope-operator)) . ((arglist-close . c-lineup(defun-block-intro (substatement-open (statement-block-intro (topmost-intro (case-label (block-open (knr-argdecl-intro ;(c-echo-syntactic-information-p . t) ) "Burgess Programming Style") ;; Customizations for all of c-mode, c++-mode, and objc-mode (defun burgess-c-mode-common-hook () ;; add my personal style and set it for the current buffer (c-add-style "BURGESS" burgess-c-style t) ;; offset customizations not in burgess-c-style (c-set-offset 'member-init-intro '++) ;; other customizations ;; keybindings for C, C++, and Objective-C. We can put these in ;; c-mode-map because c++-mode-map and objc-mode-map inherit it (define-key c-mode-map "\C-m" 'newline-and-indent) ) (add-hook 'c-mode-common-hook 'burgess-c-mode-common-hook) . . . . . . . 1) 3) 0) -1) 0) 0) -)))

(c-cleanup-list (c-offsets-alist arglist)

;;; ;;; Lite hack for slippe skrive inn kompileringskommandoen i c, ;;; (hvis ikke Makfile eksisterer) ;;; samt en fancy heading hvis det er en ny fil. ;;; (add-hook 'c-mode-hook (lambda () ; (local-set-key "\C-c\C-c" 'compile) (cond ((not (file-exists-p (buffer-file-name))) (insert-string (concat "/*" (make-string 75 ?*) "*/\n"

name)) time-string)) > */\n" */\n"

"/*" (make-string 75 ? ) "*/\n" (format "/* File: %-67s */\n" (buffer"/*" (make-string 75 ? ) "*/\n" (format "/* Created: %-64s */\n" (current"/*" (make-string 75 ? ) "*/\n" "/* Author: */\n" "/*" (make-string 75 ? ) "*/\n" "/* Revision: $Id$ "/*" (make-string 75 ? ) "*/\n" "/* Description: "/*" (make-string 75 ? ) "*/\n" "/*" (make-string 75 ?*) "*/\n" "\n#include <stdio.h>\n")))) (outline-minor-mode 1) (or (file-exists-p "makefile") (file-exists-p "Makefile") (set (make-local-variable 'compile-command) (concat "gcc -o " (substring (file-name-nondirectory buffer-file-name) 0 (string-match "\\.c$" (file-name-nondirectory buffer-file-

name))) name)))))) ;;; ;;; Samme for C++

" " (file-name-nondirectory buffer-file-

(add-hook 'c++-mode-hook (lambda () ; (local-set-key "\C-c\C-c" 'compile) (cond ((not (file-exists-p (buffer-file-name))) (insert-string (concat "/*" (make-string 75 ?=) "*/\n" "/*" (make-string 75 ? ) "*/\n" (format "/* File: %-67s */\n" (buffername)) "/*" (make-string 75 ? ) "*/\n" (format "/* Created: %-64s */\n" (currenttime-string)) "/*" (make-string 75 ? ) "*/\n" "/* Author: */\n" "/*" (make-string 75 ? ) "*/\n" "/* Revision: $Id$ */\n" "/*" (make-string 75 ? ) "*/\n" "/* Description: */\n" "/*" (make-string 75 ? ) "*/\n" "/*" (make-string 75 ?=) "*/\n" "/* */\n" "/*" (make-string 75 ?=) "*/\n"

name))) name))))))

"\n#include <iostream.h>\n")))) (outline-minor-mode 1) (or (file-exists-p "makefile") (file-exists-p "Makefile") (set (make-local-variable 'compile-command) (concat "g++ -o " (substring (file-name-nondirectory buffer-file-name) 0 (string-match "\\.C$" (file-name-nondirectory buffer-file" " (file-name-nondirectory buffer-file-

;;; Mark hacks ( setq perl-mode-hook '(lambda() (setq perl-indent-level 0) (setq perl-continued-statement-offset 3) (setq perl-continued-brace-offset -3) (setq perl-brace-offset 3) (setq perl-brace-imaginary-offset 0) (setq perl-label-offset -3) (define-key perl-mode-map "\C-m" 'newline-and-indent) ) ) ( setq java-mode-hook '(lambda() (setq java-indent-level 0) (setq java-continued-statement-offset 3) (setq java-continued-brace-offset -4) (setq java-brace-offset 3) (setq java-brace-imaginary-offset 0) (setq java-label-offset -4) (setq java-statement-block-intro . +) (setq java-knr-argdecl-intro . 3) (setq java-substatement-open . 0) (setq java-label . 0) (setq java-statement-case-open . 0) (setq java-statement-cont . 0) (define-key java-mode-map "\C-m" 'newline-and-indent) ) )

Node:Answers to questions, Next:Index, Previous:Emacs style file, Up:Top

Answers to questions
Chapter 1 1) A tool which translates high level language into machine language. 2) By typing the name of an executable file. 3) By typing something like "cc filename" 4) NO! 5) Compiler errors and runtime errors. Chapter 3 1) printf ("Wow big deal"); 2) printf ("22"); 3) printf ("The 3 wise men"); printf ("The %d wise men",3); 4) Most facilities are held in libraries Chapter 4 1) To provide a basic set of facilities to the user 2) The filename used by a computer to reference a device 3) accounts.c 4) accounts.x (or perhaps accounts.EXE) 5) By typing the name in 4) Chapter 5 1) #include <filename> or #include "filename" 2) stdio.h 3) No. Only macro names can be used if the header file is not included. 4) Header file. Chapter 7 1) A group of statements enclosed by curly braces {}. 2) Comments, preprocessor commands, functions, declarations, variables, statements. (This is a matter of opinion, of course.) 3) Not necessarily. It starts wherever main() is. 4) It signifies the end of a block, the return of control to somethng else. 5) The semi-colon (;)

Chapter 8 1) The compiler thinks the rest of the program is all one Chapter 9 1) function (a,b) int a,b; { return (a*b); } 2) No. 3) The value is discarded. 4) The result is garbage. 5) By using "return". Chapter 10 1) A name for some variable, function or macro 2) a,c,f 3) int i,j; 4) double is twice the length of float and can hold significantly larger values. 5) int can have values + or -. Unsigned can only be + and can hold slightly larger + values than int. 6) I = 67; 7) int 8) At the function defintion and in the calling function. 9) printf ("%d",(int)23.1256); 10) No. Chapter 11 1) With variable parameters or with return() 2) Where a function is definned, after its name: e.g. function (...) <-- here { } 3) Yes. 4) No and it is illegal. comment!

5) * means "the contents of" and & means "the address of" 6) No. Chapter 12 1) A global variable can be accessed by any part of a program. 2) A local variable canonly be accessed by a select part of a program. 3) Local variables cannot leak out. Nothing outside them can reach local variables. 4) Variable parameters do. Value parameters use their own local copies, so they do not. 5) int i,j; main () { float x,y; another(x,y); } another(x,y) float x,y; { } There are 6 storage spaces altogether. Chapter 13 1) #define birthday 19 2) #include <math.h> 3) false 4) false Chapter 14 1) A variable which holds the address of another variable 2) With a * character. e.g. int *i; 3) Any type at all! 4) doubleptr = (double *)chptr; 5) Because number has not been initialized. This expression initializes the place that number points to, not number itself. (See main text) Chapter 15 printf

1) #include <stdio.h> main () { printf ("%2e",6.23); } 2) This depends on individual compilers 3) a) b) c) d) scanf 1) space, newline or tab 5) true. Low level I/O 1) The statement is possible provided putchar() is not implemented as a macro. It copies the input to the output: a simple way of writing on the screen. (Note however that the output is buffered so characters may not be seen on the output for some time!) 2) ch = getchar(); putchar (ch); Chapter 16 1) The thing(s) an operator acts upon. 2) printf ("%d",5 % 2); 3) rem = 5 % 2; 4) variable = 10 - -5; 5) if (1 != 23) { printf ("Thank goodness for mathematics"); } Chapter 18 1) Three: while, do..while, for 2) while : at the start of each loop do : at the end of each loop for : at the start of each loop 3) do..while 4) #include <stdio.h> #define TRUE 1 No conversion string Conversion string without matching value Probably nothing Conversion string without matching value

main () { char ch; while (true) { ch = getchar(); putchar (ch); } Chapter 19 1) The array identifier (without square brackets) is a pointer to the first element in the array. 2) You pass the array identifier, without square brackets. No! Arrays are always variable parameters. 3) double array[4][5]; Valid array bounds from array[0][0] to array[3][4] Chapter 20 1) Arrays of characters. Pointers to arrays of characters. 2) static char *strings[]; Could then initialize with braces {} and item list. (See main text) 3) See the Morse code example. Chapter 22 1) double 2) Probably true. This is implementation dependent. The actual types are double, long float and int. 3) The length of a string (excluding NULL byte) 4) Joins two strings. 5) Overflow, underflow, domain error, Loss of accuracy and division by zero. Chapter 23 1) ++, -- and any assignment or unary operator 2) It could make a program too difficult to read 3) No. The function would return before the vaue could be incremented. Chapter 23 1) FILE is defined by stdio.h It is reserved only when this file is included. It is not a built in part of the language. 2) FILE *fp;

3) False. They are meant for comparitive purposes only. It does not make sense to do arithmetic with enumerated data. 4) Yes. It provides a generic pointer. i.e. one which can be assigned to any other pointer type. 5) volatile 6) typedef double real; 7) True. Chapter 24 1) Nothing -- only the way it is used. Yes, every variable is a bit pattern. It is normal to use integer or character types for bit patterns. 2) Inclusive OR is true if all possiblilities are true simultaneously. Exclusive OR is false if all possibilites are true simultaneously. 3) Some kind of flag message perhaps. A bit pattern for certain. 4) a) b) c) d) e) 00000111 00000001 00001111 00001111 00001111 & & & & & 00000010 00000001 00000011 00000111 00000111 == 00000010 == 2 == 00000001 == 1 == 00000011 == 3 == 00000111 == 7 & 00000011 == 00000011 = 3

5) a) 00000001 | 00000010 == 00000011 == 3 b) 00000001 | 00000010 | 00000011 == 00000011 == 3 6) a) 1 & (~1) == 00000001 & 11111110 == 0 b) 23 & ~23 == 00011111 & 11100000 == 0 c) similarly 0: n & (NOT n) is always zero Chapter 26 1) a) a string which labels a file b) a variable of type *fp which points to a FILE structure c) the number of a file "portal" in the I/O array 2) High level filing performs translations to text. Low level files untranslated bit data. 3) fp = fopen ("filename","r"); 4) fd = open ("filename",O_WRONLY); 6) fprintf () Chapter 27 1) A structure can hold several values at the same time. A union holds only one value at any one time. 2) A part of a structure, or a possible occupant of a union. 3) x.mem

4) ptr->mem 5) False. Chapter 28 1) A diagram which shows how structures are put together. 2) With pointers. 3) False. Pointers are used to reference variables and data structures are built in such a way as to require only one name for a whole structure. 4) With pointers. ptr->member etc... 5) ptr=(struct Binary Tree *)malloc(sizeof(struct Binary Tree)); Chapter 29 1) A function which is defined in terms of itself. 2) A data structure run by the C-language for keeping track of function calls and for storing local data. 3) A lot of memory is used as stack space. Chapter 31 1) Declarations are in the wrong place. 2) Missing closing brace } 3) Missing semi-colon after closing brace };

Node:Index, Previous:Answers to questions, Up:Top

Index

operator: Pointers a.out: The compiler Address of variables: Pointers Array pointer: Function pointers Arrays: Arrays ASCII codes: Character Conversion Table Assignment, hidden: Example 28 Binary tree: Example Structures Bit operations: Machine Level Operations Black boxes: Functions Braces: Form of a C program C library: Operating systems Calling functions: Functions case statement: switch Case, upper and lower: Use of Upper and Lower Case cast operator: Types and The Cast Operator
&

Casting pointers: Example 8 char: Variables Character classification: Character Identification Character constants: char Character conversion table: Character Conversion Table Comments: Comments Compiler: Basic ideas Compiler phases: The compiler Compiling a program: Filenames Conditional compilation: Other Preprocessor commands const, constants: const Constant expressions: Special Constant Expressions Constants and macros: Preprocessor Control characters: char Control characters, printf: Formatting with printf Conversion characters, scanf: Conversion characters Conversion table: Character Conversion Table Curly braces: Form of a C program Data structures: Data structures Debugging: Errors and debugging Decisions: Decisions Declarations: Use of Upper and Lower Case Devices: Files devices do while: do while End of file: feof enum type: FILE Enumerated data: FILE Environment variables: envp Environment variables in C: envp Eratosthenes sieve: Arrays and for loops Errors, diagnosing: Errors and debugging Errors, files: File Errors Errors, of purpose: Errors Errors, programming: Errors Escaping from an program: Command languages Example code: Example Programs chapter exit function: The exit function Expressions: Functions with values Extern class: Storage class register static and extern FILE: FILE File descriptors: File Handles File extensions: Filenames File handles: File Handles File, detecting end of: feof File, opening: Opening files Files: Files and Devices Files and devices: Files devices Files as abstractions: Standard Output and Standard Input Format specifiers, printf: Formatting with printf Formatting text and variables: printf Function names: Functions Functions: Functions

Functions with values: Functions with values Game of life: Example 20 gcc: The compiler getchar: getchar and putchar getenv() function: envp gets: gets and puts, gets() Global variables: Global variables Global variables and recursion: Advantages and Disadvantages of Recursion GNU compiler: The compiler Header files: Libraries Hidden assignment: Example 28 High level: Levels Identifier names: Functions if: Decisions if statement: if Initialization of arrays: Initializing Arrays Initializing structures: Pre-initializing Static Structures Initializing variables: Declarations and Initialization int: integers, Variables Integer types: Variables Integers: integers Interrupting a program: Command languages Keyboard input: Standard Output and Standard Input Layout: Programming style Levels of detail: Levels Libraries: Reserved words & example Libraries of functions: Libraries Linked list: Example Structures Linker: The compiler Local environment: Levels Local variables: Where to declare things, Local variables Logical errors: Errors long: Variables, integers Loop variables: Choosing Variables Loops: Loops Low level: Levels Machine level operations: Machine Level Operations Macros: Preprocessor main function: Form of a C program Mainframe: Operating systems malloc: Creating Memory for Dynamical struct Types Math errors: Maths Errors Mathematical functions: Mathematical Functions Memory allocation, dynamical: Creating Memory for Dynamical struct Types Multidimensional arrays: Arrays Of More Than One Dimension Names, for identifiers: Functions Nested ifs: Nested ifs and logic Non-printable characters: char Opening a file: Opening files Operating system: Operating systems Operators: Assignments Expressions and Operators Operators, hidden: Hidden Operators

Output, formatting: printf Panic button: Command languages Parameters to functions: Parameters Parsing strings: sscanf() Phases of compilation: The compiler Poem: Example 1 Pointers: Pointers Pointers to functions: Function pointers Preprocessor: Preprocessor Prime number generator: Arrays and for loops printf function: printf, printf again Printing: printf Printing formatted to strings: sprintf() Prototyping: Value parameters putchar: getchar and putchar puts: gets and puts, puts() Records: Structures and Unions Records (structures): struct again Recursion: Recursion Recursion and global variables: Advantages and Disadvantages of Recursion Reserved words: Reserved words & example Returning values: Functions with values scanf: scanf scanf, dangers: The dangerous function Scope: Where to declare things, Scope Screen editor: Basic ideas Screen output: Standard Output and Standard Input Shell: Basic ideas, Operating systems short: integers, Variables Snakes and ladders: Program listing Special characters: Special Control Characters again, char Stack: Functions and The Stack Standard error: Files devices Standard input: Files devices Standard input/output: Standard Output and Standard Input Standard output: Files devices Static initialization of arrays: Initializing Arrays Static variables: Storage class register static and extern stderr: Standard Output and Standard Input stdin: Standard Output and Standard Input stdio.h: Libraries stdout: Standard Output and Standard Input strcmp: Handling strings strcpy: Handling strings Streams: Standard Output and Standard Input, How does scanf see the input String handling functions: Handling strings Strings: Conventions and Declarations strlen: Handling strings strstr: Handling strings Structure: struct again Structure of a C program: Form of a C program Structured data: Structures and Unions

Structures: Structures and Unions Structures, initializing: Pre-initializing Static Structures Style: Programming style, if, Cautions about Style Style, global variables: Style note Substrings, searching for: Handling strings switch case: switch Syntax error: Errors Tables: Arrays Terminating a program: The exit function Tests: Decisions Type conversion: Types and The Cast Operator Types: Types and The Cast Operator Types, advanced: More on Data Types Union: Unions Unions: union, Structures and Unions Value parameters: Value parameters Variable names: Variables Variable types: Types and The Cast Operator Variables: Variables, Use of Upper and Lower Case Variables, declaring: Declarations Variables, initializing: Declarations and Initialization void: void volatile: volatile Whiet space: Comments while loop: while White space: Scope

Motivation & DBMS Architecture Overview

Why databases? Why DB research?

The technology trend angle: emphasis in CS research has shifted from computation to information management.

Evidence:
o

o o

Hardware: high-performance computer companies on hard times (Thinking Machines, KSR, Cray, SGI?). The exemplary success story in massive parallelism: Teradata (now sold by NCR). Been around since the 70's. "Shared-Nothing" (sometimes called "clusters" or "NOW/COW/WOW" etc.) Successes have been largely databasecentric. "Low-end" users: scramble to webspace reflects desire to give/receive info. Success of these efforts is questionable, and the disorganization will get worse as things grow. "High-end" users: scientists, the biggest users of high-powered computation, now have data management problems that exceed their appetite for cycles Other researchers: architecture, OS, theoreticians, AI are all moving this way. PS: you will see all this in the job market!

The utilitarian angle: "Database: the boring part of accounting"? Not anymore! Interesting, world-changing apps: o digital libraries o digital ``asset mgmt'' -- i.e., multimedia & entertainment o digital mapping & geo apps o scientific applications: earth science, DNA, molecular docking, experiment management, etc. o decision-support, data analysis & "mining" o your mom and pop care about this stuff! (as do funding agencies, companies, etc.) The intellectual angle: o Big, beautiful ideas: relational model & languages, concurrency control, query processing, etc. o Real, meaty systems work: the serious 24x7, high performance, complex systems engineering domain o Room for both kinds of contributions, separately and simultaneously plenty of room to take an idea from theory to practice o lots of useful research left to do

The database-centric view of the CS research universe (take with a large grain of salt):

OS & Architecture are ``finished'': Ascendancy of Linux and FreeBSD. If Microsoft & Intel can mass-market these, it must be easy. PL has become arcane Theory is ... theoretical. AI is that which cannot be done etc... while you may (should!) disagree with all the above in some respects, it is true that DB research is notably relevant and fertile these days. Lots of meaty problems remain that people care about.

An outline of ongoing database research:

Big: massive datasets o Tertiary storage: EOSDIS 1 Tb/day, keep it all for 15 years o Parallelism: data parallelism is natural in a DBMS. How to do DB operations in parallel and balance load well? WalMart (365 node, 6Tb online, 4billion row table, 200million updates daily, 4000 queries/day, 1500 users/week, 4 min DS response time w/ avg. 60000 rows out) o Data Analysis, Data Mining: given huge amounts of data, try to find interesting information in the data. What is the "killer query"? Wide: wide-scale distribution o World-Wide Web (a bad example) o Distributed databases for the 00's: autonomous "pay & play" databases Complex: complex datatypes and their associated lookups o complex base types: geographic data, multimedia, scientific data, CAD data o complex objects o extensible query processing engines o indexing new data types Old & hetero: the data integration problem o schema integration: trying to figure out how different schemas fit together. Hard!!! o DBMS integration: trying to semi-transparently glue different kinds of database systems together

DBMS History

late 60's: network (CODASYL) & hierarchical (IMS) DBMS. o Low-level ``record-at-a-time'' DML, i.e. physical data structures reflected in DML (no data independence) 1970: Codd's paper. The most influential paper in DB research. Set-at-a-time DML. Data independence. Allows for schema and physical storage structures to change under the covers''. Truly important theory, led to "paradigm shift" in thinking and in practice. (Papadimitriou: "as clear a paradigm shift as we can hope to find in computer science"). Turing award. early-to-mid-70's: raging debate between the two camps. "great debate" in 1975 mid 70's: 2 full-function (sort of) prototypes. Ancestors of essentially all today's commercial systems Ingres: UCB 1974-77 o a ``pickup team'', including Stonebraker & Wong. early and pioneering. begat Ingres Corp (CA), Sybase, MS SQL Server, Britton-Lee, Wang's PACE. System R: IBM San Jose (now Almaden) o 15 PhDs. begat IBM's SQL/DS & DB2, Oracle, HP's Allbase, Tandem's Non-Stop SQL. System R arguably got more stuff ``right'' Both were viable starting points, proved practicality of relational approach. Beautiful example of theory -> practice!! early 80's: commercialization of relational systems mid 80's: SQL becomes ``intergalactic standard''. o DB2 becomes IBM's flagship product.

IMS ``sunseted'' today: network & hierarchical essentially dead (though commonly in use!) o relational is mainstream, not even sexy o SQL (& perhaps RDBMS) too flawed to last in current form. semantically flawed in various ways (Date, 1985). anemic in an effort to fix it up, standards committees are making a mess design by committee leads to kitchen sink standards body as designers, rather than codifiers leads to wasting time (Sybase) or irrelevance of standard (Informix & IBM shipping SQL3 before standardized) o various players in research, industry and both scrambling to standardize the "next thing"
o

Modern DBMS taxonomy

Two axes:

Functionality: RDBMS, OODBMS, ORDBMS. o RDBMS: query in, data out. simple data model: tables with rows and columns, simple data types. widely standardized definitions, languages clean mathematical foundation o OODBMS: term is somewhat nebulous. usually, a persistent programming environment no queries (or only VERY simple ones). data model comes from PL, includes lots of good OO stuff. theoretical "foundations" after the fact, very complicated. o ORDBMS: term is getting better defined as products mature (Informix, IBM) an attempt to provide best of both worlds: queries & rich data types. query interface. Rich data types with lots of OO features, esp. object identity, type-extensibility and inheritance. Basic ``outer'' data type is relation, with extensible data types in the fields. relational theory applies to outer operations only Implementation: o Single-Site (i.e. traditional) o Parallel: lots of tightly-coupled machines solve one query together. A database supercomputer. o Distributed: geographically distributed machines, each "hosting" different data, participate in a more loosely coupled manner

Motivation & DBMS Architecture Overview

Why databases? Why DB research?

The technology trend angle: emphasis in CS research has shifted from computation to information management. Evidence:
o

o o

Hardware: high-performance computer companies on hard times (Thinking Machines, KSR, Cray, SGI?). The exemplary success story in massive parallelism: Teradata (now sold by NCR). Been around since the 70's. "Shared-Nothing" (sometimes called "clusters" or "NOW/COW/WOW" etc.) Successes have been largely databasecentric. "Low-end" users: scramble to webspace reflects desire to give/receive info. Success of these efforts is questionable, and the disorganization will get worse as things grow. "High-end" users: scientists, the biggest users of high-powered computation, now have data management problems that exceed their appetite for cycles Other researchers: architecture, OS, theoreticians, AI are all moving this way. PS: you will see all this in the job market!

The utilitarian angle: "Database: the boring part of accounting"? Not anymore! Interesting, world-changing apps: o digital libraries o digital ``asset mgmt'' -- i.e., multimedia & entertainment o digital mapping & geo apps o scientific applications: earth science, DNA, molecular docking, experiment management, etc. o decision-support, data analysis & "mining" o your mom and pop care about this stuff! (as do funding agencies, companies, etc.) The intellectual angle: o Big, beautiful ideas: relational model & languages, concurrency control, query processing, etc. o Real, meaty systems work: the serious 24x7, high performance, complex systems engineering domain o Room for both kinds of contributions, separately and simultaneously plenty of room to take an idea from theory to practice o lots of useful research left to do

The database-centric view of the CS research universe (take with a large grain of salt):

OS & Architecture are ``finished'': Ascendancy of Linux and FreeBSD. If Microsoft & Intel can mass-market these, it must be easy. PL has become arcane Theory is ... theoretical. AI is that which cannot be done etc... while you may (should!) disagree with all the above in some respects, it is true that DB research is notably relevant and fertile these days. Lots of meaty problems remain that people care about.

An outline of ongoing database research:

Big: massive datasets o Tertiary storage: EOSDIS 1 Tb/day, keep it all for 15 years o Parallelism: data parallelism is natural in a DBMS. How to do DB operations in parallel and balance load well? WalMart (365 node, 6Tb online, 4billion row table, 200million updates daily, 4000 queries/day, 1500 users/week, 4 min DS response time w/ avg. 60000 rows out) o Data Analysis, Data Mining: given huge amounts of data, try to find interesting information in the data. What is the "killer query"? Wide: wide-scale distribution o World-Wide Web (a bad example) o Distributed databases for the 00's: autonomous "pay & play" databases Complex: complex datatypes and their associated lookups o complex base types: geographic data, multimedia, scientific data, CAD data o complex objects o extensible query processing engines o indexing new data types Old & hetero: the data integration problem o schema integration: trying to figure out how different schemas fit together. Hard!!! o DBMS integration: trying to semi-transparently glue different kinds of database systems together

DBMS History

late 60's: network (CODASYL) & hierarchical (IMS) DBMS. o Low-level ``record-at-a-time'' DML, i.e. physical data structures reflected in DML (no data independence) 1970: Codd's paper. The most influential paper in DB research. Set-at-a-time DML. Data independence. Allows for schema and physical storage structures to change under the covers''. Truly important theory, led to "paradigm shift" in thinking and in practice. (Papadimitriou: "as clear a paradigm shift as we can hope to find in computer science"). Turing award. early-to-mid-70's: raging debate between the two camps. "great debate" in 1975 mid 70's: 2 full-function (sort of) prototypes. Ancestors of essentially all today's commercial systems

Ingres: UCB 1974-77 o a ``pickup team'', including Stonebraker & Wong. early and pioneering. begat Ingres Corp (CA), Sybase, MS SQL Server, Britton-Lee, Wang's PACE. System R: IBM San Jose (now Almaden) o 15 PhDs. begat IBM's SQL/DS & DB2, Oracle, HP's Allbase, Tandem's Non-Stop SQL. System R arguably got more stuff ``right'' Both were viable starting points, proved practicality of relational approach. Beautiful example of theory -> practice!! early 80's: commercialization of relational systems mid 80's: SQL becomes ``intergalactic standard''. o DB2 becomes IBM's flagship product. o IMS ``sunseted'' today: network & hierarchical essentially dead (though commonly in use!) o relational is mainstream, not even sexy o SQL (& perhaps RDBMS) too flawed to last in current form. semantically flawed in various ways (Date, 1985). anemic in an effort to fix it up, standards committees are making a mess design by committee leads to kitchen sink standards body as designers, rather than codifiers leads to wasting time (Sybase) or irrelevance of standard (Informix & IBM shipping SQL3 before standardized) o various players in research, industry and both scrambling to standardize the "next thing"

Modern DBMS taxonomy

Two axes:

Functionality: RDBMS, OODBMS, ORDBMS. o RDBMS: query in, data out. simple data model: tables with rows and columns, simple data types. widely standardized definitions, languages clean mathematical foundation o OODBMS: term is somewhat nebulous. usually, a persistent programming environment no queries (or only VERY simple ones). data model comes from PL, includes lots of good OO stuff. theoretical "foundations" after the fact, very complicated. o ORDBMS: term is getting better defined as products mature (Informix, IBM) an attempt to provide best of both worlds: queries & rich data types. query interface. Rich data types with lots of OO features, esp. object identity, type-extensibility and inheritance. Basic ``outer'' data type is relation, with extensible data types in the fields. relational theory applies to outer operations only

Implementation: o Single-Site (i.e. traditional) o Parallel: lots of tightly-coupled machines solve one query together. A database supercomputer. o Distributed: geographically distributed machines, each "hosting" different data, participate in a more loosely coupled manner

Motivation & DBMS Architecture Overview

Why databases? Why DB research?

The technology trend angle: emphasis in CS research has shifted from computation to information management. Evidence:
o

o o

Hardware: high-performance computer companies on hard times (Thinking Machines, KSR, Cray, SGI?). The exemplary success story in massive parallelism: Teradata (now sold by NCR). Been around since the 70's. "Shared-Nothing" (sometimes called "clusters" or "NOW/COW/WOW" etc.) Successes have been largely databasecentric. "Low-end" users: scramble to webspace reflects desire to give/receive info. Success of these efforts is questionable, and the disorganization will get worse as things grow. "High-end" users: scientists, the biggest users of high-powered computation, now have data management problems that exceed their appetite for cycles Other researchers: architecture, OS, theoreticians, AI are all moving this way. PS: you will see all this in the job market!

The utilitarian angle: "Database: the boring part of accounting"? Not anymore! Interesting, world-changing apps: o digital libraries o digital ``asset mgmt'' -- i.e., multimedia & entertainment o digital mapping & geo apps o scientific applications: earth science, DNA, molecular docking, experiment management, etc. o decision-support, data analysis & "mining" o your mom and pop care about this stuff! (as do funding agencies, companies, etc.) The intellectual angle:

o o o o

Big, beautiful ideas: relational model & languages, concurrency control, query processing, etc. Real, meaty systems work: the serious 24x7, high performance, complex systems engineering domain Room for both kinds of contributions, separately and simultaneously plenty of room to take an idea from theory to practice lots of useful research left to do

The database-centric view of the CS research universe (take with a large grain of salt):

OS & Architecture are ``finished'': Ascendancy of Linux and FreeBSD. If Microsoft & Intel can mass-market these, it must be easy. PL has become arcane Theory is ... theoretical. AI is that which cannot be done etc... while you may (should!) disagree with all the above in some respects, it is true that DB research is notably relevant and fertile these days. Lots of meaty problems remain that people care about.

An outline of ongoing database research:

Big: massive datasets o Tertiary storage: EOSDIS 1 Tb/day, keep it all for 15 years o Parallelism: data parallelism is natural in a DBMS. How to do DB operations in parallel and balance load well? WalMart (365 node, 6Tb online, 4billion row table, 200million updates daily, 4000 queries/day, 1500 users/week, 4 min DS response time w/ avg. 60000 rows out) o Data Analysis, Data Mining: given huge amounts of data, try to find interesting information in the data. What is the "killer query"? Wide: wide-scale distribution o World-Wide Web (a bad example) o Distributed databases for the 00's: autonomous "pay & play" databases Complex: complex datatypes and their associated lookups o complex base types: geographic data, multimedia, scientific data, CAD data o complex objects o extensible query processing engines o indexing new data types Old & hetero: the data integration problem o schema integration: trying to figure out how different schemas fit together. Hard!!! o DBMS integration: trying to semi-transparently glue different kinds of database systems together

DBMS History

late 60's: network (CODASYL) & hierarchical (IMS) DBMS.

Low-level ``record-at-a-time'' DML, i.e. physical data structures reflected in DML (no data independence) 1970: Codd's paper. The most influential paper in DB research. Set-at-a-time DML. Data independence. Allows for schema and physical storage structures to change under the covers''. Truly important theory, led to "paradigm shift" in thinking and in practice. (Papadimitriou: "as clear a paradigm shift as we can hope to find in computer science"). Turing award. early-to-mid-70's: raging debate between the two camps. "great debate" in 1975 mid 70's: 2 full-function (sort of) prototypes. Ancestors of essentially all today's commercial systems Ingres: UCB 1974-77 o a ``pickup team'', including Stonebraker & Wong. early and pioneering. begat Ingres Corp (CA), Sybase, MS SQL Server, Britton-Lee, Wang's PACE. System R: IBM San Jose (now Almaden) o 15 PhDs. begat IBM's SQL/DS & DB2, Oracle, HP's Allbase, Tandem's Non-Stop SQL. System R arguably got more stuff ``right'' Both were viable starting points, proved practicality of relational approach. Beautiful example of theory -> practice!! early 80's: commercialization of relational systems mid 80's: SQL becomes ``intergalactic standard''. o DB2 becomes IBM's flagship product. o IMS ``sunseted'' today: network & hierarchical essentially dead (though commonly in use!) o relational is mainstream, not even sexy o SQL (& perhaps RDBMS) too flawed to last in current form. semantically flawed in various ways (Date, 1985). anemic in an effort to fix it up, standards committees are making a mess design by committee leads to kitchen sink standards body as designers, rather than codifiers leads to wasting time (Sybase) or irrelevance of standard (Informix & IBM shipping SQL3 before standardized) o various players in research, industry and both scrambling to standardize the "next thing"
o

Modern DBMS taxonomy

Two axes:

Functionality: RDBMS, OODBMS, ORDBMS. o RDBMS: query in, data out. simple data model: tables with rows and columns, simple data types. widely standardized definitions, languages clean mathematical foundation o OODBMS: term is somewhat nebulous. usually, a persistent programming environment no queries (or only VERY simple ones). data model comes from PL, includes lots of good OO stuff. theoretical "foundations" after the fact, very complicated.

ORDBMS: term is getting better defined as products mature (Informix, IBM) an attempt to provide best of both worlds: queries & rich data types. query interface. Rich data types with lots of OO features, esp. object identity, type-extensibility and inheritance. Basic ``outer'' data type is relation, with extensible data types in the fields. relational theory applies to outer operations only

Implementation: o Single-Site (i.e. traditional) o Parallel: lots of tightly-coupled machines solve one query together. A database supercomputer. o Distributed: geographically distributed machines, each "hosting" different data, participate in a more loosely coupled manner

Access Methods: B-Trees, R-Trees & GiSTs

First, a Review of Disks

hardware: platters, arms, spindle. Often a cache (often 1 track) -- very devicespecific. data layout: sectors, blocks, tracks, cylinders "access time": ha! seek time, rotational delay, transfer time random vs. sequential I/O Examples:
o

o o

Seagate Hawk 2XL (2.15Mb) -- 5.5 msec average rotation delay, 9 msec average seek time (varies from 1 to 22 msec), 5MB/sec transfer rate Note: average disk access time is about 10-15 msec, RAM access is more like 5 microsecs! Another note: while transfer rate and capacity are growing wildly every year, seek time is not shrinking nearly as fast!

Access Methods: General Issues

Come in two flavors: heap files and indexes Support AMI interface: o open (possibly with selection condition) o get_next o close

insert, delete, update_field indexes usually do two things: o partition: partition a dataset or domain into buckets o label: provide a label for each bucket o this is sometimes done hierarchically (trees), sometimes not (hashing) o the whole game boils down to how you partition & label! index's utility defined by queries, not data! typically random I/O in index Performance goals: o cold vs. hot lookups o # of I/Os vs. # of blocks worth of output o because of random vs. sequential, should always compare to heap file! Implementation issues: o concurrency & recovery o cost estimation o bulk loading
o

The "Ubiquitous" B-tree

Query load: equality & linear range predicates Basics

Basic B-tree properties: balanced, high fanout, some minimal fill factor (order) o in practice, may want minimal fill factor to be less than 50% (delete at empty has been shown good in TP applications) B-tree vs. B+-tree. Note that B+-tree internal keys simply "direct traffic"; this is very useful. intra-node operations

Compression

rear & front key compression pointer compression Very common in practice, since B+-tree is on critical path of TP apps

Variable-length keys

not too hard -- fill factor now expressed in terms of bytes, not entries.

Multiuser & Recovery issues

tricky, solved problem high concurrency locking: need not be 2PL watch repeatable read, too! we will revisit this issue in detail later

Leaves usually contain record pointers (as in VSAM) VSAM

unified B+-tree-based implementation for a file-system (recall Stonebraker's gripe) store sibling leaves and parent in one cylinder replicate parent along one track to minimize rotation time

R-trees
Query load: Equality and Range predicates in n dimensions

point in polygon polygon in polygon Overlaps polygon Contains polygon

Applications: GIS, VLSI, n-d range queries over traditional data where n is typical Notation. R-tree

Structure: like a B+-tree, but keys are MBRs (rectilinear minimum bounding rectangles), one key per pointer Search: Traverse multiple paths! (DFS) Insertion: o Where: ChooseLeaf. Least enlargement. Ties broken by picking smallest key. o Node splitting as in B+-tree. Question: what goes left, what goes right? o AdjustTree required. Why not in B+-tree? Deletion: o Seek and destroy o CondenseTree: throw entries from under-full pages back into tree. Make sure they end up at same level. Why different from B+-tree? Update: delete, update, reinsert Split algorithms: o exhaustive o quadratic: start with most distant seeds, greedily add based on maximal preference o linear: like quadratic, but picks seeds differently Performance study o RISC-II VLSI data set o In practice, people (Postgres & SHORE) seem to use M/2, quadratic split

R-Tree Variants
R*-tree (Beckman, et al., SIGMOD 90)

like R-tree, but 3 distinctions: o insertion minimizes different badness metric (depends on tree level)

new & improved split algorithm (nlogn, more effective) forced reinsert of 30% most "extreme" entries decreases overlap of siblings improves storage utilization splits less often shapes tend to be more quadratic (pack better, generate smaller parents, minimizes split of associated pixels) Popular, since it out-performs R-tree on search. Concurrency problems because of reinsertion.
o o

R+-Tree (Sellis, et al, VLDB87)

instead of overlapping keys, use multiple inserts (i.e. replicate items in leaves) speeds search, complicates insert/delete/update buggy! NOTE: data vs. space partitioning. R-trees partition data. R+-trees partition space. other space-partitioning trees: K-D-B trees, Quad trees

TV(telescopic vector)-tree (Lin, et al, VLDB Journal, 1994)

Goal: solve high-d problems Keys: compressed (center, radius) pairs in n-d compression "telescopes" out to only the appropriate dimensions at each level (other dimensions are "wildcards") The "dimensionality curse" remains! (See discussion of indexability below)

hB (holey-brick B) trees (Lomet & Salzburg, TODS 90)

solve three problems o naturally handles swiss-cheese polygons o speeds intra-node search o space partitioning & data partitioning idea: put a main-memory k-d tree (space-partitioning tree) on each node. This tree can represent a holey brick! Neat idea, pretty complicated. Not really a tree -- can be a DAG. Journal paper is buggy. Not necessarily more effective than R*-trees

...and a recent flurry of new trees. In practice, most 2-d access methods have proved to be within 20-30% of each other, though no systematic benchmarking has been done.

Generalized Search Trees

Indexing in OO/OR Systems

Quick access to user-defined objects Support queries natural to the objects Two previous approaches o Specialized Indices (ABCDEFG-trees) redundant code: most trees are very similar concurrency control, etc. tricky!

Extensible B-trees & R-trees (Postgres/Illustra) B-tree or R-tree lookups only! E.g. WHERE movie.video < Terminator 2

A Third Approach

A generalized search tree. Must be: o Extensible in terms of queries o General (B+-tree, R-tree, etc.) o Easy to extend o Efficient (match specialized trees) o Highly concurrent, recoverable, etc.

Uses for GiSTs

New indexes needed for new apps... o find all supersets of S o find all molecules that bind to M o your favorite query here (multimedia?) ...and for new queries over old domains: o find all points in region from 12 to 1 oclock o find all strings that match R. E.

Structure: balanced tree of (p, ptr) pairs

p is a key predicate p holds for all objects below ptr keys on a page may overlap Key predicates: a user-defined class o This is the only extensibility required!

Key Methods

Search Consistent(E,q): E.p && q? (no/maybe) Labeling o Union(P): new key that holds for all tuples in P Partitioning o Penalty(E1,E2): penalty of inserting E2 in subtree E1 o PickSplit(P): split P into two groups of entries
o

Search

General technique: o traverse tree where Consistent is TRUE For fancier things, see [Aoki98].

Insert

descend tree along least increase in Penalty if theres room at leaf, insert there

else split according to PickSplit propagate changes using Union Notes: o on overflow, can do R*-tree style reinsert

Delete

find the entry via Search, and delete it propagate changes using Union on underflow: o ordered keys, do B+-tree style borrow/coalesce o else reinsert stuff on page and delete page

GiSTS over (B+-trees)

Logically, keys represent ranges [x,y) Queries: o Contains([a,b), v) Consistent(E,q): (x<b) && (y > a) Union(P): [MIN(xi), MAX(yi)) Penalty(E1, E2): return MAX(y2 - y1, 0) + MAX(x1 - x2, 0) o if E1 is leftmost or rightmost, drop a term PickSplit(P): split evenly in order

Key Compression

Keys may take up too much room on a page Two extra key methods: o Compress(E)/Decompress(E) o Compression can be lossy: over-generalization OK B+-tree Compression
o o

Compress(E=([x,y), ptr)): if E is leftmost return NULL, else return x Decompress(E=(p, ptr)): if E is leftmost, let x = -infinity, else let x = p. if E is rightmost, let y = infinity, else let y be the value stored in the next key on the right. if E is rightmost on a leaf page, let y = x+1.

GiSTs over R2 (R-tree)

Logically, keys represent bounding boxes Queries: Contains, Overlaps, Equals Consistent(E,q): E.p overlap/contain q? Union(P): bounding box of all entries Compress(E): form bounding box Decompress(E): identity function Penalty(E,F): size(Union({E,F}) - size(E) PickSplit(P): R-tree or R*-tree methods

GiSTs over P(Z) (RD-tree)

Logically, keys represent bounding sets Queries: Contains, Overlaps, Equals Consistent(E,q): Union(P): set-union of keys Compress(E): Bloom filters, rangesets, etc. Decompress(E): match compress Penalty(E,F): |E.p U F.p| - |E.p| PickSplit(P): R-tree algorithms

Research Issues

Concurrency, Recovery, Consistency Kornacker, et al '97 Generalizing Search -- Aoki '98 Indexability -- Hellerstein/Papadimitriou/Koutsoupias '97, Koutsoupias/Taylor '98, Tsamoladas/Miranker '98 AM development tools: amdb (Kornacker/Shah, demo at SIGMOD '98)

Query Processing
A quick primer on the relational algebra

selection projection cartesian product join (in particular, theta-join)

Unary Operations

sorting: see Knuth, or Shapiro paper. We will study high-performance parallel sorting later. Uses? hashing: a minor variant of hybrid-hash join works well. Uses?

Binary Operations (i.e. Joins)

Join algorithms apply to almost any form of combining multiple collections. Some commonly used join variants (alternative logical algebra operators):

semi-join: R semi-join S ~= R join remove-dups(S) projected to the columns of R o S basically serves as a filter o logically, selection is a "virtual semijoin" outer join: R outer-join S: compute R join S, and for each tuple of R that has no match send it to the output with the S columns filled in with NULLs. Left, Right, and Full Outer Joins. intersection & difference

These logical algebra operators can be implemented as minor variations on the same join algorithms! The "Guy Lohman Test" for join operators:

does the operator work for joining 3 inputs without storing the output of 2 of the inputs?

Nested Loops Join

for each tuple r of R for each tuple s of S if rs satisfies join predicate output rs

R is the outer relation (left) S is the inner relation (right)

works for any join predicate inner input must be stored

Refinement 1: Block Nested Loops Join

for each block BR of R for each tuple s of S for each tuple r of BR such that rs satisfies join predicate output rs

Further refinements to nested loops:

load R in chunks of M-K pages "pin" K pages of the inner into memory alternate scan direction on inner ("boustrophedonism")

Refinement 2: Index Nested Loops Join

for each tuple r of R probe index over S; output all s s.t. rs satisfies join predicate;

Notes:

Still called "nested loops", S is still referred to as the "inner" relation any join predicate can be supported if the index supports the predicate!
SELECT cities.name FROM cities, forests WHERE cities.boundary overlaps forests.boundary;

can convert all nested-loops joins to index-nested-loops by indexing inner on the fly

(Sort-)Merge Join

Works for equijoin, "band" joins we will assume here you know how to do a 2-pass sort [see Knuth or Shapiro] idea: if R & S are sorted on the join column(s), we can "simply" merge them But duplicates complicate things (as usual). R 1 2 2 3 5 S 1 2 4 5 5 R join S 1,1 2,2 2,2 5,5 5,5

sort R; sort S; R.open(); S.open(); r = R.next(); s = S.next(); while (r != NULL && s != NULL) { while (r.c < s.c) r = R.next(); if (r.c = s.c) { while (r.c = s.c) { output rs; r = R.next(); } "rewind" r to first tuple of R where r.c = s.c; s = S.next(); } while (r.c > s.c) s = S.next(); }

Refinement: do merging of R & S while merging runs from each.

Requires enough buffers for merging both R and S simultaneously.

Note: Sort-merge is symmetric, so "outer", "inner", "left", "right" are arbitrary

Classic Hash Join

Works for equijoin only Let R be the smaller relation
Hash R into VM; for each tuple of S probe hashtable over R and output all rs s.t. s.c = r.c

In the spirit of indexed nested-loop join, S is "outer" (left), and R is "inner" (right) Warning: at least one important paper calls S the right relation, R the left!!! I prefer to call S left/outer, R right/inner

note that hash join is very similar to nested-loop + index-on-the-fly (which Sybase used to do.) If we call the probing relation there the "outer", we should do the same here! Safest to refer to "building" and "probing" relations
o

Simple Hash Join

Repeat steps 1 and 2 with R, S replaced by the passed over tuples. Advantages:

Great when R is small Works for memory of any size

Disadvantages:

you tell me!

Grace Hash Join

Phase 1 is repeated with S in place of R

THEN

Advantages:

works well for big R, esp. when RAM is just big enough for output buffers of R

Disadvantages:

you tell me!

Hybrid Hash
Original paper: DeWitt, Katz, Olken, Shapiro, Stonebraker, Wood, SIGMOD '84.

Phase 2 as in Grace Join Hybrid Hash Advantages:

As good as simple hash for small R As good as Grace for big R If RAM is bigger than # of output buffers, improves on Grace

Disadvantages:

you tell me

Handling Partition Overflow:

If a partition of R overflows, recursively partition it, along with the corresponding bucket of S Note that size of S does not affect level of recursion! o Makes hash-join particularly effective if |R| << |S| (compare to sortmerge)

Variation: Virtual Memory

This is a Shapiro detail that is mostly ignored today... Define Hot Set as those pages which should be "pinned" in memory In our case, the hot set is the current hashtable in memory Any other memory (e.g. partitions) could be managed by VM! Problems: o LRU paging leaves the "wrong end" of partitions in memory

LRU screws up phase 2

Additional Tricks: Filters Idea: build a filter based on R so you stage less of S to disk

Babb Array filter Bloom filter These are like lossy semijoins! Any superset of S semijoin R will do.

Buffer Management
DBMIN (Chou & DeWitt)
Theme: There are just a few basic access patterns in a query processing system. Make straightforward observations about locality behavior of these access patterns, and expose them to your buffer manager. Old Stuff:

Background: blocks, frames, pages, "pinning" Stochastic OS replacement policies: LRU, MRU, FIFO, LIFO, Clock, etc. None is uniformly appropriate for typical DBMS access patterns (see Stonebrakers "OS Dump"). Domain Separation: Split up work/memory into categories, and do LRU within your category. If there are no pages in your chunk of memory, borrow from somewhere else. Example domains: a non-leaf level of a B+-tree. 8-10% improvement over global LRU(?) o domains are static o pages belong to partitions regardless of how theyre being used (e.g. no distinction between heap file page for sequential scan and for nested loop) o does not prevent over-utilization of memory by multiple users, since theres no notion of "users" or "processes" o needs an orthogonal load-control facility Group LRU: Like DS, but prioritize domains. Free buffers are stolen in order of priority (low high) o optimization: adjust sizes of domains using a "working-set" judgment (i.e. pages in last Ti refs are kept for domain i) o Effelsberg & Haerder: no convincing evidence that any of this works better than LRU or Clock. The "New" Algorithm: Modification of INGRES. o Two observations: 1. priority not a property of a page, but of a relation

2. each relation needs a working set o Note: this is a query-based intuition! Anticipates DBMIN. o Subdivide memory into a chunk per relation, and prioritize chunks. o Assign an empty resident set per relation. o Use MRU per relation, but each relation gets one buffer at all times. o Heuristics available to reprioritize chunks. o Simulation study looked good, but implementation in INGRES didnt beat LRU. Hot Set o Also uses query-based info o A set of pages which are accessed over and over form a "hot set". If you graph buffer size against # of page faults, you see "hot points". If your hot set fits in memory, you win! Otherwise you lose. Example: NL-join, the hot set is |inner| + 1 Dont let a query into the system unless its hot set fits in memory. Each query is given its hot set worth of buffers. The idea assumes LRU is going on. But MRU is better for looping access, and makes the "hot point" go away Using LRU over-allocates (i.e. under-utilizes) memory, since the "hot point" analysis can be fixed with MRU.

o o o o o

DBMIN Based on the Query Locality Set Model (QLSM), which characterizes DBMS reference patterns in 3 ways:

Sequential: Straight Sequential (SS) & Clustered Sequential (CS) Random: Independent Random (IR) and Clustered Random (CR) Hierarchical: Straight Hierarchical (SH), Hierarchical with Straight Sequential (H/SS), Hierarchical with Clustered Sequential (H/CS), Looping Hierarchical (LH)

Questions: which query processing operators correspond to each category? Do the categories cover all the operators?

The DBMIN Algorithm:

associate a chunk of memory with each "file instance" (more like each table in the FROM clause). This is called the file instances locality set. estimate max locality set sizes via looking at query plan & database stats. A query is allowed to run if the sum of its locality sets fits in free buffers. a global page table and global free list is kept in addition to locality sets on page request o if page in global table & the locality set, just update usage stats of page

o o o

else if page in memory but not in LocSet, grab page, and if not in another LocSet put it in our LocSet else read page into LocSet (using a page from global free list) If locality set gets bigger than max needed, choose a page to toss according to a LocSet-specific policy (to be discussed next)

Locality Set size & replacement policies for different reference patterns:

SS: LocSet size = 1. Replace that page as soon as needed. CS: LocSet size = (#tuples in largest cluster)/(# of tuples per page). FIFO or LRU replacement work. LS: LocSet size = size of relation. MRU is best. IR: odds of revisit are low, so LocSet either 1, or the magic number b from the "Yao" formula. Residual value r = (k - b)/b of a page can be used to choose between 1 and b (i.e. k is number of accesses, b is # of pages that will be referenced, so this is # of revisits over # of pages). Replacement policy? CR: Just like CS, but pages are not packed onto blocks. So its just # of tuples in largest cluster. SH, H/SS: like SS H/CS: like CS, but replace tuples with (key,ptr) pairs LH: at each level h of an index, you have random access among pages. Use Yao to figure out how many pages youll access at each level in k lookups. LocSet is sum of these over all levels that you choose to worry about (maybe only the root!) LIFO with a few (4-5) buffers probably an OK replacement strategy.

A Detailed Simulation Study (Welcome to Wisconsin!)

trace-based & distribution-based: traces used to model individual queries, but workload synthesized based on different distributions. Traces done on a popular benchmark database (from the Wisconsin Benchmark). Queries of 1 and 2 tables. simulator models CPU, one I/O device, and RAM access. Simulation tuned to micro-benchmark of WiSS. Performance metric is query throughput. 3 levels of sharing modeled: full, half, and no sharing Disk arm was jostled around randomly. Memory set big enough to hold about 8 concurrent working sets. statistical confidence intervals on validity of results guarantee things within 5% of the mean (how often do you see that in CS performance studies these days?! Why not?) comparison of RAND, FIFO, CLOCK, HOT, Working Set, DBMIN Bottom line: o as expected, DBMIN is the top line on every graph o Stochastic techniques are no good, though feedback-based load control helps Later work on such load control shows it's VERY tricky in mixed workloads. o WS not a big winner either, though a popular OS choice o DBMIN beats HOT by 7-13% more throughput

Later work: LRU-K, by O'Neil & O'Neil.

SQL Query Optimization

Basics
Given: A query joining n tables The "Plan Space": Huge number of alternative, semantically equivalent plans. The Perils of Error: Running time of plans can vary by many orders of magnitude Ideal Goal: Map a declarative query to the most efficient plan tree. Conventional Wisdom: Youre OK if you avoid the rotten plans. Industrial State of the art: Most optimizers use System R technique and work "OK" up to about 10 joins. Wide variability in handling complex queries with aggregation, subqueries, etc. Wait for rewrite paper.

Approach 1: The Optimization Oracle

(definitely not to be confused with the company of the same name) Youd like to get the following information, but in 0 time:

Consider each possible plan in turn. Run it & measure performance. The one that was fastest is the keeper.

Approach 2: Make Up a Heuristic & See if it Works

University INGRES

always use NL-Join (indexed inner whenever possible) order relations from smallest to biggest

Oracle 6

"Syntax-based" optimization

OK,OK. Approach 3: Think!

Three issues:

define plan space to search do cost estimation for plans find an efficient algorithm to search through plan space for "cheapest" plan

Selinger & the System R crowd the first to do this right. The Bible of Query Optimization.

SQL Refresher
SELECT {DISTINCT} <list of columns> FROM <list of tables> {WHERE <list of "Boolean Factors" (predicates in CNF)>} {GROUP BY <list of columns> {HAVING <list of Boolean Factors>}} {ORDER BY <list of columns>};

Semantics are: 1. take Cartesian product (a/k/a cross-product) of tables in FROM clause, projecting to only those columns that appear in other clauses 2. if theres a WHERE clause, apply all filters in it 3. if theres a GROUP BY clause, form groups on the result 4. if theres a HAVING clause, filter groups with it 5. if theres an ORDER BY clause, make sure output is in the right order 6. if theres a DISTINCT modifier, remove dups Of course the plans dont do this exactly; query optimization interleaves 1 & 2 into a plan tree. GROUP BY, HAVING, DISTINCT and ORDER BY are applied at the end, pretty much in that order.

Plan Space
All your favorite query processing algorithms:

sequential & index (clustered/unclustered) scans NL-join, (sort)-merge join, hash join sorting & hash-based grouping plan flows in a non-blocking fashion with get-next iterators

Note some assumptions folded in here:

selections are "pushed down" projections are "pushed down" all this is only for single query blocks

Some other popular assumptions (System R)

only left-deep plan trees avoid Cartesian products

Cost Estimation
The soft underbelly of query optimization.

Requires:

estimation of costs for each operator based on input cardinalities o both I/O & CPU costs to some degree of accuracy estimation of predicate selectivities to compute cardinalities for next step assumption of independence across predicates decidedly an inexact science. "Selingerian" estimation (no longer state of the art, but not really so far off.) o # of tuples & pages o # of values per column (only for indexed columns) These estimations done periodically (why not all the time?) o back of envelope calculations: CPU cost is based on # of RSS calls, no distinction between Random and Sequential IO o when you cant estimate, use the "wet finger" technique New alternative approaches: o Sampling: so far only concrete results for base relations o Histograms: getting better. Common in industry, some interesting new research. o Controlling "error propagation"

Searching the Plan Space

Exhaustive search Dynamic Programming (prunes useless subtrees): System R Top-down, transformative version of DP: Volcano, Cascades (used in MS SQL Server?) Randomized search algorithms (e.g. Ioannidis & Kang) Job Scheduling techniques In previous years we read many of these (they're fun!), but it is arguably more relevant to talk about query rewriting.

The System R Optimizers Search Algorithm

Look only at left-deep plans: there are n! plans (not factoring in choice of join method) Observation: many of those plans share common prefixes, so dont enumerate all of them Sounds like a job for Dynamic Programming! 1. Find all plans for accessing each base relation o Include index scans when available on "SARGable" predicates 2. For each relation, save cheapest unordered plan, and cheapest plan for each "interesting order". Discard all others. 3. Now, try all ways of joining all pairs of 1-table plans saved so far. Save cheapest unordered 2-table plans, and cheapest "interesting ordered" 2-table plans. o note: secondary join predicates are just like selections that cant be pushed down

4. Now try all ways of combining a 2-table plan with a 1-table plan. Save cheapest unordered and interestingly ordered 3-way plans. You can now throw away the 2-way plans. 5. Continue combining k-way and 1-way plans until you have a collection of full plan trees 6. At top, satisfy GROUP BY and ORDER BY either by using interestingly ordered plan, or by adding a sort node to unordered plan, whichever is cheapest. Some additional details:

dont combine a k-way plan with a 1-way plan if theres no predicate between them, unless all predicates have been used up (i.e. postpone Cartesian products) Cost of sort-merge join takes existing order of inputs into account.

Evaluation:

Only brings complexity down to about n2n-1, and you store plans But no-Cartesian-products rule can make a big difference for some queries. For worst queries, DP dies at 10-15 joins adding parameters to the search space makes things worse (e.g. expensive predicates, distribution, parallelism, etc.)

Simple variations to improve plan quality:

bushy trees: plans, DP complexity is 3n - 2n + 1 + n + 1 need to store n 2 plans (actually its worse for subtle reasons) consider cross products: maximizes optimization time

Subqueries 101: Selinger does a very complete job with the basics.

subqueries optimized separately uncorrelated vs. correlated subqueries o uncorrelated subqueries are basically constants to be computed once in execution o correlated subqueries are like function calls

SQL Query Rewrite & Decorrelation

Problem: Nested SQL queries are not really declarative! Contributing factors:

SQL is not the relational algebra: o duplicate semantics o NULLs o grouping and aggregation o subqueries and correlation

Selinger-style optimization is based on relational algebra (in fact, just selectproject-join)

Solutions:

Toss out SQL Solve the whole optimization enchilada Decompose the problem and solve greedily: o query rewrite to do "stable" rewrite optimizations o Selinger per block (or Selinger++) o execution techniques to minimize the cross-block costs

Rewrite Engine Architecture

An endless supply of rewrite tricks? A perfect application of rule systems

Query Representations

SQL Parse Tree QGM: IBM's Query Graph Model KOLA: a combinator ("variable-free") algebra

The "Notorious" COUNT bug

Won Kim paper on rewrite published in TODS back in '82 perpetrated the following error, exposed by Ganski & Wong, SIGMOD '87:
SELECT D.Name FROM DEPT D WHERE D.Budget <= SELECT COUNT(E.eno)*100000 FROM EMP E WHERE E.Dno = D. Dno; ==> SELECT D.Name FROM DEPT D, EMP E WHERE D.Dno = E.Dno AND D.Budget <= COUNT(E.Eno) GROUP BY D.Name

What's wrong here? Note: Won Kim is a smart guy. This is just tricky, picky stuff! (BTW, this is an example of "decorrelation"!) QGM on One Foot Note: QGM started out as a wacky research idea. It is now IBM corporate religion (i.e. a research success!) If you think this is confusing, imagine it in C++! Yet IBM product division was convinced that the benefits outweigh the cost of complexity.

"boxes" have head (declarative description of output) and body (high-level implementation details) The standard box is the "SELECT" (actually, select/project/join + format output + optionally remove dups) o other boxes are mostly trivial (UNION, INTERSECT, EXCEPT, GROUPBY) o can always add boxes (e.g. OUTERJOIN) body contains "quantifiers" ranging over other boxes o F ("ForEach"): really an iterator. F quantifiers over SELECT boxes are like views (query in FROM clause) o E (Existential): the existential quantifier. like subqueries (in WHERE clause) -- which? o A ("All"): the universal quantifier: like subqueries (in WHERE clause) -- which? quantifiers are connected in a hypergraph o in a SELECT box, hyperedges are Boolean Factors body has a DISTINCT property (Enforce/Preserve/Permit) to describe how to process dups (Permit="don't care") head has a DISTINCT property (T/F) to specify whether or not the output contains dups quantifiers have DISTINCT property (Enforce/Preserve/Permit) to describe whether they require their input to remove or keep dups body has a list of "head expressions" to generate output head has a list of output columns (schema info)

Query Rewrite Rules

Should go from one valid QGM to another, semantically equivalent QGM. Should be "stable", i.e. never raise cost of final plan.

The Main Rewrite Rule: Merge SELECT boxes

This is essentially INGRES' query modification for merging views, updated to handle SQL duplicate semantics.
if (in a SELECT box (upper) a quantifier has type F and ranges over a SELECT box (lower) and no other quantifier ranges over lower and (upper.head.distinct = TRUE || upper.body.distinct = PERMIT || lower.body.distinct != ENFORCE)) { merge lower box into upper; if (lower.body.distinct = ENFORCE && upper.body.distinct != PERMIT) upper.body.distinct = ENFORCE; }

Most of the other rules in the first section exist only to make the "if" here evaluate to TRUE as often as possible, being very careful about duplicate semantics and NULLs (thus avoiding the notorious COUNT bug.)

Decorrelation
Sometimes you can't flatten subqueries. In that case, you should at least decorrelate:

correlated query can be executed just once better for parallelism

Magic Decorrelation details are tricky. The basic idea is simple: 1. order the joins in the upper box (how? when? See Seshadri, et al., SIGMOD '96) 2. generate a subquery containing the "magic" set of correlation-variable bindings 3. this subquery is shared by rest of upper query and by new, decorrelated version of lower query The complexity comes from making this general (intervening boxes of various sorts), and stepwise (i.e. never generates a bad or non-equivalent QGM.)

Concurrency Control 1: Locking and Degrees of Consistency

Transaction Refresher
Statement of problem:

Database: a fixed set of named resources (e.g. tuples, pages, files, whatever) Consistency Constraints: must be true for DB to be considered "consistent". Examples: o sum(account balances) = sum(assets) o P is index pages for R o ACCT-BAL > 0 o each employee has a valid department Transaction: a sequence of actions bracketed by begin and end statements. Each transaction ("xact") is assumed to maintain consistency (this can be guaranteed by the system). Goal: Concurrent execution of transactions, with high throughput/utilization, low response time, and fairness.

The ACID test for transaction management:

Atomicity: Either all actions within a transaction occur, or none do ("all or nothing at all") Consistency: Each transaction takes the database from one consistent state to another. No intermediate states are observable. (We will refine this definition today)

Isolation: Events within a transaction must be invisible to other transactions. This allows transactions to be aborted in isolation (i.e. avoids cascaded aborts) Durability: Once a transaction is committed, its results must be preserved even in the case of failures.

C & I guaranteed by concurrency control. A & D guaranteed by recovery.

A transaction schedule: T0 read(A) A = A - 50 write(A) T1

read(A) temp = A*0.1 A = A - temp write(A) read(B) B = B + 50 write(B) read(B) B = B + temp write(B)

The system "understands" only reads and writes; cannot assume any semantics of other operations. Arbitrary interleaving can lead to:

temporary inconsistencies (ok, unavoidable) "permanent" inconsistencies, that is, inconsistencies that remain after transactions have completed.

Some definitions:

Schedule: A "history" or "audit trail" of all actions in the system, the xacts that performed them, and the objects they affected. Serial Schedule: A schedule is serial if all the actions of each single xact appear together. Equivalence of schedules: Two schedules S1, S2 are considered computationally equivalent if:

1. The set of transactions that participate in S1 and S2 are the same.

2. For each data item Q in S1, if transaction Ti executes read(Q) and the value of Q read by Ti was written by Tj, then the same will hold in S2. [reads are all the same] 3. For each data item Q in S1, if transaction Ti executes the last write(Q) instruction, then the same holds in S2. [the same writers "win"]

Serializability: A schedule S is serializable if there exists a serial schedule S such that S and S are computationally equivalent.

One way to think about concurrency control in terms of dependencies: 1. T1 reads N ... T2 writes N: a RW dependency 2. T1 writes N ... T2 reads N: a WR dependency 3. T1 writes N ... T2 writes N: a WW dependency Can construct a "serialization graph" for a schedule S (SG(S)):

nodes are transactions T1, ..., Tn Edges: Ti -> Tj if there is a RW, WR, or WW dependency from Ti to Tj

Theorem: A schedule S is serializable iff SG(S) is acyclic.

Locking
A technique to ensure serializability, but hopefully preserve high concurrency as well. The winner in industry. Basics:

A "lock manager" records what entities are locked, by whom, and in what "mode". Also maintains wait queues. A well-formed transaction locks entities before using them, and unlocks them some time later.

Multiple lock modes: Some data items can be shared, so not all locks need to be exclusive. Lock compatibility table 1: Assume two lock modes: shared (S) and exclusive (X) locks. S X S T F X F F If you request a lock in a mode incompatible with an existing lock, you must wait. Two-Phase Locking (2PL):

Growing Phase: A transaction may obtain locks but not release any lock.

Shrinking Phase: A transaction may release locks, but not obtain any new lock. (in fact, locks are usually all released at once to avoid "cascading aborts".)

Theorem: If all xacts are well-formed and follow 2PL, then any resulting schedule is serializable (note: this is if, not if and only if!)

Gray, et al.: Granularity of Locks

Theme: Correctness and performance

Granularity tradeoff: small granularity (e.g. field of a tuple) means high concurrency but high overhead. Large granularity (e.g. file) means low overhead but low concurrency. Possible granularities: o DB o Areas o Files o Pages o Tuples (records) o fields of tuples Want hierarchical locking, to allow "large" xacts to set large locks, "small" xacts to set small locks Problem: T1 S-locks a record in a file, then T2 X-locks the whole file. How can T2 discover that T1 has locked the record? Solution: "Intention" locks NL NL IS IX S SIX X IS IX S SIX X

o o o o o

o o

IS and IX locks T1 obtains S lock on record in question, but first gets IS lock on file. Now T2 cannot get X lock on file However, T3 can get IS or S lock on file (the reason for distinguishing IS and IX: if there were only I, T3 couldnt get an S lock on file) For higher concurrency, one more mode: SIX. Intuitively, you read all of the object but only lock some subparts. Allows concurrent IS locks (IX alone would not). Note: gives S access, so disallows IX to others. requires that xacts lock items root to leaf in the hierarchy, unlock leaf to root generalization to DAG of resources: X locks all paths to a node, S locks at least one.

Some implementation background (not in paper):

maintain a lock table as hashed main-mem structure lock/unlock must be atomic operations (protected by critical section) typically costs several hundred instructions to lock/unlock an item suppose T1 has an S lock on P, T2 is waiting to get X lock on P, and now T3 wants S lock on P. Do we grant T3 an S lock?

No! (starvation, unfair, etc.) So... Manage FCFS queue for each locked object with outstanding requests all xacts that are adjacent and compatible are a compatible group The front group is the granted group group mode is most restrictive mode amongst group members Conversions: often want to convert (e.g. S to X for "test and modify" actions). Should conversions go to back of queue? No! Instant deadlock (more notes on deadlock later). So put conversions right after granted group.
o o o o

Gray, et al.: Degrees of Consistency

First, a definition: A write is committed when transaction if finished; otherwise, the write is dirty. A Locking-Based Description of Degrees of Consistency: This is not actually a description of the degrees, but rather of how to achieve them. But its easier to understand (?)

Degree 0: set short write locks on updated items Degree 1: set long write locks on updated items ("long" = EOT) Degree 2: set long write locks on updated items, and short read locks on items read Degree 3: set long write and read locks

A Dirty-Data Description of Degrees of Consistency Transaction T sees degree X consistency if...

Degree 0: T does not overwrite dirty data of other transactions Degree 1: a. T sees degree 0 consistency, and b. T does not commit any writes before EOT

Degree 2: a. T sees degree 1 consistency, and b. T does not read dirty data of other transactions

Degree 3:

a. T sees degree 2 consistency, and b. Other transactions do not dirty any data read by T before T completes. Examples of Inconsistencies prevented by Various Degrees

Garbage reads: T1: write(X); T2: write(X) Who knows what value X will end up being? Solution: set short write locks (degree 0)

Lost Updates: T1: write(X) T2: write(X) T1: abort (restores X to pre-T1 value) At this point, the update to T2 is lost (note: log contains T1.X.oldval, newval) Solution: set long write locks (degree 1)

Dirty Reads: T1: write(X) T2: read(X) T1: abort Now T2s read is bogus. Solution: set long X locks and short S locks (degree 2) Many systems do long-running queries at degree 2.

Unrepeatable reads: T1: read(X) T2: write(X) T2: end transaction T1: read(X) Now T2 has read two different values for X. Solution: long read locks (degree 3)

Phantoms: T1: read range [x - y] T2: insert z, x < z < y T2: end transaction T1: read range [x - y] Z is a "phantom" data item (eek!) Solution: ?? NOTE: two-phase well-formed implies degree-3 consistency. (Why?)

Further reading on consistency levels: Berenson, et al., SIGMOD 1995.

Notes on Deadlock

In OS world, deadlock usually due to errors or overloads In DB/xact world with 2PL, theyre inherent. Most common causes: o Differing access orders T1: X-lock P T2: X-lock Q T1: X-lock Q // block waiting for T2 T2: X-lock P // block waiting for T1
o

lock-mode upgrades T1: S-lock P T2: S-lock P T1: convert S-lock on P to X-lock // block T2: convert S-lock on P to X-lock // block

Usual DB solution: deadlock detection (other option: deadlock avoidance) o Use "waits-for" graph and look for cycles o Empirically, in actual systems the waits-for graph shows: cycles fairly rare cycle length usually 2, sometimes 3, virtually never >3 use DFS to find cycles o When to look for cycles?

a. whenever a xact blocks b. periodically c. never (use timeouts) Typically, in centralized systems, run deadlock detection whenever blocking occurs. Arguably this is cheap: most recently blocked transaction T must be the one that caused the deadlock, so just DFS starting from T In distributed systems, even this can be expensive, so often do periodic detection (more later)

Who to restart ("victim selection") a. b. c. d. e. current blocker youngest XACT least resources used fewest locks held (common) fewest number of restarts

Concurrency Control 2: Optimistic Concurrency Control

Kung & Robinson
Attractive, simple idea: optimize case where conflict is rare. Basic idea: all transactions consist of three phases: 1. Read. Here, all writes are to private storage (shadow copies). 2. Validation. Make sure no conflicts have occurred. 3. Write. If Validation was successful, make writes public. (If not, abort!) When might this make sense? Three examples: 1. All transactions are readers. 2. Lots of transactions, each accessing/modifying only a small amount of data, large total amount of data. 3. Fraction of transaction execution in which conflicts "really take place" is small compared to total pathlength. The Validation Phase

Goal: to guarantee that only serializable schedules result. Technique: actually find an equivalent serializable schedule. That is,

1. Assign each transaction a TN during execution.

2. Ensure that if you run transactions in order induced by "<" on TNs, you get an equivalent serial schedule. Suppose TN(Ti) < TN(Tj). Then if one of the following three conditions holds, its serializable: 1. Ti completes its write phase before Tj starts its read phase. 2. WS(Ti) RS(Tj) = and Ti completes its write phase before Tj starts its write phase. 3. WS(Ti) RS(Tj) = and WS(Ti) WS(Tj) = and Ti completes its read phase before Tj completes its read phase. Is this correct? Each condition guarantees that the three possible classes of conflicts (W-R, R-W, W-W) go one way only. 1. For condition 1 this is obvious (true serial execution!) 2. For condition 2,
o o o

No W-R conflicts since WS(Ti) RS(Tj) = In all R-W conflicts, Ti precedes Tj, since the write phase (and hence the read phase) of Ti precedes that of Tj. In all W-W conflicts, Ti precedes Tj by definition.

For condition 3,
o o o

No W-R conflicts since WS(Ti) RS(Tj) = . No W-W conflicts since WS(Ti) WS(Tj) = . In all R-W conflicts, Ti precedes Tj, since the read phase of Ti precedes the write phase of Tj.

Assigning TN's: at beginning of transactions is not optimistic; do it at end of read phase. Note: this satisfies second half of condition (3). Note: a transaction T with a very long read phase must check write sets of all transactions begun and finished while T was active. This could require unbounded buffer space. Solution: bound buffer space, toss out when full, abort transactions that could be affected.

Gives rise to starvation. Solve by having starving transaction write-lock the whole DB!

Serial Validation Only checks properties (1) and (2), since writes are not going to be interleaved. Simple technique: make a critical section around <get xactno; validate (1) or (2) for everybody from your start to finish; write>. Not great if:
o

write takes a long time

SMP might want to validate 2 things at once if theres not enough reading to do

Improvement to speed up validation: repeat as often as you want { get current xactno. Check if youre valid with everything up to that xactno. } <get xactno; validate with new xacts; write>. Note: read-only xacts dont need to get xactnos! Just need to validate up to highest xactno at end of read phase (without critical section!)

Parallel Validation
Want to allow interleaved writes. Need to be able to check condition (3).
o o o

Save active xacts (those which have finished reading but not writing). Active xacts cant intersect your read or write set. Validation: <get xactno; copy active; add yourself to active> check (1) or (2) against everything from start to finish; check (3) against all xacts in active copy If alls clear, go ahead and write. <bump xact counter, remove yourself from active>.

Small critical section. Problems:

o

a member of active that causes you to abort may have aborted can add even more bookkeeping to handle this can make active short with improvement analogous to that of serial validation

Concurrency Control 3: Locking vs. Optimistic

Agrawal/Carey/Livny Performance Study: Locking vs. Optimistic

Previous work had conflicting results:

Carey & Stonebraker (VLDB84), Agrawal & DeWitt (TODS85): blocking beats restarts Tay (Harvard PhD) & Balter (PODC82): restarts beat blocking Franaszek & Robinson (TODS85): optimistic beats locking

Goal of this paper:

Do a good job modeling the problem and its variants Capture causes of previous conflicting results Make recommendations based on variables of problem

Methodology:

simulation study, compare Blocking (i.e. 2PL), Immediate Restart (restart when denied a lock), and Optimistic (a la Kung & Robinson) pay attention to model of system: o database system model: hardware and software model (CPUs, disks, size & granule of DB, load control mechanism, CC algorithm o user model: arrival of user tasks, nature of tasks (e.g. batch vs. interactive) o transaction model: logical reference string (i.e. CC schedule), physical reference string (i.e. disk block requests, CPU processing bursts). Probabilistic modeling of each. They argue this is key to a performance study of a DBMS.

logical queuing model physical queuing model

Measurements

measure throughput, mostly pay attention to variance of response time, too pick a DB size so that there are noticeable conflicts (else you get comparable performance)

Experiment 1: Infinite Resources

as many disks and CPUs as you want blocking thrashes due to transactions blocking numerous times restart plateaus: adaptive wait period (avg response time) before restart o serves as a primitive load control! optimistic scales logarithmically standard deviation of response time under locking much lower

Experiment 2: Limited Resources (1 CPU, 2 disks)

Everybody thrashes blocking throughput peaks at mpl 25 optimistic peaks at 10 restart peaks at 10, plateaus at 50 as good or better than optimistic at super-high mpl (200), restart beats both blocking and optimistic o but total throughput worse than blocking @ mpl 25 o effectively, restart is achieving mpl 60 o load control is the answer here adding it to blocking & optimistic makes them handle higher mpls better

Experiment 3: Multiple Resources (5, 10, 25, 50 CPUs, 2 disks each)

optimistic starts to win at 25 CPUs o when useful disk utilization is only about 30%, system begins to behave like infinite resources even better at 50

Experiment 4: Interactive Workloads Add user think time.

makes the system appear to have more resources so optimistic wins with think times 5 & 10 secs. Blocking still wins for 1 second think time.

Questioning 2 assumptions:

fake restart biases for optimistic o fake restarts result in less conflict. o cost of conflict in optimistic is higher o issue of k > 2 transactions contending for one item will have to punish k-1 of them with real restart write-lock acquisition o recall our discussion of lock upgrades and deadlock o blind write biases for restart (optimistic not an issue here), particularly with infinite resources (blocking holds write locks for a long time; waste of deadlock restart not an issue here). o with finite resources, blind write restarts transactions earlier (making restart look better)

Conclusions

blocking beats restarting, unless resource utilization is low possible in situations of high think time mpl control important. Restarts adaptive load control is too clumsy, though. false assumptions made blocking look relatively worse Final quote by Wulf!

Concurrency Control 3: Locking vs. Optimistic

Agrawal/Carey/Livny Performance Study: Locking vs. Optimistic
Previous work had conflicting results:

Carey & Stonebraker (VLDB84), Agrawal & DeWitt (TODS85): blocking beats restarts Tay (Harvard PhD) & Balter (PODC82): restarts beat blocking Franaszek & Robinson (TODS85): optimistic beats locking

Goal of this paper:

Do a good job modeling the problem and its variants Capture causes of previous conflicting results Make recommendations based on variables of problem

Methodology:

simulation study, compare Blocking (i.e. 2PL), Immediate Restart (restart when denied a lock), and Optimistic (a la Kung & Robinson) pay attention to model of system: o database system model: hardware and software model (CPUs, disks, size & granule of DB, load control mechanism, CC algorithm o user model: arrival of user tasks, nature of tasks (e.g. batch vs. interactive) o transaction model: logical reference string (i.e. CC schedule), physical reference string (i.e. disk block requests, CPU processing bursts). Probabilistic modeling of each. They argue this is key to a performance study of a DBMS.

logical queuing model physical queuing model

Measurements

measure throughput, mostly pay attention to variance of response time, too pick a DB size so that there are noticeable conflicts (else you get comparable performance)

Experiment 1: Infinite Resources

as many disks and CPUs as you want

blocking thrashes due to transactions blocking numerous times restart plateaus: adaptive wait period (avg response time) before restart o serves as a primitive load control! optimistic scales logarithmically standard deviation of response time under locking much lower

Experiment 2: Limited Resources (1 CPU, 2 disks)

Everybody thrashes blocking throughput peaks at mpl 25 optimistic peaks at 10 restart peaks at 10, plateaus at 50 as good or better than optimistic at super-high mpl (200), restart beats both blocking and optimistic o but total throughput worse than blocking @ mpl 25 o effectively, restart is achieving mpl 60 o load control is the answer here adding it to blocking & optimistic makes them handle higher mpls better

Experiment 3: Multiple Resources (5, 10, 25, 50 CPUs, 2 disks each)

optimistic starts to win at 25 CPUs o when useful disk utilization is only about 30%, system begins to behave like infinite resources even better at 50

Experiment 4: Interactive Workloads Add user think time.

makes the system appear to have more resources so optimistic wins with think times 5 & 10 secs. Blocking still wins for 1 second think time.

Questioning 2 assumptions:

fake restart biases for optimistic o fake restarts result in less conflict. o cost of conflict in optimistic is higher o issue of k > 2 transactions contending for one item will have to punish k-1 of them with real restart write-lock acquisition o recall our discussion of lock upgrades and deadlock o blind write biases for restart (optimistic not an issue here), particularly with infinite resources (blocking holds write locks for a long time; waste of deadlock restart not an issue here). o with finite resources, blind write restarts transactions earlier (making restart look better)

Conclusions

blocking beats restarting, unless resource utilization is low possible in situations of high think time mpl control important. Restarts adaptive load control is too clumsy, though. false assumptions made blocking look relatively worse Final quote by Wulf!

Intro to Recovery
Haerder & Reuter
Recall that Recovery guarantees Atomicity and Durability. Paper outlines issues and options for recovery. Types of Failure

Transaction failure System failure Media failure

Overwriting Options

ATOMIC: a transactions updates become visible on disk all at once. System Rs "shadow paging" scheme did this. Pros/cons? NOT ATOMIC: parts of transactions can be on disk without other parts. Pros/cons?

Buffer Pool Eviction Options

STEAL: a transactions updates may be flushed from the buffer at arbitrary times, since another transaction is allowed to "steal" a buffer pool frame. Pros/cons? NO STEAL: all of a transactions updates remain in the buffer pool until commit time. Pros/cons? FORCE: at commit time, all modified pages are forced (flushed) to disk. Pros/cons? NO FORCE: modified pages may remain in the buffer pool even after commit. Pros/cons?

Log Data

Depending on the option chosen above, need some of REDO and UNDO log records to support recovery. Log records can be logical (e.g. "inserted a tuple t into relation R), or physical (e.g. "byte 74 of page 255 used to be r and now is s"). Pros/cons of each? Physical log records can be before/after images of pages (subpages), or diffs.

Checkpoints

In order to speed up recovery, its nice to have "checkpoint" records that limit the amout of log that needs to be processed during recovery. It can be tricky to do efficient checkpoints.

State of the Art (as exemplified by ARIES)

Focus on speed and generality, rather than simplicity. NOT ATOMIC, STEAL, NO FORCE. This allows the buffer manager to make intelligent (i.e. efficient) decisions about when and what to flush to disk. Log data is "physiological" i.e. some is physical (e.g. B-tree page splits), and some is logical (heap-tuple insertions.)

Many more details in ARIES paper!

Intro to Recovery
Haerder & Reuter
Recall that Recovery guarantees Atomicity and Durability. Paper outlines issues and options for recovery. Types of Failure

Transaction failure System failure Media failure

Overwriting Options

ATOMIC: a transactions updates become visible on disk all at once. System Rs "shadow paging" scheme did this. Pros/cons? NOT ATOMIC: parts of transactions can be on disk without other parts. Pros/cons?

Buffer Pool Eviction Options

STEAL: a transactions updates may be flushed from the buffer at arbitrary times, since another transaction is allowed to "steal" a buffer pool frame. Pros/cons? NO STEAL: all of a transactions updates remain in the buffer pool until commit time. Pros/cons? FORCE: at commit time, all modified pages are forced (flushed) to disk. Pros/cons? NO FORCE: modified pages may remain in the buffer pool even after commit. Pros/cons?

Log Data

Depending on the option chosen above, need some of REDO and UNDO log records to support recovery. Log records can be logical (e.g. "inserted a tuple t into relation R), or physical (e.g. "byte 74 of page 255 used to be r and now is s"). Pros/cons of each? Physical log records can be before/after images of pages (subpages), or diffs.

Checkpoints

In order to speed up recovery, its nice to have "checkpoint" records that limit the amout of log that needs to be processed during recovery. It can be tricky to do efficient checkpoints.

State of the Art (as exemplified by ARIES)

Focus on speed and generality, rather than simplicity. NOT ATOMIC, STEAL, NO FORCE. This allows the buffer manager to make intelligent (i.e. efficient) decisions about when and what to flush to disk. Log data is "physiological" i.e. some is physical (e.g. B-tree page splits), and some is logical (heap-tuple insertions.)

Many more details in ARIES paper!

Distributed DBMS: Overview & Concurrency Control

Goal: a small-scale (dozens of sites) distributed DBMS
1. Location transparency: users dont know where the data is 2. Performance transparency: performance independent of submission site 3. Copy transparency: objects can be copied, and copies are maintained automatically. Critical for availability 4. Transaction transparency: looks like single-site xacts 5. Fragment "transparency": tables can be fragmented to different sites 6. Schema change transparency: schema updates at a single site affect global schema 7. Local DBMS transparency: shouldnt matter what DBMS is running at each site (ha!)

Nobody provides all of this (#7 & #3 are still major research field) IBM Almadens R* was the most "real" DDBMS prototype.

VTAM/CICS provided communication Good optimizer, concurrency control

Others:

SDD-1: done at CCA, never really ran. PDP-10s on Arpanet. Strange design decisions based on unreasonably slow network. Distributed INGRES: UCB, also never ran. Lack of UNIX networking software (pre TCP/IP!). Commercial vendors beginning to provide this functionality

The R* Distributed DBMS

Goals

location transparency local privacy & control local performance on local queries no central dependencies: site autonomy o no central catalogs o no central scheduler o no central deadlock detector

Tables can be

dispersed replicated partitioned: o horizontal partitioning (by predicate) o vertical partitioning (but a key at each partition) snapshots

Table names: user_name@user_site.object_name@birth_site Catalogs store info on local objects, and objects born locally (along with pointers to current sites) Remote catalog info is cached as hints:
o o

invalid cache entries are detected easily and updated catalog entries have version numbers

a catalog entry for a table:

o o o o o

System-Wide Name type info access paths view definition (if its a view) optimizer stats

To find a catalog entry:

1. check local catalog & cache 2. check birth site 3. check the place the birth site points to

An overview of Distributed CC:

Every transaction gets an xactno: site_name.time (or SN) deadlocks: shoot the youngest, largest-numbered xact Commit Protocol you cant just decide to commit:
o o

some site may not be able to you cant distingush inability from slowness!

Two Phase Commit (2PC): 1. 2. 3. 4. coordinator says to participants: want to commit? (N-1 messages) Participants all vote yea/nay (N-1 messages) coordinator tells everyone to commit/abort (N-1 messages) participants acknowledge message (N-1 messages)

This does the right thing, but its pretty costly. Next paper improves this. Distributed Deadlock Detection:

find and resolve local deadlocks pass your waits-for graph around to detect global deadlocks next paper does this more efficiently

Distributed Transactions & Replication

Transaction Management in R*
Unravels details of logging & messages sent. Assumptions

update in place, WAL batched force of log records

Desired Characteristics

guaranteed xact atomicity ability to "forget" outcome of commit ASAP

minimal log writes & message traffic optimized performance in no-failure case exploitation of completely or partially R/O xacts maximize ability to perform unilateral abort

In order to minimize logging and comm:

rare failures do not deserve extra overhead in normal processing hierarchical commit better than 2P

Normal Processing (2PC)

Coordinator Log Messages PREPARE prepare*/abort* VOTE YES/NO commit*/abort* COMMIT/ABORT commit*/abort* ACK end Subordinate Log

since subords force abort (& commit) before ACKing, they never need to ask coord about final outcome. Rule: never need to ask something you used to know; log before ACKing. Guarantees atomicity. Total cost: subords: 2 forced log-writes (prepare/commit), 2 messages (YES/ACK) coord: 1 forced log write (commit), 1 async log write (end), 2 messages/subord (prepare/commit) 2PC & Failures Recovery process per site handles xacts committing at crash, as well as incoming recovery messages. 1. on restart, read log and accumulate committing xact info in main mem 2. if you discover a local xact in the prepared state, contact coord to find out what to do

3. if you discover a local xact that didnt get prepared, UNDO it, write abort record, and forget 4. if a local xact was committing (i.e. this is the coord), then send out COMMIT msgs to subords that havent ACKed. Similar for aborting. Upon discovering a failure elsewhere: If a coord discovers that a subord is unreachable... while waiting for its vote: coord aborts xact as usual while waiting for an ACK: coord gives xact to recovery manager If a subord discovers that a coord is unreachable... if it hasnt sent a YES vote yet: abort ("unilateral abort") if it has sent a YES vote, subord gives xact to recovery manager If a recovery mgr receives an inquiry from a subord in prepared state if main mem info says xact is committing or aborting, send COMMIT/ABORT if main mem info says nothing...? An Aside: Hierarchical 2PC If you have a tree-shaped process graph. root (which talks to user) is a coordinator leaves are subordinates interior nodes are both after receiving PREPARE, propagate it to children vote after children; any NO below causes a NO vote after receiving a COMMIT record, force-write log, ACK to parent, and propagate to children. Similar for ABORT.

Presumed Abort recall... if main-mem info say nothing, coord says ABORT SO... coord can forget a xact immediately after deciding to abort it! (write abort record, THEN forget) abort can be async write no ACKS required from subords on ABORT no need to remember names of subords in abort record, nor write end record after abort if coord sees subord has failed, need not pass xact to recovery system; can just ABORT. Now, look at R/O xacts: subords who have only read send READ VOTEs instead of YES VOTEs, release locks, write no log records logic is: READ & YES = YES, READ & NO = NO, READ & READ = READ if all votes are READ, theres no second phase commit record at coord includes only YES sites Tallying up the R/O work: nobody writes log records

nonleaf processes send one message to children children send one message (to parent) Presumed Commit Idea: lets invert the logic above, since commit is the fast path: require ACK for ABORT, not COMMIT subords force abort records, not commit no information? Presume commit! Problem: 1. 2. 3. 4. 5. subord prepares to commit coord crashes on restart, coord aborts transaction and forgets it subord asks about transaction, coord says "no info = commit!" subord commits, everyone else does not

Solution: coord records names of subords on stable storage before allowing them to prepare ("collecting" record) then it can tell them about aborts on restart everything else analogous (mirror) to Presumed Abort Tallying up the R/O work: nonleaf writes collecting (forced) and commit (async) nonleaf sends one message to all children (PREPARE) children send one message (to parent) Performance analysis in paper: PA > 2PC (> = "beats") PA > PC for R/O transactions for xacts with only one write subord, PC > PA (equal log writes, PA needs an ACK from subord) for n-1 writing subords, PC >> PA (equal logging, but PA forces n-1 times when PC does not commit records of subords. Also PA send n extra messages) choice between PA and PC could be made on a transaction-by transaction basis

Gray, et al on Replication
The Upshot: Deadlock/reconciliation rates grow exponentially with replication factor. This gets even worse with disconnected operation (mobile computers). Background

eager replication (deadlocks) vs. lazy replication (reconciliations) group (update anywhere) vs. master (primary copy) scaleup pitfall: replication looks fine on small demos, dies when you scale up system delusion: lots of inconsistent versions floating around, too hard to reconcile them all

Important Observations

in group mode, every update generates an update at all nodes (i.e. nodes times more work). This generates actions2 more work! Eager Group replication: o deadlocks grow like nodes3 o deadlocks grow like actions5 (where actions = # of actions per transaction) o mobile nodes cannot run when disconnected Eager Master replication o Reduces deadlocks: like a single-site system with higher TPS o Still grow like actions5 Lazy Group replication o transaction which would wait under eager needs to be reconciled here Assume wait is rare. Deadlock is rare2 (i.e. very unlikely) so reconciliation is much more common than deadlock o reconciliations grow like TPS2 (actions nodes)3 Lazy Master replication o like RPC mechanism of previous paper (reads should send read-lock requests to master) o deadlock rate at master grows with (TPS Nodes)2 actions4 o mobile nodes cant run while disconnected

Intuitions for a Solution

checkbook example Lotus Notes example: convergence semantics (with no new updates & connectivity, everybody will eventually get the same state). Uses timestamps. lost update problem (more recent account update wins, old one is lost) o solution: commutative operations (e.g. increment/decrement)

Two-Tier Replication

The world consists of base nodes and mobile nodes mobile nodes contain 2 versions of objects: a (maybe stale) master version, and a tentative version 2 types of xacts: o base xacts work only on master data, involve at most 1 mobile node o tentative xacts work only on local tentative data. Only involve data mastered at base nodes or the local node (no other mobile nodes) o on reconnect: tentative versions are removed tentative xacts are rerun as real xacts before committing the base xacts, an acceptance criterion is used to make sure the results are close enough to the original tentative versions Features: o mobile nodes may make tentative updates o base xacts execute with single-copy serializability o xact is durable when the base xact completes o replicas at all sites converge to base system state

if all xacts commute, there are no reconciliations Their "solution" to the dangers: o use lazy master with timestamps & commutativity to avoid high deadlock rates o use 2-tier replication to handle disconnected operation
o

Distributed Transactions & Replication

Transaction Management in R*
Unravels details of logging & messages sent. Assumptions

update in place, WAL batched force of log records

Desired Characteristics

guaranteed xact atomicity ability to "forget" outcome of commit ASAP minimal log writes & message traffic optimized performance in no-failure case exploitation of completely or partially R/O xacts maximize ability to perform unilateral abort

In order to minimize logging and comm:

rare failures do not deserve extra overhead in normal processing hierarchical commit better than 2P

Normal Processing (2PC)

Coordinator Log Messages PREPARE prepare*/abort* VOTE YES/NO commit*/abort* COMMIT/ABORT Subordinate Log

commit*/abort* ACK end

since subords force abort (& commit) before ACKing, they never need to ask coord about final outcome. Rule: never need to ask something you used to know; log before ACKing. Guarantees atomicity. Total cost: subords: 2 forced log-writes (prepare/commit), 2 messages (YES/ACK) coord: 1 forced log write (commit), 1 async log write (end), 2 messages/subord (prepare/commit) 2PC & Failures Recovery process per site handles xacts committing at crash, as well as incoming recovery messages. 1. on restart, read log and accumulate committing xact info in main mem 2. if you discover a local xact in the prepared state, contact coord to find out what to do 3. if you discover a local xact that didnt get prepared, UNDO it, write abort record, and forget 4. if a local xact was committing (i.e. this is the coord), then send out COMMIT msgs to subords that havent ACKed. Similar for aborting. Upon discovering a failure elsewhere: If a coord discovers that a subord is unreachable... while waiting for its vote: coord aborts xact as usual while waiting for an ACK: coord gives xact to recovery manager If a subord discovers that a coord is unreachable... if it hasnt sent a YES vote yet: abort ("unilateral abort") if it has sent a YES vote, subord gives xact to recovery manager If a recovery mgr receives an inquiry from a subord in prepared state if main mem info says xact is committing or aborting, send COMMIT/ABORT if main mem info says nothing...? An Aside: Hierarchical 2PC If you have a tree-shaped process graph. root (which talks to user) is a coordinator leaves are subordinates

interior nodes are both after receiving PREPARE, propagate it to children vote after children; any NO below causes a NO vote after receiving a COMMIT record, force-write log, ACK to parent, and propagate to children. Similar for ABORT.

Presumed Abort recall... if main-mem info say nothing, coord says ABORT SO... coord can forget a xact immediately after deciding to abort it! (write abort record, THEN forget) abort can be async write no ACKS required from subords on ABORT no need to remember names of subords in abort record, nor write end record after abort if coord sees subord has failed, need not pass xact to recovery system; can just ABORT. Now, look at R/O xacts: subords who have only read send READ VOTEs instead of YES VOTEs, release locks, write no log records logic is: READ & YES = YES, READ & NO = NO, READ & READ = READ if all votes are READ, theres no second phase commit record at coord includes only YES sites Tallying up the R/O work: nobody writes log records nonleaf processes send one message to children children send one message (to parent) Presumed Commit Idea: lets invert the logic above, since commit is the fast path: require ACK for ABORT, not COMMIT subords force abort records, not commit no information? Presume commit! Problem: 1. 2. 3. 4. 5. subord prepares to commit coord crashes on restart, coord aborts transaction and forgets it subord asks about transaction, coord says "no info = commit!" subord commits, everyone else does not

Solution: coord records names of subords on stable storage before allowing them to prepare ("collecting" record) then it can tell them about aborts on restart everything else analogous (mirror) to Presumed Abort Tallying up the R/O work:

nonleaf writes collecting (forced) and commit (async) nonleaf sends one message to all children (PREPARE) children send one message (to parent) Performance analysis in paper: PA > 2PC (> = "beats") PA > PC for R/O transactions for xacts with only one write subord, PC > PA (equal log writes, PA needs an ACK from subord) for n-1 writing subords, PC >> PA (equal logging, but PA forces n-1 times when PC does not commit records of subords. Also PA send n extra messages) choice between PA and PC could be made on a transaction-by transaction basis

Gray, et al on Replication
The Upshot: Deadlock/reconciliation rates grow exponentially with replication factor. This gets even worse with disconnected operation (mobile computers). Background

eager replication (deadlocks) vs. lazy replication (reconciliations) group (update anywhere) vs. master (primary copy) scaleup pitfall: replication looks fine on small demos, dies when you scale up system delusion: lots of inconsistent versions floating around, too hard to reconcile them all

Important Observations

in group mode, every update generates an update at all nodes (i.e. nodes times more work). This generates actions2 more work! Eager Group replication: o deadlocks grow like nodes3 o deadlocks grow like actions5 (where actions = # of actions per transaction) o mobile nodes cannot run when disconnected Eager Master replication o Reduces deadlocks: like a single-site system with higher TPS o Still grow like actions5 Lazy Group replication o transaction which would wait under eager needs to be reconciled here Assume wait is rare. Deadlock is rare2 (i.e. very unlikely) so reconciliation is much more common than deadlock o reconciliations grow like TPS2 (actions nodes)3 Lazy Master replication o like RPC mechanism of previous paper (reads should send read-lock requests to master) o deadlock rate at master grows with (TPS Nodes)2 actions4 o mobile nodes cant run while disconnected

Intuitions for a Solution

checkbook example Lotus Notes example: convergence semantics (with no new updates & connectivity, everybody will eventually get the same state). Uses timestamps. lost update problem (more recent account update wins, old one is lost) o solution: commutative operations (e.g. increment/decrement)

Two-Tier Replication

The world consists of base nodes and mobile nodes mobile nodes contain 2 versions of objects: a (maybe stale) master version, and a tentative version 2 types of xacts: o base xacts work only on master data, involve at most 1 mobile node o tentative xacts work only on local tentative data. Only involve data mastered at base nodes or the local node (no other mobile nodes) o on reconnect: tentative versions are removed tentative xacts are rerun as real xacts before committing the base xacts, an acceptance criterion is used to make sure the results are close enough to the original tentative versions Features: o mobile nodes may make tentative updates o base xacts execute with single-copy serializability o xact is durable when the base xact completes o replicas at all sites converge to base system state o if all xacts commute, there are no reconciliations Their "solution" to the dangers: o use lazy master with timestamps & commutativity to avoid high deadlock rates o use 2-tier replication to handle disconnected operation

DB Machines, Parallel DBMS & Gamma

History
The database machine (taken from Boral/DeWitt '83, "Database Machines: An Idea Whose Time has Passed?")

Processor Per Track (PPT): Examples: CASSM, RAP, RARES. Too expensive. Wait for bubble memory, charge-couple devices (CCDs)? Still waiting...

PP Head: Examples: DBC, SURE. Helps with selection queries. Depends on parallel readout disks (read all heads at once). This has become increasingly difficult over time ("settle" is increasingly tricky as disks get faster and smaller). Off-the-disk DB Machines: Examples: DIRECT, RAP.2, RDBM, DBMAC, INFOPLEX. Precursors of today's parallel DBs. A controller CPU, and special-purpose query processing CPUs with shared disks and memory. o 1981: Britton-Lee Database Machine o "There was 1 6Mhz Z8000 Database processor, up to 4 Z8000 communications processors. Up to 4 "Intellegent" disk controllers (they could optimize their own sector schedule, I think). I think you could put up to 16 .5Gig drives on the thing and probably 4-6 Megs of memory. About a year later we upped the DBP to 10Mhz (yeilding a mighty 1 mips). " -- Mike Ubell

All failed. Why?

these don't help much with sort, join, etc. special-purpose hardware is a losing proposition o prohibitively expensive (no economy of scale) o slow to evolve

Is it time to revisit all this??

IDISK @ Berkeley Think more about this next lecture

Parallel DB 101
Performance metrics:

Speedup Scaleup o Transaction scaleup: N times as many TPC-Cs for N machines o Batch scaleup: N times as big a query for N machines

2 kinds of parallelism
o o

pipelined (most are short) partition

3 barriers to linearity:
o o o

startup overheads interference skew

3 basic architectures
o o

shared-memory shared-disk

shared-nothing

DeWitt et al., GAMMA DB Machine

Gamma: "shared-nothing" multiprocessor system Other shared-nothing systems: Bubba (MCC), Volcano (Colorado), Teradata, Tandem, Informix, IBM Parallel Edition Version 1.0 (1985) o token ring connecting 20 VAX 11/750's. o Eight processors had a disk each. o Issues: Token Ring packet size was 2K, so they used 2K disk blocks. Mistake. Bottleneck at Unibus (1/20 the bandwidth of the network itself, slower than disk!) Network interface also a bottleneck only buffered 2 incoming packets Only 2M RAM per processor, no virtual memory Version 2.0 (1989) o Intel iPSC-2 hypercube. (32 386s, 8M RAM each, 1 333M disk each.) o networking provides 8 full duplex reliable channels at a time o small messages are datagrams, larger ones form a virtual circuit which goes away at EOT o Usual multiprocessor story: tuned for scientific apps OS supports few heavyweight processes Solution: write a new OS! (NOSE) actually a thread package SCSI controller transferred only 1K blocks o ported GAMMA 1.0 some flaky porting stuff o Other complaints want disk-to-memory DMA. As of now, 10% of cycles wasted copying from I/O buffer, and CPU constantly interrupted to do so (13 times per 8K block!) "The Future", circa 1990: CM-5, Intel Touchstone Sigma Machine. "The Future", circa 1995: COW/NOW/SHRIMP with high-speed LAN (Myrinet, ATM, Fast Ethernet, etc.) "The Future", circa 2000??? How does this differ from a distributed dbms? o no notion of site autonomy o centralized schema o all queries start at "host" Storage organization: all relations are "horizontally partitioned" across all disk drives in four ways: o round robin: default, used for all query outputs o hashed: randomize on key attributes o range partitioned (specified distribution), stored in range table for relation o range partitioned (uniform distribution sort and then cut into equal pieces.) o Within a site, store however you wish. o indices created at all sites o A better idea: heat (Bubba?)

hotter relations get partitioned across more sites why is this better? "primary multiprocessor index" identifies where a tuple resides.

Query Execution

An operator tree constructed, each node assigned one or more operator processes at each site. o These operator processes never die Hash-based algorithms only for join & grouping. "Left-deep" trees (I call this "right-deep". Think of it as "building-deep") o pipelines at most 2 joins deep o this was done because of lack of RAM o "right-deep" (my "left-deep", or "probing-deep") is probably better Life of a query: o parsed and optimized at host (Query Manager) o if a single-site query, send to that site for execution o otherwise sent to a dispatcher process (load control) o dispatcher process gives query to scheduler process o scheduler process passes pieces to operator processes at the sites o results sent back to scheduler who passes them to Query Manager for display More detail: o the example query in the paper o split table used to route result tuples to appropriate processors. Three types: Hashed Range-Partitioned. Round-Robin. o selection: uses indices, compiled predicates, prefetching. o join: basic idea is to use hybrid hash, with one bucket per processor. tuples corresponding to each logical bucket should fit in the aggregate mem of participating processors each logical bucket after the first is split across all disks (may be diskless processors) o aggregation: done piecewise. Groups accumulated at individual nodes for final result. Question: can all aggs be done piecewise? o updates: as usual, unless it requires moving a tuple (when?) o control messages: 3x as many as operators in the plan tree scheduler: Initiate operator: ID of port to talk to operator: Done Thats all the coordination needed!! CC: 2PL with bi-granularity locking (file & page). Centralized deadlock detection. ARIES-based recovery, static assignment of processors to log sites. Failure Management: Chained Declustering o nodes belong to relation clusters o relations are declustered within a relation cluster o backups are declustered "one disk off"

tolerates failure of a single disk or processor discussion in paper vs. interleaved declustering these tradeoffs have been beaten to death in RAID world o glosses over how indexes are handled in case of failure (another paper) Simplified picture: Gamma gets parallelism by o Running multiple small queries in parallel at (hopefully) disjoint sites. o Running big queries over multiple sites --- idea here is: Logically partition problem so that each subproblem is independent, and Run the partitions in parallel, one per processor. Examples: hash joins, sorting, ...
o o

Performance results: o a side-benefit of declustering is that small relations mean fewer & smaller seeks o big queries get linear speedup! not perfect, but amazingly close o pretty constant scaleup! o some other observations hash-join goes faster if already partitioned on join attribute but not much! Redistribution of tuples isnt too expensive as you add processors, you lose benefits of short-circuited messages, in addition to incurring slight overhead for the additional processes Missing Research Issues (biggies!): o query optimization (query rewriting for subqueries, plan optimization) o load balancing: inter-query parallelism with intra-query parallelism o disk striping, reliability, etc. o admin utilities & database design o skew handling for non-standard data types

DB Machines, Parallel DBMS & Gamma

History
The database machine (taken from Boral/DeWitt '83, "Database Machines: An Idea Whose Time has Passed?")

Processor Per Track (PPT): Examples: CASSM, RAP, RARES. Too expensive. Wait for bubble memory, charge-couple devices (CCDs)? Still waiting... PP Head: Examples: DBC, SURE. Helps with selection queries. Depends on parallel readout disks (read all heads at once). This has become increasingly difficult over time ("settle" is increasingly tricky as disks get faster and smaller). Off-the-disk DB Machines: Examples: DIRECT, RAP.2, RDBM, DBMAC, INFOPLEX. Precursors of today's parallel DBs. A controller CPU, and special-purpose query processing CPUs with shared disks and memory. o 1981: Britton-Lee Database Machine o "There was 1 6Mhz Z8000 Database processor, up to 4 Z8000 communications processors. Up to 4 "Intellegent" disk controllers (they could optimize their own sector schedule, I think). I think you could put up to 16 .5Gig drives on the thing and probably 4-6 Megs of memory. About a year later we upped the DBP to 10Mhz (yeilding a mighty 1 mips). " -- Mike Ubell

All failed. Why?

these don't help much with sort, join, etc. special-purpose hardware is a losing proposition o prohibitively expensive (no economy of scale) o slow to evolve

Is it time to revisit all this??

IDISK @ Berkeley Think more about this next lecture

Parallel DB 101
Performance metrics:

Speedup Scaleup o Transaction scaleup: N times as many TPC-Cs for N machines o Batch scaleup: N times as big a query for N machines

2 kinds of parallelism
o o

pipelined (most are short) partition

3 barriers to linearity:
o o o

startup overheads interference skew

3 basic architectures

o o o

shared-memory shared-disk shared-nothing

DeWitt et al., GAMMA DB Machine

Gamma: "shared-nothing" multiprocessor system Other shared-nothing systems: Bubba (MCC), Volcano (Colorado), Teradata, Tandem, Informix, IBM Parallel Edition Version 1.0 (1985) o token ring connecting 20 VAX 11/750's. o Eight processors had a disk each. o Issues: Token Ring packet size was 2K, so they used 2K disk blocks. Mistake. Bottleneck at Unibus (1/20 the bandwidth of the network itself, slower than disk!) Network interface also a bottleneck only buffered 2 incoming packets Only 2M RAM per processor, no virtual memory Version 2.0 (1989) o Intel iPSC-2 hypercube. (32 386s, 8M RAM each, 1 333M disk each.) o networking provides 8 full duplex reliable channels at a time o small messages are datagrams, larger ones form a virtual circuit which goes away at EOT o Usual multiprocessor story: tuned for scientific apps OS supports few heavyweight processes Solution: write a new OS! (NOSE) actually a thread package SCSI controller transferred only 1K blocks o ported GAMMA 1.0 some flaky porting stuff o Other complaints want disk-to-memory DMA. As of now, 10% of cycles wasted copying from I/O buffer, and CPU constantly interrupted to do so (13 times per 8K block!) "The Future", circa 1990: CM-5, Intel Touchstone Sigma Machine. "The Future", circa 1995: COW/NOW/SHRIMP with high-speed LAN (Myrinet, ATM, Fast Ethernet, etc.) "The Future", circa 2000??? How does this differ from a distributed dbms? o no notion of site autonomy o centralized schema o all queries start at "host" Storage organization: all relations are "horizontally partitioned" across all disk drives in four ways: o round robin: default, used for all query outputs o hashed: randomize on key attributes o range partitioned (specified distribution), stored in range table for relation o range partitioned (uniform distribution sort and then cut into equal pieces.) o Within a site, store however you wish.

indices created at all sites A better idea: heat (Bubba?) hotter relations get partitioned across more sites why is this better? "primary multiprocessor index" identifies where a tuple resides.
o o

Query Execution

An operator tree constructed, each node assigned one or more operator processes at each site. o These operator processes never die Hash-based algorithms only for join & grouping. "Left-deep" trees (I call this "right-deep". Think of it as "building-deep") o pipelines at most 2 joins deep o this was done because of lack of RAM o "right-deep" (my "left-deep", or "probing-deep") is probably better Life of a query: o parsed and optimized at host (Query Manager) o if a single-site query, send to that site for execution o otherwise sent to a dispatcher process (load control) o dispatcher process gives query to scheduler process o scheduler process passes pieces to operator processes at the sites o results sent back to scheduler who passes them to Query Manager for display More detail: o the example query in the paper o split table used to route result tuples to appropriate processors. Three types: Hashed Range-Partitioned. Round-Robin. o selection: uses indices, compiled predicates, prefetching. o join: basic idea is to use hybrid hash, with one bucket per processor. tuples corresponding to each logical bucket should fit in the aggregate mem of participating processors each logical bucket after the first is split across all disks (may be diskless processors) o aggregation: done piecewise. Groups accumulated at individual nodes for final result. Question: can all aggs be done piecewise? o updates: as usual, unless it requires moving a tuple (when?) o control messages: 3x as many as operators in the plan tree scheduler: Initiate operator: ID of port to talk to operator: Done Thats all the coordination needed!! CC: 2PL with bi-granularity locking (file & page). Centralized deadlock detection. ARIES-based recovery, static assignment of processors to log sites. Failure Management: Chained Declustering o nodes belong to relation clusters

relations are declustered within a relation cluster backups are declustered "one disk off" tolerates failure of a single disk or processor discussion in paper vs. interleaved declustering these tradeoffs have been beaten to death in RAID world o glosses over how indexes are handled in case of failure (another paper) Simplified picture: Gamma gets parallelism by o Running multiple small queries in parallel at (hopefully) disjoint sites. o Running big queries over multiple sites --- idea here is: Logically partition problem so that each subproblem is independent, and Run the partitions in parallel, one per processor. Examples: hash joins, sorting, ...
o o o o

Performance results: o a side-benefit of declustering is that small relations mean fewer & smaller seeks o big queries get linear speedup! not perfect, but amazingly close o pretty constant scaleup! o some other observations hash-join goes faster if already partitioned on join attribute but not much! Redistribution of tuples isnt too expensive as you add processors, you lose benefits of short-circuited messages, in addition to incurring slight overhead for the additional processes Missing Research Issues (biggies!): o query optimization (query rewriting for subqueries, plan optimization) o load balancing: inter-query parallelism with intra-query parallelism o disk striping, reliability, etc. o admin utilities & database design o skew handling for non-standard data types

DB Machines, Parallel DBMS & Gamma

History
The database machine (taken from Boral/DeWitt '83, "Database Machines: An Idea Whose Time has Passed?")

Processor Per Track (PPT): Examples: CASSM, RAP, RARES. Too expensive. Wait for bubble memory, charge-couple devices (CCDs)? Still waiting...

PP Head: Examples: DBC, SURE. Helps with selection queries. Depends on parallel readout disks (read all heads at once). This has become increasingly difficult over time ("settle" is increasingly tricky as disks get faster and smaller). Off-the-disk DB Machines: Examples: DIRECT, RAP.2, RDBM, DBMAC, INFOPLEX. Precursors of today's parallel DBs. A controller CPU, and special-purpose query processing CPUs with shared disks and memory. o 1981: Britton-Lee Database Machine o "There was 1 6Mhz Z8000 Database processor, up to 4 Z8000 communications processors. Up to 4 "Intellegent" disk controllers (they could optimize their own sector schedule, I think). I think you could put up to 16 .5Gig drives on the thing and probably 4-6 Megs of memory. About a year later we upped the DBP to 10Mhz (yeilding a mighty 1 mips). " -- Mike Ubell

All failed. Why?

these don't help much with sort, join, etc. special-purpose hardware is a losing proposition o prohibitively expensive (no economy of scale) o slow to evolve

Is it time to revisit all this??

IDISK @ Berkeley Think more about this next lecture

Parallel DB 101
Performance metrics:

Speedup Scaleup o Transaction scaleup: N times as many TPC-Cs for N machines o Batch scaleup: N times as big a query for N machines

2 kinds of parallelism
o o

pipelined (most are short) partition

3 barriers to linearity:
o o o

startup overheads interference skew

3 basic architectures
o o

shared-memory shared-disk

shared-nothing

DeWitt et al., GAMMA DB Machine

Gamma: "shared-nothing" multiprocessor system Other shared-nothing systems: Bubba (MCC), Volcano (Colorado), Teradata, Tandem, Informix, IBM Parallel Edition Version 1.0 (1985) o token ring connecting 20 VAX 11/750's. o Eight processors had a disk each. o Issues: Token Ring packet size was 2K, so they used 2K disk blocks. Mistake. Bottleneck at Unibus (1/20 the bandwidth of the network itself, slower than disk!) Network interface also a bottleneck only buffered 2 incoming packets Only 2M RAM per processor, no virtual memory Version 2.0 (1989) o Intel iPSC-2 hypercube. (32 386s, 8M RAM each, 1 333M disk each.) o networking provides 8 full duplex reliable channels at a time o small messages are datagrams, larger ones form a virtual circuit which goes away at EOT o Usual multiprocessor story: tuned for scientific apps OS supports few heavyweight processes Solution: write a new OS! (NOSE) actually a thread package SCSI controller transferred only 1K blocks o ported GAMMA 1.0 some flaky porting stuff o Other complaints want disk-to-memory DMA. As of now, 10% of cycles wasted copying from I/O buffer, and CPU constantly interrupted to do so (13 times per 8K block!) "The Future", circa 1990: CM-5, Intel Touchstone Sigma Machine. "The Future", circa 1995: COW/NOW/SHRIMP with high-speed LAN (Myrinet, ATM, Fast Ethernet, etc.) "The Future", circa 2000??? How does this differ from a distributed dbms? o no notion of site autonomy o centralized schema o all queries start at "host" Storage organization: all relations are "horizontally partitioned" across all disk drives in four ways: o round robin: default, used for all query outputs o hashed: randomize on key attributes o range partitioned (specified distribution), stored in range table for relation o range partitioned (uniform distribution sort and then cut into equal pieces.) o Within a site, store however you wish. o indices created at all sites o A better idea: heat (Bubba?)

hotter relations get partitioned across more sites why is this better? "primary multiprocessor index" identifies where a tuple resides.

Query Execution

An operator tree constructed, each node assigned one or more operator processes at each site. o These operator processes never die Hash-based algorithms only for join & grouping. "Left-deep" trees (I call this "right-deep". Think of it as "building-deep") o pipelines at most 2 joins deep o this was done because of lack of RAM o "right-deep" (my "left-deep", or "probing-deep") is probably better Life of a query: o parsed and optimized at host (Query Manager) o if a single-site query, send to that site for execution o otherwise sent to a dispatcher process (load control) o dispatcher process gives query to scheduler process o scheduler process passes pieces to operator processes at the sites o results sent back to scheduler who passes them to Query Manager for display More detail: o the example query in the paper o split table used to route result tuples to appropriate processors. Three types: Hashed Range-Partitioned. Round-Robin. o selection: uses indices, compiled predicates, prefetching. o join: basic idea is to use hybrid hash, with one bucket per processor. tuples corresponding to each logical bucket should fit in the aggregate mem of participating processors each logical bucket after the first is split across all disks (may be diskless processors) o aggregation: done piecewise. Groups accumulated at individual nodes for final result. Question: can all aggs be done piecewise? o updates: as usual, unless it requires moving a tuple (when?) o control messages: 3x as many as operators in the plan tree scheduler: Initiate operator: ID of port to talk to operator: Done Thats all the coordination needed!! CC: 2PL with bi-granularity locking (file & page). Centralized deadlock detection. ARIES-based recovery, static assignment of processors to log sites. Failure Management: Chained Declustering o nodes belong to relation clusters o relations are declustered within a relation cluster o backups are declustered "one disk off"

tolerates failure of a single disk or processor discussion in paper vs. interleaved declustering these tradeoffs have been beaten to death in RAID world o glosses over how indexes are handled in case of failure (another paper) Simplified picture: Gamma gets parallelism by o Running multiple small queries in parallel at (hopefully) disjoint sites. o Running big queries over multiple sites --- idea here is: Logically partition problem so that each subproblem is independent, and Run the partitions in parallel, one per processor. Examples: hash joins, sorting, ...
o o

Performance results: o a side-benefit of declustering is that small relations mean fewer & smaller seeks o big queries get linear speedup! not perfect, but amazingly close o pretty constant scaleup! o some other observations hash-join goes faster if already partitioned on join attribute but not much! Redistribution of tuples isnt too expensive as you add processors, you lose benefits of short-circuited messages, in addition to incurring slight overhead for the additional processes Missing Research Issues (biggies!): o query optimization (query rewriting for subqueries, plan optimization) o load balancing: inter-query parallelism with intra-query parallelism o disk striping, reliability, etc. o admin utilities & database design o skew handling for non-standard data types

The Intelligent Disk (IDISK): A Revolutionary Approach to Database Computing Infrastructure

Kimberly Keeton, David A. Patterson, Joseph Hellerstein, John Kubiatowicz, and Katherine Yelick

UC Berkeley Computer Science Division {kkeeton,patterson,jmh,kubitron,yelick}@cs.berkeley.edu http://iram.cs.berkeley.edu/

2

Motivation: Server Market Breakdown

1995 Server Market Volume
32% 32% 16% 9% 6% 5% Database File server Scientific & engineering Print server Media & email Other

2000 Server Market Volume

39% 25% 14% 7% 8% 7% Database File server Scientific & engineering Print server Media & email Other

Source: Stenstrom, et al., IEEE Computer, December 1997

2
3

Motivation: Decision Support Databases

Characteristic DSS OLTP Business question Historical: support for forming business decisions Operational: day-today business transactions Industry benchmark TPC-D TPC-C Query complexity Long, very complex queries Short, moderately complex queries Portion of DB accessed per query Large Small Type of data access Read-mostly Read-write How are updates propagated? Periodic batch runs or background trickle streams Through most

transaction types
4

Motivation: Database Demand vs. Processor/DRAM speed and Disk Capacity

1 10 100 1996 1997 1998 1999 2000

Proc speed, disk capacity 2X / 18 months Processor-Memory Performance Gap: Database I/O demand: 2X / 9 months DRAM speed 2X /120 months DB-Proc./Disk Gregs Law Performance Gap: Moores Law 3
5

Motivation: Increasing Compute and I/O Needs

V Gregs Law: Greg Papadopoulos, CTO, Sun Microsystems
l l

DSS database I/O demand growth: 2X / 9 months I/O capacity and associated processing Collect richer data (i.e., more detailed)

V Contributing factors:
l

Just-in-time inventory: connect sales to suppliers

Keep longer historical record Growth of digital data l Business consolidation

l l

V Winter VLDB Survey (1997):

Telecomm., retail & financial DBs ~doubled from 1996 to 1997 Wal-Mart says that a major obstacle to its VLDB plans is that hardware vendors can barely keep up with its growth!
l l

Motivation: Intelligent Disk (IDISK)

V IDISK: Processor+memory+fast network per disk V Push processing to data, rather than data to CPU V Allows processing of system to scale with increasing storage demand V Fast network allows direct IDISK-to-IDISK comm. V Trades expensive central processor MIPS for less expensive disk processor MIPS IRAM 4
7

Outline
V Decision support DB servers today

V V V V
8

Computer architecture trends IDISK proposal for decision support databases Case study: TPC-D Q1 Conclusions

Current Decision Support Server Architecture

Sun Enterprise 10k (Oracle 8):
1 TB TPC-D result SMP: 64 CPUs, 64GB DRAM, 603 disks Disks,encl. $2,348k DRAM $2,328k Boards,encl. $983k CPUs $912k Cables,I/O $139k Misc. $65k HW total $6,775k
l l

data crossbar switch 4 address buses

12.4 GB/s
s c s i s c s

i bus bridge s c s

1
s c s i s c s i s c s i

bus bridge

23 Mem Xbar
bridge

Proc
s

1 PPrroocc Proc Mem Xbar

bridge

Proc
s

16 PPrroocc Proc

2.6 GB/s 6.0 GB/s

s c s i s c s i s c s i s c s i

Enet Enet

5
9

Limitations of Current DSS Architecture

V Experimentally, central CPUs are the bottleneck
l

Processing, storage capacity dont scale easily w/ Gregs Law Somewhat better memory system behavior than TPC-C

V Desktop processors not tailored to DB applications

l

CPI for TPC-D query 1 ~1.35 (Pentium Pro) CPI for TPC-C ~3.39 (Pentium Pro)

V Limited I/O bus growth rates V Memory system performance: bandwidth and latency V Expensive!
l l

Central processors and dense memory Cabinets and plumbing handle max configuration

10

Execution Characteristics of TPC-D Workload

V Experimentally, central CPUs are the bottleneck
l

Somewhat better memory system behavior than TPC-C

CPI for TPC-D query 1 ~1.35 CPI for TPC-C ~3.39

V Typical operations
Scan, aggregates (e.g., min, max, count) & 1-pass, 2-pass sort/join Avg. 500-2000 instructions/record (DBMS-, query-specific) l 200 B/record; <= 6 tables/query l Records are read 1-2x, written 0-1x
l l

V I/O pattern tends to be sequential

l l

8KB - 4MB reads; 8KB - 64KB writes (DBMS-specific) Index scan may be more random (DBMS-specific)

V Full storage requirements about 2-5x database size 6

11

Architecture Today: Disk Trends

V Increased disk-resident memory
l

Ex: Seagate Cheetah drive: 1 MB RAM, 4 MB optional ASIC for ECC, SCSI General purpose processor next? Intels Intelligent I/O (I20) initiative

V Increased disk-resident processing

l l

(see NSIC/NASD)
l

V Fast serial lines replacing busses

Fibre Channel Arbitrated Loop (FC-AL), Serial Storage Architectures (SSA) l Intels Gbit/sec serial bus s follow-on to 64b, 66 MHz PCI
l

V More modularized disk design

12

Architecture Today: Communication Trends

V Serial communication advances
l

Fast (Gbps) serial I/O lines [YangHorowitz96], [DallyPoulton96] Standardized I/O devices Switched Ethernet, ATM, Myrinet Fast single-chip switches

State of the art: 4 - 5 Gbps

l

V Switched networks overtake bus-based networks

l l

State of the art: 16-way, GHz / link [Seitz98]

7
13

Architecture Today: Processor Trends

V Processors designed for desktop
l l

Desktop processors have volume Servers also use desktop MPUs

V Desktop processors targeted towards SPEC, Windows apps V Desktop processors less effective for database workloads V Rise of the embedded processor
l

Embedded vs. desktop Dhrystone, SPECint95: < 2x differences

Ex: R5000 vs. R10K: SPECInt95 ratio: 1.6 Dhrystone ratio: 1.0
l l

Fraction of cost Order of magnitude lower power Mitsubishi, LSI Logic, NeoMagic Berkeleys IRAM project

V Integrated logic and DRAM on same chip

l l

14

Embedded vs. Desktop Processors

Processor Digital SA-110 MIPS R5000 MIPS R10000 Digital

21164 Intel Pentium II

Clock rate 233 MHz 200 MHz 200 MHz 600 MHz 300 MHz Cache size 16K/16K 32K/32K/ 512K 32K/32K/ 4M 8K/8K/96K/ 2M 16K/16K/ 512K IC process 0.35 3M 0.35 3M 0.35 4M 0.35 4M 0.28 4M Die size 50 mm2 84 mm2 298 mm2 209 mm2 203 mm2 SPEC95 base (i/f) n/a 4.7/4.7 10.7/17.4 16.3/19.9 11.6/6.8 Dhrystone 268 MIPS 260 MIPS 203 MIPS 920 MIPS (est.) n/a Power 0.36 W 10 W 30 W 25 W 30 W Est. mfrs. cost $18 $25 $160 $125 $90 Source: Microprocessor Report, Summer 1997

Berkeley IRAM Target Parameters

Characteristic IRAM-I (1999) IRAM-II (2002)
DRAM Generation 256 Mbit 1 Gbit On-Chip Memory (MB) 24 96 On-Chip Memory B/W (GB/s) 50 200 50 200 I/O B/W via N Serial Lines (GB/s) 0.5 2.0 0.5 4.0 Individual Serial Line B/W (GB/s) 0.25 0.5 On-Chip Memory Latency (ns) 20 30 20 30 Processor Speed (MHz) 300 500 500 1000 Vector Performance (GFLOPS) 4 16 Vector CPU + $ I/O MM MM Net I/F

+ High on-chip memory B/W, low on-chip memory latency - Small on-chip memory capacity
15 16

Proposal: Intelligent Disk (IDISK)

V Processor+DRAM+fast network per disk: IRAM! V Move function to data instead of data to CPU
Traditional relational operators: scan, sort, join, Newer object-relational operators: image, audio, l Other functionality: reorganize multi-dim. data, DB loading
l l

V Potential benefits for DSS databases:

Offloads processing from overutilized CPU to disk processors Reduces data movement through I/O system l Allows processing of system to scale with increasing storage demand
l l

V Prediction: IDISK variant can be future commodity disk part/option

l

Need to demonstrate high performance at low cost, power

9
17

Disk Processing for Databases Not a New Idea

V Late 70s, early 80s: database machines V Several flavors: central host processor +
l l

Processor per {head, track, disk} Multiprocessor cache

V After much enthusiasm, failed because:

Didnt use commodity hardware Tremendous performance gains for scans didnt justify cost l Provided performance improvements only for simple operations (e.g., scans), but not for more complicated operations (e.g., joins)
l l

V Why should IDISK succeed now?

l l

Disk with processor+memory can be commodity part Algorithmic advances of last 15 - 20 years

Parallel, cluster-based (shared nothing) join, sort algorithms

18

Evolutionary IDISK Architecture (ISTORE)

V 1 IRAM/disk V I/O interconnect: crossbar with serial lines V Trade inexpensive IDISK processing for expensive central MPU processing V Retain centralized processing
Simplify programming model l Accept and optimize user queries l Assists in executing queries too complex for IDISK alone
l

data crossbar switch 4 address buses

Mem Proc Xbar 1 PPrroocc Proc

cross bar

IRAM IRAM IRAM IRAM

IRAM IRAM IRAM IRAM

cross bar
Enet s

10
19

IDISK Software Architecture

V What is software model for IDISK? Alternatives:
Run complete DB server + OS on each disk processor Run all of storage/data manager on each disk node l Run small portion of storage manager on each disk node (*)
l l

Each disk contains library of kernel operations (scan, join, etc.) Download arbitrary user code Use secure programming environment (e.g., Java)?

V Leverage algorithms that trade I/O bandwidth for memory capacity

How well will these work in very memory-constrained environment?
l

20

Case Study: Future SMPs

SMP Characteristic 1 TB System 3 TB System 10 TB System Processors 64 * 1000 MHz 64 * 1000 MHz 64 * 1000 MHz Memory capacity 64 GB 256 GB 256 GB SMP Interconnect B/W 30,000 MB/s 30,000 MB/s 30,000 MB/s Memcpy B/W 15,000 MB/s 15,000 MB/s 15,000 MB/s Disk capacity 200 * 36 GB 600 * 36 GB 1800 * 36 GB Disk transfer rate 29 MB/s 29 MB/s 29 MB/s I/O interconnect 16*2*64b,66 MHz PCI 16*2*64b,66 MHz PCI 16*2*64b,66 MHz PCI I/O interconnect B/W 9600 MB/s 9600 MB/s 9600 MB/s

11
21

Case Study: Future IDISK Servers

IDISK Characteristic 1 TB System 3 TB System 10 TB System Processors 4 * 1000 MHz 4 * 1000 MHz 4 * 1000 MHz Memory capacity 8 GB 8 GB 8 GB SMP Interconnect B/W 30,000 MB/s 30,000 MB/s 30,000 MB/s Disk capacity 200 * 36 GB 600 * 36 GB 1800 * 36 GB IDISK Memcpy B/W 200*5,000 MB/s 600*5,000 MB/s 1800*5,000 MB/s IDISK interconnect B/W 200*2,000 MB/s 600*2,000 MB/s 1800*2,000 MB/s Disk transfer rate 29 MB/s 29 MB/s 29 MB/s Disk processor speed 500 MHz 500 MHz 500 MHz Disk memory 32 MB 32 MB 32 MB IDISK serial lines 8*2Gbit/s 8*2Gbit/s 8*2Gbit/s
22

V Scan 95% - 97% of largest table (lineitem) and compute aggregates V Tremendous reduction in data movement

Case Study for IDISK: TPC-D Query 1

Scale Factor 1 TB 3 TB 10 TB Lineitem cardinality (rows) ~6 bill. ~18 bill. ~60 bill. Lineitem size/Total data moved in SMP (GB) ~870 ~2610 ~8700 Total data moved in IDISK (KB) ~59 ~176 ~527 12
23

Case Study for IDISK: TPC-D Query 1

Inst. per tuple 200 500 1000 1500 2000 3000 6000 1 TB SMP 157 (s) 157 157 215 285 426 849 IDISK 144 144 144 144 159 253 505 IDISK speedup 1.1x 1.1x 1.1x 1.5x 1.7x 1.7x 1.7x 3 TB SMP 293 293 418 620 823 1228 2443 IDISK 144 144 144 144 169 253 505 IDISK speedup 2.0x 2.0x 2.9x 4.3x 4.9x 4.9 4.8x 10 TB SMP 922 922 1329 1987 2645 3961 7910 IDISK 160 160 160 160 188 281 562 IDISK speedup 5.8x 5.8x 8.3x 12.4x 14.1x 14.1x 14.1x

*Table shows seconds to scan and process lineitem table *Speedups from embarrassingly parallel nature of task *IDISK processing scales better than SMP processing
24

Ongoing Research & Open Questions (I)

V How does IDISK server performance and price compare to
l l

Cluster-based shared nothing server? (e.g., NCR WorldMark) CC-NUMA architecture? (e.g., SGI Origin, Sequent NUMA-Q)

V What is right ratio between embedded processors, memory and disks? V What is performance of large-scale (600-1000 nodes) serial line interconnect? 13
25

Ongoing Research & Open Questions (II)

V How much processing can we push down into disk?
l l

Scans and certain object manipulations are obvious wins What about sort, join, and aggregation operations?

V Does Amdahls Law limit IDISK performance gains?

Exactly how much time is spent doing operations that could be pushed to disk? l Answer question by profiling commercial database and doing atomic benchmarking
l

V Whats the right programming model? How to safely download code into disk? V How do we get commercial databases to modularize code so that operations can be downloaded to disk processor?
26

Conclusions
V Decision support databases
Increasingly important workload Storage and related computation requirements growing faster than processor speed increases l Central server processors saturated: current system bottleneck
l l

V IDISK offers architectural alternative

Push processing to disk, rather than bringing data to CPU Allows processing of system to scale with increasing storage demand l Overcomes pitfalls of previous research attempts
l l

V IDISK advantages
Improved performance from exploiting data parallelism Incredible reduction in data movement l Reduced cost: trade expensive MIPs for cheap MIPs
l l

V Evolutionary path to completely decentralized system 14

27

Backup Slides
(These slides used to help answer questions.)
28

Revolutionary IDISK: Scalable Decision Support?

6.0 GB/s
V 1 IRAM/disk + xbar + fast serial link v. conventional SMP V Network latency = f(SW overhead), not link distance V Move function to data v. data to CPU (scan, sort, join,...) V Looks like cluster (shared nothing) V Cheaper, faster, more scalable (~1/3 $, 3X perf)

cross bar

IRAM IRAM IRAM IRAM

IRAM IRAM IRAM IRAM

75.0 GB/s
cross bar cross bar cross bar cross bar IRAM Enet 15
29

Berkeley IRAM Vision Statement

Microprocessor & DRAM on a single chip:
on-chip memory latency 5-10X, bandwidth 50-100X l improve energy efficiency 2X-4X (no off-chip bus) l serial I/O 5-10X v. buses
l

l l

smaller board area/volume adjustable memory size/width D

R A M f a b Proc Bus DRAM $$ Proc L2$ L o g i c f a Bus b DRAM I/O I/O I/O I/O Bus
30

V-IRAM-2: 0.13 m, Fast Logic, 1GHz 16 GFLOPS(64b)/64 GOPS(16b)/128MB

Memory Crossbar Switch M M M M M M M M M M M M M M M M M M M M M M M M M

M M M M M + Vector Registers x

Load/Store 8K I cache 8K D cache 2-way Superscalar Vector Processor 8 x 64 8 x 64 8 x 64 8 x 64 8 x 64 8 x 64 or 16 x 32 or 32 x 16 8 x 64 8 x 64 Queue Instruction

I/O I/O I/O I/O

Serial I/O

16
31

V-IRAM Benefits Database Operations

V Vectorized radix sort (Zagha and Blelloch, Supercomputing 91) V Vectorized hash join (Rich Martin, UCB) V Data mining
l

Statistical operations looking for trends in data

V Image/video object manipulations

Format conversion Compression l Query by
l l

32

TPC-D Q1
SELECT L_RETURNFLAG, L_LINESTATUS, SUM(L_QUANTITY) AS SUM_QTY, SUM(L_EXTENDEDPRICE) AS SUM_BASE_PRICE, SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)) AS SUM_DISC_PRICE, SUM(L_EXTENDEDPRICE*(1-L_DISCOUNT)*(1+L_TAX)) AS SUM_CHARGE, AVG(L_QUANTITY) AS AVG_QTY, AVG(L_EXTENDEDPRICE) AS AVG_PRICE, AVG(L_DISCOUNT) AS AVG_DISC, COUNT(*) AS COUNT_ORDER FROM LINEITEM WHERE L_SHIPDATE <= DATE 12/1/98 - INTERVAL delta DAYS GROUP BY L_RETURNFLAG, L_LINESTATUS ORDER BY L_RETURNFLAG, L_LINESTATUS;

17
33

1995 Market Volume by Machine Price

1 55

2 3 19 25 24 9 8 0 5 10 15 20 25 30 <$10K <$50K <$250K <$1M >$1M Scientific & engineering Commercial

Source: Stenstrom, et al., IEEE Computer, December 1997

34

TPC-D Performance Metrics

V Power (QppD)
single user query processing power (1 * 3600 * SF) / geometric_mean(Q1Q17,UF1,UF2) l SF = scale factor (e.g., 1, 10, 30, 100, 300, 1000, 3000 GB)
l l

V Throughput (QthD)
multi-user query throughput (S * 17 * 3600 * SF) / Ts l S = # concurrent users, each executing all 17 read-only queries l Ts = total elapsed time
l l

V Queries per hour (QPhd)

l

square_root(QppD * QthD)

Extensibility & Object-Relational Systems

Motivation & Politics
In the early 80s, it became clear that Relational systems were not robust enough for non-administrative data-intensive applications of the day:
o o o o

CAD/CAM CASE GIS etc.

Two buzz-phrases began to emerge: "Object-Oriented" and "Extensible"

Much vision & politics ensued:

o o o o

Various data models (NF2, ER, Functional, Semantic) Object-Oriented DB System Manifesto (OO-ness) Third-Generation DB System Manifesto (Extensibility) Many query languages proposed

Systems were built, companies started, etc. Today, the field has settled down into two arenas:
o o o

Persistent OO PL systems (e.g. EXODUS, ObjectStore, Objectivity, Versant, etc.) Query-based systems with OO features (e.g. Starburst, Postgres, Illustra, Informix & ORacle "Universal Servers", DB/2 UDB) Almost nobody does both well

Few people continue to argue in terms of the paradigms. (Exhaustion?)

Systems History
Three influential research systems:
o o o o

Starburst (IBM Almaden) POSTGRES (Berkeley) EXODUS (Wisconsin) Others include O2 (Altair), ORION (MCC), Iris (HP), Genesis (Texas)

We will focus on POSTGRES & EXODUS, discuss Starburst briefly.

EXODUS
o

o o o

EXODUS was intended to be both a persistent PL system and a query system "Toolkit" extensibility query processing engine never got built, though the EXODUS optimizer architecture was influential (Graefe & DeWitt) Ended up focusing on OODB stuff, so we will discuss it more later. SHORE (follow-on to EXODUS) is delivering on EXODUS promises, but rather late. Persistent C++ Query Processing (w/ GIS features): Paradise Extensible Optimizer: Opt++ Parallelism We'll see Exodus/SHORE work on pointer swizzling, client-server caching

Object-Relational Systems

Informix's buzzphrase for Illustra and look-alikes (courtesy Roger Sippl). Didn't patent the name! Query-based, extensible systems with some OO features like inheritance and OIDs. SQL3 draft is very much like this. Stonebrakers application matrix:

Query No Query

RDBMS File Sys. Simple Data

ORDBMS OODBMS Complex Data

Argues that the upper right is growing, and will engulf upper left and lower right.

Hot Applications:
o o o o o

Web servers, full-text collections Time-series data "Asset Management" GIS image DB

Players:
o o o o o o

Informix Universal Server (head of illustra, body of informix). Shipping now. IBM DB2 UDB (head of Starburst, body of DB2). Extensibility features coming along. UniSQL (Won Kim of ORION fame). Went out of business recently. Oracle Universal Server (marketing-ware). Shipping now. NCR (Teradata) just bought Wisconsin's Paradise ORDBMS (and DeWitt/Naughton/students) Other big R vendors are late (Sybase, Tandem, etc.) O vendors still not running queries

Overview: Things needed in an Object-Relational DBMS

(From the "Third-Generation Database System Manifesto")

The relational data model (as implemented!) is "semantically impoverished":

o o o

fixed set of base types (integer, float, etc) only structuring allowed is normal-form relations only operations are relational algebra, using comparators from base types

Instead, people want a "semantically rich" data model

o o o o

extensible ADTs complex types based on type constructors, and methods for those types inheritance "Object identity"

Zaniolo: GEM
One of the nicest, cleanest language papers, influential in the design of objectrelational data models and query languages (starting with Postquel)

ref types, nested dots, implicit joins (implication about OIDs) generalization hierarchies (converted to "collection inheritence") set-valued attrs (non-1st-normal-form!)

Stonebraker: Extensible ADTs

Seminal paper. Idea: you should be able to add new atomic types to the system, along with methods for the types, and new access methods. Type is defined by:
o o o o

storage size (can be variable) input method output method any other methods a user wishes to provide

Then you could do standard relational processing over those types. Example: add 2d spatial operators to RDBMS Engineering issues (some from the paper, some not):
o o

Parsing must know about user-defined types and methods (table-driven) Optimization must be able to compute selectivity for user-defined predicates must know about cost of user-defined methods, and consider predicate pullup must know how to match user-defined predicates to fancy new access methods

o o

must know whether user-defined join predicates can be evaluated by hash or merge Execution: must have dynamic linking support (24x7 operation) methods called via "function pointers", "functors" or some similar construct can slow down standard processing support for "untrusted" functions (hot topic today) support for "large objects" caching for expensive methods (a la subquery caching) extensible aggregations 3 functions: init, iter, end Access Methods Stonebraker: An access method is a generic object that provides open, get-first, get-next, close, insert, delete if its fancy, it takes SARGs and evaluates them quickly needs to provide cost estimates to optimizer Problems integration with CC can be solved by physical logging can open up logging interface for AM-specific logging integration with Buffer Manager In practice, almost nobody used the Postgres Access method extensibility GiST! Buffer Management Large objects require new schemes Transactions integration with access methods

Note: Extensible ADTs do not fundamentally change a relational system

o o o

they fit naturally with the relational model as Stonebraker shows, they fit naturally into RDBMS query processing for relational "believers" this is all you should need to solve all problems

POSTGRES
Stonebraker, Rowe, a few staff and many students, 1986-1994. Post-INGRES. The Postgres Data Model 1. Co-opt the OO terminology o class = relation o instance = tuple o object-id = tuple-id o method = attribute or function of attributes 2. Support extensible ADTs o extensible procedures using C functions

binary operators, which interface to extensible AM 3. Support type constructors o trick: use queries columns can be parameterized Postquel functions (returns setof, or tuple) queries can live in fields of a tuple (returns setof or tuple) another exploitation of the view paradigm! these derived objects can optionally be cached (never implemented) nested-dots used to traverse complex object structures leverages EXISTING techniques for relational processing. Cute! o added array support directly 4. added class inheritance (gives method inheritance and collection hierarchies)
o

Implementation Details

originally written in LISP, then ran Lisp2C, resulting in a horrible built-in inheritance mechanism over C only access methods added to Postgres were done "in house" o B-tree and R-tree early o linear hashing late o GiST added in the last couple years "Fast Path" to AMs, as an alternative to "Persistent X" o never well-documented or used outside Berkeley ADTs as described above No Overwrite Storage, time travel, etc Research project was "shut down" in 94. 2 Berkeley students did a major cleanup (remove lispisms, remove a number of theses), migrated to SQL, and released Postgres95. This was picked up by freeware hackers on the net, and now PostgreSQL seems to be the serious freeware db of choice (www.postgresql.org).

More Stuff

Postgres Rule System: Active Database support (we will skip this topic this year) Shared-mem parallel version, with new optimization techniques "Inversion" file system PICASSO UI (Rowe & students) Support for tertiary memory Method Indexing Partial Indexes Expensive predicate optimization Commercialized as Illustra Concepts ported into Informix

Editorial Comments

Postgres code was a mess. Miraculous that it worked. More miraculous that Illustra salvaged it! Data model was sloppy but clever. Burst some OO bubbles. o Roger King: "My cat is object-oriented"

Missed an important distinction between Class and Collection. No-overwrite storage was not as bad as you think. Expect the concept to resurface. In many ways, the most successful of the 3 projects o used to be written off as goofy research, but Illustra is now ahead of the game o ADT extensibility & dynamic linking are very useful Illustras "DataBlades", Oracle's "Data Cartridges" o though not all aspects worked well, almost all were novel and influential o Many radical ideas in one system!
o

Starburst
Original goal: build a nice playpen for whatever comes next. Extensible "in-house". Not by users! No one survey paper seems to capture the work they did. Best bet: "Starburst MidFlight: As The Dust Clears", Haas, et al., TKDE 1990 Plumbing:
o o o o o o o

clean internal query representation (QGM). Key to Query Rewrite! non-normalized catalogs for efficiency normalized view for users uniform record structure across RSS & RDS a single expression evaluator for RSS & RDS WAL instead of shadow pages B+-tree compression Buffer Pool Manager accepts hints from optimizer (a la DBMIN)

Extensibility features:
o

o o

User-defined functions: table expressions: queries or C functions scalar functions no dynamic linking Rule-based query rewrite engine a little rule system with QGM as "working memory" conditions and actions are C functions that check and change QGM some nifty rule control mechanisms (rule classes, rule budgets, multiple conflict res.) Extensible access methods (as in POSTGRES) "Attachments": routines to be automatically called before/after dealing with an access method used by Starburst Rule System to generate transition logs used to implement pre-computed joins (see below) Complex objects implemented in a "wrapper" (SQL-XNF), translated down to Starburst SQL

"Proofs" of Extensibility:
o

o o o o o

added "signature" attachment to automatically tag tuples with some derived values though writing the attachment was hard, hooking it in took only 1 day added Outer Join not so simple required adding things in QGM, optimizer, and executor IMS attachment pre-computed joins using pointers, which are maintained written by "outsiders" Mike Carey & Beau Shekita from Wisconsin in a summer visit to Almaden Grammar-like rule-based query optimizer combined with System R bottom-up construction 2nd Rule System called "Alert", based on infinitely-running queries. Lightweight and pretty effective. Persistent C++ support (SMRC) Recursive query processing! Lots of technology for query rewriting

Many of the Starburst folks took a "vacation" from research the last few years, and merged Starburst technology into DB2 UDB. Editorial Comments
o o o o o

Probably the best code of the bunch The worlds most effective query optimizer? The DB2 experience good for all concerned Attention paid to details, shows up in complex parts of system A long-term advantage? Depends on market, IBMs development culture...

Extensibility & Object-Relational Systems

Motivation & Politics
In the early 80s, it became clear that Relational systems were not robust enough for non-administrative data-intensive applications of the day:

o o o o

CAD/CAM CASE GIS etc.

Two buzz-phrases began to emerge: "Object-Oriented" and "Extensible" Much vision & politics ensued:
o o o o

Various data models (NF2, ER, Functional, Semantic) Object-Oriented DB System Manifesto (OO-ness) Third-Generation DB System Manifesto (Extensibility) Many query languages proposed

Systems were built, companies started, etc. Today, the field has settled down into two arenas:
o o o

Persistent OO PL systems (e.g. EXODUS, ObjectStore, Objectivity, Versant, etc.) Query-based systems with OO features (e.g. Starburst, Postgres, Illustra, Informix & ORacle "Universal Servers", DB/2 UDB) Almost nobody does both well

Few people continue to argue in terms of the paradigms. (Exhaustion?)

Systems History
Three influential research systems:
o o o o

Starburst (IBM Almaden) POSTGRES (Berkeley) EXODUS (Wisconsin) Others include O2 (Altair), ORION (MCC), Iris (HP), Genesis (Texas)

We will focus on POSTGRES & EXODUS, discuss Starburst briefly.

EXODUS
o

o o o

EXODUS was intended to be both a persistent PL system and a query system "Toolkit" extensibility query processing engine never got built, though the EXODUS optimizer architecture was influential (Graefe & DeWitt) Ended up focusing on OODB stuff, so we will discuss it more later. SHORE (follow-on to EXODUS) is delivering on EXODUS promises, but rather late. Persistent C++ Query Processing (w/ GIS features): Paradise

Extensible Optimizer: Opt++ Parallelism We'll see Exodus/SHORE work on pointer swizzling, client-server caching

Object-Relational Systems
Informix's buzzphrase for Illustra and look-alikes (courtesy Roger Sippl). Didn't patent the name! Query-based, extensible systems with some OO features like inheritance and OIDs. SQL3 draft is very much like this. Stonebrakers application matrix:

Query No Query

RDBMS File Sys. Simple Data

ORDBMS OODBMS Complex Data

Argues that the upper right is growing, and will engulf upper left and lower right.

Hot Applications:
o o o o o

Web servers, full-text collections Time-series data "Asset Management" GIS image DB

Players:
o o o o o o

Informix Universal Server (head of illustra, body of informix). Shipping now. IBM DB2 UDB (head of Starburst, body of DB2). Extensibility features coming along. UniSQL (Won Kim of ORION fame). Went out of business recently. Oracle Universal Server (marketing-ware). Shipping now. NCR (Teradata) just bought Wisconsin's Paradise ORDBMS (and DeWitt/Naughton/students) Other big R vendors are late (Sybase, Tandem, etc.)

O vendors still not running queries

Overview: Things needed in an Object-Relational DBMS

(From the "Third-Generation Database System Manifesto") The relational data model (as implemented!) is "semantically impoverished":
o o o

fixed set of base types (integer, float, etc) only structuring allowed is normal-form relations only operations are relational algebra, using comparators from base types

Instead, people want a "semantically rich" data model

o o o o

extensible ADTs complex types based on type constructors, and methods for those types inheritance "Object identity"

Zaniolo: GEM
One of the nicest, cleanest language papers, influential in the design of objectrelational data models and query languages (starting with Postquel)

ref types, nested dots, implicit joins (implication about OIDs) generalization hierarchies (converted to "collection inheritence") set-valued attrs (non-1st-normal-form!)

Stonebraker: Extensible ADTs

Seminal paper. Idea: you should be able to add new atomic types to the system, along with methods for the types, and new access methods. Type is defined by:
o o o o

storage size (can be variable) input method output method any other methods a user wishes to provide

Then you could do standard relational processing over those types. Example: add 2d spatial operators to RDBMS Engineering issues (some from the paper, some not):
o

Parsing

o o

must know about user-defined types and methods (table-driven) Optimization must be able to compute selectivity for user-defined predicates must know about cost of user-defined methods, and consider predicate pullup must know how to match user-defined predicates to fancy new access methods must know whether user-defined join predicates can be evaluated by hash or merge Execution: must have dynamic linking support (24x7 operation) methods called via "function pointers", "functors" or some similar construct can slow down standard processing support for "untrusted" functions (hot topic today) support for "large objects" caching for expensive methods (a la subquery caching) extensible aggregations 3 functions: init, iter, end Access Methods Stonebraker: An access method is a generic object that provides open, get-first, get-next, close, insert, delete if its fancy, it takes SARGs and evaluates them quickly needs to provide cost estimates to optimizer Problems integration with CC can be solved by physical logging can open up logging interface for AM-specific logging integration with Buffer Manager In practice, almost nobody used the Postgres Access method extensibility GiST! Buffer Management Large objects require new schemes Transactions integration with access methods

Note: Extensible ADTs do not fundamentally change a relational system

o o o

they fit naturally with the relational model as Stonebraker shows, they fit naturally into RDBMS query processing for relational "believers" this is all you should need to solve all problems

POSTGRES
Stonebraker, Rowe, a few staff and many students, 1986-1994. Post-INGRES. The Postgres Data Model 1. Co-opt the OO terminology

class = relation instance = tuple object-id = tuple-id method = attribute or function of attributes 2. Support extensible ADTs o extensible procedures using C functions o binary operators, which interface to extensible AM 3. Support type constructors o trick: use queries columns can be parameterized Postquel functions (returns setof, or tuple) queries can live in fields of a tuple (returns setof or tuple) another exploitation of the view paradigm! these derived objects can optionally be cached (never implemented) nested-dots used to traverse complex object structures leverages EXISTING techniques for relational processing. Cute! o added array support directly 4. added class inheritance (gives method inheritance and collection hierarchies)
o o o o

Implementation Details

originally written in LISP, then ran Lisp2C, resulting in a horrible built-in inheritance mechanism over C only access methods added to Postgres were done "in house" o B-tree and R-tree early o linear hashing late o GiST added in the last couple years "Fast Path" to AMs, as an alternative to "Persistent X" o never well-documented or used outside Berkeley ADTs as described above No Overwrite Storage, time travel, etc Research project was "shut down" in 94. 2 Berkeley students did a major cleanup (remove lispisms, remove a number of theses), migrated to SQL, and released Postgres95. This was picked up by freeware hackers on the net, and now PostgreSQL seems to be the serious freeware db of choice (www.postgresql.org).

More Stuff

Postgres Rule System: Active Database support (we will skip this topic this year) Shared-mem parallel version, with new optimization techniques "Inversion" file system PICASSO UI (Rowe & students) Support for tertiary memory Method Indexing Partial Indexes Expensive predicate optimization Commercialized as Illustra Concepts ported into Informix

Editorial Comments

Postgres code was a mess. Miraculous that it worked. More miraculous that Illustra salvaged it! Data model was sloppy but clever. Burst some OO bubbles. o Roger King: "My cat is object-oriented" o Missed an important distinction between Class and Collection. No-overwrite storage was not as bad as you think. Expect the concept to resurface. In many ways, the most successful of the 3 projects o used to be written off as goofy research, but Illustra is now ahead of the game o ADT extensibility & dynamic linking are very useful Illustras "DataBlades", Oracle's "Data Cartridges" o though not all aspects worked well, almost all were novel and influential o Many radical ideas in one system!

Starburst
Original goal: build a nice playpen for whatever comes next. Extensible "in-house". Not by users! No one survey paper seems to capture the work they did. Best bet: "Starburst MidFlight: As The Dust Clears", Haas, et al., TKDE 1990 Plumbing:
o o o o o o o

clean internal query representation (QGM). Key to Query Rewrite! non-normalized catalogs for efficiency normalized view for users uniform record structure across RSS & RDS a single expression evaluator for RSS & RDS WAL instead of shadow pages B+-tree compression Buffer Pool Manager accepts hints from optimizer (a la DBMIN)

Extensibility features:
o

User-defined functions: table expressions: queries or C functions scalar functions no dynamic linking Rule-based query rewrite engine a little rule system with QGM as "working memory" conditions and actions are C functions that check and change QGM some nifty rule control mechanisms (rule classes, rule budgets, multiple conflict res.) Extensible access methods (as in POSTGRES)

"Attachments": routines to be automatically called before/after dealing with an access method used by Starburst Rule System to generate transition logs used to implement pre-computed joins (see below) Complex objects implemented in a "wrapper" (SQL-XNF), translated down to Starburst SQL

"Proofs" of Extensibility:
o

o o o o o

added "signature" attachment to automatically tag tuples with some derived values though writing the attachment was hard, hooking it in took only 1 day added Outer Join not so simple required adding things in QGM, optimizer, and executor IMS attachment pre-computed joins using pointers, which are maintained written by "outsiders" Mike Carey & Beau Shekita from Wisconsin in a summer visit to Almaden Grammar-like rule-based query optimizer combined with System R bottom-up construction 2nd Rule System called "Alert", based on infinitely-running queries. Lightweight and pretty effective. Persistent C++ support (SMRC) Recursive query processing! Lots of technology for query rewriting

Many of the Starburst folks took a "vacation" from research the last few years, and merged Starburst technology into DB2 UDB. Editorial Comments
o o o o o

Probably the best code of the bunch The worlds most effective query optimizer? The DB2 experience good for all concerned Attention paid to details, shows up in complex parts of system A long-term advantage? Depends on market, IBMs development culture...

Data Warehousing, Decision Support & OLAP

Overview & Buzz Words

This area evolved via consultants, RDBMS vendors, and startup companies. All had something to prove, had to "differentiate their product". As a result, the area is a mess. Researchers making a little (but just a little) headway cleaning up the mess.

A "data warehouse" is an organization-wide snapshot of data, typically used for decision-making. A DBMS that runs these decision-making queries efficiently is sometimes called a "Decision Support System" DSS DSS systems and warehouses are typically separate from the on-line transaction processing (OLTP) system. By contrast, one class of DSS queries is sometimes called on-line analytic processing (OLAP) o the Return of Codd (as Essbase shill?) A "data mart" is a mini-warehouse -- typically a DSS for one aspect or branch of a company, with lots of relatively homogeneous data (i.e. a straight DSS)

Warehouse/DSS properties

Very large: 100gigabytes to many terabytes (or as big as you can go) Tends to include historical data Workload: mostly complex queries that access lots of data, and do many scans, joins, aggregations. Tend to look for "the big picture". Some workloads are canned queries (OLAP), some are ad-hoc (general DSS). Parallelism a must. Updates pumped to warehouse in batches (overnight) Data may be heavily summarized and/or consolidated in advance (must be done in batches too, must finish overnight). This is where the lion's share of the research work has been (e.g. "materialized views") -- a small piece of the problem.

A typical data warehousing architecture (c. 1996):

Data Cleaning

o o o

Data Migration: simple transformation rules (replace "gender" with "sex") Data Scrubbing: use domain-specific knowledge (e.g. zip codes) to modify data. Try parsing and fuzzy matching from multiple sources. Data Auditing: discover rules and relationships (or signal violations thereof). Not unlike data "mining".

Data Load: can take a very long time! (Sorting, indexing, summarization, integrity constraint checking, etc.) Parallelism a must. o Full load: like one big xact change from old data to new is atomic. o Incremental loading ("refresh") makes sense for big warehouses, but transaction model is more complex have to break the load into lots of transactions, and commit them periodically to avoid locking everything. Need to be careful to keep metadata & indices consistent along the way.

OLAP Overview To facilitate analysis and visualization, data is often modeled multidimensionally

Think n-dimensional spreadsheet rather than relational table

E.g. for a sales warehouse we have dimensions time_of_sale, sales_district, salesperson, product Dimensions can be organized hierarchically into more detail

e.g. time_of_sale may be "rolled up" into day-month-quarter-year product "rolled up" into product-category-industry opposite of "rollup": "drill-down" Other fun ops: o Slice_and_dice (i.e. selection & projection in the dimensions) o Pivot (re-orient the multidimensional view)

The values stored in the multidimensional cells are called numeric measures

E.g. sales, budget, revenue, inventory, ROI (return on investment), etc. These are things over which you might aggregate

ROLAP vs. MOLAP

ROLAP (Relational OLAP) uses standard relational tables & engine to do OLAP
o o o o o

Requires denormalized schema Star Schema: Fact table + table per dimension Snowflake Schema: off of the dimensions, have rolled-up versions Products: MicroStrategy, Metacube (Informix), Information Advantage. Uses standard relational query processing, with lots of indexes and precomputation

MOLAP (Multidimensional OLAP) actually stores things in multi-d format

o

Special index structures are used to support this

o o

o o

Note that much of the cube is empty! (no sales of Purple Chevy Trucks in June in Reno) Identify the "dense" and "sparse" dimensions. Build an index over combos of sparse dimensions. Within a combo, find a dense subcube of the dense dimensions. Zhao, et al. propose a different model. Products: Essbase (Arbor), Express (Oracle), Lightship (Pilot) Essentially everything is precomputed

More recently, HOLAP (Hybrid OLAP) to combine the best of both

o o o

Microsoft Plato due out soon, will make OLAP commonplace Some vendors (e.g. Informix/Essbase) talking about MOLAP ADTs inside an ORDBMS Keep this in mind as we read Zhao/Deshpande/Naughton

The Data Cube

Gray, et al. present a ROLAP language approach to cubing. The full data cube is all the base data in the cube, plus all the subaggregates obtained by projection. CUBE as relational operator implies need to model projection with a magic ALL value (ick!). Recall the mess with NULLs -- this is worse. We will come back to computing the cube later (Zhao/Deshpande/Naughton.)

Variant Indexes

Index support for DSS & ROLAP. General question: what indexing tricks can you play when you know there will be no updates? Big picture: broadly view an index as a) a form of redundant storage, b) a form of (partial) precomputation. Given there are no updates, you can precompute some pretty tricky stuff and store it on the side for when you need it! Background: Model 204, PC databases ("patented Rushmore technology"). Now in data warehousing and ROLAP products. Traditional ("Value-List") Indexes B+-trees. If you use them for small domains, at the leaves you get lots of records per distinct key. Bitmap-based B+-trees ("Bitmap Indexes") Idea: instead of storing a list of all RIDS with key=x at the leaves, store a bitmaps of the entire relation, where bit i is on iff the i'th tuple in the relation has key=x. Quibble: isnt this awfully wasteful in space? Not necessarily. Consider a column whose domain has 32 values. If there are n tuples, then on average each RID-list at the leaves is n/32*sizeof(RID) bits long. If a RID is 4 bytes (32 bits), thats n bits long, i.e. the same size as the bitmap. Refined Quibble: well, even with that few values there wont be a uniform distribution some RID-lists will be long, but some will still be short! This is true. In practice you use a hybrid solution RID-lists for rare values, bitmaps for popular ones. Remember, low storage means less to read at query time. There are other benefits to bitmaps besides storage! In particular, you can take multiple predicates and do ANDing and ORing across predicates quickly this in contrast to (a) using an index for only 1 predicate, and (b) using multiple indexes and doing RID-list union/intersection. NOTing works too, with a bit of extra data (existence bitmap). Refinement: Segmentation. Split the relation into segments, and the bitmaps into fragments. If a fragment holds x bits, a segment will be x rows long. A good value for x here might be the number of bits you can pack on a page (model 204 got about 48K bits). Two benefits: 1. Storage: For rare values (stored as RID-lists), can guarantee the RID-list fits on a page (1/32 of 48K is about 1.5K). Also, can store compressed RIDS since we know what segment were dealing with at any time, we only need the least significant 2 bytes of RID to distinguish the tuples in that segment. 2. Combining predicates: this is done segment-by-segment. Some indexes may have no entries on some segments (e.g. for retail customer data if a table is

ordered by customer age, and the index is on department, the bitmap of customers who bought liquor should contain no index entries for people under 21 i.e. the first many segments). In the case of an AND on liquor purchases and something else, one neednt fetch bitmaps from any index on those segments. Projection Indexes Idea: store a copy of a column of a table, in the same order as the table. Benefit: dont have to read other columns if you dont need em. Bit-Sliced Indexes Idea: take a projection index, and slice it vertically into columns of bits. Each column is a bit-sliced index. Example: the SALES column contains all sales made in the last month, in pennies. Assuming no item cost more than 2^20 1 pennies (= $10485.75), this is 20 bit-slice indexes. Benefits: 1. Storage: can use a number of bits per value which is not a power of 2 (e.g. 20 bits per sale). 2. Computing Aggregates, checking predicates: you can play some base-2 arithmetic games that well see soon. Applications of Variant Indexes Aggregate computation: We assume a bitmap called the foundset from the predicate evaluation. 1. SUM with each index? o Traditional: for each value in index, determine number of entries in foundset with that value, multiple value*count. Sum up all those numbers. o Projection Indexes: add up all the numbers corresponding to bits turned on in foundset o Bit-Sliced Index: AND slice with foundset, count bits in result, multiple by appropriate power of 2. Sum these. o Bottom Line: Bit-Slice beats Projection in I/Os due to storage savings, both beat Bitmap (though Projecion doesnt beat it by much!) 2. COUNT? 3. AVG? 4. MAX/MIN? 5. MEDIAN/N-TILE? o Traditional usually best, a tricky solution for Bit-Sliced sometimes pays off (not in paper)

6. SUM(COLUMN-PRODUCT): projection indexes are big winners (column values from a single tuple brought together easily) Range Predicate Evaluation WHERE rangep AND q; 1. Traditional: loop through all values in rangep and test for q 2. Projection Index: for each row in foundset of q, lookup in proj. index and check rangep 3. Bit-Sliced Index: tricky. From most to least significant bit, you let foundset AND/OR/NOT= bitslice. 4. Bottom Line: for small ranges, traditional is best. For large ranges, bit-sliced is best. ROLAP Star schema, join it up and group it. Summary tables can help (e.g. store the sums, counts, etc. per group at various levels of detail for various groupings.) But this can get out of hand and doesnt always help if you have "non-dimensional" selection. Join index is an index on a table T based on a quantity that involves a column value from another table S via some standard join. Example: Star Join Index indexes the fact table on combinations of values in detail tables. Again, this can get out of hand. Bitmap join index solves the problem. Index on T based a single column of S. Multiple columns can be combined by bitmap techniques to give the behavior of multi-column join indexes. Note: join indexes can be of any of the 3 kinds listed at the top! With the right kinds of join indexes, a ROLAP query can be done "without any joins": selection conditions handled via traditional or bit-sliced join indexes, projection join indexes to retrieve dimensional values in SELECT list. Groupsets: partition bits in foundset by group.

Traditional (2-column GROUP BY):

for each entry in 1st grouping column for each entry in 2nd grouping column AND the bitmap from 1st column with that from 2nd compute aggregate on the resulting bitmap

Can be very inefficient when there are lots of groupsets and rows per groupset are randomly distributed due to computation of aggregate on sparse bitmaps.

Projection (join) indexes on fact table: for each element of foundset, lookup value in projection index and find group. Then use other projection indexes to get aggregation columns of that element, and update the running aggregate for that group. If the groups dont all fit in memory, can do this with some staging to disk (sort or hash).

Additional benefit for grouping via segmentation & clustering (well skip it.) Also groupset indexes.

Computing the Cube Efficiently

ROLAP: Use hashing or sorting (Agrawal, et al., VLDB 96). Be sure to reuse aggs computed for small groups to compute aggs over big groups. MOLAP: Carefully walk the basic cube to compute projected aggs with minimal memory requirements (Zhao, et al. SIGMOD 97) Zhao, et al. point out interesting interplay (coming up at the end.) Starting point: storing cubes

Chunk the cube into n-dimensional subcubes For dense cubes (> 40% full), store them as is (fixed size.) For sparse cubes (< 40% full), compress them with "chunk-offset compression" (<offset-in-Chunk, value> pairs). An algorithm is given for converting a relation to a chunked array representation.

First, a simple array cubing alg

Given: cube of dimensions A,B,C. Compute: aggregates for BC. This is like "sweeping a plane" through the A dimension, and bringing the values into the BC plane. Do this in a chunkwise fashion compute the agg on a chunk, then on the chunk in front of it, and so on. When youre done with a "front cell", write it to disk. This uses only 1 chunk-size piece of memory at a time. Then compute individual aggs (B and C) from the agg for BC.

Question: what do we use to compute subaggs? Think of the following

lattice: The decision corresponds to a choice of a spanning tree in the lattice. We are given

dimension sizes and chunk sizes, so we can compute the size to store each node in lattice, as well as how much memory needed to use a node to compute its child. Minimum Spanning Tree (MST) of Lattice is defined as follows: For each node n, its parent in the MST is the parent node n with minimum size. The full nave algorithm: 1. Construct the MST 2. Compute each node from its parent in the MST as illustrated above. Problem: we rescan things too often (to compute AB, AC, and BC we scan ABC 3 times) The Multi-Way Array Algorithm Goal: compute all the subaggs in a single pass of the underlying cube, using minimal memory. Dimension Order: an ordering on the dimensions corresponding to a "row major" traversal of chunks (ABC in the following picture):

This determines memory requirements:

The BC values require 1 chunk BCs each to aggregate away the A's The AC values require 4 chunks ACs each to aggregate away the B's The AB values require 16 chunks ABs each to aggregate away the C's More generally, if array element size is u, dimension X chunk size is Xc, and dimension Y size is Yd, we need |Bc||Cc|u + |Ad||Cc|u + |Ad||Bd|u memory to do this simultaneously. This generalizes naturally to more dimensions: to compute a projection on one dimension, you need to look at product of sizes of dimensions in the ordering prefix before the projected dimension, times the product of the chunk sizes of the remaining dimensions. I.e. it's harder to aggregate away dimensions that are later in the dimension order

This leads to the Minimum Memory Spanning Tree (MMST):

For each node below the root of the lattice, choose the parent who requires the least memory during traversal (Note: this depends on the chosen dimension ordering.)

To project out more than 1 dimension, you use the same logic to go from k dimensions to k-1 as you did from all dimensions to all-1. E.g. to compute C from AC, you need to look at |Ad| values simultaneously. This happens as you write the k-d aggs to disk. Theres a simple formula for memory requirements for the MMST.

What is the optimal dimension order?

Can be computed by optimizing that simple formula. Turns out to be simple: order by increasing dimension size! Memory requirement is independent of the size of the largest dimension (which is aggregated away 1st).

And if you dont have enough memory for that?

Theres a multi-pass version, which heuristically chooses all but some subtrees of the MMST to compute at a time.

Upshot: Not only does their algorithm work well and beat the best known ROLAP technique, it suggests a better ROLAP technique: 1. Scan table and convert to array 2. Do MOLAP processing 3. Dump resulting cube to a table. Wild! A wacky new "non-relational" query processing operation to be used in relational DBMSs! A benefit of thinking in a new data model??

Data Mining
Emerging hot area (see this!), linking DBMS with AI and stats. Vision: O Great Database! Tell me what I want to know. Practice (c. 1998): a small collection of statistical "queries" (with wide open parameters) 1. Clustering/Segmentation: group together similar items (separating dissimilar items) 2. Predictive Modeling: predict value of some attributes based on values of others (and previous "training") o e.g. classification: assign items to pre-specified categories 3. Dependency Modeling: model joint probability densities (relationships among columns) o another example: Association Rules (e.g. find items that get purchased together)

4. Data Summarization: find compact descriptions of subsets of data 5. Change and Deviation Analysis: find significant changes in sequences of data (e.g. stock data). Sequence-oriented. Applications of Mining:

Marketing Fraud detection Telecomm Science (astronomy, remote sensing, protein sequencing) NBA stats (IBM Advanced Scout)

DBMS spin: a taxonomy of queries:

o o o

find anomalies/points (traditional queries?) find common stuff (trend analysis/aggregation/OLAP) find pretty common stuff (mining)

Is there some insight here?

Association Rules
Why look at this example of data mining?

It's statistically "lite" (no background required) It has a relational query processing flavor Probably for both of these reasons, there have been a ton of follow-on papers (arguably too many...)

"Basket data" Association Rule: X ==> Y (X and Y are disjoint sets of items)
o o

confidence c: c% of transactions that contain X also contain Y (rulespecific) support s: s% of all transactions contain both X and Y (relative to all data)

Problem: Efficiently find all rules with support > minsup, confidence > minconf Anecdotes:
o o

diapers & beer Wendys & McDonalds (pb & j)

One algorithm from the paper: Apriori L1 = {large 1-itemsets}

for (k = 2; Lk -1 != ; k++) { Ck = apriori-gen(Lk-1); // candidate k-itemsets forall transactions t in the database { Ct = subset(Ck, t); // Candidates from Ck contained in t forall candidates c in Ct c.count++; } Lk = {c in Ck | c.count >= minsup} } Answer = kLk apriori-gen(Lk-1) { // Intuition: every subset of a large itemset must be large. // So combine almost-matching pairs of large (k-1)-itemsets, // and prune out those with non-large (k-1)-subsets. join: insert into Ck select p.item1, , p.itemk-1, q.itemk-1 from Lk-1 p, Lk-1 q where p.item1 = q.item1 and and p.itemk-2 = q.itemk-2 and p.itemk-1 < q.itemk-1; prune: // delete itemsets such that some (k-1)-subset is not in Lk-1 forall itemsets c in Ck forall (k-1)-subsets s of c if (s not in Lk-1) { delete c from Ck; break; } } Example from paper (minsup = 2)

TID 100 200 300 400

Items 134 235 Itemset Support 2 3 3 3 Itemset {1 2} {1 3} {1 5} {2 3} {2 5} {3 5} Itemset {1 3} {2 3} {2 5} {3 5} Support 2 2 3 2

1 2 3 5 {1} 25 {2} {3} {5}

Itemset {2 3 5}

Support 2 Itemset {2 3 5}

Efficiently Implementing subset: The hash tree.

Leaves contain a list of itemsets Internal nodes contain a hash table, with each bucket pointing to a child. Root is at depth 1. When inserting into interior node at depth d, we choose child by applying hash function to dth item in itemset. To check all candidate subsets of transaction t: If at leaf, find which itemsets there are in t (i.e. nave check) If weve reached an internal node by hashing on item i, hash on each item after i in turn, and continue recursively for each.

Variation: AprioriTid generates a new "database" at each step k, with items being <TID, all subsets of size k in that xact>, and scans that rather than the original database

next time around. Benefit: database gets fewer rows at each stage. Cost: databases generated may be big.

Hybrid: AprioriHybrid: run Apriori for a while, then switch to AprioriTid when the generated DB would fit in mem.

Analysis shows that AprioriHybrid is fastest, beating old algorithms and matching Apriori and AprioriTid when either one wins.

Later Work:

If you have an IsA hierarchy, do association rules on it (you may not be able to say that pampers -> beer, but if you know pampers IsA diaper, and luvs IsA diaper, maybe youll find that diapers->beer.) Parallel versions of this stuff. Quantitative association rules: "10% of married people between age 50 and 60 have at least 2 cars." Online association rules: you should be able to change support and confidence threshholds on the fly, and be able to incrementally change answers as new transactions stream in (Christian Hidber, ICSI/UCB).

Benchmarking
Goals

What's a benchmark for? 1. What do you want to learn? 2. How much are you prepared to invest? 1. programmer time 2. machine resources 3. management time 4. lost opportunity 3. What are you prepared to give up?

Side-benefits to the community...

Types of Benchmarks

Generic benchmarks try to give a broad idea of price/performance across many applications. Application-Specific benchmarks try to focus in on a restricted class of workloads. Some of the main consortia defining these: o SPEC (System Performance Evaluation Cooperative): scientific workloads, designed to measure workstation performance o The Perfect Club: scientific workloads on exotic parallel architectures o TPC (Transaction Processing Council): transaction processing and decision support workloads, architecture-independent

Gray's criteria for a good app-specific benchmark:

Relevant: It must measure the peak performance and price/performance of systems when performing typical operations within that problem domain. Portable: It should be easy to implement the benchmark on many different systems and architectures. Scaleable: The benchmark should apply to small and large computer systems. It should be possible to scale the benchmark up to larger systems, and to parallel computer systems as computer performance and architecture evolve. Simple: The benchmark must be understandable, otherwise it will lack credibility.

A counter-example: MIPS (Millions of Instructions Per Second)

irrelevant: doesn't translate directly to useful work not portable: IntelMIPS != SunMIPS not scalable: how does it apply to multiprocessors?

The importance of benchmark acceptance

Benchmarketing Wisconsin benchmark history: DeWitt vs. Ellison. The DeWitt clause. Anon, et al. Benchmark wars: escalating numbers lead to escalating tricks, wasted time ("just let us tune it as much as Vendor X did") o if you're doing your own, avoid this with "no reruns, no excuses"

Transaction Processing Council

TPC arose in 1988 (headed by consultant Omri Serlin), consortium of 35 hardware/software companies

define benchmarks for TP and DSS, define cost/performance metrics, provide official audits Today: TPC-C is the TP benchmark (succeeding A and B), TPC-D is the decision-support benchmark

Focus on as real-world a scenario as possible: o performance from terminals to server and back, as opposed (say) to SPEC. all aspects of the system. o note the overhead of doing this! hard to set up requires auditor to report results functionality as well as performance

Writing a benchmark

try to measure core speed and potential bottlenecks o "micro-benchmarks", e.g. Sort try to expose functionalities or lack thereof o e.g. Wisconsin, TPC-D o can backfire for adaptive parts of a system. Optimizers are notoroiusly hard to "benchmark" try to measure end-to-end performance on a realish workload o e.g. TPC-C, TPC-D. a statistical note: don't use average performance! consider variance. if you must give one number, give 90th percentile performance.

DB Benchmarks to be aware of:

Wisconsin (mostly for history) TPC-C TPC-D Set Query: Pat O'Neil's complex query benchmark (shows off Model 204) 007: OODBMS benchmark from Wisconsin and some of the O-vendors Sequoia: app-specific benchmark for Earth Science from Sequoia project at Berkeley (Stonebraker) and UCSB (earth scientists). Used as an early ObjectRelational and GIS benchmark (mostly R-trees and user-defined functions, but also one transitive closure query (!)) Bucky: an Object-Relational benchmark from Wisconsin focusing on structured types (refs and nested sets) OR-1: a yet-to-be-completed Object-Relational benchmark from Wisconsin, Informix (Stonebraker) and IBM (Carey). Was supposed to fix Bucky with more focus on "what matters to customers". Gray's Benchmark Handbook now lives on the web

Benchmarking
Goals

What's a benchmark for? 1. What do you want to learn? 2. How much are you prepared to invest? 1. programmer time 2. machine resources 3. management time 4. lost opportunity 3. What are you prepared to give up? o Side-benefits to the community...

Types of Benchmarks

Generic benchmarks try to give a broad idea of price/performance across many applications.

Application-Specific benchmarks try to focus in on a restricted class of workloads. Some of the main consortia defining these: o SPEC (System Performance Evaluation Cooperative): scientific workloads, designed to measure workstation performance o The Perfect Club: scientific workloads on exotic parallel architectures o TPC (Transaction Processing Council): transaction processing and decision support workloads, architecture-independent

Gray's criteria for a good app-specific benchmark:

Relevant: It must measure the peak performance and price/performance of systems when performing typical operations within that problem domain. Portable: It should be easy to implement the benchmark on many different systems and architectures. Scaleable: The benchmark should apply to small and large computer systems. It should be possible to scale the benchmark up to larger systems, and to parallel computer systems as computer performance and architecture evolve. Simple: The benchmark must be understandable, otherwise it will lack credibility.

A counter-example: MIPS (Millions of Instructions Per Second)

irrelevant: doesn't translate directly to useful work not portable: IntelMIPS != SunMIPS not scalable: how does it apply to multiprocessors?

The importance of benchmark acceptance

Benchmarketing Wisconsin benchmark history: DeWitt vs. Ellison. The DeWitt clause. Anon, et al. Benchmark wars: escalating numbers lead to escalating tricks, wasted time ("just let us tune it as much as Vendor X did") o if you're doing your own, avoid this with "no reruns, no excuses"

Transaction Processing Council

TPC arose in 1988 (headed by consultant Omri Serlin), consortium of 35 hardware/software companies

define benchmarks for TP and DSS, define cost/performance metrics, provide official audits Today: TPC-C is the TP benchmark (succeeding A and B), TPC-D is the decision-support benchmark Focus on as real-world a scenario as possible: o performance from terminals to server and back, as opposed (say) to SPEC. all aspects of the system. o note the overhead of doing this! hard to set up requires auditor to report results

functionality as well as performance

Writing a benchmark

try to measure core speed and potential bottlenecks o "micro-benchmarks", e.g. Sort try to expose functionalities or lack thereof o e.g. Wisconsin, TPC-D o can backfire for adaptive parts of a system. Optimizers are notoroiusly hard to "benchmark" try to measure end-to-end performance on a realish workload o e.g. TPC-C, TPC-D. a statistical note: don't use average performance! consider variance. if you must give one number, give 90th percentile performance.

DB Benchmarks to be aware of:

Wisconsin (mostly for history) TPC-C TPC-D Set Query: Pat O'Neil's complex query benchmark (shows off Model 204) 007: OODBMS benchmark from Wisconsin and some of the O-vendors Sequoia: app-specific benchmark for Earth Science from Sequoia project at Berkeley (Stonebraker) and UCSB (earth scientists). Used as an early ObjectRelational and GIS benchmark (mostly R-trees and user-defined functions, but also one transitive closure query (!)) Bucky: an Object-Relational benchmark from Wisconsin focusing on structured types (refs and nested sets) OR-1: a yet-to-be-completed Object-Relational benchmark from Wisconsin, Informix (Stonebraker) and IBM (Carey). Was supposed to fix Bucky with more focus on "what matters to customers". Gray's Benchmark Handbook now lives on the web

Using Samba
Robert Eckstein, David Collier-Brown, Peter Kelly 1st Edition November 1999 1-56592-449-5, Order Number: 4495 416 pages, $34.95

1. Learning the Samba

Contents: What is Samba What Can Samba Do For Me? Getting Familiar with a SMB/CIFS Network Microsoft Implementations An Overview of the Samba Distribution How Can I Get Samba? What's New in Samba 2.0? And That's Not All... If you are a typical system administrator, then you know what it means to be swamped with work. Your daily routine is filled with endless hardware incompatibility issues, system outages, data backup problems, and a steady stream of angry users. So adding another program to the mix of tools that you have to maintain may sound a bit perplexing. However, if you're determined to reduce the complexity of your work environment, as well as the workload of keeping it running smoothly, Samba may be the tool you've been waiting for. A case in point: one of the authors of this book used to look after 70 Unix developers sharing 5 Unix servers. His neighbor administered 20 Windows 3.1 users and 5 OS/2 and Windows NT servers. To put it mildly, the Windows 3.1 administrator was swamped. When he finally left -- and the domain controller melted -- Samba was brought to the rescue. Our author quickly replaced the Windows NT and OS/2 servers with Samba running on a Unix server, and eventually bought PCs for most of the company developers. However, he did the latter without hiring a new PC administrator; the administrator now manages one centralized Unix application instead of fifty distributed PCs. If you know you're facing a problem with your network and you're sure there is a better way, we encourage you to start reading this book. Or, if you've heard about Samba and you want to see what it can do for you, this is also the place to start. We'll get you started on the path to understanding Samba and its potential. Before long, you can provide Unix services to all your Windows machines -- all without spending tons of extra time or money. Sound enticing? Great, then let's get started.

What is Samba?
Samba is a suite of Unix applications that speak the SMB (Server Message Block) protocol. Many operating systems, including Windows and OS/2, use SMB to perform client-server networking. By supporting this protocol, Samba allows Unix servers to get in on the action, communicating with the same networking protocol as Microsoft Windows products. Thus, a Samba-enabled Unix machine can masquerade as a server on your Microsoft network and offer the following services:

Share one or more filesystems

Share printers installed on both the server and its clients Assist clients with Network Neighborhood browsing Authenticate clients logging onto a Windows domain Provide or assist with WINS name server resolution

Samba is the brainchild of Andrew Tridgell, who currently heads the Samba development team from his home of Canberra, Australia. The project was born in 1991 when Andrew created a fileserver program for his local network that supported an odd DEC protocol from Digital Pathworks. Although he didn't know it at the time, that protocol later turned out to be SMB. A few years later, he expanded upon his custommade SMB server and began distributing it as a product on the Internet under the name SMB Server. However, Andrew couldn't keep that name -- it already belonged to another company's product -- so he tried the following Unix renaming approach:
grep -i 's.*m.*b' /usr/dict/words

And the response was:

salmonberry samba sawtimber scramble

Thus, the name "Samba" was born. Which is a good thing, because our marketing people highly doubt you would have picked up a book called "Using Salmonberry"! Today, the Samba suite revolves around a pair of Unix daemons that provide shared resources -- or shares -- to SMB clients on the network. (Shares are sometimes called services as well.) These daemons are: smbd A daemon that allows file and printer sharing on an SMB network and provides authentication and authorization for SMB clients. nmbd A daemon that looks after the Windows Internet Name Service (WINS), and assists with browsing. Samba is currently maintained and extended by a group of volunteers under the active supervision of Andrew Tridgell. Like the Linux operating system, Samba is considered Open Source software (OSS) by its authors, and is distributed under the GNU General Public License (GPL). Since its inception, development of Samba has been sponsored in part by the Australian National University, where Andrew Tridgell earned his Ph.D. [1] In addition, some development has been sponsored by independent vendors such as Whistle and SGI. It is a true testament to Samba that both commercial and noncommercial entities are prepared to spend money to support an Open Source effort. At the time of this printing, Andrew had completed his Ph.D. work and had joined San Francisco-based LinuxCare.

Microsoft has also contributed materially by putting forward its definition of SMB and the Internet-savvy Common Internet File System (CIFS), as a public Request for Comments (RFC), a standards document. The CIFS protocol is Microsoft's renaming of future versions of the SMB protocol that will be used in Windows products -- the two terms can be used interchangeably in this book. Hence, you will often see the protocol written as "SMB/CIFS." 1.1 Learning Samba Back to: Using Samba
oreilly.com Home | O'Reilly Bookstores | How to Order | O'Reilly Contacts International | About O'Reilly | Affiliated Companies | Privacy Policy 2001, O'Reilly & Associates, Inc.

An Oracle White Paper September 2009

A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Sun Oracle Exadata Storage Server and Database Machine.............2 Todays Limits On Database I/O.........................................................2 Exadata Product Family......................................................................4 Sun Oracle Exadata Storage Server...............................................4 Sun Oracle Database Machine.......................................................8 Exadata Architecture.........................................................................12 Database Server Software............................................................13 Enterprise Manager Plug-In For Exadata......................................14 Exadata Software..........................................................................14 Exadata Smart Scan Processing..................................................15 I/O Resource Management With Exadata.....................................19 Accelerated Performance With Exadata.......................................21 Exadata Storage Virtualization......................................................22 CONCLUSION..................................................................................27White Paper A Technical
Overview of the Sun Oracle Exadata Storage Server and Database Machine

Sun Oracle Exadata Storage Server and Database Machine

The Sun Oracle Exadata Storage Server (Exadata) is a storage product optimized for use with Oracle Database applications and is the storage building block of the Sun Oracle Database Machine. Exadata delivers outstanding I/O and SQL processing performance for online transaction processing (OLTP), data warehousing (DW) and consolidation of mixed workloads. Extreme performance is delivered for all types of database applications by leveraging a massively parallel grid architecture and Exadata Smart Flash Cache to dramatically accelerate Oracle Database processing and speed I/O operations. The Exadata storage products are a combination of software and hardware used to store and access Oracle databases. Exadata provides database aware storage services, such as the ability to offload database processing from the database server to storage, and provides this while being transparent to SQL processing and database applications. The Exadata Storage Servers are also packaged in a complete end-to-end database solution the Sun Oracle Database Machine. The Sun Oracle Database Machine is an easy to deploy out of the box solution for hosting the Oracle Database for all applications and delivers the highest levels of performance available. Database Machine and Exadata storage delivers breakthrough performance with linear I/O scalability, is simple to use and manage, and delivers mission-critical availability and reliability to the enterprise. Exadata is a joint offering from Oracle and Sun Microsystems. Sun is providing the hardware technology used in the Database Machine and Exadata Storage Server. Oracle is providing the software to impart database intelligence to the storage and Database Machine and is tightly integrated with the Oracle Database and all its features. The Sun servers combine the power of the latest generation of Intel Xeon processors with Sun's system engineering expertise. These servers offer the needed density and expandability to satisfy the most demanding datacenter applications. The Oracle and Sun partnership makes possible the delivery of the Sun Oracle Database Machine and Exadata Storage Server and the revolutionary capabilities it provides.

Todays Limits On Database I/O

The foundation of the Database Machine and Exadata is smart database software to handle the complex applications deployed to drive the most demanding enterprise business needs. The

2 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Oracle Database provides an incredible amount of functionality to implement the most sophisticated OLTP and DW applications and to consolidate mixed workload environments. But to access terabytes databases with high performance, augmenting the smart database software with powerful hardware provides tremendous opportunities to deliver more database processing, faster, for the enterprise. Having powerful hardware to provide the required I/O rates and bandwidth for todays applications, in addition to smart software, is key to the extreme performance delivered by the Exadata family of products. Traditional storage devices offer high storage capacity but are relatively slow and can not sustain the I/O rates for the transaction load the enterprise requires for its applications. Instead of hundreds of IOPS (I/Os per second) per disk enterprise applications require their systems deliver at least an order of magnitude higher IOPS to deliver the service enterprise end-users expect. This problem gets magnified when hundreds of disks reside behind a single storage controller. The IOPS that can be executed are severely limited by both the speed of the mechanical disk drive and the number of drives per storage controller. Traditional storage products provide the Oracle Database a narrow and limited interface to database storage. Many bottlenecks exist in the database I/O path restricting data bandwidth, and hence limiting overall database performance when using traditional storage products. Database servers need many Storage Area Network (SAN) Host Bus Adapters (HBA) to provide the bandwidth necessary to deliver data, from storage to the database, at an adequate rate. Very often the number of HBAs required to deliver adequate performance cannot be supported in the server, or are too costly to provide. And SAN switch cost and complexity increases dramatically to provide the required bandwidth and redundancy. In addition large storage arrays cannot deliver adequate bandwidth to the hundreds of disks they house. This results in the potential performance of disks being artificially capped well below what they are capable of providing. Disk performance is bottlenecked on the Fibre Channel Loops (FCL) to disk and the processing capacity of the storage array. Traditional storage devices are also unaware that a database is residing on the storage and therefore cannot provide any database-aware I/O or SQL processing. When the database requests rows and columns what is returned from the storage are data blocks rather than the result set of a database query. Traditional storage has no database intelligence to discern the particular rows and columns actually requested. So, when processing I/O on behalf of the database, traditional storage consumes bandwidth returning data that is not relevant to the database query that was issued. Exadata products address the key dimensions of database I/O that can hamper database performance. Exadata is based on a massively parallel architecture which provides more pipes to deliver more data faster between the database servers and the storage servers.

3 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

 Exadata

is built using wider pipes that provide extremely high bandwidth between the database servers and the storage servers. Exadata is database aware and can ship just the data required to satisfy SQL requests resulting in less data being sent between the database servers and the storage servers. Exadata overcomes the mechanical limits of disk drive technology by automatically caching frequently accessed data delivering unprecedented levels of bandwidth and IOPS.

Exadata Product Family

There are two members of the Exadata product family. The foundation of the Exadata family of products is the Sun Oracle Exadata Storage Server. It is used as the storage for the Oracle Database when building custom database systems. The second member of the Exadata product family is the Sun Oracle Database Machine (Database Machine). The Database Machine is a complete and fully integrated database system that includes all the components to quickly and easily deploy any enterprise database application requiring the best performance, and includes Exadata storage. Sun Oracle Exadata Storage Server The Sun Oracle Exadata Storage Server is a database storage device running the Exadata Storage Server Software provided by Oracle. The hardware components of the Exadata Storage Server (also referred to as an Exadata cell) were carefully chosen to match the needs of high performance database processing. The Exadata software is optimized to take the best possible advantage of the hardware components and Oracle Database. Each Exadata cell delivers outstanding I/O performance and bandwidth to the database. The Sun Oracle Exadata Storage Server is a fast, reliable, high capacity, industry- standard storage server. Each Exadata cell comes preconfigured with: two Intel Xeon E5540 quad-core processors, 384 GB of Exadata Smart Flash Cache, twelve disks connected to a storage controller with 512MB battery-backed cache, 24 GB memory, dual port InfiniBand connectivity, management interface for remote access, dual-redundant hot-swappable power supplies, all the software preinstalled, and takes up 2U in a typical 19inch rack.

4 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Figure 1: Exadata Storage Cell

Exadata Smart Flash Cache

Each Exadata cell comes with 384 GB of Exadata Smart Flash Cache. This solid state storage delivers dramatic performance advantages with Exadata storage. It provides a ten-fold improvement in response time for reads over regular disk; a hundred-fold improvement in IOPS for reads over regular disk; and is a less expensive higher capacity alternative to memory. Overall it delivers a tenfold increase performing a blended average of read and write operations. The Exadata Smart Flash Cache manages active data from regular disks in the Exadata cell but it is not managed in a simple Least Recently Used (LRU) fashion. The Exadata Storage Server Software in cooperation with the Oracle Database keeps track of data access patterns and knows what and how to cache data and avoid polluting the cache. This functionality is all managed automatically and does not require manual tuning. If there are specific tables or indexes that are known to be key to the performance of a database application they can optionally be identified and pinned in cache.
Exadata Storage Server Performance, Bandwidth and IOPS

The Sun Oracle Exadata Storage Server comes with either twelve 600 GB Serial Attached SCSI (SAS) disks or twelve 2 TB Serial Advanced Technology Attachment (SATA) disks. SAS based Exadata Storage Servers provide up to 2 TB of uncompressed user data capacity, and up to 1.5 GB/second of raw data bandwidth. SATA based Exadata Storage Servers provide up to 7 TB of uncompressed user data capacity, and up to 0.85 GB/second of raw data bandwidth. When stored in compressed format, the amount of user data and the amount of data bandwidth delivered by each cell increases up to 10 times. User data capacity is computed after mirroring all the disk space, and setting aside space for database structures like logs, undo, and temp space. Actual user data varies by application. The performance that each cell delivers is extremely high due to the Exadata Smart Flash Cache. The automated caching of the Flash cache enables each Exadata cell to deliver up to 3.6 GB/second bandwidth and 75,000 IOPS when accessing uncompressed data. When data is stored in compressed format, the amount of user data capacity, the amount of data bandwidth

5 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

and IOPS achievable, often increases up to ten times. This represents a significant improvement over traditional storage devices used with the Oracle Database. The performance specifications of the Exadata Storage Server are shown below. SAS Based User Data Up to 3.6 GB/sec Exadata Storage Capacity Effective Data Bandwidth with Server (without data Flash Cache and Data SATA Based compression) Compression Exadata Storage Up to 2 TB Up to 36 GB/sec Server Up to 7 TB Up to 36 GB/sec Exadata Smart Raw Disk Data Flash Cache IOPS Flash Cache Bandwidth Up to 75,000 384 GB Up to 1.5 GB/sec Up to 75,000 384 GB Up to 0.85 GB/sec Disk IOPS Raw Disk Effective Data Up to 3,600 Capacity Bandwidth with Up to 1,440 7.2 TB Flash 24 TB Up to 3.6 GB/sec
InfiniBand and the Exadata Storage Server

Oracle Exadata storage uses a state of the art InfiniBand interconnect between the servers and storage. An Exadata cell has dual port Quad Data Rate (QDR) InfiniBand connectivity for high availability. Each InfiniBand link provides 40 Gigabits of bandwidth - many times higher than traditional storage or server networks. Further, Oracle's interconnect protocol uses direct data placement (DMA - direct memory access) to ensure very low CPU overhead by directly moving data from the wire to database buffers with no extra data copies being made. The InfiniBand network has the flexibility of a LAN network, with the efficiency of a SAN. By using an InfiniBand network, Oracle ensures that the network will not bottleneck performance. The same InfiniBand network also provides a high performance cluster interconnect for the Oracle Database Real Application Cluster (RAC) nodes.
Exadata Storage Server Configuration

In figure 2 below, a small Exadata storage based database environment is shown. Two Oracle Databases, one RAC and one single instance, are sharing three Exadata cells. All the components for this configuration database servers, Exadata cells, InfiniBand switches, Ethernet switches, and other support hardware can be housed in, and take up less than half of, a typical 19-inch rack.

6 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Exadata CellInfiniBand Switch/NetworkSingle-InstanceDatabaseRAC DatabaseExadata CellExadata CellExadata CellExadata CellExadata Cell

Figure 2: Exadata Storage Cell Based Configuration

Oracle Exadata is architected to scale-out to any level of performance. To achieve higher performance and greater storage capacity, additional Exadata cells are added to the configuration. As more cells are added, capacity and performance increases linearly. No cell-to-cell communication is ever done or required in an Exadata configuration. Oracle Automatic Storage Management (ASM) is used as the file system and volume manager for Exadata. The disk mirroring provided by ASM, combined with hot swappable Exadata disks, ensure the database can tolerate the failure of individual disk drives. Data is mirrored across cells to ensure that the failure of a cell will not result in loss of data, or inhibit data accessibility. This massively parallel architecture delivers unbounded scalability and high availability. When using Exadata, SQL processing is offloaded from the database server to the Exadata server. Exadata enables function shipping from the database instance to the underlying storage in addition to providing traditional block serving services to the database. One of the unique things the Exadata storage does compared to traditional storage is return only the rows and columns that satisfy the database query rather than the entire table being queried. Exadata pushes SQL processing as close to the data (or disks) as possible and gets all the disks operating in parallel. This reduces CPU consumption on the database server, consumes much less bandwidth moving data between database servers and storage servers, and returns a query result set rather than entire tables. Eliminating data transfers and database server workload can greatly benefit data warehousing queries that traditionally become bandwidth and CPU constrained. Eliminating data transfers can also have a significant benefit on online transaction processing (OLTP) systems that often include large batch and report processing operations. Exadata storage is totally transparent to the application using the database. Existing SQL statements, whether ad hoc or in packaged or custom applications, are unaffected and do not require any modification when Exadata storage is used. The offload processing and bandwidth

7 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

advantages of the solution are delivered without any modification to your application. And all features of the Oracle Database are fully supported with Exadata. Exadata works equally well with single-instance or Real Application Cluster deployments of the Oracle Database. Functionality like Oracle Data Guard, Oracle Recovery Manager (RMAN), Oracle Streams, and other database tools are administered the same, with or without Exadata. Users and database administrators leverage the same tools and knowledge they are familiar with today because they work just as they do with traditional non-Exadata storage. Both Exadata and non-Exadata storage may be concurrently used for database storage to facilitate migration to, or from, Exadata storage. The nature of traditional storage products encourages inefficient deployments of storage for each database in the IT infrastructure. The Exadata architecture ensures all the bandwidth and I/O resources of the Exadata storage subsystem can be made available whenever, and to whichever, database or class of work needs it. I/O bandwidth is metered out to the various classes of work, or databases, sharing the Exadata server based on user defined policies and service level agreements (SLAs). The Oracle Database Resource Manager (DBRM) has been enhanced for use with Exadata storage to manage user-defined intra and inter-database I/O resource usage to ensure customer defined SLAs are met. The I/O resource management capabilities of Exadata storage enable tailoring the I/O resources to the business priorities of the organization, and to build a shared storage grid for the Oracle databases in the environment. Sun Oracle Database Machine Oracle is also offering a fully integrated platform for all your database applications. The Sun Oracle Database Machine is an easy to deploy out of the box solution for hosting the Oracle Database. A fully integrated solution ready to be turned on day one takes a lot of integration work, cost and time, out of the application deployment process. The benefit of a common infrastructure to deploy any application on, whether OLTP, DW of a mix of the two, creates tremendous efficiencies in the datacenter.
Figure 3: Database Machine Full Rack

8 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

There are four models of the Database Machine Database Machine Full Rack, Database Machine Half Rack, Database Machine Quarter Rack, and Database Machine Eighth Rack are offered. Depending on the size and purpose of the database to be deployed, and the processing and I/O bandwidth required there is a system available to meet any need. Each Database Machine runs the same software, is upgradeable and includes common hardware components. Common to all Database Machines are: Exadata Storage Servers, either SAS or SATA. Industry standard Oracle Database 11g database servers with: two Intel Xeon dual-socket quad-core E5540 processors running at 2.53 Ghz processors, 72 GB RAM, four 146 GB SAS drives, dual port InfiniBand Host Channel Adapter (HCA), four 1 Gb/second Ethernet ports, and dual-redundant, hot-swappable power supplies. Sun Quad Data Rate (QDR) InfiniBand switches and cables to form a 40 Gb/second InfiniBand fabric for database server to Exadata storage server communication and RAC internode communication. The ratio of components to each other has been chosen to maximize performance and ensure system resiliency. The hardware composition of each model of Database Machine is depicted in the following table. Sun Oracle Sun Machine Sun Oracle Database Machine Database Oracle Half Rack Quarter Rack Machine Database Sun Oracle Database Machine Full Rack Basic System 1 Database Servers InfiniBand Switches 8 3 4 2 2 2 1 1 Exadata Upgradability Storage Connect multiple Full Racks via included InfiniBand fabric Servers Field upgrade from Half Rack to Full Rack 14 Field upgrade from Quarter Rack to Half Rack 7 Custom field upgrade 3 9 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

The performance and capacity characteristics of each model of Database Machine is depicted in the following table. Sun Oracle Machine Quarter Sun Oracle Database Machine Database Half Rack Rack Basic System Machine Sun Exadata Smart Flash Cache Full Rack Oracle 5.3 TB Sun Oracle Database 2.6 TB Database Machine 1.1 TB 384 GB Raw SAS 100 50 21 TB Disk TB TB 72 TB Capacit SATA 336 168 7.2 TB y TB TB 24 TB User (with Up Up Up to Data out to to 6 TB Capacit data 28 14 21 TB y comp TB TB Up to SAS ressi 100 50 2 TB SATA on) TB TB 7 TB Raw Disk Up to Up to Up to Up to Data 21 10.5 4.5 1.5 GB/sec Bandwidth GB/sec GB/sec GB/sec 0.85 GB/sec SAS 12 6.0 2.5 SATA GB/sec GB/sec GB/sec Effective with Flash Up to 25 Up to 3.6 Data Cache GB/sec GB/sec Bandwidth Up to 50 Up to 11 GB/sec GB/sec Effective Data 10,800 Bandwidth 4,300 with Flash Up to Cache and 3,600 Data 1,440 Compression 10 White Paper A Technical Overview of Up to the Sun Oracle Exadata Storage Server and 500 GB/sec Database Machine Up to 250 GB/sec Up to 110 GB/sec Up to 36 GB/sec Flash Cache IOPS Up to 1,000,000 Up to 500,000 Up to 225,000 Up to 75,000 Disk IOPS SAS SATA Up to 50,000 20,000 Up to 25,000 10,000 Up to

In summary the Exadata products address the key dimensions of database I/O that can hamper performance. More pipes: Exadata is based on a massively parallel architecture which provides more pipes to deliver more data faster between the database servers and

storage servers. As Exadata servers are added to the database configuratio n bandwidth scales linearly. Wider pipes: InfiniBand is 8 times faster than Fibre Channel. Exadata is built using wider InfiniBand pipes that

provide extremely high bandwidth between the database servers and storage servers. More IOPS: With the intelligent and automatic use and manageme nt of Exadata Smart

Flash Cache to avoid physical I/O effective IOPS scale to handle the largest most demanding application s. Smart software: With the Smart Scan processing less data needs to be shipped through

the pipes by performing data processing in storage. Exadata is database aware and can ship just the data required to satisfy SQL requests resulting in less data being sent between the database servers and the storage servers.

11 White Paper A
Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

iDB Protocol over InfiniBand with Path Failover

Exadata Architecture
The hardware environment for a typical Exadata based storage grid was shown in Figure 2. Each Exadata cell is a self-contained server which houses disk storage and runs the Exadata software provided by Oracle. Databases are deployed across Exadata cells, and multiple databases can share Exadata cells. The database and Exadata cells communicate via a high-speed InfiniBand interface. The collection of Exadata cells shared between a set of databases is referred to as an Exadata Realm. The set of three cells in figure 2 is an example of a realm. Realms ensure the isolation, and hence protection, across a given set of databases. Mechanisms are provided to move disks and whole cells between realms in a controlled and safe manner. The architecture of the Exadata solution includes components on the database server and in the Exadata cell. The overall architecture is shown below.
DB ServerDB InstanceDBRMASMSingle-InstanceDatabaseRAC DatabaseDB ServerDB InstanceDBRMASMDB ServerDB InstanceDBRMASMOELCELLSRVMSRSIORMExadata Cell
Figure 4: Exadata Software Architecture

12 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Database Server Software Oracle Database 11g Release 2 has been significantly enhanced to take advantage of Exadata storage. The Exadata software is optimally divided between the database servers and Exadata cells. The database servers and Exadata Storage Server Software communicate using the iDB the Intelligent Database protocol. iDB is implemented in the database kernel and transparently maps database operations to Exadata-enhanced operations. iDB implements a function shipping architecture in addition to the traditional data block shipping provided by the database. iDB is used to ship SQL operations down to the Exadata cells for execution and to return query result sets to the database kernel. Instead of returning database blocks, Exadata cells return only the rows and columns that satisfy the SQL query. Like existing I/O protocols, iDB can also directly read and write ranges of bytes to and from disk so when offload processing is not possible Exadata operates like a traditional storage device for the Oracle Database. But when feasible, the intelligence in the database kernel enables, for example, table scans to be passed down to execute on the Exadata server so only requested data is returned to the database server. iDB is built on the industry standard Reliable Datagram Sockets (RDSv3) protocol and runs over InfiniBand. ZDP (Zero-loss Zero-copy Datagram Protocol), a zero-copy implementation of RDS, is used to eliminate unnecessary copying of blocks. Multiple network interfaces can be used on the database servers and Exadata cells. This is an extremely fast low-latency protocol that minimizes the number of data copies required to service I/O operations. Automatic Storage Management (ASM) is the storage management foundation of Exadata. ASM virtualizes the storage resources and provides the advanced volume management and file system capabilities of Exadata. Striping database files evenly across the available Exadata cells and disks results in uniform I/O load across all the storage hardware. The ability of ASM to perform non-intrusive resource allocation, and reallocation, is a key enabler of the shared grid storage capabilities of Exadata environments. And the ASM mirroring and failure group functionality provides much of the data protection and resiliency across the Exadata environment. With ASM, data is mirrored across cells to ensure high availability in the event of cell failure. The Database Resource Manager (DBRM) feature in Oracle Database 11g has been enhanced for use with Exadata. DBRM lets the user define and manage intra and inter-database I/O bandwidth in addition to CPU, undo, degree of parallelism, active sessions, and the other resources it manages. This allows the sharing of storage between databases without fear of one database monopolizing the I/O bandwidth and impacting the performance of the other databases sharing the storage. Consumer groups are allocated a percent of the available I/O bandwidth and the DBRM ensures these targets are delivered. This is implemented by the database tagging I/O with the associated database and consumer group. This provides the database with a complete view of the I/O priorities through the entire I/O stack. The intra-database consumer group I/O allocations are defined and managed at the database server. The inter-database I/O allocations are defined within the software in the Exadata cell and managed

13 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

by the I/O Resource Manager (IORM). The Exadata cell software ensures that inter-database I/O resources are managed and properly allocated within, and between, databases. Overall, DBRM ensures each database receives its specified amount of I/O resources and user defined SLAs are met. Enterprise Manager Plug-In For Exadata Exadata has been integrated with the Oracle Enterprise Manager (EM) Grid Control to easily monitor the Exadata environment. By installing an Exadata plug-in to the existing EM system, statistics and activity on the Exadata server can be monitored, and events and alerts can be sent to the system administrator. The advantages of integrating the EM system with Exadata include: Monitoring Oracle Exadata storage Gathering storage configuration and performance information Raising alerts and warnings based on thresholds Providing rich out-of-box metrics and reports based on historical data All the functions users have come to expect from the Oracle Enterprise Manager work along with Exadata. By using the EM interface, users can easily manage the Exadata environment along with other Oracle Database environments traditionally used with the Enterprise Manager. DBAs can use the familiar EM interface to view reports to determine the health of the Exadata system, and manage the configuration of the Exadata storage. Exadata Software Like any storage device the Exadata server is a computer with CPUs, memory, a bus, disks, NICs, and the other components normally found in a server. It also runs an operating system (OS), which in the case of Exadata is Oracle Enterprise Linux (OEL) 5.3. The Exadata Storage Server Software resident in the Exadata cell runs under OEL. OEL is accessible in a restricted mode to administer and manage the Exadata cell. CELLSRV (Cell Services) is the primary component of the Exadata software running in the cell and provides the majority of Exadata storage services. CELLSRV is multi-threaded software that communicates with the database instance on the database server, and serves blocks to databases based on the iDB protocol. It provides the advanced SQL offload capabilities, serves Oracle blocks when SQL offload processing is not possible, and implements the DBRM I/O resource management functionality to meter out I/O bandwidth to the various databases and consumer groups issuing I/O. Two other components of Oracle software running in the cell are the Management Server (MS) and Restart Server (RS). The MS is the primary interface to administer, manage and query the status of the Exadata cell. It works in cooperation with the Exadata cell command line interface

14 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

(CLI) and EM Exadata plug-in, and provides standalone Exadata cell management and configuration. For example, from the cell, CLI commands are issued to configure storage, query I/O statistics and restart the cell. Also supplied is a distributed CLI so commands can be sent to multiple cells to ease management across cells. Restart Server (RS) ensures the ongoing functioning of the Exadata software and services. It is used to update the Exadata software. It also ensures storage services are started and running, and services are restarted when required. Exadata Smart Scan Processing With traditional, non-iDB aware storage, all database intelligence resides in the database software on the server. To illustrate how SQL processing is performed in this architecture an example of a table scan is shown below.
Figure 5: Traditional Database I/O and SQL Processing Model

The client issues a SELECT statement with a predicate to filter and return only rows of interest. The database kernel maps this request to the file and extents containing the table being scanned. The database kernel issues the I/O to read the blocks. All the blocks of the table being queried are read into memory. Then SQL processing is done against the raw blocks searching for the rows that satisfy the predicate. Lastly the rows are returned to the client. As is often the case with the large queries, the predicate filters out most of the rows read. Yet all the blocks from the table need to be read, transferred across the storage network and copied into memory. Many more rows are read into memory than required to complete the requested SQL

15 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

operation. This generates a large number of data transfers which consume bandwidth and impact application throughput and response time. Integrating database functionality within the storage layer of the database stack allows queries, and other database operations, to be executed much more efficiently. Implementing database functionality as close to the hardware as possible, in the case of Exadata at the disk level, can dramatically speed database operations and increase system throughput. With Exadata storage, database operations are handled much more efficiently. Queries that perform table scans can be processed within Exadata with only the required subset of data returned to the database server. Row filtering, column filtering and some join processing (among other functions) are performed within the Exadata storage cells. When this takes place only the relevant and required data is returned to the database server. Figure 6 below illustrates how a table scan operates with Exadata storage.
Figure 6: Smart Scan Offload Processing

The client issues a SELECT statement with a predicate to filter and return only rows of interest. The database kernel determines that Exadata storage is available and constructs an iDB command representing the SQL command issued and sends it the Exadata storage. The CELLSRV component of the Exadata software scans the data blocks to identify those rows and columns that satisfy the SQL issued. Only the rows satisfying the predicate and the requested columns are read into memory. The database kernel consolidates the result sets from across the Exadata cells. Lastly, the rows are returned to the client. Smart scans are transparent to the application and no application or SQL changes are required. The SQL EXPLAIN PLAN shows when Exadata smart scan is used. Returned data is fully consistent and transactional and rigorously adheres to the Oracle Database consistent read functionality and behavior. If a cell dies during a smart scan, the uncompleted portions of the smart scan are transparently routed to another cell for completion. Smart scans properly handle

16 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

the complex internal mechanisms of the Oracle Database including: uncommitted data and locked rows, chained rows, compressed tables, national language processing, date arithmetic, regular expression searches, materialized views and partitioned tables. The Oracle Database and Exadata server cooperatively execute various SQL statements. Moving SQL processing off the database server frees server CPU cycles and eliminates a massive amount of bandwidth consumption which is then available to better service other requests. SQL operations run faster, and more of them can run concurrently because of less contention for the I/O bandwidth. We will now look at the various SQL operations that benefit from the use of Exadata.
Smart Scan Predicate Filtering

Exadata enables predicate filtering for table scans. Only the rows requested are returned to the database server rather than all rows in a table. For example, when the following SQL is issued only rows where the employees hire date is after the specified date are sent from Exadata to the database instance. SELECT * FROM employee_table WHERE hire_date > 1-Jan-2003; This ability to return only relevant rows to the server will greatly improve database performance. This performance enhancement also applies as queries become more complicated, so the same benefits also apply to complex queries, including those with subqueries.
Smart Scan Column Filtering

Exadata provides column filtering, also called column projection, for table scans. Only the columns requested are returned to the database server rather than all columns in a table. For example, when the following SQL is issued, only the employee_name and employee_number columns are returned from Exadata to the database kernel. SELECT employee_name, employee_number FROM employee_table; For tables with many columns, or columns containing LOBs (Large Objects), the I/O bandwidth saved can be very large. When used together, predicate and column filtering dramatically improves performance and reduces I/O bandwidth consumption. In addition, column filtering also applies to indexes, allowing for even faster query performance.
Smart Scan Join Processing

Exadata performs joins between large tables and small lookup tables, a very common scenario for data warehouses with star schemas. This is implemented using Bloom Filters, which are a very efficient probabilistic method to determine whether a row is a member of the desired result set.

17 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

New Smart Scan Offload I/O Optimizations and Functionality

With Oracle Database 11g Release 2, several new powerful Smart Scan and offload capabilities are provided with Exadata storage. These include: Storage Indexing technology, Smart Scan offload of new Hybrid Columnar Compressed Tables, Smart Scan offload of encrypted tablespaces and columns, and offload of data mining model scoring.
Storage Indexing

Storage Indexes are a very powerful capability provided in Exadata storage that helps avoid I/O operations. The Exadata Storage Server Software creates and maintains a Storage Index in Exadata memory. The Storage Index keeps track of minimum and maximum values of columns for tables stored on that cell. When a query specifies a WHERE clause, but before any I/O is done, the Exadata software examines the Storage Index to determine if rows with the specified column value exists in the cell by comparing the column value to the minimum and maximum values maintained in the Storage Index. If the column value is outside the minimum and maximum range, scan I/O for that query is avoided. Many SQL Operations will run dramatically faster because large numbers of I/O operations are automatically replaced by a few in-memory lookups. To minimize operational overhead, Storage Indexes are created and maintained transparently and automatically by the Exadata Storage Server Software.
Smart Scan of Hybrid Columnar Compressed Tables

Another new feature of Oracle Database 11g Release 2 is Hybrid Columnar Compressed Tables. These new tables offer a high degree of compression for data that is bulk loaded and queried. Smart Scan processing of Hybrid Columnar Compressed Tables is provided and column projection and filtering are performed within Exadata. In addition, the decompression of the data is offloaded to Exadata eliminating CPU overhead on the database servers. Given the typical ten-fold compression of Hybrid Columnar Compressed Tables, this effectively increases the I/O rate ten-fold compared to uncompressed data.
Smart Scan of Encrypted Tablespaces and Columns

New in Exadata is the Smart Scan offload processing of Encrypted Tablespaces (TSE) and Encrypted Columns (TDE). While the prior release of Exadata fully supported the use of TSE and TDE on Exadata it did not benefit from Exadata offload processing. This enhancement increases performance when accessing confidential data.
Offload of Data Mining Model Scoring

Another new function offloaded to Exadata is Data Mining model scoring. This makes the deployment of data warehouses on Exadata or Database Machine an even better and more performant data analysis platform. All data mining scoring functions (e.g., prediction_probability) are offloaded to Exadata for processing. This will not only speed warehouse analysis but reduce

18 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

database server CPU consumption and the I/O load between the database server and Exadata storage.
Other Exadata Smart Scan Processing

Two other database operations that are offloaded to Exadata are incremental database backups and tablespace creation. The speed and efficiency of incremental database backups has been significantly enhanced with Exadata. The granularity of change tracking in the database is much finer when Exadata storage is used. Changes are tracked at the individual Oracle block level with Exadata rather than at the level of a large group of blocks. This results in less I/O bandwidth being consumed for backups and faster running backups. With Exadata the create file operation is also executed much more efficiently. For example, when issuing a Create Tablespace command, instead of operating synchronously with each block of the new tablespace being formatted in server memory and written to storage, an iDB command is sent to Exadata instructing it to create the tablespace and format the blocks. Host memory usage is reduced and I/O associated with the creation and formatting of the tablespace blocks is offloaded. The I/O bandwidth saved with these operations means more bandwidth is available for other business critical work. I/O Resource Management With Exadata With traditional storage, creating a shared storage grid is hampered by the inability to prioritize the work of the various jobs and users consuming I/O bandwidth from the storage subsystem. The same occurs when multiple databases share the storage subsystem. The DBRM and I/O resource management capabilities of Exadata storage can prevent one class of work, or one database, from monopolizing disk resources and bandwidth and ensures user defined SLAs are met when using Exadata storage. The DBRM enables the coordination and prioritization of I/O bandwidth consumed between databases, and between different users and classes of work. By tightly integrating the database with the storage environment, Exadata is aware of what types of work and how much I/O bandwidth is consumed. Users can therefore have the Exadata system identify various types of workloads, assign priority to these workloads, and ensure the most critical workloads get priority. In data warehousing, or mixed workload environments, you may want to ensure different users and tasks within a database are allocated the correct relative amount of I/O resources. For example you may want to allocate 70% of I/O resources to interactive users on the system and 30% of I/O resources to batch reporting jobs. This is simple to enforce using the DBRM and I/O resource management capabilities of Exadata storage. An Exadata administrator can create a resource plan that specifies how I/O requests should be prioritized. This is accomplished by putting the different types of work into service groupings called Consumer Groups. Consumer groups can be defined by a number of attributes including

19 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

the username, client program name, function, or length of time the query has been running. Once these consumer groups are defined, the user can set a hierarchy of which consumer group gets precedence in I/O resources and how much of the I/O resource is given to each consumer group. This hierarchy determining I/O resource prioritization can be applied simultaneously to both intra-database operations (i.e. operations occurring within a database) and inter-database operations (i.e. operations occurring among various databases). When Exadata storage is shared between multiple databases you can also prioritize the I/O resources allocated to each database, preventing one database from monopolizing disk resources and bandwidth to ensure user defined SLAs are met. For example you may have two databases sharing Exadata storage as depicted below. Exadata CellExadata CellExadata CellDatabase B(RAC)Database A(Single-Instance) CellExadata CellExadata
CellExadata
Figure 7: Inter-Database I/O Resource Management with Exadata

Business objectives dictate that each of these databases has a relative value and importance to the organization. It is decided that database A should receive 33% of the total I/O resources available and that database B should receive 67% of the total I/O of resources. To ensure the different users and tasks within each database are allocated the correct relative amount of I/O resources, various consumer groups are defined.

20 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

 Two

consumer groups are defined for database A 60% of the I/O resources are reserved for interactive marketing activities 40% of the I/O resources are reserved for batch marketing activities Three consumer groups are defined for database B 60% of the I/O resources are reserved for interactive sales activities 30% of the I/O resources are reserved for batch sales activities 10% of the I/O resources are reserved for major account sales activities These consumer group allocations are relative to the total I/O resources allocated to each database. In essence, Exadata I/O Resource Manager has solved one of the challenges traditional storage technology does not address: creating a shared grid storage environment with the ability to balance and prioritize the work of multiple databases and users sharing the storage subsystem. Exadata I/O resource management ensures user defined SLAs are met for multiple databases sharing Exadata storage. This ensures that each database or user gets the correct share of disk bandwidth to meet business objectives. Accelerated Performance With Exadata Exadata storage provides unmatched performance improvements for typical data warehousing workloads. Full table scans will receive an arbitrarily large improvement due to smart scan filtering and the balanced hardware used for Exadata-based data warehouses. Exadata storage servers deliver a scale-out architecture such that as cells are added to the configuration, bandwidth increases. This, coupled with faster InfiniBand interconnect and the reduction of data transferred due to the offload processing, yields very large performance improvements. Often a ten-fold speed up in these operations is seen when using Exadata storage compared to storage products traditionally used with the Oracle Database but in many cases a 50-fold, or greater, speedup is achieved. Two examples of real world performance improvements follow.

21 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

28x 16x
01020304050607080CDR Full Table ScanWarehouse Inventory ReportCRM Service Order ReportCRM Customer Discount ReportHandset to Customer Mapping ReportIndex CreationTablespace Creation
Figure 8: Telecommunications Application Performance Improvement with Exadata of 10X to 72X
-5.010.015.020.025.030.035.040.045.050.0Recall QueryGift Card ActivationsSales and Customer CountsPrompt04 Clone for ACL auditDate to Date MovementComparison - 53 weeksMaterialized Views RebuildMerchandising Level 1 Detail byWeekSupply Chain Vendor - Year - ItemMovementMerchandising Level 1 Detail:Current - 52 weeksMerchandising Level 1 Detail:Period Ago

Figure 9: Retail Application Performance Improvement of 3X to 48X

Exadata Storage Virtualization Exadata provides a rich set of sophisticated and powerful storage management virtualization capabilities that leverage the strengths of the Oracle Database, the Exadata software, and Exadata hardware.
Exadata Storage Software

As discussed earlier, the Exadata cell is a server that runs the Oracle Enterprise Linux as well as the Oracle provided Exadata software. When first started, the cell boots up like any other computer into Exadata storage serving mode. The first two disk drives have a small Logical Unit Number (LUN) slice called the System Area, approximately 13 GB of size, reserved for the OEL

22 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

operating system, Exadata software, and configuration metadata. The System Area contains Oracle Database 11g Automatic Diagnostic Repository (ADR) data, and other metadata about the Exadata cell. The administrator does not have to manage the System Area LUN, as it is automatically created. Its contents are automatically mirrored across the physical disks to protect against drive failures, and to allow hot disk swapping. The remaining portion of these two disk drives is available for user data.
Exadata User Storage Virtualization

Automatic Storage Management (ASM) is used to manage the storage in the Exadata cell. ASM volume management, striping, and data protection services make it the optimum choice for volume management. ASM provides data protection against drive and cell failures, the best possible performance, and extremely flexible configuration and reconfiguration options. A Cell Disk is the virtual representation of the physical disk, minus the System Area LUN (if present), and is one of the key disk objects the administrator manages within an Exadata cell. A Cell Disk is represented by a single LUN, which is created and managed automatically by the Exadata software when the physical disk is discovered. Cell Disks can be further virtualized into one or more Grid Disks. Grid Disks are the disk entity assigned to ASM, as ASM disks, to manage on behalf of the database for user data. The simplest case is when a single Grid Disk takes up the entire Cell Disk. But it is also possible to partition a Cell Disk into multiple Grid Disk slices. Placing multiple Grid Disks on a Cell Disk allows the administrator to segregate the storage into pools with different performance or availability requirements. Grid Disk slices can be used to allocate hot, warm and cold regions of a Cell Disk, or to separate databases sharing Exadata disks. For example a Cell Disk could be partitioned such that one Grid Disk resides on the higher performing portion of the physical disk and is configured to be triple mirrored, while a second Grid Disk resides on the lower performing portion of the disk and is used for archive or backup data, without any mirroring. An Information Lifecycle Management (ILM) strategy could be implemented using Grid Disk functionality.
PhysicalDiskCellDiskGridDiskGridDiskGrid DiskGrid DiskPhysicalDiskPhysicalDiskPhysicalDiskCellDiskCellDiskGridDiskGridDiskGridDiskGrid DiskGridDiskGrid Disk
Figure 10: Grid Disk Virtualization

The following example illustrates the relationship of Cell Disks to Grid Disks in a more comprehensive Exadata storage grid.

23 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Once the Cell Disks and Grid Disks are configured, ASM disk groups are defined across the Exadata configuration. Two ASM disk groups are defined; one across the hot grid disks, and a second across the cold grid disks. All of the hot grid disks are placed into one ASM disk group and all of the cold grid disks are placed in a separate disk group. When the data is loaded into the database, ASM will evenly distribute the data and I/O within disk groups. ASM mirroring can be activated for these disk groups to protect against disk failures for both, either, or neither of the disk groups. Mirroring can be turned on or off independently for each of the disk groups.
Exadata CellExadata CellHotHotHotHotHotHotColdColdColdColdColdCold CellHotHotHotHotHotHotHotHotHotHotHotHotColdColdColdColdColdColdColdColdColdColdColdCold

Hot ASMDisk GroupCold ASMDisk GroupExadata Hot GroupHot GroupCold Group

Figure 11: Example ASM Disk Groups and Mirroring

Lastly, to protect against the failure of an entire Exadata cell, ASM failure groups are defined. Failure groups ensure that mirrored ASM extents are placed on different Exadata cells.
Exadata CellExadata CellHotHotHotHotHotHotColdColdColdColdColdColdASMDisk CellHotHotHotHotHotHotHotHotHotHotHotHotColdColdColdColdColdColdColdColdColdColdColdColdASMDisk Figure 12: Example ASM Mirroring and Failure Groups

GroupASMFailure GroupASMFailure GroupExadata Group

24 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

With Exadata and ASM: Configuration of Cell Disks (LUN creation) is automated by Exadata software. Optionally, multiple Grid Disks can co-exist on the physical disks to tailor performance to the needs of the database application or construct an ILM strategy with Exadata. ASM automatically stripes the database data across Exadata disks and cells to ensure a balanced I/O load and optimum performance. ASM dynamic add and drop capability enables non-intrusive cell and disk allocation, deallocation, and reallocation. ASM mirroring, and the hot swap capability of the Exadata cell, provides transparent data protection and access across disk failures. ASM provides for double or triple mirroring to tailor the protection to the criticality of the data. ASM failure groups are automatically created with Exadata to provide transparent data protection and access across cell failures.
Migrating to Exadata Storage

Exadata storage can be used in addition to the storage arrays and products traditionally used to store the Oracle database. A single database can be partially stored on Exadata storage and partially on traditional storage devices. Tablespaces can reside on Exadata storage, non-Exadata storage, or a combination of the two, and is transparent to database operations and applications. But to benefit from the Smart Scan capability of Exadata storage, the entire tablespace must reside on Exadata storage. This co-residence and co-existence is a key feature to enable online migration to Exadata storage. An online non-disruptive migration to Exadata storage can be done for an existing database if the existing database is deployed on ASM and is using ASM redundancy. The steps to accomplish this are: 1. Add an Exadata grid disk to the existing ASM disk group. 2. ASM then automatically rebalances the data within the disk group moving a proportional amount of data to the newly added Exadata grid disk. 3. Then a non-Exadata disk is dropped from the ASM disk group. ASM would then rebalance or migrate the data from the non-Exadata disk to other disks in the disk group. 4. The above is repeated until the entire database has been migrated onto Exadata storage. In addition, migration can be done using Oracle Recovery Manager (RMAN) to backup from traditional storage and restore the data onto Exadata. Oracle Data Guard can also be used to facilitate a migration. This is done by first creating a standby database based on Exadata storage.

25 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

The standby can be using Exadata storage and the production database can be on traditional storage. By executing a fast switchover, taking just seconds, you can transform the standby database into the production database. All these approaches provide a built-in safety net as you can undo the migration very gracefully if unforeseen issues arise.
Additional Data Protection With Exadata

Exadata has been designed to incorporate the same standard of high availability (HA) customers have come to expect from Oracle products. With Exadata, all database features and tools work just as they do with traditional non-Exadata storage. Users and database administrators will use familiar tools and be able to leverage their existing Oracle Database knowledge and procedures. With the Exadata architecture, all single points of failure are eliminated. Familiar features such as mirroring, fault isolation, and protection against drive and cell failure have been incorporated into Exadata to ensure continual availability and protection of data. Other features to ensure high availability within the Exadata server are described below.
Hardware Assisted Resilient Data (HARD) built into Exadata

Oracle's Hardware Assisted Resilient Data (HARD) Initiative is a comprehensive program designed to prevent data corruptions before they happen. Data corruptions are very rare, but when they happen, they can have a catastrophic effect on a database, and therefore a business. Exadata has enhanced HARD functionality embedded in it to provide even higher levels of protection and end-to-end data validation for your data. Exadata performs extensive validation of the data stored in it including checksums, block locations, magic numbers, head and tail checks, alignment errors, etc. Implementing these data validation algorithms within Exadata will prevent corrupted data from being written to permanent storage. Furthermore, these checks and protections are provided without the manual steps required when using HARD with conventional storage.
Data Guard

Oracle Data Guard is the software feature of Oracle Database that creates, maintains, and monitors one or more standby databases to protect your database from failures, disasters, errors, and corruptions. Data Guard works unmodified with Exadata and can be used for both production and standby databases. By using Active Data Guard with Exadata storage, queries and reports can be offloaded from the production database to an extremely fast standby database and ensure that critical work on the production database is not impacted while still providing disaster protection.
Flashback

Exadata leverages Oracle Flashback Technology to provide a set of features to view and restore data back in time. The Flashback feature works in Exadata the same as it would in a nonExadata

26 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine 27

environment. The Flashback features offer the capability to query historical data, perform change analysis, and perform self-service repair to recover from logical corruptions while the database is online. In essence, with the built-in Oracle Flashback features, Exadata allows the user to have snapshot-like capabilities and restore a database to a time before an error occurred.
Recovery Manager (RMAN) and Oracle Secure Backup (OSB)

Exadata works with Oracle Recovery Manager (RMAN), a command-line and Enterprise Managerbased tool, to allow efficient Oracle database backup and recovery. All existing RMAN scripts work unchanged in the Exadata environment. RMAN is designed to work intimately with the server, providing block-level corruption detection during backup and restore. RMAN optimizes performance and space consumption during backup with file multiplexing and backup set compression, and integrates with Oracle Secure Backup (OSB) and third party media management products for tape backup.

CONCLUSION
Businesses today increasingly needs to leverage a unified database platform to enable the deployment and consolidation of all their applications onto one common infrastructure. Whether OLTP, DW or mixed workload a common infrastructure delivers the efficences and reusability the datacenter needs and provides the reality of cloud computing in-house. Building or using custom special purpose systems for different applications is wasteful and expensive. The need to process more data increases every day while corporations are also finding their IT budgets being squeezed. Examining the total cost of ownership (TCO) for IT software and hardware lead one to choose high performance common infrastructure for deployments of all applications. By incorporating Exadata and the Database Machine into the IT infrastructure, companies will: Accelerate database performance and be able to do much more in the same amount of time. Handle change and growth in scalable and incremental steps by consolidating deployments on to a common infrastructure. Deliver mission-critical data availability and protection. Exadata and the Database Machine provide this solution.

White Paper Title September 2009 Author: Ronald Weiss

Contributing Authors: Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 oracle.com Copyright 2009, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. 0109

An Oracle White Paper September 2009

A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Sun Oracle Exadata Storage Server and Database Machine.............2 Todays Limits On Database I/O.........................................................2 Exadata Product Family......................................................................4 Sun Oracle Exadata Storage Server...............................................4 Sun Oracle Database Machine.......................................................8 Exadata Architecture.........................................................................12 Database Server Software............................................................13 Enterprise Manager Plug-In For Exadata......................................14 Exadata Software..........................................................................14 Exadata Smart Scan Processing..................................................15 I/O Resource Management With Exadata.....................................19 Accelerated Performance With Exadata.......................................21 Exadata Storage Virtualization......................................................22 CONCLUSION..................................................................................27White Paper A Technical
Overview of the Sun Oracle Exadata Storage Server and Database Machine

Sun Oracle Exadata Storage Server and Database Machine

The Sun Oracle Exadata Storage Server (Exadata) is a storage product optimized for use with Oracle Database applications and is the storage building block of the Sun Oracle Database Machine. Exadata delivers outstanding I/O and SQL processing performance for online transaction processing (OLTP), data warehousing (DW) and consolidation of mixed workloads. Extreme performance is delivered for all types of database applications by leveraging a massively parallel grid architecture and Exadata Smart Flash Cache to dramatically accelerate Oracle Database processing and speed I/O operations. The Exadata storage products are a combination of software and hardware used to store and access Oracle databases. Exadata provides database aware storage services, such as the ability to offload database processing from the database server to storage, and provides this while being transparent to SQL processing and database applications. The Exadata Storage Servers are also packaged in a complete end-to-end database solution the Sun Oracle Database Machine. The Sun Oracle Database Machine is an easy to deploy out of the box solution for hosting the Oracle Database for all applications and delivers the highest levels of performance available. Database Machine and Exadata storage delivers breakthrough performance with linear I/O scalability, is simple to use and manage, and delivers mission-critical availability and reliability to the enterprise. Exadata is a joint offering from Oracle and Sun Microsystems. Sun is providing the hardware technology used in the Database Machine and Exadata Storage Server. Oracle is providing the software to impart database intelligence to the storage and Database Machine and is tightly integrated with the Oracle Database and all its features. The Sun servers combine the power of the latest generation of Intel Xeon processors with Sun's system engineering expertise. These servers offer the needed density and expandability to satisfy the most demanding datacenter applications. The Oracle and Sun partnership makes possible the delivery of the Sun Oracle Database Machine and Exadata Storage Server and the revolutionary capabilities it provides.

Todays Limits On Database I/O

The foundation of the Database Machine and Exadata is smart database software to handle the complex applications deployed to drive the most demanding enterprise business needs. The

2 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Oracle Database provides an incredible amount of functionality to implement the most sophisticated OLTP and DW applications and to consolidate mixed workload environments. But to access terabytes databases with high performance, augmenting the smart database software with powerful hardware provides tremendous opportunities to deliver more database processing, faster, for the enterprise. Having powerful hardware to provide the required I/O rates and bandwidth for todays applications, in addition to smart software, is key to the extreme performance delivered by the Exadata family of products. Traditional storage devices offer high storage capacity but are relatively slow and can not sustain the I/O rates for the transaction load the enterprise requires for its applications. Instead of hundreds of IOPS (I/Os per second) per disk enterprise applications require their systems deliver at least an order of magnitude higher IOPS to deliver the service enterprise end-users expect. This problem gets magnified when hundreds of disks reside behind a single storage controller. The IOPS that can be executed are severely limited by both the speed of the mechanical disk drive and the number of drives per storage controller. Traditional storage products provide the Oracle Database a narrow and limited interface to database storage. Many bottlenecks exist in the database I/O path restricting data bandwidth, and hence limiting overall database performance when using traditional storage products. Database servers need many Storage Area Network (SAN) Host Bus Adapters (HBA) to provide the bandwidth necessary to deliver data, from storage to the database, at an adequate rate. Very often the number of HBAs required to deliver adequate performance cannot be supported in the server, or are too costly to provide. And SAN switch cost and complexity increases dramatically to provide the required bandwidth and redundancy. In addition large storage arrays cannot deliver adequate bandwidth to the hundreds of disks they house. This results in the potential performance of disks being artificially capped well below what they are capable of providing. Disk performance is bottlenecked on the Fibre Channel Loops (FCL) to disk and the processing capacity of the storage array. Traditional storage devices are also unaware that a database is residing on the storage and therefore cannot provide any database-aware I/O or SQL processing. When the database requests rows and columns what is returned from the storage are data blocks rather than the result set of a database query. Traditional storage has no database intelligence to discern the particular rows and columns actually requested. So, when processing I/O on behalf of the database, traditional storage consumes bandwidth returning data that is not relevant to the database query that was issued. Exadata products address the key dimensions of database I/O that can hamper database performance. Exadata is based on a massively parallel architecture which provides more pipes to deliver more data faster between the database servers and the storage servers.

3 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

 Exadata

is built using wider pipes that provide extremely high bandwidth between the database servers and the storage servers. Exadata is database aware and can ship just the data required to satisfy SQL requests resulting in less data being sent between the database servers and the storage servers. Exadata overcomes the mechanical limits of disk drive technology by automatically caching frequently accessed data delivering unprecedented levels of bandwidth and IOPS.

Exadata Product Family

There are two members of the Exadata product family. The foundation of the Exadata family of products is the Sun Oracle Exadata Storage Server. It is used as the storage for the Oracle Database when building custom database systems. The second member of the Exadata product family is the Sun Oracle Database Machine (Database Machine). The Database Machine is a complete and fully integrated database system that includes all the components to quickly and easily deploy any enterprise database application requiring the best performance, and includes Exadata storage. Sun Oracle Exadata Storage Server The Sun Oracle Exadata Storage Server is a database storage device running the Exadata Storage Server Software provided by Oracle. The hardware components of the Exadata Storage Server (also referred to as an Exadata cell) were carefully chosen to match the needs of high performance database processing. The Exadata software is optimized to take the best possible advantage of the hardware components and Oracle Database. Each Exadata cell delivers outstanding I/O performance and bandwidth to the database. The Sun Oracle Exadata Storage Server is a fast, reliable, high capacity, industry- standard storage server. Each Exadata cell comes preconfigured with: two Intel Xeon E5540 quad-core processors, 384 GB of Exadata Smart Flash Cache, twelve disks connected to a storage controller with 512MB battery-backed cache, 24 GB memory, dual port InfiniBand connectivity, management interface for remote access, dual-redundant hot-swappable power supplies, all the software preinstalled, and takes up 2U in a typical 19inch rack.

4 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Figure 1: Exadata Storage Cell

Exadata Smart Flash Cache

Each Exadata cell comes with 384 GB of Exadata Smart Flash Cache. This solid state storage delivers dramatic performance advantages with Exadata storage. It provides a ten-fold improvement in response time for reads over regular disk; a hundred-fold improvement in IOPS for reads over regular disk; and is a less expensive higher capacity alternative to memory. Overall it delivers a tenfold increase performing a blended average of read and write operations. The Exadata Smart Flash Cache manages active data from regular disks in the Exadata cell but it is not managed in a simple Least Recently Used (LRU) fashion. The Exadata Storage Server Software in cooperation with the Oracle Database keeps track of data access patterns and knows what and how to cache data and avoid polluting the cache. This functionality is all managed automatically and does not require manual tuning. If there are specific tables or indexes that are known to be key to the performance of a database application they can optionally be identified and pinned in cache.
Exadata Storage Server Performance, Bandwidth and IOPS

The Sun Oracle Exadata Storage Server comes with either twelve 600 GB Serial Attached SCSI (SAS) disks or twelve 2 TB Serial Advanced Technology Attachment (SATA) disks. SAS based Exadata Storage Servers provide up to 2 TB of uncompressed user data capacity, and up to 1.5 GB/second of raw data bandwidth. SATA based Exadata Storage Servers provide up to 7 TB of uncompressed user data capacity, and up to 0.85 GB/second of raw data bandwidth. When stored in compressed format, the amount of user data and the amount of data bandwidth delivered by each cell increases up to 10 times. User data capacity is computed after mirroring all the disk space, and setting aside space for database structures like logs, undo, and temp space. Actual user data varies by application. The performance that each cell delivers is extremely high due to the Exadata Smart Flash Cache. The automated caching of the Flash cache enables each Exadata cell to deliver up to 3.6 GB/second bandwidth and 75,000 IOPS when accessing uncompressed data. When data is stored in compressed format, the amount of user data capacity, the amount of data bandwidth

5 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

and IOPS achievable, often increases up to ten times. This represents a significant improvement over traditional storage devices used with the Oracle Database. The performance specifications of the Exadata Storage Server are shown below. SAS Based User Data Up to 3.6 GB/sec Exadata Storage Capacity Effective Data Bandwidth with Server (without data Flash Cache and Data SATA Based compression) Compression Exadata Storage Up to 2 TB Up to 36 GB/sec Server Up to 7 TB Up to 36 GB/sec Exadata Smart Raw Disk Data Flash Cache IOPS Flash Cache Bandwidth Up to 75,000 384 GB Up to 1.5 GB/sec Up to 75,000 384 GB Up to 0.85 GB/sec Disk IOPS Raw Disk Effective Data Up to 3,600 Capacity Bandwidth with Up to 1,440 7.2 TB Flash 24 TB Up to 3.6 GB/sec
InfiniBand and the Exadata Storage Server

Oracle Exadata storage uses a state of the art InfiniBand interconnect between the servers and storage. An Exadata cell has dual port Quad Data Rate (QDR) InfiniBand connectivity for high availability. Each InfiniBand link provides 40 Gigabits of bandwidth - many times higher than traditional storage or server networks. Further, Oracle's interconnect protocol uses direct data placement (DMA - direct memory access) to ensure very low CPU overhead by directly moving data from the wire to database buffers with no extra data copies being made. The InfiniBand network has the flexibility of a LAN network, with the efficiency of a SAN. By using an InfiniBand network, Oracle ensures that the network will not bottleneck performance. The same InfiniBand network also provides a high performance cluster interconnect for the Oracle Database Real Application Cluster (RAC) nodes.
Exadata Storage Server Configuration

In figure 2 below, a small Exadata storage based database environment is shown. Two Oracle Databases, one RAC and one single instance, are sharing three Exadata cells. All the components for this configuration database servers, Exadata cells, InfiniBand switches, Ethernet switches, and other support hardware can be housed in, and take up less than half of, a typical 19-inch rack.

6 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Exadata CellInfiniBand Switch/NetworkSingle-InstanceDatabaseRAC DatabaseExadata CellExadata CellExadata CellExadata CellExadata Cell

Figure 2: Exadata Storage Cell Based Configuration

Oracle Exadata is architected to scale-out to any level of performance. To achieve higher performance and greater storage capacity, additional Exadata cells are added to the configuration. As more cells are added, capacity and performance increases linearly. No cell-to-cell communication is ever done or required in an Exadata configuration. Oracle Automatic Storage Management (ASM) is used as the file system and volume manager for Exadata. The disk mirroring provided by ASM, combined with hot swappable Exadata disks, ensure the database can tolerate the failure of individual disk drives. Data is mirrored across cells to ensure that the failure of a cell will not result in loss of data, or inhibit data accessibility. This massively parallel architecture delivers unbounded scalability and high availability. When using Exadata, SQL processing is offloaded from the database server to the Exadata server. Exadata enables function shipping from the database instance to the underlying storage in addition to providing traditional block serving services to the database. One of the unique things the Exadata storage does compared to traditional storage is return only the rows and columns that satisfy the database query rather than the entire table being queried. Exadata pushes SQL processing as close to the data (or disks) as possible and gets all the disks operating in parallel. This reduces CPU consumption on the database server, consumes much less bandwidth moving data between database servers and storage servers, and returns a query result set rather than entire tables. Eliminating data transfers and database server workload can greatly benefit data warehousing queries that traditionally become bandwidth and CPU constrained. Eliminating data transfers can also have a significant benefit on online transaction processing (OLTP) systems that often include large batch and report processing operations. Exadata storage is totally transparent to the application using the database. Existing SQL statements, whether ad hoc or in packaged or custom applications, are unaffected and do not require any modification when Exadata storage is used. The offload processing and bandwidth

7 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

advantages of the solution are delivered without any modification to your application. And all features of the Oracle Database are fully supported with Exadata. Exadata works equally well with single-instance or Real Application Cluster deployments of the Oracle Database. Functionality like Oracle Data Guard, Oracle Recovery Manager (RMAN), Oracle Streams, and other database tools are administered the same, with or without Exadata. Users and database administrators leverage the same tools and knowledge they are familiar with today because they work just as they do with traditional non-Exadata storage. Both Exadata and non-Exadata storage may be concurrently used for database storage to facilitate migration to, or from, Exadata storage. The nature of traditional storage products encourages inefficient deployments of storage for each database in the IT infrastructure. The Exadata architecture ensures all the bandwidth and I/O resources of the Exadata storage subsystem can be made available whenever, and to whichever, database or class of work needs it. I/O bandwidth is metered out to the various classes of work, or databases, sharing the Exadata server based on user defined policies and service level agreements (SLAs). The Oracle Database Resource Manager (DBRM) has been enhanced for use with Exadata storage to manage user-defined intra and inter-database I/O resource usage to ensure customer defined SLAs are met. The I/O resource management capabilities of Exadata storage enable tailoring the I/O resources to the business priorities of the organization, and to build a shared storage grid for the Oracle databases in the environment. Sun Oracle Database Machine Oracle is also offering a fully integrated platform for all your database applications. The Sun Oracle Database Machine is an easy to deploy out of the box solution for hosting the Oracle Database. A fully integrated solution ready to be turned on day one takes a lot of integration work, cost and time, out of the application deployment process. The benefit of a common infrastructure to deploy any application on, whether OLTP, DW of a mix of the two, creates tremendous efficiencies in the datacenter.
Figure 3: Database Machine Full Rack

8 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

There are four models of the Database Machine Database Machine Full Rack, Database Machine Half Rack, Database Machine Quarter Rack, and Database Machine Eighth Rack are offered. Depending on the size and purpose of the database to be deployed, and the processing and I/O bandwidth required there is a system available to meet any need. Each Database Machine runs the same software, is upgradeable and includes common hardware components. Common to all Database Machines are: Exadata Storage Servers, either SAS or SATA. Industry standard Oracle Database 11g database servers with: two Intel Xeon dual-socket quad-core E5540 processors running at 2.53 Ghz processors, 72 GB RAM, four 146 GB SAS drives, dual port InfiniBand Host Channel Adapter (HCA), four 1 Gb/second Ethernet ports, and dual-redundant, hot-swappable power supplies. Sun Quad Data Rate (QDR) InfiniBand switches and cables to form a 40 Gb/second InfiniBand fabric for database server to Exadata storage server communication and RAC internode communication. The ratio of components to each other has been chosen to maximize performance and ensure system resiliency. The hardware composition of each model of Database Machine is depicted in the following table. Sun Oracle Sun Machine Sun Oracle Database Machine Database Oracle Half Rack Quarter Rack Machine Database Sun Oracle Database Machine Full Rack Basic System 1 Database Servers InfiniBand Switches 8 3 4 2 2 2 1 1 Exadata Upgradability Storage Connect multiple Full Racks via included InfiniBand fabric Servers Field upgrade from Half Rack to Full Rack 14 Field upgrade from Quarter Rack to Half Rack 7 Custom field upgrade 3 9 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

The performance and capacity characteristics of each model of Database Machine is depicted in the following table. Sun Oracle Machine Quarter Sun Oracle Database Machine Database Half Rack Rack Basic System Machine Sun Exadata Smart Flash Cache Full Rack Oracle 5.3 TB Sun Oracle Database 2.6 TB Database Machine 1.1 TB 384 GB Raw SAS 100 50 21 TB Disk TB TB 72 TB Capacit SATA 336 168 7.2 TB y TB TB 24 TB User (with Up Up Up to Data out to to 6 TB Capacit data 28 14 21 TB y comp TB TB Up to SAS ressi 100 50 2 TB SATA on) TB TB 7 TB Raw Disk Up to Up to Up to Up to Data 21 10.5 4.5 1.5 GB/sec Bandwidth GB/sec GB/sec GB/sec 0.85 GB/sec SAS 12 6.0 2.5 SATA GB/sec GB/sec GB/sec Effective with Flash Up to 25 Up to 3.6 Data Cache GB/sec GB/sec Bandwidth Up to 50 Up to 11 GB/sec GB/sec Effective Data 10,800 Bandwidth 4,300 with Flash Up to Cache and 3,600 Data 1,440 Compression 10 White Paper A Technical Overview of Up to the Sun Oracle Exadata Storage Server and 500 GB/sec Database Machine Up to 250 GB/sec Up to 110 GB/sec Up to 36 GB/sec Flash Cache IOPS Up to 1,000,000 Up to 500,000 Up to 225,000 Up to 75,000 Disk IOPS SAS SATA Up to 50,000 20,000 Up to 25,000 10,000 Up to

In summary the Exadata products address the key dimensions of database I/O that can hamper performance. More pipes: Exadata is based on a massively parallel architecture which provides more pipes to deliver more data faster between the database servers and

storage servers. As Exadata servers are added to the database configuratio n bandwidth scales linearly. Wider pipes: InfiniBand is 8 times faster than Fibre Channel. Exadata is built using wider InfiniBand pipes that

provide extremely high bandwidth between the database servers and storage servers. More IOPS: With the intelligent and automatic use and manageme nt of Exadata Smart

Flash Cache to avoid physical I/O effective IOPS scale to handle the largest most demanding application s. Smart software: With the Smart Scan processing less data needs to be shipped through

the pipes by performing data processing in storage. Exadata is database aware and can ship just the data required to satisfy SQL requests resulting in less data being sent between the database servers and the storage servers.

11 White Paper A
Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

iDB Protocol over InfiniBand with Path Failover

Exadata Architecture
The hardware environment for a typical Exadata based storage grid was shown in Figure 2. Each Exadata cell is a self-contained server which houses disk storage and runs the Exadata software provided by Oracle. Databases are deployed across Exadata cells, and multiple databases can share Exadata cells. The database and Exadata cells communicate via a high-speed InfiniBand interface. The collection of Exadata cells shared between a set of databases is referred to as an Exadata Realm. The set of three cells in figure 2 is an example of a realm. Realms ensure the isolation, and hence protection, across a given set of databases. Mechanisms are provided to move disks and whole cells between realms in a controlled and safe manner. The architecture of the Exadata solution includes components on the database server and in the Exadata cell. The overall architecture is shown below.
DB ServerDB InstanceDBRMASMSingle-InstanceDatabaseRAC DatabaseDB ServerDB InstanceDBRMASMDB ServerDB InstanceDBRMASMOELCELLSRVMSRSIORMExadata Cell
Figure 4: Exadata Software Architecture

12 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Database Server Software Oracle Database 11g Release 2 has been significantly enhanced to take advantage of Exadata storage. The Exadata software is optimally divided between the database servers and Exadata cells. The database servers and Exadata Storage Server Software communicate using the iDB the Intelligent Database protocol. iDB is implemented in the database kernel and transparently maps database operations to Exadata-enhanced operations. iDB implements a function shipping architecture in addition to the traditional data block shipping provided by the database. iDB is used to ship SQL operations down to the Exadata cells for execution and to return query result sets to the database kernel. Instead of returning database blocks, Exadata cells return only the rows and columns that satisfy the SQL query. Like existing I/O protocols, iDB can also directly read and write ranges of bytes to and from disk so when offload processing is not possible Exadata operates like a traditional storage device for the Oracle Database. But when feasible, the intelligence in the database kernel enables, for example, table scans to be passed down to execute on the Exadata server so only requested data is returned to the database server. iDB is built on the industry standard Reliable Datagram Sockets (RDSv3) protocol and runs over InfiniBand. ZDP (Zero-loss Zero-copy Datagram Protocol), a zero-copy implementation of RDS, is used to eliminate unnecessary copying of blocks. Multiple network interfaces can be used on the database servers and Exadata cells. This is an extremely fast low-latency protocol that minimizes the number of data copies required to service I/O operations. Automatic Storage Management (ASM) is the storage management foundation of Exadata. ASM virtualizes the storage resources and provides the advanced volume management and file system capabilities of Exadata. Striping database files evenly across the available Exadata cells and disks results in uniform I/O load across all the storage hardware. The ability of ASM to perform non-intrusive resource allocation, and reallocation, is a key enabler of the shared grid storage capabilities of Exadata environments. And the ASM mirroring and failure group functionality provides much of the data protection and resiliency across the Exadata environment. With ASM, data is mirrored across cells to ensure high availability in the event of cell failure. The Database Resource Manager (DBRM) feature in Oracle Database 11g has been enhanced for use with Exadata. DBRM lets the user define and manage intra and inter-database I/O bandwidth in addition to CPU, undo, degree of parallelism, active sessions, and the other resources it manages. This allows the sharing of storage between databases without fear of one database monopolizing the I/O bandwidth and impacting the performance of the other databases sharing the storage. Consumer groups are allocated a percent of the available I/O bandwidth and the DBRM ensures these targets are delivered. This is implemented by the database tagging I/O with the associated database and consumer group. This provides the database with a complete view of the I/O priorities through the entire I/O stack. The intra-database consumer group I/O allocations are defined and managed at the database server. The inter-database I/O allocations are defined within the software in the Exadata cell and managed

13 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

by the I/O Resource Manager (IORM). The Exadata cell software ensures that inter-database I/O resources are managed and properly allocated within, and between, databases. Overall, DBRM ensures each database receives its specified amount of I/O resources and user defined SLAs are met. Enterprise Manager Plug-In For Exadata Exadata has been integrated with the Oracle Enterprise Manager (EM) Grid Control to easily monitor the Exadata environment. By installing an Exadata plug-in to the existing EM system, statistics and activity on the Exadata server can be monitored, and events and alerts can be sent to the system administrator. The advantages of integrating the EM system with Exadata include: Monitoring Oracle Exadata storage Gathering storage configuration and performance information Raising alerts and warnings based on thresholds Providing rich out-of-box metrics and reports based on historical data All the functions users have come to expect from the Oracle Enterprise Manager work along with Exadata. By using the EM interface, users can easily manage the Exadata environment along with other Oracle Database environments traditionally used with the Enterprise Manager. DBAs can use the familiar EM interface to view reports to determine the health of the Exadata system, and manage the configuration of the Exadata storage. Exadata Software Like any storage device the Exadata server is a computer with CPUs, memory, a bus, disks, NICs, and the other components normally found in a server. It also runs an operating system (OS), which in the case of Exadata is Oracle Enterprise Linux (OEL) 5.3. The Exadata Storage Server Software resident in the Exadata cell runs under OEL. OEL is accessible in a restricted mode to administer and manage the Exadata cell. CELLSRV (Cell Services) is the primary component of the Exadata software running in the cell and provides the majority of Exadata storage services. CELLSRV is multi-threaded software that communicates with the database instance on the database server, and serves blocks to databases based on the iDB protocol. It provides the advanced SQL offload capabilities, serves Oracle blocks when SQL offload processing is not possible, and implements the DBRM I/O resource management functionality to meter out I/O bandwidth to the various databases and consumer groups issuing I/O. Two other components of Oracle software running in the cell are the Management Server (MS) and Restart Server (RS). The MS is the primary interface to administer, manage and query the status of the Exadata cell. It works in cooperation with the Exadata cell command line interface

14 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

(CLI) and EM Exadata plug-in, and provides standalone Exadata cell management and configuration. For example, from the cell, CLI commands are issued to configure storage, query I/O statistics and restart the cell. Also supplied is a distributed CLI so commands can be sent to multiple cells to ease management across cells. Restart Server (RS) ensures the ongoing functioning of the Exadata software and services. It is used to update the Exadata software. It also ensures storage services are started and running, and services are restarted when required. Exadata Smart Scan Processing With traditional, non-iDB aware storage, all database intelligence resides in the database software on the server. To illustrate how SQL processing is performed in this architecture an example of a table scan is shown below.
Figure 5: Traditional Database I/O and SQL Processing Model

The client issues a SELECT statement with a predicate to filter and return only rows of interest. The database kernel maps this request to the file and extents containing the table being scanned. The database kernel issues the I/O to read the blocks. All the blocks of the table being queried are read into memory. Then SQL processing is done against the raw blocks searching for the rows that satisfy the predicate. Lastly the rows are returned to the client. As is often the case with the large queries, the predicate filters out most of the rows read. Yet all the blocks from the table need to be read, transferred across the storage network and copied into memory. Many more rows are read into memory than required to complete the requested SQL

15 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

operation. This generates a large number of data transfers which consume bandwidth and impact application throughput and response time. Integrating database functionality within the storage layer of the database stack allows queries, and other database operations, to be executed much more efficiently. Implementing database functionality as close to the hardware as possible, in the case of Exadata at the disk level, can dramatically speed database operations and increase system throughput. With Exadata storage, database operations are handled much more efficiently. Queries that perform table scans can be processed within Exadata with only the required subset of data returned to the database server. Row filtering, column filtering and some join processing (among other functions) are performed within the Exadata storage cells. When this takes place only the relevant and required data is returned to the database server. Figure 6 below illustrates how a table scan operates with Exadata storage.
Figure 6: Smart Scan Offload Processing

The client issues a SELECT statement with a predicate to filter and return only rows of interest. The database kernel determines that Exadata storage is available and constructs an iDB command representing the SQL command issued and sends it the Exadata storage. The CELLSRV component of the Exadata software scans the data blocks to identify those rows and columns that satisfy the SQL issued. Only the rows satisfying the predicate and the requested columns are read into memory. The database kernel consolidates the result sets from across the Exadata cells. Lastly, the rows are returned to the client. Smart scans are transparent to the application and no application or SQL changes are required. The SQL EXPLAIN PLAN shows when Exadata smart scan is used. Returned data is fully consistent and transactional and rigorously adheres to the Oracle Database consistent read functionality and behavior. If a cell dies during a smart scan, the uncompleted portions of the smart scan are transparently routed to another cell for completion. Smart scans properly handle

16 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

the complex internal mechanisms of the Oracle Database including: uncommitted data and locked rows, chained rows, compressed tables, national language processing, date arithmetic, regular expression searches, materialized views and partitioned tables. The Oracle Database and Exadata server cooperatively execute various SQL statements. Moving SQL processing off the database server frees server CPU cycles and eliminates a massive amount of bandwidth consumption which is then available to better service other requests. SQL operations run faster, and more of them can run concurrently because of less contention for the I/O bandwidth. We will now look at the various SQL operations that benefit from the use of Exadata.
Smart Scan Predicate Filtering

Exadata enables predicate filtering for table scans. Only the rows requested are returned to the database server rather than all rows in a table. For example, when the following SQL is issued only rows where the employees hire date is after the specified date are sent from Exadata to the database instance. SELECT * FROM employee_table WHERE hire_date > 1-Jan-2003; This ability to return only relevant rows to the server will greatly improve database performance. This performance enhancement also applies as queries become more complicated, so the same benefits also apply to complex queries, including those with subqueries.
Smart Scan Column Filtering

Exadata provides column filtering, also called column projection, for table scans. Only the columns requested are returned to the database server rather than all columns in a table. For example, when the following SQL is issued, only the employee_name and employee_number columns are returned from Exadata to the database kernel. SELECT employee_name, employee_number FROM employee_table; For tables with many columns, or columns containing LOBs (Large Objects), the I/O bandwidth saved can be very large. When used together, predicate and column filtering dramatically improves performance and reduces I/O bandwidth consumption. In addition, column filtering also applies to indexes, allowing for even faster query performance.
Smart Scan Join Processing

Exadata performs joins between large tables and small lookup tables, a very common scenario for data warehouses with star schemas. This is implemented using Bloom Filters, which are a very efficient probabilistic method to determine whether a row is a member of the desired result set.

17 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

New Smart Scan Offload I/O Optimizations and Functionality

With Oracle Database 11g Release 2, several new powerful Smart Scan and offload capabilities are provided with Exadata storage. These include: Storage Indexing technology, Smart Scan offload of new Hybrid Columnar Compressed Tables, Smart Scan offload of encrypted tablespaces and columns, and offload of data mining model scoring.
Storage Indexing

Storage Indexes are a very powerful capability provided in Exadata storage that helps avoid I/O operations. The Exadata Storage Server Software creates and maintains a Storage Index in Exadata memory. The Storage Index keeps track of minimum and maximum values of columns for tables stored on that cell. When a query specifies a WHERE clause, but before any I/O is done, the Exadata software examines the Storage Index to determine if rows with the specified column value exists in the cell by comparing the column value to the minimum and maximum values maintained in the Storage Index. If the column value is outside the minimum and maximum range, scan I/O for that query is avoided. Many SQL Operations will run dramatically faster because large numbers of I/O operations are automatically replaced by a few in-memory lookups. To minimize operational overhead, Storage Indexes are created and maintained transparently and automatically by the Exadata Storage Server Software.
Smart Scan of Hybrid Columnar Compressed Tables

Another new feature of Oracle Database 11g Release 2 is Hybrid Columnar Compressed Tables. These new tables offer a high degree of compression for data that is bulk loaded and queried. Smart Scan processing of Hybrid Columnar Compressed Tables is provided and column projection and filtering are performed within Exadata. In addition, the decompression of the data is offloaded to Exadata eliminating CPU overhead on the database servers. Given the typical ten-fold compression of Hybrid Columnar Compressed Tables, this effectively increases the I/O rate ten-fold compared to uncompressed data.
Smart Scan of Encrypted Tablespaces and Columns

New in Exadata is the Smart Scan offload processing of Encrypted Tablespaces (TSE) and Encrypted Columns (TDE). While the prior release of Exadata fully supported the use of TSE and TDE on Exadata it did not benefit from Exadata offload processing. This enhancement increases performance when accessing confidential data.
Offload of Data Mining Model Scoring

Another new function offloaded to Exadata is Data Mining model scoring. This makes the deployment of data warehouses on Exadata or Database Machine an even better and more performant data analysis platform. All data mining scoring functions (e.g., prediction_probability) are offloaded to Exadata for processing. This will not only speed warehouse analysis but reduce

18 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

database server CPU consumption and the I/O load between the database server and Exadata storage.
Other Exadata Smart Scan Processing

Two other database operations that are offloaded to Exadata are incremental database backups and tablespace creation. The speed and efficiency of incremental database backups has been significantly enhanced with Exadata. The granularity of change tracking in the database is much finer when Exadata storage is used. Changes are tracked at the individual Oracle block level with Exadata rather than at the level of a large group of blocks. This results in less I/O bandwidth being consumed for backups and faster running backups. With Exadata the create file operation is also executed much more efficiently. For example, when issuing a Create Tablespace command, instead of operating synchronously with each block of the new tablespace being formatted in server memory and written to storage, an iDB command is sent to Exadata instructing it to create the tablespace and format the blocks. Host memory usage is reduced and I/O associated with the creation and formatting of the tablespace blocks is offloaded. The I/O bandwidth saved with these operations means more bandwidth is available for other business critical work. I/O Resource Management With Exadata With traditional storage, creating a shared storage grid is hampered by the inability to prioritize the work of the various jobs and users consuming I/O bandwidth from the storage subsystem. The same occurs when multiple databases share the storage subsystem. The DBRM and I/O resource management capabilities of Exadata storage can prevent one class of work, or one database, from monopolizing disk resources and bandwidth and ensures user defined SLAs are met when using Exadata storage. The DBRM enables the coordination and prioritization of I/O bandwidth consumed between databases, and between different users and classes of work. By tightly integrating the database with the storage environment, Exadata is aware of what types of work and how much I/O bandwidth is consumed. Users can therefore have the Exadata system identify various types of workloads, assign priority to these workloads, and ensure the most critical workloads get priority. In data warehousing, or mixed workload environments, you may want to ensure different users and tasks within a database are allocated the correct relative amount of I/O resources. For example you may want to allocate 70% of I/O resources to interactive users on the system and 30% of I/O resources to batch reporting jobs. This is simple to enforce using the DBRM and I/O resource management capabilities of Exadata storage. An Exadata administrator can create a resource plan that specifies how I/O requests should be prioritized. This is accomplished by putting the different types of work into service groupings called Consumer Groups. Consumer groups can be defined by a number of attributes including

19 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

the username, client program name, function, or length of time the query has been running. Once these consumer groups are defined, the user can set a hierarchy of which consumer group gets precedence in I/O resources and how much of the I/O resource is given to each consumer group. This hierarchy determining I/O resource prioritization can be applied simultaneously to both intra-database operations (i.e. operations occurring within a database) and inter-database operations (i.e. operations occurring among various databases). When Exadata storage is shared between multiple databases you can also prioritize the I/O resources allocated to each database, preventing one database from monopolizing disk resources and bandwidth to ensure user defined SLAs are met. For example you may have two databases sharing Exadata storage as depicted below. Exadata CellExadata CellExadata CellDatabase B(RAC)Database A(Single-Instance) CellExadata CellExadata
CellExadata
Figure 7: Inter-Database I/O Resource Management with Exadata

Business objectives dictate that each of these databases has a relative value and importance to the organization. It is decided that database A should receive 33% of the total I/O resources available and that database B should receive 67% of the total I/O of resources. To ensure the different users and tasks within each database are allocated the correct relative amount of I/O resources, various consumer groups are defined.

20 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

 Two

consumer groups are defined for database A 60% of the I/O resources are reserved for interactive marketing activities 40% of the I/O resources are reserved for batch marketing activities Three consumer groups are defined for database B 60% of the I/O resources are reserved for interactive sales activities 30% of the I/O resources are reserved for batch sales activities 10% of the I/O resources are reserved for major account sales activities These consumer group allocations are relative to the total I/O resources allocated to each database. In essence, Exadata I/O Resource Manager has solved one of the challenges traditional storage technology does not address: creating a shared grid storage environment with the ability to balance and prioritize the work of multiple databases and users sharing the storage subsystem. Exadata I/O resource management ensures user defined SLAs are met for multiple databases sharing Exadata storage. This ensures that each database or user gets the correct share of disk bandwidth to meet business objectives. Accelerated Performance With Exadata Exadata storage provides unmatched performance improvements for typical data warehousing workloads. Full table scans will receive an arbitrarily large improvement due to smart scan filtering and the balanced hardware used for Exadata-based data warehouses. Exadata storage servers deliver a scale-out architecture such that as cells are added to the configuration, bandwidth increases. This, coupled with faster InfiniBand interconnect and the reduction of data transferred due to the offload processing, yields very large performance improvements. Often a ten-fold speed up in these operations is seen when using Exadata storage compared to storage products traditionally used with the Oracle Database but in many cases a 50-fold, or greater, speedup is achieved. Two examples of real world performance improvements follow.

21 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

28x 16x
01020304050607080CDR Full Table ScanWarehouse Inventory ReportCRM Service Order ReportCRM Customer Discount ReportHandset to Customer Mapping ReportIndex CreationTablespace Creation
Figure 8: Telecommunications Application Performance Improvement with Exadata of 10X to 72X
-5.010.015.020.025.030.035.040.045.050.0Recall QueryGift Card ActivationsSales and Customer CountsPrompt04 Clone for ACL auditDate to Date MovementComparison - 53 weeksMaterialized Views RebuildMerchandising Level 1 Detail byWeekSupply Chain Vendor - Year - ItemMovementMerchandising Level 1 Detail:Current - 52 weeksMerchandising Level 1 Detail:Period Ago

Figure 9: Retail Application Performance Improvement of 3X to 48X

Exadata Storage Virtualization Exadata provides a rich set of sophisticated and powerful storage management virtualization capabilities that leverage the strengths of the Oracle Database, the Exadata software, and Exadata hardware.
Exadata Storage Software

As discussed earlier, the Exadata cell is a server that runs the Oracle Enterprise Linux as well as the Oracle provided Exadata software. When first started, the cell boots up like any other computer into Exadata storage serving mode. The first two disk drives have a small Logical Unit Number (LUN) slice called the System Area, approximately 13 GB of size, reserved for the OEL

22 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

operating system, Exadata software, and configuration metadata. The System Area contains Oracle Database 11g Automatic Diagnostic Repository (ADR) data, and other metadata about the Exadata cell. The administrator does not have to manage the System Area LUN, as it is automatically created. Its contents are automatically mirrored across the physical disks to protect against drive failures, and to allow hot disk swapping. The remaining portion of these two disk drives is available for user data.
Exadata User Storage Virtualization

Automatic Storage Management (ASM) is used to manage the storage in the Exadata cell. ASM volume management, striping, and data protection services make it the optimum choice for volume management. ASM provides data protection against drive and cell failures, the best possible performance, and extremely flexible configuration and reconfiguration options. A Cell Disk is the virtual representation of the physical disk, minus the System Area LUN (if present), and is one of the key disk objects the administrator manages within an Exadata cell. A Cell Disk is represented by a single LUN, which is created and managed automatically by the Exadata software when the physical disk is discovered. Cell Disks can be further virtualized into one or more Grid Disks. Grid Disks are the disk entity assigned to ASM, as ASM disks, to manage on behalf of the database for user data. The simplest case is when a single Grid Disk takes up the entire Cell Disk. But it is also possible to partition a Cell Disk into multiple Grid Disk slices. Placing multiple Grid Disks on a Cell Disk allows the administrator to segregate the storage into pools with different performance or availability requirements. Grid Disk slices can be used to allocate hot, warm and cold regions of a Cell Disk, or to separate databases sharing Exadata disks. For example a Cell Disk could be partitioned such that one Grid Disk resides on the higher performing portion of the physical disk and is configured to be triple mirrored, while a second Grid Disk resides on the lower performing portion of the disk and is used for archive or backup data, without any mirroring. An Information Lifecycle Management (ILM) strategy could be implemented using Grid Disk functionality.
PhysicalDiskCellDiskGridDiskGridDiskGrid DiskGrid DiskPhysicalDiskPhysicalDiskPhysicalDiskCellDiskCellDiskGridDiskGridDiskGridDiskGrid DiskGridDiskGrid Disk
Figure 10: Grid Disk Virtualization

The following example illustrates the relationship of Cell Disks to Grid Disks in a more comprehensive Exadata storage grid.

23 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

Once the Cell Disks and Grid Disks are configured, ASM disk groups are defined across the Exadata configuration. Two ASM disk groups are defined; one across the hot grid disks, and a second across the cold grid disks. All of the hot grid disks are placed into one ASM disk group and all of the cold grid disks are placed in a separate disk group. When the data is loaded into the database, ASM will evenly distribute the data and I/O within disk groups. ASM mirroring can be activated for these disk groups to protect against disk failures for both, either, or neither of the disk groups. Mirroring can be turned on or off independently for each of the disk groups.
Exadata CellExadata CellHotHotHotHotHotHotColdColdColdColdColdCold CellHotHotHotHotHotHotHotHotHotHotHotHotColdColdColdColdColdColdColdColdColdColdColdCold

Hot ASMDisk GroupCold ASMDisk GroupExadata Hot GroupHot GroupCold Group

Figure 11: Example ASM Disk Groups and Mirroring

Lastly, to protect against the failure of an entire Exadata cell, ASM failure groups are defined. Failure groups ensure that mirrored ASM extents are placed on different Exadata cells.
Exadata CellExadata CellHotHotHotHotHotHotColdColdColdColdColdColdASMDisk CellHotHotHotHotHotHotHotHotHotHotHotHotColdColdColdColdColdColdColdColdColdColdColdColdASMDisk Figure 12: Example ASM Mirroring and Failure Groups

GroupASMFailure GroupASMFailure GroupExadata Group

24 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

With Exadata and ASM: Configuration of Cell Disks (LUN creation) is automated by Exadata software. Optionally, multiple Grid Disks can co-exist on the physical disks to tailor performance to the needs of the database application or construct an ILM strategy with Exadata. ASM automatically stripes the database data across Exadata disks and cells to ensure a balanced I/O load and optimum performance. ASM dynamic add and drop capability enables non-intrusive cell and disk allocation, deallocation, and reallocation. ASM mirroring, and the hot swap capability of the Exadata cell, provides transparent data protection and access across disk failures. ASM provides for double or triple mirroring to tailor the protection to the criticality of the data. ASM failure groups are automatically created with Exadata to provide transparent data protection and access across cell failures.
Migrating to Exadata Storage

Exadata storage can be used in addition to the storage arrays and products traditionally used to store the Oracle database. A single database can be partially stored on Exadata storage and partially on traditional storage devices. Tablespaces can reside on Exadata storage, non-Exadata storage, or a combination of the two, and is transparent to database operations and applications. But to benefit from the Smart Scan capability of Exadata storage, the entire tablespace must reside on Exadata storage. This co-residence and co-existence is a key feature to enable online migration to Exadata storage. An online non-disruptive migration to Exadata storage can be done for an existing database if the existing database is deployed on ASM and is using ASM redundancy. The steps to accomplish this are: 1. Add an Exadata grid disk to the existing ASM disk group. 2. ASM then automatically rebalances the data within the disk group moving a proportional amount of data to the newly added Exadata grid disk. 3. Then a non-Exadata disk is dropped from the ASM disk group. ASM would then rebalance or migrate the data from the non-Exadata disk to other disks in the disk group. 4. The above is repeated until the entire database has been migrated onto Exadata storage. In addition, migration can be done using Oracle Recovery Manager (RMAN) to backup from traditional storage and restore the data onto Exadata. Oracle Data Guard can also be used to facilitate a migration. This is done by first creating a standby database based on Exadata storage.

25 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine

The standby can be using Exadata storage and the production database can be on traditional storage. By executing a fast switchover, taking just seconds, you can transform the standby database into the production database. All these approaches provide a built-in safety net as you can undo the migration very gracefully if unforeseen issues arise.
Additional Data Protection With Exadata

Exadata has been designed to incorporate the same standard of high availability (HA) customers have come to expect from Oracle products. With Exadata, all database features and tools work just as they do with traditional non-Exadata storage. Users and database administrators will use familiar tools and be able to leverage their existing Oracle Database knowledge and procedures. With the Exadata architecture, all single points of failure are eliminated. Familiar features such as mirroring, fault isolation, and protection against drive and cell failure have been incorporated into Exadata to ensure continual availability and protection of data. Other features to ensure high availability within the Exadata server are described below.
Hardware Assisted Resilient Data (HARD) built into Exadata

Oracle's Hardware Assisted Resilient Data (HARD) Initiative is a comprehensive program designed to prevent data corruptions before they happen. Data corruptions are very rare, but when they happen, they can have a catastrophic effect on a database, and therefore a business. Exadata has enhanced HARD functionality embedded in it to provide even higher levels of protection and end-to-end data validation for your data. Exadata performs extensive validation of the data stored in it including checksums, block locations, magic numbers, head and tail checks, alignment errors, etc. Implementing these data validation algorithms within Exadata will prevent corrupted data from being written to permanent storage. Furthermore, these checks and protections are provided without the manual steps required when using HARD with conventional storage.
Data Guard

Oracle Data Guard is the software feature of Oracle Database that creates, maintains, and monitors one or more standby databases to protect your database from failures, disasters, errors, and corruptions. Data Guard works unmodified with Exadata and can be used for both production and standby databases. By using Active Data Guard with Exadata storage, queries and reports can be offloaded from the production database to an extremely fast standby database and ensure that critical work on the production database is not impacted while still providing disaster protection.
Flashback

Exadata leverages Oracle Flashback Technology to provide a set of features to view and restore data back in time. The Flashback feature works in Exadata the same as it would in a nonExadata

26 White Paper A Technical Overview of the Sun Oracle Exadata Storage Server and Database Machine 27

environment. The Flashback features offer the capability to query historical data, perform change analysis, and perform self-service repair to recover from logical corruptions while the database is online. In essence, with the built-in Oracle Flashback features, Exadata allows the user to have snapshot-like capabilities and restore a database to a time before an error occurred.
Recovery Manager (RMAN) and Oracle Secure Backup (OSB)

Exadata works with Oracle Recovery Manager (RMAN), a command-line and Enterprise Managerbased tool, to allow efficient Oracle database backup and recovery. All existing RMAN scripts work unchanged in the Exadata environment. RMAN is designed to work intimately with the server, providing block-level corruption detection during backup and restore. RMAN optimizes performance and space consumption during backup with file multiplexing and backup set compression, and integrates with Oracle Secure Backup (OSB) and third party media management products for tape backup.

CONCLUSION
Businesses today increasingly needs to leverage a unified database platform to enable the deployment and consolidation of all their applications onto one common infrastructure. Whether OLTP, DW or mixed workload a common infrastructure delivers the efficences and reusability the datacenter needs and provides the reality of cloud computing in-house. Building or using custom special purpose systems for different applications is wasteful and expensive. The need to process more data increases every day while corporations are also finding their IT budgets being squeezed. Examining the total cost of ownership (TCO) for IT software and hardware lead one to choose high performance common infrastructure for deployments of all applications. By incorporating Exadata and the Database Machine into the IT infrastructure, companies will: Accelerate database performance and be able to do much more in the same amount of time. Handle change and growth in scalable and incremental steps by consolidating deployments on to a common infrastructure. Deliver mission-critical data availability and protection. Exadata and the Database Machine provide this solution.

White Paper Title September 2009 Author: Ronald Weiss

Contributing Authors: Oracle Corporation World Headquarters 500 Oracle Parkway Redwood Shores, CA 94065 U.S.A. Worldwide Inquiries: Phone: +1.650.506.7000 Fax: +1.650.506.7200 oracle.com Copyright 2009, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any means, electronic or mechanical, for any purpose, without our prior written permission. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. 0109