Turning away from talking heads

lNlfRcHr93 24-29 April1993 ‘lbrning Away from Talking Heads: The Use of Video-as-Data in Neurosurgery Bonnie A. Nardi, Heinrich Schwarz, AlIan Kuchinsky, Robert Leichner Hewlett-Packard Laboratories 1501 Page Mill Road, Palo Alto, CA 94304 U.S.A Internee [email protected] Steve Whittaker Hewlett-Packard Stoke Gifford, Robert Department University of Neurosurgery of Pittsburgh, Pittsburgh, Studies of video as a support for collaborative work have provided little hard evidence of its utility for either task performance or fostering telepresence, i.e. the conveyance of a faceto-face like sociat presence for remotely located participants. To date, most research on the value of video has concentrated on “taLking heads” video in which the video images are of remote participants conferring or performing some task together. In contrast to talking heads video, we studied video-as-data in which video images of the workspace and work objects are the focus of interest, and convey criticat information about the work. The use of video-as-data is intended to enhance task performance, rather than to provide telepresence. We studied the use of video during neurosurgery within the operating room and at remote locations away from the operating room. The workspace shown in the video is the surgical field (brain or spine) that the surgeon is operating on. We discuss our findings on the use of live and recorded video, and suggest extensions to video-as-data including its integration with computerized timebased information sources to educate and co-ordinate complex actions among distributed workgroups. Keywords: Multimedia, video, collaborative computers and medicine. work, United Kingdom Sclabassi Abstract coordination, Laboratories BristolBS126QZ task Introduction PA 15213, U.S.A Talking heads video presents images of remote participants conferring or performing some task together. A large body of research has shown, however, that it does not enhance performance for a variety of tasks such as information transmission and collaborative problem solving [2, 4, 5, 15]. Alternatively, investigators have been hopeful that talking heads video will foster lelepresence, that is, give users a rich physical and psychological sense of the other people with whom they are remotely interacting via cues obtained from gaze, gesture, facial expressions, body language [5, 15].1 But this has been difficult to achieve [7] and thus far little in the way of concrete value has been demonstrated for telepresence video [2, 4,5, 15]. For example, Egido [4] reported on the history of videoconferencing and Picturephone, noting that videoconferencing remains “a smatl conglomeration of ‘niche’ markets.” Only about 75 companies in the U.S. have videoconferencing systems, and those include telephone and videoconferencing vendors. The Picturephone is a well-known failure, and while there are many possible reasons for this, Egido observed that trial use of Picturephone was met with “disturbing reports of phenomena such as users’ feelings of instant dislike toward parties they had never seen before, self-consciousness about ‘beiilg on TV’ . . . and resulting low acceptance” [4]. What about “desktop video” systems such as Portholes [3], Cruiser [6], CAVECAT [1 1], and Hydra [14]? Will they prove a more successful application of video than videoconferencing and Picturephone? Possibly they will, but to date there is little research to rely on for objective evatttation. Reports of the actual use of the desktop systems are often anecdotal and may involve both use and evaluation of the system by its developers and colleagues of the developers – a highly biased situation (e.g. [3, 1l]). A few systems have been tested in short trials on The integration of video into groupware systems seems a logicat next step in the quest for more effective computer support for collaborative work, Many such systems are currently under development, such as Portholes [3], Cruiser [6], ClearBoard [9], SharedView [10],CAVECAT[11], and Hydra [14]. (See [1] for an overview.) Videoconferencing systems in which participants gather in specially equipped conference rooms have been in existence for over 30 years [4]. The Picturephone from Bell Laboratories was introduced at the 1964 World’s Fair. use it in differentways.Thereis a needfor a word to precisely denote remote human presence, as opposed to virtual reahty or remote shared task space. It Permission is in the sense of human presence that we use telepresence. Used m tfus way, telepresence resonates with terms m common usage such as “social presence,” granted direct to copy provided commercial title of the that copying that and/or spacific e 1993 ACM all or part otherwise, of this ara not made the ACM and its date is by permission To copy fee tha copias advantage, publication Machinery. without or distributed copyright appear, matarial notica and notice of the Association is for and the is given for Computing or to republish, requires 1me tem “telepresence”does not yet have a firm definition, and PeoPk “stage presence” and simply “presence.” Webster’s defines the latter as “a quality of poise and effectiveness that enables a performer to achieve a close relationship with his audience,” which is very much like what we are trying to achieve via technology for remotely located participants. a fee permission. 327 0-89791 -575-51931000410327 . . ..$1 .50 lNTfRcH1’ 24-29 April1993 students or paid research subjects (e.g. [6, 14]), but the results have not been convincingly positive. None of the current research systems as they are now configured provides strong evidence that video is an important support for collaborative work. For example, in a four-week trial of Cruiser it was found that Cruiser did not supply value added beyond telephone or email; Fish et at. [6] reported that, “For the most part, people perceived and used [Cruiser] like a telephone or an electronic mail system ...” Fish et at. [6] suggest extensions to Cruiser which may improve its utility in the future, but it would be difficult to convince a disinterested observer of the value of video based on the implementations and related empiricaJ findings of the current tatking heads systems, as they are reported in the literature. While available evidence on the utility of video for collaborative work promotes skepticism more than enthusiasm, does this mean that video is an unimportant technology for collaborative work? We believe that the answer is no for two reasons. First, current systems that attempt to facilitate telepresence have many problems such as lack of support for eye gaze, or requiring a move to a special (expensively outfitted) room to utilize the technology. These problems will be overcome in time. Systems such as Hydra [14] which support natural eye gaze and leave a small footprint on the desktop seem promising for remote meetings and informal conversations among four or fewer remotely located people. ClearBoard [9] also atlows for gaze awareness, and provides an innovative shared drawing space for two users. Once the technical challenges of the telepresence systems are met, and the systems are more specialized to support specific tasks (just as ClearBomd supports intensive interaction between two people working closely together on a shared task), a more objective assessment of their worth will be possible. A large question will be the cost-effectiveness of telepresence video. Cost-effectiveness will have to be monitored as the technology and its price structure evolve over time. Second, we think that video is a valuable support for collaborative work because a different aspect of video usage – video-asdata – has been in use for many years in medical and industrial settings, and has become indispensable in many applications. Video-as-data stands in contrast to talking heads video: it is not used to support telepresence, but rather provides images of the workspace – the “data” – to convey critical information about the work. People can see the work objects, and how they are changing and being manipulated within the work context. These images are used by teams of workers to coordinate demanding, highly technicti tasks in real-time situations, and to support research and technicat training. For example, in power plants, live video of remote locations is used to monitor plant operations [16]. Video is used in telerobotics and remote surveillance [12]. The possibilities for extensions and enhancements to basic video capabilities are many. Milgram et al. [12] have a system that combines stereoscopic video and stereoscopic computer graphics so that users can point to, measure and annotate objects within the video. Tani et al. [16] have proposed “object- 328 oriented video” in which the real-world objects in the video become computer-based objects that can be manipulated so that users will be able to reference, overlay, highlight, and annotate them, as well as use the objects for control and information browsing. In Thni et at.’s prototype system for power plants, users can, for example, point to a burner on a boiler in the live video and bring up a document that explains how the ignition system of the boiler works. Users can get a more detailed video or related video of an object by pointing to the object, obviating the need to directly control remote cameras. Users can control remote devices through direct manipulation techniques such as clicking and dragging; for example, “pushing” a button on the video image engages a rest button on the remote device [16]. Such uses of video are quite distinctive from talking heads video, and open up a whole new realm of video-based applications. In this paper we report on an ethnographic study of the use of video-as-data in a medical setting, where live color video is used to coordinate team activity during neurosurgery, and both live and recorded video are used for training in neurosurgery. During the critical parts of neurosurgery, such as the removal of a brain tumor, the neurosurgeon looks through a stereoscopic microscope to view the brain as he2 works. A video camera co-mounted with the optics of the microscope displays a video image of what the surgeon sees on a monitor on a cable TV link. Thus everyone in the operating room (hereinafter abbreviated “OR,” the term used in the hospital) can see what the surgeon sees, though the video image is 2D and is a somewhat smaller view of the surgical field than what the surgeon sees. This technology has been in existence (though not universally available) for over twenty years, and is now an indispensable part of OR activity in many hospitals. A microphone mounted on the microscope provides audio capability for remote broadcast. In this paper we examine the ways in which the live video image of the brain or spine coordinates the demanding work of neurosurgery among the many neurosurgical team members. The technical complexity of neurosurgery is remarkable, and in the teaching hospital setting that we studied, the neurosurgical team includes an attending neurosurgeon, resident or fellow neurosurgeon(s), surgical technician, scrub nurse, circulating nurse, nurse-anesthetist, anesthesiologist, neurophysiologist, neurotechnician and sometimes an anesthesiology resident. We also examine how the live video in the OR promotes education: student nurses, medical students, residents, fellows, and visiting neurophysiologists and physicians are often present in the OR during an operation. In addition, we studied the use of remote video and audio facilities to enable remote collaboration during neurosurgery. Our main argument is that video-as-data is an important application of video technology in computer-based systems, and that we should not lose sight of its potential by over-focusing on talking heads video, and in particular, by placing too much ‘Our use of the prenoun “he” is strictly for convenience’ sake; any other constmction would make It very awkward to describe individual roles in the operating room. We alternate he and she as genencs. 3NeuroPhys1010gists usually have Ph.D.’s not M.D.’s., an M.D. degree. though some have lllTfRrJir93 24-29 April1993 emphasis on the lack of demonstrated utility of talking heads systems to date. We will describe how live video coordinates tasks within the OR, provides critical educational opportunities, and is used for remote monitoring by neurophysiologists. We discuss the extensions to both live and recorded video which will be useful for research, training, diagnostic, legat, and archivat purposes across abroad range of applications that potentially go far beyond the medical domain. In particular, the need to integrate and synchronize video images with other time-based data sources will be criticat. Methodology We conducted an ethnographic study comprised of observations in the OR, audio-taped, semi-structured interviews; informat interaction (such as going to lunch with informants and casual conversation in hallways, offices, etc.); and “shadowing.” The shadowing technique involved following around a single individwd for several days to track and record his or her activity in as much detail as possible. We used this technique with the neurophysiologists to study their use of the remote video. We had originally hoped to quantify this information in terms of times-per-task, but because of the complexities of hospital life we would have needed at least 3-6 months of shadowing to iron out anomaties and make statistically valid statements. The shadowing was nevertheless very informative as we learned a great cleat about the daily activities of neurophysiologists, and had many opportunities for informal conversation. The fieldwork team included six investigators – three psychologists, two anthropologists, and one computer scientist. A total of 14 person-weeks of fieldwork was conducted. One investigator was in the field for five weeks and one for three weeks (the anthropologists). During these weeks the observations and some shadowing were done, as well as many of the interviews. The other investigators contributed interviews and shadowing. Over 500 pages of transcripts resulted from interviews with about 35 informants. The operating At the beginning of an operation the patient is tranquilized, anesthetized and connected to a variety of monitors and drips. The attending anesthesiologist plans the general course of the 41n ethnographic studies, participants are called inform the investigator, subjectparticipants to an experiment “informants” in the sense rather than that the investigator (as ur psychology), is to in which case they are subjects. (The American Psychological Association has very recently recommended the use of the word “participant” in lieu of subject.) The essential notion is that the investigator the iuformarrt, observation. The beginning of the operation is also the time when the patient, after being anesthetized, is connected to the electrodes that will be used to monitor muscle and nerve activity. Neurophysiological monitoring is a relatively recent innovation in neurosurgery which reduces morbidity by constantly tracking and providing feedback on central nervous system activity to see that it is maintained within acceptable parameters. During neurosurgery, there is a high risk of damage due to the surgery itselfi for example, cutting, stretching or compressing a nerve, or cutting off the blood supply to parts of the brain. Neurophysiological monitoring helps prevent such untoward events. The neurophysiologist and neurotechnician apply the electrodes which provide “evoked potential” data. Throughout the course of the operation the patient is given electrical stimulation (electrical potential for activity is actuatly evoked by stimulation) to make sure that muscles and nerves are responding appropriately, and are not being damaged by the surgery. The neurophysiologist supervises the neurotechnician and all on-going cases, and is ultimately responsible for the interpretation of the neurophysiological data. The neurotechnician does the more routine monitoring. She sits in front of a computer screen viewing data from a networked computer system that processes the neurophysiological data, showing it as plotted line graphs: room It is necessary to provide some background on work flow and work roles in the operating room to be able to make sense of the discussion of the use of the video in the next section of the paper. Our observations were conducted in the operating room during a series of brain and spine surgeries. In some cases we observed complete surgeries, and in others we spent a period of hours in the OR (neurosurgeries can last from about 5 to 24 hours). that they are to anesthesia to be used for the operation, and is usually present during the “prep” period. The attendhg anesthesiologist works with the nurse-anesthetist and/or resident anesthesiologist to administer the anesthesia and insert the appropriate intravenous lines for blood, and a catheter for urine. After the initial set-up, the attending anesthesiologist generally leaves the OR to attend to another operation or to take care of other tasks. The resident and/or nurse-anesthetist then monitor thepatient’s basic physiological functions: heart rate, blood gases, blood pressure, breathing, urine concentration, etc. The attending anesthesiologist returns to the OR when necessary. He can be reached by phone via his pager, and he makes “check-in” visits to see how things are going. is ignorant of the understandings possessed by but wishes to learn as much as possible through interaction The neurophysiological data can atso be viewed in other locations outside the OR because the graphs can be displayed on remote nodes on the networked computer system. The system allows neurophysiologiststo monitor operations remotely from many nodes: their offices, other operating rooms, or conference rooms where the system is installed. The system can display all of the neurophysiological data for any operating room where the system is connected. A neurophysiologist, when on catl, thus usuatly spends apart of the day in the various ORS and a part of the day in his office, monitoring the evoked potentiah via the computer displays. Neurophysiologists typically monitor as many as six operations concurrently (they have a back up person assigned to help in case of overload). When not in the OR, they communicate with the neurotechnician in the OR via telephone. After the prepping of the patient, the neurophysiologist, like the attending anesthesiologist, may leave the OR to go to other operations that he is monitoring. Or he may go back to his office and 5For more info~ation on rhk S@SM, L_kW con~ct the ‘imt autior. 329 IN1-RCHI’ 24-29 April1993 to monitor the operation remotely. If during the course of the operation, the neurotechnician suspects a problem, she reports it to the surgeon, and she may also telephone the neurophysiologist if he is not in the OR at the time. The neurophysiologist then returns to the OR to evaluate the data and possibly communicate with other members of the surgical team such as the attending neurosurgeon or anesthesiologist. After the prep, the patient is “opened – that is, the incision made – by the resident or neurosurgical fellow. The resident or fellow then continues to cut and drill until he is down to the point in the brain or spine where the most delicate surgery is required; for example, the brain tissue that must be “picked through” to reach a tumor, aneurysm or blood vessel compressing a nerve. At this point the attending neurosurgeon arrives in the OR to take over. Often the procedures used by the neurosurgeon are “micro-procedures,” i.e. those requiring the use of the microscope. The resident or fellow neurosurgeon watches the operation through a second (2D) lens on the microscope, while the attending neurosurgeon views the surgical field through the main stereoscopic lens. When the microscope is being used, the video is on as well, so those in the OR can watch the surgery on the TV monitor. The audio portion of the system is on as soon as the microscope is turned on, typically at the beginning of the operation, long before the microscope is needed. A recent innovation is that some neurophysiologists have a cable TV link in their offices so that they can access video and audio broadcast from the OR. This enables them to see the microscope video and hear much of what is said and done in the OR. The operation may or may not be recorded, according to the discretion of the attending neurosurgeon. Throughout the surgery the scrub nurse hands the surgeon the instruments and supplies that he requests. The circulating nurse makes sure that the scrub nurse has everything she needs; the circulating nurse is abridge between the sterile operating area and non-sterile areas of the OR. At the hospital we studied, scrub nurses and circulating nurses are cross-trained, so each can do the other’s job. When the attending neurosurgeon has finished his work, the patient is “closed” – that is, the incision is repaired – by the resident or fellow. The patient is revived from the anesthesia in the OR, and asked to wiggle his toes and say something. He is then wheeled to recovery. In addition to the visual displays used to monitor physiology and neurophysiology function, some of the equipment provides (intentionally or not) auditory cues to the progress of the operation or the patient’s state. For example, the suction device tells everyone in the OR how much blood is being suctioned; a lot of blood might indicate a problem. The audio broadcast to remote locations clearly transmits the sounds of much of the OR equipment. the offices of neurophysiologists. Task coordination Video is used in the neurosurgical setting to coordinate and educate. In this section we examine how the live video supports task coordination and education within the OR. We also report preliminary findings on the use of remote video and audio in 330 room “Coordination” is distinct from communication and from collaboration. By coordination we mean the smooth enactment of actions requiring more than one person, or requiring information from another’s actions. Collaboration, by contrast, is at a higher level of abstraction than an action, and involves shared goals and the enactment of a web of actions that allows goals to be fulfilled. Communication refers to the transmission of information. We do not have space hereto fully explore these concepts, but they are delimited inactivity theory (see [13] for an overview of activity theory and a bibliography). The live video is used in the OR to coordinate activity during the most critical part of the surgery when the neurosurgeon is working deep in the brain or spine on very small structures that he sees only by looking through the microscope. In a sense, even though OR personnel are co-located, the video provides “remote” access; the surgical field is invisible, without an intervening technology, to atl but the surgeons. The video is used by the scrub nurse, anesthesiologist, nurse-anesthetist, circulating nurse, neurophysiologist, and neurotechnician. The most important function of the live video in the OR is to allow the scrub nurse to anticipate which instruments and supplies the surgeon will need. As one scrub nurse said, the video is “the only indication we have of what’s going on in the head.” The circulating nurse positions the monitor so that the scrub nurse has an unimpeded view. During the criticat parts of the surgery, events move very quickly, and the surgeon must be able to work steadily and without interruption. He changes instruments as often as every few seconds, and he needs to work in tight coordination with the scrub nurse who is selecting an instrument from over one hundred instruments arrayed on the sterile table near the operating table. The scrub nurse may atso need to hand the surgeon one of hundreds of types of supplies (sutures, sponges, teflon pads, etc.) brought to her by the circulating nurse. The work of a neurosurgicat operation is extremely detailed and fast-paced, and the better idea the scrub nurse has of the surgeon’s needs, the more smoothly the operation proceeds. Even with the video, the surgeon calls out the instrument or supply he needs next, but the ability of the nurse to anticipate what the neurosurgeon will want is considered very important by OR personnel, One neurosurgeon used a sports metaphor to explain how the video supports neurosurgeon-scrub nurse coordination: Neurosurgeon . . . an operation is like team work, [for example], ice hockey – the center brings the puck around, to the appropriate it. Findings in the operating position, and the forward and the puck is coming . . . Surgeon and scrub nurse . . .– it’s mutual team work good scrub nurse looks at the video and knows what’s — instrument in and OUL instrument giving and taking. work, [like] sports activily. way to do so [i.e. harmonious work. goes in and he hits . . . So a coming next It’s all team So if you don’t have the video, there’s no coordinate activity quickly]. . . . So it’s uniform, INTERCHI’93 24-29 April1993 As she watches the video, the scrub nurse is tracking the course of the operation and looking for unusual events to which she must respond with the correct instrument or supply. For example, she may know that the surgeon is approaching a time in the operation when a clip will be needed. Or she may see the surgeon nick some tissue, in which case a cautery device will likely be called for to repair the nick. The scrub nurse’s effective use of the video depends on her own knowledge and understanding of what she is seeing; the presence of the video image is not a guarantee that she will be able to anticipate the surgeon’s needs and respond quickly. There is an interaction between her level of expertise and understanding, and the presence of the video in the environment. As one neurosurgeon explained: Neurosurgeon: . . . Some scrub nurses are excellent when they look at the video, they know what’s next and they are very good. But other scrub nursesare not at that level yet so [1] have b telt her what I need and even if she’s looking, [she] is not at level ye~ so it is more time consuming. Because the scrub nurse is listening to the surgeon, selecting instruments and supplies and handing them to the surgeon, her use of the video involves very quick glances at the monitor to see what is happening. All the more reason she must instantly understand what she is seeing. In contrast to the scrub nurse’s quick glances at the monitor, the others in the OR who watch the video may watch it intently for long stretches of time. Their use of the video helps them keep track of the progress of the surgery, but generally they do not rely on the video for split-second reactions as does the scrub nurse. Anesthesiologists, nurse-anesthetists, circulating nurses, and neurotechnicians watch the video in part to remain attentive to the surgery, to maintain interest and concentration at times when they may have very little to do. For example Interviewer: . . . What does [the video] tell you about what you have to do? Anesthesiologist In the neurosurgical procedure, the microscopic part acturdty is quite long and boring usualty for us because once we get to that part of i~ . . . the patient usually is very stable. . . . It’s nice requirements vary depending on the surgeon’s actions: Nurse-anesthetist: through . . . The anesthetic bone are different they are working from requirements the anesthetic [for] requirements dritling when inside the head, where there are not pain fibers. In this example, the actions of anesthesia personnel must be coordinated with those of the surgeon, and depend critically on what he is doing at a given moment in the surgical field. The video provides this information to anesthesia personnel. Neurophysiologists and neurotechnicians interpret the graphs they watch on the computer display in concert with events shown on the video. One neurotechnician explained that they can “decipher the responses better” when they know what the surgeon is doing; for example, when a retractor is placed, a delayed response may result which should not necessarily be attributed to nerve damage, but may have been caused by the retractor itself. Interpreting the neurophysiological data is difficult because its meaning can be affected by signal noise, the type and amount of anesthesia used, surgical events, and random variation. The video provides an important source of information for making better inferences in a highly interpretive task. Again, the use of the video allows tasks to be coordinated appropriately by supplying neurophysiologists and nettrotechnicians with critical information about the neurosurgeon’s actions. Education in the operating room At a teaching hospital, education is of critical importance. Anesthesiology and neurosurgical residents and fellows, student nurses, and neurophysiologistsand neurotechnicians-intraining observe and/or take part in operations as a criticat part of their education. While the neurosurgical resident or fellow uses the second 2D lens on the microscope to view the operation when the attending neurosurgeon is operating, others in the room watch the video. We observed students, residents and fellows training at the hospital watching the video, and also visiting students, residents and fellows from other hospitals. On several occasions they entered the OR, parked themselves in front of the video monitor and watched for the duration of the micro-procedures (which may goon for several hours). to see where they are, how much longer are they going to be. Is he [the surgeon] stitl dissecting or is he [finishing] up? I don’t have to ask the surgeon that. Because many of the operations performed at the hospital we studied are in the neurosurgical vanguard, the OR accommodates neurosurgeons, anesthesiologists and neurophysiologists, many of them eminent in their own specialties, from other institutions around the world who come to learn about the new procedures. One of their main activities in the OR is to watch the video. Many anesthesiologists, nurse-anesthetists, circulating nurses, and neurotechnicians commented that watching the video was “interesting” and that it was much better than just sitting there with nothing to do. The video thus alleviates boredom and provides a focat point of attention that helps maintain shared awareness of the work being done by the surgeon. The use of remote video and audio to monitor This is critical because events can change very quickly during an operation. Suddenly what is seen on the video monitor can dictate that someone take action or that a new interpretation of an event applies. OR personnel look for a variety of events such as the placement of a retractor or clip, where the surgeon is drilling, if there is bleeding, how close to a tumor the surgeon is. For example, a nurse-anesthetist explained that anesthetic Much of the time the physical presence of the attending neurosurgeon or neurophysiologist is not needed in the OR, and an extension to the networked computer system being tested is the use of multimedia (audio and video) to enhance the remote monitoring of operations (now supported by the plotted line graphs). Only a few such multimedia links are functional in the current system (for neurophysiologists), so our findings operations 331 INTERCH 24-29 April1993 about their use are preliminary at this time. The idea behind the remote audio and video is that neurophysiologists and neurosurgeons can make better use of their time in their offices monitoring multiple operations, or working on other tasks such as research, taking calls from patients, or attending meetings. They will also be able to remotely monitor operations in other ORS from the OR they happen to be in at a given time. Using the remote monitoring, neurophysiologists and neurosurgeons should be able to simultaneously monitor a larger number of operations, spreading scarce expertise over a greater area and making more efficient use of their time. What kinds of information do the video and audio provide to neurophysiologists? The remote video provides the kind of information provided by the video within the OR, as described above. Neurophysiologists can interpret the graphic data more easily with the addition of the video information, anticipate what will happen next in the surgery, and generally keep track of what is going on in the surgery at a particular moment. For example, one neurophysiologist explained: are saying in the operating exactly what they are doing. I could problem, [One time I was watching see that the surgeon was in trouble, the remote video and] that he was having like there was a big bleed, for instance. a Then I would go to potentially With the case, the audio, you know . . . Because they talk to each other about So you can really anticipate what might happen. This is an example of “beyond being there” [8] whereat least one aspect of being remote is preferable to being co-located. The audio also allows the remotely located neurophysiologist to hear what the neurotechnician is telling the surgeon, and how the surgeon responds to that information. The neurophysiologist can see for himself what the neurotechnician sees on the graphs, but the response of the neurosurgeon is very important – does he reply that he’s not doing anything that might be causing a problem, that he doesn’t understand the response, that he will change an action he is taking, or say nothing? Or, the neurophysiologist may not agree with what he hears the neurotechnician tell the surgeon: In that case, I heard the technician say something to the surgeon that I didn’t agree with . . . [He] said there was a change in the response. There wasn’t. Interviewer: . . . So what did you do, you called? Neurophysiologist: the operating room. doing. the steps they are going to take. Neurophysiologist: Neurophysiologist: room so . . . if you don’t know you kind of guess what they’re Calted right away.. . Told the surgeon there was no change. The remote audio provides additional information from two sources: (1) what is being said in the OR, (2) the overall atmosphere in the OR. Together, the audio and video provide a much more complete picture of OR activity than the plotted line graphs alone Neurophysiologist: When you look at the computer data by itself [from a remote location], it seems to be one dimensional. add the rest of it [audio and video], When you you get a very rich picture of what’s going on [in the OR]. The audio information is very important in the remote situation. The neurosurgeon often explains what he is doing or discusses his anticipated actions with the other neurosurgeon(s). Anesthesia personnel discuss the patient’s physiological function. This information is useful to the neurophysiologist the progress of and plans for the operation revealed by the comments of the neurosurgeons, and physiological data revealed by the comments of the anesthesia team, help him to interpret the neurophysiological data he is looking at, and to anticipate what will happen next. Here the audio information directly influences the neurophysiologist’s behavio~ he telephones the OR to provide a different interpretation of the neurophysiologicat data than that given by the neurotechnician. Other audio information provides an overall impression of the atmosphere in the OR. The surgeon’s voice may sound nervous, or there may be a dead silence indicating a tense moment in the operation. As one neurophysiologist said: Neurophysiologist going welt or not. how much activity Neurophysiologist: In fact the audio is better over the network than it is in the operating room because you can ‘t hear what the surgeons 332 is very close to the surgeon so for whether he feels like the case M . . . you can hear it from his voice. You can hear is in the room, whether the people are scrambling. Again, this information influences the neurophysiologist’s behavior, in this case his decision as to whether to go to the OR from his office Neurophysiologist: on. I probably In many cases, the neurophysiologist actually has better access to what is being said when he is in a remote location than when he is in the OR. Within the OR, it is sometimes difficult to hear clearly some of what is being said because of the noise of equipment and random conversations. There may even be a radio playing. Listening to the audio in a remote location, by contrast, one gets a clear transmission of what the neurosurgeons and the anesthesia personnel are saying, as they are positioned closest to the microscope (on which the microphone is mounted). One neurophysiologist explained: The microphone I can realty get a good feeling Welt, if people are agitating, there’s a lot going would have a much lower threshold room because I’m alerted then that there’s something room, and that’s maybe an opportunity for going to the going on in the for me to make a significant contribution, Our preliminary findings indicate that the information from the remote multimedia sources concerning the course of the surgery, the surgeon’s observed and anticipated actions, the content of key comments made by personnel such as the neurotechnician, and the overall atmosphere in the OR allow the remotely located neurophysiologist to perform his job more efficiently and effectively, He can better plan and coordinate his visits to the OR because he has richer information with which to decide when he needs to visit a particular OR, or whether ll!TfRcHr93 he wants to place a telephone call. If he does need to go to the OR, he arrives with better information about the status of the operation, If the neurophysiologist is communicating with the neurotechnician via the telephone, again he has abetter idea of what is happening in the operation if he has the video and audio data in addition to the graphs of neurophysiological function. The use of remote multimedia facilities does not eliminate the need for neurophysiologists to be physically present in the OR for at least part of the operation. Rather, it re-distributes their allocation of time across ORS, their offices and other locations in the hospital such as conference rooms. The use of multimedia appears to give neurophysiologists more flexibility to move about the hospital on an as-needed basis, rather than to stay tied exclusively to a smatl number of ORS. Discussion We were struck by the extent to which the use of video-as-data in the operating room and in remote locations serves a number of highly varied functions. The overatl goal of the video is to provide a window into the unseeable world of the surgical field, but the uses to which the surgical information is put, and the way the information is gathered, vary greatly depending on the specific tasks associated with the differing roles within the operating room or remote offices. As we have seen, the video image can coordinate fast-paced exchanges of instruments and supplies between neurosurgeon and scrub nurse; it can serve as a means of maintaining attention and focus over long stretches of time during which some OR personnel are relatively inactive it helps OR personnel to choose the correct action or interpretation depending on the event portrayed in the video; it educates a variety of medical personnel; and the video plus audio may atlow neurophysiologists and neurosurgeons to decide when their presence is needed in the OR from a remote location. The use of video in neurosurge~ shows the utility of one well-chosen artifact, and the many activities it permits and coordinates. It demonstrates the utility of video-as-data, in contrast to telepresence video. Looking more closely, we see that the use of video in the neurosurgical context is quite different than some of our standard notions of what it means to support collaborative work; rather than facilitating direct interpersonal communication (as many CSC W systems are intended to do), in many crucial instances, the video permits individuals to work independently, actuatly obviating or reducing the need for interpersonal communication. The video supplies enough information so that the need for interpersonal communication is reduced or eliminated, and individuals can figure out what they need to know based on the video itself, circumventing the need to tatk to or gesture at someone. Thus we may find that in settings where video is data, the provision of visuat information at key moments provides a different channel of communication than that which would be provided through verbal, gestural or written communication, Rather than facilitating collaboration through interpersonal interaction, the video itself informs OR personnel of the collaboration – in the sense of tasks that need to be performed to advance the work – that is needed. Collaboration and coordination are enabled as each member of the neurosurgery team interprets the visual information, and proceeds to do his 24-29 April1993 or her job based upon an interpretation of that information. The video data, plus individual knowledge and understanding, combine to produce an interpretation that leads to the desired collaboration, with little or no interpersonal interaction. In other cases, the content of the video image becomes the basis for discussion and interaction, another aspect of its use as a sh~ed workspace. For example, we observed a nurseanesthetist in the OR watching the video with a student nurseanesthetist and describing to her the progress of the operation. Indeed, we ourselves profited from explanations in which the video was a key point of reference as OR personnel educated us about many aspects of neurosurgery. Visitors, residents, fellows, etc. atso discuss what is being shown on the video monitor. Our findings about the importance of the on-going use of videoas-data in a real work setting with demanding requirements (as opposed to brief experiments or testing within research labs) should encourage us to pursue our understanding of how video-as-data can be extended and used in other work settings. Within medicine, video is used in many kinds of surgery includlng orthopedic surgery and generat surgery that employs micro-procedures. Non-medicat applications of video-as-data could include monitoring and diagnostic tasks in complex mechanical or electrical systems such as the Space Station, power plants, or automated factories; and training for many aspects of using, designing, monitoring and repairing such systems. Wang, a computer and instrument manufacturer, uses video to train people in the use of their equipment [4], and it is easy to imagine many training applications for video-as-data. Real estate agents might show properties remotely, and attorneys are making increasing use of video data in courtroom presentations. There is a large number of potential applications for video-as-data. Video-as-data may change our sense of what it means to be “remote” or “co-located.” In the OR, even though people are co-located, the surgical field is remote, because it is invisible to anyone not looking directly through the microscope. The surgical field is accessible only through the video to most OR personnel. Thus it is not necessarily the location of people that is important in the video-as-data situation, but rather of the workspace. Aural information in the OR, on the other hand, is not remote, so we have a situation in which the aural and visual do not share the same valence on the dimension of co-location. One can imagine other such situations; for example the repair of a delicate piece of machine~ with many small parts might be a situation in which a view of the workspace is remote, while aural information is not. For neurosurgery, we atso have a “beyond being there” situation in which the aurat information may be richer in the remote location, via broadcast audio, as we described for the remote audio used by the neurophysiologists. Future Directions In this paper, we have concentrated on the use of live video, but many important extensions to video-as-data lie in recorded video. In the m~lcal application that we studied, recorded video is 333 INFRCHI’ 24-29 April1993 atready used for classroom teaching and to review events in past operations. Integration of video with other computerized time-based data is the next step. Uniform storage, access, and presentation methods for data will be needed. Means of visualizing complex relationships between datasets of varying types will provided added value. Informants in the study who have research interests underscored the need for future tools that will allow for the synchronization of video with other data sources, in particular the instrument data relevant to a particular specialty. Anesthesiologists, for example, would want to see the video synchronized with the physiological data they monitor such as blood pressure, blood gases, heart rate, pulse, temperature. They might want to do this during an operation, with video and instrument data they had just recorded. Neurophysiologists want to see video synchronized with the many measures of nerve and muscle response that they monitor. In general, users want to be able to “scroll through” a video/instrument dataset, Ending a particular video or instrttment event or time, so that they can view all related data for the event or time. Integration of video-as-data with other data sources would atso be useful in many applications for training, legal and archival purposes. Potential users of such video technology will want to be able to edit, browse, search, annotate, overlay, highlight, and display video data, Whether such extensions to video-as-data will be cost-effective remains to be seen; video is an expensive technology. Cttrrent applications of video-as-data such as the present study, the power plant applications described by Tani et al. [16], and Kuzuoka’s work [10] suggest that there is tremendous potential for video to enhance collaborative work. Future research should go beyond talking heads to recognize the vahte of video-as-data, and should be concerned with offering good video utility in a cost-effective manner. Acknowledgments: We would like to thank Erik Geelhoed and Bob Simon for their help with data collection. Steve Gate’s previous work on the project was of great value. Within HP, for support and encouragement, we acknowledge Dan Fishman, Mark Halloran, Nancy Kendzierski, John Limb, Jim Miller, Ian Page, Frank Prince, Dick Schulze and Bill Sharpe. Robin Jeffries, Jim Miller, Vicki O’Day, Andreas Paepcke and Bill Wllliarns gave insightful comments on earlier drafts. At the hospital we thank the secretaries who helped us to track down and schedule interviews with peripatetic medical personnel. Our many informants in the hospitaJ generously atlowed us to follow them around, ask endless questions, and watch them for hours on end at their jobs. For their good cheer and thoughtful answers to our questions, we offer grateful thanks. [3] Dourish, P., and Bly, S. Portholes: Supporting awareness in a distributed work group. Proceedings CHI’ 92. (Monterey, 3-7 May, 1992), 541-547. [4] Egido, C. Teleconferencing as a technology to support cooperative work. In Intellectual Teamwork. J. Galegher, R. Kraut, C. Egido, Eds. Lawrence Erlbaum, Hillsdale, NJ, 1990,351-371. [5] Gale, S. Adding audio and video to an office environment. Studies in Computer Supported Cooperative Work Theory, Practice and Design. J. Bowers and S. Benford, Eds. North-Holland: Amsterdam, 1991,49-62. 6] Fish, R., Kraut, R., Root, R., and Rice, R. Evaluating video as technology for informal communication. Proceedings CHI’92. (Monterey, 3-’7 May, 1992), 37-48. 7] Heath, C., and Luff, P. Disembodied conduce Communication through video in a multi-media office environment. Proceedings CHI’ 91. (New Orleans, 27 April-2 May, 1991), 99-103. [8] Hollan, J., and Stometta, S. Bevond being there. Proceedings CHI’92. (Monterey, 3-7-May, 1992), 119-125. [9] Ishii, H., and Kobayashi, M. ClearBoard: A seamless medium for shared drawing and conversation with eye contact. Proceedings CHI’ 92. (Monterey, 3-7 May, 1992), 525-532. [10] Kuzuoka, H. Spatial workspace collaboration: A SharedView video support system for a remote collaboration capability. Proceedings CHI’ 92. (Monterey, 3-7 May, 1992), 533-540. [11] Mantei, M., Backer, R., Sellen, A. Buxton, W., Milligan, T., and Wellman, B. Experiences in the use of a m~ia space. Proceedings CHI’ 91. (New Orleans, 27 April-2 May, 1991), 203-215. 12] Milgram, P., Drastic, D., and Grodski, J. A virtual stereoscopic pointer for a real three dimensional video world. Proceedings Interact’ 90. (Cambridge, UK, 27-31 August, 1990), 695-700. 13] Nardi, B. Studying contexc A comparison of activity theory, situated action models, and distributed cognition. In Proceedings, St. Petersburg Human Computer Interaction Work~hop. (St. Petersburg, Russia. 4-8 August, 1992), 352-359. [14] Sellen, A. Speech patterns in video-mediated conversations. Proceedings CHI’92. (Monterey, 3-7 May, 1992), 49-59. [15] Short, J., Williams, E., and Christie, B. The Social Psychology of Telecommunications. John Wiley & Sons: London, 1976. [16] Tani, M., Yamaashi, K., Tanikoshi, K. Futakawa, M., and Tanifuji, S. Object-oriented video: Interaction with realworld objects through live video. Proceedings CHI’ 92. (Monterey, 3-7 May, 1992), 593-598, References [1] Buxton, W. Telepresence Integrating shared task and shared person spaces. Proceedings of Graphics Interface’92 (Vancouver, 11-15 May, 1992), 123-129. [2] Chapanis, A. Interactive human communication. tific American, 232 (1975), 36-42. 334 Scien-