Augmented Reality Inside VR-games

HERLANDER ALVES ELIAS JOÃO DESSAIN SARAIVA University of Beira Interior, Covilhã [email protected] [email protected] Augmented Reality Inside VR-games vol 9 / Dic.2013 209-228 pp Recibido: 01-11-2013 - revisado 15-11-2013 - aceptado: 25-11-2013 © Copyright 2012: Servicio de Publicaciones de la Universidad de Murcia. Murcia (España) ISSN edición impresa: 1889-979X. ISSN edición web (htp://revistas.um.es/api): 1989-8452 Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva AUGMENTED REALITY INSIDE VIDEOGAMES ABSTRACT The new media environment brings digital data to real life through augmented reality technologies giving users a new vision, where the binary world interacts directly with the real one. A new age is upon us with challenges, doubts and mainly new ideas that start to take shape. One of these is whether or not McLuhan’s theory of a medium integraing another is sill a working paradigm should we tesify that augmented reality elements appear inside videogames. This aricle is composed by an analysis of the third stage of the web and how it expands with augmented reality, the origins and concepts of interface design and how entertainment can be a sandbox of new ideas ending with a chapter on the electric environment of hybrid media, augmentaion categories and Rancière’s theories over the emancipated spectator. This issue culminates in an object analysis regarding augmented reality inside VR-games, where a videogame analysis is carried out, being followed by a review on the top four best devices. The whole data gathered provides us the framework to analyse McLuhan’s theory regarding how virtual technologies’ future might be when it comes to virtual outputs inside other virtual media. Keywords Virtual reality, augmented, videogame, sci-i, interface, hardware, spectator, web 3. RESUMEN El nuevo entorno de los medios de comunicación aporta datos digitales a la vida real mediante tecnologías de realidad aumentada, lo que proporciona a los usuarios una nueva visión en la que el mundo binario interactúa directamente con el real. Una nueva era está llegando: con desaíos, con dudas y principalmente con ideas nuevas que empiezan a tomar forma. Una de ellas es si la teoría de McLuhan de un medio integrando a otro sigue siendo un paradigma viable, como vemos en los elementos de realidad aumentada que aparecen dentro de los videojuegos. Este arículo está compuesto de un análisis de la tercera etapa de la web y esta se expande con la realidad aumentada; los orígenes y los conceptos de diseño de interfaz y como el entretenimiento puede ser un entorno de pruebas para las nuevas ideas. Sigue un capítulo sobre el ambiente eléctrico de los medios híbridos, las categorías de aumentación y las teorías de Rancière sobre el espectador emancipado. El tema culmina en un análisis de la realidad aumentada en el interior de VR-juegos, donde se hace un análisis de videojuegos y es seguido por un comentario sobre los cuatro mejores disposiivos. Todos los datos recogidos nos proporcionaron el marco de trabajo para analizar la teoría de McLuhan sobre cómo podría ser el futuro de las tecnologías virtuales cuando se trata de producciones virtuales dentro de otros medios virtuales. Palabras Clave Realidad virtual, aumentada, videojuego, ciencia icción, interface, hardware, espectador, web 3. 211 INTRODUCTION & PROBLEM In this day and age, AR (Augmented Reality), a new media form, is not only acknowledged, but trendy as well. It has captured the imaginaion of game developers, arists, technicians and users who are thus rendered enthusiasts. The issue in this paper is how McLuhan’s idea of a medium integraing another medium is sill a working paradigm, as AR is also appearing inside VR-games, which stand for videogames as the mainstream version of VR (Virtual Reality), for instance. Due to the evoluion of social media, Web 3.0 concepts are becoming more real, and following the same pursuit of the audiences as previous media systems. Right now there is more hardware, smart gear, next-generaion game consoles, iAccessories and iDevices simulaing connecivity and cloud compuing than before. This is deinitely a new age. Images and data are more intrusive than ever, and if there is a context truly expanding that is that of digital world, in its audiovisual, sensorial, wireless, gridlike, data-structure form. In addiion to the informaion and entertainment dimension, now “things” are also becoming cyberneically tagged. Cyberspace is in every “thing”. We should recall that we did not go to live into the digital world. The digital world is overlaying the real world with its tracking systems, labels, locks and bits. We are way beyond geography. This is the age of a “coninuous geography in digital media” (Elias, 2013, p.156), as objects, hardware and sotware are produced to belong to an ecosystem of machines. Hence it is possible to speak of a “technopolia”, because we are being indulged by technology. As a result, the ofthe-grid world, or the “non-mediated world” became a lost place. It seems like in the near future, AR will solve some user issues in data-interfacing, in case we speak of cloud compuing, driving, itness, entertainment, guidance, or military applicaions; not to menion arts and engineering. AR helps expanding data-interfacing with users. It is a new space, yet augmented. Informaion appears on top of reality. As new fashion items like Google Glass and iWatches are being released, the AR concept meets real proposiions. We believe interface design is intertwined with science icion, not only working on compuing devices or sotware design. A hybrid space is the by-product of the present trend, and smartphones triggered the event since the launch of Blackberry Curve and Apple iPhones. Unlike previous igures such as speakers, TV viewers, podcasters, VJs, gamers, photographers and directors, the people who use now the new mobile media, entertainment plaforms and gaming accessories, are emancipaing themselves from the then-already engaging TV culture of the 1960s. They are way ahead of video culture formats. In theory, AR brings new forms of interfacing with data, because the opical relaion is diferent, but in reality it is sill either sotware or hardware-dependent. The interesing part is that in VR-games, where narraive, moion-controller gamepads and high-deiniion graphics collide, there are AR models being displayed, as a suggesion for interface design to be released for the average consumer products. This is why this aricle is enitled “Augmented Reality Inside VRgames”, because due to the ambiguous, and at the same ime progressive nature of images, there is sill something at play between the visible and the invisible, as virtualiies generate augmentaions. Augmented Reality Inside Videogames 1 Herlander Alves Elias & João Dessain Saraiva WEB 3 The Web in its third stage works mostly in the form of voice interfaces, sensors in every object of everyday life, geomedia, and also, in AR. So, in Web 3.0, the current stage of the Web, augmented realiies are a possibility. Much as VR was a brand, a system of ideas, a technological trend and a marriage between hardware and sotware in the 1990s, Augmented Reality performs something in the same highlight since the irst decade of the 21st century. Yet, now it is geing more obvious how easily, bold and fun it could be for any user or enthusiast. Before Augmented Reality the trend was for all things to migrate at least in the form of data onto Virtual Reality’s environments. On today’s augmented environments, digital overlays of informaion are showing up in reality; not otherwise. Daniel Boorsin speaks of a “graphic revoluion”, as photography and other iconographies established a massive intrusion of imagery on the symbolic environment: photos, lyers, drawings, and ads were just part of the equaion (in Postman, 1994, p.66). But right now we’re beyond that move. Mass media were upgraded. The digital world expanded, and instead of being just another tool, it became an environment. So reality is being overlaid with digital tags, labels from the age of the Web of things. In the near future it will be weird to ind oline things much like inding oline people; anything of the grid, ulimately. According to this Web 3.0 context, one has to noice, as Gibson did, how a certain “postgeographical feeling” (in Neale, 2000, secion 1:5 Television) emerges from today’s always online and constant global linkage. Back in 1994, Postman sees it coming up too: “Informaion became a way of trash” (Postman, p.67, our translaion). For the author, we are witnessing “Technopolia”, which, in his regard, is a mental state, a stage of culture whenever culture is indulged by technology itself (Idem, Ibidem, p.69). Following this, all we have is a world hard to unplug from digital language, inclusive interfaces, augmented realiies, and a constant mediaion of media is embracing us in a technopoliical model. As a result, digital environments are overgrowing so much that they manage to expand inside themselves. As the mediated world grows more and more, “The ‘Non-Mediated World’ has become a lost country. And I think that, in some very real way, it’s a country that we cannot ind our way back to. The mediated world is now THE WORLD. We are that which perceives a mediated realit” (Gibson in Neale, 2000, secion 1:7 Mediated World, our translaion). These are concerns we share and this worldview is by no means not true. How did it all begin? Well, Web 1 introduced us the entanglement of Connected Pages, through the author’s hyperlinks and thus we got used to the Search Layer. Ater such moment, Web 2 came up. It’s also mistaken as being the phase of Social Networks. Nothing increases more in this stage as control does. Friends react to brands, for instance, and nobody is locked out of the Social Layer. But Web 3 is something new. It’s a Web of Data, where interacion with brands occurs because Semanic Agents are looking for all forms of contents and data, iltering issues and avoiding ambiguiies (Huybrechts, 2011, p.2). Not only are we speaking of the future, but we are speaking of the present as well. It looks futurisic, either way. We were told that the “future” would be when everything would follow in a “web-based environment” (Dennis, DeFleur, 2010, p.ix), and this moment includes the “next big thing” and a yet unimagined media landscape, (Idem, Ibidem). The quesion is that we are 213 already living this present-future, so science icion does not need to anicipate any form of interface design or “imaginary narraive space” (Gibson in Neale, 2000, secção 2.7: Wriing), since we are already «interfacing» with AR. And it keeps expanding, through interacion design. 2 INTERFACE DESIGN “Futurity” (Gibson, 2012, p.8) used to be the word for describing things yet to come. AR’s interface elements, “input” and “output” discussions, are not solely for technicians or arists now. The growing ields of informaion design and interacion design have shared, so far, the taste for “futurism”. One thing is sure, regardless of what the future holds for us, it sure saves a place for design and creaivity, as challenges will be new and bigger. We are told by Shedrof & Noessel that “Design and science icion do much the same thing. Sci-i uses characters in stories to describe a possible future. Similarly, the design process uses personas in scenarios to describe a possible interface. They’re both icion” (2012, p.vi). We may disagree with the argument, but not with the relaion. Design and science icion are connected, especially in organising new forms of presening informaion today. The quesion is that we are already living in “that” ime. It is true that interfaces, especially the ones featured in ilm and videogames, are “sci-i interfaces”. Regular people don’t have access to them. However, as Shedrof & Noessel may noice, these “Sci-i interfaces help create a reality that is coherent, and makes sense for audiences. In this way, audiences are a class of users” (2012, p.310). A new perspecive is at stake here. Since interfaces, especially those of AR, are part of the human/digital world interacion, and are a complex system of elements, we have to deine them as being “(...) all parts of a thing that enable its use” (Idem, Ibidem, 3). What we did not account for was that AR interfaces managed to appear, step by step, in the average user world (outside the screens) and also in the average gamer and ilm worlds (inside the screens). Somehow, like the exising tension between the spectator and the spectacle, we now have a new problem which solves the issue Rancière stands out as the separaion of the “stage” and the “auditorium” as something to be transcended (2009, p.15). Surprisingly, the new smartphones and smart gear trends are set to break any boundaries let on the hybrid sphere of mobile media. And so new grounds appear for AR to expand. AR is not popping out of nowhere. Computers were plugged to telephones. Then Internet went global. Social media invested on over-connecion. Smartphones outselling notebooks, laptops, convenional phones and desktop computers, made them to be the device of choice, the default device for digital media access. Recent product releases, such as Samsung’s Galaxy Gear smartwatch are able to implement another trend of the 1990s: “wearable compuing”. Today we are calling it “smart clothes” and “smart gear”. Should we take into account the Google Report on “The New Muli-Screen World: Understanding Cross-Plaform Consumer Behavior,” and we will understand how smartphones, tablets and the muli-gear user of today are interacing with the digital world much like as a fully-equipped soldier links up to the military grid. As it is menioned in the research, “Smartphones are the backbone of our daily media interacions. They have the highest number of user interacions per day and serve as the most common staring point for aciviies across muliple screens” (Google, 2012, p.3). Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva We have no doubts regarding this informaion. The issue is that the study highlights how “90% of all media interacions are all screen-based” (Idem, Ibidem, 8), meaning we are sill screendependent. Or perhaps we should restate it as being muli-screen-dependent. Another keyinding is that “Context drives device choice. Today, consumers own muliple devices and move seamlessly between them throughout the day” (Idem, Ibidem, p.11). And this is a typical Web 3 efect. Users are surrounded by hardware and all-day-long connecions to informaion. One of the suggesions of sci-i interfaces, as seen in videogames and ilms, is that users will access their data through a single smart device triggering links to huge amounts of data clusters. So far, “Computers keep us producive and informed” (Idem, Ibidem, p.12), which means that the desktop computer for the average user is but a form of keeping up to date. The computer is a tool for research, producion and task-oriented duies. When it comes to smartphones, rules change. Not only they keep us connected, they occupy 38% of daily interacions. It is esimated that 40% of interacions occur out of home and as well as 60% of them at home. So smartphones are perfect for 54% communicaion and 33% of entertainment. All this data comes from the same study conducted by Google (Ibidem, .13). Unil this point, nothing menions sci-i interfaces or AR. The informaion provided by the study describes how tablets are also working out a slice of the market’s atenion span. So, “tablets are for 9% of all media interacions” (Idem, Ibidem, p.14). It is an improvement. Yes, it is. By connecing the dots between what science icion proposes, what studies as Google’s show us, and considering the release of Samsung Galaxy Gear, and even the advent of the Google Glass project (amidst other hardware we have tested) we may have a glimpse of what AR will look like in the future. For sure, AR will look at least as powerful and aestheic, and trendy and fashionable, as we have seen in videogames such as Tom Clancy’s Ghost Recon Future Soldier (2012), Syndicate (2012), Heavy Rain (Quanic Dream, 2010), and more recently, Tom Clancy’s The Division (2013). These games present paradigms to thought of future human-computer interacions, since they introduce us to a real world modiied by expanding AR interfaces, smart arifacts, grids. Consequently, the users are mostly emancipated subjects. 3 THE ELECTRIC ENVIRONMENT AND THE EMANCIPATED SPECTATOR We have Rancière (2009, p.49) speaking of a “new topography of the possible”. That is exactly the aim of AR. It’s about new space, or “dataspace” over real space. But nothing of this would exist deprived of electricity, of course. Back in 1960s, Marshal McLuhan pioneered several concepts and paradigms as for media would be understood as being extensions from ourselves. One of McLuhan’s most evocaive deiniions is that of an “electric environment”. He could not tell how computer, electronics and the Internet would turn out to be, so he explains how media could work by applying a system of diferent deiniions (higher and lower, cool or hot). In return, he describes all things electric as being the beginning of something big. He lacked the words, but by no means the spirit, in trying to anicipate the world to come; our world — the Mediated world (for Gibson). McLuhan’s vision considered the automobile, human-machine interacion, clothing, ciies, and so much more we now idenify as parts of the backbone of the Internet of things (origins of cloud compuing, digital media and AR). 215 Something that in the 1960s was most present was the dichotomy of spectator vs mass media. Thus, one may understand in which waters Mcluhan’s “forecasts” were seing sail, triggering new paths to unfold. Sill today, authors such as Jacques Rancière, enjoy discussing our relaionship with the “spectacle”, the model of viewers vs paricipants. He airms that we need a theater where the opical relaion—implied in the word ‘theatron’ — would be revoluionary. For him, “What is required is a theatre without spectators, where those in atendance learn from as opposed to being seduced by images; where they become acive paricipants as opposed to passive voyeurs” (2009, p.4). Though the problem is not just about our opical relaion, it is about power and such opical relaion. Rancière coninues, saying we don›t need: “(...) a theatre played out in front of empty seats, but the theatre where the passive opical relaionship (...) is subjected to a diferent relaionship — that implied by another word, one which refers to what is produced on the stage: ‘drama’. Drama means acio” (2009, p.3). As much as Rancière›s «ight» seems to be global issues, philosophy, and let-wing poliical discourse on emancipaion, what takes us to his statement is the evocaive power of his words. Here the author calls for acion, emancipaion on the global scene, declining passive state in the name of acion, paricipaion and collecive performance. We also noice how he labels media [in our point of view] as «prisons of images». What is in quesion here is the concept of theater, as context for «seeing», viewing, not as a passive element, but as a node of acion. For Rancière, theater is not a spectacle, it is vision and acion, emancipated existence, expanded existence. In order to best understand Rancière›s framework for an «emancipated spectator», one has to remember what McLuhan said about the so-called «electric environment» in 1964. So far, both authors, in diferent generaions, and epochs, criicize media. They criicize, they see it, but media remain developing persuasion and control. What did not exist before was the ability of viewers, spectators or audiences, to control images in these “prisons of images”. At least for now, there is a diference. Mass media are not reality. And whatever futurisic media landscape is unfolding now, it is something “augmented”, as a mcluhanesque media extension of man. In other words, despite criicism, in present ime, more than ever, we are the “emancipated spectator”, because videogames, social media and AR have helped to change the world. And each of these items has produced a fast revoluion in the recent past. When it comes to speaking of this environment, “electric” in McLuhan’s lexic, we know, as he did, that “(...) in operaional and pracical fact, the medium is the message” (1994, p.7). This means that regardless of the media type at stake, the medium always imprints its code on the content, therefore formaing it as itself. It is as if the medium would funcion as a virus. That is why the real issue is always the medium rather than the content. McLuhan observed how some media are emerging from other media. It is a never-ending loop. Like he says, “The efect of the medium is made strong and intense just because it is given another medium as ‘content’” (1994, p.18). Due to this, we have the situaion we are examining in this aricle: Augmented Reality inside VR-Games [Technically it is not true VR, but videogames instead, though there is a virtual engine performing similarly]. The videogames we were able to Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva test for the accomplishment of this aricle are portraying Augmented Reality interfacing scenes in their plots and mîse-en- scène. Again, we have a mcluhanesque loop here. Videogames are inside the TV set [now a “screen”], which is turned on by the electric environment. And inside these videogames the story has AR. Ulimately it is all VR-gaming, but since Google Glass and other sci-i interfaces are not yet available for the masses, we may observe AR inside VR-games as a model of things to come, like Hollywood works: introducing models. Audiences are users, so they keep absorbing, should we follow previous studies’ ideas. Let’s summarize the key-issues, we have paricipaion, massive gaming, electronic entertainment, audiences of users, digital media and, of course, “vivid interacion” [beyond the one menioned once by McLuhan (1994, p.348)]. The only reason why this AR worlds are moving it is because programmers built the code and the graphics with “reality engines” (Manovich, 2001, p.183). Besides, more and more people are connecing online, and during more ime, and with muliple screens, as Google reports depict. If there is any thing remaining the same, since science icion literature, media criique, interface design, and technology producion are renewing, it is “space”. The more evolved the medium, the more “real” the “space” they present it is. It happened with TV, videogames, VR, and now with AR. Let’s not forget that “for the irst ime, space becomes a media type” (Manovich, 2001, p.251). This is deinitely noiceable in the objects we have analysed. Like Manovich’s terms were not enough, we have AR truly being introduced in the market slowly with new iDevices and Smart Gear. But is it in fact AR? “Augmented Reality (AR) is technology that augments a user’s percepion of the real world, with useful, addiional informaion” (Shedrof, Noessel, 2012, p.158); meaning that augmentaion, expansion, extension and connecion are more than fancy words. And let’s not forget that user’s don’t manage to just step into digital worlds. Actually what we have is digital worlds stepping out of screens. McLuhan used to say that “We return to the inclusive form of the icon” (1994, p.12). The power of the image, the driving force of the theatron is sill unsurpassable. Only screens muliplied. When it comes to the informaion AR displays, the augmented mode means “the informaion should ‘overlay’ reality” (Shedrof, Noessel, 2012, p.159). Unlike Virtual Reality, AR does not atempt to replace the user’s perceived world by hardware and sotware combinaion of means. If VR had limits, AR deies them. Data is overlaid, objects acquire tags, real world speaks digital world. There is a blending procedure. So far, there are “four categories of augmentaion: sensor display, locaion awareness, context awareness, and goal awareness” (Shedrof, Noessel, 2012, p.159). Beyond these four categories, a translucid vision is achieved. 4 OBJECT ANALYSIS 4.1 Sotware: AR Inside VR-Games More recently, despite the AR interfaces not being focused on typography and easy reading data, we found the contrary. Even in videogames like Heavy Rain, there is data labelled as infographics while the main character, a detecive oicer, uses ARI (Augmented Reality Interface) (see Fig.9) to transform an ugly and opaque police oice into a 360 augmented world (see Fig.1). Here we have sensor display, locaion awareness and context awareness (see Fig.2). 217 Figure 1. Heavy Rain - The Augmented Oice Figure 2. Heavy Rain - The Interface In Tom Clancy’s Ghost Recon Future Soldier, it happens that the future soldiers are capable of looking at the sky and see arrow signs and typographic labels (see Fig.3) displaying instrucions regarding mission goals, bearings and geographics. Here we idenify sensor display, locaion awareness, and goal awareness. While the irst case recreates AR, the second builds a sort of a “screen-sphere” overlaying everyone on the scenario. In Tom Clancy’s The Division (2013), the player controls a military agent in New York which manages an AR smartwatch to display data, maps and staisics about the mission on course. Like a real world AR device, the player overlays geodata and detailed info on the real world (stunning Hi-deiniion graphics). Again we noice sensor display, locaion awareness, and goal awareness. Perhaps here we should recall these arguments of Rancière: “The image is not the duplicate of a thing. It is a complex set of relaions between the visible and the invisible (...)” (2009, p.93). Game worlds are also not duplicaing, they are becoming originals. And these relaions between visible and invisible are already on them, at last as a theme. In the 1980s, the movie Tron was way ahead of its ime. The book novel version of Tron (the moion picture) introduces characters depicing “a recilineal landscape, incandescent, lit by electricity, [which] was casing its rectangles and edges onto the sky” (Lisberger & Daley, 1982, p.21, Translaion is ours). When we look to Tom Clancy’s The Division’s model of AR (see Fig.4) we are seeing the world of Tron being assimilated by our real world, and in its turn, the real world being assimilated by the game world. Except that it is not looking like a game anymore. It is a parallel world for many of us, augmented ones. Maybe what we are witnessing here is a return to the huning age, and at the same ime a return to the “pedestrian scale”, as McLuhan would say (1994, p.2). We would rather label the phenomenon as a quesion of “subjecive cyberspace” (Elias, 2009), since the common factor Figure 3. Tom Clancy’s Ghost Recon Future Soldier Figure 4. Tom Clancy’s The Division Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva Figure 5. Tom Clancy’s The Division we ind in games, especially shooters, is the immersive, irst person point of view, enabling the genre First Person Shooter. Games as Tom Clancy’s The Division (see Fig.5) are displaying a third person character, as we may see the most of the persona, but aiming and focusing require a subjecive point of view. What we idenify too here is what Lipovetsky & Serroy enitle as “a screen-like vision of the world” (2007, p.29, Translaion is ours). On another videogame, a reboot of an older version, Syndicate (2012), the cuing-edge graphics are groundbreaking, much of the story is introduced to us, user-players, in the early moments: 2032. The world populaion swells into 15 billion. 5% percent are chiped and linked to their preferred corporaion. The rest is digitally locked out, out of sight and out of mind. Industrial espionage reaches unparallel heights. Covert agents are mobilized to protect and serve their interests. You are one of the chosen and few, an agent. The science icion context of Syndicate is its higher value, altogether with the proposed AR interfaces. The world dominated by skyscraper corporaions, military technology and cyberneics, places the user-player in the acion context of Syndicate. Mission-oriented goals lead the character along impressive scenarios in which deadly force is presented, and answered by us, in a bold and high-tech manner. By using the dartchip interface (see Fig.6) to train in simulaions, as well as to manipulate enemy soldiers and objects, the user controls an agent of a syndicate, whose goal is to win by “whatever it takes” moto. The AR interface is relying on wireframe wires (see Fig.7), glowing characters and any object on the screen having a geotag we Figure 6 & 7. Syndicate 219 can see thanks to the dartchip. Whenever the player needs to train a task or a chip-implanted skill, he enables the dartspace. As good example of the concepts we are working here, sensor display, locaion awareness, context awareness, and goal awareness are crucial. By playing the world of Syndicate, we understand how Rancière makes sense, because he states that «faced with the hyper-theatre that wants to transform representaion into presence and passivity into acivity” (2009, p.22), we may have condiions to rise an emancipated spectator (see Fig.8). We could aggree no more. But by going back to McLuhan’s forecasing thoughts we make key-indings: “what emerges is a total ield of inclusive awareness” (1994, p.104). This is exactly what happens in Heavy Rain (see Fig.9) and Tom Clancy’s Ghost Recon Future Soldier, since both AR models presented are inclusive, surrounding, visionary, and look like sci-i interfaces. We are not playing these games, we are respecively invesigaing a crime scene or being deployed in a theatre of war. And also we are interfacing with AR. Rancière›s idea of a «hyper-theatre» is quite interesing. Even when we examine ilms such as Ironman (Jon Favreau, 2008) we see it in the scenes. The hero in the movie lies a smart military exo-suit, which happens to have all data interfaced inside the helmet’s AR holographic projecions. Game and ilm-world ideas are mistaken for one another here. In Robocop (Paul Verhoeven, 1987), for example, as an injured police oicer is the subject of a cyborg medical procedure, he happens to face reality aterwards through “video-subjecivity”. Programming Commands, Interface Graphics and Target Icons are overlaid on his vision. Another major example of AR is Minority Report (Steven Spielberg, 2002), a detecive story focused on an innocent cop (John Anderton), being blamed for a crime he did not commit at the ime oicers start the crackdown. The key moment is when John is commuing and AR ads are targeing him directly, speaking to him with tailored messages; and also when he interfaces with geodata related to any object acquired on the grid. Many are the dialogues established between game worlds and ilm worlds. It is geing harder and harder to mark a clear boundary between both of them. Science icion and avant-garde arts seem to be the common ground here. The noion of “spectacle” is deinitely being recalled too. One thing Rancière asks is: “what in fact is the essence of spectacle for Guy Debord? It is exteriority. The spectacle is the reign of vision - and vision is exteriority (...)” (2009, p.6). We are glad to accept that the noion of “exteriority” is a key-inding in AR situaions. Data overlaid onto the real world and “subjecive cyberspace” demands outdoors. Yet the reign of vision is Figure 8. Syndicate Figure 9. Heavy Rain Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva also consolidated by the way games, AR, Smart TV, videoculture, VR-Games, seem to have a common ancestor: TV screen. Both the (TV) sets and the staion networks, and more recently electronic gaming, have helped to improve the engagement with the (new) screen. Let’s see what McLuhan said in the 1960s: “TV will not work as background. It engages you. You have to be with it” (1994, 312) (see Fig.10). This is why we evolved over the 20th century from a “onescreen” situaion to an “all-screen” basis (Lipovetsky & Serroy, 2007, p.10). In case we take a closer look to some of the science icion ilms released between 1986 and 2009, we may achieve interesing conclusions. First, in movies such as Aliens (James Cameron, 1986), AR interfaces are already displayed on spaceship cockpits and military goggles and helmets. When Terminator 2 - Judgment Day (James Cameron, 1991) was launched, a key-feature of terminators was the red vision model of overlaying data analysis on top of video-recordings streamed straight from the real world, as we see in Robocop, but more sophisicated. In Predator (John McTiernan, 1987), the original, huge, and alien monster released ater his shipwreck in the jungle of tropical Earth is able to see, record and replay his version of coloured data of the real world, as he seeks for prey and tokens and trophies. It seems that these imageries, in ilm, like the videogame footage, in sotware, need something more to convince the “audiences of users”: hardware to expand these concepts and to emancipate spectators all over the world. Perhaps in the future, people will be capable of interacing with all informaion on a city space, for example, like the main character of the video game Watchdogs (2014) does, by hacking the Chicago city’s CTOS (City Operaing System), and so connecing to surveillance, databanks and other urban, e-linked “things” (see Fig.11). Figure 10. Tom Clancy’s Ghost Recon Future Soldier Figure 11. Watchdogs 221 4.2 Hardware: Expanding Sotware is the main stage for AR and VR-Gaming experiments but hardware is also becoming a key-player alongside the digital plaform. Within the scope of this aricle, we examine the hardware at our disposal, by following the four categories of augmentaion ideniied by Shedrof & Noessel (2012, p.159), being “sensor display”, “locaion awareness”, “context awareness”, and “goal awareness” the concepts to work with. Once we get on track with the authors’ work we can acknowledge the devices’ limitaions. Shedrof & Noessel (2012, p.158), for example, believe that AR is ied to reality, and so one cannot manipulate it in terms of scale, posiion or state. Also, if we consider Manovich’s analysis of the equipment, it makes sense how in the new media image the user acively goes into it (2001, p.183). Some AR elements displayed in VRgames were so capivaing that it seemed as if words, images, narraives and performances could actually change the world, as Rancière says so (2009, p.23) in other contexts. A ine example lies in the ilm Star Trek: Into Darkness (JJ Abrams, 2013) on the space-jump sequence where AR tags and circuit lines guide the hero’s stunts (see Fig.12 and 13) aiding him in achieving his goal. As previously denoted, videogames and science icion can be a birthplace for incoming technologies and since this aricle focuses on Augmented Reality inside VR-Games we have to talk about an HMD (Head Mounted Displays) called Oculus Rit. It made its debut at Kickstarter website to raise funds for the irst prototypes (and currently has John Carmack, creator of the Doom 3 videogame, working on the team as Chief Technology Oicer). The concept behind Oculus Rit is best explained in the words of its founder, Palmer Lucky: “Our visual system is by far the most powerful sense we have; it overrides prety much everything (...). I was looking for something that made it actually feel like you were inside the game...” (Edge, 2013b, p.75). Figure 12 & 13. Star Trek Into Darkness Oculus Rit features a ield of vision of 110 degrees, a stereoscopic view of the world and traces our head movements. All of this is in a light-weight device, which the gamer tends to forget he’s using. The HMD can be connected to a computer, giving the user the ability to enjoy available demos or to create his own experiments; thus simulaing the four categories of augmentaion: sensor display, locaion, context and goal awareness. This makes the Oculus Rit (see Fig.14) the perfect staging area for future Augmented Reality soluions and interacive interfaces. Despite this great features the product has its issues. Moion sickness efect has been reported by some users; it happens whenever we are standing sill while sensing movement. Also, as menioned by aricle’s authors from Edge magazine, “It’s prety clear that a keyboard and a mouse are not going to be the most natural interface for VR” (2013b, p.79). Therefore we need an immersive Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva Figure 14. Oculus Rit device that has the ability to interact with the Virtual Reality world using the users’ body movements providing them a “telepresence feeling”, as referred by Steuer (pp.73-93, 2003), bridging the gaps between both real and digital worlds. Directly related to the digital world are the smartphones as backbones of our daily media interacions for some ime now. Hence it was only a mater of ime unil a handier AR-based soluion began to be developed. Google’s eyeglasses (Glass) allow users to see digital data overlaying real world by the form of visual items, enabling them to perceive reality throughout a translucent vision. The previous denoted local and goal awareness concepts are a default in Glass. Those are put to good use with Google Now and Maps apps, which respecively display informaion according to the user’s locaion and provide direcions for a speciic desinaion. Under the umbrella of the new media image, Google’s eyeglasses (see Fig.15) suggest a “realisic representaion of a diferent reality”, should we recall the words of Manovich (2001, p.183). The device interacion is possible via voice commands or touchpad, which is becoming a standard. However, it might atract some undesired atenion. For instance, there’s the issue of the user’s privacy being exploited; as soon as something is digiized, it can barely be considered as being “private” again. As Ranciere once stated “words and images, stories and performances, can change something of the world we live in” (op.cit.). Google’s Glass are a way to capture media data, share it, and therefore change the world of those who use it and interact with the device on a daily basis. Figure 15. Google Glass 223 We have analysed a HMD and an AR device. Although vision can be considered as one of the main senses, a true virtual experience cannot be achieved with no moion interacion. The direct control of the digital world by the user is taking place with immersive technologies that now have a key-role in the virtual world environment. The Leap moion (see Fig.16) tracker is a small sensor device that detects the user’s ten ingers, allowing the manipulaion of virtual elements with a 2 millimetres precision. We might remember this concept from the movie Minority Report (Steven Spielberg, 2002) where the physical interacion with a futurisic interface is achieved by using one’s hand movement. Despite this awesome idea, rumour has it that while the sequences were being shot the actor who performed the interacion had to take several breaks since the constant physical interacion was exhausing. With a concept similar to Leap but within a diferent context, Microsot’s Kinect, is launched with the new Xbox One game console bundle. It captures the user’s body movements, facial expressions, heartbeat and even the direcion of the gaze. It embodies Manovich’s ideas and meets Minority Report’s interface style. Surely users have never been so into media as they are nowadays. These hardware reviews made us realize that Oculus Rit is a landmark in the videogame industry. However, the devices’ acceptance by the public is sill uncertain. Nevertheless, we believe it might change the way we perceive and interact with videogames. Also in the ield of viewing experience, there’s Glass, which is a new paradigm in the hybrid sphere of mobile media, boosing the interacion between distant people and giving geo-awareness a substanial gain. Despite its main feature, it is important to remark that Glass might end the muliple screen-dependency by placing digital elements in real space directly in front of the user’s eyes. Ater these two immersive devices’ analysis we came to the conclusion that Leap is a device that can be used during short periods of ime, being crucial that it comes embodied on the laptop to avoid further wires, setups and accessories. This already happens on a laptop computer model. Kinect’s (see Fig.17) goal is to physically leave gamers ired, whereas Leap allows a punctual immersive interacion, thus bringing us to conclude that Leap might be regarded as the handy soluion and Kinect as the full-body try-out experience. Today we can pracically go into the media environment and enrich the world around us with binary elements. What used to be relected only in VR-Games and VR experiments is taking shape and enriching our daily interacions. Although the cinema or videogames’ “class of users” can be eluded by a futurisic feature, the daily class needs pracical soluions. What could work on the screen may not be useful in real life, therefore the success of any of the reviewed devices will be decided by ime and the general criicism of the global consumer. Figure 16. Leap Figure 17. Microsot Kinect Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva METHODOLOGY In order to best organize research, our materials were split into theory, sotware and hardware. By choosing theories, models and concepts from interface design, science icion, new media theory, communicaion sciences and images? studies, we were able to view the object of analysis (Augmented Reality) in a new way. The challenge was to examine all VR-Game items where AR is featured and also today’s hardware for interface with AR, ranging from game consoles, moiondetecing accessories, and Web video. Beyond tradiional print media, electronic media, online documents, games and sci-i cinema, documentary ilms and magazines were also selected for informaion analysis. The current essay is the product of a closer inspecion on games, ilms, hardware and digital media theory, so the highlight is precisely the connecion between object analysis and theory. RESULTS When it comes to sotware, the videogames, apps and programs we have tested are enabling us to conclude how progressive, suggesive and construcive entertainment is nowadays. The imagery displayed in VR-Games makes us think about the future of interfaces, when it comes to human-computer interacion. Like we have foreseen, having AR inside VR-Games looks like a media loop, and it is so, yet there is something more to say about it. An interesing feature presented in both videogames and science icion ilms, for instance, is the way AR will be displayed as a sphere of data around the user. It is a suggesion, of course, but it makes us wonder about the “screen-sphere” menioned by Lipovetsky & Serroy. We are also to idenify as a result that hardware is by no means ahead of sotware, being merely Google Glass (see Fig.14) the only project to seem to be truly futurisic. As far as AR is concerned, we may say that AR shows up in VR-games like a new stage of image: that is to say, as a complex set of relaions between the visible and the invisible. Following a McLuhan’s idea, we see when we test AR in videogames, that it is as if some media were emerging from other media. And again an idea from Manovich coninues the ground paved by McLuhan, since he believes space became a media type. When players are interacing with these new games they are introduced to the inclusive form of the icon, as data is overlaid on their visual ield, defying VR’s old boundaries and TV culture’s heritage of the screen. As we have said before, real world speaks digital world in both AR sotware and hardware. CONCLUSION Under the framework of the “expanded contexts,” we have chosen to work on the subject of having AR inside VR. It is true that only now, due to the latest releases in hardware, smart gear, AR, next-generaion videogame consoles, and smartphones, we may have the exact condiions for VR, and the global market for it, as well. However, neither VR nor AR are very much developed. Despite the more contemporary trend of AR, we sill don’t have the same interfaces as we see in moion pictures or in videogames. The only catch is that the interacion design proposed by entertainment in AR is more likely to become true than we could have imagined. 225 In theoreical terms, we have noiced that issues such as the intrusion of imagery on the symbolic environment and the digital world expanded environment are becoming big concerns. Details like digital tags, the post-geographical feeling of authors or the surrounding technopolia or consumerism of media goods are something that captures our glance. In fact, culture is indulged by technology since consumer electronics turned into fashion items. The atermath of this culture is that we are heading towards an always-mediated world, and as Web 3 is undoubtedly something new, as a Web of Data, in the near future authors will ind “interfacing” with AR something logical, since it keeps expanding from current media. In ilm we have inherited the engaging posiion, but the electric environment was further developed and spectators met their paricipaion phase in videoculture, videogames, VR, Internet, and all the screens of the post-computer era. Due to VR-games, the fact is that the topography of the possible is clearer than ever. Some call it “dataspace”, others say it is a “mediated-world” or a “dart-space”, though the concern is our opical relaion with the contents. We are emancipaing ourselves from standing sill, from non-mobile screens, from desktop Internet, from oice computers, all the prisons of imagery. When it comes to AR, VR, VR-games, if the medium is [sill] the message, then in that case perhaps we are already in a hyper-theatre. The inclusive awareness proposed by AR in entertainment games and ilms is proppelled by exteriority. This is to say that, once in an outdoor situaion, things that used to look like design or science icion, are now placing us in a subjecive cyberspace. Whenever we are, and wherever we are, “We” are the same carriers of media content, and we are augmented, sill inside previous media rules. Augmented Reality’s hardware is sill in its irst average consumer days, it promises and parially delivers interesing soluions now. It seems to us that an idea might look really good in the blueprint or when it is turned into a simulation, but it needs to be adapted and mainly be useful to the target audience that is going to purchase and use it. Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva Bibliografía Aliens (James Cameron, 1986) [movie] Dennis, E. E., DeFleur, M. (2010). Understanding media in the digital age. US: Pearson. District 9 (Neill Blomkamp, 2009) [movie] Doom 3 (ID Sotware, Acivision, 2004, PC) [videogame] Edge (2013). “Leap Moion brings 3D moion control to the desktop, with mixed results”, 12 September 2013. London, UK: Future Publishing. Available in www.edge-online.com/features/ leap-motion-brings-3d-motion-control-to-the-desktop-withmixed-results/ (Access in November 2013). Edge (2013b). “Believe the hype”, nº 254, June. London, UK: Future Publishing (75-81). Elias, H. (2013). Post-Web: The coninuous geography of digital media. Odivelas, Portugal: FormalPress. Available in: www.amazon.com. (Access in November 2013). Elias, H. (2009). First person shooter - the subjecive cyberspace. Covilhã: LabCom, UBI. Available in www.livroslabcom.ubi.pt/ pdfs/20110818-elias_irst_maio_2009.pdf (Access in November 2013). Furht., B. (2011). Handbook of augmented reality. New York, NY: Springer. Gibson, W. (2012). Distrust that paricular lavour. New York, NY: G. Putnam & Sons. Glass (Google, 2014) [hardware] Google (2012). The new muli-screen world: understanding crossplaform consumer behavior, August 2012. US: Google Think Insights, Sterling Brands, Ipsos. Available in www.google.com/ think/research-studies/the-new-muli-screen-world-study.html (Access in November 2013). Google Maps (Google, 2013) [appware] Google Now (Google, 2013) [appware] 227 Heavy Rain (Quanic Dream, SCEA, 2010, PS3) [videogame] Huybrechts, F. (2011). Mobile brand strategy. Why aim mobile investments towards a 3.0 future? Made by: The Mobile Relevance Project. November, 2011. Available in: www.MobileRelevanceProject.com (Access in August 2013). Ironman (Jon Favreau, 2008) [movie] Kinect (Microsot, Xbox One, 2013) [hardware] Leap (Leap moion, PC, 2013) [hardware] Lipovetsky, G. & Serroy, J. (2007). L’écran global. Culture-médias et cinéma àml’âge hypermoderne. Paris: Seuil. Lisberger, S. & Daley, B. (1982). Tron. Odivelas, Portugal: Europress. Manovich, L. (2001). The language of new media. Cambridge, Massachusets, MA: MIT Press. McLuhan, M. (1994). Understanding media: The extensions of man [1964]. Massachusets, MA: MIT Press. Minority Report (Steven Spielberg, 2002) [movie] Neale, Mark (2000). William gibson - no maps for these territories. US: Neale. Nilsson, S. (2007). Interacion without gesture or speech – a gaze controlled AR system. Los Alamitos, California, US: IEEE Computer Society Press. Available in: htp://citeseerx.ist.psu.edu/viewdoc/ download?doi=10.1.1.127.9994&rep=rep1&type=pdf (Access in November 2013). Oculus Rit (Oculus VR, PC, 2014) [hardware] Pajot, L., Swirsky, J. (2012). Indie game: the movie. US: Gowdy Manor. Available in: htp://buy.indiegamethemovie.com/ (Access in November 2013). PC Gamer (2013). “This is the future of PC games”, nº259, August. London, UK: Future Publishing (40-47). Postman, N. (1994). Tecnopolia - quando a cultura se rende à tecnologia [1992]. Lisbon, Portugal: Difusão Cultural. Augmented Reality Inside Videogames Herlander Alves Elias & João Dessain Saraiva Predator (John McTiernan, 1987) [movie] Rancière, J. (2009). The emancipated spectator [2008], Translated by Gregory Elliot. London, UK, New York, NY: Verso. Robocop (Paul Verhoeven, 1987) [movie] Smith, Lesley (2013). The magic tricks of jj abrams: A culture show special. BBC Two, London, UK. Available in: www.bbc.co.uk/programmes/b01sjsx5 (Access in July 2013). Star Trek Into Darkness (JJ Abrams, 2013) [movie] Steuer, J. (1993). Deining Virtual reality: Dimensions determining telepresence. In Journal of communicaion nº104, October 1993. Available in: www.cybertherapy.info/pages/telepresence.pdf (Access in November 2013). Syndicate (Starbreeze Studios, EA, 2012, PS3) [videogame] T3 (September, 2013). “Google glass”, nº306. Lisbon, Portugal: Goody (61-64). Terminator 2 - Judgment Day (James Cameron, 1991) [movie] Tom Clancy’s Ghost Recon Future Soldier (Ubisot Paris, Ubisot 2012, PS3) [videogame] Tom Clancy’s The Division (Ubisot, Ubisot, 2013, PS4) [videogame] Watchdogs (Ubisot Relecions, Ubisot Montreal, Ubisot, 2014, PS4) [videogame]