Dance with the Swarming Particles - Project Report

|| | DANCING WITH SWARMING PARTICLES Real time interaction with an avatar made of a swarm of particles in a generative virtual world RODRIGO GUERREIRO VAZ GUEDES DE CARVALHO institution: Specs [Synthetic Perceptive, Emotive and Cognitive Systems group] supervisor: Anna Mura under the INOV-ART program (DGA-ARTES - Ministério da Cultura - Portugal) || || DANCING WITH SWARMING PARTICLES || FIGURE 1 // Photo from the live performance 25-02-2011 UPF ABSTRACT In which the particles morph into the avatar’s body, DANCING WITH SWARMING PARTICLES is an interactive installation and performace that intends to explore the relationship between a physical user/performer and a virtual performer the “avatar” which has the physical characteristics of morphing flocking particles. The installation’s interactive environment is a media space that includes a physical space and a virtual space. The virtual space is initially composed of floating data which, through the interaction with the physical user/ performer, will become organized structures and take the shape of complex landscapes. The avatar’s body is composed by flocking particles that initially float in the virtual space without any apparent order. It is through the energy of the physical user/performer’s movements that the particles will start to morph into the avatar’s body. The ultimate form of the avatar and of the virtual and physical spaces, will result from the data generated by the user actions/performance and will be represented by the graphic visualisation of this interaction. In this way each interaction/performance will have a unique outcome. || RODRIGO CARVALHO | SPECS - UPF | INOVART 2 || || DANCING WITH SWARMING PARTICLES || INDEX INTRODUCTION/CONCEPT pag. 04 AVATAR “ 05 THE FLOCKING “ 06 THE SPACE “ 09 INTERACTION “ 10 NARRATIVE “ 11 STORYBOARD “ 13 TECHNICAL DESCRIPTION system scheme how it works installation setup “ “ “ 15 16 18 PERFORMANCE “ 19 ANNEX [state of the art references] [additional work in Specs] [credits] “ “ “ 24 28 29 || RODRIGO CARVALHO | SPECS - UPF | INOVART 3 || || DANCING WITH SWARMING PARTICLES || INTRODUCTION / CONCEPT DANCING WITH SWARMING PARTICLES is and interactive installation, that intends to explore the interaction between a physical performer/user and a virtual performer or avatar in a mixed media space. It intends to show and mix two similar parallel words (the real and the virtual) in order to create a mixed media space. The user/performer must explore and discover the space and create a relation with his/her avatar. The physical and the virtual are not overlapping, instead they are distinct but continuous. The border between the two environments is the projection screen that acts like a dynamic all seeing bi-directional eye. This installation is an experiment in interactive narrative and explores the potential of virtual reality and augmented feedback technologies as tools for artistic expression. The space and the avatar will evolve rebuilding and mutate themselves depending on the user’s behaviour. The tracking data from the physical performer movements is used as a parameter in the generation of the virtual world content. In this way each performance will generate a different scenario, a different avatar, …. The final form of the avatar, and the scenery will be the result of the data generated by the performer interaction, that is what makes each performance unique. || RODRIGO CARVALHO | SPECS - UPF | INOVART 4 || || DANCING WITH SWARMING PARTICLES || AVATAR The Avatar’s body is composed by a cloud/swarm of particles. This particle system has a flocking behaviour attached to it and its behaviour is controlled by a learning system. A tracking systems follows the body movements thus grabbing the position points of the physical performer’s skeleton (see Figure 2) . Through the physical performer, the single particles of the avatar(s), which are pure data forms, will start to morph into a system(s) that learns and behave but that can also adapt to dramatic disruptive changes induced by the environment, that is the physical performer behaviour….. Each group of particles have rules to “fly” together and a gravitational centre that attracts them. The position points form the skeleton will define these gravitational centres. This way the particles will be attracted to them and form the avatar body. FIGURE 2 // Skeleton tracking points, and particles being attracted to this points to from the avatar body || RODRIGO CARVALHO | SPECS - UPF | INOVART 5 || || DANCING WITH SWARMING PARTICLES || THE FLOCKING The flocking is formed by a group of 22 small groups of particles. Each group is composed by around 20 – 40 particles that fly around a gravitational & target center. These centers are connected with the joints of the human skeleton to form a humanoid figure. The flocking system is based on Craig Reynolds Boids, the artificial intelligent program which simulates the flocking behaviour of birds. “Boids” refers to “bird like object”. Craig Reynolds work was presented in a paper in 1987 in ACM SIGGRAPH conference. The boids are individual agents that have attached a set of rules: - separation (steer to avoid crowding) (see Figure 4); - alignment (steer towards the average heading of other boids) (see Figure 5); - cohesion (steer to move to the center of the mass of the group) (see Figure 6); FIGURE 3 // steer angle FIGURE 4 // alignment || RODRIGO CARVALHO | SPECS - UPF | INOVART FIGURE 5 // cohesion FIGURE 6// separation 6 || || DANCING WITH SWARMING PARTICLES || FIGURE 7 // Snap shoot from the original Craign Reinolds´s boids FIGURE 8 // Snap shoot from the original Craign Reinolds´s boids As individual agents, the Boids work as cellular automata, they act autonomously and in reference to what surrounds them. So , just as in John Conway´s Game of Life, each cell/individual will act in reference to his neighbours, The behaviour of each individual is based on a set of rules that makes reference his group. Apart from the rules attached to each Boid, other rules are added to the flocking system. Each group of Boids have a target assigned. So, the all group will seek and float around a target point. These target points are the skeleton joints of the physical user. The tracking of the skeleton is achieved using the “Kinect” camera and OpenNi application. FIGURE 9 Xbox Kinect camera Kinect (see Figure 9) is a peripheral sensor device, made originally for XBOX games, that allows a cheap computer vision tracking system in 3-Dimensions. It is composed by an infra-red light, and infra-red camera and a RGB camera. The combination of the RGB camera image and the infra-red provides the depth information of each point that allows the reconstruction of the 3dscene. Apart from the 3d motion capture the NITE (www.openni.org) software allows the skeleton tracking. * Reynolds, Craig (1987), “Flocks, herds and schools: A distributed behavioral model.”, SIGGRAPH ‘87: Proceedings of the 14th annual conference on Computer graphics and interactive techniques (Association for Computing Machinery): 25–34, doi:10.1145/37401.37406, ISBN 0-89791-227-6 http://www.red3d.com/cwr/boids/ http://www.vergenet.net/~conrad/boids/pseudocode.html UNITY FLOCKING CODE http://www.unifycommunity.com/wiki/index.php?title=Flocking JOHN CONWAY´S GAME OF LIFE http://en.wikipedia.org/wiki/Conway%27s_Game_of_Life Openni [industry-led, not-for-profit organization formed to certify and promote the compatibility and interoperability of Natural Interaction devices, applications and middleware.] www.openni.org || RODRIGO CARVALHO | SPECS - UPF | INOVART 7 || || DANCING WITH SWARMING PARTICLES || FIGURE 10 - 18 // Screen captures from the swarming particles in different phases || RODRIGO CARVALHO | SPECS - UPF | INOVART 8 || || DANCING WITH SWARMING PARTICLES || THE SPACE The aesthetics of the scenery is inspired by futuristic visions and science fiction from the past i.e. movies like “Tron” (1982) or “THX 1138” (1971). In the virtual world, the white void reinforces the sensation of a parallel universe that share some similarities with the real world, but where the laws of physics and logic don’t function exactly in the same way. FIGURE 19 // Screen captures from the scenery FIGURE 20 // Screen captures from the scenery || RODRIGO CARVALHO | SPECS - UPF | INOVART FIGURE 21 // Screen captures from the scenery 9 || || DANCING WITH SWARMING PARTICLES || THE INTERACTION As said before the Kinect camera his used to track the performer body skeleton. The Kinect is able to track the skeleton in a 3,5x3,5m area. So the performance is reduced to this tracking area. The skeleton tracking algorithm needs a calibration pose from the user for 5 seconds in a way it can understands the bodyform and identify the bodypoints. This pose, it’s a “hands on the air” pose, frontal and straight position, with the hands on the air. The pose had to be integrated in the performance dance. (see Figure 22 & 23). FIGURE 22 & 23 // Integration of the “Calibration Pose” in the dance performance After the body is detected and the skeleton tracking has started the physical user/performer must explore the physical space and its interactivity to identify the features of virtual space. He/she will find in the physical world some elements (5 spotlights in this installation) that he/she must activate in order to trigger events in the virtual world. Each time the user activates these 4 elements, a new level of complexity is reached in the virtual world, and more particles morph together to give form to a complete avatar. (see Figure 24). FIGURE 24 // Dancer clicking in the spotlight botton to activate it || RODRIGO CARVALHO | SPECS - UPF | INOVART 10 || || DANCING WITH SWARMING PARTICLES || NARRATIVE The user/performer must explore and discover the space and create a relation with his/her avatar. The narrative is based in a “creation loop”, a creation of life and destruction to create again and again and again. This life cycle of the virtual space evolves around the performer interaction with the 5 elements (the spotlights that he/she must trigger) which are part of the physical world (stage and dance choreography). The performer should discover and collect and full fill the meaning of these four elements to close the “creation loop”, so the virtual world is destroyed and the cycle starts again. This life loop is disclosed in two different ways: -The physical performer interaction with the physical space, the stage and the movements, and the creation of the virtual world; -The creation of the virtual performer, the Avatar. The scenery of the virtual world is a white void waiting to be filled with data from the physical performer interaction. The characteristics of the landscape are going to be generated dynamically with random parameters and inputs from the performer behaviour. || RODRIGO CARVALHO | SPECS - UPF | INOVART 11 || || DANCING WITH SWARMING PARTICLES || NARRATIVE The physical performer must first explore and discover the physical space and though the activation of the 4 elements he/she will activate the virtual world. With the energy released by his/her movements, the particles that inhabit the virtual world will be following the performer and thus will learn his/her movement. Through this process the avatar starts to gain form. The performer is in a “creation” process. While he/she discovers the space and activates new elements the avatar accumulates more data and knowledge. Step by step the avatar body is formed. Particles are flying all around and moving nervously submitted to different and opposite forces. On the one hand the particles have their natural flocking flying behaviours, but on the other hand there forces that attract them to form the avatar body. When the physical performer has just activated all the four performative elements, and all the trees are generated, the creation phase is completed. The world is created and the avatar is completely formed. Now the world is full of data, the world is entering again in a chaos mode and will start to collapse.The lifecycle of the world is reaching the end. Strange gravitational forces are pulling the particles away and devastating the virtual world. The avatar particles fall apart and the world is destroyed. || RODRIGO CARVALHO | SPECS - UPF | INOVART 12 || || DANCING WITH SWARMING PARTICLES || STORYBOARD A) -The world is empy; -The virtual camera is floating around. B) LEVEL 0 -The User is tracked, the virtual camera changes and points to the interaction area; -The particles and dancer move around without any apparent order. C) LEVEL 1 -The dancer aproaches the first spotlight, and activates it; -A group of partciles are attached to the dancer´s body; -Dancer moves around and starts to understand that this group of particles are following him; D) LEVEL 2, 3 & 4 -Dancer goes for the next spotlights; - As the spotlights are activated more particles are being attached to the body; - Now its possible to move the arms and legs and feel the feedback from the particles; -The body is starting to form. || RODRIGO CARVALHO | SPECS - UPF | INOVART 13 || || DANCING WITH SWARMING PARTICLES || STORYBOARD E) LEVEL 5 - All point lights are activated; - The body is completly formed; - Dancer and avatar are dancing together; - Layers of sound are being added to the athmosferic sound. F) LEVEL 6 - CHAOS - The world enters in the caos mode; - Everything is out of control; - Particles explode; - The dancer moves and tries to get back the control of the particles. tries to activate again the spotligths, but they are not working anymore; -The sound is caotic, huge amount of noise and out of sync samples. G) LEVEL 5 - Dancer falls on the floor; - Swarming particles fly awaY; - The virtual camera zooms out; - Fade out to black. F) END || RODRIGO CARVALHO | SPECS - UPF | INOVART 14 || || DANCING WITH SWARMING PARTICLES || TECHNICAL DESCRIPTION System scheme FIGURE 25 // Architecture of the system || RODRIGO CARVALHO | SPECS - UPF | INOVART 15 || || DANCING WITH SWARMING PARTICLES || TECHNICAL DESCRIPTION How it works 1// INSTALLING THE KINECT To work with the Kinect camera, first you need to install the drivers, and the apllication that does the skeleton tracking (NITE) A) download the OpenNI binaries from http://www.openni.org/?q=node/2 (in this case: OpenNI 1.0 Alpha build 23 binaries for Win32); B) download the NITE SDK from http://www.primesense.com/?p=515 (in this case: NITE 1.3 Beta build 17 SDK for Windows); C) download SensorKinect from https://github.com/avin2/SensorKinect; D) install OpenNI; E) install NITE SDK and use the following free license key frop openni.org: 0KOIk2JeIBYClPWVnMoRKn5cdY4=; F) unzip the SensorKinect file; G) connect the Kinect device; H) open the device manager (Start > Run... > devmgmt.msc) and verify that ‘Xbox NUI Motor’ is in the list (but with an exclamation mark); I) Right-click on the device and choose ‘Update Driver’ and make windows look for a driver in the avin2-SensorKinect-b7cd39d\Platform\Win32\Driver folder. This will install a device called ‘Kinect Motor’.; J) reboot your computer when asked; K) After rebooting, two new devices will be found. Repeat the same process by pointing the wizard to the driver folder. One of the two new devices will fail to install (the one for the audio), the one that will succeed will add a new device called ‘Kinect Camera’; L) install the executable the zip file in the folder avin2-SensorKinect-b7cd39d\Bin; M) run C:\Program Files\OpenNI\Samples\Bin\Release\NiUserTracker.exe to test if everything works; N) if it fails, run C:\Program Files\OpenNI\Tools\vcredist_x86.exe; O) install the OSCELETON (https://github.com/Sensebloom/OSCeleton). This application sends the sckeleton information by OSC to any software with OSC receiver. * This drivers are from January 2011, it’s recommended to check for updates in the referred links || RODRIGO CARVALHO | SPECS - UPF | INOVART 16 || || DANCING WITH SWARMING PARTICLES || 2 // STARTING THE APPLICATION After the installation of the drivers we can now start our application: A) Plug in the kinect camera with the computer; B) start the application OSCELETON. (this will be running in the background. Reading the skeleton tracking and sending the info by OSC); C) start the application “Dancing with Swarming Particles” 3// INTERACTION WITH THE APPLICATION The application is divided in different levels. Starting from –1 till 6. A) Level –1 ,the beginning of the application. Virtual cameras are floating around the scenery. In this point the performer can enter in the interaction space, he will be detected by the camera, but his skeleton is not detected yet. He must do the “calibration pose” for 5 seconds to start the detection; B) Level 0,skeleton detection. When the detection is activated the space virtual camera turns off, another virtual camera located in the interaction area starts. We are now ready to perform and dance with our avatar; C) From level 1 to 5, the avatar will be sequentially created. In the first level, only a few particles are following the dancer, and in level 5 the particles have completely formed the dancer. To pass throw the levels an human operator must click in the keyboard in the keys “1”, “2”, “3”, “4” and “5”. The moment that he must click this keys, is the exact moment that the performer in the stage clicks in the spotlights; D) After some while in the level 5, the human operator must decide when is the moment to go to the next level, the chaos level (where the particles explode and the dancer loses control of his avatar). This decision is in the hands of the human operator in a way he can surprise the dancer and the explosion and chaos become more realistic. So the human operator must click in the key “6”. E) The chaos level, (level 6) is divided in different sub levels : 1 (the explosion), 2 (lost of control), 3 (second explosion), 4 (particles return to their original flocking group, and fly around), 5 (camera changes, to fly virtual camera, and zooms out). Here the user must click in the key “z” to go up in the levels. After the second explosion (sub-level 3 ) the dancer must fall in the floor, while the particles return to their original position (sub-level 4) and the camera starts to fly around and zoom out (sub-level 5). *all the files are in the “Dancing with Swarming Particles” DVD. In the installation of Kinect || RODRIGO CARVALHO | SPECS - UPF | INOVART 17 || || DANCING WITH SWARMING PARTICLES || TECHNICAL DESCRIPTION Installation setup FIGURE 26 // Setup of the installation || RODRIGO CARVALHO | SPECS - UPF | INOVART 18 || || DANCING WITH SWARMING PARTICLES || PERFORMANCE FIGURE 27 & 28 // Photo report from the performance at UPF, 26, 02, 2011 || RODRIGO CARVALHO | SPECS - UPF | INOVART 19 || || DANCING WITH SWARMING PARTICLES || PERFORMANCE FIGURE 29 & 30 // Photo report from the performance at UPF, 26, 02, 2011 || RODRIGO CARVALHO | SPECS - UPF | INOVART 20 || || DANCING WITH SWARMING PARTICLES || PERFORMANCE FIGURE 31 & 32 // After the performance, the audience was invited to try the installation || RODRIGO CARVALHO | SPECS - UPF | INOVART 21 || || DANCING WITH SWARMING PARTICLES || PERFORMANCE FIGURE 33 & 34 // After the performance, the audience was invited to try the installation || RODRIGO CARVALHO | SPECS - UPF | INOVART 22 || || DANCING WITH SWARMING PARTICLES || ANNEX || RODRIGO CARVALHO | SPECS - UPF | INOVART 23 || || DANCING WITH SWARMING PARTICLES || REFERENCES Specs – “Re[per]curso” http://www.iua.upf.edu/repercurso Performance presented in 2007 at the MACBA (Museu d’Art Contemporani de Barcelona) and in 2008 at the Art Futura festival at the Mercat de Les Flores. Also presented as talk in ACM SIGRAPH 2008 conference in Las Vegas. Mura A. et al. re(PER)curso: an interactive mixed reality chronicle. ACM SIGGRAPH 2008 Ole Kristensen – “Body Navegation” http://www.recoil-performance.org/ || RODRIGO CARVALHO | SPECS - UPF | INOVART 24 || || DANCING WITH SWARMING PARTICLES || REFERENCES Memo Akten – “Reincarnation” http://www.vimeo.com/3047446 Adrien Mondot – “Cinematique” http://www.vimeo.com/9782048 || RODRIGO CARVALHO | SPECS - UPF | INOVART 25 || || DANCING WITH SWARMING PARTICLES || REFERENCES Robert Hodgin – “Flocking birds” http://www.vimeo.com/197323 Kyle Macdonald – “Clouds Are Looming” http://www.vimeo.com/9782048 || RODRIGO CARVALHO | SPECS - UPF | INOVART 26 || || DANCING WITH SWARMING PARTICLES || REFERENCES Freider Wess http://www.frieder-weiss.de/works/all/ 1024 architecture – “Euphorie” http://www.1024architecture.net/en/2010/02/euhorie-2/ || RODRIGO CARVALHO | SPECS - UPF | INOVART 27 || || DANCING WITH SWARMING PARTICLES || ADDITIONAL WORK IN SPECS Collaboration in a project on data visualization in a mixed reality enviroment called XIM (the Experience Indiction Machine). “DEALING WITH DATA DELUGE – emboided interaction with complex neuronal data in mixedreality” We built a 3D real-time visualization system to graphically represent the massive connectivity of neuronal network models in the eXperience Induction Machine, an immersive space equipped with a number of sensors and effectors that we constructed to conduct experiments in mixed-reality. Using this infrastructure we developed an embodied interaction framework that allows the user to move freely in the space and navigate through the neuronal system. The project was submitted as technical paper for SIGGRAFF2011 (“Computer Graphics and Interactive Techniques Conference” in Vancouver). We are currently waiting for the results. In this project I worked together mainly with Alberto Betella (PhD Student at Specs). I was in charge of the 3d visualization and Interaction under Unity3d (3d engine) and it implementation in the XMI. For the submission to SIGRAFF2011 I was also in charge for the presentation video, illustrations and photos. || RODRIGO CARVALHO | SPECS - UPF | INOVART 28 || || DANCING WITH SWARMING PARTICLES || CREDITS PROJECT MADE BY Rodrigo Carvalho [[email protected]] INSTITUTION Specs [Synthetic Perceptive, Emotive and Cognitive Systems group] Universidad Pompeu Fabra - Barcelona SUPERVISOR Anna Mura under the INOV-ART program DGA-ARTES - Ministério da Cultura - Portugal Barcelona 2010 - 2011 SPECIAL THANKS FOR HELPING ME TO :: Sytse, Elena, Henrique, Alberto and everyone at Specs. || RODRIGO CARVALHO | SPECS - UPF | INOVART 29