Immersive Cinema: Narrative Structure in a Virtual Environment

The Interface Lab at ACCAD, a 10'x6' stereoscopic display and multitouch table in foreground

Project Summary

Building upon a foundation in filmmaking and animation, the proposed project incorporates emerging technologies for alternative approaches in the practice of animation within new media arts.

Technology-based tools for artists working in animation advance continuously, with applications developing in a diverse range of disciplines that involve visualization and design processes. 3D computer graphics processing technology is developing to allow the animator to move from the traditional “frame-by-frame” practice of animation to composing motion sequences “on-the-fly” in real time. In recent years, the capacity for real time rendering has increased tremendously while cost has dropped at a surprising rate. How these advances will be interpreted and incorporated into animation applications is at the forefront of development in the medium. The advancement in real time graphics rendering has progressed to the extent that it is has become more accessible to end users, but has not yet been fully exploited in the animation artist’s or designer’s set of tools.

The objectives of this proposal include:

  • The development of a low-cost, transportable screen system capable of high resolution, stereoscopic 3D computer graphics display.
  • The production of creative works developed for the above display.
  • Exploration of substantive enhancement of the viewer experience through sensory immersion and physical interaction.

The work seeks to bridge physical and visual barriers between time-based media and its audience through an extension of traditional cinematic experience with its history and strengths in storytelling and narrative structure; introducing a form of “virtual cinema”.

In addition to the creative work, expected outcomes include the incorporation of research and resources in curricular activities at the Advanced Computing Center for the Arts and Design (ACCAD), to benefit graduate students from a range of disciplines including Design, Visual and Performing Arts, and Computer Sciences. The construction of an immersive screen display system would augment and complement existing development of virtual reality applications at ACCAD, as well as providing for a standalone system that can be transported for public display at conferences or arts and technology symposiums such as SIGGRAPH or ISEA (see references). ACCAD will be providing additional support for this project including shared costs. Additional funding will be sought to further enhance hardware and software configurations and towards the production of additional related works. Success of this proposal may be assessed through presentation of completed works and/or papers/presentations related to the works and related topics including incorporation into curriculum and student research outcomes.

Overall goal

Building upon current research in animation and real time visualization projects, this proposal consists of the production of creative work utilizing real time computer graphics and the design and construction of a display system for use in public presentation and on-going multidisciplinary research. The creative work relies upon the display system for development. The display systems itself will facilitate a diverse range of visualization applications for research by faculty and students.

Specific objectives

  1. Research current developments in the field of real time graphics and its implications for the artist working in time-based animation. Additionally, this research will inform development of the curriculum in the Advanced Computing Center for the Arts and Design, keeping the program current, if not anticipatory. This stage is ongoing, with an emphasis during preliminary design stages for the following.
  1. Design and construct a large-format immersive (approx 10’) rear-screen projection system using passive 3D stereo projection (viewers wear polarized glasses), networked computer graphics rendering (multiple computers working in parallel), and gesture-based input devices (motion capture, video tracking, audio analysis).
  1. Produce a creative work for exhibition utilizing real time graphics techniques. The objective is to produce a time-based narrative animation running on a computer display in real time. The work combines characteristics of traditional cinema and narrative structure with concepts of 3D gaming technology and virtual reality applications, producing a unique experience for the viewer.

Rationale

The outcomes of this stage will facilitate further development of custom applications and resources. Future plans include additional funding for expanding capabilities of the display system and collaborative projects in applied research in visualization and the performing and visual arts

To define real time animation, “traditional” animation may be considered as inclusive of any animation work requiring a frame-by-frame creation process, whether the frames are rendered by hand or individually on a computer before being assembled into a sequence. Real-time animation is rendered on the fly, before the viewer, and with the additional capacity for navigation and change at any moment. This differentiates between the common notion of an animated film and work that becomes an immersive experience, non-linear, and responsive. In the near future real-time rendering may replace traditional rendering processes. It will not, however, replace the skill and foundations necessary for the artist working in animation.

My research in animation and real time interactive applications has to this point operated in parallel, although each consistently informing the other. Works that I create in “traditional” time-based format, typically presented on video, develop through narrative structure and the storytelling process associated with cinema. My real time graphics work has thus far addressed what are commonly accepted uses for interactive media in the areas of simulation, visualization and instructional systems. As I have applied my knowledge to the medium over time I have become more interested in a closer combination of the two disciplines, incorporating the immediacy, immersive, and non-linear characteristics of real time rendering with the temporal and narrative structure of traditional cinema.

The rear-screen display system is designed for facilitating this viewer experience, however the development of such a system has extensive potential for additional research while housed at ACCAD, providing a virtual environment display system for applications in design, simulation and performance arts. Exploring data sets, development of way finding and navigation systems, and other diverse forms of visualization would be possible. Development of templates and/or toolsets for the system could run in parallel with a course I teach in building virtual environments – offering new ground for exploration and experimentation using real time animation and simulation to represent data in a range of disciplines. Additionally, I believe this will compliment and provide additional interaction with current research trajectories at ACCAD: the wireless-VR project and Electronic Media and Motion Arts (EMMA) lab development, as well as the Interface Lab at the Ohio Supercomputer Center.

Significance

The use of real time 3D computer animation has a short history in virtual reality simulations and the game industry. However examples of it being used by artists as a medium for expression is a nascent field, with very few instances of it being re-purposed for communication of ideas that move beyond reference to its origins or issues of mainstream game culture. Largely unknown in the United States, an underground art movement has been taking place in Europe called the “Demo Scene”. This movement is popularized primarily by young people, learning how to program and work with 3D animation and gaming technology. The resulting works exploit the power of inexpensive graphics hardware available for consumer level computers. Often abstract in form, they combine two- and three-dimensional graphics, text and music to create their own style of “visual poetry” that runs in linear sequence, unlike the non-linear structure of game environments. The works produced by these artists are distributed within their communities on the Internet, with files transferred being much smaller than traditional digital video files and without the high bandwidth requirements of streaming video systems.

A recent movement in the US has been named “Machinima” (Machine and Cinema). This art form is derivative of the European Demo Scene, but considerably more limited. It is unique however in that artists are employing the graphics engines in commercial 3D game software to create narrative time-based works. Several commercial game companies have released editing software allowing users to alter their characters or environments. Machinima artists have taken this a step further than the companies expected by exploiting these tools to build entirely new sets of characters and environments, and using “demo recording” features in the games to create their own narrative movies.

These two movements help illustrate a trend which may be taking place in the recombination of cinema, animation and game technologies for a new genre in digital media. On the fringe of what is traditionally considered time-based media, the artists are developing approaches that have not been seen before.

The approach in this proposal has comparisons to these existing forms in that it will use similar technology with three-dimensional environments, objects, characters and cameras all being rendered and manipulated in real time. However, the proposed work will distinguish itself in a number of ways: Concepts and techniques of traditional cinema are to be employed as an emphasis in the work, engaging the viewer with the structures of narrative storytelling. This is augmented by the immersive experience of the display system and stereo 3D graphics. An emphasis will be placed on level of detail, quality of light, shadow and texture to elicit a tangible experience of a real-world environment. The final narrative will be part of the project development process, but the premise for the visual experience will be to situate the viewer so as to feel part of an unfolding event between several characters, instilling a heightened perception of their presence and the passage of time. The narrative and visual elements will explore the irony between simulation and detailed depiction of commonplace environments and conventional activity, raising questions about representation of real time and space versus the rapid pace and heightened reality of contemporary cinema. The intent is to find a balance between these modes of representation, making the viewer aware of the circumstances in question while engaging him or her in a narrative sequence that simultaneously depicts the perspectives of the virtual characters as they strive to comprehend their own relationship to their simulated environment.

Further experimentation with the capabilities of rendering animation in real time and the development of successful works in this format may ultimately result in artists working in an entirely new approach. With higher profile, public forms of distribution and the acceptance of a format in which sets and characters are generated entirely within the computer in real time, there exists the potential of establishing a new form of virtual cinema.

Methods

My process as an animator involves use of computer software for 3D modeling and rendering. Using these tools to create geometry and materials, this data will be exported for use in authoring software designed for game development or virtual reality applications. An emphasis will be on creation of shaders, lighting, and environmental effects with the goal to achieve a visual quality attributed to software rendered 3D animation.

The projection system is to be constructed of components from a variety of sources. An attribute common with other virtual reality systems is that it must integrate multiple technologies for graphics display and input devices to create a seamless and transparent experience for the user. Special rear-screen material is required that will not scatter polarized light. A frame for the screen will need be to be easily assembled for transportation. Circular polarized filters for projectors and a set of polarized glasses for viewing are necessary for the 3D stereo display. Two computers, networked together for simultaneous rendering of the scene for each eye, will be connected to two DLP data projectors. Use of existing motion tracking devices, as well as development of custom video camera tracking and audio input devices will allow for interaction with the display.

The proposed creative work will use the display system in the form of an installation recreating a traditional theater environment, with seating arranged in front of the screen in an enclosed, darkened space. While it is possible the animation could be presented on a single computer and conventional display, the goal is to present the animation in stereoscopic 3D on the rear-screen projection system to create the illusion of being immersed within the represented space. Movement, sounds, or other gestures made by the audience may affect elements of the animation such as camera position, character movements or lighting. An audience member may stand up to leave, triggering new events in the narrative sequence.

It is important to note that the proposal for the display system includes planning for use in a diverse range of applications. It will be designed to be reconfigurable, with implementation of technologies for input and interaction that will allow for directed use as a design and visualization tool. A designer and client could be interacting with 3D models as they respond to each other while immersed in the virtual space. The enhanced depth perception and real time interaction provided by the system can augment a wide range of applications requiring understanding of complex surfaces or data sets.

Outcomes expected and measures of success

This project directly relates to my primary research interests. As faculty at The Ohio State University, the opportunity to implement fundamental components for facilitating this research will allow development at an accelerated pace. I anticipate funding for my continuing work in this area specific to the development of creative works, and have been a past recipient of artist’s media and exhibition awards. This proposal presents a unique opportunity to create new facilities on campus not only for my own creative work but for collaborative and multidisciplinary initiatives.

Outcomes of this proposal are represented by two principle achievements: (1) a creative work developed for exhibition in art and technology related venues and (2) an immersive display system for which the creative work is developed in addition to its use for ongoing development of similar and other visualization projects.

The display system will provide for continued research and development opportunities, complimenting existing technologies at ACCAD.  It will provide for collaborative endeavors with the Ohio Supercomputer Center’s Interface Lab and will expand academic research within the multidisciplinary course offerings at ACCAD. If funding is awarded, the majority of the budget will be applied to the construction of the display system.

The production of the creative work will be designed specifically for the immersive display system. Not only will this work operate as a proof of concept for applications utilizing the display, but is intended to present an approach that combines characteristics of virtual reality applications, game technology and traditional narrative structure, addressing implications for new forms of interactive and/or immersive environments combined with cinematic form.