From physical to virtual spaces and back

Instructor: Jean-Marc Gauthier - Interactive Telecommunications Program, Tisch School of the Arts, New York University - updated 11/10/07

Description: This course will look at virtual reality, video games, simulations and other immersive environments and will address the tools needed to produce interactive 3D projects. The course focuses on speed-prototyping of simple 3D worlds interacting with sensors from the physical world and with data from the web. We will design simple virtual worlds for interactive storytelling, immersive learning, collaborative work and gaming. You start designing basic virtual worlds that can change dynamically with sound, video cameras and other data coming from sensors located in the real world. You will gradually design more layers for your project that will provide interactive animation, navigation inside a story , ways of seeing the world, playfulness.

The last part of the course takes you back into the physical world with the design and construction of miniature immersive displays. This gives you the opportunity to experiment with small screens and displays in order to refine the quality of the viewer's experience, the feeling of immersion and the suspension of disbelief from the viewer. No prerequisite required for this class. Please note that this class covers an introduction to 3D interactive content. One hour weekly tutorials are also offered during the semester in order to give you a jump start with 3D tools including Maya, Virtools, Microsoft XNA.

Links:

blog for the class

Jean-Marc's website

3D tutorials / Maya / Maya to Virtools / Virtools / Others

 

Course Breakdown:
 

Class 1. Roadmap presentation: physical sensors, virtual spaces, immersive screens, a multi-sensorial user interface

Lecture: Creating relationships between physical worlds and virtual worlds. Presentation of recent projects, virtual worlds with user generated content, gaming environments and virtual eco-systems. Discussion in class about the roadmap.

Reading assignment online: Compare the “look and feel” of two online virtual worlds, Second Life and Google Earth. See article in Technology Review, July 2007 “Second Earth” by Wade Roush. Post your comments on the class blog.

Assignment: Record a short machinima video capture (duration approx. 4 minutes) of your first encounter with a virtual world; for example Second Life or Google Earth. Format QuickTime or AVI with ambient sound.

 

Class 2. Speed-prototyping of virtual worlds for storytelling and games

Lecture: introduction to speed-prototyping and a 3D interactive survival kit for designing virtual spaces

Tutorial: Creating 3D video textures in a virtual world. Download the tutorial

Group assignment: Compare look and feel of photographic projects attempting to present 2D images in a 3D form. Post you comments about the following projects: http://labs.live.com/photosynth/ ...

Assignment: Create a virtual diaorama with photographs and a virtual camera. Tool: Maya

 

Class 3. Webcams and textures, loading dynamic images

Lecture: Interactivity + textures: how to display images of the physical world inside a virtual world

Tutorial: Capturing the viewer's gestures and postures: motion detection and Maya paint effects tutorial

Reading assignment: Text from Luc Courchesne

Assignment: Create an virtual world with paint effects objects connected to a video input.

 

Class 4. Ways of animating virtual characters

Lecture: Animations by examples: basic transformations (translate, rotate, scale), forward kinematics, inverse kinematics, morphing

Tutorial: How to setup a 3D character with forward kinematics

Reading assignment: Text about gaming from Stephane Natkin. Check his book Video Games and Interactive Media: A Glimpse at New Digital Entertainment

Assignment: Create an animated character Download tutorial

 

Class 5. Interacting with sound: dynamic sound inputs and 3D sound

Lecture: Interactive sound design inside a virtual world

Tutorial: How to design a 3D scene with sounds

Reading assignment: Text about sound in games, also sound projects by Antoine Schmitt, http://www.gratin.org/as

Assignment: Design a 3D interactive sound environment with and without visuals

 

Class 6. Re-creating motion for interactive animations

Lecture: Introduction to 3D interactive animations

Tutorial: How to animate facial expressions following a human voice Download tutorial

Assignment: Design a simple interactive animation of a talking head controlled with a sound input using volume and pitch. Tools: Maya or Virtools

 

Class 7. Interaction with live sound input

Lecture: Live sound input and spatialized sound

Tutorial: How to animate digital 3d puppets following live sound input Download tutorial

Assignment: Design a 3D interactive puppet following live sound input.

Class 8. Creating interactive environments for storytelling

Review of Mid-term assignments

Tutorial: From maps to 3D terrains. Introduction to designing terrains in Photoshop and Maya Download tutorial

Assignment: Fast prototyping of several interactive terrains. Tool: Maya, Virtools

 

Class 9. Interactive characters, viewer

Lecture: Interactive storytelling: Playfulness and interactivity.

Tutorial: Interactive character inside a scene. Using Nintendo Wii with interactive characters

Assignment: Design a demo of an interactive character walking on a terrain, a multi-sensorial user interface blending several input devices.

 

Class 10. Playing with motion capture data

Lecture: Motion capture for interactive characters

Tutorial: Motion capture for cameras and interactive characters

Assignment: Create an interface between a virtual space and the web. Final project pitch. Download tutorial

 

Class 11. Mapping the workflow for your final project

Lecture: Designing levels of viewer's experiences for your final project

Tutorial: How to map assets, interactivity and viewer's experience for your final project. Download roadmap

Assignment: Research for Final project presentation.

 

Class 12. Designing a miniature spherical projection on a small dome and other small immersive environments

Lecture: Immersive experiences

Tutorial: How to set-up and use an immersive spherical system

Assignment: Final project 50%

 

Class 13. Final project reviews

 
Class 14. Presentation of final projects