Dr. Paul Debevec, director of the UC Institute for Creative Technologies, provided an inspiring keynote about the collection of data for realistic rendering and simulation. His work is at the forefront of what’s possible for the application of scanning and rendering for film-based animation in the entertainment industry.
Debevec provided a technology progression from his pioneering work that was done for his Ph.D. He developed some of the first software for the stitching of photos together in an automated fashion (think PhotoSynth) that applied photogrammetry techniques to create a realistic model of the Berkeley Campus back in 1997. He showed the use of kite-based aerial images along with other low-cost, low-tech capturing methods to provide a realistic model of the campus. A short movie was made that caught the eye of Hollywood and some of the techniques that were developed were used in the development of the Matrix movie series to provide the background images to the slowed-down scenes where the hero Nero dodges bullets.
From this early work, Debevec became interested in both the capture of the shape of objects and how they reflect light. His aim was to have complete control over how things are illuminated, because the shape just isn’t enough to capture realism. The project that was used to explore different techniques was a digitization of the Parthenon (http://gl.ict.usc.edu/Films/Parthenon/) where he took scans of the actual site and combined these with scans of the various sculptures in museum settings to bring back the full context of what the original building was like.
The technical hurdles to realistically render the model were significant, with 3D scans for a detailed point cloud at multiple resolutions combined with detailed capture of lighting to take out shadows for a model that could be completely manipulated to add lighting and weather. To accomplish this task, Debevec used light probes to understand all that’s needed to know about incidence of light from the sun intensity, to the direction, and also capturing indirect light. The ultimate outcome is a rendering that matches neatly to the photo, with the photo used as a tool to calibrate the texture map to make sure that he got it right.
This exploration of the properties and influence of light on the realism of models was then put to the ultimate test in his follow-up and current work to capture the human face. The complexity of dynamic human face motion is compounded by the complex surface reflectance with the oily glow of the surface compounded by the interior translucency of skin that bounces light in different directions. To understand the properties of skin and faces, Debevec and his team created a sphere with 156 white LEDs that surround and model the face of actors to get away from a plastic and synthetic rendering.
The goal was to have complete control over the lights with an ability to isolate both diffuse light to get the sub-structure of skin surfaces with the reflectant properties to capture the human glow. Very detailed scans of textures have been married with the normal reflectance of skin to render faces digitally. This technology was used to model all the Avatar characters, not to animate but to inform the animators about the properties of the actor faces so that they could create realistic scenes in different lighting and atmospheric conditions.
Debevec continues to fine-tune the realism of his models, with the ultimate goal of realistic animated computer renderings. The level of technical hurdles to capture true realism shows the many benefits that Hollywood investments will enable in other fields, including forensics, medical, AEC, etc.
