Digital Modeling of the Appearance of Materials Julie Dorsey Holly Rushmeier Yale University François Sillion ARTIS, GRAVIR/IMAG-INRIA 1 Course Description In photorealism our ultimate goal is to synthesize images that show no evidence of being computer generated. Generating realistic images requires simulating the appearance of the wide variety real world materials that surround us -- wood, leather, sand, shells, fabrics, plants, etc. Three approaches have evolved to simulate material appearance from three different communities -- artists, mathematicians and scientists, and software engineers. In an artistic approach layers of textures are painted based on careful observation. In a mathematical approach equations are developed that express material behavior in terms of basic scientific phenomena. In a software engineering approach efficiently evaluated shaders are constructed that users can control with simple parameters to achieve different visual effects. These approaches all contribute important elements to image synthesis. Our goal is to present a comprehensive view of materials modeling that embraces artistic observation, mathematical modeling and efficient computing. We begin with observation. We present a visual tour of images of real materials, and consider how they are classified by the effects that need to be modeled to realistically render them. Essential appearance concepts such as diffuse, specular, subsurface scattering, wave effects are defined and illustrated. We give a brief overview of the mathematical terms used to describe surfaces. We then discuss popularly used numerical models such as (but not restricted to) Blinn, Cook-Torrance, Ward and Lafortune. We discuss these in terms of the effects they capture and the visual impact of the parameters of each model, and not their mathematical derivations. We return to observation, and consider the complexity of aged and processed materials. We give an overview of recent work to simulate this complexity with combinations of data capture, mathematical modeling and user interfaces. Finally, we close with how material models are efficiently integrated into both real time and and offline rendering systems. Course Prerequisites The course requires only an introductory level of familiarity with computer graphics from either a previous course or practical experience. We will assume that the students understand basic terms and ideas such as setting a pixel color by specifying values of red, green and blue, and projecting a triangle onto a set of pixels given the specification of a virtual pinhole camera. Instructors Julie Dorsey is a Professor of Computer Science at Yale University, where she teaches computer graphics. Before joining the Yale faculty, she was a tenured faculty member at MIT. She received undergraduate (BS, BArch 1987) and graduate (MS 1990, PhD 1993) degrees from Cornell University. Her research interests include photorealistic image synthesis, material and texture models, illustration techniques, and interactive visualization of complex scenes. In addition to serving on numerous conference program 2 committees, she is a member of the editorial board of IEEE Transactions on Visualization and Computer Graphics and is an area editor for The Visual Computer. Holly Rushmeier is a Professor of Computer Science at Yale University. Since receiving the Ph.D. from Cornell in 1988, and she has conducted research in global illumination, data visualization, applications of perception, 3D scanning, and applications of computer graphics in cultural heritage. She has published in SIGGRAPH, ACM TOG, IEEE CG&A and IEEE TVCG. Over the past 15 years, she has organized SIGGRAPH courses on radiosity, global illumination and a scanning case study, and has lectured in SIGGRAPH courses on capturing surface properties and applying perceptual principles to rendering. François Sillion is a Senior Researcher at INRIA (Institut National de Recherche en Informatique et Automatique), where he is the head of the ARTIS research group and scientific advisor of INRIA Rhône-Alpes. He is a graduate of the Ecole Normale Supérieure Paris and received the PhD in 1989 from the Université Paris-XI. He is on the editorial board of ACM TOG and Computer Graphics Forum, is co-editor of the EUROGRAPHICS workshop series and is chairman of the EUROGRAPHICS working group on rendering. In addition to many papers on modeling and rendering, he is the coauthor "Radiosity and Global Illumination. " He is a fellow of the Eurographics Association. Additional information about our teaching and research activities can be found at: http://graphics.cs.yale.edu/ http://artis.imag.fr/~Francois.Sillion/ 3 Course Schedule and Notes Organization 1. Introduction ( 20 minutes) Very high level view of basic concepts and where models of materials fit into the image synthesis process. References will be given for people who need to fill in some of the ideas needed as background. 2. Observing and Classifying Materials (30 minutes) A visual guided tour of materials and how they are classified by how they are modeled in computer graphics. The tour is based on observing real world materials. 3. Numerical Models ( 55 minutes) A survey of popularly used material models, explaining the range of effects they capture and meaning of parameters (no derivations or mathematical analysis) 4. Processing of Materials ( 60 minutes) Introduction to and examples of appearance variations over time, due to processing/aging. 5. Integrating Material Models Into Rendering Systems (45 minutes) An overview of the relationship between material models and complete rendering systems that include various types of illumination simulation. 6. Additional Materials DORSEY, J., AND HANRAHAN, P. Modeling and rendering of metallic patinas. In Proceedings of SIGGRAPH 1996 pp. 387–396. DORSEY, J., PEDERSEN, H. K., AND HANRAHAN, P. Flow and changes in appearance. . In Proceedings of SIGGRAPH 1996 pp. 411–420. DORSEY, J., EDELMAN, A., JENSEN, H.W., LEGAKIS, J., AND PEDERSEN, H. K. Modeling and rendering of weathered stone. In Proceedings of SIGGRAPH 1999, pp. 225–234. WANG, L., WANG, W., DORSEY, J., YANG, X.,GUO, B., SHUM, H.-Y., Real-time rendering of plant leaves, Proceedings of SIGGRAPH 2005 (ACM TOG),pp. 712-719. LU, J., GEORGHIADES, A.S., RUSHMEIER, H. , DORSEY, J AND XU, C.. Synthesis of material drying history: phenonmenon modeling, transferring and rendering. In Eurographics Workshop on Natural Phenomena 2005 pp. 7-16. 7. References for Further Reading 4 Digital Modeling of the Appearance of Materials Julie Dorsey Holly Rushmeier Yale University François Sillion ARTIS, GRAVIR/IMAG-INRIA In these course notes we present basic principles of defining numerical models to be used in rendering realistic imagery of physical materials. Additional information on research in rendering and materials can be found at http://graphics.cs.yale.edu/. Updates or corrections to these notes will be posted at this site. 5 1. INTRODUCTION Digital Modeling of the Appearance of Materials: Art or Science?? The materials here are rendered with models. Artists conceived the shape, and selected materials to construct such objects in the physical world. A purely artistic approach could be used to digitally paint the shades of light and dark on the digital shapes to give the illusion of translucent stone or shiny gold metal. However, to generate these images material models are expressed numerically and rendered using lighting simulations. That is their appearance – the colors, shades of light and dark, were computed, rather than being digitally painted on the model. There are many ways in which compelling visual images can be generated of the world. Different approaches can be used to create images, all of which are valid in some set of circumstances. To define our approach in this course we distinguish between the use of digital models to define materials, and the use of techniques. Broadly, by digital models we mean a scientific or engineering approach to rendering, and by technique we mean an artistic or craft-oriented approach. We are focused here on digital models of materials. 6 Digital Models: Predictable control parameters Consistent across view and lighting conditions We define a model as taking a physically measurable input and producing a predictive output that can be verified by physical measurement. A model of a material makes possible the reliable rendering of the appearance of that material in any geometric and lighting conditions. We define a technique as taking an input which is not necessarily measurable, and produces an output that may or may not reproduce the appearance of an object under arbitrary circumstances. Human judgment is required to set the input of of a technique, and to evaluate its success. 7 Models are typically used in science or engineering for their predictive reliability. If a bridge is being designed, a model is used to predict the load that a particular beam can support given the material and dimensions of the beam. Similary, if a lighting system is being designed for a building, the appearance of objects in the building with that system is predicted by modeling the materials illuminated – carpet, paint, etc. Assurance that the system meets specifications is gained by the ability to compute specific values of incident light that can be measured in the environment. Material models may be used outside of design applications which must meet quantitative specifications. Models may be use as elements in a technique. In designing the visual look of a scene, an artist working in a digital medium may use a digital model of a material to produce a visual effect drawn from their visual experience with the everyday world without the demands of manual adjustments. Digital models may also be used by artists as a starting point to identify parameters that can be manipulated to meaningfully exaggerate or surpress aspects of appearance we normally encounter. Techniques have been used successfully by artists for centuries to produce visual impact. Techniques traditionally used to represent things like the effect of a shiny metal or matte painted surface often serve as the inspiration for the development of a model. The line between a model and a technique is not absolute. An example is the use of techniques and models to render a drawing. Many image processing techniques can be applied to produce the illusion of different materials used to make a drawing. A series of filters for blurring, edge detection and color manipulation can be applied to a digital photograph to give the appearance of a drawing made with charcoal or crayon. A user can adjust these filters to generate variations until they judge that the effect of either the charcoal or crayon material has been achieved. That effect would be valid for the direct view the user has of the image in the editing program. There would be no guarantee that the image could be say be used on a page laying on a desk in a digital scene and look realistic. A digital model of charcoal or crayon however, would take into account the physics of how the carbon or wax reflects light from various directions, and how the smale scale particles are typically deposited on the paper being used for a particular physical application of the medium. The input to the model might be the density of carbon particles, the spectral reflectance of the wax, and/or the geometric distribution of the material on the page as a function of rate of application of the drawing medium. The measurable output would be the light distribution from the page under the specified lighting conditions. The model of the drawing on paper could be inserted into a digital desk scene, and would appear reliably dull or shiny depending on the specific lighting environment. Clearly a mix is possible, and generally desirable, between the technique and the model. The digital model of wax or charcoal reflectance could be applied to a spatial distribution of the medium defined by the user using image processing techniques. An artist seeks to produce visual impact through the use of color and texture. In computer graphics this can be achieved by painting an image, or by painting or assigning acquired images as textures to 3D objects. Many techniques have been developed for created various types of effects with this approach, as described in works such as the book Digital Texturing and Painting by Owen Demers (New Riders Publishing, Indianopolis, IN, 2001.) The artist may be interested in creating a specific mood, or drawing attention to certain places in a scene. Our purpose here is not to teach art or design to create moods or direct attention. We don’t aim to present techniques, say, to make a piece of fruit look fresh and sweet. However by presenting models that predict appearance in the physical world, we provide tools for the artist to draw on. We present models for reflection of surfaces such as from a pear or apple, and for the refraction of light through small drops of water on the surface of the fruit. If an apple covered with dew produces the visual effect sought by the artists, the models we present here allow the construction of a compelling image of the apple. 8 Digital Models: Goal is to produce images that appear the same as seeing a scene or object in person Our goal is to make predictive images that give a view of a scene or object that is the same as if the person were viewing it directly. Material modeling is one aspect of this. We need to consider the object’s shape, and the light incident on it. We also need to take into account how the incident light is perceived by humans. Perception is a complex phenomenon, but some simple understanding of it makes modeling easier. The complexity of human perception is what makes digital imaging even possible. It is a remarkable fact that vastly different arrays of incident light, if adjusted properly, give us the same visual impression. 9 Our goal in this course is not to explain why things look the way they do to human beings, but to simulate the mechanisms of appearance that are relevant to the construction of digital models. Clearly there are libraries full of information of various aspects of human vision which ultimately contribute to how people perceive and evaluate materials. However, it is also clear that the appearance of an object depends on the light that reaches the viewer from the object, and the nature of human visual response. One approach would be to simulate with phenomenal accuracy the light that would reach a viewer from an object, and then build a device that would produce that same light distribution for the purpose of display. Such lighting simulations would be computationally infeasible, and such display devices do not exist. To effectively model the appearance of materials and display the results, we need to exploit the limitations of human vision. While there is no single comprehensive model of vision, it is reliably known that there are limitations to human vision. Many works discuss vision and perception in great depth, such as Seeing, edited by K.K.D. Valois, (Academic Press, San Deigo, CA, 2000.) The input to the visual system is completely defined by specifying the spatial, temporal and wavelength variations of incident light. The limits of the variations that need to be specified are defined by a “window of visibility”. There is a relatively narrow band of wavelengths and spatial and temporal variations to which people are sensitive. We exploit loose bounds on these limits in defining the accuracy and resolution required of appearance models. Because there is no ideal display device, existing devices further bound the detail that can be rendered for a particular model. Many graphics methods have been developed that exploit device limitations. Here we focus on extensible models that can be adjusted to take advantage of device limitations for computational efficiency, but which ultimately are only bounded in their application by the limits of human vision. 10