Communications of the ACM
Interactive immersion in 3D graphics
Physically Based Virtual Painting
For centuries, painters have used traditional media and tools to express their thoughts and feelings. The aesthetics of a painterly work and manipulation of materialspaint medium, brushes, knives, palettes, and supports like canvas and wood blocksare closely related. And the skills needed to manipulate these materials often have a direct effect on the finished work. However, most technological advances in computer painting systems have focused on the appearance of the resulting images, less on capturing the interaction with the paint materials. The results are often painting systems capable of creating stunning images but that employ unintuitive and unnatural interfaces requiring thick user manuals and long hours of training to learn to achieve the desired effects. Figure.
The word "painterly" also describes a fusion of feeling, action, sight, touch, intent, and purpose beyond merely producing an image that gives an artistic impression [11]. Therefore, rather than focusing solely on the rendered appearance of computer artwork, we have sought to recreate the sight, touch, action, and emotion of the artistic process itself. The goal is to design an expressive digital vehicle for interactively creating original painterly works in a setting that recreates the traditional painter's environment.
For the past several years, we've been adapting the design principle of physically based interaction to develop our prototype painting system called dAb [3] and realistic, interactive paint model called IMPaSTo [2]. This design philosophy aims to model elements of the user interface based on the laws of physics, so each element behaves and reacts as it would be expected to in the physical world. This paradigm naturally yields an intuitive user interface, since users readily transfer skills already acquired through their real-world experience.
The dAb computer painting system uses a novel physically based, deformable 3D virtual brush to give the user intuitive control of complex brush strokes similar to that of real physical brushes. It also offers haptic, or touch-enabled, feedback to further enhance the user's sense of realism and provide tactile cues to better manipulate the paintbrush (see Figure 1). dAb also features a novel bidirectional, two-layer paint model that enables easy loading of complex blends onto the 3D brush to facilitate the creation of expressive paint effects on a virtual canvas.
Paintings can be created with just the haptic stylus as physical metaphor for the brush handle and the space bar for bringing up or putting away the virtual palette.
The graphical user interface provides a minimalistic set of keystrokes and simple control, along with a great deal of expressive power. Complete paintings can be created with just the haptic stylus as physical metaphor for the brush handle and the space bar on the keyboard for bringing up or putting away the virtual palette. dAb enables users to paint directly onto a virtual canvas displayed on the screen, mix paint and clean brushes on the virtual palette, and pick from a wide selection of common brush types and shapes. A simple menu makes it easy to save and load a clean or partially painted canvas; additional digital functions include zoom, undo, and quick drying. Figure.
The dAb paintbrushes deform in a natural physical way, as the user moves a virtual brush across the canvas. The user creates strokes with the brush, which behaves much as a real brush would. The system generates brush footprints and resulting strokes based on the user's manipulation of the 3D brushes on the virtual canvas while delivering an artistic setting conceptually equivalent to a real-world painting environment. The dAb interactive painting experience provides direct interaction with the materials, as well as with the process of painting.
Modeling 3D Brushes
No item is "of greater importance to the successful execution of a painting than a sufficient quantity of the very highest-grade brushes that it is possible to find" [11]. A good set of brushes enables a competent artist to create virtually any effect imaginable, from the intricate detail of flower petals, to wispy billowing clouds, to the subtly blended shifting hues in a sunset. Brush heads are made from a variety of bristles, natural animal hairs, or synthetic fibers. Using the dAb system, users model some of the most commonly used types of brushes: rounds, flats, brights, and Filberts, as well as other types of specialty brushes (such as fans and blenders) [11].
The basic dAb brush head is modeled as a subdivision surface mesh [12] wrapped around a spring-mass particle system skeleton. This skeleton reproduces the basic dynamic behavior of a brush head, while the deformable skin mesh represents the actual shape of the brush head. We also derived a semi-implicit integration scheme that can take large time steps while maintaining stability. Though the physical model only approximates real-world physical interaction, it is designed to capture the essential properties of physical paintbrushes while preserving interactivity. We generalized this 3D brush model to generate several different common brush types and shapes (see the table on teh previous page).
Although the deformable 3D brush model of the dAb system is sufficient for recreating most desired brush stroke effects, it is not yet suitable for creating fine and detailed brush strokes. We are working on a new brush model capable of capturing the fine bristle effects of a physical paintbrush [1].
Another innovative aspect of dAb is its ability to provide force feedback that emulates the sensation of applying brush strokes to a canvas. We align the virtual paintbrush with the physical haptic stylus, positioning it so the point of force delivery coincides with the point where the head meets the handle on the virtual brush.
The force computation and the brush deformation are considered separately, the former for haptic display, the latter for graphical display, and each has different performance requirements. Consequently, dAb decouples the force simulation from the brush dynamics simulation, simplifying the force computation to run at the desired force update rates, typically in the range of 1,000Hz.
The basic force model is a simple piecewise linear function of the depth of brush penetration into the canvas based on Hooke's Law for elasticity. The spring constants can be changed to simulate brushes of varying stiffness. As frictional forces play a key role in the user's perception of surface contacts, dAb models the tangential force using a viscous model, or as a force proportional to and opposing the current brush velocity. dAb further augments this model to account for compression effects.
When a real brush contacts canvas at close to a right angle, the stiff bristles initially act as strong compressive springs, transmitting an abrupt force to the handle. As more pressure is applied, the bristles buckle, and the compressive force is reduced, as bending forces take over. When the brush makes contact at an oblique angle, compressive effects play a lesser role in the force felt by the user. Thus, dAb extends the piecewise linear force function to a piecewise Hermite curve, defined by a series of control points characterized by the amount of penetration, corresponding force magnitude, and spring stiffness constants at that point. The multi-segment Hermite curve is derived from empirical observation of how a brush head deforms under compression; Figure 2 shows the force function based on a multisegment Hermite curve for haptic display. This model exhibits a compressive force that is strongest when a brush contacts the canvas at a right angle and tapers off to zero as the brush approaches a 45-degree angle to the canvas.
We are currently developing an enhanced force-display model to capture the sensation of surface texture resulting from the accumulation of multiple layers of thick paint. We plan for dAb to account for the factors influencing the perception of surface roughness [7], as well as studies on issues contributing to the perceived instability of haptic texture rendering [4].
Full Range of Physical Behavior
Each medium used in painting reflects its own inherent characteristics. Viscous paint media (such as oil and acrylic) are popular among artists due to their versatility and ability to capture a range of expressive styles. They can be applied thinly in uniform layers to achieve deep, lustrous finishes, as in the work of Dutch genre painter Johannes Vermeer, or dabbed on thickly to achieve almost sculptural impasto effects, as in the works of French impressionist painter Claude Monet. With a scumbling technique, short choppy semiopaque brush strokes create a veil-like haze over previous layers [6].
How might an interactive model be designed to capture the full range of physical behavior of such paint? One way would be to attempt to solve the complex system of differential equations that describe viscous fluid. But it is very difficult to achieve interactive performance at high resolution using this approach. Rather than attempt to simulate paint through the exact equations, we aim instead to devise efficient approximations that capture the essential physical behavior of paint while developing heuristics that model any desired empirical behaviors not captured by the approximate physical model.
Paint is made by mixing finely ground pigments with a fluid vehicle [11]. Linseed oil is the vehicle typically used in oil paints, while in acrylics it is a polymer emulsion. Both are technically non-Newtonian fluids and, as such, are difficult to model mathematically. Basic properties of non-Newtonian fluids, like viscosity, change depending on various factors, breaking the assumptions of many fluid-solving systems. For example, the high viscosity of ketchup can be temporarily reduced by inducing a high shear rate (such as by shaking). Moreover, painting fluids interact with the complex, rough surfaces of the canvas and brush, requiring mathematics to deal with geometrically complex boundary conditions.
Paint also involves many stunning optical properties. The observed reflectance spectrum of paint results from a complex subsurface scattering and absorption phenomenon. The result is a rich, nonlinear behavior in the paint's perceived color, which depends on both the thickness of a layer of paint and the mixture of pigments being used. Both of these nonlinearities are absent in the linear, additive red-green-blue-alpha color model typically used in computer graphics. The interactive performance requirement for a computer painting system and the computational challenges in the mathematical modeling of paint media introduce strict constraints on and challenges in the design and implementation of a computational model for either an oil- or acrylic-like paint medium.
dAb paint model. dAb began in 2000, and the first version was completed in 2001 when we developed a simple paint model capable of achieving complex effects interactively. The dAb paint model incorporates variable wetness and opacity, conserves volume, and has a graphics-hardware-accelerated bidirectional paint-transfer algorithm. Even though it maintains complete interactivity, it supports a number of operations and painting techniques:
- Bidirectional transfer. Transferring paint from the brush to the canvas and back from the canvas to the brush;
- Complex brush loading. Filling the various portions of the brush head with different pigments;
- Blending. Mixing multiple pigments to obtain the desired color;
- Variable dryness. Controlling the blending of new paint onto previous layers by allowing paint to partially dry;
- Glazing. Painting with translucent layers (veils) of color over other opaque colors, or the underpainting; and
- Impasto. Painting with thick volumes of paint.
Other advanced computer painting systems have been able to generate similar end results by using complex image-compositing heuristics. With dAb, the user creates these effects directly through simple brush strokes in a manner similar to the manipulation of real brushes. However, the dAb paint model is based on simple, fast heuristics. Although it generates a pleasing, expressive painterly appearance, it does not exhibit the truly realistic physical behavior of thick paint or the optically correct composition of color pigments.
IMPaSTo paint model. Researchers have presented a variety of methods for simulating the physical properties of natural media. One group recently produced excellent results in its seminal work on watercolor simulation using a form of the shallow-water-and-diffusion equations (basically the Navier-Stokes equations with approximations to assume the level of the water is not deep) [5]. However, the formulation is specific to thin watery paint. More recently, we have developed a viscous paint model, called IMPaSTo, that captures a wider range of styles. It recreates paint media similar to oils and acrylics, yet is still fast enough for use in interactive painting systems. IMPaSTo includes both a numerical simulation to recreate the physical flow of paint and an optical model to mimic paint appearance. Figure
The IMPaSTo model allows one active wet layer and an unlimited number of dry layers.
IMPaSTo is derived from a conservative advection scheme that preserves both overall paint volume and pigment mass to simulate the basic dynamics of paint. The model is augmented with heuristics that model the remaining key paint properties. The model allows one active wet layer and an unlimited number of dry layers. Each layer is represented as both a height field and a set of per-pixel pigment concentrations.
IMPaSTo represents paintings in terms of paint pigments rather than as RGB colors, allowing users to dynamically relight paintings under any full-spectrum illuminant. IMPaSTo also incorporates an interactive implementation of the Kubelka-Munk (K-M) diffuse reflectance model [810], which captures the optical nonlinearities of real paint mixing, and uses a novel eight-component color space for greater color accuracy. Both the physical simulation and the rendering algorithms of IMPaSTo run as fragment programs on programmable graphics processors, an approach that allows for real-time interaction with the paint while calculating both the K-M reflectances and the dynamic lighting of the paint surface on the fly that would otherwise be difficult to achieve on a desktop PC.
We also incorporated the IMPaSTo paint model into the dAb painting system to demonstrate the capabilities of this physically based digital paint medium. We measured the full-spectrum reflectances of several oil paints commonly found on artists' palettes and imported them into dAb. Since we store paintings in terms of pigments rather than as colors, all pigments can be changed at any time to explore different possibilities (such as the effect of globally replacing one shade of green for another). The potential also exists for very accurate physical reproduction by using the recorded per-pixel pigment concentrations to create a hardcopy image using real paint.
Many novice users and amateur artists have now used dAb and IMPaSTo to create paintings with a variety of styles, effects, and paint textures. Since dAb and IMPaSTo provide a familiar artistic setting conceptually equivalent to a real-world painting environment, all of them were able to pick up the virtual paintbrush and begin painting almost immediately. Figure 3 shows original artwork created with dAb and IMPaSTo; more can be found at gamma.cs.unc.edu/dAb and gamma.cs. unc.edu/IMPaSTo
Most users who have worked with other painting systems have told us that dAb and IMPaSTo are more intuitive. Artists with prior painting experience have described dAb and IMPaSTo as substantially easier and simpler to adapt to than other painting systems, while offering similar digital advantages.
Visualizing Art
An artist can use dAb and IMPaSTo as a painting environment or as a practice tool for visualizing conceptual artwork. We intend to develop dAb and IMPaSTo training systems by adding a natural interface that takes advantage of skill transfer from traditional painting environments to a computer-painting program and vice versa. This real/virtual skill transfer is not possible with other existing computer painting systems. We also want to capture the paint strokes of master artists via our haptic interface and use them to train novice artists.
dAb and IMPaSTo might also be used for instruction. For example, they could help show how different types of brushes create different types of strokes, illustrate the reflection of light on various painted surfaces, and reveal how color pigments interact with one another. They could also allow artists and amateur painters alike to explore a variety of artistic concepts and creations. Moreover, they promise to allow audience members of varying physical abilities, including those with diminished motor function, to explore scientific and artistic creations in ways that would otherwise be impossible. Such physically based virtual painting systems will increasingly deliver new opportunities for a range of learners to discover the scientific processes and experiment with outcomes of computer-assisted artistic creation. Figure
References
1. Baxter, W. and Lin, M. A Versatile Interactive 3D Brush Model. Tech. Rep., University of North Carolina at Chapel Hill, 2004.
2. Baxter, W., Wendt, J., and Lin, M. IMPaSTo: A realistic, interactive model for paint. In Proceedings of the 3rd Symposium on Non-Photorealistic Animation and Rendering (Annecy, France, June 79). ACM Press, New York, 2004.
3. Baxter, W., Scheib, V., Lin, M., and Manocha, D. dAb: Interactive haptic painting with 3D virtual brushes. In Proceedings of ACM SIGGRAPH (Los Angeles, Aug. 1217). ACM Press, New York, 2001, 461468.
4. Choi, S. and Tan, H. An experimental study of perceived instability during haptic texture rendering: Effects of collision detection algorithm. In Proceedings of the 11th International Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems (Los Angeles, 2003), 197204.
5. Curtis, C., Anderson, S., Seims, J., Fleischer, K., and Salesin, D. Computer-generated watercolor. In Proceedings of ACM SIGGRAPH (Los Angeles, Aug. 38). ACM Press, New York, 1997, 421430.
6. Gair, A. The Beginner's Guide Oil Painting: A Complete Step-By-Step Guide to Techniques and Materials. New Holland Publishers, London, 1997.
7. Klatzky, R. and Lederman, S. Perceiving texture through a probe. In Touch in Virtual Environments, M. McLaughlin, J. Hespanha, and G. Sukhatme, Eds. Prentice Hall PTR, Upper Saddle River, NJ, 2002, 180193.
8. Kubelka, P. New contributions to the optics of intensely light-scattering material, Part II: Non-homogenous layers. J. Opt. Soc. 38 (1954), 330.
9. Kubelka, P. New contributions to the optics of intensely light-scattering material, Part I. J. Opt. Soc. 38 (1948), 448.
10. Kubelka, P. and Munk, F. Ein beitrag zur optik der farbanstriche. Z. Tech. Physik 12 (1931), 593.
11. Mayer, R. The Artist's Handbook of Materials and Techniques. The Viking Press, New York, 1970.
12. Schröder, P. and Zorin, D. Subdivision for modeling and animation. In ACM SIGGRAPH Course Notes. ACM Press, New York, 1998.
Footnotes
This research is supported in part by Intel Corp., National Science Foundation, Office of Naval Research, U.S. Army Research Office, and fellowships from Link Foundation, NVIDIA Corp., and the University of North Carolina at Chapel Hill. Any opinions, findings, and conclusions or recommendations expressed here are those of the authors and do not necessarily reflect the views of the research sponsors.
Figures
Figure 1. Haptic painting system components. In dAb, the artist uses a haptic stylus to paint directly on a virtual canvas.
Figure 2. The force function for haptic display.
Figure 3. Original artwork created using dAb and IMPaSTo: (right) Lily (based on a Georgia O'Keeffe painting) by Rebecca Holmberg and (left) Lady in Blue (based on an Edward Munch painting) by Heather Wendt.
Figure. Red Dragon by Sarah Hoff, created using dAb (University of North Carolina at Chapel Hill).
Figure. Real brushes, models for each of them (including skeletal structure and surface mesh), and example strokes they generate.
Figure. A painting handmade by William Baxter using the virtual canvas of dAb and IMPaSTo, after a painting by Vincent Van Gogh, Van Gogh's Room at Arles (University of North Carolina at Chapel Hill).
©2004 ACM 0002-0782/04/0800 $5.00
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. To copy otherwise, to republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee.
The Digital Library is published by the Association for Computing Machinery. Copyright © 2004 ACM, Inc.
No entries found