FIELD OF THE INVENTION

The invention relates generally to a method and apparatus for defining rigid, plastically deformable, and visco-elastically deformable surfaces in a haptic virtual realty environment and more specifically to a method and apparatus for haptically manipulating these surfaces within a virtual reality environment.

BACKGROUND OF THE INVENTION

Virtual Realty (VR) is an artificial environment constructed by a computer which permits the user to interact with that environment as if the user were actually immersed in the environment. Early VR devices permitted the user to see three dimensional (3D) depictions of an artificial environment and to move within that environment. Thus, a VR flight simulator incorporating such a device would allow a user to see a 3D view of the ground, which changed both as the virtual aircraft passed over the virtual ground and as the user's eyes looked in different directions. What the user saw in such a simulator is what a pilot would see when actually flying an aircraft.

The reality of the VR environment was enhanced by the ability of a user to manipulate virtual objects within the virtual environment using hand motions and gestures. Special gloves and devices were developed which permitted the user to interact with the virtual objects within the virtual environment. In such a system, the user typically saw an image of his or her hand within the virtual environment and was able to determine where in the virtual environment the user's hand was relative to the virtual object to be manipulated. Moving the glove or device resulted in a corresponding movement of the hand image in the virtual environment. Thus a user wearing the special gloves or using the special device would cause virtual objects to move, simply by moving the glove in such a way that the virtual object is touched by the image of the hand in the virtual environment.

The addition of force generators to the gloves or devices further enhanced the reality of VR by providing the user with a tactile response to interacting with virtual objects within the virtual environment. For example, a user moving such a force generation enhanced device in a direction such that the image of the device moved toward a virtual wall in the virtual environment, would experience a stopping force when the image of the device in the virtual environment collided with the virtual wall. Such tactile sensation creating systems, or haptic systems, thus provided a stimulus to another of the user's senses.

Although the early VR systems were oriented to providing realistic graphical interfaces, the progress in haptic VR systems has made it possible to define a haptic VR environment which may be completely independent of the graphical VR environment. As such, haptic VR environments may now be constructed which respond to manipulation by the user in the way that the early graphical VR environments responded to the visual actions of a user.

This ability to define a haptic VR space independently of a graphical, or other space, provides a greater degree of flexibility in the design and creation of such VR environments. The present invention seeks to further add to this flexibility.

SUMMARY OF THE INVENTION

Disclosed are methods for haptically deforming a virtual surface within a haptic virtual environment According to one embodiment of the invention, a haptic interactive representation including a virtual deformable surface is generated in a haptic interaction space. The virtual deformable surface may be represented by a mesh of polygons such as triangles. A position of a user in real space is sensed, for example using a haptic interface device. Next a haptic interface location is determined in the haptic interaction space in response to the position of the user in real space and a determination is made whether the virtual surface collides with the haptic interface location In the event the virtual surface does not collide with the haptic interface location, a first force is calculated and applied to the user in real space in response to this haptic interface location. The force may be nominally zero. Alternatively, if the virtual surface is determined to collide with the haptic interface location, an interaction force between the virtual surface and the user is calculated. In the event the calculated interaction force exceed a predetermined threshold force, the virtual surface is deformed and a force is calculated and applied to the user in real space in response to the interaction force.

In alternative embodiments, the step of generating a haptic interactive representation includes a virtual plastically deformable surface or a virtual visco-elastically deformable surface. According to the former method, the invention may be used to model permanently deformable surfaces in the development of industrial designs such as automobile body panels. According to the latter method, the invention may be used to model resilient compliant surfaces such as human body parts in the training of physicians and surgeons.

BRIEF DESCRIPTION OF THE DRAWINGS

This invention is pointed out with particularity in the appended claims. The above and further advantages of this invention may be better understood by referring to the following description taken in conjunction with the accompanying drawings, in which:

FIG. 1A

is a top level flowchart representation of one embodiment of a method for determining forces to be applied to a user through a haptic interface system;

FIG. 1B

is a flowchart of one embodiment of a method for generating a haptic interactive representation including a plastically deformable surface;

FIG. 1C

is an embodiment of a haptic interface system for enabling a user to generate and interface with a deformable surface in a haptic virtual reality environment;

FIG. 1D

is a schematic representation of a virtual object composed of a group of virtual surfaces for interaction with a user tool through a haptic interface;

FIG. 2

is a flowchart of one embodiment of a graphics process;

FIGS. 3A-3C

are a flowchart of one embodiment of a haptic rendering process;

FIG. 4

is a flowchart of one embodiment of a surface interaction process;

FIG. 5

is a flowchart of one embodiment of a method for finding a first penetrating vertex of a triangular mesh forming a virtual surface upon collision with a virtual tool;

FIGS. 6A-6C

are a flowchart of one embodiment of a method for determining the interaction of the virtual tool with the virtual surface when the tool mode is set to rigid interaction;

FIG. 7

is a flowchart of one embodiment of a process for backing the virtual tool out of the virtual surface;

FIG. 8

is a schematic representation of vectors calculated in the surface interaction process;

FIG. 9

is a flowchart of one embodiment of a method for determining the interaction of the virtual tool with the virtual surface when the tool mode is set to deform;

FIG. 10

is a flowchart of one embodiment of a method for determining all of the vertices of the virtual surface which penetrate the volume of the virtual tool;

FIG. 11A

is a flowchart of a method for moving a vertex of the virtual surface which penetrates the volume of the virtual tool to the surface of the virtual tool;

FIG. 11B

is a flowchart of another method for moving a vertex of the virtual surface which penetrates the volume of the vial tool to the surface of the virtual tool;

FIGS. 12A-12C

are a flowchart of a method for remeshing the virtual surface;

FIG. 12D

is a schematic representation of parent and child edges of abutting triangles;

FIG. 13A

is a schematic representation of three types of triangles;

FIG. 13B

is a listing of pseudocode for one embodiment of a method for classifying the triangles depicted in

FIG. 13A

;

FIG. 14A

is a listing of pseudocode for one embodiment of a method for subdividing a triangle one way;

FIGS. 14B-14G

are schematic representations of triangles being subdivided one way;

FIG. 15A

is a listing of pseudocode for one embodiment of a method for subdividing a triangle three ways;

FIGS. 15B-15D

are schematic representations of triangles being subdivided three ways;

FIG. 16

is a flowchart of another embodiment of a method for generating a haptic interactive representation including a visco-elastically deformable surface;

FIG. 17

is a flowchart of another embodiment of a graphics process for a visco-elastically deformable surface;

FIGS. 18A-18B

are a flowchart of a haptic interaction process for a viscoelastically deformable surface;

FIG. 19

is a flowchart of another embodiment of a method for determining all of the vertices of the virtual surface which penetrate the volume of the virtual tool;

FIG. 20

is a flowchart of an embodiment of a method for determining if each vertex within a binary spatial partitioning (BSP) tree and within a bounding sphere around the virtual tool collides with the virtual tool;

FIG. 21

is a schematic representation of the virtual tool modeled as a sphere with a penetrable outer core and a rigid inner core.

FIG. 22

is a flowchart of an embodiment of a method for adding a vertex to a list of vertices to be updated;

FIG. 23

is a flowchart of an embodiment of a method for updating the state of the virtual surface mesh,

FIGS. 24A-24B

are a flowchart of an embodiment of a method for calculating the forces on a vertex due to the spring/dashpots incident on the vertex;

FIG. 25

is a flowchart of a method for updating the state of the vertices;

FIG. 26

is a flowchart of a method for calculating the force to be applied to a user;

FIGS. 27A-27B

depict graphical representations of a virtual object in the form of a cube and an ellipsoidal virtual tool prior to and after plastic deformation of the cube; and

FIGS. 28A-28B

depict graphical representations of a virtual object in the form of a human heart and a spherical virtual tool prior to and during visco-elastic deformation of the heart.

Like reference characters in the respective drawn figures indicate corresponding elements.

DETAILED DESCRIPTION OF THE INVENTION

In brief overview, the flowchart of

FIG. 1A

shows the general steps performed by one embodiment of the method of the present invention for determining the forces to be applied to a user through a haptic interface system. Haptic interface systems include a haptic interface device and a haptic rendering process which generates a model of the environment to be “touched” and determines the forces to be applied to the user. A haptic interface device is a tactile or force-feedback device which provides the touch sensations of interacting with virtual objects. Known haptic interface devices consist of an electromechanical linkage which can exert a controllable force on a user's hand. As used herein, “haptic rendering” is defined as the creation of a virtual environment with which a user can interact through the sense of touch. “Haptic rendering process” refers to the computer application which generates the virtual environment and determines the forces to be applied to a user through a haptic interface. The haptic rendering process generates representations of real world objects in the virtual environment. The model of the environment is a computer generated representation of the real world environment. The haptic rendering process determines the forces to be applied to the user based on the model environment.

In step

1

, the haptic rendering process generates a haptic interactive representation which includes a representation of a real world object. As used herein, a “haptic interactive representation” is defined as a computer generated virtual environment which can be explored by a user through the sense of touch. As used herein, “virtual object” is defined as the representation of a real world object in the haptic interactive representation. In the embodiment illustrated by the flowchart in

FIG. 1A

, the haptic interactive representation contains only one virtual object. The haptic interactive representation may contain more than one object.

In step

2

, sensors of the haptic interface system sense the position of the user in real space. As used herein, “real space” is defined as the real world environment. In step

3

, the haptic rendering application utilizes the information obtained by the sensors to determine the haptic interface in the virtual environment. As used herein, “haptic interaction space” is defined as the region in the computer generated virtual environment with which the user can interact through the sense of touch. The haptic interaction space defines the boundaries of the haptic interactive representation with which the user can interact. The location of the haptic interface describes the position of the user in the haptic interaction space. In step

4

the haptic rendering application calculates the force to be applied to the user in real space in response to the haptic interface location. In this step, the haptic rendering application determines whether the virtual object collides with the haptic interface location. A virtual object “collides” with the haptic interface location if the surface of the virtual object is in contact with the haptic interface location or if the virtual object encompasses the haptic interface location. If the haptic interface location collides with the virtual object, the properties of the virtual object along with the ambient properties of the haptic interaction space affect the force calculation. If the haptic interface location does not collide with the virtual object, the ambient properties of the haptic interaction space affect the force calculation. After the haptic rendering application has calculated the force to be applied to the user, in step

5

this force may be generated and applied to a user in real space through a haptic interface device. After the force has been applied to the user, the haptic rendering application repeats steps

2

,

3

,

4

, and

5

for the duration of time that the user interacts with the haptic interactive representation.

The flowchart of

FIG. 1B

shows the steps performed by one embodiment of the method of the present invention for generating a haptic interactive representation including a deformable surface and enabling a user to haptically interact with and deform the deformable surface. As described above, virtual reality (VR) environments may include both a graphical VR environment and a haptic VR environment. The VR environment created by the embodiment illustrated by the flowchart in

FIG. 1B

provides both graphical and haptic feedback to a user. In other embodiments, the method may only provide haptic feedback to the user.

Classifications of deformable surfaces include plastically deformable surfaces and visco-elastically deformable surfaces. A plastically deformable surface retains the deformed shape after being deformed. A visco-elastically deformable surface may return partially or entirely to its original shape after being deformed. Additionally, a type of visco-elastically deformable surface termed an elastically deformable surface exhibits the resilience of a visco-elastically deformable surface but none of the damping thereof. In the embodiments described below, the viral plastically-deformable surfaces and the virtual visco-elastically deformable surfaces are composed of a triangular mesh. In other embodiments, the virtual surfaces may be composed of other polygonal meshes or may be defined by algebraic equations. In general, the virtual surfaces may be combines to form a virtual object having a volume. Also, in the embodiments described below, the user interacts with the virtual surfaces in the virtual environment through a volume referred to as a tool. The user may select any volume shape for the tool. The shape of the volume or tool determines how the virtual surface is deformed. The tool may be represented as a series of discrete points in vial space which form the volumetric shape of the tool.

FIG. 1D

shows an example of a virtual surface

32

and a “tool”

56

. The tool represents the user in the virtual world.

In the embodiment illustrated by the flowchart in

FIG. 1B

, the steps are performed by application software running on a computer. The application software includes a haptic rendering process, a graphics process and a surface interaction process. These three processes run simultaneously on the computer. The graphics process generates a visual display of the VR environment to the user and updates the locations of the virtual objects on a display. In one embodiment, the haptic rendering process is a computer application which generates the virtual environment and determines the forces to be applied to a user through a haptic interface. The haptic rendering process generates representations of real world objects in the virtual environment. The surface interaction process determines if the user collides with a deformable surface within the haptic interactive representation and determines how the surface should be deformed, if at all.

FIG. 1C

shows an embodiment of a haptic interface system for enabling a user

10

to generate and interface with a deformable surface in a haptic virtual reality environment. The apparatus includes a haptic interface device

12

, application software

14

running on a computer and a display

16

. As described above, a haptic interface device is a tactile or force-feedback device used by a user which provides the touch sensations of interacting with virtual objects. Known haptic interface devices consist of an electro-mechanical linkage which can exert a controllable force on a user's hand. One haptic interface device which may be used with the method of the present invention is the haptic interface device described in U.S. Pat. Nos. 5,587,937 and 5,625,576, which are incorporated herein by reference. The application software

14

includes a graphics process

18

, a haptic rendering process

20

and a surface interaction process

22

described above. The application software

14

may also include an audio process

24

for driving a speaker

26

. The haptic interface device

12

includes a sensor for sensing the position of the user

10

in real space and a force actuator for applying forces to the user

10

in real space. In other embodiments, the graphics process, the haptic rendering process, the surface interaction process, and the audio process are executed by different processors executing algorithms representing these processes.

The haptic rendering process

20

is in communication with the haptic interface device

12

. The haptic rendering process

20

executes an algorithm to generate a haptic interactive representation and determine the forces to be applied to the user

10

in real space. In one embodiment, the algorithm includes a module generating a haptic interactive representation, a module determining the user's haptic interface location in haptic interaction space, and a module calculating a force to be applied to the user in real space. The module determining the user's haptic interface in haptic interaction space translates the position of the user in real space into a position in haptic interaction space. The module calculating a force to be applied to the user determines whether the user's haptic interface location collides with any of the virtual objects. If the user's haptic interface location does not collide with any virtual objects, then this module calculates a force to be applied to the user in response to the user's haptic interface location. If the user's haptic interface location does collide with at least one virtual object then this module calculates a force to be applied to the user in response to the collision with the virtual objects. The haptic interface device

12

produces the force calculated by the haptic rendering process

20

and applies the calculated force to the user

10

.

The graphics process

18

is in communication with the haptic rendering process

20

and the surface interaction process

22

. The graphics process

18

displays the representation of the virtual object created by the haptic rendering process on a display

16

. The display

16

may be a computer screen, television screen, or any other device known in the art for displaying images of objects. In one embodiment, the apparatus also includes the audio process

24

and the speaker

26

. The audio process

24

is in communication with the haptic rendering process

20

. The audio process

24

produces audio sounds in response to the interactions of the virtual objects and the user in the virtual environment. The speaker

26

is in electrical communication with the audio process

24

. The speaker

26

outputs the sounds produced by the audio process

24

.

In one embodiment, the haptic rendering process

20

, the surface interaction process

22

, the graphics process

18

and the audio process

24

are executed by different processors. In another embodiment, the haptic rendering processor

20

, the display processor

18

and the audio processor

24

are executed by the same processor. In yet another embodiment, the module generating a haptic interactive representation, the module determining the user's haptic interface location in haptic interaction space, and the module calculating a force to be applied to the user in real space are separate devices.

Referring again to

FIG. 1B

, in step

28

, the application software initializes the graphics process and the haptic rendering process. In step

30

, the application software either creates a new virtual surface or loads a virtual surface for interaction with a user. As used herein, “virtual surface” is defined as the representation of a real world surface in the haptic interactive representation. The haptic interactive representation may contain only one virtual surface or a plurality of virtual surfaces. The virtual surfaces may be combined to form virtual objects. A “virtual object” is defined as the representation of a real world object in the haptic interactive representation.

FIG. 1D

shows an example of a virtual object

32

composed of a group of virtual surfaces for interaction with a user through a haptic interface. The virtual object

32

is a cube. In other embodiments, other virtual objects or virtual surfaces may be used.

After the initial virtual surface, i.e. virtual object

32

, is loaded, the application software initiates the graphics process, haptic rendering process and surface interaction process (step

34

). The graphics process, haptic rendering process and surface interaction process will be described in more detail below. Next, the application software proceeds to step

36

. The application software repeats steps

36

,

37

,

38

and

39

until the user requests to quit the application. In step

37

, the application software waits for an input by the user. Once the application software receives an input from the user, the application software proceeds to step

38

and performs the appropriate action based on the type of operation the user selected. Step

38

illustrates certain operations that the user may select. In other embodiments, the user may have less or more options from which to select. The operations the user may select in the embodiment illustrated in

FIG. 1B

include: save the existing surface to a file (step

40

), quit the application (step

42

), select a tool for interacting with the virtual surface (step

44

), scale the size of the selected tool (step

46

), select the mode of the tool (step

48

), select the deform mode of the virtual surface (step

50

), and rotate, scale or translate the virtual surface (step

52

). Each of these options will be described in more detail below.

One of the options the user may select is to save the surface to a file (step

40

). In this step, the application software saves the current shape of the virtual object

32

to a file. A second option the user may select is to quit the application (step

42

). If the user selects this option, the application software proceeds through step

39

to step

54

and stops the graphics process, haptic rendering process and surface interaction process. Another option the user may select is to select the tool for interacting with the virtual surface (step

44

). A user selects this option to determine the shape of the virtual object that will represent the user in the haptic interactive representation. For example, in the embodiment shown in

FIG. 1D

, the user is represented by a virtual sphere

56

. The user's motions in real space will be followed by the sphere's motions in virtual space. The user can select any shape to interact with the virtual surface. For example, the user may select that the tool be represented by a virtual cube or a virtual toroid. The user may change the tool which represents the user at any time while the application software is running.

The user may also select to scale the size of the selected tool (step

46

). The user may scale the tool uniformly so that all the dimensions of the tool scale at the same rate. The user may also scale the tool non-uniformly such that different dimensions scale at different rates. For example, in the embodiment shown in

FIG. 1D

, the user may select to scale the virtual sphere

56

uniformly such that the virtual sphere

56

increases in diameter. Alternatively, the user may select to scale the virtual sphere

56

non-uniformly such that the virtual sphere

56

increases only in the x-direction and becomes oblong-shaped or ellipsoidal. The user may scale the size of the selected tool (step

46

) at any time while the software application is running.

The user may further select the mode of the tool (step

48

). Each virtual tool

56

has three modes: rigid, deform, and no interaction. If the user selects rigid mode, the virtual tool

56

will interact with the virtual object

32

as a solid surface and will not deform any of the surfaces of the virtual object

32

. If the user selects deform mode, the virtual tool

56

may be able to deform surfaces of the virtual object

32

, depending on the characteristics of the surfaces of the virtual object

32

. Finally, if the user selects the no interaction mode, the virtual tool

56

will not interact with the virtual object

32

, but will simply pass through the virtual object

32

upon collision with the virtual object

32

.

In addition the user may select the deform mode of the virtual surface (step

50

). Each virtual surface has two deform modes: user input mode and force threshold mode. If the user selects the user input mode, the deformability of the virtual surface is determined by a user input. In this mode, the user must make an explicit action to change the state of the surface from non-deformable to deformable. For example, the user may press a switch on the haptic interface device, press a predetermined key on a keyboard which interfaces with the application software, input a voice command or perform any other explicit input action to change the state of a surface from non-deformable to deformable. If the user selects the force threshold mode, the virtual surface will not deform until the force that the user is applying to the virtual surface through the haptic interface device exceeds a predetermined threshold.

Finally, the user may select to rotate, scale or translate the virtual surface (step

52

). For example, the user may select to rotate, translate or scale the size of the virtual cube

32

. As described above in other embodiments, the user may have more, less or different options.

The application software repeats steps

36

,

37

,

38

and

39

until the user selects the quit option (step

42

). As described above, once the user selects the quit option (step

42

), the application software proceed through step

39

to step

54

and stops the graphics process, the haptic rendering process and the surface interaction process. The application software then proceeds to step

55

and exits.

In more detail and referring to

FIG. 2

, a flowchart illustrates a more detailed sequence of steps performed by the graphics process

18

in one embodiment of the present invention to display to the user the interaction of the user with virtual objects in the haptic interactive representation. The graphics process

18

draws a representation of the virtual surface (step

58

) and a representation of the tool representing the user (step

60

). For example, the graphics process repeatedly draws a representation of the virtual cube

32

and virtual sphere

56

and displays these representations to a user on a display. To draw the representations of the virtual surface and virtual tool, the graphics process determines the current position, orientation, and shape of the virtual surface and virtual tool. Implicit surfaces may be used, for example, a sphere primitive from a conventional graphics library as known by those skilled in the art The graphics process repeats these steps until the application software stops the graphics process (step

54

above). In one embodiment, the graphics process repeats these steps at a rate of 30 Hz. In other embodiments, the graphics process may repeat these steps at faster or slower rates.

In more detail and referring to

FIGS. 3A

,

3

B and

3

C, a flowchart illustrates a more detailed sequence of steps performed by the haptic rendering process

20

in one embodiment of the present invention to determine the force to be applied to the user through a haptic interface device

12

. In step

62

, sensors of the haptic interface system sense the position of the user in real space. As used herein, “real space” is defined as the real world environment. The haptic rendering application utilizes the information obtained by the sensors to determine the haptic interface in haptic interaction space. As used herein, “haptic interaction space” is defined as the region in the computer generated virtual environment with which the user can interact through the sense of touch. The haptic interaction space defines the boundaries of the haptic interactive representation with which the user can interact. As used herein, a “haptic interface” (HI) is defined as the location and orientation of the physical haptic interface device as sensed by the encoders of the VR system translated into the haptic interaction space. The location of the haptic interface describes the position of the user in the haptic interaction space.

Next, in step

64

, the haptic rendering process sets the direction of the virtual tool (tool

direction

), i.e. virtual sphere

56

, equal to the displacement between the position of the haptic interface (HI

position

) and the current tool position (tool

position

). The vector tool direction is a vector which points in the direction that the virtual tool needs to move in order to correspond with the current position of the haptic interface. In step

66

, the haptic rendering application determines the mode the user has selected for the tool. As described above, the tool has three possible modes: rigid, deform, and no interaction. If the user has selected the no interaction mode, the haptic process proceeds to step

68

. As described above, when the user selects the no interaction mode, the virtual tool does not interact with the vital surface. If the virtual tool “collides” with the virtual surface, the virtual tool simply passes through the vial surface. In step

68

, the haptic process sets the position of the virtual tool (tool

position

) equal to the position of the haptic interface (HI

position

) and the orientation of the virtual tool (tool

orientation

) equal to the orientation of the haptic interface (HI

orientation

). As the virtual tool will not interact with the virtual surface, the force (HI

force

) applied to the user through the haptic interface device will be set to a default value (step

70

). In one embodiment, the default value is zero Newtons. In other embodiments, the default value may be set to another value to simulate a force being applied to the user by the virtual environment such as gravity or another ambient condition. Once the haptic process has calculated the force to be applied to the user, the haptic process sends the calculated force (HI

force

) to the haptic interface device (step

72

). The haptic interface device then applies the calculated force to the user.

If the user has selected the deform or rigid mode, the haptic process proceeds to step

74

. As described above, if the user selects rigid mode, the virtual tool will interact with the virtual object as a solid surface and will not deform any of the surfaces of the virtual object. If the user selects deform mode, the virtual tool may be able to deform surfaces of the virtual object, depending on the characteristics of the surfaces of the virtual object. In step

74

the haptic process determines the deform mode of the virtual surface. As described above, each virtual surface has two deform modes: user input mode and force threshold mode. If the user selects the user input mode, the deformability of the virtual surface is determined by a user input. In this mode, the user makes an explicit action to change the state of the surface from non-deformable to deformable. For example, the user may press a switch on the haptic interface device, press a predetermined key on a keyboard which interfaces with the application software, input a voice command or perform any other explicit input action to change the state of a surface from nondeformable to deformable. If the user selects the force threshold mode, the virtual surface will not deform until the force that the user is applying to the virtual surface through the haptic interface device exceeds a predetermined threshold.

If the user has selected the user input mode, the haptic process proceeds to step

76

. In step

76

, the haptic process determines if the user has input the appropriate command that specifies that the tool is to deform the surface. For example, if the command is a key sequence on the keyboard, the haptic process determines if the user has input the key sequence; if the input is to press a switch on the haptic interface device, the haptic process determines if the user has pressed the switch. If the user has not input the appropriate command, the haptic process sets the mode of the virtual tool to rigid and the virtual tool will not deform the virtual surface, but will interact with the virtual surface as though the virtual surface is solid (step

78

). In step

80

, the haptic process sets the anchor position (anchor

position

) equal to the position of the virtual tool (tool

position

). The anchor position is used by the haptic process to calculate the force applied to the tool due to a collision with the virtual object. The anchor position may be defined as any fixed point from which a change in position may be used to calculate a spring force and from which a velocity may be used to calculate a damping force, as will be discussed in greater detail below. In step

82

, the haptic process calculates the tool interaction force (TIF

force

) using the anchor position (anchor

position

) and the state of the haptic interface device. In one embodiment, the haptic rendering process uses Hooke's law to calculate the tool interaction force as illustrated by equation (1) below, in which SPRING_CONSTANT is the stiffness of the virtual surface and HI

position

is the position of the haptic interface.

TIF

force

=SPRING_CONSTANT(anchor

position

−HI

position

)  (1)

Once the tool interaction force (TIF

force

) has been calculated, the haptic process sets the force to be applied by the user by the haptic interface device (HID

force

) equal to the tool interaction force (TIF

force

) (step

84

). Finally, the force to be applied to the user by the haptic interface device (HID

force

) is sent to the haptic interface device and is applied to the user (step

72

).

Referring again to step

76

, if the user has selected the user input mode as the tool mode, and has also input the appropriate command to cause the virtual tool to be in deform mode, the haptic process proceeds to step

86

. In step

86

, the haptic process determines if the amount that the virtual tool must move to intersect with the position of the haptic interface is greater than a predetermined stepsize (STEPSIZE) according to equation (2) below.

|tool

position

−HI

position

|<STEPSIZE  (2)

In one embodiment, the predetermined stepsize is about 0.25 mm. The stepsize may be selected to have a value small enough that it is not haptically perceivable and there will be no change in force detected for minor errors in calculation.

If the magnitude of the difference in position between the tool position tool

position

) and the haptic interface position (HI

position

) is less than the predetermined stepsize, the haptic process sets the current step size equal to the magnitude of the difference in position between the tool position (tool

position

) and the haptic interface position (HI

position

) (step

88

). If the magnitude is greater than the predetermined stepsize, the haptic process sets the current step size equal to the predetermined step size (step

90

). The purpose of steps

86

,

88

, and

90

is to ensure that the tool does not move more than a predetermined amount during any cycle to ensure that substantially all collisions are detected.

Once the size of the step that the virtual tool may take is set (currentStepSize) either in step

88

or step

90

, the haptic process sets the mode of the virtual tool to deform and allows the tool to deform the virtual surface when the tool collides with the virtual surface (step

92

). As the haptic process interacts with the surface process communicating where the virtual tool is being directed, the surface process checks for any collisions with the virtual surface and communicates with the haptic process in the event of a collision. Next the haptic process calculates the next position (P

deformation

) that the tool will move to according to equation (3) below in which tool

position

represents the current tool position, currentStepSize represents the magnitude of the distance the virtual tool may move, and tool

direction

represents a unit vector in the direction that the virtual tool must move to intersect the haptic interface (step

94

). The calculation of the vector tool

direction

was described above in the discussion of step

64

.

P

deformation

=tool

position

+currentStepSize*tool

direction

  (3)

In steps

96

and

98

, the haptic process sets the anchor position equal to the next position that the virtual tool will move to (P

deformation

) and calculates the tool interaction force using the anchor position and the state of the haptic interface device according to equation (1) above. Once the tool interaction force (TIF

force

) has been calculated, the haptic process sets the force to be applied by the user by the haptic interface device (HID

force

) equal to the tool interaction force (TIF

force

) (step

100

). Finally, the force to be applied to the user by the haptic interface device (HID

force

) is sent to the haptic interface device and is applied to the user (step

72

).

Referring again to step

74

in

FIG. 3A

, if the user selects the tool mode to be the force threshold mode, the haptic process proceeds to steps

102

and

104

. As described above, if the user selects the force threshold mode, the virtual surface will not deform until the force that the user is applying to the virtual surface through the haptic interface device exceeds a predetermined threshold. In steps

102

and

104

, the haptic process sets the anchor position equal to the position of the virtual tool (tool

position

) and calculates the tool interaction force using the anchor position and the state of the haptic interface device according to equation (1) above.

In step

106

the haptic process determines if the surface interaction process has signaled that it is ready to receive a new position for the tool to move to and deform the virtual surface. If the surface interaction process is not ready, the mode of the virtual tool is set to rigid (step

108

) and the haptic process sets the force to be applied by the user by the haptic interface device (HID

force

) equal to the tool interaction force (TIF

force

) previously calculated (step

110

). In step

110

, the tool interaction force (TIF

force

) is based on the previous position the virtual tool was to move to. Finally, the force to be applied to the user by the haptic interface device (HID

force

) is sent to the haptic interface device and is applied to the user (step

72

).

If the surface interaction process is ready to receive a new position for the tool (P

deformation

) in step

106

, the haptic process determines if the forces applied by the haptic interface device is greater than the predetermined threshold force for deforming the virtual object (step

112

). In one embodiment, the predetermined threshold force may be set at about two Newtons; however, the threshold force may be set as low as zero or as high as six Newtons or higher, as desired. If the haptic interface device has not exceeded the threshold force, the virtual surface remains rigid and the haptic process sets the next position for the virtual tool to be in (P

deformation

) equal to the current position of the tool (tool

position

) (step

114

) and proceeds through steps

108

,

110

and

72

as described above.

If the haptic interface device has exceed the threshold force, the virtual surface may be deformed and the haptic process calculates the next position (P

deformation

) in step

116

that the tool will move to according to equation (3) described above. In steps

118

and

120

, the haptic process sets the anchor position equal to the next position that the virtual tool will move to (P

deformation

) and calculates the tool interaction force using the anchor position and the state of the haptic interface device according to equation (1) above. The haptic process next checks if the tool interaction force is less than the force threshold for deforming the virtual surface (step

122

). If the tool interaction force is greater than the force threshold, the haptic process sets the tool mode to deform and allows the virtual tool to deform the virtual surface further (step

124

). The haptic process also sets the force to be applied to the user through the haptic interface device equal to the tool interaction force (step

110

) and sends this force to the haptic interface device to be applied to the user (step

72

).

If the tool interaction force (TIF

force

) is greater than the force threshold for deforming the virtual surface in step

122

, the haptic process sets the mode of the virtual tool to deform and allows the virtual surface to be deformed further (step

126

). The haptic process also determines the next position (P

deformation

) toward which the virtual tool will move. P

deformation

is set such that the tool interaction force to be applied to the user is equal to the force threshold for deforming the virtual surface (step

128

). In one embodiment, the haptic process calculates P

deformation

according to equation (4) below in which P

deformation

represents the next position to which the tool will move, HI

position

represents the position of the haptic interface, DEFORM

force

—

threshold

represents the threshold force for the virtual surface to deform, tool

spring

—

constant

represents the stiffness of the virtual tool and tool

position

represents the position of the virtual tool.

P

deformation

=HI

position

−(DEFORM

force

—

threshold

/tool

spring

—

constant

)*(HI

position

−tool

position

)  (4)

Next, the haptic process sets the force to be applied to the user (HID

force

) equal to the threshold force for deforming the virtual surface (step

130

) and sends this force to the haptic interface device to be applied to the user (step

72

).

After the force to be applied to the user (HID

force

) has been sent to the haptic interface device, the haptic process proceeds to step

132

and repeats the entire process outlined by the flowchart in

FIGS. 3A

,

3

B and

3

C. In one embodiment, the haptic process repeats the process at a rate of 1 KHz.

In more detail and referring to

FIG. 4

, a flowchart illustrates a more detailed sequence of steps performed by the surface interaction process

22

in one embodiment of the present invention to deform the virtual surface in response to interactions with the virtual tool. Before the surface interaction process attempts to move the position of virtual tool, the surface interaction process will obtain the current orientation of the haptic interface device from the haptic process described above and will attempt to orient the virtual tool toward the same orientation as the haptic interface device (step

134

). After orienting the virtual tool in the same orientation as the haptic interface device, the surface interaction process searches for the first vertex (V

i

) of the virtual surface that penetrates the volume of the virtual tool (step

136

). An embodiment of a method for determining the first vertex which penetrates the volume of the virtual tool will be described in detail below in the discussion of FIG.

5

. The surface interaction process will check the vertices of the virtual surface until a penetrating vertex is found. Once a single penetrating vertex is found, the surface interaction process will proceed to step

138

and will not check all of the vertices composing the virtual surface. The surface interaction process will also proceed to step

138

once it checks all the vertices of the virtual surface and does not locate a penetrating vertex.

In step

138

the surface interaction process determines if a penetrating vertex V

i

was located. If the surface interaction process locates a penetrating vertex V

i

, the surface interaction process sets the current orientation of the virtual tool (tool

orientation

) to the orientation of the tool during the previous cycle (step

140

). The surface interaction process then determines the mode of the virtual tool which was set by the haptics process

20

described above in the discussion of

FIGS. 3A

,

3

B, and

3

C (step

142

). If the mode of the virtual tool is set to no interaction, the surface interaction process sets the position of the virtual tool (tool

position

) equal to the position of the haptic interface (HI

position

) and the orientation of the virtual tool (tool

orientation

) equal to the orientation of the haptic interface device ((HI

orientation

) (step

144

). The surface interaction process then repeats steps

134

,

136

,

140

, and

142

.

If the mode of the virtual tool is set to rigid, the surface interaction process proceeds to step

148

and determines the interaction of the virtual tool with a rigid surface. The rigid interaction will be discussed below in the discussion of

FIGS. 6A

,

6

B, and

6

C. After the surface interaction process determines the rigid interaction, it repeats steps

134

,

136

,

140

, and

142

.

If the mode of the virtual tool is set to deform, the surface interaction process proceeds to step

150

and determines the deformation tool interaction. In this step, the surface interaction process determines how the virtual tool interacts with and deforms the virtual surface. An embodiment of a method for determining the interaction and deformation of the virtual surface will be described below in the discussion of FIG.

9

. Next, in step

152

, the surface interaction software remeshes the virtual surface as necessary. An embodiment of a method for remeshing the virtual surface will be described below in the discussion of

FIGS. 12A

,

12

B,

12

C, and

12

D. After the surface interaction process has remeshed the virtual surface, it repeats steps

134

,

136

,

140

and

142

(step

146

).

In more detail and referring to

FIG. 5

, a flowchart illustrates a more detailed sequence of steps performed by the surface interaction process in one embodiment of the present invention to find the first vertex of a triangular mesh forming the virtual surface. In one embodiment, the surface interaction process creates a binary spatial partitioning (BSP) tree which holds the vertices of the triangular mesh forming the virtual surface. Other spatial partitionings may be used to hold the vertices such as an octree or quadtree as known by those skilled in the art.

The surface interaction process also creates a bounding sphere around the virtual tool. The bounding sphere is the smallest sphere in which the virtual tool may be disposed. In step

154

, the surface interaction process interacts the bounding sphere surrounding the tool the with BSP tree. The surface interaction process then checks each vertex of the virtual surface which is in the BSP tree and within the bounding sphere to determine if the vertex is also within the volume of the virtual tool (repeat loop including steps

156

,

158

,

160

). In one embodiment of the method of the present invention, the volume of the virtual tool is defined by an implicit equation. In an alternative embodiment, the virtual surfaces or the virtual object may be defined by one or more implicit equations and the virtual tool may be defined as a polygonal mesh.

To determine if the vertex of the virtual surface is within the volume of the virtual tool, the implicit equation is evaluated for the coordinate of the vertex. If the solution to the equation is less than zero, the vertex is within the tool volume. If the solution to the equation is greater than zero, the vertex is outside the tool volume. Other embodiments may use a voxel approach. Once the surface interaction process finds a vertex V

i

within the volume of the tool, the surface interaction process exits the repeat loop and proceeds to step

162

and is done. The surface interaction process saves the penetrating vertex as V

i

. The surface interaction process then uses the V

i

penetrating the volume of the virtual tool as described above in the discussion of FIG.

4

.

If the surface interaction process checks all of the vertices within the bounding sphere and within the BSP tree and does not locate a vertex within the volume of the tool, the surface interaction process begins checking the vertices not located in the BSP tree to determine if they are located within the volume of the virtual tool (steps

164

,

166

,

168

). The surface interaction process continues checking vertices until it finds a vertex located within the volume of the tool. Once the surface interaction process locates a vertex within the volume of the tool, it proceeds to step

162

. The surface interaction process then uses the V

i

penetrating the volume of the virtual tool as described above in the discussion of FIG.

4

.

FIGS. 6A

,

6

B, and

6

C illustrate one embodiment of a method of the present invention for determining the interaction of the virtual tool with the virtual surface when the tool mode is set to rigid interaction as in step

148

of

FIG. 4

described above. In this embodiment, when the virtual tool collides with the virtual surface, the tool is not allowed to penetrate or deform the virtual surface. The virtual tool is only allowed to slide across the surface. In step

170

, the surface interaction process sets the direction of the virtual tool (tool

direction

), i.e. virtual sphere

56

, equal to the displacement between the position of the haptic interface (HI

position

) and the current tool position tool

position

). In step

172

, the haptic process determines if the amount that the virtual tool must move to intersect with the position of the haptic interface is greater than a predetermined stepsize (STEPSIZE) according to equation (5) below.

|tool

position

−HI

position

|<STEPSIZE  (5)

In one embodiment, the predetermined stepsize is 0.25 mm. If the magnitude of the difference in position between the tool position (tool

position

) and the haptic interface position (HI

position

) is less than the predetermined stepsize, the haptic process sets the current step size equal to the magnitude of the difference in position between the tool position (tool

position

) and the haptic interface position (HI

position

) (step

174

). If the magnitude is greater than the predetermined stepsize, the haptic process sets the current step size equal to the predetermined step size (step

176

). The purpose of steps

172

,

174

,

176

is to ensure that the tool does not move more than a predetermined amount during any cycle.

Next the surface interaction process determines the updated position of the virtual tool (tool

updated

—

position

) according to equation (6) below in which tool

position

represents the current tool position, currentStepSize represents the magnitude of the distance the virtual tool may move, and tool

direction

represents a unit vector in the direction that the virtual tool must move to intersect the haptic interface (step

178

). The calculation of the vector tool

direction

was described above in the discussion of step

170

.

tool

updated

—

position

=tool

position

+currentStepSize*tool

direction

  (6)

Next, in step

180

, the surface interaction process attempts to find a vertex (V

i

) of the triangular mesh forming the virtual surface which penetrates the volume of the virtual tool. In one embodiment, the surface interaction process uses the method illustrated in FIG.

5

and described above to find the first penetrating vertex V

i

. In step

182

the surface interaction process determines if a penetrating vertex was located. If a penetrating vertex was not found, the surface interaction process is complete for that cycle (step

183

). The virtual tool does not need to be moved out of the virtual surface as none of the vertices of the virtual surface are located within the virtual object.

If a penetrating vertex is found, the surface interaction process moves the virtual tool outside of the virtual surface (step

184

).

FIG. 7

shows one embodiment of a process for backing the virtual tool out of the virtual surface. In step

186

, the surface interaction application moves the via tool to a position halfway between the current position of the virtual tool and the last position of the virtual tool in which none of the vertices of the virtual surface penetrated the volume of the virtual tool. Next in step

188

, the surface interaction application attempts to find a vertex of the triangular mesh forming the virtual surface which penetrates the virtual tool. In one embodiment, the surface interaction process uses the steps illustrated in the flowchart of FIG.

5

and described above to locate a penetrating vertex. In step

190

, the surf interaction process determines if a penetrating vertex was found. If a penetrating vertex was not found, the surface interaction process is done and does not need to move the virtual tool any further from the virtual surface because the virtual tool no longer intersects with the virtual surface. In another embodiment, the surface interaction process may iteratively move the virtual tool back closer to the virtual surface to find the position closest to the virtual surface at which none of the vertices of the virtual surface penetrate the volume of the virtual tool. If the surface interaction process locates a penetrating vertex, the surface interaction process again moves the tool to a point halfway between the point to which the tool had just been moved and the last position of the virtual tool in which none of the vertices of the virtual surface penetrated the volume of the virtual tool. The surface interaction process then again checks for penetrating vertices. The haptic interaction process repeats steps

186

,

188

and

190

until none of the vertices of the virtual surface penetrate the virtual tool.

After the tool has been backed out of the surface, the surface interaction process slides the virtual tool in a direction substantially tangent to the surface. To prevent the virtual tool from prematurely penetrating the virtual surface due to any error in calculating the normal vector to the virtual surface, the surface interaction process moves the virtual tool across and away from the surface. The extra movement away the surface is resolved in the next iteration of this process when the virtual tool is brought down directly toward the position of the haptic interface. To determine the direction to move the virtual tool, in step

194

the surface interaction process first sets the surface normal to the virtual surface equal to the surface normal at the penetrating vertex (V

i

) located in step

180

.

The surface normal of the penetrating vertex may be determined according to several methods. As mentioned above, in one embodiment, the virtual surface is defined as a mesh and the virtual tool as the volume. The volumetric representation is used to capture penetrations of the surface mesh vertices and compel the penetrating vertices toward the surface of the tool volume. Given a point in Cartesian coordinate space, this method determines whether or not the point is inside of or outside of the tool volume.

Assuming a point in Cartesian coordinate space guaranteed to be close to the surface of the tool volume, the method generates an approximation to the surface normal at the closest point to the surface. Accordingly, the closer to the tool surface the point is, the less error in the surface normal approximation. While it may not always be possible to calculate an exact point on the surface, there will typically be known certain positions that are guaranteed to be within a known distance above or below the surface.

Virtual tools may be implemented as a sphere, a cuboid, a toroid, or other volumetric representations. An implicit function (e.g. S(P)=0) may be used to represent these tool volumes. Implicit functions have the property that S(P) equals zero if the point P lies on the surface, S(P) is less than zero if the point P is internal to the volume, and S(P) is greater than zero if the point P is external to the volume. The first derivative of S(P) with respect to Cartesian coordinates x, y, and z returns a vector pointing toward the surface of S when P is close to the surface, as long as the distance from P to the surface is relatively small with respect to the local curvature of the surface near this point. Accordingly, the first derivative may be used to generate a vector as an approximation to the surface normal. For values of S(P) less than zero, the negative of the first derivative is used as the surface normal approximation. By way of example, for a sphere, S(P) is equal to the sum of the squares of x, y, and z, plus one and the first derivative is two times the sum of x, y, and z. For a cuboid, S(P) is equal to the sum of each of x, y, and z to the eighth power, plus one and the first derivative is eight times the sum of each of x, y, and z to the seventh power.

In another embodiment for generating the surface normal, a voxel representation may be used. According to this method, a volumetric or voxel object is represented by dividing space into discrete areas known as voxels. This division of space encodes every possible point in the space as being enclosed in a voxel. For each voxel, a value is used to specify the occupancy of material through an integer or decimal point value. The surface of this volume is then defined as an isosurface of these values. Values below the isosurface value are then defined as being internal to the volume and values above the isosurface value are considered external to the volume. Voxelized volumes are known to those skilled in computer graphics and can be used readily to represent arbitrary volumes such as virtual tools with intricate shapes.

By specifying any point enclosed in a voxel having a value less than the isosurface value as inside the volume and any point enclosed in a voxel having a value greater than the isosurface value as outside the volume, an efficient algorithm may be developed. Further, using a method known to those skilled in the art as trilinear interpolation, a gradient vector similar to the vector given by the first derivative of an implicit function S(P) can be calculated. If the vector is calculated at a point exterior to the isosurface then the negative of the vector gives the normal approximation.

Once the surface interaction process has determined the surface normal at V

i

, in step

196

the surface interaction process determines the tangent direction (tangent

direction

) to the virtual surface at V

i

according to the dot product vector calculation of equation (7) below in which surface normal represents the surface normal calculated in step

194

and tool

direction

represents the unit vector pointing in the direction in which the virtual tool must move to intersect with the haptic interface location (determined above in step

170

).

tangent

direction

=surface

Normal

−(surface

Normal

*tool

direction

)*tool

direction

  (7)

See

FIG. 8

for a schematic representation of the vector relationship.

The surface interaction process then calculates the new position of the virtual tool by adding a portion of the surface normal to the virtual surface according to equation (8) below in which &lgr; represents a predetermined value (step

198

).

tool

new

—

position

=tool

position

+currentStepSize*(&lgr;*tangent

direction

+(1−&lgr;)*surface

normal

)  (8)

In one embodiment, &lgr; has a value of about 0.75.

After the virtual tool has been moved to tool

new

—

position

calculated above in equation (8), the surface interaction process again attempts to find a vertex of the virtual surface which penetrates the virtual object (step

200

). In one embodiment of the method of the present invention, the surface interaction process uses the steps illustrated in FIG.

5

and described above to locate a penetrating vertex. In step

202

the surface interaction process determines if a penetrating vertex was found. If a penetrating vertex was not found, the virtual tool does not need to be moved and the surface interaction process is complete for this cycle (step

204

). If a penetrating vertex is found, the surface interaction process backs the virtual tool out of the virtual surface according to the steps illustrated by the flowchart of FIG.

7

and described above (step

206

). Once the virtual tool is backed out of the virtual surface, the surface interaction process determines if the virtual tool was moved by a distance more than epsilon (step

208

). Epsilon is some small number used for the approximation of zero. For example, epsilon may be about 0.001, or about two orders of magnitude smaller than the predetermined stepsize. A small value is chosen for epsilon so that there is no perceivable movement of the haptic interface. If the virtual tool was moved by a distance greater than epsilon, the virtual tool does not need to be moved and the surface interaction process is complete for this cycle (step

204

).

If the virtual tool was moved by a distance less than epsilon, the virtual tool may be at an edge of the virtual surface. The surface interaction process then searches for an edge constraint to slide the virtual tool along. In step

210

the surface interaction process begins a series of steps that will be repeated N times. N is a predetermined integer. In one embodiment, N is equal to ten. In step

212

, the surface interaction process sets a second surface normal (surface

normal2

) equal to the surface normal at the most recently found penetrating vertex V

i

. The surface interaction process then calculates a vector along the edge (edgeVector) that the virtual tool has collided with according to equation (9) below in which surface

normal

represents the surface normal at the first penetrating vertex that was located in step

200

and surface

normal2

represents the surface normal calculated in step

212

.

edgeVector=surface

normal

×surface

normal2

  (9)

The vector edgeVector calculated in equation (9) is a unit vector.

Next, in step

216

, the surface interaction process calculates the new position of the virtual tool (tool

newposition

) according to equation (10) below in which tool

position

represents the current position of the tool, currentStepSize represents the magnitude of the distance that the virtual tool may move, and edgeVector represents the unit vector calculated in equation (9) and which falls along the edge.

tool

newposition

=tool

position

+currentStepSize·edgeVector  (10)

Next, in step

218

, the surface interaction process steps the virtual tool out of the virtual surface according the steps illustrated in FIG.

7

and described above. The surface interaction process then again determines if the virtual tool was moved by a distance more than epsilon (step

220

). If the virtual tool was moved by a distance greater than epsilon, the virtual tool does not need to be moved again and the surface interaction process is complete for this cycle (step

222

). If the virtual tool was moved by a distance less than epsilon, the first surface normal ) is set equal to the second surface normal (surface

normal2

) (step

224

) and the surface interaction process proceeds to step

226

. In step

226

, the surface interaction process determines if steps

210

,

212

,

214

,

216

,

218

,

220

, and

224

have been repeated N times. As described above, in one embodiment N is equal to ten. If these steps have not been repeated N times, the surface interaction process returns to step

210

and repeats the series of steps. If these steps have been repeated N times the surface interaction process does not try any further to locate a direction to move the virtual tool (step

228

).

In another embodiment, in order to find a direction to move the virtual tool so that none of the vertices of the virtual surface penetrate the virtual tool, the surface interaction process selects random vectors to determine the movement of the virtual tool. The surface interaction process continues to select a new random vector until the desired result is achieved, i.e., none of the vertices of the virtual surface penetrate the virtual tool.

Referring again to

FIG. 4

, if the tool mode has been set to deform, the surface interaction process determines the deformation of the virtual surface in response to collisions with the virtual tool (step

150

).

FIG. 9

illustrates one embodiment of a method of the present invention for determining the interaction of the virtual tool with the virtual surface when the tool mode is set to deform. In step

230

, the surface interaction process moves the virtual tool to the deformation position (P