BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the field of computer implemented user interfaces. More specifically, the present invention relates to the field of computer implemented recognition of user input information.

2. Related Art

In modern computing technology, there is always a need to provide mechanisms to facilitate user interaction with computing devices. By increasing the mechanisms by which persons interact with computer systems, the number of potential computer users and the number of potential computer applications expand. Further, by facilitating user interface mechanisms, applications become easier to use and more efficient. Today, users can communicate with computer systems using a number of various devices including refreshable display screens (cathode ray tubes, liquid crystal displays etc.), alphanumeric keyboards, keypads, cursor directing devices, microphones, etc. Keyboards are used for conveying information in alphanumeric form to a computer from a user.

The cursor directing device is used in conjunction with an animated cursor image that is rendered on the display screen. The cursor image is animated in that it can move across the display screen in real-time tracing the motion of the cursor directing device. Cursor directing devices, e.g., mouse devices, trackballs, etc., are used to direct the position of the cursor image on a display screen according to user interaction. In operation, a hand held or user directed mouse device is displaced across a mouse pad and the corresponding displacement and direction are simultaneously traced out by the cursor image as rendered on the display screen. When the cursor image is directed over a particular portion of the display screen, one or more buttons on the mouse device can be depressed to “activate” the cursor image which generally invokes a computer action related to the screen portion. The areas on the display screen that invoke a computer command when the cursor image is positioned thereon and activated have been called “hot spots.” In the past, to convey information to the computer system, cursor directing devices have been used in conjunction with hot spots located on the display screen.

More particularly, when the cursor image is activated in prior art user interfaces, the computer system performs a relatively routine task of checking the screen coordinates of the cursor image against the screen coordinates of a number of recorded hot spots to determine which enabled hot spot was selected by the cursor activation. In performing the check to determine which hot spot is selected, the computer system typically does not care about the screen path in which the cursor image passes through in order to reach the hot spot. Further, in performing the check to determine which hot spot is selected, the computer system typically does not care about the speed in which the cursor image was directed to the hot spot. All that is checked by the computer system is the screen coordinate of the cursor image when the cursor image is activated (e.g., when a mouse button is depressed). Thus, in the past, the real-time displacements of the mouse device between mouse activations are largely ignored.

Since the cursor directing device allows a substantial amount of real-time free style user-directed movement within the plane of the mouse pad, it would be advantageous to provide a user interface mechanism that could make use of more information from the cursor directing device than merely the display screen coordinate of the cursor image upon cursor activation.

Accordingly, the present invention provides a system and method of interfacing a user with a computer system that provides more information to the computer system from a cursor directing device than merely the screen coordinate of the cursor image upon cursor activation. The present invention provides a system and method of interfacing a user with a computer system that conveys information regarding the path through which a cursor directing device is spatially displaced and the relative speed of such spatial displacement. The above information provides a unique and advantageous mechanism by which information can be conveyed to the computer system from the user. These and other advantages of the present invention not specifically mentioned above will become clear within discussions of the present invention presented herein.

SUMMARY OF THE INVENTION

A computer implemented method and system are described for gesture category recognition and training. In one embodiment, a cursor directing device is used. Generally, a gesture is a hand or body initiated movement of a cursor directing device which outlines a particular pattern, in particular directions, and can comprise one or more strokes. The present invention allows a computer system to accept input data, originating from a user, in the form gesture data that are made using a cursor directing device. In one embodiment, a mouse device is used, but the present invention is equally well suited for use with other cursor directing devices (e.g., a track ball, a finger pad, an electronic stylus, optical tracking device, etc.). In one embodiment, the computer system is queued to accept a new gesture data by pressing a key on the keyboard and then moving the mouse (e.g., while a mouse button is depressed) to trace out a gesture that is associated with a gesture category. Coordinate position information of the mouse and time stamps are recorded as gesture data in memory based on the user gesture. More than one gesture can be associated with a gesture category (e.g., as negative and positive examples).

The present invention then determines a multi-dimensional feature vector based on the gesture data. The multi-dimensional feature vector is then passed through a gesture category recognition engine that, in one implementation, uses a radial basis function neural network to associate the feature vector to a preexisting gesture category. Once identified, a set of user commands that are associated with the gesture category (e.g., a macro) are applied to an application of the computer system. The user commands can originate from an automatic process that extracts commands that are associated with the menu items of a particular application program. The present invention also allows user training so that userdefined gesture categories, and the computer commands associated therewith, can be programmed into the computer system.

More specifically, an embodiment of the present invention includes a method of providing a user interface in an electronic system having a processor, a memory unit, an alphanumeric input device, the method comprising the computer implemented steps of: a) receiving gesture data representing a gesture performed by a user with the cursor directing device, the gesture data comprising coordinate positions and timing information and having one or more individual strokes; b) generating a multi-dimensional feature vector based on the gesture data; c) providing the multi-dimensional feature vector to a radial basis function neural network for recognition, the radial basis function neural network associating the multi-dimensional feature vector with a gesture category from a predefined plurality of gesture categories and supplying the gesture category as an output value; and d) applying a set of predetermined commands to the electronic system, the set of predetermined commands being associated with the gesture category output from the radial basis function neural network.

Embodiments include the above and wherein the step b) comprises the steps of: b1) normalizing the gesture data; b2) dividing each stroke of the gesture data into a plurality of segments, N; b3) determining first feature elements for each stroke of the gesture data based on an end point of a respective stroke and a start point of a next stroke; b4) determining second feature elements for each segment of each stroke of the gesture data based on an orientation of each segment with respect to a reference line, wherein the multi-dimensional feature vector comprises: the number of strokes of the gesture data; the first feature elements; and the second feature elements.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1

illustrates a general purpose computer system in which embodiments of the neural network based gesture category recognition process of present invention can be implemented.

FIG. 2

is a high level data flow diagram of aspects of the gesture category recognition process of the present invention.

FIG. 3

is a high level data flow diagram of the gesture category recognition engine of the present invention using a radial basis function neural network.

FIGS. 4A

,

4

B,

4

C and

4

D illustrate exemplary geometric gestures that can be used in accordance with the present invention.

FIGS. 5A

,

5

B and

5

C illustrate exemplary alphabetical gestures that can be used in accordance with the present invention.

FIGS. 6A

,

6

B and

6

C illustrate further exemplary alphabetical gestures that can be used in accordance with the present invention.

FIGS. 7A and 7B

illustrate gesture differentiation based on displacement direction.

FIGS. 7C and 7D

illustrate gesture differentiation based on starting point and displacement speed, respectively.

FIG. 8A

, FIG.

8

B and

FIG. 8C

illustrate a flow chart of steps used in the gesture data input and gesture category recognition processes of the present invention.

FIG.

9

A and

FIG. 9B

are logical diagrams which illustrate gesture data input and normalization in accordance with the present invention.

FIG. 10A

illustrates an exemplary gesture being divided into an array of segments for computation of the feature vector in accordance with the present invention.

FIG. 10B

illustrates an individual segment of a gesture being analyzed in accordance with the present invention for feature vector extraction.

FIG. 10C

illustrates the components of a feature vector in accordance with one embodiment of the present invention.

FIGS. 11A

,

11

B,

11

C,

11

D and

11

E illustrates exemplary steps performed during the training of the computer system of the present invention using a radial basis function neural network for gesture category recognition.

FIG. 12

illustrates a flow chart of steps in accordance with one embodiment of the present invention for gesture category training.

FIG. 13

illustrates a flow chart of steps in accordance with an embodiment of the present invention for gesture category training based on extracted computer commands from menu items of an application program.

FIG. 14

is an exemplary user interface display for training the gesture category recognition process of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the present invention, a system and method for gesture category recognition within a computer system, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be recognized by one skilled in the art that the present invention may be practiced without these specific details or with equivalents thereof. In other instances, well known methods, procedures, components, and circuits have not been described in detail as not to unnecessarily obscure aspects of the present invention.

Notation and Nomenclature

Some portions of the detailed descriptions which follow are presented in terms of procedures, steps, logic blocks, processing, and other symbolic representations of operations on data bits within a computer memory. These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. A procedure, computer executed step, logic block, process, etc., is here, and generally, conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as “processing” or “computing” or “translating” or “calculating” or “determining” or “displaying” or “recognizing” or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Computer Syetem

112

Aspects of the present invention, described below, are discussed in terms of steps executed on a computer system (e.g., process

800

, process

1200

and process

1300

). Although a variety of different computer systems can be used with the present invention, an exemplary computer system

112

is shown in FIG.

1

.

In general, computer systems

112

that can be used by the present invention comprise an address/data bus

100

for communicating information, a central processor

101

coupled with the bus for processing information and instructions, a volatile memory

102

(e.g., random access memory) coupled with the bus

100

for storing information and instructions for the central processor

101

and a non-volatile memory

103

(e.g., read only memory) coupled with the bus

100

for storing static information and instructions for the processor

101

. Computer system

112

also includes a data storage device

104

(“disk subsystem”) such as a magnetic or optical disk and disk drive coupled with the bus

100

for storing information and instructions and a display device

105

coupled to the bus

100

for displaying information to the computer user.

Also included in computer system

112

is an alphanumeric input device

106

including alphanumeric and function keys coupled to the bus

100

for communicating information and command selections to the central processor

101

. Generally, alphanumeric input device

106

is called a keyboard or keypad. System

112

also includes a cursor control or directing device

107

coupled to the bus for communicating user input information and command selections to the central processor

101

. The cursor directing device

107

is typically displaced through user movement which causes a cursor image displayed on screen

105

to move accordingly. Within the context of the present invention, the cursor directing device

107

can include a number of implementations including a mouse device, for example, a trackball device, a joystick, a finger pad (track pad), an electronic stylus, an optical beam directing device with optical receiver pad, an optical tracking device able to track the movement of a user's finger, etc., or any other device having a primary purpose of moving a displayed cursor across a display screen based on user displacements.

Using a mouse device, the mouse

107

is moved by a user's hand relative to a fixed pad thereby causing the cursor to move with respect to the mouse movement. As the mouse is moved, in real-time, coordinate displacement information is sent to the computer system. The computer

112

samples the displacements over a selected sampling frequency and therefore each coordinate displacement is associated with a particular time stamp. Therefore, for each sample an (x, y) coordinate displacement is recorded along with a relative timestamp. This information can be stored in computer memory

102

as gesture data.

Computer system

112

of

FIG. 1

can also include an optional signal generating device

108

coupled to the bus

100

for interfacing with other networked computer systems. The display device

105

of

FIG. 1

utilized with the computer system

112

of the present invention may be a liquid crystal device, other flat panel display, cathode ray tube, or other display device suitable for creating graphic images and alphanumeric .characters recognizable to the user. In one embodiment of the present invention, computer system

112

is a Windows Operating System based computer system having an x86 architecture processor

101

.

Gesture Category Recognition System of the Present Invention

With reference to

FIG. 2

, a high level data flow diagram

200

of the present invention gesture category recognition system is illustrated. User-originated gesture data is received at

205

from the cursor directing device

107

(FIG.

1

). At

205

, the user indicates to the computer system

112

that a gesture is to be input and then inputs the gesture data. The term “gesture data” includes a variable-sized set of points of space of any dimension, parameterized with time and relates to a user movement called a “gesture.” The movement can be any movement continuous or discontinues of any object in space of any dimension, and includes the intentional movement made by hand used for symbolic communication. Continuous parts of the gesture, disconnected form each other, are referred to as strokes. The term “input gesture” is gesture data that needs recognition. In this context, an input gesture can be viewed as gesture vectors because coordinate position and timing information can be used to differentiate gestures.

The term “gesture category” corresponds to a predetermined category or “name” defined by a user. The “gesture category” can have multiple different gestures defined within as samples (e.g., examples). There can be positive examples and negative examples used to define the gesture category. The samples expand the space of the gesture category and during recognition, the present invention attempts to associate an input gesture to a particular predefined gesture category. Computer commands can also be associated with a gesture category.

In one embodiment of the present invention, the gesture data is formed by the user displacing a mouse device over a fixed surface (e.g., a mouse pad). As the cursor directing device

107

is displaced during gesture input, real-time coordinate displacement information in (x, y) format is forwarded to the computer system

112

. The computer system

112

timestamps the coordinate information based on sample points and a sample frequency. As shown, coordinate (x, y) and timestamp information

207

is then stored in a computer readable memory unit

102

of the computer system

112

.

It is appreciated that in one embodiment of the present invention, a particular key on the keyboard

106

is depressed to gather the gesture data. The keyboard

106

is used as a trigger to indicate to the computer system

112

that gesture data is to be input. When the key is released, the gesture is then complete.

When configured to performed gesture category recognition, a computer implemented gesture category recognition process

800

of

FIG. 2

receives the gesture data

205

, transforms the gesture data

205

into a feature vector and uses the feature vector for determining which of a number of predefined gesture categories best represents the gesture data

205

. As shown in

FIG. 2

, a number of predefined gesture categories

210

are stored in a computer readable memory unit (e.g., unit

102

). In the example of

FIG. 2

, predefined gesture categories

210

a

-

210

z

exist. As described further below, each gesture category has an associated “bounded area” within a multi-dimensional space of the neural network. The multi-dimensional space is indexed by the input feature vector. The gesture category recognized by the gesture category recognition process

800

of the present invention depends on which bounded area is pointed to by the feature vector of the gesture data

205

.

The gesture category that is recognized by the gesture category recognition process

800

of the present invention is referred to as the output gesture category. In the example of

FIG. 2

, gesture category

210

b

is the output gesture as represented by arrow

250

. List

220

, maintained in computer readable memory

102

, includes a separate entry, e.g.,

220

a

-

220

z

, for each of the predefined gesture categories of list

210

. Each entry within database

220

represents a set of computer commands, e.g., instructions and/or commands of a macro, that are to be applied to computer system

112

when the user inputs the gesture that corresponds to that set of commands.

FIG. 2

illustrates the commands being applied to an application executed by processor

101

(FIG.

1

). An optional confirmation can be required before certain commands are applied (e.g., those that alter or remove information from computer system

112

). For example, the set of commands within

220

b

corresponds to predefined gesture category

210

b

. When the input gesture is recognized and represented by output gesture category is

210

b

, then the set of commands

220

b

are applied to processor

101

. The same is true for each of the predefined gesture categories

210

a

-

210

z

and their corresponding sets of computer commands

220

a

-

220

z.

As such, when configured for gesture category recognition (and not in training mode), the typical usage of the present invention is the following. A user manipulates the cursor directing device

107

(e.g., mouse) in such a way as to create gesture data and then, once recognized, a corresponding set of computer commands are automatically applied to processor

101

. For instance, each time a user wants a word processor application to save a document, the mouse device

107

is displaced to trace out a predetermined gesture, recognized as one of the gesture categories, and the application program is automatically directed to save. Alternatively, each time a user wants to read electronic mail, a different predetermined gesture is traced out with the mouse device

107

causing computer system

112

to interface with an external system (e.g., the email application on the internet) which then downloads the required mail.

FIG. 3

illustrates a data flow diagram

300

of elements used by the gesture category recognition process

800

of the present invention. The feature vector

310

corresponds to the gesture data

205

. Feature vector

315

is multi-dimensional and as such is made up of several feature elements represented by exemplary feature elements

315

a

-

315

d

. As described further below, each element of

315

a

-

315

d

is derived according to the process

800

of FIG.

8

A and FIG.

8

B. The feature vector

315

is input to a neural network

320

that, in one embodiment, utilizes a radial basis function to identify the output gesture category. The radial basis function neural network

320

interfaces with a multi-dimensional space, stored in memory unit

102

, that is filled with boundary areas; each bounded area corresponds with a particular predetermined gesture category. The size of the bounded area depends on the number of example gestures defined as associated with the gesture category and the number of predefined gesture categories.

The radial basis function as applied within a neural network

320

is a well known process for performing recognition. It is described in one work entitled “Introduction to Radial Basis Function Networks” by Mark J. L. Orr, published by the Centre for Cognitive Science, University of Edinburgh, 2, Buccleuch Place, Edinburgh EH8 9LW, Scotland. In the radial basis function neural network

320

, the multi-dimensional feature vector

315

is applied within the multi-dimensional space and points within this space to a particular location. This location can be positioned within one of the bounded areas of the multi-dimensional space. The predetermined gesture that corresponds to this bounded area is then the output gesture

250

.

EXAMPLE GESTURES

FIGS. 4A

,

4

B,

4

C and

4

D illustrate some exemplary “single stroke” geometric gestures that can be used in accordance with the present invention. It is appreciated that almost any geometric pattern can be used as a gesture within the present invention and that the following gestures are exemplary only. These gestures are “single stroke” because they can be made while holding down the mouse button and using a single continuous mouse movement. Exemplary gesture

405

of

FIG. 4A

is generally a square shape and made by starting at position

405

a

and tracing the cursor directing device

107

clock-wise through the square shape to the end (the arrow). It is appreciated that in one embodiment of the present invention, a particular key on the keyboard

106

is depressed while the gesture is being traced out on the cursor directing device

107

.

Exemplary gesture

410

of

FIG. 4B

is generally a diamond shape and made with the cursor directing device

107

starting at position

410

a

on the mouse pad and tracing the cursor directing device

107

clock-wise through the diamond shape to the end (the arrow). Exemplary gesture

415

of

FIG. 4C

is generally a circle in shape and is made starting at position

415

a

and tracing the cursor directing device

107

clock-wise in the circle shape to the end (the arrow). Exemplary gesture

420

of

FIG. 4D

is generally a triangle shape and made with the cursor directing device

107

starting at position

420

a

on the mouse pad and tracing the cursor directing device

107

counter clock-wise through the triangle shape to the end (the arrow).

In each of the exemplary gestures

405

,

410

,

415

and

420

, the speed of displacement of the cursor directing device

107

is maintained relatively uniform through the gesture. Typically, when using a key of the keyboard

106

as the gesture triggering event (described below), the position of the cursor directing device

107

when the key is depressed is used to indicate the start position (e.g.,

405

a

,

410

a

,

415

a

and

420

a

). Gestures

405

,

410

,

415

and

420

are generally closed geometry because the ends return to the start points (

405

a

,

410

a

,

415

a

and

420

a

). A gesture need not be closed in geometry within the present invention.

FIGS. 5A

,

5

B, and

5

C illustrate some exemplary alphanumeric gestures that can be used in accordance with the present invention. It is appreciated that almost any alphanumeric pattern can be used as a gesture within the present invention and that the gestures of

FIGS. 5A

,

5

B and

5

C are exemplary only. Exemplary gesture

510

of

FIG. 5A

is generally in the shape of a “B” and is made starting at position

510

a

and tracing the cursor directing device

107

counter clock-wise in the “B” shape to the end (the arrow). Gesture

510

is a “double stroke” gesture requiring two separate strokes of the mouse

107

and releasing the mouse button in between strokes. Exemplary gesture

515

of

FIG. 5B

is generally in the shape of a “C” and is made starting at position

515

a

and tracing the cursor directing device

107

counter clock-wise in the “C” shape to the end (the arrow). In this case, the gesture

515

is an open geometry because the end does not return to the start

515

a

. Exemplary gesture

520

of

FIG. 5C

is generally in the shape of a “D” and is made starting at position

520

a

and tracing the cursor directing device

107

counter clock-wise in the “D” shape to the end (the arrow). Gesture

520

is a “double stroke” gesture requiring two separate strokes of the mouse

107

and releasing the mouse button in between strokes. It is appreciated that each of the gestures

510

,

515

and

520

could also be drawn in the clock-wise direction which yields different gesture definitions.

FIGS. 6A

,

6

B, and

6

C illustrate additional exemplary alphanumeric gestures that can be used in accordance with the present invention. Exemplary gesture

610

of

FIG. 6A

is generally in the shape of a “G” and is made starting at position

610

a

and tracing the cursor directing device

107

counter clock-wise in the “G” shape to the end (the arrow). Exemplary gesture

615

of

FIG. 6B

is generally in the shape of a “J” and is made starting at position

615

a

and tracing the cursor directing device

107

clock-wise in the “J” shape to the end (the arrow). Exemplary gesture

620

of

FIG. 6C

is generally in the shape of an “L” and is made starting at position

620

a

and tracing the cursor directing device

107

counter clock-wise in the “L” shape to the end (the arrow). It is appreciated that gestures

610

and

620

can also be drawn in the clock-wise direction which would yield different gesture definitions. Further, gesture

615

can also be drawn in the counter clock-wise direction which would yield a different gesture definition in accordance with the present invention.

Gesture Differentiation

FIG.

7

A and

FIG. 7B

illustrate two exemplary gestures

710

and

715

, respectively. Gestures

710

and

715

illustrate that in accordance with the present invention, gestures can be differentiated based on the displacement direction by which they are traced using displacements of the cursor directing device

107

. For example, both gestures

710

and

715

are square in shape and of the same relative dimensions. However, gesture

710

is formed from start point

710

a

in a clock-wise direction while, in contrast, gesture

715

is formed from (the same) start point

715

a

in a counter clock-wise direction. The manner in which gestures are decomposed to form a feature vector within the present invention preserves information regarding the direction in which a gesture is traced using displacements of the cursor directing device

107

. Therefore, gestures can be differentiated based on trace direction.

FIG.

7

A and

FIG. 7C

illustrate gestures

710

and

720

, respectively. Gestures

710

and

720

illustrate that in accordance with the present invention, gestures can be differentiated based on their start points. For example, both gestures

710

and

720

are square in shape, of the same relative dimensions, and both are traced in the same direction (clock-wise). However, gesture

710

is formed from start point

710

a

(upper left) while, in contrast, gesture

720

is formed from start point

720

a

which is located in the upper right. The manner in which gestures are decomposed to form a feature vector within the present invention preserves information regarding the relative time in which each segment of the gesture is made. For instance, the upper left comer of gesture

710

is formed first while the upper left portion of gesture

720

is formed three quarters in time into the total time required to form gesture

720

. Therefore, gestures can be differentiated based on their relative start positions within the present invention.

FIG.

7

A and

FIG. 7D

illustrate gestures

710

and

725

, respectively. Gestures

710

and

725

illustrate that in accordance with the present invention, gestures can be differentiated based on the speed in which different parts of the gesture are formed using displacements of the cursor directing device

107

. For example, both gestures

710

and

725

are square in shape, of the same relative dimensions and have the same start point. However, gesture

710

is formed from start point

710

a

(upper left) at a relatively uniform creation speed throughout. That is to say, the cursor directing device

107

is moved at a relatively uniform speed across the mouse pad (for example) as the rectangle is formed. In contrast, gesture

725

is formed from start point

725

a

which is located in the upper right and the top right comer portion

730

is formed using a very fast stroke while the bottom left comer portion

735

is formed using a much slower stroke. The manner in which gestures are decomposed to form a feature vector within the present invention preserves information regarding the relative times in which the start and end points of a stroke of a gesture are made. Therefore, gestures can be differentiated based on speeds in which different sections of the gesture are made.

As shown below, gestures can also be differentiated based on the number and type of strokes that are used to make up the gesture. The following discussion describes the gesture category recognition and training modes available within the embodiments of the present invention.

Gesture Category Recognition Process

800

FIG. 8A

, FIG.

8

B and

FIG. 8C

illustrate steps within gesture category recognition process

800

of the present invention. It is appreciated that process

800

is implemented by computer system

112

executing instructions that are stored in a computer readable memory unit (e.g., unit

102

). Process

800

can operate in the background simultaneously with other application programs that are running in system

112

. Process

800

is invoked upon a user selecting the gesture category recognition mode of the present invention (as opposed to gesture training modes described further below).

Referring to

FIG. 8A

, process

800

commences at step

805

. At step

805

, the present invention checks if a triggering event occurs. The triggering event is typically user initiated and indicates to the present invention that a gesture is to be input and needs to be captured for recognition. In one embodiment, the triggering event is accomplished when the user holds down a gesture key (the trigger event key) on the keyboard

106

while moving the cursor directing device

107

with the mouse button pressed. Any keyboard key can be defined as the gesture or trigger key. It is appreciated that the present invention is equally suited to respond to a number of different triggering events. For instance, one triggering event can be defined by a particular set of mouse buttons that are depressed while the mouse device is moved. The triggering event can also originate from a user spoken command or other unique user interface mechanism. Process

800

returns until the trigger event is detected at step

805

.

Upon detection of the trigger event at

805

, step

810

is entered. At step

810

, the present invention sets a stroke index, s, to zero. The stroke index, s, is used to account for the number of strokes within the input gesture. A stroke is a part of the mouse input while the mouse button is continuously pressed. Strokes can be generated by taking the sections of the mouse input where the mouse button is pressed continuously. At step

815

, process

800

checks if the mouse button of mouse

107

is pressed. If it is not pressed, then step

830

is entered. If the mouse button is pressed at step

815

, then step

820

is entered where the stroke index, s, is incremented by one to indicate a first stroke of the input gesture data.

Step

825

is then entered where the gesture data is recorded into memory

102

. The gesture data is associated with the particular stroke indicated by the stroke index, s. The present invention utilizes a number of well known mechanisms by which cursor directing information is received by computer system

112

and stored into a computer memory

102

.

Step

825

continues while the mouse button remains pressed so that the gesture's stroke can be recorded. In one embodiment, a mouse device is used as cursor directing device

107

. The mouse device transmits coordinate (x, y) position information to the computer system

112

which is sampled at a predetermined sample rate over time. From the trigger event, each coordinate position data from the mouse device is recorded in memory

102

along with its associated timestamp. In one implementation of step

825

, the WM_MOUSEMOVE Windows message is handled, and the device coordinates are transformed into the logical coordinates (x, y) by the CDC::DPtoLP MFC function. The length of this sequence is about 200 for a typical gesture's stroke.

When the mouse button is released, but the gesture key is still depressed, step

830

is held until the mouse button is pressed again at step

815

of FIG.

8

A. In this case, step

820

is entered which increments the stroke index, s, and another stroke of the input gesture is obtained at step

825

. However, if the mouse button is released and also the gesture key is no longer pressed, then step

835

is entered. At time, all of strokes (if more than one) of the gesture are captured and the input gesture is done. The variable, context, is set equal to s which is the number of captured strokes and the variable context is recorded into memory

102

.

From the relative coordinate displacement information stored in memory

102

, a two dimensional spatial path, through which the strokes of the input gesture are formed, is recorded by the computer system

112

. Each bit of data is also tagged with a stroke tag indicating the stroke number to which it is associated. The data representing the (x, y) coordinates, timestamps and stroke tag for a particular input gesture is the “gesture data.”

FIG. 9A

illustrates an exemplary two-stroke input gesture including a first stroke

930

(starting at

930

a

and ending a

930

b

) followed by a second stroke

935

(starting at

935

a

and ending at

935

b

). Outline

927

represents the two dimensional space of a typical mouse pad. The (x, y) coordinate space through which input gesture is made is recorded in memory

102

.

At step

840

of

FIG. 8A

, the present invention determines a small spatial window in which the input gesture is stored.

FIG. 9A

illustrates the small window

925

that is determined by the present invention. This small window

925

is determined so that the displacement information of the input gesture can be normalized. In one embodiment of the present invention, the x-length of small window

925

is determined to be substantially the difference between the maximum x coordinate of the input gesture and the minimum x coordinate of the input gesture. The y-length of small window

925

is determined to be substantially the difference between the maximum y coordinate of the input gesture data and the minimum y coordinate of the input gesture data.

At step

840

, each of the coordinates of the input gesture data are normalized according to a square

926

which is shown in FIG.

9

B. The normalized square

926

contains Y divisions along the horizontal and Y divisions along the vertical. In one implementation Y is 1000, but could be any number as memory and processing speed allows. Normalization is performed as follows. The minimal and maximal x and y values (xmin, ymin, xmax and ymax) are determined by taking absolute values for all strokes and ignoring sections between strokes. Then MAX is set as the larger between xmax and ymax. All (x, y) values of the input gesture data are then scaled according to the following:

x′=Y*(x−xmin)/(MAX−xmin)

y′=Y*(y−ymin)/(MAX−ymin)

The input gesture data is then characterized by the new coordinates (x′, y′). Optionally at step

840

, the associated time stamps of all sample points of the gesture data are shifted such that they are each relative to the time of the start point

930

a

(

FIG. 9A

) of the input gesture data. For instance, the start point

930

a

is time (t=0).

Process

800

continues with step

845

of

FIG. 8B

where the number of segments, N, per stroke is determined. Each stroke is divided into the same number N of segments. Two feature elements are associated to each segment. The total number of feature elements (the length of the feature vector) is fixed in one embodiment, to a value, X. In one embodiment, the following is used to determine the number of segments, N, per stroke:

N=(X/context−2)/2

where context is the number of strokes of the input gesture and X is a maximum feature length, e.g., 24 in one implementation.

At step

845

, the present invention divides each stroke into N separate segments along the stroke length. Although a number of methods could be used to divide a stroke into N segments, the following method is used in one embodiment. The number of coordinate pairs in a stroke is not necessarily an integer multiple of the number of segments in a stroke. Therefore, the segmentation of a stroke is done according to the following process. The starting point of the first segment is the first (x′, y′) pair of the stroke. The average size of a segment is determined by:

SegmentSize=(StrokeSize−1)/N

where StrokeSize is the number of coordinate pairs in the given stroke and N is the number of segments per stroke. The end-point of the ith segment is the nth coordinate pair in the stroke according to:

n=1+Round[(i−1)*SegmentSize]

and the Round function rounds its argument to the closest integer. The end-point of each segment is the starting point of the next segment. It follows from the above that the end-point of the Nth segment is the last coordinate pair of the stroke.

For example, as shown in

FIG. 10A

, stroke

930

is divided into N segments

1010

-

1030

and stroke

935

is divided into N segments

1035

-

1055

where N equals 5 in this example. In one embodiment of the present invention an input gesture is limited to three strokes maximum. In this embodiment: a single-stroke gesture has 11 segments per stroke; a double-stroke gesture has 5 segments per stroke; and a triple-stroke gesture has three segments per stroke. Each stroke is now analyzed by the present invention to determine feature vector elements associated with the strokes.

At step

850

of

FIG. 8B

the stroke index, s, is set to zero. At step

855

, is it checked if the current stroke index, s, is equal the variable, context. If so, step

890

is entered and if not, step

860

is entered. At step

860

, the stroke index, s, is incremented by one to select the first stroke (e.g.,

930

) on the first pass through. At step

865

, the global features G1(s) and G2(s) are determined based on the current stroke, s, and the next stroke s+1. If the current stroke, s, is the last stroke, then the “next stroke” is the first stroke.

Global features, G1(s) and G2(s), code the relations between successive strokes. In particular, two feature elements are reserved for each stroke to store the distance vector between the current stroke's endpoint (e.g.,

930

b

) and the next stroke's starting point (e.g.,

935

a

). More specifically, if (Xc and Yc) are the last coordinate pair

930

b

in the current stroke, s, and (Xn and Yn) are the first coordinate pair

935

a

of the next stroke, s+1, then the global feature elements of the current stroke, s, are determined at step

865

according to:

G1(s)=(Xn−Xc)/H

G2(s)=(Yn−Yc)/H

where H is a normalization factor used because the rest of the feature elements are in a smaller range. Assuming Y is 1000, then H is 400 in one embodiment. Step

865

determines two feature elements per stroke.

Feature elements for the stroke segments are now computed. At step

870

of

FIG. 8B

, the present invention sets the segment index, u, to zero. At step

875

, it is checked if u is equal to N. If so, step

855

is entered and if not, step

880

is entered. At step

880

, the segment index, u, is incremented by one. At step

885

the feature elements for a particular segment, u, of a particular stroke, s, are determined.

FIG. 10B

illustrates a particular segment

1025

of a stroke

930

. At step

875

, each segment (e.g.,

1025

) is treated as if it was a straight line

1025

c

between its starting point

1025

a

and its end point

1025

b

. The two feature elements, F1(s, u) and F2(s, u), belonging to a segment are the sine and cosine values of the directed angle between the segment

1025

(e.g., straight line

1025

c

) and the horizontal reference direction

1060

(FIG.

10

B). If the starting point coordinates are (Xs, Ys) and the end-point coordinates are (Xe, Ye), then the two feature elements, F1(s, u) and F2(s, u), are determined by:

F1(s, u)=(Xe−Xs)/L

F2(s, u)=(Ye−Ys)/L

where L is the length of the segment as L=sqrt[(Xe−Xs)

2

+(Ye−Ys)

2

]. After step

885

, step

875

is entered to access the next segment of the current stroke until the last segment, N, is reached. Upon the last segment, N, being processed, step

885

is entered to obtain the next stroke. This process continues until all feature elements for all segments of all strokes have been determined at which time step

890

is entered.

At step

890

, a multi-dimensional feature vector associated with the input gesture data is generated and stored in memory

102

. The feature vector is used to recognize the input gesture data as one of the preset gesture categories

210

(FIG.

2

).

FIG. 10C

illustrates one exemplary feature vector

315

. Feature vector

315

contains a feature element

315

a

that indicates the number of strokes of the input gesture, as represented by the variable “context.” Feature element

315

b

includes the global features G1(s) and G2(s) computed at step

865

(

FIG. 8B

) and there are

2

*context in number within

315

b

. Feature element

315

c

includes the stroke features F1(s, u) and F2(s, u) of each segment of each stroke and there are 2*context*N in number within

315

c

. The order of the elements of the feature vector

315

is arbitrary, but once determines remains fixed within the present invention.

It is appreciated that at step

890

a number of different recognition processes can be used including storing each predefined feature vector

315

into memory

102

and then comparing the new feature vector against each stored feature vector to uncover a best match. However,

FIG. 10C

illustrates a preferred method

890

of recognizing the input gesture using the determined feature vector

315

and a radial basis function neural network.

At step

905

of

FIG. 8C

, the present invention utilizes the multi-dimensional feature vector (FV)

315

to point to a particular space within a multi-dimensional space within a radial basis function neural network

320

(FIG.

3

). The multi-dimensional space of the radial basis function network

320

contains multiple bounded areas, each area having a set of multi-dimensional points that all correspond to a respective predefined gesture category (e.g., gesture categories

210

a

-

210

z

of FIG.

2

). By determining to which bounded area the feature vector (FV)

315

points, the present invention recognizes the input gesture

920

to one category. In other words, the present invention at step

890

recognizes the input gesture to be that predetermined gesture category having a bounded area that includes the multi-dimensional point that is pointed to by the feature vector (FV)

315

. The identified predetermined gesture category is called the “output gesture category.” The present invention utilizes the well known radial basis function network to perform this recognition process. Use of the radial basis function network is described in a work entitled “Introduction to Radial Basis Function Networks” by Mark J. L. Orr, published by the Centre for Cognitive Science, University of Edinburgh, 2, Buccleuch Place, Edinburgh EH8 9LW, Scotland. Automatic gesture training, used with the radial basis function network of the present invention, is further described with respect to

FIGS. 11A-11E

.

At step

910

of

FIG. 8C

, the present invention locates the set of computer commands from memory list

220

(

FIG. 2

) that are associated with the output gesture category. These commands are retrieved from memory unit

102

. At optional step

915

the present invention requests user confirmation before applying the set of computer commands associated with the output gesture category. In certain cases, it is desirable to have user confirmation before certain computer commands are applied to the computer system

112

. For instance, confirmation is typically requested before performing file erase and file deletions. At step

920

, the present invention applies the set of computer commands identified at step

910

, and associated with the output gesture category, to the computer system

112

. In one implementation, the set of computer commands can be applied to a particular application program with which the user is concurrently involved.

Geture Category Training Process

1200

FIG. 12

illustrates a generalized flow diagram of steps performed by the present invention for training the gesture category recognition process

800

to recognize gesture categories. More generally, gesture training process

1200

is used by the present invention for defining the predetermined gesture categories

210

a

-

210

z

and their associated sets of computer commands

210

a

-

210

z

as shown in FIG.

2

. It is appreciated that process

1200

is implemented by computer system

112

executing instructions that are stored in a computer readable memory unit (e.g., unit

102

). Like process