CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of the filing of U.S. Provisional Patent Application Serial No. 60/163,993, entitled “System and Software for Classification and Recognition Based on Spatial Frequency Sampling and Neural Networks”, filed on Nov. 8, 1999, and the specification thereof is incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention (Technical Field)

The present invention relates to image classification and recognition based on spatial frequency sampling as well as image domain sampling.

2. Background Art

In the early 1970's, the methods used in automatic pattern recognition could be grouped into two categories: those based on edge and edge-angle correlation; and those based on power-spectral density. In connection with aerial images and remote sensing, U.S. Pat. No. 3,687,772 disclosed a robotic-eye photodetector called a ring-wedge photodetector. As shown in

FIG. 1

, this photodetector preferably has 32 separate annular semi-rings for sampling power spectral density, independently of rotation, and 32 pie-shaped segments devised for readout of edges and edge-angle correlation. The wedge data are scale-invariant. In the recognition system disclosed, the photodetector is placed in the back focal plane of a Fourier-optical processor. The entire system includes a laser illuminator, an input picture being inspected, a Fourier-transform lens, and the ring-wedge photodetector in the back focal plane that is also known as the optical-transform plane. Each of the 64 photodetectors on the ring-wedge photodetector has a separate amplifier and digitizer so that the sampled Fourier transform signal can be coupled into an electronic digital computer.

By gathering data in rings and wedges, very complicated pictures with ten million or so pixels could be coarsely sampled in the optical Fourier transform space and automatic recognition tasks could be accomplished using only 64 feature values. During the following two decades, this analog hybrid opto-electronic system was successful in establishing high accuracy classification and recognition at high speeds (on the order of one decision per millisecond). In recognition of high-resolution photographs it is comparable in accuracy to the matched-filter (an earlier form of prior art), but it is much easier to implement. Notable successes were sorting of photographs of cats versus dogs, black-lung disease determinations, sharpness analysis of hypodermic syringes, wafer inspection, CD inspection, surface roughness determinations, and particle sizing. This system is particularly applicable when the decision/recognition depends upon fine-scale features or texture.

The present invention provides a software system both for recognition and classification of digital images automatically and rapidly. This system is particularly suited to all-digital optical robotics. It can be implemented as a stand-alone system or as a “tool-kit” in a general image processing environment that includes other means for processing, filing, and sorting images. The all-digital system of the invention, using both spatial transform features and image features, can automatically classify or recognize any image group with high accuracy, matching the best performance that human photointerpreters are capable of. The software recognition system is highly advantageous over the prior art in that it is affordable, costing only a small fraction of that for the laser-optical hybrid of the prior art. It is user friendly because neural network training routines yield superior decision performance. The invention has demonstrated superior performance in a wide variety of applications including classification of image quality in a manner that is widely independent of scene content, recognition of handwriting in a manner widely independent of textual content, and classification into multiple bins or categories. Go and no-go production testing quality of images used in photolithography and automatic sizing of particles as in talcum or pharmaceuticals are all viable application areas for this invention. The present invention is also applicable in a robotic control of the shutter of a smart camera, i.e., a picture is taken only if the recognition system okays the scene content, erg., for sharpness of image, that eyes are all open, for smiling expressions, and the like.

SUMMARY OF THE INVENTION (DISCLOSURE OF THE INVENTION)

The present invention is of computer software for, computer media with computer software for, and a method of calculating ring-wedge data from a digital image comprising performing a discrete Fourier transform of the digital image. In the preferred embodiment, a calculation is also performed of discrete autocorrelation, discrete cosine transform, and/or Hadamard transform. The ring wedge sampling preferably comprises calculating

m

j

=

∑

u

=

0

N

-

1

⁢

⁢

∑

v

=

0

M

-

1

⁢

⁢

&LeftBracketingBar;

F

~

⁢

⁢

(

u

,

v

)

&RightBracketingBar;

⁢

⁢

M

~

j

⁢

⁢

(

u

,

v

)

,

where m

j

is a jth measurement over a sampling area to which each pixel's degree of membership is given by {tilde over (M)}

j

(u,v),

F

~

⁢

⁢

(

u

,

v

)

=

∑

n

=

0

N

-

1

⁢

⁢

∑

m

=

0

M

-

1

⁢

⁢

f

⁢

⁢

(

n

,

m

)

⁢

⁢

exp

⁡

[

-

2

⁢

⁢

π

⁢

⁢

(

un

N

+

vm

N

)

]

,

where f(n,m) comprises digital image pixel values with 0≦n<N, 0≦u<N, 0≦m<M, and 0≦v<M. The sampling calculation preferably determines each pixel's degree of membership by employing sampling regions for ring and wedge regions defined as:

R

j

=

{

(

f

x

,

f

y

)

⁢

:

⁢

⁢

ρ

j

≤

f

x

2

+

f

y

2

<

ρ

j

+

Δ

⁢

⁢

ρ

j

,

φ

min

≤

tan

-

1

⁢

⁢

f

y

f

x

<

φ

min

+

π

}

,

⁢

and

R

j

=

{

(

f

x

,

f

y

)

⁢

:

⁢

⁢

ρ

min

≤

f

x

2

+

f

y

2

<

ρ

max

,

φ

j

≤

tan

-

1

⁢

⁢

f

y

f

x

<

φ

j

+

Δ

⁢

⁢

φ

j

}

,

where p

j

is the radial distance from the origin to the inner radius of the jth detector region, and &Dgr;&rgr;

j

is its radial width, and &phgr;

j

is the angular distance from the f

x

axis to the leading edge of the jth detector region and &Dgr;&phgr;

j

is its angular width. The sampling may be accomplished by determining each pixel's degree of membership as appropriate for either of two preferred methods: bin-summing or mask-summing, as in the descriptions to follow. The ring-wedge data is preferably provided to a neural network (most preferably a fully connected, three-layer, feed-forward neural network with sigmoidal activation functions) to perform pattern recognition on the data. The neural network may be implemented in hardware or software, as well understood by one of ordinary skill in the art. The ring-wedge data may be used in analysis of an images such as fingerprint images, images of particles, images of human faces, and satellite images, and the analysis may be for tasks such as object recognition, image quality assessment, and image content classification.

A primary object of the invention is to allow for the rapid prototyping of practical recognition systems by providing a highly effective, consistent data format for making recognition decisions.

Another object of the invention is to provide the ability to apply either of two preferred methods to any number of subshades from a single input image. In this way spatial location and other image domain information can be combined with the ring-wedge format to produce superior recognition.

A primary advantage of the invention is the provision of a digitally calculated set of dimensionally reduced data to permit practical machine learning methods to determine appropriate means of separating these data into redefined groupings.

Another advantage of the invention is that its straightforward input-to-output data flow admits it to a modular design structure allowing easy interaction with other data processing methods.

Other objects, advantages and novel features, and further scope of applicability of the present invention will be set forth in part in the detailed description to follow, taken in conjunction with the accompanying drawings, and in part will become apparent to those skilled in the art upon examination of the following, or may be learned by practice of the invention. The objects and advantages of the invention may be realized and attained by means of the instrumentalities and combinations particularly pointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated into and form a part of the specification, illustrate several embodiments of the present invention and, together with the description, serve to explain the principles of the invention. The drawings are only for the purpose of illustrating a preferred embodiment of the invention and are not to be construed as limiting the invention. In the drawings:

FIG. 1

is a schematic diagram of the prior art ring-wedge geometry preferably used in the all-digital calculation of the ring-wedge data format of the invention from digital input images;

FIG. 2

is a representational diagram of the calculations involved in calculating the ring-wedge data format using the bin-summing method of the invention;

FIG. 3

is a representational diagram of the calculations involved in calculating the ring-wedge data format using the mask-summing method of the invention;

FIG. 4

is a block diagram of the invention configured as a discrete Fourier spectral sampling system;

FIG. 5

is a block diagram of the system/method of the invention as a discrete Fourier spectral sampling incorporating image preprocessing;

FIG. 6

is a block diagram of the system/method of the invention as a discrete Fourier spectral sampling incorporating both image- and transform-domain preprocessing;

FIG. 7

is a block diagram of the system/method of the invention as a discrete Fourier spectral sampling incorporating both image- and transform-domain preprocessor, as well as ring-wedge data postprocessing;

FIG. 8

is a block diagram of the system/method of the invention tailored for image quality assessment using two separate ring-wedge data channels;

FIG. 9

is a block diagram of the system/method of the invention as a multichannel discrete Fourier spectral sampling system incorporating different image- and transform-domain preprocessing, as well as ring-wedge data postprocessing for each of the channels;

FIG. 10

is a block diagram of the system/method of the invention as a multichannel sampling system incorporating different spatial transforms, image- and transform-domain preprocessing, and ring-wedge data postprocessing for each of the channels;

FIG. 11

is a block diagram at the system of the invention as a multichannel sampling system incorporating various image domain features, as well as ring-wedge data calculated using different spatial transforms, image- and transform-domain preprocessing, and ring-wedge data postprocessing for each of the channels;

FIG. 12

is a schematic representation of image tiling according to the invention; and

FIG. 13

is a schematic representation of image rastering according to the invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS (BEST MODES FOR CARRYING OUT THE INVENTION)

The present invention is of an automatic recognition system and method that has the capability to classify or recognize a wide variety of images grouped objectively by photointerpreters or other means. The invention can be configured for stand-alone use in a digital computer or incorporated as a tool-kit in a general image processing environment.

The methods of the invention include spatial frequency sampling of an arbitrary image with selected pre-processing on one or several separate channels. These data are coupled into a neural network that is trained using a learning set of images that have been selected to span the base of classification or recognition. For the spatial frequency sampling according to the invention, the preferred embodiment is an all-digital form of the ring-wedge photodetector (772-George) and incorporating two important variations. For the neural network, excellent results are obtained using a fully connected, three-layer, feed-forward configuration with sigmoidal activation functions. The invention can also include the use of nonlinear data processing such as rank ordering to provide robustness, edge extraction preprocessing, and error images as well as the original image.

Further, the invention comprises methods and software for calculating the ring-wedge data format from digital images. The methods and software are useful for constructing automatic pattern recognition systems that use pictorial imagery as input data. These applications include but are not limited to object recognition, image quality assessment, and image content classification. The methods can be applied generally to grayscale, color, and/or abstract images that may consist of an arbitrary number of data channels. The software also allows for the ring-wedge data format to be applied to any number of subimages taken from a single input image. In this way the software allows for systems to be constructed that use both spatial location and other image domain information with ring-wedge data.

The data provided by the invention typically supply information about spatial frequency, however several transformations can be considered in addition to a discrete Fourier transform, including a discrete cosine transform and a Hadamard transform. The software, method, and system of the invention permits the inclusion of arbitrary image preprocessing and ring-wedge data post-processing, as well as various machine learning techniques through a modular design.

To reiterate, the present invention comprises methods and software for calculating numerically the ring-wedge data format from digital images. This includes but is not limited to calculations involving a discrete Fourier transform and a discrete autocorrelation, as well as other discrete transforms such as a discrete cosine transform or a Hadamard transform, which provide a point-centered frame of reference on which to perform the ring-wedge sampling.

In the preferred embodiment, the ring-wedge sampling is performed using the discrete Fourier transform (DFT) of the input digital imagery, defined as follows:

F

~

⁢

⁢

(

u

,

v

)

=

∑

n

=

0

N

-

1

⁢

⁢

∑

m

=

0

M

-

1

⁢

⁢

f

⁢

⁢

(

n

,

m

)

⁢

⁢

exp

⁡

[

-

2

⁢

⁢

π

⁢

⁢

(

un

N

+

vm

M

)

]

,

(

1

)

where f(n,m) is the digital array image with 0≦n<N, 0≦u<N, 0≦m<M, and 0≦v<M.

In general using this all-digital system, the computer system can utilize both the phase as well as the amplitude of the DFT, {tilde over (F)}(u, v). This possibility produces interesting variations of the system. As an example of the system, the following discussion presents a system that uses the absolute value of the transform, i.e, |{tilde over (F)}(u, v)|. For this example, the ring-wedge data are described by the following equation:

m

j

=

∑

u

=

0

N

-

1

⁢

⁢

∑

v

=

0

M

-

1

⁢

⁢

&LeftBracketingBar;

F

~

⁢

⁢

(

u

,

v

)

&RightBracketingBar;

⁢

⁢

M

j

⁢

⁢

(

u

,

v

)

.

~

(

2

)

where m

j

is the jth measurement over the sampling area to which each pixel's degree of membership is defined by the sampling geometry given by {tilde over (M)}

j

(u, v). The nature of this sampling geometry is described in the following paragraph.

The ring-wedge detector geometry consists of semi-annular sampling areas on one half of the sampling plane and wedge-shaped sampling areas on the other. The separate samplings over 180 degrees are based on the use of the assertion that sampling be polar symmetric, as is the case with the amplitude transmittance of the DFT of a real valued input image. When using the DFT, the radial sampling of the ring areas provides orientation-independent information about the distribution of spatial frequencies in the image while the angular sampling of the wedge areas provides scale-independent information about the orientations of those spatial frequencies. These interpretations, however, are variant with use of alternate transformations of the input image data into other appropriate sampling domains. In any case, the ring-wedge data format efficiently summarizes the information contained within an input image into a few measurements, representing a large reduction in the amount of data that the recognition system must analyze. This is an important advantage when considering machine learning methods.

When considering the digital calculation of the ring-wedge data format, a direct approach is to treat the pixels in the sampling plane as isolated points and to calculate the ring-wedge measurements by summing together the contributions from each of the sample points within corresponding detector regions of the sampling geometry.

To develop this approach, consider it in the Fourier domain as a method of approximating the following integral definition of each of the detector measurements:

m

j

=

∫

R

j

⁢

∫

⁢

ⅆ

f

x

⁢

⁢

ⅆ

f

y

⁢

⁢

&LeftBracketingBar;

F

⁢

⁢

(

f

x

⁢

⁢

f

y

)

&RightBracketingBar;

,

(

3

)

where F(f

x

,f

y

) is the continuous Fourier transform of the ideal continuous reconstruction of the sample's array image used as input to the system.

The approximation is made by subdividing the Fourier plane on a regular rectangular grid with spacings of &Dgr;f

x

and &Dgr;f

y

in the f

x

and f

y

directions, respectively. For each rectangle covering a sampling region R

j

, employ the center point (u&Dgr;f

x

,v&Dgr;f

y

) and take the product of F(u&Dgr;f

x

,v&Dgr;f

y

) with the area of the rectangle, &Dgr;f

x

&Dgr;f

y

. The sum of all such products over the region gives the corresponding measurement, as described by the equation:

m

j

≈

∑

u

⁢

⁢

∑

v

⁢

⁢

&LeftBracketingBar;

F

⁢

⁢

(

u

⁢

⁢

Δ

⁢

⁢

f

x

,

v

⁢

⁢

Δ

⁢

⁢

f

y

)

&RightBracketingBar;

⁢

⁢

Δ

⁢

⁢

f

x

⁢

⁢

Δ

⁢

⁢

f

y

.

(

4

)

In Eq. (4), the summation is over all ordered pairs of the integers u and v belonging to the discrete set {(u, v): (u&Dgr;f

x

, v&Dgr;f

y

)∈R

j

}. Note what is done with a rectangle that is on a boundary of R

j

with only part of its area contained within the detector region. In this case, the rectangle is considered to be within the detector region only if the center point of the rectangle lies within the region.

One can describe the ring-wedge sampling geometry explicitly. The sampling regions for the ring-detector regions are described as follows:

R

j

=

{

(

f

x

,

f

y

)

⁢

:

⁢

⁢

ρ

j

≤

f

x

2

+

f

y

2

<

ρ

j

+

Δ

⁢

⁢

ρ

j

,

φ

min

≤

tan

-

1

⁢

⁢

f

y

f

x

<

φ

min

+

π

}

,

(

5

)

where &rgr;

j

is the radial distance from the origin to the inner radius of the jth detector region, and &Dgr;&rgr;

j

is its radial width. Note that the angular extent of all of the ring detectors is assumed to be &pgr; radians oriented as given by &phgr;min. The sampling regions for the wedge detector regions are described as follows:

R

j

=

{

(

f

x

,

f

y

)

⁢

:

⁢

⁢

ρ

min

≤

f

x

2

+

f

y

2

<

ρ

max

,

φ

j

≤

tan

-

1

⁢

⁢

f

y

f

x

<

φ

j

+

Δ

⁢

⁢

φ

j

}

,

(

6

)

where &phgr;

j

is the angular distance from the f

x

axis to the leading edge of the jth detector region and &Dgr;&phgr;

j

is its angular width. Note that the radial extent of all of the wedge detectors is assumed to be equal, as given by &rgr;

min

and &rgr;

max

.

By approximating F(u&Dgr;f

x

,v&Dgr;f

y

) with the two-dimensional discrete Fourier transform and by recalling the implicit periodicity of the DFT, one obtains Eq. (2) with the pixel membership of the detector regions defined such that {tilde over (M)}

j

(u, v) has a value of unity for a pixel that is contained in the sampling region and zero otherwise. This approach is known as bin-summing because it can be implemented as a simple pooling of pixel values for cases where the detector regions do not overlap.

Referring to

FIG. 2

, the bin-summing method of the invention is preferably implemented as follows: First, the membership function for all of the sampling regions are defined according to the prescribed detector geometry and a given image size. Second, for each pixel location the polar coordinates are calculated according to its relative position from the detector center. The region in which the pixel belongs is then determined using these coordinates. This information can either be stored in a lookup table or used directly in calculating the ring-wedge samples. Last, for a particular image the magnitude of the DFT coefficient at each of the pixel locations is summed into its corresponding bin according to the determined region membership.

Once a lookup table is generated for a particular sampling geometry and image size, a second execution stage is entered in which images are transformed and ring-wedge data calculated by a scan through the DFT coefficients. This two-stage approach to the method eliminates the need to calculate repetitively the polar coordinates of a each pixel location during the ring-wedge calculations of every image considered. When dealing with a large number of images, eliminating these extra coordinate calculations represents a substantial time savings over a single stage implementation of the bin-summing method.

While computationally efficient, bin-summing can contain artifacts which compromise desired invariances and degrade recognition performance. This is especially true when dealing with small images. Bin-summing is most appropriate when working with images larger than 128×128 pixels. With smaller sized imagery, it is preferred to use an alternate method of the invention known as mask-summing, representationally illustrated in FIG.

3

.

In developing this method, one should consider a result from sampling theory, namely, that |F(u, v)| can be represented by samples taken on a rectangular grid with horizontal spacing &Dgr;f

x

and vertical spacing &Dgr;f

y

:

&LeftBracketingBar;

F

⁢

⁢

(

f

x

,

f

y

)

&RightBracketingBar;

≈

∑

u

=

-

∞

∞

⁢

⁢

∑

v

=

-

∞

∞

⁢

⁢

&LeftBracketingBar;

F

⁢

⁢

(

u

⁢

⁢

Δ

⁢

⁢

f

x

,

v

⁢

⁢

Δ

⁢

⁢

f

y

)

&RightBracketingBar;

⁢

⁢

I

⁢

⁢

(

f

x

-

u

⁢

⁢

Δ

⁢

⁢

f

x

,

f

y

-

v

⁢

⁢

Δ

⁢

⁢

f

y

)

,

(

7

)

where I(f

x

,f

y

) is an interpolating function.

Substituting Eq. (7) into Eq. (3) results in the following:

m

j

≈

∫

R

j

⁢

∫

⁢

ⅆ

f

x

⁢

⁢

ⅆ

f

y

⁡

[

∑

u

=

-

∞

∞

⁢

⁢

∑

v

=

-

∞

∞

⁢

⁢

&LeftBracketingBar;

F

⁢

⁢

(

u

⁢

⁢

Δ

⁢

⁢

f

x

,

v

⁢

⁢

Δ

⁢

⁢

f

y

)

&RightBracketingBar;

⁢

⁢

I

⁢

⁢

(

f

x

-

u

⁢

⁢

Δ

⁢

⁢

f

x

,

f

y

-

v

⁢

⁢

Δ

⁢

⁢

f

y

)

]

.

(

8

)

Further, by bringing the integral inside of the summation, one obtains the following result:

m

j

≈

∑

u

=

-

∞

∞

⁢

⁢

∑

v

=

-

∞

∞

⁢

⁢

&LeftBracketingBar;

F

⁢

⁢

(

u

⁢

⁢

Δ

⁢

⁢

f

x

,

v

⁢

⁢

Δ

⁢

⁢

f

y

)

&RightBracketingBar;

⁡

[

∫

R

j

⁢

∫

⁢

ⅆ

f

x

⁢

⁢

ⅆ

f

y

⁢

⁢

I

⁢

⁢

(

f

x

-

u

⁢

⁢

Δ

⁢

⁢

f

x

,

f

y

-

v

⁢

⁢

Δ

⁢

⁢

f

y

)

]

.

⁢

(

9

)

Thus, by defining

M

j

⁢

⁢

(

u

⁢

⁢

Δ

⁢

⁢

f

x

,

v

⁢

⁢

Δ

⁢

⁢

f

y

)

⁢

⁢

Δ

⁢

⁢

f

x

⁢

⁢

Δ

⁢

⁢

f

y

=

∫

R

j

⁢

∫

⁢

ⅆ

f

x

⁢

⁢

ⅆ

f

y

⁢

⁢

I

⁢

⁢

(

f

x

-

u

⁢

⁢

Δ

⁢

⁢

f

x

,

f

y

-

v

⁢

⁢

Δ

⁢

⁢

f

y

)

(

10

)

one obtains:

m

j

≈

∑

u

=

-

∞

∞

⁢

⁢

∑

v

=

-

∞

∞

⁢

⁢

&LeftBracketingBar;

F

⁢

⁢

(

u

⁢

⁢

Δ

⁢

⁢

f

x

,

v

⁢

⁢

Δ

⁢

⁢

f

y

)

&RightBracketingBar;

⁢

⁢

M

j

⁢

⁢

(

u

⁢

⁢

Δ

⁢

⁢

f

x

,

v

⁢

⁢

Δ

⁢

⁢

f

y

)

⁢

⁢

Δ

⁢

⁢

f

x

⁢

⁢

Δ

⁢

⁢

f

y

.

(

11

)

Assuming that F(f

x

,f

y

) can be approximated using discrete Fourier transform and accounting for periodicity, one again obtains Eq. (2), where the individual ring-wedge measurements are calculated by summing together values from all of the DFT coefficients, which have first been multiplied by corresponding mask values. As described above, the mask values are calculated numerically by approximating the contribution of each pixel to a particular detector region. There is a separate mask for each of the sampling regions in a given geometry.

The mask-summing method of the invention is preferably implemented as follows: First, the mask values for each of the sampling regions are calculated according to the prescribed detector geometry. For this step, an interpolating function is chosen and Eq. (10) is evaluated numerically.

After the mask values of each of the detector regions are calculated for a particular image size and detector geometry, ring-edge data can be generated for a particles image by successively multiplying the DFT coefficients by each mask in the detector and summing over the affected pixels.

For either of the two calculation methods, input image data is transformed into one of several possible sampling domains and a reduced set of data measurements is calculated. As such, the methods permit any combination of image preprocessing and ring-wedge data postprocessing to allow the software to be incorporated in a practical recognition system. Several permutations of this are presented in accompanying drawings (FIGS.

4

-

11

).

FIG. 4

is a block diagram of the invention configured as a discrete Fourier spectral sampling system with the following components: O, input object; L, imaging lens; I, image sampling system; FFT, fast Fourier transform; RW, digital ring-wedge detector; and NS, neural-network software, hardware, or hybrid system.