FIELD OF THE INVENTION

This invention relates generally to detecting anomalies in complex data and, more particularly, to an artificial neural network based system useful in analyzing hierarchical data such as wavelet processed image data.

BACKGROUND OF THE INVENTION

Neural network based systems are known, however, known systems typically are not capable of analyzing a large collection of complex data. Rather, neural network based systems typically operate on subsets of the complex data. For example, and to detect abnormalities in image data, known neural network systems typically only analyze subsets of an entire data set, which reduces the effectiveness of the analysis.

One particular application in which this limitation of known neural network based systems has a significant adverse consequence is in mammography screening. Particularly, breast cancer is now estimated to strike one in eight adult American women, and many national institutions are promoting large-scale breast cancer screening programs. Even with the above described limitations, computer-aided diagnosis techniques have been applied to mammography screening programs and such techniques may offer substantial benefits in terms of cost reduction and increased effectiveness of the screening process. The use of computers to directly prescreen mammograms may eventually permit a substantial reduction in the number of studies that must be viewed by a radiologist. Of course, to achieve such a reduction, computers must be able to directly interpret digitized images and the process must be fully automated.

With respect to digitized images, mammograms show only an estimated 3% of their actual information content. Improvements in the visibility of mammographic information content will probably improve detection of small tumors. It is unlikely, however, that state-of-the-art screen-film radiography alone can be improved to display more information.

Wavelet transformation, a known image enhancement technique, has been used successfully to enhance the visibility of image information content in mammograms, including both masses and microcalcifications. Wavelet image representations also permit high magnitudes of data compression without loss of important image features.

To interpret the digitized images, several rule-based systems use thresholding, subtraction, or both. These techniques have been hampered by high false-positive detection rates. Artificial neural networks (ANN) are an alternative to traditional rule-based (symbolic) methods for computer-aided detection of mammographic lesions. ANNs learn the significance of image features based upon example training images. In general, ANNs are adept at pattern recognition problems.

As explained above, however, most all known ANN systems perform direct digitized data analysis of mammograms using only small regions of interest (ROI) selected from an entire image. Other ANN systems extract features, either qualitative or quantitative, for network training, or have incorporated ANNs into other computer-aided diagnosis schemes to improve lesion detection.

It would be desirable to provide a fully automated system for detecting anomalies in complex data, including for analyzing entire images to identify lesions. It also would be desirable to provide such a system which does not generate a high false-positive detection rate.

SUMMARY OF THE INVENTION

These and other objects may be attained by fully automated methods and systems for processing images to identify anomalies in which an entire set of hierarchical data (i.e., an ordered data set) is processed so that the entire set can be analyzed by a neural network. Such processing is performed utilizing wavelet processing. Particularly, and in one embodiment, multiresolution (five-level) and multidirection (two-dimensional) wavelet analysis with quadratic spline wavelets is performed to transform the data set. For example, and for image data, the wavelets are a first-order derivative of a smoothing function and enhance the edges of image objects. Because two-dimensional wavelet transformations quantize an image in terms of space and spatial frequency and can be ordered linearly, images can be processed recursively to determine prominent features.

In another aspect, the present invention relates to processing the preprocessed data so that the entire complex data set can be analyzed by a neural network. Particularly, the wavelet coefficients form a hierarchy which is linearized as a sequence of triplets: (coefficient, hierarchical level, position in level) by canonical topological sorting and parsed using a neural network approach based on sequential recursive auto-associative memory (SRAAM). Since the wavelet coefficients are continuous, linear output instead of sigmoidal output is used in the SRAAM. This variation is therefore referred to as linear output sequential recursive auto-associative memory, or LOSRAAM.

At the completion of LOSRAAM training, the coefficients and their context units so derived can be used in the network to predict the preceding wavelet coefficient triplet within a predetermined error tolerance. The activation pattern associated with the context units constitute a vector in state space. The input half of the LOSRAAM neural network maps the coefficient triplets and state space vectors into other state space vectors. The objective of LOSRAAM training is to make this mapping reversible using the output half of the LOSRAAM network. Thus, given any but the first wavelet coefficient information, the network can approximately predict the prior wavelet coefficient, its hierarchical level, its position within the level, and the activation of the context units that will predict the preceding wavelet coefficient triplet. Thus, the entire set of wavelet coefficients theoretically can be predicted from the last pair of hidden (context) units.

The LOSRAAM training process is repeated several times varying the number of context units and other training parameters, and each resultant LOSRAAM network may provide a different interpretation of the internal structure of the data. The set of context unit activation patterns arising from the LOSRAAM hidden layer defines a set of vectors in a state space. These vectors are subjected to cluster analysis.

The cluster analysis yields identifiable and discrete states. From these states, a feature vector is created for each image. Each element in the feature vector represents the number of times the corresponding state from the above cluster analysis is found in each image. The feature vectors are further processed from discrete counts to continuous, fuzzy values by weighting them according to the average fit each state vector achieves with the model state vector for that cluster. Thus, each image may be represented as a fuzzy feature vector (FFV).

Then, several feed forward neural networks (FFNNs) are trained to classify the FFVs. FFNNs differ according to their starting weights, the use of hint units, and other training parameters. Such FFNN training may be performed using the conjugate gradient method modified for use with a selfscaling error function, optimized to reduce the number of derivative computations, and monitored for restart conditions. The FFNN may also contain special (hint) output units whose target values are chosen to constrain training results by introducing coarse positional output information.

The above described system is fully automated for detecting anomalies in complex data, such as for prescreening images. For mammography studies, therefore, the system is capable of analyzing entire images to identify lesions. The system also does not generate a high false-positive detection rate.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1

is a high level flow chart illustrating the process steps for image processing in accordance with one embodiment of the present invention.

FIG. 2

is a high level flow chart illustrating the process steps for image preparation using area resampling in accordance with one embodiment of the present invention.

FIG. 3

is a high level flow chart illustrating the process steps for image preparation using wavelets in accordance with one embodiment of the present invention.

FIG. 4

is a diagram of wavelet hierarchy showing the number of coefficients that may result at each level under one choice of wavelet transformation.

FIG. 5

is a high level flow chart illustrating the process steps for structure extraction using LOSRAAM neural networks and clustering in accordance with one embodiment of the present invention.

FIG. 6

is an exemplary schematic illustration of a LOSRAAM neural network using two context units.

FIG. 7

is a graph illustrating exemplary LOSRAAM clusters for one specific clustering run.

FIG. 8

is a table illustrating the results of the exemplary LOSRAAM clustering of wavelet coefficients shown in FIG.

7

.

FIG. 9

is a high level flow chart illustrating classification and evaluation in accordance with one embodiment of the present invention.

DETAILED DESCRIPTION

In one aspect, the present invention relates to fully automated methods and systems for processing images to identify anomalies. As described below in more detail, the system includes wavelet processing, recursive processing to determine prominent features, and feed forward neural network (FFNN) processing to classify feature vectors generated in the wavelet and recursive processing. The present invention, however, is not limited to processing image data, and can be utilized to analyze any collection of data.

In addition, and in another aspect, the present invention relates to the preprocessing of a set of complex data for further analysis. Specifically, difficulties associated with compressing a large set of complex data without loss of resolution has limited the use of neural network based systems. By preprocessing the data as described below, even a very large and complex ordered data set can be analyzed by a neural network based system. In addition, the preprocessed data could be utilized in rule-based systems and probability based systems, for example, and is not limited to use in neural network based systems.

FIG. 1

is a high level flow chart

100

illustrating the process steps for image processing in accordance with one embodiment of the present invention. As shown in

FIG. 1

, digital data is first collected

102

. In the exemplary embodiment, the data is image data and can be collected using known digitizing systems. One known digitizing system is commercially available from Eastman Kodak, Rochester, N.Y., and this known system incorporates a 100-um focal spot film digitizer (a model 200 digitizer commercially available from Lumisys, Sunnyvale, Calif.). After digitization, the images may be on the order of 2,400×1,800 pixels×12 bits.

After collection of the digital image data, wavelet preprocessing is performed

104

. As described below in more detail, and in connection with wavelet preprocessing, two dimensional wavelet selection and two dimensional wavelet processing (spatial and frequency quantization) are performed. In particular, and in one embodiment, multiresolution (five-level) and multidirectional (two-dimensional) wavelet analysis with quadratic spline wavelets may be utilized to transform each square image. Such wavelet processing is described, for example, in Mallat et al., Characterization Of Signals From Multiscale Images, IEEE Trans Pattern Anal Machine Intell, 1992, 14:710-732, which is incorporated herein, in its entirety, by reference. The wavelets are a first-order derivative of a smoothing function and enhance the edges of image objects.

Subsequent to wavelet preprocessing, structure extraction operations

106

are performed. The structure extraction process generally includes linear output sequential recursive auto-associative memory, or LOSRAAM, training. The output of the LOSRAAM operations is a set of state vectors, and these extracted state vectors are subjected to cluster analysis.

In the cluster analysis, M identifiable and discrete states are determined, and from these states, a M-element feature vector may be created

108

. In this example, M=6. The feature vectors are further processed from discrete counts to continuous, fuzzy values by weighting them according to the average fit each state vector achieves with the model state vector for that cluster. Each image is then represented as an M-element fuzzy feature vector (FFV). In the example, representing each image as a six-element fuzzy feature vector (FFV), with each element contained in 4 bytes, gives a final compression ratio which enables analysis of large complex data sets in feed forward neural network (FFNN) training.

To further illustrate the digital image collection process,

FIG. 2

is a high level flow chart

110

illustrating the process steps for image preparation using area resampling for mammography. Particularly, in mammography, images can either originate from a digital x-ray machine

112

, or from film (i.e., analogue x-ray)

114

which is then digitized

116

. Usually, mammograms for screening consist of two analogue x-ray images, and each image is scanned and digitized using, for example, a film digitizer system. One known digitizing system is commercially available from Eastman Kodak, Rochester, N.Y., and this known system incorporates a 100-um focal spot film digitizer (a model 200 digitizer commercially available from Lumisys, Sunnyvale, Calif.). After digitization, the images may be on the order of 2,400×1,800 pixels×12 bits.

Subsequent to scanning (if necessary) and digitizing the image, all further image processing and neural network training may be performed on a computer such as a SPARC Center 2000, commercially available from Sun Microsystems, Sunnyvale, Calif. Such a computer has twenty available processors.

Each image is resized

118

using an area-resampling algorithm. Specifically, the area-resampling algorithm determines the value of each pixel in the new image in proportion to the area from which it is derived in the original image. An example of area-resampling is described in Newman et al., Principles of Interactive Computer Graphics, New York, N.Y., McGraw-Hill, 1979, pages 402-404. As a result of area-resampling, the image is re-sized from a rectangular matrix to a square matrix (e.g., 1,280×1,280 pixels, each 1 byte in depth).

FIG. 3

is a high level flow chart

120

illustrating the process steps for image preparation using wavelets. As shown in

FIG. 3

, two dimensional wavelet selection

122

and two dimensional wavelet processing (spatial and frequency quantization)

124

are performed. In particular, and in one embodiment, multiresolution (five-level) and multidirectional (two-dimensional) wavelet analysis with quadratic spline wavelets may be utilized to transform each square image. Such wavelet processing is described, for example, in Mallat et al., Characterization Of Signals From Multiscale Images, IEEE Trans Pattern Anal Machine Intell, 1992, 14:710-732, which is incorporated herein, in its entirety, by reference. The wavelets are a first-order derivative of a smoothing function and enhance the edges of image objects.

The wavelets transform the data set referenced above to, for example, 1,092,800 4-byte floating point coefficients within the interval (−1,1) and distributed across five levels as shown in FIG.

4

. The wavelet coefficients can be viewed hierarchically, as illustrated, descending along the two dimensions (x followed by y) at each of the five levels of resolution, terminating with the remaining (nondecomposed) coefficients at level five. The total number of coefficients and the applicable multiplier is shown at each level.

For high quality mammogram reconstruction, a truncated wavelet hierarchy can be utilized wherein coefficients below a certain threshold are discarded, as described by Laine et al., Wavelet Processing Techniques For Digital Mammography, Conference on Visualization in Biomedical Computing, Chapel Hill, N.C., October, 1992, which is incorporated herein, in its entirety, by reference. In this example, a coefficient threshold of 0.25 provides acceptable results for subsequent processing as described below.

After truncation of the wavelet transform, with use of 0.25 as the coefficient threshold, a data set of 702 images may be represented by 3,386,421 triplets (wavelet coefficient, hierarchy level, position within level) of 4-byte floating point values with an average of 4,824 triplets per image. Thus, after wavelet transformation and truncation, a compression ratio of 75.5:1 is achieved.

Because two-dimensional wavelet transformations quantize an image in terms of space and spatial frequency and can be ordered linearly, images can be processed recursively to determine prominent features. In the present invention, the wavelet coefficients form a hierarchy which is linearized as a sequence of triplets: (coefficient, hierarchical level, position in level) by canonical topological sorting and parsed using a neural network approach based on sequential recursive auto-associative memory (SRAAM). This linearization method is fully described in Kwasny et al., Tail Recursive Distributed Representations and Simple Recurrent Neural Networks, Connection Science, 1995, 7:61,80 which is incorporated herein, in its entirety, by reference. Since the wavelet coefficients are continuous, linear output instead of sigmoidal output is used in the SRAAM, which extends the [+1,−1] bounds of the output values indefinitely. This variation is therefore referred to as linear output sequential recursive auto-associative memory, or LOSRAAM.

FIG. 5

is a high level flow chart

130

illustrating the process steps for structure extraction using LOSRAAM neural networks and clustering. An objective of auto-associative training

132

is to have the output exactly match the input. Therefore, the LOSRAAM network is constructed by using an input external layer and a mirror-image output external layer with each fully connected to a hidden layer. Each external layer has three input-output units which correspond to the values in wavelet triplets, and a number of context units, as shown in FIG.

6

. In

FIG. 6

, linear input and output units are represented as ellipses and context input and output units are represented as circles. Between these two external layers is a hidden layer with the same number of hidden units as context units, two in this example. The hidden units are represented as rectangles. The number of hidden and context units chosen for the LOSRAAM architecture may be determined empirically by testing architectures with differing numbers of units. The activation values of the context units in the input layer are set by using the values developed in the hidden units from input of the immediately preceding wavelet coefficient triplet. The values of the initial context units associated with the first wavelet coefficient triplet in the series are set to zero. Each of the three layers is fully connected with adjacent layers in a feed-forward direction as shown in FIG.

6

. Training is carried out on the entire data set until a plateau is reached and no better prediction can be obtained. Because this network is used as a classification system for the wavelet coefficients, it is tested only on data used in training the network.

At the completion of LOSRAAM training, the coefficients and their context units so derived can be used in the network to predict each preceding wavelet coefficient triplet within a predetermined error tolerance. The activation pattern associated with the context units constitute a vector in a state space. The input half of the LOSRAAM neural network maps the coefficient triplets and state space vectors into other state space vectors. The objective of LOSRAAM training is to make this mapping reversible using the output half of the LOSRAAM network. Thus, given any but the first wavelet coefficient information, the network can approximately predict the prior wavelet coefficient, its hierarchical level, its position within the level, and the values of the context units that will predict the preceding wavelet coefficient triplet. Thus, the entire set of wavelet coefficients theoretically can be predicted from the last pair of hidden (context) units. However, the residual network prediction error would be propagated additively by using just this single vector. Consequently, the entire set of context unit activation patterns is captured.

The set of context unit activation patterns arising from the LOSRAAM hidden layer defines a set of vectors in a state space. These vectors are subjected to cluster analysis

134

using, for example, a method such as described in Fu, Neural Networks in Computer Intelligence, McGraw-Hill, St. Louis, Mo., 1994, which is incorporated herein, in its entirety, by reference, modified to handle data that cluster at any point, including at the origin. This analysis, when applied during a previous, simpler study which used only two hidden units and a sensitivity factor of 0.5, yielded six identifiable and discrete states as shown in FIG.

7

and the accompanying numeric table in FIG.

8

.

Referring again to

FIG. 5

, and from these states, an N-element feature vector may be created

136

for each image. The process described in Dai, Wavelet Applications In Process Sensor Data Analysis, Doctoral Dissertation, Washington University, St. Louis, Mo., 1996, which is incorporated herein, in its entirety, by reference, may be utilized to create the feature vector. Each element in the feature vector represents the number of times the corresponding state from the above cluster analysis is encountered under LOASRAAM processing of each image.

The feature vectors are further processed from discrete counts to continuous, fuzzy values by weighting them according to the average fit each state vector achieves with the model state vector for each cluster. Representing each image as a six-element fuizzy feature vector (FFV), in this example, with each element contained in 4 bytes, gives a final compression ratio for this data set on the order of 218,000:1. This level of compression makes it feasible to use large complex data sets in FFNN training.

The above described process for compressing a data set can be used to reduce a complex data set to practical size for presentation to a FFNN. Therefore, it should be understood that such process is not limited to just mammography analysis, nor just to use with image data. The above described process can be used to compress most any complex data set.

Singular value decomposition (SVD), a well-known mathematical process, may be used to preprocess the FFVs which are the inputs to the FFNNs. SVD transforms the columns of the matrix formed by the set of input vectors into an orthogonal matrix. Training is believed to proceed better with the transformed inputs. SVD assigns a singular value (SV) to each column of the transformed matrix. If the SV is below an arbitrary threshold, 1000 for example, the input unit for this column corresponds to input noise and that unit is removed from the FFNN. This helps in controlling the number of variables in the FFNN and may help in improving the generalization performance of the FFNNs. Because SVD is a well-defined linear transformation, it has a well-defined inverse. This inverse transformation can be used to back transform the weights on the input layer connections yielding a network which accepts inputs in their original orientation.

A process of training LOSRAAM neural networks as well as feed forward neural networks (FFNNs) is described in Kalman et al., High Performance Training Of Feed Forward And Simple Recurrent Networks, Neurocomputing, 1997, 14:63-84, which is incorporated herein, in its entirety, by reference. Such neural network training may be performed using the conjugate gradient method modified for use with a self-scaling error function, optimized to reduce the number of derivative computations, and monitored for restart conditions with a technique described by Powell, Restart Procedures For The Conjugate Gradient Method, Math Programming, 1977, 12:241,254, which is incorporated herein, in its entirety, by reference.

The LOSRAAM training process described above is repeated several times (

FIG. 5

) varying the number of context units and other training parameters, including but not limited to changing the set of random start weights and training set data. Each resultant LOSRAAM network may provide a different interpretation of the internal structure of the data. Subsequent processing described below occurs for each LOSRAAM network so derived.

The architecture of a FFNN is configured by connecting all units in one layer with those in another layer. This process may result in connections which do not contribute in any significant way to the task. Once a FFNN is trained, these superfluous connections may be identified and removed. This technique is a type of network surgery in which any connection (except threshold connections) which has a value near zero is removed. The network so reduced is then re-trained to yield a network with fewer adjustable parameters. This process may have little affect on the training performance level, but potentially can affect the ability of the network to generalize to unseen cases.

FIG. 9

is a high level flow chart

140

illustrating classification and evaluation. Particularly,

FIG. 9

illustrates classification

142

, voting

144

, and evaluation

146

steps. Several feed forward neural networks (FFNNs) are trained to classify FFVs. The PFNNs may contain special output units referred to as hint units whose target values are chosen to constrain training results, as described in Yu, Extra Output Biased Learning, Technical Report AI90-128, University of Texas, March, 1990, Austin, Tex., which is incorporated herein, in its entirety, by reference. A hint may represent information about the spatial location of the anomaly, for example.

Each of the multiple LOSRAAM solutions leads to multiple FFNN solutions for their own particular part of the process using different restart conditions and with or without the availability of hints. Each combination of solutions represents a sample classifier from the enormous solution space generated by the nonlinear, multivariable optimization task of classifying images. If it is assumed that these classifiers are at least somewhat independent, they may be combined through voting

144

, according to the binomial probability distribution, to produce a single classifier with improved performance. Having the multiple networks vote to predict whether a given mammographic pair of images shows a mass, for example, is much better than relying on the prediction of a single best network.

In this example, each of three LOSRAAM networks generates 4 FFNNs resulting in 12 network classifiers. The number of networks is a realistic tradeoff determined by diminishing returns as the number of networks increases. In practice, the overall system with voting on samples representing non-masses (specificity) is believed to significantly outperform a system that uses a single best network. Performance on samples representing masses (sensitivity) is about the same as with voting as with a single network. A single network may predict 57% of the non-mass images correctly and 67% of the mass images. Using the 12 network classifiers and voting, in this example, 61% of the non-mass cases and 81% of the mass cases may be predicted correctly. Using the single best network may predict only 50% of the non-mass cases and 82% of the mass cases.

The above described system is fully automated for prescreening images and is capable of analyzing entire images to identify lesions. Such a system also does not generate a high false-positive detection rate.

From the preceding description of various embodiments of the present invention, it is evident that the objects of the invention are attained. Although the invention has been described and illustrated in detail, it is to be clearly understood that the same is intended by way of illustration and example only and is not to be taken by way of limitation. Accordingly, the spirit and scope of the invention are to be limited only by the terms of the appended claims.