Background of the invention

The present invention relates to a method of processing a moving picture, and particularly relates to a picture-processing method of extracting an image of a moving object, which is suitable for a process in which an edge image of a moving object is extracted from a moving picture inputted successively in real time from an image pick-up element moving within a plane perpendicular to an optical axis.

Conventional systems for processing pictures relating to an image of a moving object execute the process by using values obtained from time differentiation or other similar operations with respect to each picture element of a moving picture inputted successively. The time differentiation or similar operations by an electronic circuit can be executed at a very high speed, since this circuit has a simplified construction. It is difficult, however, to extract an image of a moving object only from the values obtained from the time differentiation or similar operations. When a circular area shifts its position before a plain background, for instance, crescent-shaped area having different signs are outputted in the front and the rear in the shifting direction, and it is hard to extract the circular area from these pieces of information.

There is another method, in this connection, in which images obtained from an image pick-up element in a conventional manner are stored once in a storage device, pictures are then taken out one by one from the storage device, and the respective corresponding point of each picture is calculated so as to extract an image of a moving object. While the image of a moving object can be extracted correctly according to this method when the corresponding point can be found out, since an amount of movement of the object is calculated correctly in this case, a longer time is required for processing said point. It is very difficult, in addition, to detect the corresponding point in the case that there is a rotation, a change in shape or others of the object.

There is still another known method in which only an image of a background wherein the image of a moving object is not contained is stored beforehand in a device, and images inputted successively are compared with the image of the background so as to extract the image of the moving object. This method can not be applied at all when the image pick-up element is moved or when the background changes. In Japanese Patent Laid-Open No. 8087/l984, a mechanism for renewing an image of a background is contained. By using this mechanism, instantaneous images not containing an image of a moving object are taken successively in a device and stored as a background image therein. This is a contrivance not to regard a slow change in time of the background mistakenly as the image of the moving object. According to this method, however, the renewal of the image of the background is not conducted at all when the moving object is present constantly within a pick-up scope, for instance, and it is impossible to cope with a sharp instantaneous change of the background. In addition, this method is also ineffective when the image pick-up element is moved.

Summary of the invention

An object of the present invention is to furnish a picture-processing system which enables the extraction of an edge image of a moving object in real time and with high precision from a moving picture obtained from an image pick-up element which is moved within a plane perpendicular to an optical axis.

The present invention resides in that an unchanged feature portion or a slowly-changing feature component of an input image, obtained from an image pick-up element is taken and stored as a background, an image of the background is transferred vertically and horizontally so that this image of the background may correspond to an image inputted afresh, the stored data on the background made to correspond to the inputted image by the transfer are compared with feature parameters of a picture inputted every moment, and information on a moving object is extracted from the comparison.

Brief Description of the Drawings

• Fig. l is a block diagram of an embodiment of a system for extracting an image of a moving object according to the present invention;
• Fig. 2 is illustrational views of picture frames for showing processes wherein an image of a moving object is extracted from an input picture;
• Figs. 3(a) and 3(b) are illustrative diagrams showing coefficients of a spatial filter employed when an edge is extracted;
• Figs. 4(a) and 4(b) are illustrative views showing a construction of a memory;
• Fig. 5 is a block diagram of an address information forming mechanism;
• Fig. 6 is a block diagram of a comparison and background edge image renewal unit;
• Figs. 7(a) to 7(e) are illustrational diagrams for explaining processes wherein an edge image of a moving object is formed;
• Fig. 8 is illustrational views of picture frames which are objects for computation of correlation values; and
• Fig. 9 is illustrational views of picture frames for showing processes wherein a background image is formed.

Description of the Invention

An embodiment of the present invention is described hereunder in detail, referring to the drawings.

In Fig. l showing a system for extracting an image of a moving object according to the present invention, numeral l0l denotes an image pick-up element which is movable in a plane perpendicular to an optical axis picks up pictures so as to cover a prescribed pick-up scope wherein a moving object is contained, and delivers outputs la such as ordinary video signals from the image pick-up element l0l to a picture memory l02.

The image pick-up element l0l moving in a plane perpendicular to an optical axis picks up such scenes 20l, 202 and 203 as illustrated in Fig. 2. In the pictures picked up, a moving object (a child, a cat or a bird, in this case) is contained in a stationary or slowly changing background (a house, in this case, which is slowly changed in the picture).

The picture memory l02 is an ordinary memory in which the video signals la are converted into digital signals through an A/D converter not shown, and these signals of picture data are then stored therein by one frame. The picture memory l02 outputs digital signals lb of an input image stored in the picture memory l02 sequentially one by one of the picture elements of the input image to a feature extracting unit l03. The feature extracting unit l03 extracts an edge enhanced image of the input image stored in the picture memory l02. Namely, the feature extracting unit l03 takes in picture element data lb sequentially from the picture memory l02 and extracts edges from the input image, and then the results of the extraction, which are data lc relating to the edge image of the input image data lb, are forwarded to a picture memory l04 for storing an edge-enhanced image. The data lc of the edge image are stored in the picture memory.

It is known generally in the field of psychology that contours play a very important role in the recognition of an object by a man. Based on this knowledge, the feature extracting unit l03 is provided for obtaining an edge image of a moving object. In the field of picture processing, a spatial filter as shown in Fig. 3(a) has been used heretofore for extracting a contour (a portion wherein lightness is sharply varied spatially). Each numerical value in Fig. 3 represents a coefficient of convolutional integration. In general, the lightness is varied sharply on the boundary of an object, and when this filter is applied to the vicinity of said boundary, the value of the convolutional integration turns to be a large positive or negative value. The output of the convolutional integration turns to be a value approximating to zero inside the object, since the lightness is varied gently therein. Contours can be extracted by conducting said convolutional integration around all the picture elements of the picture data stored in the memory l02. When applied to such a moving picture dealt with by the present invention, however, the spatial filter of Fig. 3(b) brings forth problems as stated below:

• (l) A slippage of about one picture element often occurs in a picture owing to a slippage of a sync signal between frames; and
• (2) The filter of Fig. 3(b) acts to enhance the noises that are superposed on each separate picture element of the picture due to various factors.

Therefore, in the present invention, a spatial filter as shown in Figure 3(a) is applied as a filter suitable for extracting the edge image of a moving object in a moving picture. In the filter of Fig. 3(a), the expansion of a weight coefficient is made larger than that in the filter of Fig. 3(b) so as to cope with the problems of synchronization of the picture between frames and of separate noises. The arithmetic device of the spatial filter is a publicly-known processing device which is provided with sixteen multipliers and executes at high speed sum-of-products operations of sixteen-neighbors with respect to the picture.

The picture memory l04 is a storage device for storing the data of the edge image of the input image, which is stored in the picture memory l02 and outputted from the feature extracting unit l03, by one picture plane or one frame of an edge image.

The picture memory l05 stores one frame of the edge image corresponding to a background equivalent in dimensions to a scope that can be picked up at one time by the image pick-up element l0l, that is, it stores an edge-enhanced background image by one frame. Namely, the picture memory l05 recollects background images such as images 204, 205, 206 in Fig. 2, corresponding to the input images 20l, 202, 203. The picture memory l05 has a specific structure in which an edge image in a scope equivalent to the scope which can be covered at one time by the image pick-up element l0l can be stored by one picture frame, and in which the top and bottom of a picture plane or picture frame, and the right and left thereof are connected to each other, respectively, thereby to be made continuous, and the picture can be transferred in parallel with ease.

A picture is usually stored in a region having consecutive addresses in a storage device, as shown in Fig. 4b. Accordingly, the right and left of the stored picture are usually connected together as shown in Fig. 4a. When a mechanism as shown in Figure 5 is attached, as an address computing mechanism, to said storage device, a picture storage plane is made to have a structure wherein the top and bottom thereof are connected together, and moreover, the picture stored therein can be transferred in parallel on an assumed basis. Thus formed storage device is the above-mentioned picture memory l05.

In the mechanism shown in Fig. 5, numerals 50l, 504 denote input values of an x-coordinate and a y-coordinate on the picture respectively, while numerals 502 and 505 denote values representing which position on the actual storage device the origin of the picture comes to. A y-­coordinate value is multiplied by a length of one line by a multiplier 508, and the address on the storage device is computed by adding an x-coordinate value to the product of the above multiplication. From the value thus calculated, the residue is calculated dividing the value with the size of one picture plane as a divisor, using a residue operation unit 5l0. When the picture plane is constructed by 256 × 256, for instance, the calculation of said residue is required only to put out lower l6 bits in dual representation of an output of 509, and therefore it can be realized very easily. By using the picture memory associated with the aforesaid mechanism, the origin of the picture information stored in the memory l05 can be transferred easily, and as the result, the picture can be transferred in parallel.

Numeral l06 in Fig. l denotes a mechanism or a corresponding area indicating unit which calculates an amount of the parallel transfer of a background edge enhanced image stored in the memory l05 so that the background edge image may correspond to the edge image stored in the picture memory l04. The unit l06 generates signals for recollecting the background images as shown by 204 to 206 of Fig. 2. Namely, the corresponding area indicating unit l06 calculates and indicates the direction of the transfer of the background edge image in the picture memory l05, which direction of transfer is necessary for making the image therein correspond to the edge image in the picture memory l04, by computing a correlation function of the edge image in the picture memory l04 and the background edge image in the picture memory l05. The computation of the corrrelation function requires much time in general, and the execution thereof at a high speed requires very much time and a lot of hardware. In the present invention, therefore, high-­speed correlation computation is enabled by little hardware on the basis of the following assumption.

The speed of picture processing is higher enough than the speed of movement of the image pick-up element l0l is the plane perpendicular to the optical axis, and so the amount of transfer of a background between successive pictures is equivalent at most to one picture element of the picture taken in by the picture pick-up element l0l.

Based on the above-stated assumption, an area wherein the correlation is taken is limited to an area deviated by one picture element from the preceding corresponding picture. Deviation is assumed to occur in eight directions, i.e. upward, downward, right, left, right-­upward, right-downward, left-upward and left-downward directions, and, in addition, the case of the position of a picture being unvaried can be assumed. Accordingly, the correlation is computed with respect to nine-­neighbors. A background image is shifted in the direction in which a correlation value is maximum, and it is so modified as to correspond to an input picture. This operation is executed every time a picture is inputted, and thereby a background image corresponding to an input picture can be outputted constantly.

Numeral l07 denotes a comparison and background edge image renewal unit which compares with corresponding picture elements of two images (for example, an image 20l, 202, 203 in Fig. 2 and background image 204, 205, 206) stored in the corresponding regions of the picture memories l04 and l05, respectively, and delivers an edge image of the moving object (for example, a moving image 207, 208, 209 in Fig. 2), while renewing the data stored in the picture memory l05 simultaneously.

The comparison and background edge image renewal unit l07 has a function of delivering information on an image of a moving object based on the comparison of each picture element in the picture memory l04 with a corresponding picture element of the edge image of a background in the picture memory l05 which is shifted in the direction of parallel transfer indicated by the unit l06, and a function of renewing the edge image of a background stored in the picture memory l05, based on the image in the picture memory l04.

The detailed construction of the comparison and background edge image renewal unit l07 is illustrated by means of a block diagram in Fig. 6. In Fig. 6, the unit l07 receives signals lg and lh of the corresponding picture elements sequentially from the picture memories l04 and l05, and computes the difference between signals lg and lh by a difference calculator 60l to output a signal 6a of the difference. The signal 6a is inputted to a comparator 602 to compare it with a signal 6b of a predetermined value T (zero or little larger than zero), thereby taking the inputted values as they are when they are lager than T, and zeros when they are smaller than T. The picture inputted from the picture memory l04 contain the edge of the background and that of the moving object, while the picture inputted from the picture memory l05 contains only the edge of the background. With respect to the positive portions of the images of these edges, the output of the picture memory l05 is subtracted from the output of the picture memory l04 for each corresponding picture element, and the value lj of the results is outputted as it is as an edge image regarding the image of the moving object when it is positive, while zero is outputted when said value is smaller than T. Figs. 7(a) to 7(e) show schematically the respective processings of the signals. It is assumed that an input picture and a background image are given signals having such luminance levels as shown in the highest places of the figure, respectively. In each of Figs. 7(a) to 7(e), the axis of abscissa is an axis of space, while the axis of ordinate shows the intensity of light. When an input picture including a background image as shown in Fig. 7(a) is taken is the edge image of the input picture and that of the background image as shown in Fig. 7(b) as well as the positive portions thereof as shown in Fig. 7(c), are obtained. When the output of the picture memory l05 is subtracted from the output of the picture memory l04, an edge which exists in the input picture and not in a background picture (i.e. an edge relating to the moving object) shows a positive value, and an edge which does not exist in the input picture, but exists in the background picture (i.e. an edge of the background hidden behind the moving object) shows a negative value, as shown in Fig. 7(d). Accordingly, an edge image relating to the moving object can be obtained from the positive portions of these data, as shown in Fig. 7(e). This information can be outputted as it is.

Referring back to Fig. 6, the comparison and background edge image renewal unit l06 calculates a weighted mean with respect to each picture element in the picture memories l04 and l05 and makes it a value of the picture memory l05 afresh, thereby forming or renewing the edge image of the background.

Namely, a multiplier 603 executes multiplication of a preset value (2ⁿ-l) and data of picture elements obtained from the picture memory l05, and outputs the result to an adder 604. The adder 604 adds the output from the multiplier 603 and the data lg of picture element from the picture memory l04 and outputs the result to a n-bit shifter 605. The n-bit shifter 605 shifts the output from the adder 604 to the right by a preset value (n) and outputs the result to the picture memory l05. A memory device controller 606 controls address lines and control lines so as to correctly execute writing and reading to the picture memories l04 and l05. The controller 606 outputs addresses of picture elements to be processed to the picture memories l04 and l05 through signals 6g, 6h and set the picture memories l04 and l05 in a state to be ready to read. The data of the corresponding picture elements of the picture memories l04 and l05 are outputted as signals lg and lh. The data are processed in the difference calculator 60l and the comparator 602 to output a signal lj relating to a moving edge image. The controller 606 is left in the state to output the above-mentioned addresses, and set the picture memory l05 in a state to be ready to write in a proper time. As a result, the background edge image stored in the picture memory l05 can be renewed through a signal li as follows: , wherein BK_new(x,y): data of picture elements relating to renewed background enhanced edge image in the coordinates (x,y),

BK_old(x,y): data of picture elements relating to the background edge image before the renewal in the coordinates (x,y),

I(x,y): data of the picture elements relating to an edge image of the input picture in the coordinates (x,y), and

n: a preset value.

The above operation is repeated by one frame.

Referring to Figs. 8 and 9, operations of extracting the image of a moving object which are executed by the system having the above-described construction will be shown hereunder schematically. 806 of Fig. 8 shows a background edge image stored in the picture memory l05. The representation by broken lines shows that the background edge image has not been completed. When an image 80l is inputted, the correlation values thereof with images 802 to 8l0 are calculated by the unit l06. In the present case, the correlation values with 80l to 807 are the highest irrespective of the presence of noise (the fragment of a roof) caused by the parallel transfer of the image and the presence of a human image. After the image stored in picture memory l05 is transferred in parallel, the image 80l is compared with each picture element. As the result, only the human image contained in the image 80l is extracted as the image of a moving object. Next, the background edge image in the picture memory l05 is renewed by using the image 80l. Fig. 9 shows processes in which this background edge image is formed. 80l, 802, 803 and 804 show edge images of an input picture, while 805, 806, 807and 908 show processes in which edge image of a background is formed. It is assumed that the picture memory l05 in which the edge image of the background is stored is cleared at the beginning. When an input picture 90l containing images of a man and a house is inputted, these images are stored a little as a background image as shown in 905. When an input image 902 containing images of a bird and the house is inputted subsequently, correlation values are calculated as shown in Fig. 8, the image of 905 is transferred in parallel in accordance with these values, and an image thus obtained is renewed by an the image 902. In other words, the image of the house being present continuously is enhanced as shown in 906, and the image of the bird is stored a little, while the image of a man not being present this time turns faint. When an input image 903 containing images of a cat and the house is inputted, likewise, the image of the house is further enhanced. Finally, an image 908 is formed as the edge image of the background irrespective of the movement of the image pick-up element l0l.

According to the present embodiment, a picture processing system can be constructed in which only the edge image of a moving object can be extracted irrespective of the movement of the image pick-up element l0l within a plane perpendicular to an optical axis.

While the amount of movement of the image pick-up element l0l is assumed to be equivalent to one picture element on a picture plane in the present embodiment, the same mechanism can be designed also with respect to two, three or more picture elements, and a system for extracting the image of a moving object can be constructed in the same way.

The preset invention can be utilized for picture processing relating to a moving object, etc., since the edge image of the moving object can be extracted without using a large-scale system, such as a large-sized computer, and at high speed.