VIDEO IMAGE PROCESSING

This invention relates to video image processing, and more particularly to the generation of motion vectors indicating the direction and magnitudes of apparent movement to different regions of an image. Such vectors can then be used to assist in the generation of desired output images.

In our United Kingdom Patent GB-B-2188510 a method of processing a video image is described which produces a list of motion vectors which are applicable over a measurement block in a video image. An appropriate one of these motion vectors may be assigned to each region of the image. Each region may be as small as a picture element (pixel) or it may comprise a plurality of picture elements or an arrangement block of the picture.

Such motion vectors are particularly useful when it is desired to generate output video fields which correspond to instants in time intermediate two input fields. Such output fields are necessary in video image processing systems which transfer video images to or from film, in standards conversion, or in producing slow motion effects. In our International Patent Application No.

PCT/GB92/00952 methods for generating the motion vectors for assignment to either a particular input field or over a particular interfield interval are described.

In a conventional PAL television signal successive fields are interlaced. That is to say each field contains data to be displayed on alternate lines of a video display. This is known as 2:1 interlace. We have found that when vectors are derived between a pair of successive interlaced video fields errors can occur in the vectors derived therefrom as a result of the fact that data is on alternate interlaced picture lines.

For this reason it is preferable to perform the derivation of vectors by methods such as phase correlation between successive frames of the video image, e.g. by correlating between alternate fields of a 2:1 interlaced signal such that there is no vertical difference between the lines of data which are being correlated. If such an approach is adopted, however, the maximum motion speed which can be measured is half that which could be measured using interfield correlation.

Preferred embodiments of the present invention, therefore, seek to provide a solution to these problems such that interfield

correlation can be performed without any significant errors occuring in the motion vectors determined thereby.

The present invention is defined in the appended claims to which reference should now be made.

Preferred embodiments of the invention will now be described in detail by way of example with reference to the accompanying figures in which:

Figure 1 a) and b) show a pair of interlaced video fields and the result of a vertical shift imposed on them in an embodiment of the invention;

Figure 2 shows a first system embodying the invention;

Figure 3 shows a second system embodying the invention;

Figure 4 shows an alternative embodiment of the system of figure 3.

Figure 5 shows a two-tap interpolation scheme for use in an embodiment of the invention; and

Figure 6 shows a six-tap interpolation scheme for use in an embodiment of the invention. Figure 1a) shows a pair of successive interlaced video fields at their respective intervals in time. The lines of the first video field are represented by x's and the lines of the second video field by x's in circles. The vertical displacement of the interlaced video fields by one line can be seen clearly.

In figure 1b) the same pair of video fields can be seen after shifting by an embodiment of the present invention. Each of the fields of figure la has been shifted by a half picture line, the first field having been shifted downwards and the second field have been shifted upwards by this amount. As a result the picture lines of both fields are vertically aligned in figure 1b). Arrows in figure 1a) indicate the direction and magnitude of the shift.

The two vertically aligned fields can then be correlated according to the method described in our United Kingdom Patent GB-B-2188510 to derive a set of motion vectors for assignment to areas of the picture. Other methods of deriving vectors such as block matching could also be used.

The process by which the two fields are aligned has been described above as shifting the input fields. The process by which this is achieved is in practice more complex than simply moving data from positions shown in Fig. 1a) to those shown in Fig. 1b). The processing circuitry does not in fact have an indication of the vertical position data will occupy on a display. What in fact occurs is that data in the line positions of Fig. 1b) is

interpolated from data in the line positions of Fig.1a). The resultant line positions of Fig. 1b) have data assigned to them which would in fact have been present if the signal had been produced by scanning those line positions.

The schemes by which new lines of a video signal may be

interpolated will be familiar to those skilled in the art and may be implemented using conventional hardware or software. What is required for the purposes of the embodiments of the invention described herein is a spatial interpolation taking data from other lines in the same field as the new lines to be

generated, multiplying the data from each other line by a constant and adding the resultant sums together. The constants are selected to sum substantially to unity so that the data for a new line position will be of the same order of magnitude as that in the old lines of the video signal.

For example, Figure 5 shows a simple two-tap interpolation producing the line positions of Figure lb) from those of Figure 1a). The lines which contribute to the new lines have arrows originating from them which end at the new lines to which they contribute. Each arrow has a multiplication constant corresponding to the constant by which the line from which it originates is multiplied. In this example each new line is comprised of data from two adjacent lines multiplied by constants of 0.25 and 0.75. The nearer of the original lines has the constant of 0.75 assigned to it.

It is preferable to use a more sophisticated interpolator to derive data for the new line positions. It has been found that a six-tap interpolator gives good results. An example of such an

interpolator is shown schematically in Figure 6 for a single field. Again the new lines to be derived are at one quarter field line positions. The generation of one new line is shown. This new line comprises data from each of the six nearest lines in the field and multiplied by constants K1 - K6 respectively. The values of the constants vary with distance from the new line position and are chosen to give different weights to different lines. Various different weighting schemes may be used. We have found that if there are no aliasing components in the input video signal, a high order interpolator (e.g. six-tap) will give the correct value at the inbetween points where new lines are to be generated.

The effect of this vertical re-alignment and subsequent correlation of the fields is to double the range of maximum velocity which can be detected in the sequence of video images. There is also a reduction in the amount of noise on the phase correlation surface derived. This reduction in the noise is a result of there being less uncorrelated material in the correlation surface as a result of the higher sampling rate achieved. It will be appreciated that vectors derived between successive fields are better able to follow complex motion in the video sequence and rapid accelerations than would be possible with correlations produced at the frame sampling rate.

We have discovered that some small vertical motions are detected by the system which were not present in the original input video signal. These are probably due to alias components in the original source material. This effect can be removed with an additional temporal fileter as described below.

In order to improve further the correlation noise a temporal filter may be used with the invention and two possible types are described in our International Patent Application No. PCT/GB92/00952. The advantages of using such a temporal filter are to improve the signal to noise ratio of the correlation surface, to retime the correlation surface prior to assignment, and to reduce the alternate vertical errors which arise from the fact that successive correlation surfaces are derived from alternate interline positions in the video signal.

Figure 2 shows a first embodiment of a circuit including a temporal filter. This circuit operates according to what is referred to as the three field method of our International Patent Application No. PCT/GB92/00952. It produces an output list of vectors suitable for assignment to the central field of three successive input fields, the field present at point A in the figure.

Input video fields pass through successive field delays 2, thus making three successive video fields available for phase correlation. In operation this results in a first pair of video fields being applied to the respective down and up inputs of a field shifter 4 (interpolator). This shifts the field data as shown in figure 1 and produces output fields which form the input to a phase correlator 6. At the same time the field of the pair which is present at point A also passes to the down input a second field shifter 8 (interpolator) which has its up input receiving a field which has been delayed by a further field period than the field at point A. This second field shifter shifts the field data in a direction opposite to that of figure 1. This produces half line shifted fields which are input to a further input phase correlator 10. Thus the line positions in the fields input to correlator 8 are interlaced with those input to correlator 10. Thus correlation surfaces are derived between first and second, and second and third fields in the video sequence.

It will be appreciated that the fields output by field shifter 4 are at vertical positions which are interlaced with the fields output by field shifter 8. Thus a field input to the system of figure 2 is subjected to first a down and then an up shift by field shifter 4 and a down and then an up shift by field shifter 8. It is this alternate shifting (interpolating) of fields down and up which reduces the alternate vertical errors in an output picture.

The phase correlation surfaces produced by the phase correlators 6 and 10 are each multiplied by 0.5 in multipliers 12 and the outputs of these are then combined in an adder 14 the output of which is sent to a peak motion vector detector 16. This produces a list of vectors which can then be assigned to elemental areas of the video image.

Figure 3 shows a second system embodying the invention in which a further stage has been added to the system of figure 2. This implements what is known as the two field algorithm of our

International Patent Application No. PCT/GB92/00952 and produces a set of vectors which are applicable over the interfield interval A shown in the figure. Operation of the system is as described in relation to figure 2 except that a further delay means 2 has been added and this enables field shifting (interpolating) and phase correlation of a further pair of fields to be performed. Therefore an additional field shifter 18 (interpolator), phase correlator 20 and multiplier have also been added. The co-efficients by which the multipliers in the system operate have also been changed. The central multiplier 12 has a co-efficient of 0.5 whilst the other two have co-efficients of 0.25. All the outputs are combined in adder 14 which provides a correlation surface which is input to the peak vector detector 16 which produces a set of vectors ready for assignment to the video signal.

An alternative embodiment of the system shown in figure 3 is illustrated in figure 4. In this a +/- picture shifter circuit 22 (interpolator) shifts alternate input video field data up or down by half a picture line. The shifted fields then pass to a phase correlator 24 which produces a phase correlation surface between the pairs of video fields. Two delayed versions of the phase correlation surface output by correlator 24 are produced by field delays 26. The outputs of the two field delays and the phase correlator are multiplied by co-efficients in the three multipliers 28. This central multiplier has a co-efficient of 0.5 whilst the other two have co-efficients of 0.25. The outputs of these multipliers are combined in an adder 14 before passing to a peak vector detector 16 which produces a list of vectors for assignment to the picture.

This circuit can be modified to mimic the circuitry of figure 2 simply by setting the co-efficient on the right hand multiplier to 0 and those on the other two multipliers to 0.5. This will then produce a set of vectors for use in the three field algorithm.

The intention with both temporal filters is that odd co-efficients should total the same as the even co-efficients. Thus in the system shown in figure 3 and 4 the odd co-efficients (the outer pair) sum to 0.5, the same as the even central co-efficients. Thus the same weight is given to vectors derived from each shifted field position used for phase correlation.

Systems embodying the present invention have been described in terms of hardware circuitry but it will be readily apparent to a man skilled in the art that they could also be implemented in general purpose hardware programmed with appropriate software.