VIDEO SIGNAL PROCESSING

This invention relates to video signal processing and more particularly to motion compensated processes.

A variety of motion estimation techniques have been proposed. In certain approaches, a list of trial or candidate motion vectors is created in a first processing step and - in a subsequent step - motion vectors are assigned by determining for each pixel, which of the trial vectors produces the best match. A particularly useful example of the first processing step is the known technique of phase correlation.

It has previously been appreciated that the phase correlation procedure can be expanded to create, for each of the trial vectors, a confidence level which may be related to the height of the associated peak in a correlation surface. As explained in copending application WO 94/01970, this confidence level can be used in subsequent processing. It is thus suggested that a motion vector is scaled by reference to its confidence level. It may at first be surprising that any benefit, or indeed any sensible result, can be achieved by varying the length of a motion vector in dependence upon a confidence level. It has been found, however, that in circumstances where accurate estimation of motion is impossible, the approach provides a smooth transfer between "conventional" motion compensation where the confidence level in a motion vector is assumed to be 100%, and a non-motion compensated process where the confidence level is effectively zero.

It is an object of this invention to provide an improved method of video signal processing in which motion vectors are assigned still more effectively.

Accordingly, the present invention consists in a method of video signal processing in which an output motion vector [vector(p)] is required to be associated with a picture element p, there being N trial vectors [vector_n(p)] for that picture element , wherein the output motion vector is derived according to:- ∑ ? vecto ) * bias„(p) vectoι p) =

where [bias_n(p)] is a bias value associated with the n'th trial vector.

Thus, instead of selecting the vector offering the best match from a list of trial vectors, the present invention takes a vector sum of the trial vectors, each weighted by a bias factor which may, for example, be related to the degree of matching it offers.

In an extreme case, in which one trial vector offers a very much higher degree of fit than the others, the method according to the present invention will produce an output vector which tends toward the vector produced conventionally. In less ideal conditions, it is believed that the present invention will - largely through making greater use of the information available to it - produce a motion vector with a picture result which is more visually acceptable than the conventional approach.

Suitably, the bias value [bias_n(p)] for each vector is related to the degree of matching [match_n(p)] afforded by that vector. Advantageously, each trial vector [vector_n(p)] has a probability factor

[probability_n(p)] derived during determination of the trial vectors and the bias value [bias_n(p)j is related to the probability factor [probabillty- .

In this way, the vector sum is able to make use of the information in the probability factors, which may be derived from the height of the corresponding peaks in the correlation surface.

Preferably the bias value is given by:-

bias_n(p) = probability_n(p) * match_n(p)

In one form, the present invention recognizes that with processing usually being conducted on a block-by-block basis, the accuracy or relevance of a trial vector may vary with the location of the corresponding pixel within the block Thus, more attention should usually be paid to information derived from towards the centre of a block. With pixels at the edge of a block, for example, the "correct" motion vector may well take the pixel outside the block in question and may therefore not appear in the list of trial vectors. Since the blocks will usually overlap to a considerable degree, that "correct" motion vector should appear as a trial vector in another block which contains that pixel, towards its centre. More generally, a vector derived in a block-based process is - according to this invention - no longer regarded as having equal relevance to all picture elements in the block; its relevance to a particular picture element is now to be regarded as a function of the location of that picture element in the block.

Accordingly, in a preferred form of this invention, the bias value [bias_n(p)] is related to a block factor [block(p)] which is larger towards the centre of the block.

In one form of the invention, the bias value is given by:-

blas_n(p) = probability_n(p) * match_π(p) * block(p)

In any particular arrangement, a decision may be made to omit one of the three factors probability-, (p)/match_n(p)/block(p) or, indeed, in an extreme case to omit two of the factors. Thus, a situation can be envisaged in which trial vectors are being produced with uniformly high probability factors derived, for example from the heights of the associated peaks in a correlation surface. It might then be the case that the amount of additional information provided by attempts to match pixels at each trial vector is small and insufficient to justify the processing overhead which is involved. Then, a vector would be derived according to:-

, „ , ∑ ;.T v∞to ) * block p vectotiβ) = — —

The probability factor can optionally also be taken into account.

It is a further object of the present invention to provide an improvement in a method of motion estimation, such as phase correlation, which operates block by block.

Accordingly, the present invention consists, in still a further aspect, in a method of motion estimation between fields, comprising dividing each field into overlapping blocks such that each picture element appears in B blocks, where B is greater than one, performing a correlation between fields for each block to provide at least one vector [vector_b(p)] for each picture element; associating with each vector a block weighting factor [block_b(p)] which varies with the position of the corresponding picture element within the block and deriving a motion vector by summing according to:-

vectonjή = ∑ vector_b p) * block_b(p)

The block weighting factors [block_b(p)j can, in a situation where the division of a field into overlapping blocks is fixed, be arranged so as to sum to unity for every picture element. Alternatively, the sum can be normalised as described above. The factors probability(p); match (p) and block(p) which have been mentioned in the various aspects of this invention can be derived in a variety of ways and examples will be given. In circumstances where there is a prospect of the normalisation coefficient summing to zero, an appropriate constant can be added, thus:-

, . , ∑ 7" vector „ ) * bias_n(p) vectotip) =

∑^? biasj ή ₊ constant

In a further aspect the present invention addressees the manner in which match errors are calculated.

In the conventional approach, a match error of a fixed number of luminance units is given the same significance, irrespective of picture content. In other words, it is assumed that inter-field luminance difference is a measure of vector error that applies universally. The present invention recognises that a particular vector error will produce a relatively small inter- field luminance difference when the rate of change of luminance is low, but a relatively high inter-field luminance difference where the rate of change of luminance is high. Thus the conventional approach, when testing two vectors of equal error might accept the vector in a region of low rate of change of luminance but reject the vector in a region of high rate of change of luminance. This is felt to be inappropriate.

Accordingly, the present invention consists in this aspect in a method of determining inter-field differences, picture element by picture element, comprising subtracting picture values from respective fields and comparing the subtraction result with a measure of the rate of change of picture value.

The invention will now be described by way of example with reference to the accompanying drawings in which:-

Figure 1 is a diagram illustrating the division of fields into blocks for a block-based correlation procedure.

Figures 2, 3 and 4 are plots of weighting functions for block, probability and match factors, respectively.

Figure 5 is a diagram illustrating the range concept in accordance with one aspect of the present invention. Figure 6 shows a pre-processor according to one embodiment of the present invention; and

Figure 7 is a series of diagrams illustrating the computation of match error.

The example is taken of motion estimation using phase correlation, operating on blocks of, typically, 64 by 64 pixels. The technique of phase correlation does not require description here and reference is directed, for example, to GB 2 188 510.

Blocks are overlapped both horizontally and vertically, so that each pixel is covered by four blocks (ie. B=4). Within each block, trial vectors are identified by detecting peaks in a correlation surface. Various strategies can be adopted for controlling the number of trial vectors which are identified in a block. For example, all peaks above a threshold level can be detected; the highest, two, three, four or more peaks can be detected or a combination of these two approaches. For the sake of this example, the simplifying assumption will be made that two trial vectors will be defined in each block. Accordingly, for each pixel, eight trial vectors are defined. According to the prior art approach, those trial vectors are applied in turn to the pixel and a comparison made between fields. The trial vector providing the smallest difference between fields is taken to be the correct motion vector. In contrast, the present invention proposes taking a vector sum of all eight vectors, weighted accorded to a number of factors. These factors will be considered in turn.

The division of field F, and F₂ into overlapping blocks is shown in Figure 1. The blocks may conveniently be numbered as b=1 ; b=2 (horizontally displaced by half a block); b=3 (vertically displaced by half a block) and b=4 (horizontally and vertically displaced by half a block). The division of a field into blocks for a process such as phase correlation takes into account a number of factors. If the block size is too small in comparison with typical displacements arising through motion in one field interval, no meaningful correlation can be expected. However, the FFT and inverse FFT operations required in phase correlation are highly processor intensive and there is accordingly a practical limit on the size of block that can be processed.

Turning now to Figure 2, there is shown a plot of the block weighting factor [block (p)] against horizontal displacement over the blocks b=1 and b=2. The block weighting factor is seen be fall linearly from the block centre. For a pixel such as P, (seen in Figure 1), the weighting in block b=1 would be, say, 0.9 and the weighting in block b=2 would be 0J . Accordingly, much greater weight would be applied to a trial vector at pixel P, which was just generated in a process based on block b=1 than one generated in a process based on block b=2. A similar weighting can be applied in the vertical direction to block b=3, whilst block b=4 would receive both horizontal and vertical weightings. Thus a pyramidal weighting surface can be envisaged dealing with both horizontal and vertical departures from the block centre.

It will be recognised that the weighting function need not be linear. Also, it may be convenient for the weighting function to fall to a non-zero value at the block boundary. This would reflect the fact that blocks are typically defined in a windowing function that provides "soft" block edges.

Turning now to the probability weighting factor, this can be arranged to include information from various stages in the process by which trial vectors are identified. The simplest measure of probability in the phase correlation process is probably the height of the corresponding peak in the correlation surface. With other techniques, there will be alternative measures of motion estimation error. The function chosen to relate the probability weighting factor to the measure of error or confidence in the vector will depend upon the nature of the measure. An example is given in Figure 3 of a probability weighting function based on peak height in a correlation surface.

The degree of match (which is the measure of vector "correctness" in prior art systems) is typically generated by a simple subtraction of luminance values between pixels in the respective fields linked by the trial vector. In the present invention, this match error is then related to the degree of matching factor [match (p)] by a function such as that shown in Figure 4.

However, the present invention also contemplates a fresh approach to the determination of match error.

In the conventional approach, a match error of a fixed number of luminance units is given the same significance, irrespective of picture content. In other words, it is assumed that inter-field luminance difference is a measure of vector error that applies universally. The present invention recognises that a particular vector error will produce a relatively small inter- field luminance difference when the rate of change of luminance is low, but a relatively high inter-field luminance difference where the rate of change of luminance is high. Thus the conventional approach, when testing two vectors of equal error, might accept the vector in a region of low rate of change of luminance but reject the vector in a region of high rate of change of luminance. In the approach described above according to this invention, where vectors are weighted according, among other things, to a measure of match error, utilising the same approach would apply a falsely depressed weighting in the region of high rate of change of luminance. In a new approach, the present invention - when considering match error - will take into account not only absolute inter-field differences but also the rate of change of luminance.

In one example, illustrated in Figure 5, a measure of luminance range is taken about a pixel P. Thus, utilising a fixed interval, high (R_H) and low (R_L) range values are taken and a value R generated by subtraction.

Conveniently, an average value is computed as the mid-point of R_H and R_L. These computations can be conducted in a pre-processor, as shown in Figure 6. This processor can, through appropriate taps, take into consideration both horizontal and vertical variations. It is found that vertical variations are considerably more important. This is primarily because of the problems associated with interlace. Thus, the fact that successive fields contain alternate lines means that high vertical frequencies and horizontal edges can give rise to very large inter-field differences which bear no relation to motion. The interiace problem is of course not encountered in the horizontal dimension; further, the signal will usually have undergone some form of horizontal filtering. The pre-processor is preferably arranged and optimised so as to deal with the most acute problem, that is to say interlace. Appropriate non-linear processing can be employed to that end.

The conventional calculation for producing match error (ME) may be written in the form:-

ME = | P. - P₂ | According to this aspect of the present invention, the match error is defined as:-

ME = | Av- - Av₂ | - R, - R₂ where:- _ Ru ~ n,

R = — V— — -

R_u

Av •H + R,

In this calculation, negative match errors are ignored. The consequences of calculating match error in this new way are best understood with reference to diagrams of Figure 7. Essentially, the calculation is checking whether the vertical (in the diagram) separation between the two average values in sufficiently large that the associated ranges do not overlap. Thus in case (a) of Figure 7, the difference between the average values is smaller than the sum of the ranges and a match error of zero would be output. In case (b), the difference between the average values exceeds the sum of R, and R₂ by an amount which is taken as the match error ME. Cases (c) and (d) illustrate the cases in which the difference between average values is of the opposite sign.

The described approach will effectively ignore a match error unless it is large compared with the measured ranges. The calculation that has been quoted is of course only one of a large number of possible examples. Calculating a range value at each picture element is a convenient method of providing a measure of the rate of change but other alternatives exist. Similarly, the arithmetic approach by which match error is calculated as the difference between the highest range point in one case and the lowest range point in another, whilst effective and simple, is only one example of how rate of change of luminance information can be taken into account. It would be possible to compute range values, or conduct other rate of change processing, at the same time as calculating match error. However, considerable advantage is seen in the described approach of a pre-processor. Since, pre-processing is conducted once, there can be a considerable saving in processing overhead where a large number of blocks are treated simultaneously. It is also ensured that range information is taken from both fields, which might be less easy where all processing was required to be conducted at the same time. There will be an increase in memory requirement, since both a range value and an average value have to be held for each pixel. However, this is likely to be less significant than the reduction in processing overhead.