BACKGROUND OF THE INVENTION

This invention relates generally to multiple target tracking, and more particularly to a method and system for tracking multiple targets in a surveillance system.

Karl J Molnar et al, IEEE Transactions on Signal Processing, Vol. 46, No. 1, January 1998 discloses an iterative procedure for time-recursive multitarget/multisensory tracking.

Jordan M I et al, Machine Learning, Kluwer Academic Publishers, Boston, US, Vol. 37, No. 2, November 1999, pages 183-234 discloses the use of variational methods for inference and learning in graphical modes.

Tracking multiple targets is important in many applications, such as, for example, video surveillance, traffic monitoring, human activity analysis, sports video analysis and so forth. In addition to tracking the location of a target, other properties of the target such as its velocity, scale etc. can also be tracked. Analysis of the track of a target enables prediction of the future path of the target so that appropriate action can be taken. For example, tracking human activities in a crowded area such as an airport is important so that unusual activities may be detected and any possible damage may be prevented.

It is easier to track targets whose appearances are distinctive since multiple independent single-target trackers can be used to track them. In such a situation, all targets other than a specific target can be viewed as background due to their distinct appearance. However, it is difficult to track multiple targets whose appearances are similar such as people in crowded spaces. Multiple target tracking is fundamentally different from single target tracking and requires complex data association logic to partition detected measurements to each individual data source, and establish their correspondence with the maintained trackers. This implies two important processes that decide the success of a multi-target tracking algorithm - tracker-measurement association and tracker filtering, which are, in essence, two interleaved properties. Further, such multiple target tracking has to deal with target occlusion, in addition to other problems associated with single target tracking. In other words, a target must be recognized and tracked even while it is occluded or blocked by other objects.

Common approaches to tackling this problem take a centralized representation of a joint association vector, which is then estimated either by exhaustive enumerations, such as joint probabilistic data association (JPDA) filter, or by probabilistic Monte Carlo optimization. However, in these methods, the computational complexity involved is tremendous, especially when a large amount of tracks and measurement data needs to be handled. Sampling-based approaches have also been proposed to model the joint likelihood function, thus estimating the combined state of all targets directly. Without resorting to explicitly computing the data association, the sampling-based approaches demonstrate the capabilities of tracking multiple targets when complex motions are present. However, due to the centralized nature of the joint state representation, the complexity of these approaches grows exponentially as the number of targets to be tracked increases.

In light of the above discussion, there is a need for a method providing reduced computational complexity for tracking multiple targets.

BRIEF DESCRIPTION OF THE INVENTION

The present invention is defined in claims 1 and 6.

An exemplary embodiment of the invention provides a method and system for distributed tracking of multiple targets in a surveillance system using a variational Expectation-Maximization (EM) algorithm. For each successive frame received, a detecting module detects multiple targets in the received frame and provides the detections to a tracking module. The tracking module includes a plurality of trackers. Each tracker calculates its own motion state variable in the E-step of the variational EM algorithm. Further, each tracker calculates its data association variable with one of the multiple target detections in the M-step of the variational EM algorithm. The distributed tracking system poses constraints on the values of data association variables of the plurality of trackers thereby preventing unreasonable data associations. Based on the calculated motion state variable and data association variable, each tracker tracks its corresponding target.

Another exemplary embodiment of the invention provides a tracker capable of calculating its own motion state variable and data association variable. Each tracker calculates its own motion state variable in the E-step of a variational EM algorithm. Further, each tracker calculates its data association variable associating the tracker with one of the multiple targets in the M-step of the variational EM algorithm and provides the information related to its calculated data association variable to other trackers in the distributed tracking system. Based on the calculated motion state variable and data association variable, each tracker tracks its corresponding target. Further, each tracker updates its own data association variable on the basis of the information received from the other trackers and is capable of tracking a target even when the target is partially occluded by an object or by another target.

These and other advantages and features will be more readily understood from the following detailed description of preferred embodiments of the invention that is provided in connection with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

• FIG. 1 depicts an environment in which embodiments of the invention may be practiced.
• FIG. 2 is a block diagram depicting a system for multiple target tracking in accordance with an embodiment of the invention.
• FIG. 3 depicts pair-wise data association constraint of multiple trackers in accordance with an embodiment of the invention.
• FIG. 4 is a block diagram depicting a tracker in accordance with various embodiments of the invention.
• FIG. 5A, 5B and 5C depict association between trackers and targets in consecutive frames in accordance with various embodiments of the invention.
• FIG. 6 is a flowchart depicting a method for multiple target tracking in accordance with an embodiment of the invention.
• FIG. 7A and 7B are flowcharts illustrating a method for multiple target tracking in accordance with another embodiment of the invention.

DETAILED DESCRIPTION OF THE INVENTION

Various embodiments of the invention provide a method and system for distributed tracking of multiple targets. A variational Expectation-Maximization (EM) algorithm is used to calculate motion state variables and data association variables of a plurality of trackers. The plurality of trackers track their corresponding targets on the basis of the calculated motion state variables and data association variables.

FIG. 1 depicts an environment 100 in which embodiments of the invention may be practiced. Environment 100 includes three targets 102a, 102b and 102c, and a display 104. Display 104 displays the targets being continuously tracked on its screen. Display 104 may be employed in a crowded area such as an airport, so that human activities at the airport can be tracked and unusual activities may be detected. Video images displayed on display 104 may be monitored by a human operator and/or reviewed later by a human operator. Such video images may also be monitored by an automated system, thus saving human labor and improving detection rates. Although FIG. 1 shows only three targets being tracked and displayed on a display, any suitable number of targets may be tracked and displayed on the display. Further, a number of displays also may be employed to facilitate multiple target tracking.

FIG. 2 is a block diagram depicting a system 200 for multiple target tracking in accordance with an embodiment of the invention. System 200 includes an input module 202, a frame 204, a detecting and tracking module 206, and a display 208. Detecting and tracking module 206 includes a detecting module 210 and a tracking module 212. Input module 202 provides frame 204 to detecting and tracking module 206. Detecting and tracking module 206 detects and tracks multiple targets present in frame 204 by using a variational Expectation-Maximization algorithm. Detecting and tracking module 206 provides its output to a display 208 that displays the multiple targets being tracked.

In accordance with various embodiments of the invention, input module 202 may be a normal CCTV video source or any other video source. Frame 204 includes multiple targets that need to be tracked. Thus, targets that are being tracked are received as a sequence of frames 204. Frame 204 comprises pixels and each pixel can have different characteristics such as brightness, contrast, color, and so forth. Display 208 may be any screen capable of displaying targets being tracked.

Detecting module 210 detects targets within frame 204. Detecting module 210 may detect the targets by techniques well known in the art. Tracking module 212 tracks the multiple targets by using a plurality of trackers that are initialized after the multiple targets are detected. The multiple targets are tracked by the plurality of trackers by using the variational EM algorithm iteratively. In particular, for every frame received, the multiple targets are detected and each tracker predicts the position of its corresponding target for the next frame and tracks its corresponding target using the variational EM algorithm. Each of the multiple targets may be shown in a rectangular region on display 208. The exact process of tracking the multiple targets is explained in detail in the paragraphs below.

Consider that, in a current frame t received from input module 202, m_t measurements are detected by detecting module 210 and which are denoted by Z_t, where Z_t = {z_{1, t}, z_{2, t}. ..., z_{mt, t}}. m_t Each measurement refers to a different target being detected in the current frame t. The measurement data collected over a complete set of frames is depicted by Z^t, where Z^t = {Z₁, Z₂, ..., Z_t}.

In accordance with various embodiments of the invention, M trackers are represented in a distributed manner and each tracker i, where i represents tracker identifier and i ε {1, 2, ..., M}, has two unknown variables to be estimated, {a_{i, t}, x_{i, t}}. In other words, every tracker i is considered to be associated with a data association variable, a_{i, t} and a motion state variable, x_{i, t} which need to be determined for tracking a target correctly. a_{i, t} denotes data association variable of tracker i and can take values from a discrete set {0, 1, ..., m_t}. Thus, the tracker i can associate itself with every possible measurement z_{ai, t}, t from Z_t, or associate itself with noting, that is, a_{i, t} = 0. In particular, a_{i, t} = 0 indicates missing detection of a target or the target disappearing from the field of view. Similarly, x_{i, t} denotes motion state variable of the tracker i. Depending on each specific application scenario, x_{i, t} may describe properties of the target, such as target location, velocity, scale, and the like. The data association and motion state variables of the M trackers are denoted as a_t = {a_{1, t}, a_{2, t}, ..., a_{M, t}}, x_t = {x_{1, t}, x_{2, t}, ..., x_{M, t}}. Each of the M trackers is assigned a unique identification (ID).

In essence, multiple target tracking algorithms deal with the problem of estimating a posteriori probability p(x_t, a_t |(Z^t) which requires complex computation capability due to the heavily interleaved nature of {a_t, x_t}. In accordance with various embodiments of the invention, a marginal posteriori is estimated over one variable and the other variable is treated as hidden under missing data formulation. Further, the multiple targets tracking problem is solved by iteratively repeating the variational EM algorithm.

In accordance with various embodiments of the invention, the motion state variable of the tracker, that is, x_t is considered as the missing variable. This provides a continuously increased estimation of a probabilistic distribution over x_t in the E-step of EM iterations. Next, in the M-step, a point estimate (maximum a posteriori) of the data association variable a_t is calculated which may be optimized by graph-based optimization techniques, such as multi-way graph cut algorithm and max-product belief propagation algorithm. Although the invention has been explained with respect to M-step following the E-step, it is apparent that the M-step may be performed prior to performing the E-step. The order in which the two steps are performed does not affect the output of the EM algorithm.

Accordingly, the multiple target tracking problem is formulated as a maximum a posteriori (MAP) estimation problem of the data association variable a_t as follows: $$a_t^{*}=\underset{a_t}{\mathrm{arg\; max}}E\left(a_t\right)=\underset{a_t}{\mathrm{arg\; max}}\log p\left(a_t|Z^t\right)$$

where E(a_t) represents the original objective function that needs to be maximized. Eqn. (1) can also be represented in the following form: $$a_t^{*}=\underset{a_t}{\mathrm{arg\; max}}\log \underset{x_t}{\int }p\left(a_t,x_t|Z^t\right)d{x}_t$$

From Jensen's inequality, a function Q(x_t) is introduced in Eqn. (2) as $$\begin{array}{l}{a}_t^{*}=\underset{a_t}{\mathrm{arg\; max}}\log \underset{x_t}{\int }Q\left(x_t\right)\frac{p\left(a_t,x_t|Z^t\right)}{Q\left(x_t\right)}d{x}_t\\ \ge \underset{a_t,Q\left(x_t\right)}{\mathrm{arg\; max}}\underset{x_t}{\int }Q\left(x_t\right)\frac{p\left(a_t,x_t|Z^t\right)}{Q\left(x_t\right)}d{x}_t\\ =\underset{a_t,Q\left(x_t\right)}{\mathrm{arg\; max}}\tilde{E}\left(a_t,Q\left(x_t\right)\right)\end{array}$$

where the equality holds only when optimal association a_t^* is determined and Q(x_t) = p(a_t^*, x_t | Z^t). Maximizing the original objective function E(a_t) can be achieved by iteratively maximizing the lower bound function Ẽ(a_t, Q(x_t)) over its two unknown properties, a_t and Q(x_t).

In principle, though Q(x_t) can be defined as any valid probabilistic distribution over x_t, in this case, Q(x_t) is defined as $$Q\left(x_t\right)={\displaystyle \prod _i^M}{Q}_i\left(x_{i,t}\right)$$

where each factorial Q_i(x_{i, t}) approximates the unknown marginal probabilities p(x_{i, t})| Z^t).

From Eqn. (3), the maximization of the lower bound function Ẽ(a_t,Q(x_t)) can be expressed as $$\begin{array}{ll}\underset{a_t,Q\left(x_t\right)}{\max }\tilde{E}\left(a_t,Q\left(x_t\right)\right)& =\underset{a_t,Q\left(x_t\right)}{\max }\underset{x_t}{\int }Q\left(x_t\right)\log {}_{}p\left(a_t,x_t|Z^t\right)d{x}_t+H\left(Q\left(x_t\right)\right)\\ & =\underset{a_t,Q\left(x_t\right)}{\max }\underset{x_t}{\int }Q\left(x_t\right)\log {}_{}p\left(a_t,x_t|Z^t\right)d{x}_t+H\left(Q\left(x_t\right)\right)+\log p\left(Z^t|Z^{t-1}\right)\\ & =\underset{a_t,Q\left(x_t\right)}{\max }\underset{x_t}{\int }Q\left(x_t\right)\log p\left(a_t,x_t,Z^t|Z^{t-1}\right)d{x}_t+H\left(Q\left(x_t\right)\right)\end{array}$$

where H(Q(x_t)) is the entropy of Q(x_t) and p(Z_t | Z^t-1) is an added constant. Applying the chain rule to the term p(a_t, x_t, Z_t | Z^t-1), $$p\left(a_t,x_t,Z_t|Z^{t-1}\right)=p\left(x_t|Z^{t-1}\right)p\left(a_t|x_t,Z^{t-1}\right)p\left(Z_t|a_t,x_t,Z^{t-1}\right)$$

Using a Markovian assumption, the priori probability of the data association variable p(a_t |x_t, Z^t-1) may be simplified as p(a_t | x_t, Z^t-1) = p(a_t |x_t) and the likelihood model as p(Z_t |a_t, x_t, Z^t-1) = p(Z_t |a_t, x_t). Maximization of the lower bound function in Eqn. (5) can then be expressed as $$\begin{array}{c}\underset{a_t,Q\left(x_t\right)}{\max }\tilde{E}\left(a_t,Q\left(x_t\right)\right)=\underset{a_t,Q\left(x_t\right)}{\max }H\left(Q\left(x_t\right)\right)+\underset{x_t}{\int }Q\left(x_t\right)\log \left[p\left(x_t|Z^{t-1}\right)p\left(a_t|x_t\right)p\left(Z_t|a_t,x_t\right)\right]\end{array}d{x}_t$$

Thus, to solve the multiple target tracking problem, each of the three distributions, namely, prediction probability p(x_t | Z^t-1), priori probability of the data association variable p(a_t| x_t, Z^t-1), and likelihood model p(Z_t | a_t, x_t, Z^t-1) need to be modeled.

Motion Prediction, p(x_t |Z^t-1): It can be seen that the term p(x₁ | Z^t-1) in Eqn. (6) is the motion prediction model of the trackers and can be expressed as $$p\left(x_t|Z^{t-1}\right)=\underset{x_{t-1}}{\int }p\left(x_t|x_{t-1}\right)p\left(x_{t-1}|Z^{t-1}\right)d{x}_{t-1}$$

This joint motion posteriori p(x_t-1 |Z^t-1) can be suitably approximated via the product of its marginal components p(x_{i, t-1} |Z^t-1) as $$p\left(x_{t-1}|Z^{t-1}\right)\approx {\displaystyle \prod _{i=1}^M}p\left(x_{i,t-1}|Z^{t-1}\right)$$

Assuming that the optimal Q-function Q_i^*(x_{i, t-1}) for tracker i from frame t-1 is a good approximation of the tracker's motion posteriori p(x_{i, t-1} |Z^t-1) and also employs an independent dynamics model, where $p\left(x_t|x_{t-\mathit{1}}\right)={\displaystyle \prod _{i=1}^M}p\left(x_{i,t}|x_{i,t-\mathit{1}}\right),$the joint motion prediction model p(x_t|Z^t-1) may be simplified as: $$p\left(x_t|Z^{t-1}\right)\approx {\displaystyle \prod _{i=1}^M\underset{x_i,t-1}{\int }}p\left(x_{i,t}|x_{i,t-1}\right)Q_i^{*}\left(x_{i,t-1}\right)d{x}_{i,t-1}$$

Association Priori, p(a_t |x_t): The association priori, p(a_t | x_t), is the priori probability of the association variable a_t = {a_{I, t}, a_2, t, ..., a_{M, t}} and is explained in conjunction with FIG. 3. Likelihood model, p(Z_t | a_t, x_t): The likelihood model, p(Z_t |a_t, x_t), x_t), is the joint likelihood model of the measurement data Z_t, conditioned on (a_t, x_t). Further, p(Z_t | x_t)$\mathit{\ne }{\displaystyle \mathit{\prod }_{\mathit{i}\mathit{=}\mathrm{1}}^{\mathit{M}}}\mathit{p}\left({\mathit{z}}_{\mathit{i}\mathit{,}\mathit{t}}\mathit{|}{\mathit{x}}_{\mathit{i}\mathit{,}\mathit{t}}\right)\mathit{.}$ If a_t is provided, this joint likelihood model can be factorized, since it is known which measurement data Z_{a i,t,t} is generated from each tracker x_i,t, that is, $$p\left(Z_t|a_t,x_t\right)={\displaystyle \prod _{i=1}^M}p\left(z_{a_{i,t}},t|x_{i,t}\right)$$

FIG. 3 depicts pair-wise data association constraint of multiple trackers in accordance with an embodiment of the invention. FIG. 3 includes association variables 302a, 302b, 302c, 302d and 302e, all logically connected to each other. Conditioned on motion state variables x_t, p(a_t |x_t) is the priori probability of the association variable a_t = {a_{I, t}, a_{2, t}, ..., a_{M, t}} and can be formulated as a fully-connected but distributed pair-wise graph as shown in the figure. The corresponding probabilistic model can be expressed as $$p\left(a_t|x_t\right)=\frac{1}{Z_{x_t}}{\displaystyle \prod _{\left(i,j\right)\in E}}\mathrm{\Psi }\left(a_{i,t},a_{j,t}|x_t\right)$$

where E denotes the set of neighboring trackers in which the association constraint is introduced, and ψ(a_{i, t}, a_{j, t} |x_t) is the pair-wise constraint between a_{i, t} and a_{j, t}. Z_xt is a partition function which is introduced to make p(a_t | x_t) a proper probability distribution. In FIG. 3, each circle depicts an association variable of a tracker, and the edges connecting them represent the existence of pair-wise association constraints. The link edges between {a₁(k), a₂(k)} and {a₃(k), a₄(k), a₅(k)} are thickened to illustrate that the trackers in each of these two groups are spatially closer to each other.

In accordance with an embodiment of the invention, p(a_t |x_t) is assumed to be independent of the trackers' motions x_t. Thus, $$p\left(a_t|x_t\right)=p\left(a_t\right)=\frac{1}{Z}{\displaystyle \prod _{\left(i,j\right)\in E}}\mathrm{\Psi }\left(a_{i,t},a_{j,t}\right)$$

where $$\mathrm{\Psi }\left(a_{i,t},a_{j,t}\right)=\{\begin{array}{cc}0,& \left\{a_{i,t}=a_{j,t}\ne 0\right\}\hfill \\ 1,& \mathit{otherwise}\hfill \end{array}$$

Both a_{i, t} and a_{j, t} can choose values from the discrete measurement set {0, 1, ..., m_t}.

From the above discussion, it can be seen that the motion prediction, p(x_t | z_t-1), the association priori p(a_t | x_t), and the likelihood model p(Z_t | a_t, x_t) all take factorized or distributed forms. Further, eqn. (7) can be written as $$\begin{array}{ll}\left\{a_t^{*},Q^{*}\left(x_t\right)\right\}& =\underset{a_t,Q\left(x_t\right)}{\mathrm{arg\; max}}{\displaystyle \sum _{\left(i,j\right)\in E}\mathrm{log\Psi }\left(a_{i,t},a_{j,t}\right)}-\log Z\\ & +{\displaystyle \sum _{i=1}^M}\underset{x_{i,t}}{\int }{Q}_i\left(x_{i,t}\right)\left[\log p\left(x_{i,t}|Z^{t-1}\right)p\left(z_{a_{i,t},t}|x_{i,t}\right)\right]d{x}_{i,t}\\ & +{\displaystyle \sum _i^{M}H\left(Q_i\left(x_{i,t}\right)\right)}\end{array}$$

Thus, the EM solution to the problem designated by Eqn. (15) involves solving two iterative steps, one step computing a better Q'(x_t), and the other finding a better association a_t'. These two steps, that is, the E-step and the M-step, are solved by each tracker to track its corresponding target, and are explained in further detail with reference to FIG. 4.

FIG. 4 depicts a block diagram of a tracker 400 in accordance with various embodiments of the invention. Tracker 400 includes an E-step execution module 402 and an M-step execution module 404. An E-step execution module 402 and an M-step execution module 404 calculate motion state variable and data association variable of tracker 400 respectively. Output of the E-step execution module 402 is given as input to the M-step execution module 404 and vice versa. Thus, these two modules implement the variational EM algorithm in tracker 400.

In particular, the E-step execution module 402 computes a better $\mathit{Qʹ}\left(x_t\right)={\displaystyle \prod _{i=1}^M}{\mathit{Qʹ}}_i\left(x_{i,t}\right)$ over the trackers' motions x_t to maximize E(a_t, Q(x_t)). Further, M-step execution module 404 finds a better association a_t' = {a'_{1, t}, a'_{2, t}, ..., a'_{M, t}} to maximize E(a_t, Q'(x_t)).

In the E-step, the partial derivative of the objective in Eqn. (15) over Q_i(x_{i, t}) is taken. Further, the constraint that each Q_i(x_{i, t}) must be a valid probabilistic distribution is applied, that is, $$\underset{x_{i,t}}{\int }{Q}_i\left(x_{i,t}\right)d{x}_{i,t}=1$$

This constraint may be enforced which gives the E-step updating equation for each tracker i as $$Q_i^{ʹ}\left(x_{i,t}\right)\propto p\left(z_{a_{i,t},t}|x_{i,t}\right)p\left(x_{i,t}|Z^{t-1}\right)$$

In this case, the measurement data z_{ai, t,t} used to filter x_{i, t} is conditioned on the association variable a_{i, t}, which is estimated from the following M-step. It can be seen that the above E-step updating is composed with M independent updating equations, where each equation is for an individual tracker. A Kalman filter may be directly applied to compute the E-step in Eqn. (16) if the joint motion prediction model p(x_t | Z_t-1) in Eqn. (10) and the likelihood model p(Z_t |a_t, x_t) in eqn. (11) are assumed to be Gaussian in nature.

In the M-step, an updated set of association variables a₁ = {a_{1, t}, a_{2, t}, ..., a_{M, t}} is determined to increase the objective given the already updated Q'(x_t) from the E-step. $$a_t^{ʹ}=\underset{a_t}{\mathrm{arg\; max}}{\displaystyle \sum _{\left(i,j\right)\in E}\mathrm{log\; \Psi }}\left(a_{i,t},a_{j,t}\right)+{\displaystyle \sum _{i=1}^M\underset{x_{i,t}}{\int }}{Q}_i^{ʹ}\left(x_{i,t}\right)\log p\left(z_{a_{i,t},t}|x_{i,t}\right){\mathit{dx}}_{i,t}$$

The following two terms are defined: $$\begin{array}{l}{f}_{i,j}\left(a_{i,t},a_{j,t}\right)=\mathrm{\Psi }\left(a_{i,t},a_{j,t}\right)\\ g_i\left(a_{i,j}\right)=\exp \left\{\underset{x_{i,t}}{\int }{Q}_i^{ʹ}\left(x_{i,t}\right)\log p\left(z_{a_{i,t},t}|x_{i,t}\right)d{x}_{i,t}\right\}\end{array}$$

which are the functions of (a_{i, t}, a_{j, t}) and a_{i, t} respectively. Eqn. (17) can then be written as $$a_t^{ʹ}=\underset{a_t}{\mathrm{argmax}}{\displaystyle \prod _{\left(i,j\right)\in E}}{f}_{i,j}\left(a_{i,t},a_{j,t}\right){\displaystyle \prod _{i=1}^M}{g}_i\left(a_{i,t}\right)$$

The values {a_{1, t}, a_{2, t}, ..., a_{M, t}} are from a discrete value set, and hence, both f_{i, j}(a_{i, t}, a_j,t) and g_i(a_i,t) can be computed before M-step optimization. The integral computation involved in Eqn. (18) may be analytically computed by assuming Q_i(_{xi, t}) to be Gaussian. Further, Eqn. (19) can be solved by algorithms such as multi-way graph cut algorithm, max-product belief propagation, and so forth. The max-product belief propagation (BP) algorithm and its variants are distinguished with their distributed and parallel computational paradigm and impart distributed computation to the M-step.

In accordance with an embodiment of the invention, in order to track a target even when it is partially occluded, we consider a K-part decomposition of the target. K-part detectors are trained to detect the K parts by collecting training data of a corresponding part. Let us consider the case where the target is divided into three parts, head-shoulder, torso and legs (K=3). Then, the data association variable of a tracker is formed by K parts, that is, a_{i, t} = (a_{i, 1, t}, a_{i, 2, t}, ..., a_{i, k, t}), where each a_{i, k, t}, k ∈K describes an association that assigns a part detection from a corresponding part detector to tracker i. The motion state variable of the tracker is x_{i, t}. Conditioned on x_{i, t} and data association variable a_{i, t}, the likelihood function p(z_{ai, t, t} | x_{i, t}) may be expressed as $$p\left(z_{a_{i,t},t}|x_{i,t}\right)={\displaystyle \prod _{k=1}^K}p\left(z_{a_{i,k,t},t}|x_{i,t}\right)$$

Further, the association priori p(a_t) may be denoted as $$p\left(a_t\right)=\frac{1}{Z}{\displaystyle \prod _{\left(i,j\right)\in E}}{\displaystyle \prod _{k=1}^K}\mathrm{\Psi }\left(a_{i,k,t},a_{j,k,t}\right)$$

Thus, in case of K-part decomposition, K graph optimizations can be carried out to obtain the optimal part associations a'_{i, t} = (a'_{i, I, t}, a'_{i, 2, t}, ..., a'_{i, K, t} ) simultaneously.

In accordance with another embodiment of the invention, the tracker i can be used in 3-dimensional (3D) space and the motion state variable of each 3D tracker contains depth information. For example, let x_{i, t} < x_{j, t} denote a motion hypothesis that tracker i is closer to a camera than tracker j and let there be four detections Z_t = {z_{1, t}, ..., z_{4, t}} that are returned. Conditioned on x_{i, t} < x_{j, t}, the four detections are partitioned into Z_t¹ = {z_{1, t}, z_{2, t}, z_{3, t}} and Z_t² = {z_{4, t}}, depending on whether a detection is covered by the projection of the front tracker i. In this case, besides the common constraint {a_{i, t} = a_{j, t} ≠ 0}, all configurations with a_{j, t} = 4 are unacceptable since the motion hypothesis is x_i, < x_{j, t}. Particle filters may be used to run the trackers due to the non-linearity involved. Thus, the variational probability Q_i(x_{i, t}) will be represented by a weighted particle set, and all integral computations discussed earlier will be represented by summations instead.

The E-step can then be written as $$Q_i^{ʹ}\left(x_{i,t}\right)\propto {\displaystyle \prod _{j\in N\left(i\right)}}\exp \left\{\underset{x_{j,t}}{\int }{Q}_j\left(x_{j,t}\right)\log p\left(a_{i,t}^{ʹ},a_{j,t}^{ʹ}|x_{i,t},x_{j,t}\right)d{x}_{j,t}\right\}\times p\left(z_{a_{i,t},t}|x_{i,t}\right)\underset{x_{i,t-1}}{\int }p\left(x_{i,t}|x_{i,t-1}\right)p\left(x_{i,t-1}|Z^{t-1}\right)d{x}_{i,t-1}i\in \left\{1,\dots ,M\right\}$$

where updating of Q_i (x_{i, j}) takes each neighboring tracker's Q_j(x_{j, t}) into consideration.

The M-step objective remains the same, however, the way of pre-computing f_{i, j}(a_{i, t}, a_{j, t}) is modified as $$\begin{array}{c}{f}_{i,j}\left(a_{i,t},a_{j,t}\right)=\exp \left\{\underset{x_{i,t},x_{j,t}}{\int }{Q}_i^{ʹ}\left(x_{i,t}\right)Q_j^{ʹ}\left(x_{i,t}\right)p\left(a_{i,t},a_{j,t},|x_{i,t},x_{j,t}\right)d{x}_{i,t}d{x}_{j,t}\right\}\end{array}$$

where an integral evaluated over motions of pair-wise trackers (x_{i, t}, x_{j, t}) are required to pre-compute f_{i, j}(a_{i, t}, a_{j, t}).

FIG. 5A, 5B and 5C depict association between trackers and targets in consecutive frames in accordance with various embodiments of the invention. FIG. 5A is the output of the variational EM algorithm applied to a frame and includes trackers 502a, 502b, 502c and 502d along with their corresponding targets in a frame. A tracker is represented by a square while each target is represented by a circle having a cross within. As shown in FIG. 5A, each tracker is following a target. Thus, trackers 502a, 502b, 502c and 502d are each associated with a different target present in the frame.

FIG. 5B shows the next frame received from a video surveillance camera. From FIG. 5B it can be seen that each of the targets has moved from its earlier position. However, each of the trackers has still maintained an association with its corresponding target. FIG. 5C shows the output of applying the variational EM algorithm to the frame shown in FIG. 5B. As shown, the trackers have also left their earlier positions and are following their corresponding targets. This process continues for every frame received from the video surveillance camera.

FIG. 6 is a flow chart depicting a method for multiple target tracking in accordance with various embodiments of the invention. At step 602, multiple targets to be tracked are detected in a video frame. The detection of the multiple targets is performed by a detection module and provided to a tracking module. The tracking module includes a plurality of trackers.

At step 604, each of the plurality of trackers calculates its motion state variable. The motion state may refer to any of the target properties such as target location, velocity, scale, and the like. The motion state variable is calculated in the E-step of a variational Expectation-Maximization (EM) algorithm.

At step 606, each of the plurality of trackers calculates its data association variable in the M-step of the variational EM algorithm. When calculating its data association variable, each of the plurality of trackers also informs the other trackers about its current estimation of the data association variable. The other trackers update their data association variable based on the information received.

At step 608, each of the plurality of trackers tracks its corresponding target in the given frame with the help of the variational EM algorithm.

FIG. 7A and 7B are flowcharts illustrating a method for multiple target tracking, in accordance with another embodiment of the invention. At step 702, a set of trackers is assigned to track multiple targets detected in a frame. At step 704, the next frame is received.

At step 706, it is determined if there is any unassociated measurement in the received frame. In other words, it is checked if there is any target that is not associated with any tracker.

At step 708, if there is an unassociated measurement in the received frame, a new tracker is initialized as a temporary tracker. However, if no unassociated measurement is detected, no new tracker is initialized and the process continues at step 714.

At step 710, it is determined if the data association of the temporary tracker is valid for a predefined number of frames. At step 712, if the data association is determined to be valid, the temporary tracker is marked as an established tracker. However, if the data association is not valid, the temporary tracker is terminated, at step 718

At step 714, for every established tracker, it is determined if there exists a valid data association with its corresponding target for a selected number of frames. If there exists a valid data association, the process continues and the targets are continuously tracked, at step 716. However, if the valid data association does not exist, the established tracker is terminated, at step 718.

Thus, the invention provides a method and system for multiple target tracking. A variational Expectation-Maximization (EM) algorithm is used to calculate motion state variables and data association variables of a plurality of trackers. The plurality of trackers track their corresponding targets on the basis of the calculated motion state variables and data association variables. Both the motion state variable and the data association variable are calculated in a distributed manner, thus reducing computational complexities.

One skilled in the art of computer science will easily be able to combine the software created as described with appropriate general purpose or special purpose computer hardware, such as a microprocessor, to create a computer system or computer subsystem embodying embodiments of the invention. An apparatus in accordance with embodiments of the invention may be one or more processing systems including, but not limited to, a central processing unit (CPU), memory, storage devices, communication links and devices, servers, I/O devices, or any sub-components of one or more processing systems, including software, firmware, hardware or any combination or subset thereof, which include embodiments of the invention.

The computer program product of an embodiment of the invention is executable on a computer system for causing the computer system to perform a method of filtering an image including an image filtering method of the invention. The computer system includes a microprocessor, an input device, a display unit and an interface to either the Internet or a network such as Ethernet, and Intranet. The microprocessor is connected to a communication bus. The computer also includes a memory. The memory may include Random Access Memory (RAM) and Read Only Memory (ROM). The computer system further comprises a storage device. The storage device can be a hard disk drive or a removable storage drive such as a floppy disk drive, optical disk drive, etc. The storage device can also be other similar means for loading computer programs or other instructions into the computer system. The computer system also includes a communication unit. The communication unit allows the computer to connect to other databases and the Internet through an I/O interface. The communication unit allows the transfer as well as reception of data from other databases. The communication unit may include a modem, an Ethernet card, or any similar device that enables the computer system to connect to databases and networks such as LAN, MAN, WAN, and the Internet. The computer system facilitates inputs from a user through an input device, accessible to the system through the I/O interface. The various modules may also be in the form of hardware units.

The computer system executes a set of instructions that are stored in one or more storage elements to process input data. The set of instructions may be a program instruction means. The storage elements may also hold data or other information as desired. The storage element may be in the form of an information source or a physical memory element present in the processing machine.

The set of instructions may include various commands that instruct the processing machine to perform specific tasks such as the steps that constitute an embodiment of the method of the invention. The set of instructions may be in the form of a software program. Further, the software may be in the form of a collection of separate programs, a program module with a larger program or a portion of a program module. The software may also include modular programming in the form of object-oriented programming. The processing of input data by the processing machine may be in response to user commands, results of previous processing or a request made by another processing machine.

Accordingly, the invention is not to be seen as limited by the foregoing description, but its scope is defined by the appended claims.