DETAILED DESCRIPTION

[0063] FIG. 1 shows a multicamera motion tracking and control system 100 interfaced with an image viewing system. In this implementation two cameras 101 and 102 scan a region of interest 103 . A controlled or known background 104 surrounds the region of interest 103 . An object of interest 105 is tracked by the system when it enters the region of interest 103 . The object of interest 105 may be any generic object inserted into the region of interest 103 , and is typically a hand or finger of a system user. The object of interest 105 also may be a selection device such as a pointer.

[0064] The series of video images acquired from the cameras 101 and 102 are conveyed to a computing device or image processor 106 . In this implementation, the computing device is a general-purpose computer that runs additional software that provides feedback to the user on a video display 107 .

[0065] FIG. 2A illustrates a typical implementation of the multicamera control system 100 . The two cameras 101 and 102 are positioned outside of the region of interest 103 . The cameras are oriented so that the intersection 204 of their field of views ( 205 for camera 101 , 206 for camera 102 ) completely encompasses the region of interest 103 . The orientation is such that the cameras 101 , 102 are rotated on axes that are approximately parallel. In this example, a floor or window ledge and sidewalls provide a controlled background 104 having distinct edges. The corresponding view captured by camera 101 is shown in FIG. 2B . While not shown, it should be understood that the view captured by camera 102 is a mirror image of the view captured by camera 101 . The controlled background 104 may not cover the camera's entire field of view 205 . For each camera, an active image region 208 is found that is entirely contained within the controlled background 104 , and also contains the entire region of interest 103 . The background 104 is controlled so that a characteristic of the background can be modeled, and the object of interest 105 , either in part or in whole, differs from the background 104 in that characteristic. When the object of interest 105 appears within the region of interest 103 , the object 105 will occlude a portion of the controlled background 104 within the active image region 208 of each camera 101 , 102 . In the location of the occlusion, either as a whole or in parts, the captured images will, in terms of the selected characteristic, be inconsistent with the model of the controlled background 104 .

[0066] In summary, the object of interest 105 is identified and, if found, its position within the active image region 208 of both cameras is calculated. Using the position data of each camera 101 , 102 , as well as the positions of the cameras relative to the region of interest 103 , and parameters describing the cameras, the position of the object of interest 105 within the region of interest 103 is calculated.

[0067] The processes performed by the image processor 106 ( FIG. 1 ), which may be implemented through a software process, or alternatively through hardware, are generally shown in FIG. 3 . The camera images are simultaneously conveyed from the cameras 101 , 102 and captured by image acquisition modules 304 , 305 (respectively) into image buffers 306 , 307 (respectively) within the image processor 106 . Image detection modules 308 , 309 independently detect the object of interest 105 in each image, and determine its position relative to the camera view. The relative position information 310 , 311 from both camera views is combined by a combination module 312 and optionally refined by a position refinement module 313 , to determine at block 314 , the global presence and position of the object of interest 105 within the region of interest 103 . Optionally, specific gestures performed by the user may be detected in a gesture detection module 315 . The results of the gesture detection process are then conveyed to another process or application 316 , either on the same image processor 106 or to another processing device. The process of gesture detection is described in greater detail below.

[0068] Image detection modules 308 and 309 are identical in the processes that they execute. An implementation of these image detection modules 308 , 309 is shown in FIG. 4 . In block 402 , the image processor 106 extracts, from the captured image data stored in the image buffers 306 or 307 , the image data that corresponds to the active image region 208 (of FIG. 2B ). The image may be filtered in a filtering process 403 to emphasize or extract the aspects or characteristics of the image where the background 104 and object of interest 105 differ, but are otherwise invariant within the background 104 over time. In some implementations, the data representing the active image region may also be reduced by a scaling module 404 in order to reduce the amount of computations required in later processing steps. Using the resulting data, the background 104 is modeled by one or more instances of a background model process at block 405 to produce one or more descriptions represented as background model data 406 of the controlled background 104 . Therefore the background 104 is modeled in terms of the desired aspects or characteristics of the image. The background model(s) 406 are converted into a set of criteria in process 407 . In a comparison process 408 , the filtered (from process 403 ) and/or reduced (from module 404 ) image data is compared to those criteria (from process 407 ), and the locations where the current data is inconsistent with the background model data 406 , that is where the criteria is not satisfied, are stored in an image or difference map 409 . In detection module 410 , the difference map 409 is analyzed to determine if any such inconsistencies qualify as a possible indication of an object of interest 105 and, if these criteria are satisfied, its position within the camera view ( 205 or 206 ) is determined. The position of the object 105 may be further refined (optionally) at block 411 , which produces a camera-relative presence and position output 310 or 311 associated with the object of interest 105 (as described above with respect to FIG. 3 ).

[0069] In block 402 of FIG. 4 , image processor 106 extracts the image data that corresponds to the active image region 208 (of FIG. 2B ). The image data may be extracted by cropping, shearing, rotating, or otherwise transforming the captured image data. Cropping extracts only the portion of the overall image that is within the active image region 208 . Bounds are defined, and any pixels inside the bounds are copied, unmodified, to a new buffer, while pixels outside of the bounds are ignored. The active image region 208 may be of arbitrary shape. Shearing and rotation reorder the data into an order that is more convenient for further processing, such as a rectangular shape so that it may be addressed in terms of rows and columns of pixels.

[0070] Rotation causes the contents of an image to appear as if the image has been rotated. Rotation reorders the position of pixels from (x,y) to (x′,y′) according to the following equation: 1 $\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}\cos \theta & -\sin \theta & 0\\ \sin \theta & \cos \theta & 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$

[0071] where θ is the angle that the image is to be rotated.

[0072] If the cameras 101 and 102 are correctly mounted with respect to the region of interest 103 , the desired angle of rotation will typically be small. If the desired angle of rotation is small, shearing may be used to provide an approximation that is computationally simpler than rotation. Shearing distorts the shape of an image such that the transformed shape appears as if the rows and columns have been caused to slide over and under each other. Shearing reorders the position of pixels according to the following equations: 2 $\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}1& {\mathrm{sh}}_x& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]\mathrm{and}\left[\begin{array}{c}{x}^{\prime }\\ y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ {\mathrm{sh}}_y& 1& 0\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}x\\ y\\ 1\end{array}\right]$

[0073] where sh _x represents the amount of horizontal shear within the image, and sh _y represents the amount of vertical shear within the image.

[0074] An implementation of the multicamera control system 100 applies in scenarios where the object of interest 105 , either in whole or in part, is likely to have either higher or lower luminance than the controlled background 104 . For example, the background 104 may be illuminated to create this scenario. A filtering block 403 passes through the luminance information associated with the image data. A single background model 406 represents the expected luminance of the background 104 . In practice, the luminance of the controlled background 104 may vary within the active image region 208 , therefore the background model 406 may store the value of the expected luminance for every pixel within the active image region 208 . The comparison criteria generation process 407 accounts for signal noise (above that which may be accounted for within the background model) and minor variability of the luminance of the controlled background 104 by modifying each luminance value from the background model 406 , thus producing the minimal luminance value that may be classified as being consistent with the background model 406 . For example, if the luminance of the controlled background 104 is higher than the luminance of the object of interest 105 , then processes block 407 decreases the luminance value of each pixel by an amount greater than the expected magnitude of signal noise and variability of luminance.

[0075] In some implementations of system 100 , the region of interest 103 is sufficiently narrow such that it may to be modeled as a region of a plane. The orientation of that plane is parallel to the front and rear faces of the dotted cube that represents the region of interest 103 in FIG. 1 . The active image region 208 may be reduced to a single row of pixels in the optional scaling module 404 if two conditions are satisfied: 1) the object of interest 105 , when it is to be detected, will occlude the background 104 in all rows of some columns of the active image region 208 , and 2) a single set of values in the background model 406 sufficiently characterizes an entire column of pixels in the active image region 208 . The first condition is usually satisfied if the active image region 208 is thinner than the object of interest 105 . The second condition is satisfied by the implementation of blocks 403 , 405 , 406 and 407 described above. Application of the scaling module 404 reduces the complexity of processing that is required to be performed in later processes, as well as reducing the storage requirements of the background model(s) 406 .

[0076] The particular implementation of the scaling module 404 depends on the specifics of processing blocks 403 , 405 , 406 and 407 . If the luminance of the controlled background 104 is expected to be higher than that of the object of interest 105 , as described above, one implementation of the scaling module 404 is to represent each column by the luminance of greatest magnitude within that column. That is to say, for each column, the highest value in that column is copied to a new array. This process has the added benefit that the high-luminance part of the controlled background 104 need not fill the entire controlled background 104 .

[0077] An alternative implementation applies in scenarios where the controlled background 104 is static, that is, contains no motion, but is not otherwise limited in luminance. A sample source image is included in FIG. 5A as an example. In this case, the object of interest, as sensed by the camera, may contain, or be close in magnitude to, the luminance values that are also found within the controlled background 104 . In practice, the variability of luminance of the controlled background 104 (for example, caused by a user moving in front of the apparatus thereby blocking some ambient light) may be significant in magnitude relative to the difference between the controlled background 104 and the object of interest 105 . Therefore, a specific type of filter may be applied in the filtering process 403 that produces results that are invariant to or de-emphasize variability in global luminance, while emphasizing parts of the object of interest 105 . A 3×3 Prewitt filter is typically used in the filtering process 403 . FIG. 5B shows the result of this 3×3 Prewitt filter on the image in FIG. 5A . In this implementation, two background models 406 may be maintained, one representing each of the high and low values, and together representing the range of values expected for each filtered pixel. The comparison criteria generation process 407 then decreases the low-value and increases the high-value by an amount greater than the expected magnitude of signal noise and variability of luminance. The result is a set of criterion, an example of which, for the low-value, is shown in FIG. 5 C, and an example of which, for the high-value, is shown in FIG. 5D . These modified images are passed to the comparison process 408 , which classifies pixels as being inconsistent to the controlled background 104 if their value is either lower than the low-value criterion ( FIG. 5C ) or higher than the high-value criterion ( FIG. 5D ). The result is a binary difference map 409 , of which example corresponding to FIG. 5B is shown in FIG. 5E .

[0078] The preceding implementation allows the use of many existing surfaces, walls or window frames, for example, as the controlled background 104 where those surfaces may have arbitrary luminance, textures, edges, or even a light strip secured to the surface of the controlled background 104 . The above implementation also allows the use of a controlled background 104 that contains a predetermined pattern or texture, a stripe for example, where the above processes detect the lack of the pattern in the area where the object of interest 105 occludes the controlled background 104 .

[0079] The difference map 409 stores the positions of all pixels that are found to be inconsistent with the background 104 by the above methods. In this implementation, the difference map 409 may be represented as a binary image, where each pixel may be in one of two states. Those pixels that are inconsistent with the background 104 are identified or “tagged” by setting the pixel in the corresponding row and column of the difference map to one of those states. Otherwise, the corresponding pixel is set to the other state.

[0080] An implementation of the detection module 410 , which detects an object of interest 105 in the difference map 409 , shown in FIG. 6 . Another scaling module at block 603 provides an additional opportunity to reduce the data to a single dimensional array of data, and may optionally be applied to scenarios where the orientation of the object of interest 105 does not have a significant effect on the overall bounds of the object of interest 105 within the difference map 409 . In practice, this applies to many scenarios where the number of rows is less than or similar to the typical number of columns that the object of interest 105 occupies. When applied, the scaling module at block 603 reduces the difference map 409 into a map of one row, that is, a single dimensional array of values. In this implementation, the scaling module 603 may count the number of tagged pixels in each column of the difference map 409 . As an example, the difference map 409 of FIG. 7A is reduced in this manner and depicted as a graph 709 in FIG. 7B . Applying this optional processing step reduces the processing requirements and simplifies some of the calculations that follow.

[0081] Continuing with this implementation of the detection module 410 , it is observed that the pixels tagged in the difference map ( 409 in example FIG. 7A ) that are associated with the object of interest 105 will generally form a cluster 701 , however the cluster is not necessarily connected. A cluster identification process 604 classifies pixels (or, if the scaling module 603 has been applied, classifies columns) as to whether they are members of the cluster 701 . A variety of methods of finding clusters of samples exist and may be applied, and the following methods have been selected on the basis of processing simplicity. It is noted that, when the object of interest 105 is present, it is likely that the count of correctly tagged pixels will exceed the number of false-positives. Therefore the median position is expected to fall somewhere within the object of interest 105 . Part of this implementation of the cluster identification process 604 , when applied to a map of one row (for example, where the scaling module at block 603 or 404 has been applied), is to calculate the median column 702 and tag columns as part of the cluster 701 ( FIG. 7B ) if they are within a predetermined distance 703 that corresponds to the maximum number of columns expected to be occupied. Part of this implementation of the cluster identification process 604 , when applied to a map of multiple rows, is to add tagged pixels to the cluster 703 if they meet a neighbor-distance criterion.

[0082] In this implementation, a set of criteria is received by a cluster classification process 605 and is then imposed onto the cluster 701 to verify that the cluster has qualities consistent with those expected of the object of interest 105 . Thus, process 605 determines whether the cluster 701 should be classified as belonging to the object of interest 105 . Part of this implementation of the cluster classification process 605 is to calculate a count of the tagged pixels within the cluster 701 and to calculate a count of all tagged pixels. The count within the cluster 701 is compared to a threshold, eliminating false matches in clusters having too few tagged pixels to be considered as an object of interest 105 . Also, the ratio of the count of pixels within the cluster 701 relative to the total count is compared to a threshold, further reducing false matches.

[0083] If the cluster 701 passes these criteria, a description of the cluster is refined in process block 606 by calculating the center of gravity associated with the cluster 701 in process 607 . Although the median position found by the scaling module 603 is likely to be within the bounds defining the object of interest 105 , it is not necessarily at the object's center. The weighted mean 710 , or center of gravity, provides a better measure of the cluster's position and is optionally calculated within process 606 , as sub-process 607 . The weighted mean 710 is calculated by the following equation: 3 $\stackrel{\_}{x}=\frac{\sum _{x=0}^{c-1}x·C\left[x\right]}{\sum _{x=0}^{c-1}C\left[x\right]}$

[0084] where:

[0085] {overscore (x)} is the mean

[0086] c is the number of columns

[0087] C[x] is the count of tagged pixels in column x.

[0088] The cluster's bounds 704 may also be optionally calculated within process 606 , shown as process 608 . The cluster 703 may include some false-positive outliers, so as part of this implementation, the bounds may be defined as those that encompass a predetermined percentile of the tagged pixels, or, in scenarios where relatively few pixels are expected to be tagged, encompasses those tagged pixels (or columns, if scaling module 603 is applied) that form tight sub-clusters, that is those tagged pixels (or columns) that have neighbors that are also tagged.

[0089] In addition to the middle and bound coordinates, the orientation of the object of interest 105 may optionally be inferred by calculation of the moments of the cluster. This calculation is represented by a cluster orientation calculation process at sub-process 609 within process 606 .

[0090] In some applications of the system 100 , the object of interest 105 is used as a pointer. In this case, the “pointing end” of the object 105 is desired and may also be determined by a pointing end calculation sub-process within process 606 if the region of interest 103 contains a sufficient number of rows and the number of rows has not been reduced. An example is depicted in FIG. 7C . The object of interest 105 will typically enter, or be constrained to enter, the active image region 208 from a known border of that region. The pointing end 705 (for example the user's fingertip) of the object of interest 105 is likely to be the portion of the cluster 701 that is furthest from the region of entry 706 into the active image region 208 . The cluster 701 may include some false-positive outliers. As such, the pointing end 705 may be defined as the region 707 within the cluster 701 that encompasses multiple tagged pixels near the furthest bounding side of the cluster 701 , or, in scenarios where relatively few pixels are expected to be tagged, encompasses the furthest tagged pixels that form a tight sub-cluster; that is those tagged pixels that have neighbors that are also tagged. This sub-cluster is identified by a sub-cluster pointing end process 610 , and the position of the sub-cluster is found in process 611 .

[0091] Continuing with this implementation, a process implemented by a smoothing module 612 may optionally be applied to any or all of the positions found in process 606 . Smoothing is a process of combining the results with those solved previously so they move in a steady manner from frame to frame. The weighted mean coordinate 710 , found by the center of gravity determination process 607 , is dependent on many samples and therefore is inherently steady. The bound 704 , found by the cluster bounding dimension determination process 608 , and pointing end 705 , found by 611 , coordinates are dependent on relatively fewer members of the cluster, and the state of a single pixel may have a significant effect. Since the size of the region occupied by the object of interest 105 is expected to remain relatively steady, smoothing may be applied to the distance between the bounds 704 measured relative to the cluster's weighted mean coordinate 710 . Since the shape and orientation of the object of interest 105 is expected to change less rapidly than the overall position object of interest 105 , smoothing may be applied to the distance of the pointing end 705 measured relative to the cluster's weighted mean coordinate 710 .

[0092] A process used in the center of gravity process 607 is Eq. 1 as follows:

s ( t )=( a×r ( t ))+((1 −a )× s ( t− 1))

[0093] In Eq. 1, the smoothed value at time t (s(t)) is equal to one minus the scalar value (a) multiplied by the smoothed value at time minus one (t−1). This amount is added to the raw value at time t (r(t)) multiplied by a scalar (a) that is between zero and one.

[0094] Referring to FIG. 8 , implementations of system 100 make use of, as described above, one or more background models 406 ( FIG. 4 ). An implementation of the background model process or component 405 that generates the background model data 406 is shown in FIG. 8 . This implementation of the background model component 405 automatically generates and dynamically updates the background model, allowing unattended operation of the system.

[0095] Input data 802 is provided by the output of scaling module 404 for this implementation of the background model component 405 . Input is available every frame, and is sampled in a sampling process 803 . The sample may contain the object of interest 105 occluding part of the controlled background 104 . For each pixel, a range of values may be a better representative of the background 104 than a single value. By including the effects of this range in the background model, the expansion in process 407 may be made tighter. Contributing multiple frames of data to the sample allows this range to be observed, but also increases the portion of the background 104 that is occluded by the object of interest 105 if the object of interest 105 is in motion while the frames are being sampled. The optimal number of frames to use is dependent on the expected motion of the object of interest 105 in the particular application of the system. In practice, for systems that are tracking a hand, 10 frames, representing approximately 0.33 seconds, is sufficient to observe the majority of that range without allowing motion of the object of interest to occlude an undue portion of the background. If the particular background model is to be compared in comparison process 408 as the upper bound on values that are considered to be consistent with the background 104 , then the maximum value of each pixel observed in the multiple frames may be recorded as the sample value. If the particular background model 406 is to be compared in process 408 as the lower bound on values that are considered to be consistent with the background 104 , then the minimum value of each pixel observed in the multiple frames may be recorded as the sample value.

[0096] In this implementation of the background model component 405 , samples from the sampling process 803 are added to a buffer 804 having storage locations to store n samples, where the oldest sample in the history is replaced. The history therefore contains n sampled values for each pixel. The span of time, d, represented in the buffer is dependent on the rate that new samples are acquired and added to the history, r, by Eq. 2, described as follows: 4 $d=\frac{n}{r}$

[0097] In this implementation, a median process block 805 selects, for each pixel, a value that it determines is representative of the controlled background 104 at the location represented by that pixel. One method of selecting a value representative of the controlled background 104 within process block 805 is to select the median value of the n samples of each pixel. For any pixel, a number of the n sampled values in the buffer 804 may represent the object of interest 105 . Duration d is selected so that it is unlikely that the object of interest 105 will occlude any one pixel of the controlled background 104 for an accumulated duration of d/2 or longer within any time-span of d. Therefore, for any pixel, the majority of the sampled values will be representative of the background 104 , and therefore the median of the sampled values will be a value representative of the background 104 .

[0098] The background model component 405 is adaptive, and any changes to the background 104 will be reflected in the output of median process block 805 once they have been observed for time of d/2. This system does not require that the entire controlled background 104 be visible when initialized, the object of interest 105 may be present when initialized, however it does require that samples be observed for time of d before providing output. Optionally, the constraint may be applied that the object of interest 105 must be absent when the system is initialized, in which case the first observed sample values may be copied into all n samples of the buffer 804 , allowing the system to produce an output sooner.

[0099] The duration that any one pixel of the controlled background 104 will be occluded by the object of interest 105 , and therefore the duration d, is dependent on the particular application of the system. The number of samples, n, can be scaled for the memory buffer and processing power available.

[0100] The preceding discussion presents one implementation of obtaining the position of the object of interest 105 within and relative to the images acquired by the cameras 101 and 102 . If the object of interest 105 was successfully detected and its coordinates found in both cameras views 205 and 206 by detection modules 308 and 309 of FIG. 3 , then the combination of these coordinates is sufficient to recover the position of the object of interest 105 within the region of interest 103 . In the implementation outlined in FIG. 3 , the position of the object of interest 105 is calculated in combination module 312 .

[0101] Turning to FIGS. 9A and 9B , an implementation of the combination module 312 is shown. For each camera 101 and 102 , the position p 902 of the object of interest 105 on the camera's image plane 904 is converted to an angle 905 , which is referred in this description as beta (β), and is measured on the reference plane whose normal is defined by the axes of the rotations of the cameras 101 , 102 . (In practice, the axes are not precisely parallel and do not exactly define a single plane, however the process described herein is tolerant of that error). By approximating the camera 101 , 102 as an ideal pinhole model of the camera, that angle (β), relative to the vector 906 defining the orientation of the camera, is approximated.

[0102] Eq. 3, as shown in FIG. 9 A, illustrates an approximation calculation as follows: 5 $\beta ={\tan }^{-1}\left(\frac{f}{p}\right)$

[0103] To approximate the angle beta (β), the inverse tangent is applied to the quantity of the focal length (f) divided by the position p on the image plane projected onto the intersection of the reference plane and the image plane.

[0104] For maximum precision, the intrinsic camera parameters (location of the principal point and scale of image) and radial distortion caused by the lens should be corrected for by converting the distorted position (as represented by the relative position information 310 , 311 ) to the ideal position. More specifically, the ideal position is the position on the image plane 904 that the object 105 would be projected if the camera 101 , 102 had the properties of an ideal pinhole camera, whereby Eq. 3 will produce the exact angle. One set of correction equations are presented in Z. Zhang, A Flexible New Technique for Camera Calibration, Microsoft Research, http://research.microsoft.com/˜zhang, which is incorporated by reference. For many applications of the system, the approximation has been found to provide sufficient precision without this correction noted above.

[0105] Continuing with the description of combination module 312 , a reference vector 907 , as illustrated in FIG. 9 B, is defined such that it passes through the positions of both cameras 101 and 102 on the reference plane where the reference plane is defined such that the axis of rotation of the cameras define the normal of the reference plane. The angles 908 that the cameras are rotated are measured relative to the reference vector 907 .

[0106] A formula for measurement of the angles is shown in Eq. 4:

α=β ₀ +β

[0107] Measurement of the angle alpha (α) is equal to the angle beta_not (β ₀ ) and the angle beta (β).

[0108] Eq. 4 is applied to measure the angles 909 of the object of interest 105 relative to the reference vector 907 . That angle is referred to by the alpha (α) symbol herein. The angle alpha 909 for each camera 101 and 102 , and the length of the reference vector 907 , are sufficient to find the position of the object of interest 105 on the reference plane, by Eq. 5 and Eq. 6.

[0109] Eq. 5 calculates the offset of the object of interest (y) by the formula: 6 $y=\frac{w\tan {\alpha }_A\tan {\alpha }_B}{\tan {\alpha }_A+\tan {\alpha }_B}$

[0110] The offset (y) is equal to the reciprocal of the tangent of the angle (α _A ) for camera A 101 and the tangent of the angle (α _B ) for camera B 102 multiplied by the vector length 907 (w), the tangent of the angle (α _A ) for camera A 101 and the tangent of the angle (α _B ) for camera B 102 .

[0111] Eq. 6 calculates the offset of the object of interest (x _A ) as follows: 7 $x_A=\frac{y}{\tan {\alpha }_A}$

[0112] In Eq. 6, the offset (x _A ) is measured by the offset from Eq. 5 (y) divided by the tangent of the angle (α _A ) for camera A 101 .

[0113] The position of the object 105 on the axis perpendicular to the reference plane may be found by Eq. 7, which is applied to the position in each image, using the distance of the object of interest 105 from the camera. 8 $z=l\frac{p}{f}$

[0114] In Eq. 7, the position (z) is calculated as the position (p) on the image plane projected onto the vector of the image plane perpendicular to that use in Eq. 3 divided by the focal length (f) multiplied by the distance of the object of interest 105 from the camera (l).

[0115] These relations provide a coordinate of the object of interest 105 relative to Camera A 101 . Knowing the position and size of the region of interest 103 relative to Camera A 101 , the coordinate may be converted so that it is relative to the region of interest 103 , 312 of FIG. 3 .

[0116] Smoothing may optionally be applied to these coordinates in refinement module 313 of the implementation of this system shown in FIG. 3 . Smoothing is a process of combining the results with those solved previously so that motion is steady from frame to frame. One method of smoothing for these particular coordinate values (x _A , y, z found by combination module 312 ) is described herein. Each of the components of the coordinate values associated with the object of interest 105 , that is x, y, and z, are smoothed independently and dynamically. The degree of dampening S is calculated by Eq. 8, where S is dynamically and automatically adjusted in response to the change in position is calculated as follows: 9 $S=\{\begin{array}{cc}{S}_A& \mathrm{if}\left(D\le D_A\right)\\ \alpha S_B+\left(1-\alpha \right)S_A\mathrm{where}\alpha =\frac{D-D_A}{D_B-D_A}& \mathrm{if}\left(D_A<D<D_B\right)\\ S_B& \mathrm{if}\left(D\ge D_B\right)\end{array}D=r\left(t\right)-s\left(t-1\right)$

[0117] In Eq. 8, s(t) is the smoothed value at time t, r(t) is the raw value at time t, D _A and D _B are thresholds, and S _A and S _B define degrees of dampening.

[0118] Two distance thresholds, D _A and D _B , as shown in FIG. 10 , define three ranges of motion. A change in position that is less than D _A , motion is heavily dampened 1001 by S _A , thereby reducing the tendency of a value to switch back and forth between two nearby values (a side effect of the discrete sampling of the images). A change in position greater than D _B is lightly dampened 1002 by S _B , or not dampened. This reduces or eliminates lag and vagueness that is introduced in some other smoothing procedures. The degree of dampening is varied for motion between D _A and D _B , the region marked as 1003 , so that the transition between light and heavy dampening is less noticeable. The scalar a, which is applied to Eq. 1, is found by Eq. 9 as follows: 10 $a=\frac{e\left(1-S\right)}{S}$

[0119] In Eq. 9, scalar (a) is bound such that equal to or greater than zero, and less than or equal to one, the dampening value of S is found by Eq. 8, and e is the elapsed time since the previous frame.

[0120] These coordinates 314 of the object of interest 105 , if found, are typically conveyed to another process such as a user application program 316 for use. They may be conveyed to another process executing on the same image processor 106 as the above calculations where performed, or to another computing device. The method in which the data are conveyed to the application program 316 may include emulation of a traditional user input device (including mouse and keyboard), allowing the system to provide control of existing control functions within the application program 316 . The coordinates 314 of the object of interest 105 may be calculated for every video frame captured by the cameras, where one video frame is typically captured 30 times or more every second. This results in little latency between the user's actions and the application's reactions.

[0121] In a typical implementation of the system, the application program 316 provides user feedback by displaying to the video display 107 a visual representation of an indicator. The indicator is caused to move such that its position and motion mimics the motion of the object of interest 105 (typically the user's hand).

[0122] In one variation of this form of user interface, the indicator, such as a mouse pointer, is shown in front of other graphics, and its movements are mapped to the two dimensional space defined by the surface of the screen. This form of control is analogous to that provided by a computer mouse, such as that used with the Microsoft® Windows® operating system. An example feedback image of an application that uses this style of control is shown as 1102 in FIG. 11A .

[0123] Referring to FIG. 11A (and briefly to FIG. 3 ), the image processor 106 also includes an optional coordinate re-mapping process 317 ( FIG. 3 ). The coordinate re-mapping process 317 is operable to remap the global presence and position coordinates 314 (associated with the object of interest 105 ) into the position where the indicator 1101 (such as a cursor or mouse pointer) is overlaid onto the image 1102 by way of Eq. 10 for the x coordinate, and the equivalent of this equation for the y coordinate, as follows: 11 $x_c=\left\{\begin{array}{ccc}0& \mathrm{if}& x_h<b_l\\ \frac{x_h-b_l}{b_r-b_l}& \mathrm{if}& b_l\le x_h\le b_r\\ 1& \mathrm{if}& x_h>b_r\end{array}\right\}$

[0124] In Eq. 10, x _h is the coordinate position 314 associated with the object 105 , x _c