DETAILED DESCRIPTION

[0079] FIG. 1 shows one implementation of a video-based image control system 100 . A person (or multiple people) 101 locates him or herself in, or reaching with his hand or hands into, a region of interest 102 . The region of interest 102 is positioned relative to an image detector 103 so as to be in the overall field of view 104 of the image detector. The region of interest 102 contains a hand detection region 105 within which parts of the person's body, if present and detectable, are located and their positions and motions measured. The regions, positions and measures are expressed in a three-dimensional x, y, z coordinate or world-coordinate system 106 which does not need to be aligned to the image detector 103 . A series of video images generated by the image detector 103 are processed by a computing apparatus 107 , such as a personal computer, capable of displaying a video image on a video display 108 .

[0080] As will be described in greater detail below, the computing apparatus 107 processes the series of video images in order to analyze the position and gestures of an object such as the user's hand. The resulting position and gesture information then is mapped into an application program, such as a graphical user interface (GUI) or a video game. A representation of the position and gestures of the user's hand (such as a screen pointer or cursor) is presented on the video display 108 and allows functions within the GUI or video game to be executed and/or controlled. An exemplary function is moving the cursor over a screen button and receiving a “click” or “press” gesture to select the screen button. The function associated with the button may then be executed by the computing apparatus 107 . The image detector 103 is described in greater detail below. System 100 may be implemented in a variety of configurations including a desktop configuration where the image detector 103 is mounted on a top surface of the video display 108 for viewing the region of interest 102 , or alternatively an overhead camera configuration where the image detector 103 is mounted on a support structure and positioned above the video display 108 for viewing the region of interest 102 .

[0081] FIG. 2 shows the video image analysis process 200 , that may be implemented through computer software or alternatively computer hardware, involved in a typical implementation of the system 100 . The image detector or video camera 103 acquires stereo images 201 of the region of interest 102 and the surrounding scene. These stereo images 201 are conveyed to the computing apparatus 107 (which may optionally be incorporated into the image detector 103 ), which performs a stereo analysis process 202 on the stereo images 201 to produce a scene description 203 . From the scene description 203 , computing apparatus 107 or a different computing device, uses a scene analysis process 204 to calculate and output hand/object position information 205 of the person's (or people's) hand(s) or other suitable pointing device and optionally the positions or measures of other features of the person's body. The hand/object position information 205 is a set of three-dimensional coordinates that are provided to a position mapping process 207 that maps or transforms the three-dimensional coordinates to a scaled set of screen coordinates. These screen coordinates produced by the position mapping process 207 can then be used as screen coordinate position information by an application program 208 that runs on the computing apparatus 107 and provides user feedback 206 .

[0082] Certain motions made by the hand(s), which are detected as changes in the position of the hand(s) and/or other features represented as the hand/object position information 205 , may also be detected and interpreted by a gesture analysis and detection process 209 as gesture information or gestures 211 . The screen coordinate position information from the position mapping process 207 along with the gesture information 211 is then communicated to, and used to control, the application program 208 .

[0083] The detection of gestures may be context sensitive, in which case an application state 210 may be used by the gesture detection process 209 , and the criteria and meaning of gestures may be selected by the application program 208 . An example of an application state 210 is a condition where the appearance of the cursor changes depending upon its displayed location on the video screen 108 . Thus, if the user moves the cursor from one screen object to a different screen object, the icon representing the cursor may for example change from a pointer icon to a hand icon. Typically, the user receives feedback 206 as changes in the image presented on the video display 108 . In general, the feedback 206 is provided by the application program 208 and pertains to the hand position and the state of the application on the video display 108 .

[0084] The image detector 103 and the computing device 107 produce scene description information 203 that includes a three-dimensional position, or information from which the three-dimensional position is implied, for all or some subset of the objects or parts of the objects that make up the scene. Objects detected by the stereo cameras within the image detector 103 may be excluded from consideration if their positions lie outside the region of interest 102 , or if they have shape or other qualities inconsistent with those expected of a person in a pose consistent with the typical use of the system 100 . As a result, few limitations are imposed on the environment in which the system may operate. The environment may even contain additional people who are not interacting with the system. This is a unique aspect of the system 100 relative to other tracking systems that require that the parts of the image(s) that do not make up the user, that is the background, be static and/or modeled.

[0085] Also, few limitations are imposed on the appearance of the user and hand, as it is the general three-dimensional shape of the person and arm that is used to identify the hand. The user 101 may even wear a glove or mitten while operating system 100 . This is also a unique aspect of system 100 , as compared to other tracking systems that make use of the appearance of the hand, most commonly skin color, to identify the hand. Thus, system 100 can be considered more robust than methods relying on the appearance of the user and hand, because the appearance of bodies and hands are highly variable among poses and different people. However, it should be noted that appearance may be used by some implementations of the stereo analysis process 202 that are compatible with the system 100 .

[0086] Typically, the scene description information 203 is produced through the use of stereo cameras. In such a system, the image detector 103 consists of two or more individual cameras and is referred to as a stereo camera head. The cameras may be black and white video cameras or may alternatively be color video cameras. Each individual camera acquires an image of the scene from a unique viewpoint and produces a series of video images. Using the relative positions of parts of the scene of each camera image, the computing device 107 can infer the distance of the object from the image detector 103 , as desired for the scene description 203 .

[0087] An implementation of a stereo camera image detector 103 that has been used for this system is described in greater detail below. Other stereo camera systems and algorithms exist that produce a scene description suitable for this system, and it should be understood that it is not intended that this system be limited to using the particular stereo system described herein.

[0088] Turning to FIG. 3 each camera 301 , 302 of the image detector or stereo camera head 103 detects and produces an image of the scene that is within that camera's field of view 304 , 305 (respectively). The overall field of view 104 is defined as the intersection of all the individual field of views 304 , 305 . Objects 307 within the overall field of view 104 have the potential to be detected, as a whole or in parts, by all the cameras 301 , 302 . The objects 307 may not necessarily lie within the region of interest 102 . This is permissible because the scene description 203 is permitted to contain objects, or features of objects, that are outside the region of interest 102 . With respect to FIG. 3 , it should be noted that the hand detection region 105 is a subset of the region of interest 102 .

[0089] With respect to FIG. 4 , each image 401 and 402 of the pair of images 201 , is detected by the pair of cameras 103 . There exists a set of lines in the image 401 , such that for each line 403 of that set, there exists a corresponding line 404 in the other image 402 . Further, any common point 405 in the scene that is located on the line 403 , will also be located on the corresponding line 404 in the second camera image 402 , so long as that point is within the overall field of view 104 and visible by both cameras 301 , 302 (for example, not occluded by another object in the scene). These lines 403 , 404 are referred to as epipolar lines. The difference in position of the point on each of the epipolar lines of the pair is referred to as disparity. Disparity is inversely proportional to distance, and therefore provides information required to produce the scene description 203 .

[0090] The epipolar line pairs are dependent on the distortion in the cameras' images and the geometric relationship between the cameras 301 , 302 . These properties are determined and optionally analyzed through a pre-process referred to as calibration. The system must account for the radial distortion introduced by the lenses used on most cameras. One technique for resolving those camera characteristics that describe this radial distortion is presented in Z. Zhang, A Flexible New Technique for Camera Calibration, Microsoft Research, http://research.microsoft.com/˜zhang, which is incorporated by reference, and may be used as the first step of calibration. This technique will not find the epipolar lines, but it causes the lines to be straight, which simplifies finding them. A subset of the methods described in Z. Zhang, Determining the Epipolar Geometry and its Uncertainty: A Review, The International Journal of Computer Vision 1997, and Z. Zhang, Determining the Epipolar Geometry and its Uncertainty: A review, Technical Report 2927, INRIA Sophia Antipolis, France, July 1996, both of which are incorporated by reference, may be applied to solve the epipolar lines, as the second step of calibration.

[0091] One implementation of a stereo analysis process 202 that has been used to produce the scene description 203 is described in FIG. 5 . The image pair 201 includes a reference image 401 and a comparison image 402 . Individual images 401 and 402 are filtered by an image filter 503 and broken into features at block 504 . Each feature is represented as an 8×8 block of pixels. However it should be understood that the features may be defined in pixel blocks that are larger or smaller than 8×8 and processed accordingly.

[0092] A matching process 505 seeks a match for each feature in the reference image. To this end, a feature comparison process 506 compares each feature in the reference image to all features that lie within a predefined range along the corresponding epipolar line, in the second or comparison image 402 . In this particular implementation, a feature is defined as an 8×8 pixel block of the image 401 or 402 , where the block is expected to contain a part of an object in the scene, represented as a pattern of pixel intensities (which, due to the filtering by the image filter 503 , may not directly represent luminance) within the block. The likelihood that each pair of features matches is recorded and indexed by the disparity. Blocks within the reference image 401 are eliminated by a feature pair filter 507 if the best feature pair's likelihood of a match is weak (as compared to a predefined threshold), or if multiple feature pairs have similar likelihood of being the best match (where features are considered similar if the difference in their likelihood is within a predefined threshold). Of remaining reference features, the likelihood of all feature pairs is adjusted by a neighborhood support process 508 by an amount proportional to the likelihood found for neighboring reference features with feature pairs of similar disparity. For each reference feature, the feature pair with the best likelihood may now be selected by a feature pair selection process 509 , providing a disparity (and hence, distance) for each reference feature.

[0093] Due to occlusion, a reference feature (produced by process 504 ) may not be represented in the second or comparison image 402 and the most likely matching feature that is present will be erroneous. Therefore, in a two camera system, the features selected in comparison image 402 are examined by a similar procedure (by applying processes 506 , 507 , 508 , and 509 in a second parallel matching process 510 ) to determine the best matching features of those in reference image 401 , a reversal of the previous roles for images 401 and 402 . In a three camera system (i.e., a third camera is used in addition to cameras 301 and 302 ), the third camera's image replaces the comparison image 402 , and the original reference image 401 continues to be used as the reference image, by a similar procedure (by applying processes 506 , 507 , 508 , and 509 in the second parallel matching process 510 ) to determine the best matching features of those in the third image. If more than three cameras are available, this process can be repeated for each of the additional camera images. Any reference feature whose best matching paired feature has a more likely matching feature in the reference image 401 is eliminated in a comparison process 511 . As a result, many erroneous matches, and therefore erroneous distances, caused by occlusion are eliminated.

[0094] The result of the above procedure is a depth description map 512 that describes the position and disparity of features relative to the images 401 , 402 . These positions and disparities (measured in pixels) are transformed by a coordinate system transformation process 513 to the arbitrary three-dimensional world coordinate system (x, y, z coordinate system) ( 106 of FIG. 1 ) by applying Eq. 1, Eq. 2 and Eq. 3, which are presented below. Disparity can be difficult to work with because it is non-linearly related to distance. For this reason, these equations generally are applied at this time so that the coordinates of the scene description 203 are described in terms of linear distance relative to the world coordinate system 106 . Application of these equations, however, will re-distribute the coordinates of the features in such a way that the density of features in a region will be affected, which makes the process of clustering features (performed in a later step) more difficult. Therefore, the original image-based coordinates typically are maintained along with the transformed coordinates.

[0095] This transformed depth description map produced by transformation process 513 is the scene description 203 (of FIG. 2 ). It is the task of the scene analysis process 204 to make sense of this information and extract useful data. Typically, the scene analysis process 204 is dependent on the particular scenario in which this system is applied.

[0096] FIG. 6 presents a flow diagram that summarizes an implementation of the scene analysis process 204 . In the scene analysis process 204 , features within the scene-description 203 are filtered by a feature cropping module 601 to exclude features with positions that indicate that the features are unlikely to belong to the user or are outside the region of interest 102 . Module 601 also eliminates the background and other “distractions” (for example, another person standing behind the user).

[0097] Typically, the region of interest 102 is defined as a bounding box aligned to the world-coordinate system 106 . When this is the case, module 601 may easily check whether the coordinates of each feature are within the bounding box.

[0098] Often, parts of the background can be detected to be within the region of interest 102 , or a box-shaped region of interest may be incapable of definitively separating the user 101 from the background (particularly in confined spaces). When it is known that no user is within the region of interest 102 , the scene description 203 is optionally sampled and modified by a background sampling module 602 to produce a background reference 603 . The background reference 603 is a description of the shape of the scene that is invariant to changes in the appearance of the scene (for example, changes in illumination). Therefore, it is typically sufficient to sample the scene only when the system 100 is setup, and that reference will remain valid as long as the structure of the scene remains unchanged. The position of a feature forming part of the scene may vary by a small amount over time, typically due to signal noise. To assure that the observed background remains within the shape defined by the background reference 603 , the background sampling module 602 may observe the scene description 203 for a short period of time (typically 1 to 5 seconds), and record the features nearest to the cameras 103 for all locations. Furthermore, the value defined by those features is expanded further by a predetermined distance (typically the distance corresponding to a one pixel change in disparity at the features' distances). Once sampling is complete, this background reference 603 can be compared to scene descriptions 203 , and any features within the scene description 203 that are on or behind the background reference are removed by the feature cropping module 601 .

[0099] After feature cropping, the next step is to cluster the remaining features into collections of one or more features by way of a feature clustering process 604 . Each feature is compared to its neighbors within a predefined range. Features tend to be distributed more evenly in their image coordinates than in their transformed coordinates, so the neighbor distance typically is measured using the image coordinates. The maximum acceptable range is pre-defined, and is dependent on the particular stereo analysis process, such as stereo analysis process 202 , that is used. The stereo analysis process 202 described above produces relatively dense and evenly distributed features, and therefore its use leads to easier clustering than if some other stereo processing techniques are used. Of those feature pairs that meet the criteria to be considered neighbors, their nearness in the axis most dependent on disparity (z-axis in those scenarios where the cameras are positioned in front of the region of interest, or the y-axis in those scenarios where the cameras are positioned above the region of interest) is checked against a predefined range. A cluster may include pairs of features that do not meet these criteria if there exists some path through the cluster of features that joins those features such that the pairs of features along this path meet the criteria.

[0100] Continuing with this implementation, clusters are filtered using a cluster filtering process 605 to assure that the cluster has qualities consistent with objects of the kinds expected to be present within the region of interest 102 , and are not the result of features whose position (or disparity) has been erroneously identified in the stereo processing routine. Also, as part of the cluster filtering process 605 , clusters that contain too few features to provide a confident measure of their size, shape, or position are eliminated. Measurements of the cluster's area, bounding size, and count of features are made and compared to predefined thresholds that describe minimum quantities of these measures. Clusters, and their features, that do not pass these criteria are removed from further consideration.

[0101] The presence or absence of a person is determined by a presence detection module 606 in this implementation. The presence detection module 606 is optional because the information that this component provides is not required by all systems. In its simplest form, the presence detection module 606 need only check for the presence of features (not previously eliminated) within the bounds of a predefined presence detection region 607 . The presence detection region 607 is any region that is likely to be occupied in part by some part of the user 101 , and is not likely to be occupied by any object when the user is not present. The presence detection region 607 is typically coincident to the region of interest 102 . In specific installations of this system, however, the presence detection region 607 may be defined to avoid stationary objects within the scene. In implementations where this component is applied, further processing may be skipped if no user 101 is found.

[0102] In the described implementation of system 100 , a hand detection region 105 is defined. The method by which this region 105 is defined (by process 609 ) is dependent on the scenario in which the system is applied, and is discussed in greater detail below. That procedure may optionally analyze the user's body and return additional information including body position(s)/measure(s) information 610 , such as the position of the person's head.

[0103] The hand detection region 105 is expected to contain nothing or only the person's hand(s) or suitable pointer. Any clusters that have not been previously removed by filtering and that have features within the hand detection region 105 are considered to be, or include, hands or pointers. A position is calculated (by process 611 ) for each of these clusters, and if that position is within the hand detection region 105 , it is recorded (in memory) as hand position coordinates 612 . Typically, the position is measured as a weighted mean. The cluster's feature (identified by 1005 of the example presented in FIG. 10 ) that is furthest from side of entry ( 1002 in that example) of the hand detection region 105 is identified, and its position is given a weight of 1 based on the assumption that it is likely to represent the tip of a finger or pointer. The remaining weights of cluster features are based on the distance back from this feature, using the formula of Eq. 4 provided below. If only one hand position is required by the application and multiple clusters have features within the hand detection region 105 , the position that is furthest from the side of entry 1002 is provided as the hand position 612 and other positions are discarded. Therefore, the hand that reaches furthest into the hand detection region 105 is used. Otherwise, if more than two clusters have features within the hand detection region 105 , the position that is furthest from the side of entry 1002 and the position that is second furthest from the side of entry 1002 are provided as the hand positions 612 and other positions are discarded. Whenever these rules cause a cluster to be included in place of a different cluster, the included clusters are tagged as such in the hand position data 612 .

[0104] In those scenarios where the orientation of the cameras is such that the person's arm is detectable, the orientation is represented as hand orientation coordinates 613 of the arm or pointer, and may optionally be calculated by a hand orientation calculation module 614 . This is the case if the elevation of the cameras 103 is sufficiently high relative to the hand detection region 105 , including those scenarios where the cameras 103 are directly above the hand detection region 105 . The orientation may be represented by the principal axis of the cluster, which is calculated from the moments of the cluster.

[0105] An alternative method that also yields good results, in particular when the features are not evenly distributed, is as follows. The position where the arm enters the hand detection region 105 is found as the position where the cluster is dissected by the plane formed by that boundary of the hand detection region 105 . The vector between that position and the hand position coordinates 612 provides the hand orientation coordinates 613 .

[0106] A dynamic smoothing process 615 may optionally be applied to the hand position coordinate(s) 612 , the hand orientation(s) coordinates 613 (if solved), and any additional body positions or measures 610 . Smoothing is a process of combining the results with those solved previously so that motion is steady from frame to frame. The one particular of smoothing for these particular coordinate values, each of the components of the coordinate, that is x, y, and z, are smoothed independently and dynamically. The degree of dampening S is calculated by Eq. 5, which is provided below, where S is dynamically and automatically adjusted in response to the change in position. Two distance thresholds, D _A and D _B , as shown in FIG. 7 , define three ranges of motion. For a change in position that is less than D _A , motion is heavily dampened in region 701 by S _A , thereby reducing the tendency of a value to switch back and forth between two nearby values (a side effect of the discrete sampling of the images). A change in position greater than D _B is lightly dampened in region 702 by S _B , or not dampened. This reduces or eliminates lag and vagueness that is introduced in some other smoothing procedures. The degree of dampening is varied for motion between D _A and D _B , the region marked as 703 , so that the transition between light and heavy dampening is less noticeable. Eq. 6, which is provided below, is used to solve the scalar a, which is used in Eq. 7 (also provided below) to modify the coordinate(s). The result of dynamic smoothing process 615 is the hand/object position information 205 of FIG. 2 . Smoothing is not applied when process 611 has tagged the position as belonging to a different cluster than the previous position, since the current and previous positions are independent.

[0107] The described method by which the hand detection region 105 is determined at step 609 is dependent on the scenario in which the image control system 100 is applied. Two scenarios are discussed here.

[0108] The simplest hand detection region 105 is a predetermined fixed region that is expected to contain either nothing or only the person's hand(s) or pointer. One scenario where this definition applies is the use of system 100 for controlling the user interface of a personal computer, where the hand detection region 105 is a region in front of the computer's display monitor 108 , and above the computer's keyboard 802 , as depicted in FIG. 8 . In the traditional use of the computer, the user's hands or other objects do not normally enter this region. Therefore, any object found to be moving within the hand detection region 105 may be interpreted as an effort by the user 101 to perform the action of “pointing”, using his or her hand or a pointer, where a pointer may be any object suitable for performing the act of pointing, including, for example, a pencil or other suitable pointing device. It should be noted that specific implementation of the stereo analysis process 202 may impose constraints on the types or appearance of objects used as pointers. Additionally, the optional presence detection region, discussed above, may be defined as region 801 , to include, in this scenario, the user's head. The image detector 103 may be placed above the monitor 108 .

[0109] In some scenarios, the hand detection region 105 may be dynamically defined relative to the user's body and expected to contain either nothing or only the person's hand(s) or pointer. The use of a dynamic region removes the restriction that the user be positioned at a predetermined position. FIG. 1 depicts a scenario in which this implementation may be employed.

[0110] FIG. 9 shows an implementation of the optional dynamic hand detection region positioning process 609 in greater detail. In this process, the position of the hand detection region 105 on each of three axes is solved, while the size and orientation of the hand detection region 105 are dictated by predefined specifications. FIGS. 10 A- 10 C present an example that is used to help illustrate this process.

[0111] Using the cluster data 901 (the output of the cluster filtering process 605 of FIG. 6 ), the described procedure involves finding, in block 902 , the position of a plane 1001 (such as a torso-divisioning plane illustrated in the side view depicted in FIG. 10C ) whose orientation is parallel to the boundary 1002 of the hand detection region 105 through which the user 101 is expected to reach. If the features are expected to be evenly distributed over the original images (as is the case when the implementation of the stereo analysis process 202 described above is used), then it is expected that the majority of the remaining features will belong to the user's torso, and not his hand. In this case, the plane 1001 may be positioned so that it segments the features into two groups of equal count. If the features are expected to be unevenly distributed (as is the case when some alternative implementations of the stereo analysis process 202 are used), then the above assumption may not be true. However, the majority of features that form the outer bounds of the cluster are still expected to belong to the torso. In this case, the plane 1001 may be positioned so that it segments the outer-most features into two groups of equal count. In either case, the plane 1001 will be positioned by the torso-divisioning process in block 902 so that it is likely to pass through the user's torso.

[0112] Process block 903 determines the position of the hand detection region 105 along the axis that is defined normal to plane 1001 found above. The hand detection region 105 is defined to be a predetermined distance 1004 in front of plane 1001 , and therefore in front of the user's body. In the case of FIG. 1 , distance 1004 determines the position of the hand detection region 105 along the z-axis.

[0113] If the user's head is entirely within the region of interest 102 , then the position of the topmost feature of the cluster is expected to represent the top of the user's head (and therefore to imply the user's height), and is found in process block 904 of this implementation. In process block 905 , the hand detection region 105 is positioned based on this head position, a predefined distance below the top of the user's head. In the case of FIG. 1 , the predefined distance determines the position of the hand detection region along the y-axis. If the user's height cannot be measured, or if the cluster reaches the border of the region of interest 102 (implying that the person extends beyond the region of interest 102 ), then the hand detection region 105 is placed at a predefined height.

[0114] In many scenarios, it can be determined whether the user's left or right arm is associated with each hand that is detected in the position calculation block 611 of FIG. 6 . In process block 906 , the position where the arm intersects a plane that is a predefined position in front of plane 1001 is determined. Typically, this plane is coincident to the hand detection region boundary indicated by 1002 . If no features are near this plane, but if some features are found in front of this plane, then it is likely that those features occlude the intersection with that plane, and the position of intersection may be assumed to be behind the occluding features. By shortest neighbor distances between the features of the blocks, each intersection is associated with a hand point.

[0115] The position of the middle of the user's body and the bounds of the user's body are also found in process block 907 . Typically, this position is, given evenly distributed features, the mean position of all the features in the cluster. If features are not expected to be evenly distributed, the alternative measure of the position halfway between the cluster's bounds may be used.

[0116] In process block 908 , the arm-dependent position found by process block 906 is compared to the body centric position found by process block 907 . If the arm position is sufficiently offset (e.g., by greater than a predefined position that may be scaled by the cluster's overall width) to either the left or right of the body-center position, then it may be implied that the source of the arm comes from the left or right shoulder of the user 101 . If two hands are found but only one hand may be labeled as “left” or “right” with certainty, the label of the other hand may be implied. Therefore, each hand is labeled as “left” or “right” based on the cluster's structure, assuring proper labeling in many scenarios where both hands are found and the left hand position is to the right of the right hand position.

[0117] If one hand is identified by process block 908 , then the hand detection region 105 may be placed (by process block 909 ) so that all parts of the hand detection region 105 are within an expected range of motion associated with the user's hand. The position of the hand detection region 105 along the remaining axis may be biased towards the arm of the arm as defined by Eq. 8 (which is provided below). If process block 908 failed to identify the arm, or if it is otherwise desired, the position of the hand detection region 105 along the remaining axis may be positioned at the center of the user's body as found by 907 . In scenarios where tracking of both hands is desired, the hand detection region 105 may be positioned at the center of the user's body.

[0118] Process blocks 903 , 906 and 909 each solve the position of the hand detection region 105 in one axis, and together define the position of the hand detection region 105 within three-dimensional space. That position is smoothed by a dynamic smoothing process 910 by the same method used by component 615 (using Eq. 5, Eq. 6, and Eq. 7). However, a higher level of dampening may be used in process 910 .

[0119] The smoothed position information output from the dynamic smoothing process 910 , plus predefined size and orientation information 911 , completely defines the bounds of the hand detection region 105 . In solving the position of the hand detection region 105 , process blocks 905 , 907 , and 908 find a variety of additional body position measures 913 ( 610 of FIG. 6 ) of the user.

[0120] In summary, the above implementation described by FIG. 6 , using all the optional components including those of FIG. 9 , produces a description of person(s) in the scene (represented as the hand/object position information 205 of FIG. 2 ) that includes the following information:

[0121] Presence/absence or count of users

[0122] For each present user:

[0123] Left/Right bounds of the body or torso

[0124] Center point of the body or torso

[0125] Top of the head (if the head is within the region of interest)

[0126] For each present hand:

[0127] The hand detection region

[0128] A label of “Left”, “Right” (if detectable)

[0129] The position of the tip of the hand

[0130] The orientation of the hand or forearm

[0131] Given improvements in the resolution of the scene description 203 , the implementations described here may be expanded to describe the user in greater detail (for example, identifying elbow positions).

[0132] This hand/object position information 205 , a subset of this information, or further information that may be implied from the above information, is sufficient to allow the user to interact with and/or control a variety of application programs 208 . The control of three applications is described in greater detail below.

[0133] Through processing the above information, a variety of human gestures can be detected that are independent of the application 208 and the specific control analogy described below. An example of such a gesture is “drawing a circle in the air” or “swiping the hand off to one side”. Typically, these kinds of gestures be detected by the gesture analysis and detection process 209 using the hand/object position information 205 .

[0134] A large subset of these gestures may be detected using heuristic techniques. The detection process 209