DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] An embodiment of the present invention will be described hereinbelow. FIG. 1 is an external view of the constitution of the close region image extraction device according to the present embodiment. This close region image extraction device is constituted by a mounted portion 10 that is mounted on the regions of the face of a cameraman (a photographer) P, and a wearable computer 20 which is attached at the waist, for example, of the cameraman P. The mounted portion 10 and the wearable computer 20 are electrically connected by an electrical cable, but could be communicably connected wirelessly rather than by an electrical cable.

[0024] The mounted portion 10 comprises a photographic unit 101 , a control unit 102 , and a display unit 103 . The photographic unit 101 comprises a beam splitter 13 that receives light reflected by a target object, and infrared light source sections 15 which are arranged above and below the beam splitter 13 . The beam splitter 13 is disposed close to the cameraman's temples so as to be capable of receiving light reflected by the target object within the field of view of the cameraman P. This close region image extraction device photographs a nearby object OB grasped by the cameraman P by means of the color video camera, extracts an image of the nearby object OB (object image) from the color moving image thus obtained, and causes the display unit 103 to display this object image as a moving image.

[0025] The infrared light source sections 15 are constituted having infrared light emitting diodes arranged in predetermined columns and rows. The control unit 102 comprises a field multiplexer and a sync circuit (described subsequently).

[0026] The display unit 103 comprises eyeglasses 104 and a monocular display 105 which is disposed on the frame of the eyeglasses 104 and in front of either the left or right eye of the cameraman P (the right eye of the cameraman P in FIG. 1 ).

[0027] FIG. 2 is a block diagram showing the overall constitution of this close region image extraction device. As shown in FIG. 2 , the mounted portion 10 comprises a color camera 111 (as an image capture means), an infrared camera 12 , the beam splitter 13 , a field multiplexer 14 , the infrared light source sections 15 , a sync circuit 16 , an infrared transmitting filter 17 and a display device 18 .

[0028] The color camera 11 is disposed above the beam splitter 13 in the figure and acquires color moving images at a predetermined field rate. The color camera 11 comprises an area CCD according to which pixels are arranged in predetermined columns and rows so as to match the pixels of the display device 18 . The infrared camera 12 is provided on the right-hand side of the beam splitter 13 in the figure and acquires infrared images at the same field rate as the color camera 11 . The infrared camera 12 comprises an area CCD with a high sensitivity to the infrared light band and according to which pixels are arranged in predetermined columns and rows so as to match the pixels of the display device 18 .

[0029] The beam splitter 13 reflects an optical image of the nearby object OB such that this optical image is directed toward the color camera 11 , and transmits the light reflected by the nearby object OB such that this light is directed toward the infrared camera 12 . As a result, the optical axes of the color camera 11 and the infrared camera 12 coincide with each another and both cameras are able to photograph the same target object. The infrared transmitting filter 17 is disposed between the beam splitter 13 and the infrared camera 12 . Thus, only the infrared light component contained in the light reflected by the target object is extracted and directed toward the infrared camera 12 . Further, in a case where a beam splitter that possesses the characteristic of transmitting only the infrared light and directing this light toward the infrared camera 12 is adopted as the beam splitter 13 , the infrared transmitting filter 17 is no longer required.

[0030] The infrared light source sections 15 are constituted by infrared light emitting diodes and a drive circuit for supplying a drive current to the infrared light emitting diodes, and so forth, and illuminate the nearby object OB by repeatedly turning an infrared light alternately ON and OFF under the control of the sync circuit 16 .

[0031] The sync circuit 16 causes the infrared light source sections 15 to repeatedly turn the infrared light ON and OFF in sync with the timing with which the color camera 11 acquires field images and thus causes the infrared camera 12 to acquire infrared images in sync with the timing with which the color camera 11 acquires field images. As a result, the infrared camera 12 is capable of alternately acquiring a lit infrared image which is an infrared image when the infrared light source sections 15 are ON, and an unlit infrared image which is an infrared image when the infrared light source sections 15 are OFF, in sync with the timing with which the color camera 11 acquires field images. The perpendicular resolving power of each of a lit infrared image and an unlit infrared image is half the perpendicular resolving power of the frame image.

[0032] Further, in the description that follows, when the color camera 11 acquires an odd-numbered field image (an image representing only the odd-numbered columns of a single frame image), the infrared camera 12 acquires a lit infrared image. Further, when the color camera 11 acquires an even-numbered field image (an image representing only the even-numbered columns of a single frame image), the infrared camera 12 acquires an unlit infrared image. However, the present invention is not limited to such image acquisition, that is, image acquisition according to which the infrared camera 12 acquires an unlit infrared image when the color camera 11 acquires an odd-numbered field image and the infrared camera 12 acquires an unlit infrared image when the color camera 11 acquires an even-numbered field image is also possible.

[0033] The field multiplexer 14 forms a single image (synthesized image) by synthesizing a total of four images which are an odd-numbered field image and a lit infrared image acquired at the same time as the odd-numbered field image, and an even-numbered field image and an unlit infrared image acquired at the same time as the even-numbered field image, and then outputs this single image to the wearable computer 20 . The four images can thus be efficiently outputted to the wearable computer 20 . The field multiplexer 14 generates this synthesized image every time these four images are acquired, and outputs this image to the wearable computer 20 . Therefore, supposing that the cycle in which a field image is acquired is ΔT, the synthesized image is generated in the cycle 2ΔT.

[0034] FIG. 3 shows an example of a synthesized image generated by the field multiplexer 14 . As shown in FIG. 3 , the field multiplexer 14 divides a single image into two areas A 1 and A 2 which are disposed left and right of the center in a horizontal direction. Further, in area A 1 , odd-numbered field images and even-numbered field images in which the number of pixels in the horizontal direction has been reduced by half are arranged alternately line by line in sequence starting from an odd-numbered field image. Also, in area A 2 , lit infrared images and unlit infrared images in which the number of pixels in the horizontal direction has been reduced by half are arranged alternately line by line in sequence starting from a lit infrared image. In this case, an arrangement is also acceptable according to which, in area A 1 , odd-numbered field images and even-numbered field images are arranged alternately line by line in sequence starting from an even-numbered field image, and, in area A 2 , lit infrared images and unlit infrared images are arranged alternately line by line in sequence starting from an unlit infrared image. In addition, odd-numbered field images and even-numbered field images may be arranged in area A 1 , while lit infrared images and unlit infrared images may be arranged in area A 2 .

[0035] The wearable computer 20 shown in FIG. 2 is constituted by a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random-Access Memory), and an external storage device and the like (all of which are omitted from the figure). An auxiliary storage device stores an operating system and a control program that allows the wearable computer 20 to function as the processing section of the close region image extraction device, and so forth. By executing the control program under the control of the operating system, the CPU allows the wearable computer 20 to function as an image memory 21 , an absolute value differential image acquisition section 22 , a close region image extraction section 23 and an object image extraction section 24 .

[0036] The image memory 21 is constituted by a RAM, for example, and temporarily stores the synthesized images generated by the field multiplexer 14 .

[0037] The absolute value differential image acquisition section 22 reads a single synthesized image from the image memory 21 , acquires a differential image obtained by subtracting an unlit infrared image from the lit infrared image contained in the synthesized image as an absolute value differential image, reads out the next synthesized image from the image memory 21 , and then multiplies, by −1, a differential image obtained by subtracting the lit infrared image contained in the next synthesized image from the unlit image contained in the synthesized image to acquire an absolute differential image. Differential image as it is used here means an image which is obtained by calculating the luminance differential between pixels corresponding to a lit infrared image and an unlit infrared image.

[0038] The close region image extraction section 23 compares the luminance of an absolute value differential image acquired by the absolute value differential image acquisition section 22 with a predetermined threshold value, extracts regions whose luminance exceeds this threshold value as a close region image, subjects a corresponding odd-numbered field image or even-numbered field image to masking processing with the image thus extracted serving as a mask image, and thus extracts a close region image from the odd-numbered field image or even-numbered field image.

[0039] The object image extraction section 24 extracts an object image from the close region image extracted by the close region image extraction section 23 by removing the image of the cameraman's hand that grasps the nearby object in the close region image by eliminating skin-colored pixel regions from the close region image.

[0040] The display device 18 is constituted by the monocular display 105 and displays object images extracted by the object image extraction section 24 .

[0041] Further, in this close region image extraction device, the field multiplexer 14 is equivalent to an image synthesizer, the sync circuit 16 is equivalent to lighting control means, and the close region image extraction section 23 and object image extraction section 24 are equivalent to extracting means.

[0042] Next, the operation of this close region image extraction device will be described with reference to FIGS. 4 to 6 . FIG. 4 serves to illustrate the processing performed by this close region image extraction device. Further, FIG. 5 is a chronological representation of odd-numbered field images and even-numbered field images acquired by the color camera 11 and a chronological representation of lit infrared images and unlit infrared images acquired by the infrared camera 12 . In addition, FIG. 6A shows color moving images acquired by the color camera 11 ; FIG. 6B shows an absolute value differential image generated by using a lit infrared image and an unlit infrared image acquired by the infrared camera 12 ; FIG. 6C shows a close region image; and FIG. 6D shows an object image.

[0043] In FIG. 4 , synthesized images JI 1 , JI 2 , . . . are shown chronologically in a downward direction in a first column; an odd-numbered field image OI 1 , an even-numbered field image EI 1 , a lit infrared image LR 1 and an unlit infrared image OR 1 , . . . are shown chronologically in a downward direction in a second column; absolute value differential images SI 1 , SI 2 , . . . are shown chronologically in a downward direction in a third column; mask images M 1 , M 2 , . . . are shown chronologically in a downward direction in a fourth column; and close region images EX 1 , EX 2 , . . . are shown chronologically in a downward direction in a fifth column. Further, in FIG. 5 , odd-numbered field images OI 1 , OI 2 , . . . and even-numbered field images EI 1 , EI 2 , . . . are shown alternately in chronological order, and lit infrared images LR 1 , LR 2 , . . . and unlit infrared images OR 1 , OR 2 , . . . are shown chronologically. Also in FIG. 5 , the absolute value differential images SI 1 , SI 2 , . . . and close region images EX 1 , EX 2 , . . . are shown chronologically. Further, images which have been assigned the same reference numerals in FIGS. 4 and 5 represent the same image.

[0044] As shown in FIG. 4 , the absolute value differential image acquisition section 22 first subtracts the unlit infrared image OR 1 from the lit infrared image LR 1 contained in the synthesized image JI 1 and thus calculates the absolute value differential image SI 1 . In this case, the absolute value differential image shown in FIG. 6B is acquired. This absolute value differential image is a monochrome image that contains only luminance information.

[0045] Because the absolute value differential image is calculated in this manner, robustness during outdoor usage can be ensured. In an outdoor environment, a large infrared light component is contained in sunlight and therefore the infrared image of the nearby object OB contains an infrared component resulting from sunlight as well as an infrared component caused by illumination infrared light. When the infrared component resulting from sunlight is contained in a large quantity in the infrared image of the nearby object OB, the luminance of the background image excluding the nearby object OB also increases, and, based on this luminance, extraction of only the nearby object OB from the infrared image is difficult. Hence, in order to accurately extract the nearby object OB, it is necessary to remove the infrared component resulting from the sunlight from the infrared image of the nearby object OB so as to extract only the infrared component caused by the illumination infrared light. A lit infrared image comprises an infrared component resulting from sunlight as well as an infrared component caused by illumination infrared light. On the other hand, an unlit infrared image comprises only an infrared component caused by sunlight. Therefore, when the unlit infrared image is extracted from the lit infrared image, the infrared component resulting from the sunlight contained in both infrared images is canceled out, whereby the infrared component caused by illumination infrared light alone can be extracted. Robustness in an outdoor environment is thus ensured.

[0046] Next, the close region image extraction section 23 extracts the mask image Ml from the absolute value differential image SI 1 by comparing the luminance of the pixels of the absolute value differential image SI 1 with a predetermined threshold value. An infrared image possesses the characteristic that the luminance thereof is inversely proportional to the distance squared. In other words, the nearby object OB located near the infrared camera 12 is shown with high luminance in the infrared image, and the background image is shown with luminance that is extremely low in comparison with the nearby object OB. For this reason, the close region image can be extracted from color moving images by setting a predetermined value between the luminance of the nearby object OB and the luminance of the background image as the threshold value and then comparing the luminance of the pixels with this threshold value.

[0047] Next, the close region image extraction section 23 uses the mask image Ml to subject the odd-numbered field image OI 1 to masking, and thus extracts the close region image EX 1 from the odd-numbered field image OI 1 . In this case, the nearby object OB as shown in FIG. 6C and an image of the hand H of the cameraman P are extracted as the close region image.

[0048] Next, the object image extraction section 24 removes the image of the hand H of the cameraman P by eliminating skin-colored pixels from the close region image EX 1 and thus extracts the object image. Here, as shown in FIG. 6D , an image of the nearby object OB (cup) alone is extracted, that is, the object image is extracted. The object image thus extracted is displayed by the display device 18 .

[0049] Then, as shown in FIG. 4 , the absolute value differential image acquisition section 22 acquires the absolute value differential image SI 2 by subtracting the unlit infrared image contained in the synthesized image JI 1 from the lit infrared image LR 2 contained in the synthesized image JI 2 which is generated after the synthesized image JI 1 . In other words, as shown in FIG. 5 , when the absolute value differential image acquisition section 22 acquires even-numbered differential images (absolute value differential images SI 2 , SI 4 , . . . ), absolute value differential images are obtained by multiplying, by −1, differential images which are produced by subtracting a lit infrared image from an unlit infrared image, and then affording the luminance of the pixels of the differential image a positive value. As a result, the absolute value differential images SI 1 , SI 2 , . . . can be obtained at the field rate.

[0050] The display device 18 sequentially displays the object images thus extracted. As a result, the display device 18 displays images of the nearby object OB within the field of view of the cameraman P in compliance with the operation of the cameraman P.

[0051] According to the close region image extraction device described hereinabove, because an absolute value differential image is employed when a differential image of a lit infrared image and an unlit infrared image is calculated in order to ensure robustness outdoors, the object image can be obtained at the field rate (ΔT).

[0052] Further, the assumption is made that, because the color camera 11 and the infrared camera 12 are contained in the mounted portion 10 which is mounted on regions of the user's face, head movement (neck turning movement) is frequently generated and that, here, the image variation for each image acquired is large. However, because this close region image extraction device is capable of acquiring absolute value differential images at the field rate, the object image can be extracted highly accurately.

[0053] Further, because the beam splitter 13 is used to match the optical axes of the color camera 11 and the infrared camera 12 , both cameras are capable of acquiring the same target object, whereby the accuracy of extraction of the close region image can be raised still further.

[0054] In addition, because the field multiplexer 14 is used to generate a synthesized image which is then outputted to the wearable computer 20 , odd-numbered field images, even-numbered field images, lit infrared images and unlit infrared images can be efficiently outputted to the wearable computer 20 .

[0055] Furthermore, because the photographic unit 102 can be mounted such that the beam splitter 13 is located close to the temples of the cameraman P, cameraman P is able to photograph the nearby object OB by observing the nearby object OB. For this reason, without being conscious of the photographic process, the cameraman P is able to use the nearby object OB to acquire an image of the nearby object OB while performing work of some kind.

[0056] In addition, because the wearable computer 20 is used, the cameraman P is able to move about freely with the close region image extraction device thus mounted.

[0057] The present invention may adopt the following embodiment.

[0058] (1) In the above embodiment, the color camera 11 and infrared camera 12 are not limited to acquiring field images and infrared images with the same timing. For example, the synchronization (“in sync”) described in claim 1 also includes an embodiment according to which field images and infrared images are acquired with the timing shifted by a fixed time interval (ΔT) such that the odd-numbered field image OI 1 , lit infrared image LR 1 , even-numbered field image EI 1 , and lit infrared image OR 1 , . . . are acquired in this order as shown in FIG. 7 . In this case, when the conventional method is adopted, that is, a method in which only an image produced by subtracting an unlit infrared image from a lit infrared image is the differential image, differential images are acquired in the cycle 4ΔT. On the other hand, if the method according to the present invention is adopted, because the absolute value differential images SI 1 , SI 2 , SI 3 , . . . are acquired in the cycle 2ΔT, the object image can be extracted at high speed even in this modified example.

[0059] (2) Although, in the above embodiment, the field multiplexer 14 acquires the synthesized image shown in FIG. 3 , other types of synthesized image are equally possible. As shown in FIG. 8 , in area A 1 , lit infrared images and odd-numbered field images in which the number of pixels in the horizontal direction has been reduced by half may be arranged alternately line by line, and, in area A 2 , unlit infrared images and even-numbered field images in which the number of pixels in the horizontal direction has been reduced by half may be arranged alternately line by line.

[0060] (3) Although, in the above embodiment, an interface-mode color camera 11 that acquires images such that same are divided into odd-numbered field images and even-numbered field images are adopted, the present invention is not limited to or by such a camera and may adopt a progressive-mode color camera 11 . Here, in the same way as when the interface-mode color camera is adopted, the field multiplexer 14 may generate a synthesized image by employing a total of four images which are two images acquired by the progressive-mode color camera and a lit infrared image and an unlit infrared image which are acquired in sync with these two images. In this case also, synthesized images can be acquired at the frame rate (2ΔT).

[0061] According to the present invention as described hereinabove, because absolute value differential images of lit infrared images and unlit infrared images are used to extract the close region image, outdoor robustness can be ensured and close region images can be extracted at the field rate.

[0062] Summing up, the present invention was conceived to provide a close region image extraction device which comprises: capture means for acquiring color moving images of the nearby object by using visible light; an infrared light source for irradiating the nearby object with infrared light; lighting control means that repeatedly turn the infrared light source alternately ON and OFF, in sync with the timing with which the capture means acquire field images; infrared image acquiring means that alternately acquire a lit infrared image which is an infrared image of the nearby object when the infrared light source is lit, and an unlit infrared image which is an infrared image of the nearby object when the infrared light source is unlit, in sync with the timing with which the capture means acquire field images; absolute value differential image acquiring means, which acquires an absolute value image for the difference between the lit infrared image and the unlit infrared image acquired in chronological succession, and wherein said absolute image is obtained by multiplying the subtracted values of the lit infrared image from that of the unlit infrared image by minus 1 when the infrared image which corresponds to the current field image is a lit infrared image and the infrared image which corresponds to the previous field is an unlit infrared image; and extracting means for extracting the close region image from the color moving image on the basis of the absolute value differential image acquired by the absolute value differential image acquiring means.

[0063] According to this constitution, color moving images are taken by the capture means, the infrared light source is repeatedly turned alternately ON and OFF in sync with the timing with which the capture means acquire field images, and lit infrared images and unlit infrared images are alternately taken repeatedly by the infrared image acquiring means. Further, an absolute value image for the difference between chronologically successive lit infrared images and unlit infrared images (an absolute value differential image) is acquired. Here, in a case where the infrared image which corresponds to the previous field image is a lit infrared image and the infrared image which corresponds to the current field image is an unlit infrared image, the absolute value differential image is acquired by subtracting the unlit infrared image from the lit infrared image, and, in a case where the infrared image which corresponds to the previous field image is an unlit infrared image and the infrared image which corresponds to the current field image is a lit infrared image, the absolute value differential image is acquired by multiplying, by minus 1, the differential image produced by subtracting the lit infrared image from the unlit infrared image.

[0064] It is thus possible to obtain a differential image for the lit infrared image and the unlit infrared image at the field rate. Further, the close region image is extracted on the basis of this differential image and the corresponding color moving image is then subjected to masking processing with the close region image serving as a mask image, whereby the close region image is extracted from the color moving image.

[0065] Further, the optical axes of the capture means and the infrared image acquiring means are preferably provided so as to coincide with each other. According to this constitution, because the infrared image acquiring means and the capture means are capable of photographing the same target object, the accuracy of extraction of the close region image extracted from the color moving image can be raised still further.

[0066] Also, it is preferable that this close region image extraction device further comprise: an image synthesizer, which synthesizes two chronologically successive field images that are acquired by the capture means, and a lit infrared image and an unlit infrared image acquired in sync with these two field images, to form a single image, and outputs this image to the absolute value differential image acquiring means, wherein the image synthesizer synthesizes the two field images, the lit infrared image and the unlit infrared image by reducing same such that the number of pixels thereof in the horizontal direction is halved, so as to form a single image.

[0067] According to this constitution, because two successive color images, a lit infrared image and an unlit infrared image are synthesized to form a single image by being reduced such that the number of pixels in the horizontal direction in these images is halved, these four images can be efficiently outputted to the absolute value differential image acquiring means.

[0068] Further, the extracting means preferably extract an object image that represents the nearby object by eliminating skin-colored regions from the close region image. According to this constitution, the extracting means are capable of removing the image of the cameraman's hand that grasps the nearby object contained in the close region image and therefore of extracting an object image that represents the nearby object from the color moving image.

[0069] In addition, it is preferable that this close region image extraction device further comprise: an head-mounted display for displaying an object image extracted by the extracting means, wherein the capture means, the infrared light source and the infrared image acquiring means are integrated with the head-mounted display and are provided so that the respective optical axes thereof lie within the field of view of the cameraman.

[0070] According to this constitution, the capture means and the infrared image acquiring means permit the cameraman to grasp a nearby object and to take an image while observing the nearby object thus grasped.

[0071] The close region image extraction method according to the present invention is a close region image extraction method for extracting a close region image that comprises a nearby object located in the vicinity of a cameraman from color moving images, comprising the steps of: using the capture means to take color moving images of the nearby object; repeatedly turning an infrared light source that irradiates the nearby object with infrared light alternately ON and OFF, in sync with the timing with which the capture means acquire field images; using the infrared image acquiring means to alternately acquire a lit infrared image which is an infrared image of the nearby object when the infrared light source is lit, and an unlit infrared image which is an infrared image of the nearby object when the infrared light source is unlit, in sync with the timing with which the capture means acquire field images; acquiring, when the infrared image which corresponds to the current field image is a lit infrared image and the infrared image which corresponds to the previous field is an unlit infrared image, an absolute value image for the difference between the lit infrared image and the unlit infrared image which are in chronological succession, by rendering an image, which is obtained by multiplying the difference of the lit infrared image from the unlit infrared image by minus 1, an absolute value differential image; and extracting the close region image from the color moving image acquired by the capture means on the basis of this absolute value differential image.

[0072] According to this constitution, color moving images are taken by the capture means, the infrared light source is repeatedly turned alternately ON and OFF in sync with the timing with which the capture means acquire field images, and lit infrared images and unlit infrared images are alternately taken repeatedly by the infrared image acquiring means. Further, an absolute value image for the difference between chronologically successive lit infrared images and unlit infrared images (an absolute value differential image) is acquired. Here, in a case where the infrared image which corresponds to the previous field image is a lit infrared image and the infrared image which corresponds to the current field image is an unlit infrared image, the absolute value differential image is acquired by subtracting the unlit infrared image from the lit infrared image, and, in a case where the infrared image which corresponds to the previous field image is an unlit infrared image and the infrared image which corresponds to the current field image is a lit infrared image, the absolute value differential image is acquired by multiplying, by minus 1, the differential image produced by subtracting the lit infrared image from the unlit infrared image.

[0073] It is thus possible to obtain a differential image for the lit infrared image and the unlit infrared image at the field rate. Further, the close region image is extracted on the basis of this differential image and the corresponding color moving image is then subjected to masking processing with the close region image serving as a mask image, whereby the close region image is extracted from the color moving image.

[0074] This application is based on Japanese patent application serial no. 2003-98098, filed in Japan Patent Office on Apr. 1, 2003, the contents of which are hereby incorporated by reference.

[0075] Although the present invention has been fully described by way of example with reference to the accompanying drawings, it is to be understood that various changes and modifications will be apparent to those skilled in the art. Therefore, unless otherwise such changes and modifications depart from the scope of the present invention hereinafter defined, they should be construed as being included therein.