The present invention contains subject matter related to Japanese Patent Application JP 2006-199925 filed in the Japanese Patent Office on Jul. 21, 2006, the entire contents of which being incorporated herein by reference.
1. Field of the Invention
The present invention relates to an image processing apparatus, an image processing method, and a computer program. More particularly, the present invention relates to an image processing apparatus, an image processing method, and a computer program that make it possible to obtain high-quality images in a digital still camera and the like.
2. Description of the Related Art
For example, when a user photographs an image with a digital still camera or the like held by the hand, if an exposure time inevitably becomes long because an amount of light is insufficient, an image photographed by the digital still camera may be blurred because of hand shake. In order to prevent such a blurred image from being formed, there is a method of obtaining an image without a blur by, so to speak, superimposing plural dark images continuously photographed with an exposure time short enough for preventing the image from being affected by hand shake (see, for example, JP-A-2005-38396).
In the method disclosed in JP-A-2005-38396, plural times of photographing are temporally continuously performed by a digital still camera to obtain temporally continuous plural photographed images as plural input images. With one of the plural photographed images set as a reference image, overall movements of the respective plural photographed images with respect to the reference image are calculated. Positioning of the plural photographed images is performed on the basis of the movements. One image (an output, image) is obtained by superimposing (combining) the plural photographed images after the positioning.
FIG. 1 snows a method of obtaining an output image in the case in which there are two photographed images.
In FIG. 1, two photographed images P 1 and P 2 are photographed images continuously photographed by a digital still camera. In the photographed images P 1 and P 2 , positions of subjects deviate from each other because of hand shako or the like.
When, for example, the photographed image P 1 of the two photographed images P 1 and P 2 is set as a reference image, movements of the respective photographed images P 1 and P 2 with respect to the reference image are calculated. Positioning of the photographed images P 1 and P 2 is performed on the basis of the movements to superimpose the subjects in the two photographed images P 1 and P 2 . An output image P out is obtained by super imposing the photographed images P 1 and P 2 after the positioning.
In this case, the plural photographed images are photographed with a short exposure time. However, the plural photographed images may foe photographed with proper exposure. When the plural photographed images are photographed with the proper exposure and superimposed as described above, it is possible to obtain an output image with a high S/N (signal to Noise ratio).
When the positioning of the plural photographed images is performed and the plural photographed images after the positioning are superimposed to generate one output image as described above, positions of pixels of the plural photographed images after the positioning do not always coincide with positions of pixels of the output image.
Therefore, if a pixel for which a pixel value is calculated among the pixels of the output image is referred to as pixel of interest, superimposition of the plural photographed images after the positioning is performed by interpolating the pixel value of the pixel of interest using, among the pixels of the plural photographed images (hereinafter also referred to as photographed pixels as appropriate) after the positioning, pixel values of photographed pixels in positions near the position of the pixel of interest.
Examples of a method of the interpolation of the pixel value of the pixel of interest include a method of performing a simple addition for directly adding up pixel values of one or more photographed pixels in positions near the position of the pixel of interest and a method of performing interpolation using pixel values of one or more photographed pixels in positions near the position of the pixel of interest and an interpolation function.
The interpolation function is a function that changes according to a relative position of the photographed pixel used for the interpolation with respect to the pixel of interest of (a distance between the pixel used for the interpolation and the pixel of interest). For example, a linear function represented by a primary expression, a cubic function, and the like are used. The simple addition is equivalent to using a function with a value of 1 as the interpolation function regardless of the (relative) position of the photographed pixel used for the interpolation.
When an output image is calculated by interpolation performed by using photographed images, granular noise called zipper noise and false colors may appear in an output image unless high-frequency components of the photographed images used for the interpolation are controlled to some extent. As a result, the output, image may be an unnatural image.
In particular, when, for example, a single plate sensor having a color array such as the Bayer array is adopted as an imaging device of a digital still camera used for the photographing of the photographed images, the photographed images are images in which respective pixels have, as a pixel value, only one color signal (color component) among an R (Red) signal, a G (Green) signal, and a B (Blue) signal. In interpolation performed by using such photographed images, zipper noise and false colors may appear conspicuously.
On the other hand, if the high-frequency components of the photographed images used for the interpolation are controlled excessively, edges are blurred and an image quality of an output image is deteriorated.
Therefore, it is desirable to make it possible to obtain a high-quality image according to interpolation.
According to an embodiment of the present invention, there is provided an image processing apparatus that generates an output image according to interpolation performed by using an input image, the image processing apparatus including pixel-value calculating means for calculating a pixel value of a pixel of the output image according to interpolation performed by using pixel values of pixels of the input image and an interpolation function, edge determining means for determining an edge direction, which is a direction of an edge in the pixel of the output image, using the input image, and adjusting means for adjusting the interpolation function such that a degree of pixels of the input image present in a direction along the edge direction contributing to the interpolation is large and a degree of pixels of the input, image present, in a direction orthogonal to the edge direction contributing to the interpolation is small.
According to another embodiment of the invention, there is provided an image processing method of generating an output image according to interpolation performed by using an input image or a computer program for causing a computer to execute image processing for generating an output image according to interpolation performed by using an input image, the image processing method or the computer program including a pixel-value calculating step of calculating a pixel value of a pixel of the output image according to interpolation performed by using pixel values of pixels of the input image and an interpolation function, an edge determining step of determining an edge direction, which is a direction of an edge in the pixel of the output image, using the input, image, and an adjusting step of adjusting the interpolation function such that a degree of pixels of the input image present in a direction along the edge direction contributing to the interpolation is large and a degree of pixels of the input image present in a direction orthogonal to the edge direction contributing to the interpolation is small.
It is possible to record the computer program in various recording media. It is possible to transmit the computer program via various transmission media.
According to the embodiments of the present invention, a pixel value of a pixel of the output image is calculated by the interpolation performed by using pixel values of pixels of an input image and the interpolation function. In calculating the pixel value, an edge direction, which is a direction of an edge in the pixel of the output image, is determined using the input image. The interpolation function is adjusted such that a degree of pixels of the input image present in a direction along the edge direction contributing to the interpolation is large and a degree of pixels of the input image present in a direction orthogonal to the edge direction contributing to the interpolation is small.
According to the embodiments of the present invention, it is possible to obtain a high-quality image according to the interpolation.
FIG. 1 is a diagram for explaining a method of obtaining an output image in the case in which there are two photographed images;
FIG. 2 is a block diagram showing an example of a structure of a digital still camera according to a first embodiment of the present invention;
FIG. 3 is a flowchart for explaining photographing processing of a digital still camera 1 in FIG. 2;
FIG. 4 is a diagram showing an array of pixels of an imaging device 4 in FIG. 2;
FIG. 5 is a block diagram showing an example of a detailed structure of a signal processing circuit 7 in FIG. 2;
FIG. 6 is a diagram showing a reference coordinate system in which positions of pixels are plotted;
FIG. 7 is a diagram for explaining an interpolation method of interpolating a G signal Lg(I′,J′) in a position (I′,J′);
FIG. 8 is a diagram showing eight G pixels present in a 4×4 contributing area of a photographed image of the Bayer array;
FIG. 9 is a waveform chart showing a linear function Linear(z);
FIG. 10 is a waveform chart showing a cubic function Cubic(z);
FIG. 11 is a flowchart for explaining image generation processing;
FIG. 12 is a diagram showing a reference image having an edge in the vertical direction;
FIG. 13 is a diagram showing a cubic function Cubic(p/scaleP) with a contributing parameter scaleP adjusted to a small value;
FIG. 14 is a diagram showing a cubic function Cubic(p/scaleP) with a contributing parameter scaleP adjusted to a large value;
FIG. 15 is a diagram showing a reference image having an edge in a right oblique direction;
FIG. 16 is a diagram showing a rotated xy coordinate system;
FIG. 17 is a flowchart for explaining image generation processing;
FIG. 18 is a diagram showing a contributing area with a variable size;
FIG. 19 is a diagram showing a contributing area with a variable size;
FIG. 20 is a flowchart for explaining processing of edge determination;
FIG. 21 is a flowchart for explaining image generation processing;
FIG. 22 is a diagram for explaining a blend ratio;
FIG. 23 is a flowchart for explaining the image generation processing;
FIG. 24 is a flowchart for explaining the processing for edge determination;
FIG. 25 is a flowchart for explaining processing in steps S 207 to S 210 ; and
FIG. 26 is a block diagram showing an example of a structure of a computer according to an embodiment of the present invention.
Embodiments of the present invention will be hereinafter explained. A correspondence relation between elements of the present invention and the embodiments described or shown in the specification or the drawings is described as follows. This description is a description for confirming that the embodiments supporting the present invention are described or shown in the specification or the drawings. Therefore, even if there is an embodiment that is described or shown in the specification or the drawings but is not described herein as an embodiment corresponding to an element of the present invention, this does not means that the embodiment does not correspond to the element. Conversely, even if an embodiment is described herein as an embodiment corresponding to an element of the present invention, this does not means that the embodiment does not correspond to elements other than the element.
An image processing apparatus according to an embodiment of the present invention is an image processing apparatus (e.g., a digital still camera 1 in FIG. 2) that generates an output image according to interpolation performed by using an input image. The image processing apparatus includes pixel-value calculating means (e.g., an arithmetic circuit 24 in FIG. 5 that executes processing in steps S 106 , S 108 , and 8110 in FIG. 17 and processing in steps S 202 to S 206 and step S 210 in FIG. 23) for calculating a pixel value of a pixel of the output image according to interpolation performed by using pixel values of pixels of the input image and an interpolation function, edge determining means (e.g., the arithmetic circuit 24 in FIG. 5 that executes processing in step S 102 in FIG. 17 and processing in step S 207 in FIG. 23) for determining an edge direction, which is a direction of an edge in the pixel of the output image, using the input image, and adjusting means (e.g., the arithmetic circuit 24 in FIG. 5 that executes processing in step S 104 in FIG. 17 and processing in steps S 202 to S 206 in FIG. 23) for adjusting the interpolation function such that a degree of pixels of the input image present in a direction along the edge direction contributing to the interpolation is large and a degree of pixels of the input, image present in a direction orthogonal to the edge direction contributing to the interpolation is small.
In the image processing apparatus, it is possible to provide detecting means (e.g., a signal processing circuit 7 in FIG. 2 that executes processing in step S 3 in FIG. 3) for detecting a positional relation among plural input images continuously photographed by an imaging unit which takes an image.
In this case, it is possible to cause the adjusting means to adjust the interpolation function such that, among pixels of the plural input images after positioning obtained by performing positioning of the plural input images on the basis of the positional relation, a degree of pixels present in a direction along the edge direction contributing to the interpolation is large and a degree of pixels present in a direction orthogonal to the edge direction contributing to the interpolation is small. It is possible to cause the pixel-value calculating means to calculate a pixel value of a pixel of the output image according to interpolation performed by using pixel values of pixels of the plural input images and the interpolation function.
An image processing method or a computer program according to another embodiment of the invention is an image processing method of generating an output image according to interpolation performed by using an input image or a computer program for causing a computer to execute image processing for generating an output image according to interpolation performed by using an input image. The image processing method or the computer program includes a pixel-value calculating step (e.g., processing in steps S 106 , S 108 , and S 110 in FIG. 17 and processing in steps S 202 to S 206 and step S 210 in FIG. 23) of calculating a pixel value of a pixel of the output image according to interpolation performed by using pixel values of pixels of the input image and an interpolation function, an edge determining step (e.g., processing in step S 102 in FIG. 17 and processing in step S 207 in FIG. 23) of determining an edge direction, which is a direction of an edge in the pixel of the output image, using the input image, and an adjusting step (e.g., processing in step S 104 in FIG. 17 and processing in steps S 202 to S 206 in FIG. 23) of adjusting the interpolation function such that a degree of pixels of the input, image present in a direction along the edge direction contributing to the interpolation is large and a degree of pixels of the input image present in a direction orthogonal to the edge direction contributing to the interpolation is small.
Embodiments of the present invention will be hereinafter explained with reference to the drawings.
FIG. 2 is a block diagram showing an example of a structure of a digital still camera according to an embodiment of the present invention.
A digital still camera 1 in FIG. 2 includes a lens 2 , a stop 3 , an imaging device 4 , a correlated double sampling circuit 5 , an A/D (Analog/Digital) converter 6 , a signal processing circuit 7 , a timing generator 8 , a D/A (Digital/Analog) converter 9 , a video encoder 10 , a monitor 11 , a CODEC 12 , a memory 13 , a bus 14 , a CPU (Central Processing Unit) 15 , and an input device 16 . The signal processing circuit 7 has a frame memory 22 .
Light from a not-shown subject passes through an optical system of the lens 2 , the stop 3 , and the like and is made incident, on the imaging device 4 . The imaging device 4 is constituted by a single plate sensor formed by a CCD (Charge Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor), or the like and has a predetermined number of pixels (light receiving elements).
The imaging device 4 receives the light of the subject made incident, thereon at a predetermined interval and for a predetermined time (shutter time) in accordance with an exposure timing signal supplied from the timing generator 8 . Moreover, the imaging device 4 converts a light reception amount of the light received by the respective light receiving elements serving as the pixels into an image signal as an electric signal according to photoelectric conversion and supplies the image signal to the correlated double sampling circuit 5 . The imaging device 4 is, for example, the single plate sensor. The image signal supplied from the imaging device 4 to the correlated double sampling circuit 5 is a color signal (data) of any one of an R signal, a G signal, and a B signal for one pixel.
Even if camera shake (hand shake) occurs, in order to output a clearer image, the imaging device 4 performs, by performing photographing once (operating a release button once), imaging K times at speed higher than a shutter speed in proper exposure (in a shutter time shorter than a shutter time (exposure time) in proper exposure), Consequently, the imaging device 4 outputs image signals of R photographed images in time series as input images to be inputs to the signal processing circuit 7 at the post stage.
The correlated double sampling circuit 5 removes noise components of the image signals of the photographed images supplied from the imaging device 4 according to correlated double sampling and supplies the image signals to the A/D converter 6 .
A/D converter 6 subjects the image signals supplied from the correlated double sampling circuit 5 to A/D conversion, i.e., sampling and quantizes the image signals.
The A/D converter 6 subjects (the digital image signals of) the photographed images after the A/D conversion to, for example, bit shift to increase a gain of the photographed images to have photographed images of proper exposure and supplies the photographed images to the signal processing circuit 7 .
The signal processing circuit 7 is constituted by, for example, a DSP (Digital Signal Processor). The Signal processing circuit 7 temporarily stores the photographed images supplied from the A/D converter 6 in the frame memory 22 built therein and applies predetermined image processing to the photographed images.
As described above, the imaging device 4 outputs the N photographed images in time series in one photographing. Thus, the M photographed images are sequentially supplied to the signal processing circuit 7 from, the imaging device 4 through the sampling circuit 5 and the A/D converter 6 .
The signal processing circuit 7 supplies the N photographed images supplied thereto to the frame memory 22 built therein and causes the frame memory 2 to temporarily store the N photographed images. Moreover, the signal processing circuit 7 applies predetermined image processing to the N photographed images stored in the frame memory 22 .
The signal processing circuit 7 sets, for example, a first photographed image among the N photographed images as a reference image and sets second to Nth photographed images as target images, respectively. The signal processing circuit 7 detects what kind of positional deviation the targets images cause with respect to the reference image, respectively, i.e., a positional relation between the reference image and the target images (a positional relation of an identical subject in the reference image and the target images).
The signal processing circuit 7 obtains an output image having all a G signal, an R signal, and a B signal for one pixel, which is one clear image with camera shake corrected, on the basis of the positional relation between the reference image and the target images. The signal processing circuit 71 supplies the output image to one or both, of the D/A converter 9 and the CODEC 12 .
The timing generator 8 supplies an exposure timing signal to the imaging device 4 , the correlated double sampling circuit 5 , the A/D converter 6 , and the signal processing circuit 7 such that high-speed imaging for the N photographed images is performed at predetermined intervals in one photographing. A user can change an exposure time of the high-speed imaging (or the number N of photographed images imaged by the high-speed imaging) according to, for example, brightness of a subject. When the user changes the exposure time of the high-speed imaging, the user operates the input device 16 to supply a changed value of the exposure time determined by the CPU 15 from the CPU 15 to the timing generator 8 through the bus 14 .
The D/A converter 9 subjects an image signal of an output image supplied from the signal processing circuit 7 to D/A conversion and supplies the image signal to the video encoder 10 .
The video encoder 10 converts the image signal (the analog signal) supplied from the D/A converter 9 into an image signal that can be displayed on the monitor 11 and supplies the image signal to the monitor 11 . The monitor 11 plays a role of a finder or the like of the digital still camera 1 . The monitor 11 is constituted by an LCD or the like and displays an image signal supplied from the video encoder 10 . Consequently, the output image is displayed on the monitor 11 .
The CODEC 12 encodes the image signal of the output image supplied from the signal processing circuit 7 in accordance with a predetermined system such as the JPEG (Joint Photographic Experts Group) system and supplies the image signal to the memory 13 .
The memory 13 is constituted by a semiconductor memory such as a flash memory and stores (records) the encoded image signal supplied from the CODEC 12 . It is possible to use a recording medium such as a magnetic disk or an optical (magneto-optical) disk instead of the memory 13 . The memory 13 or the recording medium, used instead of the memory 13 is detachably insertable in the digital still camera 1 . It is possible to provide both the recording medium built in the digital still camera 1 and the recording medium detachably insertable in the digital still camera 1 .
The CPU 15 supplies control signals to the respective units through the bus 14 and controls various kinds of processing. For example, the CPU 15 supplies the control signals to the respective units such that the subject is photographed in accordance with an operation signal for starting photographing supplied from the input device 16 according to operation of the user and an output image finally obtained by the photographing is stored in the memory 13 .
The input device 16 has operation buttons such as a release button provided in a main body of the digital still camera 1 . Various operation signals generated by the operation of the operation buttons by the user are supplied from the input device 16 to the CPU 15 through the bus 14 . The CPU 15 controls the respective units to execute processing conforming to the various operation signals supplied from the input device 16 through the bus 14 . It is possible to display one or more operation buttons of the input device 16 on the monitor 11 . For example, a transparent tablet is provided on the monitor 11 . It is possible to detect, the operation of the operation buttons displayed on the monitor 11 using the tablet.
Photographing processing of the digital still camera I will be explained with reference to a flowchart in FIG. 3.
First, in step S 1 , the imaging device 4 photographs a subject. In photographing per formed by depressing the release button (a shutter button) once, the imaging device 4 receives light of the subject continuously made incident thereon N times at predetermined intervals in accordance with an exposure timing signal supplied from the timing generator 8 and photoelectrically converts the light to perform high-speed imaging N times. Therefore, H photographed images are obtained in one photographing and the respective photographed images are dark images with exposure equal to or lower than (or lower than) proper exposure. Image signals of the N photographed images obtained by the photoelectric conversion in the imaging device 4 are sequentially supplied to the correlated double sampling circuit 5 and, after noise components are removed, supplied to the A/D converter 6 .
Thereafter, the processing proceeds from step S 1 to step S 2 . The A/D converter 6 subjects image signals of the N photographed images sequentially supplied from the correlated double sampling circuit 5 to A/D conversion. Thereafter, the A/D converter 6 subjects the dark photographed images with exposure equal to or lower than the proper exposure to bit shift to convert the dark photographed images into image signals with brightness of the proper exposure and supplies the image signals to the signal processing circuit 7 . The processing proceeds to step S 3 .
In step S 3 , the signal processing circuit 7 sets, for example, a first photographed image among the N photographed images from the A/D converter 6 as a reference image and detects what kind of positional deviation the N photographed images cause with respect to the reference image, respectively, i.e., a positional relation of the N photographed images to the reference image. The processing proceeds to step S 4 .
In step S 4 , the signal processing circuit 7 performs image generation processing for generating one output image from the N photographed images on the basis of the N photographed images and the positional relation of the N photographed images detected in step S 3 . The processing proceeds to step S 5 .
Although details of the image generation processing will be described later, an output image having all of a G signal, an R signal, and a B signal for one pixel, which is one clear image without camera shake (with little camera shake) and with the proper exposure, is generated by this image generation processing. An image signal of the output image obtained by the image generation processing is supplied from the signal processing circuit 7 to one or both of the D/A converter 9 and the CODEC 12 .
In step S 5 , the output image obtained by the signal processing circuit 7 is displayed on the monitor 11 and recorded in the memory 13 such as a flash memory. Then, the processing is finished.
In step S 5 , the image signal supplied from the signal processing circuit 7 to the D/A converter 9 in step S 4 is converted into an analog signal and supplied to the video encoder 10 . Moreover, in step S 5 , the video encoder 10 converts the analog image signal supplied from the D/A converter 9 into an image signal that can be displayed on the monitor 11 and supplies the image signal to the monitor 11 . In step S 5 , the monitor 11 displays an output image on the basis of the image signal supplied from the video encoder 10 . In step S 5 , predetermined encoding of JPEG or the like is applied to the image signal supplied from the signal processing circuit 7 to the CODEC 12 in step S 4 and the image signal is recorded in the memory 13 such as a flash memory.
FIG. 4 shows an array of pixels of the imaging device 4 in FIG. 2.
In FIG. 4, pixels in a portion at the upper left of the imaging device 4 (six pixels in the horizontal direction and the four pixels in the vertical direction; twenty-four pixels in total) are shown. Pixels in other portions are arranged in the same way.
In FIG. 4, with the center (the center of gravity) of the pixels at the upper left of the imaging device 4 as an origin, an xy coordinate system with the horizontal (right) direction set as an x direction and the vertical (down) direction set as a y direction is set. It is assumed that the pixels have a rectangular shape and the lengths (the widths) in the horizontal and vertical direction of one pixel is 1, respectively.
When a position of a pixel is represented by a coordinate of the center of gravity of the rectangle as the pixel having the lengths in the horizontal and vertical direction of 1, (a coordinate of) a position (x,y) of a pixel ith from the left and jth from the top can be represented as (i−1,j−1).
In FIG. 4, an array of the pixels of the imaging device 4 is a so-called Bayer array. The array of the pixels of the imaging device 4 is not 1 limited to the Bayer array and may be other arrays.
An image having pixel values of color signals corresponding to positions of pixels is outputted from the imaging device 4 of the Bayer array.
In the Bayer array, as pixels from which the G signal can be extracted, a pixel GOO that is a pixel first in the x direction and first in the y direction from the origin, a pixel G 02 that is a pixel third in the x direction and first in the y direction from the origin, a pixel G 04 that is a pixel fifth in the x direction and first in the y direction from the origin, and a pixel G 11 that is a pixel second in the x direction and second in the y direction from the origin are arranged. In the same manner, a pixel G 13 , a pixel G 15 , a pixel G 20 , a pixel G 22 , a pixel G 24 , a pixel G 31 , a pixel G 33 , and a pixel G 35 are arranged.
As pixels from which the R signal can be extracted, a pixel R 01 that is a pixel second in the x direction and first in the y direction from the origin, a pixel R 03 that is a pixel fourth in the x direction and first in the y direction from the origin, a pixel R 05 that is a pixel sixth in the x direction and first in the y direction from the origin, and a pixel R 21 that is a pixel second in the x direction and third in the y direction from the origin are arranged. In the same manner, a pixel R 23 and a pixel R 25 are arranged.
As pixels from, which the B signal can be extracted, a pixel B 10 that is a pixel first in the x direction and second in the y direction from the origin, a pixel B 12 that is a pixel third in the x direction and second in the y direction from the origin, a pixel B 14 that is a pixel fifth in the x direction and second in the y direction from the origin, and a pixel B 30 that is a pixel first in the x direction and fourth in the y direction from the origin are arranged. In the same manner, a pixel B 32 and a pixel B 34 are arranged.
It is assumed here that the imaging device A is an imaging device in which the G signal, the R signal, and the B signal are obtained in respective pixels. An ideal image photographed without camera shake and with the proper exposure using such an imaging device 4 is assumed. The G signal, the R signal, and the B signal of the ideal image are represented as Lg(x,y), Lr(x,y), and Lb(x,y), respectively, using a position (x,y) on an xy coordinate system with the imaging device 4 set as a reference.
This ideal image is an output image that is desired to be obtained in the image generation processing in step S 4 in FIG. 3. The G signal, the R signal, and the B signal of an “i+1th and j+1th pixel” i+1th from the left and j+1th from the top of the output image can be represented as Lg(i,j), Lr(i,j), Lb(i,j), respectively.
When the “i+1th and j+1th pixel” i+1th from the left and j+1th from the top of the imaging device 4 is a pixel that outputs the G signal, i and j representing a position of the pixel are also described as ig and jg, respectively. Similarly, when the “i+1th and j+1th pixel” i+1th from the left and j+1th from the top of the imaging device 4 is a pixel that outputs the R signal, i and j representing a position of the pixel are also described as ir and j r, respectively. When the “i+1th and j+1th pixel” i+1th from the left and j+1th from the top of the imaging device 4 is a pixel that outputs the B signal, i and j representing a position of the pixel are also described, as in and jb, respectively.
A combination of the variables ig and jg is equal to a combination of the variables i and j representing the position of the pixel that outputs the G signal. A combination of the variables ir and jr is equal to a combination of the variables i and j representing the position of the pixel that outputs the R signal. A combination of the variables ib and jb is equal to a combination of the variables i and j representing the position of the pixel that outputs the B signal.
When the imaging device 4 is an imaging device of the Bayer array as described above, the variables ig and jg are the variables i and j that satisfy a condition that a difference (i−j) between the variables i and j is an even number. The variables ir and jr are the variables i and j that satisfy a condition that the variable i is an even number and a difference (i−j) between the variables i and j is an odd number. Moreover, the variables ib and jb are the variables i and j that satisfy a condition that the variable i is an odd number and a difference (i−j) between the variables i and j is an even number.
However, when the imaging device 4 is a single plate sensor of an array other than the Bayer array, conditions of the variables i and j forming the variables ig and jg, the variables ir and jr, and the variables ib and jb are different according to characteristics of the array.
When an “if 1th and j+1th pixel” i+1th from the left and j+1th from the top of a kth (k=1, 2, . . . , N) photographed image among the N photographed images outputted by the imaging device 4 in one photographing is a pixel having only the G signal as a pixel value, the G signal as the pixel value is represented as Gobs(k,i,j).
Similarly, when the “i+1th and j+1th pixel” i+1th from the left and j+1th from the top of the kth photographed image is a pixel having only the R signal as a pixel value, the R signal as the pixel value is represented as Robs(k,i,j). When the “i+1th and j+1th pixel” i+1th from the left and j+1th from the top of the kth photographed image is a pixel having only the B signal as a pixel value, the S signal as the pixel value is represented as Bobs(k,i,j).
The pixel values Gobs(k,i,j), Robs(k,i,j), and Bobs(k,i,j) can also be represented as Gobs(k,ig,jg), Robs(k,ir,jr), and Bobs(k,ib,jb).
FIG. 5 shows an example of a detailed structure of a part of the signal processing circuit 7 in FIG. 2.
The signal processing circuit 7 includes the frame memory 22 , a motion detecting circuit 23 , an arithmetic circuit 24 , and a controller 25 . The frame memory 22 includes N frame memories 22 1 to 22 N . The motion detecting circuit 23 includes N−1 motion detecting circuits 23 1 to 23 N-1 ).
As described above, the N photographed images are sequentially supplied from the A/D converter 6 to the frame memory 22 . The frame memory 22 1 temporarily stores a first photographed image supplied from the A/D converter 6 . The frame memory 22 ; stores a second photographed image supplied from the A/D converter 6 . In the same manner, the frame memory 22 k stores a kth photographed image supplied from the A/D converter 6 .
The frame memory 22 1 supplies the first photographed image stored therein to the arithmetic circuit 24 and the motion detecting circuits 23 1 to 23 N-1 at predetermined timing. The frame memory 22 2 supplies the second photographed image stored therein to the arithmetic circuit 24 and the motion detecting circuit 23 1 at predetermined timing. In the same manner, the frame memory 22 k supplies the kth photographed image stored therein to the arithmetic circuit 24 and the motion detecting circuits 23 k-1 at predetermined timing.
The motion detecting circuit 23 detects a positional relation between two photographed images. The motion detecting circuit 23 sets the first photographed image as a reference image serving as a reference for detection of the positional relation and sets the second to Nth photographed images as target images. The motion detecting circuit 23 detects deviation amounts of positional deviation of the target images with respect to the reference image indicating what kind of positional deviation the targets images (the second to Nth images) cause with respect, to the reference image. The deviation is caused by hand shake.
The motion detecting circuit 23 detects a positional relation between the reference image and the target images on the basis of the deviation amounts of the positional deviation of the target images with respect, to the reference image.
In a state in which the camera is aimed at the subject, as components of the positional deviation of the images caused by hand shake, there are a translation component generated when the camera deviates to the left and right and a rotation component around an optical axis of a lens generated when the camera rotates in the clockwise direction or the counterclockwise direction. There are also a rotation component around an axis perpendicular to the optical axis of the lens of the camera and an expansion and reduction component due to the movement in the depth direction of the camera.
The positional relation between the reference image and the target image in which hand shake occurs can be represented by, for example, affine transformation. In the affine transformation, a positional relation between a position (x,y) on the reference image and a position (x′,y′) on the target images is represented by the following Equation (1).
For example, when a=K×cos θ, b=−K×sin θθ, c=K×sin θ, and d=K×cos θ in Equation (1), Equation (1) represents the affine transformation for applying rotation at an angle θ, translation of (s,t), and expansion and reduction of K times with respect to the position (x′,y′).
A matrix (a,b,c,d) and a two-dimensional vector (s,t) of the affine transformation are collectively referred to as transformation parameters (a,b,c,d,s,t) as appropriate.
The affine transformation of Equation (1) defined by the transformation parameters represents a positional relation between the reference image and the target image. The motion detecting circuit 23 calculates the transformation parameters defining Equation (1), for example, as described below.
The motion detecting circuit 23 divides the target images into plural blocks and detects motion vectors of the respective blocks with respect, to the reference image as deviation amounts of positional deviation of the target images with respect to the reference image.
The motion detecting circuit 23 calculates, as the positional relation, the transformation parameters (a,b,c,d,s,t) of Equation (1) for minimizing a sum of square errors between a position (x″,y″) after moving (positions) of the respective pixels (x′,y′) of the target images onto the reference image in accordance with the motion vectors of the respective blocks of the target images and positions (x,y) after converting the respective pixels x′,y′) of the target images into positions (x,y) on the reference image according to Equation (1).
Specifically, the first photographed image as the reference image is supplied to the motion detecting circuit 23 1 from the frame memory 22 1 . The second photographed image as the target image is supplied to the motion detecting circuit 23 1 from the frame memory 22 2 .
The motion detecting circuit 23 1 detects motion vectors indicating which positions of the first photographed image the respective blocks obtained by dividing the second photographed image into plural blocks correspond to. In other words, the motion detecting circuit 23 1 detects a position on the first photographed image in which a portion identical with a portion of the subject projected in a certain position of the second photographed image is projected. The motion detecting circuit 23 1 calculates, on the basis of the motion vectors as a result of the detection, transformation parameters (a 2 ,b 2 ,c 2 ,d 2 ,s 2 ,t 2 ) defining Equation (2) identical with Equation (1) representing a positional relation between the first photographed image and the second photographed image and supplies the transformation parameters to the arithmetic circuit 24 .
As in the case of the imaging device 4 in FIG. 4, as a coordinate system of an image, with the center of pixels at the upper left of the image set as an origin, an xy coordinate system with the horizontal direction (the right direction) set as an x direction and the vertical direction (the down direction) set as a y direction is defined. Then, in Equation (2), (x 2 ,y 2 ) represents a position of a pixel of the second photographed image on the coordinate system of the second photographed image and (x 1(2) ,y 1(2) ) represents the position at the time when the position (x 2 ,y 2 ) of the pixel of the second photographed image is converted into a position where an identical portion of the subject is projected on the coordinate system of the first photographed image. The subscript (2) in the position (x 1(2) , y 1(2) ) indicates that the position (x 2 ,y 2 ) on the coordinate system of the second photographed image is converted into a position on the coordinate system of the first photographed image. A portion identical with the portion of the subject projected in the position (x 2 ,y 2 ) of the pixel of the second photographed image is (ideally) projected in the position (x 1(2) ,y 1(2) ) on the coordinate system of the first photographed image.
The first photographed image as the reference image is supplied to the motion detecting circuit 23 2 from the frame memory 22 1 . The third photographed image as the target image is supplied to the motion defecting circuit 23 2 from the frame memory 22 3 .
Like the motion detecting circuit 23 1 , the motion detecting circuit 23 2 detects motion vectors indicating which positions of the first photographed image respective blocks obtained by dividing the third photographed image into plural blocks correspond to. The motion detecting circuit 23 2 calculates, on the basis of the motion vectors, transformation parameters (a 3 ,b 3 ,c 3 ,d 3 ,s 3 ,t 3 ) defining the affine transformation of Equation (3) identical with Equation (1) representing a positional relation between the first photographed image and the third photographed image and supplies the transformation parameters to the arithmetic circuit 24 .
In Equation (3), (x 3 ,y 3 ) represents a position of a pixel of the third photographed image on the coordinate system of the third photographed image and (x 1(3) ,y 1(3) ) represents the position at the time when the position (x 3 ,y 3 ) of the pixel of the third photographed image is converted into a position where an identical portion of the subject is projected on the coordinate system of the first photographed image. As in the case of the Equation (2), the subscript (3) in the position (x 1(3) ,y 1(3) ) indicates that the position (x 3 ,y 3 ) on the coordinate system of the third photographed image is converted into a position on the coordinate system of the first photographed image.
In the same manner, the motion detecting circuit 23 k-1 detects a positional relation between the first photographed image and the kth photographed image and supplies the positional relation to the arithmetic circuit 24 .
The first photographed image as the reference image is supplied to the motion detecting circuit 23 k-1 from the frame memory 22 1 . The kth photographed image as the target image is supplied to the motion detecting circuit 23 k-1 from the frame memory 22 k .
The motion detecting circuit 23 k-1 detects motion vectors of respective blocks of the kth photographed image with respect to the first photographed image. The motion detecting circuit 23 k-1 calculates, on the basis of the motion vectors, transformation parameters (a k ,b k ,c x ,d x ,s x ,t k ) defining the affine transformation of Equation (4) identical with Equation (1) representing a positional relation between the first photographed image and the kth photographed image and supplies the transformation parameters to the arithmetic circuit 24 .
In Equation (4), (x k ,y k ) represents a position of a pixel of the kth photographed image on the coordinate system of the kth photographed image and (x 1(k) ,y 1(k) ) represents the position at the time when the position (x k ,y k ) of the pixel of the kth photographed image is converted into a position where an identical portion of the subject is projected on the coordinate system of the first photographed image. As in the case of the Equation (2), the subscript (k) in the position (x 1(k) ,y 1(k) ) indicates that the position (x k ,y k ) on the coordinate system of the kth photographed image is converted into a position on the coordinate system of the first photographed image.
The N photographed images are supplied to the arithmetic circuit 24 from the frame memories 22 1 to 22 N . The transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) representing the positional relation between the first photographed image and the kth photographed image are supplied to the arithmetic circuit 24 from the motion detecting circuits 23 1 to 23 N-1 .
The arithmetic circuit 24 calculates the G signal, the R signal, and the B signal as the pixel values of the pixels of an output image using at least the pixel values of the pixels of the photographed images supplied from the frame memories 22 1 to 22 N and the interpolation function that changes according to positions of the pixels of the photographed images after positioning. The positions of the pixels are obtained by performing positioning of the N photographed images on the basis of the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) representing the positional relation between each of the second to Nth photographed images supplied from the motion detecting circuits 23 1 to 23 N-1 and the first photographed image. The arithmetic circuit 24 performs image generation processing for generating an output image and supplies the output image obtained as a result of the image generation processing to the D/A converter 9 or the CODEC 12 .
Each of the N photographed images supplied from the A/D converter 6 to the signal processing circuit 7 is an image, one pixel of which has a pixel value of any one of the G signal, the R signal, and the B signal. On the other hand, the output image generated by the arithmetic circuit 24 is an image having three pixel values (color signals) of the G signal, the R signal, and the B signal for one pixel.
The controller 25 performs control of the frame memories 22 1 to 22 N , the motion detecting circuits 23 1 to 23 N-1 , the arithmetic circuit 24 , and the like in the signal processing circuit 7 in accordance with the control by the CPU 15 .
In the signal processing circuit 7 constituted as described above, in step S 3 in FIG. 3, the motion detecting circuit 23 detects transformation parameters as a positional relation among the N photographed images from the A/D converter 6 .
Moreover, in the signal processing circuit 7 , in step S 4 in FIG. 3, the arithmetic circuit 24 calculates the G signal, the R signal, and the B signal as pixel values of pixels of an output image using pixel values of the pixels of the photographed images, the interpolation function that changes according to positions of the pixels of the photographed images after positioning obtained by performing positioning of the N photographed images on the basis of the transformation parameters, and the like. The arithmetic circuit 24 performs the image generation processing for generating an output image.
In other words, in the image generation processing in step S 4 , the arithmetic circuit 24 generates, from the N photographed images, a photographed image serving as a reference in detecting the positional relation among the N photographed images, i.e., an image in a range of the subject photographed in the first photographed image as an output image.
In generating the output image from the N photographed images, the arithmetic circuit 24 calculates pixel values of respective pixels of the output image by interpolation.
In order to calculate the pixel values of the output image by interpolation in this way, the arithmetic circuit 24 performs positioning for converting (positions of) the pixels of the N photographed images into positions on the output image, i.e., positions on the first photographed image as the reference image such that respective portions of the subject projected on the respective N photographed images coincide with (correspond to) one another.
In the following explanation, a kth photographed image among the N photographed images used for generation of the output image is also referred to as a kth image as appropriate.
In the arithmetic circuit 24 , the conversion of the position of the pixels of the M photographed images into the positions on the output image, which are the positions on the first photographed image as the reference image, i.e., the positioning of the N photographed images is performed according to the affine transformation of Equation (1) defined by the transformation parameters calculated by the motion detecting circuit 23 (FIG. 5).
The arithmetic circuit 24 calculates the G signal Lg(i,j) among the pixel values of the pixels in the positions (i,j) on the coordinate system of the output image by interpolation performed by using the G signal Gobs(k,i,j)=Gobs(k,ig,jg) (k=1, 2, . . . , N) among the pixel values of the pixels in the positions after the affine transformation obtained by affine-transforming the positions of the pixels of the N photographed images.
Similarly, the arithmetic circuit 24 calculates the R signal Lr(i,j) among the pixel values of the pixels in the positions (i,j) on the coordinate system of the output image by interpolation performed by using the R signal Robs(k,i,j)=Robs(k,ir,jr) among the pixel values of the pixels in the positions after the affine transformation obtained by affine-transforming the positions of the pixels of the N photographed images. The arithmetic circuit 24 calculates the B signal Lb(i,j) among the pixel values of the pixels in the positions (i,j) on the coordinate system of the output image by interpolation performed by using the B signal Bobs(k,i,j)=Bobs(k,ib,jb).
In the coordinate system of the output image, which is the coordinate system of the reference image, a position (i−1,j−1) of an “ith and jth pixel” of the output image is represented as (I′,J′). In other words, I′=i−1 and J′=j−1, I′ and J′ are integers equal to or larger than 0.
In the following explanation, the coordinate system of the output image, which is the coordinate system of the reference image, is also referred to as a reference coordinate system as appropriate. The pixels of the output image are also referred to as output pixels as appropriate.
The arithmetic circuit 24 affine-transforms (the positions) of the pixels of the first to Nth images into the positions on the reference coordinate system. The arithmetic circuit 24 calculates a G signal Lg(I′,J′) of the output pixel in the position (I′,J′) on the reference coordinate by interpolation performed by using the G signals Gobs(k,ig,jg) in the positions after the affine transformation.
However, accuracy of interpolation is deteriorated if all the G signals Gobs(k,ig,jg) of the pixels in the positions after the affine transformation onto the reference coordinate system of the pixels of the first to Nth images are used for the interpolation of the G signal Lg(I′,J′) of the output pixel in the position (I′,J′) on the reference coordinate system.
Thus, the arithmetic circuit 24 specifies pixels of the first to Nth images, positions of which after the affine transformation onto the reference coordinate system of the pixels of the first to Nth images are near the position (I′,J′) of the output pixel for interpolating the G signal Lg(I′,J′), as pixels used for the interpolation of the G signal Lg(I′,J′). The arithmetic circuit 24 interpolates the G signal Lg(I′,J′) using the G signals Gobs(k,ig,jg) of the pixels of the first to Nth images specified.
Specifically, the arithmetic circuit 24 sets an area near the position (I′,J′) of the reference coordinate system as a contributing area in which pixels contributing to interpolation of a pixel value of the output pixel in the position (I′,J′) are present. The arithmetic circuit 24 specifies pixels of the first to Nth images, positions of which after the affine transformation onto the reference coordinate system are in the contributing area, as pixels used for the interpolation of the pixel value of the output pixel in the position (I′,J′).
FIG. 6 shows a reference coordinate system in which positions of the pixels of the first to Nth images used for the interpolation of the pixel value of the output pixel in the position (I′,J′) by the arithmetic circuit 24 are plotted.
The arithmetic circuit 24 sets, for the position (I′,J′) on the reference coordinate system, an area of a range 2×2 around the position (I′,J′) satisfying, for example, an expression I′−1≦x<I′+1 and an expression J′−1≦y<J′+1 as a contributing area. The arithmetic circuit 24 specifies pixels of the first to Nth images, positions of which after the affine transformation onto the reference coordinate system are in the contributing area, as pixels used for the interpolation of the G signal Lg(I′,J′) of the output pixel.
In other words, the arithmetic circuit 24 calculates, for the position (I′,J′), all sets of integers k, ig, and jg, with which the positions (x,y) on the reference coordinate system obtained by affine-trans forming the position (ig−1,jg−1) with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) satisfy the expression I′−1≦x<I′+1 and the expression J′−1≦y<J′+1. The arithmetic circuit 24 specifies pixels represented by (k,ig,jg) as pixels used for the interpolation of the G signal Lg(I′,J′) of the output pixel.
In FIG. 6, there are five G pixels A, B, C, D, and E as pixels, positions of which after the affine transformation onto the reference coordinate system are in the contributing area in the range of the expression I′−1≦x<I′+1 and the expression J′−1≦y<J′+1, among pixels having the G signals as pixels values (hereinafter also referred, to as G pixels as appropriate) in the pixels of the first to Nth images.
Therefore, the arithmetic circuit 24 specifies the five G pixels A to E as pixels used for the interpolation of the G signal Lg(I′, J′).
The arithmetic circuit 24 interpolates the G signal Lg(I′,J′) of the output pixel in the position (I′,J′) using the pixel values (G signals) Gobs(k,ig,jg) of the respective G pixels A to E.
In FIG. 6, the area of the range of 2×2 around the position (I′,J′) is adopted as the contributing area for the position (I′,J′). However, the contributing area for the position (I′,J′) only has to be an area near the position (I′,J′) and is not limited to the area of the range of 2×2 around the position (I′,J′). In other words, in FIG. 6, as the contributing area for the position (I′,J′), other than the area of the range of 2×2 around the position it is possible to adopt, for example, an area of a range of 4×4 around the position (I′,J′), i.e., an area satisfying an expression I′−2≦x<1′+2 and an expression J′−2≦y<J′+2, and an area of a range of 1×1.
An interpolation method of interpolating the G signal Lg(I′,J′) in the position (I′,J′) using the pixel values (G signals) Gobs(k,ig,jg) of the G pixels, the positions of which after the affine transformation onto the reference coordinate is in the contributing area, among the G pixels of the first to Nth images will be explained with reference to FIG. 7.
In FIG. 7 (and in the FIG. 8 described later), an area of a range of 4×4 is set as the contributing area.
The arithmetic circuit 24 calculates the G signal Lg(I′,J′) of the output pixel in the position (I′,J′) by interpolation indicated by the following equation using the pixel values Gobs (k,ig,jg) of the G pixels in the contributing area for the position (I′,J′) and an interpolation function that changes according to the positions of the G pixels in the contributing area.
Σ in Equation (5) indicates a sum for all the G pixels, the positions of which after the positioning of the N photographed images are in the contributing area. In other words, Σ indicates a sum for sets of (k, ig, jg), with which the positions (x,y) on the reference coordinate system obtained by affine-transforming the positions (ig,jg) of the G pixels of the photographed images with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t x ) satisfy the expression I′−2≦x<I′+2 and the expression J′−2≦y<J′+2.
In Equation (5), w((x,y),(I′,J′)) is an interpolation function having, as arguments, the positions (x,y) on the reference coordinate obtained by affine-transforming the positions (ig,jg) of the G pixels of the photographed images with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) and the position (I′,J′) of the pixel for interpolating the G signal Lg(I′,J′). In this way, the interpolation function w((x,y),(I′,J′)) has, as the argument, the positions (x,y) on the reference coordinate obtained by affine-trans forming the positions (ig,jg) of the G pixels of the photographed images with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ), i.e., the positions (x,y) of the G pixels after the positioning of the photographed images. Thus, the interpolation function w((x,y),(I′,J′)) is a function that, changes according to the positions (x,y) of the G pixels after the positioning of the photographed images.
For example, when a variable “p” is defined by an equation p=x−I′ and a variable “q” is defined by an equation q=y−J′, as shown in FIG. 7, (p,q) indicates a relative position of the G pixels after the positioning of the photographed images with the position (I′,J′) set as a reference.
The arithmetic circuit 24 calculates an R signal Lr(I′,J′) and a B signal Lb(I′,J′) of the output pixel in the position (I′,J′) by interpolation in the same manner as the calculation of the G signal Lg(I′,J′). The arithmetic circuit 24 calculates the R signal Lr(I′,J′) and the B signal Lb(I′,J′) in accordance with Equation (6) and Equation (7) similar to Equation (5).
Σ in Equation (6) indicates a sum for all pixels having only the R signal as pixel values (hereinafter also referred to as R pixels as appropriate), positions of which after the positioning of the N photographed images are in the contributing area. In other words, Σ indicates a sum for sets of (k, ir, jr), with which the positions (x,y) on the reference coordinate system obtained by affine-transforming the positions (ir,jr) of the R pixels of the photographed images with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) satisfy the expression I′−2≦x<I′+2 and the expression J′−2≦y<J′+2.
Σ in Equation (7) indicates a sum for all pixels having only the B signal as pixel values (hereinafter also referred to as B pixels as appropriate), positions of which after the positioning of the N photographed images are in the contributing area. In other words, Σ indicates a sum for sets of (k, ib, jb), with which the positions (x,y) on the reference coordinate system obtained by affine-transforming the positions (ib,jb) of the B pixels of the photographed images with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) satisfy the expression I′−2≦x<I′+2 and the expression J′−2≦y<J′+2.
Since the imaging device 4 of the Bayer array is adopted, if the photographed images after the positioning overlap over the entire contributing area of 4×4 for the position (I′,J′), for example, as indicated by circles in FIG. 8, eight pixels are present as G pixels for one photographed image after the positioning in the contributing area. On the other hand, four pixels are present as R pixels and B pixels, respectively, for one photographed image after the positioning in the contributing area.
The interpolation function w((x,y),(I′,J′)) of Equations (5) to (7) will be explained.
As described above, the variable “p” is defied by the equation p=x−I′, the variable “q” is defined by the equation q=y−J′, and a function f(p,q) having the variables “p” and “q” as arguments is adopted as the interpolation function w((x,y),(I′,J′)).
In this case, it is possible to adopt, for example, a bilinear function and a bicubic function as the interpolation function f(p,q).
A bilinear function (p,q) is a product of two linear functions Linear(z) and represented by, for example, Equation (8).
The linear function Linear (z) of Equation (8) is shown in FIG. 9.
The bicubic function Bicubic(p,q) is a product of two cubic functions Cubic(z) and represented by, for example, Equation (9).
The cubic function Cubic(z) of Equation (9) is shown in FIG. 10.
For example, the bicubic function of Equation (9) is adopted as the interpolation function f(p,q), the G signal, the R signal, or the B signal, which is a pixel value of a pixel in the position (i,j) of the kth image among the N photographed images, is represented as inputPixel(k,i,j), and the G signal, the R signal, or the E signal, which is a pixel value of an output pixel in the position (I′,J′), is represented as outputPixel(I′,J′). Then, Equations (5) to (7) can be represented by Equation (10).
However, in Equation (10), p(k,i,j) and q(k,i,j) are represented by the following equation with a position (a position on the reference coordinate system) after the positioning of the pixel in the position (i,j) of the kth image set as (x,y).
p ( k,i,j )= x−I′
q ( k,i,j )= y−J′
According to Equation (11), (p(k,i,j),q(k,i,j)) represents a coordinate (a relative coordinate) of the pixel in the position (i,j) of the kth image with the position (I′,J′) of the output pixel set as a reference (an origin).
In Equation (10), Σ before parentheses on the right-hand side indicates a sum for the N photographed images.
Moreover, in Equation (10), the G signal among the G signal, the R signal, and the B signal is calculated as the pixel value outputPixel(I′,J′) of the output pixel in the position (I′,J′). In this case, Σ of the denominator and the numerator of the fraction on the right-hand side of Equation (10) indicates a sum for all the G pixels in the contributing area among the pixels of the N photographed images after the positioning, InputPixel(k,i,j) indicates a pixel value of the G pixel in the position (i,j) of the kth image, a position of which after the positioning is a position in the contributing area.
In Equation (10), the R signal is calculated as the pixel value outputPixel(I′,J′) of the output pixel in the position (I′,J′). In this case, Σ of the denominator and the numerator of the fraction on the right-hand side of Equation (10) indicates a sum for all the R pixels in the contributing area among the pixels of the N photographed images after the positioning. InputPixel (k,i,j) indicates a pixel value of the R pixel in the position (i,j) of the kth image, a position of which after the positioning is a position in the contributing area.
Moreover, in Equation (10), the B signal is calculated as the pixel value outputPixel(I′,J′) of the output pixel in the position (I′,J′). In this case, Σ of the denominator and the numerator of the fraction on the right-hand side of Equation (10) indicates a sum for all the B pixels in the contributing area among the pixels of the N photographed images after the positioning. InputPixel(k,i,j) indicates a pixel value of the B pixel in the position (i,j) of the kth image, a position of which after the positioning is a position in the contributing area.
When, for example, the G signal among the G signal, the R signal, and the B signal of the pixel value outputPixel (I′, J′) of the output pixel is calculated, only pixel values of the G pixels among the pixels of the photographed images, i.e., only the G signals are used. However, it is also possible to calculates the G signal of the pixel value outputPixel (I′,J′) of the output pixel using the R signals as the pixel values of the R pixels or the B signals as the pixel values of the B pixels other than the G signals as the pixel values of the G pixels among the pixels of the photographed images. The R signal and the B signal of the pixel value outputPixel (I′,J′) of the output pixel are calculated in the same manner.
The image generation processing in step S 4 in FIG. 3 for generating an output, image by interpolating the G signal, the R signal, and the B signal as the pixel values of the output pixel will be explained with reference to a flowchart in FIG. 11.
First, in step S 71 , the arithmetic circuit 24 selects, with a certain output pixel on the reference coordinate system set as a pixel of interest, a position (I′,J′) of the pixel of interest as a position of interest (I′,J′).
The arithmetic circuit 24 proceeds from step S 71 to step S 72 . The arithmetic circuit 24 calculates sets of (k, ig, jg), with which the positions (x,y) on the reference coordinate system obtained by affine-transforming the positions (ig−1,jg−1) of the G pixel of the kth image (a pixel of the G signal Gobs(k,ig,jg)) with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) satisfy the expression I′−2≦x<I′+2 and the expression J′−2≦y<J′+2 representing the contributing area for the position of interest (I′,J′) for all of the first to Nth images. The arithmetic circuit 24 specifies the G pixels represented by (k,ig,jg) as contributing pixels contributing to interpolation of the pixel of interest and proceeds to step S 73 .
The transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) in affine-transforming the position of the G pixel of the kth image to the position (x,y) on the reference coordinate system is supplied from the motion detecting circuit 23 k-1 to the arithmetic circuit 24 . For the first image as the reference image, i.e., for the case of k=1, (1,0,0,1,0,0) is sued as the transformation parameters (a 1 ,b 1 ,c 1 ,d 1 ,s 1 ,t 1 ). Therefore, the first image is not substantially affine-transformed.
The position (x,y) after the affine transformation of the position of the pixel of the kth image onto the reference coordinate system is also referred to as a transformed position (x,y) as appropriate.
In step S 73 , the arithmetic circuit 24 calculates Equation (5) (Equation (10)) using all the sets of (k,ig,jg) calculated in step S 72 to calculate a G signal Lg(I′,J′) (outputPixel(I′,J′)) of the pixel value of the pixel of interest and proceeds to step S 74 .
The arithmetic circuit 24 calculates the G signal Lg(I′,J′) (outputPixel(I′,J′)) of the pixel value of the pixel of interest by interpolation of Equation (5) (Equation (10)) using the G signals Gobs(k,ig,jg) as all pixel values of the contributing pixels specified by (k,ig,jg) calculated in step S 72 and a bicubic function Bicubic (p(k,i,j),q(k,i,j)) as the interpolation function w((x,y),(I′,J′)) that changes according to the transformed position (x,y).
In step S 74 , the arithmetic circuit 24 calculates sets of (k, ir, jr), with which the positions (x,y) on the reference coordinate system obtained by affine-trans forming the positions (ir−1,jr−1) of the R pixel of the kth image (a pixel of the R signal Robs(k,ir,jr)) with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) satisfy the expression I′−2≦x<I′+2 and the expression J′−2≦y<J′+2 representing the contributing area for the position of interest (I′,J′), for all of the first to Nth images. The arithmetic circuit 24 specifies the R pixels represented by (k,ir,jr) as contributing pixels contributing to interpolation of the pixel of interest and proceeds to step S 75 .
In step S 75 , the arithmetic circuit 24 calculates Equation (6) (Equation (10)) using all the sets of (k,ir,jr) calculated in step S 74 to calculate an R signal Lr(I′,J′) (outputPixel(I′,J′)) of the pixel value of the pixel of interest and proceeds to step S 76 .
The arithmetic circuit 24 calculates the R signal Lr(I′,J′) (outputPixel(I′,J′)) of the pixel value of the pixel of interest by interpolation of Equation (6) (Equation (10)) using the R signals Robs(k,ir,jr) as all pixel values of the contributing pixels specified by (k,ir,jr) calculated in step S 74 and the bicubic function Bicubic(p(k,i,j),q(k,i,j)) as the interpolation function w((x,y),(I′,J′)) that changes according to the transformed position (x,y).
In step S 76 , the arithmetic circuit 24 calculates sets of (k, ib, jb), with which the positions (x,y) on the reference coordinate system obtained by affine-trans forming the positions (ib−1,jb−1) of the B pixel of the kth image (a pixel of the B signal Bobs(k,ib,jb)) with the transformation parameters (a k ,b k ,c k ,d k ,s k ,t k ) satisfy the expression I′−2≦x<I′+2 and the expression J′−2≦y<+2 representing the contributing area for the position of interest (I′,J′), for all of the first to Nth images. The arithmetic circuit 24 specifies the B pixels represented by (k,ib,jb) as contributing pixels contributing to interpolation of the pixel of interest and proceeds to step S 77 .
In step S 77 , the arithmetic circuit 24 calculates Equation (7) (Equation (10)) using all the sets of (k,ib,jb) calculated in step S 76 to calculate a B signal Lb(I′,J′) (outputPixel(I′,J′)) of the pixel value of the pixel of interest and proceeds to step S 78 .
The arithmetic circuit 24 calculates the B signal Lb(I′,J′) (outputPixel (I′,J′)) of the pixel value of the pixel of interest by interpolation of Equation (7) (Equation (10)) using the B signals Gobs(k,ib,jb) as all pixel values of the contributing pixels specified by (k,ib,jb) calculated in step S 76 and a bicubic function Bicubic(p(k,i,j),q(k,i,j)) as the interpolation function w((x,y),(I′,J′)) that changes according to the transformed position (x,y).
In step S 78 , the arithmetic circuit 24 determines whether all the output pixels of the output image have been set as the pixel of interest, i.e., whether the G signal Lg(I′,J′), the R signal Lr(I′,J′), and the B signal Lb(I′,J′), which are the pixel values of all the output pixels of the output image, have been calculated.
When it is determined in step S 78 that there is an output pixel that has not been set as the pixel of interest, the arithmetic circuit 24 returns to step S 71 and the processing in steps S 71 to S 78 is repeated. The arithmetic circuit 24 sets the output pixel, which has not been set as the pixel of interest yet, as a new pixel of interest and calculates the G signal Lg(I′,J′), the R signal Lr(I′,J′), and the B signal Lb(I′,J′) of the new pixel of interest.
On the other hand, when it is determined in step S 78 that all the output pixels have been set as the pixel of interest, the arithmetic circuit 24 proceeds to step S 79 . The arithmetic circuit 24 applies necessary processing such as filter processing, color correction processing, and opening correction to an output image having the G signal Lg(I′,J′), the R signal Lr(I′,J′), and the B signal Lb(I′,J′) calculated for all the output pixels as pixel values. In step S 80 , the arithmetic circuit 24 outputs the output image to the D/A converter 9 or the CODEC 12 and returns to the start of the processing.
As described above, the positional relation among the plural photographed images obtained by high-speed imaging is detected, the pixel values of the output pixels are calculated using the pixel values of the plural photographed images after the positioning subjected to the positioning on the basis of the positional relation and the interpolation function that changes according to the positions of the pixels of the plural photographed images after the positioning, and the output image is generated from the pixel values. Consequently, it is possible to obtain a clear output image without camera shake.
In the image processing in FIG. 11, among the pixels of the first to Nth images, pixels, positions of which after the positioning, i.e., the transformed positions (x,y) of which after the affine transformation are in the contributing area for the position of interest (I′,J′), are specified as contributing pixels contributing to the interpolation of the pixel of interest. The pixel value outputPixel(I′,J′) of the pixel of interest is calculated according to the interpolation of Equation (10) performed by using pixel values of the contributing pixels and the bicubic function Bicubic(p(k,i,j),q(k,i,j)) as the interpolation function w((x,y),(I′,J′)).
As explained about Equation (11), (p(k,i,j),q(k,i,j)) represents a relative coordinate of the pixel in the position (i,j) of the kth image with the position (I′,J′) of the output pixel set as a reference. This relative coordinate (p(k,i,j),q(k,i,j)) is represented as (p,q) as appropriate.
To simplify the explanation, when the denominator of Equation (10) is neglected, in the interpolation of Equation (10), the contributing pixels, the transformed positions (x,y) of which are a position (p,q), contribute to the interpolation by an amount of a function value of a bicubic function Bicubic(p,q) obtained by calculating Equation (9).
A degree of (the pixel values of) the contributing pixels contributing to the interpolation is hereinafter referred to as a contribution rate as appropriate.
The bicubic function Bicubic (p,q) of Equation (9) as the interpolation function is a product of two cubic functions Cubic (p) and Cubic (q). Thus, regardless of what kind of pixel of the output image the pixel of interest is, the contributing pixels, the transformed positions (x,y) of which are the position (p,q), contribute to the interpolation at an identical contribution rats.
However, as the pixels of the output image, there are a pixel forming an edge (hereinafter also referred to as edge pixel as appropriate), a pixel that does not form an edge and on which, for example, a fiat texture is displayed (hereinafter also referred to as non-edge pixel), and the like.
When the pixel of interest is, for example, the edge pixel, it is possible to control zipper noise and artifact caused in an edge direction as a direction of the edge by setting the contribution rate large for the contributing pixels present in the direction along an edge direction among contributing pixels.
Moreover, when the pixel of interest is the edge pixel, among the contributing pixels, it is possible to maintain a high-frequency component in the edge direction and prevent an edge from blurring by setting the contribution rate small for the contributing pixels present, in a direction orthogonal to the edge direction.
When the pixel of interest is the non-edge pixel on which a flat texture is displayed, it is possible to control noise caused in a fiat section where the fiat texture is displayed by setting the contribution rate large for all the contributing pixels present around the pixel of interest.
As described above, it is possible to prevent the edge from blurring and control noise and the like to obtain a high-quality output image by controlling the contribution rate of the contributing pixels contributing to the interpolation.
Thus, the arithmetic circuit 24 can determine an edge direction in the pixel of interest and control the contribution rate of the contributing pixels contributing to the interpolation according to a result of the determination.
The arithmetic circuit 24 performs the determination of the edge direction and the control of the contribution rate, for example, as described below.
FIG. 12 shows a reference image.
Positions of respective pixels of the output image coincide with positions of respective pixels of the reference image. Thus, the arithmetic circuit 24 performs edge determination for determining an edge direction in the pixel of interest, which is a pixel of the output image, using the reference image.
Specifically, in FIG. 12, in the reference image, a pixel value of 3×5 pixels on the right side including a pixel in the position of the pixel of interest is a certain value va and a pixel value of 2×5 pixels on the left side is a certain value vb.
Roughly speaking, in FIG. 12, when a difference between the pixel value va of the 3×5 pixels or the like arranged on the right side and the pixel value vb of the 2×5 pixels or the like arranged on the left side is equal to or larger than a fixed value, the arithmetic circuit 24 determines that the edge direction in the pixel of interest is the vertical direction (a y direction). When the difference is smaller than the fixed value, the arithmetic circuit 24 determines that the pixel of interest is the non-edge pixel.
In the same manner, the arithmetic circuit 24 determines, on the basis of a difference between a pixel value of pixels arranged on the upper side of the position (I′,J′) of the pixel of interest and a pixel value of pixels arranged on the lower side, that the edge direction in the pixel of interest is the horizontal direction (an x direction) or determines that the pixel of interest is the non-edge pixel.
The arithmetic circuit 24 controls the contribution rate of the contributing pixels contributing to the interpolation, for example, as described below on the basis of the determination that the edge direction in the pixel of interest is the vertical direction or the horizontal direction or the determination that the pixel of interest is the non-edge pixel.
When the pixel of interest is an edge pixel on which an edge with an edge direction in the vertical direction or the horizontal direction is displayed, the arithmetic circuit 24 controls the contribution rate such that, among the contributing pixels, the contribution rate of the contributing pixels present in the direction along the edge direction is large and the contribution rate of the contributing pixels present in the direction orthogonal to the edge direction is small.
When the pixel of interest is the non-edge pixel, the arithmetic circuit 24 controls the contribution rate such that the contribution rate of all the contributing pixels is large.
The control of the contribution rate is performed, for example, as described below.
When the arithmetic circuit 24 controls the contribution rate, for example, a bicubic function Bicubic (p,q) represented by Equation (12) is adopted as the bicubic function Bicubic(p,q) as the interpolation function.
In Equation (12), scaleP and scaleQ are contribution parameters for controlling a contribution rate of a pixel value inputPixel(k,i,j) of the contributing pixels, which are multiplied by the bicubic function Bicubic(p,q) as the interpolation function, contributing to the interpolation. The arithmetic circuit 24 controls the contribution rate by adjusting (the bicubic function Bicubic(p,q) of Equation (12) defined by) the contribution parameters scaleP and scaleQ.
The contribution parameters scaleP and scaleQ are also parameters for adjusting high-pass characteristics of the bicubic function Bicubic(p,q) as the interpolation function. As the contribution parameters scaleP and scaleQ are larger, an effect of the interpolation function as a low-pass filter is larger. In other words, as the contribution parameters scaleP and scaleQ are larger, the interpolation function is a low-pass filter that controls the high-frequency component more.
As described above, when the pixel of interest is the edge pixel on which the edge with the edge direction in the vertical direction or the horizontal direction is displayed, the arithmetic circuit 24 controls the contribution rate such that, among the contributing pixels, the contribution rate of the contributing pixels present in the direction along the edge direction is large and the contribution rate of the contributing pixels present in the direction orthogonal to the edge direction is small. When the pixel of interest is the non-edge pixel, the arithmetic circuit 24 controls the contribution rate such that the contribution rate of all the contributing pixels is large. The control of the contribution rate is performed by adjusting (setting) the contribution parameters scaleP and scaleQ as described below.
When the pixel of interest is the edge pixel on which the edge with the edge direction in the horizontal direction is displayed, the arithmetic circuit 24 adjusts the contribution parameter scaleP to, for example, a value equal to or lager than 1 (a large value) v big and adjusts the contribution parameter scaleQ to, for example, a value larger than 0 and equal to or smaller than the value v big (a small value) v small .
Moreover, when the pixel of interest is the edge pixel on which the edge with the edge direction in the vertical direction is displayed, the arithmetic circuit 24 adjusts the contribution parameter scaleQ to, for example, the value equal to or larger than 1 (the large value) v big and adjusts the contribution parameter scaleP to, for example, the value larger than 0 and equal to or smaller than the value v big (the small value) v small .
When the pixel of interest is the non-edge pixel, the arithmetic circuit 24 adjusts the contribution parameters scaleP and scaleQ to, for example, a value equal to or larger than 1 (a large value) v N .
As the large values v big and v N equal to or larger than 1, it is possible to adopt, for example, 1.5. However, the values v big and v N do not need to be identical. As the small value v small larger than 0 and equal to or smaller than the value v big , it is possible to adopt, for example, 0.5.
Among two cubic functions C