DETAILED DESCRIPTION OF THE PRESENTLY PREFERRED EMBODIMENTS

[0015] The present invention is broadly directed to methods and apparatus which automatically detect and determine if received speech is that of the user of the speech recognition system and, if so, for generating a signal to start the operation of the speech recognition system without requiring the speaker to first utter any activating control words or to depress or operate a start-stop button or switch. Thus, the occurrence of human speech is automatically detected and such detection is entirely transparent to the speaker. The method of the invention is preferably implemented in a digital computer based speech recognition system capable of recognizing speech data, and of at least temporarily storing recognized speech data in a memory. A typical speech recognition system receives speech as a collection or stream of speech data segments. As each speech data segment is vocalized by a user, the automated speech recognition system recognizes and stores a data element that corresponds to that speech data segment. In accordance with the present invention, as soon as speech is detected and is identified as being from the user, the speech recognition system is activated and remains so until speech from the user is no longer detected. No overt or conscious act on the part of the speaker is, therefore, required to activate and deactivate the system.

[0016] Referring to FIG. 1 , there is illustrated a block diagram of an embodiment of the present invention. The speech recognition system is normally implemented in or incorporated within a computer system which comprises bus 100 , keyboard controller 101 , external memory 102 , mass storage device 103 and processor 104 . Bus 100 can be a single bus or a combination of multiple buses, and provides communication links between the various components of the computer system. Keyboard controller 101 may be a dedicated device or can reside in another component such as a bus controller or another controller such as a hand held device, i.e., a Palm Pilot or Compaq iPAQ. The keyboard controller accommodates coupling of a keyboard to the computer system and transmits signals from a keyboard to the computer system. If the keyboard is located or implemented in the hand held device, it may be coupled to the keyboard controller by infrared, radio waves or the like. Memory 102 stores information from mass storage device 102 and processor 104 for use by processor 104 . Mass storage device 103 can be a hard disk drive, a floppy disk drive, a CD-ROM device, or a flash memory device. Processor 104 provides information to memory 102 , and may be a microprocessor operable for decoding and executing a computer program such as an application program or operating system. An audio input device 105 is provided and includes a microphone 106 to receive sound and convert it to a digital form that can be processed by the system, and in particular, by processor 104 . The audio input device is preferably located within the hand held device. A video input device 107 , which includes a video camera 108 positioned to view a visual field, is also located in or on the hand held device. The video input device outputs a digital video signal that can be processed by processor 104 .

[0017] FIG. 2 depicts a block diagram of an exemplary speech recognition system 200 according to an embodiment of the invention. As shown in FIG. 2, a user 202 is positioned within the field of view of camera 108 and within the audio range of microphone 106 that are located in or otherwise associated with a hand held device 109 . This positioning of the microphone and the camera normally results when a user picks up the hand held device and begins to speak. Audio input device 105 and video input device 107 respectively output digital information to a speech recognition unit 204 and a video recognition unit 206 . Video recognition unit 206 provides an input to speech recognition unit 204 , and the speech recognition unit and video recognition unit provide inputs to an audio-video analyzer 320 . The video recognition unit 206 , speech recognition unit 204 and audio-video analyzer 320 together form the user recognition unit 330 .

[0018] FIG. 3 is a flow chart illustrating the operation of user recognition unit 3 . The inventive method begins at step 300 and proceeds to step 302 at which the video signal is received, frame by frame. The characteristics of the received video signal for each frame are obtained and serially stored, frame by frame, at step 306 . At step 304 the audio signal is received and time indexed to synchronize the audio signal, frame by frame, to the video signal. Thus, for each frame of video information there is a corresponding “frame” of audio signal information, where the information in each of the two corresponding frames were obtained at the same time. The characteristics of the received audio signal are serially stored at step 308 . At step 310 , the frames of video information are examined, sequentially, frame by frame until a face is recognized or detected; an examination of that frame is then carried out, at step 312 , to identify movement of the speech articulators such as motion or displacement of the lip, mouth and/or tongue. Upon detection that one or more of the speech articulators has moved or is moving, an estimate of that movement is made at step 314 . Then, using the estimate of the motion and the position of the speech articulators, the characteristics of the sound (i.e., the speech) that such motion of the speech articulators is expected to produce are determined at step 316 . The frame of video information used in step 316 to obtain the sound characteristics is identified and the frame of audio signal characteristics that corresponds in time with that video frame is selected for analysis. At step 318 , the frame of audio signal characteristics which has been selected is analyzed and the characteristics of the received sound (i.e., the speech) stored in that frame are obtained. The characteristics of the actual sound reviewed and analyzed at step 318 from the frame of audio information is then compared, at step 320 , with the estimated sound that the form of the speech articulators is expected to produce from step 316 . If there is no match, an examination of the successive frames of video information continues. When there is a match, on the other hand, a signal is generated to indicate that the speech recognition system should begin operating, or to trigger such operation. Suitable time delays can be incorporated to maintain operation of the speech recognition system for a preset time interval while a search is carried out for another match of the video and audio information. If a match is found within the preset time interval, the speech recognition system continues to operate. If, however, another match is not found within the preset time interval, then the speech recognition system ceases operation.

[0019] There has accordingly been disclosed a method for inputting speech to a handheld device simultaneously using a microphone and a camera, where the microphone and camera are located on or in the handheld device. It should nevertheless be understood that the method herein disclosed can also be used with other devices, such as desktop computers or in IP telephony, that are equipped or associated with a microphone and camera. It is again noted that the inventive method is totally passive as it does not require that the user or others perform any unusual steps or other function in addition to the normal action of talking into the handheld or other device.

[0020] Although the invention has been described herein with respect to specific embodiments, it should be understood that these embodiments are exemplary only, and that it is contemplated that the described methods and apparatus of the invention can be varied widely while still maintaining the advantages of the invention. Thus, the disclosure should not be understood as limiting in any way the intended scope of the invention. In addition, as used herein, the term “unit” is intended to refer to a digital device that may, for example, take the form of a hardwired circuit, or software executing on a processor, or a combination thereof. For example, the units 204 , 206 , 208 , 310 , 312 , 314 , 316 , 318 , 320 , may by illustrative example be implemented by software executing in processor 104 , or all or some of the functionality of these components can be provided by hardware alone. Furthermore, as used herein, the term machine readable medium is intended to include, without limitation, a storage disk, CD-ROM, RAM or ROM memory, or an electronic signal propagating between components in a system or network.

[0021] Thus, while there have shown and described and pointed out fundamental novel features of the invention as applied to a preferred embodiment thereof, it will be understood that various omissions and substitutions and changes in the form and details of the devices illustrated, and in their operation, may be made by those skilled in the art without departing from the spirit of the invention. For example, it is expressly intended that all combinations of those elements and/or method steps which perform substantially the same function in substantially the same way to achieve the same results are within the scope of the invention. Moreover, it should be recognized that structures and/or elements and/or method steps shown and/or described in connection with any disclosed form or embodiment of the invention may be incorporated in any other disclosed or described or suggested form or embodiment as a general matter of design choice. It is the intention, therefore, to be limited only as indicated by the scope of the claims appended hereto.