DETAILED DESCRIPTION

[0021] The present invention's universal pointing control system consists of a handheld pointing device 100 , a display control unit 200 and a command deliver unit 300 as shown in FIG. 1 . In this example, the display control unit 200 is connected to a television 400 and a video component device 500 , which can be a digital TV set-top box, a VCR, or a DVD player, through video cables 520 and 510 , respectively. The display control unit 200 can also be embedded inside the TV or other video component device in alternative embodiments. The handheld pointing device 100 is aimed at the television screen 420 indicated by a line of sight 10 . On the other end of this line, a pointer 410 is displayed on the screen. When the user points the device to an arbitrary position of the screen, a set of orientation sensors inside the pointing device 100 , which will be described in later, detects the device's current orientation and generate the pointing direction signal. The pointing direction signal is encoded and sent to the display control unit 200 through a transmission link 50 . This transmission link can be any form of signal linkage. For example, it could be implemented by using radio frequency (RF) wireless link, infrared (IR) link, or even a wired cable. Upon receiving the signal, a central process unit (CPU) inside the display unit 200 decodes and analyzes the pointing direction and determines the new coordinates of the pointer on the screen. A pointer is drawn at the calculated coordinate and the pointer image is then superimposed onto the input video signal, which is input from video component device 500 through cable 570 . A set of menus and control items 430 are also drawn and superimposed to the video signal. The composite video is then output to the television 400 through the output video cable 520 and displayed on the television screen 420 . As a result, the pointer 470 is shown at a new location on the screen where the user points to. The user perceives that the pointer is moved following the aiming line of sight 70 .

[0022] Buttons are located on the handheld pointing device to collect the user's selection activities. Three buttons are shown in this example, one for command selection ( 101 ), one to show and hide screen pointer ( 102 ), and another one for calibration purpose ( 703 ). When the user uses the device at first time, a calibration procedure is performed. The user aims the device at the center of the screen and presses button 703 . The device's pointing direction information is recorded and stored into the display control unit as the screen center reference. Any subsequent pointing information is then compared with this reference, and the difference will be calculated as the pointer displacement distance away from the screen center.

[0023] During normal usage, as the user points and clicks the selection button, the on-screen menu or selectable items under the pointer are processed by the CPU in the display control unit. Selection information is generated and forwarded to the command delivery unit 300 by means of transmission link 60 . The link 60 , again, can be any form of signal linkage. The command deliver unit 300 can be a stand-alone device facing the TV 400 and other equipments ( 500 , 570 , 520 ), or can be embedded inside the pointing device 100 . All remote control command codes for the devices in the home entertainment system are prerecorded in a memory module in the command delivery unit 300 . Upon receiving selection information, the command delivery unit issues the corresponding command by searching the memory module, and emits the command infrared (IR) signal through the IR emitter 351 to the controlled equipments. The target equipment performs a task as if it had received a command directly from its original remote control device.

[0024] FIG. 2 shows the pointing control system as used in a computer presentation scenario. In this case, the presentation is projected onto the screen 720 by a projector 700 , which receives the video input from a computer 600 though a video cable 620 . The display control unit 200 is connected to the peripheral port of a computer 600 through the cable 610 . The presenter aims the pointing device at the screen 720 by a line of sight 10 . The aiming direction information generated by a set of orientation sensors in the pointing device 100 is transmitted to display control unit 200 through transmission link 50 . The CPU in the display control unit interprets the direction information, sends the pointer move command to the computer's peripheral port, and instructs the computer to move the pointer 710 on screen to the aimed place. This is analogous to moving the pointer by moving a regular computer mouse device, except that the moving information is in absolute coordinates instead of relative steps. The buttons 101 , 102 , and 103 on the pointing device allow the presenter to select and execute a command remotely.

[0025] FIG. 3 exposes the components inside of the handheld pointing device. On the top face of the device are buttons 101 , 102 , and 103 for collecting user selection activities. A set of orientation sensors 120 and 130 mounted on the print circuit board 160 detect device's orientation changes. Note that the sensors are mounted orthogonally to each other. The sensor 120 detects the device's yaw (azimuth) angle and sensor 130 detects device's pitch (inclination) angle. Additional sensors (not show in the picture) could be used to detect device's roll angle which may provide an additional dimension of control. A microcontroller 110 provides computation power for calculating and encoding the orientation signal output from the orientation sensors. It also provides logic control for the transmitter 140 and other electronic components. The device is powered by batteries 170 .

[0026] The orientation sensors' mechanisms are shown in FIGS. 4 a and 4 b . The orientation sensor demonstrated in FIG. 4 a is a magnetic field sensor, whereas the one in FIG. 4 b is an accelerometer sensor. However, the orientation detection may not be limited to these types of sensors. Other sensors, for example, a gyro sensor, can also be used in the pointing control system. In FIG. 4 a , a two-axis magnetic field sensor 120 is used to detect the device's orientation relative to the direction of the earth's magnetic field 25 . The sensor contains two magnetic field detectors which are arranged orthogonal to each other. The sensor is mounted on the device's circuit board so that the two magnetic field detectors are laid on the x-z plane as shown in the picture. The azimuth angle φ between the device's heading direction and the earth's North Pole direction can be calculated from the sensor's x and z output: φ=arc tan(x/z). When the user performs calibration, the device records the azimuth angle φ ₀ as the reference angle as the user points the device to center of the screen. When the device is rotated about the y-axis and the pointing direction is moved away from screen's center, the azimuth angle difference from the reference angle is φ−φ _0. This difference is interpreted by the display control unit as the degree of the pointer's horizontal departure from the screen center. The amount by which the pointer moves horizontally ( 22 ) can be adjusted in the display control unit proportionally to the change in the pointer's azimuth angle 21 .

[0027] The orientation sensor 130 uses a similar method to detect the device's inclination angle. The sensor could be an accelerometer or another orientation sensor that can sense the device's heading change in the y-z plane. An accelerometer sensor which can detect static acceleration is described in detail here. The accelerometer sensor 130 contains two orthogonally arranged acceleration detectors. The sensor is mounted perpendicular to the circuit board's plane so that one detector in the sensor detects y-axis acceleration and the other detects z-axis acceleration. Earth's gravity 26 exerts a static acceleration on these detectors. When the device is placed on a horizontal level, the accelerometer's z-axis detector outputs zero acceleration, while the y-axis outputs the maximum acceleration (1 g). If the device is rotated about the x-axis, the z and y channel outputs of the sensor are changed according to the inclination angle. The inclination angle E thus can be calculated: ε=arc tan(z/y). During the calibration, the device's inclination angle to the screen center ε ₀ is recorded and stored as a reference angle. Any inclination angles sampled thereafter is compared with this reference angle by determining the offset, ε−ε ₀ . This difference is interpreted by the display control unit as a degree of the pointer's departure from the screen's center in the vertical direction. The amount by which the pointer moves vertically ( 32 ) can be adjusted in the display control unit proportionally to the change in the pointer's inclination angle 31 .

[0028] For a simplified version, a one-axis accelerometer sensor can be used. In such a case, the acceleration detector is mounted along the device's z-axis. The inclination angle ε thus can be calculated: ε=arc sin(z).

[0029] FIG. 5 is the functional block diagram of the handheld pointing device. The signal conditioning circuit for sensor 120 consists of two amplifiers 121 , 123 and two low pass filters 122 , 124 . Because we are interested in the static position and low frequency movement of the device, the high frequency noises of the amplified x-axis and y-axis signals are filtered in order to get a higher resolution of the azimuth angle changes. Two amplifiers 131 , 133 and two low pass filters 132 , 134 are for conditioning the sensor 130 output's x-axis and z-axis signals. We are interested in the sensor's static output relative to earth's constant gravity. Therefore, the high frequency noises of the signals are also filtered in order to get a higher resolution of the inclination angle changes. The conditioned signals from sensors 120 and 130 are then sent to an analog-to-digital converter (ADC) 111 by an analog multiplexer 112 . The digitized sensors data are then sent to a microcontroller (MCU) 110 for further signal processing. Some variations of the orientation sensor convert the analog signal internally to a digital or time period-based signal. In those cases, the signals can be directly sampled by the microcontroller without an ADC chip. The MCU 110 computes the azimuth and inclination angles. Buttons 101 , 102 , and 103 produce activity signals that are sampled by MCU 100 . The sensor orientation data and buttons activities are coded in such a way that the display control unit can decode them later. The encoded data is passed to a modulator 113 to modulate a carrier frequency for transmission. The transmitter 140 emits the modulated signal 50 to the display control unit 200 . The circuit is powered by batteries 170 . A battery manager unit 171 conditions the voltage for all components of the circuit. The MCU 110 constantly monitors the changes of sensor outputs. If the sensor outputs are not changed for a period, the MCU interprets the device as not been used. The MCU instructs the battery manager unit 171 to shut down the battery power supply to the transmitter and other components in order to save power consumption during the idle stage.

[0030] FIG. 6 a and FIG. 6 b show the functional block diagrams of display control unit 200 for a computer and a television set, respectively. A central process unit (CPU) 210 , a receiver 221 , a demodulator 231 , and a memory module 270 are common for both cases. The transmitted signal 50 from the pointing device, which includes handheld device orientation and user selection activities, is intercepted by the receiver module 221 . After being demodulated by a demodulator 231 , the pointing device data is sent to CPU 210 for further processing. The CPU compares the device's azimuth and inclination angle data with the reference angles, which are sampled and stored in the memory module 270 during the calibration procedure. The difference angles calculated are translated into screen coordinates and the target device is instructed to move the pointer to the new location. The interface components of the display control unit are different for each control target. In FIG. 6 a , a computer peripheral interface module is used to connect to a computer port. The pointer coordinates are sent to the computer and, by the computer's processor and video card, the pointer on the screen is moved to the corresponding location. The button activities are also sent to the computer though this interface and trigger certain actions for the computer.

[0031] FIG. 6 b demonstrates the display control interfaces to a television. The input video signal, which may come from other home entertainment devices such as digital TV set-top boxes, DVD players, etc., is decoded by a video decoder 251 frame by frame. A new pointer image is drawn at the coordinate calculated by CPU 210 . The pointer image, along with menus and other control item images pre-stored in the memory module 270 , are sent to a graphic-video multiplexer 250 to superimpose onto a video frame. The composite video frame is then encoded by a video encoder 252 and sent to the television for display. The process is about 30 frames per second. As the result, a pointer is moved on top of the video following the handheld device's pointing direction. If the CPU 210 senses a button click activity while the pointer is moved on top a menu or a controllable item, it sends a command to a transmitter 222 through a modulator 232 . The modulated transmission signal 60 is forwarded to the command delivery unit 300 for controlling the television and other home entertainment equipment.

[0032] FIG. 7 a is the functional block diagram of command delivery unit 300 . A receiver 320 intercepts the transmitted signal from the display control unit 200 . The signal is sent to a microcontroller (MCU) 310 after demodulation by a demodulator 330 . In the command delivery unit, there is a non-volatile memory module which stores all the control command codes for varieties of home entertainment equipments. These command codes can preset by the vendor or stored by the user during programming or training procedures. The command codes are stored in such way that each command is coupled with an identification number (key). The arriving signal from the display control unit is served as the key so that MCU 310 can look up the key's record in memory and fetch the corresponding command code. For example, if the pointer is moved on top of a VCR play button on the screen and user clicks the selection button, the display control unit sends a value equal to 100 to the command delivery unit. By looking up the key value 100 in the memory module, the MCU 310 fetches the pre-stored VCR play command code. The command code is sent to an infrared transmitter 350 to drive an infrared emitter 351 . The infrared-carried command code is then sent to the home entertainment equipment, in this case a VCR, to control their functions.

[0033] FIG. 7 b demonstrates the programming (training) procedure of the command delivery unit. When the user adds new home entertainment equipment, he/she can program the command delivery unit to learn commands for the equipment. By moving the pointer on the screen and selecting a new control item, the control display unit prompts the user to train the command delivery unit with the equipment's original remote control. The control display unit also sends an identification key value to the command delivery unit. At this moment, the user can point the equipment's original remote control 800 to the command deliver unit 300 and push a corresponding remote control button. An infrared signal is sent to an infrared sensor 361 on the command delivery unit. On FIG. 7 a , the infrared signal is converted to electronic code by the infrared receiver 360 . This code is stored with the identification key value into the memory module 370 . The code stored is retrieved later by the command delivery unit to control the equipment. Because the infrared command received by the home entertainment equipment received is exactly the same as their original native code, any infrared controlled equipment can be controlled by the command delivery unit.

[0034] In an alternative implementation, the control command codes can be stored in the display control unit instead of the command delivery unit as shown in FIG. 8 a . In this case, an infrared receiver 260 and an infrared sensor 267 are added to the unit. During the programming procedure shown in FIG. 8 b , the home entertainment equipment's original remote control 800 is pointed to the display control unit 200 . The infrared signal is received by the infrared sensor 261 and infrared receiver 260 , and the command codes are stored in the memory module 270 with an identification key value. During control session, the MCU retrieves the command code using a key value according to the user's selection on the screen. The command code is sent to the command delivery unit through the modulator 232 and transmitter 222 . The command delivery unit in this case simply forwards the command to the target home entertainment equipment.