BEST MODE FOR CARRYING OUT THE INVENTION

[0077] Preferred embodiments of the present invention will now be described in detail in accordance with the accompanying drawings.

First Embodiment

[0078] The first embodiment of the present invention will be described below with reference to FIGS. 1 and 2 .

[0079] FIG. 1 is a block diagram showing the system arrangement of an image processing apparatus according to this embodiment. Although the internal arrangements of most of devices in FIG. 1 are not described, each of these devices comprises a controller and communication unit, and cooperates with other devices via communications. The communication function of the communication unit of any device can be changed by exchanging a module. Therefore, the communication units may be connected via wired or wireless connections. In FIG. 1 , the solid lines with arrows indicate the flow of control data, the dotted lines with arrows indicate the flow of CG (computer graphics) data or a CG image, and the broken line with an arrow indicates the flow of a real image or composite image.

[0080] In FIG. 1 , reference numeral 101 denotes a image input device (image input means); 102 , a position/posture sensor (image input parameter acquisition means); 103 , a moving device; 104 , a distance sensor; 105 , an HMD (head-mounted display) serving as display means; 106 , a position/posture sensor (display parameter acquisition means); 107 , a data processor; 108 , a CG data management unit; 109 , a CG image generator (CG image generation means); 110 , a CG geometric information calculation unit (CG geometric information calculation means); 111 , a moving device controller; 112 , a control command input device; 113 , a image composition device (image composition means); and 114 , a image display device.

[0081] The image input device 101 and distance sensor 104 are attached to the moving device 103 . Also, the position/posture sensor 102 is attached to the image input device 101 . The relationship among these attached devices will be described later using FIG. 2 .

[0082] The moving device 103 controls the movement and posture in accordance with control information received from the moving device controller 111 . In this way, the image input device 101 can take images in every directions from an arbitrary position. The image input device 101 sends a real image to the image composition device 113 .

[0083] The position/posture sensor 102 measures the position and posture of the image input device 101 in a predetermined coordinate system in a real world, and sends measured data (image input position/posture information) to the moving device controller 111 . The image input position/posture information is also sent to the CG image generator 109 via the moving device controller 111 .

[0084] The distance sensor 104 measures the distance to an object which is present in a predetermined direction and within a predetermined distance range from a predetermined position on the moving device 103 , converts the measured data into distance data (obstacle information) from the viewpoint of the image input device 101 , and sends the converted data to the moving device controller 111 . The obstacle information is also sent to the CG image generator 109 via the moving device controller 111 .

[0085] The position/posture sensor 106 is attached to the HMD 105 . The position/posture sensor 106 measures the position and posture of the HMD 105 in a predetermined coordinate system in a real world, and sends measured data (image input position/posture information of the HMD 105 ) to the moving device controller 111 . The image input position/posture information of the HMD 105 is also sent to the CG image generator 109 via the moving device controller 111 . Note that the moving device controller 111 does not always require the position/posture information of the HMD 105 . Hence, the position/posture information of the HMD 105 may be directly sent to the CG image generator 109 (without going through the moving device controller 111 ).

[0086] The control command input device 112 inputs commands (control commands) for controlling the actions of a virtual object or CG character that appears in a CG image or the position/posture of the moving device 101 . As a control command input method, various methods such as key depression, mouse operation, joystick operation, touch panel depression, voice input using a speech recognition technique, gesture input using an image recognition technique, and the like, are available, and any of these methods may be used.

[0087] The data processor 107 has the CG data management unit 108 , CG image generator 109 , CG geometric information calculation unit 110 , and moving device controller 111 . FIG. 1 illustrates the data processor 107 as a single device, but the data processor 107 may comprise a group of a plurality of devices. For example, the CG data management unit 108 may be arranged in the first device, the CG image generator 109 and CG geometric information calculation unit 110 for generating CG data from the viewpoint of the image input device 101 (to be described later) may be arranged in the second device, the CG image generator 109 and CG geometric information calculation unit 110 for generating CG data from the viewpoint of the HMD 105 may be arranged in the third device, and the moving device controller 111 may be arranged in the fourth device. As the data processor 107 described in this embodiment, arbitrary data processing devices such as a personal computer, workstation, versatile computer, dedicated computer or dedicated hardware, and the like may be used.

[0088] In this embodiment, CG geometric information is calculated in the CG image generation process. Therefore, the CG geometric information calculation unit 110 is included in the CG image generator 109 , but the CG geometric information calculation unit 110 need not always be included in the CG image generator 109 . Hence, the CG image generator 109 and CG geometric information calculation unit 110 may be independent modules as long as they can appropriately exchange data.

[0089] The CG data management unit 108 manages storage, update, and the like of various data required to generate CG images. The CG geometric information calculation unit 110 calculates geometric information (information of position, shape, size, and the like) upon virtually laying out a virtual object or CG character expressed by CG data read out from the CG data management unit 108 via the CG image generator 109 in a predetermined coordinate system in a real world.

[0090] The CG image generator 109 reads and writes some CG data from the CG data management unit 108 as needed. The CG image generator 109 moves and modifies a virtual object or CG character in accordance with a control command. In this case, since a portion of the CG data is rewritten, the CG image generator 109 passes the rewritten CG data to the CG geometric information calculation unit 110 and controls it to calculate CG geometric information.

[0091] The CG image generator 109 calculates using the CG geometric information and obstacle information whether or not a portion of the virtual object or CG character virtually collides against an obstacle (real object). If any collision is detected, the generator 109 changes the shape, color, and the like of the virtual object or CG character in accordance with the degree of collision. In this case, since a portion of the CG data is rewritten, the CG image generator 109 passes the rewritten CG data to the CG geometric information calculation unit 110 and controls it to calculate CG geometric information. After that, the updated geometric information is sent from the CG geometric information calculation unit 110 to the moving device controller 111 .

[0092] The CG image generator 109 then generates a CG image (a image of the virtual object or CG character) from the viewpoint of the image input device 101 using the updated CG data, updated CG geometric information, and image input position/posture information. Also, he CG image generator 109 generates a CG image from the viewpoint of the HMD 105 using the updated CG data, updated CG geometric information, and image input position/posture information of the HMD 105 . The CG image from the viewpoint of the image input device 101 is sent to the image composition device 113 , and the CG image from the viewpoint of the HMD 105 is sent to the HMD 105 .

[0093] The moving device controller 111 calculates control information using the control command, obstacle information, updated CG geometric information, and image input position/posture information so as to prevent the moving device 103 from colliding against the obstacle (real object) and the virtual object or CG character, and to control to stably change the position and posture of the moving device 103 , and sends the control information to the moving device 103 .

[0094] The HMD 105 is a see-through type HMD (an HMD of a type that allows external light to transmit through a region where no image is displayed). The HMD 105 displays the CG image received from the CG image generator 109 , but external light is transmitted through the region where no CG image is displayed. Hence, the user who wears the HMD 105 can observe a composite image of the CG image and a scene in front of him or her. Therefore, the user who wears the HMD 105 can act interactively with the CG image.

[0095] The image composition device 113 composites the real image received from the image input device 101 and the CG image received from the CG image generator 109 , and sends the composite image to the image display device 114 . The image display device 114 displays the composite image. As the image display device 114 , arbitrary display devices such as various types of displays (CRT display, liquid crystal display, plasma display, and the like), various type of projectors (forward projection type projector, backward projection type projector, and the like), non-transmission type HMD (an HMD of a type that does not allow external light to transmit through), and the like can be used.

[0096] Normally, the image display device 114 is set near a person (operator) who inputs a control command to the control command input device 112 , and the operator inputs the control command while observing the composite image. In this manner, the operator can issue a control command to interactively move the real image and CG image. That is, the CG character can freely touch or dodge the obstacle in the real image, attack or dodge the virtual object in the CG image, and dance with a performer (who wears the HMD) in the real image.

[0097] Note that the operator of the control command input device 112 may be an expert operator or an end user. Also, a plurality of operators may be present, and the operator may be present in a site different from the image input site. For example, when operators are a plurality of end users who live in distant places, the control command input device 112 and image display device 114 can be set in each user's home.

[0098] In such case, a device that combines control commands received from a plurality of control command input devices 112 into one, and a image distributor for distributing the composite image sent from the image composition device 113 to a plurality of image display devices 114 must be added.

[0099] FIG. 2 is a schematic view showing a image input scene upon generating the composite image using the image processing apparatus according to this embodiment.

[0100] Referring to FIG. 2 , reference numeral 201 denotes a image input device; 202 , a position/posture sensor; 203 , a moving device; 204 , a distance sensor; 205 , a image input device; 206 , a position/posture sensor; 207 , a moving device; 208 , a distance sensor; 209 , an HMD; 210 , a position/posture sensor; 211 , a performer (who wears the HMD); 212 , a CG character; and 213 and 214 , virtual objects.

[0101] The image input device 201 and distance sensor 204 are attached to the moving device 203 . Also, the position/posture sensor 202 is attached to the image input device 201 . The moving device 203 is a self-running robot which mounts a battery, and can move around the image input site in arbitrary directions, since it is remote-controlled via wireless communications. Since the moving device 203 has a support base of the image input device 201 , which is rotatable in the horizontal and vertical directions, it can freely change the posture of the image input device 201 .

[0102] As the distance sensor 204 , a compact infrared ray sensor, ultrasonic sensor, or the like may be used. If such sensor is used, the distance to an object, which is present within a given range in front of the sensor, can be measured.

[0103] In FIG. 2 , since the moving device 203 is vertically elongated, two distance sensors 204 (one each on upper and lower front portions) are attached to the front portion of the moving device 203 to broaden the distance measurement range vertically. With this arrangement, since the distance to an object, which is present in front of the moving device 203 and image input device 201 , can be measured, the moving device can move while dodging an obstacle and person, and can approach their neighbors, as described in FIG. 1 .

[0104] The image input device 205 , position/posture sensor 206 , moving device 207 , and distance sensor 208 respectively have the same functions as the image input device 201 , position/posture sensor 202 , moving device 203 , and distance sensor 204 mentioned above.

[0105] In this case, the moving device 207 is attached to the ceiling of a building or a support member such as a crane or the like, and can freely change the position and posture of the image input device 205 within a predetermined range.

[0106] Note that the moving devices 203 and 207 are not limited to the illustrated examples. In addition, moving devices having various functions and forms such as remote-controllable flying objects (airplane, helicopter, balloon, and the like), waterborne objects (boat, Hovercraft, amphibian, or the like), underwater moving objects (submarine, underwater robot, and the like), and so forth may be used.

[0107] The position/posture sensor 210 is attached to the HMD 209 , and can measure the viewpoint position and line-of-sight direction of the performer (who wears the HMD) 211 . Also, the position/posture sensor 210 is attached to a hand of the performer 211 , and can measure the position and direction of the hand of the hand of the performer 211 .

[0108] The CG character 212 is virtually laid out in a real world to have a position and size that can cover the image input device 201 , position/posture sensor 202 , moving device 203 , and distance sensor 204 (a set of these devices will be referred to as image input device group A hereinafter), so that image input device group A cannot be seen in a composite image from the viewpoints of the image input devices 201 and 205 , and the CG character 212 alone can be seen. Also, in a CG image from the viewpoint of the HMD 209 , the CG character 212 is displayed at a position where it covers image input device group A.

[0109] The virtual object 213 is virtually laid out in a real world to have a position and size that can cover the image input device 205 , position/posture sensor 206 , image input device 207 , and distance sensor 208 (a set of these devices will be referred to as image input device group B hereinafter), so that image input device group B cannot be seen in a composite image from the viewpoints of the image input devices 201 and 205 , and the virtual object 213 alone can be seen. Also, in a CG image from the viewpoint of the HMD 209 , the CG character 212 is displayed at a position where it covers image input device group B.

[0110] The virtual object 214 is displayed at a position where it looks as if it is held by the hand of the performer 211 . For example, a CG image can be generated in such a manner that when the performer 211 has made a predetermined hand action, the display position of the virtual object 214 moves to a position where the object is supposedly held by the hand of the G character 212 . In this case, in order to display the virtual object 214 at a position where it looks as if it is held by the hand of the performer 211 , measurement data obtained from the position/posture sensor 210 attached to the hand of the performer 211 can be used.

[0111] On the other hand, in order to display the virtual object 214 at a position where it looks as if it is held by the hand of the CG character 212 , the CG geometric information described in FIG. 1 can be used.

[0112] In FIG. 2 , since there are two image input devices, a viewer can selectively watch one of the composite images from the viewpoints of the image input devices 201 and 205 . Or when the image display device 114 described in FIG. 1 has a two-split screen display function, the viewer can watch the composite images from the two viewpoints, which are simultaneously displayed on the screen.

[0113] The present invention relates to an image processing method and apparatus, which comprise both image input means and CG image generation means to naturally composite a real image and CG image, and can be used to provide novel services to every viewing sites including the image input site and remote places in the fields that exploit images such as shooting and rehearsal of a movie and television program, play, game, KARAOKE, and the like.

Second Embodiment

[0114] The second embodiment of the present invention will be described below with reference to FIGS. 3 to 15 .

[0115] FIG. 3 is a block diagram showing the system arrangement of a studio apparatus which comprises an image processing apparatus according to this embodiment.

[0116] Referring to FIG. 3 , reference numeral 301 denotes a studio (MR studio) serving as a image input site; 302 , a studio setting placed in the studio 301 ; 303 , a performer; 304 , a image input camera (image input means); 305 , a head-mounted display (to be abbreviated as an HMD hereinafter) that the performer 303 wears on his or her head; 306 , a position sensor (display parameter acquisition means) built in the HMD 305 ; 307 , virtual objects ( 307 a , a virtual object as a main character upon shooting, and 307 b , a virtual object corresponding to viewers) which are superimposed on a image to be observed by the performer 303 and a image taken by the camera 304 ; 308 , a image generation device for generating a image to be observed by the performer 303 ; 309 , a image superimpose device for superimposing a image of the virtual objects 307 on a image taken by the camera 304 ; 310 , an operating device for managing and operating the states of the virtual objects 307 ; 311 , a network for connecting the image generation device 308 , image superimpose device 309 , and operating device 310 ; 312 , a viewer information management device for managing information of viewers by receiving communications from the viewers; and 313 , an interactive broadcast device (transmission device) for transmitting or broadcasting the output from the image superimpose device 309 , and receiving information (reactions) from the viewers.

[0117] As the position sensor 306 , for example, devices such as a magnetic position/direction sensor, Fastrak available from Polhemus Incorporated, and the like may be used. The image generation device 308 or image superimpose device 309 can comprise a combination of a PC (personal computer), a video capture card, and a video card with a CG rendering function. The operating device 310 can comprise a normal PC.

[0118] The number of sets of the HMD 305 , image generation device 308 , and the like can be increased in correspondence with the number of performers or the number of staff members who observe at the same time, and the number of sets of the camera 304 , image superimpose device 309 , and the like can be increased in correspondence with the number of image input cameras.

[0119] FIG. 4 shows the internal structure of the HMD 305 . The HMD 305 comprises a first prism optical element 401 for guiding incoming external light to an image sensor, an image sensing element 402 for receiving and sensing the light, a display element 403 for presenting a image, a second prism optical element 404 for guiding the displayed image to the eye, and the like, since it has functions of both a display device and an image sensing device.

[0120] As shown in FIG. 3 , the studio setting 302 is placed in the studio 301 , and the performer 303 acts in that studio. The performer 303 wears the HMD 305 with the built-in position sensor 306 , which outputs the position information. The operating device 310 receives instructions for displaying and moving the virtual objects 307 , and transfers these instructions to the image generation device 308 and image superimpose device 309 via the network 311 .

[0121] The image generation device 308 generates a CG image in correspondence with the instructed states of the virtual objects 307 and the head position information obtained from the position sensor 306 or the like, composites it with sensed image data obtained from the HMD 305 , and outputs the composite image to the HMD 305 . By watching the composite image displayed on the HMD 305 , the performer 303 can observe the virtual objects 307 as if they were present in the studio setting 302 . The camera 304 senses the state of the studio 301 including the performer 303 and studio setting 302 , and outputs the sensed image data. The image superimpose device 309 generates a CG image corresponding to the state of the virtual objects 307 according to an instruction from the operating device 310 , and the position and posture of the camera 304 , and composites that image with image data obtained from the camera 304 , thus generating an output image.

[0122] The image generated by the image generation device 308 and image superimpose device 309 is not only watched by a player in the studio 301 but also broadcasted to viewers via the interactive broadcast device 313 .

[0123] The interactive broadcast device 313 comprises the Internet and BS digital broadcast, which are building components which are known to those who are skilled in the art. More specifically, as for BS digital broadcast, downstream video distribution (from the studio to home) is made via a satellite, and upstream communications are made via the Internet using a cable, telephone line, or dedicated line. If the Internet allows a broadband communication, downstream video distribution can also be made via the Internet. The studio and home are interconnected via these upstream and downstream communications.

[0124] At this time, information in response to broadcast contents, which is sent from each viewer (home) is received and managed by the viewer information management device 312 .

[0125] FIG. 5 is a block diagram showing details of the operation of the image processing apparatus according to this embodiment shown in FIG. 3 .

[0126] Referring to FIG. 5 , reference numeral 501 denotes an HMD which has a so-called see-through function, and comprises an image sensing unit and image display unit. Reference numeral 502 denotes a first image composition means; 503 , a first CG rendering means that renders a CG image from the viewpoint of the HMD 501 ; 504 , a prohibited region processing means controls the existence range of a CG object; 505 , a scenario management means; 506 , a position adjustment means including a position sensor and the like; 507 , a CG data management means; 508 , a image input means such as a camera or the like; 509 , a second image composition means; 510 , a second CG rendering means that renders a CG image from the viewpoint of the image input means 508 ; 511 , an image display means; and 512 , a viewer information management means.

[0127] An image sensed by the HMD 501 is composited with a CG image generated by the first CG rendering means 503 by the first image composition means 502 , and that composite image is displayed on the HMD 501 . An image sensed by the HMD 501 is also sent to the position adjustment means 506 , which calculates the position/direction of the HMD (i.e., the head) on the basis of that information and tracking information obtained from a position sensor or the like, and sends the calculated information to the first CG rendering means 503 .

[0128] The first CG rendering means 503 renders a CG image from the viewpoint of the HMD 501 on the basis of the position/direction information of the head obtained by the position adjustment means 506 and CG data obtained from the CG data management means 507 . The scenario management means 505 sends information required for a scene configuration to the CG data management means 507 in accordance with information obtained from the prohibited region processing means 504 , the progress of a rehearsal or action or operator's instructions, and the like.

[0129] The CG data management means 507 instructs the first or second CG rendering means 503 or 510 to render CG data in accordance with the received information. The same applies to a process for an image obtained by the image input means 508 . That is, an image sensed by the image input means 508 is composited with a CG image generated by the second CG rendering means 510 by the second image composition means 509 , and the obtained composite image is displayed on the image display means 511 . An image sensed by the image input means 508 is also sent to the position adjustment means 506 , which calculates the position/direction of the image input means (i.e., a camera) on the basis of that information and tracking information obtained from a position sensor or the like, and sends the calculated information to the second CG rendering means 510 .

[0130] The second CG rendering means 510 renders a CG image from the viewpoint of the image input means 508 on the basis of the position/direction information of the image input means 508 obtained by the position adjustment means 506 and CG data obtained from the CG data management means 507 . The position adjustment means 506 sends the calculated position/direction data of the HMD (i.e., the head) and the position/direction data of the image input means (i.e., the camera) 508 to the prohibited region processing means 504 . The prohibited region processing means 504 corrects the position of the CG object based on these data in accordance with the range where the CG object is to exist.

[0131] Information required for CG rendering, which is managed by the scenario management means 505 , is information that can manage the state of a virtual world around the player in correspondence with each scene and progress of a scenario. More specifically, that information includes the number of a CG model to be displayed, reference position/posture data, the number indicating the type of action, parameters associated with the action, and the like for each individual character to be displayed.

[0132] The scenario is managed for each scene, and the aforementioned data set is selected in accordance with the status values of each character such as characteristics, state, and the like, the action of the performer, and the like in each scene. Furthermore, character information (number, positions, states) of viewer is managed as a portion of a CG environment around the player. The characters of viewer depend on information sent from the viewer information management means 512 independently of the progress of a scenario.

[0133] Note that the see-through function of the HMD 501 can also be implemented by arranging the HMD to allow the user to see through the external field (optical see-through scheme). In this case, the aforementioned first image composition means 502 is omitted.

[0134] The position adjustment means 506 may comprise means that detects a 3D position/posture such as a mechanical encoder or the like, the aforementioned magnetic position sensor, or optical position adjustment means or that using image recognition or the like. The position adjustment of the image input means 508 and that of the HMD 501 may be done by independent position adjustment means.

[0135] FIG. 6 shows an example of a camera device using a mechanical encoder. Referring to FIG. 6 , reference numeral 601 denotes a camera; 602 , a dolly that carries the camera 601 ; and 603 , a measurement device such as a rotary encoder or the like, which is provided to each joint. The measurement device 603 can measure and output the position and direction of the camera 601 from the position of the dolly 602 .

[0136] Note that the output from the second image composition means 509 in FIG. 5 can be displayed on a viewfinder of the camera 601 . In this manner, the cameraman can make camerawork in correspondence with a virtual world.

[0137] FIG. 7 shows an example of a hand-held camera device that uses a magnetic position/direction sensor.

[0138] Referring to FIG. 7 , reference numeral 701 denotes a camera to which a magnetic receiver (measurement device) 702 is fixed. The 3D position and direction of the camera 701 are calculated based on the magnetic state measured by the receiver 702 . For example, Fastrak available from Polhemus Incorporated mentioned above can be used for this purpose.

[0139] Reference numeral 703 denotes an HMD, the position and direction of which are calculated by the same method as the camera 701 . In case of such hand-held camera device, a cameraman wears an HMD 703 (or its single-eye version), and the output from the image composition means is displayed on the HMD 703 , thus allowing camerawork in correspondence with a virtual world.

[0140] In case of a camera device with a zoom function, zoom information of a zoom lens is sent to an external processing apparatus. Furthermore, whether or not viewer information is superimposed and displayed on the camera device can be selected by the cameraman as needed.

[0141] The CG data management means 507 in FIG. 5 records 3D CG models, animation data, and image data of real images and the like, e.g., 3D animation data of a CG character. The CG data management means 507 selects a CG model or animation to be displayed in accordance with the number of a CG model, the reference position/posture data, the number indicating the type of action, parameters associated with the action, and the like for each character, which are received from the scenario management means 505 , and sets parameters of the position, posture, and the like of the selected CG model, thus changing a scene graph used in CG rendering.

[0142] The scenario management means 505 stores information such as a script, lines, comments, and the like required to help actions, and lays out viewer characters on an auditorium in accordance with the states of viewers obtained from the viewer information management means. The means 505 sends required information to the CG data management means 507 in accordance with each scene. The CG data management means 507 instructs the CG rendering means 503 and 510 to execute a rendering process according to such information.

[0143] Each scene progresses using an arbitrary user interface (mouse, keyboard, voice input, or the like).

[0144] The operation for generating a image to be displayed on the HMD 305 that the performer 303 wears in FIG. 3 will be described below using FIGS. 3 and 8 .

[0145] FIG. 8 is a flow chart showing the flow of the operation for generating a image to be displayed on the HMD 305 that the performer 303 wears in FIG. 3 . In FIG. 8 , steps S 810 to S 812 are implemented by threads, which run independently and parallelly, using a parallel processing program technique, which is widespread in the art in recent years.

[0146] Once the process shown in FIG. 8 starts, it runs as an infinite loop until it is interrupted, and another process starts after interrupt.

[0147] A process in the image generation device 308 executes an internal status update process as a process for updating status flags (the type, position, and status of an object to be displayed) for rendering a CG in accordance with an instruction obtained from the operating device 310 (step S 801 ). Head position information obtained by a head position determination process (to be described later) is fetched (step S 802 ). The latest image obtained from the video capture card is captured as a background image (step S 803 ). CG data is updated on the background image in accordance with the internal status data set in step S 801 , and a CG is rendered to have the head position set in step S 802 as the position of a virtual camera used in CG generation (step S 804 ). Finally, a CG command for displaying a composite image as the rendering result is supplied to the video card, thus displaying the composite image on the HMD (step S 805 ). After that, the flow returns to step S 801 .

[0148] Note that step S 810 is a thread for receiving instruction data from the operating device 310 via the network 311 . Step S 811 is a thread for receiving information from the position sensor 306 and determining the head position using the received information and image data obtained from the video capture card together. Furthermore, step S 812 is an image capture thread for periodically reading out image data from the video capture card.

[0149] The head position determination operation will be described below using FIGS. 3 and 9 .

[0150] FIG. 9 is a flow chart showing the flow of the head position determination operation. In FIG. 9 , step S 910 is a thread for reading data from the sensor, and step S 911 is a thread for receiving a marker position message.

[0151] Note that data from the position sensor 306 is a data communication to a normal RS232C port, and data at that port is periodically read out in step S 910 . The message in step S 911 is sent using a general network communication protocol (TCP-IP).

[0152] Once the process shown in FIG. 9 starts, it runs as an infinite loop until it is interrupted, and another process starts after interrupt.

[0153] The image generation device 308 updates the head position to a position corresponding to the latest position information obtained from the position sensor 306 (step S 901 ). Then, a specific marker image is recognized from image data obtained by the camera of the HMD 305 to acquire correction information of the head position, and direction data of the head is updated in accordance with the correction information (step S 902 ). Finally, the obtained position data (including direction) of the head is passed to step S 811 as the head position determination thread (step S 903 ). After that, the flow returns to step S 901 .

[0154] The head direction is corrected as follows. That is, a predicted value (x 0 , y 0 ) which indicates the position of a marker in an image is calculated based on the 3D position and direction of the head (viewpoint) in a world coordinate system, which are obtained from the position sensor 306 , and the 3D position of the marker. A motion vector from this predicted value (x 0 , y 0 ) to the actual marker position (x 1 , y 1 ) in the image is calculated. Finally, a value obtained by rotating the direction of the head through an angle that looks in this vector as a correction value is output as the direction of the HMD 305 .

[0155] FIG. 10 shows an example of the marker adhered in the studio 301 for position measurement. A monochrome marker may be used. However, this embodiment uses a marker having three rectangular color slips 1001 , 1002 , and 1003 with a specific size, which are laid out to have a specific positional relationship. For respective color slips 1001 , 1002 , and 1003 , arbitrary colors can be selected. Using such marker, a large number of types of markers can be stably detected.

[0156] The marker position determination operation will be described below using FIGS. 3 and 11 .

[0157] FIG. 11 is a flow chart showing the flow of the marker position determination operation. In FIG. 11 ,