DETAILED DESCRIPTION OF THE ILLUSTRATIVE EMBODIMENTS

[0040] The present invention can be used with substantially any mobile body. However, the present description proceeds with respect to an illustrative embodiment in which the invention is implemented on a motor vehicle as a driver vision assist device (or virtual mirror). FIG. 1 is a simplified block diagram of one embodiment of vision assist device 10 in accordance with the present invention. Assist device 10 includes controller 12, vehicle location system 14, geospatial database 16, ranging system 18, operator interface 20 and display 22. FIG. 1 also shows that controller 12 receives, as an input, virtual mirror characteristics 23.

[0041] In one embodiment, controller 12 is a microprocessor, microcontroller, digital computer, or other similar control device having associated memory and timing circuitry. It should be understood that controller 12 can be implemented as one or more processors or computers, and that the memory can be integrated with controller 12, or be located separately therefrom. The memory, of course, may include random access memory, read only memory, magnetic or optical disc drives, tape memory, or any other suitable computer readable medium.

[0042] Operator interface 20 is illustratively a graphical user interface (GUI) keyboard, a touch-sensitive screen, a point and click user input device (e.g. a mouse), a keypad, a voice activated interface, joystick, physical input used to move the device (e.g., the virtual mirror instrumented to measure its position and orientation relative to the drive and vehicle), or any other type of user interface suitable for receiving user commands, and providing those commands to controller 12, as well as providing a user viewable indication of operating conditions from controller 12 to the user. The operator interface may also include, for example, the steering wheel and the throttle and brake pedals suitably instrumented to detect the operator's desired control inputs of heading angle and speed. Operator interface 20 may also include, for example, a LCD screen, LEDs, a plasma display, a CRT, audible noise generators as alarms or warning systems triggered by objects in the virtual mirror, or any other suitable operator interface display or speaker unit. These can include haptic, or vibratory tactile signals applied to the driver's hands through the steering wheel, the seat or throttle pedals and brake. The alarms or warnings are triggered when objets in the virtual mirror get dangerously close to the driver's vehicle and are set based on pre-programmed thresholds.

[0043] As is described in greater detail later in the specification, vehicle location system 14 determines and provides a vehicle location signal, indicative of the vehicle location in which assist device 10 is mounted, to controller 12. Thus, vehicle location system 14 can include a global positioning system receiver (GPS receiver) such as a differential GPS receiver, an earth reference position measuring system, a dead reckoning system (such as odometery and an electronic compass), an inertial measurement unit, IMU (such as accelerometers, inclinometers, or rate gyroscopes), etc. In any case, vehicle location system 14 periodically provides a location signal to controller 12 which indicates the location of the vehicle on the surface of the earth.

[0044] Geospatial database 16 contains a digital map which digitally locates road boundaries, lane boundaries, possibly some landmarks (such as road signs, water towers, or other landmarks) and any other desired items (such as road barriers, bridges etc . . . ) and describes a precise location and attributes of those items on the surface of the earth.

[0045] The digital map stored in the geospatial database 16 contains a series of numeric location data of, for example, the center line and lane boundaries of a road on which system 10 is to be used, as well as construction data which is given by a number of shape parameters including, starting and ending points of straight paths, the center of circular sections, and starting and ending angles of circular sections. While the present system is described herein in terms of starting and ending points of circular sections it could be described in terms of starting and ending points and any curvature between those points. For example, a straight path can be characterized as a section of zero curvature. Each of these items is indicated by a parameter marker, which indicates the type of parameter it is, and has associated location data giving the precise geographic location of that point on the map.

[0046] In one embodiment, each road point of the digital map in database 16 was generated at uniform 10 meter intervals. In one embodiment, the road points represent only the centerline of the road, and the lane boundaries are calculated from that centerline point. In another embodiment, both the center line and lane boundaries are mapped. Of course, geospatial database 16 also illustratively contains the exact location data indicative of the exact geographical location of street signs and other desirable landmarks. Database 16 can be obtained by manual mapping operations or by a number of automated methods such as, for example, placing a GPS receiver on the lane stripe paint spraying nozzle or tape laying mandrel to continuously obtain locations of lane boundaries.

[0047] It should be noted that there are many possible coordinate systems that can be used to express a location on the surface of the earth, but the most common coordinate frames include the longitudinal and latitudinal angles, the state coordinate system, and the county coordinate system.

[0048] Because the earth is approximately spherical in shape, it is convenient to determine a location on the surface of the earth if the location values are expressed in terms of an angle from a reference point. Longitude and latitude are the most commonly used angles to express a location on the earth's surface or in orbits around the earth. Latitude is a measurement on a globe of location north or south of the equator, and longitude is a measurement of the location east or west of the prime meridian at Greenwich, the specifically designated imaginary north-south line that passes through both geographic poles of the earth and Greenwich, England. The combinations of meridians of longitude and parallels of latitude establishes a framework or grid by means of which exact positions can be determined in reference to the prime meridian and the equator. Many of the currently available GPS systems provide latitude and longitude values as location data.

[0049] Even though the actual landscape on the earth is a curved surface, it is recognized that land is utilized as if it is a flat surface. A Cartesian coordinate system whose axes are defined as three perpendicular vectors is usually used. Each state has its own standard coordinate system to locate points within their state boundaries. All construction and measurements are done using distance dimensions (such as meters or feet). Therefore, a curved surface on the earth needs to be converted into a flat surface and this conversion is referred to as a projection. There are many projection methods used as standards for various local areas on the earth's surface. Every projection involves some degree of distortion due to the fact that a surface of a sphere is constrained to be mapped onto a plane. One standard projection method is the Lambert Conformal Conic Projection Method. This projection method is extensively used in an ellipsoidal form for large scale mapping of regions of predominantly east-west extent, including topographic, quadrangles for many of the U.S. state plane coordinate system zones, maps in the International Map of the World series and the U.S. State Base maps. The method uses well known, and publicly available, conversion equations to calculate state coordinate values from GPS receiver longitude and latitude angle data.

[0050] Ranging system 18 is configured to detect sensed objects in the vicinity of the vehicle in which system 10 is implemented (i.e., the host vehicle), and also to detect a location (such as range, range rate and azimuth angle) of the detected objects, relative to the vehicle. Sensed objects are illustratively objects which must be monitored because they may collide with the mobile body either due to motion of the body or of the object. In one illustrative embodiment, ranging system 18 is a radar system such as the one that is commercially available from Eaton Vorad. However, ranging system 18 can also include a passive or active infrared system (which could also provide the amount of heat emitted from the detected objects) or laser based ranging system, or a directional ultrasonic system, or other similar systems. Another embodiment of system 18 is an infrared sensor calibrated to obtain a scaling factor for range, range rate and azimuth which is used for transformation to an eye coordinate system. Such a system can be used to provide the lateral or cross sectional shape of the object that is being ranged.

[0051] Inter-vehicle communication can also be used to detect other vehicles and the positions of those vehicles. Vehicle location, speed, size, weight and shape can be communicated to all other vehicles. Vehicle characteristics are encoded in the vehicle identification number (VIN) of each vehicle while vehicle dynamics and braking characteristics can be determined or inferred from the VIN as well, so communicating the VIN from vehicle-to-vehicle is one way of communicating this information. This can be done using a suitable wireless communication device.

[0052] Display 22 can, in one embodiment, be a flat panel display that receives a signal from controller 12 and displays a mirror view. Display 22 can also be another type of display that gives depth of viewing, such as one mounted in eye glasses or any other such type of display. The display 22 can be mounted in conventional mirror locations such as those shown in FIG. 3 or in other locations.

[0053] Alternatively, display 22 can include a projection unit and one or more combiners which are described in greater detail later in the specification. Briefly, the projection unit receives a video signal from controller 12 and projects video images onto one or more combiners. The projection unit illustratively includes a liquid crystal display (LCD) matrix (such as a computer screen) and a high-intensity light source similar to a conventional video projector, except that it is small so that it fits near the driver's seat space. The combiner is a partially-reflective, partially transmissive beam combiner formed of optical glass or polymer for reflecting the projected light from the projection unit back to the driver. In one embodiment, the combiner is positioned such that the driver looks through the combiner, when looking at a rear view or side view mirror, so that the driver can see both the actual reflection in the mirror, as well as the computer generated images projected onto the combiner. In one illustrative embodiment, the computer-generated images substantially overlay the actual images.

[0054] It should also be noted, however, that combiners or other similar devices or flat screen displays can be placed about the driver to cover fields of view directed to both rear view and side view mirrors. This can illustratively be implemented using a plurality of projectors or a single projector with appropriate optics to scan the projected image across the appropriate fields of view. It can also be accomplished by simply providing the controller signal to multiple flat screen or flat panel displays.

[0055] Before discussing the operation of system 10 in greater detail, it is worth pointing out that system 10 can also, in one illustrative embodiment, be varied, as desired. For example, FIG. 2 illustrates that controller 12 may actually be formed of first controller 24 and second controller 26 (or any number of controllers with processing distributed among them, as desired). In that embodiment, first controller 24 performs the primary data processing functions with respect to sensory data acquisition, and also performs database queries in the geospatial database 16. This entails obtaining velocity and heading information from the GPS receiver and correction system 28. First controller 24 also performs processing of the signal from radar ranging system 18.

[0056] FIG. 2 also illustrates that vehicle location system 14 may illustratively include a differential GPS receiver and correction system 28 as well as an auxiliary inertial measurement unit (IMU) 30 (although other approaches would also work). Second controller 26 processes signals from auxiliary IMU 30, where necessary, and handles graphics computations for providing the appropriate video signal to display 22.

[0057] In a specific illustrative embodiment, differential GPS receiver and correcting system 28 is illustratively a Novatel RT-20 differential GPS (DGPS) system with a 20-centimeter accuracy, while operating at a 5 Hz sampling rate or Trimble MS 750 with 2 cm accuracy operating at 10 Hz sampling rate. FIG. 2 also illustrates that system 10 can include optional vehicle orientation detection system 31 and driver's eye position system 32. Some of the illustrations depicted herein were generated using a camera which acts to simulate the driver's eye. Therefore, the camera is used interchangeably with the driver's eye (e.g., camera position is used interchangeably with eye position, etc.). Vehicle orientation detection system 31 detects the orientation (such as roll and pitch) of the host vehicle. The roll angle refers to the rotational orientation of the vehicle about its longitudinal axis (which is parallel to its direction of travel) The roll angle can change, for example, if the vehicle is driving over a banked road, or on uneven terrain. The pitch angle is the angle that the vehicle makes in a vertical plane along the longitudinal direction. The pitch angle becomes significant if the vehicle is climbing up or descending down a hill. Taking into account the pitch and roll angles can make the projected image more accurate, and more closely conform to the actual image seen by the driver.

[0058] Optional driver's eye (or camera) position tracking system 32 can be provided to accommodate for movements in the driver's head or eye position relative to the vehicle. Of course, in one illustrative embodiment, the actual head and eye position of the driver is not monitored. Instead, the dimensions of the cab or operator compartment of the host vehicle are taken and used, along with ergonomic data, such as the height and eye position of an operator, given the dimension of the operator compartment, and the image is projected on display 22 such that the displayed images will substantially overlie the actual mirrored images for an average operator. Specific measurements can be taken for any given operator as well, such that such a system can more closely conform to any given operator.

[0059] Alternatively, optional driver's eye position tracking system 32 is provided. System 32 tracks the position of the operator's head, and eyes, (or camera) in real time.

[0060] FIG. 3 illustrates that display 22 includes projector 40, and one or more combiners 42. FIG. 3 also illustrates an operator's head position 44 with an operator sitting in an operator compartment that includes seat 46 and that is partially defined by windshield 48 and side window 49.

[0061] Projector 40 receives the video display signal from controller 12 and projects road data onto one or more of the combiners 42 or controller 12 simply renders it on a flat panel display (in which case the combiner need not be used). In an embodiment where the image is combined with an image in an actual mirror, combiners 42 are partially reflective and partially transmissive. Therefore, the operator looks forward or sideways through combiners 42 to a virtual focal plane that coincides with the location of side view mirrors 50 or rear view mirror 51. The road data (such as lane boundaries) are projected from projector 40 in proper perspective onto combiners 42 such that the lane boundaries appear to substantially overlie those which the operator actually sees in the mirror (or would see if the mirrors had unobstructed view), in the correct perspective. Of course, additional coverage can also be provided to subsequently eliminate mirror blind spots. In this way, when the operator's view of the actual lane boundaries through the mirror becomes obstructed, the operator can still determine what an unobstructed mirror view would look like.

[0062] It should be noted that combiners 42, in one illustrative embodiment, are hinged to an upper surface or side surface or other structural part, of the operator compartment. Therefore, combiners 42 can be pivoted up and out of the view of the operator and down again when the operator desires to look through combiners 42.

[0063] Each combiner 42, while being partially reflective, is essentially a transparent, optically correct, coated glass or polymer lens. Light reaching the eyes of operator 44 is a combination of light passing through the lens (such as light from the mirror) and light reflected off of the lens from the projector. With an unobstructed mirror view, the driver actually sees two images accurately superimposed together. The image passing through the combiner 42 comes from the reflection from the mirror, while the reflected image is generated by the graphics processor portion of controller 12. The optical characteristics of combiner 42 allow the combination of elements to generate the virtual screen, or virtual focal plane. This feature results in a virtual focus on the mirror which may be outside the vehicle and ensures that the driver's eyes are not required to focus back and forth between the real image and the virtual image, thus reducing eyestrain and fatigue. Of course, where display 22 is a flat panel display, the virtual mirror display is simply rendered on the flat panel display. Flat panel displays such as those from Dimensional Technologies, of Rochester, N.Y. can be used as well as others.

[0064] Another embodiment is a helmet supported visor (or eyeglass device) on which images are projected, through which the driver can still see. Such displays might include technologies such as those available from Kaiser Electro-Optics, Inc. of Carlsbad, Calif., The MicroOptical Corporation of Westwood, Mass., Universal Display Corporation of Ewing, N.J., Microvision, Inc. of Bothell, Wash., IODisplay System LLC of Menlo Park, Calif.

[0065] FIG. 4A is a simplified flow diagram illustrating the operation of system 10. Controller 12 first receives vehicle location data in the form of heading and position. Of course, this data is received from vehicle location system 14 and is indicated by block 56 in FIG. 4A.

[0066] The heading angle of the vehicle can be estimated from the past history of the GPS location data. Alternatively, a rate gyroscope can be used to determine vehicle heading as well. As noted initially, though heading angle estimation by successive differentiation of GPS data can be used, any other suitable method to measure an absolute heading angle can be used as well, such as a magnetometer (electronic compass) or an inertial measurement unit.

[0067] Once controller 12 has received the vehicle location data, controller 12 also optionally receives head or eye location information, as well as optional vehicle orientation data. As briefly discussed above, the vehicle orientation information can be obtained from a roll rate gyroscope to obtain the roll angle, and a tilt sensor (such as an accelerometer) to obtain the pitch angle as well as a yaw rate sensor to obtain yaw angle. Obtaining the head or eye (or camera) location data and the vehicle orientation data are illustrated by optional blocks 72 and 74 in FIG. 4A.

[0068] A number of transformation matrices are constructed, as described in greater detail below, from the location and heading angle of the moving vehicle, and from the optional driver's head or eye data and vehicle orientation data, where that data is sensed. The location data is converted into a local coordinate measurement using the transformation matrices, and is then fed into the perspective projection routines to calculate and draw the road shape and sensed object icons in the computer's graphic memory. The road shape and sensed object icons are then projected as a virtual view in the driver's visual field, as illustrated in FIG. 3 above.

[0069] In any case, after the vehicle position data is received, the ranging information from ranging system 18 is also received by controller 12 (shown in FIG. 2). This is indicated by block 76 in FIG. 4A. The ranging data illustratively indicates the presence and location of sensed objects around the vehicle. For example, the radar ranging system 18 developed and available from Eaton Vorad, or Delphi, or Celsius Tech, or other vendors provides a signal indicative of the presence of a radar sensed object, its range, its range rate and the azimuth angle of that sensed object with respect to the radar apparatus.

[0070] Based on the position signal, controller 12 queries the digital road map in geospatial database 16 and extracts local road data. The local road data provides information with respect to road boundaries as seen by the operator at the position of the vehicle, and also other potential radar sensed objects, such as road signs, bridge abutments, or underpasses, guard rails, road barriers, landmarks such as water towers, etc. Accessing geospatial database 16 (which can be stored on the vehicle and receive periodic updates or can be stored remotely and accessed wirelessly) is indicated by block 78 in FIG. 4A.

[0071] Controller 12 determines whether the sensed objects indicated by sensed object data 83 are expected sensed objects. Controller 12 does this by examining the information in geospatial database 16. In other words, if the sensed objects correspond to road signs, road barriers, bridges, or other information which would provide a radar return to ranging system 18, but which is expected because it is mapped into database 16 and does not need to be brought to the attention of the driver, that information can optionally be filtered out such that the driver is not alerted to every single possible item on the road which would provide a radar return. Certain objects may a priori be programmed to be brought to the attention of the driver. Such items may be guard rails, bridge abutments, etc . . . and the filtering can be selective, as desired. If, for example, the driver were to exit the roadway, all filtering can be turned off so all objects are brought to the driver's attention. The driver can change filtering based on substantially any predetermined filtering criteria, such as distance from the road or driver, location relative to the road or the driver, whether the objects are moving or stationary, or substantially any other criteria. Such criteria can be invoked by the user through the user interface, or they can be pre-programmed into controller 12.

[0072] However, where the geospatial database does not indicate an expected sensed object in the present location, then the sensed object information is determined to correspond to an unexpected sensed object, such as a moving vehicle behind or to the side of the host vehicle, such as a stalled car or a pedestrian on the side of the road, or such as some other transitory sensed object that has not been mapped to the geospatial database as a permanent, or expected. It has been found that if all expected sensed objects are brought to the operator's attention, this substantially amounts to noise so that when real sensed objects are brought to the operator's attention, they are not as readily perceived by the operator. Therefore, optional filtering of sensed objects not posing a threat to the driver can be performed as is illustrated by block 80 in FIG. 4A.

[0073] Once such sensed objects have been filtered, the transformation is performed using the transformation matrices. The result of the transformation provides the road boundary data, as well as the sensed object data, seen from the driver's eye perspective through the mirror. The road boundary and sensed object data is output, as illustrated by block 82 in FIG. 4A. Based on the output road and sensed object data, the road and sensed object shapes are generated by processor 12 for projection in the proper perspective. Generation of road and sensed object shapes and the perspective projection is illustrated by block 84 in FIG. 4A.

[0074] It should also be noted that the actual image projected is clipped such that it only includes that part of the road which would be visible by the operator with an unobstructed mirror view. Clipping is described in greater detail below.

[0075] FIG. 4B is a more detailed flow diagram illustrating how the display of the virtual mirror is generated (as indicated by block 84 in FIG. 4A). In one illustrative embodiment, the information contained in the geospatial database 16 represents lane boundaries, guard rails, shoulders, traffic islands, traffic signs, and other such road “furniture” as a linked list of lines. The vehicle wire frames associated with the host vehicle as well as sensed object vehicles, are bounded by boxes that are also stored as a linked list of lines.

[0076] In order to generate the virtual mirror display, controller 12 first selects line data indicative of a road line, a sensed object bounding box, etc. This is indicated by block 86 in FIG. 4B. Next, controller 12 optionally determines whether the line data should be trivially rejected. That is, the scene displayed by the virtual mirror includes a plurality of lines. Each line must be transformed, clipped and then projected to form the virtual mirror display. The transformation, as is discussed in greater detail below, involves multiplying end points of the lines by a transformation matrix. The line is clipped against six faces of a perspective canonical volume and is projected by multiplying the end points of the projection matrix followed by homogeneous division. It can thus be seen that the computations necessary to display a line can be quite time consuming to perform. Therefore, if the line which has been selected at block 86 lies far away from the host vehicle, so that it is too far away to substantially affect the final rendered view, then the line can be rejected before the extensive computations are undertaken. Optional trivial rejection of the line is indicated by block 88 in FIG. 4B.

[0077] If the line is trivially rejected, then processing returns to block 86 where new line data is selected. However, if the line is not trivially rejected, controller 12 first performs a mirror transformation on the line data as indicated by block 90. The mirror transformation adapts the mirror optics to a conventional forward transformation. In other words, the mirror transformation performs the basic reflection transformation that transforms a vertex given in world coordinates to its reflected image. The mirror transformation also includes normalization, clipping and projection which are described in greater detail below.

[0078] The lines which will be visible on the mirror display after the mirror transformation is performed are output and controller 12 next performs a stencil transformation as indicated by block 92. The stencil transformation is a two-dimensional (2D) space to 2D space transformation. It transforms the 2D projected view of the mirror transformation to the 2D desired shape of the generated virtual mirror.

[0079] Once the stencil transformation is performed, the points in the virtual mirror display can be treated like ordinary points in the forward field of view. That is, they can undergo the entire sequence of forward transformation and be mapped for viewing. The forward transformation enables perspective projection so that a three-dimensional world can be represented on a two-dimensional space, while retaining the feeling of “depth” in the scene. Thus, parallel lines seem to meet at a point called the vanishing point. Objects appear larger if they are closer to the point of view than if they are further away. This is referred to herein as “perspective foreshortening”. Performance of the forward transformation is indicated by block 94 in FIG. 4B.

[0080] After the forward transformation has been performed, the line is rendered on the virtual mirror display based on the transformed data. Rendering the line is illustrated by block 96 in FIG. 4B.

[0081] Blocks 88-96 in FIG. 4B will now be discussed in greater detail. The forward transformation indicated by block 94 is the final stage in the algorithmic sequence, but forms a basic stage in the development of mathematical treatment of the virtual mirror display. Therefore, the forward transformation will be discussed first. Rendering and displaying is discussed in conjunction with the forward transformation. The next logical stage in mathematical development of the present invention is the mirror transformation illustrated in block 90 of FIG. 4B. The mirror transformation is thus discussed next. The third stage in the mathematical development of the virtual mirror is the stencil transformation which bounds the mirror and forward transformations. This will thus be discussed after the mirror transformation. Finally, the optional trivial rejection illustrated by block 88 is discussed in greater detail.

Forward Transformation

[0082] As briefly discussed above, the aim of the forward transformation is to achieve perspective projection so that the 3D world can be projected on to a 2D space. The procedure to accomplish this involves projecting all vertices in the world coordinate system onto an infinite 2D plane referred to as the projection plane by straight lines referred to as projectors. For perspective projection, all projectors meet at a single point referred to as the projection reference point (PRP). The view reference coordinate (VRC) system is an arbitrary coordinate system defined so that the projection plane becomes the XY plane of the view reference coordinate system, and the projection reference point (PRP) is a positive Z coordinate in this coordinate system. The view reference coordinate system is given by the homogeneous transformation matrix X_vrc relative to the world coordinate system.

[0083] The term window is a rectangular area on the projection plane. The projection plane and the window lie in the Z=0 plane of the view reference coordinate system. The window is not to be confused with desktop windows or computer screen windows launched by window managers. The window limits the view seen from the camera position (PRP). The edges of the window are parallel to the X and Y axes of the view reference coordinate system. Therefore, it is possible to define a window by the minimum and maximum limits of the X and Y coordinates of the VRC:

WINDOW=[WNDLEFT WNDRIGHT WINDBOTTOM WNDTOP], Eq. 1

[0084] where the WINDOW is represented by a four element vector,

[0085] WNDLEFT=the minimum x limit;

[0086] WNDRIGHT=the maximum x limit;

[0087] WNDBOTTOM=the minimum y limit; and

[0088] WNDTOP=the maximum y limit.

[0089] FIG. 5 better illustrates a number of the terms defined above. The infinite plane 98 corresponds to the projection plane. Area 100 represents the window through which the operator is viewing. Element 102 corresponds to the PRP. Therefore, all lines, such as line 104 (having end points A and B) lying inside a viewing volume 106 are projected by means of straight lines 108 (the projectors) on the two-dimensional projection plane 98. All of the projectors meet at the PRP 102. The field of view visible from the PRP is referred to as the viewing volume 106. The cross section of the viewing volume 106 and projection plane 98 is referred to as window 100. The mathematical transformations discussed below map the projected contents of window 100 onto a two-dimensional rectangular region referred to as viewport 110. FIG. 5 also illustrates the elements of Equation 1 above.

[0090] Four projectors 112, 114, 116 and 117 are drawn from the PRP 102 passing through four corners of the window 100 to form an infinitely high four sided pyramid. The pyramid is truncated by two planes parallel to the projection plane 98, one near the apex and one near the base, to form a frustrum. The top plane of the frustrum is referred to as the near clipping plane 118 and the base plane of the frustrum is referred to as the far clipping plane 120. This frustrum defines the viewing volume 106.

[0091] By the very manner in which the frustrum was generated, it is evident that the window 100 forms a cross section of the viewing volume 106. The near clipping plane 118 is parallel to the window 100 and given by the equation z=F and the far clipping plane 120 is also parallel to the window 100 and given by the equation z=B. In the discussions below, the near clipping plane 118 has been chosen to be located at z=0, so that the near clipping plane 118 coincides with window 100.

[0092] The viewport 110 is a rectangular area on a computer screen (such as an LCD matrix) onto which the final scene is rendered. The vertices projected onto the window 100 are ultimately mapped into the viewport 110. The viewport 110 is VIEWPORT_W pixels wide and VIEWPORT_H pixels high. This means that viewport coordinates are integers that range from 0 to VIEWPORT_W−1 in the X direction and 0 to VIEWPORT_H−1 in the Y direction. Unless specified otherwise, illustratively, VIEWPORT_W=640 and VIEWPORT_H=400.

[0093] All coordinate systems are defined in terms of 4×4 homogenous transformation matrices. The first, second and third columns are unit vectors that give the directions of the X, Y and Z axes of the coordinate system with respect to the axes of the World Coordinate System. The fourth column gives the position of the origin of the coordinate system with respect to the World Coordinate System. The fourth row of the homogeneous transformation matrices is always [0 0 0 1].

[0094] The notation used here to denote transformation matrices and their inverses is described as follows:

[0095] Homogeneous transformation matrices are 4×4 matrices that describe 3 dimensional rotation, translation and scaling. These matrices transform one coordinate system to another coordinate system as follows:

[0096] Translation to point {x, y, z}; x, y and z in meters: 1

[0097] Rotation about the X axis through an angle of φ radians: 2

[0098] Rotation about the Y axis through an angle of φ radians: 3

[0099] Rotation about the Z axis through an angle of φ radians: 4

[0100] Scaling about the X, Y and Z axes by a factor a, b and c respectively: 5

[0101] Any number of translation, rotation and scaling matrices can be multiplied together to form a composite or aggregate matrix. The fourth row is always [0 0 0 1] for the composite matrix.

[0102] Given that a homogeneous transformation matrix transforms coordinate system A to coordinate system B, the matrix inverse transforms coordinate system B to coordinate system A.

[0103] Inverse of translation matrix:

Trans⁻¹(x, y, z)=Trans(−x,−y,−z) Eq. 7

[0104] Pure rotational matrices are orthonormal, and hence their inverses are just their transposes.

Rotz⁻¹(φ )=Rotz(−φ)=Rotz^T(φ) Eq. 8

[0105] Since the scaling matrix is a diagonal matrix, its inverse is a diagonal matrix with the elements of the diagonal being reciprocals of the elements in the original matrix. 6

[0106] Composite matrices formed by multiplying any number of translating, rotational and scaling matrices have a simple inverse. 7

[0107] The 3×3 principal sub-matrix of the inverse is the transpose of the 3×3 principal sub-matrix of the original matrix. The first three elements of the fourth column of the inverse are the negatives of the dot products of the fourth column of the original matrix with the first three columns of the original matrix. This avoids the necessity to use generic but computationally expensive methods for computing the matrix inverse like the known Gauss-Jordan elimination.

[0108] For purposes of this discussion, the World Coordinate System (WCS) is a fixed reference coordinate system with respect to which all other coordinate systems are defined. The host coordinate system is given by the homogeneous transformation matrix X_GPS with respect to the World Coordinate System.

[0109] FIG. 6 is a depiction of a plan view of host vehicle 122, and FIG. 7 depicts host vehicle 122 from the front. Similar items to those in previous figures are similarly numbered. The origin of X_GPS is the GPS antenna 28 on the host vehicle 122. The X axis points towards the right side 124 of the host vehicle 122, the Y axis points along the longitudinal axis 125 of the host vehicle 122 towards the front and the Z axis points vertically upward.

[0110] Given the following:

[0111] 1. {^WCSx, ^WCSy} are the coordinates of the host vehicle GPS antenna 28 in the WCS (in meters).

[0112] 2. The GPS antenna 28 is at a height of GPSHeight meters above the ground (z=0 plane of the WCS). In one example embodiment, GPSHeight=3.4036 m. (134″)

[0113] 3. θ₂ is the angle (in radians) between the longitudinal axis 125 of the host vehicle 122 and east direction (the X axis of the WCS). leads to:

X_GPS=Trans(^WCS_x, ^WCS_y ,GPSHeig ht)·Rotz(0₂−π/2) Eq. 12

[0114] Measurements of the eye position (or camera position) and field of view need to be taken with respect to some coordinate system located inside the host vehicle cabin because the driver's eye (or camera) is located there. Therefore, we introduce another coordinate system called the windshield coordinate system.

[0115] The windshield coordinate system is given by the homogeneous transformation matrix X_SCS with respect to World Coordinate System. The origin of X_SCS is the midpoint of the lower edge of the windshield as shown in FIG. 7. The windshield lies in the XY plane of the X_SCS. The windshield plane is assumed to be perpendicular to the ground (z=0 plane of the WCS) in order to simplify measurements. The X axis points right, the Y axis point vertically up and the Z axis of the X_SCS points inward into the vehicle (towards the driver's eye).

[0116] This means that if X_GPS is rotated through an angle of π/2 about the X axis and then translated to the X_SCS origin, then X_GPS will coincide with X_SCS. Hence, given the following:

[0117] 1. ^SCSgps_x=X coordinate of GPS antenna in X_SCS coordinates=0

[0118] 2. ^SCSgps_y=Y coordinate of GPS antenna in X_SCS coordinates=1.3716 m (54″)

[0119] 3. ^SCSgps_z=Z coordinate of GPS antenna in X_SCS coordinates=1.143 m (45″) this leads to:

X_SCS=X_GPS·Rotx(π/2)·Trans(−SCSgps_—x,−^SCSgps _—y,−< /highlight>^SCS< italic>gps_—z) Eq. 13

[0120] The following parameters are required to calculate the forward transformation. The location of the camera is conveniently measured in the X_SCS coordinate system. Given the following:

[0121] 1. PRPx=X coordinate of camera lens in X_SCS coordinates

[0122] 2. PRPy=Y coordinate of camera lens in X_SCS coordinates

[0123] 3. PRPz=Z coordinate of camera lens in X_SCS coordinates

[0124] The camera position in X_SCS is:

^SCSPRP=[PRPx PRPy PRPz 1]′ Eq. 14

[0125] The window 100 hence lies on the windshield, i.e. the XY plane of X_SCS. The hatched rectangle in FIG. 7 is the window. Given that,

[0126] 1. WNDLEFT=Minimum X coordinate on the windshield corresponding to the window;

[0127] 2. WNDRIGHT=Maximum X coordinate on the windshield corresponding to the window;

[0128] 3. WNDBOTTOM=Minimum Y coordinate on the windshield corresponding to the window;

[0129] 4. WNDTOP=Maximum Y coordinate on the windshield corresponding to the window; where all coordinates are X_SCS coordinates, the window is given by Eq. 1.

[0130] The near clipping plane 118 (z=F), coincides with the window (z=0). Hence, all items between the eye and the windshield will be clipped out. Thus,

F=< /highlight>0 Eq. 15

[0131] The viewing volume 106 illustratively has a height of 1 km, i.e. the forward view will include all vertices within 1 km of the windshield. Therefore the far clipping plane 120 (z=B) is at a distance of 1 km from the window (z=0). Thus,

B=−1000 Eq. 16

[0132] Assuming that the eye (or camera) is in a fixed position and does not change orientation, X_VRC=X_{SCS< /sub>}. However any subsequent change in orientation changes the projection plane 98 as well. The windshield no longer remains the projection plane 98. The changes in orientation are accommodated by two angles, the yaw angle camyaw and roll angle camroll, given in degrees. 8

[0133] The above transformation can readily be recognized as a similarity transformation. The yaw and roll take place in a temporary coordinate system defined at the camera lens such that the axes of this coordinate system are parallel to X_SCS. This coordinate system is obtained by translating to the camera position. The coordinate system is then rolled about the host longitudinal axis 125 and yawed about the vertical axis and then translated back to the same vector that the X_SCS initially made with the camera lens. This allows the camera (or drivers head) to roll and yaw, thereby removing the restraint that the projection plane 98 be the windshield. However, the projection plane is still located at the same distance from the camera. It is important to note that the window stays the same because it is defined with respect to the X_VRC coordinate system and lies in the XY plane of the X_VRC coordinate system and so is immune to changes in the X_VRC coordinate system itself. Also note that for zero yaw and zero roll, 9