DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

[0022] While the methods and systems described herein may be applied to any object to be examined, the preferred embodiment to be described is the examination of an organ in the human body, specifically the colon. The colon is long and twisted, which makes it especially suited for a virtual examination saving the patient monetary expense as well as the discomfort and increased hazard of a physical probe. Other examples of organs that can be examined include the lungs, stomach and portions of the gastrointestinal system, the heart and blood vessels.

[0023] As shown in FIG. 1, a method for performing a virtual examination of an object such as a colon is indicated generally by the reference numeral 100. The method 100 illustrates the steps necessary to perform a virtual colonoscopy using volume visualization techniques. Step 101 prepares the colon to be scanned in order to be viewed for examination if required by either the doctor or the particular scanning instrument. This preparation could include cleansing the colon with a “cocktail” or liquid, which enters the colon after being orally ingested and passed through the stomach. The cocktail forces the patient to expel waste material that is present in the colon. One example of a substance used is Golytcly. Additionally, in the case of the colon, air or carbon dioxide can be forced into the colon in order to expand it to make the colon easier to scan and examine. This is accomplished with a small tube placed in the rectum with approximately 1,000 cc of air pumped into the colon to distend the colon. Depending upon the type of scanner used, it may be necessary for the patient to drink a contrast substance such as barium to coat any unexpunged stool in order to distinguish the waste in the colon from the colon walls themselves. Alternatively, the method for virtually examining the colon can remove the virtual waste prior to or during the virtual examination as explained later in this specification. Step 101 does not need to be performed in all examinations as indicated by the dashed line in FIG. 1.

[0024] Step 103 scans the organ that is to be examined. The scanner can be an apparatus well known in the art, such as a spiral CT-scanner for scanning a colon or a Zenith MRI machine for scanning a lung labeled with xenon gas, for example. The scanner must be able to take multiple images from different positions around the body during suspended respiration, in order to produce the data necessary for the volume Visualization. For example, data can be acquired using a GE/CTI spiral mode scanner operating in a helical mode of 5 mm, 1.5-2.0:1 pitch, reconstructed in 1 mm slices, where the pitch is adjusted based upon the patient's height in a known manner. A routine imaging protocol of 120 kVp and 200-280 ma can be utilized for this operation. The data can be acquired and reconstructed as 1 mm thick slice images having an array size of 512×512 pixels in the field of view, which varies from 34 to 40 cm depending on the patient's size. The number of such slices generally varies under these conditions from 300 to 450, depending on the patient's height. The image data set is converted to volume elements or voxels.

[0025] An example of a single CT-image would use an X-ray beam of 5 mm width, 1:1 to 2:1 pitch, with a 40 cm field-of-view being performed from the top of the splenic flexure of the colon to the rectum.

[0026] Discrete data representations of the object can be produced by other methods besides scanning. Voxel data representing an object can be derived from a geometric model by techniques described in U.S. Pat. No. 5,038,302 entitled “Method of Connecting Continuous Three-Dimensional Geometrical Representations into Discrete Three-Dimensional Voxel-Based Representations Within a Three-Dimensional Voxel-Based System” by Kaufman, issued Aug. 8, 1991, filed Jul. 26,1988, which is hereby incorporated by reference in its entirety. Additionally, data can be produced by a computer model of an image, which can be converted to three-dimensional voxels and explored in accordance with this disclosure.

[0027] Step 104 converts the scanned images into three-dimensional volume elements (“voxels”). In a preferred embodiment for examining a colon, the scan data is reformatted into 5 mm thick slices at increments of 1 mm or 2.5 mm and reconstructed in 1 mm slices, with each slice represented as a matrix of 512 by 512 pixels. By doing this, voxels of approximately 1 cubic mm are created. Thus a large number of 2D slices are generated depending upon the length of the scan. The set of 2D slices is then reconstructed to 3D voxels. The conversion process of 2D images from the scanner into 3D voxels can either be performed by the scanning machine itself or by a separate machine such as a computer implementing techniques that are well known in the art (see, e.g., U.S. Pat. No. 4,985,856 entitled “Method and Apparatus for Storing, Accessing, and Processing Voxel-based Data” by Kaufman et al.; issued Jan. 15, 1991, filed Nov. 11, 1988; which is hereby incorporated by reference in its entirety).

[0028] Step 105 allows the operator to define the portion of the selected organ to be examined. A physician may be interested in a particular section of the colon likely to develop polyps. The physician can view a two dimensional slice overview map to indicate the section to be examined. A starting point and finishing point of a path to be viewed can be indicated by the physician/operator. A conventional computer and computer interface (e.g., keyboard, mouse or spaceball) can be used to designate the portion of the colon that is to be inspected. A grid system with coordinates can be used for keyboard entry or the physician/operator can “click” on the desired points. The entire image of the colon can also be viewed if desired.

[0029] Step 107 performs the planned or guided navigation operation of the virtual organ being examined. Performing a guided navigation operation is defined as navigating through an environment along a predefined or automatically predetermined flight path, which can be manually adjusted by an operator at any time. After the scan data has been converted to 3D voxels, the inside of the organ is traversed from the selected start to the selected finishing point. The virtual examination is modeled on having a tiny viewpoint or “camera” traveling through the virtual space with a view direction or “lens” pointing towards the finishing point. The guided navigation technique provides a level of interaction with the camera, so that the camera can navigate through a virtual environment automatically in the case of no operator interaction, and at the same time, allow the operator to manipulate the camera when necessary. The preferred embodiment of achieving guided navigation is to use a physically based camera model that employs potential fields to control the movement of the camera, as is further detailed with respect to FIG. 2.

[0030] Step 109, which can be performed concurrently with step 107, displays the inside of the organ from the viewpoint of the camera model along the selected pathway of the guided navigation operation. Three-dimensional displays can be generated using techniques well known in the art such as the marching cubes technique, for example. In order to produce a real time display of the colon, a technique is used that reduces the vast number of data computations necessary for the display of the virtual organ.

[0031] The method described in FIG. 1 can also be applied to scanning multiple organs in a body at the same time. For example, a patient may be examined for cancerous growths in both the colon and lungs. The method of FIG. 1 would be modified to scan all the areas of interest in step 103 and to select the current organ to be examined in step 105. For example the physician/operator may initially select the colon to virtually explore and later explore the lung. Alternatively, two different doctors with different specialties may virtually explore different scanned organs relating to their respective specialties. Following step 109, the next organ to be examined is selected and its portion will be defined and explored. This continues until all organs that need examination have been processed.

[0032] The steps described in conjunction with FIG. 1 can also be applied to the exploration of any object that can be represented by volume elements. For example, an architectural structure or inanimate object can be represented and explored in the same manner.

[0033] Turning to FIG. 2, a “submarine camera” model that performs guided navigation in a virtual organ is indicated generally by the reference numeral 200. The model 200 depicts a viewpoint control model that performs the guided navigation technique of step 107. When there is no operator control during guided navigation, the default navigation is similar to that of planned navigation that automatically directs the camera along a flight path from one selected end of the colon to another. During the planned navigation phase, the camera stays at the center of the colon for obtaining better views of the colonic surface. When an interesting region is encountered, the operator of the virtual camera using guided navigation can interactively bring the camera close to a specific region and direct the motion and angle of the camera to study the interesting area in detail, without unwillingly colliding with the walls of the colon. The operator can control the camera with a standard interface device such as a keyboard, mouse or nonstandard device such as a spaceball. In order to fully operate a camera in a virtual environment, six degrees of freedom for the camera are required. The camera must be able to move in the horizontal, vertical, and depth or Z direction (axes 217), as well as being able to rotate in another three degrees of freedom (axes 219) to allow the camera to move and scan all sides and angles of a virtual environment.

[0034] Methods for computing a centerline inside the area of interest are well known in the art (see, e.g., U.S. Pat. No. 5,971,767 entitled “SYSTEM AND METHOD FOR PERFORMING A THREE-DIMENSIONAL VIRTUAL EXAMINATION” by Kaufman et al.; issued Oct. 26, 1999 and incorporated by reference herein in its entirety).

[0035] Referring to FIG. 3, a two dimensional cross-section of a volumetric colon containing a flight path is indicated generally by the reference numeral 300. The cross-section 300 includes the final flight path for the camera model down the center of the colon, as indicated by “x”s, and at least one starting location 301 or 303 near one end of the colon.

[0036] Turning now to FIG. 4, a system used to perform a virtual examination of a human organ in accordance with the disclosure is indicated generally by the reference numeral 400. The system 400 is for performing the virtual examination of an object such as a human organ using the techniques described herein. A patient 401 lays on a platform 402, while a scanning device 405 scans the area that contains the organ or organs to be examined. The scanning device 405 contains a scanning portion 403 that takes images of the patient and an electronics portion 406. The electronics portion 406 includes an interface 407, a central processing unit 409, a memory 411 for temporarily storing the scanning data, and a second interface 413 for sending data to a virtual navigation platform or terminal 416. The interfaces 407 and 413 may be included in a single interface component or may be the same component. The components in the portion 406 are connected together with conventional connectors.

[0037] In the system 400, the data provided from the scanning portion 403 of the device 405 is transferred to unit 409 for processing and is stored in memory 411. The central processing unit 409 converts the scanned 2D data to 3D voxel data and stores the results in another portion of the memory 411. Alternatively, the converted data may be directly sent to the interface unit 413 to be transferred to the virtual navigation terminal 416. The conversion of the 2D data could also take place at the virtual navigation terminal 416 after being transmitted from the interface 413. In the preferred embodiment, the converted data is transmitted over a carrier 414 to the virtual navigation terminal 416 in order for an operator to perform the virtual examination. The data may also be transported in other conventional ways, such as storing the data on a storage medium and physically transporting it to terminal 416 or by using satellite transmissions, for example.

[0038] The scanned data need not be converted to its 3D representation until the visualization-rendering engine requires it to be in 3D form. This saves computational steps and memory storage space.

[0039] The virtual navigation terminal 416 includes a screen for viewing the virtual organ or other scanned image, an electronics portion 415 and an interface control 419 such as a keyboard, mouse or spaceball. The electronics portion 415 includes an interface port 421, a central processing unit 423, optional components 427 for running the terminal and a memory 425. The components in the terminal 416 are connected together with conventional connectors. The converted voxel data is received in the interface port 421 and stored in the memory 425. The central processing unit 423 then assembles the 3D voxels into a virtual representation and runs the submarine camera model as described for FIG. 2 to perform the virtual examination. As the submarine camera travels through the virtual organ, the visibility technique is used to compute only those areas that are visible from the virtual camera, and display them on the screen 417. A graphics accelerator can also be used in generating the representations. The operator can use the interface device 419 to indicate which portion of the scanned body is desired to be explored. The interface device 419 can further be used to control and move the submarine camera as desired as detailed for FIG. 2. The terminal portion 415 can be, for example, the Cube-4 dedicated system box, generally available from the Department of Computer Science at the State University of New York at Stony Brook.

[0040] The scanning device 405 and terminal 416, or parts thereof, can be part of the same unit. A single platform would be used to receive the scan image data, connect it to 3D voxels if necessary and perform the guided navigation.

[0041] An important feature in system 400 is that the virtual organ can be examined at a later time without the presence of the patient. Additionally, the virtual examination could take place while the patient is being scanned. The scan data can also be sent to multiple terminals, which would allow more than one doctor to view the inside of the organ simultaneously. Thus a doctor in New York could be looking at the same portion of a patient's organ at the same time with a doctor in California while discussing the case. Alternatively, the data can be viewed at different times. Two or more doctors could perform their own examination of the same data in a difficult case. Multiple virtual navigation terminals could be used to view the same scan data. By reproducing the organ as a virtual organ with a discrete set of data, there are a multitude of benefits in areas such as accuracy, cost and possible data manipulations.

[0042] Some of the applicable techniques may be further enhanced in virtual colonoscopy applications through the use of a number of additional techniques that are described in U.S. Pat. No. 6,343,936 entitled “SYSTEM AND METHOD FOR PERFORMING A THREE-DIMENSIONAL VIRTUAL EXAMINATION, NAVIGATION AND VISUALIZATION” by Kaufman et al.; issued Feb. 7, 2002, which is incorporated herein by reference in its entirety. These improvements, described briefly below, include improved colon cleansing, volume rendering, additional fly-path determination techniques, and alternative hardware embodiments.

[0043] An improved electronic colon cleansing technique employs modified bowel preparation operations followed by image segmentation operations, such that fluid and stool remaining in the colon during a computed tomographic (“CT”) or magnetic resonance imaging (“MRI”) scan can be detected and removed from the virtual colonoscopy images. Through the use of such techniques, conventional physical washing of the colon, and its associated inconvenience and discomfort, is minimized or completely avoided.

[0044] In addition to image segmentation and texture mapping, volume-rendering techniques may be used in connection with virtual colonoscopy procedures to further enhance the fidelity of the resulting image. Methods for volume rendering are well known to those of ordinary skill in the pertinent art.

[0045] Referring to FIG. 5, an exemplary representation of a colon and accompanying flight-path generated according to an embodiment of the present disclosure is indicated generally by the reference numeral 500. The representation 500 depicts a human colon 502 showing a centerline flight path 504. As the operator travels through the virtual organ along this flight path, two-dimensional images of the current position are displayed.

[0046] As shown in FIG. 6, an exemplary display of a two-dimensional slice of scan data according to an embodiment of the present disclosure is indicated generally by the reference numeral 600. The slice 600 is shown while advancing along the flight path, and the operator interface displays the virtual organ along with the slice for the current “z” coordinate and pans the image of that slice so that the current “x, y” position are in the center of the image. Thus, in this arrangement, the two-dimensional slices are axial slices, where convention has the z-axis pointing towards the head. However, once the scan data has been converted into a three-dimensional volume, two-dimensional slices oriented on other planes can be generated and viewed as well. For example, the two-dimensional images displayed to the operator can be oriented on the sagittal plane, the coronal plane, or perpendicular to the flight path.

[0047] Turning to FIG. 7, the colon of FIG. 5 intersected by a plane oriented perpendicular to the flight-path is indicated generally by the reference numeral 700. The intersected colon 700, for example, depicts a plane 704 indicating a two-dimensional image perpendicular to the flight path 702 through the colon 703.

[0048] Referring to FIG. 8, an exemplary operator interface screen according to embodiments of the present disclosure is indicated generally by the reference numeral 2100. The screen 2100 includes a number of sub-windows that simultaneously provide an operator with graphical information from a number of different perspectives. The center sub-window 2104 displays inside the virtual organ. An arrow or marker 2105 helps orient the operator along the projected flight path. A complete view of this flight path, along with the entire organ is depicted in sub-window 2102. Operator controls 2108 are near the bottom of the screen 2100 and are useful to control the travel through the virtual organ. The rendering of the virtual organ, as well as the control of flight through the organ, have been described earlier and are not repeated here. Four other sub-windows are shown 2106, 2114, 2112 and 2110 that provide two-dimensional images along the perpendicular plane, axial plane, sagittal plane and coronal plane, respectively. The displayed two-dimensional image is based on the current position along the flight path through the virtual organ. Each of these windows can include a marker, for example 2115, 2113 and 2111, to help orient the operator along the flight path.

[0049] Furthermore, each window 2106, 2114, 2112, and 2110 has a respective control for scrolling through two-dimensional images such as scroll bar 2115. Accordingly, the operator can traverse the flight path manually, in either direction, using this scroll bar.

[0050] The screen 2100 is exemplary in nature and a skilled artisan would recognize many equivalent alternatives within the scope of the present disclosure. For example, not all sub-windows 2106, 2114, 2112 and 2110 need to be displayed.

[0051] Turning to FIG. 9, a system embodiment based on a personal computer bus architecture is indicated generally by the reference numeral 900. The system 900 includes an alternate hardware embodiment suitable for deployment on a personal computer (“PC”), as illustrated. The system 900 includes a processor 910 that preferably takes the form of a high speed, multitasking processor, such as, for example, a Pentium III processor operating at a clock speed in excess of 400 MHZ. The processor 910 is coupled to a conventional bus structure 920 that provides for high-speed parallel data transfer. Also coupled to the bus structure 920 are a main memory 930, a graphics board 940, and a volume rendering board 950. The graphics board 940 is preferably one that can perform texture mapping, such as, for example, a Diamond Viper v770 Ultra board manufactured by Diamond Multimedia Systems. The volume rendering board 950 can take the form of the VolumePro board from Mitsubishi Electric, for example, which is based on U.S. Pat. Nos. 5,760,781 and 5,847,711, which are hereby incorporated by reference in their entirety. A display device 945, such as a conventional SVGA or RGB monitor, is operably coupled to the graphics board 940 for displaying the image data. A scanner interface board 960 is also provided for receiving data from an imaging scanner, such as an MRI or CT scanner, for example, and transmitting such data to the bus structure 920. The scanner interface board 960 may be an application specific interface product for a selected imaging scanner or can take the form of a general-purpose input/output card. The PC based system 900 will generally include an I/O interface 970 for coupling I/O devices 980, such as a keyboard, digital pointer or mouse, and the like, to the processor 910. Alternatively, the I/O interface can be coupled to the processor 910 via the bus 920.

[0052] Embodiments of the present disclosure provide a user interface displaying both two-dimensional and three-dimensional data. Organs within the body are, by nature, three-dimensional. Conventional medical imaging devices, however, as explained herein, create stacks of two-dimensional images when acquiring scan data. Radiologists and other specialists, therefore, have historically been trained to review and analyze these two-dimensional images. As a result, most doctors are comfortable viewing two-dimensional images even if three-dimensional reconstructions or virtualizations are available.

[0053] However, many organs are not simple convex objects but, instead, can be tortuous or have many branches. While a doctor may be comfortable analyzing two-dimensional images, performing navigation through complex organs is very difficult using merely two-dimensional images. Navigating using the two-dimensional images would include manually scrolling through the images to move in the “z” direction (along the major axis of the body) and panning the images to move in the “x” and “y” direction. In this manner an operator can traverse the organ looking for areas of interest.

[0054] On the other hand, three-dimensional flight paths, as described herein, are intuitive, efficient tools to virtually travel through volumetric renderings of human organs either automatically or manually. During a flight path tour, each point along the flight path is represented by a coordinate (x, y, z). According to embodiments of the present disclosure, these coordinates are used to automatically scroll and pan the series of two-dimensional images that doctors are used to analyzing. Thus, the operator does not have to manually navigate through an organ in two dimensions but, instead, can let the present virtualization system advance along the organ while the operator concentrates on analyzing each two-dimensional image.

[0055] The foregoing merely illustrates the principles of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise numerous systems, apparatus and methods which, although not explicitly shown or described herein, embody the principles of the disclosure and are thus within the spirit and scope of the disclosure as defined by its claims.

[0056] For example, the methods and systems described herein could be applied to virtually examine an animal, fish or inanimate object. Besides the stated uses in the medical field, applications of the technique could be used to detect the contents of sealed objects that cannot be opened. The technique could also be used inside an architectural structure such as a building or cavern and enable the operator to navigate through the structure.

[0057] These and other features and advantages of the present disclosure may be readily ascertained by one of ordinary skill in the pertinent art based on the teachings herein. It is to be understood that the teachings of the present disclosure may be to implemented in various forms of hardware, software, firmware, special purpose processors, or combinations thereof.

[0058] Most preferably, the teachings of the present disclosure are implemented as a combination of hardware and software. Moreover, the software is preferably implemented as an application program tangibly embodied on a program storage unit. The application program may be uploaded to, and executed by, a machine comprising any suitable architecture. Preferably, the machine is implemented on a computer platform having hardware such as one or more central processing units (“CPU”), a random access memory (“RAM”), and input/output (“I/O”) interfaces. The computer platform may also include an operating system and microinstruction code. The various processes and functions described herein may be either part of the microinstruction code or part of the application program, or any combination thereof, which may be executed by a CPU. In addition, various other peripheral units may be connected to the computer platform such as an additional data storage unit and a printing unit.

[0059] It is to be further understood that, because some of the constituent system components and methods depicted in the accompanying drawings are preferably implemented in software, the actual connections between the system components or the process function blocks may differ depending upon the manner in which embodiments of the present disclosure are programmed. Given the teachings herein, one of ordinary skill in the pertinent art will be able to contemplate these and similar implementations or configurations of the present invention.

[0060] Although the illustrative embodiments have been described herein with reference to the accompanying drawings, it is to be understood that the present invention is not limited to those precise embodiments, and that various changes and modifications may be effected therein by one of ordinary skill in the pertinent art without departing from the scope or spirit of the present disclosure. All such changes and modifications are intended to be included within the scope of the present invention as set forth in the appended claims.