Description of the Preferred Embodiment

[0040] Referring firstly to FIG. 1 , this illustrates an embodiment of the system of the present invention, as set up for operation. The system/apparatus comprises, in combination, a printed or scribed document 1 , which might, for example, be a sheet of paper that is a printed page from a holiday brochure, positioned on a work surface (eg. a desk/table top that is suitably flat) facing upwardly; a video camera 2 that is held above the document 1 by a stand 3 and focuses down on the document 1 ; a processor/computer 4 to which the camera 2 is linked, the computer suitably being a conventional PC having an associated VDU/monitor 6 ; and a pointer 7 with a pressure sensitive tip and which is linked to the computer 4 .

[0041] The document 1 is freely variably positioned on the work surface—it is not placed on the work surface in a pre-defined position and orientation dictated by guides or markings on the work surface but can be at any location and angle of orientation that is flat and face-up on the work surface within the field of view of the camera 2 . The user placing the document 1 on the work surface may have rotated the document 1 through any angle about an axis perpendicular to the work surface/document 1 . He need not have placed it in alignment with himself or the apparatus. To compensate for the freely variable positioning of the document, the document 1 differs from a conventional printed brochure page in that firstly it bears a set of four widely spaced apart and prominently visible specialised calibration marks 8 a − 8 d , one mark 8 a - d proximate each corner of the page in the margins and outside of the primary information content of the page; secondly, it has a two-dimensional bar code which serves as a readily machine-readable page identifier mark 9 and which is located in the margin at the top of the document 1 substantially centrally between the top edge pair of calibration marks 8 a, 8 b.

[0042] The calibration marks 8 a - 8 d are position reference marks that are designed to be easily differentiable and localisable by the processor of the computer 4 in the electronic images of the document 1 captured by the overhead camera 2 .

[0043] The illustrated calibration marks 8 a - 8 d are simple and robust, each comprising a black circle on a white background with an additional black circle around it as shown in FIG. 3 . This gives three image regions that share a common centre (central black disc with outer white and black rings). This relationship is approximately preserved under moderate perspective projection as is the case when the target is viewed obliquely.

[0044] It is easy to robustly locate such a mark 8 in the image taken from the camera 2. The black and white regions are made explicit by thresholding the image using either a global or preferably a locally adaptive thresholding technique. Examples of such techniques are described in:

[0045] Gonzalez R. & Woods R. Digital Image Processing, Addison-Wesley, 1992, pages 443-455; and Rosenfeld A. & Kak A. Digital Picture Processing (second edition), Volume 2, Academic Press, 1982, pages 61-73.

[0046] After thresholding, the pixels that make up each connected black or white region in the image are made explicit using a component labelling technique. Methods for performing connected component labelling/analysis both recursively and serially on a raster by raster basis are described in: Jain R., Kasturi R. & Schunk B. Machine Vision, McGraw-Hill, 1995, pages 4247 and Rosenfeld A. & Kak A. Digital Picture Processing (second edition), Volume 2, Academic Press, 1982, pages 240-250.

[0047] Such methods explicitly replace each component pixel with a unique label.

[0048] Black components and white components can be found through separate applications of a simple component labelling technique. Alternatively it is possible to identify both black and white components independently in a single pass through the image. It is also possible to identify components implicitly as they evolve on a raster by raster basis keeping only statistics associated with the pixels of the individual connected components (this requires extra storage to manage the labelling of each component).

[0049] In either case what is finally required is the centre of gravity of the pixels that make up each component and statistics on its horizontal and vertical extent. Components that are either too large or too small can be eliminated straight off. Of the remainder what we require are those which approximately share the same centre of gravity and for which the ratio of their horizontal and vertical dimensions agrees roughly with those in the calibration mark 8 . An appropriate black, white, black combination of components identifies a calibration mark 8 in the image. Their combined centre of gravity (weighted by the number of pixels in each component) gives the final location of the calibration mark 8 .

[0050] The minimum physical size of the calibration mark 8 depends upon the resolution of the sensor/camera 2 . Typically the whole calibration mark 8 must be more than about 60 pixels in diameter. For a 3MP digital camera 2 imaging an A4 document there are about 180 pixels to the inch so a 60 pixel target would cover 1/3 ^rd of an inch. It is particularly convenient to arrange four such calibration marks 8a-d at the corners of the page to form a rectangle as shown in the illustrated embodiment of FIG. 2 .

[0051] For the simple case of fronto-parallel (perpendicular) viewing it is only necessary to correctly identify two calibration marks 8 in order to determine the location and orientation of the documents and thereby compensate for the freely variable positioning of the documents. These can also be used to determine the scale of the document. Whereas for a camera 2 with a fixed viewing distance the scale of the document 1 is also fixed, in practice the thickness of the document, or pile of documents, affects the viewing distance and, therefore, the scale of the document.

[0052] In the general case the position of two known calibration marks 8 in the image is used to compute a transformation from image co-ordinates to those of the document 1 (e.g. origin at the top left hand corner with the x and y axes aligned with the short and long sides of the document respectively). The transformation is of the form: 1 $\left[\begin{array}{c}{X}^{\prime }\\ Y^{\prime }\\ 1\end{array}\right]=\left[\begin{array}{ccc}k\cos \theta & -\sin \theta & t_x\\ \sin \theta & k\cos \theta & t_y\\ 0& 0& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]$

[0053] Where (X, Y) is a point in the image and (X′, Y′) is the corresponding location on the document ( 1 ) with respect to the document page co-ordinate system. For these simple 2D displacements the transform has three components: an angle θ a translation (t _x ,t _y ) and an overall scale factor k. These can be computed from two matched points and the imaginary line between them using standard techniques (see for example: HYPER: A New Approach for the Recognition and Positioning of Two-Dimensional Objects, IEEE Trans. Pattern Analysis and Machine Intelligence, Volume 8, No. 1, January 1986, pages 44-54).

[0054] With just two identical calibration marks 8 a, 8 b it may be difficult to determine whether they lie on the left or right of the document or the top and bottom of a rotated document 1 (or in fact at opposite diagonal corners). One solution is to use non-identical marks 8 , for example, with different numbers of rings and/or opposite polarities (black and white ring order). This way any two marks 8 can be identified uniquely.

[0055] Alternatively a third mark 8 can be used to disambiguate. Three marks 8 must form an L-shape with the aspect ratio of the document 1 . Only a 180 degree ambiguity then exists for which the document 1 would be inverted for the user and thus highly unlikely to arise.

[0056] Where the viewing direction is oblique (allowing the document 1 surface to be non-fronto-parallel or extra design freedom in the camera 2 rig) it is necessary to identify all four marks 8 a - 8 d in order to compute a transformation between the viewed image co-ordinates and the document 1 page co-ordinates.

[0057] The perspective projection of the planar document 1 page into the image undergoes the following transformation: 2 $\left[\begin{array}{c}x\\ y\\ w\end{array}\right]=\left[\begin{array}{ccc}a& b& c\\ b& e& f\\ g& h& 1\end{array}\right]\left[\begin{array}{c}X\\ Y\\ 1\end{array}\right]$

[0058] Where X′=x/w and Y′=y/w.

[0059] Once the transformation has been computed then it can be used for a range of purposes which may include firstly assisting in locating the document page identifier bar code 9 from expected co-ordinates for its location that may be held in a memory in or linked to the computer 4 . The computed transformation can also be used to map events (e.g. pointing) in the image to events on the page (in its electronic form).

[0060] The flow chart of FIG. 5 shows a sequence of actions that are suitably carried out in using the system and which is initiated by triggering a switch associated with a pointing device 7 for pointing at the document 1 within the field of view of the camera 2 . The triggering causes capture of an image from the camera 2 , which is then processed by the computer 4

[0061] As noted above, in the embodiment of FIG. 1 the apparatus comprises a tethered pointer 7 with a pressure sensor at its tip that may be used to trigger capture of an image by the camera 2 when the document 1 is tapped with the pointer 7 tip. This image is used for calibration, calculating the mapping from image to page co-ordinates; for page identification from the barcode 9 ; and to identify the current location of the end of the pointer 7 . The calibration and page identification operations are best performed in advance of mapping any pointing movements in order to reduce system delay. The easiest way to identify the tip of the pointer 7 is to use a readily differentiated and, therefore, readily locatable and identifiable special marker at the tip. However, other automatic methods for recognising long pointed objects could be made to work. Indeed, pointing may be done using the operator's finger provided that the system is adapted to recognise it and respond to a signal such as tapping or other distinctive movement of the finger or operation of a separate switch to trigger image capture. The apparatus described above enables efficient use of the printed or scribed document 1 as an interface with the computer 4 for instructing the computer to call up any of a range of different screen displays or for a multitude of other functions which may be initiated simply by placing the document 1 under the camera 2 , or by placing the document 1 under the camera 2 and pointing to an area of the document 1 that the computer 4 will recognise as having a computer action or linkage associated with it. The calibration marks 8 a - 8 d and suitably also the page identification mark/bar code 9 are best printed onto the document 1 substantially at the same time as the primary information content of the document 1 ( i.e. its text/pictorial or other informational content) is printed. The document 1 with the calibration markings 8 and identification marking 9 is suitably prepared in a computer such as computer 4 and with an editor in the computer 4 configuring the layout of the calibration marks 8 and bar code 9 relative to the primary information content of the document before exporting the formatted data to a printer for printing. Direct association of the calibration marks 8 and page identification mark 9 with the information to be printed before the document is printed out enables the relationship between the marks 8 , 9 and the primary information content of the document 1 to be properly controlled and configured and, furthermore, enables information about the configuration relationship to be embedded in the printed document 1 , for example in the bar code 9 . This latter facility may be of particular use when the printed document 1 is a modified version of the document as originally held in the computer 4 prior to printing out. The electronic version of the document that is held for initial printing may be of any printable form (provided it is WYSIWYG) that may, for example, be a web page, a Word or Excel document or a PDF file.

[0062] Importantly, when some documents are printed—for example, HTML web pages—the layout of the printed document is substantially different to that on screen. For web pages the format varies with the parameters of the browser used to view it. By embedding in the markings 8 , 9 on the printed document 1 details of the relationship between the geometry of the physical form of the printed document and the electronic material that generated it, this information can be retrieved when the printed document 1 is used as an interface with a computer 4 in order to link the printed document 1 to the underlying electronic information.

[0063] Although in the present invention it is preferred that the printed document 1 has both calibration marks 8 and a page identification mark 9 , in an alternative embodiment the computer 4 may be programmed to recognise the printed document 1 by other attributes of the document 1 such as, for example, paragraph layout. A page identification mark such as a bar code 9 does, however, represent a particularly efficient basis for page recognition and is also useful for imparting further information such as the above-mentioned mapping link between the geometry of the physical form of the printed document and the electronic material that generated it.

[0064] In preparing the printed document 1 additional information may be included in the document 1 such as, for example, the identity of the user or system by which the document is printed as well as any further useful information.