DETAILED DESCRIPTION OF EXEMPLARY EMBODIMENTS

[0087] Reference is now made to FIGS. 1A and 1B which schematically illustrate parts of a computer system 10 comprising a display screen 12 , a standard computer keyboard 100 and a computer (not shown). In accordance with some exemplary embodiments of the invention, computer system 10 does not require a mouse. Optionally, mouse-substitution is provided by the installation of a software in computer system 10 (henceforth, the mapping software), which converts standard keyboard 100 into a dual-purpose-keyboard by providing it with two operational modes. Optionally, in the first mode, a typing mode, keyboard 100 has the functions of a standard keyboard, and in the second mode, a pointing-clicking mode, keyboard 100 is used as a pointing device, to move a pointing cursor and/or emulate the clicking of buttons on a pointing device (e.g., “clicking”). Additional and/or hybrid pointing modes may also be defined, for example as described below.

[0088] It should be noted that two cursors are usually provided, a selection cursor 40 , which is used to indicate the place where newly typed text will be inserted and a mouse cursor 41 which indicates the screen area referred to by the mouse. In many typical applications, clicking on the mouse when mouse cursor 41 is over a text portion will move text cursor 40 to the mouse location. The selection cursor may also be used to indicate a currently selected icon.

[0089] In the typing mode keyboard 100 optionally comprises four groups of keys:

[0090] keys 110 for typing alpha-numeric characters and symbols, such as: “a”, “5”, “:”, and “$”;

[0091] keys 120 for carrying out functions, especially general functions, for example editing, but also application specific functions, such as: Del, Insert, Enter (editing), Esc, F1 (application specific);

[0092] keys 130 that are used in conjunction with other keys to modify their meaning, such as: Shift, Ctrl, Alt; and

[0093] keys 140 for controlling cursor movement (both text and mouse, depending on the system mode), such as: Home, ←, ↑, Tab, Backspace.

[0094] In an exemplary embodiment of the invention, in the pointing-clicking mode, keys 110 (and possibly keys 130 ) are optionally used to address areas on the screen, keys 140 are optionally used to perform relative movements of the pointing cursor, and keys 120 are used to perform clicking actions and/or other editing actions. However, other key usage schemes may be provided. In particular, where a key stroke is suggested below, a key combination and/or a key sequence may be used instead.

[0095] In exemplary embodiments of the invention, any of keys 110 - 140 and/or combinations thereof may be used to toggle between the modes. Optionally, however, keys 130 are used for the toggling. In some embodiments, only one toggling direction is required, from text to pointing mode, as the mode snaps back, for example after a time delay in which no keys were pressed or after the pointing is achieved.

[0096] In some application software, some keys and/or key-combinations may already be defined to have a function. Optionally, this function is not overridden by the mapping software. One method of avoiding overriding is by defining a prefix key combination required to enter a non-standard keyboard mode. Alternatively or additionally, the assignment of keys for toggling and/or for the pointing mode takes into account common shortcut keys. Alternatively or additionally, a user may redefine key-functions so as to avoid conflict between application and the mapping software. Optionally a configuration utility is provided, for example for use during the installation of the software.

[0097] In some exemplary embodiments of the invention, certain key assignments of function keys 120 are maintained both for the typing mode and for the pointing-clicking mode. For example, a key 121 (F1) is generally used as “HELP”, and may be used as such both for other applications and for the software. In an exemplary embodiment of the invention, a key “Fn”, indicated by reference 118 is used for toggling, however, many keyboards do not include this key. Alternatively, a “right-alt” key (or other composition key) may be used for toggling, for example, by depressing it alone, possibly for a minimal defined duration. In some embodiments of the invention, for example in laptop computers, a dedicated toggle key 118 is provided. Optionally, other keys may be used for returning to a default mode (typing or pointing), for example, an “esc” key.

[0098] In an exemplary embodiment of the invention, the pointing clicking mode provides a direct addressing scheme, rather than, or in addition to a relative addressing scheme (such as provided using a mouse). Thus, the mouse cursor (and/or the text cursor) is optionally moved to a new location, rather than shifted in a certain direction by the keyboard. In some cases, fine tuning of the cursor location will be achieved using relative techniques, such as arrow keys.

[0099] In an exemplary embodiment of the invention, one or both of the following addressing schemes are provided:

[0100] a. a temporary-text mode, in which key-presses are used to bounce the cursor to matching text on the display, and

[0101] b. an area-designation mode, in which each screen area is mapped to a specific key or keys.

[0102] c. an indexing mode in which points of interest on the screen are tagged with an index, for example, responsive to the entry of a partial address (of any type).

[0103] In an exemplary embodiment of the invention, one of the modes is designated a default mode which the mapping software uses to translates the key strokes. Alternatively or additionally, a user may define using a certain key stroke (or key strokes) which mode to enter. Alternatively or additionally, the last used mode may be defined as a default. Possibly, one of the key combinations is used to open an interaction window in which a user can define the default and/or the current behavior of the mapping software.

[0104] In an exemplary embodiment of the invention, in the temporary-text mode, when the user enters a string of text using one or more keystrokes, cursor 40 bounces to the location of the string on screen 12 . For example, as we look at screen 12 in FIG. 1 A, cursor 40 is located near a word “Preferably”. To bring the cursor to a title 28 the user will type: “SUMMARY”, whereupon, cursor 40 will bounce to title 28 , optionally to the bottom left corner of it. Alternatively, cursor 40 may bounce to the right of it, or to another point in relation to it. Optionally, the string is treated as a set of characters, whose order is not important. Alternatively or additionally, a user may enter a pattern (e.g., including wildcards), rather than a string. Some wildcards may represent icons or other graphical elements, rather than letters.

[0105] Alternatively or additionally, to bouncing text cursor 40 , the pointing cursor 41 may be bounced. Optionally, a certain keystroke (which may emulate a “click”) may be used to make the two cursors match-up. In an exemplary embodiment of the invention, the cursor does not select the entered text, when it moves. Alternatively, the cursor does select the text. Possibly, the selection behavior and/or positioning relative to the word is determined by user entered defaults or by the user pressing a certain key.

[0106] The cursor may begin moved as soon as the user starts typing keys. Alternatively, the bouncing may wait until there is a pause in typing. In an exemplary embodiment of the invention, as the user types more keys, the cursor is moved to the nearest sequence which matches all the typed keys. Optionally, the user ends the string with a special key, for example, key 128 (enter).

[0107] Optionally, as a default case, the computer assumes that keystroke entries apply a new string. When the user wishes to extend a previous string (from a previous pointing or after a certain delay), the user indicates this by a special keystroke, for example, a key 123 (F3). Alternatively, one special keystroke will precede a new string, and another will precede a continuation of a previous string. Alternatively still, the computer will pose a written question, for example, “New string?”. Optionally however, the software automatically determines if a current keystroke is a continuation or a new string, for example, based on a time out or based on activities of the user between the two keystroke sets. Alternatively or additionally, a time out may be used to switch back to a default mode, for example a normal typing mode.

[0108] In some exemplary embodiments of the invention, a third option “Next string” is also available, either by a special keystroke, for example, a key 124 (F4), or in response to a computer question. The selection of this option will bounce the cursor to the next string on display screen 12 , which matches the keystrokes. Alternatively or additionally, other methods of choosing between two matches are provided, for example, by the computer posing a question or by the cursor blinking between the matches until the user presses a key. Alternatively, an indexing method, as described below, may be used.

[0109] Optionally, when scanning the display screen for a string, the computer begins scanning from the top left corner, and scans first across, a line of text, then down one character. Alternatively, especially where a language that is written from left to right is used, the computer begins scanning from the top right corner. Alternatively, the computer begins scanning from the top left corner, and scans down a line of text, then to the right one character. Alternatively or additionally, the scanning starts from the current text location. Possibly, the scanning is spiral. Alternatively, any other order of scanning maybe used.

[0110] In an exemplary embodiment of the invention, the search is case sensitive. Alternatively, the search is case insensitive. Possibly, a special key is provided to indicate a language of the text to be matched. Alternatively, the language may be determined from the computer settings and/or based on the displayed text and/or characters. Optionally, non-ASCII characters and/or icons may be represented using keystroke combinations. These features may also be controlled using user defaults. Optionally, the keyboard mapping changes, responsive to the language mode the computer or window are in and/or responsive to the major language displayed on the screen.

[0111] Accessing an icon may follow the regular rules, as applied to the associated text. Alternatively, the user may limit his addresses to those referring to icons, window controls, menu items and/or other screen display object subsets, for example, by entering the address with an indication (e.g., a special key stroke or the key sequence “icon”) that it is an icon address. In an exemplary embodiment, a trio of keys, such as “print screen”, “scroll lock” and “pause” represent the icons in the upper right corner of a window in a Windows 95 window.

[0112] Optionally, a display may be provided to show the keystrokes entered by the user. Alternatively or additionally, text editing keys, such as “backspace” may be used to “edit” the entered keystrokes and thus modify the screen address represented by them. A standard key, such as “esc” may be provided to cancel the current mode and/or the last entry and/or sub-mode change (e.g., screen enlargement).

[0113] In an exemplary embodiment of the invention, address codes for the area-designation mode are provided by the software. In an exemplary embodiment of the invention, the keyboard layout is mapped to the screen or to a portion thereof, so that each key corresponds to one or more screen areas. It is noted that the keyboard has more columns than rows. In an exemplary embodiment of the invention, the keyboard is mapped to a third of the display. In an exemplary embodiment of the invention, the user can select, for example using a certain key stroke or based on the user's previous position, which screen portion is being addressed. Alternatively or additionally, the mapping moves on the screen, for example, once for each keystroke.

[0114] In an exemplary embodiment of the invention, a map of the keyboard is overlaid on the screen to indicate the address codes to a user. The over-laying may be immediate when entering the mode, after a delay or possibly, responsive to a particular key stroke. Alternatively or additionally, other address designations, such as a grid designation, may be used, in which case the address indications may be relegated to the sides of the screen.

[0115] In an exemplary embodiment of the invention, the characters are embossed on the display, so that they minimally interfere with reading the display. Alternatively, other display methods, such as inverse-video, may be used. In different embodiments different character sizes, fonts and styles may be used. In particular, an outline character may be used.

[0116] Reference is now made to FIGS. 2 A- 2 D, which illustrate several manners of address assignment and superimposing characters on a display screen, in accordance with some exemplary embodiments of the present invention. It should be noted that although these techniques may be applied to a single software application or application window, in an exemplary embodiment of the invention, these techniques are applied to the screen as a whole, without reference to the underlying windows and/or applications, except to the extent in which they might aid in temporary text addressing techniques.

[0117] FIG. 2A illustrates a display screen division in which the screen is divided into 16 rectangles, each marked by an alphanumeric character that references its center. The marking may reference other parts of the rectangle instead, for example its upper right cornet. Alternatively or additionally, the marking may be changed by setting up defaults and/or by applying a suitable keystroke.

[0118] In an exemplary embodiment of the invention, the screen division matches the physical keyboard layout, for example a QWERTY or a Dvorak layout, however, this is not essential. In some embodiments, the layout is vertically repeated, as the aspect ratio and/or spatial resolution of the keyboard does not match that of a screen. A special key may be provided for selecting which part of the screen is mapped by the keyboard. Alternatively or additionally, the layout is horizontally repeated. Optionally the screen division is in a grid shape, however, the screen division may also be non-grid, for example, exactly matching the keyboard geometry. This may require a user to select the model of keyboard that he uses, from a list during a configuration stage.

[0119] It is noted that geometric shapes other than rectangles and/or other numbers of rectangles may be used. The aspect ratio of the rectangles may be the same as the screen or it may be different. The reference point may be marked or unmarked. In some cases, more than one character is needed to reference each rectangle. FIG. 2B illustrates a screen division to 10×6 squares, each marked by an alpha-numerical character sequence that references its upper left corner. FIG. 2C illustrates a color-coded screen division, in which the addressed areas are differentiated by using different colors. Colors will generally not mask the text or images on the display screen. FIG. 2D illustrates a checker-board pattern of light gray and white, referenced by alphabetic characters on a top ruler 13 and numeric characters on a side ruler 15 .

[0120] The same or different sets of characters may be used for the two axes. If different ones are used, the address designation may be entered in any order and even corrected by typing a new key. Grid lines may be shown or not. Various combinations of the above methods and/or other addressing methods may also be used.

[0121] In an exemplary embodiment of the invention, the addressing grid conforms to the location of objects of interest on the screen, for example, icons on a desktop, menus and window controls. This may include shifting of the grid, distorting the grid and/or varying the resolution of the grid for different parts of the screen. Alternatively or additionally, only objects of interest are tagged, with addresses that can be shown on the screen.

[0122] When a column or row is selected, the entire selected row or column may be marked. Similarly, when a cell is selected, the cell may be marked and/or highlighted.

[0123] In some cases, no screen display of the mapping is needed, as a user may remember the screen mapping and/or use the feedback from the cursor movement to correct his pointing activity.

[0124] As can be appreciated, the pointing resolution using keyboard-mapping may be insufficient for certain uses. In an exemplary embodiment of the invention, various mechanisms are provided for fine-tuning the pointing. Alternatively or additionally, the pointing location is automatically corrected to a portion of the addressed area, based on the content of the area.

[0125] Reference is now made to FIGS. 3 and 4 , which illustrate a method of fine tuning in which the screen is enlarged after an area is addressed. Optionally enlarged screen 12 is re-divided using a same scheme as used to address the area, for example into 16 rectangles, using the same letter-addresses. Alternatively, a different mapping scheme may be used, for example, using different letters or even a different addressing scheme. Thus, for example, letters may be used for main-areas and numbers for sub areas. Once the screen is enlarged, the addressing method can remain the same or it can be changed, for example from a temporary-text scheme to a direct addressing scheme. Optionally only two levels of resolution are required, however, in some embodiments more levels are provided and may be accessed, for example using a special key, such as a key 126 (F6).

[0126] In some embodiments, the grid is made finer alternatively or additionally to actually enlarging a portion of the screen. Optionally, a finer grid and/or an enlargement “pop-up” when a partial address is entered and no selection is made.

[0127] In one example, if the user wishes to change a font size, he will strike the “3” key to reach the font size box, whereupon the cursor will bounce to the rectangle designated by “3”, to the “size” window, which is the most relevant area for text input. Thus, a zooming step is possibly not required. Alternatively, rectangle “3” (and/or nearby rectangles) is enlarged to fill the whole of screen 12 as shown in a FIG. 4 . The user can strike the “J” key to reach the font selection window. As FIG. 4 shows, the cursor may bounce to a point near, but not quite on a black triangle of the font selection window. In an exemplary embodiment of the invention, the mapping software causes the cursor to be attracted to a nearest useful graphical element or point of interest. Thus, when striking “J”, the cursor will be attracted to the black triangle of the font selection window.

[0128] Typically, the useful elements are those that can be activated, such as buttons and other user interface objects or those that can be selected, such as text and/or graphic lines. In an exemplary embodiment of the invention, the interest level of a symbol or image portion is determined by analyzing the display presentation of the symbol, for example text (e.g., as compared to a dictionary of keywords), color shape and/or combinations thereof.

[0129] Upon reaching the desired point, a user may emulate a click (or double-click) with a left mouse key, for example, using a key 138 (Window key). Alternatively, he may highlight an area between a previous left-mouse click and a current cursor position using a same or a different key stroke, for example with a key 132 (Shift)+key 138 . Alternatively or additionally, the user may emulate a click with a right mouse key, for example, using a key 136 (Right mouse key). Alternatively or additionally, the user may request a shortcut key to the position of cursor 40 , for example, with a key 127 (F7).

[0130] As noted above, the mapping software may include a “sticky point” feature in which an address location is automatically fine tuned to the nearest item which might be manipulated by a mouse, for example an icon, a link or a button. Shifting between several relevant items may be achieved, for example using a key such as “Tab”. In an exemplary embodiment of the invention, the identification of the items is by analysis of the screen frame-buffer memory or by tracking the operation of functions that draw or write to the screen. Optionally, a hierarchy of importance between such items is defined, to assist in automatically selecting the most relevant object. Alternatively or additionally, tabs or other mechanisms are provided for jumping between displayed objects, for example words.

[0131] In an exemplary embodiment of the invention, the set of objects that can be selected between is defined by a dictionary. In one example, a direct address is limited to words that appear in a dictionary. Such a dictionary may be global, per application or operating system and/or provided by a user.

[0132] Another optional feature is the identification of text in images, for defining temporary text address codes for an image or portions thereof. In an exemplary embodiment of the invention, if a text string is not found, or possibly even if one match is found, image portions of the screen are analyzed to determine their text content, to allow a user to bounce the cursor to them. Such “embedded” text is common, for example in Internet images and in icons. Many methods of OCR are known in the art and may be used to detect such embedded text. In an exemplary embodiment of the invention, a degraded OCR is used, which only matches the image to the search string and does not attempt to extract the complete text string if it appears not to match the search string.

[0133] Alternatively, in an exemplary embodiment of the invention, the image may include therein an encoding of its content. In one example, such encoding is achieved by modifying the LSB bits of the image, for example 2 bits in a 24 bit image. The encoding may include, for example, the text content of the image or description of objects shown in the image. Optionally, the description includes coordinates and/or extents of the objects. Thus, the required information is available in the frame buffer. In one example, an image including a horse may include an embedding of the text “horse”. If a user types “horse”, the cursor will move to the image of the horse. Such embedding of information may be used for uses other than cursor control, for example for selecting from a menu which includes a textual description of the images or for generating such a menu. Alternatively or additionally, object recognition techniques may be used to generate the embedded text, or, similar to the OCR techniques described herein, to allow matching a text input to the image. Alternatively, a user may enter a description of screen objects, such as “hexagon” or “angle”, and screen objects matching these descriptions will be recognized by the cursor movement software, for example by tracking graphical drawing commands or by using feature recognition software.

[0134] The following is a summary of an exemplary assignment of functions to keys in accordance with an exemplary embodiment of the invention. However, the invention does not require all these key assignments or even the functionality of the keys. In some embodiments, only some of the functions are provided and/or different key-assignments are used.

[0135] 1. toggle key “CHANGE OPERATION MODE” toggles between the typing mode and the pointing clicking mode—key 118 (Fn);

[0136] 2. “LEFT-MOUSE CLICK”—key 138 (Window Key);

[0137] 3. “HIGHLIGHT CLICK”, to highlight an area between a previous left-mouse click and a current cursor position—key 132 (Shit)+key 138

[0138] 4. “RIGHT-MOUSE CLICK”—key 136 (Right-Mouse Key);

[0139] 5. “AREA-DESIGNATION-POINTING-MODE” tells the computer to point by area-designation—key 122 (F2);

[0140] 6. a “RETURN-TO-TEMPORARY—TEXT-POINTING-MODE”, a key 125 (F5);

[0141] 7. an “AREA-DESIGNATION” key —any of keys 110 ;

[0142] 8. “ENLARGE SCREEN” key—key 126 (F6);

[0143] 9. “CONTINUE STRING” key—key 123 (F3)

[0144] 10. a “STRING CHARACTER”—any of keys 110 ;

[0145] 11. “END STRING” key —key 128 (Enter);

[0146] 12. “NEXT STRING” key —key 124 (F4);

[0147] 13. “SHORTCUT-ASSIGNMENT” key —key 127 (F7).

[0148] 14. “HELP” key —key 121 (F1)

[0149] Optionally, a keyboard overlay or key caps (or stickers) in a set, are provided to mark the new functions of the keys. Alternatively or additionally, additional designated keys may be provided, for example in new keyboards or in laptop computers. Optionally, keys for specific activities may be arranged in a manner that mimics their screen appearance, for example, keys for controlling a window in the operating system Windows95, are arranged in a trio, in the order of “minimize”, “change size” and “close window”. A nearby key may be marked “move”. The keys may be so marked, as well.

[0150] Alternatively, keys with changing displays (on them or near them), for example miniature LCD or LED displays are used to show the instant or possible function of the key.

[0151] Let us follow an example, in which a user has just checked his e-mail, as illustrated in FIG. 1 A, and now wishes to close a Windows Messaging document 14 and resume his work with a Word document 12 . Assuming for the present example, the use of the shortcut key assignment delineated above, the user's procedure, can be as follows:

[0152] Step 1: Key 118 (Fn) to initiate pointing-clicking using temporary-text pointing mode;

[0153] Step 2: “xyz” (a string) followed by key 128 (Enter);

[0154] The computer will bounce the cursor to box 19 of Word, which is not the desired location.

[0155] Step 3: Key 124 (F4), to indicate, NEXT STRING.

[0156] The computer will bounce the cursor to box 35 , the desired location; At this point the user will click with key 138 and toggle out with key 118 .

[0157] Reference is now made to FIGS. 5 A- 5 D that comprise a detailed flow chart 200 for implementing one exemplary embodiment of the invention. Although many actions by the user are allowed in the flowchart, in some exemplary embodiments of the invention these actions are not performed, rather, a default is assumed or the possibility for action is blanked out, to facilitate simpler operation of the mapping software. Also, the order of the steps in the flowchart should not be considered limiting to any particular implementation, a person skilled in the art will appreciate that many orders can be used to effect exemplary embodiments of the invention as described above.

[0158] In general, the described process checks if the key is to be treated other than in a standard (prior art fashion), checks if the key is used to modify the mapping software behavior (and changes it) and then determines the address indicated by the key (and suitably moves the cursor). In some cases, the process is re-entered by a user applying a multi-key sequence. Optionally, the software remembers, at least for a certain minimum time, the state it was in after the last key was typed, to facilitate multi-key sequences.

[0159] First, a keystroke is entered ( 202 ).

[0160] Step 204 checks if the key changes the operation mode. If so, the mode of operation will be switched between the typing mode and the pointing-clicking mode, as described in a box 206 , and the computer will wait for another keystroke.

[0161] If the computer is in typing mode ( 208 ) the key is transmitted ( 210 ) to an application program (or the operating system).

[0162] If the computer is not in typing mode, key is analyzed to determine if it is meant to modify the functionality of the mapping software or if it is an address code for the mapping software to use.

[0163] Step 212 checks if the key requests the area designation mode. If so, the computer will prepare the screen for the area-designation pointing mode ( 214 ), for example, by dividing the screen to a plurality of areas and superimposing a grid and addresses on the screen. The screen division is optionally static, however, it can be dynamically assigned. Dynamic division or assignment of addresses may be dependent for example on the keyboard language, since, in some multi-lingual systems, when the language is changed, some keyboard key mappings move.

[0164] In an exemplary embodiment of the invention, when this key, or its associated key, ENLARGE SCREEN, are not struck, the computer assumes that the temporary-text pointing mode should be used. Alternatively or additionally, when the enlarge screen key is struck, the mode is switched to area designation mode. Thus, the operational mode can follow the modifying keys of that mode.

[0165] If the key emulates a left mouse click ( 216 ), the clicking operation will be performed ( 218 ). Optionally, but not necessarily, a location of the click will be saved in a pointing buffer (not shown) for a possible region-highlight request by a future key entry. Optionally, the pointing buffer will save only one location, which will be stored over any previous clicking location in the pointing buffer. Alternatively, the most recent two, three, or some other selected number of previous locations will be stored in the pointing buffer, for example to allow shifting between pointer locations, even between applications. Optionally, the location that will be saved will be the frame-buffer address of a point on the screen where the clicking operation took place. Alternatively, when using temporary-text pointing mode, the location that is saved will be the temporary-text location, even thought its frame buffer address may have changed. Alternatively or additionally, a location relative to an enclosing window is saved.

[0166] If a right click is emulated ( 220 ), the right click will be performed ( 222 ).

[0167] If the key represents a highlight (or select) function ( 224 ), text and/or image portions may be selected, for example the area between the most recent left-mouse click (stored in the pointing buffer) and a present cursor location ( 226 ). Alternatively, different keys (or repetitions) may be used to emulate letter, word and sentence selection.

[0168] Also, a “drag” key and/or other keys that emulate mouse functions such as known in the art of mouse emulation, may be provided. Correct entry by a user of area addressing codes may be important, for example, if the entered keys are meant to emulate an area-selection function or a drag function of a mouse.

[0169] If the software is in temporary-text pointing mode ( 228 ), the left side of the flowchart applies, otherwise, the software is in area-designation pointing mode, and the right side of the flowchart applies.

[0170] In temporary text mode, if the key means “NEXT STRING?” ( 230 ), the computer will then bounce the cursor ( 232 ) to the next screen location of the current string and save the new frame-buffer address, optionally, over the previous address, in a string buffer (not shown).

[0171] If the key means “BEGIN STRING?” ( 234 ), the computer will clear the string buffer from a previous string and string address and instruct the string buffer to receive a new string, as described in a box 236 . The computer will then wait for another keystroke.

[0172] If the key means “CONTINUE STRING?” ( 238 ), the next keystroke(s) are added to the string buffer to add any forthcoming characters to its existing string ( 240 ).

[0173] If the key is a printable character or one that represents a screen element ( 242 ), the character is added to the string buffer ( 244 ).

[0174] If the key means “END STRING?” ( 246 ), the string buffer is closed for updating and a search for the string on the screen is performed ( 248 ). If the string is found, the cursor is bounced to a location of the display screen associated with the string, for example, the bottom left corner of it. If the string is not found, the computer will perform an OCR conversion to any image stored in the frame buffer and repeat the search for the string. Optionally, an OCR conversion is performed with the first string request. Alternatively, it is performed when a pointing mode is requested. Alternatively or additionally, it is performed at regular intervals. Alternatively, the OCR it is performed on demand, when a string that was requested was not found in the frame buffer. When the string is found, the string and its current frame-buffer address will be stored in the string buffer.

[0175] Optionally, if the string is not found, the computer will print a message to this effect ( 249 ). Alternatively or additionally, a notification sound may be played. Alternatively or additionally, the cursor will not be moved.

[0176] Returning to step ( 228 ), if the mode is temporary text pointing mode, step 250 is performed.

[0177] If the key is an address for a screen area ( 250 ), the cursor is bounced to a point on the designated area, for example, the center of it ( 252 ). In some exemplary embodiments of the invention, areas are designated by more than one key, for example, “B5”. When the computer determines that it has received only a portion of the area designation, it will store that portion in an area-designation buffer (not shown) and wait for the remainder portion of the address before acting upon it.

[0178] If the key means “ENLARGE SCREEN?” ( 254 ), the screen will be zoomed around the area of the current cursor position ( 256 ). This area may then be divided and marked.

[0179] Optionally, if a key is undefined, the computer will print a message to that effect ( 258 ), play a sound and/or ignore the key. Alternatively, the software changes back to a standard keyboard mode.

[0180] In some exemplary embodiments, several keys may be struck one after the other, as one step, and the computer will check all these keys and perform the tasks associated with them, before waiting for another keystroke.

[0181] In some exemplary embodiments of the invention, the user may fine-tune a cursor position, for example using the arrow keys. Optionally, these keys move the cursor one character position, or a fixed number of pixels with each keystroke. Alternatively, the step sizes increase and/or decrease automatically, for example as a function of the time between presses or as a function of the count of the correction. In another example, on a tool bar, a tab will move the cursor to the next symbol, a backspace will move the cursor to a previous symbol, and an up or down arrow key will move the cursor to the upper or lower tool bar. Alternatively or additionally, the mapping methods described herein may apply to a toolbar or a set of toolbars, for example, each letter corresponding to a linear position along the toolbar.

[0182] In an exemplary embodiment of the invention, an indexing mode is provided. When indexing, a partial address (or even no address) are used to generate index-entries for all the relevant objects to be selected. The user can select a particular one of the relevant objects by entering its index value. In one example, typing “s” will select all the words starting (or ending, or containing) “s”, as relevant objects. Each such word may be assigned an index, for example a single digit or character or a numerical code. In an exemplary embodiment of the invention, digits and function keys are used as index entries. Typing the index code will bounce the cursor to the particular word.

[0183] Optionally, keys that may comprise a rest of an address (e.g., letters or digits, depending on the screen contents), do not form index entries. Alternatively, in some applications, multiple pressings of a same key can be expected, so that key is not used as an index entry.

[0184] When the number of index entries is not sufficient, a “next key” as described above, or the original partial address, may be typed to prompt marking the next set of relevant words. As noted above, the set of relevant words may be limited to words (or graphical objects) that appear in a dictionary. Such a dictionary may include individual examples as well as groups (e.g., “all icons” or “all bold words”). Optionally, the sets of words and/or indexing within words are selected in order of relevance, rather than in order of screen appearance.

[0185] In an exemplary embodiment of the invention, the intentions of a user may be guessed, or at least prioritized. In one example, an open menu, a modal dialog box or a single word on the screen will suggest that any entry probably refers to that object.

[0186] In an exemplary embodiment of the invention, indexing can also be used for selecting an icon. In an exemplary embodiment of the invention, a text string is associated with an icon, for example the text “start” is associated with the windows “start” icon. When indexing is generated, also that icon may be marked. Such associating may also be used for other addressing schemes.

[0187] In an exemplary embodiment of the invention, non-addressed items are also marked with an index, for example, window controls such as scroll bars or other items that a user is likely to want to access.

[0188] In an exemplary embodiment of the invention, the pointing mode may be a permanent mode, a temporary mode or a hybrid mode, for example one that allows both typing and pointing. In an exemplary embodiment of the invention, the following is a description of methods of carrying out typical user interface interactions, using a pointing mode.

[0189] Direct Pointing. Pressing a key button for 0.3-0.5 sec results in indication a beep. If the key is released, indexing tags appear (at words starting with the key letter). Otherwise, a normal repeat sequence of the key is initiated. Pressing one of indexing tags results in cursor movement to the location of the tag, after which the tags disappear. In a voice version (described in more detail below): saying “point at” or “jump”, followed by the first letter of a word or the whole word.

[0190] Relational Cursor Movement. One of two modes of cursor movement is selected by a toggle key (Alt or Ctrl):

[0191] a. Navigation between objects.

[0192] b. Movement in pixel or character steps.

[0193] When in one of these modes, pressing the toggle key simultaneously with arrow key results in cursor movement in the opposite mode. Voice version: saying “change mode”.

[0194] Left/Right click, Double-click. In a laptop keyboard can use the dedicated “mouse buttons”. In a regular keyboard can use any two (configurable) buttons. Voice version: saying “left click”, “right click”.

[0195] Scroll (mouse wheel), Auto Scroll (x-button). Right double click enters auto scroll mouse mode (an indicating cursor appears). Manual scroll (mouse wheel) is activated by pressing Alt and arrow key simultaneously. Voice version: saying, “auto scroll”.

[0196] Drag-and-Drop (including window move/resize, drag/drop of selected areas). When the cursor is on an object or on a selected area, prolonged press of the left mouse button results in displaying a “virtual keyboard” layout on the screen and beginning of “drag” operation. The user may then either press cursor keys, or press one of the tags, both resulting in dragging the object to the desired destination. Also, the user may choose direct pointing in order to reach the destination. When the user releases the left mouse button, he performs the “drop” part of the action, and tags disappear. Voice version: saying “drag” selects the object and displays the virtual keyboard, as described above. Saying “drop”, drops the object.

[0197] Area/object(s) selection can be performed similarly to the drag and drop operation.

[0198] In an exemplary embodiment of the invention, a word is defined as in standard word processors. Alternatively or additionally, a word may be defined as any sequence of characters, with font style changes and/or spaces indicating a change in the word. In an exemplary embodiment of the invention, a user can select whether the operation will reach a word start, end, center, select the word or be any other position relative to the word. Such selection may be, for example, by default definition, automatic, based on a system assumption or manually, by using a suitable key stroke(s).

[0199] In some exemplary embodiments of the invention, when there is a text cursor on the screen, in addition to the pointing cursor, the text cursor remains in place, while the pointing cursor is bounced to a new location by the keyboard. Alternatively, the two cursors are bounced together. Alternatively, the text cursor joins the pointing cursor upon a left-mouse click. Alternatively, the user may specify whether to bounce the text cursor or leave it in tact.

[0200] In some exemplary embodiments of the invention, a user may request an interactive mode of operation. Only three key assignments are made: toggle switch between the typing mode and the pointing clicking mode, for example, key 118 (Fn), a key to indicate “YES” by the user, for example, key 128 (Enter), and a key to indicate “NO” by the user, for example key 129 (Esc). In this exemplary embodiment, once the toggle key is in the pointing mode, the computer interacts with the user, by questions to which the user may reply with yes or no. For example, after the toggle switch is struck to indicate pointing mode, the computer will ask: “Point by area-designation?” The method of this embodiment may be slightly more time-consuming, but the user is spared the need to remember the special key assignments.

[0201] In some exemplary embodiments of the invention, toggle key 118 and/or other keys which define the functionality of the pointing mode, are replaced by a typed command (which can be captured by the mapping software), a keyboard chord, a voice command to a microphone connected to the computer, a mechanical switch added to the keyboard, or even an external switch or a foot paddle which may be connected to the computer (for example, via the mouse socket).

[0202] In some exemplary embodiments of the invention, when using the area-designation pointing mode, only a portion of the screen area is addressed. Alternatively or additionally, the resolution of addressing is different for different parts of the screen, for example responsive to their content, frequency of access and/or their distance from the current cursor position.

[0203] The mapping software may be provided for many graphical operating systems, for example MS WINDOWS, X11, Mac-OS, and OS/2. In an exemplary embodiment of the invention, a single interface is provided for many such systems, to allow a user to be. comfortable with many such systems.

[0204] The above-described mapping software can be integrated with a computer in various ways. In an exemplary embodiment of the invention, especially suitable for MS windows, the mapping software is implemented as a keyboard driver. Alternatively or additionally, the mapping software is implemented as a mouse driver. Possibly, the mouse can continue working in parallel with the mapping software, however, in some cases, a user may desire to disable the mouse. Possibly, the mapping software captures window draw functions, as is known in the art, so as to keep track of the display. Alternatively or additionally, the mapping software reads the required information directly from the frame-buffer. Alternatively, the mapping software may be integrated into the operating system, possibly as a patch. In laptop computers, for example, the mapping software may be implemented on a hardware level, so as to generate suitable mouse and keyboard signals to the motherboard.

[0205] In an exemplary embodiment of the invention, the mapping software comprises operating system dependent and operating system independent modules. In an exemplary embodiment of the invention, the operating system independent modules include modules for managing the interaction with the user, for matching addresses to content and for modifying and for retrieving screen content. Operating system dependent modules can include, for example, the specific interfaces to the keyboard (or other input device) and the screen, and a module for interacting with the operating system for determining what is being drawn on the screen.

[0206] In an exemplary embodiment of the invention, a user can designate, for example by keystroke or based on mouse focus, a window to which to limit the mapping and positioning. Alternatively or additionally, different maps and/or map resolutions may be provided for each window. In an exemplary embodiment of the invention, the mapping covers the entire window, including menus and/or window controls. Although the pointing function is preferably provided at the operating system level, so that it can be independent of application specifics, in some embodiments of the invention, the pointing may be provided at an application level, at least for some of the features described herein.

[0207] In an alternative embodiment, a smart keyboard is provided that receives an indication of the screen contents and locally processes keystrokes using this indication to determine a position for a cursor. In one example, the indication comprises a stream of text content of the frame buffer transmitted by RS232 from the computer (or other device, such as a TV) to the keyboard. The processing may be as described above.

[0208] The above invention has been described mainly with reference to standard PC keyboards, however, it may be applied to devices with no standard keyboards especially such devices with a limited or no graphical pointing ability or in which a mouse or other dedicated pointing device is inconvenient to use, for example, laptops, PDAs, devices with folding keyboards, Java machines, set-top boxes (e.g., using a remote control), digital TVs and cellular telephones. In such devices, other selections of keys and mapping of keys may be provided for.

[0209] In an exemplary embodiment of the invention, the keyboard is limited with respect to the number of available keys (or distinct recognizable sounds, in a speech entry system or a DTMF system). In an exemplary embodiment of the invention, a recursive grid-type mapping is used, as described above. Alternatively or additionally, each key can represent multiple characters, for example, “2” can be any one of {2, a, b, c}. In an exemplary embodiment of the invention, these other possibilities are not used for generating an index, to allow for multiple entry of the same key, to select a letter. Alternatively or additionally, each key entry is assumed to represent the entire set, so, for example, all words starting with one of {2, a, b, c} are selected for indexing or mapping, when the “2” key is pressed. This is a type of pattern matching which is indicated above as a possibility in address entry. It is expected that, in general, any original ambiguity between possibilities will be narrowed down to a small number as the user enters more characters.

[0210] In addition, the above methods of cursor motion control may be used as to fine tune cursor commands entered by other means, such a pointing devices, eye-gaze devices, touch screens and/or speech commands. Alternatively or additionally, these alternate input means may be used to fine-tune a cursor position entered as described herein.

[0211] Further, it is noted that alternatively to using a keyboard to enter text, other means, such as speech and pen entry means may be used. An additional benefit of pen entry is the ability to draw geometrical shapes that correspond to screen portions. Optionally, such input entry is used to navigate over the entire screen, rather than within a particular application. However, in an exemplary embodiment of the invention, once a particular window has been selected, a within-window or within application navigation scheme may be used, possibly even for hidden parts of the window.

[0212] With regard to speech entry, in some embodiments of the invention a lower quality speech entry system is used. In one example, all that is necessary is to recognize letters and digits, for example for use in direct or indirect addressing. Alternatively or additionally, speech may be used for mode switching. Alternatively or additionally, a voice mouse mode is used for relative motion of the cursor.

[0213] In an exemplary embodiment of the invention, when a speech pattern is received, a template matching method is used to recognize the speech content. However, matching is only to templates of words that are on the screen, so there is less matching actions to be performed and a greater latitude in the speech signal can be allowed. Possibly, only constants are matched.

[0214] In a combined mode, the index and/or a partial address are entered in one input modality, for example voice or keyboard, and the rest is entered in another modality, for example keyboard or speech.

[0215] In an exemplary embodiment of the invention, matching templates for the screen contents, or a list of templates to use, are provided prior to the entry by the user, for example, with the display page (e.g., an Internet), or them being calculated as the display is generated.

[0216] A particular application which can utilize speech control is a virtual reality application, in which the user's display comprises goggle that display a virtual world or an overlay. The “mapping” can optionally be provided using the display goggles. In a cellular telephone application, or in an application where a cable TV set top box (or other display device) is programmed via telephone connection, voice and DTMF can utilize the existing microphone.

[0217] In an exemplary embodiment of the invention, the above methods of text recognition on a computer screen are used to automatically alert a user or perform some other task responsive to text appearing on a display. In one example, such a software can be used as a censor to blacken a screen if sexually explicit language appears on the screen. In another example, a user that is inundated by the data flowing through the screen can be assured that when a desired key word appears, it position will be marked and he will be alerted.

[0218] The above description has focused on pointing using a cursor. However, in some embodiments of the invention, the system being interfaced with does not use a cursor. The above method scan, however, be applied to such a system, if the pointing method (e.g., direct addressing) is used to indicate a location to the system internal functions. Once the location is noted by the system, it maybe used to affect control of the system, for example by selecting an icon.

[0219] It should be appreciated that the above-described embodiments contain many features, not all of which need be practiced in all embodiments of the invention. Rather, various embodiments of the invention will utilize only some of the above described techniques, features or methods and/or combinations thereof. Further in addition, various modifications will be readily apparent to and may be readily accomplished by persons skilled in the art without departing from the spirit and the scope of the above teachings.

[0220] The present invention has been described in terms of exemplary, non-limiting embodiments thereof. It should be understood that features described with respect to one embodiment may be used with other embodiments and that not all embodiments of the invention have all of the features shown in a particular figure. Although some exemplary embodiments may have been described only as method, the scope of the invention includes software and/or hardware required to perform the methods, typically a personal computer. Additionally, the scope of the invention includes diskettes, CDs and/or other computer storage media including thereon representations of software suitable for carrying out at least one embodiment of the present invention. In particular, the scope of the invention is not defined by the exemplary embodiments but by the following claims. When used in the following claims, the terms “comprises”, “comprising”, “includes”, “including” or the like mean “including but not limited to”.