Convertible runtime graphical user interface
Kind Code:

Method to change Graphical User Interface that support runtime modification of the application's GUI without access to the application source code and/or needs no recompilation/restart of the application executable. This method allows the end-user to make runtime modifications of a widget's attributes and/or runtime conversion of widgets into different types of widgets

Tertitski, Leonid Mark (Los Gatos, CA, US)
Ivanov, Andrei (Mountain View, CA, US)
Application Number:
Publication Date:
Filing Date:
Tertitski, Leonid Mark (287 Belvue Drive, Los Gatos, CA, US)
Ivanov, Andrei (2255 Showers Drive #363, Mountain View, CA, US)
Primary Class:
Other Classes:
345/419, 345/427, 345/629, 717/111, 717/113, 717/163, 717/165
International Classes:
G06F3/048; G06F9/44; G06T15/00; G06T15/10; G09G5/00; (IPC1-7): G06F9/44; G06T15/00; G06T15/10; G09G5/00
View Patent Images:
Related US Applications:
20050289522Semantic programming languageDecember, 2005Chang et al.
20070006202Dynamic mapping of shared librariesJanuary, 2007Mikkelsen et al.
20080010633Knowledge representation language and knowledge processing environmentJanuary, 2008Streeter et al.
20090164981Template Based Asynchrony Debugging ConfigurationJune, 2009Heidasch et al.
20080155519Code translatorJune, 2008Mulvaney et al.
20090193406Bulk Search Index UpdatesJuly, 2009Williams
20060225047Generic software requirements analyzerOctober, 2006Brothers et al.
20020010913Program profilingJanuary, 2002Ronstrom

Primary Examiner:
Attorney, Agent or Firm:
1. A method of managing graphical user interfaces associated with application programs operable on a computing device having a visual display device and a user input device for controlling a pointer on said display device, said graphical user interfaces having a plurality of windows containing widgets for visualization and manipulation of application program data (variables and/or callback functions).

2. A method of claim 1, comprising the usage of dynamic links (connections) between said data and their corresponding representations (widgets) in the graphical user interface.

3. A method of claim 2, comprising the possibilities to create new, remove or replace existing said dynamic links.

4. A method of claim 1, comprising the possibilities for GUI modification during an execution of said application programs.

5. A method of claim 4, comprising the default said dynamic links for newly implemented said data.

6. A method of claim 4, comprising the change of default attributes for widgets used by said dynamic links.

7. A method of claim 4, comprising read/write procedures for previously established said dynamic links and/or the widget attributes.

8. A method of claim 4, comprising the replacement of a widget in said dynamic link by another widget from a predefined list.

9. A method of claim 8, wherein said predefined list is constructed according to the type of data associated with the said dynamic link.

10. A method of claim 4, wherein the GUI's element (widget) representing an integer editable limited value can be converted into the slider type element (widget).

11. A method of claim 1, comprising the recording/replaying functionality for user's interaction with convertible GUI.

12. A method of claim 11, wherein a prerecorded sequence of said user's interaction for any initial state of GUI can be replayed for any new (converted) state of the same GUI.

13. A method of claim 1, comprising the support of widget library, designed to work with the said dynamic links.

14. A method of claim 6, comprising the automatic recalculation of coordinates and sizes for widget during resizing of its parent widget.

15. A method of claim 8, comprising the automatic rule based conversion of attributes of replaced widget.

16. A method of claim 15, comprising the customization of rules for said rule based conversion of attributes of replaced widget.

17. A method of claim 13, comprising the run time creation of new and modification of existing widgets including their associations with the callback functions.



U.S. Pat. Nos. 5,600,780/5,603,034/5,652,884/5,786,815/6,208,336


The present invention field of endeavor is user interface management systems (UIMS). Graphical User Interface (GUI) is “the Front End”, “the Face” of any application. Modern, sometimes called “configurable”, GUIs allow the end-user to modify some of the GUI widget's attributes: geometrical sizes, position, background color, and text font. Unfortunately, they provide no possibility to change the behavior of the widget associated with some program variable at runtime, or create new link(s) between this variable and new widget(s). For example, it is impossible to convert the widget “editable integer” to the widget “slider” without changing the application source code and/or without compiling a new application executable. A UIMS that would allow the runtime creation of new widget shapes and/or conversion of a widget or a group of widgets to another widget or group of widgets without recompilation and/or restart of the application executable would be desirable.


FIG. 1 shows contents of the example program HelloWorld.cpp;

FIG. 2 shows window with parameter “Param” when program HelloWorld.cpp runs for the first time;

FIG. 3 shows window with new (converted) representation of parameter “Param”;

FIG. 4 shows contents of the example program SumOfTwoNumbers.cpp;

FIG. 5 shows window with parameters of the program from FIG. 4 when it runs for the first time;

FIG. 6 shows window from FIG. 5 after the values of parameters have been changed;

FIG. 7 shows window with new (converted) representation of variables of the program presented in FIG. 4;

FIG. 8 shows window presented by FIG. 7 with duplicated parameters;

FIG. 9 shows window presented by FIG. 8 with parameters converted to sliders;

FIG. 10 provides a list of members of the class CUIParamEntry;

FIG. 11 provides a list of members of the class CUIObject;

FIG. 12 provides a list of CUI Viewer commands currently supported in “Edit” mode;


The current embodiment of this invention is a set of methods, procedures and functions implemented either in specialized Dynamic Link Library (CUI DLL) or standalone executable (CUI Viewer), and data representing Convertible User Interface Description (CUI Description).

All variables of any application can be divided into two groups with respect to their relationship with the GUI. Some are associated with some widgets, and thus they are “visible” or perhaps even “changeable” in the GUI, and the others are not. We refer to GUI-“visible” variables as CUI variables and we treat them in a special way. First of all, we implemented a special procedure for their registration, which uses their names and types. This procedure uses a special memory manager (CUI Memory Manager) for their memory allocation. Second, we provide a set of functions for the manipulation of CUI variables.

The registration of CUI variables is done automatically when the application uses special constructors. For manipulation of CUI variables inside of the application (set/get/compare value(s)) we implemented special methods and operators, which override analogous regular methods and operators used in C++ or Java. CUI DLL provides these special constructors, methods and operators. For manipulation of CUI variables outside of the application (registration, visualization in GUI, modification of their values according the user's input) we use the set of methods, procedures and functions implemented in CUI Viewer.

CUI Description is a text file used to define the application user interface as a set of descriptions of widgets with their respective sizes, locations, graphical and other attributes.

In our design CUI Viewer plays the role of a middleman between the application and its GUI, defined by GUI Description. It is used for the registration of CUI parameters, visualization and modification of the GUI(s) of one or several applications.

During its start an application sends a CUI Description file name and its own name to CUI Viewer. If CUI Viewer is able to read CUI Description file, it converts its contents into a tree of widgets and displays them according their attributes. During its work the application generates each CUI variable by using a corresponding constructor from CUI DLL, and the constructor registers each variable with CUI Viewer. At this moment CUI Viewer attempts to create a link between each registered CUI variable and its corresponding widget. If it is successful, then the widget is updated. If not, CUI Viewer creates some default widget for the new CUI variable. This new widget is added to the chain of widgets and is displayed by CUI Viewer in the window “New Parameters”.


1. An application never needs the information about the current contents or even existence of CUI Description. Generally speaking, it should work fine even when this file does not exist.

2. If the CUI Description file name provided by application is empty, CUI Viewer uses the application name to construct some default file name for CUI Description.

3. All CUI variables of current application are linked to new widgets, if CUI Viewer cannot find or open CUI Description file.

4. The storage of all CUI variables is managed by CUI Memory Manager. In some implementations CUI Viewer uses Application Shared Memory Buffer (ASMB). This design not only simplifies the treatment of CUI variables but make possible the use of common variables in different applications working with the same CUI Viewer.

5. CUI Viewer and CUI DLL do not depend on the purpose of the application. They could be written and/or compiled for any platform once or even become part of operating systems or Internet browsers.

6. All functions and procedures of CUI Viewer could be easily moved into CUI DLL. In this case this DLL could be statically linked to any application program.

7. It is very natural to use XML representation for CUI Description. However, since today XML is not very well supported on different platforms our program is capable of reading and writing CUI Description either in XML or in our own internal text format.

When the application is running the user can:

a) change different attributes of any widget: size, position, color etc.

b) move widgets from one window to another;

c) convert any widget to another widget type;

d) construct new and delete old links between GUI variables and widgets;

e) add new widgets not connected to any application variables and which can be used, for example, as a new window or button to open some window;

f) save current widget configuration as the GUI description in the text file that will be used by CUI Viewer during the next start of application.

An example of a simple “Hello World” application is presented by FIG. 1.

For simplicity we suppose that file “HelloWorld.cpp” was compiled into “HelloWorld.exe” application. When this application starts running it calls constructor CUIChar(char *) to initiate the editable CUI variable Param of character string type. This variable is registered with CUI Viewer under the name “Test”. Then the application sets the value of Param to string “Hello World”. Finally, it waits in a loop until the value of Param becomes equal to the string “End”.

When this application starts for the first time (or if Application CUI Description file is not presented), CUI Viewer displays an editable variable named “Test” having the value “Hello World” by using the default widget designed for CUI Variables of character string type in the default window “New Parameters” (FIG. 2) and waits for user input.

In this example the text “Hello World” is the value of CUI variable Param, which can be modified by user. When user inputs the text “End”, the application will end its execution.

Without exiting the application by using the capability provided by CUI Viewer, user can create a new window “Test Window” and convert the initial default widget previously used for representation of CUI variable Param into a pull-down menu “Hello World” with two options: “Hello World” and “End” (FIG. 3).

If this is done, the application stops its execution when the user selects the option “End” from the described pull-down menu. At this point CUI Viewer asks user to save new (converted) CUI Description. If the user agrees, CUI Viewer saves the new CUI Description for “HelloWorld.exe” in a text file with the name “HelloWord.CUI”. CUI Viewer will use this file during the next launch of application “HelloWorld.exe” and this application GUI will show up as in FIG. 3.

In this example (FIG. 1) the property of CUI variable Param being editable is defined by the default value of one parameter of constructor CUIChar(char *). The full set of parameters for this constructor will be described later.

Any CUI variable has two mandatory and unchangeable attributes “Name” and “Type”, which distinguish it among the other CUI variables. In other words, CUI Viewer will create only one CUI variable for any given pair (Name, Type) and it will provide read/write access to this variable attributes for any application working under its supervision.

The attribute “Name” is a non-empty character string. Currently CUI DLL and CUI Viewer support the following CUI variable types:

a) CUIint for integers;

b) CUIdouble for doubles;

c) CUIchar for character strings;

d) CUIintArray for arrays of integers;

e) CUIdoubleArray for arrays of doubles;

f) CUIcharArray for arrays of strings.

CUI DLL provides read/write access to the value of any CUI Variable and other manipulation of CUI variables by using a set of overloaded operators defined separately for each type in the corresponding C++ class.

Two CUI Variables defined by one or several applications with the same pair of attributes (Name, Type) always have references to the same address provided by CUI Memory-Manager.

FIG. 4 shows an example of a program for the calculation of the sum of two numbers.

This program defines two user editable CUI variables FirstNumber and SecondNumber and CUI variable SumOfTwoNumbers that can be modified only by the application program. These three variables have the names “A”, “B” and “Sum of A and B”, respectively. Notice that the application program does not define how and where these variables must be displayed.

The application program sets the initial values and then stays in a while loop and calculates the sum of these two numbers. The function CUISleep( ) belongs to CUI DLL. It waits for the requested number (100) of milliseconds and then checks the status of the application. If application program state is “End Application”, then CUISleep( ) returns 1. The CUI DLL function supporting “set value” function sends the message “Update Value” to the CUI Viewer only if the value of SumOfTwoNumbers was actually changed.

When the application starts for the first time (or if file “SumOfTwoNumbers.CUI” is not found) the user is presented with Initial state of window “New Parameters” (FIG. 5). User can modify the values of A and/or B and

a) CUI DLL will modify FirstNumber and/or SecondNumber;

b) application will calculate the result A+B;

c) CUI Viewer will automatically display new result (FIG. 6)

Let's describe the hypothetical sequence of user's actions modifying the GUI of this application when it is running.

1. User can create new window “Sum of A and B” and move all widgets representing CUI variables into this window (FIG. 7).

2. User can duplicate widgets representing the values of A and B (FIG. 8).

3. User can modify size and display style of the first widget representing A (B) and convert the second widget representing the same CUI variable into a slider (FIG. 9).

4. User can test the newly created interface by using sliders to modify the values of editable CUI variables A and/or B.

5. Finally, user can save the new CUI Description of the newly constructed interface and see it in the next launch of the application.

Let's describe the data structures we are using to make CUI work. For this we will rather use “functional” descriptions of class members and not their formal names.

We are using three main base classes: CUIParamEntry, CUIParam and CUIObject. The first two of these contain information about CUI variables and methods for their manipulation; the last contains information about widgets.

FIG. 10 shows the information stored by any instance of CUIParamEntry class.

“Actual Value” (FIG. 10 line 3) can be changed either by CUI Application. Note that CUI Viewer can change “Actual Value” only if an application defined the associated CUI variable as CUI_EDITABLE and the user changes the corresponding field of the widget linked with the variable. Each CUI variable is an instance of a type specific class (CUIint, CUIdouble or CUIchar) derived from class CUIParam. Any instance of CUIParam contains only one member “Index of Parameter”, which is the index of an instance of class CUIParamEntry previously allocated by CUI MemoryManager.

The constructor of class CUIParam uses one of its input parameters, Name, passed to it by the constructor of one of the derived classes (CUIint, CUIdouble or CUIchar), and corresponding Type and attempts to find an instance of CUIParamEntry with the same Name and Type in the memory allocated by CUI Memory Manager. In case of a match it makes Index of Parameter equal to the index of found instance. Otherwise the new instance of CUIParamEntry with given (Name, Type) is constructed and Index of Parameter stores its index.

All constructors of CUIint, CUIdouble, and CUIchar classes pass additional parameters to the constructor of CUIParam class, which define the initial value of constructed CUI variable and some attributes.

FIG. 11 shows the information stored by any instance of CUIObject class.

We use the following three classes derived from CUIObject to represent different types of widgets: CUIEdit, CUIButton and CUIPicture. Each of these classes contains the full description of graphical representation and location of corresponding widget.

The operation of any application working in CUI environment is based on the message exchange between some functions provided by CUI DLL, CUI Viewer and the special thread automatically launched for the application. Among these messages the most important are the “SET_VALUE” and “RUN_CALLBACK” messages. The first of these initializes the changes in the visualization of CUI Variables on the screen, and the other initializes some reactions of the application related to the user's input.

Every time the value of any CUI Variable (CUI_Var) is changed, a special CUI DLL function sends a “SET_VALUE” message with the address of corresponding instance of CUIParamEntry class to CUI Viewer. Upon receiving this message CUI Viewer performs the following sequence of operations:

a) It checks the value of the field “CUIObject Address” (FIG. 10 line 11). If it is NULL, CUI Viewer searches for an instance of CUIObject class with the same Name (FIG. 10 line 1) and Type (FIG. 10 line 2) in the tree of CUIObjects. If no such instance is found, the CUI Viewer creates a new instance of CUIObject class with the default parameters for the visualization of CUI_Var and address of the CUIParamEntry instance stored in “Reference to parameter” field (FIG. 11 line 19). The address of the newly created or found instance is stored in the field “CUIObject Address” (FIG. 10 line 11).

b) It displays the new value of CUI_Var by using the virtual function Draw( ) for the instance of CUIObject linked with CUI_Var.


1) When the search of CUIObject instance for current Name (FIG. 10 line 1) and Type (FIG. 10 line 2) is unsuccessful, the default parameters for visualization of CUI_Var by newly constructed CUIObject instance make possible the visualization of CUI_Var in the special window “New Parameters”.

2) Any CUI Variable can have more than one representation in GUI described by different CUIObject instances. All these instances comprise a doubly linked list (FIG. 11 lines 34, 35) used by CUI Viewer when it processes the “SET_VALUE” message.

When the application calls any CUI function for the first time, CUI Viewer starts a special thread CUIAppThread used for waiting and processing of “RUN_CALLBACK” messages. CUI Viewer sends such a message every time the value of CUI Variable CUI_Var is modified by user's interaction with the GUI (or by another program) and the field “Callback address” (FIG. 10 line 5) of the CUIParamEntry instance associated with CUI_Var contains a non-NULL address of some callback routine of the application.

In the special “Edit” mode CUI Viewer allows the user to convert any widget into a widget of another type or change its attributes. “Edit” mode can be selected by pressing “Ctrl-E” (“Ctrl”+“E” buttons simultaneously). When this is done the user should:

a) select the widget he wants to convert or modify by making a right mouse button click inside this widget;

b) open the CUI Viewer pull down menu “Convert To . . . ” and select the new widget type by pressing a button inside desired widget type image.

FIG. 12 shows the full list of CUI Viewer commands currently supported in “Edit” mode.


1) CUI Viewer itself is written as CUI application; therefore the user can add or modify the list of widgets in the “Convert To” menu.

2) CUI Viewer does not send any messages regarding the widget conversion to the application program, and the application continues its execution as usual.

The user can record input information to file and then play this file by using CUI Viewer commands “Record” and “Play” (FIG. 12, lines 11 and 12). In “Record” mode CUI Viewer stores only information about CUI Variables changes. Each record contains following information:

1. Variable Name;

2. Variable Value;

3. Time when user changed Variable Value.

The CUI Viewer “Play” command can be used even if the positions, sizes or types of any Widgets have been changed (claim 12). During Play CUI Viewer reads these records and simulates execution (moves mouse pointer to widget with the same Variable Name as in the record and changes Variable Value) according to recorded time.

If there is more than one widget with the same name, CUI Viewer calculates the “Time To Set Value” for each such widget. The “Time To Set Value” is calculated as the time needed to move the mouse cursor at “Mouse Simulation Speed” to the closest point of the widget plus the time for “Value Simulation Change”. If the widget is not visible (for example, it is located in a closed window), the “Time To Set Value” is equal to the time needed to move mouse cursor and make this widget visible plus time needed for “Value Simulation Change”. The widget with minimum “Time To Set Value” is used.

In the current implementation the default value of “Mouse Simulation Speed” is equal to 5 centimeter per second, “Time To Set Value” is equal to number of the Mouse/Keyboard button clicks multiplied by 0.3 seconds per click.

Some methods of current invention are based on or intensively use the external and internal representation of different CUI data in XML format. All these XML-methods can be described as “static” or “dynamic” according to their usage of transformation of XML representations involved in the corresponding processes.

The static XML-methods use XML representation for storage and reading/writing procedures of CUI Description (claim 7) and widget library (claims 13, 17). They use no transformations of XML data.

Every dynamic method uses some XSL transformation to convert the initial XML representation of some part of CUI data into its new XML representation. When the transformation is complete the CUI Viewer uses this new XML representation for visualization of the same CUI data in some other (converted) way. This approach is implemented for automatic recalculation of coordinates and sizes of widgets (claim 14) and for automatic rule based conversion of attributes of replaced widgets (claim 15).


1) The XSL files used for our dynamic XML-methods are the external text files that form the special Transformation Library. Any of these files can be modified or replaced by a new XSL files, which can provide new rules for some transformation (claim 16).

2) The use of a special naming convention for XSL files of Transformation Library makes it possible to support predefined lists for widget conversion (claims 8, 9)

3) The use of this naming convention makes the Transformation Library expandable. There is a possibility not only to replace some or all XSL files, but also to add new XSL files providing conversion rules for a new customer's widgets.

4) The names of CUI Variables can be used in XSL transformation files names.

5) While processing an XSL file, a special function from CUI DLL has access to the current value of any CUI Variable registered with CUI Viewer. This makes it possible to use not only the names of CUI Variables but also their values during XSL transformation.