Schultz, Kevin L. (Georgetown, TX, US)
Vazquez, Nicolas (Austin, TX, US)

10/166405

12/11/2003

06/10/2002

Images are available in PDF form when logged in. To view PDFs, Login  or  Create Account (Free!)

View patents that cite this patent

Click for automatic bibliography generation

715/763

(IPC1-7): G09G005/00

Conley, Rose, & Tayon, P.C.,Jeffrey C. Hood (P.O. Box 398, Austin, TX, 78767, US)

We claim:

1. A computer-implemented method for displaying operations of an application, the method comprising: including a plurality of operations in the application in response to user input, wherein the plurality of operations implement a first function; displaying a plurality of interconnected icons in response to said including the plurality of operations in the application, wherein each icon corresponds to an operation in the application, wherein the plurality of interconnected icons visually indicate the first function; and storing information representing the application in a data structure, wherein said storing the information comprises storing information specifying the plurality of operations in the application.

2. The method of claim 1, wherein said including the plurality of operations in the application comprises receiving user input individually selecting each operation for inclusion in the application; wherein said displaying a plurality of interconnected icons comprises displaying at least one icon after each instance of said user input including one of said operations in the application.

3. The method of claim 1, wherein said including and said displaying comprise: including a first operation in the application in response to user input; displaying a first icon corresponding to the first operation in response to said including the first operation in the application; including a second operation in the application in response to user input; and displaying a second icon corresponding to the second operation in response to said including the second operation in the application; and displaying a connection between the first icon and the second icon.

4. The method of claim 1, wherein said displaying the plurality of interconnected icons comprises displaying a plurality of lines interconnecting the icons, wherein each line connects at least two icons; wherein the plurality of lines indicate a relationship of the icons.

5. The method of claim 4, wherein the plurality of lines includes a first line that connects a first icon to a second icon; wherein the first line indicates that the operation corresponding to the first icon is performed before the operation corresponding to the second icon.

6. The method of claim 4, wherein the plurality of lines include a first line that connects a first icon to a second icon; wherein the first line indicates that data produced by the operation corresponding to the first icon is used by the operation corresponding to the second icon.

7. The method of claim 1, wherein said displaying the plurality of interconnected icons comprises displaying a graphical programming representation of the plurality of operations in the application.

8. The method of claim 1, wherein each icon corresponds to a node in a graphical programming development environment.

9. The method of claim 1, wherein the plurality of interconnected icons comprise a graphical program, wherein the graphical program is executable to perform the first function.

10. The method of claim 9, further comprising: including one or more additional nodes in the graphical program in response to user input, wherein the one or more additional nodes do not correspond to operations from the plurality of operations in the application.

11. The method of claim 1, further comprising: programmatically generating a graphical program based on the application; wherein said programmatically generating the graphical program comprises programmatically including a plurality of interconnected nodes in the graphical program, wherein each node corresponds to an operation from the plurality of operations in the application; wherein the plurality of interconnected nodes implement the first function.

12. The method of claim 11, wherein each of the nodes has an iconic appearance that is substantially identical to the icon corresponding to the operation to which the node corresponds.

13. The method of claim 11, further comprising: displaying the plurality of interconnected nodes in the graphical program. wherein the plurality of interconnected nodes in the graphical program appear substantially identical to the plurality of interconnected icons.

14. The method of claim 13, wherein said displaying the plurality of interconnected icons comprises displaying the plurality of interconnected icons in a first window; wherein said displaying the plurality of interconnected nodes in the graphical program comprises displaying the graphical program in a second window.

15. The method of claim 11, wherein said programmatically generating the graphical program further comprises programmatically including one or more additional nodes in the graphical program that do not correspond to operations from the plurality of operations in the application.

16. The method of claim 11, further comprising: executing the graphical program; wherein said executing the graphical program comprises executing the plurality of interconnected nodes to perform the first function.

17. The method of claim 1, wherein said including the plurality of operations in the application comprises receiving user input individually selecting each operation for inclusion in the application; wherein said displaying the plurality of interconnected icons comprises programmatically including a node in a graphical program in response to each operation selected for inclusion in the application, wherein each node is operable to perform the respective operation when the graphical program is executed.

18. The method of claim 1, wherein said including the plurality of operations in the application comprises receiving user input individually selecting each operation for inclusion in the application; wherein the method further comprises programmatically including a node in a graphical program in response to each operation selected for inclusion in the application, wherein each node is operable to perform the respective operation when the graphical program is executed.

19. The method of claim 18, wherein each node included in the graphical program in response to selecting an operation has an iconic appearance that is substantially identical to the icon corresponding to the operation.

20. The method of claim 18, further comprising: removing a first operation from the application in response to user input; and programmatically removing the node operable to perform the first operation from the graphical program in response to said removing the first operation from the application.

21. The method of claim 1, further comprising: removing a first operation from the application, wherein the first operation corresponds to a first icon; and discontinuing display of the first icon in response to said removing the first operation from the application.

22. The method of claim 1, wherein said including the plurality of operations in the application comprises receiving user input individually selecting each operation for inclusion in the application; wherein the method further comprises executing each operation as the operation is selected for inclusion in the application.

23. The method of claim 22, further comprising: for each operation, displaying results of said executing the operation, wherein said displaying the results of said executing the operation comprises displaying the results together with the icon that corresponds to the operation.

24. The method of claim 23, wherein said displaying the results together with the icon that corresponds to the operation comprises displaying a pass/fail result together with the icon that corresponds to the operation.

25. The method of claim 24, wherein the plurality of operations includes a first operation; wherein executing the first operation produces a fail result; wherein displaying the pass/fail result together with the icon that corresponds to the first operation comprises displaying information indicating the fail result; wherein the method further comprises displaying information indicating a reason for the fail result.

26. The method of claim 23, wherein one or more of the plurality of operations in the application are operable to perform a test of a unit under test (UUT); wherein, for each operation operable to perform a test of the UUT, said displaying the results together with the icon that corresponds to the operation comprises displaying a pass/fail result together with the icon that corresponds to the operation.

27. The method of claim 1, further comprising: for each operation, displaying a name of the operation together with the icon that corresponds to the operation.

28. The method of claim 1, further comprising: displaying a graphical user interface that provides access to a set of operations; and receiving user input to the graphical user interface requesting inclusion of the plurality of operations in the application, wherein the plurality of operations are selected from the set of operations.

29. The method of claim 28, wherein said receiving user input to the graphical user interface requesting inclusion of the plurality of operations in the application does not include receiving user input specifying programming language code to implement the plurality of operations.

30. The method of claim 1, further comprising: executing the application; wherein said executing the application comprises executing the plurality of operations in the application to perform the first function.

31. The method of claim 30, further comprising: highlighting execution of the interconnected icons on the display to visually indicate progression of execution of the operations in the application.

32. The method of claim 30, wherein the plurality of operations comprises a plurality of machine vision operations, wherein the plurality of machine vision operations implement a machine vision function; wherein the plurality of interconnected icons visually indicate the machine vision function; wherein said executing the plurality of operations in the application comprises performing the machine vision function.

33. The method of claim 32, wherein said performing the machine vision function includes one or more of: acquiring one or more images; and analyzing acquired images.

34. The method of claim 30, wherein the plurality of operations comprises a plurality of motion control operations, wherein the plurality of motion control operations implement a motion control function; wherein the plurality of interconnected icons visually indicate the motion control function; wherein said executing the plurality of operations in the application comprises performing the motion control function.

35. The method of claim 34, wherein said performing the motion control function includes one or more of: controlling motion of a device; and controlling a device to move an object.

36. The method of claim 30, wherein the plurality of operations comprises a plurality of data acquisition (DAQ) operations, wherein the plurality of DAQ operations implement a DAQ function; wherein the plurality of interconnected icons visually indicate the DAQ function; wherein said executing the plurality of operations in the application comprises performing the DAQ function.

37. The method of claim 36, wherein said performing the DAQ function includes acquiring measurement data from a device.

38. The method of claim 1, further comprising: receiving user input for configuring one or more of the plurality of operations in the application; wherein said receiving user input for configuring one or more of the plurality of operations in the application does not include receiving user input specifying programming language code to configure the operations; wherein, for each operation, said configuring the operation affects functionality of the operation.

39. The method of claim 38, further comprising: for each operation to be configured, displaying a graphical panel including graphical user interface elements for setting properties of the operation; and receiving user input to the graphical panel to set one or more properties of the operation.

40. The method of claim 1, further comprising: displaying a text view of the plurality of operations in the application.

41. The method of claim 40, wherein said displaying the text view of the plurality of operations in the application comprises displaying a table including text that specifies the plurality of operations in the application.

42. A computer-implemented method for displaying operations of an application, the method comprising: including a first operation in the application in response to first user input; displaying a first icon corresponding to the first operation in response to said including the first operation in the application; including a second operation in the application in response to second user input; and displaying a second icon corresponding to the second operation in response to said including the second operation in the application; displaying a connection between the first icon and the second icon; wherein the interconnected first icon and second icon form a block diagram visual representation of the application comprising the first and second operations.

43. The method of claim 42, further comprising: wherein said first user input comprises applying the first operation to an object, wherein the first operation is included in the application in response to said applying the first operation to the object; and wherein said second user input comprises applying the second operation to the object, wherein the second operation is included in the application in response to said applying the second operation to the object.

44. The method of claim 42, further comprising: applying the first operation to an object in response to said including the first operation in the application; and applying the second operation to the object in response to said including the second operation in the application.

45. The method of claim 42, wherein the first icon corresponds to a first node in a graphical programming development environment; and wherein the second icon corresponds to a second node in the graphical programming development environment.

46. The method of claim 42, further comprising: displaying a graphical user interface which is useable for selecting the operations; receiving said first user input to the graphical user interface selecting the first operation, wherein said including the first operation in the application is performed in response to said first user input; receiving said second user input to the graphical user interface selecting the second operation, wherein said including the second operation in the application is performed in response to said second user input.

47. The method of claim 46, wherein the graphical user interface displays an object; wherein said first user input comprises applying the first operation to the object, wherein the first operation is included in the application in response to said applying the first operation to the object; and wherein said second user input comprises applying the second operation to the object, wherein the second operation is included in the application in response to said applying the second operation to the object.

48. The method of claim 47, further comprising: modifying the object on the display after each of said applying the first operation to the object and said applying the second operation to the object.

49. The method of claim 46, wherein the graphical user interface is displayed in a first window; wherein the first icon and the second icon are displayed in a second window.

50. The method of claim 42, further comprising: executing a prototyping environment, wherein said executing includes displaying a graphical user interface which is useable for selecting the operations; wherein said first user input and said second user input are received to the graphical user interface of the prototyping environment.

51. A computer-implemented method for visually depicting an application which performs a function, the method comprising: displaying an object on a display; performing a first operation on the object in response to user input; displaying a first icon corresponding to the first operation, wherein said displaying the first icon is performed in response to said performing the first operation on the object; performing a second operation on the object in response to user input; displaying a second icon corresponding to the second operation, wherein said displaying the second icon is performed in response to said performing the second operation on the object; and displaying a connection between the first icon and the second icon.

52. The method of claim 51, wherein the connection visually indicates an ordering of execution of the first operation and the second operation.

53. The method of claim 51, wherein the first icon, the second icon, and the connection have the appearance of a graphical program.

54. The method of claim 51, wherein the first icon has an appearance corresponding to the first operation; wherein the second icon has an appearance corresponding to the second operation.

55. The method of claim 51, wherein the first icon, the second icon, and the connection displayed on the display operate to visually depict the first and second operations performed by the user.

56. The method of claim 51, further comprising: including the first operation in the application in response to said performing the first operation on the object; and including the second operation in the application in response to said performing the second operation on the object.

57. The method of claim 51, wherein the first icon and the second icon originate from a graphical programming development environment.

58. The method of claim 51, wherein the first icon, the second icon, and the connection displayed between the first icon and the second icon comprise a graphical program, wherein the graphical program is executable to implement the first and second operations.

59. The method of claim 51, wherein the first icon and the second icon are displayed in an order from left to right to visually indicate an ordering of the first operation and the second operation.

60. The method of claim 51, further comprising: performing a third operation on the object in response to user input; displaying a third icon corresponding to the third operation, wherein said displaying the third icon is performed in response to said performing the third operation on the object; displaying a second connection between the second icon and the third icon.

61. The method of claim 51, wherein the object is an image; wherein the first operation and the second operation are image processing operations applied to the image

62. A computer-implemented method for creating an application which performs a function, the method comprising: displaying an object on a display; performing a first operation on the object in response to user input; displaying a first icon corresponding to the first operation, wherein said displaying the first icon is performed in response to said performing the first operation on the object; performing a second operation on the object in response to user input; and displaying a second icon corresponding to the second operation, wherein said displaying the second icon is performed in response to said performing the second operation on the object.

63. The method of claim 62, wherein the first icon has an appearance corresponding to the first operation; and wherein the second icon has an appearance corresponding to the second operation.

64. The method of claim 62, further comprising: storing information regarding the first operation in a data structure in response to said performing the first operation; and storing information regarding the second operation in the data structure in response to said performing the second operation.

65. A computer-implemented method for displaying operations of a machine vision application, the method comprising: including a plurality of machine vision operations in the application in response to user input, wherein the plurality of machine vision operations implement a machine vision function; displaying a plurality of interconnected icons in response to said including the plurality of machine vision operations in the application, wherein each icon corresponds to a machine vision operation in the application, wherein the plurality of interconnected icons visually indicate the machine vision function; and storing information representing the application in a data structure, wherein said storing the information comprises storing information specifying the plurality of machine vision operations in the application.

66. The method of claim 65, wherein said including the plurality of machine vision operations in the application comprises receiving user input individually selecting each machine vision operation for inclusion in the application; wherein said displaying a plurality of interconnected icons comprises displaying at least one icon after each instance of said user input including one of said machine vision operations in the application.

67. The method of claim 65, wherein said including and said displaying comprise: including a first machine vision operation in the application in response to user input; displaying a first icon corresponding to the first machine vision operation in response to said including the first machine vision operation in the application; including a second machine vision operation in the application in response to user input; and displaying a second icon corresponding to the second machine vision operation in response to said including the second machine vision operation in the application; and displaying a connection between the first icon and the second icon.

68. The method of claim 65, wherein said displaying the plurality of interconnected icons comprises displaying a plurality of lines interconnecting the icons, wherein each line connects two icons.

69. The method of claim 68, wherein the plurality of lines includes a first line that connects a first icon to a second icon; wherein the first line indicates that performing the machine vision function includes performing the machine vision operation corresponding to the first icon before performing the machine vision operation corresponding to the second icon.

70. The method of claim 68, wherein the plurality of lines includes a first line that connects a first icon to a second icon; wherein the first line indicates that data produced by the machine vision operation corresponding to the first icon is used by the machine vision operation corresponding to the second icon.

71. The method of claim 65, wherein said displaying the plurality of interconnected icons comprises displaying a graphical programming representation of the plurality of machine vision operations in the application.

72. The method of claim 65, wherein each icon corresponds to a node in a graphical programming development environment.

73. The method of claim 65, further comprising: programmatically generating a graphical program based on the application; wherein said programmatically generating the graphical program comprises programmatically including a plurality of interconnected nodes in the graphical program, wherein each node corresponds to a machine vision operation from the plurality of machine vision operations in the application; wherein the plurality of interconnected nodes implement the machine vision function.

74. The method of claim 73, wherein said programmatically generating the graphical program comprises automatically generating the graphical program.

75. The method of claim 73, wherein, for each node, the node has an iconic appearance that is identical to the icon corresponding to the machine vision operation to which the node corresponds.

76. The method of claim 73, wherein, for each node, the node has an iconic appearance that is substantially the same as the icon corresponding to the machine vision operation to which the node corresponds.

77. The method of claim 73, further comprising: displaying the plurality of interconnected nodes in the graphical program.

78. The method of claim 77, wherein the plurality of interconnected nodes in the graphical program appears identical to the plurality of interconnected icons.

79. The method of claim 77, wherein the plurality of interconnected nodes in the graphical program appears substantially the same as the plurality of interconnected icons.

80. The method of claim 77, wherein said displaying the plurality of interconnected icons comprises displaying the plurality of interconnected icons in a first window; wherein said displaying the plurality of interconnected nodes in the graphical program comprises displaying the graphical program in a second window.

81. The method of claim 73, further comprising: executing the graphical program; wherein said executing the graphical program comprises executing the plurality of interconnected nodes to perform the machine vision function.

82. The method of claim 65, wherein said including the plurality of machine vision operations in the application comprises receiving user input individually selecting each machine vision operation for inclusion in the application; wherein the method further comprises programmatically including a node in a graphical program in response to each machine vision operation selected for inclusion in the application, wherein each node is operable to perform the respective machine vision operation when the graphical program is executed.

83. The method of claim 82, wherein each node included in the graphical program in response to selecting a machine vision operation has an iconic appearance that is identical to the icon corresponding to the machine vision operation.

84. The method of claim 82, wherein each node included in the graphical program in response to selecting a machine vision operation has an iconic appearance that is substantially the same as the icon corresponding to the machine vision operation.

85. The method of claim 82, further comprising: removing a first machine vision operation from the application; and programmatically removing the node operable to perform the first machine vision operation from the graphical program in response to said removing the first machine vision operation from the application.

86. The method of claim 65, wherein said including the plurality of machine vision operations in the application comprises receiving user input individually selecting each machine vision operation for inclusion in the application; wherein the method further comprises executing each machine vision operation as the machine vision operation is selected for inclusion in the application.

87. The method of claim 86, further comprising: for each machine vision operation, displaying results of said executing the machine vision operation, wherein said displaying the results of said executing the machine vision operation comprises displaying the results together with the icon that corresponds to the machine vision operation.

88. The method of claim 87, wherein said displaying the results together with the icon that corresponds to the machine vision operation comprises displaying a pass/fail result together with the icon that corresponds to the machine vision operation.

89. The method of claim 88, wherein the plurality of machine vision operations includes a first machine vision operation; wherein executing the first machine vision operation produces a fail result; wherein displaying the pass/fail result together with the icon that corresponds to the first machine vision operation comprises displaying information indicating the fail result; wherein the method further comprises displaying information indicating a reason for the fail result.

90. The method of claim 87, wherein one or more of the plurality of machine vision operations in the application are operable to analyze an image of a unit under test (UUT); wherein, for each machine vision operation operable to analyze an image of the UUT, said displaying the results together with the icon that corresponds to the machine vision operation comprises displaying a pass/fail result of the analysis together with the icon that corresponds to the machine vision operation.

91. The method of claim 86, further comprising: displaying an image; wherein said executing each machine vision operation comprises applying the machine vision operation to the displayed image; wherein the method further comprises re-displaying the image after applying each machine vision operation to the image.

92. The method of claim 65, further comprising: for each machine vision operation, displaying a name of the machine vision operation together with the icon that corresponds to the machine vision operation.

93. The method of claim 65, further comprising: executing the application; wherein said executing the application comprises executing the plurality of machine vision operations in the application to perform the machine vision function.

94. The method of claim 93, wherein said performing the machine vision function includes analyzing one or more images.

95. The method of claim 93, wherein said performing the machine vision function includes acquiring one or more images for analysis.

96. A computer-implemented method for displaying operations of a machine vision application, the method comprising: including a first machine vision operation in the machine vision application; displaying a first icon corresponding to the first machine vision operation in response to said including the first machine vision operation in the machine vision application; including a second machine vision operation in the machine vision application; and displaying a second icon corresponding to the second machine vision operation in response to said including the second machine vision operation in the machine vision application.

97. The method of claim 96, further comprising: displaying a connection between the first icon and the second icon.

98. The method of claim 96, further comprising: displaying an image; applying the first machine vision operation to the image in response to said including the first machine vision operation in the machine vision application; and applying the second machine vision operation to the image in response to said including the second machine vision operation in the machine vision application.

99. The method of claim 96, further comprising: displaying an image; applying the first machine vision operation to the image in response to user input, wherein the first machine vision operation is included in the machine vision application in response to said applying the first machine vision operation to the image; and applying the second machine vision operation to the image in response to user input, wherein the second machine vision operation is included in the machine vision application in response to said applying the second machine vision operation to the image.

100. The method of claim 96, wherein the first icon corresponds to a first node in a graphical programming development environment; and wherein the second icon corresponds to a second node in the graphical programming development environment.

101. The method of claim 96, further comprising: displaying first results of executing the first machine vision operation, wherein said displaying first results of executing the first machine vision operation comprises displaying the first results together with the first icon; and displaying second results of executing the second machine vision operation, wherein said displaying second results of executing the second machine vision operation comprises displaying the second results together with the second icon.

102. The method of claim 101, wherein said displaying the first results together with the first icon comprises displaying a first pass/fail result together with the first icon; and wherein said displaying the second results together with the second icon comprises displaying a second pass/fail result together with the second icon.

103. The method of claim 96, further comprising: displaying a name of the first machine vision operation together with the first icon; and displaying a name of the second machine vision operation together with the second icon.

104. A computer-implemented method for creating an application which performs a machine vision function, the method comprising: displaying an image on a display; performing a first machine vision operation on the image in response to user input; displaying a first icon corresponding to the first machine vision operation, wherein said displaying the first icon is performed in response to said performing the first machine vision operation on the image; performing a second machine vision operation on the image in response to user input; displaying a second icon corresponding to the second machine vision operation, wherein said displaying the second icon is performed in response to said performing the second machine vision operation on the image; and displaying a connection between the first icon and the second icon.

105. The method of claim 104, wherein the first icon, the second icon, and the connection have the appearance of a graphical program.

106. The method of claim 104, wherein the first icon, the second icon, and the connection have the appearance of a graphical data flow program.

107. The method of claim 104, wherein the first icon has an appearance corresponding to the first machine vision operation; wherein the second icon has an appearance corresponding to the second machine vision operation.

108. The method of claim 104, wherein the first icon, the second icon, and the connection displayed on the display operate to visually depict the first E nd second machine vision operations performed by the user.

109. The method of claim 104, further comprising: including the first machine vision operation in the application in response to said performing the first machine vision operation on the image; and including the second machine vision operation in the application in response to said performing the second machine vision operation on the image.

110. The method of claim 104, wherein the first icon and the second icon originate from a graphical programming development environment.

111. The method of claim 104, wherein the first icon, the second icon, and the connection displayed between the first icon and the second icon comprise a graphical program, wherein the graphical program is executable to implement the first and second machine vision operations.

112. The method of claim 104, wherein the first icon and the second icon are displayed in an order from left to right to visually indicate an ordering of the first machine vision operation and the second machine vision operation.

113. The method of claim 104, further comprising: performing a third machine vision operation on the image in response to user input; displaying a third icon corresponding to the third machine vision operation, wherein said displaying the third icon is performed in response to said performing the third machine vision operation on the image; and displaying a second connection between the second icon and the third icon.

114. A computer-implemented method for creating an application which performs a machine vision function, the method comprising: displaying an image on a display; performing a first machine vision operation on the image in response to user input; displaying a first icon corresponding to the first machine vision operation, wherein said displaying the first icon is performed in response to said performing the first machine vision operation on the image; performing a second machine vision operation on the image in response to user input; and displaying a second icon corresponding to the second machine vision operation, wherein said displaying the second icon is performed in response to said performing the second machine vision operation on the image.

115. The method of claim 114, wherein the first icon has an appearance corresponding to the first machine vision operation; and wherein the second icon has an appearance corresponding to the second machine vision operation.

FIELD OF THE INVENTION

[0001] The present invention relates to the fields of software prototyping environments and graphical programming. More particularly, the invention relates to displaying operations in an application by using a graphical programming visual representation. One embodiment of the invention relates to displaying machine vision operations in an application by using a graphical programming visual representation, where the machine vision operations implement a machine vision function.

DESCRIPTION OF THE RELATED ART

[0002] Traditionally, high level text-based programming languages have been used by programmers in writing application programs. Many different high level programming languages exist, including BASIC, C, Java, FORTRAN, Pascal, COBOL, ADA, APL, etc. Programs written in these high level languages are translated to the machine language level by translators known as compilers or interpreters. The high level programming languages in this level, as well as the assembly language level, are referred to herein as text-based programming environments.

[0003] Increasingly, computers are required to be used and programmed by those who are not highly trained in computer programming techniques. When traditional text-based programming environments are used, the user's programming skills and ability to interact with the computer system often become a limiting factor in the achievement of optimal utilization of the computer system.

[0004] There are numerous subtle complexities which a user must master before he can efficiently program a computer system in a text-based environment. The task of programming a computer system to model or implement a process often is further complicated by the fact that a sequence of mathematical formulas, steps or other procedures customarily used to conceptually model a process often does not closely correspond to the traditional text-based programming techniques used to program a computer system to model such a process. In other words, the requirement that a user program in a text-based programming environment places a level of abstraction between the user's conceptualization of the solution and the implementation of a method that accomplishes this solution in a computer program. Thus, a user often must substantially master different skills in order to both conceptualize a problem or process and then to program a computer to implement a solution to the problem or process. Since a user often is not fully proficient in techniques for programming a computer system in a text-based environment to implement his solution, the efficiency with which the computer system can be utilized often is reduced.

[0005] Examples of fields in which computer systems are employed to interact with physical systems are the fields of instrumentation, process control, industrial automation, and simulation. Computer measurement and control of devices such as instruments or industrial automation hardware has become increasingly desirable in view of the increasing complexity and variety of instruments and devices available for use. However, due to the wide variety of possible testing and control situations and environments, and also the wide array of instruments or devices available, it is often necessary for a user to develop a custom program to control a desired system.

[0006] As discussed above, computer programs used to control such systems traditionally had to be written in text-based programming languages such as, for example, assembly language, C, FORTRAN, BASIC, etc. Traditional users of these systems, however, often were not highly trained in programming techniques and, in addition, text-based programming languages were not sufficiently intuitive to allow users to use these languages without training. Therefore, implementation of such systems frequently required the involvement of a programmer to write software for control and analysis of instrumentation or industrial automation data. Thus, development and maintenance of the software elements in these systems often proved to be difficult.

[0007] U.S. Pat. Nos. 4,901,221; 4,914,568; 5,291,587; 5,301,301; and 5,301,336; among others, to Kodosky et al disclose a graphical system and method for modeling a process, i.e., a graphical programming environment which enables a user to easily and intuitively model a process. The graphical programming environment disclosed in Kodosky et al can be considered a higher and more intuitive way in which to interact with a computer. A graphically based programming environment can be represented at a level above text-based high level programming languages such as C, Basic, Java, etc.

[0008] The method disclosed in Kodosky et al allows a user to construct a diagram using a block diagram editor. The block diagram may include a plurality of interconnected icons such that the diagram created graphically displays a procedure or method for accomplishing a certain result, such as manipulating one or more input variables and/or producing one or more output variables. In response to the user constructing a diagram or graphical program using the block diagram editor, data structures and/or program instructions may be automatically constructed which characterize an execution procedure that corresponds to the displayed procedure. The graphical program may be compiled or interpreted by a computer.

[0009] Therefore, Kodosky et al teaches a graphical programming environment wherein a user places or manipulates icons and interconnects or “wires up” the icons in a block diagram using a block diagram editor to create a graphical “progam.” A graphical program for performing an instrumentation, measurement or automation function, such as measuring a Unit Under Test (UUT) or device, controlling or modeling instruments, controlling or measuring a system or process, or for modeling or simulating devices, may be referred to as a virtual instrument (VI). Thus, a user can create a computer program solely by using a graphically based programming environment. This graphically based programming environment may be used for creating virtual instrumentation systems, modeling processes, control, simulation, and numerical analysis, as well as for any type of general programming.

[0010] A graphical program may have a graphical user interface. For example, in creating a graphical program, a user may create a front panel or user interface panel. The front panel may include various graphical user interface elements or front panel objects, such as user interface controls and/or indicators, that represent or display the respective input and output that will be used by the graphical program or VI, and may include other icons which represent devices being controlled. The front panel may be comprised in a single window of user interface elements, or may comprise a plurality of individual windows each having one or more user interface elements, wherein the individual windows may optionally be tiled together. When the controls and indicators are created in the front panel, corresponding icons or terminals may be automatically created in the block diagram by the block diagram editor. Alternatively, the user can place terminal icons in the block diagram which may cause the display of corresponding front panel objects in the front panel, either at edit time or later at run time. As another example, the front panel may comprise front panel objects, e.g., the GUI, embedded in the block diagram.

[0011] During creation of the block diagram portion of the graphical program, the user may select various function nodes or icons that accomplish his desired result and connect the function nodes together. For example, the function nodes may be connected in one or more of a data flow, control flow, and/or execution flow format. The function nodes may also be connected in a “signal flow” format, which is a subset of data flow. The function nodes may be connected between the terminals of the various user interface elements, e.g., between the respective controls and indicators. Thus the user may create or assemble a graphical program, referred to as a block diagram, graphically representing the desired process. The assembled graphical program may be represented in the memory of the computer system as data structures and/or program instructions. The assembled graphical program, i.e., these data structures, may then be compiled or interpreted to produce machine language that accomplishes the desired method or process as shown in the block diagram.

[0012] Input data to a graphical program may be received from any of various sources, such as from a device, unit under test, a process being measured or controlled, another computer program, or from a file. Also, a user may input data to a graphical program or virtual instrument using a graphical user interface, e.g., a front panel as described above. The input data may propagate through the data flow block diagram or graphical program and appear as changes on the output indicators. In an instrumentation application, the front panel can be analogized to the front panel of an instrument. In an industrial automation application the front panel can be analogized to the MMI (Man Machine Interface) of a device. The user may adjust the controls on the front panel to affect the input and view the output on the respective indicators. Alternatively, the front panel may be used merely to view the input and output, or just the output, and the input may not be interactively manipulable by the user during program execution.

[0013] Thus, graphical programming has become a powerful tool available to programmers. Graphical programming development environments such as the National Instruments LabVIEW product have become very popular. Tools such as LabVIEW have greatly increased the productivity of programmers, and increasing numbers of programmers are using graphical programming environments to develop their software applications. In particular, graphical programming tools are being used for test and measurement, data acquisition, process control, man machine interface (MMI), supervisory control and data acquisition (SCADA) applications, simulation, image processing/machine vision applications, and motion control, among others.

[0014] As new techniques and computerized methods are developed for a given problem domain, specialized software applications for developing solutions to problems in that domain are often created. These specialized applications can provide an environment that is conducive to rapidly and conveniently prototyping a problem solution. Hence, these applications are also referred to herein as “prototyping environments”. A prototyping environment may integrate various capabilities to aid developers of problem solutions. For example, a prototyping environment may provide a library of operations that are specific to one or more problem domains and may enable a user to select various operations from the library for inclusion in an application. Prior art prototyping environments have used various techniques to display the operations in the application, such as using text information arranged in various ways, such as a table or tree view.

SUMMARY

[0015] One embodiment of the invention relates to displaying operations in an application by using a graphical programming representation (or block diagram representation). A plurality of operations may be included in the application, e.g., in response to user input received to a graphical user interface of a prototyping environment. A plurality of interconnected icons may be automatically (i.e., programmatically) displayed in response to including the plurality of operations in the application. Each icon may correspond to an operation included in the application.

[0016] In one embodiment, as each operation is added to the application, a corresponding icon may be displayed and may be visually interconnected to one or more other icons that may have been previously displayed. Each icon may be designed to represent the respective operation using an intuitive picture. The plurality of interconnected icons may visually indicate the first function implemented by the plurality of operations in the application. In one embodiment, as each operation is added to the application, the respective operation may also optionally be displayed in a graphical user interface of the application development environment.

[0017] Thus, when the user adds a first operation to the application, a corresponding first icon may be displayed representing the first operation. In addition, a name or other indicia of the respective first operation may optionally be displayed in a graphical user interface of the application development environment. When the user adds a second operation to the application, a corresponding second icon may then be displayed representing the second operation. In addition, a name or other indicia of the respective second operation may be displayed in the graphical user interface of the application development environment. The second icon may be automatically connected to the first icon to indicate the sequence of operations. The above process repeats, where new icons are added to the block diagram representation (and connected with already displayed icons) as the user adds operations to the application.

[0018] As noted above, the system may display interconnecting lines between displayed icons, where each line connects two (or more) icons. The lines may have various meanings or semantics. For example, where the plurality of lines includes a first line that connects a first icon to a second icon, the first line may indicate that performing the first function includes performing the operation corresponding to the first icon before performing the operation corresponding to the second icon. In another embodiment, the first line may indicate that data produced by the operation corresponding to the first icon is used by the operation corresponding to the second icon (data flow). In various embodiments, the interconnection lines or links may visually represent one or more of data flow, control flow, and/or execution flow for the plurality of operations in the application. Thus, the plurality of interconnected icons may be referred to as a graphical programming visual representation (or a block diagram representation) of the plurality of operations in the application. The icons may be displayed in various ways, e.g., in a left-to-right manner or top-to-bottom manner, to visually represent an ordering of the operations or to visually represent other relationships among the operations.

[0019] The plurality of operations may implement a first function. In various embodiments the plurality of operations in the application may implement any of various functions or solutions. As one example, the plurality of operations may include one or more machine vision operations that implement a machine vision function, such as for acquiring and/or analyzing images. As another example, the plurality of operations may include one or more motion control operations that implement a motion control function, such as for controlling motion of a device and/or controlling a device to move an object. As another example, the plurality of operations may include one or more data acquisition (DAQ) operations that implement a DAQ function, such as for acquiring measurement data from a device.

[0020] In one embodiment, the user may perform operations on a displayed image or object (e.g., an image of an object displayed on the display), and this may cause the operations to be stored in the application. The object may also be updated on the display to reflect the operations performed. The system may display a graphical user interface with various menus, dialogs, controls or other GUI elements that the user may use to apply operations to an image. As the user performs each respective operation on the object, a corresponding icon may be displayed to visually represent the operation. Thus, as the user performs each additional operation on the object, a corresponding icon may be immediately displayed to visually represent this operation, and the newly displayed icon may be interconnected with one or more other existing icons. Thus, as noted above, the system may also display a connection between a newly displayed icon and a previously displayed icon to represent their relationship.

[0021] In an embodiment targeted toward machine vision, the system may display an image. The system may also display a graphical user interface that may be used by the user to apply various operations (e.g., image processing or machine vision operations) to the displayed image. The user may use the GUI to perform operations on the displayed image, may draw regions of interest (ROIs) on the image, etc. These actions by the user may cause visual changes to the image according to the operation performed. As the user performs machine vision operations on the displayed image, the machine vision operations may be included or stored in the application, and a corresponding icon may also be displayed to visually represent the machine vision operation. Thus, in one embodiment, as the user performs each operation to the image, 1) the operation is stored in memory, 2) the image is changed according to the operation, 3) a new icon is displayed in the block diagram representation, where the new icon represents the operation, and 4) the new icon may be connected with a pre-existing icon (assuming the new icon does not represent the first operation). The operation may also be displayed in an alternative form in the GUI, such as the textual name of the operation being displayed in a script window.

[0022] In one embodiment, each displayed icon may correspond to a node in a graphical programming development environment. In other words, for each operation provided by the prototyping environment, there may be an equivalent node in the graphical programming development environment. Each of these nodes in the graphical programming development environment may have an iconic appearance that is identical to or substantially the same as the corresponding icon.

[0023] Information representing the application may be stored, e.g., in a data structure or file. The stored information may include information specifying the plurality of operations in the application. For example, the data structure or file may store names or other identifiers that identify the operations in the application. The data structure or file may also store information regarding various properties or parameter values configured for one or more operations in the application.

[0024] In one embodiment, each operation may be executed as the operation is included in the application. In another embodiment, the entire application may be executed together. Executing the application may include executing the plurality of operations in the application to perform the first function.

[0025] Also, in one embodiment the prototyping environment may be operable to programmatically (automatically) generate a graphical program based on the application, where the graphical program is operable to perform the first function. The generated graphical program may appear the same as or substantially the same as the displayed plurality of interconnected icons. The graphical program may then be executed to perform the first function. In another embodiment, the displayed graphical programming representation (or block diagram representation) is itself an executable graphical program that the user may run, use execution highlighting with, etc.

BRIEF DESCRIPTION OF THE DRAWINGS

[0026] A better understanding of the present invention can be obtained when the following detailed description of the preferred embodiment is considered in conjunction with the following drawings, in which:

[0027] FIG. 1 illustrates a computer system that may execute a prototyping environment application for developing an application including a plurality of operations;

[0028] FIGS. 2A and 2B illustrate representative instrumentation and process control systems including various I/O interface options;

[0029] FIG. 3 illustrates one embodiment of a machine vision inspection system;

[0030] FIG. 4 is a block diagram of the computer system of FIGS. 1, 2A , 2 B and/or 3 ;

[0031] FIG. 5 is a flowchart diagram illustrating one embodiment of a method for displaying operations in an application by using a graphical programming visual representation;

[0032] FIG. 6 is a flowchart diagram illustrating one embodiment of a method for programmatically generating a graphical program based on an application;

[0033] FIGS. 7 - 8 and 11 - 12 illustrate an exemplary graphical user interface for one embodiment of a machine vision prototyping environment application;

[0034] FIG. 9 illustrates a block diagram of a graphical program programmatically generated based on an application created using the machine vision prototyping environment application of FIGS. 7 and 8 ; and

[0035] FIG. 10 illustrates a user interface for the programmatically generated graphical program of FIG. 9 .

[0036] While the invention is susceptible to various modifications and alternative forms specific embodiments are shown by way of example in the drawings and are herein described in detail. It should be understood however, that drawings and detailed description thereto are not intended to limit the invention to the particular form disclosed, but on the contrary the invention is to cover all modifications, equivalents and alternative following within the spirit and scope of the present invention as defined by the appended

DETAILED DESCRIPTION

[0037] Incorporation by Reference

[0038] The following references are hereby incorporated by reference in their entirety as though fully and completely set forth herein.

[0039] U.S. patent application Ser. No. 09/518,492 titled “System and Method for Programmatically Creating a Graphical Program,” filed Mar. 3, 2000.

[0040] U.S. patent application Ser. No. 09/745,023 titled “System and Method for Programmatically Generating a Graphical Program in Response to Program Information,” filed Dec. 20, 2000.

[0041] U.S. patent application Ser. No. 09/587,682 titled “System and Method for Automatically Generating a Graphical Program to Perform an Image Processing Algorithm,” filed Jun. 6, 2000.

[0042] U.S. patent application Ser. No. 09/595,003 titled “System and Method for Automatically Generating a Graphical Program to Implement a Prototype,” filed Jun. 13, 2000.

[0043] U.S. patent application Ser. No. 10/051,474 titled “System and Method for Graphically Creating a Sequence of Motion Control Operations,” filed Jan. 18, 2002.

[0044] U.S. patent application Ser. No. ______ titled “System and Method for Automatically Porting a Prototype to an Embedded Device,” filed ______.

[0045] FIG. 1 —Computer System Executing a Prototyping Environment

[0046] Knowledge of computer-implemented techniques for solving problems in a wide array of fields is expanding at a rapid pace. As new techniques and computerized methods are developed for a given problem domain, specialized software programs for developing solutions to problems in that domain are often created. These specialized software programs can provide an environment that is conducive to rapidly and conveniently prototyping a problem solution (or “application”). Hence, these programs are also referred to herein as “prototyping environments”.

[0047] FIG. 1 illustrates a computer system 82 . The computer system 82 may execute a prototyping environment application for creating an application. A prototyping environment may integrate various capabilities to aid a user in developing problem solutions, depending on the particular problem domain. For example, a prototyping environment may provide a library of operations that are specific to a problem domain and may enable the user to select various operations from the library for inclusion in the application. The prototyping environment may include a graphical user interface that is streamlined for interactively experimenting with various parameters or properties associated with the selected operations and seeing the effects of adjusting the parameters. A prototyping environment may also include capabilities for simulating real-world objects or processes.

[0048] As used herein, the term “application” refers to the result that may be produced by such a prototyping environment. In other words, the term “application” refers to a set of operations, and possibly associated parameters, that may be selected by the user using a prototyping environment. The term “application” is intended to encompass a sequence, solution, prototype, algorithm, script, or similar concept. Thus, a prototyping environment may be used to generate an application which represents an algorithm or process designed by the user in the prototyping environment.

[0049] Prototyping environments are usually designed for ease of use and may be specialized for developing solutions for a particular problem domain. They may enable users to create a computer-implemented solution to a problem without requiring the users to utilize or understand traditional programming techniques. For example, the prototyping environment may aid the user in creating an application including a plurality of operations, without requiring the user to write programming language code. Instead of writing code, the user may interact at a higher level with the graphical user interface of the prototyping environment to create the application.

[0050] As described below, according to one embodiment of the present invention, the prototyping environment may be operable to display the operations included in the application by using a graphical programming visual representation of the operations and their relationship with each other. For example, a plurality of interconnected icons may be displayed in response to including operations in the application. Each icon may correspond to an operation in the application, so that the plurality of interconnected icons visually indicates a function or process performed by the operations. As each operation is added to the application, a corresponding icon may be immediately visually displayed in the graphical programming visual representation (or block diagram representation).

[0051] Prototyping environments may be developed for many different problem domains. Various exemplary prototyping environments are mentioned herein. One example is a motion control prototyping environment application for developing a sequence of motion control operations. A motion control sequence is also referred to herein as a motion control “application”.

[0052] Computer-based motion control involves precisely controlling the movement of a device or system. Computer-based motion control is widely used in many different types of applications, including applications in the fields of industrial automation, process control, test and measurement automation, robotics, and integrated machine vision, among others. A typical computer-based motion system includes components such as the moving mechanical device(s), a motor with feedback and motion I/O, a motor drive unit, a motion controller, and software to interact with the motion controller.

[0053] A motion control prototyping environment may be designed to enable a user to easily and efficiently develop/application a motion control sequence or application without requiring the user to perform programming, e.g., without needing to write or construct code in any programming language. For example, the environment may provide a graphical user interface (GUI) enabling the user to develop/application the motion control sequence at a high level, by selecting from and configuring a sequence of motion control operations using the GUI. According to one embodiment of the present invention, the motion control prototyping environment may be operable to display the motion control operations included in the motion control sequence by using a graphical programming visual representation of the motion control operations and their relationship with each other.

[0054] Another example of a prototyping environment described herein is a machine vision prototyping environment. A machine vision prototyping environment may enable a user to rapidly develop an application including a plurality of machine vision operations. As used herein, the term “machine vision operation” may include any of various types of operations related to image analysis, image processing, image acquisition, or other machine vision-related operations. A machine vision application may specify a sequence, script, process, or algorithm for a machine vision application. For example, the machine vision prototyping environment may include a graphical user interface enabling the user to easily apply various machine vision operations to an image and see the results, in order to develop the desired machine vision application.

[0055] In various embodiments, any of various types of machine vision operations may be supported, including image acquisition functions, filtering functions, morphology functions, histogram functions, particle analysis functions, edge detection functions, pattern matching functions, color matching functions, shape matching functions, optical character recognition (OCR), optical character verification (OCV), and 1D and 2D barcodes, etc. In one embodiment, the user may apply various machine vision operations to an image, and the operations may be included in the machine vision application. For example, each operation may be recorded as a step in a script. A script may essentially specify an algorithm; i.e., an algorithm may be defined by the plurality of steps or operations in a script. The user may create the machine vision application without specifying or writing programming language code, similarly as described above. According to one embodiment of the present invention, the machine vision prototyping environment may be operable to display the machine vision operations included in a machine vision application by using a graphical programming visual representation of the machine vision operations and their relationship with each other.

[0056] In various embodiments, the computer system 82 may execute a prototyping environment application for developing applications related to any of various other types of applications, in addition to motion control and/or machine vision. In general, a “prototyping environment” may refer to a specialized application that provides an environment that is conducive to rapidly and conveniently prototyping a problem solution, preferably without requiring the user to write code in a programming language or minimizing the amount of code the user would otherwise have to write. Other examples of prototyping environments include:

[0057] an instrumentation environment for interacting with hardware instruments, e.g., in order to initialize an instrument, acquire data from the instrument, analyze the acquired data, etc.

[0058] a circuit design environment for developing and testing circuit designs

[0059] a sound processing environment for applying various audio operations to a sound clip, e.g., in order to analyze the sound clip, eliminate background noise, etc.

[0060] In various embodiments, the operations in any of various types of applications may be displayed by using a graphical programming visual representation. As a few examples, the application may be related to fields such as image processing, image analysis, machine vision, process control, automation, test and measurement, simulation, motion control, robotics, audio, video, graphics, telecommunications, and workflow processes, among others.

[0061] It is noted that a prototyping environment may also enable a user to develop an application which includes operations related to multiple different fields or technologies. For example, one embodiment of a prototyping environment may provide a library of operations including one or more machine vision operations, one or more motion control operations, and one or more data acquisition (DAQ) operations. In this example, the motion control prototyping environment may be referred to as a MC/MV/DAQ prototyping environment. (The abbreviation “MC/MV/DAQ” is used herein to refer to “motion control/machine vision/DAQ”.) For example, a MC/MV/DAQ prototyping environment may be used to create an application including machine vision operations for acquiring and analyzing images of an object, motion control operations for moving the object, and DAQ operations for acquiring measurement data of the object. Such an application may be useful, for example, to specify an automated inspection process performed on manufactured objects.

[0062] Referring again to FIG. 1 , the computer system 82 may be any type of computer system, including a personal computer system, mainframe computer system, workstation, network appliance, Internet appliance, personal digital assistant (PDA), television system, smart camera (or other type of smart sensor), smart motion controller, or other device. In general, the term “computer system” can be broadly defined to encompass any device having at least one processor that executes instructions from a memory medium.

[0063] The computer system 82 may include a memory medium(s) on which one or more computer programs or software components may be stored according to one embodiment of the present invention. For example, the memory medium may store a prototyping environment application program (or portion of such an application program) such as described above. The memory medium may also store one or more applications created using the prototyping environment application. The memory medium may also store a graphical program automatically generated by the prototyping environment application based on an application, as described below. The memory medium may also store a graphical programming development environment with which a generated graphical program is associated. The memory medium may also store operating system software, as well as other software for operation of the computer system 82 .

[0064] The term “memory medium” is intended to include an installation medium, e.g., a CD-ROM, floppy disks 104 , or tape device; a computer system memory or random access memory such as DRAM, SRAM, EDO RAM, Rambus RAM, etc.; or a non-volatile memory such as a magnetic media, e.g., a hard drive, or optical storage. The memory medium may comprise other types of memory as well, or combinations thereof. In addition, the memory medium may be located in a first computer in which the programs are executed, or may be located in a second different computer which connects to the first computer over a network, such as the Internet. In the latter instance, the second computer may provide program instructions to the first computer for execution.

[0065] FIGS. 2 A and 2 B—Instrumentation and Industrial Automation Systems

[0066] FIGS. 2A and 2B illustrate embodiments involved with performing test and/or measurement functions and/or controlling and/or modeling instrumentation or industrial automation hardware. However, it is noted that the present invention can be used for a plethora of applications and is not limited to instrumentation or industrial automation applications. In other words, the following description is exemplary only, and the present invention may be used in any of various types of systems.

[0067] FIG. 2A illustrates an instrumentation control system 100 . The system 100 includes a host computer 82 that connects to one or more instruments. The host computer 82 includes a CPU, a display screen, memory, and one or more input devices such as a mouse or keyboard as shown. The computer 82 may connect through the one or more instruments to analyze, measure, or control a unit under test (UUT) or process 150 .

[0068] In one embodiment, the host computer 82 may store a prototyping environment application operable to create an application including operations that interact with or control the one or more instruments. As described below, the prototyping environment application may be operable to display the operations in the application by using a graphical programming visual representation of the operations and their relationship with each other. In one embodiment, the host computer 82 may store an application created by the prototyping environment application and/or may execute the application to interact with or control the one or more instruments. In another embodiment, the host computer 82 may store a graphical program based on an application. For example, as described below, the graphical program may be programmatically (automatically) generated based on the application. In this instance, the host computer 82 may execute the graphical program to interact with or control the one or more instruments.

[0069] The one or more instruments may include a GPIB instrument 112 and associated GPIB interface card 122 , a data acquisition board 114 and associated signal conditioning circuitry 124 , a VXI instrument 116 , a PXI instrument 118 , a video device 132 and associated image acquisition card 134 , a motion control device 136 and associated motion control interface card 138 , and/or one or more computer based instrument cards 142 , among other types of devices.

[0070] The GPIB instrument 112 is coupled to the computer 82 via the GPIB interface card 122 provided by the computer 82 . In a similar manner, the video device 132 is coupled to the computer 82 via the image acquisition card 134 , and the motion control device 136 is coupled to the computer 82 through the motion control interface card 138 . The data acquisition board 114 is coupled to the computer 82 , and may interface through signal conditioning circuitry 124 to the UUT. The signal conditioning circuitry 124 preferably comprises an SCM (Signal Conditioning eXtensions for Instrumentation) chassis comprising one or more SCXI modules 126 .

[0071] The GPIB card 122 , the image acquisition card 134 , the motion control interface card 138 , and the DAQ card 114 are typically plugged in to an I/O slot in the computer 82 , such as a PCI bus slot, a PC Card slot, or an ISA, EISA or MicroChannel bus slot provided by the computer 82 . However, these cards 122 , 134 , 138 and 114 are shown external to computer 82 for illustrative purposes.

[0072] The VXI chassis or instrument 116 is coupled to the computer 82 via a VXI bus, MXI bus, or other serial or parallel bus provided by the computer 82 . The computer 82 preferably includes VXI interface logic, such as a VXI, MXI or GPIB interface card (not shown), which interfaces to the VXI chassis 116 . The PXI chassis or instrument is preferably coupled to the computer 82 through the computer's PCI bus.

[0073] A serial instrument (not shown) may also be coupled to the computer 82 through a serial port, such as an RS-232 port, USB (Universal Serial bus) or IEEE 1394 or 1394.2 bus, provided by the computer 82 . In typical instrumentation control systems an instrument will not be present of each interface type, and in fact many systems may only have one or more instruments of a single interface type, such as only GPIB instruments.

[0074] The instruments are coupled to the unit under test (UUT) or process 150 , or are coupled to receive field signals, typically generated by transducers. The system 100 may be used in a data acquisition and control application, in a test and measurement application, a process control application, or a man-machine interface application.

[0075] FIG. 2B illustrates an exemplary industrial automation system 160 . The industrial automation system 160 is similar to the instrumentation or test and measurement system 100 shown in FIG. 2A . Elements which are similar or identical to elements in FIG. 2A have the same reference numerals for convenience. The system 160 includes a computer 82 that connects to one or more devices or instruments. The computer 82 includes a CPU, a display screen, memory, and one or more input devices such as a mouse or keyboard as shown. The computer 82 may connect through the one or more devices to a process or device 150 to perform an automation function, such as MM (Man Machine Interface), SCADA (Supervisory Control and Data Acquisition), portable or distributed data acquisition, process control, advanced analysis, or other control.

[0076] In one embodiment, the host computer 82 may store a prototyping environment application operable to create an application including operations involved with the automation function performed by the automation system 160 . As described below, the prototyping environment application may be operable to display the operations in the application by using a graphical programming visual representation of the operations and their relationship with each other. In one embodiment, the host computer 82 may store an application created by the prototyping environment application and/or may execute the application to perform the automation function. In another embodiment, the host computer 82 may store a graphical program based on an application. For example, as described below, the graphical program may be programmatically (automatically) generated based on the application. In this instance, the host computer 82 may execute the graphical program to perform the automation function.

[0077] The one or more devices may include a data acquisition board 114 and associated signal conditioning circuitry 124 , a PXI instrument 118 , a video device 132 and associated image acquisition card 134 , a motion control device 136 and associated motion control interface card 138 , a fieldbus device 170 and associated fieldbus interface card 172 , a PLC (Programmable Logic Controller) 176 , a serial instrument 182 and associated serial interface card 184 , or a distributed data acquisition system, such as the Fieldpoint system available from National Instruments, among other types of devices.

[0078] The DAQ card 114 , the PXI chassis 118 , the video device 132 , and the image acquisition card 136 are preferably connected to the computer 82 as described above. The serial instrument 182 is coupled to the computer 82 through a serial interface card 184 , or through a serial port, such as an RS-232 port, provided by the computer 82 . The PLC 176 couples to the computer 82 through a serial port, Ethernet port, or a proprietary interface. The fieldbus interface card 172 is preferably comprised in the computer 82 and interfaces through a fieldbus network to one or more fieldbus devices. Each of the DAQ card 114 , the serial card 184 , the fieldbus card 172 , the image acquisition card 134 , and the motion control card 138 are typically plugged in to an I/O slot in the computer 82 as described above. However, these cards 114 , 184 , 172 , 134 , and 138 are shown external to computer 82 for illustrative purposes. In typical industrial automation systems a device will not be present of each interface type, and in fact many systems may only have one or more devices of a single interface type, such as only PLCs. The devices are coupled to the device or process 150 .

[0079] FIG. 3 —Machine Vision Inspection System

[0080] FIG. 3 is a diagram illustrating one embodiment of a machine vision inspection system (also referred to as an image acquisition and analysis system) for inspecting objects 200 . In FIG. 3 , the objects 200 are shown in the form of display devices. However, in various embodiments, the objects 200 may have any of various types of shapes and sizes and may include any kind of objects, devices, or products.

[0081] FIG. 3 illustrates a plurality of objects 200 that move along a manufacturing apparatus 204 . The system includes one or more cameras 212 operable to acquire images of the objects 200 as they move along the manufacturing apparatus 204 . In another embodiment, the objects may not move along a manufacturing apparatus 204 . For example, the system may be utilized to acquire and analyze images of a single object 200 , or the objects 200 may be placed in front of the camera 212 in a stationary manner. In various embodiments, any number of cameras 212 may be used. The camera(s) 212 may include any type of camera or device operable to acquire images of the objects 200 .

[0082] As shown in FIG. 3 , the camera 212 may be connected to a computer system 82 , which is operable to receive the images acquired by the camera. In another embodiment, the computer system 82 may receive images acquired from multiple cameras. For example, the computer system 82 may include one or more image acquisition boards, each for capturing one or more images. The computer system 82 may then analyze the images captured by the image acquisition board(s). Alternatively, the image acquisition board(s) may include on-board processors and memory for performing a portion or all of the image analysis.

[0083] In various embodiments, the images of the objects 200 may be analyzed in any of various ways, e.g., may be analyzed for any of various kinds of characteristics or defects, using any of various machine vision or image processing techniques. As a few examples, an image of an object may be analyzed to detect: physical surface defects (scratches, etc.); one or more components located correctly on the object; a correct label on the object; a correct marking on the object; correct color information on the object, etc.

[0084] The results of the image analyses may be used to determine whether an object 200 meets desired production standards. The determination may be made based on any of various criteria, as desired for a particular application. If separate computer systems are used to analyze the images, the results from each computer system may be considered together in making this determination. If an object does not meet the desired production standards, the object may be rejected. For example, in rejecting the object, the object may be removed from the manufacturing apparatus 204 , as indicated in FIG. 3 by the rejected object 206 , and/or the system may store information indicating that the object 206 failed the inspection. Also, images of the rejected object may be stored if desired.

[0085] The computer system 82 may be a computer system of any type. In one embodiment, multiple computer systems 82 may be employed, e.g., to distribute the image processing load across multiple computer systems. In one embodiment, the computer system(s) 82 may comprise a controller or card (a “computer on a card”) housed in a PXI, VXI or other type of chassis. The chassis may further include one or more image acquisition boards which couple to one or more cameras 212 .

[0086] The computer system(s) 82 may include a memory medium on which software operable to receive and analyze the images of the objects 200 is stored. In one embodiment, this software may include a machine vision application and/or a machine vision prototyping application used to create and execute the machine vision application. In another embodiment, this software may include a program that was programmatically generated based on a machine vision application, such as a graphical program or a text-based program. It is noted that in various embodiments, the analysis of the objects 200 may be performed in any of various manners, either in software, programmable logic, or hardware, or a combination thereof. For example, at least a portion of a machine vision application may be deployed on a hardware device.

[0087] Although FIG. 3 illustrates a machine vision inspection system and machine vision-related tests are discussed above, in other embodiments the computer system 82 may be operable to perform other types of tests of an object 200 , including audio tests (e.g., to test function or quality of an audio component of the object), shock tests or noise-vibration-harshness (NVH) tests, mechanical tests (e.g., to test function or quality of a mechanical component of the object), battery tests (e.g., to test function or quality of a battery of the object), and/or current draw or other electrical tests (e.g., to test current draw or other electrical characteristics or RF parameters of the object), among other types of tests. For example, a prototyping environment may provide access to operations related to performing these types of tests, and the user may include the desired operations in an application. As described below, the operations in the application may be displayed to the user using a graphical programming visual representation. As noted above, in other embodiments, the application may be intended for any of various other applications in addition to inspecting objects 200 . Thus, FIG. 3 is exemplary only.

[0088] FIG. 4 —Computer System Block Diagram

[0089] FIG. 4 is a block diagram representing one embodiment of the computer system 82 illustrated in FIGS. 1, 2A , 2 B, and/or 3 . It is noted that any type of computer system configuration or architecture can be used as desired, and FIG. 4 illustrates a representative PC embodiment. It is also noted that the computer system may be a general purpose computer system, a computer implemented on a VXI card installed in a VXI chassis, a computer implemented on a PXI card installed in a PXI chassis, or other types of embodiments. Elements of a computer not necessary to understand the present description have been omitted for simplicity.

[0090] The computer may include at least one central processing unit or CPU 160 which is coupled to a processor or host bus 162 . The CPU 160 may be any of various types, including an ×86 processor, e.g., a Pentium class, a PowerPC processor, a CPU from the SPARC family of RISC processors, as well as others. Main memory 166 is coupled to the host bus 162 by means of memory controller 164 . In one embodiment, the main memory 166 may store a prototyping environment application for creating, configuring, and/or performing an application. The main memory 166 may also store an application created using the prototyping environment application. In another embodiment, the main memory 166 may store a program that was automatically, i.e., programmatically generated by the prototyping environment based on an application, where the program is operable to perform the application. In one embodiment, the main memory 166 may store an application development environment associated with a programmatically generated program. The main memory may also store operating system software, as well as other software for operation of the computer system.

[0091] The host bus 162 may be coupled to an expansion or input/output bus 170 by means of a bus controller 168 or bus bridge logic. The expansion bus 170 may be the PCI (Peripheral Component Interconnect) expansion bus, although other bus types can be used. The expansion bus 170 includes slots for various devices such as a data acquisition board 114 and a GPIB interface card 122 which provides a GPIB bus interface to a GPIB instrument. The computer 82 further comprises a video display subsystem 180 and hard drive 182 coupled to the expansion bus 170 .

[0092] A reconfigurable instrument 190 may also be connected to the computer. In various embodiments, the configurable logic may be included on an instrument or device connected to the computer through means other than an expansion slot, e.g., the instrument or device may be connected via an IEEE 1394 bus, USB, or other type of port. Also, the configurable logic may be comprised on a device such as the data acquisition board 114 . In one embodiment, at least a portion of an application may execute on the reconfigurable instrument 190 .

[0093] FIG. 5 —Displaying Application Operations Using a Graphical Programming Representation

[0094] FIG. 5 is a flowchart diagram illustrating one embodiment of a method for displaying operations in an application by using a graphical programming visual representation. It is noted that FIG. 5 illustrates a representative embodiment, and alternative embodiments are contemplated. Also, various elements may be combined, omitted, or performed in different orders.

[0095] In 301 , a graphical user interface for a prototyping environment may be displayed. The graphical user interface may provide graphical access to a set of operations available for inclusion in an application. In various embodiments, any of various operations may be provided. For example, as discussed above, the prototyping environment may be intended for creating applications or solutions for problems in any of various domains. Exemplary user interfaces for exemplary machine vision prototyping environments are described below. The GUI may include various menus, dialogs, controls or other GUI elements that the user may use to select operations and/or apply operations to a displayed object or image.

[0096] In 303 , user input requesting inclusion of a plurality of operations in the application may be received to the graphical user interface. The plurality of operations may implement a first function. In one embodiment, the user is not required to specify or write any program source code to implement the application or specify the operations in the application. Instead, the application may be created graphically by interacting with the graphical user interface of the prototyping environment to include operations in the application.

[0097] As described above, in various embodiments the plurality of operations in the application may implement any of various functions or solutions. As one example, the plurality of operations may include one or more machine vision operations that implement a machine vision function, such as for acquiring and/or analyzing images. As another example, the plurality of operations may include one or more motion control operations that implement a motion control function, such as for controlling motion of a device and/or controlling a device to move an object. As another example, the plurality of operations may include one or more data acquisition (DAQ) operations that implement a DAQ function, such as for acquiring measurement data from a device.

[0098] The operations in the application may be related to one another or structured in any of various ways. For example, in one embodiment, the application may comprise a sequence of operations. In one embodiment, each operation in the sequence may be performed sequentially. In another embodiment, the user may specify conditional branches that may result in some operations being skipped or performed in different orders, e.g., depending on results of performing previous operations. The user may also specify other types of constructs, such as iteration, looping, jumps, etc.

[0099] In various embodiments, the user may perform any of various actions or may interact with the graphical user interface of the prototyping environment in any of various ways to cause operations to be included in the application. For example, in one embodiment, a plurality of buttons may be displayed, each button corresponding to a particular operation. The user may press the appropriate button to add the desired operation. In another embodiment, a library of icons may be displayed on a panel or palette, where each icon corresponds to a particular operation available for inclusion in the application. The user may select the appropriate icon to add the desired operation. In other embodiments, the user may utilize other menu, keyboard, and/or voice commands to add an operation to the application.

[0100] In one embodiment, operations may be included in the application in response to the user performing the operations on an object of interest. For example, the user may select various operations to apply to the object, and this may cause the operations to be included in or stored in the application. Thus, the application may specify the operations applied to the object. The object or an image of the object may also be updated to reflect results of applying the operations to the object. In one embodiment, the user may directly perform operations on an object, such as by using a virtual reality glove or similar device to manipulate a physical object, and the operations performed by the user may be recorded as operations in the application. Head tracking and eye tracking devices, and other types of virtual reality devices, may also be used.

[0101] In one embodiment, operations may be included in the application in response to the user performing operations on an object displayed on the screen, e.g., an image of an object. The object image may represent a physical system or device. The user may graphically manipulate or otherwise selection operations to affect the object image. For example, the user may desire to create a robotic application to assemble a portion of an automobile, and the user may graphically select and place automobile parts on the screen to create the application.

[0102] In an embodiment targeted toward machine vision, the system may display an image, and the user may perform machine vision operations on the displayed image, which may cause visual changes to the displayed image according to the operation performed. As the user performs machine vision operations on the displayed image, the machine vision operations may be included in or stored in the application.

[0103] In one embodiment each operation may be interactively executed as the operation is included in the application, e.g., may be executed automatically in response to including the operation in the application or may be executed in response to a user request. For example, this may help the user to verify that each individual operation in the application performs as intended. Also, in one embodiment, each operation may be included in the application in response to executing the operation, e.g., in response to applying the operation to an object.

[0104] In one embodiment, the graphical user interface of the prototyping environment may be updated to illustrate the effect of including and/or executing each new operation. For example, as noted above, the graphical user interface of a machine vision prototyping environment may display an image, and the image may be updated to illustrate the effect of applying a new machine vision operation included in the application to the image. As another example, the graphical user interface of a motion control prototyping environment may display one or more views of motion control performed by the application, such as a two-dimensional and/or three-dimensional view of the cumulative movement specified by the application, as well as other types of views, such as a graph indicating a velocity profile. These views may be updated to illustrate the effect of adding a new motion control operation to the motion control application. In this example, the motion control operations may not actually be executed as they are added to the motion control application, e.g., may not actually cause a motion control device to move, but rather their effects may be simulated on the graphical user interface.

[0105] In one embodiment, each operation included in an application may have various associated properties, attributes, or parameters affecting the operation. For example, an arc move motion control operation may have parameters or properties such as a radius, a start angle, and a travel angle. These parameters may initially have default values. The user may configure these parameters or properties (typically using a GUI) to customize each operation.

[0106] In the preferred embodiment, the user may configure the parameters of the operations graphically, without having to write any program code. For example, a graphical panel for configuring the operation may be displayed. This panel may be automatically displayed in response to selecting the operation or adding the operation to the application, or the panel may be displayed in response to user input requesting to configure the operation. User input for configuring the operation may be received to the graphical panel. For example, the panel may include various user interface elements for changing parameter or property values of the operation, such as numeric GUI controls, check boxes, etc.

[0107] In one embodiment, the graphical user interface of the prototyping environment may be updated to illus