DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

[0079] The invention is directed to a method and system for a Virtual Assembly Design Environment (VAE) that enables users to evaluate, analyze, and plan the assembly/disassembly of parts for mechanical systems. The invention employs an immersive virtual reality (VR) environment that is tightly coupled to a computer aided design (CAD) system. The Invention includes: (1) data integration (two-way) with a parametric CAD system; (2) realistic 3D interaction of an avatar such as a virtual hand with virtual parts in the VR environment; (3) creation of valued design information in the VR environment; (4) reverse data transfer of the created design information from the VR environment to the CAD system; (5) significant interactivity in the VR environment between the virtual hand and virtual parts; (6) collision detection between virtual parts; and (7) physical world-based modeling of the interactivity between the virtual hand and the virtual parts.

[0080] The mechanical system of parts for an assembly is designed using a parametric 3D CAD system such as Pro/Engineer™. In one embodiment, a user selects an option in the CAD system that calls the VAE system to automatically export the data necessary to recreate 3D virtual parts in a virtual environment. Next, the user engages one or more VR peripheral devices to enter the virtual environment where the user is presented with a virtual assembly scene. The invention is capable of supporting a variety of virtual reality peripheral devices, e.g., a CYBERGLOVE by Virtual Technologies Inc. and a head mounted display. The various 3D virtual parts are initially located where they would be in a real assembly plant as defined by the user, which can then perform the assembly of the parts in the virtual environment.

[0081] In the virtual environment, the user can make decisions, design changes and perform a host of other engineering tasks. During this process, the virtual environment maintains a link with the CAD system and uses the capabilities of the CAD system wherever required as described in greater detail below. However, the operation of the virtual environment by the invention is not limited by the level of the interactivity with the CAD system. At the end of the VAE session, the user will have generated valued design information which is then automatically made available to the user in the CAD system.

[0082] FIG. 1 shows an overview 100 of the interactions between a VAE system 102 and a parametric CAD system 104 . In real time, the CAD system 104 provides part assembly geometry, tolerances and part attributes, e.g., center of mass and friction, to the VAE system 102 , which outputs trajectory and sequence information collected in the virtual environment to another facility 106 for analysis. The outputted trajectory and sequence information is employed to analyze the design for assembling the parts to determine if changes in the design assembly should be made. The other facility 106 forwards the trajectory and sequence information to the CAD system 104 along with any suggested design changes for assembly. Optionally, the VAE system 102 can provide an interface to one or more other systems, including a VR based training system 108 , a computer aided process planning system 110 , a robot path planning system 112 and a specialized assembly equipment design 114 .

[0083] In FIG. 2 , an overview 116 is shown of the architecture for organizing eight separate object oriented software modules in the VAE system 102 . An Interaction Manager module 118 is employed to harmonize all of the modules and features of the VAE system 102 ; and a Model Manager module 120 is used to obtain assembly model and environment model information from the CAD system 104 . Also, an Output Manager module 122 is employed to create and update a graphics display and manage a scene graph; and a Collision Manager module 124 is used to provide real-time collision detection. Additionally, an Input Manager module 126 is employed to obtain user input including tracking data, glove data and keyboard entries; and a Swept Manager module 128 is used to create and control the editing of swept volumes and part trajectories. A Design Manager module 130 is employed to enable a user to perform parametric design modifications in the virtual environment and integrate these design modifications with the CAD system 104 ; and a Dynamic Handler module 132 is used to simulate dynamic behavior for a part in the virtual environment.

[0084] FIG. 3 illustrates a virtual scene 134 produced by the invention showing a constrained sliding motion for insertion of a virtual part 140 into a base assembly 138 along three determined axes 136 . A user controlling a virtual hand 142 can select collision options in a virtual context menu 150 and swept volume generation options in another virtual context menu 152 . A virtual workbench 154 is disposed in the virtual scene 134 .

[0085] FIG. 4 illustrates a virtual scene 144 with a pendulum shaped virtual part 146 rotating about a fixed shaft virtual part 148 and translating along the axis of the virtual shaft. Disposed in the virtual scene 144 is the virtual context menu for selecting collision options in a virtual context menu 150 and the other virtual context menu 152 for selecting swept volume generation options. Also, the virtual workbench 154 is disposed in the virtual scene 144 .

[0086] FIG. 5 illustrates a displayed screen 156 in a CAD system for a swept volume 158 that has a parametric representation and which was sent back to the CAD system for generation as a feature of the assembly.

[0087] FIG. 6 illustrates a virtual scene 160 of the virtual hand 142 selecting parameter options in a virtual context menu 162 for a shaft virtual part 166 that is positioned in a base virtual part 164 and which is disposed on the virtual workbench 154 .

[0088] FIG. 7 illustrates a virtual scene 168 of a virtual part being moved along a trajectory path in the virtual environment when the swept volume and collision detection features are turned on. A beginning swept volume 170 A and an end swept volume 170 B for a virtual part are positioned at either end of the trajectory of the virtual part. In the trajectory path, exact instances of swept volume collisions 172 with the virtual workbench 154 are highlighted.

[0089] The invention can perform and/or assist a user in assembly design evaluation, analysis, and assembly sequence planning at all product realization stages: assembly plan verification (pre-product evaluation), maintenance verification, and alternative plan searching (post-production evaluation).

[0090] In the assembly plan verification stage, the Invention enables assembly to be performed in a pre-defined sequence. The user can assemble virtual parts one by one in the virtual environment using constrained motion, swept volume and collision detection. If there is any interference detected during the assembly process, the user can try to find a way to get around it in the virtual environment.

[0091] The maintenance verification stage enables the user to check disassembly of a particular part. If a part needs to be taken out of a larger assembly for maintenance, e.g. change a spark plug or an oil filter, the invention can be employed to ensure a clear trajectory path for disassembly. In one embodiment, the user removes a virtual part from its final position in a larger assembly of virtual parts and the invention checks for collision detection during the disassembly process. In another embodiment, a swept volume of the trajectory path is created during the disassembly process for a particular virtual part. This swept volume is checked for interference with other virtual parts in the larger assembly of virtual parts. By observing the disposition of the swept volume, the invention can determine how much space is available to perform a disassembly operation.

[0092] Sometimes it is necessary to find alternative plans or sequences for operations that are already being carried out in a workshop with real parts. It is also common to post-evaluate an assembly operation. Stopping the assembly line to perform the testing is not always economically feasible and often very few alternatives can be tried out in the limited time available. The invention provides a viable alternative where assembly experts can try various alternatives, choose the best one, suggest design changes, suggest fixturing changes and perform ergonomic evaluations of the assembly/disassembly process. The results of these evaluations in the VAE system can be automatically transferred back to the original CAD system so that a user can quickly perform design changes without any other data translation.

[0093] From the invention test data, several observations have been made: (1) Pure assembly time in a virtual environment is lower than actual assembly time (about 10-15%), which can be attributed to the lack of fastening operations in the virtual environment; (2) Pure assembly time for each virtual part in the assembly increases with the physical size of the part because the difficulty to handle a part appears to increase with its size; (3) Average gripping time for each part in the assembly remains almost the same for different sizes of parts and mainly depends on a user's practice and experience in the virtual environment, whereas gripping difficulty depends on the shape of the part (a thin, long shaft is more difficult to grab than a cubic block); and (4) When considering the relationship of pure assembly time and total assembly time, the correlation coefficient is low for large assembly (0.9 for small assembly, 0.98 for half size large assembly and 0.7 for large assembly), which indicates that human considerations start influencing the assembly time for larger models, e.g., moving some distance to grab the part, finding a better viewing position to look at the part and aligning the parts.

[0094] Besides quantitative information, the invention enables qualitative information to be obtained. For example, a full-size assembly in a virtual environment provides intuitive and valuable information that is impossible to obtain from conventional assembly modeling by a CAD system. The invention test data also illustrated other potential capabilities such as training, work space study and operation time study.

[0095] With the assistance of all the capabilities of the invention, a user can perform assembly design evaluation, maintenance verification, alternative assembly plan searching, and part design modification as described above. Also, since the invention involves the experience and actions of the user, the assembly plans generated by the invention automatically include input from the knowledge of experienced users.

[0096] Additionally, since the invention is typically presented in a full immersion mode using a head mounted display, it can be tiring to put the user in the environment for a long period of time. However, combining parts into sub-assemblies has been found to reduce the amount of time a user spends in the virtual environment.

[0097] Virtual assembly evaluation and planning is particularly suited for complex assembly operations that involve a person. Also, automatic assembly planning systems are well suited for assembly models with a large number of parts that require relatively simple assembly operations (involve translation and one axis rotation) which are often performed by robots. In some cases, a combination of virtual and automatic assembly evaluation can be the best solution. For example, the automatic assembly planning system could be used to find some feasible assembly process plans. Next, the user could then enter the virtual assembly environment (VAE) for evaluation, verification, consideration of practical problems related to the realization of the assembly design and optimization.

[0098] In the sections below, several aspects of the invention are explained in greater detail, including: (1) enhanced constrained motion simulation; (2) physically based modeling of a virtual environment; (3) generation of swept volumes and interactive swept volume trajectory editing; (4) parametric design modification; and (5) finger twirling.

ENHANCED CONSTRAINED MOTION SIMULATION

[0099] From the CAD system, constraints are obtained and transferred to the VAE system. For axis constraints, two points in space defining the ends of the graphical line representing the axis are obtained. For plane constraints, three unit vectors and the origin defining the plane are obtained. One of the unit vectors is the normal vector for that plane, starting at the origin of the plane. In both cases, the type of constraint (align or mate) and the offset, if any, between the two axis or planes under consideration are also obtained.

[0100] Additionally, during the assembly process, the geometry representation of the constraints of the base part and the constraints of the part being assembled are transformed into the same coordinate system to check for closeness. If a constraint meets certain criteria, it is applied and the part's motion is limited to the constrained space. For example, if the part is constrained on an axis of the base part, the part can only slide along the axis and rotate about the axis. Alternatively, if the part is constrained on a plane of the base part, the part's motion is limited to the plane.

[0101] Although the above method can correctly map the physical constraints in the virtual environment, it only works for one constraint at a time. In some cases, using the above methods will result in the loss of previously applied constraints. For example, in FIG. 8 , where two parallel axial constraints are to be applied and one has already been applied (axis A 1 ). Simply snapping the constraint of axis A 2 by translating the part will result in a loss of the previous constraint. A 1 b and A 2 b are two axes on the base part and A 1 p and A 2 p are two axes on the part. A 1 p and A 1 b have been snapped together. A simple snapping of A 2 b onto A 2 p will move A 1 b away from A 1 p, which destroys the previously applied constraint.

[0102] FIG. 9 illustrates a scene graph 176 used to represent the graphical data structure of the VAE system. The scene graph 176 provides an intuitive way to represent the hierarchical relationships between the objects in the virtual world usually the relationships between different dynamic coordinate systems (DCS). More importantly, it provides a way to edit and modify the relationships between the objects in the virtual world. For example, a part DCS 180 represents the coordinate system attached to a part, etc. A base part DCS 178 moves with a FOB bird in the user's left hand so base part DCS 178 is directly under a global DCS 186 . The grabbed part can be manipulated in the user's right hand using a CYBERGLOVE. The part's location and orientation is calculated through its relationship with the palm and a FOB bird attached on the user's right-hand wrist, so that a part DCS 180 is attached to a palm DCS 182 , then to a hand DCS 184 before it goes to the global DCS 186 .

[0103] Referring to the scene graph in FIG. 9 , the following equation is used to transform the geometry from the part DCS 180 to the global DCS 186 . The [partLocationXform] is the transformation from the part DCS 180 to the global DCS 186 , the [part_matrix] is the transformation matrix from part DCS 180 to palm DCS 182 , the [palm_matrix] is the transformation from palm DCS 182 to hand DCS 184 , and the [hand_matrix] is the transformation from the hand DCS 184 to the global DCS 186 . The [baseLocationXform] transforms geometry from base part to the global coordinate system.

[ partLocationXform]=[part _— matrix]×[palm _— matrix]×[hand matrix] (1a)

( geometry of part in global )=( geometry of part )×[ partLocationXform] (1b)

( geometry of base in global )=( geometry of base )×[ baseLocationXform] (1c)

[0104] Then, in the global coordinate system, the geometry pairs of axis or plane constraints are compared to check for align or mate status. If certain tolerance values are satisfied, they are said to be constrained and equation (2) is used to apply the axis constraint.

[ part _— matrix]=[sv _— NegXform]×[axisRotate]×[partTranslationXform] (2)

[0105] Where [sv_NegXform] takes the part's axis to the origin of the part. Next [axisRotate] makes sure that the two axis under consideration are parallel. Finally [partTranslationXform] snaps the part's axis on to the base part's axis so that they are aligned and constrained. After the axial constraint is applied, the allowable motions are along that axis and about the axis.

[ part _— matrix]=[p _— originNegXform]×[normalRotate]×[p _— originXform]×[distance _— bp _— normalXform] (3)

[0106] Equation (3) is used to apply the plane constraint where [p_originNegXform] moves the origin of the plane on the part to the origin of the part coordinate system, [normalRotate] makes sure the two planes are parallel. Then [p_originXform] takes the origin of the part plane back to its original position. Finally [distance_bp_normalXform] snaps and constrains the two planes together by moving the part plane in the required direction.

[0107] In axis and plane constraints, there is axis align (inserting), plane or surface align, and plane or surface mate, as shown in FIG. 10 . In section (a) of FIG. 10 , axis A 1 (with end points A 1 a and A 1 b ) is going to be aligned to A 2 (with end points A 2 a and A 2 b ). Also, in section (b) of FIG. 10 , P 1 (with normal n 1 ) is aligned with P 2 (with normal n 2 ) and is mated with P 3 (with normal n 3 ).

[0108] When checking the align or mate status, differentiating them can be complex especially for plane constraints. However, the plane normals on the part and the base part are in the same direction if they are aligned. An inaccurate way to differentiate is to check that the dot product of n 1 and n 2 is near +1 if they are required to be aligned and −1 if they are required to be mated. A more accurate method for getting the constraint information from the CAD system is shown in FIG. 11 , which illustrates axis (with two end points, Aa, Ab) and plane (with an point Ori, three vectors e 1 , e 2 , and e 3 ) constraints.

[0109] From the CAD system, for a plane constraint, a point, Ori, is obtained as the origin of the plane, as well as three unit vectors that are mutually perpendicular to each other. The information is in the part coordinate system. The corresponding information on the base part is obtained by the final transformation (the transformation matrix when the part is finally assembled onto the base part) between them. This is shown in FIG. 12 where the plane on part (Pp) is mated with plane on the base part (Pb). In FIG. 12 , Pp is a plane on the part (with normal np), Pb is a plane on the base part (with normal nb). And nb is calculated by Equation-4.

nb=np×[TransformMat] (4)

[0110] In the equation (4), [TransformMat] is the transformation matrix between the part and the base part when the part is assembled to it's final location. The normal on the base part is defined in base part DCS, while the normal on the part is defined in part DCS. However, when the normal vectors need to be checked, they need to be checked in the same coordinate system. When checking the constraints in the part DCS, transform nb is transformed from base part DCS to part DCS using equation-5 (nb ^p is the representation of nb in part DCS). In the equation, [baseInPartXform] is the transformation from base part DCS to part DCS. If the part is in its final location, [baseInPartXform] is equal to [TransformMat] ⁻¹ . So there is nbp=np.

nb in part DCS=nb×[baseInPartXform]=nb ^p (5)

[0111] The normal vectors look opposite to each other, however, that is because they are viewed in different coordinate systems. For example, if a point is transformed to get a point in another coordinate system, when it is transformed back, it is still the same point, therefore, if viewed in the same coordinate system, e.g. in the part coordinate system, the two normal vectors are exactly the same.

[0112] Therefore, when align status of two axes or two planes is checked, the dot product of the two axis vectors or two normal vectors are checked to be near +1. When the plane mate status is checked, the dot product of the two normals is also checked to be near +1. No −1 should be involved at all.

[0113] Another useful observation can be made from the above discussion: since the constraints on the base part are defined by the constraints on the part, the constraints on the part can be defined in an arbitrary way without affecting the final location of the part when it is assembled on to the base part. Therefore, some complicated or abstract types of constraints can be replaced with simple types of constraints. For example, a coordinate system constraint can be replaced with three axis constraints. This step simplifies the simulation task in some cases.

[0114] Axis and plane (or surface) constraints are the most frequently used constraints in assembly operations to fix a part on a base part or a subassembly. In CAD systems, the user is allowed to pick any number of axis or plane constraints as long as they are not in conflict with each other. This, however, gives rise to some redundant information in the assembly operation. In CAD systems, the final position of the part is important, the order is not. However, in real and virtual assembly, the ordering of parts does matter. By analyzing all of the possible combinations of axis and plane constraints, invention can determine which set of axis and plane constraints are enough and which are redundant.

[0115] An exemplary result is listed in a table in FIG. 13 . “A” is denoted as axis constraint and “P” as plane constraint. Also, numbers are employed to represent the order of the constraint. For example, “A 1 ” means the first one applied is an axis, “P 2 ” means the second one is a plane constraint, etc. In FIG. 13 , all of the possible combinations that can completely constrain the motion of a part on the base part are listed. In the table, a symbol of “⊥” represents perpendicular, “//” represents parallel, “n⊥” represents not perpendicular, and “n//” means not parallel. The first column shows the various possible ways in which up to 3 constraints (axis or plane) can be used in a specific sequence to fully constrain the motion of a part. The second column shows the conditions under which a specific sequence of constraints can fully constrain a part.

[0116] Careful observation of FIG. 13 leads to the three following conclusions. First, three non-redundant axis or planar constraints are sufficient to fully constrain a part. The user should not choose more than three axis or plane constraints in the assembly design. This conclusion is very useful and means that when the third constraint is applied, the part will be completely constrained. So, in constrained motion mapping, the invention maintains the first applied constraint when the second one is applied. The task of maintaining the previous constraints is greatly simplified: the invention applies the first one using the snapping method, then uses the snapping method or the methods described in the next section to apply the second one, and when the third one is to be applied, it has reached the final location and the part is placed.

[0117] Second, if two parallel axes are used, any third axis parallel to one of them is redundant. Further more, any plane parallel to them is also redundant.

[0118] Third, if a plane is used, any other plane parallel to it is redundant. If a plane and an axis parallel to it are applied, any axis parallel to the plane or the previous axis is redundant, and any plane parallel to the plane is redundant. If two planes are used, any axis parallel to the intersection line of the two planes is redundant.

[0119] From the three conclusions discussed above, it is understood that at most three constraints are needed to fix the part and only the first one needs to be maintained. In some situations, just using equations (2) and (3) will do the work. For example, FIG. 14 illustrates when P 1 is applied (a plane on part Pp is snapped with a plane on base part, Pb), the snapping of the part onto A 2 (snap an axis on part Ap onto an axis on the base part Ab) will not violate the previously applied planar constraint. From the analysis of all the situations in FIG. 13 , there are at least three cases that will need special treatment.

[0120] Case 1 : An axis constraint has been applied (Ap 1 and Ab 1 ), another axis constraint (Ap 2 and Ab 2 ), which is parallel to the first one, is going to be applied. FIG. 15 illustrates Case 1 with an axis on part (Ap 1 ) and an axis on base (Ab 1 ) that have been snapped together. Another axis on part (Ap 2 ) needs to be snapped to its corresponding axis (Ab 2 ) and a simple snapping method will move Ab out from Ap.

[0121] In this case, the snapping method is used to snap Ap 1 to Ab 1 by equation (2). Equation (1) is still used to calculate and check the align status for Ap 2 and Ab 2 . If the condition is satisfied, the invention calculates the angle θ.

[0122] As shown in FIG. 16 , e 1 b 1 and e 2 b 1 are the end point of axis 1 on the base part; e 1 b 2 and e 2 b 2 are the end points of axis 2 on the base part; and e 1 p 2 and e 2 p 2 are the end points of axis 2 on the part. Let vector r 1 =e 2 b 1 −e 1 b 1 , vector r 2 =e 1 p 2 −e 1 b 1 and vector r 3 =e 1 b 2 −e 1 b 1 . First r 1 r 2 , r 3 are normalized; then angle θ is calculated as

θ=cos ⁻¹ ((( r 1 ×( r 2 × r 1 ))•( r 1 ×( r 3 × r 1 ))) (6)

[0123] where “x” denotes cross product and “•” denotes dot product.

[0124] The transformation matrix is calculated for rotating the part about axis Ab by angle θ, [rotate_matrix_Ab_axis]. The final transformation of the part in the PalmDCS will be

[ part _— matrix]=[part _— matrix _— A 1 ]×[ rotate _— matrix _— Ab _— axis] (7)

[0125] where [part_matrix_A 1 ] is the part matrix calculated by using equation (2) when the first axis is applied. Also notice that Ab 1 does not necessarily pass through the origin of the part DCS.

[0126] Case 2 : An axis constraint has been applied, the second one is a plane, which is parallel to the applied axis. As shown in FIG. 17 , an axis on part (Ap) and an axis on base part (Ab) have been snapped together. A plane on part (Pp) needs to be snapped to a plane on base part (Pb) and a simple snapping method will move Ap away from Ab.

[0127] In the second case, the invention also uses the snapping method to snap Ap to Ab by equation (2). Equation (1) is still used to check the align status Pp and Pb. If the condition is satisfied, a transform matrix is formed by rotating about Ab by an angle. The angle is calculated as shown in FIG. 18 . In this figure, e 1 b 1 and e 2 b 1 are the end points of the first applied axis, OriPp and OriPb are the origins of the planes on the part and the base part, Let vector r 1 =e 2 b 1 −e 1 b 1 , vector r 2 =OriPp−e 1 b 1 and vector r 3 =OriPb−e 1 b 1 , First r 1 r 2 , r 3 are normalized. The angle of rotation is calculated as:

θ=cos ⁻¹ ((( r 1 ×( r 2 × r 1 ))•( r 1 ×( r 3 × r 1 ))) (8)

[0128] The final transform of the part in the PalmDCS will be

[ part _— matrix]=[part _— matrix _— A 1 ]×[ rotate _— matrix _— Ab _— plane] (9)

[0129] where [part_matrix_A 1 ] is the part_matrix calculated by using equation (2) when the first axis is applied. Similarly notice that Ab does not necessarily pass through the origin of the part DCS.

[0130] Case 3 : A plane constraint has been applied, the next one is a plane which is not perpendicular to the first one. If the second one is perpendicular to the first one, a simple snapping method can be used.

[0131] FIG. 19 illustrates Case 3 where a plane on part (Pp 1 ) and a plane on base (Pb 1 ) have been snapped together. Another plane on part (Pp 2 ) needs to be snapped onto another plane on base (Pb 2 ). A simple snapping will move Pp 1 out of Pb 1 .

[0132] In Case 3 , the snapping method is used to snap Pp 1 to Pb 1 by equation (3). Equation (1) is used to check the align status. If the condition is satisfied, [p_orginNegXform] and [p_originXform] are calculated so that Pp 2 can be oriented parallel to Pb 2 . Now the task is to figure out a transformation vector that is parallel to Pb 1 and perpendicular to the intersection line of Pb 1 and Pb 2 .

[0133] The vector translation of Case 3 is shown in FIG. 20 , where the invention calculates the intersection vector of the two planes by the cross product of the normals of the two planes on the base part, i.e.

tb=nb 2 × nb 1 (10)

[0134] next, the invention calculates the translation direction vector t 2

t 2 = nb 1 × tb (11)

[0135] t 2 is normalized to obtain the translation vector

tr =(( Ob 2 − Op 2 )• t 2 ) t 2 (12)

[0136] The vector tr is the translation vector used to form a translation matrix [translation_along_plane], putting the above calculations together

[ part _— matrix]=[part _— matrix _— P 1 ]×[ p _— originNegXform]×[normal _— rotate]×[p _— originXform]×[translation _— along _— plane] (13)

[0137] where [part_matrix_P 1 ] is the part_matrix calculated by using equation (3) when the first plane constraint is applied.

[0138] The three special cases described above are situations where special methods are needed for accuracy. In other situations, the simple snapping method can be used.

[0139] With the help of above conclusions and methods, the invention can simulate the constraints during the assembly process. The redundant constraints are processed during the constraint checking process. A work flow chart 188 is shown in FIG. 21 for processing and application of multiple constraints. In this figure, special cases and special methods refer to the cases and methods discussed above.

[0140] In one embodiment, global position and orientation tracking is done by the Ascension Flock of Birds™ system with an Extended Range Transmitter (ERT). This transmitter employs a pulsed, DC magnetic field and is capable of determining 6 DOF information from each of its receivers. Three receivers are used in this system, one to track the head so that the user can ‘look around’, another to track the right hand and the last one is held in the left hand facilitating assembly operations.

[0141] In one embodiment, the CYBERGLOVE is used to monitor the finger and wrist movements of a user. This 22 sensor glove augments the graphical representation of the right hand in the VAE system. It measures the motions of the wrist joint, the bending of the three joints on all four fingers and the thumb, the abduction between all four fingers, the arch of the palm, and the abduction of the thumb from the palm. The digitized output values from the glove sensors are converted to appropriate joint angles for a specific user's hand using a calibration routine. These joint angles are compared against a glove tolerance to facilitate releasing the part when the user stretches his/her hand to drop the part.

[0142] In one embodiment, the graphical basis for the invention is created with a Silicon Graphics IRIS Performer™ Library. IRIS Performer™ is a software toolkit for the development of real-time 3D graphics, visualization, and simulation applications. Performer™ sits “on top” of Silicon Graphics OpenGL™ libraries. It also has better optimization of its own functions and in turn allowed better performance when using complex models.

[0143] Pro/ENGINEER™ can be used for the creation of the CAD models for use in the invention. Also, Pro/DEVELOP™ is a developer's toolkit for Pro/ENGINEER™, which is designed to be used as a means to access the Pro/ENGINEER™ database. The Pro/DEVELOP™ module automates and simplifies data exchange between the CAD system and the VAE system.

CONSTRAINT MANAGEMENT

[0144] Object-oriented methods are used to abstract and represent the constraints in the invention. Humans learn about objects by studying their attributes and observing their behaviors. Object-oriented programming models real-world objects with software counterparts. Using object-oriented technologies, the invention can take advantage of object relationships where objects of a certain class have the same characteristics i.e. inheritance. Considering the constraints used in the virtual assembly processes, even though the representations of the constraints are different, they all share the same behaviors: a checking process and an application process. This becomes a typical inheritance situation. According to the analysis and abstraction of real world constraints, The invention employs a Constraint class 190 and by inheritance includes other specific constraint classes, e.g. an AxisConstraint 191 , a CSConstraint 193 and a PlaneConstraint 192 which FIG. 22 shows in an overview 194 . Thus, a part maintains a list of “constraints” without needing to know the detailed nature of the constraint. In the Constraint class two virtual functions are defined, checkConstraint and applyConstraint. In the children classes, there are defined the geometrical representations according to the type of the constraint and override checkConstraint and applyConstraint according to algorithms presented in the previous chapter.

[0145] In FIG. 21 , the assembly process is shown to be a constraint application process. The degrees of freedom of a part relative to the base part are gradually reduced as the constraints are applied. So the constraints of a part have two different states: already applied or going to be applied. However, some constraints are redundant and will never be used at all. To catch this physical meaning, for a part 195 , the invention employs three linked lists of constraints named AppliedList 196 , UnappliedList 197 and RedundantList 198 as shown in FIG. 23 . If the part 195 is not constrained and can move in the global space freely, all the constraints will be kept in the UnappliedList 197 . If a constraint is applied, it will be moved from the UnappliedList 197 to the AppliedList 196 . During the assembly process, if any one of the constraints is found to be redundant, it will be moved from the UnappliedList 197 to the RedundantList 198 . After the part is fully constrained and placed on the base part, the UnappliedList 197 should be empty (if not empty, the remaining elements must be redundant and will be moved to the RedundantList 198 ). So finally in AppliedList 196 , there is the sequence of constraint application and in the RedundantList 198 , there is the redundant information from the design model. The information in these linked lists provides the information on the status of the part: floating, partially constrained, or placed. In addition, the lists provide information on the assembly sequence.

[0146] A ConstraintManager class 230 can manage the constraints for different parts. In the ConstraintManager 230 the invention defines three linked lists of the Constraint objects to hold the constraint status and data of the part that is being manipulated. The lists in the ConstraintManager 230 provide temporary processing and swapping space for constraint checking and application.

[0147] The constraint information exchanging between the ConstraintManager 230 and one part 195 is shown in FIG. 24 . When the user grabs a part, the constraints in the AppliedList 196 , the UnappliedList 197 and the RedundantList 198 of the part 195 are handed over to their corresponding lists in the ConstraintManager 230 for handling. If the part 150 is constrained by one or more constraints, the constraints will be moved from the UnappliedList 197 to the AppliedList 196 . If the constraint is redundant, it will be moved from the UnappliedList 197 to the RedundantList 198 . When the part 195 is released, the ConstraintManager 230 returns the lists back to their corresponding lists in the part 195 . Also, when the part 195 is placed, the ConstraintManager will give a NULL value to the UnappliedList 197 in the part 195 .

[0148] As discussed above, the graphical structure of the system is represented by the scene graph shown in FIG. 9 . The part is attached to the palm DCS 182 , which is attached to the hand DCS 184 , which is attached to the global DCS 186 . The location of the part in the global space is represented by equation (2.1).

[ partLocationXform]=[part _— matrix]×[palm _— matrix]×[hand _— matrix] (2.1)

[0149] In the equation, [partLocationXform] is the transformation from the part DCS 180 to the global DCS 186 , [part_matrix] is the transformation matrix from the part DCS 180 to the palm DCS 182 , [palm_matrix] is the transformation from the palm DCS 182 to the hand DCS 184 , [hand_matrix] is the transformation from the hand DCS 84 to the global DCS 186 . In the meantime, [baseLocationXform] represents the transformation from the base DCS 178 to the global DCS 186 .

[0150] If, at some time, the user releases the part while the part is constrained, the invention wants the part to stay on the base part and move with the base part. The relative location of the part to the base part at the time of release can be calculated by equation (2.2).

[ partInBaseXform]=[part _— matrix]×[palm _— matrix]×[hand _— matrix]×[baseLocationXform] ⁻¹ (2.2)

[0151] At the same time, the initial scene graph is changed by moving the part DCS 180 to attached to the base DCS 178 as shown in FIG. 25 . In this case, the base DCS is under the global DCS 186 which provides the dynamic coordinate system for the virtual environment scene 191 . The palm DCS 182 is attached to the hand DCS 184 which is under the global DCS 186 . Constraint handling is performed according to FIG. 24 . The constraints that have been applied are stored in the AppliedList 196 of the part 195 .

[0152] When the user releases the part in his/her hands, if none of the constraints have been applied, the part DCS 180 will move under the global space DCS 180 where it is released, as shown in FIG. 26 . When the user later comes to re-grab this part, the system needs to know where the part is: in the global space or attached to the base part. The handling method will be different since there is also a computation of the relative location of the part to the hand.

[0153] The problem finding where the part is attached becomes easy by noticing the difference between the two situations: if the part is constrained before it is released, the AppliedList 196 is not empty. If the part is not constrained when it is released, the AppliedList 196 is empty. So whenever a part is grabbed, a check is performed whether the AppliedList 196 is empty or not. If the AppliedList 196 is not empty, then that part is attached on the base part. Equation 2.3 is used to compute the relative location of the part to the palm to find out the gripping matrix. If the AppliedList 196 is empty, then the part is in the global space as in FIG. 26 and equation (2.4) is used to find the gripping matrix.

[ partToPalmXform]=[partInBaseXform]×[baseLocationXfo rm]×[hand _— matrix] ⁻¹ ×[palm _— matrix] ⁻¹ (2.3)

[ partToPalmXform]=[part _— GlobalXform]×[hand _— matrix] ⁻¹ ×[palm _— matrix] ⁻¹ (2.4)

[0154] After re-grabbing, the scene graph goes back to FIG. 9 .

[0155] Mechanical system assembly designs consist of many parts. During the assembly process, the parts can be in various stages in the environment: maybe only one axis constraint is applied, maybe only one plane constraint is applied, or maybe two axis constraints are applied, or maybe the part just lies on the table. So the constraint status of different parts are different. When the user grabs a part, the invention knows the status of the constraint information of that part: which one has been applied; the location of the part relative to the base part, etc. When the part is released from the user's hand, the invention remembers the current constraint status at that time. When several parts are involved, the invention keeps track of the constraint linked lists in every part 195 by data exchange between the ConstraintManager 230 and the parts 195 .

[0156] Additionally, there are bi-directional relationships between the parts 195 and the ConstraintManager 230 . When the part 195 is released during the assembly, the ConstraintManager 230 returns all of the constraint lists to that part so that the ConstraintManager 230 can handle other parts. In this way, the invention ensures that the constraints of different parts are kept separate and the status of constrained motion of any part is maintained.

[0157] In FIG. 27, a schematic overview for the ConstraintManager 230 handling two parts ( 195 A and 195 B) is shown. In this figure, all of the arrows pointing up refer to a “when grabbed” status and all of the arrows pointing down refer to a “when released” status.

[0158] In the assembly process, the user may want to reassemble a part even after it is placed on to the base part already. The user perhaps wants to try out some other sequences, or he/she may want to assemble the part after some other parts have been assembled. The invention also provides the functionality for disassembly of assembled parts.

[0159] When the invention performs disassembly, the constraints in the part need to be rearranged. When the part is placed, the applied constraints are stored in the AppliedList 196 in the order that they are applied, the redundant constraints are in the RedundantList 198 and the UnappliedList 197 is empty. The invention moves/swaps all of the constraints in the AppliedList 196 to the UnappliedList 197 , as shown in FIG. 28 . The constraints in the RedundantList 198 , however, need not be moved to the UnappliedList 197 since these constraints are filtered out during the assembly process and will are not used again.

[0160] When the user tries to grab the part out from the base part, the invention finds out where the part is. As discussed above, the invention can use the AppliedList 196 since the list is not empty after the part is placed. The main difference between a constrained part and a placed part is the transformation matrix that is used. In the former situation, the matrix is calculated when the part is released, i.e. [partInBaseXform]. In the later situation, the matrix is the final location matrix stored in the Part object (from the original data from CAD system). The transformation matrix of the part DCS 180 to the palm DCS 182 is calculated by equation (2.5).

[ partToPalmXform]=[finalLocationMatrix]×[baseLocatio nXform]×[hand _— matrix] ⁻¹ ×[palm _— matrix] ⁻¹ (2.5)

[0161] Another problem in disassembly is that when the user grabs the part, the system will begin checking the constraints. Since all the constraints are close to their counterpart ones in the base part when the part is in the close vicinity of its final location, the part may be constrained right after the user grabs the part. This may not be what the user wants to do. To solve this problem, the invention sets a time lag for checking for constraints if the user wants to do disassembly. The invention begins checking constraints five seconds after the user disassembles the part.

[0162] If multiple parts are involved, especially when a new assembly model is loaded in to the environment, the user may not know how to assemble it. Just letting the user try the possibilities makes the system unfriendly.

[0163] It is desirable to have a guiding mechanism in the system that can provide assembly instructions to assist the assembly. The instructions should be simple, intuitive and easy to follow. First, the user needs to know where a part needs to go onto the base part when he/she picks up the part, then he/she needs to be given instructions of how to assemble the part step by step. Since the user may release the part during the assembly process, the system needs to remember the current constrained status of the part. When the user re-grabs the part, the system needs to provide hints on the next step operation based on the constrained status. Further, if the user wants to do disassembly, the system needs to remember the sequence of the previous operation and pass the information to the user to remind him/her of the previous operation sequence.

[0164] To fulfil the requirements listed above, constraint displaying functionality is provided. The geometry of the constraints are displayed when the user grabs the part: for axis, a line is displayed; for planes, a rectangle near the contact is displayed. When several constraints are involved, different colors are used. This gives the user a very intuitive feel for the assembly process. Further more, the constraints are displayed according to the status of the constraints. If one axis constraint is applied and the user lets the part follow the base part, next time when the user grabs the part again, the applied axis will not be displayed. If a redundant constraint is detected, it will not be displayed anymore. When the part is taken away from the base part, the next time when the user wants to reassemble it, all the constraints come back again except the redundant ones.

[0165] Although the requirements of the guiding mechanism are complicated, the task is not that complex because the invention recalls the information stored in the constraint lists. The method of handling this task is to make use of the constraint lists, the AppliedList 196 , the Unappliedlist 197 and the RedundantList 198 . Whenever the invention employs a displayer 232 to display the constraints, it starts with the UnappliedList 197 . This ensures that only the unapplied constraints are displayed. The number of constraints displayed is reduced as the part is being assembled, which means that the allowed degrees of freedom reduces. Also, since the invention processes the redundant constraints all the time, if the constraint is moved to the RedundantList 198 , the invention will not get it later, so gradually only the valid constraints are displayed even if the user starts all over. An overview of the guiding mechanism discussed above is shown in FIG. 29 .

[0166] A detailed scenario is presented above for managing constraints in the virtual assembly environment. The system can efficiently manipulate multiple parts for assembly evaluations. When several constraints need to be applied, all of the constraints are applied in conjunction with the previous ones. When multiple parts are involved, each part moves observing its own constraint set.

PHYSICAL BASED MODELING

[0167] The scene graph method provides an intuitive way to represent the hierarchical relationships between the objects in the virtual world (usually the relationships between different dynamic coordinate systems). More importantly, it provides a way to edit and modify the relationships between the objects in the virtual world.

[0168] One important feature of the invention is constrained motion simulation. The constraint information is extracted from CAD system and each independent constraint satisfied will reduce the number of allowable movements of the objects relative to each other. The invention can simulate axial and planar constraints during assembly design process in any kinds of order and combination. The invention employs methods that can simulate physical constraints commonly used in assembly design without using computationally expensive collision detection.

[0169] In physically based modeling, the basic equations of motion for rigid bodies used to set up the simulation model are the Newton-Euler's equations, which are as follows:

F=M V′ and N=dL/dt=Iω′+ω×L

[0170] Where F is the external force; M is the total mass of the system; V′ is the linear acceleration of the center of the mass of the system; dL/dt is the time derivative of angular momentum in the space frame( which is equal to external torque N); I is the 3×3 inertia matrix and ω′ is the angular acceleration; ω×L is the cross product of angular velocity vector and angular momentum vector. In order to solve for the acceleration, velocity and displacement of the system, the mass properties are needed, i.e. mass and inertia matrices of the part or the system.

[0171] The invention gets around calculating mass properties of polyhedral objects by getting the information directly from the CAD system when the model is designed. The mass and inertia matrices are defined (unless the object is broken or deformed) once the model is designed. After investigating the available information which can be queried from CAD systems (e.g. ProEngineer™), the developer's toolkit (e.g. ProDevelop™) can be used to extract the information. When the model geometry and constraint information are written out, the mass properties are written into a property file for each part (or subassembly if subassemblies are used) of the model. The file format and content are illustrated in FIG. 30 . Note that in the exemplary property file, the invention also includes information other than just mass properties such as units, surface finish, tolerance values, surface area, density, and volume.

[0172] When the invention loads the model into the virtual environment, it also loads the property of the parts or subassemblies at the same time. The information can be queried from the part whenever it is needed during the simulation.

[0173] Assembly models differ tremendously in terms of size and numbers of parts, from tiny motors to large aircraft. In the assembly operations for the different models, human functionality is different. For some small assemblies, assemblers may use their bare hands with assistance from tools. For large assemblies, they depend on tools, e.g. hoists, to lift some big parts and put the parts in their final locations.

[0174] In the VAE system, this fact is taken into consideration. It is easy to use one hand to grab and lift a several hundred pound truck engine in the virtual environment. But this will result in loss of feeling of realism, or even trust in the system. So the inention distinguishes and categorizes the assembly models according to human being's behaviors and abilities.

[0175] The criterion that the invention uses is the strength survey data of human beings. For workers on the assembly lines, if he/she can lift the part with one hand or both hands without difficulty, he/she will lift the part and carry the parts to the assembly. This comes from the observation of real world operations and from the concerns of productivity of industry. The invention can categorize a part into three categories by it's weight: (1) being able to be lifted by one hand; (2) being able to be lifted by two hands; or (3) need to be lifted by a tool. If the part can be lifted by one hand, when the user tries to grab the part, he/she can grab it and move it normally. If the part needs to be lifted by two hands and the user tries to grab and lift it with only one hand, the invention can inform the user that the part is too heavy for one hand lifting and suggest he/she lift it with two hands or get help from some tools. For parts that are too heavy to be lifted by assembler's bare hands, the invention can notify the user to use a tool. Although this kind of categorization is crude and simple, it can represent the real world situation. One interesting observation is that novice users tend to reach out his/her hands to pick up relatively small parts even before any explanation is provided on how to grab the parts in the environment. If he/she is put into the environment with a large part in front of him/her, the user usually stays static and waits for instructions.

[0176] FIG. 31 shows a listing of the lifting capacity for infrequent movement and short distances data used for one embodiment. Although the data for both men and women is provided, this embodiment uses the figures for women to make sure the system works for everyone. If the part is below 20 pounds, the invention indicates that the part can be lifted by one hand; if the part is between 20 and 40 pounds, it indicates that the part can be lifted by two hands; beyond that, the part is put in the category of “needs to be lifted by tools”.

[0177] As described above, constrained motion simulation is used to simulate physical constraints in the virtual environment. Although the invention can simulate physical constraints by constrained motion without using collision detection, collision detection is still a critical aspect to verify and validate assembly sequences and orders. Further, since the invention can simulate dynamic behaviors of the parts in the virtual environment, the invention can be used to determine if these behaviors improve the reality feeling in the virtual environment and help the assembly planning process.

[0178] The simple categorization of the parts in the assembly models enables the invention to define the scope of dynamic simulation of the parts in the virtual environment. In one embodiment, the invention implements dynamic simulation in cases where the models are small and the parts are not heavy, i.e., in the range of “being handled by one hand”. For larger models and parts, it is not applied since these kinds of behaviors and motions are not allowed in real industrial world anyway because of safety concerns.

[0179] During the assembly process planning of operation, certain behaviors such as object bouncing are not of major concern because the invention can assume the user will behave rationally in the assembly operation. He/she may hit a part with a hammer to adjust its shape, but will not unnecessarily hit a part with the base part or other parts. The invention can model the behavior of the part in the user's hand and on the base part. In the virtual environment, first time users may try to throw a part away to see what a virtual reality system is, but an experienced user who wants to verify his/her design would rarely behave in this way. Thus, the Invention provides models for dynamic behaviors on the part while the part is held in the user's hand and while the part is constrained on the base part.

[0180] There are three kinds of physical motions for an object discussed in detail below: (1) free motion in space; (2) translation on a plane and along an axis; and (3) rotation about an axis.

[0181] Free motion in space of an object is the simplest physical motion to model. An object just follows a ballistic trajectory as described in elementary physics texts. The equations of motion are shown in equations 3.2.1 and 3.2.2. In the equations, t is the time of motion, V ₀ and ω ₀ are the initial linear and angular velocity vectors, S 0