DESCRIPTION OF PREFERRED EMBODIMENTS

[0053] FIG. 1 shows diagrammatically a network management system 10 including a fault diagnosis system 12 , a topology mapper 14 , an impact analyzer 16 and a help desk system 18 . The network management system communicates with a communications network 20 (or application service). The network includes a set of interconnected network elements such as routers, bridges, switches, and repeaters. These network elements provide transportation of data between end stations. Furthermore, there are computers known as servers that provide services such as e-mail, accounting software, sales tools, etc. Typically, data is transmitted electronically or optically, and network elements can forward data in packets, frames or cells to the intended destination. Servers include network adapters and/or software that interpret the electronic or optical data packet into the data elements and pass these elements to the appropriate application being hosted.

[0054] The network management system 10 includes a commercially available processor (for example, Pentium microprocessor manufactured by Intel Corporation) executing an operating system providing an operating environment for a network management program. The processor and the operating system provide a computer platform for which application programs are written in higher level programming languages. The computer (or application host) interfaces with permanent data storage, such as a magnetic or optical disk drive, a disk array, non-volatile RAM disk, or a storage area network, which maintain data files such as user configurations and policies. In general, the network management program may be configured as a generic software application residing in any commercially available computing platform.

[0055] Preferably, fault diagnosis system 12 , topology mapper 14 , and help desk system 18 are software applications written in Java and running on any computer with a Java Runtime Environment (JRE). For example, a Dell laptop computer with an Intel Pentium processor running the Windows 2000 operating system, or a Sun Ultra 60 computer running Solaris v. 2.7. Alternately, fault diagnosis system 12 , topology mapper 14 , and help desk system 18 are developed in any object oriented or structured programming language, and compiled for execution on any one or many computer platforms, or could be implemented on a neural network computing device.

[0056] The computer has a network adaptor that provides communication (preferably, but not necessarily, IP) to the users on the network. The fault diagnosis engine application may share a host with help desk system 18 , and/or the topology mapper, or each can run on a separate host, in which case they communicate using a network adaptor. Topology mapper 14 determines the network topology and creates a model. The permanent data storage holds data files that describe the current network topology, and configuration files that control the performance of topology mapper 14 . A user is an end station, interfaced to access the network or services, used by a person who is using the network, or is using services provided by the network.

[0057] The network management system 10 performs a fault management process 30 shown in FIG. 2 . The entire process is part of a phased, componentized, but interconnected method, wherein all aspects of fault management are performed. The fault management process of FIG. 2 includes the following seven phases: fault detection 32 , diagnosis 40 , impact analysis 50 , prioritization 60 presentation 70 , recourse 80 , and resolution 90 .

[0058] Fault detection process 32 (performed by fault detectors 130 shown in FIG. 3 ) is the most basic part of the fault management system. Fault detectors 130 detect raw fault data. Fault detectors 130 receive information by SNMP polling, SNMP trap handling, performance monitoring, historical trend analysis, device configuration monitoring, application and system-level management tools, and help desk trouble tickets. Fault detection process 32 can also add information to the raw fault data enabling improved diagnosis of the fault. The fault data are assembled into fault objects.

[0059] Fault diagnosis 40 occurs after a “detected” fault is entered into a fault detection and management system 100 , which is a generic system for diagnosing a fault in any a mechanical, electrical or other system. A fault detection and management system 100 ( FIG. 3 ), processes and correlates detected faults with other faults to determine their relationship. Fault detection system 100 finds one or more “root cause” faults and isolates these faults. Furthermore, the system can optionally suppress other symptomatic faults that were “caused” by the root cause fault. Fault diagnosis 40 can be performed in a single step or can involve many techniques such as examining device neighbor knowledge, tracing the route of management data, examining route tables and ACLs, etc.

[0060] Fault impact analysis 50 determines the “scope” of the analyzed fault. After receiving a root cause fault determined, by fault diagnosis 40 , impact analysis 50 determines the consequences of this fault. This determination includes analyzing the network services affected by the fault, the users affected by the fault, and any other ramifications the fault has on network 20 , or the application being managed. Furthermore, impact analysis 50 may involve analyzing various logical layers that exist in a communication network and correlating a fault with its possible consequences at each layer. Impact analysis 50 may use a fault causality tree located in a fault repository 140 ( FIG. 3 ). The interpretation schemes include analyzing how a network fault affects services like web servers or e-mail, examining how a misconfigured router running OSPF affects the users in each area, etc.

[0061] The network management system may also perform fault prioritization 60 . After a fault has been diagnosed and its impact analyzed, the fault may be prioritized. Fault prioritization 60 assigns a priority/severity to each fault object and this is used to determine the proper presentation of the fault to a user. Fault prioritization process 60 may include multiple methods based on the type and scope of the fault such as examination of service level agreements and how the fault violates them, mission critical device analysis, and fault scope.

[0062] The network management system may also perform fault presentation 70 . Fault presentation 70 provides the mechanism by which the system alerts a user that a fault has occurred. Fault presentation process 70 presents all information about the fault in a user friendly manner. Fault presentation 70 may include steps and processes the systems used to diagnose the fault, thus allowing a user to verify the diagnosis and “trust” the system to accurately diagnose faults. Fault presentation 70 may also include a network monitoring alarm system.

[0063] The network management system may also include fault recourse 80 . Fault recourse 80 provides a way in which a user can change the network management based on a given fault. For example, fault recourse 80 may involve reducing or stopping polling of devices downstream from a fault, reconfiguring connectivity modeling, script invocation to fix a misconfigured static route, or configuring user groups for a different email server.

[0064] The network management system may also include fault resolution 90 . After presenting a fault to a user and fixing the problem, problem resolution 90 records the process for future fault detection and diagnosis. Fault resolution 90 can automatically trigger for any single resolved fault a re-evaluation of associated faults in the system. This re-evaluation proactively assesses the full scope of a resolved fault. If an associated fault is still not resolved, diagnosis can be re-started to determine the cause. This process is facilitated by the use of the fault causality tree created as a result of fault diagnosis process 40 .

[0065] FIG. 2A shows diagrammatically in detail fault diagnosis process 40 . A detected fault enters the fault detection and management system and a fault object is created (step 42 ). The fault diagnosis engine ( 101 in FIG. 3 ) triggers appropriate fault handlers (step 43 ). A diagnoser fault handler generates possible faults that may be causes of the previously entered fault (step 44 ). For each generated, possible fault, fault diagnosis engine 101 triggers appropriate tester fault handlers (step 45 ). Each tester fault handler performs vendor-specific and domain-specific tests to determine the existence of one or several possible faults. Next, the tester fault handler records test results (step 46 ). If possible additional faults exist, the fault diagnosis engine continues to trigger tester fault handlers and diagnoser fault handlers (step 49 ). If there are no other possible faults, the fault diagnosis engine has isolated the fault and the system proceeds to impact analysis 50 .

[0066] FIG. 3 illustrates diagrammatically a fault detection and management system 100 . One embodiment of fault detection and management system 100 is fault diagnosis system 12 ( FIG. 1 ). Fault detection and management system 100 includes five main parts: a fault diagnosis engine 101 , a fault object factory 110 , fault detectors 130 , a fault repository 140 , and fault handlers 150 . Fault detection and management system 100 has the ability to receive detected fault information from multiple sources, control the management of the faults, and produce a root cause analysis. Furthermore, the system also provides a mechanism for performing fault correlation and impact analysis. The impact assessment is not limited to the impact of the communications network, but may include disruptions in services or applications that depend on the network infrastructure.

[0067] Fault object factory 110 receives data from fault detectors 130 and creates fault objects 112 shown in FIG. 3A . Each fault object 112 is associated with a fault type and there may be many fault types. Furthermore, each instance is a separate occurrence of a problem, potential problem, or condition of a communication network or an element located in the communication network (such as a misconfiguration, a degradation of service, physical failure or other).

[0068] Referring to FIG. 3 A, the entire architecture of the fault detection and management system based on fault objects 112 , which are records representing a detected problem, a potential problem, or a condition. Fault object 112 includes information about a detected fault, that is, includes a description of the problem or condition stored in field 114 , time and date of the reported problem 116 , a fault processing state 118 , and one or more test result objects 120 . The fault structure includes a context that is a mechanism for sharing varying amounts of data related to the fault; these amounts may exist between each instantiation of a type of fault.

[0069] Referring to FIG. 3 , fault detector 130 detects a problem or potential problem on an entity in a managed system. Fault detector 130 provides a record of the condition to fault object factory 110 , which generates fault object 112 . Fault detector 130 can monitor an entity or receive unsolicited notification from an entity when a problem occurs, according to different methods known in the art. Fault detector 130 may perform a test and may provide to fault object factory 110 data with the results of the performed tests. Fault detector 130 may share a host with fault diagnosis engine 101 , or may reside externally as an agent.

[0070] Referring to FIG. 3 B, fault repository 140 is the component used by a fault detection and management system 100 to store and access fault information. fault repository 140 stores every fault object 112 present in the system. Each component of the system (detection, diagnosis, etc.) can enter new fault objects into fault repository 140 and access any fault object 112 . Preferably, fault repository 140 includes a table structure with services capable of searching and locating existing faults.

[0071] Fault repository 140 also includes fault associations 142 , which provides a mechanism for relating faults to one another. Specifically, each defined fault association relates two fault objects. One fault object is on the left side of the association, and the other fault object is on the right side as shown for fault trees below. The semantics of an association are defined by the type of the association. New fault association types can be defined and added to the system, preferably using Interface Description Language (IDL) definitions of an interface for a service that uses the Common Object Request Broker Architecture (CORBA) transport protocol.

[0072] Referring again to FIG. 3 , each fault handler 150 performs a designated type of work as a result of a fault object entering a certain processing state (shown in FIG. 5 ). Fault handlers 150 may exist internal to the system, or reside externally in a separate process. Fault handlers 150 are registered for a particular fault type and state and, as part of the registration process, each fault handler 150 has an integer priority value. Then, fault handlers 150 are sorted by their priority values so that a fault handler with the lowest priority value is triggered first and subsequent handlers are triggered in sequence, as described below. One type of fault handler 150 can test a fault object and create a test result record. Furthermore, fault handler 150 may create additional types of fault objects, create associations between fault objects, correlate fault objects that indicate a similar problem, or perform impact analysis on a fault object to determine the scope of a problem. A tester fault handler 152 performs a selected test on a fault. A diagnoser fault handler 154 creates additional types of fault objects.

[0073] Fault diagnosis engine 101 is the central component of fault detection and management system 100 since it drives the management and diagnosis of faults. Fault diagnosis engine 101 provides a generic mechanism for fault handlers 150 to register for changes in the processing state of faults of a given fault type. Fault diagnosis engine 101 may employ any mechanism to specify registrations. The preferred implementation of fault diagnosis engine 101 uses XML (Extensible Markup Language) technology.

[0074] Referring to FIG. 4 , when a fault transitions to a state for which a handler has registered, the engine triggers the handler to perform its work. Fault diagnosis engine 101 can trigger one of fault handlers 150 arbitrarily or may use some ordering mechanism. Preferably, fault diagnosis engine 101 uses a priority mechanism to order the triggering of fault handlers that are sorted by their priority value (by triggering first a fault handler with the lowest value).

[0075] Fault detection and management system 100 uses fault processing states for analyzing faults. A fault's processing state represents its status in the fault management process and provides a mechanism to control the management of the fault. A fault can have a large number of possible states, and a fault can transition from state to state using different ways, as shown in FIGS. 5 and 5 A. Preferably, the system utilizes a fault type hierarchy in which generic base fault types are defined and from which, new more specific fault types can be derived. Each fault, which exists in the system, is of some pre-defined fault type.

[0076] Referring to FIG. 3C, a test result object 120 includes a record of test results that were performed to determine the existence of the problem or condition for which the fault was created. Test result object 120 includes a textual description of the test (field 122 ), data identifying from the target of the fault (field 123 ), test data (field 124 ), any thresholds and parameters used in determining the test result (field 125 ). Test result record 125 also contains a state representing the status of the test.

[0077] While performing its work on a fault object, a fault handler may cause the processing state of the fault to be changed. In this case, no other handlers for the current state are triggered. Fault diagnosis engine 101 obtains the handlers for the new state and resumes triggering with the new handlers when the current handler completes its work.

[0078] FIG. 4 illustrates the triggering mechanism using a flow diagram. Fault diagnosis engine 101 provides a triggering mechanism and controls and manages the diagnosis process.

[0079] Referring to FIG. 5 , fault diagnosis engine 101 utilizes processing states of a fault to control the flow of diagnosis for that fault. As described above, fault handlers 150 are triggered for a fault based on the current processing state. The transition diagram of FIG. 5 defines the following processing states: An initial state 180 begins the life-cycle of a fault object. A detected state 182 indicates that an external fault detector 130 or an internal handler 150 positively determined the condition (that the fault represents) as a problem. A testing state 184 indicates the fault is unverified; that is, the condition that the fault represents requires testing to determine if it is a problem. A completed state 184 indicates that fault diagnosis has completed for the fault.

[0080] Fault diagnosis engine 101 may allow fault handlers 150 to directly transition a fault between states, wherein preferably the processing state is hidden from fault handlers 150 . The engine transitions a fault's processing state based on the state of the current result of the fault as provided by the handlers. These are the following three test result states (shown in FIG. 5 A): PROBLEM indicates a test has identified the fault to be a problem; NO_PROBLEM indicates a test has verified the condition that the fault represents does not or no longer exists; and UNKNOWN indicates a test could not be completed for some reason or the condition that the fault represents requires verification.

[0081] FIG. 5A illustrates transition of the processing states (shown in FIG. 5 ) based on test results of an analyzed fault. For example, fault diagnosis engine 101 triggers tester fault handler 152 ( FIG. 3 ) for testing state 184 and fault handler diagnoser 154 for detected state 182 . Furthermore, fault handler diagnoser 154 may also be triggered for testing state 184 if there are no tester fault handlers that can perform a direct test. There may also be fault handlers for completed state 184 , which would not perform diagnosis, but would perform other tasks such as correlating faults that share a common root cause (described below) or notifying a presentation system to display the diagnosis results when performing presentation process 70 .

[0082] Fault diagnosis engine 101 may employ further rules that govern the triggering of fault handlers when there are multiple handlers (or types of handlers) for a particular processing state. If there are multiple types of handlers, the engine may impose an ordering such that all handlers of one type are triggered before any handlers of another type. Furthermore, if a handler provides a concrete result, as defined by the various result states, the engine may suppress remaining handlers of that type from being triggered and/or may suppress handlers of other types.

[0083] According to the preferred embodiment, since there may be both tester fault handlers 152 and diagnoser fault handlers 154 registered for testing state 184 , fault diagnosis engine 101 imposes a rule that all tester fault handlers are triggered before any diagnoser fault handler. This is because a tester fault handler can directly determine the existence or nonexistence of a problem, but a diagnoser fault handler cannot. In addition, if a tester fault handler or diagnoser fault handler provides a concrete result, then fault diagnosis engine 101 suppresses remaining handlers for the current processing state. A concrete result is one whose state is either PROBLEM or NO_PROBLEM. A result state of UNKNOWN is not concrete, that is a result could not be positively determined, as shown in FIG. 5A .

[0084] Fault detection and management system 100 utilizes a decomposition approach in the diagnosis of a fault to determine the root cause. Fault detector 130 enters a problem or potential problem into fault object factory 110 , which creates a fault object treated as a symptom fault. The symptomatic fault is decomposed into one or more constituent faults that further refine the symptom as shown in FIG. 6 . Each constituent fault represents a possible suspect that may be causing the symptom. For each constituent fault, tests may be performed to determine the existence of a problem or the fault may be decomposed into further suspects. The process continues until all faults have been completely decomposed and there are no more suspects.

[0085] The end result of this process is a hierarchy of faults in the form of a tree with the original symptomatic fault at the root as shown in FIG. 6 . The fault tree includes a root fault level, one or several intermediate fault levels, and a leaf fault level. Each fault in the tree, except the root, has at least one parent fault from which it was decomposed. Each fault also has zero or more child faults that were spawned from it. A child fault represents a possible cause of its parent. A fault that has children but is not the root is termed an intermediate fault. A fault that has no children, that is one that could not be further decomposed, is termed a leaf fault. A leaf fault that indicates a problem is a probable cause of the root symptom. There may be more than one root cause.

[0086] Referring to FIG. 6 , fault A is the original root symptom. Fault A is decomposed into faults B and C, fault B is decomposed into faults D and E, and fault C is decomposed into faults F and G. Faults B and C are intermediate faults because they have children but are not the root. Faults D, E, F, and G are all leaf faults.

[0087] Fault tree 200 enables fault detection and management system 100 to locate one or several root causes of any fault in the tree by traversing the children of that fault and compiling the leaf fault(s) that indicate a problem. Fault tree 200 as a whole also embeds the entire diagnosis process. By traversing the entire sub-tree of any fault, one can compile a complete log of the steps taken and the results of tests performed to diagnosis the fault. Thus, a presentation process 70 can display the root cause(s) of a fault and/or can present a diagnosis log allowing an end user to verify the process.

[0088] Referring to FIG. 3 , fault diagnosis engine 101 manages the structure of fault tree 200 . Fault handlers 150 provide the contents and semantics of the tree. For each fault in fault tree 200 , one or more fault handlers 150 are triggered. Fault handler 150 may perform a specific test on the fault and provide a result of the test to the engine or it may create one or more child faults to find a possible cause. Each new child fault creates a new branch in the fault tree. A branch may be represented preferably by a fault association called MaybeCausedBy shown in FIG. 6 .

[0089] Tester fault handler 152 performs a direct test and a diagnoser fault handler 154 spawns possible suspect faults. Other types of handlers may correlate similar faults or perform impact analysis. Fault handler 150 could be both test fault handler 152 , and diagnoser fault handler 154 , which can perform a test, provide a result and also attempt to find the cause. Preferably, a handler is not both test fault handler 152 and diagnoser fault handler 154 . Furthermore, if diagnoser fault handler 154 does not provide a direct result for a fault object, a composite result is computed from the results of the fault's children (shown, for example, in FIG. 6 ). Fault detection engine 101 or diagnoser fault handler 154 may compute the composite result according to rules provided below.

[0090] Referring to FIG. 6 A, fault A is entered into the system by detector 130 as a problem. Also referring to FIG. 5 A, the fault begins in initial state 180 and is transitioned to detected state 182 . Diagnoser fault handler 154 creates two child faults, B and C. A MaybeCausedBy association is created between faults A-B and A-C, as shown in FIG. 6A .

[0091] The current result state for each fault is shown in the upper right corner of each fault box. The result state for faults B and C is indicated by a question mark (?) because a result has not yet been computed. Since faults B and C are unverified, the engine transitions these faults to testing state 184 . Diagnoser fault handler 154 creates for fault B new faults D and E, and creates for fault C new faults F and G. Next, fault diagnosis engine 101 triggers tester fault handlers 154 for faults D, E, F, and G and these testers assert results shown in FIG. 6B . Since results have been computed for faults D, E, F, and G, fault diagnosis engine 101 transitions these faults to the completed state 186 ( FIG. 5A ). Next, fault diagnosis engine 101 computes a composite result for faults B and C according to the following default rules for composite result computation:

[0092] 1. If any child fault result state is PROBLEM, then the parent fault's result state is PROBLEM.

[0093] 2. If all child fault result states are NO_PROBLEM, then the parent fault's result is NO_PROBLEM.

[0094] 3. Otherwise, the parent fault's result is UNKNOWN.

[0095] Using the above rules, the composite result for fault B is NO_PROBLEM and the composite result for fault C is PROBLEM. Thus fault F is the cause of fault C. The system indicates this causality with another association called CausedBy as shown in FIG. 6C .

[0096] The fault diagnosis is now complete on faults B and C so they transition to completed state 186 . The composite result for A is PROBLEM since the result state for fault C is PROBLEM and a CausedBy association is created between A and C as shown in FIG. 6C .

[0097] As described above, the system executed root cause analysis and determined the root cause for symptomatic fault A is fault F. The diagnosis log for fault A shows that the conditions tested for faults D and E did not indicate a problem and that a result for fault G could not be determined, possibly because of the problem on fault F.

[0098] If the resulting fault tree did not find a root problem, then the composite result for fault A would indicate NO_PROBLEM. Such result would contradict the original assertion of a PROBLEM. In this case, the engine would throw out the composite result and leave the original PROBLEM result. Such a problem may have been intermittent and resolved itself quickly, or the detector was faulty, or the diagnosis was incomplete, perhaps requiring additional testing and diagnosis.

[0099] Referring to FIG. 7 , according to another situation, multiple symptomatic faults may be caused by the same root problem. This may result in portions of multiple fault trees being identical, that is the diagnosis for multiple symptomatic faults would create the same faults with possibly the same results. Consider two symptomatic faults, A and B shown in FIG. 7 . Both faults enter the system at the same time and fault handlers for both faults A and B attempt to create same fault C. The engine can handle this scenario in several different ways shown in FIGS. 7 and 7 A. A simple implementation would create two copies of fault C, one for each fault tree. Each fault tree would only be aware of its copy and, thus, each copy of fault C would be diagnosed separately. This is depicted in FIG. 7 with C being the copy. The above approach may result in tests being performed for the same condition twice, which may not be desirable for performance reasons.

[0100] Alternatively, the system creates two copies of fault C but “reuses” the results of test(s). For example, consider that fault C is created first and subsequently tested. A short time later fault C′ is created. Instead of performing the same test again, the engine would use the test result from fault C for fault C′. A drawback to this approach, however, is that, depending on the semantics of the test, a significant amount of time may have passed such that the result computed for fault C may be invalid for fault C′, that is the result for C is now “stale”. To alleviate this issue, the system may employ certain rules or heuristics to determine when and if a test result can be reused. The system may only reuse the results from certain tests or may only reuse a result depending on its state. For example, using the test result states defined above in the preferred implementation, a NO_PROBLEM result may always be re-tested but a PROBLEM or UNKNOWN result may be reused regardless of the time elapsed. The engine may also “age” test results. For example, if fault C′ occurs within a certain amount of time after fault C as determined by some heuristic “aging” factor, then the result for C can be used. Otherwise fault C′ is re-tested. An “aging” factor may be defined system-wide or an “aging” factor may be specified per fault type or per test. A system implementation may utilize only one set of rules or heuristics for test result reuse or may use a combination of approaches.

[0101] FIG. 7A depicts another approach to handling identical fault tree portions or sub-trees is to share sub-trees amongst multiple symptomatic faults. In this embodiment, instead of multiple fault super-trees maintaining their own copy of a particular sub-tree, the super-trees would “intersect” at certain faults and share a common sub-tree. Therefore fault C is shared by faults A and B.

[0102] Thus, faults A and B would share fault C and its associated test result(s). A similar issue exists, however, regarding “stale” test results, as described above. Similar rules or heuristics can be applied here as well. If fault B intersects with fault C some time after fault A, these rules can be applied to determine if fault C needs to be re-tested. FIG. 7B illustrates a more complete example using the test result states defined above in the preferred implementation. In the diagram of FIG. 7 B, faults A and F intersect at fault C and share its PROBLEM test result. The root cause of both A and F is fault D.

[0103] According to another important aspect, fault management and diagnosis system 100 enables resolution and possible re-evaluation of a previously detected problem. The system reacts when a monitoring system detects that a problem has been resolved or no longer exists. The system may remove faults and notify presentation 70 of the resolution. This process can be automated and/or require user intervention. If a user instructs the system that a fault has been resolved, the system could merely remove the fault or could choose to re-evaluate the condition represented by the fault and verify resolution.

[0104] The fault tree hierarchy can facilitate resolution and re-evaluation of faults. The system provides a mechanism allowing an observer or detector to specify that a fault originally entered into the system as a problem has subsequently been resolved. Additionally, problems detected by internal handlers in the system may monitor the condition to detect resolution. When a fault is deemed resolved, the engine would re-test all the faults, if any, in the sub-tree of the resolved fault and propagate the new computed result “up” to its parent fault. The parent fault would then re-test and propagate its new result. This process continues until the entire tree has been re-evaluated. This may result in the entire tree being resolved or the isolation of a new cause for the root symptomatic fault if it is determined that the root symptom is still a problem even though the original root cause has been resolved.

[0105] As shown in FIG. 6 C, the system diagnosed fault F as the root cause of symptomatic fault A. Assume the fault handler that tested fault F is monitoring the condition and, some time later, detects that the problem no longer exists. A new test result is asserted for fault F with a state of NO_PROBLEM that contradicts the original PROBLEM result shown in FIG. 6C . Fault diagnosis engine 101 then makes an “inference” on the parent fault of fault F with the new result state of fault F. An inference on a fault is not an assertion of a new result but a suggestion used to determine if the fault needs to be re-evaluated. If the inferred result state contradicts the current result state for the fault, the fault is re-evaluated. In this case, the engine infers a result state of NO_PROBLEM on the parent fault of F, fault C. Since this inference contradicts the current result state, PROBLEM, fault C is re-evaluated. Fault diagnosis engine 101 transitions the processing state of fault C from completed state 186 back to testing state 184 (shown in FIG. 5 ).

[0106] Fault diagnosis engine 101 re-triggers diagnoser fault handler 154 for fault C ( FIG. 8 ), which in turn attempts to re-create fault F and fault G. However, since faults F and G already exist in the current tree, no new faults are created. The engine then infers NO_PROBLEM on faults F and G as shown in FIG. 8A . There is a contradiction for fault G, since its current result state is UNKNOWN, so it is transitioned back to testing state 184 , but fault F remains in completed state 186 since no contradiction occurs and a new result was just asserted for it. Assume tester fault handler for fault G finds that G has been resolved and asserts the NO_PROBLEM result state. Diagnosis for fault G is complete so its processing state is changed to completed state 186 . The composite result computed for fault C would now be NO_PROBLEM and the CausedBy association between C and F removed, as shown in FIG. 8A .

[0107] The new result state for fault C is now propagated to fault A, which causes A to transition back to testing state 184 . However, since both child faults B and C are NO_PROBLEM, the composite result for fault A now becomes NO_PROBLEM and the causality association between faults A and C is removed, as shown in FIG. 8B .

[0108] The entire fault tree, shown in FIG. 8 B, was re-evaluated except for the sub-tree rooted at fault B, which was not re-evaluated because its original test result state was NO_PROBLEM and no contradiction occurred (see FIG. 6C ). If the engine employs an aging factor for test results, as described above, then fault B could be re-evaluated if its result is deemed stale.

[0109] In the embodiment performing aging factor testing, fault detection engine 101 re-evaluates fault results deemed stale. Specifically as shown in FIG. 8 B, fault detection engine 101 re-evaluates fault B's sub-tree. Assume that fault E is re-tested and the result is NO_PROBLEM, but fault D re-tests with a new result of PROBLEM. The composite result for fault B is now also PROBLEM and a causality association is created between faults B and D ( FIG. 8C ). Fault C is also re-evaluated. Fault detection engine 101 generates a composite result for fault A according to the above rules. Referring to FIG. 8 D, based on results of faults B and C, fault A is PROBLEM as determined in the fault tree of FIG. 6 C, but now its root cause is fault D.

[0110] The above-described fault diagnosis and management process may be applied to any system. In the following example, the above-described fault diagnosis and management process is applied to diagnosing a fault that occurred in a communications network 210 shown in FIG. 9 . Communications network 210 includes a router R 1 connecting a subnet 212 and a subnet 220 , and includes a router R 2 connecting a subnet 220 and a subnet 225 . Subnet 212 includes a bridge B 1 , a workstation 214 , and a server 216 . Subnet 220 includes a bridge B 2 and a workstation 222 , and subnet 225 includes a bridge B 3 and a HTTP server 228 . Workstation 214 includes client agent with IP address 10 . 253 . 100 . 10 . The client agent is monitoring HTTP server 228 with IP address 10 . 253 . 102 . 15 by periodically requesting an HTTP web page. The fault diagnosis and management process is running on workstation 222 at 10 . 253 . 2 . 104 . A DNS server resides on machine 216 having IP address 10 . 253 . 100 . 1 .

[0111] For example, the client agent 214 tries to load an HTML page with the URL “www.acme.com/index.html” from HTTP server 228 , but receives no response. The client agent, acting as fault detector 130 , sends information about the failure to fault object factory 110 . Fault object factory 130 creates a fault object of type HTTPLostFault from the information provided and its processing state is set to the detected state 182 ( FIG. 5 ).

[0112] As shown in FIG. 4 , fault diagnosis engine compiles a list of fault handlers (step 172 ) registered for the HTTPLostFault type. Instep 176 , diagnoser fault handler 154 registered for the HTTPLost fault type is triggered. Diagnoser fault handler checks for two possible causes of the problem: a DNS service failure and an HTTP service failure. The DNS is checked because name resolution may be needed on the client side to determine the IP address of the HTTP server. Thus, the diagnoser creates two faults: one of type DNSServiceFailure fault 234 and the other of type HTTPServiceFailure fault 238 , as shown in FIG. 10 .

[0113] For DNSServiceFailure fault 234 , diagnoser fault handler 154 creates two faults: one of type DNSServerDown fault 235 and the other of type FullContactLost fault 236 . Similarly, diagnoser fault handler 154 for the HTTPServiceFailure fault 238 creates two faults: one of type HTTPServerDown fault 239 and the other of type FullContactLost fault 240 , as shown in FIG. 10 .

[0114] For the HTTPServerDown fault 239 tester fault handler 152 makes a query to agent running on HTTP server machine 228 at IP 10 . 253 . 102 . 15 to verify that the HTTP server process is running. Assume HTTP server 228 is running so the result state of the test is set to NO_PROBLEM. Tester fault handler for the FullContactLost fault 240 , child of the HTTPServiceFailure fault 238 , performs an ICMP echo request (or ping) from the management station 222 at 10 . 253 . 2 . 104 to HTTP server machine 228 at 10 . 253 . 102 . 15 . Assume a response is received from the ping, thus the result state of this test is also NO_PROBLEM. At this point, fault detection engine 101 has verified that the management station can contact the HTTP server machine 228 and the HTTP service is available.

[0115] Similarly, tester fault handler 152 for the DNSServerDown fault 235 makes a query to the agent on DNS server machine 216 at IP 10 . 253 . 100 . 1 to verify that the DNS server process is running. In this case, for example, no response is received from the agent on machine 216 . As shown in FIG. 10 A, tester fault handler 152 asserts a test result with state UNKNOWN on DNSServerDown fault 235 because it could not be positively determined if the DNS service is available or not. The server process could be running but a network failure may be preventing access from the management station. Next the FullContactLost fault tester handler 152 is also triggered for the other FullContactLost fault from source IP 10 . 253 . 2 . 104 to Destination IP 10 . 253 . 100 . 1 , the DNS server 216 . Assume the DNS server 216 is down, this ping request fails and no response is received. Tester fault handler 152 asserts a result with state PROBLEM as shown by 236 P in FIG. 10A .

[0116] As shown in FIG. 10 B, according to the rules stated above, the composite result state for the HTTPServiceFailure fault 238 NP is NO_PROBLEM, and the composite result state for the DNSServiceFailure fault 234 P is PROBLEM. A CausedBy association is created between the DNSServiceFailure fault 234 P and FullContactLost fault 236 P.

[0117] A composite result state of PROBLEM is computed for the root HTTPLostFault, which agrees with the original PROBLEM assertion, and a CausedBy association is created between the HTTPLost fault 232 P and the DNSServiceFailure fault 234 P. Thus, the root cause of the HTTP request failure is that the DNS server machine is down, preventing name resolution of the HTTP server machine.

[0118] Additional, more complex diagnosis can be performed to check for other possible causes, such as a bad URL, configuration problems on the client side such as invalid TCP/IP configuration for DNS and the default gateway, and hardware problems on the server side such as a hard disk failure. Diagnoser fault handler 154 can be implemented for the FullContactLost fault to trace a path between the source and destination and create faults. Such a path dependent test tests the network devices and ports that the data traverses, at both Layer 2 and Layer 3, to further isolate a problem as is described in PCT application (docket No. A3-03WO) entitled “Systems and Methods for Diagnosing Faults in Computer Networks” filed on May 7, 2001, which is incorporated by reference.

[0119] Preferably, the FullContactLost faults 236 and 240 contain the client's IP address as the source instead of the management station's, as in the above example. Thus, the tested path is from the client's perspective. Also, in the above example, the HTTPLost fault 232 was detected by a software agent monitoring the HTTP server. Such a fault could also be injected into the fault diagnosis system via a Help Desk by a user experiencing a problem as described in the co-pending PCT application (docket no A3-05WO) entitled “Help Desk Systems and Methods for use with Communications Networks” filed on May 7, 2001, which is incorporated by reference.

[0120] Numerous other embodiments not described in detail here can apply the principles described to particular applications and are within the scope of the claims.