Completing your knowledge base projects requires testing, testing, and testing. And, hopefully, revisions between tests. What follows are some comments to make this process more systematic.
Conceptually there are 3 phases of validation/verification:
1) justifying the need to build an ES (and deploy it)
2) syntactically verifying its consistency and completeness
3) externally validating its performance.
Since we're beyond (1), these remarks focus on items (2) and (3). These comments mainly apply to KBs where the rules are certain, and they require certain adjustments to accommodate KBs that reason with uncertainty.
The diagram distributed in class (about recommending CD-ROM or Intranet-based publication) serves as the basis for these comments:
Redundant Rules: These have logically identical IFs, but the THEN part of Rule-a is a subset of the THEN-part of Rule-b. Example: Rule 1 and Rule 2 are redundant. The problem, of course, is that Rule 1, in this case, does nothing! Worse, it (and all the following problems, too) can make the KB harder to maintain. (It can also lead to "double counting" if you accumulate uncertainties.)
Conflicting Rules: These have logically identical IFs, but contradictory THENs. Example: Rule 3 and 4 conflict. Contradictory rules may mean one of your rules is erroneous. Sometimes, you want to allow multiple values (multiple recommendation, or multiple intermediate conclusions), so you may wish to preserve conflicting rules. But, usually, they mean either an outright error, or that you forgot to add an additional IF clause (or 2) to one of the conflicting rules. Eg: Rule 4 may require the additional if clause "AND Customer | travel is NO".
Subsumed Rules: These have identical THENs, but the premises of one are a subset of the other. Example: Rules 6 and 7. Why is this bad? Well, apparently the "Security | Concern" of Rule 6 isn't doing anything! Said another way: Rule 6 is just a special case of Rule 7, which subsumes it.
Unnecessary Premise: Two rules are identical, with the exception of the logical "sign" associated with one of their IF clauses. Example: Rules 8 and 9. So it looks like Information | Format is unnecessary (since it can be either Text or Non-Text, and the conclusions stays the same). So, get rid of that clause (and one of the rules).
Unachievable intermediate premise: A non-goal rules is not an a "path" to any goal. Example: Rule 10. No matter how good or important this bit of knowledge is, it does not participate in your ES' back chaining.
Unreferenced attribute values: Values from the real world that are not part of the knowledge base. Example: The object | attribute Network | Connection only references the value TCP/IP. So, any other kind of network connection is not mentioned (or used ) in this KB.
With practice, you can use an inference net and read off these difficulties. An unachievable intermediate premise is easy, since spotting paths not leading to goals is something we see very automatically. But the other rule types can be read from the net, too: For instance, the only way one rule may subsume another is if both rules have identical THENs. It's easy to see from the inference net which rules fit this bill. The other problems can be read off, too; it just takes a bit of practice. (Remember: jazz piano?)
Notes regarding uncertainly: I've mentioned that uncertainty can change things. For instance, given certain symptoms (IF ...... ) you may want to make "contradictory" conclusions. For instance, one rule may conclude: ... THEN diagnosis "common cold" 0.5; and another may conclude: ...THEN diagnosis is mono 0.3 You may like this "conflict." Similarly, you may want to retain subsumed rules. For instance, you might decide that Rule 6 "strengthens" the case for choosing an Intranet compared to Rule 7. So, the conclusion for Rule 6 might have a certainty of 1.0, whereas Rule 7 (a more general rule) has a certainty of only 0.5.
The other thing (which I alluded to earlier) that you've got to do differently when you write uncertain rules is watch how you let rules combine their influence. For instance, you may want to give Rule 7 a certainty of 0.7; and then give Rule 6 a certainty of 0.5. Your reasoning would be as follows: If just rule 7 fires, I'm fairly sure of the conclusion (0.7). Rule 6 only fires when rule 7 does (since it's a specialization of Rule 7). So, if I accumulate evidence, Rule 6 and 7's combined effects are to set the certainty to a value of 1 - (0.3) * (0.5) = 0.85.
So, as you can see if you're reading carefully (you are, aren't you?), there are subtle issues concerning Rule- and Or-Stops, choosing if you want to keep "bad" rules (conflicting rules, bad rules, ...), and what kind of uncertainty combination you wish to use. Jazz piano, anyone?
Gets to be a pain to keep entering in the same 92 values every time you test your ES, right? So, here's what to do:
1) Save the facts (at any point in an ES consultation): File | Save Facts
2) Before the next "run" do this: Edit | Options Clear-all = No.
3) Inference | Clear all -- to clear out any inferred values from a past run
4) File | Read facts -- and Open the appropriate Fact File in the list presented to you.
5) Inference | Attempt all goals -- since 2) means "read facts" wont be cleared out
In prose, this means that you can take your ES to a particular point and "freeze it" (step 1). Since File | Save Facts allows you to name this "frozen state," you can have as many of these frozen states as you want.
The other steps just pull the facts back into you KB. Steps 2 and 3 just say that you want Mahogany to clear out all existing facts when you say to (i.e., just before you import your "frozen state," not immediately after you say Inference | Attempt All Goals (which is the usual way).
At any time: you can ask Why? to a question. This gives you a peek at what your ES is doing, and, as those of you come to my office know, can give some surprising (and correct) information about what is going on.
At any time: you can do an object | display. This will show you the current values of all objects in your system, even if your ES is still part way through a run.
On Macs, you can press the Apple+(.) (apple and period keys simultaneously) in the middle of backchaining. This will pause the inference. You can then use your object | display to check on suspicious values.
And, of course, you can use the trace routines under Edit | options.
All the syntactic checking in the world can only ensure that your ES is internally consistent (no logical holes, no inconsistencies, no unnecessary rules, etc.) The ultimate proof of the pudding is how effectively your ES does its job. This is where you expert validates that the system is working as it should (given any intended limitations of its use). As I've mentioned, checking for external validity is something you do iteratively. You move from easier to harder test cases, "canned" to live, etc (as we talked about). In the real world, the knowledge engineer and expert ultimately come to mutual agreement that the ES has performed its intended task at the agreed to level of performance.