The following gem, posted to EVALTALK by Michael Britt in 1996 (original source unknown) is a nice reminder of some of the golden rules of genuine evaluation – and how breaking them gives evaluation a bad rap:
A company president was given a ticket for a performance of Schubert’s Unfinished Symphony. Since she was unable to go, she passed the invitation to the company’s Quality Assurance Manager. The next morning, the president asked him how he enjoyed it, and, instead of a few plausible observations, she was handed a memorandum which read as follows:
1. For a considerable period, the oboe players had nothing to do. Their number should be reduced, and their work spread over the whole orchestra, thus avoiding peaks of inactivity.
2. All twelve violins were playing identical notes. This seems unnecessary duplication, and the staff of this section should be drastically cut. If a large volume of sound is really required, this could be obtained through the use of an amplifier.
3. Much effort was involved in playing the demi-semiquavers. This seems an excessive refinement, and it is recommended that all notes should be rounded up to the nearest semiquaver. If this were done, it would be possible to use trainees instead of craftsmen.
4. No useful purpose is served by repeating with horns the passage that has already been handled by the strings. If all such redundant passages were eliminated, the concert could be reduced from two hours to twenty minutes.
In light of the above, one can only conclude that had Schubert given attention to these matters, he probably would have had the time to finish his symphony.
And the relevant Golden Rules of Genuine Evaluation are:
- Make sure you “get” the whole point of the evaluand before leaping into an evaluation
- Don’t leap to measurement/analysis with your favorite tools and frameworks without first asking what methods fit the evaluand and its broad purpose
- Select the evaluation team very carefully indeed – specialized knowledge may be critical, and some highly skilled people are simply not capable of (1) or (2) when they are outside their familiar disciplinary territory