Debugging, problem solving and Einstellung

Debugging, Problem solving, and Einstellung

Over 50 years ago Luchins demonstrated the powerful effect of "mental set." He found that subjects given N problems that could all be solved in the same manner would often fail to notice an obvious solution to the N+1st problem. Instead, they would attempt to apply the same procedures that worked for the first N problems. This failure to see an obvious solution was called "Einstellung" which means psychic blindness.

Martin Levine in 1975 expanded on Luchins's work and proposed that hypotheses can be meaningfully grouped into domains of related hypotheses. Levine proposed that (a) subjects will test hypotheses from the domain that contained the solution to previous problems and (b) once subjects start testing hypotheses from a. domain, they will not switch domains until they have tested all the hypotheses in the domain. This implies that if subjects are testing hypotheses from an infinitely large domain that does not contain the solution, they will never find the solution. Indeed, Fingerman and Levine (1991) showed that subjects induced to test hypotheses from the domain of position sequences (e.g., left, left, right, left, left, right, etc.) will overlook a simple solution to a problem (i.e., alway choose the stimulus with the "A" on it) and fail to solve the problem even after hundreds of trials.

I believe that real-world problem solving is often very inefficient because people have a propensity to miss obvious solutions to problems and spend a great deal of time testing complex and hard-to-test hypotheses. One finding from my early work in this area (Lane, Bleichfeld, McDaniel, and Rabinowitz, 1976) is that people are more likely to switch from the domain of simple hypotheses to the domain of complex hypotheses than vice versa. This suggests that once a person starts testing hypotheses more complex than the solution, they will have a difficult time solving the problem.

More recently, I have become interested in applying notions of hypotheses testing theory to trouble shooting/debugging. Debugging is the most time consuming and costly portion of software development. It is also a very time-consuming portion of many users' interaction with their computers. A basic issue in hypothesis testing is the order in which one chooses to test hypotheses. A very costly error is to test hypotheses that are complicated and/or take a long time to test when the solution is very simple. There is ample anecdotal eveidence that this happens frequently. Therefore, the most efficient strategy is often to test the easiest-to-test hypotheses first. In a series of studies my students and I have shown that subjects are typically not attuned to the strategey of testing hypotheses based on how hard it is to test them. Although in the "real world" there is often a trade off between the likelihood that a hypothesis is correct and the ease with which it is tested, we studied simpler situations where the probability of being correct was held constant.

Brad Ashby, in his master's thesis, presented subjects with a list of algebraic expressions and asked them to find the one expression that was incorrect. A typical expression might be: 2a+3b=24. To test whether an expression is correct, subjects would click on the expression so as to be presented with a screen containing the values of "a" and "b". The algebraic expressions varied in length (number of terms). Expressions checked by subjects were clearly marked so that subjects would not need to remember which expressions they had already checked. Since all expressions had an equal probablity of being the one that was incorrect, the most efficient strategy is to check the shorter expressions first. Few subjects used this strategy at first, although a majority of subjects learned it over a series of trials. Interestingly, when the task was modified slightly so that two of the expressions were in error, subjects failed to learn to use the "test the easiest to test items first" strategy.

This research demonstrates that even in a very simple laboratory situation, subjects are not adept at choosing the order to test hypotheses. It seems likely that people would fair even more poorly in the more complex hypothesis testing situations that arise in the real world.

More recently Tim Dammon and I (Dammon, C. T., & Lane, D. M.(1993) Transfer of training on a fault diagnosis taks. Proceedings of the Human Factors Society's 37th annual meeting, 1272-1276.) investigated the learning and transfer of the strategy of testing easiest-to-test hypotheses first. In the first experiment it was found that training on the "one bug" problems in the task used by Brad Ashby transferred readily to "two-bug" problems. The second experiment investigated whether training on this task would transfer to an entirely different hypothesis-testing situation. Subjects were first trained on the "one-bug" problem and then given a checkbook balancing task where the error could be something easy to test (such as an incorrect entry) or something harder to test (such as computing interest incorrectly. Despite the difference in the surface features, subjects were able to transfer the underlying strategy of testing easiest-to-test hypotheses first.

This result suggests that a very general problem solving strategy can be learned and then applied spontaneously in a variety of situations. This contrasts with other research on transfer has found that it is difficult for subjects to transfer learning to situations that do not share surface features with the situation in which the original learning took place.

A pilot study conducted by Tim Dammon suggests that experiencing Einstellung has a longer-lasting effect then simply learning about it. Two problems that have easily-missed simple solutions were used in this study. Half of the subjects were given one of these problems while the other half were told about the danger of overlooking simple solutions. Half of each of these two groups was tested the same day on the other problem; the other subjects were tested a week later. Subjects tested the same day readily found the simple solution. The key finding was that subjects tested a week later were much more likely to find the simple solution if they had been given the first problem than if they had simple been told about the danger of missing simple solutions.