What does the research say about…expertise and problem-solving?

In this series, I read through some academic research, try to understand it, and think of what it might mean for the classroom. This is not a literature review or extensive analysis: just a classroom teacher, trying to learn more.

Problem-solving is one of those 21st century skills we are suppose to cultivate in our students, but while these skills are frequently promoted, people seldom dig into what they mean and look like.

To better understand what it means to be a skilled problem-solver, I’ll be looking at The Cambridge Handbook of Expertise and Expert Performance, specifically the section on expertise within professional domains. Of particular interest is problem-solving being domain-general or domain-specific, and implications for how we teach students to be effective problem solvers.

Background Information

Now, while I endeavoured to make the research accessible, I lean on concepts that may be unfamiliar to you. So, here is a brief tour of some terms that I use in my synopses and analysis.

Problem

First of all, what is a problem? It might seem like a basic word, but, as I tell my students, it is important to define your terms. Using the definition provided by Cognitive Psychology, 4th ed. (Goldstein, 2015)

A problem occurs when there is an obstacle between the present state and a goal and its not immediately obvious how to get around the obstacle (Duncker, 1945; Lovett, 2002). Thus, a problem, as defined by psychologists, is difficult, and the solution is not immediately obvious. (pg. 336) (emphasis in original)

This element of difficulty and ambiguity in a problem is why I excluded from my summary here the chapter on mathematical expertise, as it focused on individuals who are highly skilled at mental calculations. While these are incredible feats, those problems have one right answer, and can be completed by an Excel spreadsheet.

Prototype and Exemplar Theory

In prototype theory, our minds categorize things we encounter by comparing them to a prototype,which is a typical member of the category. In exemplar theory, categorization is done by comparing to multiple examples from the category.

Our minds make use of both prototypes and exemplars. We initially develop prototypes by averaging together exemplars, and as our understanding of a concept or category grows, we use exemplars to help us identify exceptions to the prototype.

In terms of the chapters discussed here, the distinction between them is not that important, as the basic idea is the same: we understand something in the context of what we already know.

Mental Models

When we become familiar with situations, we develop mental models of them. A model is a simplified representation of a real-world phenomena. With a mental model, we know the key features of a situation, and can make predictions.

Domain-General vs. Domain-Specific

Whether something is domain-specific or domain-general is ultimately a question of transfer: whether a given skill set can move easily between domains or disciplines or if it is specific to just one.

Driving vehicles is an example of domain-specificity. Governments license you for a specific vehicle type; being licensed to drive a car does not mean you can legally drive a motorbike, a bus with air brakes, or an airplane. This recognizes the limitation of transferring driving skills.

Reading is somewhat of an example of a domain-general skill. If you are a proficient reader, you can read a romance novel, a science book meant for a general audience, and a New York Times current events article, moving between the different forms with ease. However, if you are reading an academic paper in a particular domain, then having background knowledge in the discipline is important for comprehension. I struggled at times reading these chapters, particularly the one on software engineers, because I am unfamiliar both with research on expertise and these domains. Background knowledge is important for general reading skill as well. It is just hard to think of a truly domain-general skill.

The Research

Expertise in Medicine and Surgery (Norman, G., Eva, K., Brooks, L., & Hamstra, S., 2006)

Medicine is a complex field with complex problems. Practitioners not only need to prescribe an effective treatment, but also accurately diagnose, which is essentially problem definition. We will see this in other domains, how experts spend more time on defining the problem, but the importance is most apparent in the medical field. Medical practitioners also need to recall and apply formal and experiential knowledge and stay up-to-date with a changing field.

In the past, research on medical expertise viewed diagnosis as a general thinking skill, and that finding a general strategy would be key for training future practitioners. However, when researchers compared first-year students to experienced clinicians, they found both groups utilized similar strategies. The experts had higher diagnosis accuracy, not because of a general strategy they used, but because of their breadth and organization of knowledge.

Knowledge for medical practitioners can be broken down into three categories: causal knowledge (of basic science and biological mechanisms), analytical knowledge (of signs and symptoms of various conditions), and experiential knowledge (from prior cases).

In studies that have practitioners think-aloud through their analysis of cases, experts have greater diagnostic accuracy and more coherent explanations than novices, but make less use of causal knowledge in those explanations. On the surface level, it would appear that causal knowledge is not as important for medical diagnoses. However, on closer analysis, this assumption falls apart. Many misconceptions among practitioners can be traced back to shaky knowledge of basic science. As well, when faced with a difficult diagnosis, experts will rely more on causal knowledge. This is true for medical students as well. For straightforward cases, there is no difference in diagnosis ability between students who learn the causal explanation of signs and symptoms and those who learn just the symptoms; when the cases become difficult, those who have causal knowledge perform better.

Reduced reliance on basic science for experienced practitioners shows a shift to utilizing a different kind of knowledge: experience. Rather than carefully analyzing symptoms or thinking through causal mechanisms, experts in medicine solve the problem “rapidly, and unconsciously, recognizing its similarity to an already-solved problem” (344), or an exemplar stored in memory. The expert acquires, in long-term memory, a rich array of examples to compare the current case to. When unusual cases are presented, and there are no corresponding examples in memory, then the expert turns to causal and analytical knowledge.

There is a downside to this increased reliance on experiential knowledge. While it can be quick and accurate, it can also result in less flexible thinking. Experienced practitioners may be less open to competing diagnoses, and more likely to categorize cases into areas where they have the most experience.

Expertise and Transportation (Durson, F.T., & Dattel, A.R., 2006)

Transportation is another interesting area of expertise, first of all because so many of us have firsthand experience in the domain. Personally, I drive a semi-automatic motorbike to and from work here in Phnom Penh, and it is consistently terrifying. As well, decision-making and problem-solving in the transportation domain happen in a dynamic, changing environment, which means perceptual skills become more important as well. As an example of the latter, more experienced drivers have more effective strategies for scanning the road. This research covers road drivers and other forms of transportation, such as pilots.

One factor in the difference between novice and expert drivers is the management of cognitive resources. Some cognitive resources are freed up through automatization, such as shifting gears, so they take up less space in working memory. Automatization is learning a skill to the point where you don’t need to think about it anymore, such as holding a pencil. Managing cognitive resources also takes place at a conscious level, and involves knowing when and how to focus attention. In one study, participants were asked to complete another task while driving; experienced drivers were better at keeping their eye on the road while completing the task. Military pilots are taught how to make use of ‘free time’ during demanding flights.

Automatization and the development of vehicle control skills do not necessarily lead to better drivers. More important is what the freed cognitive resources are used for, such as hazard detection. More experienced drivers are better able to gather and interpret cues from the environment for hazard detection, and have greater knowledge of possible locations for potential threats.

Experienced operators are also better able to develop situational awareness and mental models. In one study, pilots listened to recordings of conversations between air traffic controllers and pilots; experienced pilots were better able to provide a plausible conceptualization of the discussion, likely through matching the dialogue to models from long-term memory. Experienced pilots are also better at quickly modifying their mental models of a situation, such as recognizing when weather conditions are deteriorating.

Finally, the mental models of pilots help them know what information they need in a given situation. This was tested in an interesting study, where pilots had menu items on a computer that contained categories of information about the scenario. More-experienced pilots searched fewer menu items. Less-experienced pilots not only searched more menu items, but also were more likely to repeat views; they seemed to develop their strategy as they went, while more-experienced pilots had their strategy in advance. To me, this seems to have implications for how novices and experts manage a Google search for information as well.

When faced with difficult situations, experts in transportation have more strategies overall, more abstract strategies that can be applied flexibly, and can think of strategies faster, compared to novices. They are also more likely to alternate between working on different parts of a problem, while novices will work through a problem sequentially.

What all this shows is the strong cognitive element of transportation. Vehicle control is a small component and learned much more quickly. The chapter authors argue that, with the advent of cheap personal computers, simulations can be used to train operators in detecting hazards, choosing strategies, and managing cognitive resources.

Expertise in Software Design (Sonnentag, S., Niessen, C., and Volmer, J., 2006)

Like for medicine, problem definition is very important in software design. Software designers need to determine what demands the program needs to meet, and they might not all be found in the client’s request. More experienced software designers spend more time determining program requirements, and will infer more requirements from the what the client wants. Experienced designers were also more likely to use a top-down approach in design, using the range of programming plans they hold in long-term memory, while novices or experienced designers facing an unusual request are more likely to work backwards from the goals.

Program comprehension is also an important part of the design process. Designers need to understand a program in order to meet the goals and to test and debug the software. Studies show that more experienced programmers understand their programs on a more abstract level. What is interesting is how these skilled programmers gain this comprehension:

Pennington (1987) compared highly and poorly performing professional programmers. When trying to understand a program, high performers showed a “cross-referencing strategy” characterized by systematic alterations between systematically studying the computer program; translating it to domain terms, and subsequently verifying domain terms back in program terms. In contrast, poorer performers exclusively focused on program terms or on domain terms without building connections between the two “worlds.” Thus, it seems that connecting various domains and relating them to each other is crucial for arriving at a comprehensive understanding of the program and the underlying problem. (378-379)

So, skilled programmers move between the abstract and the concrete to build program comprension and a mental model of the software.

This mental model helps them search for bugs and identify underlying problems. Experienced programmers found more errors and found errors more quickly. They worked more systematically and actively tested for problems.

The chapter talked about how experience also leads to better metacognition. I have a hunch about how metacognition may be somewhat domain-specific, so this part intrigued me. However, when reading through the sample of good metacognitive knowledge in action, I simply could not understand it. One of those moments where I understood each individual word, but not how they fit together. Can’t translate that research for you.

Finally, high performing programmers have good communication and cooperation skills. They spent more time educating their team about the task, and, during the design time, more time in consultation with team members. The differences in communication and cooperation skill levels is more apparent in ill-structured meetings, compared to highly structured meetings. Programming skill alone does not make a person a great programmer.

However, the research on the relationship between programming performance and interpersonal skills is still in its infancy:

It is not clear whether communication and cooperation competencies are a prerequisite of excellent software performance or a positive by-product of being an excellent software designer. Furthermore, third variables might cause the relationship between communication and cooperation competencies and excellent software performance (e.g. generalized problem-solving skills). (381)

Overall, the researchers conclude that this research on programming expertise holds implications for both hiring and training, focusing on identifying and developing mental models, domain-specific and metacognitive knowledge, and interpersonal skills.

Professional Judgements and “Naturalistic Decision Making” (Ross, K.G., Shafer, J.L., & Klein, G., 2006)

If any of these chapters would provide evidence for problem-solving as a domain-general skill, it would be the chapter that is not specific to any domain. Naturalistic decision making (NDM) research focuses on solving ill-defined, high stakes problems under time pressure, in dynamic environments with multiple players. Exciting, right? These circumstances would certainly challenge anyone’s problem solving skills.

First, there are characteristics of expertise that NDM researchers have decided are important. Some show the importance of domain-specific knowledge and experience, such as routines and declarative knowledge, but also perceptual skills, sense of typicality, and mental simulation, which would make use of the range of mental models the expert has within the domain. Other characteristics are more general strategies, like spending more time assessing the situation and managing uncertainty. Metacognition is also important, but I am not ready to categorize it as domain-general or domain-specific.

The chapter focuses on one model of NDM: Recognition-Primed Decision (RPD) Model. Under this model, when experts face a problem under time pressure, they do not use careful analysis of all the options, but instead rely on their experience. The situation is quickly compared to the prototypes the expert has in long-term memory until a match is found. With that prototype comes a cognitive package of what to expect from the situation, possible courses of action, and different things to look for. The goal is not to find what will work best, but simply what will work. If the prototype is not enough, then mental simulations are run to imagine how a particular solution will play out. This also relies on experience of different solutions and their impacts from previous cases.

This model has been supported in studies of various professions. In a study on decisions made by fire-fighting commanders, it was found that 80% of their decisions were recognition based. Among design engineers, who are not under the same time pressure, recognition is still the most common strategy used.

This research has been used to change decision-making processes in the US military. The previous process was more analytical, involving generating and evaluating different courses of action, which was cumbersome to use in the field; some staff have “reported anecdotally that there is one main COA [course of action] they are working on and two that are more like “straw men” just to satisfy the process” (410). Recognitional Planning Model (RPM) was developed, based on RPD, to better reflect how military personnel were actually making decisions.

The new model worked. Plans were made faster, of equal or better quality than plans made under the previous process, and were better adapted to situational demands. In the case of the military, having a formal analytical problem-solving structure hampered decision making.

There are implications of RPD for training, with more of a focus on building an experience bank and an array of mental models. The training suggested in the chapter focuses on working through scenarios with feedback, and is “based on the premise that expert knowledge is largely tacit knowledge and can be difficult for the expert to share when asked” (412). I wonder the extent to which that premise varies between different domains and disciplines.

Synthesis

So, what are the main take away from this? First, while there are areas of commonality between the domains in how experts approach problem-solving, that does not mean problem-solving is a general skill. If anything, those similarities just further stress how problem-solving is domain-specific.

Experts spend more time defining the problem, but their ability to pick out key features of the environment (or patient, or…) depends on the domain-specific mental models they hold in long-term memory; they know what to look for. When experts choose and implement a solution, they are likely recalling it from previous situations. All these strategies are born out of domain-specific knowledge and experience.

This makes me think of the practice of withholding information from students, so they can think of their own solutions to a real-world problem. If what makes experts highly skilled is not thinking of their own solutions but collecting scenarios and solutions, then when we withhold real world solutions so students can think of their own, are we doing them any favours? We are getting students to practice a problem-solving strategy that people use when they lack knowledge and experience in a discipline, rather than putting them on the path to expertise. A problem-solving strategy that is easy to learn, rather than spending time on the more substantial task of developing a base of knowledge.

Different types of knowledge are also important, which are developed in different ways, and the relative importance of different types of knowledge varies between the domains. Medicine relies more on book knowledge, such as causal knowledge of biological mechanisms, while transportation is more orientated around experiential knowledge, built through simulations and, well, experience.

The question for the classroom, on whether we spend more time on book knowledge through instruction or experiential knowledge through designing experiences, is best answered by “it depends.” It depends on the discipline. You wouldn’t train a doctor the same way as a software designer, so you shouldn’t teach history the same way you teach drama. Let the structure of the discipline guide choosing the best teaching methods. Of the four domains looked at here, which ones are analogous for the types of problem-solving students do in school?

The final point of commonality between the chapters is looking at knowledge organization. Researchers seem very interested in how expert knowledge is organized differently from novice knowledge. Some differences found include more ‘elegant’ organizational structures, more connections, and more abstraction of concepts.

The question is raised of whether we can teach knowledge organization, and there does not seem to be a clear answer yet. One study discussed in the medicine chapter suggested that knowledge organization is simply a byproduct of knowledge acquisition and is not something a teacher has direct influence over. I think teachers should strive to make connections for students, because it helps make stronger memories, and to guide them from the concrete to the abstract, because it makes the knowledge more flexible, and hopefully there will be further research on whether teachers can directly influence knowledge organization.

References

Durson, F.T., & Dattel, A.R. (2006). Expertise and Transportation. In Ericsson, K.A., Charness, N., Feltovich, P.J., and Hoffman, R.R. (Eds.), The Cambridge Handbook of Expertise and Expert Performance (pp. 355-371). Cambridge: Cambridge University Press.

Goldstein, E.B. (2015). Cognitive Psychology 4th ed. Stamford: Cengage Learning.

Norman, G., Eva, K., Brooks, L., & Hamstra, S. (2006). Expertise in Medicine and Surgery. In Ericsson, K.A., Charness, N., Feltovich, P.J., and Hoffman, R.R. (Eds.), The Cambridge Handbook of Expertise and Expert Performance (pp. 339-353). Cambridge: Cambridge University Press.

Ross, K.G., Shafer, J.L., & Klein, G. (2006). Professional Judgements and “Naturalistic Decision Making”. In Ericsson, K.A., Charness, N., Feltovich, P.J., and Hoffman, R.R. (Eds.), The Cambridge Handbook of Expertise and Expert Performance (pp. 403-419). Cambridge: Cambridge University Press.

Sonnentag, S., Niessen, C., and Volmer, J. (2006). Expertise in Software Design. In Ericsson, K.A., Charness, N., Feltovich, P.J., and Hoffman, R.R. (Eds.), The Cambridge Handbook of Expertise and Expert Performance (pp. 373-387). Cambridge: Cambridge University Press.

 

Advertisements

2 thoughts on “What does the research say about…expertise and problem-solving?

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s