top of page
Writer's pictureJason G. Freund

All Models are Wrong, but Some are Useful

Updated: Aug 27

George E.P. Box was an English statistician probably best known for his time at the University of Wisconsin where he was Vilas Professor of Statistics. He married R.A. Fisher's daughter - those of you that have taken a statistics class know of Fisher - commonly referred to as the father of modern statistics. And those of you the least bit familiar with Wisconsin know the Vilas name. William Vilas was a politician, Civil War veteran, University of Wisconsin law school professor, and a host of other things. Vilas County, the Henry Vilas Zoo (named for his son), Vilas Hall on the University of Wisconsin campus, and a number of other places are named for him or his family. The George Box Medal is an award for European business and industry statisticians.

George Box, By DavidMCEddy at en.wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=14941622


George Box, is probably most famous for his paper, Science and Statistics in 1976 where he writes largely about his father-in-law and how we try to understand the world. He wrote,

Since all models are wrong, the scientist must be alert to what is importantly wrong. It is inappropriate to be concerned about mice when there are tigers abroad.

This quote from his 1976 paper, often simplified to "all models are wrong, but some are useful" evolved into later statements by Box and others. Of course, George Box was not the first to recognize that our models are wrong.

The common misconception is that models are useless and only scientists use them. That could not be further from the truth, we use models all the time in our daily lives. Models are simply representations of reality and models take many forms. Take maps as an example - they are never completely accurate and being overly precise would make a map more difficult to use. Or your drive to work or play. The speed limit is 70 on the interstate, how fast are you going? I am guessing you are not doing the speed limit, but how far above it are you willing to go? That is a model - you are weighing the risks and rewards. We use these sort of models all the time. Of course, this model is not that simple because there are many other inputs - how fast is everyone else driving? Am I near a known speed trap (Rosendale, I'm looking at you...)? Is that semi really trying to pass another semi at 70.1 mph? We use similar mental models all the time - they are our way of shaping our behavior, solving problems (how do I better understand a situation I have little, if any, experience with?), and it is how we get work done.

Above: Savage Flies take on the Close Carpet Fly, originated by George Close (my great uncle). A link to a better version - one I tied many years ago for Hans Weilenmann's website.


Models might be classified as physical - that map you use to find your favorite trout stream access points, the chemistry kits that helped you get through organic chemistry, the model cars or airplanes you made as a kid (and maybe still make), or a runway model. Our flies are a model of something else - meant to represent a specific insect species (i.e. the Close Carpet Fly is a Brown Drake (Ephemera simulans) imitation) or a general attractor - like Andrew Grillo's Hippie Stomper - that represents some unspecified "trout food" (for more, In Praise of Andrew Grillos' Hippie Stomper). Flies are examples of physical models, what fly we decide to tie on is more of a conceptual model. Conceptual models - like the more abstract models we keep in our heads about how fast we are going to travel down the interstate or mathematical models used in all parts of our lives.


What Makes for a Useful Model?


Models are judged by their accuracy, their transferability, and their simplicity and it is often said they can be only two of the three at once.

  • Accuracy - how well does the model represent reality?

  • Transferability - how effectively does the model work in other places and situations.

  • Simplicity - models with fewer parameters are preferable (parsimony).


What I find most interesting about models is how these characteristics fight against one another. Your model for how far above the speed limit you are willing to travel can be simple - you decide you are OK with going 9 above the limit. While simple, it is not necessarily transferable - certainly not to Rosendale! And to make it more accurate as to where you are, simplicity suffers. Or like a map, as we strive to make the map more accurate, we sacrifice readability (simplicity). But make a map too simple and we may miss important information (accuracy).

In science we have ways to measure how well models fit the data. Simple linear regression is commonly used to explain the relationship between a single explanatory (independent / X) and response (dependent / Y) variable. The resulting equation - Y = mX + b where m is the slope and b is the Y-intercept - explains this relationship mathematically. The slope tells us about the relationship and for how Y changes as a function of X and whether that relationship is direct (positive slope) or inverse (negative slope). And the R^2 value - the correlation coefficient or coefficient of determination - describes the strength of the relationship which is a measure of the accuracy of the model.

Quite often, simple linear regression is not sufficient to inform us about relationships in nature. As I wrote about in more detail in a post about how ecological questions are complex and need multivariate (multiple explanatory and response variables) solutions. We can make more complex mathematical models (candidate models) and weigh them against one another in a process referred to as model selection. There are a number of different criteria that can be used in model selection. On popular method is Akaike information criterion (AIC) which weighs how well a model fits the data (accuracy) but penalizes models that add more parameters - simplicity is rewarded. In AIC, we have a method to compare complex models to one another. The potential problem with AIC is that models may be selected that are not explainable ecologically. This is where professional judgement or limiting model selection to candidate models that are based on prior knowledge.

Transferability is maybe the most difficult of the model criteria to achieve and assess. Many conceptual models are built to be generalized, which tends to make them less accurate but more transferable. But here, the exceptions are often as informative as the general model. As an example, Vannote et al. (1980) River Continuum Concept, which I wrote about previously, is a conceptual model that explains how rivers change physically and biologically as they make their way downstream. This model was developed in mountain environments and has been applied universally. Because there are mathematical components - relationships between productivity and respiration - it allows researchers to test the model. It is not perfectly transferable - there are many exceptions - but as a general model, it provides a starting point to better understand streams which is why it is one of the most cited papers in stream ecology.

Essentially all scientific papers are models. We apply what we learn from others and see how well those ideas fit in our situation. In laboratory studies, we should get the same results because we can control the explanatory variables. The question with manipulative experiments is always how transferable are they to nature? With observational studies, we wonder how transferable results are to other situations in different places. That is, because we found a particular effect of Brown Trout on Brook Trout, would others find similar effects in streams that they sample?


Most typically, the reason that models are wrong is not because the model is wrong but that our inputs, our knowledge of the situation, are not sufficient to create a better model. That is a much more significant issue and one that is more difficult to deal with. However, like boats, all models are compromises.


Literature Cited / Reading List



Burnham, K.P. and Anderson, D.R. eds., 2002. Model selection and multimodel inference: a practical information-theoretic approach. New York, NY: Springer New York.


Chave, J., 2013. The problem of pattern and scale in ecology: what have we learned in 20 years?. Ecology Letters, 16, pp.4-16.


Levin, S.A., 1992. The problem of pattern and scale in ecology: the Robert H. MacArthur award lecture. Ecology, 73(6), pp.1943-1967.



Turner, M.G., 1989. Landscape ecology: the effect of pattern on process. Annual Review of Ecology and Systematics, 20(1), pp.171-197.


Wu, J. and Loucks, O.L., 1995. From balance of nature to hierarchical patch dynamics: a paradigm shift in ecology. The Quarterly Review of Biology, 70(4), pp.439-466.


164 views1 comment

Recent Posts

See All

1 Comment


I would love to see you write a blog on coevolution in the stream environment on riparian zone trees in the Coulee Region. A "model" on how the riparian zone trees influence the ecology of the Coulee Region stream environment. ----------------------- Literature cited/reading material/ Trout are Made of Trees by April Pulley Sayre.

Edited
Like
bottom of page