Peeking, Tweaking and Cross-Validation

The classical Null Hypothesis Testing (NHT) paradigm (see my prior posts on NHT here) is focused almost exclusively on the critical experiment. The critical experiment, for example the gold-standard clinical trial, is set up to insure that a hypothesis is given a definitive test. Does administration of a drug actually produce a significant reduction in the target disease. Competing explanations are controlled through random sampling and random assignment. Subjects are chosen randomly from some population and are assigned randomly to treatment and control. There may actually be multiple control groups, some receiving placebos, some receiving competitor drugs and others receiving nothing. For this purpose, classical NHT is well suited, if not not very cost effective.

In the everyday world of science, however, classical NHT is easy to teach but hard to implement. Research, by its very definition, is mostly exploratory. Any interesting scientific question starts as a somewhat poorly formed idea. Concepts are a little unclear. Measurement of the concepts are open to question. Causal processes may be obscure and even unobservable. There needs to be a period of exploration, development and more casual testing. A lot of this period will involve exploratory data analysis.

Unfortunately, NHT doesn't really allow data exploration. Significance tests and associated probability levels are based on the assumption that you have not peeked at your data. The probability levels also assume that you have not estimated your model once, found it inadequate and gone on to tweak the model until it fits your sample data. In other words, NHT assumes that you are not involved in post-hoc analysis. Is there a solution to this dilemma?

In preparation for a research project I'm currently working on, I started re-reading the literature on Path Analysis. I haven't worked in this area since the 1980's, so I started back there and began reading forward. I ran into a 1983 article by Norman Cliff titled Some Cautions Concerning the Application of Causal Modeling Methods. In the article, a number of interesting general points are made that make the article seem quite current.

To the problem of peeking and tweaking, a strong recommendation is made to use cross-validation. In cross-validation, a large sample is drawn, larger than is needed for adequate statistical power. Part of the sample is then sub-sampled and used for exploratory model development. The remaining part of the sample is used to test the tweaked model. Since one has not peeked at this data, there is the possibility of fairly testing the validity of the model, that is, cross validating the model.

The project I am currently working on involves studying Globalization in Latin America. Multi-country analysis provides another type of cross-validation. In this case, the principal investigator has extensive expertise on a few countries, the greatest expertise being in Mexico. Since a lot of his understanding of the research problem is based on experience in Mexico, one approach would be to use Mexico as the model development laboratory. Another approach would be to take some other country, say Argentina, and develop there using ones understanding of Mexico to guide model development. This approach could possibly produce a somewhat more general model.

In a future post, I will talk about how cross-validation worked in this specific context. I will also talk about the issue of statistical power analysis and sample size determination if a cross-validation is being attempted. There are also a number of other interesting points in the Cliff(1983) paper, particularly on causality, that would be worth discussion.

How Complicated Models Broke the NHT Paradigm

In a prior post (here) I pointed out the weaknesses of the Null Hypothesis Testing (NHT) paradigm using a simple data set on automobile mileage (here). What I demonstrated was that if you arrive at the end of the NHT process (described here) with an insignificant result, you can always try to increase the sample size (if this is possible) until you can detect as small a difference as you observed. The ability to produce significant results based on manipulation of effect size and sample size should make everyone a little uneasy about the body of scientific work that has been generated under the NHT paradigm. In this post, I will return to the beginning of the NHT process and ask what types of theories generate the research hypothesis on which NHT is based. The conclusion here is that if the theories always generate "injection models," then all the weaknesses of NHT apply. However, if the theories generate multiple models, rather than research and null hypotheses, there is a way out of the NHT dead end.

At first, it might seem that there could not possibly be a uniform class of theories that generate research hypotheses given the wide breadth and depth of scientific theories in current use. If we try to classify the type of theories most used in classical NHT by the types of models involved, however, there is a general class of models called "injection models" that are well suited to the NHT paradigm.

By a model I mean a structural equation model and its associated directed graph. For example, the model underlying the research hypothesis about automobile miles per gallon (MPG) is displayed in the directed path above. The independent variable we focused on was the type of transmission, but obviously there are a lot of other factors that go into the determination of MPG. These types of models are always easily described by a general linear model and can often be tested by simple linear regression. They are all "injection models" because the effects of the independent variables are "injected" into the dependent variable. The null hypothesis is always that there is no injection.

There is a problem, however, with the causal implications of regression models expressed in mathematical terms. The first equation above is the standard regression equation. If y = MPG and x = [0,1] depending on whether we do or do not have an automatic transmission, then we have an injection model for automobile mileage. Mathematical objects, however, are symmetric and nothing prevents us from re-expressing the model to predict the type of transmission from MPG. While predicting the type of transmission from MPG might be an interesting exercise, we know a priori that mileage doesn't "cause" a car to have a particular type of transmission. We need something beyond mathematics to describe causal relationships.

Causal diagrams make the direction of causality clear from the direction of the arrow connecting two variables. The importance of path diagrams for encoding causal information was discovered by geneticist Sewell Wright in the 1920's. Wright retired to the University of Wisconsin in 1955 where he had an influence on economists and sociologists.

With path diagrams it was relatively easy to develop complicated causal models that do not lend themselves to re-statement as a simple null hypothesis. For example, in the path diagram above (from Pedhazur and Kerlinger, 1982 and Karl Wuensch, here), socioeconomic status (SES) is taken as an exogenous variable that influences IQ, need achievement (nAch) and resulting grade point average (GPA). The variables that are the result of causal arrows (IQ, nAch and GPA) are considered endogenous. It is important to this theory of grade point determination to note that IQ and nAch are mediating influences on SES. That is, you will find lower SES students with high GPAs as a result of their IQ and their need for achievement. You will also find some high SES students with neither high IQ nor high need achievement, but they will probably still have better GPAs than low SES students.

In the NHT paradigm, path models have to be converted into injection models as in the graphic above. In the "testable" NHT model, SES, IQ and nAch are all considered exogenous variables determining grade point. The simple null hypothesis would be that none of these factors have a statistically significant impact on GPA. The idea of mediating or intervening variables (IQ and nAch mediating the influence of SES on grade point) is dropped. The reformulated model above, however, is still not enough for hard-core experimentalists.

For the hard-core experimentalists, Pedhzur and Kerlinger have to give up any ideas about drawing causal links from SES and IQ to GPA because these variables cannot be experimentally manipulated. It would not be possible to conduct an Eliza Doolittle experiment where subjects were randomly assigned to have their SES manipulated so changes in GPA could be observed. It would also not be practical to manipulate IQ. Since it might be possible to manipulate nAch (annotated as do_nAch using Judea Pearl's terminology) what the experimentalists require is that all extraneous factors be eliminated through random assignment. The fact that we are no longer testing Pedhazur and Kerlinger's theory is not an issue because the theory is not really testable within the NHT paradigm.

Unfortunately, there are an infinite number of important research questions where experimentation is neither possible nor desirable, global climate change being just one example (we simply cannot manipulate the world system). Instead, we are forced to study intact populations or intact systems using models. Path diagrams provide a useful and general (that is, nonparametric) way for describing causality based on an understanding of the system being studied. Contending models always exist in various stages of testing. NHT is well suited for handling the critical experiment where a hypothesis can be directly tested. Even here we never really test the research hypothesis, only the null alternative. The focus of the NHT statistician is on hypotheses while the focus of scientists is on models.

Returning to Pedhazur and Kerlinger's path model, not only did the absence of experimental design generate howls of protest from the experimental and statistical scolds (here and here, basically "correlation cannot prove causation") but the approach to estimating path diagrams was also held up for criticism because technical statistical requirements (the assumptions of classical normal distribution theory) were being violated. I'll get into these issues in a future post. For now, what is important to understand is that multiple, competing causal models (rather than hypotheses or statistical distributions) are the important parts of science that always exist in various states of confirmation and acceptance. Statistical technique needs to be able to test models in addition to merely testing hypotheses. Multi-model development, testing and selection provides one way out of the NHT dead end.