Confounded COVID Vaccine Trial

Figure 1. COVID incidence in RCT (source: VRBPAC briefing document)

December 10, 2011. Today, the FDA Biological Products Advisory Committee (VRBPAC) met to discuss the request for emergency use authorization (EUA) of a COVID-19 messenger RNA vaccine from Pfizer, Inc. My guess is that the graph above will be critical to approval of the vaccine. One week after injection, the experimental and control (placebo) groups diverged. The control group (red squares) showed continued COVID infections while the experimental group that received the vaccine, showed very few new cases. Even though the graph appears to be powerful evidence of the vaccine's effectiveness, I have some questions about the study's methodology.

Figure 2. Questions for the FDA

I was unable to find answers to my questions in the documentation submitted to VRBPAC so I submitted my questions to the FDA (graphic above). My general concern is about confounding, that is, other explanations that could account for the results of the Randomized Control Trial (RCT).  

Figure 3. US COVID Daily Counts (source Healthdata.org)

My concerns are based mainly on the graph above that shows a plot of actual and projected Daily COVID case counts under four conditions: (1) Mandates for masks and social distancing easing (red), (2) Universal Masks (green), and (3) Rapid Vaccine Rollout (blue). The Current Projection is presented in purple. What catches my attention in the graph is that Rapid Vaccine Rollout is not much better than the current projection after four months (the farthest into the future the projections are being made). The most effective way to control daily COVID case counts is universal masking (green line vs red line).

My questions to the FDA are focused on the behavior of all subjects after vaccination. Assume that Figure 3 rather than Figure 1, was the result of an imaginary RCT. The experimental group (Rapid Vaccine Rollout) is now the blue line and there are three control groups: Current Projection (purple), Mandate Easing (read) and Universal masks (green). In this imaginary experiment, the vaccine would still be more effective than the control groups, but not by much compared to Current Projection and Universal masking. Look back at the Y-axis of Figure 1. The difference between experimental and control at Day 112 is about 0.02 (cumulative incidence), which looks a lot like the difference between Rapid Vaccine Rollout and Universal Masks in Figure 3.

In RCT, it is typically not possible to control what the subjects do after vaccination. John Yang, a PBS NewHour reporter who was a subject in the RCT, reports that he was told to go home and continue his routine, which involved staying home, social distancing and mask wearing. He knew very quickly that he was in the experimental group because he developed rather strong symptoms after vaccination. If his was a common experience, than blinding in the RCT was broken and may have influenced the behavior of the experimental subjects. Subjects maintain a diary where they list side-effects and events. What is not clear is whether subjects recorded social distancing and masking.

There are a few other issue I have with statistical analysis of the data: (1) Trials were conducted in multiple countries (Argentina, Brazil, South Africa and the United States) and multiple sites. The design is called a Multicenter Clinical Trial (MCT). The appropriate statistical model is a Hierarchical Linear Model (HLM) that allows for the analysis and control of differences across sites and countries. For example, mask use differs across countries: Argentina (90%), Brazil (60%), South Africa (70%) and the United States (70%) (source: Heathdata.org). The HLM controls these and other differences across centers. For example, imagine that all the COVID cases in the control group were from Brazil. Whether or not an HLM was used in the analysis of the Pfizer study is not made clear in the documentation. (2) Testing of the assumptions underlying the statistical model are not reported. The dependent measure is a risk ratio comparing experimental and control groups. Ratios are known not be be normally distributed and must be transformed prior to analysis. The effect of the transformation in normalizing the data should be tested.

If I learn anything more from listening to the ongoing  FDA VRBPAC hearing, I will report them as comments.