How to approach baseline imbalance in cross-randomised studies?

Höper and colleagues have recently published the results of their study on the influence of transcutaneous auricular vagus nerve stimulation (tVNS) on prefrontal cortex oxygenation and autonomic nervous system (ANS) activity in adolescents. In a within-subject-cross-randomised and sham-controlled study, they compared 15-minute electrical stimulation at the concha (tVNS) with stimulation at the ear lobe (sham) and measured heart rate (HR) and heart rate variability (HRV) as indexes of ANS activity. They found an increase in HRV and a decrease in HR during tVNS compared to sham and proposed a therapeutic potential of tVNS for disorders associated with altered ANS functioning (). However, we believe there could be a potential bias in the intervention effect estimates due to baseline imbalance. We appreciate a revisiting of the data and additional statistical analysis for a better understanding of the results and methodological challenges in the field.

Höper and colleagues have used a cross-randomised design and included a baseline measurement of the response at the start of each block, also known as the period-dependent baseline (). Despite the careful design and cross-randomisation, there was already a significant difference in the period-dependent baseline measures of HR and HRV between the tVNS and sham, where no stimulation was applied (see Fig. 4 in the paper). The size of this difference was almost the same as the size of the difference between the two conditions at follow-ups during 5 min of stimulation. To rule out baseline influence, the authors have excluded the baseline time-point from the statistical model, which was included first as another repeated-measure of the outcome. Removing the baseline measure of the response variable (i.e., baseline as dependent variable) was indeed the right statistical decision, as we do not expect an intervention effect when it is not yet present, however, this exclusion does not minimise its potential confounding (imbalance) effect. On the other hand, including the subject as a random factor in the model controls for non-independence from nested structure (repeated measures) in the data; however, does not control the period-dependent baseline imbalance. As the baseline measure of the ANS response could be time-dependent, the imbalance might happen by chance during any of the periods or systematically related to the conditions. In this situation, the recommended methodologies to minimise the bias and improve precision are to include baseline as a covariate (i.e., baseline as an independent variable) or to use change scores (not common though), i.e., to use the baseline information and not exclude it (; ).

The reason for baseline imbalance in this cross-randomised study is not clear but is very insightful to know for (1) choosing an appropriate statistical analysis to minimise the bias and (2) understanding the methodological challenges in this field of research. Bias can be due to chance particularly when a small sample size is used and/or can be systematic due to order, carry-over, and condition by order interactions (). Cross-randomisation design does not guarantee to prevent such biases. Importantly, a contextual, environmental, and experimenter factor associated with any of the conditions (tVNS or sham), and with a direct effect on the response variables, can generate a systematic bias that cannot be fully controlled by cross-randomisation. To realize what is the source of baseline imbalance in this study we need to separately look at the period-dependent baselines, across the study period and not only aggregated data, to identify the period(s) with imbalance. For example, a lower HR during tVNS (vs. sham) at both baseline 1 and baseline 2 may indicate a systematic error due to an external factor co-occurring with tVNS or sham, which can be still present during the follow-up recordings, and influences the outcomes.

A recent meta-analysis on the influence of tVNS on HRV provides strong evidence that acute tVNS does not alter HRV compared to sham (). Therefore, when new evidence emerges with different results, it is important to find the potential reasons behind such differences to identify parameters that drive the effects (e.g., stimulation location, duration, frequency, intensity) and to improve methodologies in this field. We appreciate it if the authors could provide additional data visualisation and analysis by controlling for period-dependent baseline measures as a covariate in the model or using other recommended methods (; ). Also, to prevent an increase in the risk of type I error due to multiple comparisons, a method of p-value adjustment might be necessary.