Advertisement

By trying different approaches (and transparent reporting)! Response to: How to approach baseline imbalance in cross-randomised studies?

  • Julian Koenig
    Correspondence
    Corresponding author.
    Affiliations
    University of Cologne, Faculty of Medicine and University Hospital Cologne, Department of Child and Adolescent Psychiatry, Psychosomatics and Psychotherapy, Cologne, Germany
    Search for articles by this author
  • Michael Kaess
    Affiliations
    University of Bern, University Hospital of Child and Adolescent Psychiatry and Psychotherapy, Bern, Switzerland

    University Hospital Heidelberg, Center for Psychosocial Medicine, Department of Child and Adolescent Psychiatry, Heidelberg, Germany
    Search for articles by this author
Published:December 05, 2022DOI:https://doi.org/10.1016/j.autneu.2022.103055
      Gholamrezaei and Jafari raised an issue in their Letter to the Editor (
      • Gholamrezaei A.
      • Jafari H.
      How to approach baseline imbalance in cross-randomised studies?.
      ) and simultaneously on Twitter (), addressing the statistical analyses of our paper (
      • Höper S.
      • Kaess M.
      • Koenig J.
      Prefrontal cortex oxygenation and autonomic nervous system activity under transcutaneous auricular vagus nerve stimulation in adolescents.
      ), recently published in Autonomic Neuroscience: Basic and Clinical. The authors are concerned that our results carry potential bias in the intervention effect estimate due to baseline imbalance in measurements of heart rate (HR) and heart rate variability (HRV). Referring to Fig. 4 of our paper (
      • Höper S.
      • Kaess M.
      • Koenig J.
      Prefrontal cortex oxygenation and autonomic nervous system activity under transcutaneous auricular vagus nerve stimulation in adolescents.
      ), the authors state “there was already a significant difference in the period-dependent baseline measures of HR and HRV”. We have to clarify, that neither the difference in baseline HR nor baseline HRV between tVNS and sham is statistically significant (neither using parametric nor nonparametric tests). A better depiction of the individual raw data is now provided in Fig. 1. The authors state, that “[t]he size of this difference was almost the same as the size of the difference between the two conditions at follow-ups during 5 min of stimulation”. Precisely, the mean difference in HR between tVNS and sham at baseline is 3.17 beats per minute (BPM) [95%CI: −2.38; 8.72] and at 5 min of stimulation is 3.62 BPM [95%CI: −1.43; 8.66]. The mean difference in HRV at baseline is −8.78 milliseconds (ms) [−18.93; 1.37] and at 5 min of stimulation is −9.06 ms [−19.83; 1.70]. We agree, that these differences are almost the same (but slightly larger at 5 min of stimulation).
      Fig. 1
      Fig. 1Individual subject data by stimulation condition (sham versus tVNS) and time (baseline, 5 min, 10 min, and 15 min of stimulation); p-values refer to differences between baseline segments between conditions based on pairwise t-tests as implemented in the raincloud shiny app.
      While the authors agree with our decision to take out the baseline measurement in sensitivity analyses, they suggest to “include baseline as a covariate”, following respective suggestions discussed elsewhere and frequently applied in clinical studies (with respective long-term follow-up and repeated assessments of outcomes) (
      • Kenward M.G.
      • Roger J.H.
      The use of baseline covariates in crossover studies.
      ;
      • Metcalfe C.
      The analysis of cross-over trials with baseline measurements.
      ). We are not aware of such approach in analysing physiological data, when greater inter-correlation and temporal dependency between repeated assessments should be assumed (i.e., given the continuous nature of the physiological signal that is only segmented for analytical reasons). Further, such approach interferes with our statistical model including the subject ID as random effect. Anyway, we followed the respective suggestion, reproducing our sensitivity analyses, but this time considering baseline HR and HRV respectively as covariate. First, models were specified as mixed effect models as in the original report (baseline HR/HRV as fixed factor). For HR (χ2(6) = 126.53; p < .0001) and HRV (χ2(6) = 205.08; p < .0001), overall model fit improved. Models for HR (coef. = −3.62; p < .0001) and HRV (coef. = 9.064; p < .0001) still showed a significant main effect of STIMULATION. Baseline HR (coef. = 0.68; p < .0001) and baseline HRV (coef. = 0.87; p < .0001), were significant covariates. To derive posthoc contrast adjusted for baseline HR/HRV, in violation of the analytical plan, simple regression models were used (without the random effect, but accounting for individual baseline HR/HRV). Models again showed significant main effects of STIMULATION on HR (F(6) = 10.35; p = .002) and HRV (F(6) = 15.33; p = .0001). For HR, a significant effect of STIMULATON in contrasts adjusted for baseline HR was only present at 5 min (F(1) = 6.51; p = .012) of stimulation (10 min: F(1) = 3.58; p = .060; 15 min: F(6) = 1.27; p = .261). Concerning HRV, significant contrasts of STIMULATION adjusted for baseline HRV, were also only present at 5 min (F(1) = 10.34; p = .002) of stimulation (10 min: F(1) = 3.84; p = .052; 15 min: F(1) = 2.57; p = .111). To conclude, when applying the suggested method of baseline correction, we find significant effects of tVNS on HR and HRV. However, unlike suggested in the original report (when modelling the random effect), the effects of tVNS seen early in the course of stimulation seem to fade out faster than previously suggested. However, to enable adjustment of the respective posthoc contrasts we had to apply a different statistical approach, neglecting the random effect of the individual subject (that is confounded with the individual baseline HR/HRV).
      Although we cannot provide a definite answer to the question “how to [best] approach baseline imbalance in cross-randomised studies?”, we hope we were able to satisfy the authors request for additional data visualization and analyses by following their detailed suggestions. We conclude that although there are baseline differences in HR and HRV - as transparently reported in our paper - these were not statistically significant. Further, when controlling for baseline HR and HRV respectively, the main effect of tVNS remained significant. When adjusting for baseline HR/HRV as suggested using linear regression, effects of tVNS on HR/HRV that appear early in the course of stimulation seemed to fade out quite fast. Importantly – and unlike suggested by Gholamrezaei and Jafari - our data (as originally published (
      • Koenig J.
      • Parzer P.
      • Haigis N.
      • Liebemann J.
      • Jung T.
      • Resch F.
      • Kaess M.
      Effects of acute transcutaneous vagus nerve stimulation on emotion recognition in adolescent depression.
      )) were already included in the cited meta-analysis (
      • Wolf V.
      • Kühnel A.
      • Teckentrup V.
      • Koenig J.
      • Kroemer N.B.
      Does transcutaneous auricular vagus nerve stimulation affect vagally mediated heart rate variability? A living and interactive bayesian meta-analysis.
      ). Although we can only speculate on why we found significant effects (of very small size) of tVNS on HR and HRV and others did not, it is of interest to note that our sample is the only underage sample included in the living meta-analysis yet. Age related differences in the capacity of the autonomic nervous system to adapt to experimental stimulation might account for this effect. Still, our results need further replication. In particular, the present exchange with Gholamrezaei and Jafari illustrates that analytical decision may alter findings of small effects. Motivated by the authors' interest in the specific effects of tVNS on HR/HRV, we will look into more advanced ways to analyse our data, addressing individual trajectories in HR/HR under tVNS using more fine-grained approaches (i.e., shorter segments, sliding windows) and more advanced models to quantify temporal effects and dependencies between signals.

      References

        • Gholamrezaei A.
        Tweet by Ali Gholamrezaei.
        (June 21)
        • Gholamrezaei A.
        • Jafari H.
        How to approach baseline imbalance in cross-randomised studies?.
        Auton. Neurosci. 2022;
        • Höper S.
        • Kaess M.
        • Koenig J.
        Prefrontal cortex oxygenation and autonomic nervous system activity under transcutaneous auricular vagus nerve stimulation in adolescents.
        Auton. Neurosci. 2022; 241103008https://doi.org/10.1016/j.autneu.2022.103008
        • Kenward M.G.
        • Roger J.H.
        The use of baseline covariates in crossover studies.
        Biostatistics. 2010; 11: 1-17https://doi.org/10.1093/biostatistics/kxp046
        • Koenig J.
        • Parzer P.
        • Haigis N.
        • Liebemann J.
        • Jung T.
        • Resch F.
        • Kaess M.
        Effects of acute transcutaneous vagus nerve stimulation on emotion recognition in adolescent depression.
        Psychol. Med. 2021; 51: 511-520https://doi.org/10.1017/S0033291719003490
        • Metcalfe C.
        The analysis of cross-over trials with baseline measurements.
        Stat. Med. 2010; 29: 3211-3218https://doi.org/10.1002/sim.3998
        • Wolf V.
        • Kühnel A.
        • Teckentrup V.
        • Koenig J.
        • Kroemer N.B.
        Does transcutaneous auricular vagus nerve stimulation affect vagally mediated heart rate variability? A living and interactive bayesian meta-analysis.
        Psychophysiology. 2021; 58e13933https://doi.org/10.1111/psyp.13933