Minimal reporting improvement after peer review in reports of COVID-19 prediction models: systematic review

OBJECTIVES

To assess improvement in the completeness of reporting coronavirus (COVID-19) prediction models after the peer review process.

STUDY DESIGN AND SETTING

Studies included in a living systematic review of COVID-19 prediction models, with both preprint and peer-reviewed published versions available, were assessed. The primary outcome was the change in percentage adherence to the transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) reporting guidelines between pre-print and published manuscripts.

RESULTS

Nineteen studies were identified including seven (37%) model development studies, two external validations of existing models (11%), and 10 (53%) papers reporting on both development and external validation of the same model. Median percentage adherence among preprint versions was 33% (min-max: 10 to 68%). The percentage adherence of TRIPOD components increased from preprint to publication in 11/19 studies (58%), with adherence unchanged in the remaining eight studies. The median change in adherence was just 3 percentage points (pp, min-max: 0-14 pp) across all studies. No association was observed between the change in percentage adherence and preprint score, journal impact factor, or time between journal submission and acceptance.

CONCLUSIONS

The preprint reporting quality of COVID-19 prediction modeling studies is poor and did not improve much after peer review, suggesting peer review had a trivial effect on the completeness of reporting during the pandemic.

External validation of a prediction model for estimating fat mass in children and adolescents in 19 countries: individual participant data meta-analysis

OBJECTIVE

To evaluate the performance of a UK based prediction model for estimating fat-free mass (and indirectly fat mass) in children and adolescents in non-UK settings.

DESIGN

Individual participant data meta-analysis.

SETTING

19 countries.

PARTICIPANTS

5693 children and adolescents (49.7% boys) aged 4 to 15 years with complete data on the predictors included in the UK based model (weight, height, age, sex, and ethnicity) and on the independently assessed outcome measure (fat-free mass determined by deuterium dilution assessment).

MAIN OUTCOME MEASURES

The outcome of the UK based prediction model was natural log transformed fat-free mass (lnFFM). Predictive performance statistics of R², calibration slope, calibration-in-the-large, and root mean square error were assessed in each of the 19 countries and then pooled through random effects meta-analysis. Calibration plots were also derived for each country, including flexible calibration curves.

RESULTS

The model showed good predictive ability in non-UK populations of children and adolescents, providing R² values of >75% in all countries and >90% in 11 of the 19 countries, and with good calibration (ie, agreement) of observed and predicted values. Root mean square error values (on fat-free mass scale) were <4 kg in 17 of the 19 settings. Pooled values (95% confidence intervals) of R², calibration slope, and calibration-in-the-large were 88.7% (85.9% to 91.4%), 0.98 (0.97 to 1.00), and 0.01 (-0.02 to 0.04), respectively. Heterogeneity was evident in the R² and calibration-in-the-large values across settings, but not in the calibration slope. Model performance did not vary markedly between boys and girls, age, ethnicity, and national income groups. To further improve the accuracy of the predictions, the model equation was recalibrated for the intercept in each setting so that country specific equations are available for future use.

CONCLUSION

The UK based prediction model, which is based on readily available measures, provides predictions of childhood fat-free mass, and hence fat mass, in a range of non-UK settings that explain a large proportion of the variability in observed fat-free mass, and exhibit good calibration performance, especially after recalibration of the intercept for each population. The model demonstrates good generalisability in both low-middle income and high income populations of healthy children and adolescents aged 4-15 years.

Body mass index trajectories in childhood and incidence rates of type 2 diabetes and coronary heart disease in adulthood: A cohort study

AIMS

We examined associations between five body mass index (BMI) trajectories from ages 6-15 years and register-based adult-onset type 2 diabetes mellitus (T2D) and coronary heart disease (CHD) with and without adjustment for adult BMI.

METHODS

Child and adult BMI came from two Danish cohorts and 13,205 and 13,438 individuals were included in T2D and CHD analyses, respectively. Trajectories were estimated by latent class modelling. Incidence rate ratios (IRRs) were estimated with Poisson regression.

RESULTS

In models without adult BMI, compared to the lowest trajectory, among men the T2D IRRs were 0.92 (95 %CI:0.77-1.09) for the second lowest trajectory and 1.51 (95 %CI:0.71-3.20) for the highest trajectory. The corresponding IRRs in women were 0.92 (95 %CI:0.74-1.16) and 3.58 (95 %CI:2.30-5.57). In models including adult BMI, compared to the lowest trajectory, T2D IRRs in men were 0.57 (95 %CI:0.47-0.68) for the second lowest trajectory and 0.26 (95 %CI:0.12-0.56) for the highest trajectory. The corresponding IRRs in women were 0.60 (95 %CI:0.48-0.75) and 0.59 (95 %CI:0.36-0.96). The associations were similar in direction, but not statistically significant, for CHD.

CONCLUSIONS

Incidence rates of adult-onset T2D were greater for a high child BMI trajectory than a low child BMI trajectory, but not in models that included adult BMI.

A new prognostic model for predicting 30-day mortality in acute oncology patients

INTRODUCTION

Acute oncology services (AOS) provide rapid review and expedited pathways for referral to specialist care for cancer patients. Blood tests may support AOS in providing estimates of prognosis. We aimed to develop and validate a prognostic model of 30-day mortality based on routine blood markers to inform an AOS decision to actively treat or palliate patients.

METHODS AND MATERIALS

Using clinical data from 752 AOS referrals, multivariable logistic regression analysis was conducted to develop a 30-day mortality prognostic model. Internal validation and then internal-external cross-validation were used to examine overfitting and generalizability of the model's predictive performance.

RESULTS

Urea, alkaline phosphatase, albumin and neutrophils were the strongest predictors of outcome. The model separated patients into distinct prognostic groups from the cross-validation (C Statistic: 0.70; 95% CI: 0.64-0.76). Admission year was included as a predictor in the model to improve the model calibration.

CONCLUSION

The developed prediction model was able to classify patients into distinct prognostic risk groups, which is clinically useful for delivering an evidence-based AOS. Collation of data from other AOS centers would allow for the development of a more generalizable prognostic model.

Is child weight status correctly reported to parents? Cross-sectional analysis of National Child Measurement Programme data using ethnic-specific BMI adjustments

Background

BMI underestimates and overestimates body fat in children from South Asian and Black ethnic groups, respectively.

Methods

We used cross-sectional NCMP data (2015–17) for 38 270 children in three inner-London local authorities: City & Hackney, Newham and Tower Hamlets (41% South Asian, 18.8% Black): 20 439 4–5 year-olds (48.9% girls) and 17 831 10–11 year-olds (49.1% girls). We estimated the proportion of parents who would have received different information about their child’s weight status, and the area-level prevalence of obesity—defined as ≥98th centile—had ethnic-specific BMI adjustments been employed in the English National Child Measurement Programme (NCMP).

Results

Had ethnic-specific adjustment been employed, 19.7% (3112/15 830) of parents of children from South Asian backgrounds would have been informed that their child was in a heavier weight category, and 19.1% (1381/7217) of parents of children from Black backgrounds would have been informed that their child was in a lighter weight category. Ethnic-specific adjustment increased obesity prevalence from 7.9% (95% CI: 7.6, 8.3) to 9.1% (8.7, 9.5) amongst 4–5 year-olds and from 17.5% (16.9, 18.1) to 18.8% (18.2, 19.4) amongst 10–11 year-olds.

Conclusions

Ethnic-specific adjustment in the NCMP would ensure equitable categorization of weight status, provide correct information to parents and support local service provision for families.

Association of Childhood Fat Mass and Weight With Adult-Onset Type 2 Diabetes in Denmark

IMPORTANCE

Childhood obesity, defined by cutoffs based on the weight-based marker of body mass index, is associated with adult type 2 diabetes (T2D) risk. Whether childhood fat mass (FM) is the driver of these associations is currently unknown.

OBJECTIVE

To quantify and compare height-independent associations between childhood FM and weight with adult T2D risk in a historic Danish cohort.

DESIGN, SETTING, AND PARTICIPANTS

This population-based retrospective cohort study included schoolchildren from The Copenhagen School Health Records Register born between January 1930 and December 1985 with follow-up to adulthood through December 31, 2015. Analyses were based on 269 913 schoolchildren aged 10 years with 21 896 established adult T2D cases and 261 192 children aged 13 years with 21 530 established adult T2D cases for whom childhood height and weight measurements, as well as predicted FM, were available. Statistical analyses were performed between April 2019 to August 2020.

EXPOSURES

Childhood FM and weight at ages 10 and 13 years.

MAIN OUTCOMES AND MEASURES

Diagnoses of T2D were established by linkage to national disease registers for adults aged at least 30 years. Sex-specific Cox regression quantified associations, adjusted for childhood height, which were evaluated within 5 birth-cohort groups. Group-specific results were pooled using random-effects meta-analyses accounting for heterogeneity across group-specific associations.

RESULTS

This cohort study analyzed data from 269 913 children aged 10 years (135 940 boys [50.4%]) with 21 896 established adult T2D cases and 261 192 children aged 13 years (131 025 boys [50.2%]) with 21 530 established adult T2D cases. After adjusting for childhood height, increases in FM and weight (per kilogram) among boys aged 10 years were associated with elevated T2D risks at age 50 years of 12% (hazard ratio [HR], 1.12; 95% CI, 1.10-1.14) and 7% (HR, 1.07; 95% CI, 1.05-1.09), respectively, and among girls aged 10 years of 15% (HR, 1.15; 95% CI, 1.13-1.17) and 10% (HR, 1.10; 95% CI, 1.08-1.11), respectively. Among children aged 13 years, increases in FM and weight (per kilogram) were associated with increased T2D risks at age 50 years of 10% (HR, 1.10; 95% CI, 1.09-1.10) and 6% (HR, 1.06; 95% CI, 1.05-1.07) for boys, respectively, and of 10% (HR, 1.10; 95% CI, 1.10-1.11) and 7% (HR, 1.07; 95% CI, 1.06-1.08), respectively, for girls.

CONCLUSIONS AND RELEVANCE

This cohort study found that a 1-kg increase in childhood FM was more strongly associated with increased adult T2D risk than a 1-kg increase in weight, independent of childhood height. Information on FM, rather than weight-based measures, focuses on a modifiable component of weight that may be associated with adult T2D risk. These findings support the assessment of childhood FM in adiposity surveillance initiatives in an effort to reduce long-term T2D risk.

Minimum sample size for external validation of a clinical prediction model with a continuous outcome

Clinical prediction models provide individualized outcome predictions to inform patient counseling and clinical decision making. External validation is the process of examining a prediction model's performance in data independent to that used for model development. Current external validation studies often suffer from small sample sizes, and subsequently imprecise estimates of a model's predictive performance. To address this, we propose how to determine the minimum sample size needed for external validation of a clinical prediction model with a continuous outcome. Four criteria are proposed, that target precise estimates of (i) R2 (the proportion of variance explained), (ii) calibration-in-the-large (agreement between predicted and observed outcome values on average), (iii) calibration slope (agreement between predicted and observed values across the range of predicted values), and (iv) the variance of observed outcome values. Closed-form sample size solutions are derived for each criterion, which require the user to specify anticipated values of the model's performance (in particular R2 ) and the outcome variance in the external validation dataset. A sensible starting point is to base values on those for the model development study, as obtained from the publication or study authors. The largest sample size required to meet all four criteria is the recommended minimum sample size needed in the external validation dataset. The calculations can also be applied to estimate expected precision when an existing dataset with a fixed sample size is available, to help gauge if it is adequate. We illustrate the proposed methods on a case-study predicting fat-free mass in children.

Quantifying childhood fat mass: comparison of a novel height-and-weight-based prediction approach with DXA and bioelectrical impedance

Accurate assessment of childhood adiposity is important both for individuals and populations. We compared fat mass (FM) predictions from a novel prediction model based on height, weight and demographic factors (height–weight equation) with FM from bioelectrical impedance (BIA) and dual-energy X-ray absorptiometry (DXA), using the deuterium dilution method as a reference standard. FM data from all four methods were available for 174 ALSPAC Study participants, seen 2002–2003, aged 11–12-years. FM predictions from the three approaches were compared to the reference standard using; R², calibration (slope and intercept) and root mean square error (RMSE). R² values were high from ‘height–weight equation’ (90%) but lower than from DXA (95%) and BIA (91%). Whilst calibration intercepts from all three approaches were close to the ideal of 0, the calibration slope from the ‘height–weight equation’ (slope = 1.02) was closer to the ideal of 1 than DXA (slope = 0.88) and BIA (slope = 0.87) assessments. The ‘height–weight equation’ provided more accurate individual predictions with a smaller RMSE value (2.6 kg) than BIA (3.1 kg) or DXA (3.4 kg). Predictions from the ‘height–weight equation’ were at least as accurate as DXA and BIA and were based on simpler measurements and open-source equation, emphasising its potential for both individual and population-level FM assessments.

Development and validation of a prediction model for fat mass in children and adolescents: meta-analysis using individual participant data

OBJECTIVES

To develop and validate a prediction model for fat mass in children aged 4-15 years using routinely available risk factors of height, weight, and demographic information without the need for more complex forms of assessment.

DESIGN

Individual participant data meta-analysis.

SETTING

Four population based cross sectional studies and a fifth study for external validation, United Kingdom.

PARTICIPANTS

A pooled derivation dataset (four studies) of 2375 children and an external validation dataset of 176 children with complete data on anthropometric measurements and deuterium dilution assessments of fat mass.

MAIN OUTCOME MEASURE

Multivariable linear regression analysis, using backwards selection for inclusion of predictor variables and allowing non-linear relations, was used to develop a prediction model for fat-free mass (and subsequently fat mass by subtracting resulting estimates from weight) based on the four studies. Internal validation and then internal-external cross validation were used to examine overfitting and generalisability of the model's predictive performance within the four development studies; external validation followed using the fifth dataset.

RESULTS

Model derivation was based on a multi-ethnic population of 2375 children (47.8% boys, n=1136) aged 4-15 years. The final model containing predictor variables of height, weight, age, sex, and ethnicity had extremely high predictive ability (optimism adjusted R²: 94.8%, 95% confidence interval 94.4% to 95.2%) with excellent calibration of observed and predicted values. The internal validation showed minimal overfitting and good model generalisability, with excellent calibration and predictive performance. External validation in 176 children aged 11-12 years showed promising generalisability of the model (R²: 90.0%, 95% confidence interval 87.2% to 92.8%) with good calibration of observed and predicted fat mass (slope: 1.02, 95% confidence interval 0.97 to 1.07). The mean difference between observed and predicted fat mass was -1.29 kg (95% confidence interval -1.62 to -0.96 kg).

CONCLUSION

The developed model accurately predicted levels of fat mass in children aged 4-15 years. The prediction model is based on simple anthropometric measures without the need for more complex forms of assessment and could improve the accuracy of assessments for body fatness in children (compared with those provided by body mass index) for effective surveillance, prevention, and management of clinical and public health obesity.

Exploring the use of adjusted body mass index thresholds based on equivalent insulin resistance for defining overweight and obesity in UK South Asian children

Background

Body mass index (BMI) overweight/obesity thresholds in South Asian (SA) adults, at equivalent type-2 diabetes risk are lower than for white Europeans (WE). We aimed to define adjusted overweight/obesity thresholds for UK–SA children based on equivalent insulin resistance (HOMA-IR) to WE children.

Methods

In 1138 WE and 1292 SA children aged 9.0–10.9 years, multi-level regression models quantified associations between BMI and HOMA-IR by ethnic group. HOMA-IR levels for WE children were calculated at established overweight/obesity thresholds (at 9.5 years and 10.5 years), based on UK90 BMI cut-offs. Quantified associations in SA children were then used to estimate adjusted SA weight-status thresholds at the calculated HOMA-IR levels.

Results

At 9.5 years, current WE BMI overweight and obesity thresholds were 19.2 kg/m², 21.3 kg/m² (boys) and 20.0 kg/m², 22.5 kg/m² (girls). At equivalent HOMA-IR, SA overweight and obesity thresholds were lower by 2.9 kg/m² (95% CI: 2.5–3.3 kg/m²) and 3.2 kg/m² (95% CI: 2.7–3.6 kg/m²) in boys and 3.0 kg/m² (95% CI: 2.6–3.4 kg/m²) and 3.3 kg/m² (95% CI: 2.8–3.8 kg/m²) in girls, respectively. At these lower thresholds, overweight/obesity prevalences in SA children were approximately doubled (boys: 61%, girls: 56%). Patterns at 10.5 years were similar.

Conclusions

SA adjusted overweight/obesity thresholds based on equivalent IR were markedly lower than BMI thresholds for WE children, and defined more than half of SA children as overweight/obese.

Reassessing Ethnic Differences in Mean BMI and Changes Between 2007 and 2013 in English Children

OBJECTIVE

National body fatness (BF) data for English South Asian and Black children use BMI, which provides inaccurate ethnic comparisons. BF levels and time trends in the English National Child Measurement Programme (NCMP) between 2007 and 2013 were assessed by using ethnic-specific adjusted BMI (aBMI) for South Asian and Black children.

METHODS

Analyses were based on 3,195,323 children aged 4 to 5 years and 2,962,673 children aged 10 to 11 years. aBMI values for South Asian and Black children (relating to BF as in White children) were derived independently. Mean aBMI levels and 5-year aBMI changes were obtained by using linear regression.

RESULTS

In the 2007-2008 NCMP, mean aBMIs in 10- to 11-year-old children (boys, girls) were higher in South Asian children (20.1, 19.9 kg/m² ) and Black girls, but not in Black boys (18.4, 19.2 kg/m² ) when compared with White children (18.6, 19.0 kg/m² ; all P < 0.001). Mean 5-year changes (boys, girls) were higher in South Asian children (0.16, 0.32 kg/m² per 5 y; both P < 0.001) and Black boys but not girls (0.13, 0.15 kg/m² per 5 y; P = 0.01, P = 0.41) compared with White children (0.02, 0.11 kg/m² per 5 y). Ethnic differences at 4 to 5 years were similar. Unadjusted BMI showed similar 5-year changes but different mean BMI patterns.

CONCLUSIONS

BF levels were higher in South Asian children than in other groups in 2007 and diverged from those in White children until 2013, a pattern not apparent from unadjusted BMI data.

Cohort profile: Examining Neighbourhood Activities in Built Living Environments in London: the ENABLE London-Olympic Park cohort

PURPOSE

The Examining Neighbourhood Activities in Built Living Environments in London (ENABLE London) project is a natural experiment which aims to establish whether physical activity and other health behaviours show sustained changes among individuals and families relocating to East Village (formerly the London 2012 Olympics Athletes' Village), when compared with a control population living outside East Village throughout.

PARTICIPANTS

Between January 2013 and December 2015, 1497 individuals from 1006 households were recruited and assessed (at baseline) (including 392 households seeking social housing, 421 seeking intermediate and 193 seeking market rent homes). The 2-year follow-up rate is 62% of households to date, of which 57% have moved to East Village.

FINDINGS TO DATE

Assessments of physical activity (measured objectively using accelerometers) combined with Global Positioning System technology and Geographic Information System mapping of the local area are being used to characterise physical activity patterns and location among study participants and assess the attributes of the environments to which they are exposed. Assessments of body composition, based on weight, height and bioelectrical impedance, have been made and detailed participant questionnaires provide information on socioeconomic position, general health/health status, well-being, anxiety, depression, attitudes to leisure time activities and other personal, social and environmental influences on physical activity, including the use of recreational space and facilities in their residential neighbourhood.

FUTURE PLANS

The main analyses will examine the changes in physical activity, health and well-being observed in the East Village group compared with controls and the influence of specific elements of the built environment on observed changes. The ENABLE London project exploits a unique opportunity to evaluate a 'natural experiment', provided by the building and rapid occupation of East Village. Findings from the study will be generalisable to other urban residential housing developments, and will help inform future evidence-based urban planning.