Development of ARCADIA: a tool for assessing the quality of peer-review reports in biomedical research

OBJECTIVE

To develop a tool to assess the quality of peer-review reports in biomedical research.

METHODS

We conducted an online survey intended for biomedical editors and authors. The survey aimed to (1) determine if participants endorse the proposed definition of peer-review report quality; (2) identify the most important items to include in the final version of the tool and (3) identify any missing items. Participants rated on a 5-point scale whether an item should be included in the tool and they were also invited to comment on the importance and wording of each item. Principal component analysis was performed to examine items redundancy and a general inductive approach was used for qualitative data analysis.

RESULTS

A total of 446 biomedical editors and authors participated in the survey. Participants were mainly male (65.9%), middle-aged (mean=50.3, SD=13) and with PhD degrees (56.4%). The majority of participants (84%) agreed on the definition of peer-review report quality we proposed. The 20 initial items included in the survey questionnaire were generally highly rated with a mean score ranging from 3.38 (SD=1.13) to 4.60 (SD=0.69) (scale 1-5). Participants suggested 13 items that were not included in the initial list of items. A steering committee composed of five members with different expertise discussed the selection of items to include in the final version of the tool. The final checklist includes 14 items encompassed in five domains (Importance of the study, Robustness of the study methods, Interpretation and discussion of the study results, Reporting and transparency of the manuscript, Characteristics of peer reviewer's comments).

CONCLUSION

Assessment of Review reports with a Checklist Available to eDItors and Authors tool could be used regularly by editors to evaluate the reviewers' work, and also as an outcome when evaluating interventions to improve the peer-review process.

Accuracy in detecting inadequate research reporting by early career peer reviewers using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process: a cross-sectional diagnostic study

BACKGROUND

The peer review process has been questioned as it may fail to allow the publication of high-quality articles. This study aimed to evaluate the accuracy in identifying inadequate reporting in RCT reports by early career researchers (ECRs) using an online CONSORT-based peer-review tool (COBPeer) versus the usual peer-review process.

METHODS

We performed a cross-sectional diagnostic study of 119 manuscripts, from BMC series medical journals, BMJ, BMJ Open, and Annals of Emergency Medicine reporting the results of two-arm parallel-group RCTs. One hundred and nineteen ECRs who had never reviewed an RCT manuscript were recruited from December 2017 to January 2018. Each ECR assessed one manuscript. To assess accuracy in identifying inadequate reporting, we used two tests: (1) ECRs assessing a manuscript using the COBPeer tool (after completing an online training module) and (2) the usual peer-review process. The reference standard was the assessment of the manuscript by two systematic reviewers. Inadequate reporting was defined as incomplete reporting or a switch in primary outcome and considered nine domains: the eight most important CONSORT domains and a switch in primary outcome(s). The primary outcome was the mean number of domains accurately classified (scale from 0 to 9).

RESULTS

The mean (SD) number of domains (0 to 9) accurately classified per manuscript was 6.39 (1.49) for ECRs using COBPeer versus 5.03 (1.84) for the journal's usual peer-review process, with a mean difference [95% CI] of 1.36 [0.88-1.84] (p < 0.001). Concerning secondary outcomes, the sensitivity of ECRs using COBPeer versus the usual peer-review process in detecting incompletely reported CONSORT items was 86% [95% CI 82-89] versus 20% [16-24] and in identifying a switch in primary outcome 61% [44-77] versus 11% [3-26]. The specificity of ECRs using COBPeer versus the usual process to detect incompletely reported CONSORT domains was 61% [57-65] versus 77% [74-81] and to identify a switch in primary outcome 77% [67-86] versus 98% [92-100].

CONCLUSIONS

Trained ECRs using the COBPeer tool were more likely to detect inadequate reporting in RCTs than the usual peer review processes used by journals. Implementing a two-step peer-review process could help improve the quality of reporting.

TRIAL REGISTRATION

Clinical.Trials.gov NCT03119376 (Registered April, 18, 2017).

A protocol of a cross-sectional study evaluating an online tool for early career peer reviewers assessing reports of randomised controlled trials

INTRODUCTION

Systematic reviews evaluating the impact of interventions to improve the quality of peer review for biomedical publications highlighted that interventions were limited and have little impact. This study aims to compare the accuracy of early career peer reviewers who use an innovative online tool to the usual peer reviewer process in evaluating the completeness of reporting and switched primary outcomes in completed reports.

METHODS AND ANALYSIS

This is a cross-sectional study of individual two-arm parallel-group randomised controlled trials (RCTs) published in the BioMed Central series medical journals, BMJ, BMJ Open and Annals of Emergency Medicine and indexed with the publication type 'Randomised Controlled Trial'. First, we will develop an online tool and training module based (a) on the Consolidated Standards of Reporting Trials (CONSORT) 2010 checklist and the Explanation and Elaboration document that would be dedicated to junior peer reviewers for assessing the completeness of reporting of key items and (b) the Centre for Evidence-Based Medicine Outcome Monitoring Project process used to identify switched outcomes in completed reports of the primary results of RCTs when initially submitted. Then, we will compare the performance of early career peer reviewers who use the online tool to the usual peer review process in identifying inadequate reporting and switched outcomes in completed reports of RCTs at initial journal submission. The primary outcome will be the mean number of items accurately classified per manuscript. The secondary outcomes will be the mean number of items accurately classified per manuscript for the CONSORT items and the sensitivity, specificity and likelihood ratio to detect the item as adequately reported and to identify a switch in outcomes. We aim to include 120 RCTs and 120 early career peer reviewers.

ETHICS AND DISSEMINATION

The research protocol was approved by the ethics committee of the INSERM Institutional Review Board (21 January 2016). The study is based on voluntary participation and informed written consent.

TRIAL REGISTRATION NUMBER

NCT03119376.

Diagnostic accuracy of percent free prostate-specific antigen in prostatic pathology and its usefulness in monitoring prostatic cancer patients

INTRODUCTION

The aim of our study was to evaluate the clinical usefulness of percent free prostate-specific antigen (PSA) [ratio of free PSA (fPSA) to total PSA (tPSA); f/tPSA] in prostatic pathology and its usefulness in monitoring prostatic cancer patients.

PATIENTS AND METHODS

Our prospective study was carried out on 470 consecutive male patients referred to our outpatient urological clinic for observation. We looked for relationships between tPSA, fPSA and percent free PSA and the patient's age, prostatic volume and histologic diagnosis as assessed by prostatic biopsies or surgical specimens (benign prostatic hypertrophy, carcinoma, hypertrophy with inflammation). In all cases, we calculated the specificity, sensitivity and diagnostic accuracy of percent free PSA in the diagnosis of prostatic diseases, using cutoff values ranging from 14 to 20%. In prostatic cancer patients, we considered the relationships between the various PSA molecular forms and staging, grading and follow-up values. We also evaluated the effects of hormonosuppressive therapy on the serum markers and noted for which tPSA value percent free PSA possessed the greatest diagnostic accuracy.

RESULTS

While tPSA and fPSA values appeared to be correlated with patient age and prostatic volume, percent free PSA did not show a relationship with these parameters. The specificity, sensitivity and overall diagnostic accuracy were better assuming a 16% cutoff value for percent free PSA than with other cutoff values. Prostatic inflammation associated with benign hypertrophy can cause false positives in both tPSA and f/tPSA measurements, since 60% of these patients have an f/tPSA ratio below 16%. In diagnosing carcinoma, the diagnostic accuracy of percent free PSA is 100% when tPSA is between 2.5 and 4.0 ng/ml. Percent free PSA is not linked with staging in prostatic cancer, but it does appear to be related to the Gleason score. In patients receiving hormonosuppressive treatment, f/tPSA decreased significantly, and more so in patients with a higher Gleason score. In patients with disease in rapid progression, percent free PSA was lower than in patients in a stable condition.

CONCLUSIONS

Based on our experience, 16% as the f/tPSA cutoff value for discriminating between benign and malignant pathologies is the best possible choice, as it provides the highest overall values of sensitivity, specificity and diagnostic accuracy (80, 61.5 and 84.5%, respectively) in the diagnosis of prostatic cancer. We believe that f/tPSA is not a definitive test for diagnosing prostatic cancer. Our observations on the behavior of percent free PSA in relation to prostatic carcinoma grading and staging and in the follow-up of carcinoma patients are interesting; however, further studies are needed to define the appropriate role of f/tPSA in patients with an established diagnosis of prostatic carcinoma and in the follow-up of patients with prostatic cancer.

[Urinary NMP22 as a new marker in patients with transitional cell carcinoma]

To evaluate urinary NMP22 dosage as a marker of urothelial tumours, we have selected a group of 90 patients (85 males and 5 females, mean age 66 years) with clinical suspicion of transitional cell carcinoma (TCC), with microscopic or macroscopic hematuria, flank pain, urographic abnormalities and dysuria. All the patients have been evaluated by urinary cytology, renal and bladder ultrasound, cystoscopy. When a bladder tumour has been detected, bladder biopsies and, when required, I.V.P., CT or retrograde pyelography have been performed. A urine sample has been collected between midnight and noon; all samples from patients who were undergoing invasive procedures such as cystoscopy, were collected before or at least 5 days after the procedure. The test has been performed according to ELISA NMP22 (Matritech) technique; the test is specific for the nuclear matrix protein/nuclear mitotic apparatus protein expressed by cancer cells. When performing the test, 30 patients presented macroscopic hematuria. 22 patients resulted to have benign urinary tract conditions, 65 patients had TCC and 3 patients had a final evaluation suspicious for TCC. The NMP22 values were respectively 7.1 +/- 4.7 U/ml, 21.9 +/- 21.0 U/ml and 16 +/- 8.0 U/ml. From our study the sensitivity of the test is 67% (with a threshold value of 10 U/ml) and 55% (with a threshold value of 20 U/ml), while the urinary cytology resulted to have a sensitivity of 26% (p < 0.05). The sensitivity of the test in relation to staging was as follow: Tis 66% with a mean NMP22 value of 23.3 U/ml, Ta 26% with a mean NMP22 value of 13.2 U/ml, T1 100% with a mean NMP22 value of 40 U/ml, T2 73% with a mean NMP22 value of 36.4 U/ml. The specificity of the test was 100% with a threshold value of 20 U/ml. When considering a threshold value of 10 U/ml, the NMP22 test has a sensitivity higher than cytology, especially in low stage TCC. Macroscopic hematuria does not affect its sensitivity; the diagnostic efficacy of the test is not increased by the association of urinary cytology. It has an important role in the diagnosis of residual disease after TURB and in the follow-up evaluation of bladder cancer patients, since its dosage is not influenced by inflammatory conditions of the mucosa. We believe therefore that NMP22 is useful in clinical practice.

Predictive value of interleukin 6 (IL-6), interleukin 8 (IL-8) and gastric intramucosal pH (pH-i) in major abdominal surgery

OBJECTIVE

To study plasma concentrations of interleukin 6 (IL-6) and interleukin 8 (IL-8) in patients with splanchnic hypoxia, as documented by gastric intramucosal measurements (pH-i), during major abdominal surgery and the relationship between IL-6 and IL-8 concentrations and postoperative complications as well as clinical outcome.

DESIGN

A prospective study.

PATIENTS

Twelve patients scheduled for major abdominal surgery with no evidence of coexisting infectious disease.

RESULTS

Six out of seven samples from patients with postoperative complications showed intraoperative pH-i levels lower than 7.32 and IL-6 levels higher than 300 pg/ml. Seven out of nine samples from patients without complications showed pH-i levels higher than 7.32 and IL-6 levels lower than 300 pg/ml. The difference in the pattern of distribution was statistically significant (p < 0.01). Only two out of seven samples of patients with postoperative complications showed intraoperative pH-i levels lower than 7.32 and IL-8 levels higher than 60 pg/ml. It was not possible to identify a clear distribution pattern of data points for IL-6 and IL-8 during the postoperative period.

CONCLUSIONS

Intraoperative splanchnic ischemia, as documented by gastric intramucosal pH-i, is directly correlated to the increase of IL-6 plasma levels and to the incidence of postoperative complications, while IL-8 levels showed no correlation with surgical complications.

Serum markers for monitoring of prostatic carcinoma

BACKGROUND

Serum TPS (tissue polypeptide-specific antigen) has been observed to be characteristic of carcinoma proliferation, and increased levels of TPS seem to be closely related to tumor progression. In this study we wanted to evaluate the importance of the tumor-marker TPS in the diagnosis and follow-up of patients with prostatic carcinoma, and to compare it with prostate-specific antigen (PSA).

METHODS

We considered 39 patients with clinically confined disease, who underwent neoadjuvant hormonal therapy and thereafter radical prostatectomy, and 45 patients who did not undergo surgery and underwent hormonal adjuvant therapy alone. PSA and TPS were measured at the time of diagnosis and at regular intervals in the follow-up; TPS was measured in a control group of patients as well.

RESULTS

We were able to observe that, in untreated patients, PSA correlates with clinical stage, increasing with increasing tumor stage; a similar correlation was not observed when considering TPS. After androgen ablation we observed a decrease in PSA, but the serum values of TPS remained higher, suggesting that activity still exists inside the tumor. The evaluation of TPS appeared to be of particular interest in the follow-up after radical prostatectomy, especially in patients undergoing hormonal therapy; in fact, we were able to observe that relapse of the disease can be suspected early by the increase of TPS in hormonally treated patients.

CONCLUSIONS

We assert that TPS can add useful information on the state of neoplastic illness, especially in patients following adjuvant androgen-suppressive hormonal therapy, after radical prostatectomy; serial measurements of this marker could be useful in the early diagnosis of a relapse.

[Diagnostic efficacy of free SPA/total PSA ratio in the diagnosis of prostatic carcinoma]

Prostate specific antigen, specific organ and tissue marker, is a glycoprotein present in serum in different molecular forms, i.e. not protein bound and bound to proteins (PSA-ACT and PSA-AMG). The total PSA is expressed by the sum of the non protein bound value (free-PSA) and PSA-ACT. The aim of our study was to evaluate the hypothesis that measurement of free/total PSA ratio may be helpful in the differential diagnosis of prostatic pathology. Our study was conducted on 350 patients, to whom the total-PSA, free-PSA and f/t PSA had been performed; 250 patients showed a total PSA between 2.5 and 10 ng/ml and 185 of them had symptoms of bladder out-flow obstruction. In all of the 250 patients digital rectal examination, transrectal ultrasound and prostatic biopsy were performed. 100 patients were controls. The cut-off to differentiate between benign and malignant prostatic disease was 16%. The pathologic diagnosis was related to the f/t PSA ratio, and in particular those patients with a f/t PSA lower than 16% were expected to be prostatic carcinoma, while those with a f/t PSA higher than 16% were expected to be benign prostatic hypertrophy. The diagnostic accuracy of the ratio was calculated, and it was observed that it was 88.65% in the diagnosis of benign prostatic hypertrophy, while in the diagnosis of prostatic carcinoma it was 84.5%. We can therefore assume that f/t PSA can add useful information on prostatic pathology, eventually sparing unnecessary prostatic biopsies.

The effect of different glucose endovenous administrations on amylin and insulin blood concentration in healthy subjects

Islet amyloid polypeptide (IAPP) or amylin is a 37 amino-acid peptide involved in carbohydrate metabolism. It is a hormone secreted from the pancreatic cells which is reported to be co-secreted with insulin. Its function and secretion is not well known. A paracrine inhibition of insulin secretion could be the main function of this hormone. Different hypotheses are considered regarding IAPP co-localization with insulin in the secretory granules and its subsequent secretion. In the attempt to clarify these controversial findings we evaluated the plasma concentrations of IAPP and insulin during different intravenous glucose administrations in healthy patients. Twenty normoglycemic patients (10 females, 10 males, age +/- SD 37 +/- 9 years) underwent two different endovenous glucose administrations: a) "long duration" infusion, in which 500 ml of 33% glucose solution was administered in 60 minutes, b) a standard intra-venous glucose tolerance test (IVGTT) characterized by an infusion of 0.33 g/Kg of glucose in 4 minutes. Blood samples were collected at fixed times (from 0 to 180 minutes) to assay IAPP, insulin and glucose concentration. The IAPP secretion seemed to be related more to the glycemic variation than to the insulin secretion. A proportional behavior of IAPP and insulin was observed only in the first 90 minutes of the long duration infusion. The highest increases of IAPP concentrations were found when there was a rapid glucose decrease in the blood. These findings suggest a significant role of IAPP in regulating the rapid decreases of glycemia.

Blood catecholamine levels and lymphocyte beta-adrenoceptors following acute noise stress

Blood hormonal levels and lymphocyte beta-adrenoceptor characteristics have been studied in subjects exposed to acute noise stress. Cortisol and DHEAS show a peak at 15 min after the beginning of the stimulus. Catecholamine levels show an increase in 4 out of 8 subjects. However, no statistically significant changes have been observed in beta-adrenoceptor characteristics.

Prognostic significance of basement membrane antigens and of soluble interleukin 2 receptor serum concentrations in heart transplant rejection

Endomyocardial biopsy is still considered the only reliable method for diagnosing acute rejection. However, because of its invasive nature, this procedure cannot be performed on a daily basis. Therefore, for the noninvasive monitoring of transplanted patients, we tested the sensitivity of serum levels of basement membrane antigens, type IV collagen fragment NC1 and laminin fragment P1 (LP1), of antibodies against laminin and type IV collagen, and of soluble interleukin 2 receptor (sIL2R). In 17 patients who underwent heart transplantation no correlation was found between the degree of rejection and the levels of antibodies to type IV collagen and laminin. Serum laminin P1 and sIL2-R values were found increased in all the study groups even in absence of rejection; however, sIL2R levels were higher in patients with more severe rejection. NC1 levels were found significantly higher in patients with mild or moderate rejection than in those with no rejection or in controls. These preliminary data suggest a possible predictive role of basement membrane antigens in cardiac rejection, but further studies in a larger group of transplanted patients are needed.

[Effects of insular hormones on the secretion of atrial natriuretic factor]

Aim of this study was to verify the existence of a correlation between the insular hormones and the atrial natriuretic factor (ANF). We studied 70 subjects (20 control, 20 obese, 20 non insulin-dependent diabetic obese, 10 insulin-dependent diabetic subjects) submitted to a glucagon test (1 mg i.v.). Blood samples were collected at -15, 0, 3, 6, 12, 15, 30, 60, 120, 150 minutes to assay insulin, C-peptide, serum electrolytes and ANF levels. The results to point out are: the ANF basal values are significantly higher (p < 0.01) in non insulin-dependent obese patients than in controls; the obese subjects also present a significant difference (p < 0.05). After glucagon injection no variations have been found in the ANF values until the 15th minute; then the controls, the obese and, above all, the non insulin-dependent diabetic obese subjects showed a significant increase of the ANF values between 60' and 90' (basal values 38 +/- 4 ng/ml; 90' values 85 +/- 7 ng ml). As these high values appear only after the induction of hyperinsulinism in our experiment and are not present in the type-1 diabetic subjects, it's probable that insulin, rather than glucagon, stimulates, directly or indirectly, the ANF secretion. If this hypothesis is confirmed, the correlation between insulin and ANF should deserve attention from a therapeutic point of view in subjects with glycometabolic imbalance.

[ANP changes induced by varying TSH]

Aim of our study was estimating if TSH could influence the timing of ANP release. About it we observed 70 male subjects: 40 were euthyroid patients aged 42 +/- 5 years (group A); 20 patients treated by thyroxine in doses sufficient for inhibiting TSH release, aged 45 +/- 7 years (group B). A third group (C) was composed by 10 subjects with high basal levels of serum T3 and T4 with no thyrostatic therapy, aged 40 +/- 9 years. These subjects underwent a TRH test estimating at -30', 0', 30', 60', 120' plasma concentrations of TSH, ANP, T3, T4. In reply to physiological stimulus induced on patients of group A we observed a significant increase of TSH values (max at 60') and ANP levels (max at 120'). No significant variations occurred during the TRH test in T3 and T4 concentrations. In groups B and C no important modifications were observed neither in TSH nor in ANP plasma levels. ANP secretion seems to be dependent from the secretory condition of hypothalamus-pituitary-thyroid axis both in physiological and pathological conditions.

Serum levels of soluble interleukin-2 receptor in patients with systemic lupus erythematosus and systemic idiopathic vasculitis

Soluble interleukin 2 receptor (sIL2R) may be used as an index of immune perturbation. We report on the correlation between the serum levels of sIL2R, as assessed with a sandwich-ELISA, and disease activity in 61 patients with Systemic Lupus Erythematosus (SLE) and in 15 with systemic idiopathic vasculitis (SIV). The mean levels of sIL2R in SLE and SIV patients were significantly higher than in healthy controls and higher values were detected in patients with active disease or severe organ involvement or infection. We also studied patients with clinically silent SLE, characterized by the presence of several immunologic abnormalities. The sIL2R mean level in this group did not differ from that of quiescent SLE patients, suggesting that immunologic alterations are present even in inactive SLE. Finally, the sIL2R assay showed higher sensitivity and specificity than most of the common immunologic parameters.

[The soluble IL-2 receptor in malignant hemopathies]

The increase in the serum levels of the IL-2 receptors is due to its release both in vivo and in vitro from activated cells or neoplastic cells expressing it constitutively. The diagnostic, prognostic and physiopathologic significance of the sIL-2R was investigated by testing the serum of 271 haemopathic patients in various stages of the disease. In HCL the elevated sIL-2R level has a diagnostic value. In HD the sIL-2R level appears to be directly correlated with the extent of the disease and is equally important in the follow up of patients with HCL, NHL, HD, AL and MDS, where the serum level of the soluble receptor is usually associated with the biological and clinical activity of the disease. Unlike other B lymphoproliferations, patients with Multiple Myeloma on average show only slightly elevated levels of soluble receptor with no significant differences related to the stage or evolution. As for the chronic myeloproliferative disorders, we found only slightly elevated values in ET and PV, with frankly pathological values in CML during a blastic crisis or in the accelerated phase and in MFI during the clinically active phase of the disease.

Interleukin-2, soluble interleukin-2 receptor and tumor necrosis factor in sera from patients with rheumatoid arthritis

Interleukin-2 (IL-2), soluble interleukin-2 receptor (IL-2R) and tumor necrosis factor (TNF) have been measured in sera from 47 patients affected by classic rheumatoid arthritis (RA) using an enzyme-linked immunosorbent assay. The patients were divided into 4 groups as follows: group A, 18 patients with inactive disease; group B, 19 patients with active disease under treatment with non-steroidal antiinflammatory drugs (NSAID) and second-line drugs; group C, 5 patients with active disease under treatment with NSAID and cyclosporine A (CSA) for at least 4 months; group D, 5 patients in the same condition as patients of group C, but treated with azathioprine (AZA) instead of CSA. IL-2 was undetectable in all patients except two, both characterized by active disease. Soluble IL-2R levels were above the upper limit of the normal range in most of the patients studied, but the mean value ( +/- 1 SD) was significantly higher in patients of group B (1,288 +/- 421 U/ml) than in patients of group A (686 +/- 205 U/ml) and group C (842 +/- 414 U/ml). In two patients affected by active RA treated with pulse methylprednisolone therapy (1 g/day for 3 alternate days) the values of soluble IL-2R dropped from 948 to 662 U/ml and from 660 to 518 U/ml, respectively. No statistically significant correlation was observed between the serum level of IL-2R and the RF titre or percentage of C1q-binding activity, respectively. TNF was found within the normal range in all patients except one, who was characterized by active arthritis, high number of rheumatoid skin nodules and extremely high RF titre.(ABSTRACT TRUNCATED AT 250 WORDS)