An experimental evaluation of a stopping rule aimed at maximizing cost-quality trade-offs in surveys

Surveys face difficult choices in managing cost-error trade-offs. Stopping rules for surveys have been proposed as a method for managing these trade-offs. A stopping rule will limit effort on a select subset of cases to reduce costs with minimal harm to quality. Previously proposed stopping rules have focused on quality with an implicit assumption that all cases have the same cost. This assumption is unlikely to be true, particularly when some cases will require more effort and, therefore, more costs than others. We propose a new rule that looks at both predicted costs and quality. This rule is tested experimentally against another rule that focuses on stopping cases that are expected to be difficult to recruit. The experiment was conducted on the 2020 data collection of the Health and Retirement Study (HRS). We test both Bayesian and non-Bayesian (maximum-likelihood or ML) versions of the rule. The Bayesian version of the prediction models uses historical data to establish prior information. The Bayesian version led to higher-quality data for roughly the same cost, while the ML version led to small reductions in quality with larger reductions in cost compared to the control rule.

Can artificial intelligence separate the wheat from the chaff in systematic reviews of health economic articles?

OBJECTIVES

Artificial intelligence-powered tools, such as ASReview, could reduce the burden of title and abstract screening. This study aimed to assess the accuracy and efficiency of using ASReview in a health economic context.

METHODS

A sample from a previous systematic literature review containing 4,994 articles was used. Previous manual screening resulted in 134 articles included for full-text screening (FT) and 50 for data extraction (DE). Here, accuracy and efficiency was evaluated by comparing the number of identified relevant articles with ASReview versus manual screening. Pre-defined stopping rules using sampling criteria and heuristic criteria were tested. Robustness of the AI-tool's performance was determined using 1,000 simulations.

RESULTS

Considering included stopping rules, median accuracy for FT articles remained below 85%, but reached 100% for DE articles. To identify all relevant articles, a median of 89.9% of FT articles needed to be screened, compared to 7.7% for DE articles. Potential time savings between 49 and 59 hours could be achieved, depending on the stopping rule.

CONCLUSIONS

In our case study, all DE articles were identified after screening 7.7% of the sample, allowing for substantial time savings. ASReview likely has the potential to substantially reduce screening time in systematic reviews of health economic articles.

Finite nucleos(t)ide analogue therapy in hepatitis B e antigen-negative chronic hepatitis B: From an "option" to an "active recommendation"

Nucleos(t)ide analogue (Nuc) including entecavir, tenofovir disoproxil fumarate, and tenofovir alafenamide may suppress hepatitis B virus (HBV) DNA profoundly but have no direct action on covalently closed circular DNA, which is a very stable template for HBV production. Therefore, decades of long-term Nuc therapy are required to maintain HBV suppression and to achieve hepatitis B surface antigen (HBsAg) loss in hepatitis B e antigen (HBeAg)-negative patients. However, there are concerns including financial burden, adherence, and willingness for indefinite long-term Nuc therapy. Patients lost to follow-up and hence not monitored may risk severe relapse that may deteriorate to hepatic decompensation or even hepatic failure. Cessation of Nuc therapy in HBeAg-negative patients was initially considered in early 2000s. Earlier findings in Asian patients that finite Nuc therapy over 2-3 years is feasible and safe have founded Asian-Pacific Association for the Study of Liver stopping rule since 2008. Subsequent studies have confirmed the feasibility and safety of the strategy of finite Nuc therapy, which has finally been accepted as "an option" by American and European liver associations since 2016. More recent large studies since 2018 have further confirmed the pivotal finding of greatly increased HBsAg loss rate (~5-year 39%) after stopping Nuc therapy. With the high HBsAg loss rate as the main justification, the paradigm shift from indefinite long-term therapy to finite Nuc therapy in HBeAg-negative patients has been changing from an "option" to an "active recommendation" aiming to achieve HBsAg loss. More studies are needed to fine-tuning the strategy, including research for the optimal duration of consolidation therapy, timing to stop, and to start retreatment.

Monitored Tomographic Reconstruction-An Advanced Tool to Study the 3D Morphology of Nanomaterials

Detailed and accurate three-dimensional (3D) information about the morphology of hierarchically structured materials is derived from multi-scale X-ray computed tomography (XCT) and subsequent 3D data reconstruction. High-resolution X-ray microscopy and nano-XCT are suitable techniques to nondestructively study nanomaterials, including porous or skeleton materials. However, laboratory nano-XCT studies are very time-consuming. To reduce the time-to-data by more than an order of magnitude, we propose taking advantage of a monitored tomographic reconstruction. The benefit of this new protocol for 3D imaging is that the data acquisition for each projection is interspersed by image reconstruction. We demonstrate this new approach for nano-XCT data of a novel transition-metal-based materials system: MoNi4 electrocatalysts anchored on MoO2 cuboids aligned on Ni foam (MoNi4/MoO2@Ni). Quantitative data that describe the 3D morphology of this hierarchically structured system with an advanced electrocatalytically active nanomaterial are needed to tailor performance and durability of the electrocatalyst system. We present the framework for monitored tomographic reconstruction, construct three stopping rules for various reconstruction quality metrics and provide their experimental evaluation.

Stopping rules for multi-category computerized classification testing

Computerized classification testing (CCT) aims to classify persons into one of two or more possible categories to make decisions such as mastery/non-mastery or meet most/meet all/exceed. A defining feature of CCT is its stopping criterion: the test terminates when there is enough confidence to make a decision. There is abundant research on CCT with a single cut-off, and two common stopping criteria are the sequential probability ratio test (SPRT) statistic and the generalized likelihood ratio statistic (GLR). However, there is a relative scarcity of research extending the SPRT to the multi-hypothesis case for when there is more than one cut-off. In this paper, we propose a new multi-category GLR (mGLR) statistic as well as a stochastically curtailed version of the CCT with three or more categories. A simulation study was conducted to show that the mGLR statistic outperformed the existing stopping rules by generating shorter average test length without sacrificing classification accuracy. Results also revealed that the stochastically curtailed mGLR successfully increased test efficiency in certain testing conditions.

Statistical Guideline No. 5. Include Results of a Power Analysis; if a Power Analysis Was Not Performed, Describe the Stopping Rule for Recruitment

This is one in a series of statistical guidelines designed to highlight common statistical considerations in behavioral medicine research. The goal is to briefly discuss appropriate ways to analyze and present data in the International Journal of Behavioral Medicine (IJBM). Collectively the series will culminate in a set of basic statistical guidelines to be adopted by IJBM and integrated into the journal's official Instructions for Authors, and also to serve as an independent resource. If you have ideas for a future topic, please email the Statistical Editor, Suzanne Segerstrom at segerstrom@uky.edu.

Optimizing interim analysis timing for Bayesian adaptive commensurate designs

In developing products for rare diseases, statistical challenges arise due to the limited number of patients available for participation in drug trials and other clinical research. Bayesian adaptive clinical trial designs offer the possibility of increased statistical efficiency, reduced development cost and ethical hazard prevention via their incorporation of evidence from external sources (historical data, expert opinions, and real-world evidence), and flexibility in the specification of interim looks. In this paper, we propose a novel Bayesian adaptive commensurate design that borrows adaptively from historical information and also uses a particular payoff function to optimize the timing of the study's interim analysis. The trial payoff is a function of how many samples can be saved via early stopping and the probability of making correct early decisions for either futility or efficacy. We calibrate our Bayesian algorithm to have acceptable long-run frequentist properties (Type I error and power) via simulation at the design stage. We illustrate our approach using a pediatric trial design setting testing the effect of a new drug for a rare genetic disease. The optimIA R package available at https://github.com/wxwx1993/Bayesian_IA_Timing provides an easy-to-use implementation of our approach.

Baseline and kinetics of serum hepatitis B virus RNA predict response to pegylated interferon-based therapy in patients with hepatitis B e antigen-negative chronic hepatitis B

Serum hepatitis B virus (HBV) RNA has emerged as a novel biomarker of treatment response. This study aimed to investigate the role of this marker in predicting long-term outcome of patients with hepatitis B e antigen (HBeAg)-negative chronic hepatitis B (CHB) receiving pegylated interferon (PEG-IFN)-based therapy. Serial serum samples from 91 patients with HBeAg-negative CHB previously treated with PEG-IFN alone or combined with entecavir in a randomized trial were retrospectively analysed. HBV RNA quantification was examined by droplet digital PCR. At the end of 3 years post-treatment follow-up, maintained virological response (MVR, HBV DNA < 2000 IU/mL), and hepatitis B surface antigen (HBsAg) clearance were achieved in 37.4% (34/91) and 7.7% (7/91), respectively. Baseline serum HBV RNA concentrations correlated with HBV DNA and covalently closed circular DNA but did not correlate with HBsAg levels. Multiple regression analysis showed that pre-treatment HBV RNA and HBsAg were independently associated with MVR and HBsAg clearance. Baseline HBV RNA (cut-off 2.0 log₁₀  copies/mL) had a positive predictive value (PPV) and a negative predictive value in predicting MVR of 80.8% and 80.0%, respectively. At the same cut-off value, PPV and NPV for predicting HBsAg clearance were 30.8% and 95.4%, respectively. At week 12 during therapy, HBV RNA level ≥ 2 log₁₀ copies/mL displayed high NPVs of achieving MVR and HBsAg clearance (95% and 100%, respectively). In conclusion, the measurement of HBV RNA prior to PEG-IFN-based therapy could identify patients with high probability of MVR. In addition, HBV RNA kinetics may serve as a promising "stopping rule" in patients infected with HBV genotypes B or C.

An Investigation of Exposure Control Methods With Variable-Length CAT Using the Partial Credit Model

The purpose of this simulation study was to investigate the effect of several different item exposure control procedures in computerized adaptive testing (CAT) with variable-length stopping rules using the partial credit model. Previous simulation studies on CAT exposure control methods with polytomous items rarely considered variable-length tests. The four exposure control techniques examined were the randomesque with a group of three items, randomesque with a group of six items, progressive-restricted standard error (PR-SE), and no exposure control. The two variable-length stopping rules included were the SE and predicted standard error reduction (PSER), along with three item pools of varied sizes (43, 86, and 172 items). Descriptive statistics on number of nonconvergent cases, measurement precision, testing burden, item overlap, item exposure, and pool utilization were calculated. Results revealed that the PSER stopping rule administered fewer items on average while maintaining measurement precision similar to the SE stopping rule across the different item pool sizes and exposure controls. The PR-SE exposure control procedure surpassed the randomesque methods by further reducing test overlap, maintaining maximum exposure rates at the target rate or lower, and utilizing all items from the pool with a minimal increase in number of items administered and nonconvergent cases.

Stopping Rules for Computer Adaptive Testing When Item Banks Have Nonuniform Information

The standard error (SE) stopping rule, which terminates a computer adaptive test (CAT) when the SE is less than a threshold, is effective when there are informative questions for all trait levels. However, in domains such as patient reported outcomes, the items in a bank might all target one end of the trait continuum (e.g., negative symptoms), and the bank may lack depth for many individuals. In such cases, the predicted standard error reduction (PSER) stopping rule will stop the CAT even if the SE threshold has not been reached, and can avoid administering excessive questions that provide little additional information. By tuning the parameters of the PSER algorithm, a practitioner can specify a desired tradeoff between accuracy and efficiency. Using simulated data for the PROMIS Anxiety and Physical Function banks, we demonstrate that these parameters can substantially impact CAT performance. When the parameters were optimally tuned, the PSER stopping rule was found to outperform the SE stopping rule overall and particularly for individuals not targeted by the bank, and presented roughly the same number of items across the trait continuum. Therefore, the PSER stopping rule provides an effective method for balancing the precision and efficiency of a CAT.

Threshold point utilisation in juror decision-making

This study aims to identify whether a model of juror decision-making (i.e. the threshold point model) that encompasses both rational and intuitive decision-making exists. A total of 60 participants were selected who are eligible for jury duty in Scotland. These individuals read nine vignettes and rated the evidence of each vignette separately by placing the evidence in either a guilty, a not guilty or a not proven (a verdict type specific to Scotland) counter. Participants were asked after being presented with each piece of information to state how likely they thought the suspect was of being guilty, on a scale from 1 to 100. The data are best described using a flexible model (i.e. a diffusion model) that allows for information integration. Future research should examine whether or not the diffusion model can explain cognitive fallacies, such as confirmation bias, that are commonly studied in decision science.

Projection-Based Stopping Rules for Computerized Adaptive Testing in Licensure Testing

The confidence interval (CI) stopping rule is commonly used in licensure settings to make classification decisions with fewer items in computerized adaptive testing (CAT). However, it tends to be less efficient in the near-cut regions of the θ scale, as the CI often fails to be narrow enough for an early termination decision prior to reaching the maximum test length. To solve this problem, this study proposed the projection-based stopping rules that base the termination decisions on the algorithmically projected range of the final θ estimate at the hypothetical completion of the CAT. A simulation study and an empirical study were conducted to show the advantages of the projection-based rules over the CI rule, in which the projection-based rules reduced the test length without jeopardizing critical psychometric qualities of the test, such as the θ and classification precision. Operationally, these rules do not require additional regularization parameters, because the projection is simply a hypothetical extension of the current test within the existing CAT environment. Because these new rules are specifically designed to address the decreased efficiency in the near-cut regions as opposed to for the entire scale, the authors recommend using them in conjunction with the CI rule in practice.

The claim that China's fertility restrictions contributed to the use of prenatal sex selection: A sceptical reappraisal

Most observers assume that China's fertility restrictions contribute to the use of prenatal sex selection. I question the logic and evidence underlying that assumption. Experts often stress that China's low fertility is largely voluntary, and that fertility restrictions are an unneeded safety valve. Others claim that China's '1.5-child' loophole, common throughout rural areas, reinforces son preference or intensifies prenatal sex discrimination by hardening fertility constraints. These claims defy logic upon closer examination. Moreover, almost two-thirds of the exceptional distortion of the sex ratio in 1.5-child areas results from excess underreporting of daughters and enforced sex-specific stopping. Prenatal sex selection may explain the remaining third but probably reflects the stronger rural son preference that led to the 1.5-child loophole itself. The recent surge in sex selection of first births that has perpetuated the distortions also seems unrelated to policy. Some son-preferring parents who formerly wanted two children may now genuinely want only one.

Treatment outcomes and validation of the stopping rule for response to peginterferon in chronic hepatitis B: A Thai nationwide cohort study

BACKGROUND AND AIMS

Peginterferon has demonstrated effectiveness in clinical trials in patients with chronic hepatitis B (CHB). However, its efficacy in real-life settings remains unclear. We investigated the efficacy of peginterferon for CHB and validated the performance of previously identified response predictors in clinical practice.

METHODS

We analyzed prospectively collected data from a Thai nationwide cohort of CHB patients treated with peginterferon alfa-2a (180 µg/week, 48 weeks).

RESULTS

Among a total of 233 patients, mostly with genotype B or C, sustained response was observed in 23% of 135 hepatitis B e antigen (HBeAg)-positive patients (HBeAg seroconversion with hepatitis B virus [HBV] DNA < 2000 IU/mL) and 42% of 98 HBeAg-negative patients (HBV DNA < 2000 IU/mL with aminotransferase normalization) at 24 weeks after treatment. Age, sex, presence of cirrhosis, genotype, and pretreatment levels of aminotransferase, HBV DNA, and hepatitis B surface antigen (HBsAg) were not identified as significant predictors of sustained response. In HBeAg-positive patients, HBsAg > 20 000 IU/mL at week 12 provided a good stopping rule, with a negative predictive value of 96%. In HBeAg-negative patients, the performance of 12-week stopping rules of no decline in HBsAg with a < 2log₁₀ decline in HBV DNA and a < 10% log₁₀ decline in HBsAg showed modest negative predictive values of 80% and 66%, respectively, for achieving sustained response.

CONCLUSION

Outcomes in CHB patients treated with peginterferon in a clinical setting are similar to those demonstrated in clinical trials. Application of the early stopping rule based on HBsAg quantification may allow individualization of therapy, particularly in HBeAg-positive patients.

Estimation of the Coefficient of Variation with Minimum Risk: A Sequential Method for Minimizing Sampling Error and Study Cost

The coefficient of variation is an effect size measure with many potential uses in psychology and related disciplines. We propose a general theory for a sequential estimation of the population coefficient of variation that considers both the sampling error and the study cost, importantly without specific distributional assumptions. Fixed sample size planning methods, commonly used in psychology and related fields, cannot simultaneously minimize both the sampling error and the study cost. The sequential procedure we develop is the first sequential sampling procedure developed for estimating the coefficient of variation. We first present a method of planning a pilot sample size after the research goals are specified by the researcher. Then, after collecting a sample size as large as the estimated pilot sample size, a check is performed to assess whether the conditions necessary to stop the data collection have been satisfied. If not an additional observation is collected and the check is performed again. This process continues, sequentially, until a stopping rule involving a risk function is satisfied. Our method ensures that the sampling error and the study costs are considered simultaneously so that the cost is not higher than necessary for the tolerable sampling error. We also demonstrate a variety of properties of the distribution of the final sample size for five different distributions under a variety of conditions with a Monte Carlo simulation study. In addition, we provide freely available functions via the MBESS package in R to implement the methods discussed.

A randomized clinical trial of peginterferon alpha-2b with or without entecavir in patients with HBeAg-negative chronic hepatitis B: Role of host and viral factors associated with treatment response

Combining peginterferon (PEG-IFN) and a potent nucleoside/nucleotide analogue might improve treatment response in patients with chronic hepatitis B (CHB). The aims of this study were to compare the efficacy of PEG-IFN alpha-2b with or without entecavir in HBeAg-negative CHB and to investigate predictors of response. A total of 126 treatment-naïve patients were randomly assigned to receive monotherapy (n = 63) or combination therapy (n = 63) for 48 weeks. Virological response (VR) was defined as HBV DNA level <2000 IU/mL at week 96. Baseline factors including polymorphisms in the IFNL3 (rs12979860) and HLA-DPA1 (rs3077) genes and on-treatment viral kinetics were determined. At week 48, rates of undetectable HBV DNA were lower in the monotherapy than combination groups, but rates of HBsAg clearance and decline were comparable. At week 96, there was no difference between the corresponding groups regarding virological response (41.3% vs 38.1%, P = 0.856), HBsAg clearance (9.5% vs 4.8%, P = 0.491) and HBsAg decline. Baseline HBsAg level [odds ratio (OR): 3.14 (1.34-7.69), P = 0.012] and rs3077 polymorphism [OR: 2.78 (1.27-6.11), P = 0.011] were independent predictors of response. Patients carried GG genotype of rs3077 with low baseline HBV (<1000 IU/mL) had high probability of achieving VR (76.5%) and HBsAg clearance (29.4%). None of the patients without decrease in HBsAg combined with <2 log10 HBV DNA decline at week 12 achieved a virological response. In conclusion, the combination therapy lead to greater on-treatment HBV DNA suppression but did not improve virological response and HBsAg clearance/decline over monotherapy. Host and viral factors could help optimize decision-making at baseline and during PEG-IFN-based therapy.

Trial design for evaluating novel treatments during an outbreak of an infectious disease

Tragically, the outbreak of Ebola that started in West Africa in 2014 has been far more extensive and damaging than any previous outbreaks. The duration of the outbreak has, for the first time, allowed the clinical evaluation of Ebola treatments. This article discusses the designs used for two such clinical trials which have recruited patients in Liberia and Sierra Leone. General principles are outlined for trial designs intended to be deployed quickly, adapt flexibly and provide results soon enough to influence the course of the current epidemic rather than just providing evidence for use should Ebola break out again. Lessons are drawn for the conduct of clinical research in future outbreaks of infectious diseases, where the sequence of events may or may not be similar to the West African Ebola epidemic.

Advances in Statistical Approaches Oncology Drug Development

We describe some recent developments in statistical methodology and practice in oncology drug development from an academic and an industry perspective. Many adaptive designs were pioneered in oncology, and oncology is still at the forefront of novel methods to enable better and faster Go/No-Go decision making while controlling the cost.

An Application of Endpoint Detection to Bivariate Data in Tau-Path Order

The Fligner and Verducci (1988) multistage model for rankings is modified to create the moving average maximum likelihood estimator (MAMLE), a locally smooth estimator that measures stage-wise agreement between two long ranked lists, and provides a stopping rule for the detection of the endpoint of agreement. An application of this MAMLE stopping rule to bivariate data set in tau-path order (Yu, Verducci and Blower (2011)) is discussed. Data from the National Cancer Institute measuring associations between gene expression and compound potency are studied using this application, providing insights into the length of the relationship between the variables.

A Bayesian Stopping Rule for Sequential Monitoring of Serious Adverse Events

In an ongoing clinical trial, there will always be a risk for unanticipated critical safety problems, such as excessive occurrence of serious adverse events. When such a problem arises, the trial administrators must conduct an immediate evaluation to determine whether the trial should be terminated to protect patients. This decision is complicated but may be aided by statistical stopping rules. Sequential stopping rules are appropriate for immediate decisions, but frequentist approaches may not be useful because the unknown truncated end of the trial makes it impossible to define type I errors. Thus, a Bayesian stopping rule is proposed that is based on the posterior distribution with an informative prior distribution, and a guideline to construct this stopping rule is presented. Some operating characteristics are evaluated and compared with those of the modified sequential probability ratio test (SPRT), the maximized SPRT, and Pocock's test. The proposed method has flexibility for construction and could provide a more desirable performance than the other compared methods.

How Stable Are Top Choices Over Time? An Investigation Into Preferences Among Popular Baby Names In The United States

For the problem of assessing initial agreement between two rankings of long lists, inference in the Fligner and Verducci (1988) multistage model for rankings is modified to provide a locally smooth estimator of stage-wise agreement. An extension to the case of overlapping but different sets of items in the two lists, and a stopping rule to identify the endpoint of agreement, are also provided. Simulations show that this approach performs very well under several conditions. The methodology is applied to a database of popular names for newborns in the United States and provides insights into trends as well as differences in naming conventions between the two sexes.

Multiple-stage sampling procedure for covariate-adjusted response-adaptive designs

Covariate-adjusted response-adaptive (CARA) design becomes an important statistical tool for evaluating and comparing the performance of treatments when targeted medicine and adaptive therapy become important medical innovations. Due to the nature of the adaptive therapies of interest and how subjects accrue to a sampling procedure, it is of interest how to control the sample size sequentially such that the estimates of treatment effects have satisfactory precision in addition to its asymptotic properties. In this paper, we apply a multiple-stage sequential sampling method to CARA design in such a way that the control of the sample size is more feasible. The theoretical properties of the proposed method, including the estimates of regression parameters and the allocation probabilities under this randomly stopped sampling procedure, are discussed. The numerical results based on synthesized data and a real example are presented.

On becoming a Bayesian: early correspondences between J. Cornfield and L. J. Savage

Jerome Cornfield was arguably the leading proponent for the use of Bayesian methods in biostatistics during the 1960s. Prior to 1963, however, Cornfield had no publications in the area of Bayesian statistics. At a time when frequentist methods were the dominant influence on statistical practice, Cornfield went against the mainstream and embraced Bayes. The goals of this paper are as follows: (i) to explore how and why this transformation came about and (ii) to provide some sense as to who Cornfield was and the context in which he worked.

A New Stopping Rule for Computerized Adaptive Testing

The goal of the current study was to introduce a new stopping rule for computerized adaptive testing. The predicted standard error reduction stopping rule (PSER) uses the predictive posterior variance to determine the reduction in standard error that would result from the administration of additional items. The performance of the PSER was compared to that of the minimum standard error stopping rule and a modified version of the minimum information stopping rule in a series of simulated adaptive tests, drawn from a number of item pools. Results indicate that the PSER makes efficient use of CAT item pools, administering fewer items when predictive gains in information are small and increasing measurement precision when information is abundant.