Inter-rater Agreement Between Exposure Assessment Using Automatic Algorithms and Using Experts

The association between the incidence of postmenopausal breast cancer and occupational exposure to selected organic solvents, Montreal, Canada, 2008-2011

BACKGROUND

Breast cancer is the most diagnosed cancer among women and recognized risk factors explain 25%-47% of cases. Organic solvents are used widely in the workplace and exposure may increase the risk of developing breast cancer, yet there are insufficient data to confirm this hypothesis. We sought to determine whether past occupational exposures to selected organic solvents were associated with the incidence of invasive breast cancer in postmenopausal women in Montréal, Canada.

METHODS

From a population-based case-control study (2008-2011), using in-depth interviews we elicited information on risk factors and lifetime occupational histories. Industrial hygienists and chemists translated job descriptions into specific chemical and physical exposures. We assessed 11 individual solvents and four solvent groups. Unconditional logistic regression was used to estimate adjusted odds ratios (ORs) and 95% confidence intervals (CIs) for metrics of past exposures to selected solvents. Exposure metrics included any previous exposure, average frequency in hours per week, duration in years, and average cumulative concentration weighted by hours per workweek exposed.

RESULTS

We enrolled 695 cases and 608 controls. We found increased ORs for average cumulative concentration of exposure to mononuclear aromatic hydrocarbons (OR: 1.52, 95% CI: 1.04, 2.28), chlorinated alkanes (OR: 2.42, 95% CI: 1.23, 5.68), toluene (OR: 1.59, 95% CI: 1.02, 2.59), and a group of organic solvents with reactive metabolites (OR: 1.53, 95% CI: 1.08, 2.24). Positive associations were found across all exposure metrics and were higher among women with estrogen-positive/progesterone-negative tumors.

CONCLUSION

Our findings suggest occupational exposure to certain organic solvents may increase the risk of incident postmenopausal breast cancer.

Comparison of agreement in asthmagen exposure assessments between rule-based automatic algorithms and a job exposure matrix in healthcare workers in Australia and Bhutan

Background:

Assessment of occupational exposures is an integral component of population-based studies investigating the epidemiology of occupational diseases. However, all the available methods for exposure assessment have been developed, tested and used in high-income countries. Except for a few studies examining pesticide exposures, there is limited research on whether these methods are appropriate for assessing exposure in LMICs. The aim of this study is to compare a task-specific algorithm-based method (OccIDEAS) to a job-specific matrix method (OAsJEM) in the assessment of asthmagen exposures among healthcare workers in a high-income country and a low- and middle- income country (LMIC) to determine an appropriate assessment method for use in LMICs for future research.

Methods:

Data were obtained from a national cross-sectional survey of occupational asthmagens exposure in Australia and a cross-sectional survey of occupational chemical exposure among Bhutanese healthcare workers. Exposure was assessed using OccIDEAS and the OAsJEM. Prevalence of exposure to asthmagens and inter-rater agreement were calculated.

Results:

In Australia, the prevalence was higher for a majority of agents when assessed by OccIDEAS than by the OAsJEM (13 versus 3). OccIDEAS identified exposures to a greater number of agents (16 versus 7). The agreement as indicated by κ (Cohen’s Kappa coefficient) for six of the seven agents assessed was poor to fair (0.02 to 0.37). In Bhutan, the prevalence of exposure assessed by OccIDEAS was higher for four of the seven agents and κ was poor for all the four agents assessed (-0.06 to 0.13). The OAsJEM overestimated exposures to high-level disinfectants by assigning exposures to all participants from 10 (Bhutan) and 12 (Australia) ISCO-88 codes; whereas OccIDEAS assigned exposures to varying proportions of participants from these ISCO-codes.

Conclusion:

There was poor to fair agreement in the assessment of asthmagen exposure in healthcare workers between the two methods. The OAsJEM overestimated the prevalence of certain exposures. As compared to the OAsJEM, OccIDEAS appeared to be more appropriate for evaluating cross-country exposures to asthmagens in healthcare workers due to its inherent quality of assessing task-based determinants and its versatility in being adaptable for use in different countries with different exposure circumstances.

Testing and Validating Semi-automated Approaches to the Occupational Exposure Assessment of Polycyclic Aromatic Hydrocarbons

INTRODUCTION

When it is not possible to capture direct measures of occupational exposure or conduct biomonitoring, retrospective exposure assessment methods are often used. Among the common retrospective assessment methods, assigning exposure estimates by multiple expert rater review of detailed job descriptions is typically the most valid, but also the most time-consuming and expensive. Development of screening protocols to prioritize a subset of jobs for expert rater review can reduce the exposure assessment cost and time requirement, but there is often little data with which to evaluate different screening approaches. We used existing job-by-job exposure assessment data (assigned by consensus between multiple expert raters) from a large, population-based study of women to create and test screening algorithms for polycyclic aromatic hydrocarbons (PAHs) that would be suitable for use in other population-based studies.

METHODS

We evaluated three approaches to creating a screening algorithm: a machine-learning algorithm, a set of a priori decision rules created by experts based on features (such as keywords) found in the job description, and a hybrid algorithm incorporating both sets of criteria. All coded jobs held by mothers of infants participating in National Birth Defects Prevention Study (NBDPS) (n = 35,424) were used in developing or testing the screening algorithms. The job narrative fields considered for all approaches included job title, type of product made by the company, main activities or duties, and chemicals or substances handled. Each screening approach was evaluated against the consensus rating of two or more expert raters.

RESULTS

The machine-learning algorithm considered over 30,000 keywords and industry/occupation codes (separate and in combination). Overall, the hybrid method had a similar sensitivity (87.1%) as the expert decision rules (85.5%) but was higher than the machine-learning algorithm (67.7%). Specificity was best in the machine-learning algorithm (98.1%), compared to the expert decision rules (89.2%) and hybrid approach (89.1%). Using different probability cutoffs in the hybrid approach resulted in improvements in sensitivity (24-30%), without the loss of much specificity (7-18%).

CONCLUSION

Both expert decision rules and the machine-learning algorithm performed reasonably well in identifying the majority of jobs with potential exposure to PAHs. The hybrid screening approach demonstrated that by reviewing approximately 20% of the total jobs, it could identify 87% of all jobs exposed to PAHs; sensitivity could be further increased, albeit with a decrease in specificity, by adjusting the algorithm. The resulting screening algorithm could be applied to other population-based studies of women. The process of developing the algorithm also provides a useful illustration of the strengths and potential pitfalls of these approaches to developing exposure assessment algorithms.

OccIDEAS: web-based assessment of occupational agent exposure

Most epidemiological studies of disease aetiology do not consider potential risk factors at work. This may be because work is a complex exposure: people usually have a series of different jobs over their working lifetime; within each job there may be a range of different tasks; and there may be a variety of exposures in each job. Because of this complexity, many epidemiologists do not have the expertise or training to assess occupational exposures accurately. Our web-based application, OccIDEAS, manages the process of occupational agent assessment in epidemiological studies. The epidemiologist chooses the agents of interest for the study and OccIDEAS provides an online set of questionnaires that are tailored to those agents. The participant is asked specific questions about their job and evidence-based algorithms provide an assessment of exposure to each agent. OccIDEAS puts the world’s best occupational epidemiological expertise within reach of any researcher.

Using Decision Rules to Assess Occupational Exposure in Population-Based Studies

PURPOSE OF REVIEW

Population-based studies increasingly link task-based occupational questionnaire responses collected from subjects to exposure estimates via transparent, programmable decision rules. We reviewed recent applications and methodological developments of rule-based approaches.

RECENT FINDINGS

Agent-specific decision rules require interviews incorporating work-task-based questions. Some studies have developed rules before the interviews took place, while others developed rules after the interviews were completed. Agreement between rule-based estimates and exposures assigned using job-by-job expert review were generally moderate to good (Kappa = 0.4-0.8). Rules providing quantitative intensity levels using measurement data or that integrate multiple independent exposure sources for the same job represent further advances to improve the characterization of occupational exposures in population studies. Decision rules have provided transparent and reproducible assessments, reduce job-by-job review, and facilitate sensitivity analyses in epidemiologic studies. Future studies should consider the development of decision rules concurrent with the questionnaire design to facilitate occupational exposure assessment efforts.