What do measures of agreement (κ) tell us about quality of exposure assessment? Theoretical analysis and numerical simulation

Does recall bias explain the association of mood disorders with workplace harassment?

Purpose

To determine the contribution of recall bias to the observed excess in mental ill-health in those reporting harassment at work.

Methods

A prospective cohort of 1885 workers in welding and electrical trades was contacted every six months for up to 5 years, asking whether they were currently anxious or depressed and whether this was made worse by work. Only at the end of the study did we ask about any workplace harassment they had experienced at work. We elicited sensitivity and specificity of self-reported bullying from published reliability studies and formulated priors that reflect the possibility of over-reporting of workplace harassment (exposure) by those whose anxiety or depression was reported to be made worse by work (cases). We applied the resulting misclassification models to probabilistic bias analysis (PBA) of relative risks.

Results

We observe that PBA implies that it is unlikely that biased misclassification due to the study subjects' states of mind could have caused the entire observed association. Indeed, the results demonstrated that doubling of risk of anxiety or depression following workplace harassment is plausible, with the unadjusted relative risk attenuated with understated uncertainty.

Conclusions

It seems unlikely that risk of anxiety or depression following workplace harassment can be explained by the form of recall bias that we proposed.

Impact of Variability in Job Coding on Reliability in Exposure Estimates Obtained via a Job-Exposure Matrix

OBJECTIVES

The use of a job-exposure matrix (JEM) to assess exposure to potential health hazards in occupational epidemiological studies requires coding each participant's job history to a standard occupation and/or industry classification system recognized by the JEM. The objectives of this study were to assess the impact of inter-coder variability in job coding on reliability in exposure estimates derived from linking the job codes to the Canadian job-exposure matrix (CANJEM) and to identify influent parameters.

METHOD

Two trained coders independently coded 1000 jobs sampled from a population-based case-control study to the ISCO-1968 occupation classification at the five-digit resolution level, of which 859 could be linked to CANJEM using both assigned codes. Each of the two sets of codes was separately linked to CANJEM and thereby generated, for each of the 258 occupational agents available in CANJEM, two exposure estimates: exposure status (yes/no) and intensity of exposure (low, medium, and high) for exposed jobs only. Then, inter-rater reliability (IRR) was computed (i) after stratifying agents in 4 classes depending, for each, on the proportion of occupation codes in CANJEM defined as 'exposed' and (ii) for two additional scenarios restricted to jobs coded differently: the first one using experts' codes, the other one using codes randomly selected. IRR was computed using Cohen's kappa, PABAK and Gwet's AC1 index for exposure status, and weighted kappa and Gwet's AC2 for exposure intensity.

RESULTS

Across all agents and based on all jobs, median (Q1, Q3; Nagents) values were 0.68 (0.59, 0.75; 220) for kappa, 0.99 (0.95, 1.00; 258) for PABAK, and 0.99 (0.97, 1.00; 258) for AC1. For the additional scenarios, median kappa was 0.28 (0.00, 0.45; 209) and -0.01 (-0.02, 00; 233) restricted to jobs coded differently using experts' and random codes, respectively. A similar decreasing pattern was observed for PABAK and AC1 albeit with higher absolute values. Median kappa remained stable across exposure prevalence classes but was more variable for low prevalent agents. PABAK and AC1 decreased with increasing prevalence. Considering exposure intensity and all exposed jobs, median values were 0.79 (0.68, 0.91; 96) for weighted kappa, and 0.95 (0.89, 0.99; 102) for AC2. For the additional scenarios, median kappa was, respectively, 0.28 (-0.04, 0.42) and -0.05 (-0.18, 0.09) restricted to jobs coded differently using experts' and random codes, with a similar though attenuated pattern for AC2.

CONCLUSION

Despite reassuring overall reliability results, our study clearly demonstrated the loss of information associated with jobs coded differently. Especially, in cases of low exposure prevalence, efforts should be made to reliably code potentially exposed jobs.

Systematic review of biomonitoring data on occupational exposure to hexavalent chromium

Occupational exposure to hexavalent chromium (Cr(VI)) can cause serious adverse health effects such as lung cancer and irritation of the skin and airways. Although assessment of chromium (Cr) in urine is not specific for Cr(VI) exposure, the total amount of Cr in urine is the most used marker of exposure for biomonitoring of Cr(VI). The purpose of this systematic review was fourfold: (1) to assess current and recent biomonitoring levels in subjects occupationally exposed to Cr(VI), with a focus on urinary Cr levels at the end of a working week, (2) to identify variables influencing these biomonitoring levels, (3) to identify how urinary Cr levels correlate with other Cr(VI) exposure markers and (4) to identify gaps in the current research. To address these purposes, unpublished and published biomonitoring data were consulted: (i) unpublished biomonitoring data comprised urinary Cr levels (n = 3799) of workers from different industries in Belgium collected during 1998-2018, in combination with expert scores indicating jobs with Cr exposure and (ii) published biomonitoring data was extracted by conducting a systematic literature review. A linear mixed effect model was applied on the unpublished biomonitoring data, showing a decreasing time trend of 30% in urinary Cr levels. Considering the observed decreasing time trend, only articles published between January 1, 2010 and September 30, 2020 were included in the systematic literature search to assess current and recent biomonitoring levels. Twenty-five studies focusing on human biomonitoring of exposure to Cr(VI) in occupational settings were included. Overall, the results showed a decreasing time trend in urinary Cr levels and the need for more specific Cr(VI) biomarkers. Furthermore, this review indicated the importance of improved working conditions, efficient use of personal protective equipment, better exposure control and increased risk awareness to reduce Cr levels in biological matrices. Further investigation of the contribution of the different exposure routes is needed, so that better guidance on the use of control measures can be provided. In addition, this review support the call for more harmonization of human biomonitoring.

Correction of odds ratios in case-control studies for exposure misclassification with partial knowledge of the degree of agreement among experts who assessed exposures

OBJECTIVES

Estimates of association between exposures and diseases are often distorted by error in exposure classification. When the validity of exposure assessment is known, this can be used to adjust these estimates. When exposure is assessed by experts, even if validity is not known, we sometimes have information about interrater reliability. We present a Bayesian method for translating the knowledge of interrater reliability, which is often available, into knowledge about validity, which is often needed but not directly available, and applying this to correct odds ratios (OR).

METHODS

The method allows for inclusion of observed potential confounders in the analysis, as is common in regression-based control for confounding. Our method uses a novel type of prior on sensitivity and specificity. The approach is illustrated with data from a case-control study of lung cancer risk and occupational exposure to diesel engine emissions, in which exposure assessment was made by detailed job history interviews with study subjects followed by expert judgement.

RESULTS

Using interrater agreement measured by kappas (κ), we estimate sensitivity and specificity of exposure assessment and derive misclassification-corrected confounder-adjusted OR. Misclassification-corrected and confounder-adjusted OR obtained with the most defensible prior had a posterior distribution centre of 1.6 with 95% credible interval (Crl) 1.1 to 2.6. This was on average greater in magnitude than frequentist point estimate of 1.3 (95% Crl 1.0 to 1.7).

CONCLUSIONS

The method yields insights into the degree of exposure misclassification and appears to reduce attenuation bias due to misclassification of exposure while the estimated uncertainty increased.

Developing a job-exposure matrix with exposure uncertainty from expert elicitation and data modeling

Job exposure matrices (JEMs) are tools used to classify exposures for job titles based on general job tasks in the absence of individual level data. However, exposure uncertainty due to variations in worker practices, job conditions, and the quality of data has never been quantified systematically in a JEM. We describe a methodology for creating a JEM which defines occupational exposures on a continuous scale and utilizes elicitation methods to quantify exposure uncertainty by assigning exposures probability distributions with parameters determined through expert involvement. Experts use their knowledge to develop mathematical models using related exposure surrogate data in the absence of available occupational level data and to adjust model output against other similar occupations. Formal expert elicitation methods provided a consistent, efficient process to incorporate expert judgment into a large, consensus-based JEM. A population-based electric shock JEM was created using these methods, allowing for transparent estimates of exposure.

Wishful Thinking? Inside the Black Box of Exposure Assessment

BACKGROUND

Decision-making processes used by experts when undertaking occupational exposure assessment are relatively unknown, but it is often assumed that there is a common underlying method that experts employ. However, differences in training and experience of assessors make it unlikely that one general method for expert assessment would exist. Therefore, there are concerns about formalizing, validating, and comparing expert estimates within and between studies that are difficult, if not impossible, to characterize. Heuristics on the other hand (the processes involved in decision making) have been extensively studied. Heuristics are deployed by everyone as short-cuts to make the often complex process of decision-making simpler, quicker, and less burdensome. Experts' assessments are often subject to various simplifying heuristics as a way to reach a decision in the absence of sufficient data. Therefore, investigating the underlying heuristics or decision-making processes involved may help to shed light on the 'black box' of exposure assessment.

METHODS

A mixed method study was conducted utilizing both a web-based exposure assessment exercise incorporating quantitative and semiqualitative elements of data collection, and qualitative semi-structured interviews with exposure assessors. Qualitative data were analyzed using thematic analysis.

RESULTS

Twenty-five experts completed the web-based exposure assessment exercise and 8 of these 25 were randomly selected to participate in the follow-up interview. Familiar key themes relating to the exposure assessment exercise emerged; 'intensity'; 'probability'; 'agent'; 'process'; and 'duration' of exposure. However, an important aspect of the detailed follow-up interviews revealed a lack of structure and order with which participants described their decision making. Participants mostly described some form of an iterative process, heavily relying on the anchoring and adjustment heuristic, which differed between experts.

CONCLUSION

In spite of having undertaken comparable training (in occupational hygiene or exposure assessment), experts use different methods to assess exposure. Decision making appears to be an iterative process with heavy reliance on the key heuristic of anchoring and adjustment. Using multiple experts to assess exposure while providing some form of anchoring scenario to build from, and additional training in understanding the impact of simple heuristics on the process of decision making, is likely to produce a more methodical approach to assessment; thereby improving consistency and transparency in expert exposure assessment.

Effects of non-differential exposure misclassification on false conclusions in hypothesis-generating studies

Despite the theoretical success of obviating the need for hypothesis-generating studies, they live on in epidemiological practice. Cole asserted that “… there is boundless number of hypotheses that could be generated, nearly all of them wrong” and urged us to focus on evaluating “credibility of hypothesis”. Adopting a Bayesian approach, we put this elegant logic into quantitative terms at the study planning stage for studies where the prior belief in the null hypothesis is high (i.e., “hypothesis-generating” studies). We consider not only type I and II errors (as is customary) but also the probabilities of false positive and negative results, taking into account typical imperfections in the data. We concentrate on a common source of imperfection in the data: non-differential misclassification of binary exposure classifier. In context of an unmatched case-control study, we demonstrate—both theoretically and via simulations—that although non-differential exposure misclassification is expected to attenuate real effect estimates, leading to the loss of ability to detect true effects, there is also a concurrent increase in false positives. Unfortunately, most investigators interpret their findings from such work as being biased towards the null rather than considering that they are no less likely to be false signals. The likelihood of false positives dwarfed the false negative rate under a wide range of studied settings. We suggest that instead of investing energy into understanding credibility of dubious hypotheses, applied disciplines such as epidemiology, should instead focus attention on understanding consequences of pursuing specific hypotheses, while accounting for the probability that the observed “statistically significant” association may be qualitatively spurious.

Agreement of experts and non-experts in a desktop exercise evaluating exposure to asthmagens in the cotton and textile, and other industries

In the absence of personal exposure measurements, expert assessment, generally on a case-by-case basis, is often used to estimate exposures. However, the decision processes of individual experts when making assessments are unknown, making it difficult to assess the quality of these assessments or to compare different assessments to each other. We conducted a study in primarily the textile and cotton industries, but also in baking, metal work, and agriculture industries in which we assessed agreement between experts assessing intensity and probability of exposure in the absence of exposure measurements to compare how well their performance compares to agreement of non-desktop-based exercises reported in literature. In addition, agreement was compared with that of non-experts undertaking the same exercise, and results were further stratified to assess the impact of factors expected of affected assessments. Intraclass correlation coefficients of absolute agreement (ICC1) and consistency (ICC3) between raters were calculated. Sensitivity and specificity were estimated using a probabilistic simulation methodology developed previously. Fourteen occupational hygienists and exposure assessors with complete data for all 48 job descriptions and 8 non-experts participated. Although confidence intervals about correlation-coefficient differences are not reported, the individual limits were found to be so broad as to suggest that no statistically significant comparisons can be made. Nevertheless, preliminary observations are presented here as suggested by the computed means. Absolute agreement between expert raters was fair-good, but was somewhat better for intensity (ICC1 = 0.61) than for probability (ICC1 = 0.44) of exposure and was better for experts than non-experts. Estimated sensitivity was 0.95 and specificity 0.82 for intensity, and 0.91 and 0.78 for probability of exposure, respectively. Stratification for factors hypothesized to affect agreement did not show statistically significant differences, but consistent patterns of point estimates indicated that agreement between raters (both expert on non-experts) dropped for medium levels of information compared with little or extensive information. Inclusion of a photo or video generally improved agreement between experts but not between non-experts, whereas the year of the job description had no influence on the assessments. These data indicate that the desktop exposure assessment exercise was of similar quality to previously reported levels of agreement. Agreements between experts' assessments were independent of the time period of the job and can be improved by inclusion of visual material. Agreement between experts as well as the non-experts does not increase with the detail of provided job information.