Inter-rater agreement in the assessment of exposure to carcinogens in the offshore petroleum industry

Exposure to fibres and risk of pleural mesothelioma in the Norwegian Offshore Petroleum Workers cohort

OBJECTIVES

Pleural mesothelioma is a rare respiratory cancer, mainly caused by inhalation of asbestos fibres. Other inorganic fibres are also suggested risk factors. We aimed to investigate the association between exposure to asbestos or refractory ceramic fibres (RCFs) and pleural mesothelioma among male Norwegian offshore petroleum workers.

METHODS

Among 25 347 men in the Norwegian Offshore Petroleum Workers (NOPW) cohort (1965-1998), 43 pleural mesothelioma cases were identified through the Cancer Registry of Norway (1999-2022). A case-cohort study was conducted with 2095 randomly drawn non-cases from the cohort. Asbestos and RCF exposures were assessed with expert-made job-exposure matrices (JEMs). Weighted Cox regression was used to estimate HRs and 95% CIs, adjusted for age at baseline and pre-offshore employment with likely asbestos exposure.

RESULTS

An increased risk of pleural mesothelioma was indicated for the highest versus lowest tertile of average intensity of asbestos (HR=1.21, 95% CI: 0.57 to 2.54). Pre-offshore asbestos exposure (vs no such exposure) was associated with increased risk of pleural mesothelioma (HR=2.06, 95% CI: 1.11 to 3.81). For offshore workers with no pre-offshore asbestos exposure, an increased risk of pleural mesothelioma was found for the highest tertile of average intensity of asbestos (HR=4.13, 95% CI: 0.93 to 18), versus the lowest tertile. No associations were found between RCF and pleural mesothelioma.

CONCLUSIONS

Associations between JEM-based offshore asbestos exposure and pleural mesothelioma were confirmed in the NOPW cohort. Pleural mesothelioma risk was also associated with asbestos exposure before work in the offshore petroleum industry.

Benzene exposure and risk of lung cancer in the Norwegian Offshore Petroleum Worker cohort: a prospective case-cohort study

OBJECTIVE

The objective of our study was to examine whether occupational exposure to benzene is associated with lung cancer among males in the Norwegian Offshore Petroleum Workers cohort.

METHODS

Among 25 347 male offshore workers employed during 1965-1998, we conducted a case-cohort study with 399 lung cancer cases diagnosed between 1999 and 2021, and 2035 non-cases sampled randomly by 5-year birth cohorts. Individual work histories were coupled to study-specific job-exposure matrices for benzene and other known lung carcinogens. Weighted Cox regression was used to estimate HRs and 95% CIs for the associations between benzene exposure and lung cancer, by major histological subtypes, adjusted for age, smoking and occupational exposure to welding fumes, asbestos and crystalline silica. Missing data were imputed.

RESULTS

For lung cancer (all subtypes combined), HRs (95% CIs) for the highest quartiles of benzene exposure versus unexposed were 1.15 (0.61 to 2.35) for cumulative exposure, 1.43 (0.76 to 2.69) for duration, and 1.22 (0.68 to 2.18) for average intensity (0.280≤P-trend≤0.741). For 152 adenocarcinoma cases, a positive trend was observed for exposure duration (P-trend=0.044).

CONCLUSIONS

In this cohort of offshore petroleum workers generally exposed to low average levels of benzene, we did not find an overall clear support for an association with lung cancer (all subtypes combined), although an association was suggested for duration of benzene exposure and adenocarcinoma. The limited evidence might be due to restricted statistical power.

Night shift work and risk of aggressive prostate cancer in the Norwegian Offshore Petroleum Workers (NOPW) cohort

BACKGROUND

Night shift work may acutely disrupt the circadian rhythm, with possible carcinogenic effects. Prostate cancer has few established risk factors though night shift work, a probable human carcinogen, may increase the risk. We aimed to study the association between night shift work and chlorinated degreasing agents (CDAs) as possible endocrine disrupters in relation to aggressive prostate cancer as verified malignancies.

METHODS

We conducted a case-cohort study on 299 aggressive prostate cancer cases and 2056 randomly drawn non-cases in the Norwegian Offshore Petroleum Workers cohort (1965-98) with linkage to the Cancer Registry of Norway (1953-2019). Work history was recorded as years with day, night, and rollover (rotating) shift work, and CDA exposure was assessed with expert-made job-exposure matrices. Weighted Cox regression was used to estimate hazard ratios (HRs) and 95% confidence intervals (CIs) for aggressive prostate cancer, adjusted for education and year of first employment, stratified by 10-year birth cohorts, and with 10, 15, and 20 years of exposure lag periods.

RESULTS

Compared with day work only, an increased hazard of aggressive prostate cancer (HR = 1.86, 95% CI 1.18-2.91; P-trend = 0.046) was found in workers exposed to ≥19.5 years of rollover shift work. This persisted with longer lag periods (HR = 1.90, 95% CI 0.92-3.95; P-trend = 0.007). The exposure-hazard curve for a non-linear model increased linearly (HRs ≥1.00) for 18-26 years of rollover shift work. No association was found with CDA exposure.

CONCLUSIONS

Long-term exposure to rollover shift work may increase the hazard of aggressive prostate cancer in offshore petroleum workers.

Night shift work, chemical coexposures and risk of female breast cancer in the Norwegian Offshore Petroleum Workers (NOPW) cohort: a prospectively recruited case-cohort study

OBJECTIVES

This study examined the association between night shift work and risk of breast cancer, overall and by hormone receptor subtype, among females in the Norwegian Offshore Petroleum Workers (NOPW) cohort. We also examined the association of coexposure (chlorinated degreasers and benzene) and breast cancer risk, and possible interaction with work schedule.

DESIGN

Prospectively recruited case-cohort study within the NOPW cohort.

SETTING

Female offshore petroleum workers active on the Norwegian continental shelf.

PARTICIPANTS

600 female workers (86 cases and 514 non-cases) were included in the study. We excluded workers that died or emigrated before start of follow-up, had missing work history, were diagnosed with breast cancer or other prior malignancy (except non-melanoma skin cancer) before start of follow-up.

RESULTS

No overall association was found between breast cancer risk and work schedule (HR 0.87, 95% CI 0.52 to 1.46 for work schedule involving night shift vs day shift only). There was no significant association between work schedule and risk of any breast cancer subtype. No significant interactions were found between work schedule and chemical coexposures (breast cancer overall P_{interaction chlorinated degreasers}=0.725 and P_{interaction benzene}=0.175).

CONCLUSIONS

Our results did not provide supporting evidence that work schedule involving night shift affects breast cancer risk in female offshore petroleum workers, but should be considered cautiously due to few cases. Further studies with larger sample sizes are warranted.

Validity of retrospective occupational exposure estimates of lead and manganese in a case-control study

OBJECTIVES

The validity of surrogate measures of retrospective occupational exposure in population-based epidemiological studies has rarely been evaluated. Using toenail samples as bioindicators of exposure, we assessed whether work tasks and expert assessments of occupational metal exposure obtained from personal interviews were associated with lead and manganese concentrations.

METHODS

We selected 609 controls from a case-control study of bladder cancer in New England who had held a job for ≥1 year 8-24 months prior to toenail collection. We evaluated associations between toenail metal concentrations and five tasks extracted from occupational questionnaires (grinding, painting, soldering, welding, working near engines) using linear regression models. For 139 subjects, we also evaluated associations between the toenail concentrations and exposure estimates from three experts.

RESULTS

We observed a 1.9-fold increase (95% CI 1.4 to 2.5) in toenail lead concentrations with painting and 1.4-fold increase (95% CI 1.1 to 1.7) in manganese concentrations with working around engines and handling fuel. We observed significant trends with increasing frequency of both activities. For lead, significant trends were observed with the ratings from all three experts. Their average ratings showed the strongest association, with subjects rated as possibly or probably exposed to lead having concentrations that were 2.0 and 2.5 times higher, respectively, than in unexposed subjects (ptrend <0.001). Expert estimates were only weakly associated with manganese toenail concentrations.

CONCLUSIONS

Our findings support the ability of experts to identify broad contrasts in previous occupational exposure to lead. The stronger associations with task frequency and expert assessments support using refined exposure characterisation whenever possible.

Estimation of Lead Exposure Prevalence in Korean Population through Combining Multiple Experts' Judgment based on Objective Data Sources

Objective

Estimating carcinogen exposure prevalence is important for preventing occupational cancers. To develop the Korean version of CARcinogen EXposure (CAREX), a carcinogen surveillance system used in many countries, we estimated lead exposure prevalence in the Korean working population.

Methods

We used three Korean nationwide data sources to obtain objective database-derived prevalences of lead exposure across industries: airborne lead measurement data from the work environment measurement database (WEMD), blood lead measurement data from the special health examination database (SHED), and lead exposure prevalence computed using data from the work environment condition survey (WECS), which is a nationwide occupational exposure survey. We also asked a panel of 52 experts with ≥20 years of experience in industrial hygiene practice for their judgment about lead exposure prevalence across industries after they reviewed the database-derived prevalences computed from the three exposure databases. We developed and compared various estimation methods for combining the experts' judgments. The 2010 census was used as the reference population to estimate the number of lead-exposed workers in 228 industries by multiplying the exposure prevalence by the number of workers in each industry.

Results

The database-derived prevalences of lead exposure in the 228 industries were calculated using data collected between 2009 and 2011 from the WEMD and SHED and from the 2009 WECS. From the various estimation methods assessed, the median values of experts' responses were selected as our estimates of lead exposure prevalence in each industry. As a result, it was estimated that 129,250 Korean workers were exposed to lead in 2010.

Conclusions

Based on objective databases, we developed a method for estimating exposure prevalence for the CAREX system by combining experts' judgments. This work may offer an unbiased approach to the development process that accounts for the uncertainty in exposure.

Aromatic hydrocarbons and risk of skin cancer by anatomical site in 25 000 male offshore petroleum workers

BACKGROUND

We prospectively examined skin cancer risk according to occupational exposure to aromatic hydrocarbons with adjustment for ultraviolet radiation exposure, in a cohort of 24 917 male offshore petroleum workers.

METHODS

Hazard ratios (HRs) and 95% confidence intervals were estimated with Cox regression adapted to a stratified case-cohort design.

RESULTS

During 13.5 years of follow-up, 112 cutaneous melanomas (CMs) and 70 non-melanoma skin cancers were identified. Increased risks of CM and of squamous cell carcinoma on the forearm and hand were seen among those ever exposed to crude oil or benzene. For skin cancers of the forearm and hand combined, cumulative and duration metrics of benzene exposure showed P_trends of 0.031 and 0.003, respectively.

CONCLUSIONS

Our results support an association between exposure to crude oil or benzene and skin cancer risk on hands and forearms among offshore petroleum workers. Dermal uptake of polycyclic aromatic hydrocarbons or benzene may explain this association.

Combining Decision Rules from Classification Tree Models and Expert Assessment to Estimate Occupational Exposure to Diesel Exhaust for a Case-Control Study

OBJECTIVES

To efficiently and reproducibly assess occupational diesel exhaust exposure in a Spanish case-control study, we examined the utility of applying decision rules that had been extracted from expert estimates and questionnaire response patterns using classification tree (CT) models from a similar US study.

METHODS

First, previously extracted CT decision rules were used to obtain initial ordinal (0-3) estimates of the probability, intensity, and frequency of occupational exposure to diesel exhaust for the 10 182 jobs reported in a Spanish case-control study of bladder cancer. Second, two experts reviewed the CT estimates for 350 jobs randomly selected from strata based on each CT rule's agreement with the expert ratings in the original study [agreement rate, from 0 (no agreement) to 1 (perfect agreement)]. Their agreement with each other and with the CT estimates was calculated using weighted kappa (κ w) and guided our choice of jobs for subsequent expert review. Third, an expert review comprised all jobs with lower confidence (low-to-moderate agreement rates or discordant assignments, n = 931) and a subset of jobs with a moderate to high CT probability rating and with moderately high agreement rates (n = 511). Logistic regression was used to examine the likelihood that an expert provided a different estimate than the CT estimate based on the CT rule agreement rates, the CT ordinal rating, and the availability of a module with diesel-related questions.

RESULTS

Agreement between estimates made by two experts and between estimates made by each of the experts and the CT estimates was very high for jobs with estimates that were determined by rules with high CT agreement rates (κ w: 0.81-0.90). For jobs with estimates based on rules with lower agreement rates, moderate agreement was observed between the two experts (κ w: 0.42-0.67) and poor-to-moderate agreement was observed between the experts and the CT estimates (κ w: 0.09-0.57). In total, the expert review of 1442 jobs changed 156 probability estimates, 128 intensity estimates, and 614 frequency estimates. The expert was more likely to provide a different estimate when the CT rule agreement rate was <0.8, when the CT ordinal ratings were low to moderate, or when a module with diesel questions was available.

CONCLUSIONS

Our reliability assessment provided important insight into where to prioritize additional expert review; as a result, only 14% of the jobs underwent expert review, substantially reducing the exposure assessment burden. Overall, we found that we could efficiently, reproducibly, and reliably apply CT decision rules from one study to assess exposure in another study.

[Offshore work and the work of nurses on board: an integrative review]

OBJECTIVE

To know the production of theoretical approaches on issues related to offshore work and the work of offshore nurses.

METHOD

Integrative literature review conducted in the databases of LILACS, BDENF, MEDLINE, SciELO and Index PSI.

RESULTS

We selected 33 studies published in national and international journals between 1997 and 2014. The thematic analysis corpus resulted in four central themes: offshore work environment; amid work adversities, an escape; structuring of offshore health and safety services; in search of safe practices.

CONCLUSION

This study contributes to the offshore work of nurses in relation to the nature of work, acting amid adversities and the restless search for safe practices in the open sea.

Reliability and validity of expert assessment based on airborne and urinary measures of nickel and chromium exposure in the electroplating industry

The reliability and validity of six experts' exposure ratings were evaluated for 64 nickel-exposed and 72 chromium-exposed workers from six Shanghai electroplating plants based on airborne and urinary nickel and chromium measurements. Three industrial hygienists and three occupational physicians independently ranked the exposure intensity of each metal on an ordinal scale (1-4) for each worker's job in two rounds: the first round was based on responses to an occupational history questionnaire and the second round also included responses to an electroplating industry-specific questionnaire. The Spearman correlation (r(s)) was used to compare each rating's validity to its corresponding subject-specific arithmetic mean of four airborne or four urinary measurements. Reliability was moderately high (weighted kappa range=0.60-0.64). Validity was poor to moderate (r(s)=-0.37-0.46) for both airborne and urinary concentrations of both metals. For airborne nickel concentrations, validity differed by plant. For dichotomized metrics, sensitivity and specificity were higher based on urinary measurements (47-78%) than airborne measurements (16-50%). Few patterns were observed by metal, assessment round, or expert type. These results suggest that, for electroplating exposures, experts can achieve moderately high agreement and (reasonably) distinguish between low and high exposures when reviewing responses to in-depth questionnaires used in population-based case-control studies.

Agreement of experts and non-experts in a desktop exercise evaluating exposure to asthmagens in the cotton and textile, and other industries

In the absence of personal exposure measurements, expert assessment, generally on a case-by-case basis, is often used to estimate exposures. However, the decision processes of individual experts when making assessments are unknown, making it difficult to assess the quality of these assessments or to compare different assessments to each other. We conducted a study in primarily the textile and cotton industries, but also in baking, metal work, and agriculture industries in which we assessed agreement between experts assessing intensity and probability of exposure in the absence of exposure measurements to compare how well their performance compares to agreement of non-desktop-based exercises reported in literature. In addition, agreement was compared with that of non-experts undertaking the same exercise, and results were further stratified to assess the impact of factors expected of affected assessments. Intraclass correlation coefficients of absolute agreement (ICC1) and consistency (ICC3) between raters were calculated. Sensitivity and specificity were estimated using a probabilistic simulation methodology developed previously. Fourteen occupational hygienists and exposure assessors with complete data for all 48 job descriptions and 8 non-experts participated. Although confidence intervals about correlation-coefficient differences are not reported, the individual limits were found to be so broad as to suggest that no statistically significant comparisons can be made. Nevertheless, preliminary observations are presented here as suggested by the computed means. Absolute agreement between expert raters was fair-good, but was somewhat better for intensity (ICC1 = 0.61) than for probability (ICC1 = 0.44) of exposure and was better for experts than non-experts. Estimated sensitivity was 0.95 and specificity 0.82 for intensity, and 0.91 and 0.78 for probability of exposure, respectively. Stratification for factors hypothesized to affect agreement did not show statistically significant differences, but consistent patterns of point estimates indicated that agreement between raters (both expert on non-experts) dropped for medium levels of information compared with little or extensive information. Inclusion of a photo or video generally improved agreement between experts but not between non-experts, whereas the year of the job description had no influence on the assessments. These data indicate that the desktop exposure assessment exercise was of similar quality to previously reported levels of agreement. Agreements between experts' assessments were independent of the time period of the job and can be improved by inclusion of visual material. Agreement between experts as well as the non-experts does not increase with the detail of provided job information.

Expert assessment of historical crop specific pesticide use in the Netherlands

OBJECTIVES

Retrospective assessment of environmental pesticide exposure is challenging. Exposure measurements or information on crop-specific pesticide use are often lacking historically. We applied expert assessment to reconstruct historical pesticide use patterns in the Netherlands, and evaluated reliability and accuracy of this procedure.

METHODS

For six main crops in the Netherlands, two experts per crop individually rated the probability (percentage of farmers applying) and frequency of use of authorised active ingredients between 1961 and 2005 per 5-year period. Inter-rater agreement was investigated by the percentage overall agreement and weighted Cohen's κ's (κ(w)). Experts' ratings were compared with self-reported pesticide use from recent farmer surveys to determine accuracy of the ratings.

RESULTS

Inter-rater agreement on the probability of use varied between crops (κ(w) 0.25 to 0.69), as well as agreement on the frequency of use (κ(w) 0.32 to 0.64). Inter-rater agreement was marginally higher for herbicides and fungicides than insecticides. Comparisons with survey data indicated fair to good accuracy of the experts' ratings for the probability (κ(w) 0.48 to 0.65) and frequency of use (κ(w) 0.38 to 0.68). For all crops except fruit, the specificity of the experts' ratings was higher than the sensitivity.

CONCLUSIONS

Overall inter-rater agreement between experts was fair to good and experts' ratings were reasonably accurate. Results of this study indicate that expert assessment can be used to derive information on historical pesticide use, which is essential for epidemiological studies evaluating the effect of (past) environmental exposure to pesticides on health.

Reliability of the Advanced REACH Tool (ART)

OBJECTIVES

The aim of this study was to assess the reliability of the Advanced REACH Tool (ART) by (i) studying interassessor agreement of the resulting exposure estimates generated by the ART mechanistic model, (ii) studying interassessor agreement per model parameters of the ART mechanistic model, (iii) investigating assessor characteristics resulting in reliable estimates, and (iv) estimating the effect of training on assessor agreement.

METHODS

Prior to the 1-day workshop, participants had to assess four scenarios with the ART. During two 1-day workshops, 54 participants received 3-h training in applying the mechanistic model and the technical aspects of the web tool. Afterward, the participants assessed another four scenarios. The assessments of the participants were compared with gold standard estimates compiled by the workshop instructors. Intraclass correlation coefficients (ICCs) were calculated and per model parameter and the percentage agreement and Cohen kappa statistics were estimated.

RESULTS

The ICCs showed good agreement before and almost perfect agreement after training. However, substantial variability was observed between individual assessors' estimates for an individual scenario. After training, only 42% of the assessments lay within a factor of three of the gold standard estimate. The reliability appeared to be influenced by several factors: (i) information provided by text and video hampered the assessors gaining additional information required to make the assessments, (ii) for some parameters, the guidance documentation implemented in the tool may have been insufficient, and (iii) in some cases, the assessors were not able to implement the information explicitly provided.

CONCLUSIONS

The ART is an expert tool and extensive training is recommended prior to use. Improvements of the guidance documentation, consensus procedures, and improving the training methods could improve the reliability of ART. Nevertheless, considerable variability can be expected between assessors using ART to estimate exposure levels for a given scenario.

Comparison of algorithm-based estimates of occupational diesel exhaust exposure to those of multiple independent raters in a population-based case-control study

OBJECTIVES

Algorithm-based exposure assessments based on patterns in questionnaire responses and professional judgment can readily apply transparent exposure decision rules to thousands of jobs quickly. However, we need to better understand how algorithms compare to a one-by-one job review by an exposure assessor. We compared algorithm-based estimates of diesel exhaust exposure to those of three independent raters within the New England Bladder Cancer Study, a population-based case-control study, and identified conditions under which disparities occurred in the assessments of the algorithm and the raters.

METHODS

Occupational diesel exhaust exposure was assessed previously using an algorithm and a single rater for all 14 983 jobs reported by 2631 study participants during personal interviews conducted from 2001 to 2004. Two additional raters independently assessed a random subset of 324 jobs that were selected based on strata defined by the cross-tabulations of the algorithm and the first rater's probability assessments for each job, oversampling their disagreements. The algorithm and each rater assessed the probability, intensity and frequency of occupational diesel exhaust exposure, as well as a confidence rating for each metric. Agreement among the raters, their aggregate rating (average of the three raters' ratings) and the algorithm were evaluated using proportion of agreement, kappa and weighted kappa (κw). Agreement analyses on the subset used inverse probability weighting to extrapolate the subset to estimate agreement for all jobs. Classification and Regression Tree (CART) models were used to identify patterns in questionnaire responses that predicted disparities in exposure status (i.e., unexposed versus exposed) between the first rater and the algorithm-based estimates.

RESULTS

For the probability, intensity and frequency exposure metrics, moderate to moderately high agreement was observed among raters (κw = 0.50-0.76) and between the algorithm and the individual raters (κw = 0.58-0.81). For these metrics, the algorithm estimates had consistently higher agreement with the aggregate rating (κw = 0.82) than with the individual raters. For all metrics, the agreement between the algorithm and the aggregate ratings was highest for the unexposed category (90-93%) and was poor to moderate for the exposed categories (9-64%). Lower agreement was observed for jobs with a start year <1965 versus ≥1965. For the confidence metrics, the agreement was poor to moderate among raters (κw = 0.17-0.45) and between the algorithm and the individual raters (κw = 0.24-0.61). CART models identified patterns in the questionnaire responses that predicted a fair-to-moderate (33-89%) proportion of the disagreements between the raters' and the algorithm estimates.

DISCUSSION

The agreement between any two raters was similar to the agreement between an algorithm-based approach and individual raters, providing additional support for using the more efficient and transparent algorithm-based approach. CART models identified some patterns in disagreements between the first rater and the algorithm. Given the absence of a gold standard for estimating exposure, these patterns can be reviewed by a team of exposure assessors to determine whether the algorithm should be revised for future studies.

Comparison of two expert-based assessments of diesel exhaust exposure in a case-control study: programmable decision rules versus expert review of individual jobs

OBJECTIVES

Professional judgment is necessary to assess occupational exposure in population-based case-control studies; however, the assessments lack transparency and are time-consuming to perform. To improve transparency and efficiency, we systematically applied decision rules to questionnaire responses to assess diesel exhaust exposure in the population-based case-control New England Bladder Cancer Study.

METHODS

2631 participants reported 14 983 jobs; 2749 jobs were administered questionnaires ('modules') with diesel-relevant questions. We applied decision rules to assign exposure metrics based either on the occupational history (OH) responses (OH estimates) or on the module responses (module estimates); we then combined the separate OH and module estimates (OH/module estimates). Each job was also reviewed individually to assign exposure (one-by-one review estimates). We evaluated the agreement between the OH, OH/module and one-by-one review estimates.

RESULTS

The proportion of exposed jobs was 20-25% for all jobs, depending on approach, and 54-60% for jobs with diesel-relevant modules. The OH/module and one-by-one review estimates had moderately high agreement for all jobs (κ(w)=0.68-0.81) and for jobs with diesel-relevant modules (κ(w)=0.62-0.78) for the probability, intensity and frequency metrics. For exposed subjects, the Spearman correlation statistic was 0.72 between the cumulative OH/module and one-by-one review estimates.

CONCLUSIONS

The agreement seen here may represent an upper level of agreement because the algorithm and one-by-one review estimates were not fully independent. This study shows that applying decision-based rules can reproduce a one-by-one review, increase transparency and efficiency, and provide a mechanism to replicate exposure decisions in other studies.

Validity and reliability of exposure assessors' ratings of exposure intensity by type of occupational questionnaire and type of rater

BACKGROUND

In epidemiologic studies that rely on professional judgment to assess occupational exposures, the raters' accurate assessment is vital to detect associations. We examined the influence of the type of questionnaire, type of industry, and type of rater on the raters' ability to reliably and validly assess within-industry differences in exposure. Our aim was to identify areas where improvements in exposure assessment may be possible.

METHODS

Subjects from three foundries (n = 72) and three textile plants (n = 74) in Shanghai, China, completed an occupational history (OH) and an industry-specific questionnaire (IQ). Six total dust measurements were collected per subject and were used to calculate a subject-specific measurement mean, which was used as the gold standard. Six raters independently ranked the intensity of each subject's current job on an ordinal scale (1-4) based on the OH alone and on the OH and IQ together. Aggregate ratings were calculated for the group, for industrial hygienists, and for occupational physicians. We calculated intra-class correlation coefficients (ICCs) to evaluate the reliability of the raters. We calculated the correlation between the subject-specific measurement means and the ratings to evaluate the raters' validity. Analyses were stratified by industry, type of questionnaire, and type of rater. We also examined the agreement between the ratings by exposure category, where the subject-specific measurement means were categorized into two and four categories.

RESULTS

The reliability and validity measures were higher for the aggregate ratings than for the ratings from the individual raters. The group's performance was maximized with three raters. Both the reliability and validity measures were higher for the foundry industry than for the textile industry. The ICCs were consistently lower in the OH/IQ round than in the OH round in both industries. In contrast, the correlations with the measurement means were higher in the OH/IQ round than in the OH round for the foundry industry (group rating, OH/IQ: Spearman rho = 0.77; OH: rho = 0.64). No pattern by questionnaire type was observed for the textile industry (group rating, Spearman rho = 0.50, both assessment rounds). For both industries, the agreement by exposure category was higher when the task was reduced to discriminating between two versus four exposure categories.

CONCLUSIONS

Assessments based on professional judgment may reduce misclassification by using two or three raters, by using questionnaires that systematically collect task information, and by defining intensity categories that are distinguishable by the raters. However, few studies have the resources to use multiple raters and these additional efforts may not be adequate for obtaining valid subjective ratings. Thus, improving exposure assessment approaches for studies that rely on professional judgment remain an important research need.