Effects of prospective-user factors and sign design features on guessability of pharmaceutical pictograms

Evaluation of the design and structure of electronic medication labels to improve patient health knowledge and safety: a systematic review

INTRODUCTION

Patient misunderstanding of instructions on medication labels is a common cause of medication errors and can result in ineffective treatment. One way to better improve patient comprehension of medication labels is by optimizing the content and display of the information.

OBJECTIVES

To review comparative studies that have evaluated the design of a medication label to improve patient knowledge or safety.

METHODS

Studies were selected from systematic computerized literature searches performed in PubMed, Embase (Elsevier), Cochrane Central (EBSCO), Cumulative Index to Nursing and Allied Health Literature-CINAHL (EBSCO), and Web of Science (Thomson Reuters). Eligible studies included comparative studies that evaluated the design of a medication label to improve patient knowledge or safety.

RESULTS

Of the 246 articles identified in the primary literature search, 14 studies were selected for data abstraction. Thirteen of these studies significantly impacted the patient understanding of medication labels. Three studies included a measure of patient safety in terms of medication adherence and dosing errors. The utilization of patient-centered language, pictograms/graphics, color/white space, or font optimization was seen to have the most impact on patient comprehension.

CONCLUSION

It is essential to present medication information in an optimal manner for patients. This can be done by standardizing the content, display, and format of medication labels to improve understanding and medication usage. Evidence-based design principles can, therefore, be used to facilitate the standardization of the structure of label content for both print and electronic devices. However, more research needs to be done on validating the implications of label content display to measure its impact on patient safety.

SYSTEMIC REVIEW REGISTRATION

PROSPERO CRD42022347510 ( http://www.crd.york.ac.uk/prospero/ ).

The Effects of Animation on the Guessability of Universal Healthcare Symbols for Middle-Aged and Older Adults

OBJECTIVE

The purpose of this study was to investigate whether animation can help to improve the comprehension of universal healthcare symbols for middle-aged and older adults.

BACKGROUND

The Hablamos Juntos (HJ) healthcare symbol system is a set of widely used universal healthcare symbols that were developed in the United States. Some studies indicated that HJ healthcare symbols are not well-understood by users in non-English-speaking areas. Other studies found that animations can improve users' comprehension of complex symbols. Thus, we wanted to test whether animation could help to improve users' comprehension of HJ symbols.

METHODS

The participants included 40 middle-aged and 40 older adults in Taiwan. We redesigned the 12 HJ symbols into three visual formats-static, basic animation, and detailed animation-and compared them to find which best improved the participants' guessability scores.

RESULTS

(1) Middle-aged adults' comprehension of static and basic animated symbols was significantly better than that of older adults, but there was no significant difference in the guessability scores between the two age groups in terms of detailed animated symbols; (2) In general, both basic animation and detailed animation significantly improved the guessability score, but the effect with detailed animation was significantly greater than that with basic animation; (3) Older women were more receptive to detailed animation and showed better guessing performance.

CONCLUSION

Detailed animation contains more details and provides a more complete explanation of the concept of the static symbols, helping to improve the comprehension of HJ symbols for middle-aged and older adult users.

APPLICATION

Our findings provide a reference for the possibility of new style symbol design in the digital and aging era, which can be applied to improve symbol comprehension.

A comparative investigation of machine learning algorithms for predicting safety signs comprehension based on socio-demographic factors and cognitive sign features

This study examines whether the socio-demographic factors and cognitive sign features can be used for envisaging safety signs comprehensibility using predictive machine learning (ML) techniques. This study will determine the role of different machine learning components such as feature selection and classification to determine suitable factors for safety construction signs comprehensibility. A total of 2310 participants were requested to guess the meaning of 20 construction safety signs (four items for each of the mandatory, prohibition, emergency, warning, and firefighting signs) using the open-ended method. Moreover, the participants were asked to rate the cognitive design features of each sign in terms of familiarity, concreteness, simplicity, meaningfulness, and semantic closeness on a 0-100 rating scale. Subsequently, all eight features (age, experience, education level, familiarity, concreteness, meaningfulness, semantic closeness, and simplicity) were used for classification. Furthermore, the 14 most popular supervised classifiers were implemented and evaluated for safety sign comprehensibility prediction using these eight features. Also, filter and wrapper methods were used as feature selection techniques. Results of feature selection techniques indicate that among the eight features considered in this study, familiarity, simplicity, and meaningfulness are found to be the most relevant and effective components in predicting the comprehensibility of selected safety signs. Further, when these three features are used for classification, the K-NN classifier achieves the highest classification accuracy of 94.369% followed by medium Gaussian SVM which achieves a classification accuracy of 76.075% under hold-out data division protocol. The machine learning (ML) technique was adopted as a promising approach to addressing the issue of comprehensibility, especially in terms of determining factors affecting the safety signs' comprehension. The cognitive sign features of familiarity, simplicity, and meaningfulness can provide useful information in terms of designing user-friendly safety signs.

Guessability of standard pharmaceutical pictograms in members of the Nigerian public

Background

Pharmaceutical pictograms are standardized images used to visually convey medication instructions. Very little is known about the ability of Africans to interpret these images.

Objectives

Thus, the aim of this study was to assess the guessability (ability to correctly guess meaning) of selected International Pharmaceutical Federation (FIP) and United States Pharmacopoeia (USP) pictograms in members of the Nigerian public.

Methods

A cross-sectional survey was carried out between May and August 2021 on 400 randomly sampled members of Nigerian public. Selected pictograms (24 FIP and 22 USP pictograms) were grouped and printed on A3 sheets of paper which were used to interview members of the public who fulfilled the study's' eligibility criteria. Respondents were asked to guess the meanings of either the FIP or USP pictograms, and their answers written down verbatim. Descriptive and inferential statistics were used to report the data collected.

Results

Four hundred respondents were interviewed, with 200 respondents each assessing the guessability of the FIP and USP pictograms. The guessability of assessed FIP pictograms ranged between 3.5 and 95%, while that for the USP pictograms was 27.5-97%. Eleven FIP and Thirteen USP pictograms respectively achieved the International Organization for Standardization (ISO) comprehensibility cutoff point of 67%. Guessing performance (the total number of pictograms correctly guessed by an individual) of respondents that assessed the FIP pictograms was significantly associated with their age (p = 0.044) and highest level of education completed (p = 0.003). For the USP pictograms, guessing performance was only significantly associated with the highest educational level completed (p < 0.001).

Conclusions

Guessability of both pictogram types varied widely, but the guessability of the USP pictograms was generally better than that for the FIP pictograms. Many of the tested pictograms may however need to be redesigned before they can be correctly interpreted by members of the Nigerian public.

Comprehension of Pictograms Demonstrating the Risk of Medication Use During Pregnancy: A Systematic Review

OBJECTIVE

This study aimed to summarize the comprehension of a pictogram about the risk of medication use during pregnancy.

METHODS

A systematic review was performed using the PRISMA checklist of the PubMed, Embase, Cochrane Central Register of Controlled Trials, CINAHL, PsycInfo, LILACS, Academic Search Premier, Scopus, and Web of Science databases, grey literature (Google Scholar and OpenAIRE), ClinicalTrials.gov website, and design journals and congresses. The search was performed since the database inception, without language or year of publication restrictions.

RESULTS

Twelve studies met the inclusion criteria for this review, 2 of which were randomized clinical trials. The pictograms and methods used varied widely among studies. The comprehension of the pregnancy pictograms had a complex communication outcome with a variation of 21-96%.

CONCLUSIONS FOR PRACTICE

The lack of a standard pictogram and uniform methods to evaluate the comprehension of the pregnancy pictogram made it challenging to reach a conclusion with the studies available to date on the safety and efficacy of the pregnancy pictogram to alert the risk of medication use.

Health Literacy Level and Comprehension of Prescription and Nonprescription Drug Information

The aim of this study was to investigate the level of misunderstanding of medication information in Korean adults after stratifying by level of health literacy and to identify the factors influencing the misunderstanding of medication information and reading amounts of information on OTC drug labels. A cross-sectional survey was performed with 375 adult participants using the survey instrument. Multiple linear regression analyses were performed to identify factors which influence misunderstanding of medication information. Participants misunderstood 20% of words on OTC drug labels, 9% of prescription drug instructions, and 9% of pictograms. Participants on average read 59% of the overall contents of the OTC drug labels. As prescription drugs’ dosing regimens became more complicated, the level of misunderstanding instructions increased. The level of misunderstanding words on OTC drug labels significantly decreased as participants had adequate health literacy (β = −18.11, p < 0.001) and higher education levels (β = −6.83, p < 0.001), after adjusting for the study variables. The level of misunderstanding instructions for prescription drugs increased as participants became older (β = 8.81, p < 0.001) and had lower education levels (β = −5.05, p < 0.001), after adjusting for the study variables. The level of misunderstanding pictograms was similar to that of misunderstanding instructions for prescription drug labels. The amount of reading information on OTC drug labels significantly increased as respondents had adequate health literacy (β = 9.27, p < 0.001), were older (β = 12.49, p < 0.001), or had chronic diseases (β = 7.49, p = 0.007). Individuals’ health literacy level, reading behaviors, and complexity of medication instructions are associated with misunderstanding of medication information. Appropriate word choices in drug labels and an improved format of medication instructions could increase understanding of medication information and prevent adverse drug reactions.

Validation of pharmaceutical pictograms among older adults with limited English proficiency

OBJECTIVE

Pictograms on prescription medication labels enhance medication literacy and medication adherence. However, pictograms need to be contextually validated. We assessed the validity of 52 International Pharmaceutical Federation pictograms among 250 older Singaporeans with limited English proficiency.

METHODS

Participants were randomly assigned 11 pictograms each. For each pictogram, participants were first asked its intended meaning. Then, they were told the intended meaning and asked to rate how well the pictogram represented the meaning, on a scale of 1-7. Pictograms were classified as valid (≥66% participants assigned the pictogram interpreted its intended meaning correctly [transparency criterion] and ≥85% participants rated its representativeness as ≥5 [translucency criterion]), partially valid (only transparency criterion was fulfilled) or not valid. Open-ended questions gathered feedback to improve pictograms.

RESULTS

14 pictograms (26.9%) achieved validity and 6 pictograms (11.5%) achieved partial validity. A greater proportion of pictograms for dose and route of administration, and dosage frequency achieved validity or partial validity versus those depicting precautions, indications or side effects.

CONCLUSION

Majority (61.5%) of the assessed pictograms did not achieve validity or partial validity, highlighting the importance of contextual validation.

PRACTICE IMPLICATIONS

Low pictogram comprehension emphasizes the importance of facilitating pictogram understanding during medication counseling.

Evaluation of pharmaceutical pictograms by older "turkers": A cross-sectional crowdsourced study

BACKGROUND

Well-designed pharmaceutical pictograms may improve patients' understanding of medication instructions. However, the iterative participatory design process required to produce effective pictograms can be costly in terms of money, time, and effort. Crowdsourcing has been applied to bring down the costs of the participatory design process, but the feasibility of using this approach with older adults remains largely unknown.

OBJECTIVES

To evaluate the feasibility of using Amazon Mechanical Turk (MTurk), a leading crowdsourcing platform, for participatory pictogram evaluation with older adults (55+) and to evaluate the comprehensibility of USP pictogram, identify common misinterpretations, and explore the relationship between selected participant characteristics and their pictogram comprehension performance.

METHODS

108 older adults (56.5% female; 57-80 years of age) were recruited via MTurk to complete a cross-sectional online survey that asked them to interpret 15 USP pictograms and answer questions about their health and health literacy.

RESULTS

It was feasible to perform pictogram evaluation with older adults on MTurk, as shown by ease of recruitment and high data quality. Of the 15 pictograms tested, seven (46.7%) resulted in a comprehensibility score below the threshold established by the American National Standards Institute (ANSI), eight (53.3%) elicited common misinterpretations, and two (13.3%) resulted in ANSI-defined "critical confusion." Age (P = 0.04) was associated with pictogram comprehension performance. Certain issues with the pictogram subtitles emerged during the evaluation.

CONCLUSIONS

MTurk is a feasible platform for participatory pictogram evaluation, even for a sole target of older adults. The USP should develop a pictogram user manual, redesign pictograms confusing to older adults, and establish policies and procedures to ensure that pictogram subtitles conform to evidence-based best practices and standards for patient-centered written drug information.

Provision and Need for Medicine Information in Asia and Africa: A Scoping Review of the Literature

Published reviews of written medicine information (WMI) have mainly drawn on studies published in high-income countries, including very few Asian or African studies. We therefore set out to scope the research literature to determine the extent and type of studies concerning WMI for patients/consumers across these two continents. We sought empirical studies published between January 2004 and December 2019, conducted in any Asian or African country, as defined by the United Nations, in English or with an English abstract. The majority of the 923 papers identified were from high-income countries. We retained 26 papers from Africa and 99 from Asia. Most African studies (n = 20) involved patients in the development of PILs, in the assessment of the effectiveness of PILs or in surveys. In contrast, the highest proportion of Asian studies concerned the content of WMI (n = 42). WMI is desired, but needs to be in local languages, and there needs to be more use made of pre-tested pictograms. Existing WMI frequently does not meet local regulatory requirements, particularly locally manufactured products. A number of studies reported potentially positive impacts of providing WMI on knowledge and medicine use behaviours. Provision of medicine information is essential for safe use of medicines in all countries. Internationally agreed guidelines, incorporating good design principles, are needed to ensure the optimal content and design of WMI. The World Health Organization should support African and Asian regulatory bodies to share best practice in relation to WMI for patients/consumers and to develop and implement pan-continental guidelines that take into account consumer needs and preferences.

Comprehensibility of pharmaceutical pictograms: Effect of prospective-user factors and cognitive sign design features

BACKGROUND

The people's comprehensibility regarding the meaning of internationally recommended pictograms is an important factor in the correct usage of medications.

OBJECTIVE

To investigate the relationship between the guessability of the pharmaceutical pictograms, the cognitive sign features, and prospective-user factors.

METHODS

A total of 351 Iranian people participated in this study. Two questionnaires were used to measure guessability and cognitive design features regarding the pharmaceutical pictograms. A single-sheet questionnaire was also developed to collect demographic data.

RESULTS

According to the 67% correctness criterion suggested by ISO 3864:P3, 18 pictograms were understandable by the participants. Moreover, of the five cognitive features, "semantic closeness" and "meaningfulness" had the most correlation with the guessability score. In terms of personal factors, understanding of the pictograms' meaning was negatively correlated with age, while it had no association with the occupation.

CONCLUSIONS

Some pharmaceutical pictograms developed by reliable international organizations can be used in a community only after redesigning and testing among the prospective users. The findings indicated that some pharmaceutical pictograms were not comprehensible for most participants. It is therefore expected that using a combination of pictograms with written messages and training could help in conveying the messages by pharmaceutical pictograms.

Guessability of U.S. pharmaceutical pictograms in Iranian prospective users

Objective:

This study examined the gueassability of US pharmaceutical pictograms as well as associated demographic factors and cognitive design features among Iranian adults.

Methods:

A total of 400 participants requested to guess the meaning of 53 US pharmaceutical pictograms using the open-ended method. Moreover, the participants were asked to rate the cognitive design features of each pictorial in terms of familiarity, concreteness, simplicity, meaningfulness and semantic closeness on a scale of 0-100.

Results:

The average guessability score (standard deviation) was 66.30 (SD=24.59). Fifty-five percent of pharmaceutical pictograms understudy met the correctness criteria of 67% specified by ISO3864, while only 30% reached the criterion level of 85% set by ANSIz535.3. Low literate participants with only primary school education had substantial difficulty in the interpretation of pharmaceutical pictograms compared to those completed higher education levels. Younger adults of <30 years significantly performed better in the interpretation of pharmaceutical pictograms as compared to >31 years old participants. ‘Home patient care’ and ‘daily medication use’ had no effect on guessability performance. Concerning cognitive design features, meaningfulness better predict geussability score compared to the others.

Conclusions:

Several USP pictograms fail to be correctly interpreted by Iranian users and need to be redesigned respecting cognitive design features. Interface designers are recommended to incorporate more familiar and concrete elements into their graphics in order to create more meaningful pictorial symbols and to avoid any misinterpretation by the user. Much effective medication use is expected to be achieved by means of this approach, through the improvement of the communication property of pharmaceutical pictograms.

Recall of Pharmaceutical Pictograms by Older Adults

BACKGROUND

Low health literacy and high medication burden in the older adult population are contributing factors to the misunderstanding of medication instructions, leading to an increased risk of poor adherence and adverse events in this group of patients.

OBJECTIVE

To evaluate the ability of older adults to recall the meaning of 13 pharmaceutical pictograms 4 weeks after receipt of feedback on pictogram meaning.

METHODS

Older adults (aged 65 or older) were recruited from one community pharmacy in Canada. One-on-one structured interviews were conducted to assess the comprehensibility of 13 pharmaceutical pictograms from the International Pharmaceutical Federation's database of pictograms. Each participant was then told the meaning of each pictogram. Recall was assessed 4 weeks later.

RESULTS

A total of 58 participants met the inclusion criteria and agreed to participate. The number of pictograms meeting the ISO threshold for comprehensibility of symbols increased from 10 at the initial comprehensibility assessment to 13 at the recall assessment. Analysis of demographic data showed no associations between initial comprehensibility of the pictograms and age, sex, education level, or number of medications taken.

CONCLUSIONS

The results of this study indicate that after being informed of the meaning of pharmaceutical pictograms, older adults were able to recall the pictogram meanings for at least 4 weeks.

Users' Interpretation of Pictograms and Pictures for Conveying Instructions and Warnings on Pharmaceutical Labels

Pharmaceutical pictograms are figures that provide a visual representation of medication instructions and warnings. Pictograms were created in order to lower the continuously rising problem of poor medication adherence due to patients misunderstanding what is being instructed on medication labels. Research on pictograms show that their use could increase patients' comprehension of medication instructions, but the comprehension rates are still low. Researchers have argued that in order to improve the effectiveness of pictograms, they must be more concrete, simple, meaningful, and familiar. Pictures have these properties. Thus, the goal of this study was to examine if the use of pictures (i.e., real images) provide a better representation of what is being instructed to users on medication labels than pictograms. We hypothesized that real images would lead to a better understanding of the instructions and warnings for taking medication, and in turn, users would have higher comprehension levels. Participants completed an online survey evaluating the meaning of either pictograms or pictures. Both quantitative and qualitative data analyses were conducted on free responses to the questions and on users' ratings of the "goodness" of the stimuli. The results showed that participants were moderately accurate (74%) in their interpretation of pictograms, and the use of pictures did not improve their comprehension.

Investigation of the Contributory Factors to the Guessability of Traffic Signs

Traffic signs play an important role in traffic management systems. A variety of studies have focused on drivers’ comprehension of traffic signs. However, the travel safety of prospective users, which has been rarely mentioned in previous studies, has attracted considerable attention from relevant departments in China. With the growth of international and interregional travel demand, traffic signs should be designed more universally to reduce the potential risks to drivers. To identify key factors that improve prospective users’ sign comprehension, this study investigated eight factors that may affect users’ performance regarding sign guessing. Two hundred and one Chinese students, all of whom intended to be drivers and none of whom had experience with daily driving after obtaining a license or visits to Germany, guessed the meanings and rated the sign features of 54 signs. We investigated the effects of selected user factors on their sign guessing performance. Additionally, the contributions of four cognitive design features to the guessability of traffic signs were examined. Based on an analysis of the relationships between the cognitive features and the guessability score of signs, the contributions of four sign features to the guessability of traffic signs were examined. Moreover, by exploring Chinese users’ differences in guessing performance between Chinese signs and German signs, cultural issues in sign design were identified. The results showed that vehicle ownership and attention to traffic signs exerted a significant influence on guessing performance. As expected, driver’s license training and the number of years in college were dominant factors for guessing performance. With regard to design features, semantic distance and confidence in guessing were two dominant factors for the guessability of signs. We suggest improving the design of signs by including vivid, universal symbols. Thus, we provide several suggestions for designing more user-friendly signs.

The extent and effects of patient involvement in pictogram design for written drug information: a short systematic review

This short review provides insight into the extent and effectiveness of patient involvement in the design and evaluation of pictograms to support patient drug information. Pubmed, CINAHL, Cochrane Library, Embase, PsycINFO, Academic Search Premier and Web of Science were searched systematically; the 73 included articles were evaluated with the MMAT. We see that, usually, non-patient end-users are involved in the design of pharmaceutical pictograms - patients are more commonly involved in the final evaluation of pictogram success. Repeated involvement of (non-)patients aids the design of effective pharmaceutical pictograms, although there is limited evidence for such effects on patient perception of drug information or health behaviour.

Comprehension by older people of medication information with or without supplementary pharmaceutical pictograms

This study examined the benefits of pharmaceutical pictograms for improving comprehension of medication information for older people. Fifty Hong Kong Chinese older people completed a medical information comprehension task for five drugs. Participants in the control group were presented with text labels while those in the experimental group were given the text labels plus supplementary pharmaceutical pictograms, and then all reported their understanding of the medication information conveyed. Lower educated older people had poorer understanding of medication information. The addition of pharmaceutical pictograms significantly improved the comprehension of medication information for older people. The majority of older people tested with pictograms favored adding pictograms to text and thought the pictograms were useful for conveying medical information rather than using written text alone. The findings suggested that pharmaceutical and health care professionals should include pharmaceutical pictograms on labels to better convey instructions on medication to older people.

Improving visual search in instruction manuals using pictograms

Instruction manuals provide important messages about the proper use of a product. They should communicate in such a way that they facilitate users' searches for specific information. Despite the increasing research interest in visual search, there is a lack of empirical knowledge concerning the role of pictograms in search performance during the browsing of a manual's pages. This study investigates how the inclusion of pictograms improves the search for the target information. Furthermore, it examines whether this search process is influenced by the visual similarity between the pictograms and the searched for information. On the basis of eye-tracking measurements, as objective indicators of the participants' visual attention, it was found that pictograms can be a useful element of search strategy. Another interesting finding was that boldface highlighting is a more effective method for improving user experience in information seeking, rather than the similarity between the pictorial and adjacent textual information. Implications for designing effective user manuals are discussed. Practitioner Summary: Users often view instruction manuals with the aim of finding specific information. We used eye-tracking technology to examine different manual pages in order to improve the user's visual search for target information. The results indicate that the use of pictograms and bold highlighting of relevant information facilitate the search process.

Testing of Candidate Icons to Identify Acetaminophen-Containing Medicines

Adding icons on labels of acetaminophen-containing medicines could help users identify the active ingredient and avoid concomitant use of multiple medicines containing acetaminophen. We evaluated five icons for communication effectiveness. Adults (n = 300) were randomized to view a prescription container label or over-the-counter labels with either one or two icons. Participants saw two icon candidates, and reported their interpretation; experts judged whether these reflected critical confusions that might cause harm. Participants rated how effectively each icon communicated key messages. Icons based on abbreviations of “acetaminophen” (“Ac”, “Ace”, “Acm”) were rated less confusing and more effective in communicating the active ingredient than icons based on “APAP” or an abstract symbol. Icons did not result in critical confusion when seen on a readable medicine label. Icon implementation on prescription labels was more effective at communicating the warning against concomitant use than implementation on over-the-counter (OTC) labels. Adding an icon to a second location on OTC labels did not consistently enhance this communication, but reduced rated effectiveness of acetaminophen ingredient communication among participants with limited health literacy. The abbreviation-based icons seem most suitable for labeling acetaminophen-containing medications to enable users to identify acetaminophen-containing products.

Evaluation, Modification, and Validation of Pictograms Depicting Medication Instructions in the Elderly

Poor health literacy has been recognized as a limiting factor in the elderly's ability to comprehend written or verbal medication information and also to successfully adhere to medical regimens. The objective of this study was to validate a set of pictograms depicting medication instructions for use among the elderly to support health literacy. Elderly outpatients were recruited in 3 community pharmacies in Canada. One-on-one structured interviews were conducted to assess comprehension of 76 pictograms from the International Pharmaceutical Federation. Comprehension was assessed using transparency testing and pictogram translucency, or the degree to which the pictogram represents the intended message. A total of 135 participants were enrolled in this study, and 76 pictograms were assessed. A total of 50 pictograms achieved more than 67% comprehension. Pictograms depicting precautions and warnings against certain side effects were generally not well understood. Gender, age, and education level all had a significant impact on the interpretation scores of certain individual pictograms. When all pictograms were included, younger males had a significantly higher comprehension score than older females, and participants with a higher level of education provided significantly higher translucency scores. Even when pictograms reached the comprehension threshold set by the International Organization for Standardization in the general populations, only 50 of these pictograms achieved more than 67% comprehension among the elderly, confirming that validation in this subpopulation should be conducted prior to using specific pictograms. Accompanying pictograms with education about these pictograms and important counseling points remains extremely important.

Crowdsourcing participatory evaluation of medical pictograms using Amazon Mechanical Turk

Background

Consumer and patient participation proved to be an effective approach for medical pictogram design, but it can be costly and time-consuming. We proposed and evaluated an inexpensive approach that crowdsourced the pictogram evaluation task to Amazon Mechanical Turk (MTurk) workers, who are usually referred to as the “turkers”.

Objective

To answer two research questions: (1) Is the turkers’ collective effort effective for identifying design problems in medical pictograms? and (2) Do the turkers’ demographic characteristics affect their performance in medical pictogram comprehension?

Methods

We designed a Web-based survey (open-ended tests) to ask 100 US turkers to type in their guesses of the meaning of 20 US pharmacopeial pictograms. Two judges independently coded the turkers’ guesses into four categories: correct, partially correct, wrong, and completely wrong. The comprehensibility of a pictogram was measured by the percentage of correct guesses, with each partially correct guess counted as 0.5 correct. We then conducted a content analysis on the turkers’ interpretations to identify misunderstandings and assess whether the misunderstandings were common. We also conducted a statistical analysis to examine the relationship between turkers’ demographic characteristics and their pictogram comprehension performance.

Results

The survey was completed within 3 days of our posting the task to the MTurk, and the collected data are publicly available in the multimedia appendix for download. The comprehensibility for the 20 tested pictograms ranged from 45% to 98%, with an average of 72.5%. The comprehensibility scores of 10 pictograms were strongly correlated to the scores of the same pictograms reported in another study that used oral response–based open-ended testing with local people. The turkers’ misinterpretations shared common errors that exposed design problems in the pictograms. Participant performance was positively correlated with their educational level.

Conclusions

The results confirmed that crowdsourcing can be used as an effective and inexpensive approach for participatory evaluation of medical pictograms. Through Web-based open-ended testing, the crowd can effectively identify problems in pictogram designs. The results also confirmed that education has a significant effect on the comprehension of medical pictograms. Since low-literate people are underrepresented in the turker population, further investigation is needed to examine to what extent turkers’ misunderstandings overlap with those elicited from low-literate people.