The wisdom of the crowd playing The Price Is Right

Subjective Confidence as a Monitor of the Replicability of the Response

Confidence is commonly assumed to monitor the accuracy of responses. However, intriguing results, examined in the light of philosophical discussions of epistemic justification, suggest that confidence actually monitors the reliability of choices rather than (directly) their accuracy. The focus on reliability is consistent with the view that the construction of truth has much in common with the construction of reality: extracting reliable properties that afford prediction. People are assumed to make a binary choice by sampling cues from a "collective wisdomware," and their confidence is based on the consistency of these cues, in line with the self-consistency model. Here, however, I propose that internal consistency is taken to index the reliability of choices themselves-the likelihood that they will be repeated. The results of 10 studies using binary decisions from different domains indicated that confidence in a choice predicts its replicability both within individuals and across individuals. This was so for domains for which choices have a truth value and for those for which they do not. For the former domains, differences in replicability mediated the prediction of accuracy whether confidence was diagnostic or counterdiagnostic of accuracy. Metatheoretical, methodological, and practical implications are discussed.

Drafting strategies in fantasy football: A study of competitive sequential human decision making

Drafting is a competitive task in which a set of decision makers choose from a set of resources sequentially, with each resource becoming unavailable once selected. How people make these choices raises basic questions about human decision making, including people’s sensitivity to the statistical regularities of the resource environment, their ability to reason about the behavior of their competitors, and their ability to execute and adapt sophisticated strategies in dynamic situations involving uncertainty. Sports provides one real-world example of drafting behavior, in which a set of teams draft players from an available pool in a well-regulated way. Fantasy sport competitions provide potentially large data sets of drafting behavior. We study fantasy football drafting behavior from the 2017 National Football League (NFL) season based on 1350 leagues hosted by the http://sleeper.app platform. We find people are sensitive to some important environmental regularities in the order in which they draft players, but also present evidence that they use a more narrow range of strategies than is likely optimal in terms of team composition. We find little to no evidence for the use of the complicated but well-documented strategy known as handcuffing, and no evidence of irrational influence from individual-level biases for different NFL teams. We do, however, identify a set of circumstances for which there is clear evidence that people’s choices are strongly influenced by the immediately preceding choice made by a competitor.

Model-Based Wisdom of the Crowd for Sequential Decision-Making Tasks

We study the wisdom of the crowd in three sequential decision-making tasks: the Balloon Analogue Risk Task (BART), optimal stopping problems, and bandit problems. We consider a behavior-based approach, using majority decisions to determine crowd behavior and show that this approach performs poorly in the BART and bandit tasks. The key problem is that the crowd becomes progressively more extreme as the decision sequence progresses, because the diversity of opinion that underlies the wisdom of the crowd is lost. We also consider model-based approaches to each task. This involves inferring cognitive models for each individual based on their observed behavior, and using these models to predict what each individual would do in any possible task situation. We show that this approach performs robustly well for all three tasks and has the additional advantage of being able to generalize to new problems for which there are no behavioral data. We discuss potential applications of the model-based approach to real-world sequential decision problems and discuss how our approach contributes to the understanding of collective intelligence.

Going, going, gone: competitive decision-making in Dutch auctions

In a Dutch auction, an item is offered for sale at a set maximum price. The price is then gradually lowered over a fixed interval of time until a bid is made, securing the item for the bidder at the current price. Bidders must trade-off between certainty and price: bid early to secure the item and you pay a premium; bid later at a lower price but risk losing to another bidder. These properties of Dutch auctions provide new opportunities to study competitive decision-making in a group setting. We developed a novel computerised Dutch auction platform and conducted a set of experiments manipulating volatility (fixed vs varied number of items for sale) and price reduction interval rate (step-rate). Triplets of participants (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N=66$$\end{document}N=66) competed with hypothetical funds against each other. We report null effects of step-rate and volatility on bidding behaviour. We developed a novel adaptation of prospect theory to account for group bidding behaviour by balancing certainty and subjective expected utility. We show the model is sensitive to variation in auction starting price and can predict the associated changes in group bid prices that were observed in our data.

Does the "surprisingly popular" method yield accurate crowdsourced predictions?

The “surprisingly popular” method (SP) of aggregating individual judgments has shown promise in overcoming a weakness of other crowdsourcing methods—situations in which the majority is incorrect. This method relies on participants’ estimates of other participants’ judgments; when an option is chosen more often than the average metacognitive judgments of that option, it is “surprisingly popular” and is selected by the method. Although SP has been shown to improve group decision making about factual propositions (e.g., state capitals), its application to future outcomes has been limited. In three preregistered studies, we compared SP to other methods of aggregating individual predictions about future events. Study 1 examined predictions of football games, Study 2 examined predictions of the 2018 US midterm elections, and Study 3 examined predictions of basketball games. When applied to judgments made by objectively assessed experts, SP performed slightly better than other aggregation methods. Although there is still more to learn about the conditions under which SP is effective, it shows promise as a means of crowdsourcing predictions of future outcomes.

A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment

Research has shown that aggregation of independent expert judgments significantly improves the quality of forecasts as compared to individual expert forecasts. This “wisdom of crowds” (WOC) has sparked substantial interest. However, previous studies on strengths and weaknesses of aggregation algorithms have been restricted by limited empirical data and analytical complexity. Based on a comprehensive analysis of existing knowledge on WOC and aggregation algorithms, this paper describes the design and implementation of a static stochastic simulation model to emulate WOC scenarios with a wide range of parameters. The model has been thoroughly evaluated: the assumptions are validated against propositions derived from literature, and the model has a computational representation. The applicability of the model is demonstrated by investigating aggregation algorithm behavior on a detailed level, by assessing aggregation algorithm performance, and by exploring previously undiscovered suppositions on WOC. The simulation model helps expand the understanding of WOC, where previous research was restricted. Additionally, it gives directions for developing aggregation algorithms and contributes to a general understanding of the WOC phenomenon.

Population Size Estimation of Tramadol Misusers in Urban Population in Iran: Synthesis of Methods and Results

Background

Estimating the population who use drugs is essential for planning, monitoring, and evaluation of substance use prevention and treatment. This study aims to estimate the population who misuse tramadol in urban population in Iran.

Methods

We used the wisdom of the experts (WOTE) and network scale-up (NSU) methods to calculate the population of tramadol misusers in 10 provincial capitals of Iran, in 2016. The WOTE was conducted among pharmacists in drugstores and the personnel of traditional medicinal herbs stores. They guessed the best estimation of tramadol misusers population in their cities. The NSU method was conducted among the general population and participants were questioned about ever and daily, non-medical use of tramadol during last 12 months in their network. The median of the methods was used to calculate the proportion of the adult population (15-49 years old).

Findings

The population size of tramadol misusers in studied cities was 83300 [95% uncertainty limits (UL): 47960-256220]. This corresponded to 6.6 per 1000 (95% UL: 3.88-20.30) of the 15-49-year-old population. The projected number of tramadol misusers for all 31 provincial capitals was 118290 (95% UL: 68100-363130840) and 212440 (95% UL: 122310-653410) for all urban areas. NSU also estimated the number of people who misuse tramadol on daily basis. These numbers were 52000 (95% UL: 19940-176570) for studied cities, 73840 (95% UL: 28320-250740) for all 31 provincial capital cities, and 132610 (95% UL: 50860-450310) for all urban areas in Iran.

Conclusion

This study presents information on high prevalence of tramadol misuse in urban population. We need national control measures and demand reduction programs to control tramadol misuse.

The Expression and Transfer of Valence Associated with Social Conformity

Consensus seeking – abandoning one’s own judgment to align with a group majority – is a fundamental feature of human social interaction. Notably, such striving for majority affiliation often occurs in the absence of any apparent economic or social gain, suggesting that achieving consensus might have intrinsic value. Here, using a simple gambling task, in which the decisions of ostensible previous gamblers were indicated below available options on each trial, we examined the affective properties of agreeing with a group majority by assessing the trade-off between social and non-social currencies, and the transfer of social valence to concomitant stimuli. In spite of demonstrating near-perfect knowledge of objective reward probabilities, participant’s choices and evaluative judgments reflected a reliable preference for conformity, consistent with the hypothesized value of social alignment.

Estimating the Size of the MSM Population in Metro Vancouver, Canada, Using Multiple Methods and Diverse Data Sources

Men who have sex with men (MSM) are disproportionately affected by HIV globally, regionally in Canada, and locally in Vancouver. Lack of reliable population size estimates of MSM impedes effective implementation of health care services and limits our understanding of the HIV epidemic. We estimated the population size of MSM residing in Metro Vancouver drawing on four data sources: the Canadian Community Health Survey (CCHS), a cross-sectional bio-behavioural MSM survey, HIV testing services data from sexually transmitted infection (STI) clinics serving MSM, and online social networking site Facebook. Estimates were calculated using (1) direct estimates from the CCHS, (2) "Wisdom of the Crowds" (WOTC), and (3) the multiplier method using data from a bio-behavioural MSM survey, clinic-based HIV testing, and online social media network site Facebook. Data sources requiring greater public disclosure of sexual orientation resulted in our mid-range population estimates (Facebook 23,760, CCHS 30,605). The WOTC method produced the lowest estimate, 10,000. The multiplier method using STI clinic HIV testing data produced the largest estimate, 41,777. The median of all estimates was 27,183, representing 2.9% of the Metro Vancouver census male adult population, with an interquartile range of 1.1-4.5%. Using multiple data sources, our estimates of the MSM population in Metro Vancouver are similar to population prevalence estimates based on population data from other industrialized nations. These findings will support understanding of the HIV burden among MSM and corresponding public health and health services planning for this key population.

Population size estimation of female sex workers in Iran: Synthesis of methods and results

INTRODUCTION

Estimating the number of key populations at risk of HIV is essential for planning, monitoring, and evaluating prevention, care, and treatment programmes. We conducted this study to estimate the number of female sex workers (FSW) in major cities of Iran.

METHODS

We used three population size estimation methods (i.e., wisdom of the crowds, multiplier method, and network scale-up) to calculate the number of FSW in 13 cities in Iran. The wisdom of the crowds and multiplier methods were integrated into a nationwide bio-behavioural surveillance survey in 2015, and the network scale-up method was included in a national survey of the general population in 2014. The median of the three methods was used to calculate the proportion of the adult female population who practice sex work in the 13 cities. These figures were then extrapolated to provide a national population size estimation of FSW across urban areas.

RESULTS

The population size of FSW was 91,500 (95% Uncertainty Intervals [UIs] 61,400-117,700), corresponding to 1.43% (95% UIs 0.96-1.84) of the adult (i.e., 15-49 year-old) female population living in these 13 cities. The projected numbers of FSW for all 31 provincial capital cities were 130,800 (95% UIs 87,800-168,200) and 228,700 (95% UIs 153,500-294,300) for all urban settings in Iran.

CONCLUSIONS

Using methods of comparable rigor, our study provided a data-driven national estimate of the population size of FSW in urban areas of Iran. Our findings provide vital information for enhancing HIV programme planning and lay a foundation for assessing the impact of harm reduction efforts within this marginalized population.

The relationship between crowd majority and accuracy for binary decisions

We consider the wisdom of the crowd situation in which individuals make binary decisions, and the majority answer is used as the group decision. Using data sets from nine different domains, we examine the relationship between the size of the majority and the accuracy of the crowd decisions. We find empirically that these calibration curves take many different forms for different domains, and the distribution of majority sizes over decisions in a domain also varies widely. We develop a growth model for inferring and interpreting the calibration curve in a domain, and apply it to the same nine data sets using Bayesian methods. The modeling approach is able to infer important qualitative properties of a domain, such as whether it involves decisions that have ground truths or are inherently uncertain. It is also able to make inferences about important quantitative properties of a domain, such as how quickly the crowd accuracy increases as the size of the majority increases. We discuss potential applications of the measurement model, and the need to develop a psychological account of the variety of calibration curves that evidently exist.

Population Size Estimation of Men Who Have Sex with Men in Tbilisi, Georgia; Multiple Methods and Triangulation of Findings

INTRODUCTION

An accurate estimation of the population size of men who have sex with men (MSM) is critical to the success of HIV program planning and to monitoring of the response to epidemic as a whole, but is quite often missing. In this study, our aim was to estimate the population size of MSM in Tbilisi, Georgia and compare it with other estimates in the region.

METHODS

In the absence of a gold standard for estimating the population size of MSM, this study reports a range of methods, including network scale-up, mobile/web apps multiplier, service and unique object multiplier, network-based capture-recapture, Handcock RDS-based and Wisdom of Crowds methods. To apply all these methods, two surveys were conducted: first, a household survey among 1,015 adults from the general population, and second, a respondent driven sample of 210 MSM. We also conducted a literature review of MSM size estimation in Eastern European and Central Asian countries.

RESULTS

The median population size of MSM generated from all previously mentioned methods was estimated to be 5,100 (95% Confidence Interval (CI): 3,243~9,088). This corresponds to 1.42% (95%CI: 0.9%~2.53%) of the adult male population in Tbilisi.

CONCLUSION

Our size estimates of the MSM population (1.42% (95%CI: 0.9%~2.53%) of the adult male population in Tbilisi) fall within ranges reported in other Eastern European and Central Asian countries. These estimates can provide valuable information for country level HIV prevention program planning and evaluation. Furthermore, we believe, that our results will narrow the gap in data availability on the estimates of the population size of MSM in the region.

Critique and lessons learned from using multiple methods to estimate population size of men who have sex with men in Ghana

Population size estimation of key populations at risk of HIV is essential to every national response. We implemented population size estimation of men who have sex with men (MSM) in Ghana using a three-stage approach within the 2011 Ghana Men's Study: during the study's formative assessment, the larger integrated bio-behavioral surveillance (IBBS) survey; and during the stakeholder meeting. We used six methods in combination within the three-stage approach (literature review, mapping with census, unique object multiplier, service multiplier, wisdom of the crowd, and modified Delphi) to generate size estimates from 16 locations (4 IBBS survey sites and 12 other locations) and used the estimates from the 16 sites to extrapolate the total MSM population size of Ghana. We estimated the number of MSM in Ghana to be 30,579 with a plausible range of 21,645-34,470. The overall estimate suggests that the prevalence of MSM in Ghana is 0.48 % of the adult male population. Lessons learned are shared to inform and improve applications of the methods in future studies.

A cognitive model for aggregating people's rankings

We develop a cognitive modeling approach, motivated by classic theories of knowledge representation and judgment from psychology, for combining people's rankings of items. The model makes simple assumptions about how individual differences in knowledge lead to observed ranking data in behavioral tasks. We implement the cognitive model as a Bayesian graphical model, and use computational sampling to infer an aggregate ranking and measures of the individual expertise. Applications of the model to 23 data sets, dealing with general knowledge and prediction tasks, show that the model performs well in producing an aggregate ranking that is often close to the ground truth and, as in the "wisdom of the crowd" effect, usually performs better than most of individuals. We also present some evidence that the model outperforms the traditional statistical Borda count method, and that the model is able to infer people's relative expertise surprisingly well without knowing the ground truth. We discuss the advantages of the cognitive modeling approach to combining ranking data, and in wisdom of the crowd research generally, as well as highlighting a number of potential directions for future model development.

Using cognitive models to combine probability estimates

We demonstrate the usefulness of cognitive models for combining human estimates of probabilities in two experiments. The first experiment involves people’s estimates of probabilities for general knowledge questions such as “What percentage of the world’s population speaks English as a first language?” The second experiment involves people’s estimates of probabilities in football (soccer) games, such as “What is the probability a team leading 1–0 at half time will win the game?”, with ground truths based on analysis of large corpus of games played in the past decade. In both experiments, we collect people’s probability estimates, and develop a cognitive model of the estimation process, including assumptions about the calibration of probabilities and individual differences. We show that the cognitive model approach outperforms standard statistical aggregation methods like the mean and the median for both experiments and, unlike most previous related work, is able to make good predictions in a fully unsupervised setting. We also show that the parameters inferred as part of the cognitive modeling, involving calibration and expertise, provide useful measures of the cognitive characteristics of individuals. We argue that the cognitive approach has the advantage of aggregating over latent human knowledge rather than observed estimates, and emphasize that it can be applied in predictive settings where answers are not yet available.

Estimates of the size of key populations at risk for HIV infection: men who have sex with men, female sex workers and injecting drug users in Nairobi, Kenya

OBJECTIVES

Size estimates of populations at higher risk for HIV infection are needed to help policy makers understand the scope of the epidemic and allocate appropriate resources. Population size estimates of men who have sex with men (MSM), female sex workers(FSW) and intravenous drug users (IDU) are few or non-existent in Nairobi, Kenya.

METHODS

We integrated three population size estimation methods into a behavioural surveillance survey among MSM, FSW and IDU in Nairobi during 2010–2011. These methods included the multiplier method, ‘Wisdom of the Crowds’ and an approach that drew on published literature. The median of the three estimates was hypothesised to be the most plausible size estimate with the other results forming the upper and lower plausible bounds. Data were shared with community representatives and stakeholders to finalise ‘best’ point estimates and plausible bounds based on the data collected in Nairobi, a priori expectations from the global literature and stakeholder input.

RESULTS

We estimate there are approximately 11 042 MSM with a plausible range of 10 000–22 222, 29 494 FSW with a plausible range of 10 000–54 467 FSW and approximately 6107 IDU and plausibly 5031–10 937 IDU living in Nairobi.

CONCLUSIONS

We employed multiple methods and used a wide range of data sources to estimate the size of three hidden populations in Nairobi, Kenya. These estimates may be useful to advocate for and to plan, implement and evaluate HIV prevention and care programmes for MSM, FSW and IDU. Surveillance activities should consider integrating population size estimation in their protocols.