Learning representations for quality estimation of crowdsourced submissions

QADM: A quality-aware data model for freelancing applications based on recommendation systems

Freelancing systems such as Freelancer and Upwork have recently been increasing in popularity due to their ability to provide efficient solutions. Nevertheless, the success of freelancing depends on the delivery of high-quality freelanced output. Indeed, quality control is a major concern and is affected by the uncertainty stemming from human factors relating to the diversity of workers’ skills and the workers’ changing behaviour over time. This article proposes a quality-aware freelancing data model (QADM) that takes into account both the diversity of worker skills and workers’ changing behaviour. The QADM is comprised of three submodels: skill, task and worker. The main goal is to model worker quality appropriately, and it does so by effectively modelling (1) worker suitability for new tasks, (2) worker reputation, (3) worker accuracy in completed tasks and (4) worker expected accuracy in new tasks. To improve the worker accuracy estimation, a task-to-task similarity algorithm is developed that achieves higher accuracy than Cos(topic), Cos(tf-idf) and Jaccard similarity methods. The quality-aware task assignment decision problem is solved as a top-k task recommendation problem. The results achieved in this article show that the QADM accomplishes a high recommender system mean average precision for the assignment decisions.

A TOPSIS-Inspired Ranking Method Using Constrained Crowd Opinions for Urban Planning

Crowdsourcing has become an important tool for gathering knowledge for urban planning problems. The questions posted to the crowd for urban planning problems are quite different from the traditional crowdsourcing models. Unlike the traditional crowdsourcing models, due to the constraints among the multiple components (e.g., multiple locations of facilities) in a single question and non-availability of the defined option sets, aggregating of multiple diverse opinions that satisfy the constraints as well as finding the ranking of the crowd workers becomes challenging. Moreover, owing to the presence of the conflicting nature of features, the traditional ranking methods such as the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) cannot always be feasible as the optimal solutions in terms of multiple objectives cannot occur simultaneously for the conflicting cases (e.g., benefit and cost criteria) for urban planning problems. Therefore, in this work, a multi-objective approach is proposed to produce better compromised solutions in terms of conflicting features from the general crowd. In addition, the solutions are employed to obtain a proper ideal solution for ranking the crowd. The experimental results are validated using two constrained crowd opinion datasets for real-world urban planning problems and compared with the state-of-the-art TOPSIS models.