Presentation accuracy of Web animation methods

Web-Based Research in Psychology

Abstract. The present article reviews web-based research in psychology. It captures principles, learnings, and trends in several types of web-based research that show similar developments related to web technology and its major shifts (e.g., appearance of search engines, browser wars, deep web, commercialization, web services, HTML5…) as well as distinct challenges. The types of web-based research discussed are web surveys and questionnaire research, web-based tests, web experiments, Mobile Experience Sampling, and non-reactive web research, including big data. A number of web-based methods are presented and discussed that turned out to become important in research methodology. These are one-item-one-screen design, seriousness check, instruction manipulation and other attention checks, multiple site entry technique, subsampling technique, warm-up technique, and web-based measurement. Pitfalls and best practices are described then, especially regarding dropout and other non-response, recruitment of participants, and interaction between technology and psychological factors. The review concludes with a discussion of important concepts that have developed over 25 years and an outlook on future developments in web-based research.

Implementation of stimuli with millisecond timing accuracy in online experiments

Online experiments are growing in popularity. This study aimed to determine the timing accuracy of web technologies and investigate whether they can be used to support high temporal precision psychology experiments. A dynamic sinusoidal grating and flashes were produced by setInterval, CSS3, and requestAnimationFrame (hereafter, rAF) technologies. They were run at normal or real-time priority processing in Chrome, Firefox, Edge, and Internet Explorer on Windows, macOS, and Linux. Timing accuracies were compared with that of Psychtoolbox which was chosen as gold standard. It was found that rAF with real-time priority had the best timing accuracy compared to the other web technologies and had a similar timing accuracy as Psychtoolbox in traditional experiments in most cases. However, rAF exhibited poor timing accuracy on Linux. Therefore, rAF can be used as technical basis for accuracy of millisecond timing sequences in online experiments, thereby benefiting the psychology field.

Mental chronometry in the pocket? Timing accuracy of web applications on touchscreen and keyboard devices

Web applications can implement procedures for studying the speed of mental processes (mental chronometry) and can be administered via web browsers on most commodity desktops, laptops, smartphones, and tablets. This approach to conducting mental chronometry offers various opportunities, such as increased scale, ease of data collection, and access to specific samples. However, validity and reliability may be threatened by less accurate timing than specialized software and hardware can offer. We examined how accurately web applications time stimuli and register response times (RTs) on commodity touchscreen and keyboard devices running a range of popular web browsers. Additionally, we explored the accuracy of a range of technical innovations for timing stimuli, presenting stimuli, and estimating stimulus duration. The results offer some guidelines as to what methods may be most accurate and what mental chronometry paradigms may suitably be administered via web applications. In controlled circumstances, as can be realized in a lab setting, very accurate stimulus timing and moderately accurate RT measurements could be achieved on both touchscreen and keyboard devices, though RTs were consistently overestimated. In uncontrolled circumstances, such as researchers may encounter online, stimulus presentation may be less accurate, especially when brief durations are requested (of up to 100 ms). Differences in RT overestimation between devices might not substantially affect the reliability with which group differences can be found, but they may affect reliability for individual differences. In the latter case, measurement via absolute RTs can be more affected than measurement via relative RTs (i.e., differences in a participant's RTs between conditions).

Accuracy and precision of stimulus timing and reaction times with Unreal Engine and SteamVR

The increasing interest in Virtual Reality (VR) as a tool for neuroscientific research contrasts with the current lack of established toolboxes and standards. In several recent studies, game engines like Unity or Unreal Engine were used. It remains to be tested whether these software packages provide sufficiently precise and accurate stimulus timing and time measurements that allow inferring ongoing mental and neural processes. We here investigated the precision and accuracy of the timing mechanisms of Unreal Engine 4 and SteamVR in combination with the HTC Vive VR system. In a first experiment, objective external measures revealed that stimulus durations were highly accurate. In contrast, in a second experiment, the assessment of the precision of built-in timing procedures revealed highly variable reaction time measurements and inaccurate determination of stimulus onsets. Hence, we developed a new software-based method that allows precise and accurate reaction time measurements with Unreal Engine and SteamVR. Instead of using the standard timing procedures implemented within Unreal Engine, time acquisition was outsourced to a background application. Timing benchmarks revealed that the newly developed method allows reaction time measurements with a precision and accuracy in the millisecond range. Overall, the present results indicate that the HTC Vive together with Unreal Engine and SteamVR can achieve high levels of precision and accuracy both concerning stimulus duration and critical time measurements. The latter can be achieved using a newly developed routine that allows not only accurate reaction time measures but also provides precise timing parameters that can be used in combination with time-sensitive functional measures such as electroencephalography (EEG) or transcranial magnetic stimulation (TMS).

Gorilla in our midst: An online behavioral experiment builder

Behavioral researchers are increasingly conducting their studies online, to gain access to large and diverse samples that would be difficult to get in a laboratory environment. However, there are technical access barriers to building experiments online, and web browsers can present problems for consistent timing-an important issue with reaction-time-sensitive measures. For example, to ensure accuracy and test-retest reliability in presentation and response recording, experimenters need a working knowledge of programming languages such as JavaScript. We review some of the previous and current tools for online behavioral research, as well as how well they address the issues of usability and timing. We then present the Gorilla Experiment Builder (gorilla.sc), a fully tooled experiment authoring and deployment platform, designed to resolve many timing issues and make reliable online experimentation open and accessible to a wider range of technical abilities. To demonstrate the platform's aptitude for accessible, reliable, and scalable research, we administered a task with a range of participant groups (primary school children and adults), settings (without supervision, at home, and under supervision, in both schools and public engagement events), equipment (participant's own computer, computer supplied by the researcher), and connection types (personal internet connection, mobile phone 3G/4G). We used a simplified flanker task taken from the attentional network task (Rueda, Posner, & Rothbart, 2004). We replicated the "conflict network" effect in all these populations, demonstrating the platform's capability to run reaction-time-sensitive experiments. Unresolved limitations of running experiments online are then discussed, along with potential solutions and some future features of the platform.

Do trunk-based left/right judgment tasks elicit motor imagery?

BACKGROUND

Left/right judgment tasks (LRJTs) are used in the management of chronic pain. This use is predicated on their ability to elicit the simulation of movements (i.e. motor imagery), including those where the execution of the same movements induces pain. While established for limb-based LRJTs, the ability of trunk-based LRJTs to elicit motor imagery of trunk movements has not been demonstrated.

OBJECTIVE

To establish whether data from a trunk-based LRJT are indicative of motor imagery being elicited for the specific lateralised trunk postures presented.

DESIGN

Cross-sectional repeated measures (within-subject experiment).

METHODS

Twenty-nine unimpaired and pain-free participants completed a trunk-based LRJT typical of those used in practice. Accordingly, left/right judgements were made to images depicting a human figure with its trunk rotated or side-flexed to the left or right. The extent (amplitude) of this movement was manipulated (small, medium, large). The whole figure was also oriented to different degrees (0⁰, 45⁰, 90⁰, 135⁰, 180⁰) and along different axes (sagittal, axial, coronal).

RESULTS

Accuracy was higher and response times (RTs) faster (p < 0.001 for both) when lateralised trunk movements depicted had a larger amplitude, contrary to predictions if motor imagery was elicited. Differences in accuracy and RTs depending on the orientation of the whole figure were consistent with previous studies.

DISCUSSION

Data were not consistent with motor imagery of lateralised trunk movements being elicited by the trunk-based LRJT. The study presented here questions the value of trunk-based LRJTs in clinical practice.

Testing the accuracy of timing reports in visual timing tasks with a consumer-grade digital camera

This study tested the accuracy of a visual timing task using a readily available and relatively inexpensive consumer grade digital camera. A visual inspection time task was recorded using short high-speed video clips and the timing as reported by the task's program was compared to the timing as recorded in the video clips. Discrepancies in these two timing reports were investigated further and based on display refresh rate, a decision was made whether the discrepancy was large enough to affect the results as reported by the task. In this particular study, the errors in timing were not large enough to impact the results of the study. The procedure presented in this article offers an alternative method for performing a timing test, which uses readily available hardware and can be used to test the timing in any software program on any operating system and display.

The effect of noise-induced variance on parameter recovery from reaction times

Background

Technical noise can compromise the precision and accuracy of the reaction times collected in psychological experiments, especially in the case of Internet-based studies. Although this noise seems to have only a small impact on traditional statistical analyses, its effects on model fit to reaction-time distributions remains unexplored.

Results

Across four simulations we study the impact of technical noise on parameter recovery from data generated from an ex-Gaussian distribution and from a Ratcliff Diffusion Model. Our results suggest that the impact of noise-induced variance tends to be limited to specific parameters and conditions.

Conclusions

Although we encourage researchers to adopt all measures to reduce the impact of noise on reaction-time experiments, we conclude that the typical amount of noise-induced variance found in these experiments does not pose substantial problems for statistical analyses based on model fitting.

Accuracy and precision of visual stimulus timing in PsychoPy: no timing errors in standard usage

In a recent report published in PLoS ONE, we found that the performance of PsychoPy degraded with very short timing intervals, suggesting that it might not be perfectly suitable for experiments requiring the presentation of very brief stimuli. The present study aims to provide an updated performance assessment for the most recent version of PsychoPy (v1.80) under different hardware/software conditions. Overall, the results show that PsychoPy can achieve high levels of precision and accuracy in the presentation of brief visual stimuli. Although occasional timing errors were found in very demanding benchmarking tests, there is no reason to think that they can pose any problem for standard experiments developed by researchers.

Presentation accuracy of the web revisited: animation methods in the HTML5 era

Using the Web to run behavioural and social experiments quickly and efficiently has become increasingly popular in recent years, but there is some controversy about the suitability of using the Web for these objectives. Several studies have analysed the accuracy and precision of different web technologies in order to determine their limitations. This paper updates the extant evidence about presentation accuracy and precision of the Web and extends the study of the accuracy and precision in the presentation of multimedia stimuli to HTML5-based solutions, which were previously untested. The accuracy and precision in the presentation of visual content in classic web technologies is acceptable for use in online experiments, although some results suggest that these technologies should be used with caution in certain circumstances. Declarative animations based on CSS are the best alternative when animation intervals are above 50 milliseconds. The performance of procedural web technologies based on the HTML5 standard is similar to that of previous web technologies. These technologies are being progressively adopted by the scientific community and have promising futures, which makes their use advisable to utilizing more obsolete technologies.

Collecting response times using Amazon Mechanical Turk and Adobe Flash

Crowdsourcing systems like Amazon's Mechanical Turk (AMT) allow data to be collected from a large sample of people in a short amount of time. This use has garnered considerable interest from behavioral scientists. So far, most experiments conducted on AMT have focused on survey-type instruments because of difficulties inherent in running many experimental paradigms over the Internet. This study investigated the viability of presenting stimuli and collecting response times using Adobe Flash to run ActionScript 3 code in conjunction with AMT. First, the timing properties of Adobe Flash were investigated using a phototransistor and two desktop computers running under several conditions mimicking those that may be present in research using AMT. This experiment revealed some strengths and weaknesses of the timing capabilities of this method. Next, a flanker task and a lexical decision task implemented in Adobe Flash were administered to participants recruited with AMT. The expected effects in these tasks were replicated. Power analyses were conducted to describe the number of participants needed to replicate these effects. A questionnaire was used to investigate previously undescribed computer use habits of 100 participants on AMT. We conclude that a Flash program in conjunction with AMT can be successfully used for running many experimental paradigms that rely on response times, although experimenters must understand the limitations of the method.

Measuring software timing errors in the presentation of visual stimuli in cognitive neuroscience experiments

Because of the features provided by an abundance of specialized experimental software packages, personal computers have become prominent and powerful tools in cognitive research. Most of these programs have mechanisms to control the precision and accuracy with which visual stimuli are presented as well as the response times. However, external factors, often related to the technology used to display the visual information, can have a noticeable impact on the actual performance and may be easily overlooked by researchers. The aim of this study is to measure the precision and accuracy of the timing mechanisms of some of the most popular software packages used in a typical laboratory scenario in order to assess whether presentation times configured by researchers do not differ from measured times more than what is expected due to the hardware limitations. Despite the apparent precision and accuracy of the results, important issues related to timing setups in the presentation of visual stimuli were found, and they should be taken into account by researchers in their experiments.

Parental problem drinking predicts implicit alcohol expectancy in adolescents and young adults

The present study aimed to investigate the influence of parental problem drinking on implicit and explicit alcohol expectancy of adolescents and young adults (12-24 years). The study was conducted via the Internet, employing a between-subjects design. We measured alcohol expectancy by means of an Implicit Association Test (IAT) and a self-report questionnaire. A short version of the Children of Alcoholics Screening Test (CAST) was used to measure alcohol-related parental problem behavior. Our results showed that increased CAST-scores were correlated with a stronger implicit association between the concepts alcohol and arousal. In contrast, no such relationship was observed between parental problem drinking and self-reported expectancy of alcohol arousal. These findings provide tentative evidence that an implicit cognitive processing bias is implicated in the intergenerational transmission of addictive behaviors.

Adobe Flash as a medium for online experimentation: a test of reaction time measurement capabilities

Adobe Flash can be used to run complex psychological experiments over the Web. We examined the reliability of using Flash to measure reaction times (RTs) using a simple binary-choice task implemented both in Flash and in a Linux-based system known to record RTs with millisecond accuracy. Twenty-four participants were tested in the laboratory using both implementations; they also completed the Flash version on computers of their own choice outside the lab. RTs from the trials run on Flash outside the lab were approximately 20 msec slower than those from trials run on Flash in the lab, which in turn were approximately 10 msec slower than RTs from the trials run on the Linux-based system (baseline condition). RT SDs were similar in all conditions, suggesting that although Flash may overestimate RTs slightly, it does not appear to add significant noise to the data recorded.

Magnifying effects on visual word recognition: a dynamic display for internet-based experiments

A visual presentation procedure is introduced that presents target words followed by a dynamic mask until recognition. This form of stimulus degradation prolongs the word recognition process. Differences in word recognition latencies-which are usually quite small--are magnified, and thus can be more easily observed. The results of two experiments on the Internet with a total of 141 participants establish the task's ability to magnify differences in word recognition latencies stemming from word familiarity (Experiment 1) and word prototypicality (Experiment 2). Both factors interact with stimulus degradation, but at different presentation intervals; these results are discussed as evidence for comparing models of word recognition. The new procedure can be used for assessing individual differences, such as implicit motives and self-focused attention. Further applications are discussed.

Scientific LogAnalyzer: A Web-based tool for analyses of server log files in psychological research

Scientific LogAnalyzer is a platform-independent interactive Web service for the analysis of log files. Scientific LogAnalyzer offers several features not available in other log file analysis tools--for example, organizational criteria and computational algorithms suited to aid behavioral and social scientists. Scientific LogAnalyzer is highly flexible on the input side (unlimited types of log file formats), while strictly keeping a scientific output format. Features include (1) free definition of log file format, (2) searching and marking dependent on any combination of strings (necessary for identifying conditions in experiment data), (3) computation of response times, (4) detection of multiple sessions, (5) speedy analysis of large log files, (6) output in HTML and/or tab-delimited form, suitable for import into statistics software, and (7) a module for analyzing and visualizing drop-out. Several methodological features specifically needed in the analysis of data collected in Internet-based experiments have been implemented in the Web-based tool and are described in this article. A regression analysis with data from 44 log file analyses shows that the size of the log file and the domain name lookup are the two main factors determining the duration of an analysis. It is less than a minute for a standard experimental study with a 2 x 2 design, a dozen Web pages, and 48 participants (ca. 800 lines, including data from drop-outs). The current version of Scientific LogAnalyzer is freely available for small log files. Its Web address is http://genpsylab-logcrunsh.unizh.ch/.

Human Research and Data Collection via the Internet

Advantages and disadvantages of Web and lab research are reviewed. Via the World Wide Web, one can efficiently recruit large, heterogeneous samples quickly, recruit specialized samples (people with rare characteristics), and standardize procedures, making studies easy to replicate. Alternative programming techniques (procedures for data collection) are compared, including client-side as opposed to server-side programming. Web studies have methodological problems; for example, higher rates of drop out and of repeated participation. Web studies must be thoroughly analyzed and tested before launching on-line. Many studies compared data obtained in Web versus lab. These two methods usually reach the same conclusions; however, there are significant differences between college students tested in the lab and people recruited and tested via the Internet. Reasons that Web researchers are enthusiastic about the potential of the new methods are discussed.

What do facial expressions convey: feeling states, behavioral intentions, or action requests?

Emotion theorists assume certain facial displays to convey information about the expresser's emotional state. In contrast, behavioral ecologists assume them to indicate behavioral intentions or action requests. To test these contrasting positions, over 2,000 online participants were presented with facial expressions and asked what they revealed-feeling states, behavioral intentions, or action requests. The majority of the observers chose feeling states as the message of facial expressions of disgust, fear, sadness, happiness, and surprise, supporting the emotions view. Only the anger display tended to elicit more choices of behavioral intention or action request, partially supporting the behavioral ecology view. The results support the view that facial expressions communicate emotions, with emotions being multicomponential phenomena that comprise feelings, intentions, and wishes.

A server-side program for delivering experiments with animations

A server-side program for animation experiments is presented. The program is capable of delivering an experiment composed of discrete animation sequences in various file formats, collecting a discrete or continuous response from the observer, evaluating the appropriateness of the response, and ensuring that the user is not proceeding at an unreasonable rate. Most parameters of the program are controllable by experimenter-edited text files or simple switches in the program code, thereby minimizing the need for programming to create new experiments. A simple demonstration experiment is discussed and is freely available.