Quantitative Data From Rating Scales: An Epistemological and Methodological Enquiry

Uncovering Hidden Framings in Dark Triad Self-Ratings: What Frames-of-Reference Do People Use When Responding to Generic Dark Triad Items?

In typical Dark Triad (DT) questionnaires, generic items oftentimes refer to "others" or "people" in general. Hence, respondents have to mentally aggregate their behavior across several kinds of "others" (e.g., work colleagues, family members, and friends). It remains unknown if individuals consider different kinds of interaction partners equally or if their self-reports contain "hidden" interaction partner-specific tendencies. To shed light on this issue, we assessed generic and contextualized DT items (referring to family, friends, work, and strangers; N = 814 from the general population). The correlated trait-correlated (method - 1) model was used to investigate preregistered research questions. On average, generic DT items showed the strongest association with work-contextualized DT items and the weakest association with family-contextualized DT items. However, the associations varied considerably across DT items and traits. In sum, our results suggest that hidden framings exist in some DT items, which may impact their ability to predict relevant criteria due to contextual (a)symmetries. The generalizability of the findings to other DT instruments, items, and participant groups should be examined in future research.

Measuring the intensity of emotions

We describe a theoretical framework for the measurement of the intensity of emotional experiences and summarize findings of a series of studies that implemented this framework. Our approach is based on a realist view of quantities and combines the modern psychometric (i.e., latent-variable) view of measurement with a deductive order of inquiry for testing measurement axioms. At the core of the method are nonmetric probabilistic difference scaling methods, a class of indirect scaling methods based on ordinal judgments of intensity differences. Originally developed to scale sensations and preferences, these scaling methods are also well-suited for measuring emotion intensity, particularly in basic research. They are easy to perform and provide scale values of emotion intensity that are much more precise than the typically used, quality-intensity emotion rating scales. Furthermore, the scale values appear to fulfill central measurement-theoretical axioms necessary for interval-level measurement. Because of these properties, difference scaling methods allow precise tests of emotion theories on the individual subject level.

Beyond polarization towards dynamic balance: harmony as the core of mental health

Worldviews are culturally derived assumptions that influence individual and collective behaviors, values, and representations of reality. The study of mental functions is not exempt from this influence, as reflected in scientific theories, methodological approaches, and empirical studies. Despite acknowledging the interplay of mental processes with developmental, environmental, and cultural dimensions, psychological research is still primarily based on quantitative methods, and on the conceptualization of mental phenomena as unfolding along polarized continua. A lively epistemological debate surrounds this approach, especially underscoring the risk of blurring the distinction between constructs derived from statistical models and real-life processes and experiences. Based on this debate and on recent empirical evidence derived from the positive psychology literature, this paper is aimed at proposing an integrated view of mental health, as a holistically patterned, contextually imbedded, and dynamic phenomenon changing over time and across life events, with harmony, harmonization and dynamic balance as core qualities. The heuristic potential of investigating the qualitative configuration patterns of mental health dimensions across individuals and groups, beyond their position along a quantitative continuum, is outlined. The development of more integrated approaches and methodologies to investigate mental health as a harmonization process, taking into account personal, contextual and developmental features, would be aligned with evidence derived from the integration of traditional nomothetic and ideographic approaches, and other life sciences. However, the development of a transdisciplinary line of research requires further inputs from different epistemological views, as well as higher attention to the potential contribution of different philosophical traditions.

Capturing Between- and Within-Person Variability of Borderline Traits in Youth from a Developmentally Sensitive Situational Judgment Perspective

The significance of early maladaptive traits for understanding the roots of borderline personality disorder (BPD) is increasingly endorsed. Given the interpersonal nature of BPD and fluctuations in phenotypic symptomatology, this study aims to test the assumption that a situation-contingent contextualized assessment of borderline traits is a viable avenue to more fully capture the way underlying traits are reflected in socioaffective behavior that occurs in age-relevant situations. We aimed to explore the extent to which the variability on these responses represents a meaningful construct for understanding the clinical richness of BPD. Toward this end, a contextualized measure of DSM-5 BPD traits was developed along a situational judgment test (SJT) format, allowing us to explore both between-person and within-person variability in a more economic format than repeated measurement does. Examination of the psychometric properties of the SJT revealed distinctive correlational patterns of the situation-based traits with symptom scales of a youth BPD measure, but also indicated that for certain traits consistency in trait level across different situations is more maladaptive, whereas for other traits a greater degree of variability in trait expression is more maladaptive. Overall, this work could set the stage for further research on the potential of SJTs for understanding personality vulnerabilities both at the dispositional and the dynamic level.

Teacher attitudes toward evidence-based practices: Exploratory and confirmatory analyses of the school-adapted evidence-based practice attitude scale

Background

The Evidence-Based Practice Attitudes Scale (EBPAS) is widely used in implementation research, but it has not been adapted and validated for use among general education teachers, who are most likely to deliver evidence-based prevention programs in schools, the most common setting where youth access social, emotional, and behavioral health services.

Method

School-based stakeholders and a research team comprised of experts in the implementation of evidence-based practices in schools adapted the EBPAS for teachers (the S-EBPAS). The adapted instrument was administered to a representative sample ( n = 441) of general education teachers (grades K—5) to assess the reliability and internal consistency via factor analyses. The S-EBPAS included two forms (i.e., EBP-agnostic and EBP-specific item referents), therefore, a multiple-group confirmatory factor analysis (CFA) was also performed to establish measurement invariance between the two forms.

Results

After adaptation and refinement, a 9-item, 3-factor structure was confirmed, with the final model supporting three first-order factors that load onto a second-order factor capturing attitudes toward adopting evidence-based practices. Multiple-group CFA analyses of measurement invariance indicated there were no significant differences between the two forms.

Conclusions

Overall, this study provides a brief, flexible instrument capturing attitudes toward adopting EBPs that has high reliability and internal consistency, which support its use among general education teachers in school settings implementing evidence-based practices.

Plain Language Summary

The Evidence-Based Practice Attitudes Scale (EBPAS) is a popular instrument for measuring attitudes toward evidence-based practices (EBPs). This instrument provides valuable information during implementation initiatives, such as whether providers or front-line implementers have favorable attitudes toward a given practice. The EBPAS has been used in many different settings, such as in community-based mental health clinics, medical hospitals, and in child welfare. However, it's use in schools has been limited, and it has not yet been tested with general education teachers, who are key implementers of evidence-based practices in schools. In order to trust that the scores from an instrument are accurate, it needs to be evaluated when scaling it out to new populations and settings. One popular method to determine this is to use factor analysis, which was employed in this study. This study fills the identified gap by assessing the reliability (i.e., accuracy) and internal consistency of the EBPAS among a representative sample of general education teachers. Findings from this study indicate that the school-adapted EBPAS (S-EBPAS) is a brief, nine-item instrument that provides a reliable estimate of teachers’ attitudes toward evidence-based practices. Our results also provide evidence that the S-EBPAS can be used to capture attitudes toward specific EBPs as well as attitudes toward EBP -agnostic. This study provides a flexible instrument that can be used by school-based implementation researchers, practitioners, and intermediaries at multiple phases of implementation projects, such as when exploring a new EBP to adopt.

Rating scales institutionalise a network of logical errors and conceptual problems in research practices: A rigorous analysis showing ways to tackle psychology's crises

This article explores in-depth the metatheoretical and methodological foundations on which rating scales-by their very conception, design and application-are built and traces their historical origins. It brings together independent lines of critique from different scholars and disciplines to map out the problem landscape, which centres on the failed distinction between psychology's study phenomena (e.g., experiences, everyday constructs) and the means of their exploration (e.g., terms, data, scientific constructs)-psychologists' cardinal error. Rigorous analyses reveal a dense network of 12 complexes of problematic concepts, misconceived assumptions and fallacies that support each other, making it difficult to be identified and recognised by those (unwittingly) relying on them (e.g., various forms of reductionism, logical errors of operationalism, constructification, naïve use of language, quantificationism, statisticism, result-based data generation, misconceived nomotheticism). Through the popularity of rating scales for efficient quantitative data generation, uncritically interpreted as psychological measurement, these problems have become institutionalised in a wide range of research practices and perpetuate psychology's crises (e.g., replication, confidence, validation, generalizability). The article provides an in-depth understanding that is needed to get to the root of these problems, which preclude not just measurement but also the scientific exploration of psychology's study phenomena and thus its development as a science. From each of the 12 problem complexes; specific theoretical concepts, methodologies and methods are derived as well as key directions of development. The analyses-based on three central axioms for transdisciplinary research on individuals, (1) complexity, (2) complementarity and (3) anthropogenicity-highlight that psychologists must (further) develop an explicit metatheory and unambiguous terminology as well as concepts and theories that conceive individuals as living beings, open self-organising systems with complementary phenomena and dynamic interrelations across their multi-layered systemic contexts-thus, theories not simply of elemental properties and structures but of processes, relations, dynamicity, subjectivity, emergence, catalysis and transformation. Philosophical and theoretical foundations of approaches suited for exploring these phenomena must be developed together with methods of data generation and methods of data analysis that are appropriately adapted to the peculiarities of psychologists' study phenomena (e.g., intra-individual variation, momentariness, contextuality). Psychology can profit greatly from its unique position at the intersection of many other disciplines and can learn from their advancements to develop research practices that are suited to tackle its crises holistically.

The Intelligent Attitude: What Is Missing from Intelligence Tests

Intelligence, like creativity and wisdom, has an attitudinal component as well as an ability-based one. The attitudinal component is at least as important as the ability-based one. Theories of intelligence, in ignoring the attitudinal component of intelligence, have failed to account fully or accurately for why so many people who have relatively high levels of intelligence as an ability fail fully to deploy their ability, especially toward positive ends. The article reviews the need to view intelligence as comprising an attitude as well as an ability, and surveys reasons why people's lack of an intelligent attitude hinders their deployment of intelligence. Suggestions are made for how things could change in a positive way.

Future Objectivity Requires Perspective and Forward Combinatorial Meta-Analyses

This manuscript contributes to a future definition of objectivity by bringing together recent statements in epistemology and methodology. It outlines how improved objectivity can be achieved by systematically incorporating multiple perspectives, thereby improving the validity of science. The more result-biasing perspectives are known, the more a phenomenon of interest can be disentangled from these perspectives. Approaches that call for the integration of perspective into objectivity at the epistemological level or that systematically incorporate different perspectives at the statistical level already exist and are brought together in the manuscript. Recent developments in research methodology, such as transparency, reproducibility of research processes, pre-registration of studies, or free access to raw data, analysis strategies, and syntax, promote the explication of perspectives because they make the entire research process visible. How the explication of perspectives can be done practically is outlined in the manuscript. As a result, future research programs can be organized in such a way that meta-analyses and meta-meta-analyses can be conducted not only backward but forward and prospectively as a regular and thus well-prepared part of objectification and validation processes.

Improving Translational Paradigms in Drug Discovery and Development

Despite improved knowledge regarding disease causality, new drug targets, and enabling technologies, the attrition rate for compounds entering clinical trials has remained consistently high for several decades, with an average 90% failure rate. These failures are manifested in an inability to reproduce efficacy findings from animal models in humans and/or the occurrence of unexpected safety issues, and reflect failures in T1 translation. Similarly, an inability to sequentially demonstrate compound efficacy and safety in Phase IIa, IIb, and III clinical trials represents failures in T2 translation. Accordingly, T1 and T2 translation are colloquially termed 'valleys of death'. Since T2 translation dealt almost exclusively with clinical trials, T3 and T4 translational steps were added, with the former focused on facilitating interactions between laboratory- and population-based research and the latter on 'real world' health outcomes. Factors that potentially lead to T1/T2 compound attrition include: the absence of biomarkers to allow compound effects to be consistently tracked through development; a lack of integration/'de-siloing' of the diverse discipline-based and technical skill sets involved in drug discovery; the industrialization of drug discovery, which via volume-based goals often results in quantity being prioritized over quality; inadequate project governance and strategic oversight; and flawed decision making based on unreliable/irreproducible or incomplete data. A variety of initiatives have addressed this problem, including the NIH National Center for Advancing Translational Sciences (NCATS), which has focused on bringing an unbiased academic perspective to translation, to potentially revitalize the process. This commentary provides an overview of the basic concepts involved in translation, along with suggested changes in the conduct of biomedical research to avoid valleys of death, including the use of Translational Scoring as a tool to avoid translational attrition and the impact of the FDA Accelerated Approval Pathway in lowering the hurdle for drug approval. © 2021 Wiley Periodicals LLC.

“Are psychological attributes quantitative?” is not an empirical question: Conceptual confusions in the measurement debate

Critics of psychological measurement have accused quantitative psychologists of ignoring the empirical hypothesis that psychological phenomena are quantitative (Michell), or have claimed that it is impossible in principle to find out whether psychological phenomena are actually quantitative (Trendler). By drawing on Bennett and Hacker (2003), I argue that both criticisms do not go far enough because they sidestep the fundamental conceptual problem of the measurement debate: It is impossible to give concrete formulations of the question “Are psychological attributes quantitative?” without transgressing the boundaries of meaningful language. Conceptual confusions and questionable philosophical assumptions have contributed to the misguided idea that the quantity of psychological phenomena must or can be demonstrated empirically. First, the measurement debate is characterized by misleading examples and ambiguous terminology. Second, the idea of psychological measurement is inherently Cartesian. In summary, psychological measurement is even more problematic than Michell and Trendler have argued.

Exploration of Hospital Inpatients' Use of the Verbal Rating Scale of Pain

Background: Assessment of pain largely relies on self-report. Hospitals routinely use pain scales, such as the Verbal Rating Scale (VRS), to record patients' pain, but such scales are unidimensional, concatenating pain intensity and other dimensions of pain with significant loss of clinical information. This study explored how inpatients understand and use the VRS in a hospital setting.

Methods: Forty five participants were interviewed, with data analysed by thematic analysis, and completed a task concerned with the VRS and communication of other dimensions of pain.

Results: Participants anchored their pain experience in the physical properties of pain, its tolerability, and its impact on functioning. Their relationship to analgesic medication, personal coping styles, and experiences of staff all influenced how they used the VRS to communicate their pain.

Conclusion: Participants grounded and explained their pain in semantically similar but idiosyncratic ways. The VRS was used to combine pain intensity with multiple other elements of pain and often as a way to request analgesic medication. Pain scores need to be explored and elaborated by patient and staff, content of which will imply access to non-pharmacological resources to manage pain.

Residents’ Preferences and Perceptions toward Green Open Spaces in an Urban Area

Green open space is an important part of the natural–social ecosystem, providing ecological services that maintain the healthy development of cities and society. Residents’ perceptions of these benefits are largely related to their social-economic background as well as their familiarity with the development of green open spaces in their neighborhoods. Understanding residents’ perceptions of green open space will contribute significantly to urban planning by providing practical information that facilitates residents’ needs. Using the urban development zone (UDZ) of Wuhan, this study aims to understand residents’ preference toward green open space and their perceptions of ecological services and improvement, with the focus on the linking between social factors, preference, and views. In this study, data are collected through online questionnaire surveys and interviews. The results demonstrate how respondents’ views vary and which social factors significantly relate to them. Significant changes in natural space changes are reflected in the public’s perception of the ecological functions of these spaces. Responses to improving green open space reflect the residents’ pursuit of natural affinity and practicality. We conclude that it is better to enhance public involvement by providing residents’ views, which helps to recognize actual needs in long-term green open space planning.

Hidden framings and hidden asymmetries in the measurement of personality--A combined lens-model and frame-of-reference perspective

OBJECTIVE

The symmetry principle and the frame-of-reference perspective have each made contributions to improving the measurement of personality. Although each perspective is valuable in its own right, we argue that even greater improvement can be achieved through the combination of both. Therefore, the goal of the current article was to show the value of a combined lens-model and frame-of-reference perspective.

METHOD

We conducted a literature review to summarize relevant research findings that shed light on the interplay of both perspectives and developed an integrative model.

RESULTS

Based on the literature review and on theoretical grounds, we argue that a basic premise of the frame-of-reference literature--that personality items are open to interpretation and allow individuals to impose their own contextual framings--should be considered from a symmetry perspective. Unintended context-specificity in items may "spread" to personality facets and domains, and thus, impact the symmetry of personality measures with other criteria. As the individuals´ frames-of-reference and (a)symmetric relationships are not always apparent, we term them as "hidden."

CONCLUSIONS

The proposed combination of lens-model and frame-of-reference perspectives provides further insights into current issues in personality research and uncovers important avenues for future research.

A multi-study approach to refining ecological momentary assessment measures for use among midlife women with elevated risk for cardiovascular disease

BACKGROUND

Ample evidence indicates that everyday perceptions of the social environment can affect health behaviors; these include social comparisons (i.e., self-evaluations compared to others) and positive versus negative social interactions. However, relations between social perceptions and healthy behaviors have received little attention among specific medical populations for whom an improved understanding of behavioral determinants could inform updates to tailored interventions. Research methods that capture and differentiate between stable, person-level differences and dynamic, within-person variability in these relations would be particularly useful, both for identifying their nature in daily life and informing improvements to tailored interventions.

METHODS

We conducted a series of three formative research studies to adapt and test the measures and instructions for an ecological momentary assessment (EMA) protocol with midlife women who had elevated risk for cardiovascular disease (CVD; e.g., current diagnosis of hypertension, type 2 diabetes). Study 1 involved a pilot test of initial EMA items, sent to participants' smartphones 5 times per day for 7 days (N=13; M_Age =47, M_BMI =33.7 kg/m²), as well as brief exit interviews to identify points of confusion and suggestions for improvement. Study 2 used 1-hour, individual qualitative interviews with a new sample to elicit women's perceptions of revised items and identify additional opportunities for refinement (N=10, M_Age =52, M_BMI =29.8 kg/m²). In Study 3, a new sample of participants completed 7 days of EMA with revised items and instructions (5 times per day; N=13, M_Age =50, M_BMI =33.4 kg/m²).

RESULTS

Item performance in Study 3, including the frequencies of reporting social comparisons and interactions, was compared to that in Study 1 using multilevel modeling; these tests showed meaningful improvement in reporting patterns between Studies 1 and 3 (e.g., changes of d=0.33-0.75 where appropriate).

CONCLUSIONS

Together, findings from this series of studies demonstrate the utility of a multi-study approach to refining EMA methods for use with midlife women who have elevated CVD risk, which may generalize to other populations of interest.

Complementing Human Behavior Assessment by Leveraging Personal Ubiquitous Devices and Social Links: An Evaluation of the Peer-Ceived Momentary Assessment Method

BACKGROUND

Ecological momentary assessment (EMA) enables individuals to self-report their subjective momentary physical and emotional states. However, certain conditions, including routine observable behaviors (eg, moods, medication adherence) as well as behaviors that may suggest declines in physical or mental health (eg, memory losses, compulsive disorders) cannot be easily and reliably measured via self-reports.

OBJECTIVE

This study aims to examine a method complementary to EMA, denoted as peer-ceived momentary assessment (PeerMA), which enables the involvement of peers (eg, family members, friends) to report their perception of the individual's subjective physical and emotional states. In this paper, we aim to report the feasibility results and identified human factors influencing the acceptance and reliability of the PeerMA.

METHODS

We conducted two studies of 4 weeks each, collecting self-reports from 20 participants about their stress, fatigue, anxiety, and well-being, in addition to collecting peer-reported perceptions from 27 of their peers.

RESULTS

Preliminary results showed that some of the peers reported daily assessments for stress, fatigue, anxiety, and well-being statistically equal to those reported by the participant. We also showed how pairing assessments of participants and peers in time enables a qualitative and quantitative exploration of unique research questions not possible with EMA-only based assessments. We reported on the usability and implementation aspects based on the participants' experience to guide the use of the PeerMA to complement the information obtained via self-reports for observable behaviors and physical and emotional states among healthy individuals.

CONCLUSIONS

It is possible to leverage the PeerMA method as a complement to EMA to assess constructs that fall in the realm of observable behaviors and states in healthy individuals.

Psychology's Status as a Science: Peculiarities and Intrinsic Challenges. Moving Beyond its Current Deadlock Towards Conceptual Integration

Psychology holds an exceptional position among the sciences. Yet even after 140 years as an independent discipline, psychology is still struggling with its most basic foundations. Its key phenomena, mind and behaviour, are poorly defined (and their definition instead often delegated to neuroscience or philosophy) while specific terms and constructs proliferate. A unified theoretical framework has not been developed and its categorisation as a 'soft science' ascribes to psychology a lower level of scientificity. The article traces these problems to the peculiarities of psychology's study phenomena, their interrelations with and centrality to everyday knowledge and language (which may explain the proliferation and unclarity of terms and concepts), as well as to their complex relations with other study phenomena. It shows that adequate explorations of such diverse kinds of phenomena and their interrelations with the most elusive of all-immediate experience-inherently require a plurality of epistemologies, paradigms, theories, methodologies and methods that complement those developed for the natural sciences. Their systematic integration within just one discipline, made necessary by these phenomena's joint emergence in the single individual as the basic unit of analysis, makes psychology in fact the hardest science of all. But Galtonian nomothetic methodology has turned much of today's psychology into a science of populations rather than individuals, showing that blind adherence to natural-science principles has not advanced but impeded the development of psychology as a science. Finally, the article introduces paradigmatic frameworks that can provide solid foundations for conceptual integration and new developments.

Should We Say Goodbye to Latent Constructs to Overcome Replication Crisis or Should We Take Into Account Epistemological Considerations?

This paper discusses theoretical and epistemological problems concerning validity of psychological science in the context of latent constructs. I consider the use of latent constructs as one reason for the replicability crisis. At the moment, there exist different constructs describing the same psychological phenomena side by side, and different psychological phenomena that are reflected by the same latent construct. Hagger called them déjà-variables, which lead to a decreasing validity of measurements and inhibit a deeper understanding of psychological phenomena. To overcome this problem, I suggest a shift of theoretical and epistemological perspective on latent constructs. One main point is the explicit consideration of latent constructs as mental representations, which change objects and are changed by objects via assimilative and accommodative processes. The explicit orientation toward assimilation and accommodation allows the control of normally automatized processes that influence our understanding of psychological phenomena and their corresponding latent constructs. I argue that assimilation and accommodation are part of our research practice anyway and cause the mentioned problems. For example, taking a measurement is an assimilative process, and thus a high measurement error should lead to an increase of accommodative processes. Taking into account these considerations, I suggest consequences for research practices, for individual researchers and for the philosophy of science.