Dynamic Handwriting Analysis for Neurodegenerative Disease Assessment: A Literary Review

Multimodal Technologies for Remote Assessment of Neurological and Mental Health

PURPOSE

Automated remote assessment and monitoring of patients' neurological and mental health is increasingly becoming an essential component of the digital clinic and telehealth ecosystem, especially after the COVID-19 pandemic. This review article reviews various modalities of health information that are useful for developing such remote clinical assessments in the real world at scale.

APPROACH

We first present an overview of the various modalities of health information-speech acoustics, natural language, conversational dynamics, orofacial or full body movement, eye gaze, respiration, cardiopulmonary, and neural-which can each be extracted from various signal sources-audio, video, text, or sensors. We further motivate their clinical utility with examples of how information from each modality can help us characterize how different disorders affect different aspects of patients' spoken communication. We then elucidate the advantages of combining one or more of these modalities toward a more holistic, informative, and robust assessment.

FINDINGS

We find that combining multiple modalities of health information allows for improved scientific interpretability, improved performance on downstream health applications such as early detection and progress monitoring, improved technological robustness, and improved user experience. We illustrate how these principles can be leveraged for remote clinical assessment at scale using a real-world case study of the Modality assessment platform.

CONCLUSION

This review article motivates the combination of human-centric information from multiple modalities to measure various aspects of patients' health, arguing that remote clinical assessment that integrates this complementary information can be more effective and lead to better clinical outcomes than using any one data stream in isolation.

In Silico Decoding of Parkinson's: Speech & Writing Analysis

Background: Parkinson's disease (PD) has transitioned from a rare condition in 1817 to the fastest-growing neurological disorder globally. The significant increase in cases from 2.5 million in 1990 to 6.1 million in 2016, coupled with predictions of a further doubling by 2040, underscores an impending healthcare challenge. This escalation aligns with global demographic shifts, including rising life expectancy and a growing global population. The economic impact, notably in the U.S., reached $51.9 billion in 2017, with projections suggesting a 46% increase by 2037, emphasizing the substantial socio-economic implications for both patients and caregivers. Coupled with a worldwide demand for health workers that is expected to rise to 80 million by 2030, we have fertile ground for a pandemic. Methods: Our transdisciplinary research focused on early PD detection through running speech and continuous handwriting analysis, incorporating medical, biomedical engineering, AI, and linguistic expertise. The cohort comprised 30 participants, including 20 PD patients at stages 1-4 on the Hoehn and Yahr scale and 10 healthy controls. We employed advanced AI techniques to analyze correlation plots generated from speech and handwriting features, aiming to identify prodromal PD biomarkers. Results: The study revealed distinct speech and handwriting patterns in PD patients compared to controls. Our ParkinsonNet model demonstrated high predictive accuracy, with F1 scores of 95.74% for speech and 96.72% for handwriting analyses. These findings highlight the potential of speech and handwriting as effective early biomarkers for PD. Conclusions: The integration of AI as a decision support system in analyzing speech and handwriting presents a promising approach for early PD detection. This methodology not only offers a novel diagnostic tool but also contributes to the broader understanding of PD's early manifestations. Further research is required to validate these findings in larger, diverse cohorts and to integrate these tools into clinical practice for timely PD pre-diagnosis and management.

Computerized assessment of handwriting in de novo Parkinson's disease: A kinematic study

INTRODUCTION

Dysgraphia, a recognized PD motor symptom, lacks effective clinical assessment. Current evaluation relies on motor assessment scales. Computational methods introduced over the past decade offer an objective dysgraphia assessment, considering size, duration, speed, and handwriting fluency. Objective evaluation of dysgraphia may be of help for early diagnosis of PD.

OBJECTIVE

Computerized assessment of dysgraphia in de novo PD patients and its correlation with clinical scales.

METHODS

We evaluated 38 recently diagnosed, premedication PD patients and age-matched controls without neurological disorders. Participants wrote "La casa de Pamplona es bonita" three times on paper and once on a Wacom tablet under the paper, totaling four phrases. Writing segments of 5-10 s were analyzed. The Wacom tablet captured kinematic data, including mean velocity, mean acceleration, and pen pressure. Data were saved in.svc format and analyzed using specialized software developed by Tecnocampus Mataró. Standard clinical practice data, Hoehn & Yahr staging, and UPDRS scales were used for evaluation.

RESULTS

Significant kinematic differences existed; patients had lower mean speed (27 ± 12 vs. 48 ± 18, p < 0.0001) and mean acceleration (7.2 ± 3.9 vs. 15.01 ± 7, p < 0.0001) than controls. Mean speed and mean acceleration correlated significantly with UPDRS III scores (speed: r = -0.52, p < 0.0007; acceleration: r = 0.60, p < 0.0001), indicating kinematic parameters' potential in PD evaluation.

CONCLUSIONS

Dysgraphia is identifiable in PD patients, even de novo, indicating an early symptom and correlates with clinical scales, offering potential for objective PD patient evaluation.

How word semantics and phonology affect handwriting of Alzheimer's patients: A machine learning based analysis

Using kinematic properties of handwriting to support the diagnosis of neurodegenerative disease is a real challenge: non-invasive detection techniques combined with machine learning approaches promise big steps forward in this research field. In literature, the tasks proposed focused on different cognitive skills to elicitate handwriting movements. In particular, the meaning and phonology of words to copy can compromise writing fluency. In this paper, we investigated how word semantics and phonology affect the handwriting of people affected by Alzheimer's disease. To this aim, we used the data from six handwriting tasks, each requiring copying a word belonging to one of the following categories: regular (have a predictable phoneme-grapheme correspondence, e.g., cat), non-regular (have atypical phoneme-grapheme correspondence, e.g., laugh), and non-word (non-meaningful pronounceable letter strings that conform to phoneme-grapheme conversion rules). We analyzed the data using a machine learning approach by implementing four well-known and widely-used classifiers and feature selection. The experimental results showed that the feature selection allowed us to derive a different set of highly distinctive features for each word type. Furthermore, non-regular words needed, on average, more features but achieved excellent classification performance: the best result was obtained on a non-regular, reaching an accuracy close to 90%.

Handwriting Markers for the Onset of Alzheimer's Disease

INTRODUCTION

Alzheimer's disease has an impact on handwriting (AD). Numerous researchers reported that fact. Therefore, examining handwriting characteristics could be a useful way to screen for AD. The aim of the article is to present the reliability and effectiveness of the AD-HS tool.

METHODS

Most of the existing studies examine either linguistic manifestations of writing or certain motor functions. However, handwriting is a complex of cognitive and motor activities. Since the influence of AD on handwriting is individual, it is important to analyze the complete set of handwriting features. The AD-HS instrument is based on this principle. Validation of the AD-HS instrument for revealing cognitive impairment in AD-diagnosed persons in comparison to the control group. The study is based on the evaluation of free handwritten texts. AD-HS includes 40 handwriting and 2 linguistic features of handwritten texts. It is based on the standard protocol for handwriting analysis. The cumulative evaluation of all features builds a quantitative AD-Indicator (ADI) as a marker of possible AD conditions. The analyzed experiment includes 53 AD-diagnosed persons and a control group of 192 handwriting specimens from the existing database.

RESULTS

AD-HS shows a distinct difference in evaluated ADI for the participants (the mean value equals 0.49) and the control group (the mean value equals 0.28).

CONCLUSION

The handwriting marker of AD could be an effective supplement instrument for earlier screening. It is also useful when traditional biomarkers and neurological tests could not be applied. AD-HS can accompany therapy as an indication of its effect on a person.

Direct mass spectrometric imaging of document handwriting with laser desorption ionization and post ultraviolet photodissociation

Handwriting represents personal education and physical or psychological states. This work describes a chemical imaging technique for document evaluation that combines laser desorption ionization with post ultraviolet photo-induced dissociation (LDI-UVPD) in mass spectrometry. Taken the advantages of chromophores in ink dyes, handwriting papers were subjected to direct laser desorption ionization without additional matrix materials. It is a surface-sensitive analytical method that uses a low intensity pulsed laser at 355 nm to remove chemical components from very outermost surfaces of overlapped handwritings. Meanwhile, the transfer of photoelectrons to those compounds leads to the ionization and the formation of radical anions. The gentle evaporation and ionization property enable the dissection of chronological orders. Paper documents maintain intact without extensive damages after laser irradiation. The evolving plume resulting from the irradiation of the 355 nm laser is fired by the second ultraviolet laser at 266 nm that is in parallel to the sample surface. In contrast to collision activated dissociation in tandem MS/MS, such post ultraviolet photodissociation generates much more different fragment ions through electron-directed specific cleavages of chemical bonds. LDI-UVPD can not only provide graphic representation of chemical components but also reveal hidden dynamic features such as alterations, pressures and aging.

Handwriting Changes in Alzheimer's Disease: A Systematic Review

BACKGROUND

Handwriting is a complex process involving fine motor skills, kinesthetic components, and several cognitive domains, often impaired by Alzheimer's disease (AD).

OBJECTIVE

Provide a systematic review of handwriting changes in AD, highlighting the effects on motor, visuospatial and linguistic features, and to identify new research topics.

METHODS

A search was conducted on PubMed, Scopus, and Web of Science to identify studies on AD and handwriting. The review followed PRISMA norms and analyzed 91 articles after screening and final selection.

RESULTS

Handwriting is impaired at all levels of the motor-cognitive hierarchy in AD, particularly in text, with higher preservation of signatures. Visuospatial and linguistic features were more affected. Established findings for motor features included higher variability in AD signatures, higher in-air/on-surface time ratio and longer duration in text, longer start time/reaction time, and lower fluency. There were conflicting findings for pressure and velocity in motor features, as well as size, legibility, and pen lifts in general features. For linguistic features, findings were contradictory for error patterns, as well as the association between agraphia and severity of cognitive deficits.

CONCLUSIONS

Further re-evaluation studies are needed to clarify the divergent results on motor, general, and linguistic features. There is also a lack of research on the influence of AD on signatures and the effect of AD variants on handwriting. Such research would have an impact on clinical management (e.g., for early detection and patient follow-up using handwriting tasks), or forensic examination aimed at signatory identification.

On Extracting Digitized Spiral Dynamics’ Representations: A Study on Transfer Learning for Early Alzheimer’s Detection

This work proposes a decision-aid tool for detecting Alzheimer’s disease (AD) at an early stage, based on the Archimedes spiral, executed on a Wacom digitizer. Our work assesses the potential of the task as a dynamic gesture and defines the most pertinent methodology for exploiting transfer learning to compensate for sparse data. We embed directly in spiral trajectory images, kinematic time functions. With transfer learning, we perform automatic feature extraction on such images. Experiments on 30 AD patients and 45 healthy controls (HC) show that the extracted features allow a significant improvement in sensitivity and accuracy, compared to raw images. We study at which level of the deep network features have the highest discriminant capabilities. Results show that intermediate-level features are the best for our specific task. Decision fusion of experts trained on such descriptors outperforms low-level fusion of hybrid images. When fusing decisions of classifiers trained on the best features, from pressure, altitude, and velocity images, we obtain 84% of sensitivity and 81.5% of accuracy, achieving an absolute improvement of 22% in sensitivity and 7% in accuracy. We demonstrate the potential of the spiral task for AD detection and give a complete methodology based on off-the-shelf features.

Comparison of CNN-Learned vs. Handcrafted Features for Detection of Parkinson's Disease Dysgraphia in a Multilingual Dataset

Parkinson's disease dysgraphia (PDYS), one of the earliest signs of Parkinson's disease (PD), has been researched as a promising biomarker of PD and as the target of a noninvasive and inexpensive approach to monitoring the progress of the disease. However, although several approaches to supportive PDYS diagnosis have been proposed (mainly based on handcrafted features (HF) extracted from online handwriting or the utilization of deep neural networks), it remains unclear which approach provides the highest discrimination power and how these approaches can be transferred between different datasets and languages. This study aims to compare classification performance based on two types of features: features automatically extracted by a pretrained convolutional neural network (CNN) and HF designed by human experts. Both approaches are evaluated on a multilingual dataset collected from 143 PD patients and 151 healthy controls in the Czech Republic, United States, Colombia, and Hungary. The subjects performed the spiral drawing task (SDT; a language-independent task) and the sentence writing task (SWT; a language-dependent task). Models based on logistic regression and gradient boosting were trained in several scenarios, specifically single language (SL), leave one language out (LOLO), and all languages combined (ALC). We found that the HF slightly outperformed the CNN-extracted features in all considered evaluation scenarios for the SWT. In detail, the following balanced accuracy (BACC) scores were achieved: SL—0.65 (HF), 0.58 (CNN); LOLO—0.65 (HF), 0.57 (CNN); and ALC—0.69 (HF), 0.66 (CNN). However, in the case of the SDT, features extracted by a CNN provided competitive results: SL—0.66 (HF), 0.62 (CNN); LOLO—0.56 (HF), 0.54 (CNN); and ALC—0.60 (HF), 0.60 (CNN). In summary, regarding the SWT, the HF outperformed the CNN-extracted features over 6% (mean BACC of 0.66 for HF, and 0.60 for CNN). In the case of the SDT, both feature sets provided almost identical classification performance (mean BACC of 0.60 for HF, and 0.58 for CNN).

Automated Analysis of Drawing Process to Estimate Global Cognition in Older Adults: Preliminary International Validation on the US and Japan Data Sets

BACKGROUND

With the aging of populations worldwide, early detection of cognitive impairments has become a research and clinical priority, particularly to enable preventive intervention for dementia. Automated analysis of the drawing process has been studied as a promising means for lightweight, self-administered cognitive assessment. However, this approach has not been sufficiently tested for its applicability across populations.

OBJECTIVE

The aim of this study was to evaluate the applicability of automated analysis of the drawing process for estimating global cognition in community-dwelling older adults across populations in different nations.

METHODS

We collected drawing data with a digital tablet, along with Montreal Cognitive Assessment (MoCA) scores for assessment of global cognition, from 92 community-dwelling older adults in the United States and Japan. We automatically extracted 6 drawing features that characterize the drawing process in terms of the drawing speed, pauses between drawings, pen pressure, and pen inclinations. We then investigated the association between the drawing features and MoCA scores through correlation and machine learning-based regression analyses.

RESULTS

We found that, with low MoCA scores, there tended to be higher variability in the drawing speed, a higher pause:drawing duration ratio, and lower variability in the pen's horizontal inclination in both the US and Japan data sets. A machine learning model that used drawing features to estimate MoCA scores demonstrated its capability to generalize from the US dataset to the Japan dataset (R2=0.35; permutation test, P<.001).

CONCLUSIONS

This study presents initial empirical evidence of the capability of automated analysis of the drawing process as an estimator of global cognition that is applicable across populations. Our results suggest that such automated analysis may enable the development of a practical tool for international use in self-administered, automated cognitive assessment.

A survey of visual and procedural handwriting analysis for neuropsychological assessment

To date, Artificial Intelligence systems for handwriting and drawing analysis have primarily targeted domains such as writer identification and sketch recognition. Conversely, the automatic characterization of graphomotor patterns as biomarkers of brain health is a relatively less explored research area. Despite its importance, the work done in this direction is limited and sporadic. This paper aims to provide a survey of related work to provide guidance to novice researchers and highlight relevant study contributions. The literature has been grouped into “visual analysis techniques” and “procedural analysis techniques”. Visual analysis techniques evaluate offline samples of a graphomotor response after completion. On the other hand, procedural analysis techniques focus on the dynamic processes involved in producing a graphomotor reaction. Since the primary goal of both families of strategies is to represent domain knowledge effectively, the paper also outlines the commonly employed handwriting representation and estimation methods presented in the literature and discusses their strengths and weaknesses. It also highlights the limitations of existing processes and the challenges commonly faced when designing such systems. High-level directions for further research conclude the paper.

Patients' Self-Report and Handwriting Performance Features as Indicators for Suspected Mild Cognitive Impairment in Parkinson's Disease

Early identification of mild cognitive impairment (MCI) in Parkinson's disease (PD) patients can lessen emotional and physical complications. In this study, a cognitive functional (CF) feature using cognitive and daily living items of the Unified Parkinson's Disease Rating Scale served to define PD patients as suspected or not for MCI. The study aimed to compare objective handwriting performance measures with the perceived general functional abilities (PGF) of both groups, analyze correlations between handwriting performance measures and PGF for each group, and find out whether participants' general functional abilities, depression levels, and digitized handwriting measures predicted this CF feature. Seventy-eight participants diagnosed with PD by a neurologist (25 suspected for MCI based on the CF feature) completed the PGF as part of the Daily Living Questionnaire and wrote on a digitizer-affixed paper in the Computerized Penmanship Handwriting Evaluation Test. Results indicated significant group differences in PGF scores and handwriting stroke width, and significant medium correlations between PGF score, pen-stroke width, and the CF feature. Regression analyses indicated that PGF scores and mean stroke width accounted for 28% of the CF feature variance above age. Nuances of perceived daily functional abilities validated by objective measures may contribute to the early identification of suspected PD-MCI.

From online handwriting to synthetic images for Alzheimer's disease detection using a deep transfer learning approach

Early diagnosis of neurodegenerative disorders, such as Alzheimer's Disease (AD), is very important to reduce their effects and to improve both quality and life expectancy of patients. In this context, it is generally agreed that handwriting is one of the first skills altered by the onset of such diseases. For this reason, the analysis of handwriting and the study of its alterations have become of great interest in order to formulate the diagnosis as soon as possible. A fundamental aspect for the use of these techniques is the definition of effective features, which allows the system to distinguish the natural alterations of handwriting due to age, from those caused by neurodegenerative disorders. Starting from these considerations, the aim of our study is to verify whether the combined use of both shape and dynamic features allows a decision support system to improve performance for AD diagnosis. To this purpose, starting from a database of on-line handwriting samples, we generated for each of them an off-line synthetic color image, where the color of each elementary trait encodes, in the three RGB channels, the dynamic information associated with that trait. In order to verify the importance and the specific role played by shape information, we also generated an off-line synthetic binary image for each handwriting sample, where background pixels have white color, while those corresponding to the traits have black color. Finally, we exploited the ability of Convolutional Neural Network (CNN) to automatically extract features on both color and binary images. We carried out a large set of experiments for comparing the results obtained by using on-line features with those obtained by using the off-line features provided by CNN on both color and binary images.

Analysis of Fine Motor Skills in Essential Tremor: Combining Neuroimaging and Handwriting Biomarkers for Early Management

Essential tremor (ET) is a highly prevalent neurological disorder characterized by action-induced tremors involving the hand, voice, head, and/or face. Importantly, hand tremor is present in nearly all forms of ET, resulting in impaired fine motor skills and diminished quality of life. To advance early diagnostic approaches for ET, automated handwriting tasks and magnetic resonance imaging (MRI) offer an opportunity to develop early essential clinical biomarkers. In this study, we present a novel approach for the early clinical diagnosis and monitoring of ET based on integrating handwriting and neuroimaging analysis. We demonstrate how the analysis of fine motor skills, as measured by an automated Archimedes' spiral task, is correlated with neuroimaging biomarkers for ET. Together, we present a novel modeling approach that can serve as a complementary and promising support tool for the clinical diagnosis of ET and a large range of tremors.

Cartesian genetic programming for diagnosis of Parkinson disease through handwriting analysis: Performance vs. interpretability issues

In the last decades, early disease identification through non-invasive and automatic methodologies has gathered increasing interest from the scientific community. Among others, Parkinson's disease (PD) has received special attention in that it is a severe and progressive neuro-degenerative disease. As a consequence, early diagnosis would provide more effective and prompt care strategies, that cloud successfully influence patients' life expectancy. However, the most performing systems implement the so called black-box approach, which do not provide explicit rules to reach a decision. This lack of interpretability, has hampered the acceptance of those systems by clinicians and their deployment on the field. In this context, we perform a thorough comparison of different machine learning (ML) techniques, whose classification results are characterized by different levels of interpretability. Such techniques were applied for automatically identify PD patients through the analysis of handwriting and drawing samples. Results analysis shows that white-box approaches, such as Cartesian Genetic Programming and Decision Tree, allow to reach a twofold goal: support the diagnosis of PD and obtain explicit classification models, on which only a subset of features (related to specific tasks) were identified and exploited for classification. Obtained classification models provide important insights for the design of non-invasive, inexpensive and easy to administer diagnostic protocols. Comparison of different ML approaches (in terms of both accuracy and interpretability) has been performed on the features extracted from the handwriting and drawing samples included in the publicly available PaHaW and NewHandPD datasets. The experimental findings show that the Cartesian Genetic Programming outperforms the white-box methods in accuracy and the black-box ones in interpretability.

Handwriting Biometrics: Applications and Future Trends in e-Security and e-Health

Online handwritten analysis presents many applications in e-security, signature biometrics being the most popular but not the only one. Handwriting analysis also has an important set of applications in e-health. Both kinds of applications (e-security and e-health) have some unsolved questions and relations among them that should be addressed in the next years. We summarize the state of the art and applications based on handwriting signals. Later on, we focus on the main achievements and challenges that should be addressed by the scientific community, providing a guide for future research. Among all the points discussed in this article, we remark the importance of considering security, health, and metadata from a joint perspective. This is especially critical due to the risks inherent when using these behavioral signals.