Beyond Signatures: Leveraging Sensor Fusion for Contextual Handwriting Recognition

This paper deals with biometric identification based on unique patterns and characteristics of an individual's handwriting, focusing on the dynamic writing process on a touchscreen device. Related work in this domain indicates the dominance of specific research approaches. Namely, in most cases, only the signature is analyzed, verification methods are more prevalent than recognition methods, and the provided solutions are mainly based on using a particular device or specific sensor for collecting biometric data. In this context, our work aims to fill the identified research gap by introducing a new handwriting-based user recognition technique. The proposed approach implements the concept of sensor fusion and does not rely exclusively on signatures for recognition but also includes other forms of handwriting, such as short sentences, words, or individual letters. Additionally, two different ways of handwriting input, using a stylus and a finger, are introduced into the analysis. In order to collect data on the dynamics of handwriting and signing, a specially designed apparatus was used with various sensors integrated into common smart devices, along with additional external sensors and accessories. A total of 60 participants took part in a controlled experiment to form a handwriting biometrics dataset for further analysis. To classify participants' handwriting, custom architecture CNN models were utilized for feature extraction and classification tasks. The obtained results showed that the proposed handwriting recognition system achieves accuracies of 0.982, 0.927, 0.884, and 0.661 for signatures, words, short sentences, and individual letters, respectively. We further investigated the main effects of the input modality and the train set's size on the system's accuracy. Finally, an ablation study was carried out to analyze the impact of individual sensors within the fusion-based setup.

Privacy-Preserving Biometric Verification With Handwritten Random Digit String

Handwriting verification has stood as a steadfast identity authentication method for decades. However, this technique risks potential privacy breaches due to the inclusion of personal information in handwritten biometrics such as signatures. To address this concern, we propose using the Random Digit String (RDS) for privacy-preserving handwriting verification. This approach allows users to authenticate themselves by writing an arbitrary digit sequence, effectively ensuring privacy protection. To evaluate the effectiveness of RDS, we construct a new HRDS4BV dataset composed of online naturally handwritten RDS. Unlike conventional handwriting, RDS encompasses unconstrained and variable content, posing significant challenges for modeling consistent personal writing style. To surmount this, we propose the Pattern Attentive VErification Network (PAVENet), along with a Discriminative Pattern Mining (DPM) module. DPM adaptively enhances the recognition of consistent and discriminative writing patterns, thus refining handwriting style representation. Through comprehensive evaluations, we scrutinize the applicability of online RDS verification and showcase a pronounced outperformance of our model over existing methods. Furthermore, we discover a noteworthy forgery phenomenon that deviates from prior findings and discuss its positive impact in countering malicious impostor attacks. Substantially, our work underscores the feasibility of privacy-preserving biometric verification and propels the prospects of its broader acceptance and application.

Correlation Fuzzy measure of multivariate time series for signature recognition

Distinguishing different time series, which is determinant or stochastic, is an important task in signal processing. In this work, a correlation measure constructs Correlation Fuzzy Entropy (CFE) to discriminate Chaos and stochastic series. It can be employed to distinguish chaotic signals from ARIMA series with different noises. With specific embedding dimensions, we implemented the CFE features by analyzing two available online signature databases MCYT-100 and SVC2004. The accurate rates of the CFE-based models exceed 99.3%.

Online Signature Biometrics for Mobile Devices

This paper addresses issues concerning biometric authentication based on handwritten signatures. Our research aimed to check whether a handwritten signature acquired with a mobile device can effectively verify a user's identity. We present a novel online signature verification method using coordinates of points and pressure values at each point collected with a mobile device. Convolutional neural networks are used for signature verification. In this paper, three neural network models are investigated, i.e., two self-made light SigNet and SigNetExt models and the VGG-16 model commonly used in image processing. The convolutional neural networks aim to determine whether the acquired signature sample matches the class declared by the signer. Thus, the scenario of closed set verification is performed. The effectiveness of our method was tested on signatures acquired with mobile phones. We used the subset of the multimodal database, MobiBits, that was captured using a custom-made application and consists of samples acquired from 53 people of diverse ages. The experimental results on accurate data demonstrate that developed architectures of deep neural networks can be successfully used for online handwritten signature verification. We achieved an equal error rate (EER) of 0.63% for random forgeries and 6.66% for skilled forgeries.

Analysis of Gender Differences in Online Handwriting Signals for Enhancing e-Health and e-Security Applications

Handwriting is a complex perceptual–motor skill that is mastered around the age of 8. Although its computerized analysis has been utilized in many biometric and digital health applications, the possible effect of gender is frequently neglected. The aim of this paper is to analyze different online handwritten tasks performed by intact subjects and explore gender differences in commonly used temporal, kinematic, and dynamic features. The differences were explored in the BIOSECUR-ID database. We have identified a significant gender difference in on-surface/in-air time of genuine and skilled forgery signatures, on-surface time in cursive letters and numbers, and pressure, speed, and acceleration in text written in capital letters. Our findings accent the need to consider gender as an important confounding factor in studies dealing with online handwriting signal processing.

SynSig2Vec: Forgery-Free Learning of Dynamic Signature Representations by Sigma Lognormal-Based Synthesis and 1D CNN

Handwritten signature verification is a challenging task because signatures of a writer may be skillfully imitated by a forger. As skilled forgeries are generally difficult to acquire for training, in this paper, we propose a deep learning-based dynamic signature verification framework, SynSig2Vec, to address the skilled forgery attack without training with any skilled forgeries. Specifically, SynSig2Vec consists of a novel learning-by-synthesis method for training and a 1D convolutional neural network model, called Sig2Vec, for signature representation extraction. The learning-by-synthesis method first applies the Sigma Lognormal model to synthesize signatures with different distortion levels for genuine template signatures, and then learns to rank these synthesized samples in a learnable representation space based on average precision optimization. The representation space is achieved by the proposed Sig2Vec model, which is designed to extract fixed-length representations from dynamic signatures of arbitrary lengths. Through this training method, the Sig2Vec model can extract extremely effective signature representations for verification. Our SynSig2Vec framework requires only genuine signatures for training, yet achieves state-of-the-art performance on the largest dynamic signature database to date, DeepSignDB, in both skilled forgery and random forgery scenarios. Source codes of SynSig2Vec will be available at https://github.com/LaiSongxuan/SynSig2Vec.

Evolutionary Algorithm for Selecting Dynamic Signatures Partitioning Approach

In the verification of identity, the aim is to increase effectiveness and reduce involvement of verified users. A good compromise between these issues is ensured by dynamic signature verification. The dynamic signature is represented by signals describing the position of the stylus in time. They can be used to determine the velocity or acceleration signal. Values of these signals can be analyzed, interpreted, selected, and compared. In this paper, we propose an approach that: (a) uses an evolutionary algorithm to create signature partitions in the time and velocity domains; (b) selects the most characteristic partitions in terms of matching with reference signatures; and (c) works individually for each user, eliminating the need of using skilled forgeries. The proposed approach was tested using Biosecure DS2 database which is a part of the DeepSignDB, a database with genuine dynamic signatures. Our simulations confirmed the correctness of the adopted assumptions.

Deformation Adjustment with Single Real Signature Image for Biometric Verification Using CNN

Signature verification is the widely used biometric verification method for maintaining individual privacy. It is generally used in legal documents and in financial transactions. A vast range of research has been done so far to tackle different system issues, but there are various hot issues that remain unaddressed. The scale and orientation of the signatures are some issues to address, and the deformation of the signature within the genuine examples is the most critical for the verification system. The extent of this deformation is the basis for verifying a given sample as a genuine or forgery signature, but in the case of only a single signature sample for a class, the intra-class variation is not available for decision-making, making the task difficult. Besides this, most real-world signature verification repositories have only one genuine sample, and the verification system is abiding to verify the query signature with a single target sample. In this work, we utilize a two-phase system requiring only one target signature image to verify a query signature image. It takes care of the target signature's scaling, orientation, and spatial translation in the first phase. It creates a transformed signature image utilizing the affine transformation matrix predicted by a deep neural network. The second phase uses this transformed sample image and verifies the given sample as the target signature with the help of another deep neural network. The GPDS synthetic and MCYT datasets are used for the experimental analysis. The performance analysis of the proposed method is carried out on FAR, FRR, and AER measures. The proposed method obtained leading performance with 3.56 average error rate (AER) on GPDS synthetic, 4.15 AER on CEDAR, and 3.51 AER on MCYT-75 datasets.

Mobile Contactless Fingerprint Recognition: Implementation, Performance and Usability Aspects

This work presents an automated contactless fingerprint recognition system for smartphones. We provide a comprehensive description of the entire recognition pipeline and discuss important requirements for a fully automated capturing system. In addition, our implementation is made publicly available for research purposes. During a database acquisition, a total number of 1360 contactless and contact-based samples of 29 subjects are captured in two different environmental situations. Experiments on the acquired database show a comparable performance of our contactless scheme and the contact-based baseline scheme under constrained environmental influences. A comparative usability study on both capturing device types indicates that the majority of subjects prefer the contactless capturing method. Based on our experimental results, we analyze the impact of the current COVID-19 pandemic on fingerprint recognition systems. Finally, implementation aspects of contactless fingerprint recognition are summarized.

Online Signature Verification Systems on a Low-Cost FPGA

This paper describes three different approaches for the implementation of an online signature verification system on a low-cost FPGA. The system is based on an algorithm, which operates on real numbers using the double-precision floating-point IEEE 754 format. The double-precision computations are replaced by simpler formats, without affecting the biometrics performance, in order to permit efficient implementations on low-cost FPGA families. The first approach is an embedded system based on MicroBlaze, a 32-bit soft-core microprocessor designed for Xilinx FPGAs, which can be configured by including a single-precision floating-point unit (FPU). The second implementation attaches a hardware accelerator to the embedded system to reduce the execution time on floating-point vectors. The last approach is a custom computing system, which is built from a large set of arithmetic circuits that replace the floating-point data with a more efficient representation based on fixed-point format. The latter system provides a very high runtime acceleration factor at the expense of using a large number of FPGA resources, a complex development cycle and no flexibility since it cannot be adapted to other biometric algorithms. By contrast, the first system provides just the opposite features, while the second approach is a mixed solution between both of them. The experimental results show that both the hardware accelerator and the custom computing system reduce the execution time by a factor ×7.6 and ×201 but increase the logic FPGA resources by a factor ×2.3 and ×5.2, respectively, in comparison with the MicroBlaze embedded system.

Online signature verification using signature down-sampling and signer-dependent sampling frequency

Online signature verification considers signatures as time sequences of different measurements of the signing instrument. These signals are captured on digital devices and therefore consist of a discrete number of samples. To enrich or simplify this information, several verifiers employ resampling and interpolation as a preprocessing step to improve their results; however, their design decisions may be difficult to generalize. This study investigates the direct effect of the sampling rate of the input signals on the accuracy of online signature verification systems without using interpolation techniques and proposes a novel online signature verification system based on a signer-dependent sampling frequency. Twenty verifier configurations were created for five different public signature databases and a variety of popular preprocessing approaches and evaluated for 20–40 different sampling rates. Our results show that there is an optimal range for the sampling frequency and the number of sample points that minimizes the error rate of a verifier. A sampling frequency range of 15–50 Hz and a signature point count of 60–240 provided the best accuracies in our experiments. As expected, lower ranges showed inaccurate results; interestingly, however, higher frequencies often decreased the verification accuracy. The results show that one can achieve better or at least the same verification accuracies faster by down-sampling the online signatures before further processing. The proposed system achieved competitive results to state-of-the-art systems for different databases by using the optimal sampling frequency. We also studied the effect of choosing individual sampling frequencies for each signer and proposed a signature verification system based on signer-dependent sampling frequency. The proposed system was tested using 500 different verification methods and improved the accuracy in 92% of the test cases compared to the usage of the original frequency.

Offline Signature Identification and Verification Based on Capsule Representations

Offline signature is one of the frequently used biometric traits in daily life and yet skilled forgeries are posing a great challenge for offline signature verification. To differentiate forgeries, a variety of research has been conducted on hand-crafted feature extraction methods until now. However, these methods have recently been set aside for automatic feature extraction methods such as Convolutional Neural Networks (CNN). Although these CNN-based algorithms often achieve satisfying results, they require either many samples in training or pre-trained network weights. Recently, Capsule Network has been proposed to model with fewer data by using the advantage of convolutional layers for automatic feature extraction. Moreover, feature representations are obtained as vectors instead of scalar activation values in CNN to keep orientation information. Since signature samples per user are limited and feature orientations in signature samples are highly informative, this paper first aims to evaluate the capability of Capsule Network for signature identification tasks on three benchmark databases. Capsule Network achieves 97 96, 94 89, 95 and 91% accuracy on CEDAR, GPDS-100 and MCYT databases for 64×64 and 32×32 resolutions, which are lower than usual, respectively. The second aim of the paper is to generalize the capability of Capsule Network concerning the verification task. Capsule Network achieves average 91, 86, and 89% accuracy on CEDAR, GPDS-100 and MCYT databases for 64×64 resolutions, respectively. Through this evaluation, the capability of Capsule Network is shown for offline verification and identification tasks.

Handwriting Biometrics: Applications and Future Trends in e-Security and e-Health

Online handwritten analysis presents many applications in e-security, signature biometrics being the most popular but not the only one. Handwriting analysis also has an important set of applications in e-health. Both kinds of applications (e-security and e-health) have some unsolved questions and relations among them that should be addressed in the next years. We summarize the state of the art and applications based on handwriting signals. Later on, we focus on the main achievements and challenges that should be addressed by the scientific community, providing a guide for future research. Among all the points discussed in this article, we remark the importance of considering security, health, and metadata from a joint perspective. This is especially critical due to the risks inherent when using these behavioral signals.

Enhancing Security on Touch-Screen Sensors with Augmented Handwritten Signatures

We aim at enhancing personal identity security on mobile touch-screen sensors by augmenting handwritten signatures with specific additional information at the enrollment phase. Our former works on several available and private data sets acquired on different sensors demonstrated that there are different categories of signatures that emerge automatically with clustering techniques, based on an entropy-based data quality measure. The behavior of such categories is totally different when confronted to automatic verification systems in terms of vulnerability to attacks. In this paper, we propose a novel and original strategy to reinforce identity security by enhancing signature resistance to attacks, assessed per signature category, both in terms of data quality and verification performance. This strategy operates upstream from the verification system, at the sensor level, by enriching the information content of signatures with personal handwritten inputs of different types. We study this strategy on different signature types of 74 users, acquired in uncontrolled mobile conditions on a largely deployed mobile touch-screen sensor. Our analysis per writer category revealed that adding alphanumeric (date) and handwriting (place) information to the usual signature is the most powerful augmented signature type in terms of verification performance. The relative improvement for all user categories is of at least 93% compared to the usual signature.

iDeLog: Iterative Dual Spatial and Kinematic Extraction of Sigma-Lognormal Parameters

The Kinematic Theory of rapid movements and its associated Sigma-Lognormal model have been extensively used in a large variety of applications. While the physical and biological meaning of the model have been widely tested and validated for rapid movements, some shortcomings have been detected when it is used with continuous long and complex movements. To alleviate such drawbacks, and inspired by the motor equivalence theory and a conceivable visual feedback, this paper proposes a novel framework to extract the Sigma-Lognormal parameters, namely iDeLog. Specifically, iDeLog consists of two steps. The first one, influenced by the motor equivalence model, separately derives an initial action plan defined by a set of virtual points and angles from the trajectory and a sequence of lognormals from the velocity. In the second step, based on a hypothetical visual feedback compatible with an open-loop motor control, the virtual target points of the action plan are iteratively moved to improve the matching between the observed and reconstructed trajectory and velocity. During experiments conducted with handwritten signatures, iDeLog obtained promising results as compared to the previous development of the Sigma-Lognormal.

Anthropomorphic Features for On-Line Signatures

Many features have been proposed in on-line signature verification. Generally, these features rely on the position of the on-line signature samples and their dynamic properties, as recorded by a tablet. This paper proposes a novel feature space to describe efficiently on-line signatures. Since producing a signature requires a skeletal arm system and its associated muscles, the new feature space is based on characterizing the movement of the shoulder, the elbow and the wrist joints when signing. As this motion is not directly obtained from a digital tablet, the new features are calculated by means of a virtual skeletal arm (VSA) model, which simulates the architecture of a real arm and forearm. Specifically, the VSA motion is described by its 3D joint position and its joint angles. These anthropomorphic features are worked out from both pen position and orientation through the VSA forward and direct kinematic model. The anthropomorphic features' robustness is proved by achieving state-of-the-art performance with several verifiers and multiple benchmarks on third party signature databases, which were collected with different devices and in different languages and scripts.

Online Signature Verification Based on a Single Template via Elastic Curve Matching

Person verification using online handwritten signatures is one of the most widely researched behavior-biometrics. Many signature verification systems typically require five, ten, or even more signatures for an enrolled user to provide an accurate verification of the claimed identity. To mitigate this drawback, this paper proposes a new elastic curve matching using only one reference signature, which we have named the curve similarity model (CSM). In the CSM, we give a new definition of curve similarity and its calculation method. We use evolutionary computation (EC) to search for the optimal matching between two curves under different similarity transformations, so as to obtain the similarity distance between two curves. Referring to the geometric similarity property, curve similarity can realize translation, stretching and rotation transformation between curves, thus adapting to the inconsistency of signature size, position and rotation angle in signature curves. In the matching process of signature curves, we design a sectional optimal matching algorithm. On this basis, for each section, we develop a new consistent and discriminative fusion feature extraction for identifying the similarity of signature curves. The experimental results show that our system achieves the same performance with five samples assessed with multiple state-of-the-art automatic signature verifiers and multiple datasets. Furthermore, it suggests that our system, with a single reference signature, is capable of achieving a similar performance to other systems with up to five signatures trained.

Fusion of Neuro-Signals and Dynamic Signatures for Person Authentication

Many biometric systems based on physiological traits such as ones facial characteristics, iris, and fingerprint have been developed for authentication purposes. Such security systems, however, commonly suffer from impersonation attacks such as obfuscation, abrasion, latent samples, and covert attack. More conventional behavioral methods, such as passwords and signatures, suffer from similar issues and can easily be spoofed. With growing levels of private data readily available across the internet, a more robust authentication system is needed for use in emerging technologies and mobile applications. In this paper, we present a novel multimodal biometric user authentication framework by combining the behavioral dynamic signature with the the physiological electroencephalograph (EEG) to restrict unauthorized access. EEG signals of 33 genuine users were collected while signing on their mobile phones. The recorded sequences were modeled using a bidirectional long short-term memory neural network (BLSTM-NN) based sequential classifier to accomplish person identification and verification. An accuracy of 98.78% was obtained for identification using decision fusion of dynamic signatures and EEG signals. The robustness of the framework was also tested against 1650 impersonation attempts made by 25 forged users by imitating the dynamic signatures of genuine users. Verification performance was measured using detection error tradeoff (DET) curves and half total error rate (HTER) security matrices using true positive rate (TPR) and false acceptance rate (FAR), resulting in 3.75% FAR and 1.87% HTER with 100% TPR for forgery attempts.

Nonlinear Dynamics Tools for Offline Signature Verification Using One-class Gaussian Process

One of the major problems in biometrics and in document forensics is the offline mode of signature verification. This study aims to present a novel approach of verifying an individual’s signature through offline images of handwriting. The approach proposed here relies on a global method which considers signature images as waveforms. First, image decompositions are in terms of a series of wavelet sub-bands at some specific levels. Wavelet sub-bands are then extended so as to obtain waveforms. Each waveform is quantized by two Nonlinear Dynamics Tools in order to generate feature vectors. Multi-Resolution Box-Counting (MRBC) fractal dimension algorithm as well as probabilistic finite state automata (PFSA) are applied separately to signature waveforms. In the training and verification phase, we propose the one-class Gaussian process (GP) priors based on writer-independent approach. As one of the main parameters, optimal decision threshold is selected from False Accept Rate (FAR) and False Reject Rate (FRR) curves. The presented system was tested on two Persian databases (PHBC and UTSig) as well as on two Latin databases (MCYT-75 and CEDAR). In fact, the results produced by this method were generally better in terms of the four signature databases than the state-of-the-art results.

A Large-Scale Study of Fingerprint Matching Systems for Sensor Interoperability Problem

The fingerprint is a commonly used biometric modality that is widely employed for authentication by law enforcement agencies and commercial applications. The designs of existing fingerprint matching methods are based on the hypothesis that the same sensor is used to capture fingerprints during enrollment and verification. Advances in fingerprint sensor technology have raised the question about the usability of current methods when different sensors are employed for enrollment and verification; this is a fingerprint sensor interoperability problem. To provide insight into this problem and assess the status of state-of-the-art matching methods to tackle this problem, we first analyze the characteristics of fingerprints captured with different sensors, which makes cross-sensor matching a challenging problem. We demonstrate the importance of fingerprint enhancement methods for cross-sensor matching. Finally, we conduct a comparative study of state-of-the-art fingerprint recognition methods and provide insight into their abilities to address this problem. We performed experiments using a public database (FingerPass) that contains nine datasets captured with different sensors. We analyzed the effects of different sensors and found that cross-sensor matching performance deteriorates when different sensors are used for enrollment and verification. In view of our analysis, we propose future research directions for this problem.

On the Exploration of Information From the DTW Cost Matrix for Online Signature Verification

This paper explores the utility of information derived from the dynamic time warping (DTW) cost matrix for the problem of online signature verification. The prior works in literature primarily utilize only the DTW scores to authenticate a test signature. To the best of our knowledge, the characteristics of the warping path (used for the alignment) in the cost matrix is hardly exploited for verification of online signatures. Accordingly, we devise a score that utilizes the information from the cost matrix and warping path alignments. We subsequently consider its fusion (using a sum rule combiner) with the DTW score for authenticating the veracity of a test signature. In addition, a minor modification is suggested with regards to the set of features employed for matching the signatures. We introduce a spacing parameter for feature extraction and demonstrate its applicability in increasing the separation between the distribution of genuine and forgery signatures for an user. Our method has been tested on two publicly available online signature databases namely the SVC-2004 Task 2 and MCYT-100. We report reduction in error rates over the traditional DTW framework.