Multimodal Environmental Sensing Using AI & IoT Solutions: A Cognitive Sound Analysis Perspective

This study presents a novel audio compression technique, tailored for environmental monitoring within multi-modal data processing pipelines. Considering the crucial role that audio data play in environmental evaluations, particularly in contexts with extreme resource limitations, our strategy substantially decreases bit rates to facilitate efficient data transfer and storage. This is accomplished without undermining the accuracy necessary for trustworthy air pollution analysis while simultaneously minimizing processing expenses. More specifically, our approach fuses a Deep-Learning-based model, optimized for edge devices, along with a conventional coding schema for audio compression. Once transmitted to the cloud, the compressed data undergo a decoding process, leveraging vast cloud computing resources for accurate reconstruction and classification. The experimental results indicate that our approach leads to a relatively minor decrease in accuracy, even at notably low bit rates, and demonstrates strong robustness in identifying data from labels not included in our training dataset.

Research on multi-robot collaborative operation in logistics and warehousing using A3C optimized YOLOv5-PPO model

Introduction

In the field of logistics warehousing robots, collaborative operation and coordinated control have always been challenging issues. Although deep learning and reinforcement learning methods have made some progress in solving these problems, however, current research still has shortcomings. In particular, research on adaptive sensing and real-time decision-making of multi-robot swarms has not yet received sufficient attention.

Methods

To fill this research gap, we propose a YOLOv5-PPO model based on A3C optimization. This model cleverly combines the target detection capabilities of YOLOv5 and the PPO reinforcement learning algorithm, aiming to improve the efficiency and accuracy of collaborative operations among logistics and warehousing robot groups.

Results

Through extensive experimental evaluation on multiple datasets and tasks, the results show that in different scenarios, our model can successfully achieve multi-robot collaborative operation, significantly improve task completion efficiency, and maintain target detection and environment High accuracy of understanding.

Discussion

In addition, our model shows excellent robustness and adaptability and can adapt to dynamic changes in the environment and fluctuations in demand, providing an effective method to solve the collaborative operation problem of logistics warehousing robots.

Medical Augmented Reality: Definition, Principle Components, Domain Modeling, and Design-Development-Validation Process

Three decades after the first set of work on Medical Augmented Reality (MAR) was presented to the international community, and ten years after the deployment of the first MAR solutions into operating rooms, its exact definition, basic components, systematic design, and validation still lack a detailed discussion. This paper defines the basic components of any Augmented Reality (AR) solution and extends them to exemplary Medical Augmented Reality Systems (MARS). We use some of the original MARS applications developed at the Chair for Computer Aided Medical Procedures and deployed into medical schools for teaching anatomy and into operating rooms for telemedicine and surgical guidance throughout the last decades to identify the corresponding basic components. In this regard, the paper is not discussing all past or existing solutions but only aims at defining the principle components and discussing the particular domain modeling for MAR and its design-development-validation process, and providing exemplary cases through the past in-house developments of such solutions.

Multi-modal data collection for measuring health, behavior, and living environment of large-scale participant cohorts

BACKGROUND

As mobile technologies become ever more sensor-rich, portable, and ubiquitous, data captured by smart devices are lending rich insights into users' daily lives with unprecedented comprehensiveness and ecological validity. A number of human-subject studies have been conducted to examine the use of mobile sensing to uncover individual behavioral patterns and health outcomes, yet minimal attention has been placed on measuring living environments together with other human-centered sensing data. Moreover, the participant sample size in most existing studies falls well below a few hundred, leaving questions open about the reliability of findings on the relations between mobile sensing signals and human outcomes.

RESULTS

To address these limitations, we developed a home environment sensor kit for continuous indoor air quality tracking and deployed it in conjunction with smartphones, Fitbits, and ecological momentary assessments in a cohort study of up to 1,584 college student participants per data type for 3 weeks. We propose a conceptual framework that systematically organizes human-centric data modalities by their temporal coverage and spatial freedom. Then we report our study procedure, technologies and methods deployed, and descriptive statistics of the collected data that reflect the participants' mood, sleep, behavior, and living environment.

CONCLUSIONS

We were able to collect from a large participant cohort satisfactorily complete multi-modal sensing and survey data in terms of both data continuity and participant adherence. Our novel data and conceptual development provide important guidance for data collection and hypothesis generation in future human-centered sensing studies.

Real-Time Multi-Modal Sensing and Feedback for Catheterization in Porcine Tissue

Objective: In this study, we introduce a multi-modal sensing and feedback framework aimed at assisting clinicians during endovascular surgeries and catheterization procedures. This framework utilizes state-of-the-art imaging and sensing sub-systems to produce a 3D visualization of an endovascular catheter and surrounding vasculature without the need for intra-operative X-rays. Methods: The catheterization experiments within this study are conducted inside a porcine limb undergoing motions. A hybrid position-force controller of a robotically-actuated ultrasound (US) transducer for uneven porcine tissue surfaces is introduced. The tissue, vasculature, and catheter are visualized by integrated real-time US images, 3D surface imaging, and Fiber Bragg Grating (FBG) sensors. Results: During externally-induced limb motions, the vasculature and catheter can be reliably reconstructed at mean accuracies of 1.9±0.3 mm and 0.82±0.21 mm, respectively. Conclusions: The conventional use of intra-operative X-ray imaging to visualize instruments and vasculature in the human body can be reduced by employing improved diagnostic technologies that do not operate via ionizing radiation or nephrotoxic contrast agents. Significance: The presented multi-modal framework enables the radiation-free and accurate reconstruction of significant tissues and instruments involved in catheterization procedures.

MOSS-Multi-Modal Best Subset Modeling in Smart Manufacturing

Smart manufacturing, which integrates a multi-sensing system with physical manufacturing processes, has been widely adopted in the industry to support online and real-time decision making to improve manufacturing quality. A multi-sensing system for each specific manufacturing process can efficiently collect the in situ process variables from different sensor modalities to reflect the process variations in real-time. However, in practice, we usually do not have enough budget to equip too many sensors in each manufacturing process due to the cost consideration. Moreover, it is also important to better interpret the relationship between the sensing modalities and the quality variables based on the model. Therefore, it is necessary to model the quality-process relationship by selecting the most relevant sensor modalities with the specific quality measurement from the multi-modal sensing system in smart manufacturing. In this research, we adopted the concept of best subset variable selection and proposed a new model called Multi-mOdal beSt Subset modeling (MOSS). The proposed MOSS can effectively select the important sensor modalities and improve the modeling accuracy in quality-process modeling via functional norms that characterize the overall effects of individual modalities. The significance of sensor modalities can be used to determine the sensor placement strategy in smart manufacturing. Moreover, the selected modalities can better interpret the quality-process model by identifying the most correlated root cause of quality variations. The merits of the proposed model are illustrated by both simulations and a real case study in an additive manufacturing (i.e., fused deposition modeling) process.

Combining Electromagnetic Spectroscopy and Ground-Penetrating Radar for the Detection of Anti-Personnel Landmines

Dual mode detectors combining metal detection and ground-penetrating radar are increasingly being used during humanitarian demining operations because of their ability to discriminate metal clutter. There are many reports in the academic literature studying metal detector and ground-penetrating radar systems individually. However, the combination of these techniques has received much less attention. This paper describes the development of a novel dual modality landmine detector, which integrates spectroscopic metal detection with ground-penetrating radar. This paper presents a feature-level sensor fusion strategy based on three features extracted from the two sensors. This paper shows how the data from the two components can be fused together to enrich the feedback to the operator. The algorithms presented in this paper are targeted at automating the location of buried, visibly obscured objects; however, the system described is also capable of collecting information which could also be used for the potential classification of such items.