Wissenschaftliche Publikationen

We investigate AR-based tracking and registration of the liver surface for potential surgical applications. Our approach consists of streaming RGBD data from a Hololens2 device, RGBD segmentation using a deep learning model and registering the acquired partial liver surface point cloud with the corresponding virtual liver model. We aim to derive basic requirements for AR-guided liver surgery, thus consider several test cases of partially occluded liver as it would appear in surgical scenarios. To evaluate our approach, we use a 3D-printed phantom with basic texture and rigid structure. Our results show that the visible liver section has a substantial impact of feature extraction and matching, thus the registration process. Test cases, where specific anatomical features are visible, e.g. the right liver lobe, yielded superior outcomes compared to other cases, e.g. only the left liver lobe visible. Moreover, our results showed that large scale Hololens movements during the tracking process affected the registration performance. Our implementation achieved 2-3 frames per second for tracking and registration. We discuss the potential and limitations of utilizing Hololens2 for real-time tracking and registration of the liver surface. To our knowledge this is the first experimental approach for real-time markerless tracking and registration for AR-guided surgery guidance using the Hololens2 sensors only.

 

Link to paper

The COVID-19 pandemic has stimulated pervasive computing research and development resulting in new, impactful public health tools, including digital contact tracing, crowd dynamics analysis, and symptom tracking, which are broadly recognized by the public and expert groups alike. In the post-COVID age, focus has shifted to establish a level of pandemic preparedness. Across all preparedness measures, there is the need for interoperable data, pervasive computing tools, and data analysis methods. Considering the open technical challenges, further pervasive computing research is a key to fill the opportunity created over the last years and eventually save more lives at global scale. In this special issue, we capture new technical approaches to the pervasive tool inventory that help dealing with a pandemic situation, but also investigations that highlight opportunities for further research. We aim to motivate continued research and discussion of new ideas on pervasive computing for public health tools that spurs pandemic preparedness.

 

Link to paper

Link to publication

Background:

Abdominal auscultation, i.e. listening to Bowel Sounds (BS), can be used to analyse digestion. An automated retrieval of BS would be beneficial to assess gastro-intestinal disorders non-invasively.

Objective:

To develop a multi-scale spotting model to detect BS in continuous audio data from a wearable monitoring system.

Methods:

We designed a spotting model based on Efficient-U-Net (EffUNet) architecture to analyse 10-second audio segments at a time and spot BS with a temporal resolution of 25 ms. Evaluation data was collected across different digestive phases from 18 healthy participants and 9 patients with Inflammatory Bowel Disease (IBD). Audio data were recorded in a daytime setting with a T-Shirt that embeds digital microphones. The dataset was annotated by independent raters with substantial agreement (Cohen’s κ between 0.70 and 0.75), resulting in 136 h of labelled data. In total, 11482 BS were analysed, with BS duration ranging between 18 ms and 6.3 s. The share of BS in the dataset (BS ratio) was 0.89%. We analysed performance depending on noise level, BS duration, and BS event rate, as well as report spotting timing errors.

Results:

Leave-One-Participant-Out cross-validation of BS event spotting yielded a median F1 score of 0.73 for both, healthy volunteers and patients. EffUNet detected BS in different noise conditions with 0.73 recall and 0.72 precision. In particular, for SNR > 4 dB, more than 83% of BS were recognised, with precision ≥ 0.77. EffUNet recall dropped below 0.60 for BS duration ≥ 1.5 s. At BS ratio > 5%, our model precision was > 0.83. For both healthy participants and patients, insertion and deletion timing errors were the largest, with a total of 15.54 min insertion errors and 13.08 min of deletion errors over the total audio dataset. On our dataset, EffUNet outperform existing BS spotting models that provide similar temporal resolution.

Conclusions:

The EffUNet spotter is robust against background noise and can retrieve BS with varying duration. EffUNet outperforms previous BS detection approaches in unmodified audio data, containing highly sparse BS events.

 

Link to publication

We analyse pretrained and non-pretrained deep neural models to detect 10-seconds Bowel Sounds (BS) audio segments in continuous audio data streams. The models include MobileNet, EfficientNet, and Distilled Transformer architectures. Models were initially trained on AudioSet and then transferred and evaluated on 84 hours of labelled audio data of eighteen healthy participants. Evaluation data was recorded in a semi-naturalistic daytime setting including movement and background noise using a smart shirt with embedded microphones. The collected dataset was annotated for individual BS events by two independent raters with substantial agreement (Cohen’s Kappa κ = 0.74). Leave-One-Participant-Out cross-validation for detecting 10-second BS audio segments, i.e. segment-based BS spotting, yielded a best F1 score of 73% and 67%, with and without transfer learning respectively. The best model for segment-based BS spotting was EfficientNet-B2 with an attention module. Our results show that pretrained models could improve F1 score up to 26%, in particular, increasing robustness against background noise. Our segment-based BS spotting approach reduces the amount of audio data to be reviewed by experts from 84 h to 11 h, thus by ∼87%.

 

Link to publication

We propose a co-simulation framework comprising biomechanical human body models and wearable inertial sensor models to analyse gait events dynamically, depending on inertial sensor type, sensor positioning, and processing algorithms. A total of 960 inertial sensors were virtually attached to the lower extremities of a validated biomechanical model and shoe model. Walking of hemiparetic patients was simulated using motion capture data (kinematic simulation). Accelerations and angular velocities were synthesised according to the inertial sensor models. A comprehensive error analysis of detected gait events versus reference gait events of each simulated sensor position across all segments was performed. For gait event detection, we considered 1-, 2-, and 4-phase gait models. Results of hemiparetic patients showed superior gait event estimation performance for a sensor fusion of angular velocity and acceleration data with lower nMAEs (9%) across all sensor positions compared to error estimation with acceleration data only. Depending on algorithm choice and parameterisation, gait event detection performance increased up to 65%. Our results suggest that user personalisation of IMU placement should be pursued as a first priority for gait phase detection, while sensor position variation may be a secondary adaptation target. When comparing rotatory and translatory error components per body segment, larger interquartile ranges of rotatory errors were observed for all phase models i.e., repositioning the sensor around the body segment axis was more harmful than along the limb axis for gait phase detection. The proposed co-simulation framework is suitable for evaluating different sensor modalities, as well as gait event detection algorithms for different gait phase models. The results of our analysis open a new path for utilising biomechanical human digital twins in wearable system design and performance estimation before physical device prototypes are deployed.

 

Link to publication