Abstract
With each sensing modality exhibiting inherent strengths and limitations, multi-modal approaches for wearable Human Activity Recognition (HAR) are becoming increasingly relevant -- particularly for recognizing Activities of Daily Living (ADLs), where individual modalities often produce ambiguous signals for similar or complex activities. This work introduces HARMES, a multi-modal wearable dataset combining three wrist-recorded modalities: motion sensing via an Inertial Measurement Unit (IMU), atmospheric environmental sensors (humidity, temperature, and pressure), and audio. Collected from 20 participants performing household activities in their own homes, HARMES totals over 80 hours of recorded data, with approximately three hours of labeled activity data per participant across 15 ADL classes. To the best of our knowledge, HARMES is the first dataset to combine this particular sensor trio, and it is nearly six times larger than the previously largest wrist-inertial-acoustic HAR dataset. In an extensive benchmark, we evaluate cross-subject, cross-dataset, cross-modality, and cross-environment generalization and conduct an ablation study revealing that modality contributions are activity-dependent and can provide complementary value, particularly for activities that are ambiguous from motion data alone. HARMES is freely available at Zenodo, alongside example code for loading the dataset and training models on GitHub.
Dataset Overview
HARMES supports research on wearable Human Activity Recognition by providing synchronized motion, environmental, and acoustic sensor data recorded during everyday household activities in participants' own homes. The dataset enables researchers to investigate the complementary information provided by different sensing modalities and evaluate generalization across participants and recording conditions.
20
Participants
15
ADL Classes
80+ h
Recorded Data
Sensing modalities: wrist-worn inertial measurement units (IMUs), atmospheric environmental sensing (humidity, temperature, and pressure), and audio.
BibTeX
@misc{burchard2026harmes,
title = {HARMES: A Multi-Modal Dataset for Wearable Human Activity Recognition with Motion, Environmental Sensing and Sound},
author = {Burchard, Robin and Brückner, Pascal-André and Bock, Marius and Gall, Jürgen and Van Laerhoven, Kristof},
year = {2026},
eprint = {2605.02596},
archivePrefix = {arXiv},
primaryClass = {cs.LG},
url = {https://arxiv.org/abs/2605.02596}
}
Publications using this dataset
Research papers and preprints citing the HARMES dataset or publication.