research-article

Open access

Consumer Wearables and Affective Computing for Wellbeing Support

Authors:

Stanislaw Saganowski,

Przemyslaw Kazienko,

Maciej Dziezyc,

Patrycja Jakimow,

Joanna Komoszynska,

Weronika Michalska,

Anna Dutkowiak,

Adam Dziadek, and

Michal UjmaAuthors Info & Claims

MobiQuitous '20: MobiQuitous 2020 - 17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services

December 2020

Pages 482 - 487

https://doi.org/10.1145/3448891.3450332

Published: 09 August 2021 Publication History

All formats PDF

Abstract

Wearables equipped with pervasive sensors enable us to monitor physiological and behavioral signals in our everyday life. We propose the WellAff system able to recognize affective states for wellbeing support. It also includes health care scenarios, in particular patients with chronic kidney disease suffering from bipolar disorders. For the need of a large-scale field study, we revised over 50 off-the-shelf devices in terms of usefulness for emotion, stress, meditation, sleep, and physical activity recognition and analysis. Their usability directly comes from the types of sensors they possess as well as the quality and availability of raw signals. We found there is no versatile device suitable for all purposes. Using Empatica E4 and Samsung Galaxy Watch, we have recorded physiological signals from 11 participants over many weeks. The gathered data enabled us to train a classifier that accurately recognizes strong affective states.

References

[1]

H Alawieh, Z Dawy, E Yaacoub, and et al.2019. A Real-Time ECG Feature Extraction Algorithm for Detecting Meditation Levels within a General Measurement Setup. In 41st Annual Int. Conf. of the IEEE Eng. in Med. and Biol. Soc. IEEE, 99–103.

[2]

Amani Albraikan, Basim Hafidh, and Abdulmotaleb El Saddik. 2018. iAware: A Real-Time Emotional Biofeedback System Based on Physiological Signals. IEEE Access 6(2018), 78780–78789.

[3]

H Allahbakhshi, L Conrow, B Naimi, and R Weibel. 2020. Using Accelerometer and GPS Data for Real-Life Physical Activity Type Detection. Sensors 20, 3 (2020), 588.

[4]

G Aroganam, N Manivannan, and D Harrison. 2019. Review on wearable technology sensors used in consumer sport applications. Sensors 19, 9 (2019), 1983.

[5]

KR Arunkumar and M Bhaskar. 2020. Heart rate estimation from wrist-type photoplethysmography signals during physical exercise. Biomedical Signal Processing and Control 57 (2020), 101790.

[6]

Mahmoud Assaf, Aïcha Rizzotti-Kaddouri, and Magdalena Punceva. 2018. Sleep detection using physiological signals from a wearable device. In EAI International Conference on IoT Technologies for HealthCare. Springer, 23–37.

[7]

MI Cajita, CE Kline, and et al.2020. Feasible but Not Yet Efficacious: a Scoping Review of Wearable Activity Monitors in Interventions Targeting Physical Activity, Sedentary Behavior, and Sleep. Current Epidem. Rep. 7, 1 (2020), 25–38.

[8]

Burçin Camcı, Cem Ersoy, and Hakan Kaynak. 2019. Abnormal respiratory event detection in sleep: a prescreening system with smart wearables. Journal of biomedical informatics 95 (2019), 103218.

Digital Library

[9]

Yekta Said Can, Bert Arnrich, and Cem Ersoy. 2019. Stress detection in daily life scenarios using smart phones and wearable sensors: A survey. Journal of biomedical informatics(2019), 103139.

[10]

Yekta Said Can, Niaz Chalabianloo, Deniz Ekiz, and Cem Ersoy. 2019. Continuous stress detection using wearable sensors in real life: Algorithmic programming contest case study. Sensors 19, 8 (2019), 1849.

[11]

Rossana Castaldo, Marta Prati, Luis Montesinos, Vishwesh Kulkarni, Micheal Chappell, Helen Byrne, Pasquale Innominato, Stephen Hughes, and Leandro Pecchia. 2019. Investigating the Use of Wearables for Monitoring Circadian Rhythms: A Feasibility Study. In International Conference on Biomedical and Health Informatics. Springer, 275–280.

[12]

Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. 2002. SMOTE: synthetic minority over-sampling technique. Journal of artificial intelligence research 16 (2002), 321–357.

[13]

Gloria Cosoli, Susanna Spinsante, and Lorenzo Scalise. 2020. Wrist-worn and chest-strap wearable devices: systematic review on accuracy and metrological characteristics. Measurement (2020), 107789.

[14]

Massimiliano de Zambotti, Leonardo Rosas, Ian M Colrain, and Fiona C Baker. 2019. The sleep of the ring: comparison of the ŌURA sleep tracker against polysomnography. Behavioral sleep medicine 17, 2 (2019), 124–136.

[15]

Urtnasan Erdenebayar, Yoon Ji Kim, Jong-Uk Park, Eun Yeon Joo, and Kyoung-Joung Lee. 2019. Deep learning approaches for automatic detection of sleep apnea events from an electrocardiogram. Computer methods and programs in biomedicine 180 (2019), 105001.

[16]

Luz Fernández-Aguilar, Arturo Martínez-Rodrigo, José Moncho-Bogani, Antonio Fernández-Caballero, and José Miguel Latorre. 2019. Emotion Detection in Aging Adults Through Continuous Monitoring of Electro-Dermal Activity and Heart-Rate Variability. In Int. Work-Conference on the Interplay Between Natural and Artificial Computation. Springer, 252–261.

[17]

Arindam Ghosh, Morena Danieli, and Giuseppe Riccardi. 2015. Annotation and prediction of stress and workload from physiological and inertial signals. In 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). IEEE, 1621–1624.

[18]

G. Giannakakis, D. Grigoriadis, K. Giannakaki, O. Simantiraki, A. Roniotis, and M. Tsiknakis. 2019. Review on psychological stress detection using biosignals. IEEE Transactions on Affective Computing(2019).

[19]

Alireza Golgouneh and Bahram Tarvirdizadeh. 2019. Fabrication of a portable device for stress monitoring using wearable sensors and soft computing algorithms. Neural Computing and Applications(2019), 1–23.

[20]

Rosenberger M Grandner M. 2019. Actigraphic sleep tracking and wearables: Historical context, scientific applications and guidelines, limitations, and considerations for commercial sleep devices. In Sleep and Health. Elsevier, 147–157.

[21]

Juan Lorenzo Hagad, Kenichi Fukui, and Masayuki Numao. 2019. Deep visual models for EEG of mindfulness meditation in a workplace setting. In International Workshop on Health Intelligence. Springer, 129–137.

[22]

BP Harne, Y Bobade, RS Dhekekar, and A Hiwale. 2019. SVM classification of EEG signal to analyze the effect of OM Mantra meditation on the brain. In 2019 IEEE 16th India Council International Conference (INDICON). IEEE, 1–4.

[23]

Cheng He, Yun-jin Yao, and Xue-song Ye. 2017. An emotion recognition system based on physiological signals obtained by wearable sensors. In Wearable sensors and robots. Springer, 15–25.

[24]

RT Hurlburt and CL Heavy. 2002. Interobserver reliability of descriptive experience sampling. Cognitive Therapy and Research 26, 1 (2002), 135–142.

[25]

Eiman Kanjo, Eman MG Younis, and Chee Siang Ang. 2019. Deep learning analysis of mobile physiological, environmental and location sensor data for emotion detection. Information Fusion 49(2019), 46–56.

Digital Library

[26]

A Léonard, S Clément, C-D Kuo, and MU Manto. 2019. Changes in Heart Rate Variability During Heartfulness Meditation: A Power Spectral Analysis Including the Residual Spectrum. Frontiers in cardiovascular medicine 6 (2019), 62.

[27]

M Maier, C Marouane, and D Elsner. 2019. DeepFlow: Detecting Optimal User Experience From Physiological Data Using Deep Neural Networks. In Proc. of the 18th Int. Conf. on Autonom. Agents and MultiAgent Sys. Int. Found. for Autonom. Agents and Multiagent Sys., 2108–2110.

[28]

F Mendonca, SS Mostafa, AG Ravelo-García, F Morgado-Dias, and T Penzel. 2018. A review of obstructive sleep apnea detection approaches. IEEE journal of biomedical and health informatics 23, 2(2018), 825–837.

[29]

Andre Matthias Müller, Nan Xin Wang, Jiali Yao, Chuen Seng Tan, Ivan Cherh Chiet Low, Nicole Lim, Jeremy Tan, Agnes Tan, and Falk Müller-Riemenschneider. 2019. Heart rate measures from wrist-worn activity trackers in a laboratory and free-living setting: validation study. JMIR mHealth and uHealth 7, 10 (2019), e14120.

[30]

H Murakami, R Kawakami, S Nakae, Y Yamada, Y Nakata, and et al.2019. Accuracy of 12 Wearable Devices for Estimating Physical Activity Energy Expenditure Using a Metabolic Chamber and the Doubly Labeled Water Method: Validation Study. In JMIR mHealth and uHealth.

[31]

B. Nakisa, M. N. Rastgoo, A. Rakotonirainy, F. Maire, and V. Chandran. 2018. Long short term memory hyperparameter optimization for a neural network based emotion recognition framework. IEEE Access 6(2018), 49325–49338.

[32]

Grzegorz J Nalepa, Krzysztof Kutt, Barbara Giżycka, Paweł Jemioło, and Szymon Bobek. 2019. Analysis and Use of the Emotional Context with Wearable Devices for Games and Intelligent Assistants. Sensors 19, 11 (2019), 2509.

[33]

Sami Nikkonen, Isaac O Afara, Timo Leppänen, and Juha Töyräs. 2019. Artificial neural network analysis of the oxygen saturation signal enables accurate diagnostics of sleep apnea. Scientific reports 9, 1 (2019), 1–9.

[34]

Selena R. Pasadyn, Mohamad Soudan, Marc Gillinov, Penny Houghtaling, Dermot Phelan, Nicole Gillinov, Barbara Bittel, and Milind Y. Desai. 2019. Accuracy of commercially available heart rate monitors in athletes: a prospective study. Cardiovascular Diagnosis and Therapy 9, 4 (2019).

[35]

Reddipogu Pavani and Akshay Berad. 2019. Study of Effect of Meditation on Galvanic Skin Response in Healthy Individuals. International Journal of Medical and Biomedical Studies 3, 4(2019).

[36]

JM Peake, G Kerr, and JP Sullivan. 2018. A critical review of consumer wearables, mobile applications, and equipment for providing biofeedback, monitoring stress, and sleep in physically active populations. Frontiers in physiology 9 (2018), 743.

[37]

S Pham, D Yeap, and et al.2020. Wearable Sensor System to Monitor Physical Activity and the Physiological Effects of Heat Exposure. Sensors 20, 3 (2020).

[38]

S. Saganowski, A. Dutkowiak, A. Dziadek, M. Dziezyc, J. Komoszynska, W. Michalska, A. G. Polak, M. Ujma, and P. Kazienko. 2020. Emotion Recognition Using Wearables: A Systematic Literature Review - Work-in-progress. In 2020 Int. Conf. on Perv. Comp. and Comm. Workshops. IEEE, 1–6.

[39]

P. Schmidt, A. Reiss, R. Duerichen, C. Marberger, and K. Van Laerhoven. 2018. Introducing WESAD, a Multimodal Dataset for Wearable Stress and Affect Detection. In Proc. of the 2018 on Int. Conf. on Multimodal Interaction. ACM, 400–408.

[40]

P. Schmidt, A. Reiss, R. Dürichen, and K. Van Laerhoven. 2018. Labelling Affective States in the Wild: Practical Guidelines and Lessons Learned. In The 2018 ACM Int. Joint Conf. and 2018 Int. Symp. on Pervasive and Ubiquitous Comp. and Wearable Comp. ACM, 654–659.

[41]

F Setiawan, SA Khowaja, AG Prabono, and et al.2018. A Framework for Real Time Emotion Recognition Based on Human ANS Using Pervasive Device. In 2018 IEEE 42nd Annual Comp. Software and App. Conf., Vol. 1. IEEE, 805–806.

[42]

H. Sharma, R. Raj, and M. Juneja. 2019. EEG signal based classification before and after combined Yoga and Sudarshan Kriya. Neurosc. letters 707(2019), 134300.

[43]

Saul Shiffman, Arthur A Stone, and Michael R Hufford. 2008. Ecological Momentary Assessment. Annu. Rev. Clin. Psychol. 4 (2008), 1–32.

[44]

M Shoaib, S Bosch, OD Incel, and et al.2014. Fusion of Smartphone Motion Sensors for Physical Activity Recognition. Sensors 14, 6 (2014), 10146–10176.

[45]

MZ Soroush, K Maghooli, SK Setarehdan, and AM Nasrabadi. 2017. A Review on EEG signals based emotion recognition. Int. Clinic. Neurosc. Jour. 4, 4 (2017), 118.

[46]

J Wei and J Boger. 2019. You Are How You Sleep: Personalized Sleep Monitoring Based on Wrist Temperature and Accelerometer Data. In 13th Int. Conf. on Perv. Comp. Tech. for Healthcare-Demos and Posters. European Alliance for Innovation.

[47]

Bin Yu, Mathias Funk, Jun Hu, and Loe Feijs. 2018. Unwind: a musical biofeedback for relaxation assistance. Behaviour & Inform. Technology 37, 8 (2018), 800–814.

[48]

Roberto Zangróniz, Arturo Martínez-Rodrigo, María T López, José Manuel Pastor, and Antonio Fernández-Caballero. 2018. Estimation of mental distress from photoplethysmography. Applied Sciences 8, 1 (2018), 69.

[49]

R Zangróniz, A Martínez-Rodrigo, J Pastor, and et al.2017. Electrodermal activity sensor for classification of calm/distress condition. Sensors 17, 10 (2017), 2324.

Cited By

Sun JPortilla JOtero A(2024)A Deep Learning Approach for Fear Recognition on the Edge Based on Two-Dimensional Feature MapsIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.339237328:7(3973-3984)Online publication date: Jul-2024
https://doi.org/10.1109/JBHI.2024.3392373
Wolff MSchaathun HFougner ASteinert MVolden R(2024)Mobile Software Development Kit for Real Time Multivariate Blood Glucose PredictionIEEE Access10.1109/ACCESS.2024.334949612(5910-5919)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3349496
Abeltino ARiente ABianchetti GSerantoni CDe Spirito MCapezzone SEsposito RMaulucci G(2024)Digital applications for diet monitoring, planning, and precision nutrition for citizens and professionals: a state of the artNutrition Reviews10.1093/nutrit/nuae035Online publication date: 9-May-2024
https://doi.org/10.1093/nutrit/nuae035
Show More Cited By

Recommendations

Wearable Electronic Textiles for Healthcare, Wellbeing, and Protective Applications
CHI EA '24: Extended Abstracts of the 2024 CHI Conference on Human Factors in Computing Systems
Electronic textiles (E-textiles) can act as an exceptional substrate for incorporating sensing devices for monitoring physiological parameters, as they are comfortable to wear close to the skin. This makes them excellent for a variety of applications in ...
Read More
Usability of a Low-Cost Wearable Health Device for Physical Activity and Sleep Duration in Healthy Adults
MobileHealth '15: Proceedings of the 2015 Workshop on Pervasive Wireless Healthcare

For public health and epidemiology, an understanding of physical activity and sleep duration is important. People found decreased physical activity and short sleep duration as they aged. The development of healthy behaviors in the early life will ensure ...
Read More
Affective Wearables
ISWC '97: Proceedings of the 1st IEEE International Symposium on Wearable Computers

An "affective wearable" is a wearable system equipped with sensors and tools which enables recognition of its wearer's affective patterns. Affective patterns include expressions of emotion such as a joyful smile, an angry gesture, a strained voice or a ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

MobiQuitous '20: MobiQuitous 2020 - 17th EAI International Conference on Mobile and Ubiquitous Systems: Computing, Networking and Services

December 2020

493 pages

ISBN:9781450388405

DOI:10.1145/3448891

Editors:
Max Mühlhäuser
Technical University of Darmstadt, Germany
,
George C. Polyzos
Athens University of Economics and Business Athens, Greece
,
Florian Michahelles
Siemens Corporation, USA
,
Alejandro Sanchez Guinea
Technical University of Darmstadt, Germany
,
Lin Wang
Vrije Universiteit (VU) Amsterdam

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 August 2021

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Narodowe Centrum Nauki
Department of Computational Intelligence, Wroclaw University of Science and Technology

Conference

MobiQuitous '20

MobiQuitous '20: Computing, Networking and Services

December 7 - 9, 2020

Darmstadt, Germany

Acceptance Rates

Overall Acceptance Rate 26 of 87 submissions, 30%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
1,359
Total Downloads

Downloads (Last 12 months)429
Downloads (Last 6 weeks)49

Other Metrics

View Author Metrics

Citations

Cited By

Sun JPortilla JOtero A(2024)A Deep Learning Approach for Fear Recognition on the Edge Based on Two-Dimensional Feature MapsIEEE Journal of Biomedical and Health Informatics10.1109/JBHI.2024.339237328:7(3973-3984)Online publication date: Jul-2024
https://doi.org/10.1109/JBHI.2024.3392373
Wolff MSchaathun HFougner ASteinert MVolden R(2024)Mobile Software Development Kit for Real Time Multivariate Blood Glucose PredictionIEEE Access10.1109/ACCESS.2024.334949612(5910-5919)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3349496
Abeltino ARiente ABianchetti GSerantoni CDe Spirito MCapezzone SEsposito RMaulucci G(2024)Digital applications for diet monitoring, planning, and precision nutrition for citizens and professionals: a state of the artNutrition Reviews10.1093/nutrit/nuae035Online publication date: 9-May-2024
https://doi.org/10.1093/nutrit/nuae035
Gahlan NSethia D(2024)Federated learning in Emotion Recognition Systems based on physiological signals for privacy preservation: a reviewMultimedia Tools and Applications10.1007/s11042-024-19467-3Online publication date: 3-Jun-2024
https://doi.org/10.1007/s11042-024-19467-3
Grzeszczyk MLisowska ASitek ALisowska A(2023)Decoding Emotional Valence from Wearables: Can Our Data Reveal Our True Feelings?Proceedings of the 25th International Conference on Mobile Human-Computer Interaction10.1145/3565066.3608698(1-6)Online publication date: 26-Sep-2023
https://doi.org/10.1145/3565066.3608698
Saganowski SPerz BPolak AKazienko P(2023)Emotion Recognition for Everyday Life Using Physiological Signals From Wearables: A Systematic Literature ReviewIEEE Transactions on Affective Computing10.1109/TAFFC.2022.317613514:3(1876-1897)Online publication date: 1-Jul-2023
https://dl.acm.org/doi/10.1109/TAFFC.2022.3176135
Lisowska AWilk SPeleg M(2023)SATO (IDEAS expAnded wiTh BCIO)Journal of Biomedical Informatics10.1016/j.jbi.2022.104276138:COnline publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.jbi.2022.104276
Anwar ZJahangir RNauman MAlroobaea RAlzahrani SAli I(2022)Context-Based Emotion PredictorMobile Information Systems10.1155/2022/64888482022Online publication date: 1-Jan-2022
https://dl.acm.org/doi/10.1155/2022/6488848
Behnke MSaganowski SKunc DKazienko P(2022)Ethical Considerations and Checklist for Affective Research With WearablesIEEE Transactions on Affective Computing10.1109/TAFFC.2022.322252415:1(50-62)Online publication date: 16-Nov-2022
https://dl.acm.org/doi/10.1109/TAFFC.2022.3222524
Saganowski SKunc DPerz BKomoszynska JBehnke MKazienko P(2022)The Cold Start Problem and Per-Group Personalization in Real-Life Emotion Recognition With Wearables2022 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops)10.1109/PerComWorkshops53856.2022.9767233(812-817)Online publication date: 21-Mar-2022
https://doi.org/10.1109/PerComWorkshops53856.2022.9767233
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents