We are the Human-centric Ubiquitous Intelligence (HUI) Lab at Korea University with
Dr.-Ing Sungho Suh. Our research focuses on multimodal learning, generative models, and self-supervised learning for smart living with practical applications including wearable sensor-based human activity recognition, Human-Robot Interaction, Vision-Language-Action (VLA) model, and visual inspection of industrial modules.
We are looking for passionate new members including interns, MS, PhD students, or Postdocs to join the team.

CoSS: Co-optimizing Sensor and Sampling rate for data-efficient human activity recognition

COA-HAR: Exploring Contrastive Online test-time Adaptation for wearable sensor-based Human Activity Recognition using sensor data augmentation

Assessing Grid Penalized Reinforcement Learning for Renewable Energy Management of Power-to-X Integrated with Intermediate Storage

PACL+: Online continual learning using proxy-anchor and contrastive loss with Gaussian replay for sensor-based human activity recognition

TxP: Reciprocal Generation of Ground Pressure Dynamics and Activity Descriptions for Improving Human Activity Recognition

Past, present, and future of sensor-based human activity recognition using wearables: A surveying tutorial on a still challenging task

Exploration and exploitation in continual learning

LLM-Generated Tips Rival Expert-Created Tips in Helping Students Answer Quantum-Computing Questions

A Novel Guidance Framework for Nasal Rapid Antigen Tests with Improved Swab Keypoint Detection

Optimal operation guidelines for direct recovery of high-purity precursor from spent lithium-ion batteries: hybrid operation model of population balance equation and data-driven classifier

Non-destructive Fault Diagnosis of Electronic Interconnects by Learning Signal Patterns of Reflection Coefficient in the Frequency Domain

Remaining useful life prediction of Lithium-ion batteries using spatio-temporal multimodal attention networks

An Empirical Study on Fault Detection and Root Cause Analysis of Indium Tin Oxide Electrodes by Processing S-parameter Patterns

AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation.

A Comprehensive Evaluation of Marker-Based, Markerless Methods for Loose Garment Scenarios in Varying Camera Configurations

ContextMix: A context-aware data augmentation method for industrial visual inspection systems

Worker Activity Recognition in Manufacturing Line Using Near-body Electric Field

Human-centric artificial intelligence architecture for industry 5.0 applications

InMyFace: Inertial and mechanomyography-based sensor fusion for wearable facial activity recognition

MoCaPose: Motion Capturing with Textile-integrated Capacitive Sensors in Loose-fitting Smart Garments

TASKED: Transformer-based Adversarial learning for human activity recognition using wearable sensors via Self-KnowledgE Distillation

Two-stage generative adversarial networks for binarization of color document images

Quali-mat: Evaluating the quality of execution in body-weight exercises with a pressure sensitive sports mat

Move with the theremin: Body posture and gesture recognition using the theremin in loose-garment with embedded textile cables as antennas

Supervised segmentation with domain adaptation for small sampled orbital CT images

Generalized multiscale feature extraction for remaining useful life prediction of bearings with generative adversarial networks

Discriminative feature generation for classification of imbalanced data

Sequential targeting: A continual learning approach for data imbalance in text classification

CEGAN: Classification Enhancement Generative Adversarial Networks for unraveling data imbalance problems

Supervised health stage prediction using convolutional neural networks for bearing wear

Generative oversampling method for imbalanced data on bearing fault detection and diagnosis

Robust albedo estimation from a facial image with cast shadow under general unknown lighting

OpenMarcie: Dataset for Multimodal Action Recognition in Industrial Environments

Interpretable Multimodal Gesture Recognition for Drone and Mobile Robot Teleoperation via Log-Likelihood Ratio Fusion

Improving Wearable Sensor-Based Human Activity Recognition via Statistical Pretext Tasks

Rotation-Invariant Wearable Sensor-Based Human Activity Recognition via SO(3)-Equivariant Vector-Neuron Networks

Multimodal Wearable Sensor-Based Gesture Recognition for Reliable Robot Control in Hazardous Industrial Environments

ChairPose: Pressure-based Chair Morphology Grounded Sitting Pose Estimation through Simulation-Assisted Training

SImpHAR: Advancing impedance-based human activity recognition using 3D simulation and text-to-motion models

RAD: Region-Aware Diffusion Models for Image Inpainting

Spend More to Save More (SM2): An Energy and Hardware-Aware Implementation of Successive Halving for Sustainable Hyperparameter Optimization

PIM: Physics-Informed Multi-task Pre-training for Improving Inertial Sensor-Based Human Activity Recognition

OV-HHIR: Open Vocabulary Human Interaction Recognition Using Cross-modal Integration of Large Language Models

SPACE: SPAtial-aware Consistency rEgularization for anomaly detection in Industrial applications

MeciFace: Mechanomyography and Inertial Fusion-based Glasses for Edge Real-Time Recognition of Facial and Eating Activities

TSAK: Two-Stage Semantic-Aware Knowledge Distillation for Efficient Wearable Modality and Model Optimization in Manufacturing Lines

A Synthetic Benchmarking Pipeline to Compare Camera Calibration Algorithms

ALS-HAR: Harnessing Wearable Ambient Light Sensors to Enhance IMU-based HAR

The Power of Training: How Different Neural Network Setups Influence the Energy Demand

Sensor Data Augmentation from Skeleton Pose Sequences for Improving Human Activity Recognition

BeSound: Bluetooth-Based Position Estimation Enhancing with Cross-Modality Distillation

A Novel Local-Global Feature Fusion Framework for Body-weight Exercise Recognition with Pressure Mapping Sensors

Image Generation of Ultra-Thin Polymer Films Using Diffusion Models from Tensile Testing for Mechanical Failure Prediction

Fault Diagnosis of Indium Tin Oxide Electrodes by Multi-channel S-parameter Patterns

Real-Time and on-the-Edge Multiple Channel Capacitive and Inertial Fusion-Based Glove

Two-stage Early Prediction Framework of Remaining Useful Life for Lithium-ion Batteries

Selecting the Motion Ground Truth for Loose-fitting Wearables: Benchmarking Optical MoCap Methods

Proxy anchor-based unsupervised learning for continuous generalized category discovery

ClothFit: Cloth-Human-Attribute Guided Virtual Try-on Network Using 3D Simulated Dataset

FieldHAR: A Fully Integrated End-to-End RTL Framework for Human Activity Recognition with Neural Networks from Heterogeneous Sensors

A Knowledge Distillation Framework for Multi-Organ Segmentation of Medaka Fish in Tomographic Image

Chemical Property-Guided Neural Networks for Naphtha Composition Prediction

PressureTransferNet: Human Attribute Guided Dynamic Ground Pressure Profile Transfer using 3D simulated Pressure Maps

Estimation Of 3d Body Shape And Clothing Measurements From Frontal-And Side-View Images

Learning from the Best: Contrastive Representations Learning Across Sensor Locations for Wearable Activity Recognition

Adversarial deep feature extraction network for user independent human activity recognition

Mocapaci: Posture and gesture detection in loose garments using textile cables as capacitive antennas

Fusion of global-local features for image quality inspection of shipping label

Sequential lung nodule synthesis using attribute-guided generative adversarial networks

Tasked: transformer-based adversarial learning for human activity recognition using wearable sensors via self-knowledge distillation

Mocapose: Motion capturing with textile-integrated capacitive sensors in loose-fitting smart garments

A 0.8 µm smart dual conversion gain pixel for 64 megapixels CMOS image sensor with 12k e-full-well capacitance and low dark noise

Wearable sensor-based human activity recognition for worker safety in manufacturing line

Inmyface: Inertial and mechanomyography-based sensor fusion for wearable facial activity recognition

Proxy Anchor-based Unsupervised Learning for Continuous Generalized Category Discovery

The power of training: How different neural network setups influence the energy demand

Paul Lukowicz

Bo Zhou

Vitor fortes rey

Lala Ray

Yong Oh Lee

Hymalai Bello

Daniel Geißler

Haebom Lee

Hooman Tavakoli Ghinani

Sizhen Bian

Mengxi Liu

John Soldatos

Hansang Cho

Dunja Mladenic

Lars Krupp

Hyungmin Kim

Joel Jang

Mayank S JHA

Junmo Kim

Minsik Lee

Sungho Suh

Human-centricity is the core value behind the evolution of manufacturing towards Industry 5.0. Nevertheless, there is a lack of architecture that considers safety, trustworthiness, and human-centricity at its core. Therefore, we propose an architecture that integrates Artificial Intelligence (Active Learning, Forecasting, Explainable Artificial Intelligence), simulated reality, decision-making, and users' feedback, focussing on synergies between humans and machines. Furthermore, we align the proposed architecture with the Big Data Value Association Reference Architecture Model. Finally, we validate it on three use cases from real-world case studies.

The data imbalance problem in classification is a frequent but challenging task. In real-world datasets, numerous class distributions are imbalanced and the classification result under such condition reveals extreme bias in the majority data class. Recently, the potential of GAN as a data augmentation method on minority data has been studied. In this paper, we propose a classification enhancement generative adversarial networks (CEGAN) to enhance the quality of generated synthetic minority data and more importantly, to improve the prediction accuracy in data imbalanced condition. In addition, we propose an ambiguity reduction method using the generated synthetic minority data for the case of multiple similar classes that are degenerating the classification accuracy. The proposed method is demonstrated with five benchmark datasets. The results indicate that approximating the real data distribution using …

Bearing is a key component in industrial machinery and its failure may lead to unwanted downtime and economic loss. Hence, it is necessary to predict the remaining useful life (RUL) of bearings. Conventional data-driven approaches of RUL prediction require expert domain knowledge for manual feature extraction and may suffer from data distribution discrepancy between training and test data. In this study, we propose a novel generalized multiscale feature extraction method with generative adversarial networks. The adversarial training learns the distribution of training data from different bearings and is introduced for health stage division and RUL prediction. To capture the sequence feature from a one-dimensional vibration signal, we adapt a U-Net architecture that reconstructs features to process them with multiscale layers in the generator of the adversarial network. To validate the proposed method …

Wearable sensor-based human activity recognition (HAR) has emerged as a principal research area and is utilized in a variety of applications. Recently, deep learning-based methods have achieved significant improvement in the HAR field with the development of human–computer interaction applications. However, most of them are limited to operating in a local neighborhood in the process of a standard convolution neural network, and correlations between different sensors on body positions are ignored. In addition, even though several recent existing works considered the correlations between different sensor positions, they still face significant challenging problems with performance degradation due to large gaps in the distribution of training and test data, and behavioral differences between subjects. In this work, we propose a novel Transformer-based Adversarial learning framework for human activity …

In this study, we developed a novel data-driven fault detection and diagnosis (FDD) method for bearing faults in induction motors where the fault condition data are imbalanced. First, we propose a bearing fault detector based on convolutional neural networks (CNN), in which the vibration signals from a test bench are used as inputs after an image transformation procedure. Experimental results demonstrate that the proposed classifier for FDD performs well (accuracy of 88% to 99%) even when the volume of normal and fault condition data is imbalanced (imbalance ratio varies from 20:1 to 200:1). Additionally, our generative model reduces the level of data imbalance by oversampling. The results improve the accuracy of FDD (by up to 99%) when a severe imbalance ratio (200:1) is assumed.

Document image enhancement and binarization methods are often used to improve the accuracy and efficiency of document image analysis tasks such as text recognition. Traditional non-machine-learning methods are constructed on low-level features in an unsupervised manner but have difficulty with binarization on documents with severely degraded backgrounds. Convolutional neural network (CNN)based methods focus only on grayscale images and on local textual features. In this paper, we propose a two-stage color document image enhancement and binarization method using generative adversarial neural networks. In the first stage, four color-independent adversarial networks are trained to extract color foreground information from an input image for document image enhancement. In the second stage, two independent adversarial networks with global and local features are trained for image binarization …

We present MoCaPose, a novel wearable motion capturing (MoCap) approach to continuously track the wearer's upper body's dynamic poses through multi-channel capacitive sensing integrated in fashionable, loose-fitting jackets. Unlike conventional wearable IMU MoCap based on inverse dynamics, MoCaPose decouples the sensor position from the pose system. MoCaPose uses a deep regressor to continuously predict the 3D upper body joints coordinates from 16-channel textile capacitive sensors, unbound by specific applications. The concept is implemented through two prototyping iterations to first solve the technical challenges, then establish the textile integration through fashion-technology co-design towards a design-centric smart garment. A 38-hour dataset of synchronized video and capacitive data from 21 participants was recorded for validation. The motion tracking result was validated on multiple …

A 0.8 μm-pitch 64 megapixels ultrahighresolution CMOS image sensor has been demonstrated for mobile applications for the first time. Full-well capacity (FWC) of 6k e- was achieved in 0.8 μm pixels as the best in the world, and the advanced color filter (CF) isolation technology was introduced to overcome sensitivity degradation. Dual conversion gain (CG) technology was also first applied to mobile applications to improve the FWC performance of Tetracell up to 12k e-. In addition, highly refined deep trench isolation (DTI) and photodiode design significantly improved dark noise characteristics.

Text classification has numerous use cases including sentiment analysis, spam detection, document classification, hate speech detection, etc. In realistic settings, classification on text data confronts imbalanced data conditions where classes of interest usually compose a minor fraction. Deep neural networks used for text classification, such as recurrent neural networks and transformer networks, suffer from a lack of efficient methods addressing imbalanced data. Traditional data-level methods attempting to mitigate distributional skew include oversampling and undersampling. The oversampling methods destruct the quality of original language representation of the sparse data coming from minority classes whereas the undersampling methods fail to fully utilize the rich context of majority classes. We address such issues in data-driven approaches by enforcing continual learning on imbalanced data by partitioning the …

Improving worker safety and productivity is of paramount importance in the manufacturing industry, driving the adoption of advanced sensing and control systems. This concern is particularly relevant within the framework of Industry 5.0. In this context, wearable sensors offer a promising solution by enabling continuous and unobtrusive monitoring of workers’ activities in the manufacturing line. This book chapter focuses on wearable sensor-based human activity recognition and its role in promoting worker safety in manufacturing environments. Specifically, we present a case study on wearable sensor-based worker activity recognition in a manufacturing line with a mobile robot. As wearable sensors comprise various sensor types, we investigate and compare sensor data fusion approaches using neural network models to effectively handle the multimodal sensor data. In addition, we introduce several deep learning …

Recognizing facial activity is a well-understood (but non-trivial) computer vision problem. However, reliable solutions require a camera with a good view of the face, which is often unavailable in wearable settings. Furthermore, in wearable applications, where systems accompany users throughout their daily activities, a permanently running camera can be problematic for privacy (and legal) reasons. This work presents an alternative solution based on the fusion of wearable inertial sensors, planar pressure sensors, and acoustic mechanomyography (muscle sounds). The sensors were placed unobtrusively in a sports cap to monitor facial muscle activities related to facial expressions. We present our integrated wearable sensor system, describe data fusion and analysis methods, and evaluate the system in an experiment with thirteen subjects from different cultural backgrounds (eight countries) and both sexes (six …

Recent advances in deep learning have significantly improved the performance of various computer vision applications. However, discovering novel categories in an incremental learning scenario remains a challenging problem due to the lack of prior knowledge about the number and nature of new categories. Existing methods for novel category discovery are limited by their reliance on labeled datasets and prior knowledge about the number of novel categories and the proportion of novel samples in the batch. To address the limitations and more accurately reflect real-world scenarios, in this paper, we propose a novel unsupervised class incremental learning approach for discovering novel categories on unlabeled sets without prior knowledge. The proposed method fine-tunes the feature extractor and proxy anchors on labeled sets, then splits samples into old and novel categories and clusters on the unlabeled dataset. Furthermore, the proxy anchors-based exemplar generates representative category vectors to mitigate catastrophic forgetting. Experimental results demonstrate that our proposed approach outperforms the state-of-the-art methods on fine-grained datasets under real-world scenarios.

We present a wearable system to detect body postures and gestures that does not require sensors to be firmly fixed to the body or integrated into a tight-fitting garment. The sensing system can be used in a loose piece of clothing such as a coat/blazer. It is based on the well-known theremin musical instrument, which we have unobtrusively integrated into a standard men’s blazer using conductive textile antennas and OpenTheremin hardware as a prototype, the ”MoCaBlazer.” Fourteen participants with diverse body sizes and balanced gender distribution mimicked 20 arm/torso movements with the unbuttoned, single-sized blazer. State-of-the-art deep learning approaches were used to achieve average recognition accuracy results of 97.18% for leave one recording out and 86.25% for user independent recognition.

Early detection of faults in rotating machinery systems is crucial in preventing system failure, increasing safety, and reducing maintenance costs. Current methods of fault detection suffer from the lack of efficient feature extraction method, the need for designating a threshold producing minimal false alarm rates, and the need for expert domain knowledge, which is costly. In this paper, we propose a novel data-driven health division method based on convolutional neural networks using a graphical representation of time series data, called a nested scatter plot. The proposed method trains the model with a small amount of labeled data and does not require a threshold value to predict the health state of rotary machines. Notwithstanding the lack of datasets that show the ground truth of health stages, our experiments with two open datasets of run-to-failure bearing demonstrated that our method is able to detect the early symptoms of bearing wear earlier and more efficiently than other threshold-based health indicator methods.

Manufacturing industries strive to improve production efficiency and product quality by deploying advanced sensing and control systems. Wearable sensors are emerging as a promising solution for achieving this goal, as they can provide continuous and unobtrusive monitoring of workers’ activities in the manufacturing line. This article presents a novel wearable sensing prototype that combines IMU and body capacitance sensing modules to recognize worker activities in the manufacturing line. To handle these multimodal sensor data, we propose and compare early, and late sensor data fusion approaches for multichannel time-series convolutional neural networks and deep convolutional LSTM. We evaluate the proposed hardware and neural network model by collecting and annotating sensor data using the proposed sensing prototype and Apple Watches in the testbed of the manufacturing line. Experimental …

The data imbalance problem is a frequent bottleneck in the classification performance of neural networks. In this paper, we propose a novel supervised discriminative feature generation (DFG) method for a minority class dataset. DFG is based on the modified structure of a generative adversarial network consisting of four independent networks: generator, discriminator, feature extractor, and classifier. To augment the selected discriminative features of the minority class data by adopting an attention mechanism, the generator for the class-imbalanced target task is trained, and the feature extractor and classifier are regularized using the pre-trained features from a large source data. The experimental results show that the DFG generator enhances the augmentation of the label-preserved and diverse features, and the classification results are significantly improved on the target task. The feature generation model can …

User dependence remains one of the most difficult general problems in Human Activity Recognition (HAR), in particular when using wearable sensors. This is due to the huge variability of the way different people execute even the simplest actions. In addition, detailed sensor fixtures and placement will be different for different people or even at different times for the same users. In theory, the problem can be solved by a large enough data set. However, recording data sets that capture the entire diversity of complex activity sets is seldom practicable. Instead, models are needed that focus on features that are invariant across users. To this end, we present an adversarial subject-independent feature extraction method with the maximum mean discrepancy (MMD) regularization for human activity recognition. The proposed model is capable of learning a subject-independent embedding feature representation from multiple …

While sports activity recognition is a well studied subject in mobile, wearable and ubiquitous computing, work to date mostly focuses on recognition and counting of specific exercise types. Quality assessment is a much more difficult problem with significantly less published results. In this work, we present Quali-Mat: a method for evaluating the quality of execution (QoE) in exercises using a smart sports mat that can measure the dynamic pressure profiles during full-body, body-weight exercises. As an example, our system not only recognizes that the user is doing push-ups, but also distinguishes 5 subtly different types of push-ups, each of which (according to sports science literature and professional trainers) has a different effect on different muscle groups. We have investigated various machine learning algorithms targeting the specific type of spatio-temporal data produced by the pressure mat system. We …

This work offers a heuristic evaluation of the effects of variations in machine learning training regimes and learning paradigms on the energy consumption of computing, especially HPC hardware with a life-cycle aware perspective. While increasing data availability and innovation in high-performance hardware fuels the training of sophisticated models, it also fosters the fading perception of energy consumption and carbon emission. Therefore, the goal of this work is to raise awareness about the energy impact of general training parameters and processes, from learning rate over batch size to knowledge transfer. Multiple setups with different hyperparameter configurations are evaluated on three different hardware systems. Among many results, we have found out that even with the same model and hardware to reach the same accuracy, improperly set training hyperparameters consume up to 5 times the energy of …

We address the well-known wearable activity recognition problem of having to work with sensors that are non-optimal in terms of information they provide but have to be used due to wearability/usability concerns (e.g. the need to work with wrist-worn IMUs because they are embedded in most smart watches). To mitigate this problem we propose a method that facilitates the use of information from sensors that are only present during the training process and are unavailable during the later use of the system. The method transfers information from the source sensors to the latent representation of the target sensor data through contrastive loss that is combined with the classification loss during joint training (fig. 1). We evaluate the method on the well-known PAMAP2 and Opportunity benchmarks for different combinations of source and target sensors showing average (over all activities) F1 score improvements of …

Human-centric Ubiquitous Intelligence (HUI) Lab

교수님 소개

서성호

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

CoSS: Co-optimizing Sensor and Sampling rate for data-efficient human activity recognition

COA-HAR: Exploring Contrastive Online test-time Adaptation for wearable sensor-based Human Activity Recognition using sensor data augmentation

Assessing Grid Penalized Reinforcement Learning for Renewable Energy Management of Power-to-X Integrated with Intermediate Storage

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

서성호

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

CoSS: Co-optimizing Sensor and Sampling rate for data-efficient human activity recognition

COA-HAR: Exploring Contrastive Online test-time Adaptation for wearable sensor-based Human Activity Recognition using sensor data augmentation

Assessing Grid Penalized Reinforcement Learning for Renewable Energy Management of Power-to-X Integrated with Intermediate Storage

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집