Ah-Hyung Shin

Sung-Hoon Lee

Juwon Seo

Jae-Ho Lee

Keon-Hee Park

Min-Yeong Park

Seun-An Choe

Visual & General Intelligence Lab. at Department of AI, Korea University is interested in developing multi-modal AI algorithms associated with computer vision and natural language processing, studing continual learning techniques of modern deep learning algorithms, learning methods for future AI, deep generative models, embodied AI, and medical AI.

Detecting Unknown Objects via Energy-based Separation for Open World Object Detection

Personalized Audio-driven Whole-body Talking Avatars

Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing

Dynamic Texture Modeling of 3D Clothed Gaussian Avatars from a Single Video

Perturb a Model, Not an Image: Towards Robust Privacy Protection via Anti-Personalized Diffusion Models

Disentangled Concepts Speak Louder Than Words: Explainable Video Action Recognition

ConceptSplit: Decoupled Multi-Concept Personalization of Diffusion Models via Token-wise Adaptation and Attention Disentanglement

GeoAvatar: Adaptive Geometrical Gaussian Splatting for 3D Head Avatar

SFUOD: Source-Free Unknown Object Detection

ESSENTIAL: Episodic and Semantic Memory Integration for Video Class-Incremental Learning

When Will It Fail?: Anomaly to Prompt for Forecasting Future Anomalies in Time Series

Oral at ICML Workshop on Machine Unlearning for Generative AI (MUGen)

Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech

ESC: Erasing Space Concept for Knowledge Deletion

Universal Domain Adaptation for Semantic Segmentation

Test-Time Fine-Tuning of Image Compression Models for Multi-Task Adaptability

Multispectral Pedestrian Detection with Sparsely Annotated Label

Online Continuous Generalized Category Discovery

Versatile Incremental Learning: Towards Class and Domain-Agnostic Incremental Learning

Towards Model-Agnostic Dataset Condensation by Heterogeneous Models

Pre-trained Vision and Language Transformers Are Few-Shot Incremental Learners

Generative Unlearning for Any Identity

Open-Set Domain Adaptation for Semantic Segmentation

Online Class Incremental Learning on Stochastic Blurry Task Boundary via Mask and Visual Prompt Tuning

LFS-GAN: Lifelong Few-Shot Image Generation

LINe: Out-of-Distribution Detection by Leveraging Important Neurons

WINE : Wavelet-Guided GAN Inversion and Editing for High-Fidelity Refinement

Towards High-fidelity Head Blending with Chroma Keying for Industrial Applications

CED: Comparing Embedding Differences for Detecting Out-of-Distribution and Hallucinated Text

RADIO: Reference-Agnostic Dubbing Video Synthesis

GLAD: Global-Local View Alignment and Background Debiasing for Video Domain Adaptation

IntereStyle: Encoding an Interest Region for Robust StyleGAN Inversion

Non-Probabilistic Cosine Similarity Loss for Few-Shot Image Classification

Unsupervised Domain Adaptation for Medical Image Segmentation Using Adaptogen-Perturbation

SSMPD: Semi-Supervised Learning for Multispectral Pedestrian Detection

SR-EM: Episodic Memory Aware of Semantic Relations Based on Hierarchical Clustering Resonance Network

Adaptive Developmental Resonance Network

Convolutional Neural Network with Developmental Memory for Continual Learning

Developmental Resonance Network

Deep ART Neural Model for Biologically Inspired Episodic Memory and Its Application to Task Performance of Robots

PCBEAR: Pose Concept Bottleneck for Explainable Action Recognition

A Two-phase Multi-channel Classification Resonance Network

Context Preference-based Deep Adaptive Resonance Theory: Integrating User Preference into Episodic Memory Encoding and Retrieval

Memory-based Realization of Task Intelligence for Robots in Human Environment

One small step for generative ai, one giant leap for agi: A complete survey on chatgpt in aigc era

Pre-trained vision and language transformers are few-shot incremental learners

Line: Out-of-distribution detection by leveraging important neurons

Online class incremental learning on stochastic blurry task boundary via mask and visual prompt tuning

Open-set domain adaptation for semantic segmentation

Convolutional neural network with developmental memory for continual learning

Deep ART neural model for biologically inspired episodic memory and its application to task performance of robots

Task intelligence of robots: Neural model-based mechanism of thought and online motion planning

Lfs-gan: Lifelong few-shot image generation

Time series anomaly detection using transformer-based GAN with two-step masking

Generative unlearning for any identity

Deep adaptive resonance theory for learning biologically inspired episodic memory

Realization of task intelligence for service robots in an unstructured environment

Developmental resonance network

REM-ART: Reward-based electromagnetic adaptive resonance theory

Interestyle: Encoding an interest region for robust stylegan inversion

Wav2NeRF: Audio-driven realistic talking head generation via wavelet-based NeRF

Online continuous generalized category discovery

WaGI: wavelet-based GAN inversion for preserving high-frequency image details

Universal domain adaptation for semantic segmentation

Jong-Hwan Kim

Jinwoo Choi

Seong Tae Kim

Keonhee Park

Deokhwa Kim

Kyungwoo Song

Chaoning Zhang

Jung Uk Kim

Tae-Young Lee

Jong Hwan Ko

Hah Min Lew

Jaejun Yoo

Kyle Min

HyeongYeop Kang

Gyeong-Moon Park

OpenAI has recently released GPT-4 (a.k.a. ChatGPT plus), which is demonstrated to be one small step for generative AI (GAI), but one giant leap for artificial general intelligence (AGI). Since its official release in November 2022, ChatGPT has quickly attracted numerous users with extensive media coverage. Such unprecedented attention has also motivated numerous researchers to investigate ChatGPT from various aspects. According to Google scholar, there are more than 500 articles with ChatGPT in their titles or mentioning it in their abstracts. Considering this, a review is urgently needed, and our work fills this gap. Overall, this work is the first to survey ChatGPT with a comprehensive review of its underlying technology, applications, and challenges. Moreover, we present an outlook on how ChatGPT might evolve to realize general-purpose AIGC (a.k.a. AI-generated content), which will be a significant milestone for the development of AGI.

Few-Shot Class Incremental Learning (FSCIL) is a task that requires a model to learn new classes incrementally without forgetting when only a few samples for each class are given. FSCIL encounters two significant challenges: catastrophic forgetting and overfitting and these challenges have driven prior studies to primarily rely on shallow models such as ResNet-18. Even though their limited capacity can mitigate both forgetting and overfitting issues it leads to inadequate knowledge transfer during few-shot incremental sessions. In this paper we argue that large models such as vision and language transformers pre-trained on large datasets can be excellent few-shot incremental learners. To this end we propose a novel FSCIL framework called PriViLege Pre-trained Vision and Language transformers with prompting functions and knowledge distillation. Our framework effectively addresses the challenges of catastrophic forgetting and overfitting in large models through new pre-trained knowledge tuning (PKT) and two losses: entropy-based divergence loss and semantic knowledge distillation loss. Experimental results show that the proposed PriViLege significantly outperforms the existing state-of-the-art methods with a large margin eg+ 9.38% in CUB200+ 20.58% in CIFAR-100 and+ 13.36% in miniImageNet. Our implementation code is available at https://github. com/KHU-AGI/PriViLege.

It is important to quantify the uncertainty of input samples, especially in mission-critical domains such as autonomous driving and healthcare, where failure predictions on out-of-distribution (OOD) data are likely to cause big problems. OOD detection problem fundamentally begins in that the model cannot express what it is not aware of. Post-hoc OOD detection approaches are widely explored because they do not require an additional re-training process which might degrade the model's performance and increase the training cost. In this study, from the perspective of neurons in the deep layer of the model representing high-level features, we introduce a new aspect for analyzing the difference in model outputs between in-distribution data and OOD data. We propose a novel method, Leveraging Important Neurons (LINe), for post-hoc Out of distribution detection. Shapley value-based pruning reduces the effects of …

Continual learning aims to learn a model from a continuous stream of data, but it mainly assumes a fixed number of data and tasks with clear task boundaries. However, in real-world scenarios, the number of input data and tasks is constantly changing in a statistical way, not a static way. Although recently introduced incremental learning scenarios having blurry task boundaries somewhat address the above issues, they still do not fully reflect the statistical properties of real-world situations because of the fixed ratio of disjoint and blurry samples. In this paper, we propose a new Stochastic incremental Blurry task boundary scenario, called Si-Blurry, which reflects the stochastic properties of the real-world. We find that there are two major challenges in the Si-Blurry scenario:(1) inter-and intra-task forgettings and (2) class imbalance problem. To alleviate them, we introduce Mask and Visual Prompt tuning (MVP). In MVP, to address the inter-and intra-task forgetting issues, we propose a novel instance-wise logit masking and contrastive visual prompt tuning loss. Both of them help our model discern the classes to be learned in the current batch. It results in consolidating the previous knowledge. In addition, to alleviate the class imbalance problem, we introduce a new gradient similarity-based focal loss and adaptive feature scaling to ease overfitting to the major classes and underfitting to the minor classes. Extensive experiments show that our proposed MVP significantly outperforms the existing state-of-the-art methods in our challenging Si-Blurry scenario.

Unsupervised domain adaptation (UDA) for semantic segmentation aims to transfer the pixel-wise knowledge from the labeled source domain to the unlabeled target domain. However current UDA methods typically assume a shared label space between source and target limiting their applicability in real-world scenarios where novel categories may emerge in the target domain. In this paper we introduce Open-Set Domain Adaptation for Semantic Segmentation (OSDA-SS) for the first time where the target domain includes unknown classes. We identify two major problems in the OSDA-SS scenario as follows: 1) the existing UDA methods struggle to predict the exact boundary of the unknown classes and 2) they fail to accurately predict the shape of the unknown classes. To address these issues we propose Boundary and Unknown Shape-Aware open-set domain adaptation coined BUS. Our BUS can accurately discern the boundaries between known and unknown classes in a contrastive manner using a novel dilation-erosion-based contrastive loss. In addition we propose OpenReMix a new domain mixing augmentation method that guides our model to effectively learn domain and size-invariant features for improving the shape detection of the known and unknown classes. Through extensive experiments we demonstrate that our proposed BUS effectively detects unknown classes in the challenging OSDA-SS scenario compared to the previous methods by a large margin.

Convolutional neural networks (CNNs) are one of the most successful deep neural networks. Indeed, most of the recent applications related to computer vision are based on CNNs. However, when learning new tasks in a sequential manner, CNNs face catastrophic forgetting: they forget a considerable amount of previously learned tasks while adapting to novel tasks. To overcome this main barrier to continual learning with CNNs, we introduce developmental memory (DM) into a CNN, continually generating submemory networks to learn important features of individual tasks. A novel training method, referred to here as guided learning (GL), guides the newly generated submemory to become an expert on the new task, eventually improving the performance of the overall network. At the same time, the existing submemories attempt to preserve the knowledge of old tasks. Experiments on image classification tasks show …

Robots are expected to perform smart services and to undertake various troublesome or difficult tasks in the place of humans. Since these human-scale tasks consist of a temporal sequence of events, robots need episodic memory to store and retrieve the sequences to perform the tasks autonomously in similar situations. As episodic memory, in this paper we propose a novel Deep adaptive resonance theory (ART) neural model and apply it to the task performance of the humanoid robot, Mybot, developed in the Robot Intelligence Technology Laboratory at KAIST. Deep ART has a deep structure to learn events, episodes, and even more like daily episodes. Moreover, it can retrieve the correct episode from partial input cues robustly. To demonstrate the effectiveness and applicability of the proposed Deep ART, experiments are conducted with the humanoid robot, Mybot, for performing the three tasks of arranging toys …

The crux of the realization of task intelligence for robots is to design the memory module for storing temporal event sequences of tasks, the mechanism of thought for reasoning, and motion planning methodology for execution, among others. In this paper, task intelligence is realized using episodic memory, neural model-based mechanism of thought, and an online motion planning algorithm. Robots are taught either by demonstration or symbolic description. A behavior appropriate to the current situation is selected by the developmental episodic memory-based mechanism of thought, while a proper task is retrieved from Deep adaptive resonance theory (ART). The behaviors are executed safely and quickly with the proposed motion planning algorithm. The effectiveness and applicability of task intelligence are demonstrated through experiments with the humanoid robot, Mybot, developed in the Robot Intelligence …

We address a challenging lifelong few-shot image generation task for the first time. In this situation, a generative model learns a sequence of tasks using only a few samples per task. Consequently, the learned model encounters both catastrophic forgetting and overfitting problems at a time. Existing studies on lifelong GANs have proposed modulation-based methods to prevent catastrophic forgetting. However, they require considerable additional parameters and cannot generate high-fidelity and diverse images from limited data. On the other hand, the existing few-shot GANs suffer from severe catastrophic forgetting when learning multiple tasks. To alleviate these issues, we propose a framework called Lifelong Few-Shot GAN (LFS-GAN) that can generate high-quality and diverse images in lifelong few-shot image generation task. Our proposed framework learns each task using an efficient task-specific modulator-Learnable Factorized Tensor (LeFT). LeFT is rank-constrained and has a rich representation ability due to its unique reconstruction technique. Furthermore, we propose a novel mode seeking loss to improve the diversity of our model in low-data circumstances. Extensive experiments demonstrate that the proposed LFS-GAN can generate high-fidelity and diverse images without any forgetting and mode collapse in various domains, achieving state-of-the-art in lifelong few-shot image generation task. Surprisingly, we find that our LFS-GAN even outperforms the existing few-shot GANs in the few-shot image generation task. The code is available at Github.

Time series anomaly detection is a task that determines whether an unseen signal is normal or abnormal, and it is a crucial function in various real-world applications. Typical approach is to learn normal data representation using generative models, like Generative Adversarial Network (GAN), to discriminate between normal and abnormal signals. Recently, a few studies actively adopt Transformer to model time series data, but there is no pure Transformer-based GAN framework for time series anomaly detection. As a pioneer work, we propose a new pure Transformer-based GAN framework, called AnoFormer, and its effective training strategy for better representation learning. Specifically, we improve the detection ability of our model by introducing two-step masking strategies. The first step is Random masking: we design a random mask pool to hide parts of the signal randomly. This allows our model to learn the …

Recent advances in generative models trained on large-scale datasets have made it possible to synthesize high-quality samples across various domains. Moreover the emergence of strong inversion networks enables not only a reconstruction of real-world images but also the modification of attributes through various editing methods. However in certain domains related to privacy issues eg human faces advanced generative models along with strong inversion methods can lead to potential misuses. In this paper we propose an essential yet under-explored task called generative identity unlearning which steers the model not to generate an image of a specific identity. In the generative identity unlearning we target the following objectives:(i) preventing the generation of images with a certain identity and (ii) preserving the overall quality of the generative model. To satisfy these goals we propose a novel framework Generative Unlearning for Any Identity (GUIDE) which prevents the reconstruction of a specific identity by unlearning the generator with only a single image. GUIDE consists of two parts:(i) finding a target point for optimization that un-identifies the source latent code and (ii) novel loss functions that facilitate the unlearning procedure while less affecting the learned distribution. Our extensive experiments demonstrate that our proposed method achieves state-of-the-art performance in the generative machine unlearning task. The code is available at https://github. com/KHU-AGI/GUIDE.

Biologically inspired episodic memory is able to store time sequential events, and to recall all of them from partial information. Because of the advantages of episodic memory, the biological concepts of episodic memory have been utilized to many applications. In this research, we propose a new memory model, called Deep ART (Adaptive Resonance Theory), to make a robust memory system for learning episodic memory. Deep ART has an attribute field in the bottom layer, which is newly designed to get semantic information of inputs. After encoding all inputs with their features, events are categorized in the event field using specified inputs. Since an episode is made of a temporal sequence of events, Deep ART makes event sequences with proposed sequence encoding and decoding processes. They can encode any temporal sequence of events, even if there are duplicated events in the episode. Moreover …

In order to perform various tasks using a robot in a real environment, it is necessary to learn the tasks based on recognition, to be able to derive a task sequence suitable for the situation, and to be able to generate a behavior adaptively. To deal with this issue, this paper proposes a system for realizing task intelligence having a memory module motivated by human episodic memory, and a task planning module to resolve the current situation. In addition, this paper proposes a technique that can modify demonstrated trajectories according to current robot states and recognized target positions in order to perform the determined task sequence, as well as a technique that can generate the modified trajectory without collisions with surrounding obstacles. The effectiveness and applicability of the task intelligence are demonstrated through experiments with Mybot, a humanoid robot developed in the Robot Intelligence …

Adaptive resonance theory (ART) networks deal with normalized input data only, which means that they need the normalization process for the raw input data, under the assumption that the upper and lower bounds of the input data are known in advance. Without such an assumption, ART networks cannot be utilized. To solve this problem and improve the learning performance, inspired by the ART networks, we propose a developmental resonance network (DRN) by employing new techniques of a global weight and node connection and grouping processes. The proposed DRN learns the global weight converging to the unknown range of the input data and properly clusters by grouping similar nodes into one. These techniques enable DRN to learn the raw input data without the normalization process while retaining the stability, plasticity, and memory usage efficiency without node proliferation. Simulation results …

This paper proposes a Reward-based Electromagnetic Adaptive Resonance Theory (REM-ART) mainly for reducing the retrieval error, which can store and retrieve episodes consisting of a temporal sequence of events. With the existing EM-ART it is difficult to predict a correct episode when there are noisy inputs that are sparse or distorted. To overcome this problem, the proposed REM-ART has a reward channel along with an event channel, lending it robustness to noisy inputs. The reward channel generates a reward signal to reinforce the activation of a proper episode node. The proposed REM-ART is applied to predict and retrieve a correct episode from noisy inputs, out of episodes that are stored through the visual demonstration using an RGB-D camera.

Recently, manipulation of real-world images has been highly elaborated along with the development of Generative Adversarial Networks (GANs) and corresponding encoders, which embed real-world images into the latent space. However, designing encoders of GAN still remains a challenging task due to the trade-off between distortion and perception. In this paper, we point out that the existing encoders try to lower the distortion not only on the interest region, e . g ., human facial region but also on the uninterest region, e . g ., background patterns and obstacles. However, most uninterest regions in real-world images are located at out-of-distribution (OOD), which are infeasible to be ideally reconstructed by generative models. Moreover, we empirically find that the uninterest region overlapped with the interest region can mangle the original feature of the interest region, e . g ., a microphone overlapped with a facial …

Talking head generation is an essential task in various real-world applications such as film making and virtual reality. To this end, recent works focus on the NeRF-based methods that can capture the 3D structural information of faces and generate more natural and vivid talking videos. However, the existing NeRF-based methods fail to accurately generate the audio-synced videos. In this paper, we point out that the previous methods do not consider the audio-visual representations explicitly, which is crucial for precise lip synchronization. Moreover, the existing methods struggle to generate high-frequency details, making the generation results unnatural. To overcome these problems, we propose a novel audio-synced and high-fidelity NeRF-based talking head generation framework, named Wav2NeRF, which learns audio-visual cross-modality representations and employs the wavelet transform for better visual …

With the advancement of deep neural networks in computer vision, artificial intelligence (AI) is widely employed in real-world applications. However, AI still faces limitations in mimicking high-level human capabilities, such as novel category discovery, for practical use. While some methods utilizing offline continual learning have been proposed for novel category discovery, they neglect the continuity of data streams in real-world settings. In this work, we introduce Online Continuous Generalized Category Discovery ( OCGCD ), which considers the dynamic nature of data streams where data can be created and deleted in real time. Additionally, we propose a novel method, DEAN , D iscovery via E nergy guidance and feature A ugmentatio N , which can discover novel categories in an online manner through energy-guided discovery and facilitate discriminative learning via energy-based contrastive loss. Furthermore …

Recent GAN inversion models focus on preserving image-specific details through various methods, e.g., generator tuning or feature mixing. While those are helpful for preserving details compared to a naiive low-rate latent inversion, they still fail to maintain high-frequency features precisely. In this paper, we point out that the existing GAN inversion models have inherent limitations in both structural and training aspects, which preclude the delicate reconstruction of high-frequency features. Especially, we prove that the widely-used loss term in GAN inversion, i.e., L2, is biased to reconstruct low-frequency features mainly. To overcome this problem, we propose a novel GAN inversion model, coined WaGI, which enables to handle high-frequency features explicitly, by using a novel wavelet-based loss term and a newly proposed wavelet fusion scheme. To the best of our knowledge, WaGI is the first attempt to interpret GAN inversion in the frequency domain. We demonstrate that WaGI shows outstanding results on both inversion and editing, compared to the existing state-of-the-art GAN inversion models. Especially, WaGI robustly preserves high-frequency features of images even in the editing scenario. We will release our code with the pre-trained model after the review.

Unsupervised domain adaptation for semantic segmentation (UDA-SS) aims to transfer knowledge from labeled synthetic data (source) to unlabeled real-world data (target). Traditional UDA-SS methods work on the assumption that the category settings between the source and target domains are known in advance. However, in real-world scenarios, the category settings of the source and target are unknown due to the lack of labels in the target, resulting in the existence of target-private or source-private classes. Traditional UDA-SS methods struggle with this change, leading to negative transfer and performance degradation. To address these issues, we propose Universal Domain Adaptation for Semantic Segmentation (UniDA-SS) for the first time to achieve good performance even when the category settings of source and target are unknown. We defined the problem in the UniDA-SS scenario as that the confidence score of common Unsupervised domain adaptation for semantic segmentation (UDA-SS) aims to transfer knowledge from labeled source data to unlabeled target data. However, traditional UDA-SS methods assume that category settings between source and target domains are known, which is unrealistic in real-world scenarios. This leads to performance degradation if private private classes exist. To address this limitation, we propose Universal Domain Adaptation for Semantic Segmentation (UniDA-SS), achieving robust adaptation even without prior knowledge of category settings. We define the problem in the UniDA-SS scenario as low confidence scores of common classes in the target domain, which leads to confusion with …

Visual & General Intelligence Lab

교수님 소개

박경문

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Detecting Unknown Objects via Energy-based Separation for Open World Object Detection

Personalized Audio-driven Whole-body Talking Avatars

Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

박경문

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Detecting Unknown Objects via Energy-based Separation for Open World Object Detection

Personalized Audio-driven Whole-body Talking Avatars

Edit-As-Act: Goal-Regressive Planning for Open-Vocabulary 3D Indoor Scene Editing

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집