The SDML lab focuses on understanding and modeling the inherent dynamics within both data and models. Firstly, the primary emphasis is on data, with extensive research into probabilistic machine learning tools such as Gaussian processes and neural differential equations, thereby advancing the frontier in these domains. On the model front, the objective is to investigate the equivalence between deep learning models employing neural differential equations and the correspondence between training dynamics, aiming to reveal their theoretical properties. The applications target in diverse areas such as time series analysis, explaining sequential data, and financial modeling encompassing pricing, hedging, and risk neutralization.

Stochastic Dynamics and Machine Learning Lab

SigFormer: Signature Transformers for Deep Hedging

Global Contrastive Learning for Long-Tailed Classification

Roles of ambient temperature and relative humidity on the relationship between fine particulate matter and gaseous pollutants in the largest industrial city of Ulsan, South Korea

Learning fractional white noises in neural stochastic differential equations

Characterizing Deep Gaussian Processes via Nonlinear Recurrence Systems

Learning Compositional Sparse Gaussian Processes with a Shrinkage Prior

Discovering Latent Covariance Structures for Multiple Time Series

Confirmatory Bayesian online change point detection in the covariance structure of Gaussian processes

Automatic Construction of Nonparametric Relational Regression Models for Multiple Time Series

Automatic construction of nonparametric relational regression models for multiple time series

Learning Fractional White Noises in Neural Stochastic Differential Equations

Neural ODE Transformers: Analyzing Internal Dynamics and Adaptive Fine-tuning

Automatic Generation of Probabilistic Programming from Time Series Data

CASUAL: Conditional Support Alignment for Domain Adaptation with Label Shift

Conditional Support Alignment for Domain Adaptation with Label Shift

Towards Transparent Time Series Analysis: Exploring Methods and Enhancing Interpretability

Probabilistic Model Discovery: Relational Learning and Scalable Inference

Explain pathology in Deep Gaussian Process using Chaos Theory

Jaesik Choi

Toan Tran

Thanh Nguyen-Tang

Jiyeon Han

Kyowoon Lee

Hung Bui

Quang Tran Vuong

Truong-Son Hy

Thong Bach

Anh T Nguyen

Anh Tong

Gaussian Processes (GPs) provide a general and analytically tractable way of modeling complex time-varying, nonparametric functions. The Automatic Bayesian Covariance Discovery (ABCD) system constructs natural-language description of time-series data by treating unknown time-series data nonparametrically using GP with a composite covariance kernel function. Unfortunately, learning a composite covariance kernel with a single time-series data set often results in less informative kernel that may not give qualitative, distinctive descriptions of data. We address this challenge by proposing two relational kernel learning methods which can model multiple time-series data sets by finding common, shared causes of changes. We show that the relational kernel learning methods find more accurate models for regression problems on several real-world data sets; US stock data, US house price index data and currency exchange rate data.

Differential equations play important roles in modeling complex physical systems. Recent advances present interesting research directions by combining differential equations with neural networks. By including noise, stochastic differential equations (SDEs) allows us to model data with uncertainty and measure imprecision. There are many variants of noises known to exist in many real-world data. For example, previously white noises are idealized and induced by Brownian motions. Nevertheless, there is a lack of machine learning models that can handle such noises. In this paper, we introduce a generalized fractional white noise to existing models and propose an efficient approximation of noise sample paths based on classical integration methods and sparse Gaussian processes. Our experimental results demonstrate that the proposed model can capture noise characteristics such as continuity from various time series data, therefore improving model fittings over existing models. We examine how we can apply our approach to score-based generative models, showing that there exists a case of our generalized noise resulting in a better image generation measure.

Analyzing multivariate time series data is important to predict future events and changes of complex systems in finance, manufacturing, and administrative decisions. The expressiveness power of Gaussian Process (GP) regression methods has been significantly improved by compositional covariance structures. In this paper, we present a new GP model which naturally handles multiple time series by placing an Indian Buffet Process (IBP) prior on the presence of shared kernels. Our selective covariance structure decomposition allows exploiting shared parameters over a set of multiple, selected time series. We also investigate the well-definedness of the models when infinite latent components are introduced. We present a pragmatic search algorithm which explores a larger structure space efficiently. Experiments conducted on five real-world data sets demonstrate that our new model outperforms existing methods in term of structure discoveries and predictive performances.

Deep hedging is a promising direction in quantitative finance, incorporating models and techniques from deep learning research. While giving excellent hedging strategies, models inherently requires careful treatment in designing architectures for neural networks. To mitigate such difficulties, we introduce SigFormer, a novel deep learning model that combines the power of path signatures and transformers to handle sequential data, particularly in cases with irregularities. Path signatures effectively capture complex data patterns, while transformers provide superior sequential attention. Our proposed model is empirically compared to existing methods on synthetic data, showcasing faster learning and enhanced robustness, especially in the presence of irregular underlying price data. Additionally, we validate our model performance through a real-world backtest on hedging the S&P 500 index, demonstrating positive …

Recent advancements in large language models (LLMs) based on transformer architectures have sparked significant interest in understanding their inner workings. In this paper, we introduce a novel approach to modeling transformer architectures using highly flexible non-autonomous neural ordinary differential equations (ODEs). Our proposed model parameterizes all weights of attention and feed-forward blocks through neural networks, expressing these weights as functions of a continuous layer index. Through spectral analysis of the model's dynamics, we uncover an increase in eigenvalue magnitude that challenges the weight-sharing assumption prevalent in existing theoretical studies. We also leverage the Lyapunov exponent to examine token-level sensitivity, enhancing model interpretability. Our neural ODE transformer demonstrates performance comparable to or better than vanilla transformers across various configurations and datasets, while offering flexible fine-tuning capabilities that can adapt to different architectural constraints.

In the analysis of sequential data, the detection of abrupt changes is important in predicting future changes. In this paper, we propose statistical hypothesis tests for detecting covariance structure changes in locally smooth time series modeled by Gaussian Processes (GPs). We provide theoretically justified thresholds for the tests, and use them to improve Bayesian Online Change Point Detection (BOCPD) by confirming statistically significant changes and non-changes. Our Confirmatory BOCPD (CBOCPD) algorithm finds multiple structural breaks in GPs even when hyperparameters are not tuned precisely. We also provide conditions under which CBOCPD provides the lower prediction error compared to BOCPD. Experimental results on synthetic and real-world datasets show that our new tests correctly detect changes in the covariance structure in GPs. The proposed algorithm also outperforms existing methods for the prediction of nonstationarity in terms of both regression error and log likelihood.

Choosing a proper set of kernel functions is an important problem in learning Gaussian Process (GP) models since each kernel structure has different model complexity and data fitness. Recently, automatic kernel composition methods provide not only accurate prediction but also attractive interpretability through search-based methods. However, existing methods suffer from slow kernel composition learning. To tackle large-scaled data, we propose a new sparse approximate posterior for GPs, MultiSVGP, constructed from groups of inducing points associated with individual additive kernels in compositional kernels. We demonstrate that this approximation provides a better fit to learn compositional kernels given empirical observations. We also provide theoretically justification on error bound when compared to the traditional sparse GP. In contrast to the search-based approach, we present a novel probabilistic algorithm to learn a kernel composition by handling the sparsity in the kernel selection with Horseshoe prior. We demonstrate that our model can capture characteristics of time series with significant reductions in computational time and have competitive regression performance on real-world data sets.

Although meteorological conditions play a significant role in air pollution, research on their effects on the relationship between air pollutants is limited. In this study, trends of six criteria air pollutants were investigated from 15 air quality monitoring stations (AQMSs) in Ulsan, a multi-industrial city in South Korea, during 2015–2019. Unlike CO and O 3 , SO 2 , NO 2 , PM 10 , and PM 2.5 showed statistically significant decreasing trends over the period. The companion relationship between PM 2.5 and gaseous pollutants was evaluated by their correlations [R (PM 2.5 –GPs)]. R (PM 2.5 –NO 2 ) was relatively high at almost all AQMSs, whereas high R (PM 2.5 –SO 2 ) was observed near the petrochemical industrial complex, suggesting a great influence of local emissions (vehicles and industries). R (PM 2.5 –CO) and the standardized regression coefficients of CO obtained from the multiple linear regression model were the highest …

Probabilistic programming languages represent complex data with intermingled models in a few lines of code. Efficient inference algorithms in probabilistic programming languages make possible to build unified frameworks to compute interesting probabilities of various large, real-world problems. When the structure of model is given, constructing a probabilistic program is rather straightforward. Thus, main focus have been to learn the best model parameters and compute marginal probabilities. In this paper, we provide a new perspective to build expressive probabilistic program from continue time series data when the structure of model is not given. The intuition behind of our method is to find a descriptive covariance structure of time series data in nonparametric Gaussian process regression. We report that such descriptive covariance structure efficiently derives a probabilistic programming description accurately.

Unsupervised domain adaptation (UDA) refers to a domain adaptation framework in which a learning model is trained based on the labeled samples on the source domain and unlabelled ones in the target domain. The dominant existing methods in the field that rely on the classical covariate shift assumption to learn domain-invariant feature representation have yielded suboptimal performance under label distribution shift. In this paper, we propose a novel Conditional Adversarial SUpport ALignment (CASUAL) whose aim is to minimize the conditional symmetric support divergence between the source’s and target domain’s feature representation distributions, aiming at a more discriminative representation for the classification task. We also introduce a novel theoretical target risk bound, which justifies the merits of aligning the supports of conditional feature distributions compared to the existing marginal support alignment approach in the UDA settings. We then provide a complete training process for learning in which the objective optimization functions are precisely based on the proposed target risk bound. Our empirical results demonstrate that CASUAL outperforms other state-of-the-art methods on different UDA benchmark tasks under different label shift conditions.

Recent advances in Deep Gaussian Processes (DGPs) show the potential to have more expressive representation than that of traditional Gaussian Processes (GPs). However, there exists a pathology of deep Gaussian processes that their learning capacities reduce significantly when the number of layers increases. In this paper, we present a new analysis in DGPs by studying its corresponding nonlinear dynamic systems to explain the issue. Existing work reports the pathology for the squared exponential kernel function. We extend our investigation to four types of common stationary kernel functions. The recurrence relations between layers are analytically derived, providing a tighter bound and the rate of convergence of the dynamic systems. We demonstrate our finding with a number of experimental results.

We consider the long-tailed classification problem in which a few classes in the training data dominate the majority of the other classes. For concreteness, we focus on the visual domain in this paper. Most current methods employ contrastive learning to learn a representation for long-tailed data. In this paper, first, we investigate -positive sampling, a popular baseline method widely used to build contrastive learning models for imbalanced data. Previous works show that -positive learning, which only chooses positive samples (instead of all positive images) for each query image, suffers from inferior performance in long-tailed data. In this work, we further point out that k-positive learning limits the learning capability of both head and tail classes. Based on this perspective, we propose a novel contrastive learning framework that improves the limitation in k-positive learning by enlarging its positive selection space, so it can help the model learn more semantic discrimination features. Second, we analyze how the temperature (the hyperparameter used for tuning a concentration of samples on feature space) affects the gradients of each class in long-tailed learning, and propose a new method that can mitigate inadequate gradients between classes, which can help model learning easier. We name this framework as CoGloAT. Finally, we go on to introduce a new prototype learning framework namely ProCo based on coreset selection, which creates a global prototype for each cluster while keeping the computation cost within a reasonable time and show that combining CoGloAT with ProCo can further enhance the model learning ability on long-tailed …

Unsupervised domain adaptation (UDA) refers to a domain adaptation framework in which a learning model is trained based on the labeled samples on the source domain and unlabelled ones in the target domain. The dominant existing methods in the field that rely on the classical covariate shift assumption to learn domain-invariant feature representation have yielded suboptimal performance under the label distribution shift between source and target domains. In this paper, we propose a novel conditional adversarial support alignment (CASA) whose aim is to minimize the conditional symmetric support divergence between the source's and target domain's feature representation distributions, aiming at a more helpful representation for the classification task. We also introduce a novel theoretical target risk bound, which justifies the merits of aligning the supports of conditional feature distributions compared to the existing marginal support alignment approach in the UDA settings. We then provide a complete training process for learning in which the objective optimization functions are precisely based on the proposed target risk bound. Our empirical results demonstrate that CASA outperforms other state-of-the-art methods on different UDA benchmark tasks under label shift conditions.

This paper presents a comprehensive cross-task analysis of time series methodologies, revealing fundamental connections that are often obscured by task-specific perspectives. Our contributions are fivefold. First, we introduce seven priority properties, along with exogenous integration, that characterize methodologies independent of application domain, enabling systematic comparison across traditional and modern approaches. Second, we classify neural architectures by transparency levels determined by two characteristics: parameter time-invariance and the explicitness of mathematical formulations. Locally time-invariant operations enable mechanistic understanding, but globally time-varying operations pose fundamental challenges to achieving it. Third, our hierarchical taxonomy guides the selection of methodologies. Fourth, we comparatively evaluate explanation methods by quantifying how closely they …

This thesis studies interesting problems in compositionality for machine learning models under some settings including relational learning, scalability and deep models. Compositionality is the terminology describing the process of building small objects to complex ones. Bringing this concept into machine learning is important because it appears in many aspects from infinitesimal atomic to planetary structures. In this thesis, machine learning models center around Gaussian process of which covariance function is compositionally constructed. The proposed approach builds methods that can explore compositional model space automatically and efficiently as well as strives to address the interpretability for obtained models. The aforementioned problems are both important and challenging. Considering multivariate or relational learning is de facto in time series analysis for many domains. However, the existing methods of compositional learning are inapplicable to extend to such a setting since the explosion in model space makes it infeasible to use. Learning compositional structures is already a time-consuming task. Although there are existing approximation methods, they do not work well for compositional covariances. This makes it even harder to propose a scalable approach without sacrificing model performances. Finally, analyzing hierarchical deep Gaussian processes is notoriously difficult especially when incorporating different covariance functions. Previous work focuses on a single case of covariance function and is difficult to generalize for many other cases. The goal of this thesis is to propose solutions to the given problems. The first …

Stochastic Dynamics and Machine Learning Lab

교수님 소개

안통

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

SigFormer: Signature Transformers for Deep Hedging

Global Contrastive Learning for Long-Tailed Classification

Roles of ambient temperature and relative humidity on the relationship between fine particulate matter and gaseous pollutants in the largest industrial city of Ulsan, South Korea

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

안통

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

SigFormer: Signature Transformers for Deep Hedging

Global Contrastive Learning for Long-Tailed Classification

Roles of ambient temperature and relative humidity on the relationship between fine particulate matter and gaseous pollutants in the largest industrial city of Ulsan, South Korea

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집