The KAIST Visual AI Group, led by Minhyuk Sung, focuses on generating, manipulating, and analyzing various visual data, including images, videos, and 3D data.

성민혁 교수는 KAIST 전산학부 부교수이다. KAIST 부임 전 Adobe Research에서 연구원으로 1년동안 근무하였다. 2019년 Stanford University에서 박사학위를 취득하였으며, 2008년, 2010년에 KAIST에서 학사 및 석사학위를 취득하였다. 2010년부터 2013년까지 KIST에서 연구원으로 근무하였다. 주 연구분야는 3차원 형상 데이터 처리를 위한 기계학습이다.

Proxy-Free Gaussian Splats Deformation with Splat-Based Surface Estimation

BoxSplitGen: A Generative Model for 3D Part Bounding Boxes in Varying Granularity

Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models

Ψ-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models

Inference-Time Scaling for Flow Models via Stochastic Generation and Rollover Budget Forcing

ORIGEN: Zero-Shot 3D Orientation Grounding in Text-to-Image Generation

Moment- and Power-Spectrum-Based Gaussianity Regularization for Text-to-Image Models

Neural Green’s Functions

Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulation

MemBench: Memorized Image Trigger Prompt Dataset for Diffusion Models

VideoHandles: Editing 3D Object Compositions in Videos Using Video Generative Priors

REWIND: Real-Time Egocentric Whole-Body Motion Diffusion with Exemplar-Based Identity Conditioning

StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spaces

SyncTweedies: A General Generative Framework Based on Synchronized Diffusions

Neural Pose Representation Learning for Generating and Transferring Non-Rigid Object Poses

GrounDiT: Grounding Diffusion Transformers via Noisy Patch Transplantation

MV2Cyl: Reconstructing 3D Extrusion Cylinders from Multi-View Images

Occupancy-Based Dual Contouring

ReGround: Improving Textual and Spatial Grounding at No Cost

PartSTAD: 2D-to-3D Part Segmentation Task Adaptation

Posterior Distillation Sampling

As-Plausible-As-Possible: Semantic-Aware Shape Deformation using 2D Diffusion Priors

InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion

Split, Merge, and Refine: Fitting Tight Bounding Boxes via Learned Over-Segmentation and Iterative Search

SyncDiffusion: Coherent Montage via Synchronized Joint Diffusions

FourierHandFlow: Neural 4D Hand Representation Using Fourier Query Flow

OptCtrlPoints: Optimizing Control Points for Biharmonic 3D Shape Deformation

SALAD: Part-Level Latent Diffusion for 3D Shape Generation and Manipulation

Im2Hands: Learning Attentive Implicit Representation of Interacting Two-Hand Shapes

ShapeTalk: A Language Dataset and Framework for 3D Shape Edits and Deformations

Seg&Struct: The Interplay Between Part Segmentation and Structure Inference for 3D Shape Parsing

LADIS: Language Disentanglement for 3D Shape Editing

The Shape Part Slot Machine: Contact-based Reasoning for Generating 3D Shapes from Parts

PartGlot: Learning Shape Part Segmentation from Language Reference Games

Pop-Out Motion: 3D-Aware Image Deformation via Learning the Shape Laplacian

Point2Cyl: Reverse Engineering 3D Objects from Point Clouds to Extrusion Cylinders

Implicit LiDAR Network: LiDAR Super-Resolution via Interpolation Weight Prediction

CPFN: Cascaded Primitive Fitting Networks for High-Resolution Point Clouds

CTRL-C: Camera calibration TRansformer with Line-Classification

DeepMetaHandles: Learning Deformation Meta-Handles of 3D Meshes with Biharmonic Coordinates

MultiBodySync: Multi-Body Segmentation and Motion Estimation via 3D Scan Synchronization

Joint Learning of 3D Shape Retrieval and Deformation

DeformSyncNet: Deformation Transfer via Synchronized Shape Deformation Spaces

Deformation-Aware 3D Shape Embedding and Retrieval

Neural Geometric Parser for Single Image Camera Calibration

Pix2Surf: Learning Parametric 3D Surface Models of Objects from Images

Learning 3D Part Assembly from a Single Image

Learning and Exploring the Compositional Structure of 3D Data

Supervised Fitting of Geometric Primitives to 3D Point Clouds

GSPN: Generative Shape Proposal Network for 3D Instance Segmentation in Point Cloud

Deep Functional Dictionaries: Learning Consistent Semantic Structures on 3D Models from Functions

Learning Fuzzy Set Representations of Partial Shapes on Dual Embedding Spaces

ComplementMe: Weakly-Supervised Component Suggestions for 3D Modeling

Data-Driven Structural Priors for Shape Completion

Image Unprojection for 3D Surface Reconstruction: A Triangulation-based Approach

Finding the M-best consistent correspondences between 3D symmetric objects

A Triangulation-Invariant Method for Anisotropic Geodesic Map Computation on Surface Meshes

A Spectral Approach to Shape Matching Using a Heat Kernel Function

GSPN: Generative shape proposal network for 3d instance segmentation in point cloud

Supervised fitting of geometric primitives to 3d point clouds

Data-driven structural priors for shape completion

ComplementMe: weakly-supervised component suggestions for 3D modeling

Point2cyl: Reverse engineering 3d objects from point clouds to extrusion cylinders

Learning 3d part assembly from a single image

Syncdiffusion: Coherent montage via synchronized joint diffusions

Multibodysync: Multi-body segmentation and motion estimation via 3d scan synchronization

Salad: Part-level latent diffusion for 3d shape generation and manipulation

Ctrl-c: Camera calibration transformer with line-classification

Joint learning of 3d shape retrieval and deformation

Deformation-aware 3d model embedding and retrieval

Cpfn: Cascaded primitive fitting networks for high-resolution point clouds

Posterior distillation sampling

Deepmetahandles: Learning deformation meta-handles of 3d meshes with biharmonic coordinates

Pix2surf: Learning parametric 3d surface models of objects from images

Deep functional dictionaries: Learning consistent semantic structures on 3d models from functions

Implicit LiDAR network: LiDAR super-resolution via interpolation weight prediction

Leonidas Guibas

Vladimir G. Kim

Mikaela Angelina Uy

Juil Koo

Li Yi

Niloy J. Mitra

Tolga Birdal

Jihyun Lee

Phillip Y. Lee

Seungwoo Yoo

Anastasia Dubrovina

He Wang

Panos Achlioptas

Hao Su

Siddhartha Chaudhuri

Paul Guerrero

Radomir Mech

Ian Huang

Kunho Kim

Junho Kim

Minhyuk Sung

We introduce a novel 3D object proposal approach named Generative Shape Proposal Network (GSPN) for instance segmentation in point cloud data. Instead of treating object proposal as a direct bounding box regression problem, we take an analysis-by-synthesis strategy and generate proposals by reconstructing shapes from noisy observations in a scene. We incorporate GSPN into a novel 3D instance segmentation framework named Region-based PointNet (R-PointNet) which allows flexible proposal refinement and instance segmentation generation. We achieve state-of-the-art performance on several 3D instance segmentation tasks. The success of GSPN largely comes from its emphasis on geometric understandings during object proposal, greatly reducing proposals with low objectness.

Fitting geometric primitives to 3D point cloud data bridges a gap between low-level digitized 3D data and high-level structural information on the underlying 3D shapes. As such, it enables many downstream applications in 3D data processing. For a long time, RANSAC-based methods have been the gold standard for such primitive fitting problems, but they require careful per-input parameter tuning and thus do not scale well for large datasets with diverse shapes. In this work, we introduce Supervised Primitive Fitting Network (SPFN), an end-to-end neural network that can robustly detect a varying number of primitives at different scales without any user control. The network is supervised using ground truth primitive surfaces and primitive membership for the input points. Instead of directly predicting the primitives, our architecture first predicts per-point properties and then uses a differential model estimation module to compute the primitive type and parameters. We evaluate our approach on a novel benchmark of ANSI 3D mechanical component models and demonstrate a significant improvement over both the state-of-the-art RANSAC-based methods and the direct neural prediction.

Acquiring 3D geometry of an object is a tedious and time-consuming task, typically requiring scanning the surface from multiple viewpoints. In this work we focus on reconstructing complete geometry from a single scan acquired with a low-quality consumer-level scanning device. Our method uses a collection of example 3D shapes to build structural part-based priors that are necessary to complete the shape. In our representation, we associate a local coordinate system to each part and learn the distribution of positions and orientations of all the other parts from the database, which implicitly also defines positions of symmetry planes and symmetry axes. At the inference stage, this knowledge enables us to analyze incomplete point clouds with substantial occlusions, because observing only a few regions is still sufficient to infer the global structure. Once the parts and the symmetries are estimated, both data sources …

Assembly-based tools provide a powerful modeling paradigm for non-expert shape designers. However, choosing a component from a large shape repository and aligning it to a partial assembly can become a daunting task. In this paper we describe novel neural network architectures for suggesting complementary components and their placement for an incomplete 3D part assembly. Unlike most existing techniques, our networks are trained on unlabeled data obtained from public online repositories, and do not rely on consistent part segmentations or labels. Absence of labels poses a challenge in indexing the database of parts for the retrieval. We address it by jointly training embedding and retrieval networks, where the first indexes parts by mapping them to a low-dimensional feature space, and the second maps partial assemblies to appropriate complements. The combinatorial nature of part arrangements …

We propose Point2Cyl, a supervised network transforming a raw 3D point cloud to a set of extrusion cylinders. Reverse engineering from a raw geometry to a CAD model is an essential task to enable manipulation of the 3D data in shape editing software and thus expand their usages in many downstream applications. Particularly, the form of CAD models having a sequence of extrusion cylinders---a 2D sketch plus an extrusion axis and range---and their boolean combinations is not only widely used in the CAD community/software but also has great expressivity of shapes, compared to having limited types of primitives (eg, planes, spheres, and cylinders). In this work, we introduce a neural network that solves the extrusion cylinder decomposition problem in a geometry-grounded way by first learning underlying geometric proxies. Precisely, our approach first predicts per-point segmentation, base/barrel labels and normals, then estimates for the underlying extrusion parameters in differentiable and closed-form formulations. Our experiments show that our approach demonstrates the best performance on two recent CAD datasets, Fusion Gallery and DeepCAD, and we further showcase our approach on reverse engineering and editing.

Autonomous assembly is a crucial capability for robots in many applications. For this task, several problems such as obstacle avoidance, motion planning, and actuator control have been extensively studied in robotics. However, when it comes to task specification, the space of possibilities remains underexplored. Towards this end, we introduce a novel problem, single-image-guided 3D part assembly , along with a learning-based solution. We study this problem in the setting of furniture assembly from a given complete set of parts and a single image depicting the entire assembled object. Multiple challenges exist in this setting, including handling ambiguity among parts ( e.g. , slats in a chair back and leg stretchers) and 3D pose prediction for parts and part subassemblies, whether visible or occluded. We address these issues by proposing a two-module pipeline that leverages strong 2D-3D correspondences and …

The remarkable capabilities of pretrained image diffusion models have been utilized not only for generating fixed-size images but also for creating panoramas. However, naive stitching of multiple images often results in visible seams. Recent techniques have attempted to address this issue by performing joint diffusions in multiple windows and averaging latent features in overlapping regions. However, these approaches, which focus on seamless montage generation, often yield incoherent outputs by blending different scenes within a single image. To overcome this limitation, we propose SyncDiffusion, a plug-and-play module that synchronizes multiple diffusions through gradient descent from a perceptual similarity loss. Specifically, we compute the gradient of the perceptual loss using the predicted denoised images at each denoising step, providing meaningful guidance for achieving coherent montages. Our experimental results demonstrate that our method produces significantly more coherent outputs compared to previous methods (66.35% vs. 33.65% in our user study) while still maintaining fidelity (as assessed by GIQA) and compatibility with the input prompt (as measured by CLIP score). We further demonstrate the versatility of our method across three plug-and-play applications: layout-guided image generation, conditional image generation and 360-degree panorama generation. Our project page is at https://syncdiffusion. github. io.

We present MultiBodySync, a novel, end-to-end trainable multi-body motion segmentation and rigid registration framework for multiple input 3D point clouds. The two non-trivial challenges posed by this multi-scan multibody setting that we investigate are:(i) guaranteeing correspondence and segmentation consistency across multiple input point clouds capturing different spatial arrangements of bodies or body parts; and (ii) obtaining robust motion-based rigid body segmentation applicable to novel object categories. We propose an approach to address these issues that incorporates spectral synchronization into an iterative deep declarative network, so as to simultaneously recover consistent correspondences as well as motion segmentation. At the same time, by explicitly disentangling the correspondence and motion segmentation estimation modules, we achieve strong generalizability across different object categories. Our extensive evaluations demonstrate that our method is effective on various datasets ranging from rigid parts in articulated objects to individually moving objects in a 3D scene, be it single-view or full point clouds.

We present a cascaded diffusion model based on a part-level implicit 3D representation. Our model achieves state-of-the-art generation quality and also enables part-level shape editing and manipulation without any additional training in conditional setup. Diffusion models have demonstrated impressive capabilities in data generation as well as zero-shot completion and editing via a guided reverse process. Recent research on 3D diffusion models has focused on improving their generation capabilities with various data representations, while the absence of structural information has limited their capability in completion and editing tasks. We thus propose our novel diffusion model using a part-level implicit representation. To effectively learn diffusion with high-dimensional embedding vectors of parts, we propose a cascaded framework, learning diffusion first on a low-dimensional subspace encoding extrinsic parameters of parts and then on the other high-dimensional subspace encoding intrinsic attributes. In the experiments, we demonstrate the outperformance of our method compared with the previous ones both in generation and part-level completion and manipulation tasks.

Single image camera calibration is the task of estimating the camera parameters from a single input image, such as the vanishing points, focal length, and horizon line. In this work, we propose Camera calibration TRansformer with Line-Classification (CTRL-C), an end-to-end neural network-based approach to single image camera calibration, which directly estimates the camera parameters from an image and a set of line segments. Our network adopts the transformer architecture to capture the global structure of an image with multi-modal inputs in an end-to-end manner. We also propose an auxiliary task of line classification to train the network to extract the global geometric information from lines effectively. Our experiments demonstrate that CTRL-C outperforms the previous state-of-the-art methods on the Google Street View and SUN360 benchmark datasets. Code is available at https://github. com/jwlee-vcl/CTRL-C.

We propose a novel technique for producing high-quality 3D models that match a given target object image or scan. Our method is based on retrieving an existing shape from a database of 3D models and then deforming its parts to match the target shape. Unlike previous approaches that independently focus on either shape retrieval or deformation, we propose a joint learning procedure that simultaneously trains the neural deformation module along with the embedding space used by the retrieval module. This enables our network to learn a deformation-aware embedding space, so that retrieved models are more amenable to match the target after an appropriate deformation. In fact, we use the embedding space to guide the shape pairs used to train the deformation module, so that it invests its capacity in learning deformations between meaningful shape pairs. Furthermore, our novel part-aware deformation module can work with inconsistent and diverse part-structures on the source shapes. We demonstrate the benefits of our joint training not only on our novel framework, but also on other state-of-the-art neural deformation modules proposed in recent years. Lastly, we also show that our jointly-trained method outperforms various non-joint baselines.

We introduce a new problem of retrieving 3D models that are deformable to a given query shape and present a novel deep deformation-aware embedding to solve this retrieval task. 3D model retrieval is a fundamental operation for recovering a clean and complete 3D model from a noisy and partial 3D scan. However, given a finite collection of 3D shapes, even the closest model to a query may not be satisfactory. This motivates us to apply 3D model deformation techniques to adapt the retrieved model so as to better fit the query. Yet, certain restrictions are enforced in most 3D deformation techniques to preserve important features of the original model that prevent a perfect fitting of the deformed model to the query. This gap between the deformed model and the query induces asymmetric relationships among the models, which cannot be handled by typical metric learning techniques. Thus, to retrieve the best …

Representing human-made objects as a collection of base primitives has a long history in computer vision and reverse engineering. In the case of high-resolution point cloud scans, the challenge is to be able to detect both large primitives as well as those explaining the detailed parts. While the classical RANSAC approach requires case-specific parameter tuning, state-of-the-art networks are limited by memory consumption of their backbone modules such as PointNet++, and hence fail to detect the fine-scale primitives. We present Cascaded Primitive Fitting Networks (CPFN) that relies on an adaptive patch sampling network to assemble detection results of global and local primitive detection networks. As a key enabler, we present a merging formulation that dynamically aggregates the primitives across global and local scales. Our evaluation demonstrates that CPFN improves the state-of-the-art SPFN performance by 13-14% on high-resolution point cloud datasets and specifically improves the detection of fine-scale primitives by 20-22%. Our code is available at: https://github. com/erictuanle/CPFN

We introduce Posterior Distillation Sampling (PDS) a novel optimization method for parametric image editing based on diffusion models. Existing optimization-based methods which leverage the powerful 2D prior of diffusion models to handle various parametric images have mainly focused on generation. Unlike generation editing requires a balance between conforming to the target attribute and preserving the identity of the source content. Recent 2D image editing methods have achieved this balance by leveraging the stochastic latent encoded in the generative process of diffusion models. To extend the editing capabilities of diffusion models shown in pixel space to parameter space we reformulate the 2D image editing method into an optimization form named PDS. PDS matches the stochastic latents of the source and the target enabling the sampling of targets in diverse parameter spaces that align with a desired attribute while maintaining the source's identity. We demonstrate that this optimization resembles running a generative process with the target attribute but aligning this process with the trajectory of the source's generative process. Extensive editing results in Neural Radiance Fields and Scalable Vector Graphics representations demonstrate that PDS is capable of sampling targets to fulfill the aforementioned balance across various parameter spaces.

We propose DeepMetaHandles, a 3D conditional generative model based on mesh deformation. Given a collection of 3D meshes of a category and their deformation handles (control points), our method learns a set of meta-handles for each shape, which are represented as combinations of the given handles. The disentangled meta-handles factorize all the plausible deformations of the shape, while each of them corresponds to an intuitive deformation. A new deformation can then be generated by sampling the coefficients of the meta-handles in a specific range. We employ biharmonic coordinates as the deformation function, which can smoothly propagate the control points' translations to the entire mesh. To avoid learning zero deformation as meta-handles, we incorporate a target-fitting module which deforms the input mesh to match a random target. To enhance deformations' plausibility, we employ a soft-rasterizer-based discriminator that projects the meshes to a 2D space. Our experiments demonstrate the superiority of the generated deformations as well as the interpretability and consistency of the learned meta-handles.

We investigate the problem of learning to generate 3D parametric surface representations for novel object instances, as seen from one or more views. Previous work on learning shape reconstruction from multiple views uses discrete representations such as point clouds or voxels, while continuous surface generation approaches lack multi-view consistency. We address these issues by designing neural networks capable of generating high-quality parametric 3D surfaces which are also consistent between views. Furthermore, the generated 3D surfaces preserve accurate image pixel to 3D surface point correspondences , allowing us to lift texture information to reconstruct shapes with rich geometry and appearance. Our method is supervised and trained on a public dataset of shapes from common object categories. Quantitative results indicate that our method significantly outperforms previous work, while qualitative …

Various 3D semantic attributes such as segmentation masks, geometric features, keypoints, and materials can be encoded as per-point probe functions on 3D geometries. Given a collection of related 3D shapes, we consider how to jointly analyze such probe functions over different shapes, and how to discover common latent structures using a neural network—even in the absence of any correspondence information. Our network is trained on point cloud representations of shape geometry and associated semantic functions on that point cloud. These functions express a shared semantic understanding of the shapes but are not coordinated in any way. For example, in a segmentation task, the functions can be indicator functions of arbitrary sets of shape parts, with the particular combination involved not known to the network. Our network is able to produce a small dictionary of basis functions for each shape, a dictionary whose span includes the semantic functions provided for that shape. Even though our shapes have independent discretizations and no functional correspondences are provided, the network is able to generate latent bases, in a consistent order, that reflect the shared semantic structure among the shapes. We demonstrate the effectiveness of our technique in various segmentation and keypoint selection applications.

Super-resolution of LiDAR range images is crucial to improving many downstream tasks such as object detection, recognition, and tracking. While deep learning has made a remarkable advances in super-resolution techniques, typical convolutional architectures limit upscaling factors to specific output resolutions in training. Recent work has shown that a continuous representation of an image and learning its implicit function enable almost limitless upscaling. However, the detailed approach, predicting values (depths) for neighbor pixels in the input and then linearly interpolating them, does not best fit the LiDAR range images since it does not fill the unmeasured details but creates a new image with regression in a high-dimensional space. In addition, the linear interpolation blurs sharp edges providing important boundary information of objects in 3-D points. To handle these problems, we propose a novel network …

Editing 3D geometry is a challenging task requiring specialized skills. In this work, we aim to facilitate the task of editing the geometry of 3D models through the use of natural language. For example, we may want to modify a 3D chair model to" make its legs thinner" or to" open a hole in its back". To tackle this problem in a manner that promotes open-ended language use and enables fine-grained shape edits, we introduce the most extensive existing corpus of natural language utterances describing shape differences: ShapeTalk. ShapeTalk contains over half a million discriminative utterances produced by contrasting the shapes of common 3D objects for a variety of object classes and degrees of similarity. We also introduce a generic framework, ChangeIt3D, which builds on ShapeTalk and can use an arbitrary 3D generative model of shapes to produce edits that align the output better with the edit or deformation description. Finally, we introduce metrics for the quantitative evaluation of language-assisted shape editing methods that reflect key desiderata within this editing setup. We note that ShapeTalk allows methods to be trained with explicit 3D-to-language data, bypassing the necessity of" lifting" 2D to 3D using methods like neural rendering, as required by extant 2D image-language foundation models. Our code and data are publicly available at https://changeit3d. github. io/.

We introduce PartGlot, a neural framework and associated architectures for learning semantic part segmentation of 3D shape geometry, based solely on part referential language. We exploit the fact that linguistic descriptions of a shape can provide priors on the shape's parts--as natural language has evolved to reflect human perception of the compositional structure of objects, essential to their recognition and use. For training we use the paired geometry/language data collected in the ShapeGlot work for their reference game, where a speaker creates an utterance to differentiate a target shape from two distractors and the listener has to find the target based on this utterance. Our network is designed to solve this target discrimination problem, carefully incorporating a Transformer-based attention module so that the output attention can precisely highlight the semantic part or parts described in the language. Furthermore, the network operates without any direct supervision on the 3D geometry itself. Surprisingly, we further demonstrate that the learned part information is generalizable to shape classes unseen during training. Our approach opens the possibility of learning 3D shape parts from language alone, without the need for large-scale part geometry annotations, thus facilitating annotation acquisition.

Professor

KAIST Visual AI Group

교수님 소개

Minhyuk Sung

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Proxy-Free Gaussian Splats Deformation with Splat-Based Surface Estimation

BoxSplitGen: A Generative Model for 3D Part Bounding Boxes in Varying Granularity

Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

Minhyuk Sung

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Proxy-Free Gaussian Splats Deformation with Splat-Based Surface Estimation

BoxSplitGen: A Generative Model for 3D Part Bounding Boxes in Varying Granularity

Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Models

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집