The Brain-Machine Cognition Lab investigates the computational dynamics underlying high-level cognition in humans and machines. Research focuses on the computational principles and neural mechanisms that facilitate effortless environmental interaction. The lab aims to translate these insights into advanced machine models, envisioning a future of seamless human-machine collaboration and technology-augmented human capabilities.

뇌-기계 인지 연구실은 인간과 기계의 고차원적 인지 능력의 계산적 역학을 연구한다. 환경과의 용이한 상호작용을 가능하게 하는 계산 원리와 신경 메커니즘에 초점을 맞춘다. 궁극적으로 인간-기계 협업과 기술 증강 인간 역량을 위한 미래를 구상하며, 이러한 통찰력을 바탕으로 발전된 기계 모델을 개발하는 것을 목표로 한다.

Category-specific perceptual learning of robust object recognition modelled using deep neural networks

Bractive: A brain activation approach to human visual brain learning

Configural processing as an optimized strategy for robust object recognition in neural networks

Improved modeling of human vision by incorporating robustness to blur in convolutional neural networks

Robustness to Transformations Across Categories: Is Robustness Driven by Invariant Neural Representations?

Noise-trained deep neural networks effectively predict human vision and its neural responses to challenging images

Convolutional neural networks trained with a developmental sequence of blurry to clear images reveal core differences between face and object processing

Noise-robust neural networks and methods thereof

Test–retest reliability of spatial patterns from resting-state functional MRI using the restricted Boltzmann machine and hierarchically organized spatial patterns from the deep belief network

Developing predictive imaging biomarkers using whole-brain classifiers: Application to the ABIDE I dataset

Task-specific feature extraction and classification of fMRI volumes using a deep neural network initialized with a deep belief network: Evaluation using sensorimotor tasks

Developing predictive imaging biomarkers using whole-brain classifiers: application to the ABIDE I dataset

Test–retest reliability of spatial patterns from resting-state functional MRI using the restricted Boltzmann machine and hierarchically organized spatial patterns from the deep …

Robustness to transformations across categories: is robustness driven by invariant neural representations?

​ Can deep learning networks acquire the robustness of human recognition when faced with objects in visual noise?

Comparing humans and deep neural networks on face recognition under various distance and rotation viewing conditions

Evaluating the robustness of object recognition to visual noise in humans and convolutional neural networks

Comparing Human Face and Body Recognition at Various Distance and Rotation Viewing Conditions.

Comparing Human eye-tracking heatmaps with DNN saliency maps for faces at different spatial frequencies

Biomimetic-inspired resilient learning: Impact of progressive chromatic variations on the face recognition performance

Lack of experience with blurry visual input may cause cnns to deviate from biological visual systems

Convolutional neural networks optimized for face recognition reveal a computational basis for holistic face processing

Do deep neural networks perceive contextual visual illusions?

Hojin Jang

Feedforward deep neural networks (DNNs), artificial neural networks with multiple hidden layers, have recently demonstrated a record-breaking performance in multiple areas of applications in computer vision and speech processing. Following the success, DNNs have been applied to neuroimaging modalities including functional/structural magnetic resonance imaging (MRI) and positron-emission tomography data. However, no study has explicitly applied DNNs to 3D whole-brain fMRI volumes and thereby extracted hidden volumetric representations of fMRI that are discriminative for a task performed as the fMRI volume was acquired. Our study applied fully connected feedforward DNN to fMRI volumes collected in four sensorimotor tasks (i.e., left-hand clenching, right-hand clenching, auditory attention, and visual stimulus) undertaken by 12 healthy participants. Using a leave-one-subject-out cross-validation …

Deep neural networks (DNNs) for object classification have been argued to provide the most promising model of the visual system, accompanied by claims that they have attained or even surpassed human-level performance. Here, we evaluated whether DNNs provide a viable model of human vision when tested with challenging noisy images of objects, sometimes presented at the very limits of visibility. We show that popular state-of-the-art DNNs perform in a qualitatively different manner than humans—they are unusually susceptible to spatially uncorrelated white noise and less impaired by spatially correlated noise. We implemented a noise training procedure to determine whether noise-trained DNNs exhibit more robust responses that better match human behavioral and neural performance. We found that noise-trained DNNs provide a better qualitative match to human performance; moreover, they reliably predict human recognition thresholds on an image-by-image basis. Functional neuroimaging revealed that noise-trained DNNs provide a better correspondence to the pattern-specific neural representations found in both early visual areas and high-level object areas. A layer-specific analysis of the DNNs indicated that noise training led to broad-ranging modifications throughout the network, with greater benefits of noise robustness accruing in progressively higher layers. Our findings demonstrate that noise-trained DNNs provide a viable model to account for human behavioral and neural responses to objects in challenging noisy viewing conditions. Further, they suggest that robustness to noise may be acquired through a process of visual …

Whenever a visual scene is cast onto the retina, much of it will appear degraded due to poor resolution in the periphery; moreover, optical defocus can cause blur in central vision. However, the pervasiveness of blurry or degraded input is typically overlooked in the training of convolutional neural networks (CNNs). We hypothesized that the absence of blurry training inputs may cause CNNs to rely excessively on high spatial frequency information for object recognition, thereby causing systematic deviations from biological vision. We evaluated this hypothesis by comparing standard CNNs with CNNs trained on a combination of clear and blurry images. We show that blur-trained CNNs outperform standard CNNs at predicting neural responses to objects across a variety of viewing conditions. Moreover, blur-trained CNNs acquire increased sensitivity to shape information and greater robustness to multiple forms of …

Although convolutional neural networks (CNNs) provide a promising model for understanding human vision, most CNNs lack robustness to challenging viewing conditions, such as image blur, whereas human vision is much more reliable. Might robustness to blur be attributable to vision during infancy, given that acuity is initially poor but improves considerably over the first several months of life? Here, we evaluated the potential consequences of such early experiences by training CNN models on face and object recognition tasks while gradually reducing the amount of blur applied to the training images. For CNNs trained on blurry to clear faces, we observed sustained robustness to blur, consistent with a recent report by Vogelsang and colleagues (2018). By contrast, CNNs trained with blurry to clear objects failed to retain robustness to blur. Further analyses revealed that the spatial frequency tuning of the two CNNs was profoundly different. The blurry to clear face-trained network successfully retained a preference for low spatial frequencies, whereas the blurry to clear object-trained CNN exhibited a progressive shift toward higher spatial frequencies. Our findings provide novel computational evidence showing how face recognition, unlike object recognition, allows for more holistic processing. Moreover, our results suggest that blurry vision during infancy is insufficient to account for the robustness of adult vision to blurry objects.

We designed a modular machine learning program that uses functional magnetic resonance imaging (fMRI) data in order to distinguish individuals with autism spectrum disorders from neurodevelopmentally normal individuals. Data was selected from the Autism Brain Imaging Dataset Exchange (ABIDE) I Preprocessed Dataset.

DiCarlo, JJ, Zoccolan, D., and Rust, NC,“How does the brain solve visual object recognition?", Neuron 73, 415-434 (2012). Dodge, S., and Karam, L.,“Understanding how image quality affects deep neural networks”, In Quality of Multimedia Experience (QOMEX), 2016 Eighth International Conference on.(IEEE), pp. 1-6 (2016). Dodge, Samuel, and Lina Karam.“A study and comparison of human and deep learning recognition performance under visual distortions”, 2017 26th international conference on computer com munication and networks (ICCCN). IEEE, 2017. Field, DJ,“Relations between the statistics of natural images and the response properties of cortical cells”, J Opt Soc Am A 4,

Background Restricted Boltzmann machines (RBMs), including greedy layer-wise trained RBMs as part of a deep belief network (DBN), have the ability to identify spatial patterns (SPs; functional networks) in resting-state fMRI (rfMRI) data. However, there has been little research on (1) the reproducibility and test-retest reliability of SPs derived from RBMs and on (2) hierarchical SPs derived from DBNs. Methods We applied a weight sparsity-controlled RBM and DBN to whole-brain rfMRI data from the Human Connectome Project. We evaluated the within-session reproducibility and between-session test-retest reliability of the SPs derived from the RBM approach and compared them both with those identified using independent component analysis (ICA) and with three voxel-wise statistical measures—the Hurst exponent, entropy, and kurtosis—of the rfMRI data. We also assessed the potential hierarchy of the SPs from …

Deep convolutional neural networks (DCNNs) have demonstrated impressive robustness to recognize objects under transformations (e.g., blur or noise) when these transformations are included in the training set. A hypothesis to explain such robustness is that DCNNs develop invariant neural representations that remain unaltered when the image is transformed. However, to what extent this hypothesis holds true is an outstanding question, as robustness to transformations could be achieved with properties different from invariance; for example, parts of the network could be specialized to recognize either transformed or nontransformed images. This article investigates the conditions under which invariant neural representations emerge by leveraging that they facilitate robustness to transformations beyond the training distribution. Concretely, we analyze a training paradigm in which only some object categories …

The human brain is a highly efficient processing unit, and understanding how it works can inspire new algorithms and architectures in machine learning. In this work, we introduce a novel framework named Brain Activation Network (BRACTIVE), a transformer-based approach to studying the human visual brain. The primary objective of BRACTIVE is to align the visual features of subjects with their corresponding brain representations using functional Magnetic Resonance Imaging (fMRI) signals. It enables us to identify the brain’s Regions of Interest (ROIs) in the subjects. Unlike previous brain research methods, which can only identify ROIs for one subject at a time and are limited by the number of subjects, BRACTIVE automatically extends this identification to multiple subjects and ROIs. Our experiments demonstrate that BRACTIVE effectively identifies person-specific regions of interest, such as face and body …

Convolutional neural networks (CNNs) have attracted considerable attention for their remarkable performance at a variety of cognitive tasks, including visual object recognition. This has led to the proposal that deep learning networks may provide a biologically plausible model of human visual processing (eg, Yamins et al., 2014; Khaligh-Razavi & Kriegeskorte, 2014). Here, we investigated whether these networks are robust to noisy viewing conditions. If not, are CNNs disrupted by visual noise in a manner that resembles human performance? To this end, we systematically compared the performance of humans and machines across a range of signal-to-noise ratios by presenting object images in varying levels of Gaussian pixel noise or Fourier phase-scrambled noise. Human performance proved far more robust to noise than state-of-the-art CNNs (AlexNet, VGG, and GoogLeNet). Moreover, CNNs were more …

Humans possess impressive skills for recognizing faces even when the viewing conditions are challenging, such as long ranges, non-frontal regard, variable lighting, and atmospheric turbulence. We sought to characterize the effects of such viewing conditions on the face recognition performance of humans, and compared the results to those of DNNs. In an online verification task study, we used a 100 identity face database, with images captured at five different distances (2m, 5m, 300m, 650m and 1000m) three pitch values (00-straight ahead,+/-30 degrees) and three levels of yaw (00, 45, and 90 degrees). Participants were presented with 175 trials (5 distances x 7 yaw and pitch combinations, with 5 repetitions). Each trial included a query image, from a certain combination of range x yaw x pitch, and five options, all frontal short range (2m) faces. One was of the same identity as the query, and the rest were the most …

Convolutional neural networks (CNNs) have led to a major advance in machine-based object classification performance and are sometimes described as approaching near-human levels of recognition accuracy. CNNs are also of great interest because they provide a biologically plausible model of object recognition; however, to what extent does their performance resemble that of human observers? The goal of this study was to evaluate the similarity and robustness of human and CNN performance, by presenting objects under challenging viewing conditions with varying levels of visual white noise. For the CNN, we relied on the AlexNet model (Krizhevsky et al., 2012) and images obtained from the 2012 ImageNet Large Scale Visual Recognition Competition. We pre-selected 16 out of the 1000 object categories (ie, 8 animate and 8 inanimate categories) to compare performance across man and machine …

Deep networks trained on large face datasets have achieved impressive performance on recognition tasks. However, as we reported in last year’s meeting (Fux et al., 2023), humans still outperform DNNs when viewing conditions are challenging as with large distances, non-frontal regard, and atmospheric turbulence. In the current study, we investigate recognition performance of humans and deep networks with images of whole bodies. This task is akin to the ‘person re-identification’challenge of great interest to the machine vision community. We worked with a large database of images acquired at a variety of distances and from multiple yaw/pitch angles. We ran an online behavioral study in which participants were asked to rank whole-body images of people from most to least likely to be the same identity as the person in the query image. The query image depicts individuals in three conditions: Whole body, head …

Configural processing, the perception of spatial relationships among an object’s components, is crucial for object recognition, yet its teleology and underlying mechanisms remain unclear. We hypothesize that configural processing drives robust recognition under varying conditions. Using identification tasks with composite letter stimuli, we compare neural network models trained with either configural or local cues. We find that configural cues support robust generalization across geometric transformations (e.g., rotation, scaling) and novel feature sets. When both cues are available, configural cues dominate local features. Layerwise analysis reveals that sensitivity to configural cues emerges later in processing, likely enhancing robustness to pixel-level transformations. Notably, this occurs in a purely feedforward manner without recurrent computations. These findings with letter stimuli successfully extend to …

Deep neural network (DNN)-based Face recognition (FR) models have improved greatly over the past decades achieving, or even exceeding, human-level accuracies under certain viewing conditions, such as frontal face views. However, as we reported in last year’s meeting (XXX et al., 2023), under challenging viewing conditions (eg large distances, non-frontal regard) humans outperform DNNs. To shed light on potential explanations for these differences in FR accuracies of humans and DNNs, we turned to eye-tracking paradigms to discern potentially important zones of information uptake for observers, and compare them with DNN-derived saliency maps. Despite the conceptual similarity between human eye tracking-based heat-maps and DNN saliency maps, the literature is sparse in terms of strategic efforts to quantitatively compare the two and translate human gaze and attention strategies to improve …

As is evident from the ease with which we can recognize grayscale images, humans are remarkably resilient to changes in chromatic content. Based on results from children with late-onset sight, we have hypothesized that this resilience may be based, in part, on the developmental progression of the color system, with initially poor chromaticity eventually maturing into rich color experiences (Vogelsang et. al., under review). Computational tests of this hypothesis have involved the creation of training regimens wherein the first half of training data is devoid of color, and the second half has full color information. The results so far indicate that such a training progression yields greater resilience to color shifts than one where all training data are in rich color. While encouraging, these investigations are still quite different from true developmental progressions which are characterized by gradual changes in the amount of …

Our subjective impression of the visual world is that it appears clear when in fact large portions of the retinal image will often consist of degraded input due to optical defocus and low resolution in the periphery. Convolutional neural networks (CNNs) are believed to provide the best current model of biological vision, yet the typical training regime for CNNs predominantly consists of clear images. We hypothesized that a lack of blurry input may cause CNNs to acquire representations that rely excessively on the high-spatial frequency content of visual objects (Jang & Tong, Journal of Vision, 2021), causing deviations from biological visual systems. We sought to test this idea by comparing two types of CNNs, those trained with both blurry and clear images and those trained with clear images only. Multiple data sets were employed to compare CNN performance, including human fMRI data (Xu & Vaziri-Pashkam, 2021 …

Face recognition plays a crucial role in our daily social lives. Considerable research has suggested that faces constitute a special category of stimuli that can engage more holistic processing than other object categories. However, there is still a lack of clarity as to why holistic encoding strategies may be preferred in the case of face recognition. Does holistic processing arise from the more extensive training that people acquire for faces as compared to other objects, or might it also depend on the inherent visual structure of faces? To explore these questions, we compared convolutional neural networks (CNNs) that were trained to discriminate faces or subordinate-level objects (ie, dogs or birds) on image manipulations known to impair holistic/configural processing. Specifically, we evaluated how CNN performance was affected by misalignment of the top-and bottom-halves of the image, random scrambling of local …

Object recognition in real-world environments requires dealing with considerable ambiguity, yet the human visual system is highly robust to noisy viewing conditions. Here, we investigated the role of perceptual learning in the acquisition of robustness in both humans and deep neural networks (DNNs). Specifically, we sought to determine whether perceptual training with object images in Gaussian noise, drawn from certain animate or inanimate categories, would lead to category-specific or category-general improvements in human robustness. Moreover, might DNNs provide viable models of human perceptual learning? Both before and after training, we evaluated the noise threshold required for accurate recognition using novel object images. Human observers were quite robust to noise before training, but showed additional category-specific improvement after training with only a few hundred noisy object examples. In comparison, standard DNNs initially lacked robustness, then showed both category-general and category-specific learning after training with the same noisy examples. We further evaluated DNN models that were pre-trained with moderately noisy images to match human pre-training accuracy. Notably, these models only showed category-specific improvement, matching the overall pattern of learning exhibited by human observers. A layer-wise analysis of DNN responses revealed that category-general learning effects emerged in the lower layers, whereas category-specific improvements emerged in the higher layers. Our findings provide support for the notion that robustness to noisy visual conditions arises through learning …

Through a combination of innate circuit mechanisms and visual experience, human visual perception comes to incorporate sensitivity to contextual cues, which provide important information for interpreting the environment. While these contextual influences enable accurate perception in complex and dynamic settings, they can also give rise to systematic biases under certain conditions, as observed in phenomena like visual illusions. Classic illusions such as the Delboeuf, Ebbinghaus, Ponzo, and Müller-Lyer vividly demonstrate how context alters the perception of relative size, often leading to significant misjudgments. Modern deep neural networks (DNNs), which have demonstrated remarkable success in emulating human perceptual behaviors, raise the following question: Can artificial vision systems, that incorporate some of the architectural properties of their biological counterparts, and are trained on natural …

Brain and Machine Cognition Lab

교수님 소개

장호진

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Category-specific perceptual learning of robust object recognition modelled using deep neural networks

Bractive: A brain activation approach to human visual brain learning

Configural processing as an optimized strategy for robust object recognition in neural networks

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

장호진

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Category-specific perceptual learning of robust object recognition modelled using deep neural networks

Bractive: A brain activation approach to human visual brain learning

Configural processing as an optimized strategy for robust object recognition in neural networks

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집