The Mobile and Edge Systems (MES) Laboratory, led by Prof. Sangeun Oh, focuses on identifying practical challenges in daily life and developing edge computing technologies to create innovative user experiences. Research is conducted in three key areas: multi-device/multi-app collaboration for mobile IoT, system-level optimization for on-device AI, and LLM-enhanced interaction for mobile IoT.

오상근 교수가 이끄는 모바일 및 엣지 시스템(MES) 연구실은 일상생활의 실질적인 문제점을 파악하고 혁신적인 사용자 경험을 창출하기 위한 엣지 컴퓨팅 기술 개발에 중점을 둔다. 주요 연구 분야는 모바일 IoT를 위한 멀티 디바이스/멀티 앱 협업, 온디바이스 AI를 위한 시스템 수준 최적화, LLM 기반의 모바일 IoT 상호작용이다.

MobileGPT: Augmenting LLM with Human-like Memory for Mobile Task Automation

SPET: Transparent SRAM Allocation and Model Partitioning for Real-time DNN Tasks on Edge TPU

A-Mash: Providing Single-App Illusion for Multi-App Use through User-centric UI Mashup

FLUID-XP: flexible user interface distribution for cross-platform experience

FLUID: Flexible User Interface Distribution for Ubiquitous Multi-device Interaction

Mobile Plus: Multi-device Mobile Platform for Cross-device Functionality Sharing

RT-SDN: Adaptive Routing and Priority Ordering for Software-Defined Real-Time Networking

SCOUT: Thermal-Aware SRAM Allocation for Real-Time DNN Tasks on Edge TPU

VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification

Mitigating Resource Contention for Responsive On-device Machine Learning Inferences

EarDVFS: Environment-Adaptable RL-based DVFS for Mobile Devices

TaPIN: Reinforcing PIN Authentication on Smartphones with Tap Biometrics

CROS-RT: Cross-Layer Priority Scheduling for Predictable Inter-Process Communication in ROS2

MagPie: Extending a Smartphone’s Interaction Space via a Customizable Magnetic Back-of-Device Input Accessory

MobileGPT: Augmenting LLM with Human-like App Memory for Mobile Task Automation

Extracting Payment Tokens Out of Sounds Produced by Magnetic Field Fluctuations

Supporting Flexible and Transparent User Interface Distribution across Mobile Devices

RT-Swap: Addressing GPU Memory Bottlenecks for Real-Time Multi-DNN Inference

LiquidListener: Supporting Ubiquitous Liquid Volume Sensing via Singing Sounds

App-level Multi-surface Framework for Supporting Cross-platform User Interface Distribution

Supporting Safe Priority Adjustment for Software-Defined Real-Time Networking

MagSnoop: Listening to Sounds Induced by Magnetic Field Fluctuations to Infer Mobile Payment Tokens

GreenBag: Energy-efficient Bandwidth Aggregation for Real-time Streaming in Heterogeneous Mobile Wireless Networks

FLUID: Flexible User Interface Distribution for Ubiquitous Multi-Device Interaction

FLUID: Multi-device Mobile Platform for Flexible User Interface Distribution

Mobile Platform Design for Sharing Functionalities between Multiple Devices

Mobile Plus: Mobile platform for Transparent Sharing of Functionalities Across Devices

보청기 착용 전후 청력 수준과 설문지 점수를 이용한 최적 실이삽입 이득 예 측을 위한 머신러닝 접근법

Edge TPU 에서의 실시간 DNN 작업을 위한 SRAM 할당 및 모델 분할 기법

멀티 디바이스 상호작용을 위한 차세대 모바일 프레임워크 설계

기기 간 앱 컴포넌트 분산을 위한 다중기기 플랫폼

Edge TPU 에서의 실시간성 보장을 위한 실시간 DNN 프레임워크

다중 서피스 상호작용 지원을 위한 모바일 앱 변환 프레임워크

Edge TPU 에서의 실시간 DNN 작업을 위한 데드라인 인지 SRAM 할당 기법

AppLens : 기존 모바일 앱과 AR 환경의 호환성 지원을 위한 새로운 AR 프레임워크

다중 기기 환경에서 UI 분산을 위한 모바일 플랫폼 연구

Greenbag: Energy-efficient bandwidth aggregation for real-time streaming in heterogeneous mobile wireless networks

Mobilegpt: Augmenting llm with human-like app memory for mobile task automation

Mobile plus: Multi-device mobile platform for cross-device functionality sharing

Explore, select, derive, and recall: Augmenting llm with human-like memory for mobile task automation

Fluid: Flexible user interface distribution for ubiquitous multi-device interaction

RT-SDN: adaptive routing and priority ordering for software-defined real-time networking

Safeguarding mobile gui agent via logic-based action verification

FLUID-XP: Flexible user interface distribution for cross-platform experience

MagSnoop: Listening to sounds induced by magnetic field fluctuations to infer mobile payment tokens

A-mash: providing single-app illusion for multi-app use through user-centric UI mashup

Spet: Transparent sram allocation and model partitioning for real-time dnn tasks on edge tpu

Cros-rt: Cross-layer priority scheduling for predictable inter-process communication in ros 2

TaPIN: Reinforcing PIN Authentication on Smartphones With Tap Biometrics

Supporting flexible and transparent user interface distribution across mobile devices

Leveraging LLMs for Efficient and Personalized Smart Home Automation

App-Level Multi-Surface Framework for Supporting Cross-Platform User Interface Distribution

Sangeun Oh

Modern mobile devices are equipped with multiple network interfaces, including 3G/LTE and WiFi. Bandwidth aggregation over LTE and WiFi links offers an attractive opportunity of supporting bandwidth-intensive services, such as high-quality video streaming, on mobile devices. However, achieving effective bandwidth aggregation in mobile environments raises several challenges related to deployment, link heterogeneity, network fluctuation, and energy consumption. We present GreenBag, an energy-efficient bandwidth aggregation middleware that supports real-time data-streaming services over asymmetric wireless links, requiring no modifications to the existing Internet infrastructure and servers. GreenBag employs several techniques, including medium load balancing, efficient segment management, and energy-aware mode control, to resolve such challenges. We implement a prototype of GreenBag on …

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT 1 , an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app---explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and …

In recent years, the explosion of diverse smart devices such as mobile phones, TVs, watches, and even cars, has completely changed our lives. We communicate with friends through social network services (SNSs) whenever we want, buy stuff without visiting shops, and enjoy multimedia wherever we are, thanks to these devices. However, these smart devices cannot simply interact with each other even though they are right next to each other. For example, when you want to read a PDF stored on a smartphone on a larger TV screen, you need to do complicated work or plug in a bunch of cables. In this paper, we introduce M+, an extension of Android that supports cross-device functionality sharing in a transparent manner. As a platform-level solution, M+ enables unmodified Android applications to utilize not only application functionalities but also system functionalities across devices, as if they were to utilize them …

The advent of large language models (LLMs) has opened up new opportunities in the field of mobile task automation. Their superior language understanding and reasoning capabilities allow users to automate complex and repetitive tasks. However, due to the inherent unreliability and high operational cost of LLMs, their practical applicability is quite limited. To address these issues, this paper introduces MobileGPT, an innovative LLM-based mobile task automator equipped with a human-like app memory. MobileGPT emulates the cognitive process of humans interacting with a mobile app -- explore, select, derive, and recall. This approach allows for a more precise and efficient learning of a task's procedure by breaking it down into smaller, modular sub-tasks that can be re-used, re-arranged, and adapted for various objectives. We implement MobileGPT using online LLMs services (GPT-3.5 and GPT-4) and evaluate its performance on a dataset of 185 tasks across 18 mobile apps. The results indicate that MobileGPT can automate and learn new tasks with 82.7% accuracy, and is able to adapt them to different contexts with near perfect (98.75%) accuracy while reducing both latency and cost by 62.5% and 68.8%, respectively, compared to the GPT-4 powered baseline.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. For example, it is currently not possible for a user to dynamically partition an existing live streaming app with chatting capabilities across different devices, such that she watches her favorite broadcast on her smart TV while real-time chatting on her smartphone. In this paper, we present FLUID, a new Android-based multi-device platform that enables innovative ways of using multiple devices. FLUID aims to i) allow users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices (high flexibility), ii) require no additional development effort to support unmodified, legacy applications (ease of development), and iii) support a …

In this article, an adaptive real-time communication system is proposed that leverages the software-defined networking (SDN) paradigm to provide end-to-end deadline guarantees. SDN is an emerging networking paradigm that allows control of the network through directly programmable software controllers, offering the flexibility to manage and optimize network resources dynamically. By utilizing these SDN features, a novel networking system, called RT-SDN, is presented that can effectively support real-time communication through cooperative routing and scheduling. To this end, we present a highly efficient routing algorithm that can adaptively reconfigure the routes of existing flows to find bandwidth-guaranteed routes for all flows. We then introduce a new priority assignment scheme that can achieve better deadline guarantees (i.e., better schedulability). In addition, RT-SDN allows routing and scheduling to …

Large Foundation Models (LFMs) have unlocked new possibilities in human-computer interaction, particularly with the rise of mobile Graphical User Interface (GUI) Agents capable of interpreting GUIs. These agents promise to revolutionize mobile computing by allowing users to automate complex mobile tasks through simple natural language instructions. However, the inherent probabilistic nature of LFMs, coupled with the ambiguity and context-dependence of mobile tasks, makes LFM-based automation unreliable and prone to errors. To address this critical challenge, we introduce VeriSafe Agent (VSA): a formal verification system that serves as a logically grounded safeguard for Mobile GUI Agents. VSA is designed to deterministically ensure that an agent's actions strictly align with user intent before conducting an action. At its core, VSA introduces a novel autoformalization technique that translates natural …

Being able to use a single app across multiple devices can bring novel experiences to the users in various domains including entertainment and productivity. For instance, a user of a video editing app would be able to use a smart pad as a canvas and a smartphone as a remote toolbox so that the toolbox does not occlude the canvas during editing. However, existing approaches do not properly support the single-app multi-device execution due to several limitations, including high development cost, device heterogeneity, and high performance requirement. In this paper, we introduce FLUID-XP, a novel cross-platform multi-device system that enables UIs of a single app to be executed across heterogeneous platforms, while overcoming the limitations of previous approaches. FLUID-XP provides flexible, efficient, and seamless interactions by addressing three main challenges: i) how to transparently enable a single …

Samsung Pay, one of the most representative mobile payment services, allows mobile users to make payment transactions almost anywhere using only their smartphone. This is thanks to MST (Magnetic Secure Transmission) that supports communication between smartphones and payment terminals for magnetic cards by transferring payment tokens via magnetic waves. Several attack methods have targeted this new technology by eavesdropping on magnetic fields to intercept the tokens, but with the use of dedicated hardware. This paper raises new security concerns for mobile payment users in a different, yet more effective way; by introducing MagSnoop, a novel framework that infers payment tokens from listening to MST sounds generated during the activation of MST payment transactions. More specifically, we first explore the principle, causing the generation of MST sounds, and the fundamental …

Mobile apps offer a variety of features that greatly enhance user experience. However, users still often find it difficult to use mobile apps in the way they want. For example, it is not easy to use multiple apps simultaneously on a small screen of a smartphone. In this paper, we present A-Mash, a mobile platform that aims to simplify the way of interacting with multiple apps concurrently to the level of using a single app only. A key feature of A-Mash is that users can mash up the UIs of different existing mobile apps on a single screen according to their preferences. To this end, A-Mash 1) extracts UIs from unmodified existing apps (dynamic UI extraction) and 2) embeds extracted UIs from different apps into a single wrapper app (cross-process UI embedding) , while 3) making all these processes hidden from the users (transparent execution environment). To the best of our knowledge, A-Mash is the first work to enable UIs of …

Deep neural networks (DNNs) have been deployed in many safety-critical real-time embedded systems. To support DNN tasks in real-time, most previous studies focused on GPU or CPU. However, Edge TPU has not yet been studied for real-time guarantees. This paper presents a real-time DNNs framework for Edge TPU to satisfy multiple DNN inference tasks’ timing requirements. The proposed framework provides 1) SRAM allocation and model partitioning techniques and 2) a MIP-based algorithm that determines the amount of SRAM and the number of segments for each task. The experiment result shows that our framework provides 79% higher schedulability than the existing Edge TPU system.

The increasing complexity and memory demands of Deep Neural Networks (DNNs) for real-time systems pose new significant challenges, one of which is the GPU memory capacity bottleneck, where the limited physical memory inside GPUs impedes the deployment of sophisticated DNN models. This paper presents, to the best of our knowledge, the first study of addressing the GPU memory bottleneck issues, while simultaneously ensuring the timely inference of multiple DNN tasks. We propose RT-Swap, a real-time memory management framework, that enables transparent and efficient swap scheduling of memory objects, employing the relatively larger CPU memory to extend the available GPU memory capacity, without compromising timing guarantees. We have implemented RT-Swap on top of representative machine-learning frameworks, demonstrating its effectiveness in making significantly more DNN task …

The Robot Operating System 2 (ROS 2) is a popular middleware for distributed robotic applications. However, achieving real-time guarantees in ROS 2 is challenging due to unpredictable delays and priority inversions. We reveal that these issues arise from the lack of consistent priority propagation across ROS 2's multi-layered communication architecture, particularly down to the kernel layer. To address this, we present CROS-RT, the first cross-layer scheduler explicitly designed to tackle the unpredictability in ROS 2 inter-process communication caused by multi-layer priority misalignment. CROS-RT ensures consistent, priority-based scheduling across the application, middleware, and kernel layers, introducing mechanisms for priority propagation, kernel-level message prioritization, and dynamic kernel thread adjustment. We have implemented and evaluated CROS-RT on the current stable release of ROS 2 …

Samsung Pay, a widely-used mobile payment service, enables users to pay using just their smartphone thanks to Magnetic Secure Transmission (MST). This technology facilitates communication between smartphones and magnetic card terminals by transmitting payment tokens through magnetic waves. Intriguingly, such magnetic waves inherently produce a distinct sound pattern (called MST sound ) containing payment information, which opens up new opportunities for both potential attackers and payment users. That is, MST sound can serve either as a new side channel for attackers to eavesdrop on MST transactions or as an easily accessible communication channel that enhances the payment experience for users. Inspired by these possibilities, we aim to deeply explore the potential of MST sound across these two dimensions, presenting two frameworks with different objectives: MagSnoop and M2APay. The …

PIN authentication is the first line of defense for protecting private data on many smartphone applications, such as lock screens, messengers, and banking apps. However, existing PIN authentication systems have several constraints regarding security, usability, and robustness. To go beyond their limitations, this paper presents TaPIN, a reliable system that authenticates smartphone users with the collaborative use of PINs and tap biometrics. A user is first instructed to enter her PIN by tapping a smartphone screen for authentication. During the PIN entry, the user's fingertip collides with the screen, producing user-specific vibration and sound signals. TaPIN then senses the tap-induced signals and the collision properties, e.g., pressures and sizes, using the smartphone's built-in sensors and leverages them as biometric features. That is, it authenticates the user by verifying not only the entered PIN but also the …

The growing trend of multi-device ownerships creates opportunities to use applications across devices. However, the current methods of app development/usage remain in the single-device paradigm, which is far below user expectations. For example, it is currently impossible for users to dynamically partition an existing app across different devices to utilize multiple surfaces. We introduce , a novel multi-device platform that supports simultaneous operation of multiple devices. aims to i) distribute the user interfaces (UIs) of a single app across multiple devices, ii) support unmodified legacy apps without extra engineering, and iii) support numerous apps with customized UIs. Previous approaches, like screen mirroring and app migration, do not satisfy those goals altogether. However, is designed to satisfy the goals. It can efficiently deploy UI objects to different devices by identifying only UI states necessary for …

The proliferation of smart home devices has increased the complexity of controlling and managing them, leading to user fatigue. In this context, large language models (LLMs) offer a promising solution by enabling natural-language interfaces for Internet of Things (IoT) control. However, existing LLM-based approaches suffer from unreliable and inefficient device control due to the non-deterministic nature of LLMs, high inference latency and cost, and limited personalization. To address these challenges, we present IoTGPT, an LLM-based smart home agent designed to execute IoT commands in a reliable, efficient, and personalized manner. Inspired by how humans manage complex tasks, IoTGPT decomposes user instructions into subtasks and memorizes them. By reusing learned subtasks, subsequent instructions can be processed more efficiently with fewer LLM calls, improving reliability and reducing both latency and cost. IoTGPT also supports fine-grained personalization by adapting individual subtasks to user preferences. Our evaluation demonstrates that IoTGPT outperforms baselines in accuracy, latency/cost, and personalization, while reducing user workload.

Back-of-Device (BoD) interfaces have emerged as a promising solution to free up screen real estate in smartphones by offloading interactions from the display to the back, thereby reducing reliance on on-screen interfaces. However, existing BoD solutions face limitations, such as requiring specialized hardware, consuming excessive power, or offering limited input vocabularies. We introduce MagPie, a novel BoD interface that leverages the magnetic phenomenon induced by MagSafe, part of the wireless charging standard. Users can seamlessly attach MagPie to MagSafe-enabled smartphones and interact using tangible, modular interfaces that generate unique magnetic signals upon activation. MagPie then detects these signals and recognizes the input through magnetic sensing. Our experiments with real-world users demonstrate that i ) MagPie achieves high performance in accuracy, usability, deployability …

With the recent advances in IoT, there is a growing interest in multi-surface computing, where a mobile app can cooperatively utilize multiple devices' surfaces. We propose a novel framework that seamlessly augments mobile apps with multi-surface computing capabilities. It enables various apps to employ multiple surfaces with acceptable performance.

The growing trend of multi-device ownerships creates a need and an opportunity to use applications across multiple devices. However, in general, the current app development and usage still remain within the single-device paradigm, falling far short of user expectations. We present FLUID, a new multi-device platform that allows users to migrate or replicate individual user interfaces (UIs) of a single app on multiple devices. In addition, FLUID aims to require no extra development effort to support a wide range of legacy apps that follow the trend of using custom-made UIs. To this end, FLUID analyzes which UI states are necessary to correctly render UI objects, deploys only those states on different devices, and supports cross-device function calls transparently. In this demo, we demonstrate several interesting use cases supported by our Android-based FLUID prototype.

MES Lab. @ KU

교수님 소개

오상은

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

SCOUT: Thermal-Aware SRAM Allocation for Real-Time DNN Tasks on Edge TPU

VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification

Mitigating Resource Contention for Responsive On-device Machine Learning Inferences

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

오상은

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

SCOUT: Thermal-Aware SRAM Allocation for Real-Time DNN Tasks on Edge TPU

VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification

Mitigating Resource Contention for Responsive On-device Machine Learning Inferences

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집