Software Engineering Laboratory for Next-Gen Ecosystems

[02841] 서울 성북구 안암로 145 고려대학교 자연계캠퍼스 과학도서관 616A호

Pig: Leveraging Large Language Models for Python Library Migrations

Finding the Needle in the Crash Stack: Industrial-Scale Crash Root Cause Localization with AutoCrashFL

METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries

Lachesis: Predicting LLM Inference Accuracy using Structural Properties of Reasoning Paths

COSMosFL: Ensemble of Small Language Models for Fault Localisation

Evaluating Machine Learning-Based Test Case Prioritization in the Real World: An Experiment with SAP HANA

Just-in-Time Flaky Test Detection via Abstracted Failure Symptom Matching

A Quantitative and Qualitative Evaluation of LLM-based Explainable Fault Localization

Iterative Refactoring of Real-World Open-Source Programs with Large Language Models

Learning Test-Mutant Relationship for Accurate Fault Localisation

BUGSC++: A Highly Usable Real World Defect Benchmark for C/C++

Fonte: Finding Bug Inducing Commits from Failures

Automatically Identifying Shared Root Causes of Test Breakages in SAP HANA

FDG: A Precise Measurement of Fault Diagnosability Gain of Test Cases

Ahead of Time Mutation Based Fault Localisation using Statistical Inference

Searching for Multi-Fault Programs in Defects4J

Reducing the Search Space of Bug Inducing Commits using Failure Coverage

Assisting Bug Report Assignment Using Automated Fault Localisation: An Industrial Case Study

PyGGI 2.0: Language Independent Genetic Improvement Framework

Comparing Line and AST Granularity Level for Program Repair using PyGGI

A quantitative and qualitative evaluation of LLM-based explainable fault localization

PyGGI 2.0: language independent genetic improvement framework

Fonte: Finding bug inducing commits from failures

Comparing line and AST granularity level for program repair using PyGGI

Searching for multi-fault programs in defects4j

Iterative refactoring of real-world open-source programs with large language models

Bugsc++: A highly usable real world defect benchmark for c/c++

Reducing the search space of bug inducing commits using failure coverage

PyGGI: Python General framework for Genetic Improvement

Learning test-mutant relationship for accurate fault localisation

Just-in-time flaky test detection via abstracted failure symptom matching

Lachesis: Predicting llm inference accuracy using structural properties of reasoning paths

The state and future of genetic improvement

The 13th international workshop on genetic improvement (gi@ icse 2024)

Genetic Improvement@ ICSE 2023

Shin Yoo

Sungmin Kang

Jingun Hong

Jinhan Kim

Juyeon Yoon

Aymeric Blot

Justyna Petke

Naryeong Kim

Robert Feldt

Jeongju Sohn

Dongwon Hwang

Jinsu Choi

Jooyong Yi

Seongmin Lee

Gabin An

Fault Localization (FL), in which a developer seeks to identify which part of the code is malfunctioning and needs to be fixed, is a recurring challenge in debugging. To reduce developer burden, many automated FL techniques have been proposed. However, prior work has noted that existing techniques fail to provide rationales for the suggested locations, hindering developer adoption of these techniques. With this in mind, we propose A uto FL, a Large Language Model (LLM)-based FL technique that generates an explanation of the bug along with a suggested fault location. A uto FL prompts an LLM to use function calls to navigate a repository, so that it can effectively localize faults over a large software repository and overcome the limit of the LLM context length. Extensive experiments on 798 real-world bugs in Java and Python reveal A uto FL improves method-level acc@1 by up to 233.3% over baselines …

PyGGI is a research tool for Genetic Improvement (GI), that is designed to be versatile and easy to use. We present version 2.0 of PyGGI, the main feature of which is an XML-based intermediate program representation. It allows users to easily define GI operators and algorithms that can be reused with multiple target languages. Using the new version of PyGGI, we present two case studies. First, we conduct an Automated Program Repair (APR) experiment with the QuixBugs benchmark, one that contains defective programs in both Python and Java. Second, we replicate an existing work on runtime improvement through program specialisation for the MiniSAT satisfiability solver. PyGGI 2.0 was able to generate a patch for a bug not previously fixed by any APR tool. It was also able to achieve 14% runtime improvement in the case of MiniSAT. The presented results show the applicability and the expressiveness of the …

A Bug Inducing Commit (BIC) is a commit that introduces a software bug into the codebase. Knowing the relevant BIC for a given bug can provide valuable information for debugging as well as bug triaging. However, existing BIC identification techniques are either too expensive (because they require the failing tests to be executed against previous versions for bisection) or inapplicable at the debugging time (because they require post hoc artefacts such as bug reports or bug fixes). We propose Fonte, an efficient and accurate BIC identification technique that only requires test coverage. Fonte combines Fault Localisation (FL) with BIC identification and ranks commits based on the suspiciousness of the code elements that they modified. Fonte reduces the search space of BICs using failure coverage as well as a filter that detects commits that are merely style changes. Our empirical evaluation using 130 real-world …

P y GGI is a lightweight Python framework that can be used to implement generic Genetic Improvement algorithms at the API level. The original version of P y GGI only provided lexical modifications, i.e., modifications of the source code at the physical line granularity level. This paper introduces new extensions to P y GGI that enables syntactic modifications for Python code, i.e., modifications that operates at the AST granularity level. Taking advantage of the new extensions, we also present a case study that compares the lexical and syntactic search granularity level for automated program repair, using ten seeded faults in a real world open source Python project. The results show that search landscapes at the AST granularity level are more effective (i.e. eventually more likely to produce plausible patches) due to the smaller sizes of ingredient spaces (i.e., the space from which we search for the material to build a patch …

Defects4J has enabled numerous software testing and debugging research work since its introduction. A large part of its contribution, and the resulting popularity, lies in the clear separation and distillation of the root cause of each individual test failure based on careful manual analysis, which in turn allowed researchers to easily study individual faults in isolation. However, in a realistic debugging scenario, multiple faults can coexist and affect test results collectively. Study of automated debugging techniques for these situations, such as failure clustering or fault localisation for multiple faults, would significantly benefit from a reliable benchmark of multiple, coexisting faults. We search for versions of Defects4J subjects that contain multiple faults, by iteratively transplanting fault-revealing test cases across Defects4J versions. Out of 326 studied versions of Defects4J subjects, we report that over 95% (311 versions …

Code refactoring is a critical task for improving software quality, but it is traditionally a manual, time-consuming process. This paper demonstrates an approach to automate project-level code refactoring using Large Language Models (LLMs). The key idea is to iteratively identify methods with high cyclomatic complexity, and then use LLMs to generate refactored implementations that reduce complexity. Our evaluation using 17 open-source projects shows that the proposed automated refactoring can reduce average cyclomatic complexity by up to 10.4% within 20 iterations. These results suggest that automated project-level code refactoring is feasible using LLMs with tailored prompts.

As software systems grow larger and more complex, debugging takes up an increasingly significant portion of developers' time and efforts during software maintenance. To aid software engineers in debugging, many automated debugging and repair techniques have been proposed. Both the development and evaluation of these automated techniques depend on benchmarks of bugs. While many different defect benchmarks have been developed, only a few benchmarks are widely used due to the origin of the collected bugs as well as the usability of the benchmarks themselves, risking a biased research landscape. This paper presents BUGSC++, a new benchmark that contains 209 real-world bugs collected from 22 open-source C/C++ projects. BugsC++ aims to provide high usability by providing a similar user interface to the widely used Defects4J. Further, BugsC++ ensures the replicability of the bugs in its …

Mutation analysis can effectively capture the de-pendency between source code and test results. This has been exploited by Mutation Based Fault Localisation (MBFL) techniques. However, MBFL techniques suffer from the need to expend the high cost of mutation analysis after the observation of failures, which may present a challenge for its practical adoption. We introduce SIMFL (Statistical Inference for Mutation-based Fault Localisation), an MBFL technique that allows users to perform the mutation analysis in advance before a failure is observed, allowing the amortisation of the analysis cost. SIMFL uses mutants as artificial faults and aims to learn the failure patterns among test cases against different locations of mutations. Once a failure is observed, SIMFL requires either almost no or very small additional cost for analysis, depending on the used inference model. An empirical evaluation using DEFECTS4J …

Knowing how exactly a bug has been introduced into the code can help developers debug the bug efficiently. However, techniques currently used to retrieve Bug Inducing Commits (BICs) from the repository timeline are limited in their accuracy. Automated bisection of the version history depends on the bug revealing test case being executable against all candidate previous versions, whereas blaming the last commits that touched the same parts as the fixing commit (à la SZZ) requires that the bug has already been fixed. We show that filtering commits using the coverage of the bug revealing test cases can effectively reduce the search space for both bisection and SZZ-like blame models by 87.6% and 27.9%, respectively, significantly reducing the cost of BIC retrieval. The application of our approach to bugs in Defects4J also reveals inconsistencies in some of their BICs known in the literature.

The performance of many Fault Localisation (FL) techniques directly depends on the quality of the used test suites. Consequently, it is extremely useful to be able to precisely measure how much diagnostic power each test case can introduce when added to a test suite used for FL. Such a measure can help us not only to prioritise and select test cases to be used for FL, but also to effectively augment test suites that are too weak to be used with FL techniques. We propose FDG, a new measure of Fault Diagnosability Gain for individual test cases. The design of FDG is based on our analysis of existing metrics that are designed to prioritise test cases for better FL. Unlike other metrics, FDG exploits the ongoing FL results to emphasise the parts of the program for which more information is needed. Our evaluation of FDG with Defects4J shows that it can successfully help the augmentation of test suites for better FL. When …

We present Python General Framework for Genetic Improvement (PYGGI, pronounced ˈpɪgɪ), a lightweight general framework for Genetic Improvement (GI). It is designed to be a simple and easy to configure GI tool for multiple programming languages such as Java, C, or Python. Through two case studies, we show that PYGGI can modify source code of a given program either to improve non-functional properties or to automatically repair functional faults.

We present a case study of an industry scale application of automated fault localisation to SAP HANA2 database. When a test breaks in the Continuous Integration (CI) pipeline, the bug needs to be triaged and assigned to the appropriate development team. Given the scale and complexity of SAP HANA2, the assignment itself can be a challenging task. The current practice depends on the static mapping between test scripts and software components, as well as human domain knowledge. We apply automated fault localisation to aid the issue allocation in the CI pipeline: once a test failure is observed, the automated fault localisation technique identifies the suspicious software component using the information from the test failure. The localisation result can be used by the issue manager to allocate the incoming test failure issues more efficiently. We have analysed 137 CI test executions with at least one failing test …

Context Automated fault localisation aims to assist developers in the task of identifying the root cause of the fault by narrowing down the space of likely fault locations. Simulating variants of the faulty program called mutants, several Mutation Based Fault Localisation (MBFL) techniques have been proposed to automatically locate faults. Despite their success, existing MBFL techniques suffer from the cost of performing mutation analysis after the fault is observed. Method To overcome this shortcoming, we propose a new MBFL technique named SIMFL (Statistical Inference for Mutation-based Fault Localisation). SIMFL localises faults based on the past results of mutation analysis that has been done on the earlier version in the project history, allowing developers to make predictions on the location of incoming faults in a just-in-time manner. Using several statistical inference methods, SIMFL models the relationship …

Continuous Integration (CI) of a large-scale software system such as SAP HANA can produce a non-trivial number of test breakages. Each breakage that newly occurs from daily runs needs to be manually inspected, triaged, and eventually assigned to developers for debugging. However, not all new breakages are unique, as some test breakages would share the same root cause; in addition, human errors can produce duplicate bug tickets for the same root cause. An automated identification of breakages with shared root causes will be able to significantly reduce the cost of the (typically manual) post-breakage steps. This paper investigates multiple similarity functions between test breakages to assist and automate the identification of test breakages that are caused by the same root cause. We consider multiple information sources, such as static (i.e., the code itself), historical (i.e., whether the test results have …

We report our experience of using failure symptoms, such as error messages or stack traces, to identify flaky test failures in a Continuous Integration (CI) pipeline for a large industrial software system, SAP HANA. Although failure symptoms are commonly used to identify similar failures, they have not previously been employed to detect flaky test failures. Our hypothesis is that flaky failures will exhibit symptoms distinct from those of non-flaky failures. Consequently, we can identify recurring flaky failures, without rerunning the tests, by matching the failure symptoms to those of historical flaky runs. This can significantly reduce the need for test reruns, ultimately resulting in faster delivery of test results to developers. To facilitate the process of matching flaky failures across different execution instances, we abstract newer test failure symptoms before matching them to the known patterns of flaky failures, inspired by previous …

Large Language Models are increasingly used to build agents to perform more complex tasks. As LLMs perform more complicated reasoning through longer interactions, self-consistency, i.e., the idea that the answer obtained from sampling and marginalising a number of multiple independent inferences is more likely to be correct, has been received much attention as a simple validation technique. This paper aims to empirically verify this intuitive hypothesis by predicting the correctness of answers obtained using self-consistency from properties of the samples of reasoning paths. We introduce Lachesis, a predictive model for self-consistency based LLM inferences, and empirically evaluate it using AutoFL, a recently proposed LLM-based fault localisation technique, as the target technique that uses self-consistency. Lachesis converts collected reasoning paths from AutoFL using specifically designed reasoning …

We report the discussion session at the sixth international Genetic Improvement workshop, GI-2019 @ ICSE, which was held as part of the 41st ACM/IEEE International Confer- ence on Software Engineering on Tuesday 28th May 2019. Topics included GI representations, the maintainability of evolved code, automated software testing, future areas of GI research, such as co-evolution, and existing GI tools and benchmarks.

The GI @ ICSE 2024 workshop, held 16 April, in addition to presentations, contained a keynote on how to use Genetic Improvement to control deep AI large language models in software engineering and a tutorial on a language independent tool for GI research. We summarise these, the papers, people, prizes, acknowledgements, discussions and hopes for the future.

LLMs are rapidly being adopted to build powerful tools and agents for software engineering, but most of them rely heavily on extremely large closed-source models. This, in turn, can hinder wider adoption due to security issues as well as financial cost and environmental impact. Recently, a number of open source Small Language Models (SLMs) are being released and gaining traction. While SLMs are smaller, more energy-efficient, and therefore easier to locally deploy, they tend to show worse performance when compared to larger closed LLMs. We present COSMos, a task-level LLM ensemble technique that uses voting mechanism, to provide a broader range of choice between SLMs and LLMs. We instantiate COSMos with an LLM-based Fault Localisation technique, AutoFL, and report the cost-benefit trade-off between LLM accuracy and various costs such as energy consumption, inference time, and the …

Following the formal presentations, which included keynotes by Prof. Myra B. Cohen of Iowa State University and Dr. Sebastian Baltes of SAP as well as six papers (which are recorded in the pro- ceedings) there was a wide ranging discussion at the twelfth inter- national Genetic Improvement workshop, GI-2023 @ ICSE held on Saturday 20th May 2023 in Melbourne and online via Zoom. Topics included GI to improve testing, and remove unpleasant surprises in cloud computing costs, incorporating novelty search, large language models (LLM ANN) and GI benchmarks.

Ph.D. in Computer Science, KAIST (Distinguished Dissertation Award)

Assistant Professor, Department of Computer Science and Engineering, Korea University

Software Engineering Laboratory for Next-Gen Ecosystems

교수님 소개

안가빈

연구실 소개

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Pig: Leveraging Large Language Models for Python Library Migrations

Finding the Needle in the Crash Stack: Industrial-Scale Crash Root Cause Localization with AutoCrashFL

METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries

연구실 보유 특허

대학별 입시끈

우리 연구실 모집

교수님 소개

안가빈

연구실 키워드

알럼나이가 간 회사

연구실 지원 방법

대우 조건

연도별 논문 인용수

저널 & 학회 논문 출판 수

연구실 논문 목록

Pig: Leveraging Large Language Models for Python Library Migrations

Finding the Needle in the Crash Stack: Industrial-Scale Crash Root Cause Localization with AutoCrashFL

METAMON: Finding Inconsistencies between Program Documentation and Behavior using Metamorphic LLM Queries

연구실 보유 특허

최근 3년간 연구실적 높은 연구실

대학별 입시끈

우리 연구실 모집