-
Leveraging Environment Interaction for Automated PDDL Generation and Planning with Large Language Models
Authors:
Sadegh Mahdavi,
Raquel Aoki,
Keyi Tang,
Yanshuai Cao
Abstract:
Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning problems into the Planning Domain Definition Language (PDDL) has been proposed as a potential solution, enabling the use of automated planners. However, generating a…
▽ More
Large Language Models (LLMs) have shown remarkable performance in various natural language tasks, but they often struggle with planning problems that require structured reasoning. To address this limitation, the conversion of planning problems into the Planning Domain Definition Language (PDDL) has been proposed as a potential solution, enabling the use of automated planners. However, generating accurate PDDL files typically demands human inputs or correction, which can be time-consuming and costly. In this paper, we propose a novel approach that leverages LLMs and environment feedback to automatically generate PDDL domain and problem description files without the need for human intervention. Our method introduces an iterative refinement process that generates multiple problem PDDL candidates and progressively refines the domain PDDL based on feedback obtained from interacting with the environment. To guide the refinement process, we develop an Exploration Walk (EW) metric, which provides rich feedback signals for LLMs to update the PDDL file. We evaluate our approach on PDDL environments. We achieve an average task solve rate of 66% compared to a 29% solve rate by GPT-4's intrinsic planning with chain-of-thought prompting. Our work enables the automated modeling of planning environments using LLMs and environment feedback, eliminating the need for human intervention in the PDDL generation process and paving the way for more reliable LLM agents in challenging problems.
△ Less
Submitted 17 July, 2024;
originally announced July 2024.
-
Counterfactual Explanations for Multivariate Time-Series without Training Datasets
Authors:
Xiangyu Sun,
Raquel Aoki,
Kevin H. Wilson
Abstract:
Machine learning (ML) methods have experienced significant growth in the past decade, yet their practical application in high-impact real-world domains has been hindered by their opacity. When ML methods are responsible for making critical decisions, stakeholders often require insights into how to alter these decisions. Counterfactual explanations (CFEs) have emerged as a solution, offering interp…
▽ More
Machine learning (ML) methods have experienced significant growth in the past decade, yet their practical application in high-impact real-world domains has been hindered by their opacity. When ML methods are responsible for making critical decisions, stakeholders often require insights into how to alter these decisions. Counterfactual explanations (CFEs) have emerged as a solution, offering interpretations of opaque ML models and providing a pathway to transition from one decision to another. However, most existing CFE methods require access to the model's training dataset, few methods can handle multivariate time-series, and none can handle multivariate time-series without training datasets. These limitations can be formidable in many scenarios. In this paper, we present CFWoT, a novel reinforcement-learning-based CFE method that generates CFEs when training datasets are unavailable. CFWoT is model-agnostic and suitable for both static and multivariate time-series datasets with continuous and discrete features. Users have the flexibility to specify non-actionable, immutable, and preferred features, as well as causal constraints which CFWoT guarantees will be respected. We demonstrate the performance of CFWoT against four baselines on several datasets and find that, despite not having access to a training dataset, CFWoT finds CFEs that make significantly fewer and significantly smaller changes to the input time-series. These properties make CFEs more actionable, as the magnitude of change required to alter an outcome is vastly reduced.
△ Less
Submitted 28 May, 2024;
originally announced May 2024.
-
Causal Inference from Small High-dimensional Datasets
Authors:
Raquel Aoki,
Martin Ester
Abstract:
Many methods have been proposed to estimate treatment effects with observational data. Often, the choice of the method considers the application's characteristics, such as type of treatment and outcome, confounding effect, and the complexity of the data. These methods implicitly assume that the sample size is large enough to train such models, especially the neural network-based estimators. What i…
▽ More
Many methods have been proposed to estimate treatment effects with observational data. Often, the choice of the method considers the application's characteristics, such as type of treatment and outcome, confounding effect, and the complexity of the data. These methods implicitly assume that the sample size is large enough to train such models, especially the neural network-based estimators. What if this is not the case? In this work, we propose Causal-Batle, a methodology to estimate treatment effects in small high-dimensional datasets in the presence of another high-dimensional dataset in the same feature space. We adopt an approach that brings transfer learning techniques into causal inference. Our experiments show that such an approach helps to bring stability to neural network-based methods and improve the treatment effect estimates in small high-dimensional datasets.
△ Less
Submitted 18 May, 2022;
originally announced May 2022.
-
Multi-treatment Effect Estimation from Biomedical Data
Authors:
Raquel Aoki,
Yizhou Chen,
Martin Ester
Abstract:
This work proposes the M3E2, a multi-task learning neural network model to estimate the effect of multiple treatments. In contrast to existing methods, M3E2 can handle multiple treatment effects applied simultaneously to the same unit, continuous and binary treatments, and many covariates. We compared M3E2 with three baselines in three synthetic benchmark datasets: two with multiple treatments and…
▽ More
This work proposes the M3E2, a multi-task learning neural network model to estimate the effect of multiple treatments. In contrast to existing methods, M3E2 can handle multiple treatment effects applied simultaneously to the same unit, continuous and binary treatments, and many covariates. We compared M3E2 with three baselines in three synthetic benchmark datasets: two with multiple treatments and one with one treatment. Our analysis showed that our method has superior performance, making more assertive estimations of the multiple treatment effects.
△ Less
Submitted 5 January, 2023; v1 submitted 14 December, 2021;
originally announced December 2021.
-
Heterogeneous Multi-task Learning with Expert Diversity
Authors:
Raquel Aoki,
Frederick Tung,
Gabriel L. Oliveira
Abstract:
Predicting multiple heterogeneous biological and medical targets is a challenge for traditional deep learning models. In contrast to single-task learning, in which a separate model is trained for each target, multi-task learning (MTL) optimizes a single model to predict multiple related targets simultaneously. To address this challenge, we propose the Multi-gate Mixture-of-Experts with Exclusivity…
▽ More
Predicting multiple heterogeneous biological and medical targets is a challenge for traditional deep learning models. In contrast to single-task learning, in which a separate model is trained for each target, multi-task learning (MTL) optimizes a single model to predict multiple related targets simultaneously. To address this challenge, we propose the Multi-gate Mixture-of-Experts with Exclusivity (MMoEEx). Our work aims to tackle the heterogeneous MTL setting, in which the same model optimizes multiple tasks with different characteristics. Such a scenario can overwhelm current MTL approaches due to the challenges in balancing shared and task-specific representations and the need to optimize tasks with competing optimization paths. Our method makes two key contributions: first, we introduce an approach to induce more diversity among experts, thus creating representations more suitable for highly imbalanced and heterogenous MTL learning; second, we adopt a two-step optimization [6, 11] approach to balancing the tasks at the gradient level. We validate our method on three MTL benchmark datasets, including Medical Information Mart for Intensive Care (MIMIC-III) and PubChem BioAssay (PCBA).
△ Less
Submitted 27 May, 2022; v1 submitted 19 June, 2021;
originally announced June 2021.
-
PEAK SHIFT ESTIMATION A novel method to estimate ranking of selectively omitted examination data
Authors:
Satoshi Takahashi,
Masaki Kitazawa,
Ryoma Aoki,
Atsushi Yoshikawa
Abstract:
In this paper, we focus on examination results when examinees selectively skip examinations, to compare the difficulty levels of these examinations. We call the resultant data 'selectively omitted examination data' Examples of this type of examination are university entrance examinations, certification examinations, and the outcome of students' job-hunting activities. We can learn the number of st…
▽ More
In this paper, we focus on examination results when examinees selectively skip examinations, to compare the difficulty levels of these examinations. We call the resultant data 'selectively omitted examination data' Examples of this type of examination are university entrance examinations, certification examinations, and the outcome of students' job-hunting activities. We can learn the number of students accepted for each examination and organization but not the examinees' identity. No research has focused on this type of data. When we know the difficulty level of these examinations, we can obtain a new index to assess organization ability, how many students pass, and the difficulty of the examinations. This index would reflect the outcomes of their education corresponding to perspectives on examinations. Therefore, we propose a novel method, Peak Shift Estimation, to estimate the difficulty level of an examination based on selectively omitted examination data. First, we apply Peak Shift Estimation to the simulation data and demonstrate that Peak Shift Estimation estimates the rank order of the difficulty level of university entrance examinations very robustly. Peak Shift Estimation is also suitable for estimating a multi-level scale for universities, that is, A, B, C, and D rank university entrance examinations. We apply Peak Shift Estimation to real data of the Tokyo metropolitan area and demonstrate that the rank correlation coefficient between difficulty level ranking and true ranking is 0.844 and that the difference between 80 percent of universities is within 25 ranks. The accuracy of Peak Shift Estimation is thus low and must be improved; however, this is the first study to focus on ranking selectively omitted examination data, and therefore, one of our contributions is to shed light on this method.
△ Less
Submitted 9 March, 2021;
originally announced March 2021.
-
ParKCa: Causal Inference with Partially Known Causes
Authors:
Raquel Aoki,
Martin Ester
Abstract:
Methods for causal inference from observational data are an alternative for scenarios where collecting counterfactual data or realizing a randomized experiment is not possible. Adopting a stacking approach, our proposed method ParKCA combines the results of several causal inference methods to learn new causes in applications with some known causes and many potential causes. We validate ParKCA in t…
▽ More
Methods for causal inference from observational data are an alternative for scenarios where collecting counterfactual data or realizing a randomized experiment is not possible. Adopting a stacking approach, our proposed method ParKCA combines the results of several causal inference methods to learn new causes in applications with some known causes and many potential causes. We validate ParKCA in two Genome-wide association studies, one real-world and one simulated dataset. Our results show that ParKCA can infer more causes than existing methods.
△ Less
Submitted 11 November, 2020; v1 submitted 17 March, 2020;
originally announced March 2020.
-
Luck is Hard to Beat: The Difficulty of Sports Prediction
Authors:
Raquel YS Aoki,
Renato M Assuncao,
Pedro OS Vaz de Melo
Abstract:
Predicting the outcome of sports events is a hard task. We quantify this difficulty with a coefficient that measures the distance between the observed final results of sports leagues and idealized perfectly balanced competitions in terms of skill. This indicates the relative presence of luck and skill. We collected and analyzed all games from 198 sports leagues comprising 1503 seasons from 84 coun…
▽ More
Predicting the outcome of sports events is a hard task. We quantify this difficulty with a coefficient that measures the distance between the observed final results of sports leagues and idealized perfectly balanced competitions in terms of skill. This indicates the relative presence of luck and skill. We collected and analyzed all games from 198 sports leagues comprising 1503 seasons from 84 countries of 4 different sports: basketball, soccer, volleyball and handball. We measured the competitiveness by countries and sports. We also identify in each season which teams, if removed from its league, result in a completely random tournament. Surprisingly, not many of them are needed. As another contribution of this paper, we propose a probabilistic graphical model to learn about the teams' skills and to decompose the relative weights of luck and skill in each game. We break down the skill component into factors associated with the teams' characteristics. The model also allows to estimate as 0.36 the probability that an underdog team wins in the NBA league, with a home advantage adding 0.09 to this probability. As shown in the first part of the paper, luck is substantially present even in the most competitive championships, which partially explains why sophisticated and complex feature-based models hardly beat simple models in the task of forecasting sports' outcomes.
△ Less
Submitted 7 June, 2017;
originally announced June 2017.
