-
Evaluating the Ability of Large Language Models to Reason about Cardinal Directions
Authors:
Anthony G Cohn,
Robert E Blackwell
Abstract:
We investigate the abilities of a representative set of Large language Models (LLMs) to reason about cardinal directions (CDs). To do so, we create two datasets: the first, co-created with ChatGPT, focuses largely on recall of world knowledge about CDs; the second is generated from a set of templates, comprehensively testing an LLM's ability to determine the correct CD given a particular scenario.…
▽ More
We investigate the abilities of a representative set of Large language Models (LLMs) to reason about cardinal directions (CDs). To do so, we create two datasets: the first, co-created with ChatGPT, focuses largely on recall of world knowledge about CDs; the second is generated from a set of templates, comprehensively testing an LLM's ability to determine the correct CD given a particular scenario. The templates allow for a number of degrees of variation such as means of locomotion of the agent involved, and whether set in the first , second or third person. Even with a temperature setting of zero, Our experiments show that although LLMs are able to perform well in the simpler dataset, in the second more complex dataset no LLM is able to reliably determine the correct CD, even with a temperature setting of zero.
△ Less
Submitted 24 June, 2024;
originally announced June 2024.
-
Exploring Spatial Representations in the Historical Lake District Texts with LLM-based Relation Extraction
Authors:
Erum Haris,
Anthony G. Cohn,
John G. Stell
Abstract:
Navigating historical narratives poses a challenge in unveiling the spatial intricacies of past landscapes. The proposed work addresses this challenge within the context of the English Lake District, employing the Corpus of the Lake District Writing. The method utilizes a generative pre-trained transformer model to extract spatial relations from the textual descriptions in the corpus. The study ap…
▽ More
Navigating historical narratives poses a challenge in unveiling the spatial intricacies of past landscapes. The proposed work addresses this challenge within the context of the English Lake District, employing the Corpus of the Lake District Writing. The method utilizes a generative pre-trained transformer model to extract spatial relations from the textual descriptions in the corpus. The study applies this large language model to understand the spatial dimensions inherent in historical narratives comprehensively. The outcomes are presented as semantic triples, capturing the nuanced connections between entities and locations, and visualized as a network, offering a graphical representation of the spatial narrative. The study contributes to a deeper comprehension of the English Lake District's spatial tapestry and provides an approach to uncovering spatial relations within diverse historical contexts.
△ Less
Submitted 20 June, 2024;
originally announced June 2024.
-
Dishonesty in Helpful and Harmless Alignment
Authors:
Youcheng Huang,
Jingkun Tang,
Duanyu Feng,
Zheng Zhang,
Wenqiang Lei,
Jiancheng Lv,
Anthony G. Cohn
Abstract:
People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference. We find that this also induces dishonesty in helpful and harmless alignment where LLMs tell lies in generating harmless responses. Using the latest interpreting tools, we detect dishonesty, show how LLMs can be harmful…
▽ More
People tell lies when seeking rewards. Large language models (LLMs) are aligned to human values with reinforcement learning where they get rewards if they satisfy human preference. We find that this also induces dishonesty in helpful and harmless alignment where LLMs tell lies in generating harmless responses. Using the latest interpreting tools, we detect dishonesty, show how LLMs can be harmful if their honesty is increased, and analyze such conflicts at the parameter-level. Given these preliminaries and the hypothesis that reward-seeking stimulates dishonesty, we theoretically show that the dishonesty can in-turn decrease the alignment performances and augment reward-seeking alignment with representation regularization. Extensive results, including GPT-4 annotated win-rates, perplexities, and cases studies demonstrate that we can train more honest, helpful, and harmless LLMs. We will make all our codes and results be open-sourced upon this paper's acceptance.
△ Less
Submitted 5 June, 2024; v1 submitted 3 June, 2024;
originally announced June 2024.
-
Reframing Spatial Reasoning Evaluation in Language Models: A Real-World Simulation Benchmark for Qualitative Reasoning
Authors:
Fangjun Li,
David C. Hogg,
Anthony G. Cohn
Abstract:
Spatial reasoning plays a vital role in both human cognition and machine intelligence, prompting new research into language models' (LMs) capabilities in this regard. However, existing benchmarks reveal shortcomings in evaluating qualitative spatial reasoning (QSR). These benchmarks typically present oversimplified scenarios or unclear natural language descriptions, hindering effective evaluation.…
▽ More
Spatial reasoning plays a vital role in both human cognition and machine intelligence, prompting new research into language models' (LMs) capabilities in this regard. However, existing benchmarks reveal shortcomings in evaluating qualitative spatial reasoning (QSR). These benchmarks typically present oversimplified scenarios or unclear natural language descriptions, hindering effective evaluation. We present a novel benchmark for assessing QSR in LMs, which is grounded in realistic 3D simulation data, offering a series of diverse room layouts with various objects and their spatial relationships. This approach provides a more detailed and context-rich narrative for spatial reasoning evaluation, diverging from traditional, toy-task-oriented scenarios. Our benchmark encompasses a broad spectrum of qualitative spatial relationships, including topological, directional, and distance relations. These are presented with different viewing points, varied granularities, and density of relation constraints to mimic real-world complexities. A key contribution is our logic-based consistency-checking tool, which enables the assessment of multiple plausible solutions, aligning with real-world scenarios where spatial relationships are often open to interpretation. Our benchmark evaluation of advanced LMs reveals their strengths and limitations in spatial reasoning. They face difficulties with multi-hop spatial reasoning and interpreting a mix of different view descriptions, pointing to areas for future improvement.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
-
Advancing Spatial Reasoning in Large Language Models: An In-Depth Evaluation and Enhancement Using the StepGame Benchmark
Authors:
Fangjun Li,
David C. Hogg,
Anthony G. Cohn
Abstract:
Artificial intelligence (AI) has made remarkable progress across various domains, with large language models like ChatGPT gaining substantial attention for their human-like text-generation capabilities. Despite these achievements, spatial reasoning remains a significant challenge for these models. Benchmarks like StepGame evaluate AI spatial reasoning, where ChatGPT has shown unsatisfactory perfor…
▽ More
Artificial intelligence (AI) has made remarkable progress across various domains, with large language models like ChatGPT gaining substantial attention for their human-like text-generation capabilities. Despite these achievements, spatial reasoning remains a significant challenge for these models. Benchmarks like StepGame evaluate AI spatial reasoning, where ChatGPT has shown unsatisfactory performance. However, the presence of template errors in the benchmark has an impact on the evaluation results. Thus there is potential for ChatGPT to perform better if these template errors are addressed, leading to more accurate assessments of its spatial reasoning capabilities. In this study, we refine the StepGame benchmark, providing a more accurate dataset for model evaluation. We analyze GPT's spatial reasoning performance on the rectified benchmark, identifying proficiency in mapping natural language text to spatial relations but limitations in multi-hop reasoning. We provide a flawless solution to the benchmark by combining template-to-relation mapping with logic-based reasoning. This combination demonstrates proficiency in performing qualitative reasoning on StepGame without encountering any errors. We then address the limitations of GPT models in spatial reasoning. We deploy Chain-of-thought and Tree-of-thoughts prompting strategies, offering insights into GPT's ``cognitive process", and achieving remarkable improvements in accuracy. Our investigation not only sheds light on model deficiencies but also proposes enhancements, contributing to the advancement of AI with more robust spatial reasoning capabilities.
△ Less
Submitted 8 January, 2024;
originally announced January 2024.
-
Language Models as a Service: Overview of a New Paradigm and its Challenges
Authors:
Emanuele La Malfa,
Aleksandar Petrov,
Simon Frieder,
Christoph Weinhuber,
Ryan Burnell,
Raza Nazar,
Anthony G. Cohn,
Nigel Shadbolt,
Michael Wooldridge
Abstract:
Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, b…
▽ More
Some of the most powerful language models currently are proprietary systems, accessible only via (typically restrictive) web or software programming interfaces. This is the Language-Models-as-a-Service (LMaaS) paradigm. In contrast with scenarios where full model access is available, as in the case of open-source models, such closed-off language models present specific challenges for evaluating, benchmarking, and testing them. This paper has two goals: on the one hand, we delineate how the aforementioned challenges act as impediments to the accessibility, replicability, reliability, and trustworthiness of LMaaS. We systematically examine the issues that arise from a lack of information about language models for each of these four aspects. We conduct a detailed analysis of existing solutions and put forth a number of considered recommendations, and highlight the directions for future advancements. On the other hand, it serves as a comprehensive resource for existing knowledge on current, major LMaaS, offering a synthesized overview of the licences and capabilities their interfaces offer.
△ Less
Submitted 30 November, 2023; v1 submitted 28 September, 2023;
originally announced September 2023.
-
An Evaluation of ChatGPT-4's Qualitative Spatial Reasoning Capabilities in RCC-8
Authors:
Anthony G Cohn
Abstract:
Qualitative Spatial Reasoning (QSR) is well explored area of Commonsense Reasoning and has multiple applications ranging from Geographical Information Systems to Robotics and Computer Vision. Recently many claims have been made for the capabilities of Large Language Models (LLMs). In this paper we investigate the extent to which one particular LLM can perform classical qualitative spatial reasonin…
▽ More
Qualitative Spatial Reasoning (QSR) is well explored area of Commonsense Reasoning and has multiple applications ranging from Geographical Information Systems to Robotics and Computer Vision. Recently many claims have been made for the capabilities of Large Language Models (LLMs). In this paper we investigate the extent to which one particular LLM can perform classical qualitative spatial reasoning tasks on the mereotopological calculus, RCC-8.
△ Less
Submitted 27 September, 2023;
originally announced September 2023.
-
Dialectical language model evaluation: An initial appraisal of the commonsense spatial reasoning abilities of LLMs
Authors:
Anthony G Cohn,
Jose Hernandez-Orallo
Abstract:
Language models have become very popular recently and many claims have been made about their abilities, including for commonsense reasoning. Given the increasingly better results of current language models on previous static benchmarks for commonsense reasoning, we explore an alternative dialectical evaluation. The goal of this kind of evaluation is not to obtain an aggregate performance value but…
▽ More
Language models have become very popular recently and many claims have been made about their abilities, including for commonsense reasoning. Given the increasingly better results of current language models on previous static benchmarks for commonsense reasoning, we explore an alternative dialectical evaluation. The goal of this kind of evaluation is not to obtain an aggregate performance value but to find failures and map the boundaries of the system. Dialoguing with the system gives the opportunity to check for consistency and get more reassurance of these boundaries beyond anecdotal evidence. In this paper we conduct some qualitative investigations of this kind of evaluation for the particular case of spatial reasoning (which is a fundamental aspect of commonsense reasoning). We conclude with some suggestions for future work both to improve the capabilities of language models and to systematise this kind of dialectical evaluation.
△ Less
Submitted 22 April, 2023;
originally announced April 2023.
-
Object-agnostic Affordance Categorization via Unsupervised Learning of Graph Embeddings
Authors:
Alexia Toumpa,
Anthony G. Cohn
Abstract:
Acquiring knowledge about object interactions and affordances can facilitate scene understanding and human-robot collaboration tasks. As humans tend to use objects in many different ways depending on the scene and the objects' availability, learning object affordances in everyday-life scenarios is a challenging task, particularly in the presence of an open set of interactions and objects. We addre…
▽ More
Acquiring knowledge about object interactions and affordances can facilitate scene understanding and human-robot collaboration tasks. As humans tend to use objects in many different ways depending on the scene and the objects' availability, learning object affordances in everyday-life scenarios is a challenging task, particularly in the presence of an open set of interactions and objects. We address the problem of affordance categorization for class-agnostic objects with an open set of interactions; we achieve this by learning similarities between object interactions in an unsupervised way and thus inducing clusters of object affordances. A novel depth-informed qualitative spatial representation is proposed for the construction of Activity Graphs (AGs), which abstract from the continuous representation of spatio-temporal interactions in RGB-D videos. These AGs are clustered to obtain groups of objects with similar affordances. Our experiments in a real-world scenario demonstrate that our method learns to create object affordance clusters with a high V-measure even in cluttered scenes. The proposed approach handles object occlusions by capturing effectively possible interactions and without imposing any object or scene constraints.
△ Less
Submitted 30 March, 2023;
originally announced April 2023.
-
A Hierarchical Framework for Collaborative Artificial Intelligence
Authors:
James L. Crowley,
Joëlle L Coutaz,
Jasmin Grosinger,
Javier Vázquez-Salceda,
Cecilio Angulo,
Alberto Sanfeliu,
Luca Iocchi,
Anthony G. Cohn
Abstract:
We propose a hierarchical framework for collaborative intelligent systems. This framework organizes research challenges based on the nature of the collaborative activity and the information that must be shared, with each level building on capabilities provided by lower levels. We review research paradigms at each level, with a description of classical engineering-based approaches and modern altern…
▽ More
We propose a hierarchical framework for collaborative intelligent systems. This framework organizes research challenges based on the nature of the collaborative activity and the information that must be shared, with each level building on capabilities provided by lower levels. We review research paradigms at each level, with a description of classical engineering-based approaches and modern alternatives based on machine learning, illustrated with a running example using a hypothetical personal service robot. We discuss cross-cutting issues that occur at all levels, focusing on the problem of communicating and sharing comprehension, the role of explanation and the social nature of collaboration. We conclude with a summary of research challenges and a discussion of the potential for economic and societal impact provided by technologies that enhance human abilities and empower people and society through collaboration with Intelligent Systems.
△ Less
Submitted 14 December, 2022;
originally announced December 2022.
-
Exploring the GLIDE model for Human Action-effect Prediction
Authors:
Fangjun Li,
David C. Hogg,
Anthony G. Cohn
Abstract:
We address the following action-effect prediction task. Given an image depicting an initial state of the world and an action expressed in text, predict an image depicting the state of the world following the action. The prediction should have the same scene context as the input image. We explore the use of the recently proposed GLIDE model for performing this task. GLIDE is a generative neural net…
▽ More
We address the following action-effect prediction task. Given an image depicting an initial state of the world and an action expressed in text, predict an image depicting the state of the world following the action. The prediction should have the same scene context as the input image. We explore the use of the recently proposed GLIDE model for performing this task. GLIDE is a generative neural network that can synthesize (inpaint) masked areas of an image, conditioned on a short piece of text. Our idea is to mask-out a region of the input image where the effect of the action is expected to occur. GLIDE is then used to inpaint the masked region conditioned on the required action. In this way, the resulting image has the same background context as the input image, updated to show the effect of the action. We give qualitative results from experiments using the EPIC dataset of ego-centric videos labelled with actions.
△ Less
Submitted 1 August, 2022;
originally announced August 2022.
-
Refactoring the Whitby Intelligent Tutoring System for Clean Architecture
Authors:
Paul S. Brown,
Vania Dimitrova,
Glen Hart,
Anthony G. Cohn,
Paulo Moura
Abstract:
Whitby is the server-side of an Intelligent Tutoring System application for learning System-Theoretic Process Analysis (STPA), a methodology used to ensure the safety of anything that can be represented with a systems model. The underlying logic driving the reasoning behind Whitby is Situation Calculus, which is a many-sorted logic with situation, action, and object sorts. The Situation Calculus i…
▽ More
Whitby is the server-side of an Intelligent Tutoring System application for learning System-Theoretic Process Analysis (STPA), a methodology used to ensure the safety of anything that can be represented with a systems model. The underlying logic driving the reasoning behind Whitby is Situation Calculus, which is a many-sorted logic with situation, action, and object sorts. The Situation Calculus is applied to Ontology Authoring and Contingent Scaffolding: the primary activities within Whitby. Thus many fluents and actions are aggregated in Whitby from these two sub-applications and from Whitby itself, but all are available through a common situation query interface that does not depend upon any of the fluents or actions. Each STPA project in Whitby is a single situation term, which is queried for fluents that include the ontology, and to determine what pedagogical interventions to offer.
Initially Whitby was written in Prolog using a module system. In the interest of a cleaner architecture and implementation with improved code reuse and extensibility, the initial application was refactored into Logtalk. This refactoring includes decoupling the Situation Calculus reasoner, Ontology Authoring framework, and Contingent Scaffolding framework into third-party libraries that can be reused in other applications. This extraction was achieved by inverting dependencies via Logtalk protocols and categories, which are reusable interfaces and components that provide functionally cohesive sets of predicate declarations and predicate definitions. In this paper the architectures of two iterations of Whitby are evaluated with respect to the motivations behind the refactor: clean architecture enabling code reuse and extensibility.
△ Less
Submitted 10 August, 2021;
originally announced August 2021.
-
Spatial representability of neuronal activity
Authors:
D. Akhtiamov,
A. G. Cohn,
Y. Dabaghian
Abstract:
A common approach to interpreting spiking activity is based on identifying the firing fields---regions in physical or configuration spaces that elicit responses of neurons. Common examples include hippocampal place cells that fire at preferred locations in the navigated environment, head direction cells that fire at preferred orientations of the animal's head, view cells that respond to preferred…
▽ More
A common approach to interpreting spiking activity is based on identifying the firing fields---regions in physical or configuration spaces that elicit responses of neurons. Common examples include hippocampal place cells that fire at preferred locations in the navigated environment, head direction cells that fire at preferred orientations of the animal's head, view cells that respond to preferred spots in the visual field, etc. In all these cases, firing fields were discovered empirically, by trial and error. We argue that the existence and a number of properties of the firing fields can be established theoretically, through topological analyses of the neuronal spiking activity.
△ Less
Submitted 7 August, 2021;
originally announced August 2021.
-
Scribble-Supervised Semantic Segmentation by Uncertainty Reduction on Neural Representation and Self-Supervision on Neural Eigenspace
Authors:
Zhiyi Pan,
Peng Jiang,
Yunhai Wang,
Changhe Tu,
Anthony G. Cohn
Abstract:
Scribble-supervised semantic segmentation has gained much attention recently for its promising performance without high-quality annotations. Due to the lack of supervision, confident and consistent predictions are usually hard to obtain. Typically, people handle these problems to either adopt an auxiliary task with the well-labeled dataset or incorporate the graphical model with additional require…
▽ More
Scribble-supervised semantic segmentation has gained much attention recently for its promising performance without high-quality annotations. Due to the lack of supervision, confident and consistent predictions are usually hard to obtain. Typically, people handle these problems to either adopt an auxiliary task with the well-labeled dataset or incorporate the graphical model with additional requirements on scribble annotations. Instead, this work aims to achieve semantic segmentation by scribble annotations directly without extra information and other limitations. Specifically, we propose holistic operations, including minimizing entropy and a network embedded random walk on neural representation to reduce uncertainty. Given the probabilistic transition matrix of a random walk, we further train the network with self-supervision on its neural eigenspace to impose consistency on predictions between related images. Comprehensive experiments and ablation studies verify the proposed approach, which demonstrates superiority over others; it is even comparable to some full-label supervised ones and works well when scribbles are randomly shrunk or dropped.
△ Less
Submitted 19 February, 2021;
originally announced February 2021.
-
Defect segmentation: Mapping tunnel lining internal defects with ground penetrating radar data using a convolutional neural network
Authors:
Senlin Yang,
Zhengfang Wang,
Jing Wang,
Anthony G. Cohn,
Jiaqi Zhang,
Peng Jiang,
Peng Jiang,
Qingmei Sui
Abstract:
This research proposes a Ground Penetrating Radar (GPR) data processing method for non-destructive detection of tunnel lining internal defects, called defect segmentation. To perform this critical step of automatic tunnel lining detection, the method uses a CNN called Segnet combined with the Lovász softmax loss function to map the internal defect structure with GPR synthetic data, which improves…
▽ More
This research proposes a Ground Penetrating Radar (GPR) data processing method for non-destructive detection of tunnel lining internal defects, called defect segmentation. To perform this critical step of automatic tunnel lining detection, the method uses a CNN called Segnet combined with the Lovász softmax loss function to map the internal defect structure with GPR synthetic data, which improves the accuracy, automation and efficiency of defects detection. The novel method we present overcomes several difficulties of traditional GPR data interpretation as demonstrated by an evaluation on both synthetic and real datas -- to verify the method on real data, a test model containing a known defect was designed and built and GPR data was obtained and analyzed.
△ Less
Submitted 29 March, 2020;
originally announced March 2020.
-
Human-like Planning for Reaching in Cluttered Environments
Authors:
Mohamed Hasan,
Matthew Warburton,
Wisdom C. Agboh,
Mehmet R. Dogar,
Matteo Leonetti,
He Wang,
Faisal Mushtaq,
Mark Mon-Williams,
Anthony G. Cohn
Abstract:
Humans, in comparison to robots, are remarkably adept at reaching for objects in cluttered environments. The best existing robot planners are based on random sampling of configuration space -- which becomes excessively high-dimensional with large number of objects. Consequently, most planners often fail to efficiently find object manipulation plans in such environments. We addressed this problem b…
▽ More
Humans, in comparison to robots, are remarkably adept at reaching for objects in cluttered environments. The best existing robot planners are based on random sampling of configuration space -- which becomes excessively high-dimensional with large number of objects. Consequently, most planners often fail to efficiently find object manipulation plans in such environments. We addressed this problem by identifying high-level manipulation plans in humans, and transferring these skills to robot planners. We used virtual reality to capture human participants reaching for a target object on a tabletop cluttered with obstacles. From this, we devised a qualitative representation of the task space to abstract the decision making, irrespective of the number of obstacles. Based on this representation, human demonstrations were segmented and used to train decision classifiers. Using these classifiers, our planner produced a list of waypoints in task space. These waypoints provided a high-level plan, which could be transferred to an arbitrary robot model and used to initialise a local trajectory optimiser. We evaluated this approach through testing on unseen human VR data, a physics-based robot simulation, and a real robot (dataset and code are publicly available). We found that the human-like planner outperformed a state-of-the-art standard trajectory optimisation algorithm, and was able to generate effective strategies for rapid planning -- irrespective of the number of obstacles in the environment.
△ Less
Submitted 3 March, 2020; v1 submitted 28 February, 2020;
originally announced February 2020.
-
GPRInvNet: Deep Learning-Based Ground Penetrating Radar Data Inversion for Tunnel Lining
Authors:
Bin Liu,
Yuxiao Ren,
Hanchi Liu,
Hui Xu,
Zhengfang Wang,
Anthony G. Cohn,
Peng Jiang
Abstract:
A DNN architecture referred to as GPRInvNet was proposed to tackle the challenges of mapping the ground-penetrating radar (GPR) B-Scan data to complex permittivity maps of subsurface structures. The GPRInvNet consisted of a trace-to-trace encoder and a decoder. It was specially designed to take into account the characteristics of GPR inversion when faced with complex GPR B-Scan data, as well as ad…
▽ More
A DNN architecture referred to as GPRInvNet was proposed to tackle the challenges of mapping the ground-penetrating radar (GPR) B-Scan data to complex permittivity maps of subsurface structures. The GPRInvNet consisted of a trace-to-trace encoder and a decoder. It was specially designed to take into account the characteristics of GPR inversion when faced with complex GPR B-Scan data, as well as addressing the spatial alignment issues between time-series B-Scan data and spatial permittivity maps. It displayed the ability to fuse features from several adjacent traces on the B-Scan data to enhance each trace, and then further condense the features of each trace separately. As a result, the sensitive zones on the permittivity maps spatially aligned to the enhanced trace could be reconstructed accurately. The GPRInvNet has been utilized to reconstruct the permittivity map of tunnel linings. A diverse range of dielectric models of tunnel linings containing complex defects has been reconstructed using GPRInvNet. The results have demonstrated that the GPRInvNet is capable of effectively reconstructing complex tunnel lining defects with clear boundaries. Comparative results with existing baseline methods also demonstrated the superiority of the GPRInvNet. For the purpose of generalizing the GPRInvNet to real GPR data, some background noise patches recorded from practical model testing were integrated into the synthetic GPR data to retrain the GPRInvNet. The model testing has been conducted for validation, and experimental results revealed that the GPRInvNet had also achieved satisfactory results with regard to the real data.
△ Less
Submitted 26 September, 2021; v1 submitted 11 December, 2019;
originally announced December 2019.
-
ViTac: Feature Sharing between Vision and Tactile Sensing for Cloth Texture Recognition
Authors:
Shan Luo,
Wenzhen Yuan,
Edward Adelson,
Anthony G. Cohn,
Raul Fuentes
Abstract:
Vision and touch are two of the important sensing modalities for humans and they offer complementary information for sensing the environment. Robots could also benefit from such multi-modal sensing ability. In this paper, addressing for the first time (to the best of our knowledge) texture recognition from tactile images and vision, we propose a new fusion method named Deep Maximum Covariance Anal…
▽ More
Vision and touch are two of the important sensing modalities for humans and they offer complementary information for sensing the environment. Robots could also benefit from such multi-modal sensing ability. In this paper, addressing for the first time (to the best of our knowledge) texture recognition from tactile images and vision, we propose a new fusion method named Deep Maximum Covariance Analysis (DMCA) to learn a joint latent space for sharing features through vision and tactile sensing. The features of camera images and tactile data acquired from a GelSight sensor are learned by deep neural networks. But the learned features are of a high dimensionality and are redundant due to the differences between the two sensing modalities, which deteriorates the perception performance. To address this, the learned features are paired using maximum covariance analysis. Results of the algorithm on a newly collected dataset of paired visual and tactile data relating to cloth textures show that a good recognition performance of greater than 90\% can be achieved by using the proposed DMCA framework. In addition, we find that the perception performance of either vision or tactile sensing can be improved by employing the shared representation space, compared to learning from unimodal data.
△ Less
Submitted 13 March, 2018; v1 submitted 21 February, 2018;
originally announced February 2018.
-
CLAD: A Complex and Long Activities Dataset with Rich Crowdsourced Annotations
Authors:
Jawad Tayyub,
Majd Hawasly,
David C. Hogg,
Anthony G. Cohn
Abstract:
This paper introduces a novel activity dataset which exhibits real-life and diverse scenarios of complex, temporally-extended human activities and actions. The dataset presents a set of videos of actors performing everyday activities in a natural and unscripted manner. The dataset was recorded using a static Kinect 2 sensor which is commonly used on many robotic platforms. The dataset comprises of…
▽ More
This paper introduces a novel activity dataset which exhibits real-life and diverse scenarios of complex, temporally-extended human activities and actions. The dataset presents a set of videos of actors performing everyday activities in a natural and unscripted manner. The dataset was recorded using a static Kinect 2 sensor which is commonly used on many robotic platforms. The dataset comprises of RGB-D images, point cloud data, automatically generated skeleton tracks in addition to crowdsourced annotations. Furthermore, we also describe the methodology used to acquire annotations through crowdsourcing. Finally some activity recognition benchmarks are presented using current state-of-the-art techniques. We believe that this dataset is particularly suitable as a testbed for activity recognition research but it can also be applicable for other common tasks in robotics/computer vision research such as object detection and human skeleton tracking.
△ Less
Submitted 21 September, 2017; v1 submitted 11 September, 2017;
originally announced September 2017.
-
The STRANDS Project: Long-Term Autonomy in Everyday Environments
Authors:
Nick Hawes,
Chris Burbridge,
Ferdian Jovan,
Lars Kunze,
Bruno Lacerda,
Lenka Mudrová,
Jay Young,
Jeremy Wyatt,
Denise Hebesberger,
Tobias Körtner,
Rares Ambrus,
Nils Bore,
John Folkesson,
Patric Jensfelt,
Lucas Beyer,
Alexander Hermans,
Bastian Leibe,
Aitor Aldoma,
Thomas Fäulhammer,
Michael Zillich,
Markus Vincze,
Eris Chinellato,
Muhannad Al-Omari,
Paul Duckworth,
Yiannis Gatsoulis
, et al. (8 additional authors not shown)
Abstract:
Thanks to the efforts of the robotics and autonomous systems community, robots are becoming ever more capable. There is also an increasing demand from end-users for autonomous service robots that can operate in real environments for extended periods. In the STRANDS project we are tackling this demand head-on by integrating state-of-the-art artificial intelligence and robotics research into mobile…
▽ More
Thanks to the efforts of the robotics and autonomous systems community, robots are becoming ever more capable. There is also an increasing demand from end-users for autonomous service robots that can operate in real environments for extended periods. In the STRANDS project we are tackling this demand head-on by integrating state-of-the-art artificial intelligence and robotics research into mobile service robots, and deploying these systems for long-term installations in security and care environments. Over four deployments, our robots have been operational for a combined duration of 104 days autonomously performing end-user defined tasks, covering 116km in the process. In this article we describe the approach we have used to enable long-term autonomous operation in everyday environments, and how our robots are able to use their long run times to improve their own performance.
△ Less
Submitted 14 October, 2016; v1 submitted 15 April, 2016;
originally announced April 2016.
-
Reasoning with Topological and Directional Spatial Information
Authors:
Sanjiang Li,
Anthony G. Cohn
Abstract:
Current research on qualitative spatial representation and reasoning mainly focuses on one single aspect of space. In real world applications, however, multiple spatial aspects are often involved simultaneously.
This paper investigates problems arising in reasoning with combined topological and directional information. We use the RCC8 algebra and the Rectangle Algebra (RA) for expressing topol…
▽ More
Current research on qualitative spatial representation and reasoning mainly focuses on one single aspect of space. In real world applications, however, multiple spatial aspects are often involved simultaneously.
This paper investigates problems arising in reasoning with combined topological and directional information. We use the RCC8 algebra and the Rectangle Algebra (RA) for expressing topological and directional information respectively. We give examples to show that the bipath-consistency algorithm BIPATH is incomplete for solving even basic RCC8 and RA constraints. If topological constraints are taken from some maximal tractable subclasses of RCC8, and directional constraints are taken from a subalgebra, termed DIR49, of RA, then we show that BIPATH is able to separate topological constraints from directional ones. This means, given a set of hybrid topological and directional constraints from the above subclasses of RCC8 and RA, we can transfer the joint satisfaction problem in polynomial time to two independent satisfaction problems in RCC8 and RA. For general RA constraints, we give a method to compute solutions that satisfy all topological constraints and approximately satisfy each RA constraint to any prescribed precision.
△ Less
Submitted 1 September, 2009;
originally announced September 2009.
-
Topological Maps from Signals
Authors:
Yu. Dabaghian,
A. G. Cohn,
L. Frank
Abstract:
We discuss the task of reconstructing the topological map of an environment based on the sequences of locations visited by a mobile agent -- this occurs in systems neuroscience, where one runs into the task of reconstructing the global topological map of the environment based on activation patterns of the place coding cells in hippocampus area of the brain. A similar task appears in the context…
▽ More
We discuss the task of reconstructing the topological map of an environment based on the sequences of locations visited by a mobile agent -- this occurs in systems neuroscience, where one runs into the task of reconstructing the global topological map of the environment based on activation patterns of the place coding cells in hippocampus area of the brain. A similar task appears in the context of establishing wifi connectivity maps.
△ Less
Submitted 23 December, 2007;
originally announced December 2007.
-
Topological coding in hippocampus
Authors:
Yu. Dabaghian,
A. G. Cohn,
L. Frank
Abstract:
The proposed analysis of the currently available experimental results concerning the neural cell activity in the brain area known as hippocampus suggests a particular mechanism of spatial information and memory processing. Below it is argued that the spatial information available through the analysis of the hippocampal cell activity is predominantly of topological nature. It is pointed out that…
▽ More
The proposed analysis of the currently available experimental results concerning the neural cell activity in the brain area known as hippocampus suggests a particular mechanism of spatial information and memory processing. Below it is argued that the spatial information available through the analysis of the hippocampal cell activity is predominantly of topological nature. It is pointed out that a direct topological analysis can produce a topological invariant based classification of the cell activity patterns and a complete topological description of animal's current environment. It also provides a full first order logical system for local topological reasoning about spatial structure and animal's navigational strategies.
△ Less
Submitted 25 February, 2007;
originally announced February 2007.
