-
PEGASUS: Physically Enhanced Gaussian Splatting Simulation System for 6DoF Object Pose Dataset Generation
Authors:
Lukas Meyer,
Floris Erich,
Yusuke Yoshiyasu,
Marc Stamminger,
Noriaki Ando,
Yukiyasu Domae
Abstract:
We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DOF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting.
Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. <i>PEGASUS</i> allows the composition of new scenes by merging the respective underlying…
▽ More
We introduce Physically Enhanced Gaussian Splatting Simulation System (PEGASUS) for 6DOF object pose dataset generation, a versatile dataset generator based on 3D Gaussian Splatting.
Environment and object representations can be easily obtained using commodity cameras to reconstruct with Gaussian Splatting. <i>PEGASUS</i> allows the composition of new scenes by merging the respective underlying Gaussian Splatting point cloud of an environment with one or multiple objects. Leveraging a physics engine enables the simulation of natural object placement within a scene through interaction between meshes extracted for the objects and the environment. Consequently, an extensive amount of new scenes - static or dynamic - can be created by combining different environments and objects. By rendering scenes from various perspectives, diverse data points such as RGB images, depth maps, semantic masks, and 6DoF object poses can be extracted.
Our study demonstrates that training on data generated by PEGASUS enables pose estimation networks to successfully transfer from synthetic data to real-world data. Moreover, we introduce the Ramen dataset, comprising 30 Japanese cup noodle items. This dataset includes spherical scans that captures images from both object hemisphere and the Gaussian Splatting reconstruction, making them compatible with PEGASUS.
△ Less
Submitted 15 July, 2024; v1 submitted 4 January, 2024;
originally announced January 2024.
-
NeuralLabeling: A versatile toolset for labeling vision datasets using Neural Radiance Fields
Authors:
Floris Erich,
Naoya Chiba,
Yusuke Yoshiyasu,
Noriaki Ando,
Ryo Hanai,
Yukiyasu Domae
Abstract:
We present NeuralLabeling, a labeling approach and toolset for annotating a scene using either bounding boxes or meshes and generating segmentation masks, affordance maps, 2D bounding boxes, 3D bounding boxes, 6DOF object poses, depth maps and object meshes. NeuralLabeling uses Neural Radiance Fields (NeRF) as renderer, allowing labeling to be performed using 3D spatial tools while incorporating g…
▽ More
We present NeuralLabeling, a labeling approach and toolset for annotating a scene using either bounding boxes or meshes and generating segmentation masks, affordance maps, 2D bounding boxes, 3D bounding boxes, 6DOF object poses, depth maps and object meshes. NeuralLabeling uses Neural Radiance Fields (NeRF) as renderer, allowing labeling to be performed using 3D spatial tools while incorporating geometric clues such as occlusions, relying only on images captured from multiple viewpoints as input. To demonstrate the applicability of NeuralLabeling to a practical problem in robotics, we added ground truth depth maps to 30000 frames of transparent object RGB and noisy depth maps of glasses placed in a dishwasher captured using an RGBD sensor, yielding the Dishwasher30k dataset. We show that training a simple deep neural network with supervision using the annotated depth maps yields a higher reconstruction performance than training with the previously applied weakly supervised approach.
△ Less
Submitted 21 September, 2023;
originally announced September 2023.
-
Motion Priority Optimization Framework towards Automated and Teleoperated Robot Cooperation in Industrial Recovery Scenarios
Authors:
Shunki Itadera,
Yukiyasu Domae
Abstract:
In this study, we introduce an optimization framework aimed at enhancing the efficiency of motion priority design in scenarios involving automated and teleoperated robots within an industrial recovery context. The escalating utilization of industrial robots at manufacturing sites has been instrumental in mitigating human workload. Nevertheless, the challenge persists in achieving effective human-r…
▽ More
In this study, we introduce an optimization framework aimed at enhancing the efficiency of motion priority design in scenarios involving automated and teleoperated robots within an industrial recovery context. The escalating utilization of industrial robots at manufacturing sites has been instrumental in mitigating human workload. Nevertheless, the challenge persists in achieving effective human-robot collaboration/cooperation where human workers and robots share a workspace for collaborative tasks. In the event of an industrial robot encountering a failure, it necessitates the suspension of the corresponding factory cell for safe recovery. Given the limited capacity of pre-programmed robots to rectify such failures, human intervention becomes imperative, requiring entry into the robot workspace to address the dropped object while the robot system is halted. This non-continuous manufacturing process results in productivity loss. Robotic teleoperation has emerged as a promising technology enabling human workers to undertake high-risk tasks remotely and safely. Our study advocates for the incorporation of robotic teleoperation in the recovery process during manufacturing failure scenarios, which is referred to as "Cooperative Tele-Recovery". Our proposed approach involves the formulation of priority rules designed to facilitate collision avoidance between manufacturing and recovery robots. This, in turn, ensures a continuous manufacturing process with minimal production loss within a configurable risk limitation. We present a comprehensive motion priority optimization framework, encompassing an HRC simulator-based priority optimization and a cooperative multi-robot controller, to identify optimal parameters for the priority function. The framework dynamically adjusts the allocation of motion priorities for manufacturing and recovery robots while adhering to predefined risk limitations.
△ Less
Submitted 23 January, 2024; v1 submitted 29 August, 2023;
originally announced August 2023.
-
A Closed-Loop Bin Picking System for Entangled Wire Harnesses using Bimanual and Dynamic Manipulation
Authors:
Xinyi Zhang,
Yukiyasu Domae,
Weiwei Wan,
Kensuke Harada
Abstract:
This paper addresses the challenge of industrial bin picking using entangled wire harnesses. Wire harnesses are essential in manufacturing but poses challenges in automation due to their complex geometries and propensity for entanglement. Our previous work tackled this issue by proposing a quasi-static pulling motion to separate the entangled wire harnesses. However, it still lacks sufficiency and…
▽ More
This paper addresses the challenge of industrial bin picking using entangled wire harnesses. Wire harnesses are essential in manufacturing but poses challenges in automation due to their complex geometries and propensity for entanglement. Our previous work tackled this issue by proposing a quasi-static pulling motion to separate the entangled wire harnesses. However, it still lacks sufficiency and generalization to various shapes and structures. In this paper, we deploy a dual-arm robot that can grasp, extract and disentangle wire harnesses from dense clutter using dynamic manipulation. The robot can swing to dynamically discard the entangled objects and regrasp to adjust the undesirable grasp pose. To improve the robustness and accuracy of the system, we leverage a closed-loop framework that uses haptic feedback to detect entanglement in real-time and flexibly adjust system parameters. Our bin picking system achieves an overall success rate of 91.2% in the real-world experiments using two different types of long wire harnesses. It demonstrates the effectiveness of our system in handling various wire harnesses for industrial bin picking.
△ Less
Submitted 26 June, 2023;
originally announced June 2023.
-
Force Map: Learning to Predict Contact Force Distribution from Vision
Authors:
Ryo Hanai,
Yukiyasu Domae,
Ixchel G. Ramirez-Alpizar,
Bruno Leme,
Tetsuya Ogata
Abstract:
When humans see a scene, they can roughly imagine the forces applied to objects based on their experience and use them to handle the objects properly. This paper considers transferring this "force-visualization" ability to robots. We hypothesize that a rough force distribution (named "force map") can be utilized for object manipulation strategies even if accurate force estimation is impossible. Ba…
▽ More
When humans see a scene, they can roughly imagine the forces applied to objects based on their experience and use them to handle the objects properly. This paper considers transferring this "force-visualization" ability to robots. We hypothesize that a rough force distribution (named "force map") can be utilized for object manipulation strategies even if accurate force estimation is impossible. Based on this hypothesis, we propose a training method to predict the force map from vision. To investigate this hypothesis, we generated scenes where objects were stacked in bulk through simulation and trained a model to predict the contact force from a single image. We further applied domain randomization to make the trained model function on real images. The experimental results showed that the model trained using only synthetic images could predict approximate patterns representing the contact areas of the objects even for real images. Then, we designed a simple algorithm to plan a lifting direction using the predicted force distribution. We confirmed that using the predicted force distribution contributes to finding natural lifting directions for typical real-world scenes. Furthermore, the evaluation through simulations showed that the disturbance caused to surrounding objects was reduced by 26 % (translation displacement) and by 39 % (angular displacement) for scenes where objects were overlapping.
△ Less
Submitted 12 April, 2023;
originally announced April 2023.
-
Learning to Dexterously Pick or Separate Tangled-Prone Objects for Industrial Bin Picking
Authors:
Xinyi Zhang,
Yukiyasu Domae,
Weiwei Wan,
Kensuke Harada
Abstract:
Industrial bin picking for tangled-prone objects requires the robot to either pick up untangled objects or perform separation manipulation when the bin contains no isolated objects. The robot must be able to flexibly perform appropriate actions based on the current observation. It is challenging due to high occlusion in the clutter, elusive entanglement phenomena, and the need for skilled manipula…
▽ More
Industrial bin picking for tangled-prone objects requires the robot to either pick up untangled objects or perform separation manipulation when the bin contains no isolated objects. The robot must be able to flexibly perform appropriate actions based on the current observation. It is challenging due to high occlusion in the clutter, elusive entanglement phenomena, and the need for skilled manipulation planning. In this paper, we propose an autonomous, effective and general approach for picking up tangled-prone objects for industrial bin picking. First, we learn PickNet - a network that maps the visual observation to pixel-wise possibilities of picking isolated objects or separating tangled objects and infers the corresponding grasp. Then, we propose two effective separation strategies: Dropping the entangled objects into a buffer bin to reduce the degree of entanglement; Pulling to separate the entangled objects in the buffer bin planned by PullNet - a network that predicts position and direction for pulling from visual input. To efficiently collect data for training PickNet and PullNet, we embrace the self-supervised learning paradigm using an algorithmic supervisor in a physics simulator. Real-world experiments show that our policy can dexterously pick up tangled-prone objects with success rates of 90%. We further demonstrate the generalization of our policy by picking a set of unseen objects. Supplementary material, code, and videos can be found at https://xinyiz0931.github.io/tangle.
△ Less
Submitted 7 July, 2023; v1 submitted 16 February, 2023;
originally announced February 2023.
-
Learning Efficient Policies for Picking Entangled Wire Harnesses: An Approach to Industrial Bin Picking
Authors:
Xinyi Zhang,
Yukiyasu Domae,
Weiwei Wan,
Kensuke Harada
Abstract:
Wire harnesses are essential connecting components in manufacturing industry but are challenging to be automated in industrial tasks such as bin picking. They are long, flexible and tend to get entangled when randomly placed in a bin. This makes it difficult for the robot to grasp a single one in dense clutter. Besides, training or collecting data in simulation is challenging due to the difficulti…
▽ More
Wire harnesses are essential connecting components in manufacturing industry but are challenging to be automated in industrial tasks such as bin picking. They are long, flexible and tend to get entangled when randomly placed in a bin. This makes it difficult for the robot to grasp a single one in dense clutter. Besides, training or collecting data in simulation is challenging due to the difficulties in modeling the combination of deformable and rigid components for wire harnesses. In this work, instead of directly lifting wire harnesses, we propose to grasp and extract the target following a circle-like trajectory until it is untangled. We learn a policy from real-world data that can infer grasps and separation actions from visual observation. Our policy enables the robot to efficiently pick and separate entangled wire harnesses by maximizing success rates and reducing execution time. To evaluate our policy, we present a set of real-world experiments on picking wire harnesses. Our policy achieves an overall 84.6% success rate compared with 49.2% in baseline. We also evaluate the effectiveness of our policy under different clutter scenarios using unseen types of wire harnesses. Results suggest that our approach is feasible for handling wire harnesses in industrial bin picking.
△ Less
Submitted 7 January, 2023; v1 submitted 11 December, 2021;
originally announced December 2021.
-
Material Classification Using Active Temperature Controllable Robotic Gripper
Authors:
Yukiko Osawa,
Kei Kase,
Yukiyasu Domae,
Yoshiyuki Furukawa,
Abderrahmane Kheddar
Abstract:
Recognition techniques allow robots to make proper planning and control strategies to manipulate various objects. Object recognition is more reliable when made by combining several percepts, e.g., vision and haptics. One of the distinguishing features of each object's material is its heat properties, and classification can exploit heat transfer, similarly to human thermal sensation. Thermal-based…
▽ More
Recognition techniques allow robots to make proper planning and control strategies to manipulate various objects. Object recognition is more reliable when made by combining several percepts, e.g., vision and haptics. One of the distinguishing features of each object's material is its heat properties, and classification can exploit heat transfer, similarly to human thermal sensation. Thermal-based recognition has the advantage of obtaining contact surface information in realtime by simply capturing temperature change using a tiny and cheap sensor. However, heat transfer between a robot surface and a contact object is strongly affected by the initial temperature and environmental conditions. A given object's material cannot be recognized when its temperature is the same as the robotic grippertip. We present a material classification system using active temperature controllable robotic gripper to induce heat flow. Subsequently, our system can recognize materials independently from their ambient temperature. The robotic gripper surface can be regulated to any temperature that differentiates it from the touched object's surface. We conducted some experiments by integrating the temperature control system with the Academic SCARA Robot, classifying them based on a long short-term memory (LSTM) using temperature data obtained from grasping target objects.
△ Less
Submitted 30 November, 2021;
originally announced November 2021.
-
A Topological Solution of Entanglement for Complex-shaped Parts in Robotic Bin-picking
Authors:
Xinyi Zhang,
Keisuke Koyama,
Yukiyasu Domae,
Weiwei Wan,
Kensuke Harada
Abstract:
This paper addresses the problem of picking up only one object at a time avoiding any entanglement in bin-picking. To cope with a difficult case where the complex-shaped objects are heavily entangled together, we propose a topology-based method that can generate non-tangle grasp positions on a single depth image. The core technique is entanglement map, which is a feature map to measure the entangl…
▽ More
This paper addresses the problem of picking up only one object at a time avoiding any entanglement in bin-picking. To cope with a difficult case where the complex-shaped objects are heavily entangled together, we propose a topology-based method that can generate non-tangle grasp positions on a single depth image. The core technique is entanglement map, which is a feature map to measure the entanglement possibilities obtained from the input image. We use the entanglement map to select probable regions containing graspable objects. The optimum grasping pose is detected from the selected regions considering the collision between robot hand and objects. Experimental results show that our analytic method provides a more comprehensive and intuitive observation of entanglement and exceeds previous learning-based work in success rates. Especially, our topology-based method does not rely on any object models or time-consuming training process, so that it can be easily adapted to more complex bin-picking scenes.
△ Less
Submitted 28 February, 2022; v1 submitted 2 June, 2021;
originally announced June 2021.
-
Hierarchical Pyramid Representations for Semantic Segmentation
Authors:
Hiroaki Aizawa,
Yukihiro Domae,
Kunihito Kato
Abstract:
Understanding the context of complex and cluttered scenes is a challenging problem for semantic segmentation. However, it is difficult to model the context without prior and additional supervision because the scene's factors, such as the scale, shape, and appearance of objects, vary considerably in these scenes. To solve this, we propose to learn the structures of objects and the hierarchy among o…
▽ More
Understanding the context of complex and cluttered scenes is a challenging problem for semantic segmentation. However, it is difficult to model the context without prior and additional supervision because the scene's factors, such as the scale, shape, and appearance of objects, vary considerably in these scenes. To solve this, we propose to learn the structures of objects and the hierarchy among objects because context is based on these intrinsic properties. In this study, we design novel hierarchical, contextual, and multiscale pyramidal representations to capture the properties from an input image. Our key idea is the recursive segmentation in different hierarchical regions based on a predefined number of regions and the aggregation of the context in these regions. The aggregated contexts are used to predict the contextual relationship between the regions and partition the regions in the following hierarchical level. Finally, by constructing the pyramid representations from the recursively aggregated context, multiscale and hierarchical properties are attained. In the experiments, we confirmed that our proposed method achieves state-of-the-art performance in PASCAL Context.
△ Less
Submitted 5 April, 2021;
originally announced April 2021.
-
Planning a Sequence of Base Positions for a Mobile Manipulator to Perform Multiple Pick-and-Place Tasks
Authors:
Jingren Xu,
Yukiyasu Domae,
Toshio Ueshiba,
Weiwei Wan,
Kensuke Harada
Abstract:
In this paper, we present a planner that plans a sequence of base positions for a mobile manipulator to efficiently and robustly collect objects stored in distinct trays. We achieve high efficiency by exploring the common areas where a mobile manipulator can grasp objects stored in multiple trays simultaneously and move the mobile manipulator to the common areas to reduce the time needed for movin…
▽ More
In this paper, we present a planner that plans a sequence of base positions for a mobile manipulator to efficiently and robustly collect objects stored in distinct trays. We achieve high efficiency by exploring the common areas where a mobile manipulator can grasp objects stored in multiple trays simultaneously and move the mobile manipulator to the common areas to reduce the time needed for moving the mobile base. We ensure robustness by optimizing the base position with the best clearance to positioning uncertainty so that a mobile manipulator can complete the task even if there is a certain deviation from the planned base positions. Besides, considering different styles of object placement in the tray, we analyze feasible schemes for dynamically updating the base positions based on either the remaining objects or the target objects to be picked in one round of the tasks. In the experiment part, we examine our planner on various scenarios, including different object placement: (1) Regularly placed toy objects; (2) Randomly placed industrial parts; and different schemes for online execution: (1) Apply globally static base positions; (2) Dynamically update the base positions. The experiment results demonstrate the efficiency, robustness and feasibility of the proposed method.
△ Less
Submitted 30 September, 2020;
originally announced October 2020.
-
Development of a Shape-memorable Adaptive Pin Array Fixture
Authors:
Peihao Shi,
Zhengtao Hu,
Kazuyuki Nagata,
Weiwei Wan,
Yukiyasu Domae,
Kensuke Harada
Abstract:
This paper proposes an adaptive pin-array fixture. The key idea of this research is to use the shape-memorable mechanism of pin array to fix multiple different shaped parts with common pin configuration. The clamping area consists of a matrix of passively slid-able pins that conform themselves to the contour of the target object. Vertical motion of the pins enables the fixture to encase the profil…
▽ More
This paper proposes an adaptive pin-array fixture. The key idea of this research is to use the shape-memorable mechanism of pin array to fix multiple different shaped parts with common pin configuration. The clamping area consists of a matrix of passively slid-able pins that conform themselves to the contour of the target object. Vertical motion of the pins enables the fixture to encase the profile of the object. The shape memorable mechanism is realized by the combination of the rubber bush and fixing mechanism of a pin. Several physical peg-in-hole tasks is conducted to verify the feasibility of the fixture.
△ Less
Submitted 20 May, 2020;
originally announced May 2020.
-
Selecting and Designing Grippers for an Assembly Task in a Structured Approach
Authors:
Jingren Xu,
Weiwei Wan,
Keisuke Koyama,
Yukiyasu Domae,
Kensuke Harada
Abstract:
In this paper, we present a structured approach to selecting and designing a set of grippers for an assembly task. Compared to current experience-based gripper design method, our approach accelerates the design process by automatically generating a set of initial design options on gripper type and parameters according to the CAD models of assembly components. We use mesh segmentation techniques to…
▽ More
In this paper, we present a structured approach to selecting and designing a set of grippers for an assembly task. Compared to current experience-based gripper design method, our approach accelerates the design process by automatically generating a set of initial design options on gripper type and parameters according to the CAD models of assembly components. We use mesh segmentation techniques to segment the assembly components and fit the segmented parts with shape primitives, according to the predefined correspondence between primitive shape and gripper type, suitable gripper types and parameters can be selected and extracted from the fitted shape primitives. Moreover, we incorporate the assembly constraints in the further evaluation of the initially obtained gripper types and parameters. Considering the affordance of the segmented parts and the collision avoidance between the gripper and the subassemblies, applicable gripper types and parameters can be filtered from the initial options. Among the applicable gripper configurations, we further optimize number of grippers for performing the assembly task, by exploring the gripper that is able to handle multiple assembly components during the assembly. Finally, the feasibility of the designed grippers is experimentally verified by assembling a part of an industrial product.
△ Less
Submitted 4 January, 2021; v1 submitted 9 March, 2020;
originally announced March 2020.
-
Team O2AS at the World Robot Summit 2018: An Approach to Robotic Kitting and Assembly Tasks using General Purpose Grippers and Tools
Authors:
Felix von Drigalski,
Chisato Nakashima,
Yoshiya Shibata,
Yoshinori Konishi,
Joshua C. Triyonoputro,
Kaidi Nie,
Damien Petit,
Toshio Ueshiba,
Ryuichi Takase,
Yukiyasu Domae,
Taku Yoshioka,
Yoshihisa Ijiri,
Ixchel G. Ramirez-Alpizar,
Weiwei Wan,
Kensuke Harada
Abstract:
We propose a versatile robotic system for kitting and assembly tasks which uses no jigs or commercial tool changers. Instead of specialized end effectors, it uses its two-finger grippers to grasp and hold tools to perform subtasks such as screwing and suctioning. A third gripper is used as a precision picking and centering tool, and uses in-built passive compliance to compensate for small position…
▽ More
We propose a versatile robotic system for kitting and assembly tasks which uses no jigs or commercial tool changers. Instead of specialized end effectors, it uses its two-finger grippers to grasp and hold tools to perform subtasks such as screwing and suctioning. A third gripper is used as a precision picking and centering tool, and uses in-built passive compliance to compensate for small position errors and uncertainty. A novel grasp point detection for bin picking is described for the kitting task, using a single depth map. Using the proposed system we competed in the Assembly Challenge of the Industrial Robotics Category of the World Robot Challenge at the World Robot Summit 2018, obtaining 4th place and the SICE award for lean design and versatile tool use. We show the effectiveness of our approach through experiments performed during the competition.
△ Less
Submitted 4 March, 2020;
originally announced March 2020.
-
Planning an Efficient and Robust Base Sequence for a Mobile Manipulator Performing Multiple Pick-and-place Tasks
Authors:
Jingren Xu,
Kensuke Harada,
Weiwei Wan,
Toshio Ueshiba,
Yukiyasu Domae
Abstract:
In this paper, we address efficiently and robustly collecting objects stored in different trays using a mobile manipulator. A resolution complete method, based on precomputed reachability database, is proposed to explore collision-free inverse kinematics (IK) solutions and then a resolution complete set of feasible base positions can be determined. This method approximates a set of representative…
▽ More
In this paper, we address efficiently and robustly collecting objects stored in different trays using a mobile manipulator. A resolution complete method, based on precomputed reachability database, is proposed to explore collision-free inverse kinematics (IK) solutions and then a resolution complete set of feasible base positions can be determined. This method approximates a set of representative IK solutions that are especially helpful when solving IK and checking collision are treated separately. For real world applications, we take into account the base positioning uncertainty and plan a sequence of base positions that reduce the number of necessary base movements for collecting the target objects, the base sequence is robust in that the mobile manipulator is able to complete the part-supply task even there is certain deviation from the planned base positions. Our experiments demonstrate both the efficiency compared to regular base sequence and the feasibility in real world applications.
△ Less
Submitted 9 March, 2020; v1 submitted 22 January, 2020;
originally announced January 2020.
-
Learning Based Industrial Bin-picking Trained with Approximate Physics Simulator
Authors:
Ryo Matsumura,
Kensuke Harada,
Yukiyasu Domae,
Weiwei Wan
Abstract:
In this research, we tackle the problem of picking an object from randomly stacked pile. Since complex physical phenomena of contact among objects and fingers makes it difficult to perform the bin-picking with high success rate, we consider introducing a learning based approach. For the purpose of collecting enough number of training data within a reasonable period of time, we introduce a physics…
▽ More
In this research, we tackle the problem of picking an object from randomly stacked pile. Since complex physical phenomena of contact among objects and fingers makes it difficult to perform the bin-picking with high success rate, we consider introducing a learning based approach. For the purpose of collecting enough number of training data within a reasonable period of time, we introduce a physics simulator where approximation is used for collision checking. In this paper, we first formulate the learning based robotic bin-picking by using CNN (Convolutional Neural Network). We also obtain the optimum grasping posture of parallel jaw gripper by using CNN. Finally, we show that the effect of approximation introduced in collision checking is relaxed if we use exact 3D model to generate the depth image of the pile as an input to CNN.
△ Less
Submitted 22 May, 2018;
originally announced May 2018.
-
3D Object Discovery and Modeling Using Single RGB-D Images Containing Multiple Object Instances
Authors:
Wim Abbeloos,
Esra Ataer-Cansizoglu,
Sergio Caccamo,
Yuichi Taguchi,
Yukiyasu Domae
Abstract:
Unsupervised object modeling is important in robotics, especially for handling a large set of objects. We present a method for unsupervised 3D object discovery, reconstruction, and localization that exploits multiple instances of an identical object contained in a single RGB-D image. The proposed method does not rely on segmentation, scene knowledge, or user input, and thus is easily scalable. Our…
▽ More
Unsupervised object modeling is important in robotics, especially for handling a large set of objects. We present a method for unsupervised 3D object discovery, reconstruction, and localization that exploits multiple instances of an identical object contained in a single RGB-D image. The proposed method does not rely on segmentation, scene knowledge, or user input, and thus is easily scalable. Our method aims to find recurrent patterns in a single RGB-D image by utilizing appearance and geometry of the salient regions. We extract keypoints and match them in pairs based on their descriptors. We then generate triplets of the keypoints matching with each other using several geometric criteria to minimize false matches. The relative poses of the matched triplets are computed and clustered to discover sets of triplet pairs with similar relative poses. Triplets belonging to the same set are likely to belong to the same object and are used to construct an initial object model. Detection of remaining instances with the initial object model using RANSAC allows to further expand and refine the model. The automatically generated object models are both compact and descriptive. We show quantitative and qualitative results on RGB-D images with various objects including some from the Amazon Picking Challenge. We also demonstrate the use of our method in an object picking scenario with a robotic arm.
△ Less
Submitted 17 October, 2017;
originally announced October 2017.
