3D Feature Distillation with Object-Centric Priors

Abstract

Grounding natural language to the physical world is a ubiquitous topic with a wide range of applications in computer vision and robotics. Recently, 2D vision-language models such as CLIP have been widely popularized, due to their impressive capabilities for open-vocabulary grounding in 2D images. Recent works aim to elevate 2D CLIP features to 3D via feature distillation, but either learn neural fields that are scene-specific and hence lack generalization, or focus on indoor room scan data that require access to multiple camera views, which is not practical in robot manipulation scenarios. Additionally, related methods typically fuse features at pixel-level and assume that all camera views are equally informative. In this work, we show that this approach leads to sub-optimal 3D features, both in terms of grounding accuracy, as well as segmentation crispness. To alleviate this, we propose a multi-view feature fusion strategy that employs object-centric priors to eliminate uninformative views based on semantic information, and fuse features at object-level via instance segmentation masks. To distill our object-centric 3D features, we generate a large-scale synthetic multi-view dataset of cluttered tabletop scenes, spawning 15k scenes from over 3300 unique object instances, which we make publicly available. We show that our method reconstructs 3D CLIP features with improved grounding capacity and spatial consistency, while doing so from single-view RGB-D, thus departing from the assumption of multiple camera views at test time. Finally, we show that our approach can generalize to novel tabletop domains and be re-purposed for 3D instance segmentation without fine-tuning, and demonstrate its utility for language-guided robotic grasping in clutter.

Figure 1: Visualization of 3D features (middle), back-projected 2D features (left) and user query similarity heatmaps (right), for previous SOTA point-cloud feature distillation method OpenScene and our DROP-CLIP. OpenScene fuses pixel-wise 2D features across all views with average pooling, leading to grounding failures, segmentation imprecisions and fuzzy object boundaries. Our method resolves these issues by employing object-centric priors to fuse object-level 2D features in 3D instance masks with semantics-informed view selection.

Keywords: Open-Vocabulary 3D Segmentation, Multi-view Feature Distillation

1 Introduction

Language grounding in 3D environments plays a crucial role in realizing intelligent systems that can interact naturally with the physical world. In the robotics field, being able to precisely segment desired objects in 3D based on open language queries (object semantics, visual attributes, affordances, etc.) can serve as a powerful proxy for enabling open-ended robot manipulation. As a result, research focus on 3D segmentation methods has seen growth in recent years [1, 2, 3, 4, 5, 6]. However, related methods fall in the closed-vocabulary regime, where only a fixed list of classes can be used as queries. Inspired by the success of open-vocabulary 2D methods  [7, 8, 9, 10], recent efforts elevate 2D representations from pretrained models  [7, 11] to 3D via distillation pipelines  [12, 13, 14, 15, 16, 17, 18, 19]. However, we identify certain limitations of existing distillation approaches. On the one hand, field-based methods  [13, 20, 16, 17, 18] offer continuous 3D feature fields, but require to be trained online in specific scenes and hence cannot generalize to novel object instances and compositions, they require a few minutes to train, and need to collect multiple camera views before training, all of which hinder their real-time applicability. On the other hand, original 3D feature distillation methods and follow up work [12, 14, 21] use room scan datasets  [22, 23] to learn point-cloud encoders, hence being applicable in novel scenes with open vocabularies. However, such approaches assume that 2D features from all views are equally informative, which is not the case in highly cluttered indoors scenes (e.g. due to partial occlusions from some view), thus leading in noisy 3D features. 2D features are also usually fused point-wise from ViT patches  [9, 10, 8] or multi-scale crops [13, 6], therefore leading to the so called “patchyness” issue [24] (see Fig. 1). The latter issue is especially impactful in robot manipulation, where precise 3D segmentation is vital for specifying robust actuation goals.

To address such limitations, we revisit 2D $\rightarrow$ 3D point-based feature distillation but revise the multi-view feature fusion strategy to enhance the quality of the target 3D features. In particular, we inject both semantic and spatial object-centric priors into the fusion strategy, in three ways: (i) We obtain object-level 2D features by isolating object instances in each camera view from their 2D segmentation masks, (ii) we fuse features only at corresponding 3D object regions using 3D segmentation masks, (iii) we leverage object-level semantic information to devise an informativeness metric, which is used to weight the contribution of views and eliminate uninformative ones. Extensive ablation studies demonstrate the advantages of object-centric fusion compared to vanilla approaches. To train our method, we require a large-scale cluttered indoors dataset with many views per scene, which is currently not existent. To that end, we build MV-TOD (Multi-View Tabletop Objects Dataset), consisting of $\sim 15k$ Blender scenes from more than $3.3k$ unique 3D object models, for which we provide $73$ views per scene with $360^{\circ}$ coverage, further equipped with 2D/3D segmentations, 6-DoF grasps and textual object-level annotations. We use MV-TOD to distill our object-centric 3D CLIP  [7] features into a 3D representation, which we call DROP-CLIP (Distilled Representations with Object-centric Priors from CLIP). Our 3D encoder operates in partial point-clouds from a single RGB-D view, thus departing from the requirement of multiple camera images at test time, while offering real-time inference capabilities. We demonstrate that our learned 3D features achieve high grounding performance and segmentation crispness, while significantly outperforming previous 2D open-vocabulary approaches in the single-view setting. Further, we show that they can be leveraged zero-shot in novel tabletop domains, as well as be used out-of-the-box for 3D instance segmentation.

In summary, our contributions are fourfold: (i) we release MV-TOD, a large-scale synthetic dataset of household objects in cluttered tabletop scenarios, featuring dense multi-view coverage and semantic/mask/grasp annotations, (ii) we identify limitations of current multi-view feature fusion approaches and illustrate how to overcome them by leveraging object-centric priors, (iii) we release DROP-CLIP, a 3D model that reconstructs view-independent 3D CLIP features from single-view, and (iv) we conduct extensive ablation studies, comparative experiments and robot demonstrations to showcase the effectiveness of the proposed method in terms of 3D segmentation performance, generalization to novel domains and tasks, and applicability in robot manipulation scenarios.

2 Multi-View Tabletop Objects Dataset

Figure 2: (Left:) Example generated Blender scene, multi-view image coverage and annotations included in our dataset. (Right:) Automatic semantic annotation generation with large vision-language models.

Dataset	Layout	Multi	Clutter	Vision	Ref.Expr.	Grasp	Num.Obj.	Num.	Num.	Obj.-lvl
		View		Data	Annot.	Annot.	Categories	Scenes	Expr.	Semantics
ScanNet [22]	indoor	✔	-	RGB-D,3D	✗	✗	$17$	$800$	$-$	✗
S3DIS [25]	indoor	✔	-	RGB-D,3D	✗	✗	$13$	$6$	$-$	✗
Replica [26]	indoor	✔	-	RGB-D,3D	✗	✗	$88$	$-$	$-$	✔
STPLS3D [25]	outdoor	✔	-	3D	✗	✗	$12$	$18$	$-$	✔
ScanRefer [1]	indoor	✔	✗	RGB-D,3D	2D/3D mask	✗	$18$	$800$	$51.5k$	✗
ReferIt-3D [2]	indoor	✔	✗	RGB-D,3D	2D/3D mask	✗	$18$	$707$	$125.5k$	✗
ReferIt-RGBD [27]	indoor	✔	✗	RGB-D	2D box	✗	-	$7.6k$	$38.4k$	✗
SunSpot [28]	indoor	✗	✔	RGB-D	2D box	✗	38	$1.9k$	$7.0k$	✗
GraspNet [29]	tabletop	✗	✔	3D	✗	6-DoF	$88$	$190$	$-$	✗
REGRAD [30]	tabletop	✔	✔	RGB-D,3D	✗	6-DoF	$55$	$47k$	$-$	✗
OCID-VLG [31]	tabletop	✗	✔	RGB-D,3D	2D mask	4-DoF	$31$	$1.7k$	$89.6k$	template
Grasp-Anything [32]	tabletop	✗	✗	RGB	2D mask	4-DoF	$236$	$1M$	$-$	open
MV-TOD (ours)	tabletop	✔	✔	RGB-D,3D	3D mask	6-DoF	$149$	$15k$	$671.2k$	open

Table 1: Comparisons between MV-TOD and existing 3D datasets.

Existing 3D datasets mainly focus on indoor scenes in room layouts [33, 22, 26] and related language annotations typically cover closed-set object categories (e.g. furniture) and spatial relations [1, 2, 27, 34, 28], which are not practical for robot manipulation tasks, where cluttered tabletop scenarios and open-vocabulary language are of key importance. On the other hand, recent grasp-related efforts collect cluttered tabletop scenes, but either lack language annotations [30, 35, 29] or connect cluttered scenes with language but only for 4-DoF grasps with RGB data  [31, 32]. Further, most of such datasets lack dense multi-view scene coverage, granting them non applicable for 2D $\rightarrow$ 3D feature distillation, where we require multiple images from each scene to extract 2D features with a foundation model. To cover this gap, we propose MV-TOD, a large-scale synthetic dataset with cluttered tabletop scenes featuring dense multi-view coverage and rich language annotations at the object level. We generate a total of $15k$ scenes in Blender [36], comprising of 3379 unique object models, $99$ collected by us and the rest filtered from ShapeNet-Sem model set  [37]. The dataset features $149$ object categories, each of which includes multiple instances that vary in fine-grained details. For each object instance, we leverage GPT-4-Vision [38] to generate open-set descriptions from various perspectives, including category, color, material, state, utility, affordance, etc, which spawn over $670k$ unique referring instance queries (see Fig. 2-right and Appendix A). For each scene, we provide 2D/3D segmentation masks, 6D object poses, as well as a set of semantic concepts for each appearing object instance. Additionally, we include 6-DoF grasp annotations for each object model, originating from the ACRONYM dataset  [35]. To the best of our knowledge, MV-TOD is the first dataset to combine 3D cluttered tabletop scenes with open-vocabulary language and 6-DoF grasp annotations, which we hope will accelerate future research.

3 Methodology

Figure 3: Method Overview: Given a 3D scene and multiple camera views, we employ three object-centric priors (in red) for multi-view feature fusion: (i) extract CLIP features from 2D masked object crops, (ii) use semantic annotations to fuse 2D features across views, (iii) apply the fused feature on all points in the object’s 3D mask. The fused feature-cloud is distilled with a single-view posed RGB-D encoder and cosine distance loss. During inference, we compute point-wise cosine similarity scores in CLIP space (higher similarity towards red).

Our goal is to distill multi-view 2D CLIP features into a 3D representation, while employing an object-centric feature fusion strategy to ensure high quality 3D features. Our overall pipeline is illustrated in Fig 3. We first introduce traditional multi-view feature fusion (Sec. 3.1), present our variant with object-centric priors (Sec. 3.2) and discuss our feature distillation method (Sec. 3.3).

3.1 Multi-view 2D Feature Fusion

We assume access to a dataset of 3D scenes, where each scene is represented through a set of $\mathcal{V}$ posed RGB-D views of size $H\times W$: $\left\{I_{v}\in\mathbb{R}^{H\times W\times 3},\,D_{v}\in\mathbb{R}^{H\times W},\,T_{v}\in\mathbb{R}^{4\times 4}\right\}_{v=1}^{\mathcal{V}}$, where $T_{v}$ the camera pose from view $v$. For each scene, we first obtain the full point-cloud $P\in\mathbb{R}^{M\times 3}$ along with a 2D-3D correspondence map $\mathcal{M}_{v}\in\mathbb{R}^{M\times 2}$, mapping each point $\mathbf{x}_{i},\,i=1,\dots,M$ to a pixel location $\mathbf{u}_{v,i}=(u_{x},u_{y})^{T}$ in image view $I_{v}$. RGB images are fed to a pretrained image model $f^{2D}:\mathbb{R}^{H\times W\times 3}\rightarrow\mathbb{R}^{H\times W\times C}$ [9, 10, 8] to obtain pixel-level 2D features of size C: $Z^{2D}_{v}=f^{2D}(I_{v})$, which are then back-projected to 3D points via:

$$\textbf{z}^{2D}_{v,i}=f^{2D}\left(I_{v}(\textbf{u}_{v,i})\right)=f^{2D}\left(I_{v}(\mathcal{M}_{v}(\mathbf{x}_{i}))\right)$$ (1)

To fuse 2D features across views $Z^{3D}\in\mathbb{R}^{M\times C}$, previous works  [12, 6, 14, 21] use average pooling: $\mathbf{z}_{i}^{3D}=\frac{1}{\mathcal{V}}\sum_{v=1}^{\mathcal{V}}\mathbf{z}_{v,i}^{2D}$ (see Appendix B for a comprehensive overview). In essence, this method assumes that all views are equally informative for each point, as long as the point is visible from that view. We suggest that naively average pooling 2D features for each point leads to sub-optimal 3D features, as noisy, uninformative views contribute equally, therefore “polluting" the overall representation. We then propose to instead use a generalized version relying on weighted average:

$$\mathbf{z}^{3D}_{i}=\frac{\sum_{v=1}^{\mathcal{V}}\mathbf{z}^{2D}_{v,i}\cdot\omega_{v,i}}{\sum_{v=1}^{\mathcal{V}}\omega_{v,i}}$$ (2)

where $\omega_{v,i}\in\mathbb{R}$ is a scalar weight that represents the informativeness of view $v$ for point $i$. In the next subsection, we describe how to use text data to dynamically compute an informativeness weight for each view based on semantic object-level semantic information. Additionally, vanilla point-wise fusion with pixel-level 2D features leads to non-crisp segmentations and fuzzy object boundaries. To resolve this, we propose to also leverage dense spatial information, i.e., instance-wise 2D/3D segmentation masks, which are used for both: (a) obtaining robust object-level 2D CLIP features from each view, and (b) fusing features only at the points corresponding to the 3D object region.

3.2 Employing Object-Centric Priors

Let $\left\{S_{v}^{2D}\in\{0,1\}^{N\times H\times W}\right\}_{v=1}^{\mathcal{V}}$ be 2D instance-wise segmentation masks for each scene, where $N$ the total number of scene objects. We aggregate the 2D masks to obtain $S^{3D}\in\{0,1\}^{M\times N}$, such that for each point $i$ we can retrieve the corresponding object instance $n_{i}=\texttt{argmax}_{\nu}\;S^{3D}_{i,\nu}$.

Semantic informativeness metric Let $\mathcal{Q}=\left\{Q_{k}\right\}_{k=1}^{\mathcal{K}},\;Q_{k}\in\mathbb{R}^{N_{k}\times C}$ be a set of object-specific textual prompts, where $\mathcal{K}$ the number of dataset object instances and $N_{k}$ the number of prompts for object $k$. We use CLIP’s text encoder to embed the textual prompts in $\mathbb{R}^{C}$ and average them to obtain an object-specific prompt $\textbf{q}_{k}=1/N_{k}\cdot\sum_{j=1}^{N_{k}}Q_{k,j}$. For each scene, we map each object instance $n\in[1,N]$ to its positive prompt $\textbf{q}_{n}^{+}$, as well as a set $Q^{-}_{n}\doteq\mathcal{Q}-\{\textbf{q}_{n}^{+}\}$ of negative prompts corresponding to all other instances. We define our semantic informativeness metric as:

$$G_{v,i}=\texttt{cos}(\textbf{z}^{2D}_{v,i},\textbf{q}^{+}_{n_{i}})-\texttt{max}_{\textbf{q}\sim Q^{-}_{n_{i}}}\texttt{cos}(\textbf{z}^{2D}_{v,i},\textbf{q})$$ (3)

Intuitively, we want a 2D feature from view $v$ to contribute to the overall 3D feature of point $i$ according to how much its similarity with the correct object instance is higher than the maximum similarity to any of the negative object instances, hence offering a proxy for semantic informativeness. We clip this weight to 0 to eliminate views that don’t satisfy the condition $G_{v,i}\geq 0$. Plugging in our metric in equation (2) already provides improvements over vanilla average pooling (see Sec. 4.1), however, does not deal with 3D spatial consistency, for which we employ our spatial priors below.

Object-level 2D CLIP features For obtaining object-level 2D CLIP features, we isolate the pixels for each object $n$ from each view $v$ from $S^{2D}_{v,n}$ and crop a bounding box around the mask from $I_{v}$: $\textbf{z}^{2D}_{v,n}=f^{2D}_{cls}\left(\texttt{cropmask}(I_{v},\,S^{2D}_{v,n})\right)$ (see Appendix C for ablations in CLIP visual prompts). Here we use $f^{2D}_{cls}:\mathbb{R}^{h_{n}\times w_{n}\times 3}\rightarrow\mathbb{R}^{C}$, i.e., only the [CLS] feature of CLIP’s ViT encoder, to represent an object crop of size $h_{n}\times w_{n}$. We can now define our metric from equation (3) also at object-level:

$$G_{v,n}=\texttt{cos}(\textbf{z}^{2D}_{v,n},\textbf{q}^{+}_{n})-\texttt{max}_{\textbf{q}\sim Q^{-}_{n}}\texttt{cos}(\textbf{z}^{2D}_{v,n},\textbf{q})$$ (4)

where $G_{v,n}\in\mathbb{R}^{\mathcal{V}\times N}$ now represents the semantic informativeness of view $v$ for object instance $n$.

Fusing object-wise features A 3D object-level feature can be obtained by fusing 2D object-level features across views similar to equation (2):

$$\mathbf{z}^{3D}_{n}=\frac{\sum_{v=1}^{\mathcal{V}}\mathbf{z}^{2D}_{v,n}\cdot\omega_{v,n}}{\sum_{v=1}^{\mathcal{V}}\omega_{v,n}}=\frac{\sum_{v=1}^{\mathcal{V}}\mathbf{z}^{2D}_{v,n}\cdot\Lambda_{v,n}\cdot G_{v,n}}{\sum_{v=1}^{\mathcal{V}}\Lambda_{v,n}\cdot G_{v,n}}$$ (5)

where each view is weighted by its semantic informativeness metric $G_{v,n}$, as well as optionally a visibility metric $\Lambda_{v,n}=\sum_{\textbf{}}S^{2D}_{v,n}$ that measures the number of pixels from $n$-th object’s mask that are visible from view $v$  [6]. We finally reconstruct the full feature-cloud $Z^{3D}\in\mathbb{R}^{M\times C}$ by equating each point’s feature to its corresponding 3D object-level one via: $\mathbf{z}^{3D}_{i}=\mathbf{z}^{3D}_{n_{i}},\;n_{i}=\texttt{argmax}_{\nu}\;S^{3D}_{i,\nu}$.

3.3 View-Independent Feature Distillation

Even though the above feature-cloud $Z^{3D}$ could be directly used for open-vocabulary grounding in 3D, its construction is computationally intensive and requires a lot of expensive resources, such as access to multiple camera views, view-aligned 2D instance segmentation masks, as well as a set of text descriptions to compute informativeness metrics. Such utilities are rarely available in open-ended scenarios, especially in robotic applications, where usually only single-view RGB-D images from sensors mounted on the robot are provided. To tackle this, we wish to distill all the above knowledge from the feature-cloud $Z^{3D}$ into a 3D encoder that receives only a partial point-cloud from single-view posed RGB-D. Hence, the only assumption that we make during inference is access to camera intrinsic and extrinsic parameters, which is a mild requirement in most robotic works.

In particular, given a partial colored point-cloud from view $v$: $P_{v}\in\mathbb{R}^{M_{v}\times 6}$ (3D coordinates plus colors), we train a 3D encoder $\mathcal{E}_{\theta}:\mathbb{R}^{M_{v}\times 6}\rightarrow\mathbb{R}^{M_{v}\times C}$ such that $\mathcal{E}_{\theta}(P_{v})=Z^{3D}$. Notice that the distillation target $Z^{3D}$ is independent of view $v$. Following  [12, 15] we use cosine distance loss:

$$\mathcal{L}(\theta)=1-\texttt{cos}(\mathcal{E_{\theta}}(P_{v}),\,Z^{3D})$$ (6)

See Appendix B.2 for training implementation details. With such a setup, we can obtain 3D features that: (i) are co-embedded in CLIP text space, so they can be leveraged for 3D segmentation tasks from open-vocabulary queries via computing cosine similarities between CLIP text embeddings $Q$ and the predicted feature cloud: $\hat{S}_{i}=\texttt{argmax}_{i}\;\texttt{cos}(\hat{\mathbf{z}}_{i}^{3D},Q)$, (ii) are ensured to be optimally informative per object, due to the usage of the semantic informativeness metric to compute $Z^{3D}$, (iii) maintain 3D spatial consistency in object boundaries, due to performing object-wise instead of point-wise fusion when computing $Z^{3D}$, and (iv) are encouraged to be view-independent, as the same features $Z^{3D}$ are utilized as distillation targets regardless of the input view $v$. Importantly, no labels, prompts, or segmentation masks are needed at test-time to reproduce the fused feature-cloud, while obtaining it amounts to a single forward pass of our 3D encoder, hence offering real-time performance.

Figure 4: Open-Vocabulary 3D Referring Segmentation. Examples of learned 3D features and grounding heatmaps from open-ended language queries (class names, attributes, user affordances, and open instance-specific concepts) in scenes from MV-TOD dataset. Points are colored based on their query similarity (higher towards red). We note that table points are excluded from similarity computation in our visualizations.

4 Experiments

In our experiments, we explore the following questions: (i) Sec. 4.1: What are the contributions of our proposed object-centric priors for multi-view feature fusion? Does the dense number of views of our proposed dataset also contribute? (ii) Sec. 4.2: How does our method compare to previous open-vocabulary approaches for 3D semantic and referring segmentation tasks? Are the learned features robust to open-ended language? (iii) Sec. 4.3: What are the generalization capabilities of our learned 3D representation in novel domains and novel tasks (3D instance segmentation)? (iv) Sec. 4.4: Can we leverage our 3D learned representation for language-guided 6-DoF robotic grasping?

4.1 Multi-view Feature Fusion Ablation Studies

Fusion	$\mathbf{f^{2D}}$	$\mathbf{\Lambda_{v,i}}$	$\mathbf{G_{v,i}}$	Ref.Segm (%)
				mIoU	Pr@25	Pr@50	Pr@75
point	patch			44.2	59.9	41.4	27.0
point	patch	✓		37.3	55.4	33.7	16.7
point	patch		✓	57.0	74.1	59.5	40.9
point	patch	✓	✓	57.4	77.0	60.9	39.9
obj	obj			65.6	67.0	65.4	64.1
obj	obj	✓		67.3	68.7	67.1	65.8
obj	obj		✓	83.1	83.9	83.1	82.4
obj	obj	✓	✓	80.9	83.1	80.2	79.7

Table 2: Multi-view feature fusion ablation study for 3D referring segmentation in MV-TOD.

To evaluate the contributions of our proposed object-centric priors, we conduct ablation studies on the multi-view feature fusion pipeline, where we compare 3D referring segmentation results of obtained 3D features in held-out scenes of MV-TOD. We highlight that here we aim to establish a performance upper bound that the feature fusion method can provide for distillation, and not the distilled features themselves.

Figure 5: Referring segmentation accuracy (Pr@25 (%)) vs. number of utilized views.

We ablate: (i) patch-wise vs. object-wise fusion, (ii) MaskCLIP [8] patch-level vs. CLIP [7] masked crop features, (iii) inclusion of visibility ($\Lambda_{v,i}$) and semantic informativeness ($G_{v,i}$) metrics for view selection. Results in Table 2.

Effect of object-centric priors We observe that all components contribute positively to the quality of the 3D features. Our proposed $G_{v,i}$ metric boosts mIoU across both point- and object-wise fusion ($57.0\%$ vs. $44.2\%$ and $83.1\%$ vs. $65.6\%$ respectively). Further, we observe that the usage of spatial priors for object-wise fusion and object-level features leads to both higher segmentation crispness ($25.7\%$ mIoU delta), as well as higher grounding precision ($42.5\%$ Pr@75 delta). See qualitative comparisons in Appendix D.

Effect of the number of views We ablate the 3D referring segmentation performance based on the number of input views in Fig. 5, where novel viewpoints are added incrementally. We observe that in both setups (point- and object-wise) fusing features from more views leads to improvements, with a small plateauing behavior around 40 views. We believe this is an encouraging result for leveraging dense multi-view coverage in feature distillation pipelines, as we propose with the introduction of MV-TOD.

4.2 Open-Vocabulary 3D Segmentation Results

Method	#views	Ref.Segm. (%)				Sem.Segm (%)
		mIoU	Pr@25	Pr@50	Pr@75	mIoU	mAcc
OpenScene†  [12]	73	29.32	44.00	24.51	11.26	21.79	32.14
OpenMask3D∗†  [6]	73	65.38	73.05	63.99	57.40	59.47	66.48
DROP-CLIP† (Ours)	73	82.67	86.11	82.43	79.23	75.41	80.02
DROP-CLIP (Ours)	73	66.56	75.73	67.55	59.88	62.04	70.74
OpenSeg→3D  [9]	1	12.89	17.36	2.38	0.23	12.83	17.21
MaskCLIP→3D  [8]	1	25.64	40.36	18.69	6.95	20.97	32.09
DROP-CLIP (Ours)	1	62.31	71.96	62.75	53.85	54.48	64.41

Table 3: Referring and Semantic segmentation results on MV-TOD test split. Methods with ^† denote upper-bound 3D features, whereas DROP-CLIP denotes our distilled model. Methods with ^→3D produce 2D predictions that are projected to 3D to compute metrics. Method with $*$ denotes further usage of ground-truth segmentation masks.

In this section, we compare referring and semantic segmentation performance of our distilled features vs. previous open-vocabulary approaches, both in multi-view and in single-view settings. For multi-view, we compare our trained model with OpenScene [12] and OpenMask3D [6] methods, where the full point-cloud from all $73$ views is given as input.

Figure 6: Referring segmentation accuracy (Pr@25 (%)) vs. different language query types.

We note that for these baselines we obtain the upper-bound 3D features as before, as we observed that our trained model already outperforms them, so we refrained from also distilling features from baselines (details in Appendix C2). For single-view, we feed our network with partial point-cloud from projected RGB-D pair, and compare with 2D baselines MaskCLIP [8] and OpenSeg [9]. Our model slightly outperforms the OpenMask3D upper bound baseline in the multi-view setting ($+1.18\%$ in referring and $+2.57\%$ in semantic segmentation), while significantly outperforming 2D baselines in the single-view setting ($>30\%$ in both tasks). Importantly, single-view results closely match the multi-view ones ($\sim-4.0\%$), suggesting that DROP-CLIP indeed learns view-independent features.

Open-ended queries We evaluate the robustness of our model in different types of input language queries, organized in 4 families (class name - e.g. “cereal box", class + attribute - e.g. “brown cereal box", open - e.g. “chocolate Kellogs", and affordance - e.g. “I want something sweet‘). Comparative results are presented in Fig. 6 and qualitative in Fig. 4. We observe that our method achieves high grounding accuracy in all query types, even when using single-view.

4.3 Zero-Shot Transfer to Novel Domains / Tasks

Method	OCID-VLG  [31]		REGRAD  [30]
	IoU	Pr@25	IoU	Pr@25
MaskCLIP→3D  [8]	24.1	30.9	33.2	39.0
DROP-CLIP (Ours)	46.2	48.9	59.1	63.0

Table 4: Referring segmentation results in OCID-VLG [31] and REGRAD [30] datasets

Generalization to Novel Domains We evaluate the 3D referring segmentation performance of our trained model when applied zero-shot in novel tabletop domains. We test in 500 scenes from OCID-VLG [31] using the dataset’s instance-wise open queries, as well as in 1000 scenes from REGRAD [30], using each model’s class name as a query. Only single-view input is provided for both datasets.

Method	$mIoU$	$AP_{25}$	$AP_{50}$
SAM  [39]	70.11	95.26	79.88
DROP-CLIP (S)	80.83	91.92	86.83
Mask3D  [40]	14.41	18.65	3.41
DROP-CLIP (F)	88.37	93.13	91.47

Table 5: Zero-shot 3D instance segmentation results in MV-TOD.

We compare with MaskCLIP [8] as above and report results in Table 4. We note that test datasets contain both novel object instances (REGRAD) and classes (OCID-VLG). We observe that our method provides a significant performance boost across both domains ($22.1\%$ mIoU delta in OCID-VLG and $25.9\%$ in REGRAD).

Zero-Shot 3D Instance Segmentation Since our method has been distilled from features with object-level priors, we demonstrate that it can be used out-of-the-box for 3D instance segmentation, via clustering the 3D features (see Appendix E for implementation details). We report results in MV-TOD in Table 5, where we compare with SAM [39] with single-view images, as well as Mask3D  [41] with full point-clouds (transferred from ScanRefer [1] with room layout). Mask3D struggles to generalize to tabletop domains, whereas our method achieves comparable performance with SAM for segmenting from single-view, even without being explicitly trained for instance segmentation.

4.4 Open-Vocabulary Language-guided Robotic Grasping

Figure 7: Language-guided 6-DoF grasping trial with real robot (left), 3D features, grounding and grasp proposals (right).

.

In this section, we wish to illustrate the applicability of DROP-CLIP in a language-guided robotic grasping scenario. We integrate our method with a 6-DoF grasp detection network [42], to segment and then propose gripper poses for picking a target object indicated verbally. We randomly place 5-12 objects on a tabletop with different levels of clutter, and query the robot to pick the target object and place it in a fixed position. The user instruction is open-vocabulary and can involve open object descriptions, attributes, or affordances. We conducted 50 trials in Gazebo [43] and 10 with a real robot, and observed grounding accuracy of 84% and 80% respectively, and a final success rate of 64% and 60%, where failures were mostly due to grasp proposals that are outside of the robot’s kinematic range or motion planning that lead to a collision with other objects and the table. Our setup and example trials are shown in Fig. 7, while more details and qualitative results are provided in Appendix E. A video of robot demonstrations is provided as supplementary material.

5 Related work

3D Scene Understanding There’s a long line of works in closed-set 3D scene understanding [44, 45, 46, 47, 48, 49], applied in 3D classification [50, 51], localization [52, 1] and segmentation [53, 23, 22], using two-stage pipelines with instance proposals from point-clouds [54, 55] or RGB-D views  [56, 27], or single-stage methods [3] that leverage 3D-language cross attentions. [57] use CLIP embeddings for pretraining a 3D segmentation model, but still cannot be applied open-vocabulary.

Open-Vocabulary Grounding with CLIP Following the impressive results of CLIP [7] for open-set image recognition, followup works transfer CLIP’s powerful representations from image- to pixel-level [40, 58, 59, 60, 61, 62, 63, 9, 10, 8], extending to detection / segmentation, but limited to 2D. For 3D segmentation, the closest work is perhaps OpenMask3D [6] that extracts multi-view CLIP features from instance proposals from Mask3D [41] to compute similarities with open text queries.

3D CLIP Feature Distillation Recent works distill features from 2D foundation models with point-cloud encoders [12, 14, 21] or neural fields [13, 19, 17, 18, 19, 24], with applications in robot manipulation [20, 16] and navigation [64, 65]. However, associated works extract 2D features from OpenSeg [9], LSeg [10], MaskCLIP [8] or multi-scale crops from CLIP [7] and fuse point-wise with average pooling, while our approach leverages semantics-informed view selection and segmentation masks to do object-wise fusion with object-level features (see detailed overview in Appendix F).

6 Conclusion, Limitations and Future Work

We propose DROP-CLIP, a 2D$\rightarrow$3D CLIP feature distillation framework that employs object-centric priors to select views based on semantic informativeness and ensure crisp 3D segmentations, while working with single-view RGB-D. We also release MV-TOD, a large-scale synthetic dataset of multi-view tabletop scenes with dense annotations that can be leveraged for several downstream tasks. We hope our work can benefit the robotics community, both in terms of released resources as well as illustrating and overcoming theoretical limitations of existing 3D feature distillation works.

While our spatial object-centric priors lead to improved segmentation quality, they collapse local features in favor of a global object-level feature, and hence cannot be applied for segmenting object parts. In the future, we plan to add object part annotations in our dataset and fuse with both object- and part-level masks. Second, DROP-CLIP only provides grounding and a two-stage pipeline is needed for grasping, while our dataset already provides rich 6-DoF grasp annotations. A next step would be to also distill them, opting for a joint 3D representation for grounding and grasping.