-
All-day Depth Completion
Authors:
Vadim Ezhov,
Hyoungseob Park,
Zhaoyang Zhang,
Rishi Upadhyay,
Howard Zhang,
Chethan Chinder Chandrappa,
Achuta Kadambi,
Yunhao Ba,
Julie Dorsey,
Alex Wong
Abstract:
We propose a method for depth estimation under different illumination conditions, i.e., day and night time. As photometry is uninformative in regions under low-illumination, we tackle the problem through a multi-sensor fusion approach, where we take as input an additional synchronized sparse point cloud (i.e., from a LiDAR) projected onto the image plane as a sparse depth map, along with a camera…
▽ More
We propose a method for depth estimation under different illumination conditions, i.e., day and night time. As photometry is uninformative in regions under low-illumination, we tackle the problem through a multi-sensor fusion approach, where we take as input an additional synchronized sparse point cloud (i.e., from a LiDAR) projected onto the image plane as a sparse depth map, along with a camera image. The crux of our method lies in the use of the abundantly available synthetic data to first approximate the 3D scene structure by learning a mapping from sparse to (coarse) dense depth maps along with their predictive uncertainty - we term this, SpaDe. In poorly illuminated regions where photometric intensities do not afford the inference of local shape, the coarse approximation of scene depth serves as a prior; the uncertainty map is then used with the image to guide refinement through an uncertainty-driven residual learning (URL) scheme. The resulting depth completion network leverages complementary strengths from both modalities - depth is sparse but insensitive to illumination and in metric scale, and image is dense but sensitive with scale ambiguity. SpaDe can be used in a plug-and-play fashion, which allows for 25% improvement when augmented onto existing methods to preprocess sparse depth. We demonstrate URL on the nuScenes dataset where we improve over all baselines by an average 11.65% in all-day scenarios, 11.23% when tested specifically for daytime, and 13.12% for nighttime scenes.
△ Less
Submitted 27 May, 2024;
originally announced May 2024.
-
An Inverse Procedural Modeling Pipeline for SVBRDF Maps
Authors:
Yiwei Hu,
Chengan He,
Valentin Deschaintre,
Julie Dorsey,
Holly Rushmeier
Abstract:
Procedural modeling is now the de facto standard of material modeling in industry. Procedural models can be edited and are easily extended, unlike pixel-based representations of captured materials. In this paper, we present a semi-automatic pipeline for general material proceduralization. Given Spatially-Varying Bidirectional Reflectance Distribution Functions (SVBRDFs) represented as sets of pixe…
▽ More
Procedural modeling is now the de facto standard of material modeling in industry. Procedural models can be edited and are easily extended, unlike pixel-based representations of captured materials. In this paper, we present a semi-automatic pipeline for general material proceduralization. Given Spatially-Varying Bidirectional Reflectance Distribution Functions (SVBRDFs) represented as sets of pixel maps, our pipeline decomposes them into a tree of sub-materials whose spatial distributions are encoded by their associated mask maps. This semi-automatic decomposition of material maps progresses hierarchically, driven by our new spectrum-aware material matting and instance-based decomposition methods. Each decomposed sub-material is proceduralized by a novel multi-layer noise model to capture local variations at different scales. Spatial distributions of these sub-materials are modeled either by a by-example inverse synthesis method recovering Point Process Texture Basis Functions (PPTBF) or via random sampling. To reconstruct procedural material maps, we propose a differentiable rendering-based optimization that recomposes all generated procedures together to maximize the similarity between our procedural models and the input material pixel maps. We evaluate our pipeline on a variety of synthetic and real materials. We demonstrate our method's capacity to process a wide range of material types, eliminating the need for artist designed material graphs required in previous work. As fully procedural models, our results expand to arbitrary resolution and enable high level user control of appearance.
△ Less
Submitted 27 September, 2021; v1 submitted 13 September, 2021;
originally announced September 2021.
-
Countering Racial Bias in Computer Graphics Research
Authors:
Theodore Kim,
Holly Rushmeier,
Julie Dorsey,
Derek Nowrouzezahrai,
Raqi Syed,
Wojciech Jarosz,
A. M. Darke
Abstract:
Current computer graphics research practices contain racial biases that have resulted in investigations into "skin" and "hair" that focus on the hegemonic visual features of Europeans and East Asians. To broaden our research horizons to encompass all of humanity, we propose a variety of improvements to quantitative measures and qualitative practices, and pose novel, open research problems.
Current computer graphics research practices contain racial biases that have resulted in investigations into "skin" and "hair" that focus on the hegemonic visual features of Europeans and East Asians. To broaden our research horizons to encompass all of humanity, we propose a variety of improvements to quantitative measures and qualitative practices, and pose novel, open research problems.
△ Less
Submitted 2 June, 2022; v1 submitted 28 March, 2021;
originally announced March 2021.
-
AniCode: Authoring Coded Artifacts for Network-Free Personalized Animations
Authors:
Zeyu Wang,
Shiyu Qiu,
Qingyang Chen,
Alexander Ringlein,
Julie Dorsey,
Holly Rushmeier
Abstract:
Time-based media (videos, synthetic animations, and virtual reality experiences) are used for communication, in applications such as manufacturers explaining the operation of a new appliance to consumers and scientists illustrating the basis of a new conclusion. However, authoring time-based media that are effective and personalized for the viewer remains a challenge. We introduce AniCode, a novel…
▽ More
Time-based media (videos, synthetic animations, and virtual reality experiences) are used for communication, in applications such as manufacturers explaining the operation of a new appliance to consumers and scientists illustrating the basis of a new conclusion. However, authoring time-based media that are effective and personalized for the viewer remains a challenge. We introduce AniCode, a novel framework for authoring and consuming time-based media. An author encodes a video animation in a printed code, and affixes the code to an object. A consumer uses a mobile application to capture an image of the object and code, and to generate a video presentation on the fly. Importantly, AniCode presents the video personalized in the consumer's visual context. Our system is designed to be low cost and easy to use. By not requiring an internet connection, and through animations that decode correctly only in the intended context, AniCode enhances privacy of communication using time-based media. Animation schemes in the system include a series of 2D and 3D geometric transformations, color transformation, and annotation. We demonstrate the AniCode framework with sample applications from a wide range of domains, including product "how to" examples, cultural heritage, education, creative art, and design. We evaluate the ease of use and effectiveness of our system with a user study.
△ Less
Submitted 30 July, 2018;
originally announced July 2018.
-
Ergonomic-driven Geometric Exploration and Reshaping
Authors:
Youyi Zheng,
Julie Dorsey,
Niloy Mitra
Abstract:
The paper addresses the following problem: given a set of man-made shapes, e.g., chairs, can we quickly rank and explore the set of shapes with respect to a given avatar pose? Answering this question requires identifying which shapes are more suitable for the defined avatar and pose; and moreover, to provide fast preview of how to alter the input geometry to better fit the deformed shapes to the g…
▽ More
The paper addresses the following problem: given a set of man-made shapes, e.g., chairs, can we quickly rank and explore the set of shapes with respect to a given avatar pose? Answering this question requires identifying which shapes are more suitable for the defined avatar and pose; and moreover, to provide fast preview of how to alter the input geometry to better fit the deformed shapes to the given avatar pose? The problem naturally links physical proportions of human body and its interaction with object shapes in an attempt to connect ergonomics with shape geometry. We designed an interaction system that allows users to explore shape collections using the deformation of human characters while at the same time providing interactive previews of how to alter the shapes to better fit the user-specified character. We achieve this by first mapping ergonomics guidelines into a set of simultaneous multi-part constraints based on target contacts; and then, proposing a novel contact-based deformation model to realize multi-contact constraints. We evaluate our framework on various chair models and validate the results via a small user study.
△ Less
Submitted 21 February, 2014;
originally announced February 2014.
