Projekte

In computer vision communities such as stereo, optical flow, or visual tracking, commonly accepted and widely used benchmarks have enabled objective comparison and boosted scientific progress. In the emergent light field community, a comparable benchmark and evaluation methodology is still missing. The performance of newly proposed methods is often demonstrated qualitatively on a handful of images, making quantitative comparison and targeted progress very difficult. To overcome these difficulties, we propose a novel light field benchmark. We provide 24 carefully designed synthetic, densely sampled 4D light fields with highly accurate disparity ground truth. We thoroughly evaluate four state-of-the-art light field algorithms and one multi-view stereo algorithm using existing and novel error measures. This consolidated state-of-the art may serve as a baseline to stimulate and guide further scientific progress. We publish the benchmark website www.lightfield-analysis.net, an evaluation toolkit, and our rendering setup to encourage submissions of both algorithms and further datasets.

We propose a framework to infer complete geometry of a scene with strong reflections or hidden by partially transparent occluders from a set of 4D light fields captured with a hand-held light field camera. For this, we first introduce a variant of bundle adjustment specifically tailored to 4D light fields to obtain improved pose parameters. Geometry is recovered in a global framework based on convex optimization for a weighted minimal surface. To allow for non-Lambertian materials and semi-transparent occluders, the point-wise costs are not based on the principle of photo-consistency. Instead, we perform a layer analysis of the light field obtained by finding superimposed oriented patterns in epipolar plane image space to obtain a set of depth hypotheses and confidence scores, which are integrated into a single functional.

We present a novel variational model for intrinsic light field decomposition, which is performed on four-dimensional ray space instead of a traditional 2D image. As most existing intrinsic image algorithms are designed for Lambertian objects, their performance suffers when considering scenes which exhibit glossy surfaces. In contrast, the rich structure of the light field with many densely sampled views allows us to cope with non-Lambertian objects by introducing an additional decomposition term that models specularity. Regularization along the epipolar plane images further encourages albedo and shading consistency across views. In evaluations of our method on real-world data sets captured with a Lytro Illum plenoptic camera, we demonstrate the advantages of our approach with respect to intrinsic image decomposition and specular removal.

Disparity estimation for multi-layered light fields can robustly be performed with a statistical analysis of sparse light field coding coefficients [7]. The key idea is to explain each epipolar plane image patch with a dictionary composed of atoms with known disparity values. We significantly improve upon their approach in two ways. First, we reduce the number of necessary dictionary atoms, improving descriptive quality of each and reducing time complexity by an order of magnitude. Second, we introduce a way to explicitly handle occlusions, which is the main drawback in the previous work. Experiments demonstrate that we thus achieve substantially better results on both Lambertian as well as multi-layered scenes.

We present a novel approach to relative pose estimation which is tailored to 4D light field cameras. From the relationships between scene geometry and light field structure and an analysis of the light field projection in terms of Pluecker ray coordinates, we deduce a set of linear constraints on ray space correspondences between a light field camera pair. These can be applied to infer relative pose of the light field cameras and thus obtain a point cloud reconstruction of the scene. While the proposed method has interesting relationships to pose estimation for generalized cameras based on ray-to-ray correspondence, our experiments demonstrate that our approach is both more accurate and computationally more efficient. It also compares favourably to direct linear pose estimation based on aligning the 3D point clouds obtained by reconstructing depth for each individual light field. To further validate the method, we employ the pose estimates to merge light fields captured with hand-held consumer light field cameras into refocusable panoramas.

Although light fields are well-established as a tool in image-based rendering and computer vision, their capture is still at a relatively early stage. In this article, we search for imaging situations similar to uncalibrated integral optics, noticing that they are common in everyday life. We investigate light field capturing scenarios which are provided by commonly available items like cutlery as optical building blocks. Using a generic calibration approach based on structured light, we reconstruct the light path providing an unorthodox light field capturing setup. As the resulting data is unstructured and poorly sampled and thus unsuited for standard image-based rendering pipelines, we propose techniques for the processing of such light fields. Additionally, we have implemented a novel depth estimation scheme to guide the rendering process. We demonstrate the potential of these techniques on different scenes, both static and dynamic, recorded by combining a DSLR camera with household items.

Images of scenes which contain reflective or transparent surfaces are composed of different layers which are observed at different depths. Analyzing such a scene requires separating the image into its individual layers, which remains a challenging and important problem. While the problem is very much ill-posed when only a single image is considered, recent work has shown that depth estimation for two layers becomes quite tractable when one instead captures a 4D light field of the scene. In this paper, we propose a novel variational approach to layer separation which is based on these ideas. We formulate a linear generative model to reconstruct the light field from disparity and luminance information for the individual layers on the center view. Comparing the model with the observerd data yields a convex variational problem for layer reconstruction, which can be solved to global optimality with a primal-dual scheme. Layer disparity is estimated in a first step, for which we improve upon a model based on second order structure tensors on the epipolar plane images. In contrast to previous work, the resulting approach is robust enough to be able to deal with light fields from the Lytro Illum camera, for which we obtain a compelling separation of the reflectance layer in real-world scenes.