Abstract: Digital cameras with electronic viewfinders provide a relatively faithful depiction of the final image, providing a WYSIWYG experience. If, however, the image is created from a burst of differently captured images, or non-linear interactive edits significantly alter the final outcome, then the photographer cannot directly see the results, but instead must imagine the post-processing effects. This paper explores the notion of viewfinder editing, which makes the viewfinder more accurately reflect the final image the user intends to create. We allow the user to alter the local or global appearance (tone, color, saturation, or focus) via stroke-based input, and propagate the edits spatiotemporally. The system then delivers a real-time visualization of these modifications to the user, and drives the camera control routines to select better capture parameters.

Abstract: High-dimensional Gaussian filtering is a popular technique in image processing, geometry processing and computer graphics for smoothing data while preserving important features. For instance, the bilateral filter, cross bilateral filter and non-local means filter fall under the broad umbrella of high-dimensional Gaussian filters. Recent algorithmic advances therein have demonstrated that by relying on a sampled representation of the underlying space, one can obtain speed-ups of orders of magnitude over the naive approach. The simplest such sampled representation is a lattice, and it has been used successfully in the bilateral grid and the permutohedral lattice algorithms. In this paper, we analyze these lattice-based algorithms, developing a general theory of lattice-based high-dimensional Gaussian filtering. We consider the set of criteria for an optimal lattice for filtering, as it offers a good tradeoff of quality for computational efficiency, and evaluate the existing lattices under the criteria. In particular, we give a rigorous exposition of the properties of the permutohedral lattice and argue that it is the optimal lattice for Gaussian filtering. Lastly, we explore further uses of the permutohedral-lattice-based Gaussian filtering framework, showing that it can be easily adapted to perform mean shift filtering and yield improvement over the traditional approach based on a Cartesian grid.

Abstract: Given the frequent use of improvised explosive devices (IEDs) around the world, the study of traumatic blast injuries is of increasing interest. The ear is the most common organ affected by blast injury because it is the body's most sensitive pressure transducer. We fabricated a blast chamber to re-create blast profiles similar to that of IEDs and used it to develop a reproducible mouse model to study blast-induced hearing loss. The tympanic membrane was perforated in all mice after blast exposure and found to heal spontaneously. Micro-computed tomography demonstrated no evidence for middle ear or otic capsule injuries; however, the healed tympanic membrane was thickened. Auditory brainstem response and distortion product otoacoustic emission threshold shifts were found to be correlated with blast intensity. As well, these threshold shifts were larger than those found in control mice that underwent surgical perforation of their tympanic membranes, indicating cochlear trauma. Histological studies one week and three months after the blast demonstrated no disruption or damage to the intra-cochlear membranes. However, there was loss of outer hair cells (OHCs) within the basal turn of the cochlea and decreased spiral ganglion neurons (SGNs) and afferent nerve synapses. Using our mouse model that recapitulates human IED exposure, our results identify that the mechanisms underlying blast-induced hearing loss does not include gross membranous rupture as is commonly believed. Instead, there is both OHC and SGN loss that produce auditory dysfunction. Given the frequent use of improvised explosive devices (IEDs) around the world, the study of traumatic blast injuries is of increasing interest. The ear is the most common organ affected by blast injury because it is the body's most sensitive pressure transducer. We fabricated a blast chamber to re-create blast profiles similar to that of IEDs and used it to develop a reproducible mouse model to study blast-induced hearing loss. The tympanic membrane was perforated in all mice after blast exposure and found to heal spontaneously. Micro-computed tomography demonstrated no evidence for middle ear or otic capsule injuries; however, the healed tympanic membrane was thickened. Auditory brainstem response and distortion product otoacoustic emission threshold shifts were found to be correlated with blast intensity. As well, these threshold shifts were larger than those found in control mice that underwent surgical perforation of their tympanic membranes, indicating cochlear trauma. Histological studies one week and three months after the blast demonstrated no disruption or damage to the intra-cochlear membranes. However, there was loss of outer hair cells (OHCs) within the basal turn of the cochlea and decreased spiral ganglion neurons (SGNs) and afferent nerve synapses. Using our mouse model that recapitulates human IED exposure, our results identify that the mechanisms underlying blast-induced hearing loss does not include gross membranous rupture as is commonly believed. Instead, there is both OHC and SGN loss that produce auditory dysfunction.

Abstract: Many cameras provide insufficient control over depth of field. Some have a fixed aperture; others have a variable aperture that is either too small or too large to produce the desired amount of blur. To overcome this limitation, one can capture a focal stack, which is a collection of images each focused at a different depth, then combine these slices to form a single composite that exhibits the desired depth of field. In this paper, we present a theory of focal stack compositing, and algorithms for computing images with extended depth of field, shallower depth of field than the lens aperture naturally provides, or even freeform (non-physical) depth of field. We show that while these composites are subject to halo artifacts, there is a principled methodology for avoiding these artifacts---by feathering a slice selection map according to certain rules before computing the composite image.

Abstract: In this paper we present a robust, real-time video stabilization and rolling shutter correction technique based on commodity gyroscopes. First, we develop a unified algorithm for modeling camera motion and rolling shutter warping. We then present a novel framework for automatically calibrating the gyroscope and camera outputs from a single video capture. This calibration allows us to use only gyroscope data to effectively correct rolling shutter warping and to stabilize the video. Using our algorithm, we show results for videos featuring large moving foreground objects, parallax, and low-illumination. We also compare our method with commercial image-based stabilization algorithms. We find that our solution is more robust and computationally inexpensive. Finally, we implement our algorithm directly on a mobile phone. We demonstrate that by using the phone's inbuilt gyroscope and GPU, we can remove camera shake and rolling shutter artifacts in real-time.

Abstract: High-dimensional Gaussian filters, most notably the bilateral filter, are important tools for many computer graphics and vision tasks. In recent years, a number of techniques for accelerating their evaluation have been developed by exploiting the separability of these Gaussians. However, these techniques do not apply to the more general class of spatially varying Gaussian filters, as they cannot be expressed as convolutions. These filters are useful because the underlying data.e.g. images, range data, meshes or light fields.often exhibit strong local anisotropy and scale. We propose an acceleration method for approximating spatially varying Gaussian filters using a set of spatially invariant Gaussian filters each of which is applied to a segment of some non-disjoint partitioning of the dataset. We then demonstrate that the resulting ability to locally tilt, rotate or scale the kernel improves filtering performance in various applications over traditional spatially invariant Gaussian filters, without incurring a significant penalty in computational expense.

Abstract: Although there has been much interest in computational photography within the research and photography communities, progress has been hampered by the lack of a portable, programmable camera with sufficient image quality and computing power. To address this problem, we have designed and implemented an open architecture and API for such cameras: the Frankencamera. It consists of a base hardware specification, a software stack based on Linux, and an API for C++. Our architecture permits control and synchronization of the sensor and image processing pipeline at the microsecond time scale, as well as the ability to incorporate and synchronize external hardware like lenses and flashes. This paper specifies our architecture and API, and it describes two reference implementations we have built. Using these implementations we demonstrate six computational photography applications: HDR viewfinding and capture, low-light viewfinding and capture, automated acquisition of extended dynamic range panoramas, foveal imaging, IMU-based hand shake detection, and rephotography. Our goal is to standardize the architecture and distribute Frankencameras to researchers and students, as a step towards creating a community of photographer-programmers who develop algorithms, applications, and hardware for computational cameras.

Abstract: We present a flexible method for fusing information from optical and range sensors based on an accelerated high-dimensional filtering approach. Our system takes as input a sequence of monocular camera images as well as a stream of sparse range measurements as obtained from a laser or other sensor system. In contrast with existing approaches, we do not assume that the depth and color data streams have the same data rates or that the observed scene is fully static. Our method produces a dense, high-resolution depth map of the scene, automatically generating confidence values for every interpolated depth point. We describe how to integrate priors on object shape, motion and appearance and how to achieve an efficient implementation using parallel processing hardware such as GPUs.

Abstract: Recent advances in computational cameras achieve extension of depth of field by modulating the aperture of an imaging system, either spatially or temporally. They are, however, accompanied by loss of image detail, the chief cause of which is low and/or depth-varying frequency response of such systems. In this paper, we examine the tradeoff between achieving depth invariance and maintaining high transfer efficiency by providing a mathematical framework for analyzing the transfer function of these computational cameras. Using this framework, we prove mathematical bounds on the efficacy of the tradeoff. These bounds lead to observations on the fundamental limitations of computational cameras. In particular, we show that some existing designs are already near-optimal in our metrics.

Abstract: Many useful algorithms for processing images and geometry fall under the general framework of high-dimensional Gaussian filtering. This family of algorithms includes bilateral filtering and non-local means. We propose a new way to perform such filters using the permutohedral lattice, which tessellates high-dimensional space with uniform simplices. Our algorithm is the first implementation of a high-dimensional Gaussian filter that is both linear in input size and polynomial in dimensionality. Furthermore it is parameter-free, apart from the filter size, and achieves a consistently high accuracy relative to ground truth (> 45 dB). We use this to demonstrate a number of interactive-rate applications of filters in as high as eight dimensions.

Abstract: The past decade witnessed a rise in the ubiquity and capability of digital photography, paced by the advances in embedded devices, image processing and social media. Along with it, the popularity of computational photography also grew. Many computational photography techniques work by first capturing a coded representation of the scene---a stack of photographs with different settings, an image obtained via a modified optical path, et cetera---and then computationally decoding it later as a post-process according to the user's specification. However, the coded representation, available to the user at the time of capture, is often not sufficiently indicative of the decoded output that will be produced later. Depending on the type of the computational photography technique involved, the coded representation may appear to be a distorted image, or may not even be an image at all. Consequently, these techniques discard one of the most significant attractions of digital photography: the what-you-see-is-what-you-get (WYSIWYG) experience. In response, this dissertation explores a new kind of interface for manipulating images in computational photography applications, called viewfinder editing. With viewfinder editing, the viewfinder more accurately reflects the final image the user intends to create, by allowing the user to alter the local or global appearance of the photograph via stroke-based input on a touch-enabled digital viewfinder, and propagating the edits spatiotemporally. Furthermore, the user specifies via the interface how the coded representation should be decoded in computational photography applications, guiding the acquisition and composition of photographs and giving immediate visual feedback to the user. Thus, the WYSIWYG aspect is reclaimed, enriching the user's photographing experience and helping him make artistic decisions before or during capture, instead of after capture. This dissertation realizes and presents a real-time implementation of viewfinder editing on a mobile platform, constituting the first of its kind. This implementation is enabled by a new spatiotemporal edit propagation method that meaningfully combines and improves existing algorithms, achieving an order-of-magnitude speed-up over existing methods. The new method trades away spatial locality for efficiency and robustness against camera or scene motion. Finally, several applications of the framework are demonstrated, such as high-dynamic-range (HDR) multi-exposure photography, focal stack composition, selective colorization, and general tonal editing. In particular, new camera control algorithms for stack metering and focusing are presented, which takes advantage of the knowledge of the user's intent indicated via the viewfinder editing interface and optimizes the camera parameters accordingly.

Abstract: This thesis describes the multi-channel coded-aperture photography, a modified camera system that can extract an all-focus image of the scene along with a depth estimate over the scene. The modification consists of inserting a set of patterned color filters into the aperture of the camera lens. This work generalizes the previous research on a single-channel coded aperture, by deploying distinct filters in the three primary color channels, in order to cope better with the effect of a Bayer filter and to exploit the correlation among the channels. We derive the model and algorithms for the multi-channel coded aperture, comparing the simulated performance of the reconstruction algorithm against that of the original single-channel coded aperture. We also demonstrate a physical prototype, discussing the challenges arising from the use of multiple filters. We provide a comparison with the single-channel coded aperture in performance, and present results on several scenes of cluttered objects at various depths.