Recomendados
Más contenido relacionado
La actualidad más candente
La actualidad más candente (20)
Destacado
Destacado (14)
Similar a Multi Aperture Photography
Similar a Multi Aperture Photography (20)
Último
Último (20)
Multi Aperture Photography
- 1. Multi-Aperture Photography Paul Green – MIT CSAIL Wenyang Sun – MERL Wojciech Matusik – MERL Frédo Durand – MIT CSAIL
- 2. Motivation http://photographertips.net Portrait Landscape Small Aperture Large Aperture Depth of Field Control Shallow Depth of Field Large Depth of Field
- 3. Depth and Defocus Blur plane of focus sensor lens defocus blur depends on distance from plane of focus subject rays from point in focus converge to single pixel circle of confusion
- 4. Defocus Blur & Aperture lens plane of focus defocus blur depends on aperture size aperture http://photographertips.net sensor subject circle of confusion
- 9. 1D vs 2D Aperture Sampling u v Aperture 2D Grid Sampling http://photographertips.net
- 10. 1D vs. 2D Aperture Sampling 4 Samples Aperture 1D “Ring” Sampling 45 Samples u v Aperture 2D Grid Sampling http://photographertips.net
- 16. Aperture Splitting Incoming light Sensor Mirrors Focusing lenses Tilted Mirrors
- 17. Aperture Splitting X Ideally at aperture plane , but not physically possible! Solution: Relay Optics to create virtual aperture plane Photographic Lens Aperture Plane Relay system Aperture splitting optics New Aperture Plane
- 18. Optical Prototype Mirror Close-up main lens relay optics mirrors tilted mirrors lenses SLR Camera
- 22. DOF Navigation
- 24. DOF Extrapolation Roadmap capture estimate blur fit model extrapolate blur Blur size Aperture Diameter Largest physical aperture I E I 1 I 2 I 0 I 3
- 25. Defocus Gradient Defocus blur G is slope of this line Defocus Gradient Map Defocus Gradient Blur proportional to aperture diameter Blur size Aperture Diameter D I 1 I 2 I E I 0 σ I 3 Largest physical aperture focal length aperture diameter sensor distance object distance
- 27. Depth of Field Extrapolation
Notas del editor
- Today I will be talking about how to provide useful Depth of field controls to photographers. Depth of field is the term photographers use to describe the range of distances in the scene that are sharp in the final image. shallow DOF, as shown in the top row of images, emphasizes the subject and removes distracting backgrounds, and is often used in portrait photography . Large DOF is necessary when where there are large distances between points of interest in the scene, for example in landscape and outdoor photography. DOF of field is directly controlled by the size of the lens aperture. Large Apertures produce shallow DOF, and Small apertures produce large DOF A photographer must the choose aperture size before taking each photo. And We would like to alleviate some of this burden by allowing post-exposure depth of field control, by capturing multiple aperture settings in one exposure. Now I will go into some details of DoF
- The Job of a lens is to take all the rays leaving a point on an subject located at the plane of focus, and converge them into a single point at the sensor. However, if the subject is moved away from the focus plane, the rays no longer converge to a single point, and instead produce a blurry spot. The further the subject is from the focus plane, the larger the spot will be. [CLICK] We call the size of the blurred spot, the circle of confusion or the defocus blur.
- In addition to depending on depth, defocus blur also depends on the size of the aperture. [click] the smaller the aperture, the smaller the blur. So by controlling the aperture size, we can control the defocus blur, and hence DOF.
- Its for this reason that aperture size is a critical parameter for a photographer to set before taking each photo, and also a parameter that takes experience to control well. we want to facilitate post-exposure DOF control by allowing the photographer to explorer the aperture settings after taking the photo . Additionally, the amount of defocus blur is limited by the physical size of the aperture, so to get the Shallow depth of field that many photographers want, it is necessary to use expensive, large aperture lenses. We would like to allow extrapolation of shallow DOF beyond the constraints of the physical aperture on your lens.
- In this talk I will be presenting a new camera design that enables the capture of multiple aperture settings in a single exposure. I will also discuss some of the applications of the multi-aperture camera, with an emphasis on post-exposure depth of field control and depth of field extrapolation. And I will also describe a limited refocusing method.
- We build on many previous works in the area of passive computational cameras that allow DOF control. I will go into the details and differences of several, and discuss how they ultimately shaped our design. Also, In the second half of the talk, I will describe the software applications of our camera, many of which build on depth from defocus.
- The first body of work that really shaped our design is Plenoptic, or Lightfield Cameras that can capture the 4d lightfield entering the camera, by Sampling 2 spatial dimensions, and 2 angular dimensions. The general idea of these cameras is to trade some spatial resolution to record angular information instead. For example, The design of Ng and colleagues uses a lenslet array placed just over the camera sensor to direct the light that comes from different regions of the aperture to different pixels on the sensor. For example, The purple ray that passes through the center of the aperture is captured at one pixel, while rays that passes through the edges of the aperture are recorded at different pixels. In a sense, what you get is a 4D function, that has the standard 2 spatial dimensions, and 2 extra angular dimensions that sample the aperture in a grid like pattern that I show here. This extra information allows them to perform refocusing, and DOF control after the exposure is taken, and produces very high quality images. Unfortunately the image resolution is reduced by as much as 100 times, and adding the lenslet array has permanently modified your camera.
- Going back to the particular application of DOF control, we remember that the main controllable parameter that affects DOF is the aperture size. The set of images on the right show how the aperture size changes in a normal camera. We can mimic the same behavior with a light field camera by summing different portions of the 2d grid of samples. For example, to create an image with a large depth of field, we would only use the central sample. [click] Now to synthesize the image from slightly larger aperture, we sum the central sample plus the next “ring” of samples. And so on…
- lightfield cameras are extremely general, but if what we are really interested in is changing the aperture, then we could have saved a lot of resolution if we had stored the “rings” directly instead of the grid of samples. In other words we have stored a 2d sampling, for something that is essentially 1d. This is the key idea that shaped our camera design.
- One alternative to using a lightfield camera that can capture the 1D space of apertures is to use a network of beamsplitters and cameras, for example as described in the Optical Splitting Trees work by McGuire and colleagues. The main drawback to this approach is that you loose light. In order to get variations in the aperture settings, each camera must use a physical aperture that blocks light. It is also difficult to image onto a single sensor, which would preclude it from being used with standard SLR cameras.
- our review of previous work has added a few more goals. [CLICK] We want to capture the 1d space of aperture rings directly without using beamsplitters we want to use a single Image Sensor so that it can be used with consumer photography and finally, we would like it to be removable, so that you can take a normal photograph if you want.
- Now I will go into the details of our particular optical design
- To restate our design principles, we want to record a 3D sampling of the light entering the camera: {PAUSE} capturing 2 spatial dimensions, and 1 aperture dimension. In practice, we will capture 4 aperture rings onto the 4 quadrants of the sensor [click] For example the light that enters the smallest green aperture should form an image in the lower left corner on the sensor. [click] So the main task of our optical system is to divert the light from each aperture ring along different paths to form 4 separate images.
- Our strategy to split the light arriving at the aperture into 4 paths, is to place a set of tilted mirror surfaces at the [click] Aperture plane, With Each surface having a different orientation
- The tilted surfaces are oriented such that light striking different regions of the aperture is diverted along different directions. For example, light that strikes the center green mirror is reflected downwards, [click] While, light that hits the blue mirror is reflected upwards. Note, the mirrors are colored here for illustrative purposes, but are normal first surface mirrors. [click] Once split, we use mirrors to fold the light back towards the sensor, and [click] lenses to form an image on the sensor.
- Ideally, we would place the aperture splitting mirrors at the aperture plane of the photographic lens. [click] Unfortunately, In practice we can’t, [click] As the aperture lies inside of the lens. Our solution is to use a relay system to create a virtual aperture plane outside of the photographic lens, and place our mirrors there. A relay system is really just more lenses that make an image the aperture.
- I have brought our optical prototype to give some sense of its scale. [hold up camera as prop]. Our prototype uses an off the shelf photographic lens [click] attached to relay optics. [click] The relay optics produce an image of the aperture onto the central splitting mirrors. [click] The light is then reflected onto 1 of 4 folding mirrors [click] depending on its position in the aperture. And finally focused through lenses [click] onto the ccd sensor of a SLR camera [click]
- Our design successfully splits the aperture into four regions. Here is an example of the data we capture with our camera; where Each quadrant is an image through one of the rings.
- Here I am showing the Point Spread Functions of each of the rings for a point light source placed off the plane of focus. The ring shape of the PSFs indicates that our optical design does indeed split the light at the aperture into rings as intended. Ideally each would be perfectly circular and not contain any gaps or occlusions. These gaps in the PSF, which are particularly pronounced in the larger rings, are caused by the reflected light being occluded by bases of the other mirrors. The rightmost image, shows the 4 other images summed together, and is an illustration of the PSF of the reconstructed full aperture image.
- Now I will discuss some of the new applications and algorithms possible with our multi-aperture camera
- The first thing we can do with this data is to adjust the depth of field between the smallest and largest apertures captured. By successively summing the images from different rings, we can create images as if taken through larger apertures. [CLICK] The more rings that we sum together, the shallower the DOF becomes
- So we were able to construct a sequence of images taken with the different aperture sizes. But what if we want to synthesize an image as if taken from a larger aperture? What does this extrapolated image look like? We know that it should be similar to the other images, only blurrier, because we are using a larger aperture. But exactly how much blurrier? We can approximate the defocus blur at each pixel as a convolution with a kernel that depends on the aperture size, and the depth at each pixel. This allows to relate the blur observed already in our captured images, to the extrapolated image. assuming that I0 is the smallest aperture image, We can express the blur in the other images as a convolution of image I0. For example, this pixel in I1 is a sum of some disc shaped area in I0. The corresponding pixel in I2, is also sum, but of a slightly larger area. The blur will be even larger for the extrapolated image. The only issue is to figure out exactly how large it should be. And then do this for every pixel in the extrapolated image.
- The general roadmap of what we would like to do to extrapolate blur is as follows: First we capture data with our multi-aperture camera, as I’ve described earlier. From this data we have 4 images, each taken with a different aperture setting. Next, we would like to estimate the amount of blur, at each pixel, in each of the 4 images. Then, we would like to fit a function to the blur samples that allows us to extrapolate. Finally, we evaluate the extrapolation function for the new aperture size, and synthesize a new blurred image using the extrapolated blur size. In fact aperture size has a simple linear relation to blur size, and we can use this model to combine the estimation of blur and fitting the extrapolation function for a more robust estimation. Next, I’ll briefly discuss the linear model of aperture and blur size that we will use as our extrapolation function.
- Looking at our lens diagram geometrically, a similar triangles argument can be used to see that there is a linear relationship between aperture diameter and blur size. For example, if the aperture size is halved, the blur should also be halved. [CLICK] More formally, we can derive an equation relating the blur size, sigma, to many of the imaging parameters such as focal length, lens aperture diameter, object and sensor distances. The details of this equation really aren’t important because we are going to group all of these terms together, and replace them with one number which we call the defocus gradient. This substitution helps make the linear form become more apparent. We call G, the defocus gradient because it describes the rate at which defocus blur changes with respect to aperture diameter. [CLICK] and From this we see that G is the slope of our extrapolation line. The key benefit of Performing this substitution is that it has reduced our task to essentially fitting a line to several data points. And then to extrapolate, we just evaluate the line at the desired aperture size. [CLICK]We call the estimate of the defocus gradient at each pixel A defocus gradient map. The DGM is related to a depth map, and infact we could have solved for depth and then converted it into a defocus gradient map. But this is a simple method that directly solves for the quantity that we are interested in, specifically How does defocus change with aperture size.
- To solve for the defocus gradient map, we construct a graph problem and use graph cuts optimization. Our objective function includes a data fitting term as well as a regularization term to enforce spatial smoothness among neighbors. The data term searches for the defocus gradient value that best explains the observed blur in different aperture images. Please see our paper for more details of the optimization.
- Here is an example of our DOF extrapolation technique where the depth of field is extrapolated beyond the largest aperture size.
- We can also perform a type of synthetic refocusing. By shifting the labels in the defocus gradient map, we can synthetically move the apparent plane of focus. And then blur using the DOF extrapolation technique just described. Darker values indicate depths closer to the plane of focus This method only works if you originally focused on the nearest or furthest object in the scene. Otherwise there is an ambiguity about whether a point is in front of or behind the original plane of focus. Depth from Defocus methods often have similar restrictions.
- Here I am showing a video where the focus plane is being moved from doll to doll
- One issue with our technique, particularly for refocusing, is that the refocusing ability is limited by the depth of field present in the smallest aperture image. We can improve the sharpness by using deconvolution techniques, where the size of the kernel used to deconvolve the image Is varied at each pixel according to our estimate of the local amount of blur, taken from the defocus gradient map. The bottom two images show a close-up comparison of the original captured image data, and after our depth guided deconvolution. We used richardson-lucy deconvolution, with the kernel determined by our DGM.
- A main limitation of our optical design is the occlusion that the central splitting mirror produces. This occlusion is inherent to our design, but we believe it can be reasonably well minimized. Also, there is the potential to use is as an advantage, in for example a coded-aperture type method, which you will hear about in the next two talks. Also, there is a difficult and tedious alignment process to get the images to route correctly to the sensor. Part of the difficulty of the alignment is because our system is only a prototype, which was built entirely from custom parts made in our in-house machine shop. The main issue with our DOF extrapolation and refocusing methods is that the DOF of the output image is dependent on the DOF present in the smallest aperture image. But image sharpness can be improved with our depth guided deconvolution method. Also, similarly to many Depth from defocus methods, we need texture in the image to compute an accurate defocus gradient map. Fortunately, for DOF extrapolation, its isn’t critical to have Accurate defocus gradients in textureless regions because we are blurring them, and a textureless area looks very similar blurred or unblurred.
- In conclusion We have presented the design for a multi-aperture camera that captures a 1d sampling of aperture sizes, instead of a 2d grid. Our system doesn’t use beamsplitters which loose light And is completely removable. Our Multi-Aperture camera enables post-exposure DOF control, And in particular DOF extrapolation. In addition we can perform limited refocusing. And Finally we presented a depth guided deconvolution method to improve sharpness.
- Thank you