3D reconstruction project
Required functionalities
Each group should implement a system with the following functionality:

Gold standard Fmatrix estimation from known correspondences. (Many of the required functions exist in LAB3, and in OpenCV).

Computation of an Ematrix from
F and known K, and subsequent extraction of
R and t from E, and visiblepoints constraint.
 Robust Perspectivenpoint (PnP) estimation for adding a new view and detecting outliers.

Bundle adjustment for N cameras (N≥2) using known correspondences, intrinsic camera parameters (Kmatrix), and an initial guess for the camera poses.
 Use a sparsity mask for the Jacobian in the Bundle adjustment step (this speedup is absolutely essential if you use scipy.optimize.least_squares). Alternatively, in C++ you can use Ceres Solver, which handles the sparsity in a different way. Yet another option is to use pytorch (Ask your guide for details).
 Evaluate the robustness to noise in the implemented system, by measuring the camera pose errors of your reconstruction. Details can be found here.
In addition to the above, groups with four students should implement at least one of the following functionalities:
 Visualisation of the 3D model. Pick one of the following:
 Use Poisson surface reconstruction (or screened PSR) to obtain a volumetric representation. Try e.g. the version in Meshlab, or Kazhdan's own implementation (has more options). Both require you to estimate colour and surface normals for the 3Dpoints, and saving these in PLY format. Use visibility in cameras to determine normal signs. See Kazhdan and Hoppe for a theoretical description.
 Use space carving (as in Fitzgibbon, Cross and Zisserman) to generate a volumetric representation of your object.
 Use texture patches (e.g. PMVS) or billboards, to represent the texture of the estimated 3D model.
 Find correspondences between pairs of views, and remove false matches using epipolar geometry and crosschecking. (Allows the implemented system to use any image set with known K)
 Use your own camera to take images of an object, and reconstruct it. This requires estimation of the Kmatrix, and possibly lens distortion, followed by image rectification. Note that this also requires finding your own correspondences (functionality #2), and that you need to carefully plan how to take the images.
 Densify your initial sparse SfM model, using dense stereo between selected frame pairs. E.g. PatchMatch can be used, or using one of the approaches in the Multiview Stereo tutorial.
 Use nextbestview selection (e.g. as in Schönberger et al., or some simplified form thereof) to select the order in which cameras are added to SfM.
 Replace Incremental SfM with Global SfM. Use global estimation of all rotation matrices as an initialisation of Bundle Adjustment (as in Martinec and Pajdla).
 Write your own nonlinear solver, e.g.~LevenbergMarquardt (see the report by Madsen et al.), and use the ShurComplement Trick to speed up the computation of the update step (see the IREG compendium).
Groups with five students are required to implement two of the above, and groups with six members should implement three.
Datasets
The Visual Geometry Group at Oxford University has a number of datasets availabe on their web site, e.g., the dinosaur sequence:
These data sets contain coordinates of image points that are tracked between images, which means that there is both some amount of noise on the image coordinates and that these points are not visible over the entire sequence since they are occluded in some or even most of theÂ images. The data sets contain some type of ground truth in terms of estimated camera matrices for each view.
Notice that the dinosaur sequence is a turntable sequence that is generated by rotating the object around a fixed axis. The acquisition geometry of this dataset makes it sensitive to the quality of the used point correspondences.
For debugging it is useful to work with a noise free dataset, and for this purpose we have "cleaned" the dinosaur data set from any noise (up to the numerical accuracy of Matlab) and produced a dataset that contains 2D image points, 3D points, and the camera matrices. These are found in the BAdino2.mat file on the local ISY file system:
 /courses/TSBB15/sequences/BAdino/BAdino2.mat
The EPFL multiview stereo datasets are also highly recommended:
They all have undistorted highres images, with groundtruth camera poses and known intrinsic camera parameters. However, you have to find the correspondences yourself here.
Python code
Please look at the utility code for CE3. You can find it at /courses/TSBB15/python/ in the computer labs. Also look at the Extra exercises in the lab sheet, they are meant to help you get started with the projects.
Matlab code
In the case that you use Matlab for solving the tasks in this project there are some software packages, that will spare you from implementing all functionalities yourself:
 The Visual Geometry Group at Oxford University
 Peter Kovesi at the University of Western Australia

The LSQNONLIN optimiser in Matlab. Note that LSQNONLIN is very slow, unless a sparsity mask is used.

P3P by Laurent Kneip. Note: Documentation is weird. You need
to verify the input and output behaviour yourself on synthetic
data (e.g. for coordinate axes). See this example for how to do this in Matlab.
C code
You are absolutely NOT allowed to use complete SfM systems such as COLMAP, SBA, Bundler, Visual SFM, SSBA, and OpenMVG.On the other hand, you are encouraged to use a nonlinear least squares package, and if you intend to make a more advanced representation of your 3D model, you may use a package for this. Some useful (and allowed) packages are listed below.
 Ceres Solver, a library for efficient nonlinear optimization.
 levmar, and lmfit, LevenbergMarquardt optimisation packages.
 sparseLM, a sparse LevenbergMarquardt optimisation package (much faster, but also more complex to set up).

PMVS, package to compute 3D models from images and camera poses.
 OpenCV, implements several local invariant features, and some geometry functions.
 Lambda twist P3P by Mikael Persson and Klas Nordberg.

P3P by Laurent Kneip.
 OpenGV, a collection of geometric solvers for calibrated epipolar geometry.
Deliverables
In order to pass, each project group should do the following:

Make a design plan, and get it approved by the guide.
The design plan should contain the following: A list of the tasks/functionalities that will be implemented
 A group member list, with responsibilities for each person (e.g. what task)
 A flowchart of the system components
 A brief description of the components
 Deliver a good presentation at the seminar
 Hand in a written report to the project guide.
 A group member list, with responsibilities for each person
 A description of the problem that is solved
 How the problem is solved
 What the result is (i.e. performance relative to ground truth)

Why the result is what it is

References to used methods
We recommend that you use the CVPR LaTeX template when writing the report.
References
 C. Barnes et al. PatchMatch: A randomized correspondence algorithm for structural image editing, ACM Transactions on Graphics (Proceedings of SIGGRAPH) 2009.
 A. Fitzgibbon, G. Cross and A. Zisserman, Automatic 3D Model Construction for TurnTable Sequences , in 3D Structure from Multiple Images of LargeScale Environments , Editors Koch & Van Gool, Springer Verlag 1998, pages 155170.
 Y. Furukawa and C. Hernández, Multiview Stereo: A Tutorial. In Foundations and trends in Computer Graphics and Vision Vol. 9, No. 12, 2013.
 Michael Kazhdan and Hugues Hoppe, Screened Poisson Surface Reconstruction, ACM Transactions on Graphics 2013.
 K. Madsen et al., Methods for NonLinear Least Squares Problems, 2nd ed., DTU technical report 2004.
 Daniel Martinec and Tomáš Pajdla, Robust Rotation and Translation Estimation in Multiview Reconstruction, CVPR 2007.
 Klas Nordberg, Introduction to Representations and Estimation in Geometry (IREG).
 Johannes Schönberger and JanMichael Frahm, StructurefromMotion Revisited, CVPR 2016.
 Triggs, McLauchlan, Hartley and Fitzgibbon, Bundle Adjustment  A Modern Synthesis, in Vision Algorithms  Theory and Practice, Springer Verlag, Lecture Notes in Computer Science No 1883, 2000, pages 298372.
 Daniel Scharstein and Richard Szeliski, A Taxonomy and Evaluation of Dense TwoFrame Stereo Correspondence Algorithms, IJCV 47(13), 2002
 Brian Curless and Marc Levoy, A Volumetric Method for Building Complex Models from Range Images, SIGGRAPH'96, 1996
 William E. Lorenzen, Harvey E. Cline, Marching cubes: A high resolution 3D surface construction algorithm, SIGGRAPH'87, 1987
 R. Newcombe et al. KinectFusion: Realtime Dense Surface Mapping and Tracking. ISMAR'11, 2011
 C. Loop and Z. Zhang, Computing Rectifying Homographies for Stereo Vision, ICCV 1999.
Senast uppdaterad: 20210426