M6. 3D Vision

The goal of this module is to learn the principles of the 3D reconstruction of an object or a scene from multiple images or stereoscopic videos. For that, the basic concepts of the projective geometry and the 3-space are firstly introduced. The rest of the theoretical aspects and applications are built upon these basic tools. The mapping from the 3D world to the image plane will be studied, for that we will introduce different camera models, their parameters and how to estimate them (camera calibration and auto-calibration). The geometry that relates two pair of views will be analyzed. All these concepts will be applied to obtain a 3D reconstruction in the possible settings: calibrated or uncalibrated cameras. In particular, we will learn how to: estimate the depth of image points, extract the underlying 3D points given a set of point correspondences in the images, generate novel views, estimate the 3D object given a set of calibrated color images or binary images, and estimate a sparse set of 3D points given a set of uncalibrated images. The 3D representation in voxels and meshes will be studied. Finally, we will explain the reconstruction and modeling from Kinect data, as a particular model of sensors that provide and image of the scene together with its depths. The concepts and techniques learnt in this module are used in real applications ranging from augmented reality, object scanning, motion capture, new view synthesis, bullet-time effect, robotics, etc.
Project title: 
3D Recovery of urban scenes

The main goal of the project is the 3D reconstruction of buildings from a set of images taken from different points of view. We will work with two kind of images from different public datasets: frontal images of the façades and aerial images. The 3D reconstruction of urban scenes has many applications such as virtual navigation, urban planification and architectural design, traffic and noise simulation.
More specifically the student will learn how to rectify the perspective distortion from a single view, what is the geometry and the mathematical tools that relate points seen in two views and how to estimate them given a set of point correspondences between the images, calibrate a camera with a planar pattern, estimate the depth of points in the scene given two images, generate new views of the scene, and compute a 3D model either from a set of calibrated or uncalibrated cameras. These tools are not only specific to images of buildings and can be applied to generic images for different real applications ranging from augmented reality, object scanning, motion capture, new view synthesis, bullet-time effect, robotics, etc.

 

Module lectures: 
Academic Year 2016-2017        
Week Date Time Lecture Lecturer University Building Room
1 Tue. Feb. 28th  16:00 -18:00 Introduction and applications. 2D projective geometry Gloria Haro UPF  52  52321
1 Thu. Mar. 2nd 16:00 -17:00 2D projective geometry. Planar transformations.  Gloria Haro UPF  52  52321
1 Thu. Mar. 2nd 17:00 -18:00 Homography estimation. Affine and metric rectification Coloma Ballester UPF  52  52321
1 Thu. Mar. 2nd 18:00 - 19:00 Project Introduction

Carles Bosch

Gloria Haro

UPF  52

 52101

 52105 

               
2 Tue. Mar. 7th 16:00 -18:00 3D projective geometry and transformations. Camera models Coloma Ballester UPF  52  52321
2 Thu. Mar. 9th 16:00 -18:00 Camera calibration and the image of the absolute conic. Radial distortion Coloma Ballester UPF  52  52321
2 Thu. Mar. 9th 18:00 -19:00 Project follow-up

Carles Bosch

Gloria Haro

UPF  52  52101

 52105

               
3 Tue. Mar. 14th 16:00 -18:00  HOMEWORK        
3 Thu. Mar. 16th 16:00 -19:00 HOMEWORK        
               
4 Tue. Mar. 21st 16:00 -18:00 Epipolar geometry. Fundamental matrix. Essential matrix. Extraction of camera matrices. Javier Ruiz UPF  52 52321 
4 Thu. Mar. 23th 16:00 -18:00 Computation of the fundamental matrix. Image rectification Javier Ruiz UPF  52  52321
4 Thu. Mar. 23th 18:00 -19:00 Project follow-up

Carles Bosch

Gloria Haro

UPF  52  52101

 52105

               
5 Tue. Mar. 28th 16:00 -18:00 Triangulation methods. Depth computation. New View Synthesis Gloria Haro UPF  52  52321
5 Thu. Mar. 30th 16:00 -18:00 Voxel-Based Methods. Projective reconstruction with factorization method. Stratified methods. Gloria Haro UPF  52  52321
5 Thu. Mar. 30th 18:00 -19:00 Project follow-up

Carles Bosch

Gloria Haro

UPF 52  52101

 52105

               
6 Tue. Apr. 4th 16:00 -17:00 Auto-calibration. Bundle adjustment. Gloria Haro UPF  52  52321
6 Tue. Apr. 4th 17:00 -18:00 3D sensors (kinect) Josep Ramon Casas UPF  52  52321
6 Thu. Apr. 6th 16:00 -18:00 Point cloud processing  Josep Ramon Casas UPF  52  52321
6 Thu. Apr. 6th 18:00 -19:00 Project follow-up

Carles Bosch

Gloria Haro

UPF  52  52101

 52105

               
      Easter Holidays (from Apr. 8th to Apr.17th )        
               
7 Thu. Apr. 20th 16:00 -19:00 Project presentations

Carles Bosch

Gloria Haro

UPF  52  52101

 52105

               
8 Tue. Apr. 25th 16:00 -18:00  HOMEWORK        
Thu. Apr. 27th 16:00 -19:00  HOMEWORK        
               
9 Thu. May. 4th 16:00 -19:00 EXAM Gloria Haro UPF  52  52321
               

M6 Student Guide [here]