PhD position: Deep generative models for weakly-supervised improvement of photogrammetric models

Title: Deep generative models for weakly-supervised improvement of photogrammetric models

Contact: Maxime Lhuillier (chargé de recherches CNRS, HDR) [email protected]

Keywords: deep generative models, semi-supervised learning, surface reconstruction from images, photogrammetry for VR.

Summary:

The 3D reconstruction of a complete environment from images is useful in a lot of applications including virtual reality (https://maximelhuillier.fr) and perception of autonomous vehicles.

Several methods of computer vision and photogrammetry are needed to solve this problem.

They include the estimation of the geometry (camera parameters and cloud of 3D points) from images and the surface reconstruction.

A promising way of research is the design of deep learning methods that improve the surface reconstruction step.

We also would like to avoid supervised methods, which need dataset of environments generated by 3D scanner.

There are several reasons to do this: price/availability/experimental conditions of the scanner and time/effort of acquisition.

Thus we replace dataset obtained by 3D scanner by other dataset.

The dataset can be synthetic and collect surface segments that are known to be very probable in the environments.

The dataset can also include large environments reconstructed by a previous method (which is not deep learning), with a minority of manual corrections.

Then a network learns to replace a wrong or improbable segment of surface by a more probable one.

Thanks to the learning, we expect to improve previous surface reconstruction methods, for example when experimental conditions are difficult.

Two kinds of deep learning methods can potentially do this.

The non-generative methods (eg autoencoder) compute only one result, ie one corrected surface.

They have drawbacks: uncertainty of the result is unknown and user cannot choose the best among several results.

The generative methods (eg variational autoencoder or diffusion model) can remove these drawbacks since they provide several results, more precisely a distribution of results.

In experiment, we focus on complete outdoor environments reconstructed by using a 360 camera (fixed on a helmet or a vehicle) that moves hundreds of meters or more.

Short bibliography:

- J.Ho, A.Jain, P.Abbeel, Denoising diffusion probabilistic models, NeurIPS 2020.

- D.P.Kingma, M.Welling, Auto-encoding variational Bayes, ICLR 2014.

- M.Lhuillier, Surface reconstruction from a sparse point cloud by enforcing visibility consistency and topology constraints, CVIU 175, 2018.

- M.Lhuillier, Estimating the vertical direction in a photogrammetric 3D model, with application to visualization, CVIU 236, 2023.

- S.Peng, M.Niemeyer, L.Meschender, M.Pollefeys, A.Geiger, Convolutional occupancy networks, ECCV 2020.

- M.Prakash, A.Krull, F.Jug, Fully unsupervised diversity denoising with convolutional variational autoencoders, ICLR 2021.

- Y.Song, S.Ermon, Generative modeling by estimating gradients of the data distribution, NIPS 2019.

- R.Sulzer, L.Landrieu, R.Marlet, B.Vallet, Scalable surface reconstruction with Delaunay-graph neural networks, CGF 40(5) 2021.

Application:

Send a CV, transcripts (including scores and ranks) of BSc and MSc, motivation letter and recommendation letter(s) to [email protected] before Tuesday 30th April 2024.

Requirements: master's degree or engineer's degrees in 2024 or before, mathematical background (probability, statistics, linear algebra, optimization), programming skills in Python and Pytorch, knowledge in photogrammetry/computer vision is a plus.

Contract:

3-year contract started on october 2024, in the ComSee team at the Institut Pascal, Clermont-Ferrand, France.