FineReco: Fine grain image recognition and intra/inter-collection links

Call for expression of interest description

The Marie S. Curie Postdoctoral Fellowship (MSCA-PF) programme is a highly prestigious renowned EU-funded scheme. It offers talented scientists a unique chance to set up 2-year research and training projects with the support of a supervising team. Besides providing an attractive grant, it represents a major opportunity to boost the career of promising researchers.  

Research laboratories in Brittany arethus looking for excellent postdoctoral researchers with an international profile to write a persuasive proposal to apply for a Marie S. Curie Postdoctoral Fellowship grant in 2024 (deadline of the EU call set on 11 September 2024). The topic and research team presented below have been identified in this regard.

Main Research Field

• Information Science and Engineering (ENG)

Research sub-field(s)

Computer science

Keywords

Fine-grain image recognition, Image Instance Retrieval, Patrimonial archives, Computer vision, Deep learning, recent neural architectures, ViT, CNN

Research project description

Some photographic archives are made up of multiple images that are almost identical, taken however at different times, by different photographers, under often very different lighting, framing, focal length and composition conditions. Examples include images of built heritage, landscapes and works of art. Identifying these similar photos makes it possible to create links between the elements of vast collections of images, allowing to structure collections and enabling curators to explore them more effectively.

The state of the art for CNN- or Transformer-based instance retrieval is very diverse, with many candidate techniques available.

Established as well as emerging and innovative neural architectures propose full pipelines or specific image representations for classification and instance retrieval such as DeLF, How, MAC, RMAC, GeM, CVNet, AILIR, Reranking Transformers, IRT, Selective Local Features, PWA, MultiScale-CNN, LDD, DeepRetrieval, IME-CNN, ....

Their respective performance has only partially evaluated and many grey zones remain about their precision, recall, their ability to filter out false positives, their capacity to deal with metric learning, manifolds, their reaction to data augmentation and also their ability to scale up (indexing millions of photos) as well as about their detailed computational needs (learning, inference).

In addition to their respective strengths and weaknesses, existing contributions are challenged with relatively less studied issues such as their capacity to cope with dynamically growing collections, with continuous learning and distribution shifts, with out-of-core indexing, or with using image datasets strongly departing from well controlled lab benchmarks.

The aim of this project is to study in depth the multiple DNN-based techniques used today to perform accurate image instance retrieval. Understanding how the best contributions react when facing such difficulties will ground research aimed at compensating, to some extent, the observed flaws.

A few references:

C. Bai et al. Unsupervised adversarial instance-level image retrieval. IEEE Trans. on Multimedia, 23, 2021.

T.-T. Do et al. From selective deep convolutional features to compact binary representations for image retrieval. ACM Trans. Multimedia Computing, Communications and Applications, 15(2), 2019.

A. El-Nouby et al. Training vision transformers for image retrieval. CoRR, abs/2102.05644, 2021.

W. Chen et al. Deep learning for instance retrieval: a survey. IEEE TPAMI vol 45, n 6, 06/2023

Supervisor(s)

The Postdoctoral Fellow will be supervised by Laurent Amsaleg, senior CNRS researcher. Laurent leads the Linkmedia research team  at the IRISA -INRIA  lab in Rennes, France. His research interests include high dimensional indexing at scale and multimedia analytics, as well as the many facets of the security issues in relation with the processing of extremely large collections of multimedia material. He supervised the theses of 13 doctoral students, authored over a hundred international publication (with recent ICLR, CVPR, NeurIPS), co-founded a startup, Videntifier. In 2019, he was general chair of ACM Multimedia. He has participated in over a dozen national and international projects, and is co-author of three patents.

Department/

Research                    

IRISA is one of the largest French research laboratories (more than 850 people) in the field of computer science and information technologies. Structured into seven scientific departments, the laboratory is a research center of excellence with scientific priorities such as bioinformatics, systems security, new software architectures, virtual reality, big data analysis and artificial intelligence.

Located in Rennes, Lannion and Vannes, IRISA is at the heart of a rich regional ecosystem for research and innovation and is positioned as the reference in France with a  internationally recognized expertise through numerous European contracts and international scientific collaborations. Focused on the future of computer science and necessarily internationally oriented, IRISA is at the very heart of the digital transition of society and of innovation at the service of cybersecurity, health, environment and ecology, transport, robotics, energy, culture and artificial intelligence. IRISA is a joint-venture resulting from the collaboration between nine institutions, in alphabetical order: CentraleSupelec, CNRS, ENS Rennes, IMT Atlantique, Inria, INSA Rennes, Inserm, Universite Bretagne Sud, Universite de Rennes. More information on the IRISA website: https://www.irisa.fr/en/

The research will take place inside the Linkmedia team which is concerned with the processing of extremely large collections of multimedia material. There are about 20ppl in this team, 6 senior researchers, a dozen of PhD students as well as engineers and post-doctoral fellows.

Getting rich, meaningful and deep insight from collections, however, remains today hardly achievable because of the heterogeneity (and semantic) gap between modalities, because of the scale of the collections, because of the

complex, hidden and implicit relationships between the items they contain, etc. To that end, Linkmedia contributes multimedia analytics algorithms to automatically process collections, eventually producing knowledge usable by humans. This involves a lot of deep learning, computer vision and NLP. Linkmedia is a joint team where Inria, CNRS, Univ. Rennes 1 and Insa researchers collaborate. http://www-linkmedia.irisa.fr/

Location

IRISA (Institut de Recherche en Informatique et Systèmes Aléatoires)

Campus Université de Beaulieu

35042, Rennes, France

Suggestion for interdisciplinary / intersectoral secondments and placements

A 4--6-month secondment can be envisaged at the Ouest-France company, the largest newspaper in France, located in Rennes. Ouest-France has a continuously growing archive comprising close to 40M images and relying on fast and accurate instance retrievals as well as on meaningful navigational links is key for journalists and curators. Absolutely unique datasets exist there, strongly departing from the in vitro benchmarks used in every single academic study. No doubt that the best modern instance retrieval systems will be seriously challenged by such original in vivo settings, which in turn offers opportunities to conduct disruptive research. A tight partnership exists between Linkmedia and Ouest-France and that secondment will be co-supervised by Laurent Amsaleg (Linkmedia) and Michel Le Nouy (Ouest-France).

Deadline for sending an expression of interest

5th May 2024

Selection of the most promising application(s)

May – June 2024

Writing the MSCA-PF proposal with the support of the above-mentioned supervisor(s)

June – September 2024

  MSCA-PF 2024 call deadline

11th September 2024

Publication of the MSCA-PF evaluation results

February 2025

Start of the MSCA-PF project (if funded)

 May 2025 (at the earliest)