Image - Semantics Consistency

4c46a8b1 9a2a 476e 8a1c b2791484f0a6.png 500 500

Organization: Autonomous Systems Lab

Involved Host(s): Cadena Cesar

Abstract: In this project we will exploit generative models (e.g. GANs) to provide introspection to semantic segmentation networks in open-world settings as faced by mobile robots.

Description: Current state of the art semantic segmentation methods are based on deep learning. However, this methods are known to fail when the deployment environment does not follow the data distribution of the training dataset [1]. By taking advantage of state of the art generative models, this project will propose a way of detecting inconsistencies produced by the semantic segmentation network. The student will build on top of, and improve, previous research [2,3] with an extra objective for modularity. [1] Blum H, Sarlin PE, Nieto J, Siegwart R, Cadena C. The Fishyscapes Benchmark: Measuring Blind Spots in Semantic Segmentation. arXiv preprint arXiv:1904.03215. 2019 Apr 5. [2] Haldimann D, Blum H, Siegwart R, Cadena C. This is not what I imagined: Error Detection for Semantic Segmentation through Visual Dissimilarity. ArXiv, abs/1909.00676. 2019 March 15. [3] Lis K, Nakka K, Salzmann M, Fua P. Detecting the Unexpected via Image Resynthesis. arXiv preprint arXiv:1904.07595. 2019 Apr 16.

Work Packages: - Implement state of the art semantic segmentation networks. - Implement state of the art GANs for image generation. - Investigate distance learning and cycle consistency. - Propose, implement and validated a method for detecting inconsistencies (mistakes) on semantic segmentation pipelines.

Requirements: - Highly motivated and independent student. - Strong C++ and/or Python coding skills. - Experience with deep learning frameworks (e.g: Tensorflow, PyTorch). - Students from outside of D-MAVT (particularly, from D-INFK, D-ITET, D-PHYS, and D-MATH) are also highly encouraged to apply.

Contact Details: If you are interested, please send your transcripts and CV to Cesar Cadena ( and Hermann Blum (

Deep Learning Generative models Semantic Segmentation Robotics Scene Understanding

Labels: Semester Project Master Thesis CLS Student Project (MPG ETH CLS)
Topics: Information, Computing and Communication Sciences