Overview

The vast majority of modern consumer-grade cameras are equipped with an electronic rolling shutter (RS), which is becoming increasingly important in computer vision, robotics, and autonomous driving, etc. Although ubiquitous, its progressive exposure mechanism causes geometric distortion in the acquired images whenever the camera or the target moves. In the last two decades, researchers have made significant progress to remove the geometry distortion and improve the visual quality. In particular, (1) Since the tutorial entitled “Multi-view geometry for rolling shutter cameras” in ACCV 2016, many RS motion models and non-linear optimization methods have emerged. (2) At the same time, the rise of deep learning techniques has also facilitated research on RS-related problems in the past five years. (3) Most recently, incorporating hardware features of RS cameras with deep learning has pushed the field forward, especially for real images/videos with both camera and scene motion.

This half-day tutorial will cover the latest advances in this area from three aspects, i.e., motion modeling and optimization-based solutions, deep learning-based solutions, and joint hardware and deep learning-based solutions. Specifically, we will first systematically present geometric motion models (like discrete, continuous, and special motions) and optimization-based approaches. Then, we will introduce deep learning-based RS image processing methods, such as RS image correction and RS temporal super-resolution, with new results and benchmarks that have recently appeared. Finally, we will elaborate on the combination of hardware features of RS cameras (e.g., dual RS cameras and global reset feature) and deep learning to boost the correction of RS geometric distortions.

The course will cover the following units:

Overview of rolling shutter cameras

Geometric modeling and non-linear optimization of rolling shutter camera

• Discrete motion
• Continuous motion
• Special motion

New paradigm with deep learning

• Rolling shutter image correction
• Rolling shutter temporal super-resolution
• Building synthetic datasets

New trends with combination of hardware features and deep learning

• Dual rolling shutter cameras
• Rolling shutter with global reset feature
• Building real-world datasets

Challenges and future trends

Yuchao Dai

NWPU

Yinqiang Zheng

UTokyo

Bin Fan

NWPU

Zhihang Zhong

UTokyo

Zhixiang Wang

UTokyo / RIISE

Schedule

Title	Speaker	Time (PST)
Opening	Yuchao Dai	09:00 - 09:10
Part (1): Geometric modeling and non-linear optimization of rolling shutter camera	Yuchao Dai	09:10 - 09:40
Part (2): New paradigm with deep learning	Bin Fan	09:40 - 10:30
Coffee Break	-	10:30 - 10:45
Part (3): New trends with combination of hardware features and deep learning	Zhixiang Wang / Zhihang Zhong	10:45 - 11:30
Discussion and Final Remarks	ALL	11:30 - 11:45