EyeBAG: Accurate Control of Eye Blink and Gaze Based on Data Augmentation Leveraging Style Mixing

Bryan S Kim ¹, Jeong Young Jeong ², Wonjong Ryu ²

¹ KAIST
² Innerverz

Abstract

Recent developments in generative models have enabled the generation of photo-realistic human face images, and downstream tasks utilizing face generation technology have advanced accordingly. However, models for downstream tasks are yet substandard at eye control (e.g. blink, gaze redirection). To overcome such eye control problems, we introduce a novel framework consisting of two distinct modules: a blink control module and a gaze redirection module. We also propose a novel data augmentation method to train each module, leveraging style mixing to obtain images with desired features. We show that our framework produces eye-controlled images of high quality, and demonstrate how it can be used to improve the performance of downstream tasks.

Overview of the EyeBAG Model

Blink Control

Blink Control Module

An open eye image is encoded and input to an Adaptive Instance Normalization Residual Block (AdaIN ResBlock), along with a blink score of either 0, 0.5, or 1. Blink scores of 0 and 0.5 cause generation of closed eye images and uncertain eye images, respectively. Blink scores of 1 cause reconstruction of open eye images. These blink scores pass through a trainable multilayer perceptron (MLP). Closed eye and uncertain eye generation are learned simultaneously to effectively learn the entire blinking process, as uncertain eyes serve as the intermediate stage between being ‘open’ and ‘closed’.

Image Generation by the Blink Control Module

Gaze Redirection

Gaze Redirection Module

Data preprocessing is done for each eye image in the dataset to create an empty-eye image, cropped iris mask, and eye style image. The task of the module generator is to reconstruct the GT image based on the three inputs.

Input of horizontal shift value and vertical shift value is all that is needed to create a gaze-redirected image. Any value can be input as the horizontal or vertical shift amount, allowing for gaze redirection with complete freedom.

Eye Roll Grid

Iris Style Transfer

Iris style is preserved with high quality, and iris styles of other face images can be transferred to the original image. For the figure above, all three images on the bottom row use the same empty-eye image and cropped iris mask, but use a different eye style image.

Applications

Blink Detection

Blink score measurement with the discriminator of our blink control module is highly accurate and consistent. Accordingly, the blink control module discriminator can be used as a pre-trained blink detection network.

Performance Enhancement of Downstream Tasks

Performance of face swapping downstream tasks is enhanced by data supplement. When the face swapping target has eyes closed, models trained only on FFHQ give poor results. Supplementing the model with blink data generated by EyeBAG improves performance.

Paper

BibTeX


      @misc{kim2023eyebag,
        title={EyeBAG: Accurate Control of Eye Blink and Gaze Based on Data Augmentation Leveraging Style Mixing}, 
        author={Bryan S. Kim and Jeong Young Jeong and Wonjong Ryu},
        year={2023},
        eprint={2306.17391},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
      }