Illustration Research

Illustrations pose an important image processing challenge. Not only are sketches sparse and noisy, but there exists a large diversity of drawing styles. Furthermore, very precise results are required for real-world usage scenarios. The research here focuses on tackling these problems to aid illustrators in the content creation process.

  • Smart Inker

    Smart Inker

    We present an interactive approach for inking, which is the process of turning a pencil rough sketch into a clean line drawing. The approach, which we call the Smart Inker, consists of several “smart” tools that intuitively react to user input, while guided by the input rough sketch, to efficiently and naturally connect lines, erase shading, and fine-tune the line drawing output. Our approach is data-driven: the tools are based on fully convolutional networks, which we train to exploit both the user edits and inaccurate rough sketch to produce accurate line drawings, allowing high-performance interactive editing in real-time on a variety of challenging rough sketch images. For the training of the tools, we developed two key techniques: one is the creation of training data by simulation of vague and quick user edits; the other is a line normalization based on learning from vector data. These techniques, in combination with our sketch-specific data augmentation, allow us to train the tools on heterogeneous data without actual user interaction. We validate our approach with an in-depth user study, comparing it with professional illustration software, and show that our approach is able to reduce inking time by a factor of 1.8x while improving the results of amateur users.

  • Mastering Sketching

    Mastering Sketching

    We present an integral framework for training sketch simplification networks that convert challenging rough sketches into clean line drawings. Our approach augments a simplification network with a discriminator network, training both networks jointly so that the discriminator network discerns whether a line drawing is a real training data or the output of the simplification network, which in turn tries to fool it. This approach has two major advantages: first, because the discriminator network learns the structure in line drawings, it encourages the output sketches of the simplification network to be more similar in appearance to the training sketches. Second, we can also train the networks with additional unsupervised data: by adding rough sketches and line drawings that are not corresponding to each other, we can improve the quality of the sketch simplification. Thanks to a difference in the architecture, our approach has advantages over similar adversarial training approaches in stability of training and the aforementioned ability to utilize unsupervised training data. We show how our framework can be used to train models that significantly outperform the state of the art in the sketch simplification task, despite using the same architecture for inference. We additionally present an approach to optimize for a single image, which improves accuracy at the cost of additional computation time. Finally, we show that, using the same framework, it is possible to train the network to perform the inverse problem, i.e., convert simple line sketches into pencil drawings, which is not possible using the standard mean squared error loss.

  • Sketch Simplification

    Sketch Simplification

    We present a novel technique to simplify sketch drawings based on learning a series of convolution operators. In contrast to existing approaches that require vector images as input, we allow the more general and challenging input of rough raster sketches such as those obtained from scanning pencil sketches. We convert the rough sketch into a simplified version which is then amendable for vectorization. This is all done in a fully automatic way without user intervention. Our model consists of a fully convolutional neural network which, unlike most existing convolutional neural networks, is able to process images of any dimensions and aspect ratio as input, and outputs a simplified sketch which has the same dimensions as the input image. In order to teach our model to simplify, we present a new dataset of pairs of rough and simplified sketch drawings. By leveraging convolution operators in combination with efficient use of our proposed dataset, we are able to train our sketch simplification model. Our approach naturally overcomes the limitations of existing methods, e.g., vector images as input and long computation time; and we show that meaningful simplifications can be obtained for many different test cases. Finally, we validate our results with a user study in which we greatly outperform similar approaches and establish the state of the art in sketch simplification of raster images.

Publications

  • Real-Time Data-Driven Interactive Rough Sketch Inking
  • Learning to Restore Deteriorated Line Drawing
    • Learning to Restore Deteriorated Line Drawing
    • Kazuma Sasaki, Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa
    • The Visual Computer (Proc. of Computer Graphics International), 2018
  • Mastering Sketching: Adversarial Augmentation for Structured Prediction
    • Mastering Sketching: Adversarial Augmentation for Structured Prediction
    • Edgar Simo-Serra*, Satoshi Iizuka*, Hiroshi Ishikawa (* equal contribution)
    • ACM Transactions on Graphics (Presented at SIGGRAPH), 2018
  • Joint Gap Detection and Inpainting of Line Drawings
    • Joint Gap Detection and Inpainting of Line Drawings
    • Kazuma Sasaki, Satoshi Iizuka, Edgar Simo-Serra, Hiroshi Ishikawa
    • Conference in Computer Vision and Pattern Recognition (CVPR), 2017
  • Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup
    • Learning to Simplify: Fully Convolutional Networks for Rough Sketch Cleanup
    • Edgar Simo-Serra*, Satoshi Iizuka*, Kazuma Sasaki, Hiroshi Ishikawa (* equal contribution)
    • ACM Transactions on Graphics (SIGGRAPH), 2016

Source Code

  • Sketch Simplification Network
    • Sketch Simplification Network, 1.0 (Dec, 2017)
    • Sketch Simplification Convolutional Neural Network
    • Edgar Simo-Serra and Satoshi Iizuka

Datasets

  • Da Vinci Dataset
    • Da Vinci Dataset
    • Annotated line drawing sketches drawn by Leonardo Da Vinci.