davidbau.com

Rewriting a Deep Generative Model

Can the rules in a deep network be directly rewritten?

State-of-the-art deep nets are trained as black boxes, using huge piles of data and millions of rounds of optimization that can take weeks to complete. But what if we want change a learned program after all that training is done?

Since the start of my PhD I have been searching for a way to reprogram huge deep nets in a different way: I want to reconfigure learned programs by plugging pieces together instead of retraining them on big data sets. In my quest, I have found a lot of cool deep net structure and developed methods to exploit that structure. Yet most ideas do not work. Generalizable editing is elusive. Research is hard.

But now, I am delighted to share this finding:

Rewriting a Deep Generative Model (ECCV 2020 oral)
Website: https://rewriting.csail.mit.edu
Code: https://github.com/davidbau/rewriting
Preprint: https://arxiv.org/pdf/2007.15646.pdf
2 min video: https://www.youtube.com/watch?v=i2_-zNqtEPk

Editing a StyleGANv2 model of horses. After changing a rule, horses wear hats!

Unlike my GanPaint work, the focus of this paper is not on changing a single image, but on editing the rules of the network itself. To stretch a programming analogy: in previous work we figured out how to edit fields of a single neural database record. Now we aim to edit the logic of the neural program itself.

Here is how: to locate and change a single rule in a generative model, we treat a layer as an optimal linear associative memory that stores key-value pairs that associate meaningful conditions with visible consequences. We change a rule by rewriting a single entry of this memory. And we enable users to make these changes by providing an interactive tool.

I edit StyleGanV2 and Progressive GAN and show how to do things like redefine horses to let them wear hats, or redefine pointy towers to have buildings that sprout trees. My favorite effects are redefining kids eyebrows to be very bushy, and redefining the rule that governs reflected light in a scene, to invert the presence of reflections from windows.

Here is a 10 min video:
https://rewriting.csail.mit.edu/video/

MIT news has a story about it here today:
http://news.mit.edu/2020/rewriting-rules-machine-generated-art-0818

Posted by David at August 19, 2020 11:11 AM