August 19, 2020

Rewriting a Deep Generative Model

Can the rules in a deep network be directly rewritten?

State-of-the-art deep nets are trained as black boxes, using huge piles of data and millions of rounds of optimization that can take weeks to complete. But what if we want change a learned program after all that training is done?

Since the start of my PhD I have been searching for a way to reprogram huge deep nets in a different way: I want to reconfigure learned programs by plugging pieces together instead of retraining them on big data sets. In my quest, I have found a lot of cool deep net structure and developed methods to exploit that structure. Yet most ideas do not work. Generalizable editing is elusive. Research is hard.

But now, I am delighted to share this finding:

Rewriting a Deep Generative Model (ECCV 2020 oral)
2 min video:

Editing a StyleGANv2 model of horses. After changing a rule, horses wear hats!

Unlike my GanPaint work, the focus of this paper is not on changing a single image, but on editing the rules of the network itself. To stretch a programming analogy: in previous work we figured out how to edit fields of a single neural database record. Now we aim to edit the logic of the neural program itself.

Here is how: to locate and change a single rule in a generative model, we treat a layer as an optimal linear associative memory that stores key-value pairs that associate meaningful conditions with visible consequences. We change a rule by rewriting a single entry of this memory. And we enable users to make these changes by providing an interactive tool.

I edit StyleGanV2 and Progressive GAN and show how to do things like redefine horses to let them wear hats, or redefine pointy towers to have buildings that sprout trees. My favorite effects are redefining kids eyebrows to be very bushy, and redefining the rule that governs reflected light in a scene, to invert the presence of reflections from windows.

Here is a 10 min video:

MIT news has a story about it here today:

Posted by David at August 19, 2020 11:11 AM

Hi David~ I have an unfeeling question. That is, will you find some research job in some university? I just think your research direction is interesting and want to be your student. Thank you~

Posted by: benfen at November 6, 2020 04:07 AM

Dear David,

Thanks for your illuminating work and framing -- I enjoyed watching your PhD defense. Congrats b.t.w.

I suspect that identifying 'Markov Blankets' might be a useful way to reason about why neurons cluster -- which could have rammifications for editing/programming purposes.

Specifically see Karl Friston's neuroscience work on 'Free Energy Minimization'



Posted by: Pindar Wong at December 6, 2021 09:00 PM
Post a comment

Remember personal info?