this post was submitted on 01 Oct 2023
1368 points (97.8% liked)

Memes

45130 readers
3667 users here now

Rules:

  1. Be civil and nice.
  2. Try not to excessively repost, as a rule of thumb, wait at least 2 months to do it if you have to.

founded 5 years ago
MODERATORS
 
you are viewing a single comment's thread
view the rest of the comments
[–] CodeInvasion@sh.itjust.works 17 points 11 months ago

This is done by combining a Diffusion model with ControlNet interface. As long as you have a decently modern Nvidia GPU and familiarity with Python and Pytorch it's relatively simple to create your own model.

The ControlNet paper is here: https://arxiv.org/pdf/2302.05543.pdf

I implemented this paper back in March. It's as simple as it is brilliant. By using methods originally intended to adapt large pre-trained language models to a specific application, the author's created a new model architecture that can better control the output of a diffusion model.