How can you use stable diffusion for webcam? Lets take a look!
1. Setup Comfy UI
To start, we’re going to use a Stable Diffusion interface called Comfy UI, created by the developers of Stable Diffusion. Follow the installation instructions found on their Github Page: https://github.com/comfyanonymous/ComfyUI
2. Download the SDXL Turbo Checkpoint
Next, we’re going to use the SDXL Turbo checkpoint. This checkpoint will allow us to create images very quickly, which will be important for creating a live webcam feed. Download the model file from the Hugging Face repository here: https://huggingface.co/stabilityai/sdxl-turbo/tree/main
The file is called “sd_xl_turbo_1.0_fp16.safetensors”, place it into your Comfy UI checkpoints folder.
3. Download the Custom Node
We will need a custom node for Comfy UI that will allow us to capture webcam images and process them. Follow the installation instructions here: https://github.com/toyxyz/ComfyUI_toyxyz_test_nodes
4. Create the Workflow
To add webcam capture to your existing workflows, simply add the “CaptureWebcam” node, and encode that as your latent image. Here is a JASON file you can load into Comfy UI to set up the workflow automatically: Stable Diffusion Live Webcam.json
Here is what the workflow should look like:
5. Start the Webcam Capture
Open up your Comfy UI file directory, and then the Custom_nodes directory. Then open the “ComfyUI_toyxyz_test_nodes” directory and then the “CaptureCam” directory. And then launch “run.bat”. This will start the webcam capture software. In the Webcam drop-down menu, select the name of your webcam. Then press Start.
You may also want to adjust to size of the image captured, I usually set mine to 1344×768:
6. Start Generating Images
Back in Comfy UI, check “Extra Options”, then check “Auto Queue”. This will let it generate the next image as soon as it’s done creating one. Now click “Queue Prompt” and you should see a stable diffusion version of your webcam feed!
7. Virtual Webcam
At this point, you can see yourself as a dog in space. But we want to spice up our zoom meetings.
Download and install OBS Studio: https://obsproject.com/download
Run the application and click the plus (+) button under Sources. Select Window capture and press OK.
Then select your web browser as the window and press OK.
Now you can switch your browser tab back to Comfy UI, where you should see your workflow as well as the images you’re generating.
Using the Alt key, then dragging one of the sides of the window will allow you to select only the image preview. Then scale it so it fits the OBS preview window. Now you’re ready to click “Start Virtual Camera” and you’re done!