Stable Diffusion as a Webcam

How can you use stable diffusion for webcam? Lets take a look!

1. Setup Comfy UI

To start, we’re going to use a Stable Diffusion interface called Comfy UI, created by the developers of Stable Diffusion. Follow the installation instructions found on their Github Page:

2. Download the SDXL Turbo Checkpoint

Next, we’re going to use the SDXL Turbo checkpoint. This checkpoint will allow us to create images very quickly, which will be important for creating a live webcam feed. Download the model file from the Hugging Face repository here:

The file is called “sd_xl_turbo_1.0_fp16.safetensors”, place it into your Comfy UI checkpoints folder.

3. Download the Custom Node

We will need a custom node for Comfy UI that will allow us to capture webcam images and process them. Follow the installation instructions here:

4. Create the Workflow

To add webcam capture to your existing workflows, simply add the “CaptureWebcam” node, and encode that as your latent image. Here is a JASON file you can load into Comfy UI to set up the workflow automatically: Stable Diffusion Live Webcam.json

Here is what the workflow should look like:

Image showing the comfy UI workflow

5. Start the Webcam Capture

Open up your Comfy UI file directory, and then the Custom_nodes directory. Then open the “ComfyUI_toyxyz_test_nodes” directory and then the “CaptureCam” directory. And then launch “run.bat”. This will start the webcam capture software. In the Webcam drop-down menu, select the name of your webcam. Then press Start.

You may also want to adjust to size of the image captured, I usually set mine to 1344×768:

Image showing the webcamera capture app

6. Start Generating Images

Back in Comfy UI, check “Extra Options”, then check “Auto Queue”. This will let it generate the next image as soon as it’s done creating one. Now click “Queue Prompt” and you should see a stable diffusion version of your webcam feed!

image showing the auto queue feature in comfy ui

7. Virtual Webcam

At this point, you can see yourself as a dog in space. But we want to spice up our zoom meetings.

Download and install OBS Studio:

Run the application and click the plus (+) button under Sources. Select Window capture and press OK.

image showing the new source interface in OBS

Then select your web browser as the window and press OK.

Now you can switch your browser tab back to Comfy UI, where you should see your workflow as well as the images you’re generating.

Using the Alt key, then dragging one of the sides of the window will allow you to select only the image preview. Then scale it so it fits the OBS preview window. Now you’re ready to click “Start Virtual Camera” and you’re done!

Leave a Comment

Your email address will not be published. Required fields are marked *