How To Create Video Game Materials using Stable Diffusion (PBR)

Creating physically-based rendering materials in video games can be a challenging task, but one solution that has gained popularity recently is the use of stable diffusion in conjunction with Materialize. In this article, we will discuss the basics of using stable diffusion and Materialize for generating the material images and how we can add height maps, normal maps and how to get them working in Unreal Engine.

MitchJourn-E and other popular Stable Diffusion applications have the ability to “Tile” images, enabling them to be placed next to each other seamlessly, without appearing repetitive. The tilling image we create from Stable Diffusion can be considered the “diffuse map” in terms of video game materials.

In MitchJourn-E, the Seamless Tile check box can be found in the settings.

Creating the Material

Here’s an interesting stone texture image generated using the Seamless Tile feature. This is the prompt used:

incredibly detailed (rough rock stone)1 (flat texture)1

Negative Prompt: [(framed, ugly, tiling, blurry, bad anatomy, blurred, watermark, grainy, signature, cut off, draft)0.5] [(lowres text error cropped worst quality low quality normal quality jpeg artifacts signature watermark username blurry artist name deformed disfigured poorly drawn out of focus censorship amateur drawing bad art poor art messy drawing cartoon anime art painting ugly):1]

Steps: 50, Seed: 20610, Width: 512 Height: 512, Prompt weight: 7.5, Sampler: k_euler_a

With that, we have the most important part of our material created. Now we can start generating the other PBR components to bring our material to life.

Getting started with Materialize

From the creator of Materialize “Materialize is a stand alone tool for creating materials for use in games from images. You can create an entire material from a single image[…]” This means we can import our image and generate all the components of a PBR material without going into an image editor and manually defining the details.

Open the program and click the “O” button under the Diffuse Map section to import our texture:

Now we can click the “Create” buttons under each of the components that we want to generate. For this material, I’ll generate a height map and a normal map to give our stone some depth and additional texture. When clicking the create button, we have a bunch of options and presets to choose from. The defaults can work fine as well. Use the “Show Full Material” button to see how your changes will affect the final result.

The material after adding the Height map and Normal map

Using your Material in Unreal Engine

To export our PBR components, use the “Save Project” button to save each component to it’s own file location.

Now drag and drop the files into an Unreal Engine project folder. It will recognize the normal map texture and automatically format it accordingly. Open a new material and drag the imported files into the editor window.

Connect the diffuse map’s RGB to the Base Color node. Then connect the normal map’s RGB to the Normal node. For the Height map, first right click the background and create a Bump Offset. Then connect the height map’s alpha node to the Height node of the Bump Offset, and connect the Bump Offset to both UV nodes of the diffuse and normal map. You may also want to create a TextureCoordinate to increate the number of times to tile the image for the material. Plug that into the coordinate node on the bump offset.

Apply and save the material and we’re all set!

Stable diffusion material in Unreal Engine

In conclusion, using stable diffusion in conjunction with Materialize is a powerful solution for creating physically-based rendering materials in video games. It allows developers to create highly detailed and realistic materials by simulating the way that light interacts with surfaces, while also providing the ability to “tile” images seamlessly. Materialize is a convenient tool that can be used to generate all the components of a PBR material from a single image, making the process more efficient. With these tools and techniques, developers can create more realistic and immersive worlds for their players.

Start generating your own today

Learn how you can start generating your own incredible diffuse maps with the free and open source application MitchJourn-E. MitchJourn-E is a powerful AI image generation app that uses Invoke-AI as it’s backend to create beautiful images.

Leave a Comment

Your email address will not be published. Required fields are marked *