Tutorial:Upscale texture using ai

What is this about ?

In this article, we will explore a method for upscaling game textures using AI. This process can be partially automated, enabling batch processing of textures.

Setting up the work environment

Before proceeding, it's essential to set up the correct environment with all the necessary tools and configurations.

Note that you can adapt tools and methods based on your preferences and expertise.

Required software

Bellow are the required tools and their configuration:

Ubuntu WSL
Gimp
Paint.Net
AutoHotKey
Upscayl
RPFM or AssetEditor
Python: Any form will suffice, I personally use the Python binary built-in Ubuntu WSL

Software configuration & plugins

I. Gimp batch processing plugin

Later on, we will need to convert DDS textures to PNG to allow Upscayl to work its magic. Therefore, we'll utilize a custom Python-fu plugin to perform batch texture conversion instead of doing it manually.

To do so, put the code bellow in the following folder (make sure to add your Windows Username):

C:\Users\<username>\AppData\Roaming\GIMP\2.10\plug-ins\dds_to_png.py

Click to toggle code

from gimpfu import *


def batch_dds_to_png(directory):
    import os

    pdb.gimp_message("Processing directory: " + directory)

    # Find all DDS files in the directory
    filelist = [f for f in os.listdir(directory) if f.lower().endswith('.dds')]

    pdb.gimp_message("List of files to convert: " + ", ".join(filelist))

    if len(filelist) == 0:
        pdb.gimp_message("No DDS files found in the specified directory.")
        return

    pdb.gimp_message("Initiating batch conversion from DDS to PNG...")

    for filename in filelist:
        filepath = os.path.join(directory, filename)

        pdb.gimp_message("Currently converting: " + filename)

        # Open the DDS file
        image = pdb.file_dds_load(filepath, filepath, 1, 0)

        # Get the active layer
        drawable = pdb.gimp_image_get_active_layer(image)

        # Save as PNG
        png_filepath = os.path.join(directory, os.path.splitext(filename)[0] + '.png')
        pdb.file_png_save_defaults(image, drawable, png_filepath, png_filepath)

        # Delete the image to free memory
        pdb.gimp_image_delete(image)

    pdb.gimp_message("Batch conversion completed.")


register(
    "python-fu-batch-dds-to-png",
    "Batch DDS to PNG",
    "Convert all DDS files in a directory to PNG",
    "Toumai",
    "Toumai",
    "2023",
    "Batch DDS to PNG...",
    "",
    [
        (PF_DIRNAME, "directory", "Directory", "")
    ],
    [],
    batch_dds_to_png,
    menu="<Image>/File/Batch/"
)

main()

II. AutoHotKey script

Since we cannot re-use Gimp to convert from PNG to DDS, we will have to automate the manual conversion through Paint.Net. To do so, put the following code in :

C:\Users\<username>\Documents\AutoHotke\/Paint.Net PNG to DDS.ahk**

Click to toggle code

; Specify the directory and file extension to search for
directory := "C:\\Users\\Guild\\Documents\\4k_demo\\base_colour_upscayled_RealESRGAN_General_x4_v3_x4\\"
extension := "*.png"


; Loop through each file in the directory
Loop, Files, % directory extension
{
    ; Open the file with Paint.NET
    Run, "C:\\Program Files\\paint.net\\PaintDotNet.exe" "%A_LoopFileFullPath%"

    ; Wait for Paint.NET to become active
    Sleep, 1500

    ; Save As dialog (Ctrl+ALT+S)
        Send, ^+s

    ; wait for the dialog to open
        Sleep, 500

    ; Tab to the "Save as type" dropdown field
        Send, {Tab}

    ; Type in the desired format (.dds) and press Enter
        Send, d{Enter}

    ; Wait for export option dialog
        Sleep, 500
        Send, {Enter}

    ; Wait for the export to be done
        Sleep, 7000

    ; ; Close the active file (Ctrl+W)
        Send, ^w

    ; ; Wait for the file to close
        Sleep, 500
}

III. Custom Python script to replace old texture

This script will be used in the later part of this tutorial to replace old textures with the newly upscaled ones. Don't worry about it for now; its usage will make more sense when you reach that part of the tutorial.

For now, just create a new Python file and paste the following code:

Click to toggle code

import os
import shutil

def move_matching_files(src_dir, dst_dir):
        # Loop through all filenames in src_dir
        for filename in os.listdir(src_dir):
            src_filepath = os.path.join(src_dir, filename)

            # Skip directories
            if os.path.isdir(src_filepath):
                continue

            # Search recursively for matching files in dst_dir
            for root, dirs, files in os.walk(dst_dir):
                if filename in files:
                    dst_filepath = os.path.join(root, filename)

                    # Move the file to the matched place in dst_dir
                    shutil.move(src_filepath, dst_filepath)
                    print(f"Moved {src_filepath} to {dst_filepath}")
                    break  # Stop search as file is moved

if __name__ == "__main__":
    src_dir = "/mnt/c/Users/Guild/Documents/4k_demo/final"
    dst_dir = "/mnt/c/Users/Guild/Documents/4k_demo/grn"
    move_matching_files(src_dir, dst_dir)

Getting started

In this example, we will upscale the textures of Greenskins' trolls, including troll river, troll river hag, and their respective weapons.

Step 1: Extracting Textures

The first step is to extract all the textures that need upscaling. In this case, we'll focus on the base_colour and normal textures. To achieve this, we'll use the AssetEditor and apply a filter to find the desired textures: ^grn_troll.*[normal|base_colour].dds$

Additionally, ensure that you do not export low-poly textures, as we are only interested in upscaling high-quality textures.

Step 2: Centralizing Textures

Next, we need to centralize all the textures in the same folder or directory. Since we're using different models for each type of texture (normal and base_colour), it's a good practice to keep them separate. Below are the commands to achieve this using the shell in Ubuntu WSL:

# Create new folders to store edited textures
mkdir -p /path/to/wip/normal /path/to/wip/base_colour

# Move all the **normal** textures to the designated folder
cp **/*normal.dds /path/to/wip/normal

# Move all the **base_colour** textures to the designated folder
cp **/*base_colour.dds /path/to/wip/base_colour

Step 3: Converting DDS to PNG

Now that we have all the vanilla DDS files, we need to convert them to PNG format for upscaling.

Launch Gimp and ensure you have installed the custom plugin mentioned at the beginning of this guide.
. In Gimp, select "Batch DDS to PNG" from the options, as shown in the image below
A pop-up window will appear. Select the folder containing the vanilla normal textures and begin the conversion process, as indicated in the image bellow
Once the conversion is complete, you will see the message Batch conversion completed in the Error Console Panel as shown bellow
Once done with the normal textures, make sure to repeat this step for the base_colour folder to convert those textures to PNG format as well.

Step 4: Upscaling the textures

Now it's time to upscale the textures using your chosen models. Here are the models I personally recommend, but feel free to experiment or import your custom models:

Normal Textures: Ultramix Balanced
Base Colour Textures: Fast Real-ESRGAN

Let's get started:

To prepare for upscaling, we need to remove or move the vanilla DDS textures to avoid conflicts. You can do this using the following commands:
```
cd /path/to/wip/ && rm normal/*dds rm base_colour/*dds
```

We are now ready to launch upscayl.
In the left option panel, toggle the batch upscayl option.
Select the folder containing your normal textures.
Choose the model you want to use (in this case, Ultramix Balanced).
Do not specify an output folder.
Click the purple Upscayl button
Wait for the processing to complete.
By default Upscayl will have created a new folder containing the upscaled textures
After completing this step, be sure to repeat the process with the base_colour folder and adjust the model accordingly.

Step 5: Putting the Upscaled Textures Back into the Game

Congratulations, you have successfully upscaled the textures! Now, it's time to put these textures back into the game ; since GIMP doesn't support the necessary compression, we will use Paint.Net and automate the process with AutoHotKey.

Open Paint.Net
Open one of the normal textures (any will do)
Click Save As or Ctrl+Shift+S
In the Save as type option, change it from PNG (*.png*) to Direct Draw Surface (DDS) (*.dds)
In the Export Configuration, make sure to select the following settings:
1. Normal Textures: BC3 (sRGB, DX 10+)
2. Base Colour Textures: BC1 (sRGB, DX 10+)
Finally, click OK

Now, Paint.Net is configured to convert your normal textures, as such we can automate this process to convert all the extures using AutoHotKey:

If you followed the installation process at the beginning of this demo, you should have a script called Paint.Net PNG to DDS.ahk.
Open that file and make the following changes
1. In line 2, put the path to your DDS normal texture folder. Make sure to escape all backslashes as shown below.
2. In line 32, if your GPU is not high-end, you can increase the waiting time (7 seconds in the screenshot below).

Once you've edited the script, open the folder containing the script
Double-click on it
AutoHotKey will start simulating the process of converting the upscaled PNG to DDS. Do not interrupt the process by using your mouse or keyboard, as the script cannot recover from interruptions.
Once you are done with the normal textures, you can repeat the entire Step 5, but this time focus on the base_colour textures. Make sure to select the BC1 (sRGB, DX 10+) compression for the base_colour textures during the conversion.

Step 6: Replacing Vanilla Textures with Upscaled Ones

At this point, you should have all the upscaled DDS textures, as shown below.

It is now time to replace all the vanilla textures in the exported folder (see Step 1) with the upscaled ones. However, if you used the AssetEditor to export the textures, it may have also exported other files in the folder. Let's first remove those unnecessary files using the following commands:

cd /path/to/exported/textures
find . -name 'low_poly' -type d -exec rm -rf {} \;
find . -name '*lowlod*' -delete
find . -name '*.anim' -delete
find . -name '*.bone_inv_trans_mats' -delete
find . -name '*base_colour*' -delete
find . -name '*xml.material' -delete
find . -name '*material_map.dds' -delete
find . -name 'materials' -type d -delete
find . -name '*.wsmodel' -delete
find . -name '*mask.dds' -delete
find . -name '*rigid_model_v2' -delete

Once you've completed this step, you'll need to edit the python script transfert.py as set up at the beginning of this tutorial.

Make sure to replace the following variables:

line 24: src_dir: the folder where you can find the Upscaled DDS textures.
line 25: dst_dir: the folder exported through the AssetEditor.

Python script to replace old textures with upscaled one.jpg

After editing the script, launch it and verify that the upscaled textures have been successfully moved.

Step 7: Importing Upscaled Textures with RPFM / AssetEditor

Now, you're almost done! At this stage, you can use RPFM or AssetEditor to import the folder containing the Upscaled DDS textures:

Open RPFM
Create a new packfile
Add the folder containing the upscaled textures
You'll see the end result as shown bellow
Save as usual

Step 8: Profit

Congratulations! You've successfully upscaled and integrated your textures back into the game. Enjoy the enhanced graphics!

If you have any more questions or need further assistance, please feel free to contact me on Discord: toumai#3164