S2D2: Simple Stable Diffusion based on Diffusers

Diffusers-based simple generating image module with upscaling features for jupyter notebook, ipython or python interactive shell

Features

• ☑ Just prepare safetensors files to go
• ☑ Run Hires.fix without AUTOMATIC1111/stable-diffusion-webui
  • ☑ Latent Upscaler
  • ☒ GAN models
• ☑ Multi-stage upscaling (extension of Hires.fix)
• ☑ LoRA
• ☒ Controlnet
• ☒ Multi-batch generation (Only single generation is supported)

Schedule

• Add ControlNet (Convert SD safetensors file to diffusers model files)

Getting Started

1. Install libraries

pip install -r requirements.txt

2. Prepare safetensors files of SD models and LoRA(option)

• Only safetensors file is supported.
• Place the files in the directory of your choice.

Ex.

3. Run jupyter notebook

cd s2d2
jupyter notebook

4. Import main class and load LoRA(option)

from s2d2 import StableDiffusionImageGenerator
generator = StableDiffusionImageGenerator(
    r"C:\xxx\Counterfeit-V30.safetensors",
)
# Load LoRA (multi files)
generator.load_lora(r"C:\xxx\lora_1.safetensors", alpha=0.2)
generator.load_lora(r"C:\xxx\lora_2.safetensors", alpha=0.15)

5. Generate image using enhance features(Hires.fix and its extended upscaling)

image = generator.diffusion_enhance(
          prompt,
          negative_prompt,
          scheduler_name="dpm++_2m_karras", # [1]
          num_inference_steps=20, # [2]
          num_inference_steps_enhance=20, # [3]
          guidance_scale=10,  # [4]
          width=700, # [5]
          height=500, # [6]
          seed=-1, # [7]
          upscale_target="latent", # [8] "latent" or "pil". pil mode is temporary implemented.
          interpolate_mode="bicubic", # [9]
          antialias=True, # [10]
          upscale_by=1.8, # [11]
          enhance_steps=2, # [12] 2=Hires.fix
          denoising_strength=0.60, # [13]
          output_type="pil", # [14] "latent" or "pil"
          decode_factor=0.15, # [15] Denominator when decoding latents. Used to adjust the saturation of the image during decoding.
          decode_factor_final=0.18215, # [16] Denominator when decoding final latents.
          )
image.save("generated_image.jpg) # or just "image" to display image in jupyter

Correspondence of web ui and parameters

Parameters

🚧🚧🚧🚧🚧Under construction🚧🚧🚧🚧🚧

• Available schedulers are:

SCHEDULERS = {
    "unipc": diffusers.schedulers.UniPCMultistepScheduler,
    "euler_a": diffusers.schedulers.EulerAncestralDiscreteScheduler,
    "euler": diffusers.schedulers.EulerDiscreteScheduler,
    "ddim": diffusers.schedulers.DDIMScheduler,
    "ddpm": diffusers.schedulers.DDPMScheduler,
    "deis": diffusers.schedulers.DEISMultistepScheduler,
    "dpm2": diffusers.schedulers.KDPM2DiscreteScheduler,
    "dpm2-a": diffusers.schedulers.KDPM2AncestralDiscreteScheduler,
    "dpm++_2s": diffusers.schedulers.DPMSolverSinglestepScheduler,
    "dpm++_2m": diffusers.schedulers.DPMSolverMultistepScheduler,
    "dpm++_2m_karras": diffusers.schedulers.DPMSolverMultistepScheduler,
    "dpm++_sde": diffusers.schedulers.DPMSolverSDEScheduler,
    "dpm++_sde_karras": diffusers.schedulers.DPMSolverSDEScheduler,
    "heun": diffusers.schedulers.HeunDiscreteScheduler,
    "heun_karras": diffusers.schedulers.HeunDiscreteScheduler,
    "lms": diffusers.schedulers.LMSDiscreteScheduler,
    "lms_karras": diffusers.schedulers.LMSDiscreteScheduler,
    "pndm": diffusers.schedulers.PNDMScheduler,
}

Generated sample images

• Used Counterfeit-V30.safetensors
• Initial resolution: 696x496
• Upscale factor: 1.8
• Target resolution: 696x496(x1.8, nearest multiple of 8) = 1248x888

2-stage upscaling(Hires.fix)

N-stage upscaling(Ex.4)

Stepwise upscaling between the initial resolution and the target resolution.

Comparison of generating images without or with latent upscaling

• Without latent upscaling: Single generation@696x496

• With latent upcscaling: 2-stage generation(like Hires.fix, 696x496 to 1248x888)

• Prompt: "1girl, solo, full body, blue eyes, looking at viewer, hairband, bangs, brown hair, long hair, smile, blue eyes, wine-red dress, outdoor, night, moonlight, castle, flowers, garden"

• Negative prompt: "EasyNegative, extra fingers, fewer fingers, bad hands"

References

• AUTOMATIC1111/stable-diffusion-webui
• ddPn08/Radiata
• huggingface/diffusers/blob/main/scripts/convert_lora_safetensor_to_diffusers.py