krahets/DiffSynth

Install from PyPI (version updates may be delayed; for latest features, install from source)

pip install diffsynth

If you meet problems during installation, they might be caused by upstream dependencies. Please check the docs of these packages:

• torch
• sentencepiece
• cmake
• cupy

from diffsynth.pipelines.qwen_image import QwenImagePipeline, ModelConfig
import torch

pipe = QwenImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="transformer/diffusion_pytorch_model*.safetensors"),
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="text_encoder/model*.safetensors"),
        ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="vae/diffusion_pytorch_model.safetensors"),
    ],
    tokenizer_config=ModelConfig(model_id="Qwen/Qwen-Image", origin_file_pattern="tokenizer/"),
)
prompt = "A detailed portrait of a girl underwater, wearing a blue flowing dress, hair gently floating, clear light and shadow, surrounded by bubbles, calm expression, fine details, dreamy and beautiful."
image = pipe(prompt, seed=0, num_inference_steps=40)
image.save("image.jpg")

import torch
from diffsynth.pipelines.flux_image_new import FluxImagePipeline, ModelConfig

pipe = FluxImagePipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="flux1-dev.safetensors"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder/model.safetensors"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="text_encoder_2/"),
        ModelConfig(model_id="black-forest-labs/FLUX.1-dev", origin_file_pattern="ae.safetensors"),
    ],
)

image = pipe(prompt="a cat", seed=0)
image.save("image.jpg")

import torch
from diffsynth import save_video
from diffsynth.pipelines.wan_video_new import WanVideoPipeline, ModelConfig

pipe = WanVideoPipeline.from_pretrained(
    torch_dtype=torch.bfloat16,
    device="cuda",
    model_configs=[
        ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="diffusion_pytorch_model*.safetensors", offload_device="cpu"),
        ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="models_t5_umt5-xxl-enc-bf16.pth", offload_device="cpu"),
        ModelConfig(model_id="Wan-AI/Wan2.1-T2V-1.3B", origin_file_pattern="Wan2.1_VAE.pth", offload_device="cpu"),
    ],
)
pipe.enable_vram_management()

video = pipe(
    prompt="A documentary photography style scene: a lively puppy rapidly running on green grass. The puppy has brown-yellow fur, upright ears, and looks focused and joyful. Sunlight shines on its body, making the fur appear soft and shiny. The background is an open field with occasional wildflowers, and faint blue sky and clouds in the distance. Strong sense of perspective captures the motion of the puppy and the vitality of the surrounding grass. Mid-shot side-moving view.",
    negative_prompt="Bright colors, overexposed, static, blurry details, subtitles, style, artwork, image, still, overall gray, worst quality, low quality, JPEG compression artifacts, ugly, deformed, extra fingers, poorly drawn hands, poorly drawn face, malformed limbs, fused fingers, still frame, messy background, three legs, crowded background people, walking backwards",
    seed=0, tiled=True,
)
save_video(video, "video1.mp4", fps=15, quality=5)

Detail page: ./examples/image_synthesis/

FLUX	Stable Diffusion 3

Kolors	Hunyuan-DiT

Stable Diffusion	Stable Diffusion XL

• HunyuanVideo: ./examples/HunyuanVideo/

https://github.com/user-attachments/assets/48dd24bb-0cc6-40d2-88c3-10feed3267e9

• StepVideo: ./examples/stepvideo/

https://github.com/user-attachments/assets/5954fdaa-a3cf-45a3-bd35-886e3cc4581b

• CogVideoX: ./examples/CogVideoX/

https://github.com/user-attachments/assets/26b044c1-4a60-44a4-842f-627ff289d006

We have integrated a series of image quality assessment models. These models can be used for evaluating image generation models, alignment training, and similar tasks.

Detail page: ./examples/image_quality_metric/

• ImageReward
• Aesthetic
• PickScore
• CLIP
• HPSv2
• HPSv2.1
• MPS

• Detail page: https://github.com/modelscope/Nexus-Gen
• Paper: Nexus-Gen: Unified Image Understanding, Generation, and Editing via Prefilled Autoregression in Shared Embedding Space
• Model: ModelScope, HuggingFace
• Dataset: ModelScope Dataset
• Online Demo: ModelScope Nexus-Gen Studio

• Detail page: ./examples/ArtAug/
• Paper: ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interaction
• Model: ModelScope, HuggingFace
• Online Demo: ModelScope AIGC Tab

FLUX.1-dev	FLUX.1-dev + ArtAug LoRA

• Detail page: ./examples/EntityControl/
• Paper: EliGen: Entity-Level Controlled Image Generation with Regional Attention
• Model: ModelScope, HuggingFace
• Online Demo: ModelScope EliGen Studio
• Dataset: EliGen Train Set

Entity Control Mask	Generated Image

• Project Page: Project Page
• Paper: ExVideo: Extending Video Diffusion Models via Parameter-Efficient Post-Tuning
• Code Example: ./examples/ExVideo/
• Model: ModelScope, HuggingFace

https://github.com/modelscope/DiffSynth-Studio/assets/35051019/d97f6aa9-8064-4b5b-9d49-ed6001bb9acc

• Project Page: Project Page
• Paper: Diffutoon: High-Resolution Editable Toon Shading via Diffusion Models
• Code Example: ./examples/Diffutoon/

https://github.com/Artiprocher/DiffSynth-Studio/assets/35051019/b54c05c5-d747-4709-be5e-b39af82404dd

• Project Page: Project Page
• Paper: DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesis
• Code Example: ./examples/diffsynth/

https://github.com/Artiprocher/DiffSynth-Studio/assets/35051019/59fb2f7b-8de0-4481-b79f-0c3a7361a1ea

• June 15, 2025 ModelScope's official evaluation framework, EvalScope, now supports text-to-image generation evaluation. Try it with the Best Practices guide.

• March 25, 2025 Our new open-source project, DiffSynth-Engine, is now open-sourced! Focused on stable model deployment. Geared towards industry. Offers better engineering support, higher computational performance, and more stable functionality.

• March 31, 2025 We support InfiniteYou, an identity preserving method for FLUX. Please refer to ./examples/InfiniteYou/ for more details.

• March 13, 2025 We support HunyuanVideo-I2V, the image-to-video generation version of HunyuanVideo open-sourced by Tencent. Please refer to ./examples/HunyuanVideo/ for more details.

• February 25, 2025 We support Wan-Video, a collection of SOTA video synthesis models open-sourced by Alibaba. See ./examples/wanvideo/.

• February 17, 2025 We support StepVideo! State-of-the-art video synthesis model! See ./examples/stepvideo.

• December 31, 2024 We propose EliGen, a novel framework for precise entity-level controlled text-to-image generation, complemented by an inpainting fusion pipeline to extend its capabilities to image inpainting tasks. EliGen seamlessly integrates with existing community models, such as IP-Adapter and In-Context LoRA, enhancing its versatility. For more details, see ./examples/EntityControl.

  • Paper: EliGen: Entity-Level Controlled Image Generation with Regional Attention
  • Model: ModelScope, HuggingFace
  • Online Demo: ModelScope EliGen Studio
  • Training Dataset: EliGen Train Set

• December 19, 2024 We implement advanced VRAM management for HunyuanVideo, making it possible to generate videos at a resolution of 129x720x1280 using 24GB of VRAM, or at 129x512x384 resolution with just 6GB of VRAM. Please refer to ./examples/HunyuanVideo/ for more details.

• December 18, 2024 We propose ArtAug, an approach designed to improve text-to-image synthesis models through synthesis-understanding interactions. We have trained an ArtAug enhancement module for FLUX.1-dev in the format of LoRA. This model integrates the aesthetic understanding of Qwen2-VL-72B into FLUX.1-dev, leading to an improvement in the quality of generated images.

  • Paper: https://arxiv.org/abs/2412.12888
  • Examples: https://github.com/modelscope/DiffSynth-Studio/tree/main/examples/ArtAug
  • Model: ModelScope, HuggingFace
  • Demo: ModelScope, HuggingFace (Coming soon)

• October 25, 2024 We provide extensive FLUX ControlNet support. This project supports many different ControlNet models that can be freely combined, even if their structures differ. Additionally, ControlNet models are compatible with high-resolution refinement and partition control techniques, enabling very powerful controllable image generation. See ./examples/ControlNet/.

• October 8, 2024. We release the extended LoRA based on CogVideoX-5B and ExVideo. You can download this model from ModelScope or HuggingFace.

• August 22, 2024. CogVideoX-5B is supported in this project. See here. We provide several interesting features for this text-to-video model, including

  • Text to video
  • Video editing
  • Self-upscaling
  • Video interpolation

• August 22, 2024. We have implemented an interesting painter that supports all text-to-image models. Now you can create stunning images using the painter, with assistance from AI!

  • Use it in our WebUI.

• August 21, 2024. FLUX is supported in DiffSynth-Studio.

  • Enable CFG and highres-fix to improve visual quality. See here
  • LoRA, ControlNet, and additional models will be available soon.

• June 21, 2024. We propose ExVideo, a post-tuning technique aimed at enhancing the capability of video generation models. We have extended Stable Video Diffusion to achieve the generation of long videos up to 128 frames.

  • Project Page
  • Source code is released in this repo. See examples/ExVideo.
  • Models are released on HuggingFace and ModelScope.
  • Technical report is released on arXiv.
  • You can try ExVideo in this Demo!

• June 13, 2024. DiffSynth Studio is transferred to ModelScope. The developers have transitioned from "I" to "we". Of course, I will still participate in development and maintenance.

• Jan 29, 2024. We propose Diffutoon, a fantastic solution for toon shading.

  • Project Page
  • The source codes are released in this project.
  • The technical report (IJCAI 2024) is released on arXiv.

• Dec 8, 2023. We decide to develop a new Project, aiming to release the potential of diffusion models, especially in video synthesis. The development of this project is started.

• Nov 15, 2023. We propose FastBlend, a powerful video deflickering algorithm.

  • The sd-webui extension is released on GitHub.
  • Demo videos are shown on Bilibili, including three tasks.
    • Video deflickering
    • Video interpolation
    • Image-driven video rendering
  • The technical report is released on arXiv.
  • An unofficial ComfyUI extension developed by other users is released on GitHub.

• Oct 1, 2023. We release an early version of this project, namely FastSDXL. A try for building a diffusion engine.

  • The source codes are released on GitHub.
  • FastSDXL includes a trainable OLSS scheduler for efficiency improvement.
    • The original repo of OLSS is here.
    • The technical report (CIKM 2023) is released on arXiv.
    • A demo video is shown on Bilibili.
    • Since OLSS requires additional training, we don't implement it in this project.

• Aug 29, 2023. We propose DiffSynth, a video synthesis framework.

  • Project Page.
  • The source codes are released in EasyNLP.
  • The technical report (ECML PKDD 2024) is released on arXiv.