-
Notifications
You must be signed in to change notification settings - Fork 5.9k
MultiControlNetModel is not supported for SD3ControlNetInpaintingPipeline #11208
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
do you have a use case where you would use multiple controlnet with inpaiting for sd3? cc @asomoza here too |
Yes, I have a specific use case requiring multiple ControlNets with SD3's inpainting capabilities. I would like to use depth control alongside inpainting to better preserve the anatomical features and structure of the original image. Combining inpainting with depth control sometimes produces better results than using inpainting alone. This approach helps maintain the original image's spatial relationships while targeting specific areas for regeneration. Here are examples of what I'm trying to do with SD3:
import torch
from diffusers import FluxControlInpaintPipeline
from diffusers.models.transformers import FluxTransformer2DModel
from transformers import T5EncoderModel
from diffusers.utils import load_image, make_image_grid
from image_gen_aux import DepthPreprocessor
from PIL import Image
import numpy as np
pipe = FluxControlInpaintPipeline.from_pretrained(
"black-forest-labs/FLUX.1-Depth-dev",
torch_dtype=torch.bfloat16,
)
# GPU optimization code...
pipe.to("cuda")
prompt = "a blue robot singing opera with human-like expressions"
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")
# Create mask for the robot's head
head_mask = np.zeros_like(image)
head_mask[65:580,300:642] = 255
mask_image = Image.fromarray(head_mask)
# Process depth map
processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image = processor(image)[0].convert("RGB")
output = pipe(
prompt=prompt,
image=image,
control_image=control_image,
mask_image=mask_image,
num_inference_steps=30,
strength=0.9,
guidance_scale=10.0,
generator=torch.Generator().manual_seed(42),
).images[0] Some result example:
import torch
import numpy as np
from PIL import Image
from diffusers import StableDiffusionControlNetInpaintPipeline, ControlNetModel
from diffusers.utils import make_image_grid
import controlnet_hinter
import cv2
# Setup model
controlnet = ControlNetModel.from_pretrained("lllyasviel/sd-controlnet-depth", torch_dtype=torch.float16)
pipe = StableDiffusionControlNetInpaintPipeline.from_pretrained(
"...inpaint_model...",
controlnet=controlnet,
torch_dtype=torch.float16
)
pipe.to("cuda")
# Load images
init_image = Image.open('image.jpg')
mask_img = Image.open('mask.png')
# Prepare mask
mask_array = np.array(mask_img) > 0
mask_array = cv2.resize(mask_array.astype(np.uint8),
(init_image.size[0], init_image.size[1]),
interpolation=cv2.INTER_NEAREST).astype(bool)
mask = Image.fromarray(mask_array.astype(np.uint8) * 255)
# Generate depth map
control_image = controlnet_hinter.hint_depth(init_image)
control_image = control_image.resize(init_image.size)
# Generation configuration
generator = torch.Generator("cuda").manual_seed(42)
config = {
"negative_prompt": "bad quality, worst quality",
"num_inference_steps": 30,
"guidance_scale": 7.5,
"strength": 0.7,
"controlnet_conditioning_scale": 0.6,
"control_guidance_start": 0.6,
"control_guidance_end": 0.8
}
# Generate image
output = pipe(
prompt="Elegant blonde woman displaying refined style...",
image=init_image,
mask_image=mask,
control_image=control_image,
# other parameters...
).images[0] Some result example: Additionally, my primary concern with SD3 inpainting is its significant issues with anatomical consistency. When using SD3 inpainting without depth control, I frequently encounter severe distortions in human faces and body proportions that reduce output quality. Adding depth control would help maintain proper structural integrity while inpainting, solving these anatomical problems that are more pronounced in SD3 than previous SD generations |
thanks @DanilaAniva |
Describe the bug
When using
StableDiffusion3ControlNetInpaintingPipeline
withSD3MultiControlNetModel
, I receive an error:NotImplementedError: MultiControlNetModel is not supported for SD3ControlNetInpaintingPipeline.
Reproduction
Example reproduction code:
Logs
System Info
Versions
Python version: 3.10.16 (main, Dec 11 2024, 16:24:50) [GCC 11.2.0]
PyTorch version: 2.2.0+cu118
CUDA version: 11.8
Diffusers version: 0.32.2
Transformers version: 4.50.3
Accelerate version: 1.7.0.dev0
Who can help?
@yiyixuxu @sayakpaul
The text was updated successfully, but these errors were encountered: