Error while loading Lora

[D1-3105]

Describe the bug

Error(s) in loading state_dict for UNet2DConditionModel. I have uploaded the model on hugging face. Error appears on load_lora_weights() function.

Reproduction

from diffusers import DiffusionPipeline

pipe = DiffusionPipeline.from_pretrained("stabilityai/stable-diffusion-xl-base-1.0")
pipe.load_lora_weights("D1-3105/lora_bug-1")

prompt = "abstract portrait of 1girl,undefined gender,fragmented visual style,red and black color palette,evokes feelings of rebellion,passion,and freedom,blurred boundaries,high resolution,aesthetic,"
image = pipe(prompt).images[0]

Logs

Loading default_0 was unsucessful with the following error: 
Error(s) in loading state_dict for UNet2DConditionModel:
        size mismatch for down_blocks.1.attentions.0.proj_in.lora_A.default_0.weight: copying a param with shape torch.Size([32, 640, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 640]).
        size mismatch for down_blocks.1.attentions.0.proj_in.lora_B.default_0.weight: copying a param with shape torch.Size([640, 32, 1, 1]) from checkpoint, the shape in current model is torch.Size([640, 32]).
        size mismatch for down_blocks.1.attentions.0.transformer_blocks.0.attn2.to_k.lora_A.default_0.weight: copying a param with shape torch.Size([32, 768]) from checkpoint, the shape in current model is torch.Size([32, 2048]).
        size mismatch for down_blocks.1.attentions.0.transformer_blocks.0.attn2.to_v.lora_A.default_0.weight: copying a param with shape torch.Size([32, 768]) from checkpoint, the shape in current model is torch.Size([32, 2048]).
        size mismatch for down_blocks.1.attentions.0.proj_out.lora_A.default_0.weight: copying a param with shape torch.Size([32, 640, 1, 1]) from checkpoint, the shape in current model is torch.Size([32, 640]).
........

System Info

System Info

Diffusers version: Version: 0.33.0.dev0
Python: 3.12.9

Who can help?

@sayakpaul

[sayakpaul]

Hello,

Which trainer was used for this? Also, are we sure the base model is SDXL? The reason for my suspicion is errors like this:

size mismatch for up_blocks.1.attentions.1.transformer_blocks.0.attn2.to_v.lora_A.default_0.weight: copying a param with shape torch.Size([32, 768]) from checkpoint, the shape in current model is torch.Size([32, 2048]).
size mismatch for up_blocks.1.attentions.1.transformer_blocks.0.attn2.to_v.lora_B.default_0.weight: copying a param with shape torch.Size([1280, 32]) from checkpoint, the shape in current model is torch.Size([640, 32]).

It indicates different shapes in the base model than what SDXL UNet has