Composable QLoRA: Semantic Low-Rank Decomposition for Multi-Character Image Generation on Consumer Hardware via Modular Adaptation of Functionally-Specialized UNet Layers

We present three contributions for composable personalization of diffusion models on consumer hardware: (1) Semantic LoRA Decomposition. We discover and exploit the functional specialization of UNet transformer block layer types: cross-attention layers encode text-conditioned identity (WHO), self-attention layers encode spatial structure and composition (HOW/WHAT), and feedforward layers encode per-position style and environment (WHERE). By constraining each LoRA adapter to exactly one layer type, we guarantee zero mutual interference- adapters targeting mutually exclusive module sets can be freely composed, switched, and weighted at inference without retraining or quality degradation. (2) Int4 QLoRA for Diffusion Models on Apple Silicon. We demonstrate Int4 quantized training of SDXL's 2.6B-parameter UNet on an 8 GB Apple M1 device, achieving 3.8x compression (4,260 MB to 1,133 MB) with 0.9978 cosine similarity to FP16. We identify and resolve a critical numerical instability: gradient checkpointing with adaptive optimizers produces NaN when LoRA parameters are stored in reduced precision, due to zero-gradient moment estimate collapse. Storing adapters in float32 (+44 MB) eliminates this failure. (3) Adaptive Composition Weights. We propose treating per-layer composition weights as temporally-adaptive parameters that increase proportionally with reuse frequency, enabling the system to automatically reinforce aesthetic patterns that recur across generated outputs- a form of implicit preference learning without explicit retraining. We validate all three contributions with measured experiments on Apple M1 (8 GB): three composable layers train independently in 3.0 hours total at less than 2 GB peak memory, producing a complete multi-character generation system on minimum-spec consumer hardware. Our approach enables instant character switching (scalar weight change, no reload), linear scaling (N characters = N adapters, not N-squared combinations), and per-character training without joint optimization.