The Complete Stable Diffusion Guide: Installation, Models, ControlNet & Best Practices

What is Stable Diffusion?

Stable Diffusion is a deep learning model built on a technique called latent diffusion. Rather than working directly with full-resolution pixel data, it compresses images into a smaller latent space, applies the diffusion process there, and then decodes the result back to pixels. This architectural decision is what makes it possible to run on consumer-grade GPUs rather than requiring massive data center hardware.

The model was originally developed by CompVis at Ludwig Maximilian University of Munich, in collaboration with Stability AI and Runway. Since its public release in August 2022, Stable Diffusion has grown into the largest open-source AI art ecosystem in the world, with thousands of community-created models, extensions, and tools built around it.

What sets Stable Diffusion apart from competitors is its open-source nature. You can inspect the code, fine-tune the model on your own data, create custom LoRAs for specific styles or characters, and integrate it into production pipelines. The community has built an extraordinary ecosystem of fine-tuned checkpoints, each optimized for different aesthetic styles ranging from photorealism to anime to concept art.

How Latent Diffusion Works

The generation process starts with pure random noise in the latent space. A neural network called the U-Net is trained to predict and remove noise from this latent representation, guided by your text prompt which has been encoded by a CLIP text encoder. Through a series of iterative denoising steps (typically 20-50), the model gradually transforms noise into a coherent image that matches your description. A VAE (Variational Autoencoder) then decodes the final latent representation into a full-resolution image.

Understanding this pipeline matters because each component can be swapped, tuned, or extended. Different VAEs produce different color profiles, different text encoders handle prompts differently, and the sampling method you choose affects both quality and generation speed.

Installation: AUTOMATIC1111 & ComfyUI

Two interfaces dominate the Stable Diffusion ecosystem. Each serves a different kind of user, and many experienced artists use both depending on the task at hand.

AUTOMATIC1111 Web UI

AUTOMATIC1111 (commonly called A1111) is the most popular Stable Diffusion interface. It provides a traditional web-based UI with familiar form controls: text fields for prompts, sliders for settings, and dropdown menus for model selection. If you have used any web application before, you will feel at home with A1111.

A1111 Installation Steps

1. Install Python 3.10.x from python.org. Avoid Python 3.12+ as some dependencies may have compatibility issues.
2. Install Git from git-scm.com if not already present on your system.
3. Clone the repository: Open a terminal and run git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
4. Download a model checkpoint (such as SD 1.5 or SDXL) and place the .safetensors file in the models/Stable-diffusion/ folder.
5. Launch the UI by running webui-user.bat (Windows) or webui.sh (macOS/Linux). The first launch will download remaining dependencies automatically.
6. Open your browser to http://127.0.0.1:7860 and begin generating.

A1111 shines for its massive extension ecosystem. The built-in Extensions tab lets you install ControlNet, ADetailer (automatic face fixing), regional prompting, and hundreds of other tools with a single click. For users who want a straightforward path to generating images with advanced features, A1111 is the recommended starting point.

ComfyUI

ComfyUI takes a fundamentally different approach. Instead of a traditional interface, it presents a node-based workflow canvas where you visually connect processing blocks. Each node represents a step in the pipeline: loading a model, encoding a prompt, running the sampler, decoding the latent, and saving the image. You wire these together by dragging connections between nodes.

ComfyUI Installation Steps

1. Download ComfyUI from the GitHub releases page. Windows users can download the portable package which includes Python and all dependencies bundled together.
2. Extract the archive to your preferred location. No separate Python installation is required for the portable version.
3. Place model checkpoints in the models/checkpoints/ folder, LoRAs in models/loras/, and VAEs in models/vae/.
4. Run run_nvidia_gpu.bat (Windows) or python main.py (macOS/Linux).
5. Open your browser to http://127.0.0.1:8188 to access the node editor.

ComfyUI's node-based approach is more complex to learn but offers significant advantages. You can build reusable workflows, share them as JSON files, and achieve precise control over every step of the pipeline. ComfyUI also uses less VRAM than A1111 for equivalent operations, making it the better choice for GPUs with limited memory. Many professional and production workflows now run on ComfyUI.

If you are brand new to Stable Diffusion, start with AUTOMATIC1111. Once you understand the fundamentals of prompting, models, and samplers, consider transitioning to ComfyUI for more advanced work.

Models: SD 1.5, SDXL & SD3

Stable Diffusion is not a single model but a family of model architectures, each with its own strengths and hardware requirements. Understanding the differences between them helps you choose the right base for your projects.

Stable Diffusion 1.5

SD 1.5 remains the most widely used base model thanks to its low hardware requirements and the enormous library of fine-tuned checkpoints built on top of it. It generates images at 512x512 pixels natively and can be upscaled afterward. Thousands of community checkpoints on CivitAI and Hugging Face are based on SD 1.5, covering every conceivable style from hyperrealistic photography to stylized anime.

SD 1.5 requires as little as 4GB of VRAM, making it accessible to users with older or budget GPUs. The trade-off is that its native resolution is relatively low, and it can struggle with complex compositions, accurate human anatomy, and detailed text rendering within images.

Stable Diffusion XL (SDXL)

SDXL represents a major leap in quality. It generates images at 1024x1024 pixels natively, produces dramatically better composition and anatomy, and handles complex prompts with greater fidelity. SDXL uses a dual text encoder system (CLIP ViT-L and OpenCLIP ViT-bigG), which gives it a deeper understanding of prompt semantics.

The cost is higher hardware requirements: you need at least 8GB of VRAM for comfortable generation, with 12GB recommended. SDXL also introduced a refiner model that can be run as a second pass to enhance fine details, though many fine-tuned SDXL checkpoints produce excellent results without it.

Stable Diffusion 3 (SD3)

SD3 is the latest architecture from Stability AI, built on a Multimodal Diffusion Transformer (MMDiT) rather than the U-Net used in previous versions. This transformer-based architecture brings superior text rendering capabilities, better prompt adherence, and improved handling of spatial relationships between objects.

SD3 comes in multiple sizes. SD3 Medium is suitable for consumer hardware with 12GB+ VRAM, while larger variants require professional GPU setups. The community fine-tune ecosystem for SD3 is growing but is still smaller than what exists for SD 1.5 and SDXL.

LoRA & Embeddings

One of Stable Diffusion's most powerful features is the ability to extend base models with lightweight add-ons that teach the model new concepts without replacing or retraining the entire checkpoint.

LoRAs (Low-Rank Adaptations)

LoRAs are small files (typically 10-200 MB) that modify a subset of the model's weights to introduce a new style, character, or concept. They are the primary method for customizing Stable Diffusion output. You can find thousands of community-created LoRAs on CivitAI, covering everything from specific artistic styles to individual characters to lighting techniques.

Using a LoRA is straightforward. In AUTOMATIC1111, you include the trigger keyword in your prompt and reference the LoRA file using the syntax <lora:filename:weight>, where weight controls how strongly the LoRA influences the output (typically 0.5 to 1.0). In ComfyUI, you add a LoRA Loader node between your checkpoint loader and the rest of the pipeline.

You can stack multiple LoRAs in a single generation. For example, you might combine a style LoRA for watercolor painting with a character LoRA for a specific subject, and a detail LoRA for enhanced textures. The key is balancing the weights so no single LoRA dominates the output.

Textual Inversions (Embeddings)

Textual inversions, also called embeddings, are even smaller than LoRAs (usually under 100 KB). Instead of modifying model weights, they teach the text encoder new "words" that represent specific concepts. They are most commonly used for negative prompts, where community-created embeddings like EasyNegative or BadDream encode a broad set of quality-reducing artifacts into a single trigger word, simplifying your negative prompt significantly.

ControlNet

ControlNet is arguably the most important extension in the Stable Diffusion ecosystem. It solves one of the fundamental challenges of AI image generation: precise spatial control. While text prompts can describe what you want in an image, they are inherently imprecise about where things should be positioned and how they should be posed.

ControlNet works by adding a parallel neural network that conditions the generation process on a control signal. This signal can be derived from an existing image or created from scratch. The most commonly used control types include:

• Canny Edge Detection — Extracts edges from a reference image and uses them to guide the composition. Excellent for maintaining the overall structure of a scene while completely changing the style.
• OpenPose — Detects human body pose keypoints (skeleton) and ensures generated characters match the same pose. Essential for character art and action scenes.
• Depth Map — Uses depth information to maintain the spatial layering of a scene. Foreground and background elements stay in their proper positions.
• Scribble/Sketch — Interprets rough hand-drawn sketches as composition guides. You can draw a basic layout and have SD fill in the details.
• IP-Adapter — Uses a reference image to transfer its style or subject to a new generation. Useful for maintaining consistent character appearances across multiple images.
• Tile — Guides upscaling by preserving fine details from the original image while enhancing resolution. Critical for producing print-quality output.

ControlNet models need to match your base model architecture. SD 1.5 ControlNet models will not work with SDXL checkpoints and vice versa. Always download the correct version for the base model you are using.

Image-to-Image (img2img)

Image-to-image generation starts with an existing image rather than pure noise. Instead of beginning the denoising process from random latent noise, the model encodes your input image into latent space and adds a controlled amount of noise, determined by the denoising strength parameter. It then runs the normal denoising process, guided by your text prompt.

The denoising strength controls how much the output deviates from the input. At 0.3, the output closely resembles the original with subtle style changes. At 0.7, the composition is loosely preserved but details change significantly. At 1.0, the input image is essentially ignored and you get a fresh generation.

Common img2img Use Cases

• Style transfer — Transform a photograph into a painting, illustration, or any other artistic style while preserving the composition.
• Iterative refinement — Take a generation you like and run it through img2img with a low denoising strength (0.2-0.4) to fix minor imperfections while preserving the overall look.
• Sketch-to-render — Turn a rough sketch or color block layout into a fully rendered scene.
• Upscaling with enhancement — Use img2img at low denoising strength on an upscaled image to add convincing high-resolution detail.

Inpainting

Inpainting allows you to selectively regenerate portions of an image while keeping the rest untouched. You paint a mask over the area you want to change, write a prompt describing what should appear there, and the model fills in the masked region while seamlessly blending with the surrounding pixels.

Inpainting is indispensable for fixing common AI artifacts. Deformed hands, awkward facial features, unwanted objects, or inconsistent background elements can all be corrected with targeted inpainting passes. It transforms Stable Diffusion from a one-shot generator into an iterative editing tool.

Inpainting Tips for Best Results

• Use the dedicated inpainting model if available for your base checkpoint. These models are specifically fine-tuned to blend masked regions naturally.
• Mask slightly larger than needed — extend the mask 10-20 pixels beyond the area you want to change so the model has context to blend edges smoothly.
• Set "Inpaint at full resolution" when fixing small areas like faces or hands. This crops the masked area, processes it at higher resolution, and pastes it back.
• Match the prompt to context — describe not just the new content but also the surrounding environment, lighting, and style so the inpainted region blends seamlessly.
• Use multiple passes — complex fixes often require 2-3 rounds of inpainting with progressively smaller masks and lower denoising strength.

Prompt Engineering Best Practices

Writing effective prompts for Stable Diffusion differs from prompting cloud-based models like Midjourney. Because SD checkpoints vary dramatically in how they interpret text, developing strong prompting habits is essential for consistent results.

Prompt Structure

A well-structured Stable Diffusion prompt typically follows this pattern: subject, action/pose, environment, lighting, camera/lens, style, quality modifiers. Placing the most important elements first gives them greater weight in the generation.

Quality Modifiers and Trigger Words

Many fine-tuned checkpoints respond to specific quality trigger words. Common ones include masterpiece, best quality, highly detailed, 8k, sharp focus, and professional. However, these are not universal. Always check the model card on CivitAI or Hugging Face for the checkpoint-specific trigger words and recommended settings.

Negative Prompts

Negative prompts are just as important as positive prompts in Stable Diffusion. They tell the model what to avoid. A good baseline negative prompt addresses common artifacts: worst quality, low quality, normal quality, lowres, blurry, deformed, extra limbs, bad anatomy, bad hands, watermark, text, signature, cropped. For even better results, use quality-focused embeddings like EasyNegative in your negative prompt.

Prompt Weighting

Both A1111 and ComfyUI support prompt weighting to emphasize or de-emphasize specific terms. In A1111, wrapping a term in parentheses increases its weight: (detailed eyes:1.3) gives that concept 30% more influence. You can also decrease weight: (background:0.7). This fine-grained control helps you steer the model toward your intended result without restructuring the entire prompt.

Strengths & Weaknesses

Understanding where Stable Diffusion excels and where it falls short helps you decide when to use it versus alternative tools, and how to work around its limitations.

Video Tutorials

These curated video tutorials walk you through the setup process and core workflows visually. Watching alongside this guide will accelerate your learning significantly.

Stable Diffusion Beginner Setup Guide

Advanced Stable Diffusion Workflows