September 7, 2023 6 min read

Prompt Engineering Patterns for AI Image Generation

AI Image Generation Prompt Engineering DALL-E Azure

Introduction

Effective prompt engineering is the key to getting consistent, high-quality results from AI image generation models like DALL-E 2. This post covers proven patterns and techniques for crafting prompts that produce the images you want.

Understanding How Image Models Interpret Prompts

AI image models process prompts by:

Breaking down text into concepts and relationships
Mapping concepts to learned visual representations
Generating images that combine these elements

Understanding this helps write better prompts.

The Anatomy of an Effective Prompt

Basic Structure

[Subject] + [Style] + [Details] + [Mood/Atmosphere] + [Technical Specifications]

Example:

A golden retriever (subject)
in impressionist painting style (style)
playing in a sunny meadow with wildflowers (details)
joyful and peaceful atmosphere (mood)
soft natural lighting (technical)

Prompt Engineering Patterns

Pattern 1: Layered Description

Build your prompt in layers from general to specific:

from dataclasses import dataclass
from typing import Optional, List

@dataclass
class LayeredPrompt:
    # Layer 1: Core subject
    subject: str

    # Layer 2: Action or state
    action: Optional[str] = None

    # Layer 3: Environment
    environment: Optional[str] = None

    # Layer 4: Style
    style: Optional[str] = None

    # Layer 5: Mood
    mood: Optional[str] = None

    # Layer 6: Technical details
    technical: Optional[List[str]] = None

    def build(self) -> str:
        layers = [self.subject]

        if self.action:
            layers.append(self.action)
        if self.environment:
            layers.append(f"in {self.environment}")
        if self.style:
            layers.append(f"{self.style} style")
        if self.mood:
            layers.append(f"{self.mood} atmosphere")
        if self.technical:
            layers.extend(self.technical)

        return ", ".join(layers)

# Usage
prompt = LayeredPrompt(
    subject="A wise old wizard",
    action="reading an ancient spellbook",
    environment="a candlelit tower library",
    style="fantasy illustration",
    mood="mysterious and scholarly",
    technical=["detailed", "dramatic lighting", "8k resolution"]
)

print(prompt.build())

Pattern 2: Style Anchoring

Anchor your prompt to known styles or artists:

STYLE_ANCHORS = {
    "photorealistic": [
        "photorealistic",
        "hyperrealistic",
        "professional photography",
        "DSLR quality",
        "8k resolution"
    ],
    "illustration": [
        "digital illustration",
        "detailed artwork",
        "artstation trending",
        "concept art"
    ],
    "painting": [
        "oil painting",
        "classical painting style",
        "museum quality",
        "rich brushstrokes"
    ],
    "anime": [
        "anime style",
        "studio ghibli inspired",
        "detailed anime art",
        "vibrant colors"
    ],
    "minimalist": [
        "minimalist design",
        "clean lines",
        "simple shapes",
        "limited color palette"
    ]
}

def apply_style_anchor(subject: str, style_key: str) -> str:
    """Apply style anchoring to a subject."""
    if style_key not in STYLE_ANCHORS:
        return subject

    style_terms = ", ".join(STYLE_ANCHORS[style_key])
    return f"{subject}, {style_terms}"

# Usage
prompt = apply_style_anchor(
    "A dragon flying over mountains",
    "illustration"
)
print(prompt)
# Output: A dragon flying over mountains, digital illustration, detailed artwork,
#         artstation trending, concept art

Pattern 3: Composition Control

Control the composition and framing:

COMPOSITION_TERMS = {
    "portrait": "portrait shot, centered subject, shallow depth of field",
    "landscape": "wide landscape shot, rule of thirds, scenic view",
    "close_up": "extreme close-up, macro detail, sharp focus",
    "wide_angle": "wide angle shot, expansive view, dramatic perspective",
    "birds_eye": "bird's eye view, top-down perspective, aerial shot",
    "low_angle": "low angle shot, looking up, dramatic perspective",
    "symmetrical": "symmetrical composition, centered, balanced",
    "dynamic": "dynamic composition, action shot, motion blur"
}

def apply_composition(subject: str, composition_key: str) -> str:
    """Apply composition terms to prompt."""
    comp_terms = COMPOSITION_TERMS.get(composition_key, "")
    return f"{subject}, {comp_terms}" if comp_terms else subject

# Usage
prompt = apply_composition(
    "A samurai warrior in traditional armor",
    "low_angle"
)

Pattern 4: Lighting Specification

Lighting dramatically affects the final image:

LIGHTING_PRESETS = {
    "golden_hour": "golden hour lighting, warm tones, long shadows, sunset glow",
    "studio": "professional studio lighting, soft shadows, even illumination",
    "dramatic": "dramatic lighting, high contrast, deep shadows, spotlight effect",
    "natural": "natural daylight, soft diffused light, realistic lighting",
    "neon": "neon lighting, cyberpunk atmosphere, colorful glow, night scene",
    "moonlight": "moonlit scene, cool blue tones, night atmosphere, soft glow",
    "backlit": "backlit silhouette, rim lighting, glowing edges",
    "overcast": "overcast sky, diffused light, soft even shadows"
}

class LightingComposer:
    def __init__(self, base_prompt: str):
        self.base_prompt = base_prompt
        self.lighting = None

    def with_lighting(self, lighting_key: str) -> 'LightingComposer':
        self.lighting = LIGHTING_PRESETS.get(lighting_key, lighting_key)
        return self

    def build(self) -> str:
        if self.lighting:
            return f"{self.base_prompt}, {self.lighting}"
        return self.base_prompt

# Usage
prompt = (LightingComposer("A mysterious castle on a cliff")
    .with_lighting("moonlight")
    .build())

Pattern 5: Negative Prompting (Conceptual)

While DALL-E 2 doesn’t support negative prompts directly, you can guide away from unwanted elements:

def positive_redirect(subject: str, avoid_concepts: List[str]) -> str:
    """Redirect away from unwanted concepts using positive language."""

    REDIRECTS = {
        "blurry": "sharp focus, high detail",
        "dark": "well-lit, bright",
        "crowded": "clean composition, single subject",
        "cartoon": "realistic, detailed",
        "amateur": "professional quality, expert craftsmanship",
        "noisy": "clean image, smooth gradients"
    }

    additions = []
    for concept in avoid_concepts:
        if concept in REDIRECTS:
            additions.append(REDIRECTS[concept])

    if additions:
        return f"{subject}, {', '.join(additions)}"
    return subject

# Usage
prompt = positive_redirect(
    "A portrait of a young woman",
    ["blurry", "amateur", "dark"]
)
# Output: A portrait of a young woman, sharp focus, high detail,
#         professional quality, expert craftsmanship, well-lit, bright

Building a Prompt Library

Reusable Components

class PromptLibrary:
    def __init__(self):
        self.subjects = {}
        self.styles = {}
        self.modifiers = {}

    def add_subject(self, key: str, description: str):
        self.subjects[key] = description

    def add_style(self, key: str, style_terms: List[str]):
        self.styles[key] = style_terms

    def add_modifier(self, key: str, modifier: str):
        self.modifiers[key] = modifier

    def compose(
        self,
        subject_key: str,
        style_key: str = None,
        modifier_keys: List[str] = None
    ) -> str:
        """Compose a prompt from library components."""
        parts = []

        # Add subject
        if subject_key in self.subjects:
            parts.append(self.subjects[subject_key])
        else:
            parts.append(subject_key)

        # Add style
        if style_key and style_key in self.styles:
            parts.extend(self.styles[style_key])

        # Add modifiers
        if modifier_keys:
            for mod_key in modifier_keys:
                if mod_key in self.modifiers:
                    parts.append(self.modifiers[mod_key])

        return ", ".join(parts)

# Build library
library = PromptLibrary()

# Add subjects
library.add_subject("forest_path", "A winding path through an ancient forest")
library.add_subject("city_street", "A busy city street with tall buildings")
library.add_subject("space_station", "A futuristic space station orbiting Earth")

# Add styles
library.add_style("realistic", ["photorealistic", "detailed", "8k resolution"])
library.add_style("painterly", ["oil painting style", "artistic", "museum quality"])
library.add_style("scifi", ["sci-fi concept art", "futuristic", "detailed design"])

# Add modifiers
library.add_modifier("morning", "early morning light, misty atmosphere")
library.add_modifier("night", "night scene, dramatic lighting")
library.add_modifier("rain", "rainy weather, wet surfaces, reflections")

# Use library
prompt = library.compose(
    "forest_path",
    "painterly",
    ["morning", "rain"]
)
print(prompt)

Testing and Iteration

A/B Testing Prompts

from typing import Dict
import random

class PromptTester:
    def __init__(self, image_generator):
        self.generator = image_generator
        self.results = []

    def test_variations(
        self,
        base_subject: str,
        variations: Dict[str, str]
    ) -> List[dict]:
        """Test multiple prompt variations."""
        results = []

        for name, modifier in variations.items():
            prompt = f"{base_subject}, {modifier}"

            # Generate image
            urls = self.generator(prompt)

            results.append({
                "name": name,
                "prompt": prompt,
                "urls": urls
            })

        self.results.extend(results)
        return results

    def compare_styles(self, subject: str, styles: List[str]) -> List[dict]:
        """Compare different style applications."""
        variations = {style: style for style in styles}
        return self.test_variations(subject, variations)

# Usage example structure
# tester = PromptTester(generate_image)
# results = tester.compare_styles(
#     "A medieval knight",
#     ["photorealistic", "oil painting", "digital art", "anime style"]
# )

Best Practices Summary

Be Specific: Vague prompts produce vague results
Layer Your Description: Build from subject to style to details
Use Style Anchors: Reference known styles for consistency
Control Composition: Specify framing and perspective
Specify Lighting: Light dramatically affects mood
Iterate and Test: Compare variations to find what works
Build a Library: Reuse successful prompt components

Conclusion

Effective prompt engineering transforms AI image generation from hit-or-miss to reliable and consistent. By understanding how models interpret prompts and using structured approaches, you can achieve better results with fewer iterations.