Chapter 3: Multimodal Generation

3.1 Text-to-Image Generation

Text-to-image models generate images from natural language descriptions. This capability has transformed creative workflows and democratized visual content creation.

Leading Text-to-Image Models (2025)

Diffusion Models: The Foundation

Modern text-to-image models use diffusion, a process that learns to reverse the gradual addition of noise to images:

Latent Diffusion (Stable Diffusion Architecture)

Generating Images with Stable Diffusion

# Text-to-Image with Stable Diffusion
from diffusers import StableDiffusionPipeline
import torch

# Load model
pipe = StableDiffusionPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    variant="fp16"
)
pipe = pipe.to("cuda")

# Generate image
prompt = "A serene Japanese garden with cherry blossoms, koi pond, \
          traditional wooden bridge, morning mist, photorealistic, 8k"

negative_prompt = "blurry, low quality, distorted, ugly"

image = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    num_inference_steps=50,
    guidance_scale=7.5,
    height=1024,
    width=1024
).images[0]

image.save("japanese_garden.png")

Using DALL-E 3 API

# DALL-E 3 via OpenAI API
import openai
from PIL import Image
import requests
from io import BytesIO

client = openai.OpenAI(api_key="your-api-key")

response = client.images.generate(
    model="dall-e-3",
    prompt="A futuristic Tokyo cityscape at night with flying cars, \
            neon signs in Japanese, cyberpunk aesthetic",
    size="1024x1024",
    quality="hd",
    n=1
)

# Download and display image
image_url = response.data[0].url
image_response = requests.get(image_url)
image = Image.open(BytesIO(image_response.content))
image.save("cyberpunk_tokyo.png")

# Get the revised prompt (DALL-E 3 may modify your prompt)
print(f"Revised prompt: {response.data[0].revised_prompt}")

3.2 Video Understanding Models

Video understanding extends image understanding with temporal reasoning - understanding how scenes change over time.

Challenges in Video Understanding

• Temporal reasoning: Understanding cause-effect, sequence of events
• Long-range dependencies: Events early in video affecting later interpretation
• Computational cost: Processing hundreds of frames per video
• Action recognition: Identifying what people/objects are doing

Video Tokenization Strategies

# Video Understanding with Temporal Tokens
import torch
import torch.nn as nn
from transformers import CLIPProcessor, CLIPModel

class SimpleVideoEncoder(nn.Module):
    """Encode video by processing frames and adding temporal information"""

    def __init__(self, clip_model_name="openai/clip-vit-base-patch32"):
        super().__init__()
        self.clip = CLIPModel.from_pretrained(clip_model_name)
        self.clip_processor = CLIPProcessor.from_pretrained(clip_model_name)

        # Learnable temporal position embeddings
        self.temporal_embed = nn.Embedding(100, 512)  # Up to 100 frames

        # Temporal attention
        self.temporal_attn = nn.TransformerEncoder(
            nn.TransformerEncoderLayer(d_model=512, nhead=8, batch_first=True),
            num_layers=2
        )

    def forward(self, frames):
        """
        frames: list of PIL Images or tensor (B, T, C, H, W)
        """
        B, T = len(frames), len(frames[0])

        # Extract frame features with CLIP
        frame_features = []
        for batch in frames:
            inputs = self.clip_processor(images=batch, return_tensors="pt")
            with torch.no_grad():
                features = self.clip.get_image_features(**inputs)
            frame_features.append(features)

        # Stack: (B, T, D)
        frame_features = torch.stack(frame_features)

        # Add temporal position embeddings
        positions = torch.arange(T).unsqueeze(0).expand(B, -1)
        temporal_pos = self.temporal_embed(positions)
        frame_features = frame_features + temporal_pos

        # Apply temporal attention
        video_features = self.temporal_attn(frame_features)

        # Pool to video-level representation
        video_embedding = video_features.mean(dim=1)

        return video_embedding, video_features

State-of-the-Art: Molmo 2 (2025)

Molmo 2 from AI2 represents the current frontier in video understanding:

• Processes video frames with lightweight temporal tokens
• Interleaves visual tokens with timestamp information
• Outperforms Gemini 3 Pro on video tracking benchmarks
• Supports multi-turn conversation about video content

3.3 Any-to-Any Multimodal Models

"Any-to-Any" models can accept multiple input modalities and generate multiple output modalities:

GPT-4V/4o Capabilities

# GPT-4o: Multimodal Reasoning
import openai
import base64

client = openai.OpenAI()

def encode_image_to_base64(image_path):
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode('utf-8')

# Analyze multiple images together
def compare_images(image_paths, question):
    content = [{"type": "text", "text": question}]

    for path in image_paths:
        content.append({
            "type": "image_url",
            "image_url": {
                "url": f"data:image/jpeg;base64,{encode_image_to_base64(path)}"
            }
        })

    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": content}],
        max_tokens=500
    )

    return response.choices[0].message.content

# Example: Compare before/after images
result = compare_images(
    ["room_before.jpg", "room_after.jpg"],
    "What changes were made to this room? List specific differences."
)
print(result)

DeepSeek Janus-Pro: Open-Source Unified Model

Janus-Pro (January 2025) is a breakthrough open-source model with unified understanding and generation:

# Using Janus-Pro for Understanding and Generation
from transformers import AutoModelForCausalLM, AutoProcessor
import torch

# Load Janus-Pro
model = AutoModelForCausalLM.from_pretrained(
    "deepseek-ai/Janus-Pro-7B",
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True
)
processor = AutoProcessor.from_pretrained(
    "deepseek-ai/Janus-Pro-7B",
    trust_remote_code=True
)

# Image Understanding
def understand_image(image_path, question):
    from PIL import Image
    image = Image.open(image_path)

    conversation = [
        {"role": "user", "content": [
            {"type": "image", "image": image},
            {"type": "text", "text": question}
        ]}
    ]

    inputs = processor(conversation, return_tensors="pt").to(model.device)
    outputs = model.generate(**inputs, max_new_tokens=256)
    return processor.decode(outputs[0], skip_special_tokens=True)

# Image Generation
def generate_image(prompt):
    inputs = processor(
        text=prompt,
        return_tensors="pt",
        mode="generation"
    ).to(model.device)

    outputs = model.generate(**inputs, max_new_tokens=1024)
    image = processor.decode_image(outputs)
    return image

# Example usage
answer = understand_image("chart.png", "What trend does this chart show?")
generated = generate_image("A majestic mountain landscape at sunset")

3.4 Audio-Visual Models

Audio-visual models integrate speech, sound, and visual information:

Applications

• Audio-Visual Speech Recognition (AVSR): Using lip movements + audio for better ASR
• Video Captioning with Audio: Describing both visual and audio content
• Sound Source Localization: Identifying where sounds come from in video
• Multimodal Sentiment Analysis: Combining facial expressions, tone, and words

# Audio-Visual Processing Example
import torch
import torchaudio
from transformers import Wav2Vec2Processor, Wav2Vec2Model

class AudioVisualEncoder(nn.Module):
    """Combine audio and visual features"""

    def __init__(self, visual_dim=512, audio_dim=768, hidden_dim=512):
        super().__init__()

        # Audio encoder (Wav2Vec2)
        self.audio_processor = Wav2Vec2Processor.from_pretrained(
            "facebook/wav2vec2-base"
        )
        self.audio_encoder = Wav2Vec2Model.from_pretrained(
            "facebook/wav2vec2-base"
        )

        # Projection layers
        self.visual_proj = nn.Linear(visual_dim, hidden_dim)
        self.audio_proj = nn.Linear(audio_dim, hidden_dim)

        # Cross-modal attention
        self.cross_attn = nn.MultiheadAttention(hidden_dim, num_heads=8)

        # Fusion
        self.fusion = nn.Linear(hidden_dim * 2, hidden_dim)

    def forward(self, visual_features, audio_waveform, sample_rate=16000):
        # Process audio
        audio_inputs = self.audio_processor(
            audio_waveform,
            sampling_rate=sample_rate,
            return_tensors="pt"
        )
        audio_features = self.audio_encoder(**audio_inputs).last_hidden_state

        # Project to common space
        visual_proj = self.visual_proj(visual_features)
        audio_proj = self.audio_proj(audio_features)

        # Cross-modal attention (audio queries visual)
        attended, _ = self.cross_attn(
            audio_proj.transpose(0, 1),
            visual_proj.transpose(0, 1),
            visual_proj.transpose(0, 1)
        )
        attended = attended.transpose(0, 1)

        # Fuse modalities
        combined = torch.cat([audio_proj, attended], dim=-1)
        fused = self.fusion(combined)

        return fused

3.5 Practical Considerations

Choosing the Right Model

3.6 Summary

Exercises

Exercise 1: Prompt Engineering for Images

Generate the same concept with DALL-E and Stable Diffusion. Compare results and analyze how prompt modifications affect output quality.

Exercise 2: Video Frame Analysis

Extract keyframes from a video and use GPT-4V to describe what happens. Then implement temporal averaging to create a video summary.

Exercise 3: Model Comparison

For a visual QA task, compare responses from GPT-4V, Claude 3, and Janus-Pro. Analyze strengths and weaknesses of each.

Exercise 4: Cost Optimization

Design a system that routes image generation requests to different models based on quality requirements and budget constraints.