Architecture - PixelFlux

Architecture & Implementation

This document explains the internal architecture of PixelFlux, how Metal GPU compute shaders work, and the design decisions behind the framework.

Overview

PixelFlux is built on Apple's Metal framework, which provides low-level access to the GPU. The framework is organized into several layers:

User-Facing API: Simple Swift interfaces (BrightnessFilter, PassthroughFilter)
Core Engines: Texture handling, GPU pipeline management, error handling
GPU Layer: Metal compute shaders that perform actual image processing

Processing Pipeline

When you apply a filter to an image, PixelFlux performs these steps:

Image to Texture: Convert UIImage to Metal GPU texture
Parameter Setup: Create GPU buffers for filter parameters (e.g., brightness value)
GPU Execution: Dispatch compute shader to process texture in parallel
Result Collection: Copy processed data from GPU back to CPU memory
Texture to Image: Convert Metal texture back to UIImage

Key Components

1. Texture Management (TextureUtilities.swift)

Handles conversion between UIImage and Metal textures.

Image → Texture Conversion

// UIImage is loaded via Core Image
let ciImage = CIImage(cgImage: cgImage)

// MTKTextureLoader efficiently uploads to GPU
let texture = try textureLoader.newTexture(cgImage: cgImage, options: [...])

Why Metal textures? Metal textures reside in GPU memory and can be directly processed by compute shaders without CPU-GPU transfer overhead.

2. GPU Context (GPUContext.swift)

Singleton that manages the Metal device and command queue.

GPU Context Setup

static let shared = try GPUContext()

// Creates:
let device = MTLCreateSystemDefaultDevice()  // The GPU
let queue = device.makeCommandQueue()         // Command submission queue

All GPU operations go through a single command queue to ensure proper synchronization and command ordering.

3. Compute Pipeline (ComputePipeline.swift)

Encapsulates Metal shader compilation and command encoding.

Pipeline Initialization

public init(functionName: String) throws {
  // 1. Load the Metal shader library
  let library = try loadMetalLibrary()

  // 2. Get the compute function by name
  guard let function = library.makeFunction(name: functionName) else {
    throw PixelFluxError.functionNotFound
  }

  // 3. Create a compute pipeline state (compiled GPU code)
  self.pipelineState = try device.makeComputePipelineState(function: function)
}

4. Filter Classes

High-level interfaces that implement the Filter protocol.

BrightnessFilter

public final class BrightnessFilter: Filter {
  private let pipeline: ComputePipeline
  public var brightness: Float

  public init(brightness: Float) throws {
    self.brightness = brightness
    self.pipeline = try ComputePipeline(functionName: "brightnessKernel")
  }

  public func process(_ input: MTLTexture) throws -> MTLTexture {
    let output = makeEmptyTexture(matching: input)

    // 1. Create command buffer
    guard let cmd = GPUContext.shared.queue.makeCommandBuffer() else {
      throw PixelFluxError.commandBufferFailed
    }

    // 2. Allocate GPU buffer for brightness parameter
    let params = BrightnessParams(brightness: brightness)
    guard let paramBuffer = GPUContext.shared.device.makeBuffer(
      bytes: ¶ms,
      length: MemoryLayout<BrightnessParams>.size
    ) else {
      throw PixelFluxError.bufferCreationFailed
    }

    // 3. Encode compute commands
    pipeline.encode(commandBuffer: cmd, input: input, output: output, params: paramBuffer)

    // 4. Submit and wait
    cmd.commit()
    cmd.waitUntilCompleted()

    return output
  }
}

Metal Compute Shaders

At the heart of PixelFlux are Metal compute shaders—GPU programs that run thousands of threads in parallel.

Shader Language

PixelFlux shaders are written in Metal Shading Language (MSL), which is similar to C++. They live in .metal files in the framework bundle.

Brightness Shader Example

brightnessKernel in Metal

#include <metal_stdlib>
using namespace metal;

struct BrightnessParams {
  float brightness;
};

kernel void brightnessKernel(
  texture2d<float, access::read>  input    [[texture(0)]],
  texture2d<float, access::write> output   [[texture(1)]],
  constant BrightnessParams &params [[buffer(0)]],
  uint2 gid [[thread_position_in_grid]]
) {
  // Each GPU thread processes one pixel
  float4 pixel = input.read(gid);

  // Apply brightness adjustment
  float4 brightened = float4(
    pixel.rgb + params.brightness,
    pixel.a
  );

  // Clamp to valid color range
  output.write(clamp(brightened, 0.0, 1.0), gid);
}

How it works:

Each GPU thread processes one pixel independently
gid (thread_position_in_grid) identifies which pixel this thread processes
Textures are read/written in parallel across all threads
GPU schedules thousands of threads across cores, achieving massive parallelism

Passthrough Shader

The simplest possible filter—tests that the entire GPU pipeline works correctly:

passthroughKernel in Metal

kernel void passthroughKernel(
  texture2d<float, access::read>  input  [[texture(0)]],
  texture2d<float, access::write> output [[texture(1)]],
  uint2 gid [[thread_position_in_grid]]
) {
  // Just copy the pixel unchanged
  float4 pixel = input.read(gid);
  output.write(pixel, gid);
}

Memory Model

Texture Layout

Metal textures are 2D arrays of pixels in GPU memory. Each pixel is typically 4 bytes (RGBA).

Memory Layout

// Physical GPU memory
[R][G][B][A] [R][G][B][A] [R][G][B][A] ...  ← Row 0
[R][G][B][A] [R][G][B][A] [R][G][B][A] ...  ← Row 1
[R][G][B][A] [R][G][B][A] [R][G][B][A] ...  ← Row 2
...

// Thread processes (x, y) coordinate
uint2 gid = uint2(3, 1)  // Pixel at column 3, row 1
float4 pixel = input.read(gid)

Synchronization

PixelFlux ensures proper GPU-CPU synchronization:

Command Submission: cmd.commit() queues work to the GPU
Synchronization: cmd.waitUntilCompleted() blocks until all GPU work is done
Result Copy: After completion, output texture is safely readable on CPU

Performance Characteristics

Throughput

Modern iOS GPUs can process millions of pixels per second:

A17 Pro GPU: ~1.5 TFLOPS (trillion floating-point operations per second)
Processing a 4000×3000 pixel image: ~50ms per filter on A17 Pro
Older devices may take 100-200ms for the same operation

Bottlenecks

Texture Upload: Moving image data from CPU to GPU (happens once per filter)
Compute Time: Actual GPU processing (depends on image size and filter complexity)
Texture Download: Moving result back to CPU (unavoidable for UIImage output)

Optimization Strategies

Batch Processing: Apply multiple filters to same image, avoiding repeated uploads
Async Operations: Dispatch filters to background threads to keep UI responsive
Downsampling: Process smaller images and upscale for preview, full-res for output
Reuse Filters: Create filter once, apply multiple times

Error Handling Strategy

PixelFlux defines errors at key failure points:

Stage	Possible Error	Recovery
Input Validation	`noCGImage`	Validate image before filtering
Shader Load	`metalLibraryNotFound`	Check bundle linking and compilation
Function Lookup	`functionNotFound`	Verify shader function name
GPU Init	`commandBufferFailed`	GPU context unavailable (rare on devices)
Memory Allocation	`bufferCreationFailed`	Out of VRAM or invalid size

Design Patterns

1. Protocol-Oriented Design

All filters implement the Filter protocol, enabling composition and abstraction:

public protocol Filter {
  func process(_ input: MTLTexture) throws -> MTLTexture
}

// Any conforming type can be used generically
func applyFilter<F: Filter>(_ filter: F, to image: UIImage) throws -> UIImage? {
  let texture = try makeTexture(from: image)
  let output = try filter.process(texture)
  return textureToUIImage(output)
}

2. Singleton GPU Context

Single GPU context shared across all filters ensures consistent device state and command ordering:

static let shared = try GPUContext()

// All filters use the same GPU context
guard let cmd = GPUContext.shared.queue.makeCommandBuffer() else {
  throw PixelFluxError.commandBufferFailed
}

3. Separation of Concerns

Filter Classes: Define what to do (brightness, passthrough, etc.)
ComputePipeline: Handles how to run GPU code
TextureUtilities: Handle image ↔ texture conversions
GPUContext: Manages GPU resources

Extending PixelFlux

Creating a Custom Filter

To add a new filter (e.g., saturation adjustment):

Step 1: Create the Swift class

import Metal

public final class SaturationFilter: Filter {
  private let pipeline: ComputePipeline
  public var saturation: Float

  public init(saturation: Float) throws {
    self.saturation = saturation
    self.pipeline = try ComputePipeline(functionName: "saturationKernel")
  }

  public func process(_ input: MTLTexture) throws -> MTLTexture {
    // Similar structure to BrightnessFilter
    let output = makeEmptyTexture(matching: input)
    // ... encode GPU commands ...
    return output
  }
}

Step 2: Add the Metal shader

kernel void saturationKernel(
  texture2d<float, access::read>  input  [[texture(0)]],
  texture2d<float, access::write> output [[texture(1)]],
  constant SaturationParams &params [[buffer(0)]],
  uint2 gid [[thread_position_in_grid]]
) {
  float4 pixel = input.read(gid);

  // Convert RGB to HSL
  // Adjust saturation
  // Convert back to RGB

  output.write(result, gid);
}

Testing Strategy

PixelFlux includes comprehensive tests for:

Filter initialization with various parameters
Image dimension preservation
Error handling (invalid inputs, missing resources)
GPU availability across devices/simulators

Tests use the UIGraphicsImageRenderer with explicit scale settings to create consistent test images across different display densities.

Deployment Considerations

Device Compatibility

Minimum: iOS 16.0 (when Metal compute became widely supported)
Simulator: Metal software rendering works on M1/M2/M3 Macs with appropriate Metal Toolchain
Older Devices: A9 and later all support Metal

Distribution

PixelFlux is distributed as:

Swift Package (SPM): Source code + Metal shaders in bundle
XCFramework: Pre-compiled binary for both device and simulator

Debugging Tips

Enable Metal Debugging

In Xcode, enable Metal validation to catch GPU errors:

Scheme → Run → Diagnostics → Metal API Validation
This slows execution but provides detailed error messages

Print GPU Information

let device = MTLCreateSystemDefaultDevice()
print("GPU: \(device?.name ?? "Unknown")")
print("Max threads: \(device?.maxThreadsPerThreadgroup ?? 0)")

Verify Shader Compilation

If shaders fail to compile, check:

Metal Toolchain is installed (Xcode 26.2 requires explicit download)
Shader file is in correct target's Build Phases → Copy Bundle Resources
Function names match between Swift and Metal files