AI Inference at Scale

From Data Centers to Edge Devices

CUDAGenieXMLXllama.cpp

Explore Course Instructor

What You'll Master

Deploy AI models across three wildly different platforms with confidence

Data Center Inference

vLLM, continuous batching, PagedAttention, and cost optimization at scale

Edge Deployment

Model compression, on-device inference, power and memory constraints

Robotics Systems

Hard real-time constraints, WCET guarantees, control-loop deadlines

Hardware Optimization

CPU/GPU/NPU tradeoffs, roofline analysis, TOPS/W metrics

Key Frameworks

NVIDIA CUDA

GPU-accelerated computing for data center inference. CUDA optimization and memory management for high-throughput serving.

CUDA Toolkit

Qualcomm GenieX

On-device inference for Snapdragon NPUs. Deploy quantized models to mobile devices with power and memory constraints.

GenieX Hexagon NPU SDK

Apple MLX

Neural network inference on Apple Silicon. Leverage the Neural Engine for efficient on-device AI.

MLX Framework CoreML

What You'll Learn

Master the essential skills for production AI inference across platforms

Hardware & Architecture

Roofline model analysis
CPU/GPU/NPU tradeoffs
Memory & bandwidth optimization

Model Optimization

Quantization & compression
Pruning and distillation
Accuracy vs speed tradeoffs

Deployment Strategies

Data Center: batching strategies
Edge: Power constraints
Robotics: Real-time (WCET)

Cost & Performance

Tokens/sec/GPU metrics
Latency guarantees

Course Overview

Modern AI systems must run efficiently across vastly different environments: massive data centers serving millions, power-constrained edge devices, and real-time robotics systems. This course teaches you how to deploy AI models in production across all three platforms.

You'll master: Hardware optimization (CPU/GPU/NPU), model compression techniques, and serving strategies (batching, caching) through hands-on work with production systems.

Session Structure

Session 1: Inference Bottlenecks & Hardware

Understanding compute vs memory bounds, hardware architectures (CPU/GPU/NPU), and profiling inference workloads.

Session 2: Model Compression & Edge

Model compression techniques, on-device deployment, and measuring accuracy/speed tradeoffs.

Session 3: Data Center at Scale

High-throughput serving with vLLM, KV-cache optimization, and cost modeling for production systems.

Session 4: Robotics & Cross-Platform

Real-time constraints, deploying one model across all platforms, and designing deployment strategies.

Prerequisites

Basic Python programming
Understanding of ML concepts (inference vs training)
Comfort with command-line tools

No deep learning expertise required. Focus is on engineering and deployment, not research.