Reading List

Paper reading list

Presentation-group signup is available in the presentation signup sheet .

Explanations of the presenter, advocate, and critic group roles are available in the Resources tab.

Paper ID	Topic	Paper Title	Venue	Link	Paper Details
Paper 1	KV cache management	Efficient Memory Management for Large Language Model Serving with PagedAttention	SOSP 2023	Link to PDF	Open details Paper Description: Introduces virtual-memory-inspired KV-cache paging and sharing, making memory allocation and fragmentation first-class concerns in LLM serving. Related Readings: Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inference
Paper 2	Prefill/decode	Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve	OSDI 2024	Link to PDF	Open details Paper Description: Introduced chunked prefill to break compute-heavy prompt processing into smaller chunks, co-batching them with memory-bound decode iterations. Related Readings: NanoFlow: Towards Optimal Large Language Model Serving Throughput SLOs-Serve: Optimized Serving of Multi-SLO LLMs DeepSpeed-FastGen: High-Throughput Text Generation for LLMs via MII and DeepSpeed-Inference
Paper 3	Prefill-decode disaggregation	Splitwise: Efficient Generative LLM Inference Using Disaggregated Compute	ISCA 2024	Link to PDF	Open details Paper Description: Hardware-software co-design proving that prefill and decode stages thrive on different hardware architectures, for example FLOPS-rich compute nodes versus memory-bandwidth-rich nodes. Related Readings: DistServe: Disaggregating Prefill and Decoding for LLM Serving Mooncake: A KVCache-Centric Disaggregated Architecture for LLM Serving Efficiently Serving Large Multimodal Models Using EPD Disaggregation
Paper 4	Speculative decoding	Fast Inference from Large Language Models via Speculative Decoding	ICML 2023	Link to PDF	Open details Paper Description: Used a fast draft model to generate candidate tokens speculatively, followed by a single parallel forward pass on the target model to verify multiple tokens at once. Related Readings: Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Heads EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty Sequoia: Scalable, Robust, and Hardware-Aware Speculative Decoding
Paper 5	Kernel programming	Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations	MAPL 2019	Link to PDF	Open details Paper Description: Introduces a tile-level abstraction that raises GPU programming above CUDA threads while preserving enough control over locality, parallelism, and layouts to generate competitive AI kernels. Related Readings: ThunderKittens: Simple, Fast, and Adorable AI Kernels TileLang: A Composable Tiled Programming Model for AI Systems Triton-distributed: Programming Overlapping Kernels on Distributed AI Systems with the Triton Compiler
Paper 6	Megakernel programming	Mirage Persistent Kernel: A Compiler and Runtime for Mega-Kernelizing Tensor Programs	arXiv 2025	Link to PDF	Open details Paper Description: Replaces the kernel-per-operator execution model with an automatically generated persistent megakernel and SM-level task graph, enabling cross-operator pipelining and fine-grained compute-communication overlap. Related Readings: FlashFormer: Whole-Model Kernels for Efficient Low-Batch Inference Event Tensor: A Unified Abstraction for Compiling Dynamic Megakernel Fleet: Hierarchical Task-based Abstraction for Megakernels on Multi-Die GPUs
Paper 7	Attention kernel design and optimization	FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness	NeurIPS 2022	Link to PDF	Open details Paper Description: Establishes IO-aware algorithm design by reformulating exact attention around GPU SRAM capacity and HBM traffic rather than optimizing arithmetic count alone. Related Readings: FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-Precision FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving
Paper 8	System optimization through semantic information	Parrot: Efficient Serving of LLM-Based Applications with Semantic Variable	OSDI 2024	Link to PDF	Open details Paper Description: Demonstrates that compound and agentic applications should expose dependencies and application semantics so the serving system can optimize complete workflows rather than isolated requests. Related Readings: Towards End-to-End Optimization of LLM-Based Applications with Ayo SGLang: Efficient Execution of Structured Language Model Programs Agentix: An Efficient Serving Engine for LLM Agents as General Programs
Paper 9	Scaling large models	Insights into DeepSeek-V3: Scaling Challenges and Reflections on Hardware for AI Architectures	ISCA 2025	Link to PDF	Open details Paper Description: Connects MLA, MoE routing, FP8 training, communication overlap, and multi-plane networks to concrete hardware bottlenecks, providing a rare production-scale model-system-architecture co-design case study. Related Readings: DeepSeek-V3 Technical Report DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Paper 10	Data movement in MoE	Patterns Behind Chaos: Forecasting Data Movement for Efficient Large-Scale MoE LLM Inference	ISCA 2026	Link to PDF	Open details Paper Description: Characterizes temporal, spatial, and task-dependent regularities in expert activation and uses them to forecast MoE communication, demonstrating how workload prediction can guide expert placement and proactively hide large-scale all-to-all data movement. Related Readings: MoETuner: Optimized Mixture-of-Expert Serving with Balanced Expert Placement and Token Routing MoE-Infinity: Offloading-Efficient MoE Model Serving Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference
Paper 11	Hardware design space exploration	Timeloop: A Systematic Approach to DNN Accelerator Evaluation	ISPASS 2019	Link to PDF	Open details Paper Description: Formalizes the joint exploration of tensor mappings, dataflows, memory hierarchies, and compute structures, allowing students to understand why accelerator efficiency depends as much on mapping as on peak arithmetic throughput. Related Readings: Accelergy: An Architecture-Level Energy Estimation Methodology for Accelerator Designs Sparseloop: An Analytical Approach to Sparse Tensor Accelerator Modeling GAMMA: Automating the HW Mapping of DNN Models on Accelerators via Genetic Algorithm
Paper 12	LLM on CPUs	SMEPilot: Characterizing and Optimizing LLM Inference with Scalable Matrix Extensions	arXiv 2026	Link to PDF	Open details Paper Description: Characterizes existing LLM operators and phases, identifies their distinct compute and memory bottlenecks, and proposes operator-specific runtime, kernel, and data-layout optimizations. Related Readings: Understanding Performance Implications of LLM Inference on CPUs Inference Performance Optimization for Large Language Models on CPUs Towards Understanding, Analyzing, and Optimizing Agentic AI Execution: A CPU-Centric Perspective
Paper 13	GPU alternatives for LLMs	Corsair: An In-Memory Computing Chiplet Architecture for Inference-Time Compute Acceleration	IEEE Micro 2025	Link to PDF	Open details Paper Description: Describes d-Matrix's silicon-backed architecture combining digital in-memory compute, chiplets, block-floating-point numerics, hierarchical networks, and a dataflow software stack for bandwidth-bound generative-AI inference. Related Readings: Early Silicon of Raptor: The First 3D-DRAM Accelerator for Generative Inference Simba: Scaling Deep-Learning Inference with Multi-Chip-Module-Based Architecture A Latency Processing Unit: A Latency-Optimized and Highly Scalable Processor for Large Language Model Inference
Paper 14	Inference optimization on heterogeneous hardware platforms	Agent.xpu: Efficient Scheduling of Agentic LLM Workloads on Heterogeneous SoC	arXiv 2026	Link to PDF	Open details Paper Description: Characterizes the execution and shared-memory interference of agentic LLM workloads across CPUs, iGPUs, and NPUs, and proposes a flow-aware heterogeneous scheduler that dynamically maps and preempts work to improve end-to-end latency and throughput. Related Readings: HeteroMosaic: Exposing and Exploiting Heterogeneous Execution Opportunities for Energy-Efficient Edge LLM Inference Characterizing Mobile SoC for Accelerating Heterogeneous LLM Inference Fast On-device LLM Inference with NPUs
Paper 15	Energy/power optimization for LLM inference	DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiency	HPCA 2025	Link to PDF	Open details Paper Description: Dynamically adjusts the number of serving instances, model parallelism, and GPU frequency based on workload and SLOs, and reports roughly 53 percent energy savings. Related Readings: throttLL'eM: Predictive GPU Throttling for Energy Efficient LLM Inference Serving BEAM: Joint Resource-Power Optimization for Energy-Efficient LLM Inference under SLO Constraints GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving