Generative AI for Cache Policy Design
Question
Can a generative AI workflow propose useful last-level cache replacement policies without simulating every candidate from scratch? This team project for NC State's Generative AI for Systems course explored that question with an LLM, a learned performance surrogate, and architecture simulation.
The loop
The system began with a candidate cache-replacement policy, then compiled and evaluated selected candidates in ChampSim. The simulator measured instructions per cycle, or IPC, across workloads. Those measurements became feedback for the next generation step, so the process could refine an idea instead of treating each prompt as an isolated answer.
Memory without vector search
Rather than retrieve documents through an embedding index, the project kept structured experiment memory. Policy descriptions, design lineage, compilation outcomes, and workload-level measurements gave the next prompt direct context about what had already been tried and how it behaved. This vectorless retrieval pattern made the prior experimental record part of the design conversation.
Screen before the full run
Not every generated policy needs the same evaluation budget. A screening heuristic compared representative workload evidence with baselines and surrogate estimates to identify weak candidates early. Candidates that cleared that screen moved to broader simulation, where measured outcomes, rather than the heuristic, became the evidence used by the next iteration.