How to Be Fast and Not Furious: Looking Under the Hood of CPU Cache Prefetching

Software-based prefetching is a powerful method for tolerating access penalties that are encountered by data processing systems: memory latency. Although the idea appears straightforward—simply informing the CPU about upcoming data accesses—the intricacies of its implementation remain insufficiently understood. Existing works demonstrate how to rewrite algorithms for prefetching, yet they often overlook the limitations and hardware implications of bringing data into the cache hierarchy. In this paper, we examine software-based prefetching thoroughly by delving into its implementation and identifying pitfalls across various platforms. Furthermore, we provide actionable insights and recommendations for developers seeking to boost their applications through this technique.