Fundamentals of GPU Architecture: SIMT Core Part 3

Understanding the Two Loop Approximation in GPU Architecture

Overview of GPU Core Architecture

• Nick introduces the topic of GPU architecture, specifically focusing on the two loop approximation and its relation to core architecture.

• Recaps the one loop approximation, emphasizing how it handles instruction execution within warps and latency hiding through multiple warps.

Limitations of One Loop Approximation

• Discusses the inefficiency of the one loop approximation due to lack of dependency tracking, which requires waiting for previous instructions to complete before issuing new ones.

• Highlights that without dependency information, a large number of warps are needed to effectively hide long latency instructions.

Introduction to Two Loop Approximation

• Proposes that by incorporating dependency information, fewer warps can be used while still allowing multiple instructions from those same warps to be issued.

• Describes an instruction buffer that stores instructions fetched from cache and a scheduler that determines which instructions can be executed based on their readiness.

Instruction Buffer and Dependency Tracking

• Explains how instruction memories function as a first-level cache supported by secondary unified caches, aiding in hiding instruction cache misses.

• Introduces classical CPU methods for detecting data dependencies: scoreboards and reservation stations. Emphasizes simplicity in design for GPUs due to their massively parallel nature.

Challenges with Scoreboard Implementation

• Notes that implementing a simple inorder scoreboard incurs significant overhead due to the large number of registers required per warp (up to 8192 bits).

• Discusses complications arising from multi-threaded processors needing multiple threads to probe the scoreboard simultaneously, leading to excessive read port requirements (256 ports).

Solutions for Efficient Dependency Tracking

• Suggests limiting scheduling consideration only to certain warps as a naive solution but warns this reduces parallelism benefits.

• Introduces an alternative design from 2008 where instead of one bit per register per warp, several entries track outstanding dependencies more efficiently.

Final Thoughts on Two Loop Architecture

• Concludes with how this new structure allows better management of dependencies while maintaining performance efficiency by checking dependencies when instructions enter the buffer.

• Summarizes key ideas behind two loop architecture: reducing warp count while enabling multiple instruction issuance based on cleared dependencies.

Additional Resources

• Mentions resources available on GitHub related to GPU programming and ongoing projects including a book draft covering various programming topics.