SPECTRA: Adaptive Execution of Speculative Decoding on a Runtime-Reconfigurable Tiled Architecture

{'Problem': 'The paper addresses the challenge of efficient autoregressive decoding on edge devices, which are often limited by computational and memory resources. The authors propose a solution to enhance performance in such constrained environments. This work is presented as a preprint and has not undergone peer review.', 'Method': 'The core technical contribution is the SPECTRA architecture, which is a runtime-reconfigurable tiled architecture. It employs two levels of reconfiguration: tile-level and system-level. Tile-level reconfiguration allows for adjustments in tile count, kernel partitioning, and communication patterns, while system-level reconfiguration operates on a per-kernel basis. The compute engine within SPECTRA can switch between systolic execution for General Matrix Multiplications (GEMMs) and vector-lane execution for General Vector Operations (GEMVs). The architecture was evaluated on a 20-tile FPGA prototype using models from the Pythia, SmolLM2, and GPT-2 families.', 'Results': 'The results indicate a speedup of 2.09× from tile-level reconfiguration compared to fixed designs. Additionally, system-level adaptability provides an extra gain of 1.25× over fixed designs. These improvements demonstrate the effectiveness of the proposed architecture in enhancing decoding efficiency.', 'Limitations': 'The authors do not report any limitations in the study. However, potential limitations could include the scalability of the architecture beyond the tested 20-tile FPGA prototype and the generalizability of results across different edge device configurations.', 'Why it matters': 'The implications of this work are significant for downstream applications in edge computing, particularly in natural language processing tasks that require efficient model inference. By improving the adaptability and performance of autoregressive decoding, SPECTRA could enable more complex models to run effectively on resource-constrained devices, thus broadening the applicability of AI technologies in real-world scenarios.'}