What question did this study set out to answer?

To implement and evaluate Hidden-FFN on Apple GPUs for compact, accelerated inference and training of sparse feedforward networks across pruned language models.

August 19, 20260 citationsOpen Access

Hidden-FFN on Apple GPUs: Compact Inference and Training from Homogeneous Sparse FFNs to WANDA-Pruned Language Models

View Full Paper

MPMaxime ParkerUniversité Toulouse III - Paul Sabatier

Key Points

To implement and evaluate Hidden-FFN on Apple GPUs for compact, accelerated inference and training of sparse feedforward networks across pruned language models.
Implemented four Apple Metal GPU execution paths supporting compact active weights, gradients, and AdamW optimizer states without dense fallback.
Evaluated performance on an Apple M1 GPU across homogeneous sparse FFNs and WANDA-pruned Pythia models (70M, 160M, and 410M parameters) at various sparsity densities.
Homogeneous sparse FFN inference achieved 2.52× and 1.61× speedups at 5% and 10% density, while training reached 3.38× at 5% and 1.69× at 20% density versus iso-support PyTorch/MPS.
Pythia-410M decode demonstrated a 1.72× speedup at 10% density, and training speedups at 20% density scaled from 1.15× on 70M to 1.57× on 410M.
Full WikiText-2 evaluation maintained negative log-likelihood within 1.3 × 10⁻⁶, with 14 of 15 multi-epoch training runs passing the paired quality gate.

Abstract

This paper presents a first GPU implementation of Hidden-FFN (HFFN), whose companion CPU worksestablished an exact compact runtime for fixed sparse FFNs. Four Metal paths are evaluated: homogeneous FFNinference and training, then Pythia-70M, 160M, and 410M inference and training with fixed WANDA supports.Only FFNs are replaced; active weights, gradients, and AdamW states remain compact, with no dense fallback orpost-attention fusion.On an Apple M1 GPU, homogeneous inference reaches 2.52× and 1.61× speedup over iso-support PyTorch/MPSat 5% and 10% density; training reaches 3.38× at 5% and 1.69× at 20%. Pythia-410M decode reaches 1.72×at 10%, while training at 20% and N = 32 scales from 1.15× on 70M to 1.57× on 410M. Full WikiText-2inference preserves NLL within 1.3 × 10−6. Across five ten-epoch training attempts per model, all 15 HFFNtrajectories remain finite and 14 of 15 pass the paired final-quality gate. The resulting first-version baselinetherefore establishes both the performance opportunity and a functional compact training runtime, withouttreating its measured operating range as a GPU ceiling of HFFN.

AI से पूछें

Bookmark

View Full Paper

AI से पूछें

Bookmark

View Full Paper

Hidden-FFN on Apple GPUs: Compact Inference and Training from Homogeneous Sparse FFNs to WANDA-Pruned Language Models

Key Points

Abstract

Cite This Study