What question did this study set out to answer?

The central aim is to create a web-based architecture for distributed inference of large language models that accommodates hardware diversity and minimizes setup complexity.

March 27, 2026Open Access

An Architecture for Web-Based Distributed LLM Inference

Key Points

The central aim is to create a web-based architecture for distributed inference of large language models that accommodates hardware diversity and minimizes setup complexity.
Developed a web-based distributed architecture for LLM inference
Used Rust for orchestrating model partition assignments
Implemented ONNX Runtime Web for browser inference
Utilized WebGPU for computational acceleration
Created a dynamic partitioning algorithm for performance optimization.
Orchestration server overhead is negligible, less than 0.15% of total inference time
Networking time remains under 25% in high-bandwidth situations with up to 10 workers
Data transfer reduced by 75 MB per request through token decoding integration
Prefix caching enhances Time to First Token by up to 30%
Although throughput is lower than local engines, the solution remains efficient for accessing large models.

Abstract

LLMs reach many billions of parameters, requiring hundreds of gigabytes of memory to run inference for the largest frontier models. This restricts access to large organizations using server-grade hardware. Distributed LLM inference enables multiple devices to serve models that would be too large for just one device. However, existing solutions remain restrictive. In particular, common inference servers, such as vLLM, assume homogeneous hardware, whereas frameworks like Petals necessitate manual configuration and setup.To this end, we propose an architecture for web-based distributed LLM inference. Using the browser as a distribution tool, we enable zero-cost deployment; participants connect to a website, and a Rust-based orchestrator automatically assigns model partitions. The system uses ONNX Runtime Web for browser-based inference, WebGPU for acceleration, and Protocol Buffers over WebSockets for efficient communication. To accommodate hardware heterogeneity, we implement a dynamic partitioning algorithm that minimizes end-to-end latency based on worker capabilities.Our experiments show that the orchestration server introduces negligible overhead, accounting for less than 0.15% of the inference time. In high-bandwidth environments, networking time remains below 25% for up to 10 workers, suggesting that execution is primarily compute-bound. Integrating token decoding into the ONNX graph reduces data transfer by 75 MB per request, while prefix caching improves Time to First Token (TTFT) by up to 30%. Although throughput is lower than local inference engines such as Ollama, we empirically demonstrate that web-based distributed inference is a viable, zero-setup solution for accessing large-scale models.

Ask AI

Helpful

Bookmark

View Full Paper