An architecture demonstrates efficient distributed inference for large language models with minimal setup, suggesting wide access to advanced AI tools.
Key Points
The central aim is to create a web-based architecture for distributed inference of large language models that accommodates hardware diversity and minimizes setup complexity.
Developed a web-based distributed architecture for LLM inference
Used Rust for orchestrating model partition assignments
Implemented ONNX Runtime Web for browser inference
Utilized WebGPU for computational acceleration
Created a dynamic partitioning algorithm for performance optimization.
Orchestration server overhead is negligible, less than 0.15% of total inference time
Networking time remains under 25% in high-bandwidth situations with up to 10 workers
Data transfer reduced by 75 MB per request through token decoding integration
Prefix caching enhances Time to First Token by up to 30%
Although throughput is lower than local engines, the solution remains efficient for accessing large models.