Tensor Data Interchange for AI/ML Systems: A Survey and the Hurray Proposal

AI/ML systems move large amounts of tensor data between libraries, processes, machines, accelerators, and storage. Model weights are loaded from files into accelerator memory. Frameworks exchange tensors without copying them. Distributed training splits tensors across GPUs. Disaggregated inference transfers key-value (KV) caches between prefill and decode workers. A tensor is a multidimensional array of values. Moving a tensor from one system to another requires transferring its data, but also enough information to interpret that data: its element type, shape, and memory layout, i.e. how its elements are arranged in memory. Some tensors require additional information. Quantized tensors store values using lower-precision encodings and associated parameters such as scales. Sparse tensors store selected values together with indexes instead of storing every value. Paged KV caches use blocks whose physical order can differ from their logical sequence order. Tensors can also be split, or sharded, across several devices. Several interchange solutions cover parts of this information. DLPack describes strided tensors in memory, including their device. Apache Arrow defines dense and sparse tensor IPC (inter-process communication) representations and has added fixed- and variable-shape tensor types to its columnar data model. SafeTensors and GGUF store model tensors in files, with GGUF supporting many quantized representations. Zarr and NetCDF store large multidimensional arrays. NIXL and UCX move memory efficiently between machines and devices, while NCCL provides communication operations between GPUs. This paper compares these solutions and the tensor information they preserve. The comparison shows a practical gap for tensors whose physical representation matters to computation. Layout, quantization, device memory, paging, and sharding increasingly appear at the boundaries between runtimes, but among the systems surveyed here, no widely used interchange format combines these properties in one tensor description reusable across memory, files, and network transfers. We then present Hurray, an open-source tensor interchange project designed for this use case. Hurray defines a language-independent descriptor for a tensor's type, shape, layout, quantization, device and memory placement, buffers, and composition. The same descriptor is used for streaming and persistent files and can accompany data moved through existing communication systems. The goal is simple: when two runtimes support the same tensor representation, they should be able to exchange it without first converting it to an intermediate layout. When they do not, the difference should be explicit so that the required conversion can be selected.