Answers are grounded in this paper and cite their sources.
We present mandala , a Python library that largely eliminates the accidental complexity of scientific data management and incremental computing. While most traditional and/or popular data management solutions are based on logging , mandala takes a fundamentally different approach, using memoization of function calls as the fundamental unit of saving, loading, querying and deleting computational artifacts. It does so by implementing a compositional form of memoization, which keeps track of how memoized functions compose with one another. In this way: (1) complex computations are effectively memoized end-to-end, and become ‘interfaces’ to their own intermediate results by retracing the memoized calls; (2) all computations in a project form a single computational graph, which can be explored, queried and manipulated in high-level ways through a computation frame , which is a natural generalization of a dataframe that replaces columns by a computation graph, and rows by (partial) executions of this graph. Several features implemented on top of the core memoization data structures — such as natively and transparently handling Python collections, in-memory caching of intermediate results, and a flexible versioning system with dynamic dependency tracking — turn mandala into a practical and simple tool for managing and interacting with computational data.
No takes yet. Share an insight, caveat, or question.
Aleksandar Makelov (2024) studied this question.
Synapse has enriched 10 related papers. Consider them for comparative context: