Knowledge Compile: Incremental LLM-Powered Knowledge Extraction Without Databases, Embeddings, or Graphs

Key Points

• The aim is to explore efficient knowledge extraction methods using LLMs without traditional databases or structures.
• Compile 192 ancient texts over 1,800 years with LLMs for knowledge extraction.
• Implement a system that allows searching through compiled knowledge using grep.
• Reduce token usage by 18x to optimize performance.
• Achieved a total compilation cost of $36 for extracting knowledge across 192 texts.
• Enabled instant search capabilities for extracted concepts and FAQs.
• Demonstrated efficiency through notable token reduction without requiring database infrastructure.