Engineering notes
Since 2023 we have documented what actually works in AI systems — with numbers, sources and honest limits.
OpenAI's o1-preview and o1-mini, released September 12, 2024, trade latency and cost for accuracy by spending compute on hidden reasoning tokens at inference time. We examine where trained deliberation helps — math, code, planning — what it costs, why the chain of thought stays hidden, and what test-time compute means for engineering.
Context windows are working memory, not memory. This article separates the two concepts and reviews the three memory mechanisms in production use for LLM agents in 2024 — conversation summaries, vector memories and structured state — with their failure modes, why deliberate forgetting improves correctness, and which boundaries the GDPR sets for persistent agent memory.
Anthropic's prompt caching beta changes the unit economics of long prompts: cache writes cost 25 percent more than base input tokens, cache reads 90 percent less. We walk through the pricing, the break-even arithmetic, the limits of the five-minute TTL, and what the numbers mean for RAG pipelines and agent loops.
ColPali (June 2024) replaces the OCR parsing pipeline with direct embedding of document page images. We explain the late-interaction mechanism inherited from ColBERT, the ViDoRe benchmark results — 81.3 versus 66.1 nDCG@5 — and the storage and scaling trade-offs, and assess where visual retrieval fits in production RAG systems as of July 2024.
How invoices and contracts become schema-validated JSON with vision LLMs in 2024: what GPT-4o, Claude 3.5 Sonnet and Gemini 1.5 Pro deliver on documents, which accuracy ranges are realistic in production, why the validator is written before the prompt, and where human review remains structurally necessary.
Microsoft has published GraphRAG: an indexing pipeline that uses an LLM to extract an entity graph from text, partitions it with the Leiden algorithm, and pre-summarizes every community. We explain how this answers global questions that vector RAG cannot, what the paper actually measured, and why indexing cost is the real trade-off.
Claude 3.5 Sonnet, released on June 20 2024, beats Anthropic's flagship Claude 3 Opus at one fifth of the price and twice the speed. We examine the benchmark data, the collapsing cost-per-quality curve since GPT-4, and why Artifacts points toward working outputs instead of chat transcripts — limits included.
How to build and maintain golden test sets for LLM products in 2024: sampling real production queries, labeling with domain experts despite criteria drift, covering known failure modes, and calibrating automated judges against expert labels. And why dataset maintenance — not tooling — is the real evaluation bottleneck. With concrete numbers and dated sources.
GPT-4o, announced on 13 May 2024, processes text, vision, and audio in a single end-to-end network. We examine the verified numbers — 320 ms average voice latency, half the price of GPT-4 Turbo, five times the rate limits — and separate what the API delivers today from what the launch demos promise.
A practical guide to parameter-efficient fine-tuning as of April 2024. We cover how LoRA works, what QLoRA changes with 4-bit quantization, the current open-source toolchain, and a decision framework for tuning versus retrieval versus prompting — including what adapters do not deliver and how thousands of them run on one GPU.
Meta's Llama 3 (April 18, 2024) narrows the gap between open weights and closed APIs. We examine the 8B and 70B benchmarks, what 4-bit quantization does to hardware requirements, the utilization math of self-hosting versus API pricing, and the privacy-bound workloads where an open model is the only viable option.
Cohere's Command R, released 11 March 2024, treats grounded generation as a trained capability rather than a prompt trick: span-level citations, native tool use, 128k context, and open research weights. We examine what a RAG-tuned model changes for production systems, what it does not solve, and where the category is heading.