🧠 Backtesting with LLMs: Opportunities and Limitations in the Age of Context-Constrained Intelligence

Backtesting is the cornerstone of quantitative finance. It’s how we validate strategies, stress-test assumptions, and build confidence before risking capital. But what happens when we introduce large language models (LLMs) into the mix—agents that reason, synthesize, and debate like human analysts? The promise is enormous, but so are the constraints.

📦 The Context Window Bottleneck

At the heart of every LLM lies a context window—a finite memory span that determines how much information the model can process at once. Today’s cutting-edge models offer context windows up to 1 million tokens, which sounds vast until you try to simulate a multi-agent trading firm.

• Historical price data, news articles, earnings transcripts, insider trades, and technical indicators—all need to be packed into this window.
• Add in structured outputs, agent dialogues, and reasoning chains, and you quickly hit the ceiling.

This limitation forces trade-offs:

• Data pruning: Only the most relevant slices of history can be included.
• Temporal compression: Analysts summarize weeks of market activity into a few paragraphs.
• Tool delegation: Retrieval-augmented generation (RAG) systems offload memory to external databases, but at the cost of latency and coherence.

As context windows expand—toward 10M+ tokens and beyond—we’ll see richer simulations, longer memory chains, and more nuanced agent behavior. But for now, every token counts.

⏱️ Execution State: The Latency Wall

Another challenge is replicating the state of execution. Traditional backtesting frameworks simulate tick-level precision, order book dynamics, and latency-sensitive strategies. LLMs, however, operate at a different cadence:

• Mid-to-long term horizons: Think swing trades, not scalping.
• Mid-low frequency: Agents reason over daily or weekly intervals, not milliseconds.
• Mid-high latency: Decision loops involve multiple agents, debates, and validations—far slower than algorithmic execution engines.

This means LLM-based strategies are best suited for:

• Narrative-driven trades: Earnings season, macro shifts, sentiment swings.
• Portfolio rebalancing: Sector rotation, thematic exposure, risk overlays.
• Event anticipation: Regulatory changes, geopolitical developments, insider activity.

Trying to simulate high-frequency trading with LLMs is like racing a Formula 1 car through a chess tournament—it’s the wrong tool for the job.

🧪 Designing Realistic Backtests

To backtest LLM-driven strategies effectively, we need to rethink the framework:

• Agent orchestration: Simulate a firm-like structure—analysts, researchers, traders, risk managers—with clear roles and communication protocols.
• Temporal batching: Feed agents data in time slices (e.g., weekly snapshots) to mimic real-world decision cadence.
• Synthetic memory: Use structured logs and shared state to preserve continuity across rounds.
• Explainability-first: Capture agent rationale, tool usage, and trade-offs for auditability.

This isn’t just about performance metrics—it’s about understanding how and why decisions are made.

🧠 Key Components of an LLM-Based Backtesting Framework

1. Temporal Batching

LLMs are not designed for continuous, tick-level simulation. Instead, they operate best in discrete time intervals—daily, weekly, or monthly snapshots.

• Batching Strategy: Feed agents a curated slice of market data per round.
• State Preservation: Use structured memory to retain context across batches.

This mimics how human analysts review markets periodically and adjust positions accordingly.

2. Agent Orchestration

Rather than a monolithic model, Tangents uses multiple agents with specialized roles. A backtesting framework must simulate:

• Role-specific prompts: Each agent receives tailored data and constraints.
• Communication protocols: Agents exchange structured reports and engage in debates.
• Decision hierarchy: Final trades are approved by a fund manager agent after risk checks.

This layered structure mirrors real-world trading desks and allows for modular reasoning.

3. Context Window Management

LLMs have a finite context window (currently ~1 million tokens), which limits how much data can be processed at once.

• Data Compression: Analysts summarize long histories into concise insights.
• Selective Retrieval: Use RAG systems to fetch only relevant data.
• Tool Integration: Offload heavy computation (e.g., technical indicators) to external tools and feed results back into the LLM.

As context windows grow, frameworks will evolve to support richer, more continuous simulations.

4. Execution Modeling

LLMs are not latency-sensitive. They cannot simulate microsecond-level execution or order book dynamics.

• Strategy Scope: Focus on mid-to-long term trades (e.g., swing trading).
• Latency Tolerance: Accept that decisions may take seconds or minutes to finalize.
• Frequency Constraints: Limit strategy to low-frequency execution—daily or weekly trades.

This makes LLMs ideal for narrative-driven strategies, macro positioning, and sentiment-based trades.

5. Explainability and Logging

One of the biggest advantages of LLMs is their ability to explain decisions.

• Structured Logs: Capture agent rationale, tool usage, and trade-offs.
• Audit Trails: Preserve every step of the decision-making process.
• Debugging: Easily identify why a trade was made and what influenced it.

This is critical for compliance, model validation, and trust-building in financial applications.

6. Performance Metrics

Standard metrics still apply, but interpretation must be contextualized:

In LLM frameworks, qualitative metrics like decision coherence, agent alignment, and debate quality may also be tracked.

🔍 Considerations for Realism

• Data Integrity: Ensure historical data is timestamped and consistent across modalities.
• Tool Fidelity: Technical indicators, sentiment scores, and news parsing must be accurate and reproducible.
• Agent Constraints: Simulate realistic limits—capital allocation, risk thresholds, and execution delays.

🔮 Future Enhancements

• Live Data Feeds: Real-time backtesting with streaming inputs.
• Reinforcement Learning Loops: Agents learn from outcomes and refine strategies.
• Multi-Asset Portfolios: Simulate cross-asset reasoning and hedging behavior.
• Market Simulation Engines: Integrate synthetic environments for stress testing.