← cs
$ cat projects/Oracle.md

Oracle – Research-Grade Forecasting Pipeline for Kalshi Markets

A blind-forecasting pipeline for Kalshi prediction markets with leakage-safe research tools and calibrated backtesting.

2026-06-30
Python
view on github

Oracle: Research-Grade Forecasting Pipeline for Kalshi Markets

An engineering follow-on to my CSAIL research on LLM ensemble forecasting for prediction markets, built as a standalone, general-purpose pipeline rather than a one-off experiment harness.

What I Built

Blind-Forecasting Safety:

  • Price-quarantine checks that prevent the forecasting stage from ever seeing the market's current price before it commits to a forecast: the same class of bug (accidental price leakage) that quietly inflates a lot of "beat the market" forecasting results
  • The same quarantine logic is enforced again at the tool layer, including for external research tools reached over MCP

Research & Forecasting Engine:

  • Typed Pydantic domain schemas end-to-end: Kalshi identifier parsing, market-question compilation, structured evidence, structured forecast outputs
  • A structured LLM research planner and evidence extractor, with domain-aware deterministic planning for routing to specialized tools
  • A provider-neutral, versioned-prompt-template forecasting engine with structured-output adapters for both OpenAI and Gemini
  • An MCP research tool adapter (JSON-RPC over stdio) with its own blind-stage price-leak checks, plus opt-in domain research tools (Wikipedia, Open-Meteo, Federal Register, arXiv, SEC, ESPN) and an opt-in price-redacted Kalshi metadata tool

Aggregation & Recommendations:

  • Log-odds forecast aggregation across agents/runs
  • Fee-aware recommendation math (a forecast being "right" isn't the same as a trade being worth taking after fees)
  • Calibration-weighted aggregation hooks driven by each agent's historical Brier and log-loss scores from backtests

Backtesting & Persistence:

  • Deterministic backtest replay from JSONL files, manifests, or shard directories, with dataset validation for duplicates, scoreability, and temporal leakage
  • Backtest summaries broken out per-agent and per-phase, with replay dataset metadata retained for calibration provenance
  • SQLAlchemy persistence with sanitized, immutable config snapshots per run, and a content-addressed blob store
  • Read-only FastAPI inspection endpoints and a local HTML UI for browsing persisted runs, evaluations, and outcomes

Architecture

CLI-driven, uv-managed Python project:

uv run oracle parse KXEXAMPLE-26JUN30-YES
uv run oracle compile-question path/to/kalshi_market.json
uv run oracle forecast KXEXAMPLE-26JUN30-YES --engine llm --research-engine llm
uv run oracle record-outcome KXEXAMPLE-26JUN30-YES yes
uv run oracle backtest path/to/replay_manifest.toml --summary-json data/backtests/summary.json

uv run uvicorn oracle.api.app:app --reload
GET /runs/{run_id}
GET /runs/{run_id}/evaluations
GET /markets/{market_ticker}/outcome

Lessons Learned

  • Price leakage is easy to introduce accidentally in any pipeline that touches both "current market state" and "forecast generation" in the same process, so it's worth enforcing the quarantine at more than one layer
  • Aggregation is a separate problem from forecasting: a well-calibrated ensemble needs historical Brier/log-loss weighting, not a simple average
  • Deterministic, leakage-checked backtest replay is what makes a forecasting system's claimed edge trustworthy rather than anecdotal

Future Work

  • Live paper-trading loop against Kalshi's real-time market feed
  • Expand the calibration-weighted aggregation from backtest-only to online updating
  • Broaden the domain research tool set based on which categories of markets it forecasts best/worst on