Oracle: Forecasting Pipeline for Kalshi Markets
The hard part of forecasting with a language model isn't the model. It's not fooling yourself.
Every prediction market publishes a live price, and that price is the crowd's answer to the exact question you're asking. If any of it reaches the model, through the prompt, a retrieved article, or a tool call that quietly returns the current odds, the model stops forecasting and starts paraphrasing the market. The result looks brilliant in a backtest and means nothing. Oracle is built so that can't happen.
It began as an engineering follow-on to my CSAIL research on LLM ensemble forecasting for prediction markets, rebuilt as a standalone, general-purpose pipeline rather than a one-off experiment harness.
Keeping the price out
The forecasting stage runs behind a price quarantine: checks that fail loudly if a market's current price could reach the model before it commits to a number. The same checks run again at the tool layer, including for external research tools reached over MCP, because the tool layer is where leakage actually happens. A research tool that helpfully returns "current odds: 63%" defeats the whole experiment, and it does it quietly. One quarantine is a policy. Two is an invariant.
Kalshi metadata reaches the blind stage only through a price-redacted view, and the domain research tools (Wikipedia, Open-Meteo, Federal Register, arXiv, SEC, ESPN) are opt-in per run, so what the model could have seen is always a fact about the run config rather than a guess.
Research and forecasting
Everything crossing a stage boundary is a typed Pydantic model: Kalshi identifier parsing, the compiled market question, extracted evidence, the forecast itself. That is less about elegance than about the fact that an untyped dict passed between stages is exactly how a price ends up somewhere it shouldn't be.
A structured LLM research planner sits in front of evidence extraction, with deterministic domain-aware routing to the specialized tools. The forecasting engine is provider-neutral, with versioned prompt templates and structured-output adapters for both OpenAI and Gemini, so swapping providers doesn't silently change what the prompt said.
Aggregation and recommendations
Forecasts aggregate in log-odds rather than probability space, weighted by each agent's historical Brier and log-loss scores from backtests. Aggregation is a separate problem from forecasting, and averaging probabilities across agents whose calibration you never measured throws away most of what you paid for.
Being right is also not the same as having a trade worth taking, so the recommendation math is fee-aware and works off the post-fee edge rather than the raw probability gap.
Backtesting and persistence
Backtest replay is deterministic, driven from JSONL files, manifests, or shard directories, and the dataset is validated for duplicates, scoreability, and temporal leakage before anything gets scored. Summaries break out per agent and per phase, and the replay dataset metadata is kept, so a calibration weight can always be traced back to the runs that produced it. Deterministic replay is what makes a claimed edge checkable instead of anecdotal.
Runs persist through SQLAlchemy with a sanitized, immutable config snapshot each, plus a content-addressed blob store. Read-only FastAPI endpoints and a small local HTML UI cover browsing runs, evaluations, and outcomes.
Running it
CLI-driven, uv-managed Python project:
uv run oracle parse KXEXAMPLE-26JUN30-YES
uv run oracle compile-question path/to/kalshi_market.json
uv run oracle forecast KXEXAMPLE-26JUN30-YES --engine llm --research-engine llm
uv run oracle record-outcome KXEXAMPLE-26JUN30-YES yes
uv run oracle backtest path/to/replay_manifest.toml --summary-json data/backtests/summary.json
uv run uvicorn oracle.api.app:app --reload
GET /runs/{run_id}
GET /runs/{run_id}/evaluations
GET /markets/{market_ticker}/outcome
The next thing I want is a paper-trading loop against Kalshi's live feed, so calibration weights update online instead of only from replayed backtests.