This repo implements the system detailed in Approaching Human-Level Forecasting with Language Models by Halawi et al. This is a summary of the architecture:
The system uses large language models to generate accurate forecasts for arbitrary Yes/No questions -- like those seen on popular prediction markets such as Polymarket, Metaculus, and Manifold.
Example questions you could predict:
- Will the price of Dogecoin exceed $0.50 USD during any 48h period before 2025?
- Will Carlos Alcaraz win the 2024 French Open?
- Will NVDA beat its consensus earnings forecast for Q3 2024?
- Will Joe Biden win the popular vote in the 2024 general election?
The system uses retrieval augmented generation and hundreds of news articles obtained from the NewsCatcher API to research and answer each question it is provided.
It is the core of what powers the Cassandra0racle Twitter account.
Install dependencies:
yarn installGet API keys for:
Add your API keys:
cp .env.example .env
vim .env
To make a prediction:
yarn predict questions/spacexMoonLanding.json
Question JSON files are expected to have the following properties:
- question: A yes/no question.
- resolutionCriteria: The exact conditions under which the answer to the question should be Yes vs No.
- background: Background information relevant to answering the question.
- beginDate: The start time of the forecasting period, formatted as an ISO 8601 string.
- closeDate: The end time of the forecasting period, formatted as an ISO 8601 string.
Approximate cost to make a single prediction is currently $0.25.