🧠 Open RL Trading Agent for Binance Futures

A high-performance, research-grade reinforcement learning system for intraday trading on Binance Futures. Built using Dueling Double Deep Q-Networks (D3QN) and Prioritized Experience Replay (PER), this framework supports realistic backtesting, robust benchmarking, and scalable experimentation.

⚠️ Note: This release runs in demo mode with a lightweight ~256K-parameter model, short 10-minute sessions, and 30-minute input context — optimized for fast execution, CPU-only training, and interpretable visualizations. The full architecture (60-min sessions, 90-min context, 1M+ parameters) is still dormant. This project lays the foundation for a scalable, production-grade trading AI.

📖 Read the full technical article (English):
RL Agent for Algorithmic Trading on Binance Futures — Architecture, Backtest, and Results

📖 Article (Russian):
RL-агент для алгоритмической торговли на Binance Futures: архитектура, бэктест, результаты

This repository includes:

• ✅ A modular RL pipeline for market simulation and policy learning
• ✅ A custom Gym-compatible environment with slippage, commissions, and penalties
• ✅ A D3QN agent with PER buffer, epsilon decay, and action masking
• ✅ A complete lifecycle: training, testing, backtesting, and baseline evaluation
• ✅ An honest CNN classifier as a supervised baseline
• ✅ Config-driven experiment isolation and reproducibility

A full equity curve over the backtest period (March–June 2025):

• Mean Reward: 0.00285
• Mean PnL: +28.47 USDT
• Win Rate: 55.67%

• Final Balance Change: +144.23%
• Sharpe Ratio: 1.85
• Sortino Ratio: 2.05
• Accuracy: 69.6%
• Profit Days: 78.57%
• Max Drawdown: –22.49%
• Average Trade Size: 11,324.29 USDT
• Trades per Day: 2.00

• Mean PnL: –27.95 USDT
• Win Rate: 47.85%

A curated minute-level dataset from Binance Futures focused on high-volatility segments. Packaged in .npz format.

Each session contains 60 minutes of market activity and serves as a standalone trading window.

📂 Dataset: HuggingFace Hub

# 1. Train the RL agent
python train.py configs/alpha.py

# 2. Evaluate on the test set
python test_agent.py configs/alpha.py

# 3. Run realistic backtest
python backtest_engine.py configs/alpha.py

# 4. Train supervised CNN baseline
python baseline_cnn_classifier.py configs/alpha_baseline_cnn.py

# 5. Run Optuna config optimization
python optimize_cfg.py configs/alpha.py --trials 100 --jobs 1

# 6. Show and save top-10 trials for a given config
python get_info_from_optuna.py configs/alpha.py --n-best-trials 10

# 7. If your objective is minimized
python get_info_from_optuna.py configs/alpha.py --n-best-trials 10 --direction min

rl_trading_binance/
├── train.py               # RL training
├── test_agent.py          # Agent evaluation
├── backtest_engine.py     # Full backtest simulation
├── optimize_cfg.py        # Optuna config tuning
├── baseline_cnn_classifier.py
├── config.py              # Config model
├── configs/               # Experiment configs
├── model.py               # CNN + Dueling Q-network
├── agent.py               # D3QN logic
├── replay_buffer.py       # Prioritized replay buffer
├── trading_environment.py # Gym-compatible environment
├── utils.py               # Logging, visualization, metrics
├── data/                  # Market datasets (.npz format)
│   ├── train_data.npz
│   ├── val_data.npz
│   ├── test_data.npz
│   └── backtest_data.npz
├── output/                # Experiment results
│   └── <config_name>/
│       ├── logs/
│       ├── plots/
│       └── saved_models/

Profitable Session	Unprofitable Session

Train Example	Val Example	Test Example	Backtest Example

A more advanced version of this agent is deployed live, scanning Binance Futures in real-time and publishing trade decisions:

• Scans all symbols every minute
• Detects volatility spikes
• Predicts trade direction and confidence
• Publishes signal + final trade outcome with PnL

This is a public demonstration of production-oriented RL decision-making in streaming environments.

👉 Follow: @binance_ai_agent

Live Signal + Prediction	Verification Example

⚠️ This system is experimental and for educational purposes only.

• Replace CNN with iTransformer / Perceiver IO / Temporal Fusion Transformer (TFT)
• Integrate Model-Based RL (Dreamer, MuZero)
• Extend agent architectures: A3C / PPO / SAC / DDPG / TD3
• Real-time trade execution via Binance REST & WebSocket API
• Implement adaptive action masking + dynamic risk management
• Build full streaming pipeline with Airflow + TimescaleDB
• Enable live training on streamed data
• Expand exchange compatibility: integrate Bybit, OKX, and KuCoin APIs
• Support both Futures and Spot markets across multiple crypto exchanges
• Extend to traditional markets: equities (NASDAQ, NYSE) and major Forex pairs

If this project helps your research, please cite:

@software{Kolesnikov_2025_RL_Trading_Binance,
  author    = {Yuriy Kolesnikov},
  title     = {Open RL Trading Agent for Binance Futures (D3QN + PER)},
  year      = {2025},
  publisher = {GitHub},
  url       = {https://github.com/YuriyKolesnikov/rl-trading-binance},
  version   = {0.1.0}
}

Key methods referenced in this repository:

@inproceedings{vanHasselt2015DoubleDQN,
  title={Deep Reinforcement Learning with Double Q-learning},
  author={Hado van Hasselt and Arthur Guez and David Silver},
  booktitle={AAAI},
  year={2016},
  url={https://arxiv.org/abs/1509.06461}
}

@inproceedings{Wang2016Dueling,
  title={Dueling Network Architectures for Deep Reinforcement Learning},
  author={Ziyu Wang and Tom Schaul and Matteo Hessel and Hado van Hasselt and Marc Lanctot and Nando de Freitas},
  booktitle={ICML},
  year={2016},
  url={https://proceedings.mlr.press/v48/wangf16.html}
}

@inproceedings{Schaul2016PER,
  title={Prioritized Experience Replay},
  author={Tom Schaul and John Quan and Ioannis Antonoglou and David Silver},
  booktitle={ICLR},
  year={2016},
  url={https://arxiv.org/abs/1511.05952}
}

Licensed under the MIT License — free for commercial and non-commercial use. Attribution is appreciated.

Developed by @YuriyKolesnikov

For integration, research collaboration, or consulting — feel free to reach out.