Project
DQN vs Q-Learning Market Making Agent
- Reinforcement Learning
- DQN
- Python
- Quant
A head-to-head comparison of tabular Q-Learning and Double DQN as market-making agents inside a custom trading environment. The goal was to measure — concretely — what the move from tabular RL to deep RL actually buys you, and at what compute cost.
Overview
Designed and trained two RL agents (tabular Q-Learning and Double DQN) to act as market makers in a custom trading environment. Engineered a reward function balancing profit against inventory risk over a 25-action bid/ask offset space. DQN delivered ~4× higher profit (680.71 vs 169.77) at ~20× the compute cost, quantifying the tabular-vs-deep-RL tradeoff.
What I worked on
I built the trading environment from scratch — state space (inventory, volatility, time remaining), a 25-action bid/ask offset space, and a reward function that traded raw profit against inventory risk. Then I trained both agents on identical episodes and instrumented the training loop to capture profit, inventory excursions, and wall-clock cost.
Results
Double DQN delivered roughly 4× the profit of the tabular agent (680.71 vs 169.77 in the evaluation regime) but cost about 20× more compute to train. The interesting finding wasn’t that DQN wins — it was the shape of the tradeoff, and how much of the tabular agent’s gap came from poor coverage of the inventory dimension.