reward-function-v330
Installation
SKILL.md
Reward Function v3.3.0 - Risk-Aware Composite Reward
Overview
| Item | Details |
|---|---|
| Date | 2025-01-21 |
| Goal | Fix reward hacking and P&L gradient saturation |
| Files | alpaca_trading/gpu/vectorized_env.py, alpaca_trading/gpu/ppo_trainer_native.py |
| Status | Success |
Context
v2.5.0 introduced trading_incentive (+0.02 for non-HOLD actions) to fix HOLD bias.
However, this created the opposite problem: reward hacking via overtrading.
Additional issues:
- Direction weight (40%) dominated despite being binary (+1/-1)
- P&L weight (25%) too low despite being the actual objective
- P&L saturated via
tanh(pnl * 10)- destroyed gradient signal beyond ±0.1% moves - Exploration was limited due to low entropy coefficient (0.005)