Say we have a value network that should predict the future discounted reward for a given state at time t, which is
Then for the next timestep, we can construct a target, after the transition and getting a reward .
Train the network to reduce the error,
In a nutshell, what youβre really doing is training a network to estimate reward while itβs performing the action rather than at the end of an episode.
This gets a bit tricky because you are using the networkβs own estimate to train the network, but over the course of multiple episodes, the network starts to correct itself towards the actual reward obtained by the agent.
Another way to think about it is that, the network is self-correcting with every transition. Every step, it gets a reward.