Monthly Archives: July 2026

Problem: Let \(p(r_{t+1},\mathbf x_{t+1}|\mathbf x_t)\) be a Markov reward process. Compare constant-\(\alpha\) Monte Carlo (MC) evaluation and \(n\)-step constant-\(\alpha\) temporal difference (TD) evaluation. Solution: Essentially, TD is a bootstrapped version of MC, and indeed it turns out \(\lim_{n\to\infty}\text{TD}_{(n,\alpha)}=\text{MC}_{\alpha}\). That is, … Continue reading →