本文最后更新于：2025年2月16日 晚上
假设我们有一个个体（agent） ，其处在某个环境 中，个体在这个环境里一定会存在一个状态（state） （空间中的位置，时间中的某一刻），个体会采取某个行动（action） （例如空间中移动）导致状态更新。个体行动的方式被policy 建模。
policy的作用是使用概率建模个体在某个状态下采取某个行动的概率。假设时间步表示为t，则
a t ∼ π ( ⋅ ∣ s t ) a_t \sim \pi(\cdot|s_t)
a t ∼ π ( ⋅ ∣ s t )
即个体在状态s t s_t s t a t a_t a t π ( ⋅ ∣ s t ) \pi(\cdot|s_t) π ( ⋅ ∣ s t )
而我们给个体采取的每个行动都给予一个打分，这个打分叫做奖励（reward），奖励规则就定义了我们想要个体采取怎样的行动。强化学习的目标就是去选择一个policy，使agent根据policy行动的时候能够获得最大的回报（return）。
如果我们把agent看作LLM，state看作prompt，action看作模型选择的下一个token，那么语言模型其实完全可以和RL对应起来。LLM对于语言建模的方式是下一个token的概率，即
P ( y ∣ x ) = ∏ i p ( y i ∣ x , y < i ) P(y|x) = \prod_{i} p(y_i|x,y_{<i})
P ( y ∣ x ) = i ∏ p ( y i ∣ x , y < i )
所以模型预测下一个token就是在采取action，其state就是输入的token，即context（预测第一个token前是prompt，预测输出后是prompt+已预测的token），我们的policy就是语言模型建模的下一个token的概率分布。
但给语言模型的action打分并非易事，首先，我们打分是对于一个完整的答案进行的，例如生成答案正确的给高分，生成错误的给低分。但是当给生成风格打分的时候会很困难，因为有的人可能喜欢长的答案，有的人喜欢精简的答案，所以不同人类之间给出的分数可能会有很大的方差。
我们采取比较的方式创建reward model，而不是分数。例如给出两个答案，人类选择哪个答案更喜欢。选择的答案称为(chosen/winner)，抛弃的答案称为(reject/loser)。
在实际操作中，我们一般直接拿一个预训练好的LLM，将其head替换为一个 scalar head，其将hidden state投影到一个常数，这个常数就代表某个token的reward。我们一般取eos_token的reward作为整个回答的reward。
训练reward model的方式很简单，其优化目标为
arg min θ − log σ ( r θ ( x , y w ) − r θ ( x , y l ) ) \argmin_{\theta} -\log \sigma(r_\theta(x,y_w)-r_\theta(x,y_l))
θ a r g m i n − log σ ( r θ ( x , y w ) − r θ ( x , y l ) )
观察公式不难发现，优化就是提高模型对于y w y_w y w y l y_l y l σ \sigma σ
上文中提到，RL的目标是个体根据policy行动的最大化回报，即
π ∗ = arg max π J ( π ) \pi^*=\argmax_{\pi}J(\pi)
π ∗ = π a r g m a x J ( π )
这里J ( π ) J(\pi) J ( π )
J ( π ) = ∫ τ P ( τ ∣ π ) R ( τ ) d τ = E τ ∼ π [ R ( τ ) ] J(\pi) = \int_{\tau} P(\tau|\pi)R(\tau)d\tau = E_{\tau\sim\pi}[R(\tau)]
J ( π ) = ∫ τ P ( τ ∣ π ) R ( τ ) d τ = E τ ∼ π [ R ( τ ) ]
其中τ \tau τ
τ = ( s 0 , a 0 , s 1 , a 1 , … ) \tau = (s_0,a_0,s_1,a_1,\dots)
τ = ( s 0 , a 0 , s 1 , a 1 , … )
即模型从初始状态s 0 s_0 s 0 J ( π ) J(\pi) J ( π )
P ( τ ∣ π ) = ρ 0 ( s 0 ) ∏ t = 0 T − 1 P ( s t + 1 ∣ s t , a t ) π ( a t ∣ s t ) P(\tau|\pi) = \rho_0(s_0)\prod_{t=0}^{T-1}P(s_{t+1}|s_t,a_t)\pi(a_t|s_t)
P ( τ ∣ π ) = ρ 0 ( s 0 ) t = 0 ∏ T − 1 P ( s t + 1 ∣ s t , a t ) π ( a t ∣ s t )
即，一个Markov链，从状态s 0 s_0 s 0 ρ 0 ( s 0 ) \rho_0(s_0) ρ 0 ( s 0 ) s t + 1 s_{t+1} s t + 1 s t s_t s t a t a_t a t π ( a t ∣ s t ) \pi(a_t|s_t) π ( a t ∣ s t )
最后，我们在reward建模上有一些trick，我们建模一个有衰减的reward，而不是简单的求和
R ( τ ) = ∑ t = 0 ∞ γ t r t R(\tau) = \sum_{t=0}^\infty \gamma^t r_t
R ( τ ) = t = 0 ∑ ∞ γ t r t
其中r t r_t r t t t t a t a_t a t γ \gamma γ
语言建模中对于轨迹的定义稍有区别，由于自回归的特性，在这里action本身将成为state一部分，例如
prompt : Where is Shanghai?Response : Shanghai is in China.
"where is Shanghai?"作为状态s 0 s_0 s 0 a 0 a_0 a 0 s 1 s_1 s 1 a 1 a_1 a 1 s 2 s_2 s 2
我们已经完成了对目标函数的建模，即
J ( π θ ) = E τ ∼ π θ [ R ( τ ) ] J(\pi_\theta) = E_{\tau\sim \pi_\theta}[R(\tau)]
J ( π θ ) = E τ ∼ π θ [ R ( τ ) ]
我们的目标就是去找一个π θ \pi_\theta π θ
θ k + 1 = θ k + α ∇ θ J ( π θ ) ∣ θ k \theta_{k+1} = \theta_k + \alpha \nabla_\theta J(\pi_\theta)|_{\theta_k}
θ k + 1 = θ k + α ∇ θ J ( π θ ) ∣ θ k
在这里，因为我们是最大化目标函数，与loss的优化不一样，所以我们需要梯度上升，在这里是加上梯度的方向，而不是优化loss时减去梯度方向。
但紧接着会发现一个问题，我们计算期望需要采样出所有可能的τ \tau τ
那么，能不能尝试把这个期望的梯度∇ θ E τ ∼ π θ [ R ( τ ) ] \nabla_\theta E_{\tau \sim \pi_\theta}[R(\tau)] ∇ θ E τ ∼ π θ [ R ( τ ) ]
我们从目标函数的梯度开始做推导
∇ θ J ( π θ ) = ∇ θ E τ ∼ π θ [ R ( τ ) ] = ∇ θ ∫ τ P ( τ ∣ θ ) R ( τ ) d τ = ∫ τ ∇ θ P ( τ ∣ θ ) R ( τ ) d τ \begin{aligned}
\nabla_\theta J(\pi_\theta) &= \nabla_\theta E_{\tau \sim \pi_\theta}[R(\tau)] \\
&=\nabla_\theta \int_{\tau}P(\tau|\theta)R(\tau)d\tau \\
&=\int_{\tau}\nabla_\theta P(\tau|\theta)R(\tau)d\tau \\
\end{aligned}
∇ θ J ( π θ ) = ∇ θ E τ ∼ π θ [ R ( τ ) ] = ∇ θ ∫ τ P ( τ ∣ θ ) R ( τ ) d τ = ∫ τ ∇ θ P ( τ ∣ θ ) R ( τ ) d τ
注意到
∇ θ log P ( τ ∣ θ ) = ∇ θ P ( τ ∣ θ ) P ( τ ∣ θ ) ⇒ ∇ θ P ( τ ∣ θ ) = P ( τ ∣ θ ) ∇ θ log P ( τ ∣ θ ) \nabla_\theta \log P(\tau|\theta) = \frac{\nabla_\theta P(\tau|\theta)}{P(\tau|\theta)} \Rightarrow \nabla_\theta P(\tau|\theta) = P(\tau|\theta)\nabla_\theta \log P(\tau|\theta)
∇ θ log P ( τ ∣ θ ) = P ( τ ∣ θ ) ∇ θ P ( τ ∣ θ ) ⇒ ∇ θ P ( τ ∣ θ ) = P ( τ ∣ θ ) ∇ θ log P ( τ ∣ θ )
这里因为是优化目标所以省略log \log log
所以
∫ τ ∇ θ P ( τ ∣ θ ) R ( τ ) d τ = ∫ τ P ( τ ∣ θ ) ∇ θ log P ( τ ∣ θ ) R ( τ ) d τ = E τ ∼ π θ [ ∇ θ log P ( τ ∣ θ ) R ( τ ) ] \begin{aligned}
\int_{\tau}\nabla_\theta P(\tau|\theta)R(\tau)d\tau &= \int_\tau P(\tau|\theta)\nabla_{\theta}\log P(\tau|\theta)R(\tau)d\tau \\
&=E_{\tau\sim \pi_\theta}[\nabla_\theta \log P(\tau|\theta)R(\tau)]
\end{aligned}
∫ τ ∇ θ P ( τ ∣ θ ) R ( τ ) d τ = ∫ τ P ( τ ∣ θ ) ∇ θ log P ( τ ∣ θ ) R ( τ ) d τ = E τ ∼ π θ [ ∇ θ log P ( τ ∣ θ ) R ( τ ) ]
注意到上式还可以进一步简化，之前我们定义过
P ( τ ∣ π ) = ρ 0 ( s 0 ) ∏ t = 0 T − 1 P ( s t + 1 ∣ s t , a t ) π ( a t ∣ s t ) P(\tau|\pi) = \rho_0(s_0)\prod_{t=0}^{T-1}P(s_{t+1}|s_t,a_t)\pi(a_t|s_t)
P ( τ ∣ π ) = ρ 0 ( s 0 ) t = 0 ∏ T − 1 P ( s t + 1 ∣ s t , a t ) π ( a t ∣ s t )
则
∇ θ log P ( τ ∣ θ ) = ∇ θ ρ 0 ( s 0 ) + ∑ t = 0 T − 1 ( ∇ θ P ( s t + 1 ∣ s t ) + ∇ θ log π θ ( a t ∣ s t ) ) = ∑ t = 0 T − 1 ∇ θ log π θ ( a t ∣ s t ) \begin{aligned}
\nabla_\theta \log P(\tau|\theta) &= \nabla_\theta \rho_0(s_0) + \sum_{t=0}^{T-1}\left(\nabla_\theta P(s_{t+1}|s_t)+\nabla_\theta \log \pi_\theta (a_t|s_t)\right) \\
&=\sum_{t=0}^{T-1}\nabla_\theta\log\pi_\theta (a_t|s_t)
\end{aligned}
∇ θ log P ( τ ∣ θ ) = ∇ θ ρ 0 ( s 0 ) + t = 0 ∑ T − 1 ( ∇ θ P ( s t + 1 ∣ s t ) + ∇ θ log π θ ( a t ∣ s t ) ) = t = 0 ∑ T − 1 ∇ θ log π θ ( a t ∣ s t )
所以我们得到
∇ θ J ( π θ ) = E τ ∼ π θ [ R ( τ ) ∑ t = 0 T − 1 ∇ θ log π θ ( a t ∣ s t ) ] \nabla_\theta J(\pi_\theta)=E_{\tau\sim \pi_\theta}\left[R(\tau)\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_t|s_t)\right]
∇ θ J ( π θ ) = E τ ∼ π θ [ R ( τ ) t = 0 ∑ T − 1 ∇ θ log π θ ( a t ∣ s t ) ]
对于一个纯期望的表达式，我们可以用样本期望去逼近概率期望，即采样一部分trajectory构成集合D D D
1 ∣ D ∣ ∑ τ ∈ D R ( τ ) ∑ t = 0 T − 1 ∇ θ log π θ ( a t ∣ s t ) \frac{1}{|D|}\sum_{\tau\in D}R(\tau)\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_t|s_t)
∣ D ∣ 1 τ ∈ D ∑ R ( τ ) t = 0 ∑ T − 1 ∇ θ log π θ ( a t ∣ s t )
对于我们在上面找到的梯度
1 ∣ D ∣ ∑ τ ∈ D R ( τ ) ∑ t = 0 T − 1 ∇ θ log π θ ( a t ∣ s t ) \frac{1}{|D|}\sum_{\tau\in D}R(\tau)\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_t|s_t)
∣ D ∣ 1 τ ∈ D ∑ R ( τ ) t = 0 ∑ T − 1 ∇ θ log π θ ( a t ∣ s t )
是无偏的，即取足够多的trajectories它会收敛到真正的梯度。但当采样的trajectory不够多的时候会有很大的方差。
但注意到，我们估计的梯度实际上可以写为
1 ∣ D ∣ ∑ i = 1 ∣ D ∣ ( ∑ t = 0 T − 1 r ( s i , t , a i , t ) ) ( ∑ t = 0 T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) ) \frac{1}{|D|}\sum_{i=1}^{|D|}\left(\sum_{t=0}^{T-1}r(s_{i,t},a_{i,t})\right)\left(\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t})\right)
∣ D ∣ 1 i = 1 ∑ ∣ D ∣ ( t = 0 ∑ T − 1 r ( s i , t , a i , t ) ) ( t = 0 ∑ T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) )
这里r ( s i , t , a i , t ) r(s_{i,t},a_{i,t}) r ( s i , t , a i , t ) i i i a i , t a_{i,t} a i , t
即我们的R ( τ ) R(\tau) R ( τ ) R ( τ ) R(\tau) R ( τ )
1 ∣ D ∣ ∑ i = 1 ∣ D ∣ ( ∑ t = 0 T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) ( ∑ t ‘ = t T − 1 r ( s i , t ′ , a i , t ′ ) ) ⏟ rewards to go ) \frac{1}{|D|}\sum_{i=1}^{|D|}\left(\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t})\underbrace{\left(\sum_{t‘=t}^{T-1}r(s_{i,t'},a_{i,t'})\right)}_{\text{rewards to go}}\right)
∣ D ∣ 1 i = 1 ∑ ∣ D ∣ ⎝ ⎜ ⎜ ⎜ ⎜ ⎜ ⎛ t = 0 ∑ T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) rewards to go ( t ‘ = t ∑ T − 1 r ( s i , t ′ , a i , t ′ ) ) ⎠ ⎟ ⎟ ⎟ ⎟ ⎟ ⎞
即计算到时刻t t t t t t t t t
但是，reward本身也是有噪且稀疏的，这也会导致估计的梯度variance很高。在信号处理中我们都知道，要降低噪声，最好的办法是使用差值。所以，我们尝试在reward中减去一个baseline
1 ∣ D ∣ ∑ i = 1 ∣ D ∣ ( ∑ t = 0 T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) ( ∑ t ‘ = t T − 1 r ( s i , t ′ , a i , t ′ ) − b ) ) \frac{1}{|D|}\sum_{i=1}^{|D|}\left(\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t})\left(\sum_{t‘=t}^{T-1}r(s_{i,t'},a_{i,t'})-b\right)\right)
∣ D ∣ 1 i = 1 ∑ ∣ D ∣ ( t = 0 ∑ T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) ( t ‘ = t ∑ T − 1 r ( s i , t ′ , a i , t ′ ) − b ) )
可以证明减去baseline之后梯度估计仍然无偏。
在这里，我们定义baseline为某种value function，给定一个状态s t s_t s t s t s_t s t
我们记这个value function为V π ( s ) V^\pi(s) V π ( s ) s s s
V π ( s t ) = E a t [ r ( s t , a t ) + γ E s t + 1 [ V π ( s t + 1 ) ] ] V^\pi (s_t) = E_{a_t}[r(s_t,a_t)+\gamma E_{s_{t+1}}[V^\pi(s_{t+1})]]
V π ( s t ) = E a t [ r ( s t , a t ) + γ E s t + 1 [ V π ( s t + 1 ) ] ]
我们将value function用作baseline来计算reward和其的差值。在实现上，value function也是一个LLM+scalar head的配置，其负责预测每个时间步的价值。也就是说，我们直接用模型学一个value function出来。
在这里，我们再升级之前得到的"rewards to go"为Q function。Q function告诉我们从状态s t s_t s t a t a_t a t
Q π ( s t , a t ) = E s t + 1 [ r ( s t , a t ) + γ E a t + 1 [ Q π ( s t + 1 , a t + 1 ) ] ] Q^\pi(s_t,a_t) = E_{s_{t+1}}[r(s_t,a_t)+\gamma E_{a_{t+1}}[Q^\pi (s_{t+1},a_{t+1})]]
Q π ( s t , a t ) = E s t + 1 [ r ( s t , a t ) + γ E a t + 1 [ Q π ( s t + 1 , a t + 1 ) ] ]
即，从状态s t s_t s t a t a_t a t γ \gamma γ s t + 1 s_{t+1} s t + 1 a t + 1 a_{t+1} a t + 1
结合Q function和我们之前介绍的value function，我们的公式变为
1 ∣ D ∣ ∑ i = 1 ∣ D ∣ ( ∑ t = 0 T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) ( Q π ( s i , t , a i , t ) − V π ( s i , t ) ) ) = 1 ∣ D ∣ ∑ i = 1 ∣ D ∣ ( ∑ t = 0 T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) A π ( s i , t , a i , t ) ) \begin{aligned}
&\frac{1}{|D|}\sum_{i=1}^{|D|}\left(\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t})\left(Q^\pi(s_{i,t},a_{i,t})-V^\pi (s_{i,t})\right)\right)\\
&=\frac{1}{|D|}\sum_{i=1}^{|D|}\left(\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t})A^\pi (s_{i,t},a_{i,t})\right)
\end{aligned}
∣ D ∣ 1 i = 1 ∑ ∣ D ∣ ( t = 0 ∑ T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) ( Q π ( s i , t , a i , t ) − V π ( s i , t ) ) ) = ∣ D ∣ 1 i = 1 ∑ ∣ D ∣ ( t = 0 ∑ T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) A π ( s i , t , a i , t ) )
函数A π ( s i , t , a i , t ) = Q π ( s i , t , a i , t ) − V π ( s i , t ) A^\pi (s_{i,t},a_{i,t})=Q^\pi(s_{i,t},a_{i,t})-V^\pi (s_{i,t}) A π ( s i , t , a i , t ) = Q π ( s i , t , a i , t ) − V π ( s i , t ) s t s_t s t a t a_t a t a t a_t a t
但是计算advantage function是很困难的，主要在于Q function很难去直接计算，在这里，我们使用TD(Temporal Difference) error来近似估计advantage function。
A π ( s i , t , a i , t ) = Q π ( s i , t , a i , t ) − V π ( s i , t ) ≈ r ( s t , a t ) + γ V π ( s t + 1 ) − V π ( s t ) \begin{aligned}
A^\pi (s_{i,t},a_{i,t})&=Q^\pi(s_{i,t},a_{i,t})-V^\pi (s_{i,t}) \\
&\approx r(s_t,a_t) + \gamma V^\pi(s_{t+1}) - V^{\pi}(s_t)
\end{aligned}
A π ( s i , t , a i , t ) = Q π ( s i , t , a i , t ) − V π ( s i , t ) ≈ r ( s t , a t ) + γ V π ( s t + 1 ) − V π ( s t )
其中δ t = r ( s t , a t ) + γ V π ( s t + 1 ) − V π ( s t ) \delta_t=r(s_t,a_t) + \gamma V^\pi(s_{t+1}) - V^{\pi}(s_t) δ t = r ( s t , a t ) + γ V π ( s t + 1 ) − V π ( s t )
但仅仅有一个真实的rewardr ( s t , a t ) r(s_t,a_t) r ( s t , a t )
A ^ t π , ( 1 ) = δ t = r ( s t , a t ) + γ V π ( s t + 1 ) − V π ( s t ) A ^ t π , ( 2 ) = δ t + γ δ t + 1 = r ( s t , a t ) + γ r ( s t + 1 , a t + 1 ) + γ 2 V π ( s t + 2 ) − V π ( s t ) A ^ t π , ( 3 ) = δ t + γ δ t + 1 + γ 2 δ t + 2 = r ( s t , a t ) + γ r ( s t + 1 , a t + 1 ) + γ 2 r ( s t + 2 , a t + 2 ) + γ 3 V π ( s t + 3 ) − V π ( s t ) ⋯ \begin{aligned}
&\hat A^{\pi,(1)}_t = \delta_t = r(s_t,a_t) + \gamma V^\pi(s_{t+1}) - V^{\pi}(s_t) \\
&\hat A^{\pi,(2)}_t = \delta_t + \gamma \delta_{t+1} = r(s_t,a_t)+\gamma r(s_{t+1},a_{t+1})+\gamma^2 V^\pi (s_{t+2}) - V^\pi (s_t)\\
&\hat A^{\pi,(3)}_t = \delta_t + \gamma \delta_{t+1} + \gamma^2 \delta_{t+2} = r(s_t,a_t)+\gamma r(s_{t+1},a_{t+1})+\gamma^2 r(s_{t+2},a_{t+2}) +\gamma^3 V^\pi(s_{t+3})- V^\pi (s_t)\\
&\cdots
\end{aligned}
A ^ t π , ( 1 ) = δ t = r ( s t , a t ) + γ V π ( s t + 1 ) − V π ( s t ) A ^ t π , ( 2 ) = δ t + γ δ t + 1 = r ( s t , a t ) + γ r ( s t + 1 , a t + 1 ) + γ 2 V π ( s t + 2 ) − V π ( s t ) A ^ t π , ( 3 ) = δ t + γ δ t + 1 + γ 2 δ t + 2 = r ( s t , a t ) + γ r ( s t + 1 , a t + 1 ) + γ 2 r ( s t + 2 , a t + 2 ) + γ 3 V π ( s t + 3 ) − V π ( s t ) ⋯
当我们求和越来越多的时候，可以证明advantage function的估计的bias会越来越小。但是更多的求和项也会带来更大的variance，所以，为了平衡bias-variance，我们直接对上述的项进行加权求和
A ^ t π = ( 1 − λ ) ( A ^ t π , ( 1 ) + λ A ^ t π , ( 2 ) + λ 2 A ^ t π , ( 3 ) + ⋯ ) = ∑ l = 0 ∞ ( γ λ ) l δ t + l \begin{aligned}
\hat A^{\pi}_t &= (1-\lambda)(\hat A^{\pi,(1)}_t+\lambda \hat A^{\pi,(2)}_t+\lambda^2\hat A^{\pi,(3)}_t+\cdots) \\
&=\sum_{l=0}^\infty (\gamma \lambda)^l\delta_{t+l}
\end{aligned}
A ^ t π = ( 1 − λ ) ( A ^ t π , ( 1 ) + λ A ^ t π , ( 2 ) + λ 2 A ^ t π , ( 3 ) + ⋯ ) = l = 0 ∑ ∞ ( γ λ ) l δ t + l
上式称作Generalized Advantage Estimation(GAE)
不难发现
A ^ t π = ∑ l = 0 ∞ ( γ λ ) l δ t + l = δ t + ∑ l = 1 ∞ ( γ λ ) l δ t + l = i = l − 1 δ t + ∑ i = 0 ∞ ( γ λ ) i + 1 δ t + i + 1 ⇔ δ t + γ λ A ^ t + 1 π \begin{aligned}
\hat A^{\pi}_{t} &=\sum_{l=0}^\infty (\gamma \lambda)^l\delta_{t+l}\\
&=\delta_t + \sum_{l=1}^\infty (\gamma\lambda)^l \delta_{t+l}\\
&\overset{i=l-1}{=}\delta_t + \sum_{i=0}^\infty (\gamma\lambda)^{i+1}\delta_{t+i+1} \Leftrightarrow \delta_t + \gamma\lambda \hat A^{\pi}_{t+1}
\end{aligned}
A ^ t π = l = 0 ∑ ∞ ( γ λ ) l δ t + l = δ t + l = 1 ∑ ∞ ( γ λ ) l δ t + l = i = l − 1 δ t + i = 0 ∑ ∞ ( γ λ ) i + 1 δ t + i + 1 ⇔ δ t + γ λ A ^ t + 1 π
即GAE也存在recursive的形式。
使用上述的梯度估计方法还存在一个问题，回顾我们找到的梯度估计方法
∇ θ J ( π θ ) = 1 ∣ D ∣ ∑ i = 1 ∣ D ∣ ( ∑ t = 0 T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) A π ( s i , t , a i , t ) ) \nabla_\theta J(\pi_\theta) = \frac{1}{|D|}\sum_{i=1}^{|D|}\left(\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_{i,t}|s_{i,t})A^\pi (s_{i,t},a_{i,t})\right)
∇ θ J ( π θ ) = ∣ D ∣ 1 i = 1 ∑ ∣ D ∣ ( t = 0 ∑ T − 1 ∇ θ log π θ ( a i , t ∣ s i , t ) A π ( s i , t , a i , t ) )
我们通过梯度上升更新参数
θ k + 1 = θ k + α ∇ θ J ( π θ ) ∣ θ k \theta_{k+1} = \theta_k + \alpha \nabla_\theta J(\pi_\theta)|_{\theta_k}
θ k + 1 = θ k + α ∇ θ J ( π θ ) ∣ θ k
这样一来，我们每次更新参数的时候都要去采样trajectories，模型参数一更新，就得重新采样新的trajectories，这样一来是inefficient的。
在这里，我们使用importance sampling来重新写我们的目标函数。对于某个复合函数随机变量的期望，我们可以这样计算
E x ∼ p ( x ) [ f ( x ) ] = ∫ p ( x ) f ( x ) d x = ∫ q ( x ) q ( x ) p ( x ) f ( x ) d x = ∫ q ( x ) p ( x ) q ( x ) f ( x ) d x = E x ∼ q ( x ) [ p ( x ) q ( x ) f ( x ) ] \begin{aligned}
E_{x\sim p(x)}[f(x)] &= \int p(x)f(x)dx \\
&=\int \frac{q(x)}{q(x)}p(x)f(x)dx \\
&=\int q(x)\frac{p(x)}{q(x)}f(x)dx \\
&=E_{x\sim q(x)}\left[\frac{p(x)}{q(x)}f(x)\right]
\end{aligned}
E x ∼ p ( x ) [ f ( x ) ] = ∫ p ( x ) f ( x ) d x = ∫ q ( x ) q ( x ) p ( x ) f ( x ) d x = ∫ q ( x ) q ( x ) p ( x ) f ( x ) d x = E x ∼ q ( x ) [ q ( x ) p ( x ) f ( x ) ]
这样一来，我们对x x x p ( x ) p(x) p ( x ) q ( x ) q(x) q ( x )
对于原始带期望的目标函数，我们可以变换为
∇ θ J ( π θ ) = E τ ∼ π θ [ ∑ t = 0 T − 1 ∇ θ log π θ ( a t ∣ s t ) A π ( s t , a t ) ] = E τ ∼ π offline [ ∏ t = 0 T − 1 π online ( a t ∣ s t ) π offline ( a t ∣ s t ) ( ∑ t = 0 T − 1 ∇ θ log π online ( a t ∣ s t ) A π ( s t , a t ) ) ] where online = θ \begin{aligned}
\nabla_\theta J(\pi_\theta)&=E_{\tau\sim \pi_\theta}\left[\sum_{t=0}^{T-1}\nabla_\theta \log \pi_\theta (a_t|s_t)A^\pi (s_{t},a_{t})\right] \\
&=E_{\tau\sim \pi_{\text{offline}}}\left[\prod_{t=0}^{T-1}\frac{\pi_{\text{online}}(a_t|s_t)}{\pi_{\text{offline}}(a_t|s_t)}\left(\sum_{t=0}^{T-1}\nabla_\theta \log \pi_{\text{online}} (a_t|s_t)A^\pi (s_{t},a_{t})\right)\right] \\
&\text{where}\ \text{online} = \theta
\end{aligned}
∇ θ J ( π θ ) = E τ ∼ π θ [ t = 0 ∑ T − 1 ∇ θ log π θ ( a t ∣ s t ) A π ( s t , a t ) ] = E τ ∼ π offline [ t = 0 ∏ T − 1 π offline ( a t ∣ s t ) π online ( a t ∣ s t ) ( t = 0 ∑ T − 1 ∇ θ log π online ( a t ∣ s t ) A π ( s t , a t ) ) ] where online = θ
注意到我们梯度只针对于online模型，offline是不更新的，因为这个比例类似于加权系数，对每个样本加权就类似于对每个样本赋予一个重要度，所以叫importance sampling。
但在一些文献里面，研究者们直接优化一个更简单的目标函数
∇ θ online = 1 ∣ D ∣ ∑ i = 1 ∣ D ∣ ( ∑ t = 0 T − 1 ∇ θ online log π θ online ( a i , t ∣ s i , t ) log π θ offline ( a i , t ∣ s i , t ) A π ( s i , t , a i , t ) ) \nabla_{\theta_{\text{online}}} = \frac{1}{|D|}\sum_{i=1}^{|D|}\left(\sum_{t=0}^{T-1}\nabla_{\theta_{\text{online}}} \frac{\log \pi_{\theta_{\text{online}}} (a_{i,t}|s_{i,t})}{\log \pi_{\theta_{\text{offline}}} (a_{i,t}|s_{i,t})}A^\pi (s_{i,t},a_{i,t})\right)
∇ θ online = ∣ D ∣ 1 i = 1 ∑ ∣ D ∣ ( t = 0 ∑ T − 1 ∇ θ online log π θ offline ( a i , t ∣ s i , t ) log π θ online ( a i , t ∣ s i , t ) A π ( s i , t , a i , t ) )
笔者注 ： 可在TRPO中找到相关做法，但似乎并没有解释为什么这么做，也没有解释数学上和优化目标上的区别等等。
Schulman, John. “Trust Region Policy Optimization.” arXiv preprint arXiv:1502.05477 (2015). https://arxiv.org/abs/1502.05477
注意到上式中我们采样的D D D
从数学角度讲，原来的期望在θ \theta θ θ \theta θ
在实际操作过程中，我们的θ offline \theta_{\text{offline}} θ offline θ online \theta_{\text{online}} θ online θ online \theta_{\text{online}} θ online θ offline = θ online \theta_{\text{offline}}=\theta_{\text{online}} θ offline = θ online
PPO算法从TRPO开始改进，TRPO优化一个更为简单的目标
max θ E ^ t [ π θ ( a t ∣ s t ) π old ( a t ∣ s t ) A ^ t − β KL [ π old ( ⋅ ∣ s t ) ∣ ∣ π θ ( ⋅ ∣ s t ) ] ] \max_{\theta} \mathbb{\hat E}_t\left[\frac{\pi_{\theta}(a_t|s_t)}{\pi_{\text{old}}(a_t|s_t)}\hat A_t-\beta \text{KL}[\pi_{\text{old}}(\cdot|s_t)||\pi_\theta(\cdot|s_t)]\right]
θ max E ^ t [ π old ( a t ∣ s t ) π θ ( a t ∣ s t ) A ^ t − β KL [ π old ( ⋅ ∣ s t ) ∣ ∣ π θ ( ⋅ ∣ s t ) ] ]
注意到这里加了一个KL散度，保证训练过的模型的分布与之前模型的分布的差距不会偏移太多，这在做RLHF时是很有用的，模型可能因为RLHF的偏好对齐忘记世界知识，或者发生reward hacking，例如我们的reward model教模型输出礼貌的回答，但模型完全只输出"Thank you"，虽然能满足reward model的要求，但是其功能已经丧失。
但是，作者claim仅仅使用一个固定的β \beta β
令r t ( θ ) = π θ ( a t ∣ s t ) π θ old ( a t ∣ s t ) r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)} r t ( θ ) = π θ old ( a t ∣ s t ) π θ ( a t ∣ s t ) r t ( θ ) r_t(\theta) r t ( θ )
L CLIP ( θ ) = E ^ t [ min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) ] L^{\text{CLIP}}(\theta) = \mathbb{\hat E}_t[\min(r_t(\theta)\hat A_t,\text{clip}(r_t(\theta),1-\epsilon, 1+\epsilon)\hat A_t)]
L CLIP ( θ ) = E ^ t [ min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) ]
当A ^ t > 0 \hat A_t>0 A ^ t > 0 r t ( θ ) > 1 + ϵ r_t(\theta) > 1+\epsilon r t ( θ ) > 1 + ϵ
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 + ϵ ) A ^ t ) = ( 1 + ϵ ) A ^ t \begin{aligned}
\min(r_t(\theta)\hat A_t,\text{clip}(r_t(\theta),1-\epsilon, 1+\epsilon)\hat A_t) &= \min(r_t(\theta)\hat A_t, (1+\epsilon)\hat A_t) \\
&=(1+\epsilon)\hat A_t
\end{aligned}
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 + ϵ ) A ^ t ) = ( 1 + ϵ ) A ^ t
当A ^ t > 0 \hat A_t>0 A ^ t > 0 r t ( θ ) < 1 − ϵ r_t(\theta) < 1-\epsilon r t ( θ ) < 1 − ϵ
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 − ϵ ) A ^ t ) = r t ( θ ) A ^ t \begin{aligned}
\min(r_t(\theta)\hat A_t,\text{clip}(r_t(\theta),1-\epsilon, 1+\epsilon)\hat A_t) &= \min(r_t(\theta)\hat A_t, (1-\epsilon)\hat A_t) \\
&=r_t(\theta)\hat A_t
\end{aligned}
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 − ϵ ) A ^ t ) = r t ( θ ) A ^ t
即A ^ t > 0 \hat A_t>0 A ^ t > 0
当A ^ t < 0 \hat A_t <0 A ^ t < 0 r t ( θ ) > 1 + ϵ r_t(\theta) > 1+\epsilon r t ( θ ) > 1 + ϵ
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 + ϵ ) A ^ t ) = r t ( θ ) A ^ t \begin{aligned}
\min(r_t(\theta)\hat A_t,\text{clip}(r_t(\theta),1-\epsilon, 1+\epsilon)\hat A_t) &= \min(r_t(\theta)\hat A_t, (1+\epsilon)\hat A_t) \\
&=r_t(\theta)\hat A_t
\end{aligned}
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 + ϵ ) A ^ t ) = r t ( θ ) A ^ t
当A ^ t < 0 \hat A_t<0 A ^ t < 0 r t ( θ ) < 1 − ϵ r_t(\theta) < 1-\epsilon r t ( θ ) < 1 − ϵ
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 − ϵ ) A ^ t ) = ( 1 − ϵ ) A ^ t \begin{aligned}
\min(r_t(\theta)\hat A_t,\text{clip}(r_t(\theta),1-\epsilon, 1+\epsilon)\hat A_t) &= \min(r_t(\theta)\hat A_t, (1-\epsilon)\hat A_t) \\
&=(1-\epsilon)\hat A_t
\end{aligned}
min ( r t ( θ ) A ^ t , clip ( r t ( θ ) , 1 − ϵ , 1 + ϵ ) A ^ t ) = min ( r t ( θ ) A ^ t , ( 1 − ϵ ) A ^ t ) = ( 1 − ϵ ) A ^ t
即当A ^ t < 0 \hat A_t<0 A ^ t < 0
当r t ( θ ) r_t(\theta) r t ( θ ) [ 1 − ϵ , 1 + ϵ ] [1-\epsilon,1+\epsilon] [ 1 − ϵ , 1 + ϵ ]
PPO 的完整目标函数如下
L t CLIP + VF + S ( θ ) = E ^ t [ L t CLIP ( θ ) − c 1 L t VF ( θ ) + c 2 S [ π θ ] ( s t ) ] L_t^{\text{CLIP}+\text{VF}+\text{S}}(\theta)=\mathbb{\hat E}_t [L_t^{\text{CLIP}}(\theta)-c_1L_t^{\text{VF}}(\theta)+c_2 S[\pi_\theta](s_t)]
L t CLIP + VF + S ( θ ) = E ^ t [ L t CLIP ( θ ) − c 1 L t VF ( θ ) + c 2 S [ π θ ] ( s t ) ]
我们首先在最大化第一项，即L t CLIP ( θ ) L_t^{\text{CLIP}}(\theta) L t CLIP ( θ )
L t VF ( θ ) = ∥ V θ ( s ) − ∑ t = 0 T − 1 ( γ t r t ∣ s 0 = s ) ∥ 2 2 L_t^{\text{VF}}(\theta)=\left\|V_\theta(s)-\sum_{t=0}^{T-1}(\gamma^t r_t|s_0=s) \right\|_2^2
L t VF ( θ ) = ∥ ∥ ∥ ∥ ∥ ∥ V θ ( s ) − t = 0 ∑ T − 1 ( γ t r t ∣ s 0 = s ) ∥ ∥ ∥ ∥ ∥ ∥ 2 2
即使用回归目标让value模型学习我们估计出来的value。系数c 1 c_1 c 1 1 / 2 1/2 1 / 2
第三项是Entropy loss，即直接计算概率分布的entropy。
S [ π θ ] ( s t ) = − ∑ x π θ ( x ∣ s t ) log π θ ( x ∣ s t ) S[\pi_\theta](s_t) = -\sum_x \pi_\theta(x|s_t)\log \pi_\theta (x|s_t)
S [ π θ ] ( s t ) = − x ∑ π θ ( x ∣ s t ) log π θ ( x ∣ s t )
Entropy的最大值在概率分布等概的时候取到，而这一项在优化过程中是被放大的，所以我们是在约束policy对于所有选择也尽可能的均匀。这虽然违背我们学习的目标----学到一个能根据state做出有效判断的policy，但增大entropy也可以鼓励policy去探索不同的可能性，在学习目标的同时保证一定的多样性。