【机器学习基础】 -EM算法

灰太狼 2023-02-17 02:44 8阅读 0赞

原文链接:[https://blog.csdn.net/u010834867/article/details/90762296][https_blog.csdn.net_u010834867_article_details_90762296]

**目录**

一 样例

二 公式描述

三 参考文献

--------------------

**          最大期望算法（Expectation-maximization algorithm****，又译为期望最大化算法）**，是在概率模型中寻找参数最大似然估计或者最大后验估计的算法，其中概率模型依赖于无法观测的隐性变量。

最大期望算法经过两个步骤交替进行计算：

**第一步**是计算期望（E），利用对隐藏变量的现有估计值，计算其最大似然估计值；

**第二步**是最大化（M），最大化在E步上求得的最大似然值来计算参数的值。M步上找到的参数估计值被用于下一个E步计算中，这个过程不断交替进行。

# 一 样例       #

举个例子，抛硬币，有两个硬币，但是两个硬币的材质不同导致其出现正反面的概率不一样，目前我们只有一组观测数据，要求出每一种硬币投掷时正面向上的概率。总共投了五轮，每轮投掷五次，现在先考虑一种简单的情况，假设我们知道这每一轮用的是哪一个硬币去投掷的：

![2019060408222040.png][]

那么我们拿着这样的一组数据，就可以很轻松的估计出A硬币和B硬币出现正面的概率，如下：

PA = （3+1+2）/ 15 = 0.4

PB= （2+3）/10 = 0.5

现在把问题变得复杂一点，假设我们不知道每一次投掷用的是哪一种硬币，等于是现在的问题加上了一个隐变量，就是每一次选取的硬币的种类。

![20190604082251356.png][]

那么现在可以想一想，假设我们把每一次硬币的种类设为z,则这五次实验生成了一个5维的向量(z1,z2,z3,z4,z5)，现在问题来了，如果我们要根据观测结果去求出PA,PB，那么首先需要知道z，但是如果用最大似然估计去估计z，又要先求出PA,PB。这就产生了一个循环。那么这个时候EM算法的作用就体现出来了，EM算法的基本思想是：先初始化一个PA,PB，然后我们拿着这个初始化的PA,PB用最大似然概率估计出z，接下来有了z之后就用z去计算出在当前z的情况下的PA,PB是多少，然后不断地重复这两个步骤直到收敛。

有了这个思想之后现在用这个思想来做一下这个例子，假设初始状态下PA=0.2, PB=0.7，然后我们根据这个概率去估计出z：

![20190604082322112.png][]

按照最大似然估计，z=(B,A,A,B,A)，有了z之后我们反过来重新估计一下PA,PB：

PA = （2+1+2）/15 = 0.33

PB =（3+3）/10 = 0.6

可以看到PA,PB的值已经更新了，假设PA,PB的真实值0.4和0.5，那么你在不断地重复这两步你就会发现PA,PB在不断地靠近这两个真实值。

# 二 公式描述 #

假设目标函数表示为：

![L(\\theta)=\\prod\_\{j=1\}^\{N\}P\_\\theta(y\_i)][L_theta_prod_j_1_N_P_theta_y_i]

其中![\\theta][theta]为概率分布![P\_\\theta(y)][P_theta_y]的参数，在没有隐变量的情况下，我们求解L的最大值的套路是先对L取对数，将连乘的形式转换成累加的形式，然后就可以对未知数![\\theta][theta]进行求导，只需要求得导数为0的位置未知量的值即为目标函数的极大值。但是，如果概率分布中有隐变量存在时，我们用全概率公式把隐变量在上式中体现出来：

> 全概率公式：

![P(B)=\\sum\_\{i=1\}^\{n\}P(A\_i)P(B|A\_i)][P_B_sum_i_1_n_P_A_i_P_B_A_i]

其中A需是一组完备事件组且都有正概率，则对任意B上式都成立。

![L(\\theta)=\\prod\_\{j=1\}^\{N\}\\sum\_\{z\}P\_\\theta(z)P\_\\theta(y\_i|z)][L_theta_prod_j_1_N_sum_z_P_theta_z_P_theta_y_i_z]

两边同时取对数ln:

![lnL(\\theta)=\\sum\_\{j=1\}^\{N\}ln\\sum\_\{z\}P\_\\theta(z)P\_\\theta(y\_i|z)][lnL_theta_sum_j_1_N_ln_sum_z_P_theta_z_P_theta_y_i_z]

从上式中可以看出，如果要按照之前的套路，那就需要对上式进行求偏导，但是由于ln中还包含了求和项，在偏导数中的形式将会非常复杂，而且很难求得解析解。因此需要找到一种办法去求得解析解的近似解，这里就引入了EM算法。为了表述的方便性，在这里用![l(\\theta)][l_theta]代表![lnL(\\theta)][lnL_theta]，则：

![l(\\theta)=\\sum\_\{j=1\}^\{N\}ln\\sum\_\{z\}P\_\\theta(z)P\_\\theta(y\_i|z)][l_theta_sum_j_1_N_ln_sum_z_P_theta_z_P_theta_y_i_z]

对于求解近似解，EM算法采用的是迭代的方式不断地逼近真实值，假设第n次迭代的参数值为![\\theta\_n][theta_n]，第n+1次迭代的参数值为![\\theta\_\{n+1\}][theta_n_1]，那么其实只要满足![l(\\theta\_\{n+1\})>l(\\theta\_n)][l_theta_n_1_l_theta_n]就可以不断地进行迭代。现在假设我们已经进行到了第n次迭代，也就是说![\\theta\_n][theta_n]目前是作为已知的值，那么来看一看![l(\\theta)-l(\\theta\_n)][l_theta_-l_theta_n]

![l(\\theta)-l(\\theta\_n)=\\sum\_\{j=1\}^\{N\}ln\\sum\_\{Z\}P\_\\theta(y\_j|z)P\_\\theta(z)-\\sum\_\{j=1\}^\{N\}ln\\sum\_\{Z\}P\_\{\\theta\_n\}(y\_j|z)P\_\{\\theta\_n\}(z)][l_theta_-l_theta_n_sum_j_1_N_ln_sum_Z_P_theta_y_j_z_P_theta_z_-_sum_j_1_N_ln_sum_Z_P_theta_n_y_j_z_P_theta_n_z]

由于这里![\\theta\_n][theta_n]是作为常数的，因此右式中的第二项可以不用将隐变量体现出，化简如下：

![=\\sum\_\{j=1\}^\{N\}ln\\sum\_\{Z\}P\_\\theta(y\_j|z)P\_\\theta(z)-\\sum\_\{j=1\}^\{N\}lnP\_\{\\theta\_n\}(y\_j)][sum_j_1_N_ln_sum_Z_P_theta_y_j_z_P_theta_z_-_sum_j_1_N_lnP_theta_n_y_j]

![=\\sum\_\{j=1\}^\{N\}\\left \[ ln\\sum\_\{Z\}P\_\\theta(y\_j|z)P\_\\theta(z)-lnP\_\{\\theta\_n\}(y\_j)\\right \]][sum_j_1_N_left _ ln_sum_Z_P_theta_y_j_z_P_theta_z_-lnP_theta_n_y_j_right]

上式中，其实只需要关注求和里面的项就好了，做一个标注，记![ln\\sum\_\{Z\}P\_\\theta(y\_j|z)P\_\\theta(z)][ln_sum_Z_P_theta_y_j_z_P_theta_z]为I项，记为![lnP\_\{\\theta\_n\}(y\_j)][lnP_theta_n_y_j]II项，接下来分别对这两项进行化简：

**I项：**

![ln\\sum\_\{Z\}P\_\\theta(y\_j|z)P\_\\theta(z)=ln\\sum\_\{Z\}P\_\\theta(y\_j|z)\\frac\{P\_\{\\theta\_n\}(z|y\_j)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}][ln_sum_Z_P_theta_y_j_z_P_theta_z_ln_sum_Z_P_theta_y_j_z_frac_P_theta_n_z_y_j_P_theta_z_P_theta_n_z_y_j]

> 上式中的变换是因为![\\sum\_\{Z\}P\_\{\\theta\_n\}(z|y)=1][sum_Z_P_theta_n_z_y_1]

接下来还会用到Jensen不等式，Jensen不等式是这么描述的：

假设f(x)为凸函数，X是随机变量，那么：E\[f(X)\]>=f(E\[X\])，通俗的说法是函数的期望大于等于期望的函数。

特别地，如果f是严格凸函数，当且仅当 ![P\\left( X=EX \\right) =1][P_left_ X_EX _right_ _1] ，即 ![X][] 是常量时，上式取等号，即![E\\left\[ f\\left( X \\right) \\right\] =f\\left( EX \\right)][E_left_ f_left_ X _right_ _right_ _f_left_ EX _right]，如下图所示：

![clip\_image019][clip_image019]

再观察一下I项，感觉有一些像能够用Jensen不等式进行再一次的化简，下面写的直观一点：

![=ln\\sum\_\{Z\}P\_\\theta(y\_j|z)\\frac\{P\_\{\\theta\_n\}(z|y\_j)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}=ln\\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j)\\frac\{P\_\\theta(y\_j|z)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}][ln_sum_Z_P_theta_y_j_z_frac_P_theta_n_z_y_j_P_theta_z_P_theta_n_z_y_j_ln_sum_Z_P_theta_n_z_y_j_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j]

把![P\_\{\\theta\_n\}(y\_j)][P_theta_n_y_j]当成X，那么上式![\\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j)\\frac\{P\_\\theta(y\_j|z)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}][sum_Z_P_theta_n_z_y_j_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j]就是EX，又因为ln函数是凹函数，所以根据Jensen不等式有f(E\[X\])>=E\[f(X)\]，则

![ln\\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j)\\frac\{P\_\\theta(y\_j|z)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}\\geqslant \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j)ln\\frac\{P\_\{\\theta\}(y\_j|z)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}][ln_sum_Z_P_theta_n_z_y_j_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_geqslant _sum_Z_P_theta_n_z_y_j_ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j]

**II项：**

![lnP\_\{\\theta\_n\}(y\_j)=\\left \[ lnP\_\{\\theta\_n\}(y\_j) \\right \]\\cdot \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j)][lnP_theta_n_y_j_left _ lnP_theta_n_y_j_ _right _cdot _sum_Z_P_theta_n_z_y_j]

![= \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) lnP\_\{\\theta\_n\}(y\_j)][_sum_Z_P_theta_n_z_y_j_ lnP_theta_n_y_j]

然后再把I-II合并起来看一下：

![I-II\\geqslant \\sum\_\{j=1\}^\{N\}\\left \[ \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j)ln\\frac\{P\_\{\\theta\}(y\_j|z)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}-\\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) lnP\_\{\\theta\_n\}(y\_j) \\right\]][I-II_geqslant _sum_j_1_N_left _ _sum_Z_P_theta_n_z_y_j_ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_-_sum_Z_P_theta_n_z_y_j_ lnP_theta_n_y_j_ _right]

![=\\sum\_\{j=1\}^\{N\} \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) \\left\[ ln\\frac\{P\_\{\\theta\}(y\_j|z)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)\}-lnP\_\{\\theta\_n\}(y\_j) \\right \]][sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_-lnP_theta_n_y_j_ _right]

![=\\sum\_\{j=1\}^\{N\} \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) \\left\[ ln\\frac\{P\_\{\\theta\}(y\_j|z)P\_\\theta(z)\}\{P\_\{\\theta\_n\}(z|y\_j)P\_\{\\theta\_n\}(y\_j)\} \\right \]][sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_P_theta_n_y_j_ _right]

![=\\sum\_\{j=1\}^\{N\} \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) \\left\[ ln\\frac\{P\_\{\\theta\}(y\_j,z)\}\{P\_\{\\theta\_n\}(z,y\_j)\} \\right \]][sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right]

将上式代回![l(\\theta)-l(\\theta\_n)][l_theta_-l_theta_n]，

![l(\\theta)-l(\\theta\_n)\\geqslant \\sum\_\{j=1\}^\{N\} \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) \\left\[ ln\\frac\{P\_\{\\theta\}(y\_j,z)\}\{P\_\{\\theta\_n\}(z,y\_j)\} \\right \]][l_theta_-l_theta_n_geqslant _sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right]

![l(\\theta)\\geqslant \\sum\_\{j=1\}^\{N\}\\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) \\left\[ ln\\frac\{P\_\{\\theta\}(y\_j,z)\}\{P\_\{\\theta\_n\}(z,y\_j)\} \\right \]+l(\\theta\_n)][l_theta_geqslant _sum_j_1_N_sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right _l_theta_n]

将右边的这一大串记为![Q(\\theta|\\theta\_n)][Q_theta_theta_n]，称为**下边界函数**，EM算法的目的是要取得目标函数的极大值，那么可以通过不断地提升下边界函数值来不断地提升目标函数的值，接下来，再看一下![Q(\\theta|\\theta\_n)][Q_theta_theta_n]，将其化简为便于优化迭代的形式：

![Q(\\theta|\\theta\_n)=l(\\theta\_n)+ \\sum\_\{j=1\}^\{N\} \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) \\left\[ ln\\frac\{P\_\{\\theta\}(y\_j,z)\}\{P\_\{\\theta\_n\}(z,y\_j)\} \\right \]][Q_theta_theta_n_l_theta_n_ _sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right]

可以看到在等式的右边，由于我们之前的假设是![\\theta\_n][theta_n]是已知的，那么把已知量和未知量分开：

![= \\sum\_\{j=1\}^\{N\} \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) lnP\_\{\\theta\}(y\_j,z)+l(\\theta\_n) -\\sum\_\{j=1\}^\{N\} \\sum\_\{Z\}P\_\{\\theta\_n\}(z|y\_j) lnP\_\{\\theta\_n\}(z,y\_j)][_sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ lnP_theta_y_j_z_l_theta_n_ -_sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ lnP_theta_n_z_y_j]

上式中，等号右边的第一项中带未知项，第二项和第三项都是常数，所以接下来的过程就简单了，我们只要对这个式子求偏导![\\frac\{\\partial Q(\\theta|\\theta\_n)\}\{\\partial \\theta\}=0][frac_partial Q_theta_theta_n_partial _theta_0]，求得此时取极大值时![\\theta][theta]的取值，这个值就是进入到下一步迭代是的概率分布参数值![\\theta\_\{n+1\}][theta_n_1]，有了![\\theta\_\{n+1\}][theta_n_1]之后就可以获得![Q(\\theta|\\theta\_\{n+1\})][Q_theta_theta_n_1]，然后不断地迭代直到收敛。图示的话如下：

![1359004484_7944.jpg][]

# 三 参考文献 #

1. [如何通俗理解EM算法][EM]

[https_blog.csdn.net_u010834867_article_details_90762296]: https://blog.csdn.net/u010834867/article/details/90762296
[2019060408222040.png]: https://img-blog.csdnimg.cn/2019060408222040.png
[20190604082251356.png]: https://img-blog.csdnimg.cn/20190604082251356.png
[20190604082322112.png]: https://img-blog.csdnimg.cn/20190604082322112.png
[L_theta_prod_j_1_N_P_theta_y_i]: https://private.codecogs.com/gif.latex?L%28%5Ctheta%29%3D%5Cprod_%7Bj%3D1%7D%5E%7BN%7DP_%5Ctheta%28y_i%29
[theta]: https://private.codecogs.com/gif.latex?%5Ctheta
[P_theta_y]: https://private.codecogs.com/gif.latex?P_%5Ctheta%28y%29
[P_B_sum_i_1_n_P_A_i_P_B_A_i]: https://private.codecogs.com/gif.latex?P%28B%29%3D%5Csum_%7Bi%3D1%7D%5E%7Bn%7DP%28A_i%29P%28B%7CA_i%29
[L_theta_prod_j_1_N_sum_z_P_theta_z_P_theta_y_i_z]: https://private.codecogs.com/gif.latex?L%28%5Ctheta%29%3D%5Cprod_%7Bj%3D1%7D%5E%7BN%7D%5Csum_%7Bz%7DP_%5Ctheta%28z%29P_%5Ctheta%28y_i%7Cz%29
[lnL_theta_sum_j_1_N_ln_sum_z_P_theta_z_P_theta_y_i_z]: https://private.codecogs.com/gif.latex?lnL%28%5Ctheta%29%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7Dln%5Csum_%7Bz%7DP_%5Ctheta%28z%29P_%5Ctheta%28y_i%7Cz%29
[l_theta]: https://private.codecogs.com/gif.latex?l%28%5Ctheta%29
[lnL_theta]: https://private.codecogs.com/gif.latex?lnL%28%5Ctheta%29
[l_theta_sum_j_1_N_ln_sum_z_P_theta_z_P_theta_y_i_z]: https://private.codecogs.com/gif.latex?l%28%5Ctheta%29%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7Dln%5Csum_%7Bz%7DP_%5Ctheta%28z%29P_%5Ctheta%28y_i%7Cz%29
[theta_n]: https://private.codecogs.com/gif.latex?%5Ctheta_n
[theta_n_1]: https://private.codecogs.com/gif.latex?%5Ctheta_%7Bn&plus;1%7D
[l_theta_n_1_l_theta_n]: https://private.codecogs.com/gif.latex?l%28%5Ctheta_%7Bn&plus;1%7D%29%3El%28%5Ctheta_n%29
[l_theta_-l_theta_n]: https://private.codecogs.com/gif.latex?l%28%5Ctheta%29-l%28%5Ctheta_n%29
[l_theta_-l_theta_n_sum_j_1_N_ln_sum_Z_P_theta_y_j_z_P_theta_z_-_sum_j_1_N_ln_sum_Z_P_theta_n_y_j_z_P_theta_n_z]: https://private.codecogs.com/gif.latex?l%28%5Ctheta%29-l%28%5Ctheta_n%29%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7Dln%5Csum_%7BZ%7DP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29-%5Csum_%7Bj%3D1%7D%5E%7BN%7Dln%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28y_j%7Cz%29P_%7B%5Ctheta_n%7D%28z%29
[sum_j_1_N_ln_sum_Z_P_theta_y_j_z_P_theta_z_-_sum_j_1_N_lnP_theta_n_y_j]: https://private.codecogs.com/gif.latex?%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7Dln%5Csum_%7BZ%7DP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29-%5Csum_%7Bj%3D1%7D%5E%7BN%7DlnP_%7B%5Ctheta_n%7D%28y_j%29
[sum_j_1_N_left _ ln_sum_Z_P_theta_y_j_z_P_theta_z_-lnP_theta_n_y_j_right]: https://private.codecogs.com/gif.latex?%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7D%5Cleft%20%5B%20ln%5Csum_%7BZ%7DP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29-lnP_%7B%5Ctheta_n%7D%28y_j%29%5Cright%20%5D
[ln_sum_Z_P_theta_y_j_z_P_theta_z]: https://private.codecogs.com/gif.latex?ln%5Csum_%7BZ%7DP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29
[lnP_theta_n_y_j]: https://private.codecogs.com/gif.latex?lnP_%7B%5Ctheta_n%7D%28y_j%29
[ln_sum_Z_P_theta_y_j_z_P_theta_z_ln_sum_Z_P_theta_y_j_z_frac_P_theta_n_z_y_j_P_theta_z_P_theta_n_z_y_j]: https://private.codecogs.com/gif.latex?ln%5Csum_%7BZ%7DP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29%3Dln%5Csum_%7BZ%7DP_%5Ctheta%28y_j%7Cz%29%5Cfrac%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D
[sum_Z_P_theta_n_z_y_1]: https://private.codecogs.com/gif.latex?%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy%29%3D1
[P_left_ X_EX _right_ _1]: https://www.zhihu.com/equation?tex=+P%5Cleft%28+X%3DEX+%5Cright%29+%3D1
[X]: https://www.zhihu.com/equation?tex=+X+
[E_left_ f_left_ X _right_ _right_ _f_left_ EX _right]: https://www.zhihu.com/equation?tex=+E%5Cleft%5B+f%5Cleft%28+X+%5Cright%29+%5Cright%5D+%3Df%5Cleft%28+EX+%5Cright%29+
[clip_image019]: https://images.cnblogs.com/cnblogs_com/jerrylead/201104/201104061615564400.png
[ln_sum_Z_P_theta_y_j_z_frac_P_theta_n_z_y_j_P_theta_z_P_theta_n_z_y_j_ln_sum_Z_P_theta_n_z_y_j_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j]: https://private.codecogs.com/gif.latex?%3Dln%5Csum_%7BZ%7DP_%5Ctheta%28y_j%7Cz%29%5Cfrac%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D%3Dln%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%5Cfrac%7BP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D
[P_theta_n_y_j]: https://private.codecogs.com/gif.latex?P_%7B%5Ctheta_n%7D%28y_j%29
[sum_Z_P_theta_n_z_y_j_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j]: https://private.codecogs.com/gif.latex?%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%5Cfrac%7BP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D
[ln_sum_Z_P_theta_n_z_y_j_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_geqslant _sum_Z_P_theta_n_z_y_j_ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j]: https://private.codecogs.com/gif.latex?ln%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%5Cfrac%7BP_%5Ctheta%28y_j%7Cz%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D%5Cgeqslant%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%7Cz%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D
[lnP_theta_n_y_j_left _ lnP_theta_n_y_j_ _right _cdot _sum_Z_P_theta_n_z_y_j]: https://private.codecogs.com/gif.latex?lnP_%7B%5Ctheta_n%7D%28y_j%29%3D%5Cleft%20%5B%20lnP_%7B%5Ctheta_n%7D%28y_j%29%20%5Cright%20%5D%5Ccdot%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29
[_sum_Z_P_theta_n_z_y_j_ lnP_theta_n_y_j]: https://private.codecogs.com/gif.latex?%3D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20lnP_%7B%5Ctheta_n%7D%28y_j%29
[I-II_geqslant _sum_j_1_N_left _ _sum_Z_P_theta_n_z_y_j_ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_-_sum_Z_P_theta_n_z_y_j_ lnP_theta_n_y_j_ _right]: https://private.codecogs.com/gif.latex?I-II%5Cgeqslant%20%5Csum_%7Bj%3D1%7D%5E%7BN%7D%5Cleft%20%5B%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%7Cz%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D-%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20lnP_%7B%5Ctheta_n%7D%28y_j%29%20%5Cright%5D
[sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_-lnP_theta_n_y_j_ _right]: https://private.codecogs.com/gif.latex?%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20%5Cleft%5B%20ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%7Cz%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%7D-lnP_%7B%5Ctheta_n%7D%28y_j%29%20%5Cright%20%5D
[sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_z_P_theta_n_z_y_j_P_theta_n_y_j_ _right]: https://private.codecogs.com/gif.latex?%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20%5Cleft%5B%20ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%7Cz%29P_%5Ctheta%28z%29%7D%7BP_%7B%5Ctheta_n%7D%28z%7Cy_j%29P_%7B%5Ctheta_n%7D%28y_j%29%7D%20%5Cright%20%5D
[sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right]: https://private.codecogs.com/gif.latex?%3D%5Csum_%7Bj%3D1%7D%5E%7BN%7D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20%5Cleft%5B%20ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%2Cz%29%7D%7BP_%7B%5Ctheta_n%7D%28z%2Cy_j%29%7D%20%5Cright%20%5D
[l_theta_-l_theta_n_geqslant _sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right]: https://private.codecogs.com/gif.latex?l%28%5Ctheta%29-l%28%5Ctheta_n%29%5Cgeqslant%20%5Csum_%7Bj%3D1%7D%5E%7BN%7D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20%5Cleft%5B%20ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%2Cz%29%7D%7BP_%7B%5Ctheta_n%7D%28z%2Cy_j%29%7D%20%5Cright%20%5D
[l_theta_geqslant _sum_j_1_N_sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right _l_theta_n]: https://private.codecogs.com/gif.latex?l%28%5Ctheta%29%5Cgeqslant%20%5Csum_%7Bj%3D1%7D%5E%7BN%7D%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20%5Cleft%5B%20ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%2Cz%29%7D%7BP_%7B%5Ctheta_n%7D%28z%2Cy_j%29%7D%20%5Cright%20%5D&plus;l%28%5Ctheta_n%29
[Q_theta_theta_n]: https://private.codecogs.com/gif.latex?Q%28%5Ctheta%7C%5Ctheta_n%29
[Q_theta_theta_n_l_theta_n_ _sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ _left_ ln_frac_P_theta_y_j_z_P_theta_n_z_y_j_ _right]: https://private.codecogs.com/gif.latex?Q%28%5Ctheta%7C%5Ctheta_n%29%3Dl%28%5Ctheta_n%29&plus;%20%5Csum_%7Bj%3D1%7D%5E%7BN%7D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20%5Cleft%5B%20ln%5Cfrac%7BP_%7B%5Ctheta%7D%28y_j%2Cz%29%7D%7BP_%7B%5Ctheta_n%7D%28z%2Cy_j%29%7D%20%5Cright%20%5D
[_sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ lnP_theta_y_j_z_l_theta_n_ -_sum_j_1_N_ _sum_Z_P_theta_n_z_y_j_ lnP_theta_n_z_y_j]: https://private.codecogs.com/gif.latex?%3D%20%5Csum_%7Bj%3D1%7D%5E%7BN%7D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20lnP_%7B%5Ctheta%7D%28y_j%2Cz%29&plus;l%28%5Ctheta_n%29%20-%5Csum_%7Bj%3D1%7D%5E%7BN%7D%20%5Csum_%7BZ%7DP_%7B%5Ctheta_n%7D%28z%7Cy_j%29%20lnP_%7B%5Ctheta_n%7D%28z%2Cy_j%29
[frac_partial Q_theta_theta_n_partial _theta_0]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%5Cpartial%20Q%28%5Ctheta%7C%5Ctheta_n%29%7D%7B%5Cpartial%20%5Ctheta%7D%3D0
[Q_theta_theta_n_1]: https://private.codecogs.com/gif.latex?Q%28%5Ctheta%7C%5Ctheta_%7Bn&plus;1%7D%29
[1359004484_7944.jpg]: https://img-my.csdn.net/uploads/201301/24/1359004484_7944.jpg
[EM]: https://blog.csdn.net/v_july_v/article/details/81708386