机器学习笔记线性判别分析（LDA）（Linear Discriminate Analysis）

偏执的太偏执、 2022-03-22 09:23 366阅读 0赞

上篇博客分析了PCA（[https://blog.csdn.net/qq\_38366615/article/details/86663634][https_blog.csdn.net_qq_38366615_article_details_86663634]），这次对LDA做一下分析。

与LDA与PCA一样，都是对数据进行绛维，但是，考虑的方向不一样，PCA主要是寻找数据的主要特征，然后将数据转换到这些特征上，而LDA则将数据的标签类型考虑进来，要求数据绛维后仍然可分。从这个层面上来说，PCA是一种无监督学习而LDA是一种有监督学习。LDA主要解决的问题是找到一个投影方向，使得数据投影到该方向后数据尽可能分开。一个最直观的感受是当不同类别的数据距离较远时，分类效果更好，在数学上我们使用数据的中心距离来代表整个数据的距离，这里我们首先从二分类问题入手。假设二分类问题，两类数据的中心分别为：![\{\\mu \_1\}\{\\rm\{ = \}\}\\sum\\limits\_\{s \\in \{C\_1\}\} s][mu _1_rm_ _ _sum_limits_s _in _C_1_ s]，![\{\\mu \_2\}\{\\rm\{ = \}\}\\sum\\limits\_\{s \\in \{C\_2\}\} s][mu _2_rm_ _ _sum_limits_s _in _C_2_ s]。则可以得到两类中心投影后的距离为：

![L = \\left\\| \{\{w^T\}\{\\mu \_1\} - \{w^T\}\{\\mu \_2\}\} \\right\\|][L _ _left_ _w_T_mu _1_ - _w_T_mu _2_ _right]

这样我们就可以将LDA要解决的问题转化为带约束的优化问题：

![\\left\\\{ \{\\begin\{array\}\{\*\{20\}\{c\}\} \{\\mathop \{\\max \}\\limits\_w \\left\\| \{\{w^T\}(\{\\mu \_1\} - \{\\mu \_2\})\} \\right\\|\}\\\\ \{s.t.\{w^T\}w = 1\} \\end\{array\}\} \\right.][left_ _begin_array_20_c_ _mathop _max _limits_w _left_ _w_T_mu _1_ - _mu _2_ _right_ _s.t._w_T_w _ 1_ _end_array_ _right.]

可以看出当w方向与![\{\\mu \_1\} - \{\\mu \_2\}][mu _1_ - _mu _2]一致时能达到最大值。但是由于我们使用的是数据中心的距离，不同类别数据中心的数据较远不能代表所有数据都距离较远，数据中心是一个整体的概念，可能有一些不同类别的数据投影后还是会重叠，所以我们需要在中心距离远的基础上让各个类别的数据距离中心的距离更近，这样的话不同类别的数据重叠的概率就会比较少。所以，LDA的问题转化为最大化各个类别的中心距离同时最小化每个类别数据距离该类别中心距离。所以优化的问题为：

![L(w) = \\frac\{\{\{\{\\left\\| \{\{w^T\}(\{\\mu \_1\} - \{\\mu \_2\})\} \\right\\|\}^2\}\}\}\{\{\{D\_1\} + \{D\_2\}\}\}][L_w_ _ _frac_left_ _w_T_mu _1_ - _mu _2_ _right_2_D_1_ _ _D_2]

![\{D\_1\},\{D\_2\}][D_1_D_2]分别表示为两类中的数据距离中心的和

![\{D\_1\} = \\sum\\limits\_\{x \\in \{C\_1\}\} \{(\{w^T\}\} (x - \{\\mu \_1\})\{)^2\} = \\sum\\limits\_\{x \\in \{C\_1\}\} \{\{w^T\}\} (x - \{\\mu \_1\})\{(x - \{\\mu \_1\})^T\}w][D_1_ _ _sum_limits_x _in _C_1_ _w_T_ _x - _mu _1_2_ _ _sum_limits_x _in _C_1_ _w_T_ _x - _mu _1_x - _mu _1_T_w]

同理可以得到：

![\{D\_2\} = \\sum\\limits\_\{x \\in \{C\_2\}\} \{\{w^T\}\} (x - \{\\mu \_2\})\{(x - \{\\mu \_2\})^T\}w][D_2_ _ _sum_limits_x _in _C_2_ _w_T_ _x - _mu _2_x - _mu _2_T_w]

于是问题变为：

![L(w) = \\frac\{\{\{w^T\}(\{\\mu \_1\} - \{\\mu \_2\})(\{\\mu \_1\} - \{\\mu \_2\})^\{T\}w\}\}\{\{\\sum\\limits\_\{x \\in \{C\_i\}\} \{\{w^T\}(x - \{\\mu \_i\})(x - \{\\mu \_i\})^\{T\}w\} \}\}][L_w_ _ _frac_w_T_mu _1_ - _mu _2_mu _1_ - _mu _2_T_w_sum_limits_x _in _C_i_ _w_T_x - _mu _i_x - _mu _i_T_w_]

为了便于计算，令![A = (\{\\mu \_1\} - \{\\mu \_2\})\{(\{\\mu \_1\} - \{\\mu \_2\})^T\}][A _ _mu _1_ - _mu _2_mu _1_ - _mu _2_T]表示类别之间的散度，![B = \\sum\\limits\_\{x \\in \{C\_i\}\} \{(x - \{\\mu \_i\}\} )\{(x - \{\\mu \_i\})^T\}][B _ _sum_limits_x _in _C_i_ _x - _mu _i_ _x - _mu _i_T]表示类内之间的散度。用优化目标对w求导得：

![\\frac\{\{\\partial L(w)\}\}\{\{\\partial w\}\} = \\frac\{\{\\frac\{\{\\partial \{w^T\}Aw\}\}\{\{\\partial w\}\}\{w^T\}Bw - \\frac\{\{\\partial \{w^T\}Bw\}\}\{\{\\partial w\}\}\{w^T\}Aw\}\}\{\{\{\{(\{w^T\}Bw)\}^2\}\}\}][frac_partial L_w_partial w_ _ _frac_frac_partial _w_T_Aw_partial w_w_T_Bw - _frac_partial _w_T_Bw_partial w_w_T_Aw_w_T_Bw_2]

由于A和B都是对称阵而且![\{w^T\}Bw][w_T_Bw]和![\{w^T\}Aw][w_T_Aw]都是标量，所以令导数等于零，就可以得到：

![(\{w^T\}Bw)Aw = (\{w^T\}Aw)Bw][w_T_Bw_Aw _ _w_T_Aw_Bw]

然后我们令![\\lambda = L(w) = \\frac\{\{\{w^T\}Aw\}\}\{\{\{w^T\}Bw\}\}][lambda _ L_w_ _ _frac_w_T_Aw_w_T_Bw]，于是上面的式子可以写成：

![\{B^\{ - 1\}\}Aw = \\lambda w][B_ - 1_Aw _ _lambda w]

这里就得到了跟PCA相同的结论，最大化目标为一个矩阵的最大特征值，而投影的方向为该特征值对应的特征向量。到此，对于二分类问题的LDA分析就结束了，后面分析多分类问题的LDA。

--------------------

对于多分类问题，我们将类别扩展到n，而且我们需要找的是d维的超平面![\\\{ \{w\_1\},\{w\_2\}, \\cdots \{w\_d\}\\\}][_w_1_w_2_ _cdots _w_d]。上面方法的B矩阵仍然是适用的，只是把i从2扩大到n。我们需要重新定义A矩阵。我们定义全局散度为：

![T = \\sum\\limits\_\{i = 1\}^n \{(\{\\mu \_i\} - \\mu \} )\{(\{\\mu \_i\} - \\mu )^T\}][T _ _sum_limits_i _ 1_n _mu _i_ - _mu _ _mu _i_ - _mu _T]

其中![\\mu = \\sum\\limits\_\{i = 1\}^n \{\{\\mu \_i\}\}][mu _ _sum_limits_i _ 1_n _mu _i]。重新定义的A矩阵为：

![\{A^'\} = T - A \\\\ = \\sum\\limits\_\{i = 1\}^n \{(\{x\_i\}\} - \\mu )\{(\{x\_i\} - \\mu )^T\} - \\sum\\limits\_\{x \\in \{C\_i\}\} \{(x - \{\\mu \_i\}\} )\{(x - \{\\mu \_i\})^T\} \\\\ = \\sum\\limits\_\{j = 1\}^n \{\\sum\\limits\_\{x \\in \{C\_i\}\} \{((x - \\mu )(x - \} \} \\mu \{)^T\} - (x - \{\\mu \_j\})\{(x - \{\\mu \_j\})^T\}) = \\sum\\limits\_\{j = 1\}^n \{\{k\_j\}\} (\{\\mu \_j\} - \\mu )(\{\\mu \_j\} - \\mu )][A_ _ T - A _ _ _sum_limits_i _ 1_n _x_i_ - _mu _x_i_ - _mu _T_ - _sum_limits_x _in _C_i_ _x - _mu _i_ _x - _mu _i_T_ _ _ _sum_limits_j _ 1_n _sum_limits_x _in _C_i_ _x - _mu _x - _ _ _mu _T_ - _x - _mu _j_x - _mu _j_T_ _ _sum_limits_j _ 1_n _k_j_ _mu _j_ - _mu _mu _j_ - _mu]

所以最大化目标为：

![L(W) = \\frac\{\{tr(\{W^T\}A'W)\}\}\{\{tr(\{W^T\}BW)\}\}][L_W_ _ _frac_tr_W_T_A_W_tr_W_T_BW]

通过求解![\{B^\{ - 1\}\}A][B_ - 1_A]的最大的d个特征值所对应的特征向量就能得到最佳的超平面。这里跟PAC的第二种求解方法类似。总的来说PCA跟LAD的求解过程类似，都是通过求解矩阵的最大的前d个特征值对应的特征向量，从而求解出最佳的超平面。

[https_blog.csdn.net_qq_38366615_article_details_86663634]: https://blog.csdn.net/qq_38366615/article/details/86663634
[mu _1_rm_ _ _sum_limits_s _in _C_1_ s]: https://private.codecogs.com/gif.latex?%7B%5Cmu%20_1%7D%7B%5Crm%7B%20%3D%20%7D%7D%5Csum%5Climits_%7Bs%20%5Cin%20%7BC_1%7D%7D%20s
[mu _2_rm_ _ _sum_limits_s _in _C_2_ s]: https://private.codecogs.com/gif.latex?%7B%5Cmu%20_2%7D%7B%5Crm%7B%20%3D%20%7D%7D%5Csum%5Climits_%7Bs%20%5Cin%20%7BC_2%7D%7D%20s
[L _ _left_ _w_T_mu _1_ - _w_T_mu _2_ _right]: https://private.codecogs.com/gif.latex?L%20%3D%20%5Cleft%5C%7C%20%7B%7Bw%5ET%7D%7B%5Cmu%20_1%7D%20-%20%7Bw%5ET%7D%7B%5Cmu%20_2%7D%7D%20%5Cright%5C%7C
[left_ _begin_array_20_c_ _mathop _max _limits_w _left_ _w_T_mu _1_ - _mu _2_ _right_ _s.t._w_T_w _ 1_ _end_array_ _right.]: https://private.codecogs.com/gif.latex?%5Cleft%5C%7B%20%7B%5Cbegin%7Barray%7D%7B*%7B20%7D%7Bc%7D%7D%20%7B%5Cmathop%20%7B%5Cmax%20%7D%5Climits_w%20%5Cleft%5C%7C%20%7B%7Bw%5ET%7D%28%7B%5Cmu%20_1%7D%20-%20%7B%5Cmu%20_2%7D%29%7D%20%5Cright%5C%7C%7D%5C%5C%20%7Bs.t.%7Bw%5ET%7Dw%20%3D%201%7D%20%5Cend%7Barray%7D%7D%20%5Cright.
[mu _1_ - _mu _2]: https://private.codecogs.com/gif.latex?%7B%5Cmu%20_1%7D%20-%20%7B%5Cmu%20_2%7D
[L_w_ _ _frac_left_ _w_T_mu _1_ - _mu _2_ _right_2_D_1_ _ _D_2]: https://private.codecogs.com/gif.latex?L%28w%29%20%3D%20%5Cfrac%7B%7B%7B%7B%5Cleft%5C%7C%20%7B%7Bw%5ET%7D%28%7B%5Cmu%20_1%7D%20-%20%7B%5Cmu%20_2%7D%29%7D%20%5Cright%5C%7C%7D%5E2%7D%7D%7D%7B%7B%7BD_1%7D%20&plus;%20%7BD_2%7D%7D%7D
[D_1_D_2]: https://private.codecogs.com/gif.latex?%7BD_1%7D%2C%7BD_2%7D
[D_1_ _ _sum_limits_x _in _C_1_ _w_T_ _x - _mu _1_2_ _ _sum_limits_x _in _C_1_ _w_T_ _x - _mu _1_x - _mu _1_T_w]: https://private.codecogs.com/gif.latex?%7BD_1%7D%20%3D%20%5Csum%5Climits_%7Bx%20%5Cin%20%7BC_1%7D%7D%20%7B%28%7Bw%5ET%7D%7D%20%28x%20-%20%7B%5Cmu%20_1%7D%29%7B%29%5E2%7D%20%3D%20%5Csum%5Climits_%7Bx%20%5Cin%20%7BC_1%7D%7D%20%7B%7Bw%5ET%7D%7D%20%28x%20-%20%7B%5Cmu%20_1%7D%29%7B%28x%20-%20%7B%5Cmu%20_1%7D%29%5ET%7Dw
[D_2_ _ _sum_limits_x _in _C_2_ _w_T_ _x - _mu _2_x - _mu _2_T_w]: https://private.codecogs.com/gif.latex?%7BD_2%7D%20%3D%20%5Csum%5Climits_%7Bx%20%5Cin%20%7BC_2%7D%7D%20%7B%7Bw%5ET%7D%7D%20%28x%20-%20%7B%5Cmu%20_2%7D%29%7B%28x%20-%20%7B%5Cmu%20_2%7D%29%5ET%7Dw
[L_w_ _ _frac_w_T_mu _1_ - _mu _2_mu _1_ - _mu _2_T_w_sum_limits_x _in _C_i_ _w_T_x - _mu _i_x - _mu _i_T_w_]: https://private.codecogs.com/gif.latex?L%28w%29%20%3D%20%5Cfrac%7B%7B%7Bw%5ET%7D%28%7B%5Cmu%20_1%7D%20-%20%7B%5Cmu%20_2%7D%29%28%7B%5Cmu%20_1%7D%20-%20%7B%5Cmu%20_2%7D%29%5E%7BT%7Dw%7D%7D%7B%7B%5Csum%5Climits_%7Bx%20%5Cin%20%7BC_i%7D%7D%20%7B%7Bw%5ET%7D%28x%20-%20%7B%5Cmu%20_i%7D%29%28x%20-%20%7B%5Cmu%20_i%7D%29%5E%7BT%7Dw%7D%20%7D%7D
[A _ _mu _1_ - _mu _2_mu _1_ - _mu _2_T]: https://private.codecogs.com/gif.latex?A%20%3D%20%28%7B%5Cmu%20_1%7D%20-%20%7B%5Cmu%20_2%7D%29%7B%28%7B%5Cmu%20_1%7D%20-%20%7B%5Cmu%20_2%7D%29%5ET%7D
[B _ _sum_limits_x _in _C_i_ _x - _mu _i_ _x - _mu _i_T]: https://private.codecogs.com/gif.latex?B%20%3D%20%5Csum%5Climits_%7Bx%20%5Cin%20%7BC_i%7D%7D%20%7B%28x%20-%20%7B%5Cmu%20_i%7D%7D%20%29%7B%28x%20-%20%7B%5Cmu%20_i%7D%29%5ET%7D
[frac_partial L_w_partial w_ _ _frac_frac_partial _w_T_Aw_partial w_w_T_Bw - _frac_partial _w_T_Bw_partial w_w_T_Aw_w_T_Bw_2]: https://private.codecogs.com/gif.latex?%5Cfrac%7B%7B%5Cpartial%20L%28w%29%7D%7D%7B%7B%5Cpartial%20w%7D%7D%20%3D%20%5Cfrac%7B%7B%5Cfrac%7B%7B%5Cpartial%20%7Bw%5ET%7DAw%7D%7D%7B%7B%5Cpartial%20w%7D%7D%7Bw%5ET%7DBw%20-%20%5Cfrac%7B%7B%5Cpartial%20%7Bw%5ET%7DBw%7D%7D%7B%7B%5Cpartial%20w%7D%7D%7Bw%5ET%7DAw%7D%7D%7B%7B%7B%7B%28%7Bw%5ET%7DBw%29%7D%5E2%7D%7D%7D
[w_T_Bw]: https://private.codecogs.com/gif.latex?%7Bw%5ET%7DBw
[w_T_Aw]: https://private.codecogs.com/gif.latex?%7Bw%5ET%7DAw
[w_T_Bw_Aw _ _w_T_Aw_Bw]: https://private.codecogs.com/gif.latex?%28%7Bw%5ET%7DBw%29Aw%20%3D%20%28%7Bw%5ET%7DAw%29Bw
[lambda _ L_w_ _ _frac_w_T_Aw_w_T_Bw]: https://private.codecogs.com/gif.latex?%5Clambda%20%3D%20L%28w%29%20%3D%20%5Cfrac%7B%7B%7Bw%5ET%7DAw%7D%7D%7B%7B%7Bw%5ET%7DBw%7D%7D
[B_ - 1_Aw _ _lambda w]: https://private.codecogs.com/gif.latex?%7BB%5E%7B%20-%201%7D%7DAw%20%3D%20%5Clambda%20w
[_w_1_w_2_ _cdots _w_d]: https://private.codecogs.com/gif.latex?%5C%7B%20%7Bw_1%7D%2C%7Bw_2%7D%2C%20%5Ccdots%20%7Bw_d%7D%5C%7D
[T _ _sum_limits_i _ 1_n _mu _i_ - _mu _ _mu _i_ - _mu _T]: https://private.codecogs.com/gif.latex?T%20%3D%20%5Csum%5Climits_%7Bi%20%3D%201%7D%5En%20%7B%28%7B%5Cmu%20_i%7D%20-%20%5Cmu%20%7D%20%29%7B%28%7B%5Cmu%20_i%7D%20-%20%5Cmu%20%29%5ET%7D
[mu _ _sum_limits_i _ 1_n _mu _i]: https://private.codecogs.com/gif.latex?%5Cmu%20%3D%20%5Csum%5Climits_%7Bi%20%3D%201%7D%5En%20%7B%7B%5Cmu%20_i%7D%7D
[A_ _ T - A _ _ _sum_limits_i _ 1_n _x_i_ - _mu _x_i_ - _mu _T_ - _sum_limits_x _in _C_i_ _x - _mu _i_ _x - _mu _i_T_ _ _ _sum_limits_j _ 1_n _sum_limits_x _in _C_i_ _x - _mu _x - _ _ _mu _T_ - _x - _mu _j_x - _mu _j_T_ _ _sum_limits_j _ 1_n _k_j_ _mu _j_ - _mu _mu _j_ - _mu]: https://private.codecogs.com/gif.latex?%7BA%5E%27%7D%20%3D%20T%20-%20A%20%5C%5C%20%3D%20%5Csum%5Climits_%7Bi%20%3D%201%7D%5En%20%7B%28%7Bx_i%7D%7D%20-%20%5Cmu%20%29%7B%28%7Bx_i%7D%20-%20%5Cmu%20%29%5ET%7D%20-%20%5Csum%5Climits_%7Bx%20%5Cin%20%7BC_i%7D%7D%20%7B%28x%20-%20%7B%5Cmu%20_i%7D%7D%20%29%7B%28x%20-%20%7B%5Cmu%20_i%7D%29%5ET%7D%20%5C%5C%20%3D%20%5Csum%5Climits_%7Bj%20%3D%201%7D%5En%20%7B%5Csum%5Climits_%7Bx%20%5Cin%20%7BC_i%7D%7D%20%7B%28%28x%20-%20%5Cmu%20%29%28x%20-%20%7D%20%7D%20%5Cmu%20%7B%29%5ET%7D%20-%20%28x%20-%20%7B%5Cmu%20_j%7D%29%7B%28x%20-%20%7B%5Cmu%20_j%7D%29%5ET%7D%29%20%3D%20%5Csum%5Climits_%7Bj%20%3D%201%7D%5En%20%7B%7Bk_j%7D%7D%20%28%7B%5Cmu%20_j%7D%20-%20%5Cmu%20%29%28%7B%5Cmu%20_j%7D%20-%20%5Cmu%20%29
[L_W_ _ _frac_tr_W_T_A_W_tr_W_T_BW]: https://private.codecogs.com/gif.latex?L%28W%29%20%3D%20%5Cfrac%7B%7Btr%28%7BW%5ET%7DA%27W%29%7D%7D%7B%7Btr%28%7BW%5ET%7DBW%29%7D%7D
[B_ - 1_A]: https://private.codecogs.com/gif.latex?%7BB%5E%7B%20-%201%7D%7DA