PCA算法的最小平方误差解释

今天药忘吃喽~ 2021-12-14 12:51 229阅读 0赞

PCA算法另外一种理解角度是：最小化点到投影后点的距离平方和.

假设我们有m个样本点，且都位于n维空间 [![$x\\in \\mathbb\{R\}^n$][x_in _mathbb_R_n]][x_in _mathbb_R_n_x_in _mathbb_R_n]中，而我们要把原n维空间中的样本点投影到k维子空间W中去（k<n），并使得这m个点到投影点的距离(即投影误差)的平方和最小.我们假设投影到的k维子空间的标准正交基（orthonormal basis）为 [![$u\_1,u\_2,\\cdots,u\_k$][u_1_u_2_cdots_u_k]][u_1_u_2_cdots_u_k_u_1_u_2_cdots_u_k]，这组标准正交基组成了一个[![$n\\times k$][n_times k]][n_times k_n_times k]的矩阵U：

[![U=\\begin\{bmatrix\} u\_1\\Bigg| u\_2\\Bigg|\\cdots\\Bigg| u\_k \\end\{bmatrix\}][U_begin_bmatrix_ u_1_Bigg_ u_2_Bigg_cdots_Bigg_ u_k _end_bmatrix]][U_begin_bmatrix_ u_1_Bigg_ u_2_Bigg_cdots_Bigg_ u_k _end_bmatrix_U_begin_bmatrix_ u_1_Bigg_ u_2_Bigg_cdots_Bigg_ u_k _end_bmatrix]

则[![$P=UU^T$][P_UU_T]][P_UU_T_P_UU_T]称为子空间W 的投影矩阵（projection matrix）。

如果我们不从标准正交基出发，如何求得W的投影矩阵？设[![$a\_1,a\_2,...,a\_k$][a_1_a_2_..._a_k]][a_1_a_2_..._a_k_a_1_a_2_..._a_k]是W 的任意一组基，形成一个[![$n\\times k$][n_times k]][n_times k_n_times k]的矩阵[![A=\\begin\{bmatrix\} a\_1\\Bigg| a\_2\\Bigg|\\cdots\\Bigg| a\_k \\end\{bmatrix\}][A_begin_bmatrix_ a_1_Bigg_ a_2_Bigg_cdots_Bigg_ a_k _end_bmatrix]][A_begin_bmatrix_ a_1_Bigg_ a_2_Bigg_cdots_Bigg_ a_k _end_bmatrix_A_begin_bmatrix_ a_1_Bigg_ a_2_Bigg_cdots_Bigg_ a_k _end_bmatrix]则W的投影矩阵是[![$A(A^TA)^\{-1\}A^T$][A_A_TA_-1_A_T]][A_A_TA_-1_A_T_A_A_TA_-1_A_T]

投影矩阵具有如下性质：

[![\\begin\{aligned\} &P^n=P(n=1,2,\\cdots),\\quad P^T=P \\\\ &(I-P)^n=I-P(n=1,2,\\cdots),\\quad (I-P)^T=I-P \\end\{aligned\}][begin_aligned_ _P_n_P_n_1_2_cdots_quad P_T_P _ _I-P_n_I-P_n_1_2_cdots_quad _I-P_T_I-P _end_aligned]][begin_aligned_ _P_n_P_n_1_2_cdots_quad P_T_P _ _I-P_n_I-P_n_1_2_cdots_quad _I-P_T_I-P _end_aligned_begin_aligned_ _P_n_P_n_1_2_cdots_quad P_T_P _ _I-P_n_I-P_n_1_2_cdots_quad _I-P_T_I-P _end_aligned]

记每一个点[![$x^\{(i)\}$][x_i]][x_i_x_i]对应的投影误差为[![$e^\{(i)\}$][e_i]][e_i_e_i]，且投影误差的表达式为[![$e^\{(i)\}=(I-P)x^\{(i)\}$][e_i_I-P_x_i]][e_i_I-P_x_i_e_i_I-P_x_i]，那么我们要最小化的表达式为：

[![E'=\\sum\_\{i=1\}^\{m\}e^\{(i)T\}e^\{(i)\}][E_sum_i_1_m_e_i_T_e_i]][E_sum_i_1_m_e_i_T_e_i_E_sum_i_1_m_e_i_T_e_i]

为了后面的推导方便，我将上式除以[![\\frac\{1\}\{m\}][frac_1_m]][frac_1_m_frac_1_m]即样本个数），由于其是定值，所以不影响我们问题的求解

[![\\begin\{aligned\} E&=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}e^\{(i)T\}e^\{(i)\}\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\[(I-P)x^\{(i)\}\]^T (I-P)x^\{(i)\}\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)T\}(I-P)^T (I-P)x^\{(i)\}\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)T\}(I-P)^2 x^\{(i)\}\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)T\}(I-P)x^\{(i)\}\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)T\}x^\{(i)\}-\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\} x^\{(i)T\}Px^\{(i)\}\\\\ \\end\{aligned\}][begin_aligned_ E_frac_1_m_sum_i_1_m_e_i_T_e_i_ _frac_1_m_sum_i_1_m_I-P_x_i_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_2 x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_x_i_-_frac_1_m_sum_i_1_m_ x_i_T_Px_i_ _end_aligned]][begin_aligned_ E_frac_1_m_sum_i_1_m_e_i_T_e_i_ _frac_1_m_sum_i_1_m_I-P_x_i_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_2 x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_x_i_-_frac_1_m_sum_i_1_m_ x_i_T_Px_i_ _end_aligned_begin_aligned_ E_frac_1_m_sum_i_1_m_e_i_T_e_i_ _frac_1_m_sum_i_1_m_I-P_x_i_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_2 x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_x_i_-_frac_1_m_sum_i_1_m_ x_i_T_Px_i_ _end_aligned]

由于[![$x^\{(i)\},i=1,2,...,m$][x_i_i_1_2_..._m]][x_i_i_1_2_..._m_x_i_i_1_2_..._m]是预先给定的样本点，故上式中第一项是定值，因此我们的问题转化为了求第二项的最大值，即

[![\\max\_P \\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)T\}Px^\{(i)\}][max_P _frac_1_m_sum_i_1_m_x_i_T_Px_i]][max_P _frac_1_m_sum_i_1_m_x_i_T_Px_i_max_P _frac_1_m_sum_i_1_m_x_i_T_Px_i]

由于[![$P=UU^T$][P_UU_T]][P_UU_T_P_UU_T]（其中U是以子空间W的标准正交基为列构成的矩阵），上面的问题等价于[![\\max\_U \\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)T\}UU^Tx^\{(i)\}][max_U _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i]][max_U _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_max_U _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i]

对其进一步化简得：

[![\\begin\{aligned\} \\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)T\}UU^Tx^\{(i)\} &= \\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}(U^Tx^\{(i)\})^T(U^Tx^\{(i)\})\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}(u\_1^Tx^\{(i)\},u\_2^Tx^\{(i)\}, ...,u\_k^Tx^\{(i)\})\\cdot(u\_1^Tx^\{(i)\},u\_2^Tx^\{(i)\}, ...,u\_k^Tx^\{(i)\})^T\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^k (u\_j^Tx^\{(i)\})^2\\\\ &=\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}\\sum\_\{j=1\}^k u\_j^Tx^\{(i)\}x^\{(i)T\}u\_j\\\\ &=\\sum\_\{j=1\}^k u\_j^T(\\frac\{1\}\{m\}\\sum\_\{i=1\}^\{m\}x^\{(i)\}x^\{(i)T\}) u\_j\\\\ &=\\sum\_\{j=1\}^k u\_j^T\\Sigma u\_j \\end\{aligned\}][begin_aligned_ _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_ _ _frac_1_m_sum_i_1_m_U_Tx_i_T_U_Tx_i_ _frac_1_m_sum_i_1_m_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_cdot_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_T_ _frac_1_m_sum_i_1_m_sum_j_1_k _u_j_Tx_i_2_ _frac_1_m_sum_i_1_m_sum_j_1_k u_j_Tx_i_x_i_T_u_j_ _sum_j_1_k u_j_T_frac_1_m_sum_i_1_m_x_i_x_i_T_ u_j_ _sum_j_1_k u_j_T_Sigma u_j _end_aligned]][begin_aligned_ _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_ _ _frac_1_m_sum_i_1_m_U_Tx_i_T_U_Tx_i_ _frac_1_m_sum_i_1_m_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_cdot_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_T_ _frac_1_m_sum_i_1_m_sum_j_1_k _u_j_Tx_i_2_ _frac_1_m_sum_i_1_m_sum_j_1_k u_j_Tx_i_x_i_T_u_j_ _sum_j_1_k u_j_T_frac_1_m_sum_i_1_m_x_i_x_i_T_ u_j_ _sum_j_1_k u_j_T_Sigma u_j _end_aligned_begin_aligned_ _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_ _ _frac_1_m_sum_i_1_m_U_Tx_i_T_U_Tx_i_ _frac_1_m_sum_i_1_m_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_cdot_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_T_ _frac_1_m_sum_i_1_m_sum_j_1_k _u_j_Tx_i_2_ _frac_1_m_sum_i_1_m_sum_j_1_k u_j_Tx_i_x_i_T_u_j_ _sum_j_1_k u_j_T_frac_1_m_sum_i_1_m_x_i_x_i_T_ u_j_ _sum_j_1_k u_j_T_Sigma u_j _end_aligned] 因此，[![$\\min E$][min E]][min E_min E]等价于

[![\\begin\{aligned\} &\\max\_\{u\_1,u\_2,\\cdots,u\_k\}\\sum\_\{j=1\}^\{k\}u\_j^T\\Sigma u\_j\\\\ &s.t.\\quad u\_j^Tu\_j=1(j=1,2,\\cdots,k) \\end\{aligned\}][begin_aligned_ _max_u_1_u_2_cdots_u_k_sum_j_1_k_u_j_T_Sigma u_j_ _s.t._quad u_j_Tu_j_1_j_1_2_cdots_k_ _end_aligned]][begin_aligned_ _max_u_1_u_2_cdots_u_k_sum_j_1_k_u_j_T_Sigma u_j_ _s.t._quad u_j_Tu_j_1_j_1_2_cdots_k_ _end_aligned_begin_aligned_ _max_u_1_u_2_cdots_u_k_sum_j_1_k_u_j_T_Sigma u_j_ _s.t._quad u_j_Tu_j_1_j_1_2_cdots_k_ _end_aligned]

求解上面的[![$u\_j$][u_j]][u_j_u_j]要用到最大方差解释中使用的Lagrangian Multiplier，在此不再赘述，而最后求得的[![$u\_1,u\_2,\\cdots,u\_k$][u_1_u_2_cdots_u_k]][u_1_u_2_cdots_u_k_u_1_u_2_cdots_u_k]就是协方差矩阵[![$\\Sigma$][Sigma]][Sigma_Sigma]的前k个特征向量

转载于:https://www.cnblogs.com/wacc/p/3428110.html

[x_in _mathbb_R_n]: http://latex.codecogs.com/gif.latex?%24x%5Cin&space;%5Cmathbb%7BR%7D%5En%24
[x_in _mathbb_R_n_x_in _mathbb_R_n]: http://private.codecogs.com/eqnedit.php?latex=%24x%5Cin&space;%5Cmathbb%7BR%7D%5En%24
[u_1_u_2_cdots_u_k]: http://latex.codecogs.com/gif.latex?%24u_1,u_2,%5Ccdots,u_k%24
[u_1_u_2_cdots_u_k_u_1_u_2_cdots_u_k]: http://private.codecogs.com/eqnedit.php?latex=%24u_1,u_2,%5Ccdots,u_k%24
[n_times k]: http://latex.codecogs.com/gif.latex?%24n%5Ctimes&space;k%24
[n_times k_n_times k]: /images/20211214/5372a9de77d04e78be48c5c6c2c2c20f.png
[U_begin_bmatrix_ u_1_Bigg_ u_2_Bigg_cdots_Bigg_ u_k _end_bmatrix]: http://latex.codecogs.com/gif.latex?U=%5Cbegin%7Bbmatrix%7D&space;u_1%5CBigg%7C&space;u_2%5CBigg%7C%5Ccdots%5CBigg%7C&space;u_k&space;%5Cend%7Bbmatrix%7D
[U_begin_bmatrix_ u_1_Bigg_ u_2_Bigg_cdots_Bigg_ u_k _end_bmatrix_U_begin_bmatrix_ u_1_Bigg_ u_2_Bigg_cdots_Bigg_ u_k _end_bmatrix]: /images/20211214/c434014fe94041e5957069414b289766.png
[P_UU_T]: http://latex.codecogs.com/gif.latex?%24P=UU%5ET%24
[P_UU_T_P_UU_T]: /images/20211214/1c5e530e0b7e469fb3563a184ada86fa.png
[a_1_a_2_..._a_k]: http://latex.codecogs.com/gif.latex?%24a_1,a_2,...,a_k%24
[a_1_a_2_..._a_k_a_1_a_2_..._a_k]: http://private.codecogs.com/eqnedit.php?latex=%24a_1,a_2,...,a_k%24
[A_begin_bmatrix_ a_1_Bigg_ a_2_Bigg_cdots_Bigg_ a_k _end_bmatrix]: http://latex.codecogs.com/gif.latex?A=%5Cbegin%7Bbmatrix%7D&space;a_1%5CBigg%7C&space;a_2%5CBigg%7C%5Ccdots%5CBigg%7C&space;a_k&space;%5Cend%7Bbmatrix%7D
[A_begin_bmatrix_ a_1_Bigg_ a_2_Bigg_cdots_Bigg_ a_k _end_bmatrix_A_begin_bmatrix_ a_1_Bigg_ a_2_Bigg_cdots_Bigg_ a_k _end_bmatrix]: http://private.codecogs.com/eqnedit.php?latex=A=%5Cbegin%7Bbmatrix%7D&space;a_1%5CBigg%7C&space;a_2%5CBigg%7C%5Ccdots%5CBigg%7C&space;a_k&space;%5Cend%7Bbmatrix%7D
[A_A_TA_-1_A_T]: http://latex.codecogs.com/gif.latex?%24A%28A%5ETA%29%5E%7B-1%7DA%5ET%24
[A_A_TA_-1_A_T_A_A_TA_-1_A_T]: /images/20211214/60b1aae4cd6249198643b0c4d96ecc45.png
[begin_aligned_ _P_n_P_n_1_2_cdots_quad P_T_P _ _I-P_n_I-P_n_1_2_cdots_quad _I-P_T_I-P _end_aligned]: http://latex.codecogs.com/gif.latex?%5Cbegin%7Baligned%7D&space;&P%5En=P%28n=1,2,%5Ccdots%29,%5Cquad&space;P%5ET=P&space;%5C%5C&space;&%28I-P%29%5En=I-P%28n=1,2,%5Ccdots%29,%5Cquad&space;%28I-P%29%5ET=I-P&space;%5Cend%7Baligned%7D
[begin_aligned_ _P_n_P_n_1_2_cdots_quad P_T_P _ _I-P_n_I-P_n_1_2_cdots_quad _I-P_T_I-P _end_aligned_begin_aligned_ _P_n_P_n_1_2_cdots_quad P_T_P _ _I-P_n_I-P_n_1_2_cdots_quad _I-P_T_I-P _end_aligned]: /images/20211214/05d0a62971da4fbf8f93d4ae5e5f4336.png
[x_i]: http://latex.codecogs.com/gif.latex?%24x%5E%7B%28i%29%7D%24
[x_i_x_i]: http://private.codecogs.com/eqnedit.php?latex=%24x%5E%7B%28i%29%7D%24
[e_i]: http://latex.codecogs.com/gif.latex?%24e%5E%7B%28i%29%7D%24
[e_i_e_i]: http://private.codecogs.com/eqnedit.php?latex=%24e%5E%7B%28i%29%7D%24
[e_i_I-P_x_i]: http://latex.codecogs.com/gif.latex?%24e%5E%7B%28i%29%7D=%28I-P%29x%5E%7B%28i%29%7D%24
[e_i_I-P_x_i_e_i_I-P_x_i]: /images/20211214/dfa37dae191740b1ab7316a9b180a461.png
[E_sum_i_1_m_e_i_T_e_i]: http://latex.codecogs.com/gif.latex?E'=%5Csum_%7Bi=1%7D%5E%7Bm%7De%5E%7B%28i%29T%7De%5E%7B%28i%29%7D
[E_sum_i_1_m_e_i_T_e_i_E_sum_i_1_m_e_i_T_e_i]: /images/20211214/a7a70d2d8e36435f8338fea690ff3412.png
[frac_1_m]: http://latex.codecogs.com/gif.latex?%5Cfrac%7B1%7D%7Bm%7D
[frac_1_m_frac_1_m]: /images/20211214/30c1074ee0294fb885740b01cf37f869.png
[begin_aligned_ E_frac_1_m_sum_i_1_m_e_i_T_e_i_ _frac_1_m_sum_i_1_m_I-P_x_i_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_2 x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_x_i_-_frac_1_m_sum_i_1_m_ x_i_T_Px_i_ _end_aligned]: http://latex.codecogs.com/gif.latex?%5Cbegin%7Baligned%7D&space;E&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7De%5E%7B%28i%29T%7De%5E%7B%28i%29%7D%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%5B%28I-P%29x%5E%7B%28i%29%7D%5D%5ET&space;%28I-P%29x%5E%7B%28i%29%7D%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7D%28I-P%29%5ET&space;%28I-P%29x%5E%7B%28i%29%7D%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7D%28I-P%29%5E2&space;x%5E%7B%28i%29%7D%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7D%28I-P%29x%5E%7B%28i%29%7D%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7Dx%5E%7B%28i%29%7D-%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D&space;x%5E%7B%28i%29T%7DPx%5E%7B%28i%29%7D%5C%5C&space;%5Cend%7Baligned%7D
[begin_aligned_ E_frac_1_m_sum_i_1_m_e_i_T_e_i_ _frac_1_m_sum_i_1_m_I-P_x_i_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_2 x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_x_i_-_frac_1_m_sum_i_1_m_ x_i_T_Px_i_ _end_aligned_begin_aligned_ E_frac_1_m_sum_i_1_m_e_i_T_e_i_ _frac_1_m_sum_i_1_m_I-P_x_i_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_T _I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_2 x_i_ _frac_1_m_sum_i_1_m_x_i_T_I-P_x_i_ _frac_1_m_sum_i_1_m_x_i_T_x_i_-_frac_1_m_sum_i_1_m_ x_i_T_Px_i_ _end_aligned]: /images/20211214/6e8098b5601e4642b617d8154bd318f2.png
[x_i_i_1_2_..._m]: http://latex.codecogs.com/gif.latex?%24x%5E%7B%28i%29%7D,i=1,2,...,m%24
[x_i_i_1_2_..._m_x_i_i_1_2_..._m]: /images/20211214/8e15b505eed64045a0ece180aa5f6176.png
[max_P _frac_1_m_sum_i_1_m_x_i_T_Px_i]: http://latex.codecogs.com/gif.latex?%5Cmax_P&space;%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7DPx%5E%7B%28i%29%7D
[max_P _frac_1_m_sum_i_1_m_x_i_T_Px_i_max_P _frac_1_m_sum_i_1_m_x_i_T_Px_i]: /images/20211214/9fad6f9b0f6440a69b32512ed3401399.png
[max_U _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i]: http://latex.codecogs.com/gif.latex?%5Cmax_U&space;%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7DUU%5ETx%5E%7B%28i%29%7D
[max_U _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_max_U _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i]: /images/20211214/de874226dcd84d91ab87f13eacb60045.png
[begin_aligned_ _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_ _ _frac_1_m_sum_i_1_m_U_Tx_i_T_U_Tx_i_ _frac_1_m_sum_i_1_m_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_cdot_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_T_ _frac_1_m_sum_i_1_m_sum_j_1_k _u_j_Tx_i_2_ _frac_1_m_sum_i_1_m_sum_j_1_k u_j_Tx_i_x_i_T_u_j_ _sum_j_1_k u_j_T_frac_1_m_sum_i_1_m_x_i_x_i_T_ u_j_ _sum_j_1_k u_j_T_Sigma u_j _end_aligned]: http://latex.codecogs.com/gif.latex?%5Cbegin%7Baligned%7D&space;%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7DUU%5ETx%5E%7B%28i%29%7D&space;&=&space;%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%28U%5ETx%5E%7B%28i%29%7D%29%5ET%28U%5ETx%5E%7B%28i%29%7D%29%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%28u_1%5ETx%5E%7B%28i%29%7D,u_2%5ETx%5E%7B%28i%29%7D,&space;...,u_k%5ETx%5E%7B%28i%29%7D%29%5Ccdot%28u_1%5ETx%5E%7B%28i%29%7D,u_2%5ETx%5E%7B%28i%29%7D,&space;...,u_k%5ETx%5E%7B%28i%29%7D%29%5ET%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%5Csum_%7Bj=1%7D%5Ek&space;%28u_j%5ETx%5E%7B%28i%29%7D%29%5E2%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%5Csum_%7Bj=1%7D%5Ek&space;u_j%5ETx%5E%7B%28i%29%7Dx%5E%7B%28i%29T%7Du_j%5C%5C&space;&=%5Csum_%7Bj=1%7D%5Ek&space;u_j%5ET%28%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29%7Dx%5E%7B%28i%29T%7D%29&space;u_j%5C%5C&space;&=%5Csum_%7Bj=1%7D%5Ek&space;u_j%5ET%5CSigma&space;u_j&space;%5Cend%7Baligned%7D
[begin_aligned_ _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_ _ _frac_1_m_sum_i_1_m_U_Tx_i_T_U_Tx_i_ _frac_1_m_sum_i_1_m_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_cdot_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_T_ _frac_1_m_sum_i_1_m_sum_j_1_k _u_j_Tx_i_2_ _frac_1_m_sum_i_1_m_sum_j_1_k u_j_Tx_i_x_i_T_u_j_ _sum_j_1_k u_j_T_frac_1_m_sum_i_1_m_x_i_x_i_T_ u_j_ _sum_j_1_k u_j_T_Sigma u_j _end_aligned_begin_aligned_ _frac_1_m_sum_i_1_m_x_i_T_UU_Tx_i_ _ _frac_1_m_sum_i_1_m_U_Tx_i_T_U_Tx_i_ _frac_1_m_sum_i_1_m_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_cdot_u_1_Tx_i_u_2_Tx_i_ ..._u_k_Tx_i_T_ _frac_1_m_sum_i_1_m_sum_j_1_k _u_j_Tx_i_2_ _frac_1_m_sum_i_1_m_sum_j_1_k u_j_Tx_i_x_i_T_u_j_ _sum_j_1_k u_j_T_frac_1_m_sum_i_1_m_x_i_x_i_T_ u_j_ _sum_j_1_k u_j_T_Sigma u_j _end_aligned]: http://private.codecogs.com/eqnedit.php?latex=%5Cbegin%7Baligned%7D&space;%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29T%7DUU%5ETx%5E%7B%28i%29%7D&space;&=&space;%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%28U%5ETx%5E%7B%28i%29%7D%29%5ET%28U%5ETx%5E%7B%28i%29%7D%29%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%28u_1%5ETx%5E%7B%28i%29%7D,u_2%5ETx%5E%7B%28i%29%7D,&space;...,u_k%5ETx%5E%7B%28i%29%7D%29%5Ccdot%28u_1%5ETx%5E%7B%28i%29%7D,u_2%5ETx%5E%7B%28i%29%7D,&space;...,u_k%5ETx%5E%7B%28i%29%7D%29%5ET%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%5Csum_%7Bj=1%7D%5Ek&space;%28u_j%5ETx%5E%7B%28i%29%7D%29%5E2%5C%5C&space;&=%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7D%5Csum_%7Bj=1%7D%5Ek&space;u_j%5ETx%5E%7B%28i%29%7Dx%5E%7B%28i%29T%7Du_j%5C%5C&space;&=%5Csum_%7Bj=1%7D%5Ek&space;u_j%5ET%28%5Cfrac%7B1%7D%7Bm%7D%5Csum_%7Bi=1%7D%5E%7Bm%7Dx%5E%7B%28i%29%7Dx%5E%7B%28i%29T%7D%29&space;u_j%5C%5C&space;&=%5Csum_%7Bj=1%7D%5Ek&space;u_j%5ET%5CSigma&space;u_j&space;%5Cend%7Baligned%7D
[min E]: http://latex.codecogs.com/gif.latex?%24%5Cmin&space;E%24
[min E_min E]: /images/20211214/60e646feabea4f1b84bbd1ceb05646a8.png
[begin_aligned_ _max_u_1_u_2_cdots_u_k_sum_j_1_k_u_j_T_Sigma u_j_ _s.t._quad u_j_Tu_j_1_j_1_2_cdots_k_ _end_aligned]: http://latex.codecogs.com/gif.latex?%5Cbegin%7Baligned%7D&space;&%5Cmax_%7Bu_1,u_2,%5Ccdots,u_k%7D%5Csum_%7Bj=1%7D%5E%7Bk%7Du_j%5ET%5CSigma&space;u_j%5C%5C&space;&s.t.%5Cquad&space;u_j%5ETu_j=1%28j=1,2,%5Ccdots,k%29&space;%5Cend%7Baligned%7D
[begin_aligned_ _max_u_1_u_2_cdots_u_k_sum_j_1_k_u_j_T_Sigma u_j_ _s.t._quad u_j_Tu_j_1_j_1_2_cdots_k_ _end_aligned_begin_aligned_ _max_u_1_u_2_cdots_u_k_sum_j_1_k_u_j_T_Sigma u_j_ _s.t._quad u_j_Tu_j_1_j_1_2_cdots_k_ _end_aligned]: /images/20211214/a830e24e62904e0e9b980ff8a2511f26.png
[u_j]: http://latex.codecogs.com/gif.latex?%24u_j%24
[u_j_u_j]: http://private.codecogs.com/eqnedit.php?latex=%24u_j%24
[Sigma]: http://latex.codecogs.com/gif.latex?%24%5CSigma%24
[Sigma_Sigma]: /images/20211214/fb62cf55af6f48609e95d6772eadbd31.png