《Python机器学习》特征抽取——kPCA

╰半橙微兮° 2023-06-17 15:00 4阅读 0赞

说

明：

1.  关于本书: [《Python机器学习》][Python]
2.  本笔记侧重代码调用，只描述了一些简单概念，本书的公式推导不在这里展示
3.  接[上文][Link 1]
    
    --------------------
    
    **特征抽取** 可以将原始数据集变换到一个维度更低的新的特征子空间，在尽可能多地保持相关信息的情况下，对数据进行压缩。

## 3. 核主成份分析 Kernel Principle Component Analysis ##

### 3.1 简单介绍 ###

*  现实世界中，并不是所有数据都是线性可分的
 *  通过LDA，PCA将其转化为线性问题并不是好的方法
 *  **线性可分 VS 非线性可分**

*  引入**核主成份分析**
 *  可以通过kPCA将非线性数据映射到高维空间，在高维空间下使用标准PCA将其映射到另一个低维空间

### 3.2 原理 ###

*  定义**非线性映射函数**，该函数可以对原始特征进行非线性组合，以将原始的d维数据集映射到更高维的k维特征空间。

![20191129154120614.png][]

*  例如：

![20191129154148681.png][]

*  计算协方差矩阵 Σ的通用公式:

![\\sum =\\frac\{1\}\{n-1\}XX^\{T\}][sum _frac_1_n-1_XX_T]

*  加上核函数后，计算协方差矩阵 ΣΣ 的通用公式:

![\\sum =\\frac\{1\}\{n-1\}\\o (X)\\o (X)^\{T\}][sum _frac_1_n-1_o _X_o _X_T]

*  常用核函数：

1.  多项式核： κ(x(i),x(j))=(x(i)Tx(j)+θ)pκ(x(i),x(j))=(x(i)Tx(j)+θ)p
2.  双曲正切核（sigmoid）：κ(x(i),x(j))=thah(ηx(i)Tx(j)+θ)κ(x(i),x(j))=thah(ηx(i)Tx(j)+θ)
3.  径向基核（RBF），高斯核函数：κ(x(i),x(j))=exp(−∥x(i)−x(j)∥2σ2)κ(x(i),x(j))=exp(−‖x(i)−x(j)‖2σ2) 或 κ(x(i),x(j))=exp(−γ∥x(i)−x(j)∥2)

*  基于RBF核的kPCA算法流程：

> 计算核矩阵 ![k][]，做如下计算：![k(x^\{(i)\},x^\{j\})=exp(-\\gamma )||x^\{(i)\}-x^\{(j)\}||^\{2\})][k_x_i_x_j_exp_-_gamma _x_i_-x_j_2]。需要计算任意两样本之间的值。例如，如果数据集包含100个训练样本，将得到一个![100\*100][100_100]维的对称核矩阵。

![eq][]

> 1.  通过如下公式计算，使得核矩阵 ![k][]更为聚集：![K^\{\`\}=K-l\_\{n\}K-Kl\_\{n\}+l\_\{n\}Kl\_\{n\}][K_K-l_n_K-Kl_n_l_n_Kl_n],其中，![l\_\{n\}][l_n]是一个![n\\ast n][n_ast n]的矩阵，其所有的值均是![1/n][1_n]。
> 2.  将聚集后的核矩阵的特征值按照降序排列，选择前k个特征值所对应的特征与标准PCA不同，这里的特征向量不是主成份轴，而是将样本映射到这些轴上。

### 3.3 使用Python实现kPCA ###

### 3.4 示例一 ###

创建如下数据：

import matplotlib.pyplot as plt
    from sklearn.datasets import make_moons
    
    X, y = make_moons(n_samples=100, random_state=123)
    
    plt.scatter(X[y == 0, 0], X[y == 0, 1], color='red', marker='^', alpha=0.5)
    plt.scatter(X[y == 1, 0], X[y == 1, 1], color='blue', marker='o', alpha=0.5)
    
    plt.tight_layout()
    # plt.savefig('./figures/half_moon_1.png', dpi=300)
    plt.show()

直接用PCA，得到如下右图，线性不可分！

from sklearn.decomposition import PCA
    from sklearn.preprocessing import StandardScaler
    
    scikit_pca = PCA(n_components=2)
    X_spca = scikit_pca.fit_transform(X)
    
    fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))
    
    ax[0].scatter(X_spca[y == 0, 0], X_spca[y == 0, 1],
                  color='red', marker='^', alpha=0.5)
    ax[0].scatter(X_spca[y == 1, 0], X_spca[y == 1, 1],
                  color='blue', marker='o', alpha=0.5)
    
    ax[1].scatter(X_spca[y == 0, 0], np.zeros((50, 1)) + 0.02,
                  color='red', marker='^', alpha=0.5)
    ax[1].scatter(X_spca[y == 1, 0], np.zeros((50, 1)) - 0.02,
                  color='blue', marker='o', alpha=0.5)
    
    ax[0].set_xlabel('PC1')
    ax[0].set_ylabel('PC2')
    ax[1].set_ylim([-1, 1])
    ax[1].set_yticks([])
    ax[1].set_xlabel('PC1')
    
    plt.tight_layout()
    # plt.savefig('./figures/half_moon_2.png', dpi=300)
    plt.show()

使用kPCA，使得数据线性可分

from matplotlib.ticker import FormatStrFormatter
    
    X_kpca = rbf_kernel_pca(X, gamma=15, n_components=2)
    
    fig, ax = plt.subplots(nrows=1,ncols=2, figsize=(7,3))
    ax[0].scatter(X_kpca[y==0, 0], X_kpca[y==0, 1], 
                color='red', marker='^', alpha=0.5)
    ax[0].scatter(X_kpca[y==1, 0], X_kpca[y==1, 1],
                color='blue', marker='o', alpha=0.5)
    
    ax[1].scatter(X_kpca[y==0, 0], np.zeros((50,1))+0.02, 
                color='red', marker='^', alpha=0.5)
    ax[1].scatter(X_kpca[y==1, 0], np.zeros((50,1))-0.02,
                color='blue', marker='o', alpha=0.5)
    
    ax[0].set_xlabel('PC1')
    ax[0].set_ylabel('PC2')
    ax[1].set_ylim([-1, 1])
    ax[1].set_yticks([])
    ax[1].set_xlabel('PC1')
    ax[0].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))
    ax[1].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))
    
    plt.tight_layout()
    # plt.savefig('./figures/half_moon_3.png', dpi=300)
    plt.show()

### 3.5 使用sklearn进行核主成份分析 ###

还是调包简单！可以通过kernal参数来选择不同核函数,得到的结果与上面的左图是一致的

rom sklearn.decomposition import KernelPCA
    
    X, y = make_moons(n_samples=100, random_state=123)
    scikit_kpca = KernelPCA(n_components=2, kernel='rbf', gamma=15)
    X_skernpca = scikit_kpca.fit_transform(X)
    
    plt.scatter(X_skernpca[y == 0, 0], X_skernpca[y == 0, 1],
                color='red', marker='^', alpha=0.5)
    plt.scatter(X_skernpca[y == 1, 0], X_skernpca[y == 1, 1],
                color='blue', marker='o', alpha=0.5)
    
    plt.xlabel('PC1')
    plt.ylabel('PC2')
    plt.tight_layout()
    # plt.savefig('./figures/scikit_kpca.png', dpi=300)
    plt.show()

完整代码：

from scipy.spatial.distance import pdist, squareform
    from scipy import exp
    from numpy.linalg import eigh
    import numpy as np
    
    def rbf_kernel_pca(X, gamma, n_components):
        """
        RBF kernel PCA implementation.
    
        Parameters
        ------------
        X: {NumPy ndarray}, shape = [n_samples, n_features]
    
        gamma: float
          Tuning parameter of the RBF kernel
    
        n_components: int
          Number of principal components to return
    
        Returns
        ------------
         X_pc: {NumPy ndarray}, shape = [n_samples, k_features]
           Projected dataset
    
        """
        # Calculate pairwise squared Euclidean distances
        # in the MxN dimensional dataset.
        sq_dists = pdist(X, 'sqeuclidean')
    
        # Convert pairwise distances into a square matrix.
        mat_sq_dists = squareform(sq_dists)
    
        # Compute the symmetric kernel matrix.
        K = exp(-gamma * mat_sq_dists)
    
        # Center the kernel matrix.
        N = K.shape[0]
        one_n = np.ones((N, N)) / N
        K = K - one_n.dot(K) - K.dot(one_n) + one_n.dot(K).dot(one_n)
    
        # Obtaining eigenpairs from the centered kernel matrix
        # numpy.linalg.eigh returns them in sorted order
        eigvals, eigvecs = eigh(K)
    
        # Collect the top k eigenvectors (projected samples)
        X_pc = np.column_stack((eigvecs[:, -i]
                                for i in range(1, n_components + 1)))
    
        return X_pc
    
    import matplotlib.pyplot as plt
    from sklearn.datasets import make_moons
    X, y = make_moons(n_samples=100, random_state=123)
    '''
    
    plt.scatter(X[y == 0, 0], X[y == 0, 1], color='red', marker='*', alpha=0.5)
    plt.scatter(X[y == 1, 0], X[y == 1, 1], color='blue', marker='o', alpha=0.5)
    
    plt.tight_layout()
    plt.savefig('./figures/half_moon_1.png', dpi=300)
    plt.show()
    '''
    
    
    from sklearn.decomposition import PCA
    from sklearn.preprocessing import StandardScaler
    
    scikit_pca = PCA(n_components=2)
    X_spca = scikit_pca.fit_transform(X)
    
    fig, ax = plt.subplots(nrows=1, ncols=2, figsize=(7, 3))
    
    ax[0].scatter(X_spca[y == 0, 0], X_spca[y == 0, 1],
                  color='red', marker='^', alpha=0.5)
    ax[0].scatter(X_spca[y == 1, 0], X_spca[y == 1, 1],
                  color='blue', marker='o', alpha=0.5)
    
    ax[1].scatter(X_spca[y == 0, 0], np.zeros((50, 1)) + 0.02,
                  color='red', marker='^', alpha=0.5)
    ax[1].scatter(X_spca[y == 1, 0], np.zeros((50, 1)) - 0.02,
                  color='blue', marker='o', alpha=0.5)
    
    ax[0].set_xlabel('PC1')
    ax[0].set_ylabel('PC2')
    ax[1].set_ylim([-1, 1])
    ax[1].set_yticks([])
    ax[1].set_xlabel('PC1')
    
    plt.tight_layout()
    # plt.savefig('./figures/half_moon_2.png', dpi=300)
    plt.show()
    
    print("----------------------------------------------")
    from matplotlib.ticker import FormatStrFormatter
    
    X_kpca = rbf_kernel_pca(X,gamma=15,n_components=2)
    fig,ax=plt.subplots(nrows=1,ncols=2,figsize=(7,3))
    
    ax[0].scatter(X_kpca[y==0,0],X_kpca[y==0,1],
                  color='red',marker='^',alpha=0.5)
    ax[0].scatter(X_kpca[y==1,0],X_kpca[y==1,1],
                  color='blue',marker='o',alpha=0.5)
    
    ax[1].scatter(X_kpca[y==0,0],np.zeros((50,1))+0.02,
                  color='red',marker='^',alpha=0.5)
    ax[1].scatter(X_kpca[y==1,0],np.zeros((50,1))-0.02,
                  color='blue',marker='o',alpha=0.5)
    
    ax[0].set_xlabel('PC1')
    ax[0].set_ylabel('PC2')
    ax[1].set_ylim([-1,1])
    ax[1].set_yticks([])
    ax[1].set_xlabel('PC1')
    ax[0].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))
    ax[1].xaxis.set_major_formatter(FormatStrFormatter('%0.1f'))
    
    plt.tight_layout()
    plt.savefig('./figures/half_moon_3.png',dpi=300)
    plt.show()
    
    
    
    print('--------------------------')
    from sklearn.decomposition import KernelPCA
    
    X,y = make_moons(n_samples=100,random_state=123)
    scikit_kpca = KernelPCA(n_components=2,kernel='rbf',gamma=15)
    X_skernpca = scikit_kpca.fit_transform(X)
    
    plt.scatter(X_skernpca[y==0,0],X_skernpca[y==0,1],
                color='red',marker='^',alpha=0.5)
    plt.scatter(X_skernpca[y==1,0],X_skernpca[y==1,1],
                color='blue',marker='o',alpha=0.5)
    
    plt.xlabel('PC1')
    plt.xlabel('PC2')
    plt.savefig('./figures/scikit_kpca.png',dpi=300)
    plt.show()

[Python]: https://book.douban.com/subject/27000110/
[Link 1]: http://blog.csdn.net/weixin_40604987/article/details/79615968
[20191129154120614.png]: https://img-blog.csdnimg.cn/20191129154120614.png
[20191129154148681.png]: https://img-blog.csdnimg.cn/20191129154148681.png
[sum _frac_1_n-1_XX_T]: https://private.codecogs.com/gif.latex?%5Csum%20%3D%5Cfrac%7B1%7D%7Bn-1%7DXX%5E%7BT%7D
[sum _frac_1_n-1_o _X_o _X_T]: https://private.codecogs.com/gif.latex?%5Csum%20%3D%5Cfrac%7B1%7D%7Bn-1%7D%5Co%20%28X%29%5Co%20%28X%29%5E%7BT%7D
[k]: https://private.codecogs.com/gif.latex?k
[k_x_i_x_j_exp_-_gamma _x_i_-x_j_2]: https://private.codecogs.com/gif.latex?k%28x%5E%7B%28i%29%7D%2Cx%5E%7Bj%7D%29%3Dexp%28-%5Cgamma%20%29%7C%7Cx%5E%7B%28i%29%7D-x%5E%7B%28j%29%7D%7C%7C%5E%7B2%7D%29
[100_100]: https://private.codecogs.com/gif.latex?100*100
[eq]: http://latex.codecogs.com/gif.latex?%5Cbegin%7Bbmatrix%7D&space;%5Ckappa&space;%28x%5E%7B%281%29%7D,&space;x%5E%7B%281%29%7D%29&space;&%5Ckappa&space;%28x%5E%7B%281%29%7D,&space;x%5E%7B%282%29%7D%29&space;&...&space;&%5Ckappa&space;%28x%5E%7B%281%29%7D,&space;x%5E%7B%28n%29%7D%29&space;%5C%5C&space;%5Ckappa&space;%28x%5E%7B%282%29%7D,&space;x%5E%7B%281%29%7D%29&%5Ckappa&space;%28x%5E%7B%282%29%7D,&space;x%5E%7B%28j%29%7D%29&space;&...&space;&%5Ckappa&space;%28x%5E%7B%283%29%7D,&space;x%5E%7B%28n%29%7D%29&space;%5C%5C&space;...&space;&...&space;&...&space;&...&space;%5C%5C&space;%5Ckappa&space;%28x%5E%7B%28n%29%7D,&space;x%5E%7B%281%29%7D%29&space;&%5Ckappa&space;%28x%5E%7B%28n%29%7D,&space;x%5E%7B%282%29%7D%29&space;&...&space;&%5Ckappa&space;%28x%5E%7B%28n%29%7D,&space;x%5E%7B%28n%29%7D%29&space;%5Cend%7Bbmatrix%7D
[K_K-l_n_K-Kl_n_l_n_Kl_n]: https://private.codecogs.com/gif.latex?K%5E%7B%60%7D%3DK-l_%7Bn%7DK-Kl_%7Bn%7D&plus;l_%7Bn%7DKl_%7Bn%7D
[l_n]: https://private.codecogs.com/gif.latex?l_%7Bn%7D
[n_ast n]: https://private.codecogs.com/gif.latex?n%5Cast%20n
[1_n]: https://private.codecogs.com/gif.latex?1/n