mahout源码分析之DistributedLanczosSolver（六）完结篇

绝地灬酷狼 2022-05-12 23:20 106阅读 0赞

Mahout版本：0.7，hadoop版本：1.0.4，jdk：1.7.0\_25 64bit。

接上篇，分析完3个Job后得到继续往下：其实就剩下两个函数了：

List<Map.Entry<MatrixSlice, EigenStatus>> prunedEigenMeta = pruneEigens(eigenMetaData);
    
        saveCleanEigens(new Configuration(), prunedEigenMeta);

看pruneEigens函数：

private List<Map.Entry<MatrixSlice, EigenStatus>> pruneEigens(Map<MatrixSlice, EigenStatus> eigenMetaData) {
        List<Map.Entry<MatrixSlice, EigenStatus>> prunedEigenMeta = Lists.newArrayList();
    
        for (Map.Entry<MatrixSlice, EigenStatus> entry : eigenMetaData.entrySet()) {
          if (Math.abs(1 - entry.getValue().getCosAngle()) < maxError && entry.getValue().getEigenValue() > minEigenValue) {
            prunedEigenMeta.add(entry);
          }
        }

看到这里其实是做筛选的，三个job生成了三个eigenStatus，每个eigenStatus都有一个cosAngle和eigenValue，用这两个参数来判断是否应该保留，这三个总结如下：

第一个；
    resultantVector:
    [-285.43017035605783, -61.30237570857193, -68.94124551381431, -520.2302762811703, -3232.201254912267, -32.31785150049481, -37.63572264009423, -12.025276244275622, -28.58260635344015, -6.8801603142200065, -28.491567864130573, -68.13521243410383, 4382.173720122737]
    vector:
    [0.01671441233225078, 0.0935655369363106, 0.09132650234523473, -0.0680324702834075, -0.9461123439509093, 0.10210271255992123, 0.10042714365337412, 0.11137954332150339, 0.10331974823993555, 0.10621406378767596, 0.10586960137353602, 0.09262650242313884, 0.09059904726143547]
    eigenValue=newNorm/oldNorm=5479.061620543984/1=5479.061620543984;
    cosAngle=resultantVector.dot(vector) / newNorm * oldNorm=0.6300724679092792
    
    第二个：
    resultantVector：
    vector：
    [0.01180448947054423, 0.001703710024210367, 0.002100735590662567, 0.014221147454610283, 0.09654151173375553, 0.0025666815984826535, 0.0026147055494762234, 1.753144283209579E-4, 0.0017595900141802873, 0.0049406361794682024, 7.881250692924197E-4, 0.002873479530226361, 0.9951286321096425]
    eigenValue：6433335.386819993
    cosAngle=0.9999998030863401
    第三个：
    vector:
    [-0.2883450858059115, -0.29170231535763447, -0.29157035465385267, -0.28754185317979386, -0.26018076078737895, -0.2914154866344813, -0.2913995247546756, -0.2922103132689348, -0.2916837423401091, -0.29062644748002026, -0.2920066313645422, -0.2913135151887795, 0.03848561950058266]
    eigenValue=1442.6143913921014
    cosAngle=0.3671147029085018

可以看到只有第二个可以通过筛选，得到的prunedEigenMeta如下：

![SouthEast][]  
看下一个函数saveCleanEigens：

private void saveCleanEigens(Configuration conf, Collection<Map.Entry<MatrixSlice, EigenStatus>> prunedEigenMeta)
        throws IOException {
        Path path = new Path(outPath, CLEAN_EIGENVECTORS);
        FileSystem fs = FileSystem.get(path.toUri(), conf);
        SequenceFile.Writer seqWriter = new SequenceFile.Writer(fs, conf, path, IntWritable.class, VectorWritable.class);
        try {
          IntWritable iw = new IntWritable();
          int numEigensWritten = 0;
          for (Map.Entry<MatrixSlice, EigenStatus> pruneSlice : prunedEigenMeta) {
            MatrixSlice s = pruneSlice.getKey();
            EigenStatus meta = pruneSlice.getValue();
            EigenVector ev = new EigenVector(s.vector(),
                                             meta.getEigenValue(),
                                             Math.abs(1 - meta.getCosAngle()),
                                             s.index());
            //log.info("appending {} to {}", ev, path);
            Writable vw = new VectorWritable(ev);
            iw.set(s.index());
            seqWriter.append(iw, vw);
    
            // increment the number of eigenvectors written and see if we've
            // reached our specified limit, or if we wish to write all eigenvectors
            // (latter is built-in, since numEigensWritten will always be > 0
            numEigensWritten++;
            if (numEigensWritten == maxEigensToKeep) {
              log.info("{} of the {} total eigens have been written", maxEigensToKeep, prunedEigenMeta.size());
              break;
            }
          }
        } finally {
          Closeables.closeQuietly(seqWriter);
        }
        cleanedEigensPath = path;
      }

看保存的ev是什么吧：

![SouthEast 1][]  
还不是筛选出来的那个值，不过这里的误差就是1-cosAngle了；

分享，成长，快乐

转载请注明blog地址：[http://blog.csdn.net/fansy1990][http_blog.csdn.net_fansy1990]

[SouthEast]: /images/20220513/778768593a3a469387b246113a0c7bd5.png
[SouthEast 1]: /images/20220513/fa90467f03dc4913b02fa0107493e35b.png
[http_blog.csdn.net_fansy1990]: http://blog.csdn.net/fansy1990