Mahout初探

川长思鸟来 2022-05-27 10:27 102阅读 0赞

[mahout官网][mahout],执行下面命令,详细安装过程可参考[Mahout介绍、安装与应用案例][Mahout]

tar -xzvf apache-mahout-distribution-0.13.0.tar.gz
     ln -s apache-mahout-distribution-0.13.0 mahout

修改profile环境变量

export MAHOUT_HOME=/home/hadoop/application/mahout
    export MAHOUT_CONF_DIR=/home/hadoop/application/mahout/conf
    export PATH=$PATH:$MAHOUT_HOME/bin

mahout下载测试数据，进行验证

mkdir -p /home/hadoop/download
    cd /home/hadoop/download
    wget http://labfile.oss.aliyuncs.com/courses/237/synthetic_control.data
    mkdir -p /home/hadoop/application/mahout/testdata
    cp synthetic_control.data /home/hadoop/application/mahout/testdata/
    hadoop fs -mkdir ./testdata
    hadoop fs -put /home/hadoop/application/mahout/testdata/synthetic_control.data ./testdata
    mahout org.apache.mahout.clustering.syntheticcontrol.kmeans.Job
    hadoop fs -ls ./output # 查看输出
    mahout vectordump -i ./output/data/part-m-00000 # 查看数据内容

**1、第一个推荐程序**  
这个程序来自《Mahout in Action Mahout 实战》，Sean Owen Robin Anil著的

import java.io.File;
    import java.util.List;
    
    import org.apache.mahout.cf.taste.impl.model.file.FileDataModel;
    import org.apache.mahout.cf.taste.impl.neighborhood.NearestNUserNeighborhood;
    import org.apache.mahout.cf.taste.impl.recommender.GenericUserBasedRecommender;
    import org.apache.mahout.cf.taste.impl.similarity.PearsonCorrelationSimilarity;
    import org.apache.mahout.cf.taste.model.DataModel;
    import org.apache.mahout.cf.taste.neighborhood.UserNeighborhood;
    import org.apache.mahout.cf.taste.recommender.RecommendedItem;
    import org.apache.mahout.cf.taste.recommender.Recommender;
    import org.apache.mahout.cf.taste.similarity.UserSimilarity;
    
    public class RecommenderIntro {
    
        public static void main(String[] args) throws Exception {
            // 数据模型
            DataModel dataModel = new FileDataModel(new File("F:/mawork/demo1/src/main/java/com/bwjf/mahout/demo1/one/intro"));
            // 皮尔逊相关系数——相似度模型
            UserSimilarity similarity = new PearsonCorrelationSimilarity(dataModel);
            // 临近算法——用户领域模型
            UserNeighborhood neighborhood = new NearestNUserNeighborhood(2, similarity, dataModel);
            // 推荐引擎
            Recommender recommender = new GenericUserBasedRecommender(dataModel, neighborhood, similarity);
            List<RecommendedItem> recommendedItems =  recommender.recommend(1, 1);
            for (RecommendedItem recommendedItem: recommendedItems){
                System.out.println(recommendedItem);
            }
        }
    }

数据为

1,101,5.0
    1,102,3.0
    1,103,2.5
    2,101,2.0
    2,102,2.5
    2,103,5.0
    2,104,2.0
    3,101,2.5
    3,104,4.0
    3,105,4.5
    3,107,5.0
    4,101,5.0
    4,103,3.0
    4,104,4.5
    4,106,4.0
    5,101,4.0
    5,102,3.0
    5,103,2.0
    5,104,4.0
    5,105,3.5
    5,106,4.0

[Mahout中相似度计算方法介绍][Mahout 1]，上例用的是皮尔森相关度算法,  
**1.1 PearsonCorrelationSimilarity 皮尔斯相关系数**  
[如何理解皮尔逊相关系数（Pearson Correlation Coefficient）？][Pearson Correlation Coefficient]，两个变量之间的相关系数越高，从一个变量预测另一个变量的精确度也就越高。因为相关系数越高，变量之间的共变部分越多，[基于皮尔森相关系数的协同过滤算法][Link 1]，这篇文章解释的更为详细，做笔记给自己脑补。  
X、Y做线性相关假设，就有三种情况，无关系、正相关、负相关  
`org.apache.mahout.cf.taste.impl.similarity.AbstractSimilarity`中`userSimilarity`方法使用的是红色方框中的内容  
![5][]  
**1.2 GenericUserBasedRecommender**  
参考[GenericUserBasedRecommender推荐引擎源码分析][GenericUserBasedRecommender]  
[Mahout的taste推荐系统里的几种Recommender分析][Mahout_taste_Recommender]  
[【Machine Learning】Mahout基于协同过滤(CF)的用户推荐][Machine Learning_Mahout_CF]  
[Mahout推荐算法编程实践][Mahout 2]  
数学公式忘的差不多了，但是试想想如果你进入一个网站，你不知道你想要什么，但是你会希望别人怎么推荐给你商品呢？  
首先你告诉店主，你找几个跟我爱好类似的人吧。问题在于那些人与我的相似程度比较高呢？这里用到了皮尔逊相似系数算法，相关性系数越高，就属于同类爱好的人。  
接着说不定他们喜欢的，我也喜欢。这里就用到了临近算法，根据相似度的高低，找到K个与自己相似的邻居。  
然后你再告诉店主，和有类似爱好的人哪些商品的偏好值是多少，把最喜欢的推荐给我，说不定我也喜欢。  
**1.3 协方差与相关系数**  
[如何通俗易懂地解释「协方差」与「相关系数」的概念][Link 2]  
这篇文档我觉得写得非常清晰。  
**1.4 NearestNUserNeighborhood**  
这个紧邻代码也相对比较简单，主要采取优先队列，对皮尔逊相关系数进行排序，选出你觉得推荐的top N.  
[Java堆结构PriorityQueue完全解析][Java_PriorityQueue]

public static long[] getTopUsers(int howMany,
                                       LongPrimitiveIterator allUserIDs,
                                       IDRescorer rescorer,
                                       Estimator<Long> estimator) throws TasteException {
        Queue<SimilarUser> topUsers = new PriorityQueue<>(howMany + 1, Collections.reverseOrder());
        boolean full = false;
        double lowestTopValue = Double.NEGATIVE_INFINITY;
        while (allUserIDs.hasNext()) {
          long userID = allUserIDs.next();
          if (rescorer != null && rescorer.isFiltered(userID)) {
            continue;
          }
          double similarity;
          try {
            similarity = estimator.estimate(userID);
          } catch (NoSuchUserException nsue) {
            continue;
          }
          double rescoredSimilarity = rescorer == null ? similarity : rescorer.rescore(userID, similarity);
          if (!Double.isNaN(rescoredSimilarity) && (!full || rescoredSimilarity > lowestTopValue)) {
            topUsers.add(new SimilarUser(userID, rescoredSimilarity));
            if (full) {
              topUsers.poll();
            } else if (topUsers.size() > howMany) {
              full = true;
              topUsers.poll();
            }
            lowestTopValue = topUsers.peek().getSimilarity();
          }
        }
        int size = topUsers.size();
        if (size == 0) {
          return NO_IDS;
        }
        List<SimilarUser> sorted = new ArrayList<>(size);
        sorted.addAll(topUsers);
        Collections.sort(sorted);
        long[] result = new long[size];
        int i = 0;
        for (SimilarUser similarUser : sorted) {
          result[i++] = similarUser.getUserID();
        }
        return result;
      }

**2 RecommenderEvaluator**  
[mahout 返回主页 WEB系统架构与底层研发][mahout _ WEB]，我这篇文章对代码进行了分析，我就不再赘述，《MaHout实战》中返回结构是1.0，我的确实NaN，后来调整参数`double score = evalutor.evaluate(builder, null, dataModel, 0.9, 1.0);`，终于出现数据1.0了。  
**2.1 查准率和查全率**

相关 不相关
    检索到     A    C
    未检索到   B    D
    查全率=A/(A+B)
    查准率=A=(A+C)

`GenericRecommenderIRStatsEvaluator`定义评价好坏的阈值=μ（平均值）+σ（标准差）

[mahout]: http://mahout.apache.org/
[Mahout]: https://www.cnblogs.com/charlist/p/7063947.html
[Mahout 1]: https://www.cnblogs.com/dlts26/archive/2012/06/20/2555772.html
[Pearson Correlation Coefficient]: https://www.zhihu.com/question/19734616?sort=created
[Link 1]: https://www.cnblogs.com/hxsyl/p/4000756.html
[5]: /images/20220527/ca9922876b4548f1a27e724d7798a59e.png
[GenericUserBasedRecommender]: https://blog.csdn.net/ls386239766/article/details/42145289
[Mahout_taste_Recommender]: https://blog.csdn.net/zhoubl668/article/details/13297583
[Machine Learning_Mahout_CF]: https://www.cnblogs.com/yangxiao99/p/4753420.html
[Mahout 2]: https://blog.csdn.net/wolvesqun/article/details/52763966
[Link 2]: https://www.zhihu.com/question/20852004
[Java_PriorityQueue]: https://blog.csdn.net/u013309870/article/details/71189189
[mahout _ WEB]: https://blog.csdn.net/senvil/article/details/48916107