Spark计算相关性系数（皮尔森、斯皮尔曼、卡方检验）

Bertha 。 2022-05-11 10:44 730阅读 0赞

![70][]

![70 1][]

皮尔森、斯皮尔曼（pearson spearman）：

import spark.implicits.\_  
   import org.apache.spark.mllib.stat.Statistics  
   import spark.sql    
   val df = sql(s"select \* from xxxx ")  
      
   val columns = List("xx","xx","xx")  
   for(col <- columns)\{  
        
     val df\_real = df.select("label", col)   
     val rdd\_real = df\_real.rdd.map(x=>(x(0).toString.toDouble ,x(1).toString.toDouble ))  
     val label = rdd\_real.map(x=>x.\_1.toDouble )  
     val feature = rdd\_real.map(x=>x.\_2.toDouble )  
       
     val cor\_pearson:Double = Statistics.corr(label, feature, "pearson")  
     println( s"$\{col\}------" + cor\_pearson )  
   
     val cor\_spearman:Double = Statistics.corr(label, feature, "spearman")  
     println(s"$\{col\}------" + cor\_spearman )  
   \}

卡方检验计算卡方值：

import org.apache.spark.mllib.linalg.\{Matrix, Matrices, Vectors \}  
   import org.apache.spark.mllib.regression.LabeledPoint   
   import org.apache.spark.mllib.stat.Statistics  
   import spark.implicits.\_   
   import spark.sql  
   val df\_real = sql(s"select \* from  xxxx ")  
      
    val columns = List("xx", "xx","xx","xx" )  
    
  val featInd = columns.map(df\_real.columns.indexOf(\_))  
  val targetInd = df\_real.columns.indexOf("label")   
  val lp\_data = df\_real.rdd.map(r => LabeledPoint(  
   r.getString(targetInd).toDouble,     
   Vectors.dense(featInd.map(r.getString(\_).toDouble).toArray)   
))  
   val vd=Statistics.chiSqTest(lp\_data)   
       
  vd.foreach(x=>println(x.statistic))  
  columns.foreach(println(\_))

[70]: /images/20220505/a5a9f9d8da6347628a4dd0809cdac71c.png
[70 1]: /images/20220505/175dcf09aada4a5cb7743f868ba14115.png