MongoDB Middle Level---> 大数据量下MapReduce取代GroupBy

超、凢脫俗 2022-06-18 06:37 144阅读 0赞

[MongoDB Middle Level---> 大数据量下MapReduce取代GroupBy][MongoDB Middle Level---_ _MapReduce_GroupBy]

转自：[http://blog.csdn.net/crazyjixiang/article/details/6619911][MongoDB Middle Level---_ _MapReduce_GroupBy]

ongoDB中的MapReduce其实更类似关系型数据库中的GroupBy 。

刚做了下这样试验，对于大数据量的GroupBy（MapReduce）还是比较理想的，生成100W条3位随机字符串

[view plain][] [print][view plain] [?][view plain]

1.  **for** (**var** i=0; i<1000000; i++) 
2.  \{ 
3.  **var** x = "0123456789"; 
4.  **var** tmp=""; 
5.  **for** (**var** j=0; j<3; j++) 
6.   \{ 
7.   tmp += x.charAt(Math.ceil(Math.random()\*100000000)%x.length);| 
8.   \} 
9.  **var** u = \{\_id:i,v1:tmp\}; 
10.  db.RandomNum.insert(u); 
11. \}

然后进行对相同的随机数取Count数 所以必须GroupBy

[view plain][] [print][view plain] [?][view plain]

1.  **var** m = **function**()\{emit(**this**.v1,\{count:1\}); \}; //map key类似关系型数据的group by 第二个是value 就是要进行聚合的字段(sum...)
2.  
3.  **var** r = **function** (key,values) \{ **var** total = 0;**for** (**var** i=0; i<values.length; i++) \{ total += values\[i\].count; \} **return** \{count : total\}; \};. //reduce
4.  
5.  **var** res = db.RandomNum.mapReduce(m, r, \{out:\{replace:'Result'\}\}); 
6.  
7.  db\[res.result\].find()

测试了下时间：

[view plain][] [print][view plain] [?][view plain]

1.  **var** startTime = **new** Date(); 
2.  
3.  **var** m = **function**()\{emit(**this**.v1,\{count:1\}); \}; 
4.  
5.  **var** r = **function** (key,values) \{ **var** total = 0;**for** (**var** i=0; i<values.length; i++) \{ total += values\[i\].count; \} **return** \{count : total\}; \}; 
6.  
7.  **var** res = db.RandomNum.mapReduce(m, r, \{out:\{replace:'Result'\}\}); 
8.  
9.  db\[res.result\].find() 
10. 
11. (**new** Date().getTime()-startTime.getTime())/1000

结果如下：

[view plain][] [print][view plain] [?][view plain]

1.  > db\[res.result\].find() 
2.  \{ "\_id" : "000", "value" : \{ "count" : 1075 \} \} 
3.  \{ "\_id" : "001", "value" : \{ "count" : 1045 \} \} 
4.  \{ "\_id" : "002", "value" : \{ "count" : 1022 \} \} 
5.  \{ "\_id" : "003", "value" : \{ "count" : 968 \} \} 
6.  \{ "\_id" : "004", "value" : \{ "count" : 994 \} \} 
7.  \{ "\_id" : "005", "value" : \{ "count" : 1009 \} \} 
8.  \{ "\_id" : "006", "value" : \{ "count" : 948 \} \} 
9.  \{ "\_id" : "007", "value" : \{ "count" : 1003 \} \} 
10. \{ "\_id" : "008", "value" : \{ "count" : 983 \} \} 
11. \{ "\_id" : "009", "value" : \{ "count" : 993 \} \} 
12. \{ "\_id" : "010", "value" : \{ "count" : 987 \} \} 
13. \{ "\_id" : "011", "value" : \{ "count" : 982 \} \} 
14. \{ "\_id" : "012", "value" : \{ "count" : 957 \} \} 
15. \{ "\_id" : "013", "value" : \{ "count" : 1031 \} \} 
16. \{ "\_id" : "014", "value" : \{ "count" : 971 \} \} 
17. \{ "\_id" : "015", "value" : \{ "count" : 1053 \} \} 
18. \{ "\_id" : "016", "value" : \{ "count" : 974 \} \} 
19. \{ "\_id" : "017", "value" : \{ "count" : 975 \} \} 
20. \{ "\_id" : "018", "value" : \{ "count" : 978 \} \} 
21. \{ "\_id" : "019", "value" : \{ "count" : 1010 \} \} 
22. has more 
23. > 
24. > (new Date().getTime()-startTime.getTime())/1000
25. 63.335s
26. > bye

测试机的性能：

![0_1318666857E01d.gif][]

[MongoDB Middle Level---_ _MapReduce_GroupBy]: http://blog.csdn.net/crazyjixiang/article/details/6619911
[view plain]: http://blog.csdn.net/crazyjixiang/article/details/6619911#
[0_1318666857E01d.gif]: /images/20220618/2b67d763df5f43bf8531239345723ad6.png