记一次MongoDB性能问题(从MySQL迁移到MongoDB)

痛定思痛。 2023-10-17 16:57 129阅读 0赞

最近忙着把一个项目从MySQL迁移到MongoDB，在导入旧数据的过程中，遇到了些许波折，犯了不少错误，但同时也学到了不少知识，遂记录下来,需要的朋友可以参考下

公司为这个项目专门配备了几台高性能务器，清一色的双路四核超线程CPU，外加32G内存，运维人员安装好MongoDB后，就交我手里了，我习惯于在使用新服务器前先看看相关日志，了解一下基本情况，当我浏览MongoDB日志时，发现一些警告信息：

WARNING: You are running on a NUMA machine. We suggest launching mongod like this to avoid performance problems: numactl –interleave=all mongod \[other options\]

当时我并不太清楚[NUMA][]是什么东西，所以没有处理，只是把问题反馈给了运维人员，后来知道运维人员也没有理会这茬儿，所以问题的序幕就这样拉开了。

迁移工作需要导入旧数据。MongoDB本身有一个[mongoimport][]工具可供使用，不过它只接受json、csv等格式的源文件，不适合我的需求，所以我没用，而是用PHP写了一个脚本，平稳运行了一段时间后，我发现数据导入的速度下降了，同时PHP抛出异常：

cursor timed out (timeout: 30000, time left: 0:0, status: 0)

我一时判断不出问题所在，想想先在PHP脚本里加大Timeout的值应付一下：

可惜这样并没有解决问题，错误反倒变着花样的出现了：

max number of retries exhausted, couldn't send query, couldn't send query: Broken pipe

接着使用strace跟踪了一下PHP脚本，发现进程卡在了recvfrom操作上：

通过如下命令查询recvfrom操作的含义：

或者按照下面的方式确认一下：

此时如果查询MongoDB的[当前操作][Link 1]，会发现几乎每个操作会消耗大量的时间：

与此同时，运行[mongostat][]的话，结果会显示很高的locked值。

…

我在网络上找到一篇：[MongoDB Pre-Splitting for Faster Data Loading and Importing][]，看上去和我的问题很类似，不过他的问题实质是由于自动分片导致数据迁移所致，解决方法是使用手动分片，而我并没有使用自动分片，自然不是这个原因。

…

询问了几个朋友，有人反映曾遇到过类似的问题，在他的场景里，问题的主要原因是系统IO操作繁忙时，[数据文件预分配][Link 2]堵塞了其它操作，从而导致雪崩效应。

为了验证这种可能，我搜索了一下MongoDB日志：

我使用的文件系统是ext4（xfs也不错 ），创建数据文件非常快，所以不是这个原因，但如果有人使用ext3，可能会遇到这类问题，所以还是大概介绍一下如何解决：

MongoDB按需自动生成数据文件：先是<DB>.0，大小是64M，然后是<DB>.1，大小翻番到128M，到了<DB>.5，大小翻番到2G，其后的数据文件就保持在2G大小。为了避免可能出现的问题，可以采用事先手动创建数据文件的策略：

<table style="border-collapse:collapse; border-spacing:0px; border:1px solid rgb(204,204,204); margin-top:0.54em; text-align:center; width:660px; clear:both; background:none!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.5em!important; margin-right:0px!important; margin-bottom:1px!important; margin-left:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-size:0.1em!important; min-height:auto!important"> 
 <tbody style="background:none!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.5em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; text-align:left!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:0.1em!important; min-height:auto!important"> 
  <tr style="background:none!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.5em!important; margin:0px!important; outline:0px!important; overflow:visible!important; padding:0px!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-size:0.1em!important; min-height:auto!important"> 
   <td style="padding:0.2em 0.46em; border:1px solid rgb(204,204,204); margin:0px!important; background:none!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.5em!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-size:0.1em!important; min-height:auto!important; color:rgb(175,175,175)!important"> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      1 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      2 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      3 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      4 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      5 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      6 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      7 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      8 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      9 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      10 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      11 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      12 
    </div> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px 0.5em 0px 1em!important; border-width:0px 3px 0px 0px!important; border-top-style:initial!important; border-right-style:solid!important; border-bottom-style:initial!important; border-left-style:initial!important; border-top-color:initial!important; border-right-color:rgb(108,226,108)!important; border-bottom-color:initial!important; border-left-color:initial!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; text-align:right!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
      13 
    </div> </td> 
   <td style="padding:0.2em 0.46em; border:1px solid rgb(204,204,204); width:636px; margin:0px!important; background:none!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; line-height:1.5em!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-size:0.1em!important; min-height:auto!important"> 
    <div style="line-height:12px; font-size:14px; margin:0px!important; padding:0px!important; background:none!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:relative!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important"> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:gray!important">#!/bin/sh</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
       &nbsp; 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">DB_NAME=$1</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
       &nbsp; 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important; color:rgb(255,20,147)!important">cd</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important"> /path/to/</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">$DB_NAME</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
       &nbsp; 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:rgb(0,102,153)!important">for</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important"> INDEX_NUMBER </code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:rgb(0,102,153)!important">in</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important"> {5..50}; </code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:rgb(0,102,153)!important">do</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">&nbsp;&nbsp;</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">FILE_NAME=$DB_NAME.$INDEX_NUMBER</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important">
       &nbsp; 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">&nbsp;&nbsp;</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:rgb(0,102,153)!important">if</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important"> [ ! -e $FILE_NAME ]; </code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:rgb(0,102,153)!important">then</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">&nbsp;&nbsp;&nbsp;&nbsp;</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important; color:rgb(255,20,147)!important">head</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important"> -c 2146435072 </code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">/dev/zero</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important"> &gt; $FILE_NAME</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; min-height:auto!important">&nbsp;&nbsp;</code> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:rgb(0,102,153)!important">fi</code> 
     </div> 
     <div style="line-height:12px; margin:0px!important; padding:0px 1em!important; border:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; width:auto!important; min-height:auto!important; white-space:pre!important"> 
      <code style="margin:3px auto 0px; padding:0px 0px 0px 5px; border-left:3px solid rgb(108,226,108); line-height:20px; width:640px; clear:both; border-top:0px!important; border-right:0px!important; border-bottom:0px!important; bottom:auto!important; float:none!important; height:auto!important; left:auto!important; outline:0px!important; overflow:visible!important; position:static!important; right:auto!important; top:auto!important; vertical-align:baseline!important; font-family:Consolas,&quot;Bitstream Vera Sans Mono&quot;,&quot;Courier New&quot;,Courier,monospace!important; font-weight:bold!important; min-height:auto!important; color:rgb(0,102,153)!important">done</code> 
     </div> 
    </div> </td> 
  </tr> 
 </tbody> 
</table>

注：数值2146435072并不是标准的2G，这是INT整数范围决定的。

…

最后一个求助方式就是[官方论坛][Link 3]了，那里的国际友人建议我检查一下是不是索引不佳所致，死马当活马医，我激活了[Profiler][]记录慢操作：

不过结果显示基本都是insert操作（因为我是导入数据为主），本身就不需要索引：

…

问题始终没有得到解决，求人不如求己，我又重复了几次迁移旧数据的过程，结果自然还是老样子，但我发现每当出问题的时候，总有一个名叫[irqbalance][]的进程CPU占用率居高不下，搜索了一下，发现很多介绍irqbalance的文章中都提及了NUMA，让我一下子想起之前在日志中看到的警告信息，我勒个去，竟然绕了这么大一个圈圈！安下心来仔细翻阅文档，发现官方其实已经有了[相关介绍][Link 4]，按如下设置搞定：

关于zone\_reclaim\_mode内核参数的说明，可以参考[官方文档][Link 5]。

注：从[MongoDB1.9.2][]开始：MongoDB会在启动时自动设置zone\_reclaim\_mode。

至于NUMA的含义，简单点说，在有多个物理CPU的架构下，NUMA把内存分为本地和远程，每个物理CPU都有属于自己的本地内存，访问本地内存速度快于访问远程内存，缺省情况下，每个物理CPU只能访问属于自己的本地内存。对于MongoDB这种需要大内存的服务来说就可能造成内存不足，NUMA的详细介绍，可以参考[老外的文章][Link 6]。

理论上，MySQL、Redis、Memcached等等都可能会受到NUMA的影响，需要留意。

来源:http://www.jb51.net/article/109198.htm

[NUMA]: http://en.wikipedia.org/wiki/Non-Uniform_Memory_Access
[mongoimport]: http://www.mongodb.org/display/DOCS/Import+Export+Tools#ImportExportTools-mongoimport
[Link 1]: http://www.mongodb.org/display/DOCS/Viewing+and+Terminating+Current+Operation
[mongostat]: http://www.mongodb.org/display/DOCS/mongostat
[MongoDB Pre-Splitting for Faster Data Loading and Importing]: http://blog.zawodny.com/2011/03/06/mongodb-pre-splitting-for-faster-data-loading-and-importing/
[Link 2]: http://www.mongodb.org/display/DOCS/Excessive+Disk+Space#ExcessiveDiskSpace-DatafilePreallocation
[Link 3]: http://groups.google.com/group/mongodb-user
[Profiler]: http://www.mongodb.org/display/DOCS/Database+Profiler
[irqbalance]: http://irqbalance.org/
[Link 4]: http://www.mongodb.org/display/DOCS/NUMA
[Link 5]: http://www.kernel.org/doc/Documentation/sysctl/vm.txt
[MongoDB1.9.2]: https://jira.mongodb.org/browse/SERVER/fixforversion/10380
[Link 6]: http://jcole.us/blog/archives/2010/09/28/mysql-swap-insanity-and-the-numa-architecture/