大文件拷贝，试试 NIO 的内存映射

逃离我推掉我的手 2021-09-01 03:45 387阅读 0赞

点击上方 [Java后端][Java]，选择 设为星标

优质文章，及时送达

--------------------

作者： 何甜甜在吗

链接：juejin.im/post/6844904051516129287

最近项目里有个需求需要实现文件拷贝，在java中文件拷贝流的读写，很容易就想到IO中的InputStream和OutputStream之类的，但是上网查了一下文件拷贝也是有很多种方法的，除了IO，还有NIO、Apache提供的工具类、JDK自带的文件拷贝方法

### IO拷贝 ###

public class IOFileCopy {
    
        private static final int BUFFER_SIZE = 1024;
    
        public static void copyFile(String source, String target) {
            long start = System.currentTimeMillis();
            try(InputStream in = new FileInputStream(new File(source));
                OutputStream out = new FileOutputStream(new File(target))) {
                byte[] buffer = new byte[BUFFER_SIZE];
                int len;
                while ((len = in.read(buffer)) > 0) {
                    out.write(buffer, 0, len);
                }
    
                System.out.println(String.format("IO file copy cost %d msc", System.currentTimeMillis() - start));
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
    }

传统IO中文件读取过程可以分为以下几步：

内核从磁盘读取数据到缓冲区，这个过程由磁盘操作器通过DMA操作将数据从磁盘读取到内核缓冲区，该过程**不依赖CPU**

用户进程在将数据从内核缓冲区**拷贝**到用户空间缓冲区

用户进程从用户空间缓冲区读取数据

![format_png][]

### NIO拷贝 ###

NIO进行文件拷贝有两种实现方式，一是通过管道，而是通过文件内存内存映射

public class NIOFileCopy {
    
        public static void copyFile(String source, String target) {
            long start = System.currentTimeMillis();
            try(FileChannel input = new FileInputStream(new File(source)).getChannel();
                FileChannel output = new FileOutputStream(new File(target)).getChannel()) {
                output.transferFrom(input, 0, input.size());
            } catch (Exception e) {
                e.printStackTrace();
            }
    
            System.out.println(String.format("NIO file copy cost %d msc", System.currentTimeMillis() - start));
        }
    }

**文件内存映射：**

把内核空间地址与用户空间的虚拟地址**映射到同一个物理地址**，DMA 硬件可以填充对内核与用户空间进程同时可见的缓冲区了。用户进程直接**从内存中读取**文件内容，应用只需要和内存打交道，不需要进行缓冲区来回拷贝，大大提高了IO拷贝的效率。加载内存映射文件**所使用的内存在Java堆区之外**

public class NIOFileCopy2 {
    
        public static void copyFile(String source, String target) {
            long start = System.currentTimeMillis();
            try(FileInputStream fis = new FileInputStream(new File(source));
                FileOutputStream fos = new FileOutputStream(new File(target))) {
                FileChannel sourceChannel = fis.getChannel();
                FileChannel targetChannel = fos.getChannel();
                MappedByteBuffer mappedByteBuffer = sourceChannel.map(FileChannel.MapMode.READ_ONLY, 0, sourceChannel.size());
                targetChannel.write(mappedByteBuffer);
            } catch (FileNotFoundException e) {
                e.printStackTrace();
            } catch (IOException e) {
                e.printStackTrace();
            }
    
            System.out.println(String.format("NIO memory reflect file copy cost %d msc", System.currentTimeMillis() - start));
            File targetFile = new File(target);
            targetFile.delete();
        }
    }

NIO内存映射文件拷贝可以分为以下几步

![format_png 1][]

NIO的内存映射实际上就是少了一次从内核空间拷贝到用户空间的过程，将对用户缓冲区的读改为从内存读取

### Files\#copyFile方法 ###

public class FilesCopy {
    
        public static void copyFile(String source, String target) {
            long start = System.currentTimeMillis();
            try {
                File sourceFile = new File(source);
                File targetFile = new File(target);
                Files.copy(sourceFile.toPath(), targetFile.toPath());
            } catch (IOException e) {
                e.printStackTrace();
            }
    
            System.out.println(String.format("FileCopy file copy cost %d msc", System.currentTimeMillis() - start));
        }
    }

### FileUtils\#copyFile方法 ###

使用FileUtils之前需先引入依赖

依赖

<dependency>
        <groupId>commons-io</groupId>
        <artifactId>commons-io</artifactId>
        <version>2.4</version>
    </dependency>

FileUtils\#copyFile封装类：FileUtilsCopy.java

public class FileUtilsCopy {
    
        public static void copyFile(String source, String target) {
            long start = System.currentTimeMillis();
            try {
                FileUtils.copyFile(new File(source), new File(target));
            } catch (IOException e) {
                e.printStackTrace();
            }
    
            System.out.println(String.format("FileUtils file copy cost %d msc", System.currentTimeMillis() - start));
        }
    }

性能比较

既然有这么多种实现方法，肯定要从中选择性能最佳的

**测试环境：**

*  windows 10
 *  CPU 6核
 *  JDK1.8

**测试代码：PerformTest.java**

public class PerformTest {
    
        private static final String source1 = "input/test1.txt";
        private static final String source2 = "input/test2.txt";
        private static final String source3 = "input/test3.txt";
        private static final String source4 = "input/test4.txt";
        private static final String target1 = "output/test1.txt";
        private static final String target2 = "output/test2.txt";
        private static final String target3 = "output/test3.txt";
        private static final String target4 = "output/test4.txt";
    
        public static void main(String[] args) {
            IOFileCopy.copyFile(source1, target1);
            NIOFileCopy.copyFile(source2, target2);
            FilesCopy.copyFile(source3, target3);
            FileUtilsCopy.copyFile(source4, target4);
        }
    }

总共执行了五次，读写的文件大小分别为9KB、23KB、239KB、1.77MB、12.7MB

![format_png 2][]

注意：单位均为毫秒

从执行结果来看：

文件很小时 =>  IO > NIO【内存映射】> NIO【管道】 > Files\#copy > FileUtils\#copyFile

在文件较小时 => NIO【内存映射】> IO > NIO【管道】 > Files\#copy > FileUtils\#copyFile

在文件较大时 => NIO【内存映射】> > NIO【管道】> IO  > Files\#copy > FileUtils\#copyFile

修改IO缓冲区大小对拷贝效率有影响，但是并不是越大性能越好，稍大于拷贝文件大小即可

文件较小时，IO效率高于NIO，NIO底层实现较为复杂，NIO的优势不明显。同时NIO内存映射初始化耗时，所以在文件较小时和IO复制相比没有优势

如果追求效率可以选择NIO的内存映射去实现文件拷贝，但是对于大文件使用内存映射拷贝要格外关注系统内存的使用率。推荐：**大文件**拷贝使用**内存映射**，原文是这样的:

For most operating systems, mapping a file into memory is more
    expensive than reading or writing a few tens of kilobytes of data via
    the usual {@link #read read} and {@link #write write} methods.  From the
    standpoint of performance it is generally only worth mapping relatively
    large files into memory

绝大多数操作系统的内存映射开销大于IO开销

同时通过测试结果来看，工具类和JDK提供的文件复制方法效果并不高，如果不追求效率还是可以使用一下，毕竟能少写一行代码就少写一行代码，写代码没有摸鱼来的快乐

https://github.com/TiantianUpup/java-learning/tree/master/src/main/java/com/h2t/study/io

**- END -**

最近整理一份面试资料**《Java技术栈学习手册》**，覆盖了Java技术、面试题精选、Spring全家桶、Nginx、SSM、微服务、数据库、数据结构、架构等等。

获取方式：点“ **在看**，关注公众号 **Java后端** 并回复 **777** 领取，更多内容陆续奉上。

**推****荐****阅****读 **

***1.*** [Linux纯干货知识总结 ][Linux]

***2. ***[Docker 部署不香吗？][Docker]

***3.*** [为什么不建议在 MySQL 中使用 UTF-8？][MySQL _ UTF-8]

***4.*** [Linux 思维导图整理][Linux 1]

***5.*** [15000 字的 SQL 语句大全][15000 _ SQL]

![format_png 3][]

**喜****欢****文****章****，****点****个****在看 **![format_png 4][]

[Java]: http://mp.weixin.qq.com/s?__biz=Mzg2MjEwMjI1Mg%3D%3D&chksm=ce0da17bf97a286df119047a25b7de51621fe542816673bb4aa3ebd8ab97b09a86ed7db2e2ee&idx=3&mid=2247489272&scene=21&sn=ce74938072fc73a808d83ccb474db46d#wechat_redirect
[format_png]: /images/20210730/b6c508a79f8c4b1182bbb382e9569cda.png
[format_png 1]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vVEtIYzZGOHRzaWFqaWF6ZW9QYU94Q0hKdGFLR2ZFQ1g2SEZyU0FFNXhlcThncnhvWGVlemhpY2hHY25sZHNsVENpYkJib3g3Qk1IaWF3R1ZYcUlnaWIwbUVUZy82NDA?x-oss-process=image/format,png
[format_png 2]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X3BuZy9vVEtIYzZGOHRzaWFqaWF6ZW9QYU94Q0hKdGFLR2ZFQ1g2b2lhbVdnRjNTRERka1dWTnNWbFRtRmlhR2VLdHI1aDRNRTdxTUZpY0lWMW04YVpQS2liMEIzVHpJQS82NDA?x-oss-process=image/format,png
[Linux]: http://mp.weixin.qq.com/s?__biz=Mzg2MjEwMjI1Mg%3D%3D&chksm=ce0e4b4ef979c258e0795c4cd967d6a7e5f52ce39675fef4b3e23bbe2a2e8fb7dc7579f3f092&idx=2&mid=2247498957&scene=21&sn=82b542bc6bd8da13f1a34ff6747e25a3#wechat_redirect
[Docker]: http://mp.weixin.qq.com/s?__biz=Mzg2MjEwMjI1Mg%3D%3D&chksm=ce0e4b04f979c2127522c757d1d26fb1422aa94233686b2bad753555d41dbe3b552b3396fd13&idx=1&mid=2247498887&scene=21&sn=5d07989c92f61c486ed708722a3719aa#wechat_redirect
[MySQL _ UTF-8]: http://mp.weixin.qq.com/s?__biz=Mzg2MjEwMjI1Mg%3D%3D&chksm=ce0e4bdff979c2c96e27b60576d6b93cf105263677ae3c3086e568a17c3a8f80a268f630ade7&idx=2&mid=2247498844&scene=21&sn=0847b6f7d97de43503bf654790d7cd8a#wechat_redirect
[Linux 1]: http://mp.weixin.qq.com/s?__biz=Mzg2MjEwMjI1Mg%3D%3D&chksm=ce0e4bdff979c2c976060aa19f5e20c04eef70bd4a75dd43ce4ab7ca5f376d0639990b196927&idx=4&mid=2247498844&scene=21&sn=919afb08997fbc5cb80d11bb15955604#wechat_redirect
[15000 _ SQL]: http://mp.weixin.qq.com/s?__biz=Mzg2MjEwMjI1Mg%3D%3D&chksm=ce0e4ca6f979c5b0ccf6f2944a0d63b966e9d65f862cbe38ca91b256760a7cffb491206004fc&idx=1&mid=2247498533&scene=21&sn=3cd9b05080bfff3d3e0481eb2410c92d#wechat_redirect
[format_png 3]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2pwZy9vVEtIYzZGOHRzaGlhRVQyZVMwU2NnRXM5ejl6dTdpYU9EWjRiYkxmRUkwNmQ3ZE5RaWNTalR4YjFGWWNaUVVWcm9saWJXRE41WWliMWliUXZ6NnFCaWNEOGJJeXcvNjQw?x-oss-process=image/format,png
[format_png 4]: https://imgconvert.csdnimg.cn/aHR0cHM6Ly9tbWJpei5xcGljLmNuL21tYml6X2pwZy9DMXVETURxam4xOXd0dVFwSzNqbUpXM2JGR1dJOFl6NkZSMTd0bDFNRjhWZnFZeFB4OTkwa3YySjc0THZxd2liMjZLYXlIZE9YZDZlYnpycVlpYmJUand3LzY0MA?x-oss-process=image/format,png