Java IO与NIO实现文件拷贝

- 日理万妓 2022-06-10 12:09 121阅读 0赞

做这个实验是因为最近要做传输，虽然方向上定的是用Mina的IoBuffer来进行传值，但是在系统对接和文件备份上都要用到拷贝，原生的IO和NIO也是一个不错的选择。做这个实验主要的目的还是熟悉一下各种写法，并比较一下性能。

其实，说心里话，我一直是希望NIO有一个很不错的效率，能够大大的领先“旧”IO，可是实验的结果却并不如我所愿。我采用了如下几种文件传输方式：FileInputStream、BufferedInputStream、BufferedReader、RandomAccessFile以及FileChannel。由于这里是单线程的测试，所以没有用到管道流。

第一轮对300MB的压缩文件做复制和粘贴，结果BufferedInputStream完胜其他几个方式。速度几乎接近了系统自身的复制和粘贴。而NIO并没有想象的那么出色，和其他几种方式时间相差并不大。于是我不死心，开始第二轮的测试。文件大小为3个GB。直接导致了FileChannel中自带的transferFrom方法报错。而其他方法时间也大致相同，谁都没有特别优势。

下面贴出实验结果：

/** * a.rar 336MB * 采用传统IO FileInputStream 读取，耗时：2372 * 采用传统IO BufferedInputStream 读取，耗时：699 * 采用传统IO RandomAccessFile 读取，耗时：2302 * 采用NIO FileChannel 自带方法 读取，耗时：2724 * 采用NIO FileChannel 循环 读取，耗时：2077 * * Red Hat Enterprise Linux 5 64-bit (2).vmdk 3GB * 采用传统IO FileInputStream 读取，耗时：83519 * 采用传统IO BufferedInputStream 读取，耗时：89459 * 采用传统IO RandomAccessFile 读取，耗时：97847 * 采用NIO FileChannel 循环 读取，耗时：88136 * */

出现这样的结果也很正常，JDK已经优化了传统IO的实现方式，大部分底层都采用了NIO的方式来实现，所以差距并不会很明显。而第一次试验BufferedInputStream之所以能完胜，主要我想还是因为操作系统内存管理的问题，页的频繁切换也会产生极大地消耗。所以其实在平常的应用中旧IO也足够用了。

但是对于高并发的系统，IO始终是瓶颈，而这笔开销主要还是在于复制上。这里虽然也用到了transferTo这样的API，但是由于测试环境在WIN8上，所以并没有体现其优势，在Kafka项目的介绍中有一段关于文件Copy的说明：

Maintaining this common format allows optimization of the most important operation: network transfer of persistent log chunks. Modern unix operating systems offer a highly optimized code path for transferring data out of pagecache to a socket; in Linux this is done with the sendfile system call. Java provides access to this system call with the FileChannel.transferTo api.
    To understand the impact of sendfile, it is important to understand the common data path for transfer of data from file to socket:
    1.	The operating system reads data from the disk into pagecache in kernel space
    2.	The application reads the data from kernel space into a user-space buffer
    3.	The application writes the data back into kernel space into a socket buffer
    4.	The operating system copies the data from the socket buffer to the NIC buffer where it is sent over the network
    This is clearly inefficient, there are four copies, two system calls. Using sendfile, this re-copying is avoided by allowing the OS to send the data from pagecache to the network directly. So in this optimized path, only the final copy to the NIC buffer is needed.
    We expect a common use case to be multiple consumers on a topic. Using the zero-copy optimization above, data is copied into pagecache exactly once and reused on each consumption instead of being stored in memory and copied out to kernel space every time it is read. This allows messages to be consumed at a rate that approaches the limit of the network connection.
    For more background on the sendfile and zero-copy support in Java, see this article on IBM developerworks.

在linux系统下可以采用epoll技术区实现，具体的内容可以参考IBM的这篇论文[https://www.ibm.com/developerworks/linux/library/j-zerocopy/][https_www.ibm.com_developerworks_linux_library_j-zerocopy]。文件的顺序写基本上能满足性能上的要求，但是一旦涉及到随即读写，或者有了并发性的读写，这些都需要不断的改进和测试了。

下面贴出测试代码：

package com.a2.desktop.example3.fileread;
    
    import java.io.BufferedInputStream;
    import java.io.BufferedOutputStream;
    import java.io.BufferedReader;
    import java.io.BufferedWriter;
    import java.io.File;
    import java.io.FileInputStream;
    import java.io.FileOutputStream;
    import java.io.FileReader;
    import java.io.FileWriter;
    import java.io.RandomAccessFile;
    import java.io.Reader;
    import java.io.Writer;
    import java.nio.ByteBuffer;
    import java.nio.channels.FileChannel;
    
    /**
     * 文件拷贝各种方式的比较
     * 
     * @author Chen.Hui
     * 
     */
    public class TestFileTransport {
    
    	 public final static String FILE_PATH =
    	 "F:\\RedHetServer\\Red Hat Enterprise Linux 5 64-bit (2).vmdk";
    
    	//public final static String FILE_PATH = "F:\\apache-tomcat-7.0.11\\webapps\\ROOT\\a.rar";
    
    	public final static String FILE_PATH_OUT = "C:\\Red Hat Enterprise Linux 5 64-bit (2).vmdk";
    
    	public static void TransByCommonIoStream() throws Exception {
    
    		long beginTime = System.currentTimeMillis();
    
    		FileInputStream fis = new FileInputStream(new File(FILE_PATH));
    
    		FileOutputStream fos = new FileOutputStream(new File(FILE_PATH_OUT));
    
    		byte[] b = new byte[1024];
    
    		int len = 0;
    
    		while ((len = fis.read(b)) != -1) {
    			fos.write(b, 0, len);
    		}
    
    		fos.flush();
    
    		fis.close();
    		fos.close();
    
    		long endTime = System.currentTimeMillis();
    
    		System.out.println("采用传统IO FileInputStream 读取，耗时："
    				+ (endTime - beginTime));
    
    	}
    
    	public static void TransByCommonIoBufferedStream() throws Exception {
    
    		long beginTime = System.currentTimeMillis();
    
    		FileInputStream fis = new FileInputStream(new File(FILE_PATH));
    
    		FileOutputStream fos = new FileOutputStream(new File(FILE_PATH_OUT));
    
    		BufferedInputStream bis = new BufferedInputStream(fis);
    
    		BufferedOutputStream bos = new BufferedOutputStream(fos);
    
    		byte[] b = new byte[1024];
    
    		int len = 0;
    
    		while ((len = bis.read(b)) != -1) {
    			bos.write(b, 0, len);
    		}
    
    		bos.flush();
    
    		fis.close();
    		fos.close();
    		bis.close();
    		bos.close();
    
    		long endTime = System.currentTimeMillis();
    
    		System.out.println("采用传统IO BufferedInputStream 读取，耗时："
    				+ (endTime - beginTime));
    
    	}
    
    	public static void TransByCommonIoBuffered() throws Exception {
    
    		long beginTime = System.currentTimeMillis();
    
    		Reader br = new BufferedReader(new FileReader(new File(FILE_PATH)));
    		Writer bw = new BufferedWriter(new FileWriter(new File(FILE_PATH_OUT)));
    
    		char[] c = new char[1024];
    
    		int len = 0;
    
    		while ((len = br.read(c)) != -1) {
    			bw.write(c, 0, len);
    		}
    
    		bw.flush();
    		br.close();
    		bw.close();
    
    		long endTime = System.currentTimeMillis();
    
    		System.out.println("采用传统IO  BufferedReader 读取，耗时："
    				+ (endTime - beginTime));
    	}
    
    	public static void TransByRandomAccFile() throws Exception {
    
    		long beginTime = System.currentTimeMillis();
    
    		FileInputStream fis = new FileInputStream(new File(FILE_PATH));
    
    		RandomAccessFile raf = new RandomAccessFile(new File(FILE_PATH_OUT),
    				"rw");
    
    		byte[] b = new byte[1024];
    
    		int len = 0;
    
    		while ((len = fis.read(b)) != -1) {
    			raf.write(b, 0, len);
    		}
    
    		long endTime = System.currentTimeMillis();
    
    		System.out.println("采用传统IO RandomAccessFile 读取，耗时："
    				+ (endTime - beginTime));
    
    	}
    
    	/**
    	 * 采用FileChannel 自带方法测试 public abstract long
    	 * transferFrom(ReadableByteChannel src, long position, long count) throws
    	 * IOException;
    	 */
    	public static void TransByNioFileChannel() throws Exception {
    
    		long beginTime = System.currentTimeMillis();
    
    		FileChannel fc = new FileInputStream(new File(FILE_PATH)).getChannel();
    
    		FileChannel fco = new RandomAccessFile(new File(FILE_PATH_OUT), "rw")
    				.getChannel();
    
    		fco.transferFrom(fc, 0, fc.size());
    
    		long endTime = System.currentTimeMillis();
    
    		System.out.println("采用NIO FileChannel 自带方法  读取，耗时："
    				+ (endTime - beginTime));
    	}
    
    	public static void TransByNioFileChannelCommon() throws Exception {
    
    		long beginTime = System.currentTimeMillis();
    
    		FileChannel fc = new FileInputStream(new File(FILE_PATH)).getChannel();
    
    		FileChannel fco = new RandomAccessFile(new File(FILE_PATH_OUT), "rw")
    				.getChannel();
    
    		ByteBuffer buf = ByteBuffer.allocate(1024);
    
    		while (fc.read(buf) != -1) {
    			buf.flip();
    			fco.write(buf);
    			buf.clear();
    		}
    
    		long endTime = System.currentTimeMillis();
    
    		System.out.println("采用NIO FileChannel 循环 读取，耗时："
    				+ (endTime - beginTime));
    	}
    
    	public static void deleteFile() {
    		File f = new File(FILE_PATH_OUT);
    		if (f.exists())
    			f.delete();
    	}
    
    	public static void main(String[] args) throws Exception {
    
    		TransByCommonIoStream();
    		deleteFile();
    		TransByCommonIoBufferedStream();
    		deleteFile();
    		TransByRandomAccFile();
    //		deleteFile();
    //		TransByNioFileChannel();
    		deleteFile();
    		TransByNioFileChannelCommon();
    		deleteFile();
    	}
    
    	/**
    	 * a.rar 336MB
    	 * 采用传统IO FileInputStream 读取，耗时：2372 
    	 * 采用传统IO BufferedInputStream 读取，耗时：699
    	 * 采用传统IO RandomAccessFile 读取，耗时：2302
    	 * 采用NIO FileChannel 自带方法 读取，耗时：2724
    	 * 采用NIO FileChannel 循环 读取，耗时：2077
    	 * 
    	 * Red Hat Enterprise Linux 5 64-bit (2).vmdk 3GB
    	 * 采用传统IO FileInputStream 读取，耗时：83519
    	 * 采用传统IO BufferedInputStream 读取，耗时：89459
    	 * 采用传统IO RandomAccessFile 读取，耗时：97847
    	 * 采用NIO FileChannel 循环 读取，耗时：88136
    	 * 
    	 */
    }

测试主要是为了追求更高的性能，最近研究性的东西做的比较多，今天做完了Mina和C的对接，明天要做协议的解析，但愿能有新的发现。

[https_www.ibm.com_developerworks_linux_library_j-zerocopy]: https://www.ibm.com/developerworks/linux/library/j-zerocopy/