Java多线程下载文件

悠悠 2022-06-11 08:20 245阅读 0赞

# Java多线程下载文件 #

优化：合理利用服务器资源，将资源利用最大化，加快下载速度

一般有两种方式：

*  线程池里面有N个线程，多线程下载单个文件，将网络路径的文件流切割成多快，每个线程下载一小部分，然后写入到文件里面，组成一个文件
 *  当有很多个文件需要下载的时候，调用某个方法，有个线程池，线程池大小假定是10，当有10个文件过来的时候，每个线程去下载一个文件即可

多线程如果只知道Thread和Runnable，那你就Out了，Java1.5以后有个concurrent包，就是Java并发编程。

## N个线程下载单个文件 ##

定义一个单个线程下载的部分

public class DownloadWithRange implements Runnable { 
        private String urlLocation;
    
        private String filePath;
    
        private long start;
    
        private long end;
    
        DownloadWithRange(String urlLocation, String filePath, long start, long end) {
            this.urlLocation = urlLocation;
            this.filePath = filePath;
            this.start = start;
            this.end = end;
        }
    
        @Override
        public void run() {
            try {
                HttpURLConnection conn = getHttp();
                conn.setRequestProperty("Range", "bytes=" + start + "-" + end);
    
                File file = new File(filePath);
                RandomAccessFile out = null;
                if (file != null) {
                    out = new RandomAccessFile(file, "rw");
                }
                out.seek(start);
                InputStream in = conn.getInputStream();
                byte[] b = new byte[1024];
                int len = 0;
                while ((len = in.read(b)) >= 0) {
                    out.write(b, 0, len);
                }
                in.close();
                out.close();
            } catch (Exception e) {
                e.getMessage();
            }
    
        }
    
        public HttpURLConnection getHttp() throws IOException {
            URL url = null;
            if (urlLocation != null) {
                url = new URL(urlLocation);
            }
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            conn.setReadTimeout(5000);
            conn.setRequestMethod("GET");
    
            return conn;
        }
    
    }

定义线程Pool

public class DownloadFileWithThreadPool {
    
        public void getFileWithThreadPool(String urlLocation, String filePath, int poolLength) throws IOException {
            ExecutorService threadPool = Executors.newCachedThreadPool();
    
            long len = getContentLength(urlLocation);
            System.out.println(len);
            for (int i = 0; i < poolLength; i++) {
                long start = i * len / poolLength;
                long end = (i + 1) * len / poolLength - 1;
                if (i == poolLength - 1) {
                    end = len;
                }
                System.out.println(start+"---------------"+end);
                DownloadWithRange download = new DownloadWithRange(urlLocation, filePath, start, end);
                threadPool.execute(download);
            }
            threadPool.shutdown();
        }
    
        public static long getContentLength(String urlLocation) throws IOException {
            URL url = null;
            if (urlLocation != null) {
                url = new URL(urlLocation);
            }
            HttpURLConnection conn = (HttpURLConnection) url.openConnection();
            conn.setReadTimeout(5000);
            conn.setRequestMethod("GET");
            long len = conn.getContentLength();
    
            return len;
        }
    
    }

测试下载单个文件

public static void main(String[] args) {
            Date startDate = new Date();
            DownloadFileWithThreadPool pool = new DownloadFileWithThreadPool();
            try {
                pool.getFileWithThreadPool("http://mpge.5nd.com/2016/2016-11-15/74847/1.mp3", "D:\\1.mp3", 100);
            } catch (IOException e) {
                e.printStackTrace();
            }
            System.out.println(new Date().getTime() - startDate.getTime());
        }

测试发现，我目前的网速，下载一个3.6M左右mp3音频文件，使用多线程，下载大概需要800ms左右，而不使用多线程，则需要1600ms以上，有时候甚至3000ms

## 多文件下载，单文件单线程 ##

/** * @author Nick 带线程池的文件下载类，线程大小10 * 文件数少的情况下，体现不大 * @version V1.0.0 * @Date 2017/8/2 20:43 */
    public class FileDownConnManager { 
    
        private static final Logger logger = LoggerFactory.getLogger(FileDownConnManager.class);
    
        private static final FileDownConnManager connManager = new FileDownConnManager();
    
        private static ExecutorService executorService = Executors.newFixedThreadPool(10); //10个线程跑
    
        public static FileDownConnManager getDefaultManager() {
            return connManager;
        }
    
        public static byte[] fileDown(final String netURL) throws ExecutionException, InterruptedException {
            Future<byte[]> future = executorService.submit(new Callable<byte[]>() {
                @Override
                public byte[] call() throws Exception {
                    Date date = new Date();
                    URL url;
                    byte[] getData = new byte[0];
                    InputStream is = null;
                    try {
                        url = new URL(netURL);
                        URLConnection connection = url.openConnection();
                        is = connection.getInputStream();
                        getData = readInputStream(is);
    
                    } catch (IOException e) {
                        logger.error("从URL获得字节流数组失败 " + ExceptionUtils.getMessage(e));
                    } finally {
                        try {
                            is.close();
                        } catch (IOException e) {
                            logger.error("从URL获得字节流数组流释放失败 " + ExceptionUtils.getMessage(e));
                        }
                    }
                    return getData;
                }
            });
            return future.get();
        }
    
    }

测试

@Test
        public void test2() throws ExecutionException, InterruptedException, IOException {
            long time1 = System.currentTimeMillis();
            for(int i = 0; i < 15; i++) {
                byte[] by1 = FileDownConnManager.fileDown("http://mpge.5nd.com/2016/2016-11-15/74847/1.mp3");
                FileUtils.writeByteArrayToFile(new File("D:\\test2_"+i+".mp3"), by1);
            }
            System.out.println(System.currentTimeMillis() - time1);
        }
    
        @Test
        public void test3() throws IOException {
            long time1 = System.currentTimeMillis();
            for(int i = 0; i < 15; i++) {
                byte[] by1 = FileFromUrlUtil.getInputStreamFromUrl("http://mpge.5nd.com/2016/2016-11-15/74847/1.mp3");
                FileUtils.writeByteArrayToFile(new File("D:\\test3_"+i+".mp3"), by1);
            }
            System.out.println(System.currentTimeMillis() - time1);
        }

结论：同样的文件，测试发现，在一定数量(文件数量大概15)的文件下载的时候，每个文件是3.6M的情况下，使用多线程大概时间是30xxxms，而不实用多线程大概是38xxxms，还是有一定时间上的提升

此处的测试还是会存在一定的误差，因为ExecutorService线程池的创建是需要时间，类的初始化也是需要一定的时间，假如在Web应用中，第一次创建出来了，第二次不需要创建，优势还是稍微能够体现出来！！