HttpClient4.5使用RequestConfig配置连接信息

落日映苍穹つ 2022-05-13 15:10 224阅读 0赞

本文作者：合肥工业大学 管理学院 钱洋 email：1563178220@qq.com 内容可能有不到之处，欢迎交流。

**未经本人允许禁止转载**。

### 文章目录 ###

*  RequestConfig类
 *  超时相关配置
 *  代理配置

# RequestConfig类 #

在构建网络爬虫时，经常需要配置很多信息，例如RequestTimeout（连接池获取到连接的超时时间）、ConnectTimeout（建立连接的超时）、SocketTimeout（获取数据的超时时间)、代理、是否允许重定向等信息。  
在HttpClient，实现这些配置需要使用到RequestConfig类的一个内部类Builder。如下为Builder的源码：

public static class Builder { 
    
            private boolean expectContinueEnabled;
            private HttpHost proxy;
            private InetAddress localAddress;
            private boolean staleConnectionCheckEnabled;
            private String cookieSpec;
            private boolean redirectsEnabled;
            private boolean relativeRedirectsAllowed;
            private boolean circularRedirectsAllowed;
            private int maxRedirects;
            private boolean authenticationEnabled;
            private Collection<String> targetPreferredAuthSchemes;
            private Collection<String> proxyPreferredAuthSchemes;
            private int connectionRequestTimeout;
            private int connectTimeout;
            private int socketTimeout;
            private boolean contentCompressionEnabled;
    
            Builder() { 
                super();
                this.staleConnectionCheckEnabled = false;
                this.redirectsEnabled = true;
                this.maxRedirects = 50;
                this.relativeRedirectsAllowed = true;
                this.authenticationEnabled = true;
                this.connectionRequestTimeout = -1;
                this.connectTimeout = -1;
                this.socketTimeout = -1;
                this.contentCompressionEnabled = true;
            }
    
            public Builder setExpectContinueEnabled(final boolean expectContinueEnabled) { 
                this.expectContinueEnabled = expectContinueEnabled;
                return this;
            }
    
            public Builder setProxy(final HttpHost proxy) { 
                this.proxy = proxy;
                return this;
            }
    
            public Builder setLocalAddress(final InetAddress localAddress) { 
                this.localAddress = localAddress;
                return this;
            }
    
            /** * @deprecated (4.4) Use {@link * org.apache.http.impl.conn.PoolingHttpClientConnectionManager#setValidateAfterInactivity(int)} */
            @Deprecated
            public Builder setStaleConnectionCheckEnabled(final boolean staleConnectionCheckEnabled) { 
                this.staleConnectionCheckEnabled = staleConnectionCheckEnabled;
                return this;
            }
    
            public Builder setCookieSpec(final String cookieSpec) { 
                this.cookieSpec = cookieSpec;
                return this;
            }
    
            public Builder setRedirectsEnabled(final boolean redirectsEnabled) { 
                this.redirectsEnabled = redirectsEnabled;
                return this;
            }
    
            public Builder setRelativeRedirectsAllowed(final boolean relativeRedirectsAllowed) { 
                this.relativeRedirectsAllowed = relativeRedirectsAllowed;
                return this;
            }
    
            public Builder setCircularRedirectsAllowed(final boolean circularRedirectsAllowed) { 
                this.circularRedirectsAllowed = circularRedirectsAllowed;
                return this;
            }
    
            public Builder setMaxRedirects(final int maxRedirects) { 
                this.maxRedirects = maxRedirects;
                return this;
            }
    
            public Builder setAuthenticationEnabled(final boolean authenticationEnabled) { 
                this.authenticationEnabled = authenticationEnabled;
                return this;
            }
    
            public Builder setTargetPreferredAuthSchemes(final Collection<String> targetPreferredAuthSchemes) { 
                this.targetPreferredAuthSchemes = targetPreferredAuthSchemes;
                return this;
            }
    
            public Builder setProxyPreferredAuthSchemes(final Collection<String> proxyPreferredAuthSchemes) { 
                this.proxyPreferredAuthSchemes = proxyPreferredAuthSchemes;
                return this;
            }
    
            public Builder setConnectionRequestTimeout(final int connectionRequestTimeout) { 
                this.connectionRequestTimeout = connectionRequestTimeout;
                return this;
            }
    
            public Builder setConnectTimeout(final int connectTimeout) { 
                this.connectTimeout = connectTimeout;
                return this;
            }
    
            public Builder setSocketTimeout(final int socketTimeout) { 
                this.socketTimeout = socketTimeout;
                return this;
            }
    
            /** * @deprecated (4.5) Set {@link #setContentCompressionEnabled(boolean)} to {@code false} and * add the {@code Accept-Encoding} request header. */
            @Deprecated
            public Builder setDecompressionEnabled(final boolean decompressionEnabled) { 
                this.contentCompressionEnabled = decompressionEnabled;
                return this;
            }
    
            public Builder setContentCompressionEnabled(final boolean contentCompressionEnabled) { 
                this.contentCompressionEnabled = contentCompressionEnabled;
                return this;
            }
    
            public RequestConfig build() { 
                return new RequestConfig(
                        expectContinueEnabled,
                        proxy,
                        localAddress,
                        staleConnectionCheckEnabled,
                        cookieSpec,
                        redirectsEnabled,
                        relativeRedirectsAllowed,
                        circularRedirectsAllowed,
                        maxRedirects,
                        authenticationEnabled,
                        targetPreferredAuthSchemes,
                        proxyPreferredAuthSchemes,
                        connectionRequestTimeout,
                        connectTimeout,
                        socketTimeout,
                        contentCompressionEnabled);
            }
    
        }

# 超时相关配置 #

HttpClient中可设置三个超时：RequestTimeout（连接池获取到连接的超时时间）、ConnectTimeout（建立连接的超时）、SocketTimeout（获取数据的超时时间）。使用RequestConfig进行配置的示例程序如下：

//全部设置为10秒
    		RequestConfig requestConfig = RequestConfig.custom()
    				.setSocketTimeout(10000)
    				.setConnectTimeout(10000)
    				.setConnectionRequestTimeout(10000)
    				.build();
    		//配置httpClient
    		HttpClient httpClient = HttpClients.custom()
    				.setDefaultRequestConfig(requestConfig)
    				.build();

# 代理配置 #

RequestConfig defaultRequestConfig = RequestConfig.custom()
    				.setProxy(new HttpHost("171.97.67.160", 3128, null))
    				.build();   //添加代理
    		HttpClient httpClient = HttpClients.custom().
    				setDefaultRequestConfig(defaultRequestConfig).build();  //配置httpClient