HttpClient官方sample代码的深刻分析(链接池)

前言

以前一直使用apache的httpclient(4.5.x), 进行http的交互处理. 而httpclient实例则使用了http链接池, 而一旦涉及到链接池, 那会不会在使用上有些隐藏很深的坑. 事实上, 经过分析httpclient源码, 发现它很优雅地解决了这个问题, 同时隐藏全部的链接池细节. 今天这边在这边作下笔记.html

官方代码片断

这是apache httpclient官网提供一段代码片断:java

CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://targethost/homepage");
CloseableHttpResponse response1 = httpclient.execute(httpGet);
// 链接对象被response对象持有, 以保证内容经过response对象消费
// 确保在finally代码块添加ClosableHttpResponse#close的调用
// 值得注意的是, 若是链接没有被彻底消费干净, 该链接将不能安全复用, 将会被关闭, 被链接池丢弃 
try {
    System.out.println(response1.getStatusLine());
    HttpEntity entity1 = response1.getEntity();
    // do something useful with the response body
    // and ensure it is fully consumed
    EntityUtils.consume(entity1);
} finally {
    response1.close();
}

简单分析下代码, 很是的简练, 你丝毫看不到任何链接池操做的蛛丝马迹, 它是怎么设计, 又是怎么作到的呢?c++

常规链接池的注意点

链接池的使用须要保证以下几点, 尤为对自研的链接池.
1. Connection的get/release配对.
2. 保证一次http交互中请求/响应处理完整干净(cleanup).
好比一次请求交互中, 因某种缘由没有消费掉响应内容, 致使该内容还处于socket的缓存中. 继而使得同一个链接下的第二次交互其响应内容为第一次的响应结果, 后果十分可怕. 之前作c++开发的时候, 封装编写redis链接池的时候, 就遇到相似的问题, 印象很是的深入.redis

链接封装

httpclient引入了ConnectionHolder类, 构建了真实链接(HttpCilentConnection)和链接池(HttpClientConnectionManager)的桥梁, 同时维护了该链接的可重用(reusable)和租赁(leased)状态.apache

class ConnectionHolder implements ConnectionReleaseTrigger, 
        Cancellable, Closeable {
    private final Log log;
    private final HttpClientConnectionManager manager;
    private final HttpClientConnection managedConn;
    private final AtomicBoolean released;  // 链接池租赁状态
    private volatile boolean reusable;     // 链接是否可复用
}

该类最重要的一个方法为releaseConnection, 后续的执行流程多多少少会涉及到该方法.缓存

private void releaseConnection(boolean reusable) {
    // *) 判断租赁状态, 若已归还链接池, 则再也不执行后续的代码
    if(this.released.compareAndSet(false, true)) {
        HttpClientConnection var2 = this.managedConn;
        synchronized(this.managedConn) {
            // *) 根据可重用性分状况处理, 同时归还到链接池中
            if(reusable) {
                this.manager.releaseConnection(this.managedConn, 
                        this.state, this.validDuration, this.tunit);
            } else {
                try {
                    // *) 关闭链接
                    this.managedConn.close();
                    this.log.debug("Connection discarded");
                } catch (IOException var9) {
                    if(this.log.isDebugEnabled()) {
                        this.log.debug(var9.getMessage(), var9);
                    }
                } finally {
                    this.manager.releaseConnection(this.managedConn, 
                        (Object)null, 0L, TimeUnit.MILLISECONDS);
                }
            }
        }
    }

}

而CloseableHttpResponse又持有ConnectionHolder对象, 它close方法, 本质上就是间接调用了ConnectionHolder的releaseConnection方法.安全

class HttpResponseProxy implements CloseableHttpResponse {

    public void close() throws IOException {
        if(this.connHolder != null) {
            this.connHolder.close();
        }
    }
}

class ConnectionHolder 
        implements ConnectionReleaseTrigger, Cancellable, Closeable {

    public void close() throws IOException {
        this.releaseConnection(false);
    }

}

因而可知, 官方sample的推荐作法, 在finally中保证ClosableHttpResponse#close的调用, 可以确保链接池的get/release配对. 如果close前, 链接状态依旧为租赁状态(leased为false), 则该链接明确不被复用.less

可重用性判断

http的长链接复用, 其断定规则主要分两类.
1. http协议支持+请求/响应header指定
2. 一次交互处理的完整性(响应内容消费干净)
对于前者, httpclient引入了ConnectionReuseStrategy来处理, 默认的采用以下的约定:socket

HTTP/1.0经过在Header中添加Connection:Keep-Alive来表示支持长链接.
HTTP/1.1默认支持长链接, 除非在Header中显式指定Connection:Close, 才被视为短链接模式.

在MainClientExec类中相关的代码片断:函数

var27 = this.requestExecutor.execute(request, managedConn, context);
if(this.reuseStrategy.keepAlive(var27, context)) {
    long entity = this.keepAliveStrategy.getKeepAliveDuration(var27, context);
    if(this.log.isDebugEnabled()) {
        String s;
        if(entity > 0L) {
            s = "for " + entity + " " + TimeUnit.MILLISECONDS;
        } else {
            s = "indefinitely";
        }

        this.log.debug("Connection can be kept alive " + s);
    }

    var25.setValidFor(entity, TimeUnit.MILLISECONDS);
    var25.markReusable();
} else {
    var25.markNonReusable();
}

具体ReusableStrategy中, 其执行代码以下:

public class DefaultClientConnectionReuseStrategy 
            extends DefaultConnectionReuseStrategy {
    public static final DefaultClientConnectionReuseStrategy INSTANCE 
            = new DefaultClientConnectionReuseStrategy();

    public DefaultClientConnectionReuseStrategy() {
    }

    public boolean keepAlive(HttpResponse response, HttpContext context) {
        HttpRequest request = (HttpRequest)context
              .getAttribute("http.request");
        if(request != null) {
            // *) 寻找Connection:Close
            Header[] connHeaders = request.getHeaders("Connection");
            if(connHeaders.length != 0) {
                BasicTokenIterator ti = new BasicTokenIterator(
                        new BasicHeaderIterator(connHeaders, (String)null)
                    );

                while(ti.hasNext()) {
                    String token = ti.nextToken();
                    if("Close".equalsIgnoreCase(token)) {
                        return false;
                    }
                }
            }
        }

        return super.keepAlive(response, context);
    }
}

 而在父类的keepAlive函数中, 其实现以下:

public class DefaultConnectionReuseStrategy 
        implements ConnectionReuseStrategy {

    public boolean keepAlive(HttpResponse response, HttpContext context) {
        // 省略一段代码
        if(headerIterator1.hasNext()) {
            try {
                BasicTokenIterator px1 = new BasicTokenIterator(headerIterator1);
                boolean keepalive1 = false;

                while(px1.hasNext()) {
                    String token = px1.nextToken();
                    // *) 存在Close Tag, 则不可重用
                    if("Close".equalsIgnoreCase(token)) {
                        return false;
                    }
                    // *) 存在Keep-Alive Tag 则可重用
                    if("Keep-Alive".equalsIgnoreCase(token)) {
                        keepalive1 = true;
                    }
                }

                if(keepalive1) {
                    return true;
                }
            } catch (ParseException var11) {
                return false;
            }
        }
        // 高于HTTP/1.0版本的都复用链接  
        return !ver1.lessEquals(HttpVersion.HTTP_1_0);
    }

}

总结一下:

request首部中包含Connection:Close，不复用
response中Content-Length长度设置不正确，不复用
response首部包含Connection:Close，不复用
reponse首部包含Connection:Keep-Alive，复用
都没命中的状况下，若是HTTP版本高于1.0则复用

而对于后者(一次交互处理的完整性), 这是怎么断定的呢? 其实很简单, 就是response返回的InputStream(HttpEntity#getContent)明确调用close方法(没有引起socket的close), 即认为消费完整.
让咱们来简单分析一下EntityUtils.consume方法.

public final class EntityUtils {

    public static void consume(HttpEntity entity) throws IOException {
        if(entity != null) {
            if(entity.isStreaming()) {
                InputStream instream = entity.getContent();
                if(instream != null) {
                    instream.close();
                }
            }
        }
    }

}

让咱们在ConnectionHolder类的releaseConnection方法中添加断点.

而后具体执行一个http请求, 咱们会发现程序运行到该断点时的, 线程调用堆栈以下:

"main@1" prio=5 tid=0x1 nid=NA runnable
  java.lang.Thread.State: RUNNABLE
      at org.apache.http.impl.execchain.ConnectionHolder.releaseConnection(ConnectionHolder.java:97)
      at org.apache.http.impl.execchain.ConnectionHolder.releaseConnection(ConnectionHolder.java:120)
      at org.apache.http.impl.execchain.ResponseEntityProxy.releaseConnection(ResponseEntityProxy.java:76)
      at org.apache.http.impl.execchain.ResponseEntityProxy.streamClosed(ResponseEntityProxy.java:145)
      at org.apache.http.conn.EofSensorInputStream.checkClose(EofSensorInputStream.java:228)
      at org.apache.http.conn.EofSensorInputStream.close(EofSensorInputStream.java:172)
      at org.apache.http.client.entity.LazyDecompressingInputStream.close(LazyDecompressingInputStream.java:97)
      at org.apache.http.util.EntityUtils.consume(EntityUtils.java:90)

你会发现inputstream#close的调用, 会引起链接的归还, 而此时reusable状态值为true(前提KeepaliveStrategy判断该链接为可复用).
再额外添加一个Apache HttpClient中定义的ContentLengthInputStream类的close实现, 用于明确close会附带消费完数据, 以此打消最后的疑惑.

public class ContentLengthInputStream extends InputStream {

    // *) 该close会把剩余的字节所有消费, 才设定本身为关闭状态
    public void close() throws IOException {
        if(!this.closed) {
            try {
                if(this.pos < this.contentLength) {
                    byte[] buffer = new byte[2048];

                    while(true) {
                        if(this.read(buffer) >= 0) {
                            continue;
                        }
                    }
                }
            } finally {
                this.closed = true;
            }
        }

    }

}

总结

让咱们再回到最初的官方sample代码.

CloseableHttpClient httpclient = HttpClients.createDefault();
HttpGet httpGet = new HttpGet("http://targethost/homepage");
CloseableHttpResponse response1 = httpclient.execute(httpGet);
try {
    System.out.println(response1.getStatusLine());
    HttpEntity entity1 = response1.getEntity();

    // *) 引起releaseConnect()调用, reusable值取决于keepAliveStrategy断定, leased置为true
    EntityUtils.consume(entity1);
} finally {
    // *) 若链接leased为false, 则releaseConnect(false)调用, 明确不可复用, leased置为true
    // *) 若链接leased为true, 则do nothing
    response1.close();
}

c++会使用RAII模式, 即利用对象的构造/析构函数来自动实现资源申请和释放, java这边的话, 仍是须要明确的一个finally中, 添加保证释放的代码, ^_^.
总的来讲, 该段代码, 堪称完美. 对于官方推荐的代码, 放心大胆的使用便可.

参考文章

Http持久链接与HttpClient链接池
 关于HttpClient重试策略的研究