class Net::HTTP::Persistent

Persistent connections for Net::HTTP

Net::HTTP::Persistent maintains persistent connections across all the servers you wish to talk to. For each host:port you communicate with a single persistent connection is created.

Multiple Net::HTTP::Persistent objects will share the same set of connections.

For each thread you start a new connection will be created. A Net::HTTP::Persistent connection will not be shared across threads.

You can shut down the HTTP connections when done by calling shutdown. You should name your Net::HTTP::Persistent object if you intend to call this method.

Example:

require 'net/http/persistent'

uri = URI 'http://example.com/awesome/web/service'

http = Net::HTTP::Persistent.new 'my_app_name'

# perform a GET
response = http.request uri

# create a POST
post_uri = uri + 'create'
post = Net::HTTP::Post.new post_uri.path
post.set_form_data 'some' => 'cool data'

# perform the POST, the URI is always required
response http.request post_uri, post

SSL

SSL connections are automatically created depending upon the scheme of the URI. SSL connections are automatically verified against the default certificate store for your computer. You can override this by changing #verify_mode or by specifying an alternate cert_store.

Here are the SSL settings, see the individual methods for documentation:

certificate

This client's certificate

ca_file

The certificate-authority

cert_store

An SSL certificate store

private_key

The client's SSL private key

reuse_ssl_sessions

Reuse a previously opened SSL session for a new connection

ssl_version

Which specific SSL version to use

verify_callback

For server certificate verification

verify_mode

How connections should be verified

Proxies

A proxy can be set through proxy= or at initialization time by providing a second argument to ::new. The proxy may be the URI of the proxy server or :ENV which will consult environment variables.

See proxy= and proxy_from_env for details.

Headers

Headers may be specified for use in every request. headers are appended to any headers on the request. override_headers replace existing headers on the request.

The difference between the two can be seen in setting the User-Agent. Using http.headers['User-Agent'] = 'MyUserAgent' will send "Ruby, MyUserAgent" while http.override_headers['User-Agent'] = 'MyUserAgent' will send "MyUserAgent".

Tuning

Segregation

By providing an application name to ::new you can separate your connections from the connections of other applications.

Idle Timeout

If a connection hasn't been used for this number of seconds it will automatically be reset upon the next use to avoid attempting to send to a closed connection. The default value is 5 seconds. nil means no timeout. Set through idle_timeout.

Reducing this value may help avoid the "too many connection resets" error when sending non-idempotent requests while increasing this value will cause fewer round-trips.

Read Timeout

The amount of time allowed between reading two chunks from the socket. Set through read_timeout

Open Timeout

The amount of time to wait for a connection to be opened. Set through open_timeout.

Socket Options

Socket options may be set on newly-created connections. See socket_options for details.

Non-Idempotent Requests

By default non-idempotent requests will not be retried per RFC 2616. By setting #retry_change_requests to true requests will automatically be retried once.

Only do this when you know that retrying a POST or other non-idempotent request is safe for your application and will not create duplicate resources.

The recommended way to handle non-idempotent requests is the following:

require 'net/http/persistent'

uri = URI 'http://example.com/awesome/web/service'
post_uri = uri + 'create'

http = Net::HTTP::Persistent.new 'my_app_name'

post = Net::HTTP::Post.new post_uri.path
# ... fill in POST request

begin
  response = http.request post_uri, post
rescue Net::HTTP::Persistent::Error

  # POST failed, make a new request to verify the server did not process
  # the request
  exists_uri = uri + '...'
  response = http.get exists_uri

  # Retry if it failed
  retry if response.code == '404'
end

The method of determining if the resource was created or not is unique to the particular service you are using. Of course, you will want to add protection from infinite looping.

Connection Termination

If you are done using the Net::HTTP::Persistent instance you may shut down all the connections in the current thread with shutdown. This is not recommended for normal use, it should only be used when it will be several minutes before you make another HTTP request.

If you are using multiple threads, call shutdown in each thread when the thread is done making requests. If you don't call shutdown, that's OK. Ruby will automatically garbage collect and shutdown your HTTP connections when the thread terminates.

Public Class Methods

new(name = nil, proxy = nil) click to toggle source

Creates a new Net::HTTP::Persistent.

Set name to keep your connections apart from everybody else's. Not required currently, but highly recommended. Your library name should be good enough. This parameter will be required in a future version.

proxy may be set to a URI::HTTP or :ENV to pick up proxy options from the environment. See #proxy_from_env for details.

In order to use a URI for the proxy you may need to do some extra work beyond URI parsing if the proxy requires a password:

proxy = URI 'http://proxy.example'
proxy.user     = 'AzureDiamond'
proxy.password = 'hunter2'
# File lib/net/http/persistent.rb, line 374
def initialize name = nil, proxy = nil
  @name = name

  @debug_output     = nil
  @proxy_uri        = nil
  @headers          = {}
  @override_headers = {}
  @http_versions    = {}
  @keep_alive       = 30
  @open_timeout     = nil
  @read_timeout     = nil
  @idle_timeout     = 5
  @socket_options   = []

  @socket_options << [Socket::IPPROTO_TCP, Socket::TCP_NODELAY, 1] if
    Socket.const_defined? :TCP_NODELAY

  key = ['net_http_persistent', name].compact
  @generation_key     = [key, 'generations'    ].join('_').intern
  @ssl_generation_key = [key, 'ssl_generations'].join('_').intern
  @request_key        = [key, 'requests'       ].join('_').intern
  @timeout_key        = [key, 'timeouts'       ].join('_').intern

  @certificate        = nil
  @ca_file            = nil
  @private_key        = nil
  @ssl_version        = nil
  @verify_callback    = nil
  @verify_mode        = OpenSSL::SSL::VERIFY_PEER
  @cert_store         = nil

  @generation         = 0 # incremented when proxy URI changes
  @ssl_generation     = 0 # incremented when SSL session variables change
  @reuse_ssl_sessions = OpenSSL::SSL.const_defined? :Session

  @retry_change_requests = false

  self.proxy = proxy if proxy
end

Public Instance Methods

ca_file=(file) click to toggle source

Sets the SSL certificate authority file.

# File lib/net/http/persistent.rb, line 426
def ca_file= file
  @ca_file = file

  reconnect_ssl
end
can_retry?(req) click to toggle source

Is the request idempotent or is #retry_change_requests allowed

# File lib/net/http/persistent.rb, line 605
def can_retry? req
  retry_change_requests or idempotent?(req)
end
cert_store=(store) click to toggle source

Overrides the default SSL certificate store used for verifying connections.

# File lib/net/http/persistent.rb, line 436
def cert_store= store
  @cert_store = store

  reconnect_ssl
end
certificate=(certificate) click to toggle source

Sets this client's OpenSSL::X509::Certificate

# File lib/net/http/persistent.rb, line 417
def certificate= certificate
  @certificate = certificate

  reconnect_ssl
end
connection_close?(header) click to toggle source

Workaround for missing Net::HTTPHeader#connection_close? on Ruby 1.8

# File lib/net/http/persistent.rb, line 613
def connection_close? header
  header.connection_close?
end
connection_for(uri) click to toggle source

Creates a new connection for uri

# File lib/net/http/persistent.rb, line 466
def connection_for uri
  Thread.current[@generation_key]     ||= Hash.new { |h,k| h[k] = {} }
  Thread.current[@ssl_generation_key] ||= Hash.new { |h,k| h[k] = {} }
  Thread.current[@request_key]        ||= Hash.new 0
  Thread.current[@timeout_key]        ||= Hash.new EPOCH

  use_ssl = uri.scheme.downcase == 'https'

  if use_ssl then
    ssl_generation = @ssl_generation

    ssl_cleanup ssl_generation

    connections = Thread.current[@ssl_generation_key][ssl_generation]
  else
    generation = @generation

    cleanup generation

    connections = Thread.current[@generation_key][generation]
  end

  net_http_args = [uri.host, uri.port]
  connection_id = net_http_args.join ':'

  if @proxy_uri then
    connection_id << @proxy_connection_id
    net_http_args.concat @proxy_args
  end

  connection = connections[connection_id]

  unless connection = connections[connection_id] then
    connections[connection_id] = http_class.new(*net_http_args)
    connection = connections[connection_id]
    ssl connection if use_ssl
  else
    reset connection if expired? connection
  end

  unless connection.started? then
    connection.set_debug_output @debug_output if @debug_output
    connection.open_timeout = @open_timeout if @open_timeout
    connection.read_timeout = @read_timeout if @read_timeout

    connection.start

    socket = connection.instance_variable_get :@socket

    if socket then # for fakeweb
      @socket_options.each do |option|
        socket.io.setsockopt(*option)
      end
    end
  end

  connection
rescue Errno::ECONNREFUSED
  raise Error, "connection refused: #{connection.address}:#{connection.port}"
rescue Errno::EHOSTDOWN
  raise Error, "host down: #{connection.address}:#{connection.port}"
end
connection_keep_alive?(header) click to toggle source

Workaround for missing Net::HTTPHeader#connection_keep_alive? on Ruby 1.8

# File lib/net/http/persistent.rb, line 620
def connection_keep_alive? header
  header.connection_keep_alive?
end
error_message(connection) click to toggle source

Returns an error message containing the number of requests performed on this connection

# File lib/net/http/persistent.rb, line 533
def error_message connection
  requests = Thread.current[@request_key][connection.object_id] - 1 # fixup
  last_use = Thread.current[@timeout_key][connection.object_id]

  age = Time.now - last_use

  "after #{requests} requests on #{connection.object_id}, "        "last used #{age} seconds ago"
end
escape(str) click to toggle source

URI::escape wrapper

# File lib/net/http/persistent.rb, line 546
def escape str
  CGI.escape str if str
end
expired?(connection) click to toggle source

Returns true if the connection should be reset due to an idle timeout, false otherwise.

# File lib/net/http/persistent.rb, line 554
def expired? connection
  return false unless @idle_timeout
  return true  if     @idle_timeout.zero?

  last_used = Thread.current[@timeout_key][connection.object_id]

  Time.now - last_used > @idle_timeout
end
finish(connection, thread = Thread.current) click to toggle source

Finishes the Net::HTTP connection

# File lib/net/http/persistent.rb, line 566
def finish connection, thread = Thread.current
  if requests = thread[@request_key] then
    requests.delete connection.object_id
  end

  connection.finish
rescue IOError
end
http_version(uri) click to toggle source

Returns the HTTP protocol version for uri

# File lib/net/http/persistent.rb, line 587
def http_version uri
  @http_versions["#{uri.host}:#{uri.port}"]
end
idempotent?(req) click to toggle source

Is req idempotent according to RFC 2616?

# File lib/net/http/persistent.rb, line 594
def idempotent? req
  case req
  when Net::HTTP::Delete, Net::HTTP::Get, Net::HTTP::Head,
       Net::HTTP::Options, Net::HTTP::Put, Net::HTTP::Trace then
    true
  end
end
normalize_uri(uri) click to toggle source

Adds "http://" to the String uri if it is missing.

# File lib/net/http/persistent.rb, line 653
def normalize_uri uri
  (uri =~ %r^https?:/) ? uri : "http://#{uri}"
end
pipeline(uri, requests) { |responses| ... } click to toggle source

Pipelines requests to the HTTP server at uri yielding responses if a block is given. Returns all responses recieved.

See Net::HTTP::Pipeline for further details.

Only if net-http-pipeline was required before net-http-persistent pipeline will be present.

# File lib/net/http/persistent.rb, line 668
def pipeline uri, requests, &block # :yields: responses
  connection = connection_for uri

  connection.pipeline requests, &block
end
private_key=(key) click to toggle source

Sets this client's SSL private key

# File lib/net/http/persistent.rb, line 677
def private_key= key
  @private_key = key

  reconnect_ssl
end
proxy=(proxy) click to toggle source

Sets the proxy server. The proxy may be the URI of the proxy server, the symbol :ENV which will read the proxy from the environment or nil to disable use of a proxy. See proxy_from_env for details on setting the proxy from the environment.

If the proxy URI is set after requests have been made, the next request will shut-down and re-open all connections.

If you are making some requests through a proxy and others without a proxy use separate Net::Http::Persistent instances.

# File lib/net/http/persistent.rb, line 695
def proxy= proxy
  @proxy_uri = case proxy
               when :ENV      then proxy_from_env
               when URI::HTTP then proxy
               when nil       then # ignore
               else raise ArgumentError, 'proxy must be :ENV or a URI::HTTP'
               end

  if @proxy_uri then
    @proxy_args = [
      @proxy_uri.host,
      @proxy_uri.port,
      @proxy_uri.user,
      @proxy_uri.password,
    ]

    @proxy_connection_id = [nil, *@proxy_args].join ':'
  end

  reconnect
  reconnect_ssl
end
proxy_from_env() click to toggle source

Creates a URI for an HTTP proxy server from ENV variables.

If HTTP_PROXY is set a proxy will be returned.

If HTTP_PROXY_USER or HTTP_PROXY_PASS are set the URI is given the indicated user and password unless HTTP_PROXY contains either of these in the URI.

For Windows users, lowercase ENV variables are preferred over uppercase ENV variables.

# File lib/net/http/persistent.rb, line 730
def proxy_from_env
  env_proxy = ENV['http_proxy'] || ENV['HTTP_PROXY']

  return nil if env_proxy.nil? or env_proxy.empty?

  uri = URI normalize_uri env_proxy

  unless uri.user or uri.password then
    uri.user     = escape ENV['http_proxy_user'] || ENV['HTTP_PROXY_USER']
    uri.password = escape ENV['http_proxy_pass'] || ENV['HTTP_PROXY_PASS']
  end

  uri
end
reconnect() click to toggle source

Forces reconnection of HTTP connections.

# File lib/net/http/persistent.rb, line 748
def reconnect
  @generation += 1
end
reconnect_ssl() click to toggle source

Forces reconnection of SSL connections.

# File lib/net/http/persistent.rb, line 755
def reconnect_ssl
  @ssl_generation += 1
end
request(uri, req = nil, &block) click to toggle source

Makes a request on uri. If req is nil a Net::HTTP::Get is performed against uri.

If a block is passed request behaves like Net::HTTP#request (the body of the response will not have been read).

req must be a Net::HTTPRequest subclass (see Net::HTTP for a list).

If there is an error and the request is idempontent according to RFC 2616 it will be retried automatically.

# File lib/net/http/persistent.rb, line 787
def request uri, req = nil, &block
  retried      = false
  bad_response = false

  req = Net::HTTP::Get.new uri.request_uri unless req

  @headers.each do |pair|
    req.add_field(*pair)
  end

  @override_headers.each do |name, value|
    req[name] = value
  end

  unless req['Connection'] then
    req.add_field 'Connection', 'keep-alive'
    req.add_field 'Keep-Alive', @keep_alive
  end

  connection = connection_for uri
  connection_id = connection.object_id

  begin
    Thread.current[@request_key][connection_id] += 1
    response = connection.request req, &block

    if connection_close?(req) or
       (response.http_version <= '1.0' and
        not connection_keep_alive?(response)) or
       connection_close?(response) then
      connection.finish
    end
  rescue Net::HTTPBadResponse => e
    message = error_message connection

    finish connection

    raise Error, "too many bad responses #{message}" if
      bad_response or not can_retry? req

    bad_response = true
    retry
  rescue IOError, EOFError, Timeout::Error,
         Errno::ECONNABORTED, Errno::ECONNRESET, Errno::EPIPE,
         Errno::EINVAL, OpenSSL::SSL::SSLError => e

    if retried or not can_retry? req
      due_to = "(due to #{e.message} - #{e.class})"
      message = error_message connection

      finish connection

      raise Error, "too many connection resets #{due_to} #{message}"
    end

    reset connection

    retried = true
    retry
  ensure
    Thread.current[@timeout_key][connection_id] = Time.now
  end

  @http_versions["#{uri.host}:#{uri.port}"] ||= response.http_version

  response
end
reset(connection) click to toggle source

Finishes then restarts the Net::HTTP connection

# File lib/net/http/persistent.rb, line 762
def reset connection
  Thread.current[@request_key].delete connection.object_id
  Thread.current[@timeout_key].delete connection.object_id

  finish connection

  connection.start
rescue Errno::ECONNREFUSED
  raise Error, "connection refused: #{connection.address}:#{connection.port}"
rescue Errno::EHOSTDOWN
  raise Error, "host down: #{connection.address}:#{connection.port}"
end
shutdown(thread = Thread.current) click to toggle source

Shuts down all connections for thread.

Uses the current thread by default.

If you've used Net::HTTP::Persistent across multiple threads you should call this in each thread when you're done making HTTP requests.

NOTE: Calling shutdown for another thread can be dangerous!

If the thread is still using the connection it may cause an error! It is best to call shutdown in the thread at the appropriate time instead!

# File lib/net/http/persistent.rb, line 868
def shutdown thread = Thread.current
  generation = reconnect
  cleanup generation, thread, @generation_key

  ssl_generation = reconnect_ssl
  cleanup ssl_generation, thread, @ssl_generation_key

  thread[@request_key] = nil
  thread[@timeout_key] = nil
end
shutdown_in_all_threads() click to toggle source

Shuts down all connections in all threads

NOTE: THIS METHOD IS VERY DANGEROUS!

Do not call this method if other threads are still using their connections! Call shutdown at the appropriate time instead!

Use this method only as a last resort!

# File lib/net/http/persistent.rb, line 889
def shutdown_in_all_threads
  Thread.list.each do |thread|
    shutdown thread
  end

  nil
end
ssl(connection) click to toggle source

Enables SSL on connection

# File lib/net/http/persistent.rb, line 900
  def ssl connection
    connection.use_ssl = true

    connection.ssl_version = @ssl_version if @ssl_version

    connection.verify_mode = @verify_mode

    if OpenSSL::SSL::VERIFY_PEER == OpenSSL::SSL::VERIFY_NONE and
       not Object.const_defined?(:I_KNOW_THAT_OPENSSL_VERIFY_PEER_EQUALS_VERIFY_NONE_IS_WRONG) then
      warn "                             !!!SECURITY WARNING!!!

The SSL HTTP connection to:

  #{connection.address}:#{connection.port}

                           !!!MAY NOT BE VERIFIED!!!

On your platform your OpenSSL implementation is broken.

There is no difference between the values of VERIFY_NONE and VERIFY_PEER.

This means that attempting to verify the security of SSL connections may not
work.  This exposes you to man-in-the-middle exploits, snooping on the
contents of your connection and other dangers to the security of your data.

To disable this warning define the following constant at top-level in your
application:

  I_KNOW_THAT_OPENSSL_VERIFY_PEER_EQUALS_VERIFY_NONE_IS_WRONG = nil

"
    end

    if @ca_file then
      connection.ca_file = @ca_file
      connection.verify_mode = OpenSSL::SSL::VERIFY_PEER
      connection.verify_callback = @verify_callback if @verify_callback
    end

    if @certificate and @private_key then
      connection.cert = @certificate
      connection.key  = @private_key
    end

    connection.cert_store = if @cert_store then
                              @cert_store
                            else
                              store = OpenSSL::X509::Store.new
                              store.set_default_paths
                              store
                            end
  end
ssl_version=(ssl_version) click to toggle source

SSL version to use

# File lib/net/http/persistent.rb, line 965
def ssl_version= ssl_version
  @ssl_version = ssl_version

  reconnect_ssl
end
verify_callback=(callback) click to toggle source

SSL verification callback.

# File lib/net/http/persistent.rb, line 987
def verify_callback= callback
  @verify_callback = callback

  reconnect_ssl
end
verify_mode=(verify_mode) click to toggle source

Sets the HTTPS verify mode. Defaults to OpenSSL::SSL::VERIFY_PEER.

Setting this to VERIFY_NONE is a VERY BAD IDEA and should NEVER be used. Securely transfer the correct certificate and update the default certificate store or set the ca file instead.

# File lib/net/http/persistent.rb, line 978
def verify_mode= verify_mode
  @verify_mode = verify_mode

  reconnect_ssl
end