Advanced Usage#

Customizing Pool Behavior#

The PoolManager class automatically handles creating ConnectionPool instances for each host as needed. By default, it will keep a maximum of 10 ConnectionPool instances. If you’re making requests to many different hosts it might improve performance to increase this number:

>>> import urllib3
>>> http = urllib3.PoolManager(num_pools=50)

However, keep in mind that this does increase memory and socket consumption.

Similarly, the ConnectionPool class keeps a pool of individual HTTPConnection instances. These connections are used during an individual request and returned to the pool when the request is complete. By default only one connection will be saved for re-use. If you are making many requests to the same host simultaneously it might improve performance to increase this number:

>>> import urllib3
>>> http = urllib3.PoolManager(maxsize=10)
# Alternatively
>>> http = urllib3.HTTPConnectionPool('google.com', maxsize=10)

The behavior of the pooling for ConnectionPool is different from PoolManager. By default, if a new request is made and there is no free connection in the pool then a new connection will be created. However, this connection will not be saved if more than maxsize connections exist. This means that maxsize does not determine the maximum number of connections that can be open to a particular host, just the maximum number of connections to keep in the pool. However, if you specify block=True then there can be at most maxsize connections open to a particular host:

>>> http = urllib3.PoolManager(maxsize=10, block=True)
# Alternatively
>>> http = urllib3.HTTPConnectionPool('google.com', maxsize=10, block=True)

Any new requests will block until a connection is available from the pool. This is a great way to prevent flooding a host with too many connections in multi-threaded applications.

Streaming and I/O#

When using preload_content=True (the default setting) the response body will be read immediately into memory and the HTTP connection will be released back into the pool without manual intervention.

However, when dealing with large responses it’s often better to stream the response content using preload_content=False. Setting preload_content to False means that urllib3 will only read from the socket when data is requested.

Note

When using preload_content=False, you need to manually release the HTTP connection back to the connection pool so that it can be re-used. To ensure the HTTP connection is in a valid state before being re-used all data should be read off the wire.

You can call the drain_conn() to throw away unread data still on the wire. This call isn’t necessary if data has already been completely read from the response.

After all data is read you can call release_conn() to release the connection into the pool.

You can call the close() to close the connection, but this call doesn’t return the connection to the pool, throws away the unread data on the wire, and leaves the connection in an undefined protocol state. This is desirable if you prefer not reading data from the socket to re-using the HTTP connection.

stream() lets you iterate over chunks of the response content.

>>> import urllib3
>>> http = urllib3.PoolManager()
>>> r = http.request(
...     'GET',
...     'http://httpbin.org/bytes/1024',
...     preload_content=False)
>>> for chunk in r.stream(32):
...     print(chunk)
b'...'
b'...'
...
>>> r.release_conn()

However, you can also treat the HTTPResponse instance as a file-like object. This allows you to do buffering:

>>> r = http.request(
...     'GET',
...     'http://httpbin.org/bytes/1024',
...     preload_content=False)
>>> r.read(4)
b'\x88\x1f\x8b\xe5'

Calls to read() will block until more response data is available.

>>> import io
>>> reader = io.BufferedReader(r, 8)
>>> reader.read(4)
>>> r.release_conn()

You can use this file-like object to do things like decode the content using codecs:

>>> import codecs
>>> reader = codecs.getreader('utf-8')
>>> r = http.request(
...     'GET',
...     'http://httpbin.org/ip',
...     preload_content=False)
>>> json.load(reader(r))
{'origin': '127.0.0.1'}
>>> r.release_conn()

Proxies#

You can use ProxyManager to tunnel requests through an HTTP proxy:

>>> import urllib3
>>> proxy = urllib3.ProxyManager('http://localhost:3128/')
>>> proxy.request('GET', 'http://google.com/')

The usage of ProxyManager is the same as PoolManager.

You can connect to a proxy using HTTP, HTTPS or SOCKS. urllib3’s behavior will be different depending on the type of proxy you selected and the destination you’re contacting.

HTTP and HTTPS Proxies#

Both HTTP/HTTPS proxies support HTTP and HTTPS destinations. The only difference between them is if you need to establish a TLS connection to the proxy first. You can specify which proxy you need to contact by specifying the proper proxy scheme. (i.e http:// or https://)

urllib3’s behavior will be different depending on your proxy and destination:

  • HTTP proxy + HTTP destination

    Your request will be forwarded with the absolute URI.

  • HTTP proxy + HTTPS destination

    A TCP tunnel will be established with a HTTP CONNECT. Afterward a TLS connection will be established with the destination and your request will be sent.

  • HTTPS proxy + HTTP destination

    A TLS connection will be established to the proxy and later your request will be forwarded with the absolute URI.

  • HTTPS proxy + HTTPS destination

    A TLS-in-TLS tunnel will be established. An initial TLS connection will be established to the proxy, then an HTTP CONNECT will be sent to establish a TCP connection to the destination and finally a second TLS connection will be established to the destination. You can customize the ssl.SSLContext used for the proxy TLS connection through the proxy_ssl_context argument of the ProxyManager class.

For HTTPS proxies we also support forwarding your requests to HTTPS destinations with an absolute URI if the use_forwarding_for_https argument is set to True. We strongly recommend you only use this option with trusted or corporate proxies as the proxy will have full visibility of your requests.

Your proxy appears to only use HTTP and not HTTPS#

If you’re receiving the ProxyError and it mentions your proxy only speaks HTTP and not HTTPS here’s what to do to solve your issue:

If you’re using urllib3 directly, make sure the URL you’re passing into urllib3.ProxyManager starts with http:// instead of https://:

# Do this:
http = urllib3.ProxyManager("http://...")

# Not this:
http = urllib3.ProxyManager("https://...")

If instead you’re using urllib3 through another library like Requests there are multiple ways your proxy could be mis-configured. You need to figure out where the configuration isn’t correct and make the fix there. Some common places to look are environment variables like HTTP_PROXY, HTTPS_PROXY, and ALL_PROXY.

Ensure that the values for all of these environment variables starts with http:// and not https://:

# Check your existing environment variables in bash
$ env | grep "_PROXY"
HTTP_PROXY=http://127.0.0.1:8888
HTTPS_PROXY=https://127.0.0.1:8888  # <--- This setting is the problem!

# Make the fix in your current session and test your script
$ export HTTPS_PROXY="http://127.0.0.1:8888"
$ python test-proxy.py  # This should now pass.

# Persist your change in your shell 'profile' (~/.bashrc, ~/.profile, ~/.bash_profile, etc)
# You may need to logout and log back in to ensure this works across all programs.
$ vim ~/.bashrc

If you’re on Windows or macOS your proxy may be getting set at a system level. To check this first ensure that the above environment variables aren’t set then run the following:

$ python -c 'import urllib.request; print(urllib.request.getproxies())'

If the output of the above command isn’t empty and looks like this:

{
  "http": "http://127.0.0.1:8888",
  "https": "https://127.0.0.1:8888"  # <--- This setting is the problem!
}

Search how to configure proxies on your operating system and change the https://... URL into http://. After you make the change the return value of urllib.request.getproxies() should be:

{  # Everything is good here! :)
  "http": "http://127.0.0.1:8888",
  "https": "http://127.0.0.1:8888"
}

If you still can’t figure out how to configure your proxy after all these steps please join our community Discord and we’ll try to help you with your issue.

SOCKS Proxies#

For SOCKS, you can use SOCKSProxyManager to connect to SOCKS4 or SOCKS5 proxies. In order to use SOCKS proxies you will need to install PySocks or install urllib3 with the socks extra:

python -m pip install urllib3[socks]

Once PySocks is installed, you can use SOCKSProxyManager:

>>> from urllib3.contrib.socks import SOCKSProxyManager
>>> proxy = SOCKSProxyManager('socks5h://localhost:8889/')
>>> proxy.request('GET', 'http://google.com/')

Note

It is recommended to use socks5h:// or socks4a:// schemes in your proxy_url to ensure that DNS resolution is done from the remote server instead of client-side when connecting to a domain name.

Custom TLS Certificates#

Instead of using certifi you can provide your own certificate authority bundle. This is useful for cases where you’ve generated your own certificates or when you’re using a private certificate authority. Just provide the full path to the certificate bundle when creating a PoolManager:

>>> import urllib3
>>> http = urllib3.PoolManager(
...     cert_reqs='CERT_REQUIRED',
...     ca_certs='/path/to/your/certificate_bundle')

When you specify your own certificate bundle only requests that can be verified with that bundle will succeed. It’s recommended to use a separate PoolManager to make requests to URLs that do not need the custom certificate.

Custom SNI Hostname#

If you want to create a connection to a host over HTTPS which uses SNI, there are two places where the hostname is expected. It must be included in the Host header sent, so that the server will know which host is being requested. The hostname should also match the certificate served by the server, which is checked by urllib3.

Normally, urllib3 takes care of setting and checking these values for you when you connect to a host by name. However, it’s sometimes useful to set a connection’s expected Host header and certificate hostname (subject), especially when you are connecting without using name resolution. For example, you could connect to a server by IP using HTTPS like so:

>>> import urllib3
>>> pool = urllib3.HTTPSConnectionPool(
...     "10.0.0.10",
...     assert_hostname="example.org",
...     server_hostname="example.org"
... )
>>> pool.urlopen(
...     "GET",
...     "/",
...     headers={"Host": "example.org"},
...     assert_same_host=False
... )

Note that when you use a connection in this way, you must specify assert_same_host=False.

This is useful when DNS resolution for example.org does not match the address that you would like to use. The IP may be for a private interface, or you may want to use a specific host under round-robin DNS.

Client Certificates#

You can also specify a client certificate. This is useful when both the server and the client need to verify each other’s identity. Typically these certificates are issued from the same authority. To use a client certificate, provide the full path when creating a PoolManager:

>>> http = urllib3.PoolManager(
...     cert_file='/path/to/your/client_cert.pem',
...     cert_reqs='CERT_REQUIRED',
...     ca_certs='/path/to/your/certificate_bundle')

If you have an encrypted client certificate private key you can use the key_password parameter to specify a password to decrypt the key.

>>> http = urllib3.PoolManager(
...     cert_file='/path/to/your/client_cert.pem',
...     cert_reqs='CERT_REQUIRED',
...     key_file='/path/to/your/client.key',
...     key_password='keyfile_password')

If your key isn’t encrypted the key_password parameter isn’t required.

Certificate Validation and macOS#

Apple-provided Python and OpenSSL libraries contain a patches that make them automatically check the system keychain’s certificates. This can be surprising if you specify custom certificates and see requests unexpectedly succeed. For example, if you are specifying your own certificate for validation and the server presents a different certificate you would expect the connection to fail. However, if that server presents a certificate that is in the system keychain then the connection will succeed.

This article has more in-depth analysis and explanation.

TLS Warnings#

urllib3 will issue several different warnings based on the level of certificate verification support. These warnings indicate particular situations and can be resolved in different ways.

  • InsecureRequestWarning

    This happens when a request is made to an HTTPS URL without certificate verification enabled. Follow the certificate verification guide to resolve this warning.

  • InsecurePlatformWarning

    This happens on Python 2 platforms that have an outdated ssl module. These older ssl modules can cause some insecure requests to succeed where they should fail and secure requests to fail where they should succeed. Follow the pyOpenSSL guide to resolve this warning.

  • SNIMissingWarning

    This happens on Python 2 versions older than 2.7.9. These older versions lack SNI support. This can cause servers to present a certificate that the client thinks is invalid. Follow the pyOpenSSL guide to resolve this warning.

Making unverified HTTPS requests is strongly discouraged, however, if you understand the risks and wish to disable these warnings, you can use disable_warnings():

>>> import urllib3
>>> urllib3.disable_warnings()

Alternatively you can capture the warnings with the standard logging module:

>>> logging.captureWarnings(True)

Finally, you can suppress the warnings at the interpreter level by setting the PYTHONWARNINGS environment variable or by using the -W flag.

Google App Engine#

urllib3 supports Google App Engine with some caveats.

If you’re using the Flexible environment, you do not have to do any configuration- urllib3 will just work. However, if you’re using the Standard environment then you either have to use urllib3.contrib.appengine’s AppEngineManager or use the Sockets API

To use AppEngineManager:

>>> from urllib3.contrib.appengine import AppEngineManager
>>> http = AppEngineManager()
>>> http.request('GET', 'https://google.com/')

To use the Sockets API, add the following to your app.yaml and use PoolManager as usual:

env_variables:
    GAE_USE_SOCKETS_HTTPLIB : 'true'

For more details on the limitations and gotchas, see urllib3.contrib.appengine.

Brotli Encoding#

Brotli is a compression algorithm created by Google with better compression than gzip and deflate and is supported by urllib3 if the Brotli package or brotlicffi package is installed. You may also request the package be installed via the urllib3[brotli] extra:

$ python -m pip install urllib3[brotli]

Here’s an example using brotli encoding via the Accept-Encoding header:

>>> from urllib3 import PoolManager
>>> http = PoolManager()
>>> http.request('GET', 'https://www.google.com/', headers={'Accept-Encoding': 'br'})

Decrypting Captured TLS Sessions with Wireshark#

Python 3.8 and higher support logging of TLS pre-master secrets. With these secrets tools like Wireshark can decrypt captured network traffic.

To enable this simply define environment variable SSLKEYLOGFILE:

export SSLKEYLOGFILE=/path/to/keylogfile.txt

Then configure the key logfile in Wireshark, see Wireshark TLS Decryption for instructions.