Add TLS session resumption via SSLSessionCache - #789
Conversation
Such claims would ideally be supported by benchmarks. Could you try to create some? |
That's the goal, but you're right, I don't have any tests to prove that, removed this claim from the PR description. If I manage to create proper benchmarks I will update on that |
|
We could, if it helps, only support this for TLS 1.3. |
7281340 to
4500773
Compare
|
@dkropachev @Lorak-mmk I pushed changes with improvement from older Dmitry's PR, will update PR description soon |
dkropachev
left a comment
dkropachev
left a comment
There was a problem hiding this comment.
I rechecked the TLS session-resumption path against the current branch. The ssl_options configuration still builds a fresh SSLContext per Connection, and a cached stdlib session from the previous connection is incompatible with that new context. I reproduced the failure locally on Python 3.10.12; the session restore path raises ValueError: Session refers to a different SSLContext. Since the new code only catches AttributeError and ssl.SSLError, reconnects fail instead of falling back to a full handshake, and the regression is enabled by default because Cluster auto-creates SSLSessionCache for ssl_options.
4500773 to
d12db4a
Compare
dkropachev
left a comment
dkropachev
left a comment
There was a problem hiding this comment.
Two blocking issues from local validation:
- Twisted caches a TLS session even after hostname verification has already failed, which lets an untrusted peer populate the resumption cache.
SSLSessionCacheacceptsmax_size <= 0and then crashes on the first insert (KeyErrorfrompopitem()on an emptyOrderedDict).
d12db4a to
f8eb94d
Compare
dkropachev
left a comment
dkropachev
left a comment
There was a problem hiding this comment.
Two correctness issues need attention before this lands: the PyOpenSSL TLS 1.3 cache point is too early to capture the resumable session, and the cache can evict a live entry while expired ones remain resident.
08cabfd to
5a713f1
Compare
5a713f1 to
61f7523
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Full details: Description checkExplanation The description explains the motivation, design, configuration, limitations, tests, and linked issue. It follows the repository template and is sufficiently complete despite unchecked documentation-related items. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
2e6d4ce to
03c1717
Compare
5a037a3 to
46f065e
Compare
nikagra
left a comment
nikagra
left a comment
There was a problem hiding this comment.
Reviewed at 46f065e2, with every point checked against a build of this branch. Tags are 🔴 blocking, 🟠 nit, 🟢 question.
One 🔴 blocking: shutdown() clears the whole SSLContext out of a shared cache instead of just this cluster's entries, so one cluster shutting down empties the cache of another that is still running — the sharing pattern the ssl_session_cache docs recommend.
The rest are six 🟠 nits and one 🟢 question. One thread runs through three of them: the cache key is (ssl_context, endpoint, hostname) and carries no cluster or connection identity, but shutdown() reads it as “my entries”, _discard_tls_session reads it as “the session I offered”, and the shard-aware override reads it as “the same peer”.
The caching itself held up to everything I tried: resumption, refresh on reuse, the TLS 1.3 ticket timing and the LRU bound all behave as documented.
| kwargs_dict.setdefault('sockopts', self.sockopts) | ||
| kwargs_dict.setdefault('ssl_options', self.ssl_options) | ||
| kwargs_dict.setdefault('ssl_context', self.ssl_context) | ||
| if self.ssl_session_cache is not None: |
There was a problem hiding this comment.
🟠 nit The guard is on the cache being set, but the comment says it should be on resumption being active. Pass a cache plus a connection_class that does not derive from Connection, and __init__ warns that resumption is unavailable and then this still sends ssl_session_cache= to it — TypeError: unexpected keyword argument, and no connection opens. Guard on the same resumable flag __init__ already computed.
There was a problem hiding this comment.
[Major] 🟠 Reopening — the guard is unchanged, and the comment above it still describes a check this line does not make. The cost is in the docstring at :880, which tells users to assign the cache after construction: that path skips __init__'s resumability check, so Cluster(connection_class=AsyncioConnection, ssl_context=ctx) with a cache assigned afterwards gets one that never fills, and no warning. Gating both on a stored resumable also makes the comment unnecessary.
| Resumption is available when TLS is configured through | ||
| :attr:`~Cluster.ssl_context` and the reactor establishes TLS with the | ||
| standard library's ``ssl`` module: the ``libev`` and ``asyncore`` reactors, | ||
| which is to say the default one. |
There was a problem hiding this comment.
🟠 nit On Python 3.12 asyncore is gone from the standard library, so without the libev extension DefaultConnection resolves to AsyncioConnection and resumption is off. Worth not calling libev/asyncore “the default one” here.
| if session.has_ticket: | ||
| lifetime = session.ticket_lifetime_hint | ||
| if not lifetime: | ||
| if self._socket.version() == 'TLSv1.3': |
There was a problem hiding this comment.
🟠 nit The comment above says a reactor that does not use the stdlib ssl module only has to override _set_tls_session and _get_resumable_tls_session, but _tls_session_lifetime needs self._socket.version() too. Worth either saying so there or taking the version through the accessor layer.
There was a problem hiding this comment.
[Major] 🟠 Reopening — unchanged. What makes this worth fixing rather than documenting is that both reads fail quietly: version() raising is swallowed at :1473, so an affected reactor caches nothing and says so only at debug level, and a missing session_reused makes this method always return True, so extend=True re-stamps a full lifetime on every reuse — what extend=False was added to prevent. @dkropachev's twisted and eventlet threads are the same boundary from the other side.
There was a problem hiding this comment.
Most of this is fixed already — the comment now names three accessors, _tls_session_lifetime goes through _tls_negotiated_version(), and a test builds a Connection with no socket at all and only those three overridden, so nothing in the policy can reach for one unnoticed.
The session_reused half is gone. That comparison moved inside set() and is made by session id under the storing lock, so there's no method left to return True from and no full lifetime to re-stamp.
The silence is real and deliberate. Both call sites are under @defunct_on_error, so raising would kill a healthy connection over an optimisation, and the store runs on every connection and every reconnect — a warning there is unbounded noise for a condition that never changes after the first attempt.
5cd9f22 to
fff3f4d
Compare
fff3f4d to
f636dd4
Compare
dkropachev
left a comment
dkropachev
left a comment
There was a problem hiding this comment.
One inline correctness issue. Branch also needs a rebase onto current master; PR is currently conflicting.
f636dd4 to
a578ae3
Compare
|
The design is right where it matters — one cache per What I would like to settle before the next push is the shape of the bookkeeping around it, because each round so far has been answered with a parameter or a method rather than a change of mechanism:
Two changes would collapse most of that. 1. The cache has an owner, and the driver knows which it is. An auto-created cache is a 2. The accessor boundary should hold. Order I would suggest: 1, then 2 (which also removes the version-guessing in Separately: five threads are resolved while the code at each is unchanged — cluster.py, connection.py, pool.py, cluster.py, test_tls_resumption.py. If any of those was a decision rather than an oversight, say so and I will drop it. Deliberately not raising this round, to keep it to the shape above: docs for an on-by-default change, |
nikagra
left a comment
nikagra
left a comment
There was a problem hiding this comment.
Detail under the comment above — the two changes there would remove most of what these three sit on.
b540378 to
a9124b7
Compare
TLS clients can skip the expensive part of a handshake by replaying a session established earlier with the same peer (RFC 5077 tickets for TLS 1.2, RFC 8446 PSKs for TLS 1.3), but OpenSSL never does this on its own: the client has to hold on to the session and offer it explicitly on the next connection. Add the storage half of that: a bounded, thread-safe LRU of TLS sessions keyed by TLS peer identity, plus an EndPoint.tls_session_cache_key property that produces the key. A cached session is not consumed by being used -- one session can be replayed by any number of concurrent connections -- so get() leaves the entry in place and each successful handshake stores back over it whatever the peer handed over. Entries carry the lifetime the caller gives them and are dropped once it runs out, so a session is never offered past the point the peer said it would honour it; what that lifetime should be is for the caller to work out, since it depends on how the session resumes. Whether a store moves an existing deadline is decided here rather than asked of the caller: a session the peer handed back unchanged keeps the deadline it had, since its lifetime runs from when the peer issued it and not from when it was last replayed, and re-stamping a full lifetime on every reuse would let one ticket be offered for as long as connections keep being opened. Below TLS 1.3 an abbreviated handshake hands back the offered session itself, so that is the ordinary case; TLS 1.3 normally issues a fresh one, which is entitled to a lifetime of its own. The two are told apart by session id, compared under the same lock that stores the result, so no concurrent store can land in between. A caller can also say which session it offered on the handshake it is storing the result of, and a store of that same session is skipped where the entry no longer holds it: another connection has stored a session the peer reissued to it, or the deadline passed and a lookup dropped the entry. In neither case has this caller anything to add, while storing it would put a deadline running from now on a session the peer issued at some earlier point -- and would replace a fresher session with an older one. SNI endpoints add the server name to their key, since they all share a proxy address and port but are distinct TLS peers. Client-routes endpoints key on the node's host_id rather than the proxy address they happen to resolve to at the moment. Nothing uses the cache yet. Refs DRIVER-165
Offer the cached session for the endpoint before the handshake, and store the negotiated session once the connection is up, so that the next connection to the same node -- in particular the burst of per-shard connections a pool opens at once -- can skip the certificate exchange and signature of a full handshake. The session is stored from the ReadyMessage / AuthSuccessMessage handlers rather than right after the handshake. A TLS 1.3 server sends its NewSessionTicket as a post-handshake message, so a session read straight after connect() carries no ticket and would not resume; by the time the CQL handshake has completed the ticket has been read off the socket. Storing is idempotent, so nothing needs to track whether it already happened, and every failure in this path is logged and dropped: resumption is an optimisation, and both call sites are wrapped in @defunct_on_error, where a raised exception would kill a healthy connection. How long a session may be offered is worked out here, because it depends on how the session resumes: a ticket's lifetime is the one the server announced, while SSLSession.timeout is only the local context's default and says nothing about what the peer will still accept, so it is used solely for a session that resumes by id. RFC 8446 section 4.6.1 also caps the client at seven days however long the server asked for. A zero lifetime is read against the negotiated version, since the two RFCs disagree on it: TLS 1.3 says discard the ticket immediately, while RFC 5077 section 3.3 reserves zero for "lifetime unspecified" and leaves retention to local policy, so a TLS 1.2 ticket is kept and timed by the local timeout. The version also rules out caching a TLS 1.3 session that carries only an id: resumption there is the ticket's pre-shared key, while the id a TLS 1.3 handshake carries is the legacy_session_id_echo a server sends back for the middlebox compatibility mode of RFC 8446 appendix D.4, which resumes nothing. OpenSSL reports no id at all until a NewSessionTicket has been read, so that session is not one it produces; stating the rule against the negotiated version rather than against what one library exposes is what makes it hold regardless. OpenSSL does not apply either limit on the client's behalf -- it will offer an expired ticket and let the server refuse it. The announced lifetime is taken whole rather than reduced by the session's age: this connection established the session moments ago, so that age is one CQL handshake, and SSLSession.time is a wall-clock stamp, so subtracting it would let a clock step landing in between decide the answer -- far enough forward and nothing is cached at all. The deadline the cache keeps is monotonic, so nothing after the store can skew it either. A pool reaches a shard-aware node on a second port, which would otherwise key those connections separately from the one the control connection established, leaving the whole per-shard burst to handshake in full. The endpoint alias that _get_shard_aware_endpoint already builds for that port therefore carries the node's cache key, so both listeners share one session and nothing has to be threaded through the connection factory. The port stays part of the key by default, so two unrelated TLS servers on one address still cannot share a session; only an endpoint that names another node is exempt. The SSLContext is part of the key because a session cannot be replayed onto a different one and a cache may be shared by several clusters. It is held strongly there: a cached session already keeps its context alive on its own -- CPython's SSLSession holds a reference to the context it was established with -- so holding it weakly here would buy nothing. The key also carries the name wrap_socket() is given, which is the name the peer certificate is verified against. A resumed handshake sends no Certificate, so that name is never checked again; offering a session to a connection expecting a different name would silently skip hostname verification for it. Both the key and wrap_socket() take the name from one accessor so the two cannot drift apart. A session offered on a connection whose handshake then failed is dropped from the cache. Both RFCs have a server fall back to a full handshake rather than fail when it will not resume, so this should not happen; but nothing stores a fresh session for a connection that never came up, so an entry that did provoke a failure would otherwise be offered again by every later connection until its lifetime ran out. Only a TLS error counts: a refused or reset connection says nothing about the session. And only the session this connection offered goes: connections to one node are opened together, so another may have stored a fresh one under the same key in the meantime, and that one failed nothing. What was offered is kept past a handshake that succeeded as well, because the store hands it to the cache. Telling a session the peer reissued from the one that came back unchanged needs both sides of the exchange, and where connections to a node are opened together only the connection that offered one knows the second. Three accessors are the whole of what a reactor whose TLS does not go through the stdlib ssl module has to reimplement to take part: the policy around them asks one for the session to store, one for the negotiated version and one to restore a session onto a socket, and reads nothing off a socket itself. Connections whose SSLContext is derived from ssl_options do not participate, because a session cannot be replayed onto a different context and each of those connections builds its own. The asyncio reactor opts out entirely: its handshake happens inside loop.create_connection(), with no point at which a session could be restored. Refs DRIVER-165
Create an SSLSessionCache per Cluster whenever TLS is configured through ssl_context, and hand it to every connection the cluster opens, so that resumption is on by default with no configuration. Pass ssl_session_cache=None to turn it off, or an instance of your own to size it or share it between clusters. No cache is created where resumption cannot work: the deprecated ssl_options-only path, whose per-connection SSLContexts a session cannot be replayed onto, and reactors that report they cannot restore a session before the handshake, which today means asyncio -- which is also what the default connection class resolves to on Python 3.12 and newer with no libev extension installed, asyncore having left the standard library there. connection_class is not required to derive from Connection, so one that does not report the capability at all is treated as lacking it rather than raising. That is settled at construction, so the attribute reads as documented straight away, and again from connect(), against whatever ssl_context and connection_class are in force by then. Both are public attributes: a decision kept from the constructor would leave resumption off on a reactor that does support it, or hand the keyword to a connection class that does not take it. What the caller asked for is remembered, so a cache supplied and then declined is restored rather than lost if a later decision turns resumption back on, and an explicit None still means off. A cache put there for one of those configurations is warned about and then dropped, whether it was passed to the constructor or assigned to the attribute afterwards: what is decided is the attribute as it stands, so both arrive at the same place. Asking for resumption and silently getting none is worse than not having it: an unusable cache left in place would be handed to every connection -- which a connection class that does not take the keyword cannot even accept -- and would sit reachable and empty for anyone reading it back, which is also what a server that issues no tickets looks like. So the attribute holds a cache only where one will actually be used, and that is what decides whether connections are given it. The reason is given once for the cluster: both decisions usually see the same pair, and saying it again at connect() would be noise. Whose the cache is settles what becomes of it, and that is recorded at construction. One created here is reachable only through the attribute, so it and the sessions in it go when the cluster does and shutdown has nothing to do. One the caller supplied stays the caller's: shutdown leaves its entries alone, which is what lets clusters share sessions -- at the same time, or one after another, so that a cluster replacing an earlier one resumes rather than handshaking in full -- and keeps the driver from deleting rows in an object it does not own. Its entries hold the SSLContext their session was established with, bounded by the cache's max_size, and clear() is there for a caller who wants them gone sooner. Refs DRIVER-165
Stand up a TLS server on loopback and connect to it with the driver's own socket setup, so the restore-before-handshake and store-after-startup paths run for real and the result is read back the way OpenSSL reports it, through SSLSocket.session_reused. Covers TLS 1.2 and TLS 1.3, the latter skipped where the local OpenSSL does not offer it -- skipping the subclass rather than the base, since a skipped base would take its subclasses with it. Two of these pin down behaviour that is easy to regress: that four connections opened at once all resume from the single cached session -- the per-shard burst DRIVER-165 is about -- and that on TLS 1.3 nothing is cached until the server's NewSessionTicket has actually been read off the socket. Refs DRIVER-165
Restart the cluster with client encryption on, warm a session cache with one cluster, then hand it to a second one and require every connection it opens to have resumed -- which is the question only a real server can answer: whether it accepts one session offered concurrently by the whole batch of per-shard connections. The cluster is given a shard-aware TLS port, since that is the port those per-shard connections use and therefore where resumption has to pay off; Scylla leaves it unset by default. The certificate names every node rather than only the contact point, or the driver could not build pools to the rest of the cluster and the test would quietly examine a single host. Each Session is held for the duration of a test: Cluster.sessions is a WeakSet, so a dropped Session takes its pools -- everything worth inspecting -- with it and leaves only the control connection behind. The number of connections collected is asserted before their resumption flags, so the test cannot pass by examining almost nothing. Whether there is anything here to test at all depends on the reactor, so the skip asks the connection class Cluster will instantiate rather than reading EVENT_LOOP_MANAGER: with no selector set and no libev to import, that class is the asyncio reactor, which cannot restore a session before the handshake. Follows the reconfigure-and-remove pattern the other modules here use for cluster-level options, and generates the server certificate with cryptography so the test does not depend on an openssl binary. Refs DRIVER-165
Scylla only issues session tickets when enable_session_tickets is set in client_encryption_options, and that is off by default -- without it the cache stays empty and every connection performs a full handshake, with no indication of why. Refs DRIVER-165
What and why
A shard-aware driver opens one TLS connection per shard to every node, and each one currently
pays for a full handshake — certificate exchange plus a signature, which is the expensive part,
especially with certificate authentication. TLS lets a client skip that by replaying a session
established earlier with the same peer (RFC 5077 tickets for TLS 1.2, RFC 8446 PSKs for
TLS 1.3), but OpenSSL never does this on its own: the client has to hold on to the session and
offer it explicitly on the next connection. Neither the stdlib
sslmodule nor pyOpenSSLexposes
SSL_CTX_sess_set_new_cb, so there is no way around doing it by hand.This adds that: one
SSLSessionCacheperCluster, offered to every connection before itshandshake and refreshed after. On by default whenever
ssl_contextis set.Design notes
get()leaves the entry in place, andeach successful handshake stores a fresh session over it. Measured: one session is accepted by
four concurrent connections on TLS 1.2 and 1.3, stdlib and pyOpenSSL, and against real Scylla.
Treating tickets as single-use (removing on
get()) would mean only the first connection of aper-shard burst resumes — precisely the case this ticket is about. RFC 8446's "SHOULD NOT
reuse" concerns 0-RTT replay and tracking; the driver sends no early data.
the handshake. A TLS 1.3 server sends its NewSessionTicket as a post-handshake message;
confirmed against Scylla that
has_ticketisFalseimmediately afterconnect()andTrueafter the first CQL exchange. Storing is idempotent, so nothing needs to track whether it
already happened, and every failure in this path is logged and dropped — both call sites are
wrapped in
@defunct_on_error, where a raised exception would kill a healthy connection overan optimisation.
SSLContextis part of the cache key. A session cannot be replayed onto a differentcontext — the stdlib rejects it with
ValueError: Session refers to a different SSLContext.That is also why the deprecated
ssl_options-only path does not participate: each of thoseconnections builds its own context.
costs one full handshake, which is the fallback anyway.
_get_resumable_tls_session/_set_tls_session) so areactor not using the stdlib
sslmodule overrides only those.Not covered
loop.create_connection(..., ssl=...), which offersno point at which a session could be restored.
AsyncioConnectiondeclaressupports_tls_session_resumption = Falseand no cache is created for it.Fixes: https://scylladb.atlassian.net/browse/DRIVER-165
Pre-review checklist
./docs/source/.Fixes:annotations to PR description.