proxygen: wangle::SSLSessionCacheManager Class Reference

#include <SSLSessionCacheManager.h>

Inheritance diagram for wangle::SSLSessionCacheManager:

Public Member Functions
	SSLSessionCacheManager (uint32_t maxCacheSize, uint32_t cacheCullSize, folly::SSLContext ctx, const std::string &context, SSLStats stats, const std::shared_ptr< SSLCacheProvider > &externalCache)

virtual	~SSLSessionCacheManager ()

void	onGetSuccess (SSLCacheProvider::CacheContext *cacheCtx, const std::string &value)

void	onGetSuccess (SSLCacheProvider::CacheContext *cacheCtx, std::unique_ptr< folly::IOBuf > valueBuf)

void	onGetFailure (SSLCacheProvider::CacheContext *cacheCtx)

Static Public Member Functions
static void	shutdown ()

Private Types
using	session_callback_arg_session_id_t = unsigned char *

Private Member Functions
int	newSession (SSL ssl, SSL_SESSION session)

void	removeSession (SSL_CTX ctx, SSL_SESSION session)

SSL_SESSION *	getSession (SSL ssl, unsigned char session_id, int id_len, int *copyflag)

void	restoreSession (SSLCacheProvider::CacheContext cacheCtx, const uint8_t data, size_t length)

bool	storeCacheRecord (const std::string &sessionId, SSL_SESSION *session)

bool	lookupCacheRecord (const std::string &sessionId, folly::AsyncSSLSocket *sslSock)

void	restartSSLAccept (const SSLCacheProvider::CacheContext *cacheCtx)

Static Private Member Functions
static std::shared_ptr< ShardedLocalSSLSessionCache >	getLocalCache (uint32_t maxCacheSize, uint32_t cacheCullSize)

static int	newSessionCallback (SSL ssl, SSL_SESSION session)

static void	removeSessionCallback (SSL_CTX ctx, SSL_SESSION session)

static SSL_SESSION *	getSessionCallback (SSL ssl, session_callback_arg_session_id_t session_id, int id_len, int copyflag)

Private Attributes
folly::SSLContext *	ctx_

std::shared_ptr< ShardedLocalSSLSessionCache >	localCache_

PendingLookupMap	pendingLookups_

SSLStats *	stats_ {nullptr}

std::shared_ptr< SSLCacheProvider >	externalCache_

Static Private Attributes
static int32_t	sExDataIndex_ = -1

static std::shared_ptr< ShardedLocalSSLSessionCache >	sCache_

static std::mutex	sCacheLock_

Detailed Description

SSLSessionCacheManager handles all stateful session caching. There is an instance of this object per SSL VIP per thread, with a 1:1 correlation with SSL_CTX. The cache can work locally or in concert with an external cache to share sessions across instances.

There is a single in memory session cache shared by all VIPs. The cache is split into N buckets (currently 16) with a separate lock per bucket. The VIP ID is hashed and stored as part of the session to handle the (very unlikely) case of session ID collision.

When a new SSL session is created, it is added to the LRU cache and sent to the external cache to be stored. The external cache expiration is equal to the SSL session's expiration.

When a resume request is received, SSLSessionCacheManager first looks in the local LRU cache for the VIP. If there is a miss there, an asynchronous request for this session is dispatched to the external cache. When the external cache query returns, the LRU cache is updated if the session was found, and the SSL_accept call is resumed.

If additional resume requests for the same session ID arrive in the same thread while the request is pending, the 2nd - Nth callers attach to the original external cache requests and are resumed when it comes back. No attempt is made to coalesce external cache requests for the same session ID in different worker threads. Previous work did this, but the complexity was deemed to outweigh the potential savings.

Definition at line 136 of file SSLSessionCacheManager.h.

Member Typedef Documentation

using wangle::SSLSessionCacheManager::session_callback_arg_session_id_t = unsigned char*

private

Definition at line 255 of file SSLSessionCacheManager.h.

Constructor & Destructor Documentation

wangle::SSLSessionCacheManager::SSLSessionCacheManager	(	uint32_t	maxCacheSize,
		uint32_t	cacheCullSize,
		folly::SSLContext *	ctx,
		const std::string &	context,
		SSLStats *	stats,
		const std::shared_ptr< SSLCacheProvider > &	externalCache
	)

Constructor. SSL session related callbacks will be set on the underlying SSL_CTX. vipId is assumed to a unique string identifying the VIP and must be the same on all servers that wish to share sessions via the same external cache.

Definition at line 142 of file SSLSessionCacheManager.cpp.

References getLocalCache(), getSessionCallback(), folly::SSLContext::getSSLCtx(), wangle::SSLUtil::getSSLCtxExIndex(), localCache_, newSessionCallback(), removeSessionCallback(), folly::SSLContext::setSessionCacheContext(), and sExDataIndex_.

                                                        :
     ctx_(ctx),
     stats_(stats),
     externalCache_(externalCache) {
 
   SSL_CTX* sslCtx = ctx->getSSLCtx();
 
   SSLUtil::getSSLCtxExIndex(&sExDataIndex_);
 
   SSL_CTX_set_ex_data(sslCtx, sExDataIndex_, this);
   SSL_CTX_sess_set_new_cb(sslCtx, SSLSessionCacheManager::newSessionCallback);
   SSL_CTX_sess_set_get_cb(sslCtx, SSLSessionCacheManager::getSessionCallback);
   SSL_CTX_sess_set_remove_cb(sslCtx,
                              SSLSessionCacheManager::removeSessionCallback);
   if (!FLAGS_dcache_unit_test && !context.empty()) {
     // Use the passed in context
     ctx->setSessionCacheContext(context);
   }
 
   SSL_CTX_set_session_cache_mode(sslCtx, SSL_SESS_CACHE_NO_INTERNAL
                                  | SSL_SESS_CACHE_SERVER);
 
   localCache_ = SSLSessionCacheManager::getLocalCache(maxCacheSize,
                                                       cacheCullSize);
 }

wangle::SSLSessionCacheManager::~SSLSessionCacheManager ( )

virtual

Definition at line 174 of file SSLSessionCacheManager.cpp.

174 {

175 }

Member Function Documentation

shared_ptr< ShardedLocalSSLSessionCache > wangle::SSLSessionCacheManager::getLocalCache	(	uint32_t	maxCacheSize,
		uint32_t	cacheCullSize
	)

staticprivate

Get or create the LRU cache for the given VIP ID

Definition at line 182 of file SSLSessionCacheManager.cpp.

References g(), sCache_, and sCacheLock_.

Referenced by SSLSessionCacheManager().

                           {
 
   std::lock_guard<std::mutex> g(sCacheLock_);
   if (!sCache_) {
     sCache_.reset(new ShardedLocalSSLSessionCache(NUM_CACHE_BUCKETS,
                                                   maxCacheSize,
                                                   cacheCullSize));
   }
   return sCache_;
 }

SSL_SESSION * wangle::SSLSessionCacheManager::getSession	(	SSL *	ssl,
		unsigned char *	session_id,
		int	id_len,
		int *	copyflag
	)

private

Invoked by openssl when a client requests a stateful session resumption. Triggers a lookup in our local cache and potentially an asynchronous request to an external cache.

Definition at line 272 of file SSLSessionCacheManager.cpp.

References externalCache_, folly::AsyncSocket::getFd(), folly::AsyncSSLSocket::getFromSSL(), wangle::SSLUtil::hexlify(), localCache_, lookupCacheRecord(), folly::gen::move, folly::fibers::onFiber(), pendingLookups_, wangle::SSLStats::recordSSLSession(), folly::AsyncSSLSocket::setSessionIDResumed(), stats_, and string.

Referenced by getSessionCallback().

                    {
   VLOG(7) << "SSL get session callback";
   folly::ssl::SSLSessionUniquePtr session;
   bool foreign = false;
   std::string missReason;
 
   if (id_len < MIN_SESSION_ID_LENGTH) {
     // We didn't generate this session so it's going to be a miss.
     // This doesn't get logged or counted in the stats.
     return nullptr;
   }
   string sessionId((char*)session_id, id_len);
 
   AsyncSSLSocket* sslSocket = AsyncSSLSocket::getFromSSL(ssl);
 
   assert(sslSocket != nullptr);
 
   // look it up in the local cache first
   session.reset(localCache_->lookupSession(sessionId));
 #ifdef SSL_SESSION_CB_WOULD_BLOCK
   if (session == nullptr && externalCache_) {
     // external cache might have the session
     foreign = true;
     if (!SSL_want_sess_cache_lookup(ssl)) {
       missReason = "reason: No async cache support;";
     } else {
       PendingLookupMap::iterator pit = pendingLookups_.find(sessionId);
       if (pit == pendingLookups_.end()) {
         auto result = pendingLookups_.emplace(sessionId, PendingLookup());
         // initiate fetch
         VLOG(4) << "Get SSL session [Pending]: Initiate Fetch; fd=" <<
           sslSocket->getFd() << " id=" << SSLUtil::hexlify(sessionId);
         if (lookupCacheRecord(sessionId, sslSocket)) {
           // response is pending
           *copyflag = SSL_SESSION_CB_WOULD_BLOCK;
           return nullptr;
         } else {
           missReason = "reason: failed to send lookup request;";
           pendingLookups_.erase(result.first);
         }
       } else {
         // A lookup was already initiated from this thread
         if (pit->second.request_in_progress) {
           // Someone else initiated the request, attach
           VLOG(4) << "Get SSL session [Pending]: Request in progess: attach; "
             "fd=" << sslSocket->getFd() << " id=" <<
             SSLUtil::hexlify(sessionId);
           std::unique_ptr<DelayedDestruction::DestructorGuard> dg(
             new DelayedDestruction::DestructorGuard(sslSocket));
           pit->second.waiters.emplace_back(sslSocket, std::move(dg));
           *copyflag = SSL_SESSION_CB_WOULD_BLOCK;
           return nullptr;
         }
         // request is complete
         session.reset(
             pit->second.session); // nullptr if our friend didn't have it
         if (session) {
           SSL_SESSION_up_ref(session.get());
         }
       }
     }
   }
 #elif FOLLY_OPENSSL_IS_110
   if (session == nullptr && externalCache_) {
     foreign = true;
     DCHECK(folly::fibers::onFiber());
 
     try {
       session = externalCache_->getFuture(sessionId).get();
     } catch (const std::exception& e) {
       missReason = folly::to<std::string>("reason: ", e.what(), ";");
     }
 
     if (session) {
       localCache_->storeSession(sessionId, session.get(), stats_);
       SSL_SESSION_up_ref(session.get());
     }
   }
 #endif
 
   bool hit = (session != nullptr);
   if (stats_) {
     stats_->recordSSLSession(false, hit, foreign);
   }
   if (hit) {
     sslSocket->setSessionIDResumed(true);
   }
 
   VLOG(4) << "Get SSL session [" << ((hit) ? "Hit" : "Miss")
           << "]: " << ((foreign) ? "external" : "local") << " cache; "
           << missReason << "fd=" << sslSocket->getFd()
           << " id=" << SSLUtil::hexlify(sessionId);
 
   // We already bumped the refcount
   *copyflag = 0;
 
   return session.release();
 }

SSL_SESSION * wangle::SSLSessionCacheManager::getSessionCallback	(	SSL *	ssl,
		session_callback_arg_session_id_t	session_id,
		int	id_len,
		int *	copyflag
	)

staticprivate

Definition at line 257 of file SSLSessionCacheManager.cpp.

References folly::FATAL, getSession(), and sExDataIndex_.

Referenced by SSLSessionCacheManager().

                    {
   SSLSessionCacheManager* manager = nullptr;
   SSL_CTX* ctx = SSL_get_SSL_CTX(ssl);
   manager = (SSLSessionCacheManager*)SSL_CTX_get_ex_data(ctx, sExDataIndex_);
 
   if (manager == nullptr) {
     LOG(FATAL) << "Null SSLSessionCacheManager in callback";
   }
   return manager->getSession(ssl, (unsigned char*)sess_id, id_len, copyflag);
 }

bool wangle::SSLSessionCacheManager::lookupCacheRecord	(	const std::string &	sessionId,
		folly::AsyncSSLSocket *	sslSock
	)

private

Lookup a session in the external cache for the specified SSL socket.

Definition at line 387 of file SSLSessionCacheManager.cpp.

References externalCache_.

Referenced by getSession().

                                                                           {
   auto cacheCtx = new SSLCacheProvider::CacheContext();
   cacheCtx->sessionId = sessionId;
   cacheCtx->session = nullptr;
   cacheCtx->sslSocket = sslSocket;
   cacheCtx->guard.reset(
       new DelayedDestruction::DestructorGuard(cacheCtx->sslSocket));
   cacheCtx->manager = this;
   bool res = externalCache_->getAsync(sessionId, cacheCtx);
   if (!res) {
     delete cacheCtx;
   }
   return res;
 }

int wangle::SSLSessionCacheManager::newSession	(	SSL *	ssl,
		SSL_SESSION *	session
	)

private

Invoked by openssl when a new SSL session is created

Definition at line 207 of file SSLSessionCacheManager.cpp.

References externalCache_, wangle::SSLUtil::hexlify(), localCache_, wangle::SSLStats::recordSSLSession(), stats_, and storeCacheRecord().

Referenced by newSessionCallback().

                                                                  {
   unsigned int sessIdLen = 0;
   const unsigned char* sessId = SSL_SESSION_get_id(session, &sessIdLen);
   string sessionId((char*) sessId, sessIdLen);
   VLOG(4) << "New SSL session; id=" << SSLUtil::hexlify(sessionId);
 
   if (stats_) {
     stats_->recordSSLSession(true /* new session */, false, false);
   }
 
   localCache_->storeSession(sessionId, session, stats_);
 
   if (externalCache_) {
     VLOG(4) << "New SSL session: send session to external cache; id=" <<
       SSLUtil::hexlify(sessionId);
     storeCacheRecord(sessionId, session);
   }
 
   return 1;
 }

int wangle::SSLSessionCacheManager::newSessionCallback	(	SSL *	ssl,
		SSL_SESSION *	session
	)

staticprivate

static functions registered as callbacks to openssl via SSL_CTX_sess_set_new/get/remove_cb

Definition at line 195 of file SSLSessionCacheManager.cpp.

References folly::FATAL, newSession(), and sExDataIndex_.

Referenced by SSLSessionCacheManager().

                                                                              {
   SSLSessionCacheManager* manager = nullptr;
   SSL_CTX* ctx = SSL_get_SSL_CTX(ssl);
   manager = (SSLSessionCacheManager *)SSL_CTX_get_ex_data(ctx, sExDataIndex_);
 
   if (manager == nullptr) {
     LOG(FATAL) << "Null SSLSessionCacheManager in callback";
   }
   return manager->newSession(ssl, session);
 }

void wangle::SSLSessionCacheManager::onGetFailure ( SSLCacheProvider::CacheContext * cacheCtx )

Callback for ExternalCache to call when an async get fails, either because the requested session is not in the external cache or because of an error.

Parameters

context The context that was passed to the async get request

Definition at line 450 of file SSLSessionCacheManager.cpp.

References restartSSLAccept().

                                             {
   restartSSLAccept(cacheCtx);
   delete cacheCtx;
 }

void wangle::SSLSessionCacheManager::onGetSuccess	(	SSLCacheProvider::CacheContext *	cacheCtx,
		const std::string &	value
	)

Callback for ExternalCache to call when an async get succeeds

Parameters

context	The context that was passed to the async get request
value	Serialized session

Definition at line 434 of file SSLSessionCacheManager.cpp.

References restoreSession(), and uint8_t.

                             {
   restoreSession(cacheCtx, (uint8_t*)value.data(), value.length());
 }

void wangle::SSLSessionCacheManager::onGetSuccess	(	SSLCacheProvider::CacheContext *	cacheCtx,
		std::unique_ptr< folly::IOBuf >	valueBuf
	)

Callback for ExternalCache to call when an async get succeeds

Parameters

context	The context that was passed to the async get request
valueBuf	Serialized session stored in folly::IOBuf, this should NOT be called with valueBuf == nullptr.

Definition at line 440 of file SSLSessionCacheManager.cpp.

References folly::IOBuf::coalesce(), folly::IOBuf::data(), folly::IOBuf::length(), and restoreSession().

                                         {
   if (!valueBuf) {
     return;
   }
   valueBuf->coalesce();
   restoreSession(cacheCtx, valueBuf->data(), valueBuf->length());
 }

void wangle::SSLSessionCacheManager::removeSession	(	SSL_CTX *	ctx,
		SSL_SESSION *	session
	)

private

Invoked by openssl when an SSL session is ejected from its internal cache. This can't be invoked in the current implementation because SSL's internal caching is disabled.

Definition at line 239 of file SSLSessionCacheManager.cpp.

References wangle::SSLUtil::hexlify(), localCache_, wangle::SSLStats::recordSSLSessionRemove(), and stats_.

Referenced by removeSessionCallback().

                                                                  {
   unsigned int sessIdLen = 0;
   const unsigned char* sessId = SSL_SESSION_get_id(session, &sessIdLen);
   string sessionId((char*) sessId, sessIdLen);
 
   // This hook is only called from SSL when the internal session cache needs to
   // flush sessions.  Since we run with the internal cache disabled, this should
   // never be called
   VLOG(3) << "Remove SSL session; id=" << SSLUtil::hexlify(sessionId);
 
   localCache_->removeSession(sessionId);
 
   if (stats_) {
     stats_->recordSSLSessionRemove();
   }
 }

void wangle::SSLSessionCacheManager::removeSessionCallback	(	SSL_CTX *	ctx,
		SSL_SESSION *	session
	)

staticprivate

Definition at line 228 of file SSLSessionCacheManager.cpp.

References folly::FATAL, removeSession(), and sExDataIndex_.

Referenced by SSLSessionCacheManager().

                                                                          {
   SSLSessionCacheManager* manager = nullptr;
   manager = (SSLSessionCacheManager *)SSL_CTX_get_ex_data(ctx, sExDataIndex_);
 
   if (manager == nullptr) {
     LOG(FATAL) << "Null SSLSessionCacheManager in callback";
   }
   return manager->removeSession(ctx, session);
 }

void wangle::SSLSessionCacheManager::restartSSLAccept ( const SSLCacheProvider::CacheContext * cacheCtx )

private

Restart all clients waiting for the answer to an external cache query

Definition at line 403 of file SSLSessionCacheManager.cpp.

References pendingLookups_, folly::AsyncSSLSocket::restartSSLAccept(), wangle::SSLCacheProvider::CacheContext::session, wangle::SSLCacheProvider::CacheContext::sessionId, and wangle::SSLCacheProvider::CacheContext::sslSocket.

Referenced by onGetFailure(), and restoreSession().

                                                   {
   PendingLookupMap::iterator pit = pendingLookups_.find(cacheCtx->sessionId);
   CHECK(pit != pendingLookups_.end());
   pit->second.request_in_progress = false;
   pit->second.session = cacheCtx->session;
   VLOG(7) << "Restart SSL accept";
   cacheCtx->sslSocket->restartSSLAccept();
   for (const auto& attachedLookup: pit->second.waiters) {
     // Wake up anyone else who was waiting for this session
     VLOG(4) << "Restart SSL accept (waiters) for fd=" <<
       attachedLookup.first->getFd();
     attachedLookup.first->restartSSLAccept();
   }
   pendingLookups_.erase(pit);
 }

void wangle::SSLSessionCacheManager::restoreSession	(	SSLCacheProvider::CacheContext *	cacheCtx,
		const uint8_t *	data,
		size_t	length
	)

private

Invoked by onGetSuccess callback, to restore session from cache.

Parameters

context	The context that was passed to the async get request
data	Buffer that stores serialized session
length	Buffer size

Definition at line 420 of file SSLSessionCacheManager.cpp.

References localCache_, restartSSLAccept(), wangle::SSLCacheProvider::CacheContext::session, wangle::SSLCacheProvider::CacheContext::sessionId, and stats_.

Referenced by onGetSuccess().

                    {
   cacheCtx->session = d2i_SSL_SESSION(nullptr, &data, length);
   restartSSLAccept(cacheCtx);
 
   /* Insert in the LRU after restarting all clients.  The stats logic
    * in getSession would treat this as a local hit otherwise.
    */
   localCache_->storeSession(cacheCtx->sessionId, cacheCtx->session, stats_);
   delete cacheCtx;
 }

void wangle::SSLSessionCacheManager::shutdown ( )

static

Call this on shutdown to release the global instance of the ShardedLocalSSLSessionCache.

Definition at line 177 of file SSLSessionCacheManager.cpp.

References g(), sCache_, and sCacheLock_.

                                       {
   std::lock_guard<std::mutex> g(sCacheLock_);
   sCache_.reset();
 }

bool wangle::SSLSessionCacheManager::storeCacheRecord	(	const std::string &	sessionId,
		SSL_SESSION *	session
	)

private

Store a new session record in the external cache

Definition at line 375 of file SSLSessionCacheManager.cpp.

References ctx_, expiration(), externalCache_, folly::SSLContext::getSSLCtx(), string, uint32_t, and uint8_t.

Referenced by newSession().

                                                                     {
   std::string sessionString;
   uint32_t sessionLen = i2d_SSL_SESSION(session, nullptr);
   sessionString.resize(sessionLen);
   uint8_t* cp = (uint8_t *)sessionString.data();
   i2d_SSL_SESSION(session, &cp);
   size_t expiration = SSL_CTX_get_timeout(ctx_->getSSLCtx());
   return externalCache_->setAsync(sessionId, sessionString,
                                   std::chrono::seconds(expiration));
 }