SSL_accept / SSL_read
在 EPOLL 告知 server fd 有 EPOLLIN 事件時,代表有有一個或多個新的連線請求,我們先完成 accept 取得 client fd 然後再使用 SSL_accept 開始進行 SSL handshake。由於是 non-blocking socket,呼叫 SSL_accept 一般會回傳 SSL_ERROR_WANT_READ / SSL_ERROR_WANT_WRITE 告知該動作並無完成。接著再下一輪觸發 client fd EPOLLIN 事件出現時再次呼叫 SSL_accept 直到完成。
但有個常出錯的點,如果觸發 EPOLLIN 的是 client socket,且當 SSL_accept 完成時,需要緊接著使用 SSL_read,因為 SSL_ERROR_WANT_READ / SSL_ERROR_WANT_WRITE 隱含著程式需要將 socket buffer 內收到的資料通通使用完,而且在 ET mode 在,在讀取完 socket buffer 內的所有資料前,EPOLLIN 事件都不會再次出現。因此 SSL_accept 完成後,若不使用 SSL_read 清空 socket buffer 而直接等待下一次的 EPOLL 事件,有可能會永遠等不到下一個 EPOLLIN 事件出現。
pseudo-code
// server socket must be non-blocking
set_nonblocking( server_fd_ );
...
// add server socket into epoll with ET mode
// we only listen the EPOLLIN (incoming data) event for now
epoll_fd_ = epoll_create1(0);
event.data.fd = server_fd_;
event.events = EPOLLIN | EPOLLET;
epoll_ctl( epoll_fd_, EPOLL_CTL_ADD, server_fd_, &event );
// core task for server
while( server_keep_running_ )
{
// wait for next epoll event
num_of_events_ = epoll_wait( epoll_fd_, events, ... );
foreach incomming event
{
// server fd's EPOLLIN event means there is/are
// one or more new client sock connection requests
if ( event.fd == server_fd_ && ( event.events & EPOLLIN ))
{
// process all connect request
while ( true )
{
// socket accept
infd = accept( server_fd_, ... );
if ( infd == -1 )
{
error-handler
( errno == EAGAIN/EWOULDBLOCK ) means no more request
break;
}
set_nonblocking( infd );
// let client socket listen the EPOLLIN event
event.data.fd = infd;
event.events = EPOLLIN | EPOLLET;
epoll_ctl( epoll_fd_, EPOLL_CTL_ADD, infd, &event );
// ssl accept
client_fd_ssl = create_ssl_object( infd );
ret = SSL_accept( client_fd_ssl );
if ( ret == 1 )
{
// SSL handshake complete
}
else if ( ret == SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE )
{
// SSL handshake incomplete
}
else { error handler }
}
}
else if ( event.fd == any connected client_fd_ )
{
// client fd with EPOLLIN event means SSL_accept
// was incomplete or there is data ready for SSL_read
if ( ! SSL_is_init_finished( client_fd_ssl_ )
{
// we need finish the SSL_accept if not yet
ret = SSL_accept( cliend_fd_ssl );
if ( ret == 1 )
{
// we will immdiately do SSL_read below
}
else if ( ret == SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE )
{
// back to epoll_wait()
continue;
}
else { error handler }
}
// read all data in buffer until empty
while ( true )
{
ret = SSL_read( ... );
if ( ret <= 0 )
{
if ( SSL_get_error( ... ) == SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE )
{
// stil some data in the socket buffer
// waiting for next SSL_read
continue;
}
else { error handler, break the while-loop if no more data }
}
}
}
}
}
SSL_write
EPOLL 事件中,EPOLLOUT 代表該 socket 目前是可寫的狀態。而在 ET mode 下,EPOLLOUT 會在下面二種情況下出現:
- socket fd 加入 EPOLLET | EPOLLOUT 事件後,第一次出現 writable 狀態。
- SSL_write 回傳 SSL_ERROR_WANT_READ/WRITE 告知 write buffer 滿了導致錯誤發生以後,第一次恢復 writable 狀態。
另一個需注意的是,除非特別設定 SSL_MODE_ENABLE_PARTIAL_WRITE mode,不然 SSL_write 只有在所有的資料都寫入 socket buffer 後,才會回傳成功值。
在 ET mode (在 LT 其實也是) 下,比較好的做法是平常並不讓 socket fd 去聽 EPOLLOUT 事件,一直使用 SSL_write 直到出現 SSL_ERROR_WANT_READ/WRITE 的錯誤訊息 (i.e. write buffer is full),才將該 socket fd 註冊並等待 EPOLLOUT 出現。
pseudo-code
// use mutex to let writedata() thread-safe, however, if you want,
// openssl could be thread-safe if provided two callback functions
void writedata( int socketfd, uint8* data ... )
{
data_mutex.lock();
datalist.add( socketfd, data );
data_mutex.unlock();
}
...
while( server_keep_running_ )
{
data_mutex.lock();
foreach( data in datalist )
{
if ( !is_socket_writable( socketfd ) )
continue;
ret = SSL_write( socketfd, data );
if ( ret <= 0 )
{
if ( ssl_get_error( socketfd_ssl, ret ) ==
SSL_ERROR_WANT_READ/WRITE )
{
// write buffer is full, set unwritable
set_socket_writable( socketfd, false );
// start to listen EPOLLOUT event
event.data.fd = socketfd;
event.events = EPOLLIN | EPOLLOUT | EPOLLET;
ret = epoll_ctl(epoll_fd_, EPOLL_CTL_MOD, socketfd, &event);
}
else { error-handler }
}
else
{
// SSL_write complete, remove the data from datalist
datalist.erase( socketfd, data );
}
}
data_mutex.unlock();
// wait for next epoll event
num_of_events_ = epoll_wait( epoll_fd_, events, ... );
foreach incomming event
{
if ( event.fd == one of the socketfd )
{
if ( events[i].events & EPOLLOUT )
{
// socket is back to writable again
set_socket_writable( socketfd, true );
// no need to listen EPOLLOUT event anymore
event.data.fd = socketfd;
event.events = EPOLLIN | EPOLLET;
ret = epoll_ctl(epoll_fd_, EPOLL_CTL_MOD, events[i].data.fd, &event);
}
}
}
}
Summary
在同時處理 SSL_accept / SSL_write / SSL_read 時,要注意 EPOLLIN 和 EPOLLOUT 是會同時出現的,二者都需要處裡完後才能回到 epoll_wait(),ET mode 下事件只會觸發一次,如果沒有同時處理完會有邏輯上的錯誤。
一個 epoll ET mode / non-blocking / single thread / openssl server 的運作流程如下:
pseudo-code
while( server_keep_running_ )
{
// close all bad/dead socket fds came from error-handler
close_all_bad_fds();
// SSL_write
foreach (data in datalist)
{
if client_fd is not writable, continue
if SSL_write successful, continue
if write buffer full,
1. set client_fd not writable
2. add EPOLLOUT event into client_fd
}
// wait for next epoll event, cannot block here
// since SSL_read/SSL_write are in the same thread
epoll_wait()
if epoll_wait() returns due to timeout, continue
// handle server socket event (connection request)
if epoll event is for server_fd
if epoll event type is EPOLLIN
while( true )
accept(), then break if there is no more request
SSL_accept()
// handle client socket event
if epoll event is for client_fd
if epoll event type is EPOLLIN
do SSL_accept() if !SSL_is_init_finished()
still not init finished? continue
do SSL_read() multiple times until no more data
if epoll event type is EPOLLOUT
1. set client_fd writable
2. remove EPOLLOUT event from client_fd
}
Full code: https://github.com/meowyih/octillion-cubes/blob/master/src/server/coreserver.cpp