Implement openssl server in edged triggered (ET) mode Use SSL_accept / SSL_write / SSL_read with non-blocking socket

SSL_accept / SSL_read

在 EPOLL 告知 server fd 有 EPOLLIN 事件時,代表有有一個或多個新的連線請求,我們先完成 accept 取得 client fd 然後再使用 SSL_accept 開始進行 SSL handshake。由於是 non-blocking socket,呼叫 SSL_accept 一般會回傳 SSL_ERROR_WANT_READ / SSL_ERROR_WANT_WRITE 告知該動作並無完成。接著再下一輪觸發 client fd EPOLLIN 事件出現時再次呼叫 SSL_accept 直到完成。

但有個常出錯的點,如果觸發 EPOLLIN 的是 client socket,且當 SSL_accept 完成時,需要緊接著使用 SSL_read,因為  SSL_ERROR_WANT_READ / SSL_ERROR_WANT_WRITE 隱含著程式需要將 socket buffer 內收到的資料通通使用完,而且在 ET mode 在,在讀取完 socket buffer 內的所有資料前,EPOLLIN 事件都不會再次出現。因此 SSL_accept 完成後,若不使用 SSL_read 清空 socket buffer 而直接等待下一次的 EPOLL 事件,有可能會永遠等不到下一個 EPOLLIN 事件出現。

pseudo-code


// server socket must be non-blocking
set_nonblocking( server_fd_ );

...

// add server socket into epoll with ET mode
// we only listen the EPOLLIN (incoming data) event for now
epoll_fd_ = epoll_create1(0);
event.data.fd = server_fd_;
event.events = EPOLLIN | EPOLLET;
epoll_ctl( epoll_fd_, EPOLL_CTL_ADD, server_fd_, &event );

// core task for server
while( server_keep_running_ )
{
    // wait for next epoll event
    num_of_events_ = epoll_wait( epoll_fd_, events, ... );
    
    foreach incomming event
    {
        // server fd's EPOLLIN event means there is/are
        // one or more new client sock connection requests
        if ( event.fd == server_fd_ && ( event.events & EPOLLIN ))
        {
            // process all connect request
            while ( true )
            {
                // socket accept
                infd = accept( server_fd_, ... );
                if ( infd == -1 )
                {
                    error-handler 
                    ( errno == EAGAIN/EWOULDBLOCK ) means no more request
                    break;
                }
                set_nonblocking( infd );
                
                // let client socket listen the EPOLLIN event
                event.data.fd = infd;
                event.events = EPOLLIN | EPOLLET;
                epoll_ctl( epoll_fd_, EPOLL_CTL_ADD, infd, &event ); 
                 
                // ssl accept
                client_fd_ssl = create_ssl_object( infd );
                ret = SSL_accept( client_fd_ssl );
                if ( ret == 1 )
                {
                    // SSL handshake complete
                }
                else if ( ret == SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE )
                {
                    // SSL handshake incomplete
                }
                else { error handler }
            }
        }
        else if ( event.fd == any connected client_fd_ )
        {
            // client fd with EPOLLIN event means SSL_accept 
            // was incomplete or there is data ready for SSL_read
            if ( ! SSL_is_init_finished( client_fd_ssl_ )
            {
                // we need finish the SSL_accept if not yet
                ret = SSL_accept( cliend_fd_ssl );
                if ( ret == 1 ) 
                {
                    // we will immdiately do SSL_read below
                }
                else if ( ret == SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE )
                {
                    // back to epoll_wait()
                    continue;
                } 
                else { error handler }
            }
            
            // read all data in buffer until empty
            while ( true )
            {
                ret = SSL_read( ... );
                if ( ret <= 0 ) 
                {
                    if ( SSL_get_error( ... ) == SSL_ERROR_WANT_READ/SSL_ERROR_WANT_WRITE )
                    {
                        // stil some data in the socket buffer
                        // waiting for next SSL_read
                        continue;
                    }
                    else { error handler, break the while-loop if no more data }
                }
            }
        }
    }
}

SSL_write

EPOLL 事件中,EPOLLOUT 代表該 socket 目前是可寫的狀態。而在 ET mode 下,EPOLLOUT 會在下面二種情況下出現:

  1. socket fd 加入 EPOLLET | EPOLLOUT 事件後,第一次出現 writable 狀態。
  2. SSL_write 回傳 SSL_ERROR_WANT_READ/WRITE 告知 write buffer 滿了導致錯誤發生以後,第一次恢復 writable 狀態。

另一個需注意的是,除非特別設定 SSL_MODE_ENABLE_PARTIAL_WRITE mode,不然 SSL_write 只有在所有的資料都寫入 socket buffer 後,才會回傳成功值。

在 ET mode (在 LT 其實也是) 下,比較好的做法是平常並不讓 socket fd 去聽 EPOLLOUT 事件,一直使用 SSL_write 直到出現 SSL_ERROR_WANT_READ/WRITE 的錯誤訊息 (i.e. write buffer is full),才將該 socket fd 註冊並等待 EPOLLOUT 出現。

pseudo-code


// use mutex to let writedata() thread-safe, however, if you want,
// openssl could be thread-safe if provided two callback functions
void writedata( int socketfd, uint8* data ... )
{
    data_mutex.lock();
    datalist.add( socketfd, data );
    data_mutex.unlock();
}

...
 
while( server_keep_running_ )
{
    data_mutex.lock();
    foreach( data in datalist )
    {
        if ( !is_socket_writable( socketfd ) )
            continue;
        ret = SSL_write( socketfd, data );
        if ( ret <= 0 )
        {
            if ( ssl_get_error( socketfd_ssl, ret ) == 
                SSL_ERROR_WANT_READ/WRITE )
            {
                // write buffer is full, set unwritable
                set_socket_writable( socketfd, false );
                
                // start to listen EPOLLOUT event
                event.data.fd = socketfd;
                event.events = EPOLLIN | EPOLLOUT | EPOLLET;
                ret = epoll_ctl(epoll_fd_, EPOLL_CTL_MOD, socketfd, &event);
            }
            else { error-handler }
        }
        else
        {
            // SSL_write complete, remove the data from datalist
            datalist.erase( socketfd, data );
        }        
    }
    data_mutex.unlock();
    
    // wait for next epoll event
    num_of_events_ = epoll_wait( epoll_fd_, events, ... );
    
    foreach incomming event
    {
        if ( event.fd == one of the socketfd )
        {
            if ( events[i].events & EPOLLOUT )
            {
                // socket is back to writable again
                set_socket_writable( socketfd, true );
                
                // no need to listen EPOLLOUT event anymore
                event.data.fd = socketfd;
                event.events = EPOLLIN | EPOLLET;
                ret = epoll_ctl(epoll_fd_, EPOLL_CTL_MOD, events[i].data.fd, &event);
            }
        }
    }    
}

Summary

在同時處理 SSL_accept / SSL_write / SSL_read 時,要注意 EPOLLIN 和 EPOLLOUT 是會同時出現的,二者都需要處裡完後才能回到 epoll_wait(),ET mode 下事件只會觸發一次,如果沒有同時處理完會有邏輯上的錯誤。

一個 epoll ET mode / non-blocking / single thread / openssl server 的運作流程如下:

pseudo-code



while( server_keep_running_ )
{
    // close all bad/dead socket fds came from error-handler
    close_all_bad_fds();
    
    // SSL_write
    foreach (data in datalist)
    {
        if client_fd is not writable, continue
        if SSL_write successful, continue
        if write buffer full,
            1. set client_fd not writable
            2. add EPOLLOUT event into client_fd
    }
    
    // wait for next epoll event, cannot block here 
    // since SSL_read/SSL_write are in the same thread 
    epoll_wait()
    
    if epoll_wait() returns due to timeout, continue
    
    // handle server socket event (connection request)
    if epoll event is for server_fd 
        if epoll event type is EPOLLIN
            while( true )
                accept(), then break if there is no more request
                SSL_accept()
                
    // handle client socket event
    if epoll event is for client_fd
        if epoll event type is EPOLLIN
            
            do SSL_accept() if !SSL_is_init_finished()
               still not init finished? continue
            
            do SSL_read() multiple times until no more data
        
        if epoll event type is EPOLLOUT
            1. set client_fd writable
            2. remove EPOLLOUT event from client_fd
}

Full code: https://github.com/meowyih/octillion-cubes/blob/master/src/server/coreserver.cpp