A game server has one job that cannot wait: the frame. Many times a second it reads what the players did, moves the monsters, resolves the hits and tells everyone what happened. Anything else that happens on that thread, anything at all, every player feels as lag. I first learned this working on Helbreath Olympia, a Helbreath server where I joined as a programmer. Since then I have taken my own servers from one thread to several, one step at a time.
1. Before: one thread and a message loop
The original Helbreath server is a Windows program, and it runs the way Windows programs of its time did: one thread, waiting for messages.
int EventLoop()
{
static unsigned short _usCnt = 0;
register MSG msg;
while( 1 ) {
if( PeekMessage( &msg, NULL, 0, 0, PM_NOREMOVE ) ) {
if( !GetMessage( &msg, NULL, 0, 0 ) ) {
return msg.wParam;
}
TranslateMessage(&msg);
DispatchMessage(&msg);
UpdateScreen();
}
else WaitMessage();
}
}
Everything arrives as a window message. Each player's socket is registered with WSAAsyncSelect, so when data comes in, Windows posts a message with a number of its own for that player, WM_ONCLIENTSOCKETEVENT plus the player's index, up to 2000 of them. The game's heartbeat is a multimedia timer set to 300 milliseconds, and even it does not run the game. It only posts one more message into the same queue:
void CALLBACK _TimerFunc(UINT wID, UINT wUser, DWORD dwUSer, DWORD dw1, DWORD dw2)
{
PostMessage(G_hWnd, WM_USER_TIMERSIGNAL, wID, NULL);
}
So reading a packet, moving a monster and answering the timer all happen on the same thread, one after the other. That works because the slow work is somewhere else. The server does not touch a database at all. Every 10 minutes it sends each player's data over TCP to a separate log server and lets that one write it down:
if ((m_pMapList[m_pClientList[i]->m_cMapIndex]->m_bIsFightZone == FALSE) &&
((dwTime - m_pClientList[i]->m_dwAutoSaveTime) > (DWORD)DEF_AUTOSAVETIME)) {
bSendMsgToLS(MSGID_REQUEST_SAVEPLAYERDATA, i);
If the log server cannot be reached when a player leaves, the game server writes the character to a text file in a folder named after the date. In the whole game server there is exactly one call to CreateThread, and it opens a debug window.
It was multi-process instead of multi-threaded. The slow part, the disk, lived in another program, and the network between them was the queue.
2. SQL on the game thread
When Helbreath Empire started, in 2017, the separate log server was gone and the game server talked to MySQL itself. The main loop was simple: rest a millisecond, run a frame.
while (!g_close)
{
std::this_thread::sleep_for(std::chrono::milliseconds(1));
g_game->tick();
}
And saving a player happened inside that frame. Each save opened a brand new connection to MySQL, ran its queries and waited for every answer:
void Client::Save()
{
unique_ptr<Sql> sql(Sql::create());
// ...
}
{
//ClockTest test("Query");
mysql_query(sql()->get(), (char*)_str.c_str());
}
mysql_query blocks. While MySQL thinks, the thread waits, and since that thread is the game, the whole world waits with it. Nobody moves, no hit lands, no packet is answered. You can see the problem being worked around in the frame itself, which saves at most one player per frame, so a hundred saves due at once do not freeze the server a hundred times in a row:
if(!saved)
{
if(Clock.elapsed(p->_save_tm, 10 _m, true)) {
p->Save();
saved = true;
}
}
That spreads the cost; it does not remove it. Every save is still a small freeze that every player shares. And the queries that read the most were the worst. The item logs, every trade, drop and pick-up a player ever made, were paged like this, on the game thread:
sql->stat << "SELECT COUNT(player_id) FROM item_logs where player_id = " << p->_id << ";";
// ...
sql->stat << "select * from item_logs where player_id = " << p->_id << " limit " << limit << "," << 30 << ";";
A table that only grows, counted in full and paged with an offset that MySQL must walk past row by row. One curious player opening page 200 of their history could stop the game for everyone. Twenty days after the server went into the repository, the item logs were switched off.
One part already knew better. The login ran on its own thread from the first day, std::thread([&](){tick();}).detach();, because checking a password must never stop a game in progress.
3. The queue
In December 2017, over three days, the database moved off the game thread. The first piece was a queue that two threads can share without fighting over it:
template<typename Fnc>
void Consume(Fnc fnc)
{
{
std::lock_guard<std::mutex> g(_mutex);
if (_push_queue->empty())
return;
swap(_push_queue, _read_queue);
_ready = false;
}
while (!_read_queue->empty())
{
fnc(_read_queue->front());
_read_queue->pop();
}
}
It is two queues, not one. One side pushes into the first; the other side, under the lock, swaps the two pointers and lets go. The lock is held for the time it takes to swap two pointers, not for the time it takes to run the work, so the game thread never waits behind a query to add a new one.
The next day came the database thread, with two of those queues: requests going in and responses coming back.
void Database::Tick()
{
Log::Out("Database thread started");
while (!*(&v_close))
{
std::this_thread::sleep_for(std::chrono::milliseconds(1));
GetRequests().Consume([&](unique_ptr<Request> & req)
{
req->_callback(req);
});
}
}
The game frame, once per tick, picks up whatever answers are ready and runs their second half:
g_db->GetResponses().Consume([&](unique_ptr<DatabaseResponse> & rsp)
{
rsp->_callback(rsp);
});
Here is the part that changes how you think. Once a query runs on another thread, the world keeps going while it runs. By the time the answer comes back, the player who asked may have logged out, and another player may be sitting in the same slot. So an answer does not carry a pointer to the player. It carries the player's slot and a unique id, and finds the player again when it comes back. If the id does not match, the answer is dropped. Every job written since follows that rule.
4. Threads, each owning one thing
Today the server is a handful of threads, and each one owns one kind of work:
- The game frame: reads the players, runs the world, answers. Nothing on it waits for a disk or a database.
- The network: receives, acknowledges and sends the moment the socket has something, instead of once per frame.
- The database: every query.
- The logs: item logs, chat and the rest, written in batches.
- The login: passwords and accounts, as from the first day.
- The loaders: at startup, maps and data load in parallel over several cores.
The database thread is now kazilib's lib::JobThread, shared by both games, and its design is in its first comment:
// One worker, so jobs never overlap and always run in queue order (a save queued
// before a load is written before it is read). The worker wakes as soon as a job
// is queued. Both Helbreath games' database threads are one (hb_lib db/jobs.h).
void Start(const std::string_view name, std::function<bool()> keep_running = {})
{
_keep_running = std::move(keep_running);
_worker.Start(name, [this](Worker & w)
{
while (!w.StopRequested() && (!_keep_running || _keep_running()))
{
_requests.WaitFor(kIdleWait);
RunRequests();
}
});
}
Two decisions are packed in there. First, one worker, not a pool. A pool would run queries faster in parallel, but a player who logs out and back in quickly queues a save and then a load, and with a pool the load could win the race and read the character from before the save. One worker keeps the order. Second, it no longer checks the queue every millisecond. It sleeps until a job arrives, which is both faster to react and lighter on the CPU.
Every job is written as two halves, one per thread, and the code reads top to bottom in the order it runs:
REQ_PACK(p->_acc_id);
DB_REQ(p)
{
UNPACK1(req, AccId, acc_id); // on the database thread
// ... queries ...
RSP_PACK(found, name);
DB_RSP()
{
null_returns(auto, p, rsp->GetPlayer()); // back on the game thread
// ...
Saving a player shows the split well. The game thread, which owns the character, takes a snapshot of it as query text, which is quick. The database thread runs those queries as one transaction, which is slow but now bothers no one. Helbreath Millennium, the live server, made the same move in June 2026: until then its SQL also ran on the game thread.
The network got its own thread last, in kazilib's lib::NetworkThread, used by both servers:
// With the thread, the thread owns ENet's servicing: it wakes as soon as the
// socket has data (WSAEventSelect) or the program sends, and at least every
// kTickMs for ENet's own timers (pings, resends, timeouts). It receives, ACKs and
// sends at once and queues what it received; Service hands the program's thread
// the queue at the same point in the frame and in the same order as before, so
// the frame sees the same events in the same order, only the wire is no longer
// held by the frame.
The last sentence is the important one. The game sees exactly the same events in exactly the same order as before; only the waiting moved. A slow frame no longer delays the acknowledgements that keep every connection alive, so a heavy fight in one city no longer looks like packet loss to a player in another.
It also made the server harder to attack. Under a test flood from four clients, one round of network servicing used to take 19 seconds, the whole game held while it ran. With the thread, and a limit on how much it handles at once, it takes under one second.
5. Tables with millions of rows
Moving SQL off the game thread is what made big tables possible at all. On the game thread, a query that takes two seconds stops the world for two seconds. On the database thread, it makes one player wait two seconds for their page, and nobody else notices. That is the difference between "we can't keep item logs" and keeping every item movement in the game, millions of rows, and letting players search their own.
Off the game thread is not enough on its own, though, because there is one database thread and every save and every login queues behind whatever it is doing. The queries still have to be fast. A migration from 2026 explains what a missing index costs:
-- With only `idx_uid` present, that OR cannot become an index_merge union:
-- one side is indexed, the other is not, so the optimizer falls back to a
-- full scan of `item_logs` — for the COUNT(*) *and* the page SELECT, twice
-- per click, on a table that only grows (no pruning) and from the single
-- DB thread every save and login queues behind.
Compare today's item log search with the 2017 one from section 2:
Sql_Stmt(format("SELECT COUNT(*) AS `n` FROM (SELECT 1 FROM `item_logs` il WHERE {} LIMIT {}) c",
Pred(cid, kind, ids, from, to), ilq::kCountCap + 1).c_str());
// ...
// The page: per character its first (page + 1) x 50 rows by the sort (its index stops
// there), merged, then the page's 50 (an IN (...) + ORDER BY would sort every match)
- The count has a ceiling. It counts up to a cap and stops. Nobody needs to know they have exactly 3,481,207 entries; "more than the cap" is enough to draw the pages.
- Indexes match the questions. The searches players make (by character and type, by character and item, by character and time) each have an index in that order, so MySQL reads only the rows it returns.
- The page reads only what it shows. For each character, the index gives the first rows already in order; they are merged and the page's 50 are taken, instead of sorting every match first.
- The indexes went in without stopping the server. They were added with
ALGORITHM=INPLACE, LOCK=NONE, so a live table keeps taking writes while its index is built. - Writes come in batches. The logs thread gathers rows and inserts up to 500 at a time every 30 seconds, with a ceiling of 200,000 rows waiting, so a busy evening never becomes thousands of tiny inserts or an unbounded pile of memory.
In the end: who waits for what
| Where the slow work runs | Who waits for a slow query | |
|---|---|---|
| Original | another program (the log server) | nobody in the game, but nothing could be looked up |
| 2017 | the game thread | every player on the server |
| Late 2017 | a database thread, through queues | the player who asked |
| Today | a thread for each kind of work, in both games | the player who asked, briefly, and the network never waits for the frame |
The rule I learned at Olympia has not changed: the frame never waits. Everything since has been finding one more thing it was waiting for, and giving it a thread of its own.
The projects behind this post
Helbreath Empire
My most important project technically: an online MMORPG whose client and server I rewrote into my own engine. Millennium runs live, Empire is the next generation: OpenGL and GLSL, TCP/UDP networking off the frame, multi-threading, SSE2, an in-house UI, sprite and map editors, in-game Twitch streaming, VoIP, screen recording and more.
kazilib
My own C++ library, under every game on this page: packets and networking on ENet, threads and parallel jobs, time, an SFML 3 and ImGui layer, screen recording, WebAssembly builds for the browser, the Testerino test bridge and one FASTBuild runner. Helbreath Millennium, Helbreath Empire and my prototypes share it, so a fix in one reaches them all.