Blog

The frame never waits: from one thread to many in Helbreath's servers

15 November 2026

A game server has one job that cannot wait: the frame. Many times a second it reads what the players did, moves the monsters, resolves the hits and tells everyone what happened. Anything else that happens on that thread, anything at all, every player feels as lag. I first learned this working on Helbreath Olympia, a Helbreath server where I joined as a programmer. Since then I have taken my own servers from one thread to several, one step at a time.

1. Before: one thread and a message loop

The original Helbreath server is a Windows program, and it runs the way Windows programs of its time did: one thread, waiting for messages.

int EventLoop()
{
 static unsigned short _usCnt = 0;
 register MSG msg;

	while( 1 ) {
		if( PeekMessage( &msg, NULL, 0, 0, PM_NOREMOVE ) ) {
			if( !GetMessage( &msg, NULL, 0, 0 ) ) {
				return msg.wParam;
			}
            TranslateMessage(&msg);
            DispatchMessage(&msg);
			UpdateScreen();
		}
		else WaitMessage();
	}
}

Everything arrives as a window message. Each player's socket is registered with WSAAsyncSelect, so when data comes in, Windows posts a message with a number of its own for that player, WM_ONCLIENTSOCKETEVENT plus the player's index, up to 2000 of them. The game's heartbeat is a multimedia timer set to 300 milliseconds, and even it does not run the game. It only posts one more message into the same queue:

void CALLBACK _TimerFunc(UINT wID, UINT wUser, DWORD dwUSer, DWORD dw1, DWORD dw2)
{
	PostMessage(G_hWnd, WM_USER_TIMERSIGNAL, wID, NULL);
}

So reading a packet, moving a monster and answering the timer all happen on the same thread, one after the other. That works because the slow work is somewhere else. The server does not touch a database at all. Every 10 minutes it sends each player's data over TCP to a separate log server and lets that one write it down:

if ((m_pMapList[m_pClientList[i]->m_cMapIndex]->m_bIsFightZone == FALSE) &&
	((dwTime - m_pClientList[i]->m_dwAutoSaveTime) > (DWORD)DEF_AUTOSAVETIME)) {
	bSendMsgToLS(MSGID_REQUEST_SAVEPLAYERDATA, i);

If the log server cannot be reached when a player leaves, the game server writes the character to a text file in a folder named after the date. In the whole game server there is exactly one call to CreateThread, and it opens a debug window.

It was multi-process instead of multi-threaded. The slow part, the disk, lived in another program, and the network between them was the queue.

2. SQL on the game thread

When Helbreath Empire started, in 2017, the separate log server was gone and the game server talked to MySQL itself. The main loop was simple: rest a millisecond, run a frame.

while (!g_close)
{
	std::this_thread::sleep_for(std::chrono::milliseconds(1));
	g_game->tick();
}

And saving a player happened inside that frame. Each save opened a brand new connection to MySQL, ran its queries and waited for every answer:

void Client::Save()
{
	unique_ptr<Sql> sql(Sql::create());
	// ...
}

{
	//ClockTest test("Query");
	mysql_query(sql()->get(), (char*)_str.c_str());
}

mysql_query blocks. While MySQL thinks, the thread waits, and since that thread is the game, the whole world waits with it. Nobody moves, no hit lands, no packet is answered. You can see the problem being worked around in the frame itself, which saves at most one player per frame, so a hundred saves due at once do not freeze the server a hundred times in a row:

if(!saved)
{
	if(Clock.elapsed(p->_save_tm, 10 _m, true)) {
		p->Save();
		saved = true;
	}
}

That spreads the cost; it does not remove it. Every save is still a small freeze that every player shares. And the queries that read the most were the worst. The item logs, every trade, drop and pick-up a player ever made, were paged like this, on the game thread:

sql->stat << "SELECT COUNT(player_id) FROM item_logs where player_id = " << p->_id << ";";
// ...
sql->stat << "select * from item_logs where player_id = " << p->_id << " limit " << limit << "," << 30 << ";";

A table that only grows, counted in full and paged with an offset that MySQL must walk past row by row. One curious player opening page 200 of their history could stop the game for everyone. Twenty days after the server went into the repository, the item logs were switched off.

One part already knew better. The login ran on its own thread from the first day, std::thread([&](){tick();}).detach();, because checking a password must never stop a game in progress.

3. The queue

In December 2017, over three days, the database moved off the game thread. The first piece was a queue that two threads can share without fighting over it:

template<typename Fnc>
void Consume(Fnc fnc)
{
	{
		std::lock_guard<std::mutex> g(_mutex);
		if (_push_queue->empty())
			return;
		swap(_push_queue, _read_queue);
		_ready = false;
	}
	while (!_read_queue->empty())
	{
		fnc(_read_queue->front());
		_read_queue->pop();
	}
}

It is two queues, not one. One side pushes into the first; the other side, under the lock, swaps the two pointers and lets go. The lock is held for the time it takes to swap two pointers, not for the time it takes to run the work, so the game thread never waits behind a query to add a new one.

The next day came the database thread, with two of those queues: requests going in and responses coming back.

void Database::Tick()
{
	Log::Out("Database thread started");
	while (!*(&v_close))
	{
		std::this_thread::sleep_for(std::chrono::milliseconds(1));

		GetRequests().Consume([&](unique_ptr<Request> & req)
		{
			req->_callback(req);
		});
	}
}

The game frame, once per tick, picks up whatever answers are ready and runs their second half:

g_db->GetResponses().Consume([&](unique_ptr<DatabaseResponse> & rsp)
{
	rsp->_callback(rsp);
});

Here is the part that changes how you think. Once a query runs on another thread, the world keeps going while it runs. By the time the answer comes back, the player who asked may have logged out, and another player may be sitting in the same slot. So an answer does not carry a pointer to the player. It carries the player's slot and a unique id, and finds the player again when it comes back. If the id does not match, the answer is dropped. Every job written since follows that rule.

4. Threads, each owning one thing

Today the server is a handful of threads, and each one owns one kind of work:

The database thread is now kazilib's lib::JobThread, shared by both games, and its design is in its first comment:

// One worker, so jobs never overlap and always run in queue order (a save queued
// before a load is written before it is read). The worker wakes as soon as a job
// is queued. Both Helbreath games' database threads are one (hb_lib db/jobs.h).
void Start(const std::string_view name, std::function<bool()> keep_running = {})
{
	_keep_running = std::move(keep_running);
	_worker.Start(name, [this](Worker & w)
	{
		while (!w.StopRequested() && (!_keep_running || _keep_running()))
		{
			_requests.WaitFor(kIdleWait);
			RunRequests();
		}
	});
}

Two decisions are packed in there. First, one worker, not a pool. A pool would run queries faster in parallel, but a player who logs out and back in quickly queues a save and then a load, and with a pool the load could win the race and read the character from before the save. One worker keeps the order. Second, it no longer checks the queue every millisecond. It sleeps until a job arrives, which is both faster to react and lighter on the CPU.

Every job is written as two halves, one per thread, and the code reads top to bottom in the order it runs:

REQ_PACK(p->_acc_id);
DB_REQ(p)
{
    UNPACK1(req, AccId, acc_id);             // on the database thread
    // ... queries ...
    RSP_PACK(found, name);
    DB_RSP()
    {
        null_returns(auto, p, rsp->GetPlayer());   // back on the game thread
        // ...

Saving a player shows the split well. The game thread, which owns the character, takes a snapshot of it as query text, which is quick. The database thread runs those queries as one transaction, which is slow but now bothers no one. Helbreath Millennium, the live server, made the same move in June 2026: until then its SQL also ran on the game thread.

The network got its own thread last, in kazilib's lib::NetworkThread, used by both servers:

// With the thread, the thread owns ENet's servicing: it wakes as soon as the
// socket has data (WSAEventSelect) or the program sends, and at least every
// kTickMs for ENet's own timers (pings, resends, timeouts). It receives, ACKs and
// sends at once and queues what it received; Service hands the program's thread
// the queue at the same point in the frame and in the same order as before, so
// the frame sees the same events in the same order, only the wire is no longer
// held by the frame.

The last sentence is the important one. The game sees exactly the same events in exactly the same order as before; only the waiting moved. A slow frame no longer delays the acknowledgements that keep every connection alive, so a heavy fight in one city no longer looks like packet loss to a player in another.

It also made the server harder to attack. Under a test flood from four clients, one round of network servicing used to take 19 seconds, the whole game held while it ran. With the thread, and a limit on how much it handles at once, it takes under one second.

5. Tables with millions of rows

Moving SQL off the game thread is what made big tables possible at all. On the game thread, a query that takes two seconds stops the world for two seconds. On the database thread, it makes one player wait two seconds for their page, and nobody else notices. That is the difference between "we can't keep item logs" and keeping every item movement in the game, millions of rows, and letting players search their own.

Off the game thread is not enough on its own, though, because there is one database thread and every save and every login queues behind whatever it is doing. The queries still have to be fast. A migration from 2026 explains what a missing index costs:

-- With only `idx_uid` present, that OR cannot become an index_merge union:
-- one side is indexed, the other is not, so the optimizer falls back to a
-- full scan of `item_logs` — for the COUNT(*) *and* the page SELECT, twice
-- per click, on a table that only grows (no pruning) and from the single
-- DB thread every save and login queues behind.

Compare today's item log search with the 2017 one from section 2:

Sql_Stmt(format("SELECT COUNT(*) AS `n` FROM (SELECT 1 FROM `item_logs` il WHERE {} LIMIT {}) c",
                Pred(cid, kind, ids, from, to), ilq::kCountCap + 1).c_str());
// ...
// The page: per character its first (page + 1) x 50 rows by the sort (its index stops
// there), merged, then the page's 50 (an IN (...) + ORDER BY would sort every match)

In the end: who waits for what

Where the slow work runsWho waits for a slow query
Originalanother program (the log server)nobody in the game, but nothing could be looked up
2017the game threadevery player on the server
Late 2017a database thread, through queuesthe player who asked
Todaya thread for each kind of work, in both gamesthe player who asked, briefly, and the network never waits for the frame

The rule I learned at Olympia has not changed: the frame never waits. Everything since has been finding one more thing it was waiting for, and giving it a thread of its own.

The projects behind this post

Helbreath Empire

My most important project technically: an online MMORPG whose client and server I rewrote into my own engine. Millennium runs live, Empire is the next generation: OpenGL and GLSL, TCP/UDP networking off the frame, multi-threading, SSE2, an in-house UI, sprite and map editors, in-game Twitch streaming, VoIP, screen recording and more.

C++23, OpenGL / GLSL, ENet (UDP), MySQL, SFML, ImGui, FFmpeg, kazilib

kazilib

My own C++ library, under every game on this page: packets and networking on ENet, threads and parallel jobs, time, an SFML 3 and ImGui layer, screen recording, WebAssembly builds for the browser, the Testerino test bridge and one FASTBuild runner. Helbreath Millennium, Helbreath Empire and my prototypes share it, so a fix in one reaches them all.

C++23, ENet, SFML 3, Dear ImGui, FFmpeg, WebAssembly, FASTBuild