I Built My Own Load Balancer In Python Without AI
In the age of AI, nothing is black and white anymore.
Everyone on the internet has an opinion, and everyone shares it. AI is going to take your job. No, AI won't take your job. Junior roles are done for, but senior engineers are exactly who companies will fight over. Pick any thread on any given day and you'll find all of these being argued at once, usually with equal confidence.
This is my website, so here's mine.
I think the age of AI is great. We ship fast. We build faster. But the part I keep coming back to is this: it's the one skilled developer who's quietly making absurd money right now. The developer who persisted, who has years of real experience, who can craft a production-ready system before writing a line — that person just got a 4x multiplier on their output. AI didn't replace them. It handed them a bigger lever.
Which is slightly uncomfortable, because it means the moat isn't the typing. It's the judgment. And judgment is the one thing you can't prompt your way into overnight.
So I decided to do something about it.
The problem with letting AI drive
When you let AI write everything, something quiet happens. You stop making the small decisions. You stop feeling the friction of "wait, how should these two things talk to each other?" You get output, skim it, accept it, move on. It works. It ships. And your own architectural muscle goes soft without you noticing.
I didn't want to wake up in five years and find out I'm a vibe coder.
So I picked something I've read about a hundred times in system design material but had never built with my own hands: a load balancer.
Do we need to write our own? No. Should you run a hand-rolled one in production? Absolutely not — use nginx, HAProxy, or whatever your cloud hands you. But should you build one for learning? A hundred percent. It's small enough to finish and deep enough to teach you something real about how traffic actually gets spread across servers.
Why Python, of all things
I wrote this in Python mostly because I knew nobody writes a load balancer in Python. 😂 I found a repo and an article walking through one in Go, and Go is arguably the "correct" choice here. But I didn't want correct. I wanted to spin something up in a language I'm comfortable in so the language never got in the way of the thinking. The whole point was to wrestle with the ideas, not fight a borrowed syntax I'd have to look up every ten seconds.
The rules I set for myself
To make this worth anything, I set two ground rules and stuck to them.
No AI anywhere in the actual code. No matter how ugly it got, no matter how badly I wanted to just ask, I wrote every line by hand. I gave it five hours a week. Honestly, I think every freelancer and every developer in their growth phase should carve out five hours a week for their core skill — no client work, no shortcuts, just you and the fundamentals.
Google and AI are for lookups only. If I forget the exact syntax for, say, a class-level variable versus an instance one in Python, I'm allowed to look it up. That's just remembering vocabulary. What I'm not allowed to do is ask for architecture, logic, or "how should I structure this." The thinking has to be mine. The syntax can come from anywhere.
That line between looking up a word and outsourcing the thinking turned out to be the whole point of the exercise.
Scope of the project
For this project I did not want to implement something crazy and I wanted to wrap things up in 5 hours of coding session. So I limited the goals for this project:
-
A simple load balancer. It should be able to balance the load between the server using an algorithm in this case it will be round robin algorithm. we can replace such things into separate files later on to add more algorithms but that is out of scope for today's session.
-
Health Checks. One day you are sleeping and you realize that a server is down oh no! oh my god! what to do now! and you are using Osaf Ali Sayed's very own production-ready load balancer. nobody would want their load balancer to crash just because a server crashed do they?🙂 We need to make sure if a server goes down, our load balancer can detect it and handle it properly. for this we will write a thread which will run along side the load balancer, it will keep checking the /health endpoint on all the servers. if a server is down we do not send request to that server.
-
Server Death Checks. If by chance we hit the server before the health check was able to mark it dead. we need to detect this during the request to make the system more robust. Ideally here we would redirect to another server if request failed but for now this is out of scope.
What actually happened
It took me five hours. I wrote 200 lines of code. That's it.
And it was an eye-opener.
There's a specific feeling you get when you build something entirely yourself, where you understand every line because you had to fight for it. I hadn't felt that in a while. Somewhere in the AI-assisted daily grind, I'd traded it away for speed without realizing I'd made the trade.
For the balancing, I went with round robin the simplest, most classic load-balancing algorithm there is. You keep a list of your backend servers, and each incoming request goes to the next server in line, cycling back to the start when you reach the end. Request one to server A, request two to B, request three to C, request four back to A. No weights, no cleverness. Just fair, even rotation.
Here's the round-robin selection:
class ProxyHandler(BaseHTTPRequestHandler):
counter = 0
lock = threading.Lock()
server_state = [True for i in servers]
# ... inside do_GET, under the lock ...
with ProxyHandler.lock:
checked = 0
while True:
if checked == len(servers):
# every server is dead
self.send_response(502)
self.end_headers()
self.wfile.write(b'{"error":"all servers are dead"}')
return
ProxyHandler.counter = (ProxyHandler.counter + 1) % len(servers)
if ProxyHandler.server_state[ProxyHandler.counter]:
choosen_index = ProxyHandler.counter
break
checked += 1
server_url = servers[choosen_index]
Then a server died, and things got interesting
Plain round robin has an obvious flaw: it doesn't know or care whether a server is actually alive. If server B falls over, round robin will happily keep sending it every third request, and every one of those requests just dies.
So I added health checks.
The idea is simple. A background thread runs on a loop, and every few seconds it pings a /health endpoint on each backend. If a server answers with a 200, it's marked alive. If it refuses the connection or answers with anything else, it gets marked dead. The router then skips any server currently marked dead when it's picking where to send the next request.
The nice part is that this cuts both ways. A server going down isn't a permanent death sentence. The health-check loop keeps pinging it anyway, so the moment it comes back up and answers a 200 again, it gets flipped back to alive and starts receiving traffic like nothing happened. No restart, no manual intervention. The balancer just notices and adjusts.
Here's the health-check loop:
def health_check_loop():
while True:
for i, backend in enumerate(servers):
host, port = backend.split(":")
outgoing_headers = {'Host': backend}
try:
conn = http.client.HTTPConnection(host, int(port))
conn.request("GET", '/health', headers=outgoing_headers)
resp = conn.getresponse()
if resp.status == 200:
ProxyHandler.server_state[i] = True
else:
ProxyHandler.server_state[i] = False
except ConnectionRefusedError:
ProxyHandler.server_state[i] = False
time.sleep(5)
# started as a background daemon thread so it runs alongside the server
t = threading.Thread(target=health_check_loop, daemon=True)
t.start()
And the request handler that picks a live server and forwards the request:
def do_GET(self):
path = self.path
headers = self.headers
outgoing_headers = dict(headers)
# only the tiny critical section is locked: read + bump the shared
# counter and pick a live server. everything slow stays outside.
with ProxyHandler.lock:
checked = 0
while True:
if checked == len(servers):
self.send_response(502)
self.end_headers()
self.wfile.write(b'{"error":"all servers are dead"}')
return
ProxyHandler.counter = (ProxyHandler.counter + 1) % len(servers)
if ProxyHandler.server_state[ProxyHandler.counter]:
choosen_index = ProxyHandler.counter
break
checked += 1
server_url = servers[choosen_index]
host, port = server_url.split(":")
outgoing_headers['Host'] = server_url
outgoing_headers['X-Forwarded-For'] = self.client_address[0]
# forwarding happens outside the lock, so requests run in parallel
try:
conn = http.client.HTTPConnection(host, int(port))
conn.request("GET", self.path, headers=outgoing_headers)
resp = conn.getresponse()
status = resp.status
body = resp.read()
self.send_response(status)
self.end_headers()
self.wfile.write(body)
except ConnectionRefusedError:
# server died mid-request: mark it dead and report failure
ProxyHandler.server_state[choosen_index] = False
self.send_response(503)
self.end_headers()
self.wfile.write(b'{"error":"this server is dead!"}')
server = ThreadingHTTPServer(('127.0.0.1', 8080), ProxyHandler)
server.serve_forever()
There's also a nice fallback baked in: if the handler walks the whole list and every server is marked dead, it doesn't just hang or crash. It returns a 502 saying all servers are down. And if a request gets routed to a server that dies right at that moment, the handler catches the refused connection, marks that server dead on the spot, and returns a 503 instead of silently failing.
The part that genuinely humbled me: threading
Here's what I did not expect to be the hard part. Threading is brutal to reason about.
A load balancer has to handle multiple requests at once, that's the entire job. So it's inherently concurrent, which means multiple threads are touching the same shared counter that tracks whose turn it is. And the second you have multiple threads reading and writing the same variable, you're one bad interleaving away from two requests grabbing the same "next" server, or the counter skipping, or worse.
This is the thing we all studied in college and mostly nodded along to: the race condition. It's very different to actually hit one in code you wrote yourself.
The fix is a lock. But the real lesson wasn't that I needed a lock, it was where. My first instinct was to wrap everything in the lock and move on. That "works," but it's stupid: it makes the whole thing synchronous. One request at a time. You've just built a load balancer that can't handle load, which defeats the entire purpose.
So I had to actually think about it. I ended up repositioning my code so the lock only wraps the tiny critical section, the part that reads and bumps the shared counter and picks the index. Everything else, the slow part, the actual forwarding of the request to the backend, happens outside the lock, so requests can genuinely run in parallel. Getting that boundary right meant understanding the context of my own code: which few lines truly can't be touched by two threads at once, and which lines are fine to run concurrently.
That was the moment the whole exercise paid for itself. Not the load balancer. The forced understanding of exactly where concurrency bites.
AI is a lever. A huge one. But a lever does nothing without someone who knows where to put it. The developers who'll thrive aren't the ones who prompt the fastest they're the ones who still understand the machine well enough to catch it when it's wrong.
So build the small thing by hand. Kill the autocomplete for a few hours. Feel the friction on purpose. Five hours a week is nothing against a career, and it's the cheapest insurance you can buy against becoming a vibe coder.
Here is the github repo if you want to see one file pulled together: github.com/OsafAliSayed/load-balancer