← AI Hardware Sizer · elhashif.net

Tools · High availabilityTools · High availability

How high availability works in AI infrastructureCara kerja high availability di infrastruktur AI

What happens when a GPU server fails, and how to keep chatbots and other AI services running. Written for IT teams evaluating an on-premises AI platform.Apa yang terjadi saat satu server GPU mati, dan bagaimana menjaga chatbot serta layanan AI lain tetap berjalan. Ditulis untuk tim IT yang sedang mengevaluasi platform AI on-premises.

  • Run at least two copies of every model on different servers, both active, behind a load balancer (active-active).
  • When a server fails, the load balancer stops sending it requests within seconds and the surviving copy serves everyone. Chat history is kept because it lives in the application database, not in the model server.
  • Decide how much capacity must survive a failure: "keep running, slower" needs fewer GPUs; "full performance" (N+1) needs one extra server of spare capacity.
  • Jalankan minimal dua salinan setiap model di server berbeda, keduanya aktif, di belakang load balancer (active-active).
  • Saat satu server mati, load balancer berhenti mengirim permintaan ke server itu dalam hitungan detik dan salinan yang tersisa melayani semua pengguna. Riwayat chat tetap ada karena disimpan di database aplikasi, bukan di server model.
  • Tentukan berapa kapasitas yang harus bertahan saat gagal: "tetap jalan, lebih lambat" butuh GPU lebih sedikit; "performa penuh" (N+1) butuh satu server tambahan sebagai kapasitas cadangan.

1. The architecture1. Arsitekturnya

Model servers are stateless: they only turn a prompt into an answer. Everything that must survive (users, chat history, documents, vectors) lives in replicated databases, so any model copy can serve any request.Server model bersifat stateless: tugasnya hanya mengubah pertanyaan menjadi jawaban. Semua yang harus bertahan (pengguna, riwayat chat, dokumen, vektor) disimpan di database yang direplikasi, sehingga salinan model mana pun bisa melayani permintaan apa pun.

Active-active AI architecture: users, two load balancers, two GPU servers each with a model copy, replicated databases Users & applications Pengguna & aplikasi chat · API · WhatsApp Load balancer 1 Load balancer 2 active aktif standby · VIP 50% of requests 50% permintaan 50% of requests 50% permintaan GPU server A Server GPU A Model copy 1 (inference server) Salinan model 1 (inference server) Local model repository (NVMe) Repositori model lokal (NVMe) GPU server B Server GPU B Model copy 2 (inference server) Salinan model 2 (inference server) Local model repository (NVMe) Repositori model lokal (NVMe) App DB + vector DB (primary) DB aplikasi + vector DB (primer) users · chat history · documents pengguna · riwayat chat · dokumen App DB + vector DB (replica) DB aplikasi + vector DB (replika) kept in sync selalu tersinkron replication replikasi
Both GPU servers are active every day; nothing sits idle waiting for a failure. The two load balancers share a virtual IP (VIP): if the active one fails, the standby takes over the address.Kedua server GPU aktif setiap hari; tidak ada yang menganggur menunggu kegagalan. Dua load balancer berbagi satu virtual IP (VIP): bila yang aktif mati, yang standby mengambil alih alamat tersebut.

2. How the two model servers work together2. Bagaimana kedua server model bekerja sama

Short answer: they do not talk to each other. Each model copy is an independent, stateless inference server. Every request carries everything the model needs (question, chat history, retrieved document passages), so any copy can answer any request. The servers are coordinated by the layers around them: a router in front, shared databases behind, and identical model files and configuration on both.

Jawaban singkat: keduanya tidak saling berkomunikasi. Setiap salinan model adalah inference server yang berdiri sendiri dan stateless. Setiap permintaan membawa semua yang dibutuhkan model (pertanyaan, riwayat chat, potongan dokumen hasil pencarian), sehingga salinan mana pun bisa menjawab permintaan apa pun. Kedua server dikoordinasikan oleh lapisan di sekelilingnya: router di depan, database bersama di belakang, serta file model dan konfigurasi yang identik di keduanya.

The path of one chat questionPerjalanan satu pertanyaan chat

Request flow: user, load balancer, AI gateway, application, databases, inference server, answer stored UserPenggunabrowser / app Load balancerVIP · TLS Chat / RAG applicationAplikasi chat / RAGstateless, 2+ copiesstateless, 2+ salinan App DB: chat historyDB aplikasi: riwayat chat Vector DB: passagesVector DB: dokumen AI gateway / routerhealth, retries, routinghealth, retry, routing Inference server AInference server A Inference server BInference server B 1 2 3 4 5 or Batau B 6 answerstreams back6 jawabanmengalir balik no link between A and BA dan B tidak terhubung
  1. The user sends a question to one virtual IP.
  2. The load balancer passes it to any healthy copy of the application.
  3. The application reads the chat history from the app DB and finds relevant passages in the vector DB.
  4. It builds one complete prompt and sends it to the AI gateway.
  5. The gateway picks a healthy inference server (A or B). If that server fails before answering, the gateway can retry on the other one.
  6. The answer streams back word by word; the application saves it to the app DB, so the next question can go to either server.
  1. Pengguna mengirim pertanyaan ke satu virtual IP.
  2. Load balancer meneruskannya ke salinan aplikasi mana pun yang sehat.
  3. Aplikasi membaca riwayat chat dari DB aplikasi dan mencari potongan dokumen yang relevan di vector DB.
  4. Aplikasi menyusun satu prompt lengkap dan mengirimnya ke AI gateway.
  5. Gateway memilih inference server yang sehat (A atau B). Bila server itu gagal sebelum menjawab, gateway bisa mencoba ulang ke server lainnya.
  6. Jawaban mengalir balik kata per kata; aplikasi menyimpannya ke DB aplikasi, sehingga pertanyaan berikutnya bisa dilayani server mana pun.

What keeps the two copies consistentYang menjaga kedua salinan tetap konsisten

ConcernKebutuhanHow it is solvedSolusinyaTalks overLewat
Same model on both serversModel yang sama di kedua serverModel files pulled from one model registry or object store into each server's local NVMe, pinned by version and checksumFile model diambil dari satu model registry atau object storage ke NVMe lokal setiap server, dikunci versi dan checksum-nyaHTTPS / S3 API
Same settings (context length, precision, chat template)Setelan yang sama (panjang konteks, presisi, chat template)One configuration in Git, applied to every copy (GitOps); updates roll out one server at a timeSatu konfigurasi di Git, diterapkan ke semua salinan (GitOps); pembaruan dijalankan satu server bergantianGit · Kubernetes API
Knowing which copy is aliveMengetahui salinan mana yang hidupHealth and readiness checks from the load balancer, gateway or KubernetesHealth check dan readiness check dari load balancer, gateway, atau KubernetesHTTP
Conversation stateStatus percakapanStored in the replicated app DB; sessions and rate limits in a shared cacheDisimpan di DB aplikasi yang direplikasi; sesi dan rate limit di cache bersamaDB replicationReplikasi DB
Who owns the virtual IPSiapa pemegang virtual IPThe two load balancers elect an active one; the standby takes over the IP if it failsDua load balancer memilih yang aktif; yang standby mengambil alih IP bila yang aktif gagalVRRP
KV cache (the model's working memory)KV cache (memori kerja model)Each server keeps its own in GPU memory. It is lost on failure and rebuilt from the next prompt, which costs a little latency, not dataSetiap server menyimpan sendiri di memori GPU. Hilang saat gagal dan dibangun ulang dari prompt berikutnya: hanya menambah sedikit latensi, bukan kehilangan dataNot sharedTidak dibagi

When servers do talk: one model across serversKapan server saling bicara: satu model di beberapa server

A model too large for one server can be split across servers (tensor or pipeline parallelism). Those servers exchange data on every token over a fast network (NCCL on NVIDIA, RCCL on AMD, usually over RDMA). This is not HA: if one of them fails, that whole copy stops. For HA you still need a second complete copy.

Model yang terlalu besar untuk satu server bisa dipecah ke beberapa server (tensor atau pipeline parallelism). Server-server itu bertukar data di setiap token lewat jaringan cepat (NCCL di NVIDIA, RCCL di AMD, biasanya lewat RDMA). Ini bukan HA: bila salah satunya gagal, seluruh salinan itu berhenti. Untuk HA tetap dibutuhkan salinan lengkap kedua.

Optional: smarter routing and shared KV cacheOpsional: routing lebih cerdas dan KV cache bersama

Newer open-source projects route requests to the server that already holds the relevant KV cache, or share KV cache between servers, to speed up long chats and RAG. They improve performance and efficiency; basic HA works without them.

Proyek open source yang lebih baru mengarahkan permintaan ke server yang sudah menyimpan KV cache relevan, atau membagi KV cache antar server, untuk mempercepat chat panjang dan RAG. Ini meningkatkan performa dan efisiensi; HA dasar tetap jalan tanpa ini.

Open-source software by layerSoftware open source per lapisan

Pick one per layer. Licenses as listed on each project repository (November 2026); check them again before production.Pilih satu per lapisan. Lisensi sesuai yang tercantum di repositori setiap proyek (November 2026); cek ulang sebelum production.

LayerLapisanOpen-source options (license)Pilihan open source (lisensi)Role in HAPeran dalam HA
Virtual IP & load balancerVirtual IP & load balancerkeepalived (GPL-2.0) + HAProxy (GPL-2.0), NGINX (BSD-2), Envoy (Apache-2.0)One address for users, health checks, removes failed serversSatu alamat untuk pengguna, health check, mengeluarkan server yang gagal
AI gateway / routerAI gateway / routerLiteLLM (MIT core; enterprise features separate), Gateway API Inference Extension (Apache-2.0), vLLM production-stack (Apache-2.0)One OpenAI-compatible endpoint, retries and fallback between model serversSatu endpoint kompatibel OpenAI, retry dan fallback antar server model
Inference serverInference servervLLM (Apache-2.0), SGLang (Apache-2.0), Triton Inference Server (BSD-3)Runs the model on the GPUs; each copy is independentMenjalankan model di GPU; setiap salinan berdiri sendiri
OrchestrationOrkestrasiKubernetes (Apache-2.0) + KServe (Apache-2.0) oratau Ray Serve (Apache-2.0)Keeps 2+ copies running, restarts failed ones, rolling updatesMenjaga 2+ salinan berjalan, menyalakan ulang yang gagal, pembaruan bergilir
App databaseDatabase aplikasiPostgreSQL + Patroni (MIT)Chat history and users replicated, automatic primary failoverRiwayat chat dan pengguna direplikasi, failover primer otomatis
Shared cacheCache bersamaValkey (BSD-3)Sessions, rate limits, short-lived state shared by all app copiesSesi, rate limit, status sementara yang dipakai bersama semua salinan aplikasi
Vector databaseVector databaseQdrant (Apache-2.0), Milvus (Apache-2.0), pgvector (PostgreSQL License) on the Patroni clusterdi cluster PatroniDocument vectors replicated across serversVektor dokumen direplikasi antar server
Model filesFile modelSeaweedFS (Apache-2.0), Ceph RGW (LGPL)One source of model files, copied to each server's NVMeSatu sumber file model, disalin ke NVMe setiap server
ConfigurationKonfigurasiArgo CD (Apache-2.0)Same settings on every copy, from GitSetelan yang sama di setiap salinan, dari Git
MonitoringMonitoringPrometheus (Apache-2.0) + DCGM exporter (Apache-2.0) + Grafana (AGPL-3.0)Alerts on failed servers, GPU errors and slow answersPeringatan saat server gagal, GPU error, dan jawaban melambat
Optional: KV-cache-aware routingOpsional: routing sadar KV cachellm-d (Apache-2.0), LMCache (Apache-2.0), NVIDIA DynamoFaster long chats and RAG; not needed for basic HAChat panjang dan RAG lebih cepat; tidak wajib untuk HA dasar

Supported alternatives: the enterprise platforms in the sizer package these layers with vendor support. NVIDIA AI Enterprise ships NIM inference microservices; Red Hat OpenShift AI is built on KServe; SUSE AI runs on Rancher Prime; AMD Enterprise AI Suite is open source on Kubernetes.Alternatif dengan dukungan vendor: platform enterprise di sizer mengemas lapisan-lapisan ini beserta support resmi. NVIDIA AI Enterprise membawa NIM inference microservice; Red Hat OpenShift AI dibangun di atas KServe; SUSE AI berjalan di atas Rancher Prime; AMD Enterprise AI Suite adalah open source di atas Kubernetes.

3. What happens when a server fails3. Yang terjadi saat satu server mati

① Normal① Normal

LB 50% 50%

The load balancer checks every few seconds that each model server answers (health check) and splits requests between them.Load balancer memeriksa setiap beberapa detik apakah setiap server model merespons (health check) dan membagi permintaan di antara keduanya.

② Server B fails② Server B mati

LB 100%

Health checks fail, server B is removed, and server A receives every request. Answers being written on B are cut off; users resend them. Chat history is intact.Health check gagal, server B dikeluarkan, dan server A menerima semua permintaan. Jawaban yang sedang ditulis di B terputus; pengguna mengirim ulang. Riwayat chat tetap utuh.

③ Recovery③ Pemulihan

LB 50% ready ✓

Server B restarts and loads the model into GPU memory from its local NVMe. Only when the model is ready does the health check pass and the load balancer send it traffic again.Server B menyala lagi dan memuat model ke memori GPU dari NVMe lokal. Baru setelah model siap, health check lolos dan load balancer kembali mengirim trafik ke server itu.

4. How much capacity survives a failure?4. Berapa kapasitas yang tersisa saat gagal?

This is the business decision behind the HA option in the sizer. Capacity left after one failure = (servers − 1) ÷ servers, unless spare capacity is added.Inilah keputusan bisnis di balik opsi HA di sizer. Kapasitas tersisa setelah satu kegagalan = (jumlah server − 1) ÷ jumlah server, kecuali ditambah kapasitas cadangan.

No HATanpa HA1 server. A failure stops the service until it is repaired.1 server. Kegagalan menghentikan layanan sampai diperbaiki.
Keep running, slower (2 servers)Tetap jalan, lebih lambat (2 server)Each server carries half the peak load.Setiap server menanggung separuh beban jam sibuk.
Keep running, slower (3 servers)Tetap jalan, lebih lambat (3 server)More, smaller servers lose less when one fails.Server lebih banyak dan lebih kecil kehilangan lebih sedikit saat satu mati.
Full performance (N+1)Performa penuh (N+1)Peak load needs N servers; one extra server is spare capacity. With 1 server needed, that means 2× the hardware.Beban jam sibuk butuh N server; satu server tambahan sebagai kapasitas cadangan. Bila cukup 1 server, berarti hardware 2×.

"Slower" means: when the surviving servers are overloaded, inference servers queue requests that do not fit in GPU memory. Users wait longer for the first word of an answer; nothing is lost."Lebih lambat" artinya: saat server yang tersisa kelebihan beban, inference server mengantrekan permintaan yang tidak muat di memori GPU. Pengguna menunggu lebih lama sampai kata pertama jawaban muncul; tidak ada data yang hilang.

5. Why not just use VM high availability?5. Kenapa tidak memakai HA VM saja?

VM restart HA with GPU passthroughHA restart VM dengan GPU passthrough

  • A VM with a passed-through GPU generally cannot live-migrate.
  • Restarting it on another host needs an identical, free GPU there: expensive hardware that sits idle.
  • After restart the model must load again into GPU memory, so the service is down for that time.
  • VM dengan GPU passthrough umumnya tidak bisa live migration.
  • Restart di host lain butuh GPU identik yang kosong di sana: hardware mahal yang menganggur.
  • Setelah restart, model harus dimuat ulang ke memori GPU, jadi layanan berhenti selama waktu itu.

Active-active at the application levelActive-active di level aplikasi

  • Both copies serve users every day; GPUs are not idle.
  • Failover is a load balancer decision that takes seconds, not a VM boot.
  • The same design scales out: add a third server and both capacity and resilience grow.
  • Kedua salinan melayani pengguna setiap hari; GPU tidak menganggur.
  • Failover adalah keputusan load balancer dalam hitungan detik, bukan booting VM.
  • Desain yang sama bisa diperluas: tambah server ketiga, kapasitas dan ketahanan sama-sama naik.

VM HA still has a place for supporting services (load balancers, application servers, databases without GPUs).HA VM tetap berguna untuk layanan pendukung (load balancer, server aplikasi, database tanpa GPU).

6. What must be redundant6. Apa saja yang harus redundan

HA is only as strong as its weakest single component.HA hanya sekuat komponen tunggal yang paling lemah.

TrafficTrafik

  • Two load balancers (or ingress replicas) with a shared virtual IP
  • Health checks that test the model, not just the port
  • Dua load balancer (atau replika ingress) dengan virtual IP bersama
  • Health check yang menguji model, bukan hanya port

AI serversServer AI

  • At least two copies of every model, on different servers
  • A model repository on local NVMe in every server, so a restart does not depend on another server
  • Minimal dua salinan setiap model, di server berbeda
  • Repositori model di NVMe lokal setiap server, agar restart tidak bergantung pada server lain

DataData

  • Application database (users, chat history) replicated to a second server
  • Vector database and documents replicated, or on shared redundant storage
  • Backups on top of replication: replication also copies mistakes
  • Database aplikasi (pengguna, riwayat chat) direplikasi ke server kedua
  • Vector database dan dokumen direplikasi, atau di storage bersama yang redundan
  • Backup tetap perlu di atas replikasi: replikasi ikut menyalin kesalahan

Network & powerJaringan & daya

  • Two switches, bonded NICs on every server
  • Dual power supplies fed from different PDUs or circuits
  • Servers in different racks when possible
  • Dua switch, NIC bonding di setiap server
  • Dual power supply dari PDU atau sirkuit listrik berbeda
  • Server di rack berbeda bila memungkinkan

7. On Kubernetes-based AI platforms7. Di platform AI berbasis Kubernetes

Most enterprise AI platforms (NVIDIA AI Enterprise, Red Hat OpenShift AI, SUSE AI, AMD Enterprise AI Suite) run on Kubernetes, which provides the building blocks:Kebanyakan platform AI enterprise (NVIDIA AI Enterprise, Red Hat OpenShift AI, SUSE AI, AMD Enterprise AI Suite) berjalan di atas Kubernetes, yang menyediakan komponen dasarnya:

Kubernetes featureFitur KubernetesWhat it does for HAFungsinya untuk HA
replicas: 2+Keeps at least two copies of the model server runningMenjaga minimal dua salinan server model tetap berjalan
Pod anti-affinityPod anti-affinityForces those copies onto different serversMemaksa salinan tersebut berada di server berbeda
Readiness probeReadiness probeSends traffic only after the model is loaded in GPU memoryMengirim trafik hanya setelah model termuat di memori GPU
PodDisruptionBudgetPrevents maintenance from taking down every copy at onceMencegah maintenance mematikan semua salinan sekaligus
GPU operatorGPU operatorPrepares GPU drivers on every node, so a copy can start on any GPU serverMenyiapkan driver GPU di setiap node, agar salinan bisa berjalan di server GPU mana pun

8. Test it before you need it8. Uji sebelum dibutuhkan

Recommendation (practice, not a standard): run a failover drill before go-live and at least twice a year. Switch off one GPU server during business hours and measure how long until users notice, how slow answers become at peak, and how long until the server serves traffic again.Rekomendasi (praktik, bukan standar): lakukan uji failover sebelum go-live dan minimal dua kali setahun. Matikan satu server GPU di jam kerja, lalu ukur berapa lama sampai pengguna merasakan dampaknya, seberapa lambat jawaban di jam sibuk, dan berapa lama sampai server kembali melayani trafik.

9. The question to ask the business9. Pertanyaan untuk pihak bisnis

"If the AI is slower for a day, what does it cost the business? If it is down for a day?" If down is unacceptable but slower is fine, choose "keep running, slower". If both are unacceptable, choose N+1.

"Kalau AI-nya lebih lambat selama sehari, apa dampaknya ke bisnis? Kalau mati selama sehari?" Bila mati tidak bisa diterima tapi lambat masih boleh, pilih "tetap jalan, lebih lambat". Bila keduanya tidak bisa diterima, pilih N+1.