← PortfolioPortofolio · elhashif.net

Case study · Healthcare · Health check & HCIStudi kasus · Kesehatan · Health check & HCI

From “UNKNOWN” to highly available.Dari “UNKNOWN” menuju high availability.

Two engagements at a hospital in Indonesia. First, a health check of 6 production servers that all looked fine but were not. Second, a new 3-node hyperconverged platform on Proxmox VE and Ceph, built so that no single server, disk or switch can stop patient services.Dua pekerjaan di sebuah rumah sakit di Indonesia. Pertama, health check 6 server produksi yang terlihat baik-baik saja padahal tidak. Kedua, platform hyperconverged 3 node baru di atas Proxmox VE dan Ceph, dibangun agar tidak ada satu server, disk, atau switch yang bisa menghentikan layanan pasien.

6legacy servers health-checkedserver lama diperiksa
5 / 6with disk health unreadabledengan kesehatan disk tak terbaca
3node HCI cluster, all-NVMe Cephnode cluster HCI, Ceph all-NVMe
3×copies of every data blocksalinan setiap blok data

Summary for managementRingkasan untuk manajemen

  • Health check: all 6 servers running hospital services were online, yet the primary database was running hot and disk or memory health could not be seen on most servers. Every finding was ranked P1 to P3 with a 2-week to 3-month action plan.
  • New platform: a 3-node Proxmox VE cluster with Ceph storage keeps three copies of every block across the nodes. Each node connects to two stacked switches, so a server, a disk, a cable or a switch can fail without stopping VMs.
  • Why it matters: for a hospital, unplanned downtime means stalled patient services. Both engagements reduce that risk: one by making problems visible, the other by removing single points of failure.
  • Next: repurpose the old servers as backup infrastructure and complete a 3-2-1 backup strategy with an off-site copy.
  • Health check: keenam server yang menjalankan layanan rumah sakit dalam kondisi online, tetapi database utama berjalan terlalu panas dan kesehatan disk atau memori tidak terlihat di sebagian besar server. Setiap temuan diberi prioritas P1 sampai P3 dengan rencana tindakan 2 minggu sampai 3 bulan.
  • Platform baru: cluster Proxmox VE 3 node dengan storage Ceph menyimpan tiga salinan setiap blok di seluruh node. Setiap node terhubung ke dua switch yang di-stack, sehingga server, disk, kabel, atau switch boleh gagal tanpa menghentikan VM.
  • Mengapa penting: bagi rumah sakit, downtime tak terencana berarti layanan pasien tersendat. Kedua pekerjaan ini menekan risiko tersebut: yang satu membuat masalah terlihat, yang lain menghilangkan titik kegagalan tunggal.
  • Berikutnya: alihfungsikan server lama menjadi infrastruktur backup dan lengkapi strategi backup 3-2-1 dengan salinan off-site.

Part 1 · Health checkBagian 1 · Health check

What was checkedApa yang diperiksa

A health-check script ran on each server and collected the same five signals, followed by manual analysis of every result.Script health check dijalankan di setiap server dan mengumpulkan lima sinyal yang sama, lalu setiap hasilnya dianalisis manual.

UptimeUptimemodel, last bootmodel, boot terakhir
DisksDiskSMART health, wear, agekesehatan SMART, keausan, usia
CPUCPUtemperature vs. thresholdsuhu vs. ambang batas
MemoryMemoriECC errors (EDAC)error ECC (EDAC)
KernelKernelhardware error logslog error hardware

Status at a glanceStatus sekilas

Server rolePeran serverStatusPriorityPrioritas
Web server (HIS)Web server (SIMRS)WARNINGP2 · this weekP2 · minggu ini
Application nodeNode aplikasiWARNINGP2 · this weekP2 · minggu ini
Primary databaseDatabase utamaCRITICALP1 · nowP1 · segera
HIS v2 serverServer SIMRS v2OKP3 · routineP3 · rutin
Electronic medical records (RME)Rekam medis elektronik (RME)WARNINGP2 · this weekP2 · minggu ini
Database connection poolerConnection pooler databaseWARNINGP2 · this weekP2 · minggu ini

Key findingsTemuan utama

CRITICAL Primary database running hotDatabase utama terlalu panas

  • Processor 0 reached 72 °C against a high threshold of 83 °C: only 11 °C of margin.
  • Processor 1 ran at 44–59 °C. The imbalance points to a local cooling problem: dust, a failing fan, a loose heatsink or dried thermal paste.
  • The server had run for more than a year without a reboot.
  • Prosesor 0 mencapai 72 °C dengan ambang tinggi 83 °C: sisa margin hanya 11 °C.
  • Prosesor 1 berada di 44–59 °C. Ketimpangan ini menunjukkan masalah pendinginan lokal: debu, kipas bermasalah, heatsink longgar, atau thermal paste mengering.
  • Server berjalan lebih dari satu tahun tanpa reboot.

WARNING Disk health invisible on 5 of 6 serversKesehatan disk tak terlihat di 5 dari 6 server

  • SMART status read as UNKNOWN because the disks sit behind RAID controllers. That does not mean the disks are healthy; it means nobody can tell.
  • Fix: query disks through the controller (for example smartctl with the right device type, or the vendor RAID tool) and the out-of-band management interface.
  • Status SMART terbaca UNKNOWN karena disk berada di balik RAID controller. Itu tidak berarti disk sehat, melainkan tidak ada yang bisa mengetahuinya.
  • Solusi: baca disk melalui controller (misalnya smartctl dengan tipe device yang tepat, atau tool RAID dari vendor) dan antarmuka out-of-band management.

WARNING Memory errors unmonitored on 3 servers, including the databaseError memori tak terpantau di 3 server, termasuk database

  • No EDAC tooling on the primary database, the application node and the connection pooler. Undetected memory errors on a database can corrupt data silently.
  • Two servers also had no temperature monitoring, and firmware-level error reporting was disabled on some.
  • Tidak ada tool EDAC di database utama, node aplikasi, dan connection pooler. Error memori yang tak terdeteksi di database dapat merusak data tanpa disadari.
  • Dua server juga tanpa pemantauan suhu, dan pelaporan error di level firmware nonaktif di beberapa server.

WARNING A disk with 10 years of runtimeDisk dengan usia operasi 10 tahun

  • One application node disk showed 88,840 power-on hours, about 10.1 years. SMART still passes with no reallocated sectors, but failure risk rises sharply with age.
  • Planned replacement in the next maintenance cycle, with a verified backup first.
  • Satu disk di node aplikasi mencatat 88.840 jam operasi, sekitar 10,1 tahun. SMART masih PASSED tanpa reallocated sector, tetapi risiko kegagalan naik tajam seiring usia.
  • Penggantian direncanakan pada siklus pemeliharaan berikutnya, dengan backup terverifikasi lebih dulu.

Health check action planRencana tindakan health check

P1 within 1–2 weeksdalam 1–2 minggu

  • Inspect and clean the database server's cooling, then track CPU temperature trends.
  • Enable SMART monitoring through the RAID controllers on all servers.
  • Enable out-of-band management on all servers.
  • Periksa dan bersihkan pendinginan server database, lalu pantau tren suhu CPU.
  • Aktifkan pemantauan SMART melalui RAID controller di semua server.
  • Aktifkan out-of-band management di semua server.

P2 within 1 monthdalam 1 bulan

  • Install EDAC memory monitoring on the database, application node and pooler.
  • Install temperature sensors tooling where missing.
  • Schedule replacement of the 10-year-old disk.
  • Pasang pemantauan memori EDAC di database, node aplikasi, dan pooler.
  • Pasang tool sensor suhu di server yang belum punya.
  • Jadwalkan penggantian disk berusia 10 tahun.

P3 within 3 monthsdalam 3 bulan

  • Standardize the monitoring toolset across all servers.
  • Plan reboots for servers with more than a year of uptime to apply kernel updates.
  • Automated, threshold-based alerting (for example Zabbix or LibreNMS) instead of manual checks.
  • Standarkan toolset pemantauan di semua server.
  • Rencanakan reboot untuk server dengan uptime lebih dari setahun agar update kernel diterapkan.
  • Alerting otomatis berbasis ambang batas (misalnya Zabbix atau LibreNMS), menggantikan pengecekan manual.

Part 2 · New platformBagian 2 · Platform baru

A hyperconverged cluster with no single point of failureCluster hyperconverged tanpa titik kegagalan tunggal

Three new servers run Proxmox VE and Ceph together: every node provides compute and storage, and every block of data is kept on three nodes. Two switches, stacked into one logical switch, carry the traffic.Tiga server baru menjalankan Proxmox VE dan Ceph sekaligus: setiap node menyediakan compute dan storage, dan setiap blok data disimpan di tiga node. Dua switch yang di-stack menjadi satu switch logis membawa seluruh trafik.

Hospital routerRouter rumah sakit uplink · active-backupuplink · active-backup Switch A10/25GbE Switch B10/25GbE Node 1Proxmox VE + Ceph5 NVMe OSDs5 OSD NVMeZFS mirror bootboot ZFS mirrorNode 2Proxmox VE + Ceph5 NVMe OSDs5 OSD NVMeZFS mirror bootboot ZFS mirrorNode 3Proxmox VE + Ceph5 NVMe OSDs5 OSD NVMeZFS mirror bootboot ZFS mirror

Simplified diagram. Green dashed lines: the two switches are stacked into one logical switch. Blue lines: each node's LACP bond, with one link to each switch, so losing a switch, a cable or a port only halves bandwidth.Diagram disederhanakan. Garis hijau putus-putus: kedua switch di-stack menjadi satu switch logis. Garis biru: bond LACP setiap node, dengan satu link ke masing-masing switch, sehingga kehilangan satu switch, kabel, atau port hanya mengurangi bandwidth separuhnya.

The platform, layer by layerPlatform, lapis demi lapis

ComputeCompute

  • 3 nodes, each with 2 processors of 32 cores: 192 cores in total.
  • Proxmox VE 9.1 installed on a ZFS mirror of two boot SSDs per node.
  • The three nodes form one cluster; VMs are protected by HA and restart on another node if theirs fails.
  • 3 node, masing-masing 2 prosesor 32 core: total 192 core.
  • Proxmox VE 9.1 dipasang di ZFS mirror dari dua SSD boot per node.
  • Ketiga node membentuk satu cluster; VM dilindungi HA dan berjalan ulang di node lain jika node-nya gagal.

Storage: all-NVMe CephStorage: Ceph all-NVMe

  • 15 NVMe drives (5 per node) as Ceph OSDs, with monitors and managers on all three nodes.
  • The VM pool keeps 3 copies of every block and stays writable with 2: a whole node can be lost without losing data or stopping VMs.
  • 15 drive NVMe (5 per node) sebagai OSD Ceph, dengan monitor dan manager di ketiga node.
  • Pool VM menyimpan 3 salinan setiap blok dan tetap bisa ditulisi dengan 2: satu node utuh boleh hilang tanpa kehilangan data atau menghentikan VM.

NetworkJaringan

  • Two switches stacked into one logical switch, configured from a single point.
  • Each node has an LACP bond with one link to each switch, carrying cluster, Ceph and VM traffic.
  • The uplink to the hospital router uses an active-backup bond across both switches.
  • Dua switch di-stack menjadi satu switch logis, dikonfigurasi dari satu titik.
  • Setiap node memiliki bond LACP dengan satu link ke masing-masing switch, membawa trafik cluster, Ceph, dan VM.
  • Uplink ke router rumah sakit memakai bond active-backup melintasi kedua switch.

Operations standardsStandar operasional

  • A VM checklist for the hospital team: HA enabled, start at boot, QEMU guest agent with filesystem freeze for consistent backups, host CPU type with NUMA, and protection against accidental deletion.
  • Scheduled backup jobs from the start.
  • Full documentation: bill of materials, parameter worksheet, logical topology, rack elevation and cabling matrix.
  • Checklist VM untuk tim rumah sakit: HA aktif, start at boot, QEMU guest agent dengan freeze filesystem agar backup konsisten, tipe CPU host dengan NUMA, serta proteksi dari penghapusan tidak sengaja.
  • Backup job terjadwal sejak awal.
  • Dokumentasi lengkap: bill of materials, worksheet parameter, topologi logis, rack elevation, dan matriks kabel.

Recommended next stepsLangkah lanjutan yang direkomendasikan

CapacityKapasitas

  • Raise memory to at least 256 GB per node, so the remaining nodes can absorb all VMs during an HA event.
  • Naikkan memori menjadi minimal 256 GB per node, agar node yang tersisa mampu menampung semua VM saat terjadi HA event.

Backup 3-2-1Backup 3-2-1

  • Repurpose the old, health-checked servers as on-site backup infrastructure.
  • Add an off-site copy in cloud storage or colocation.
  • Copy 1: cluster · Copy 2: on-site · Copy 3: off-site.
  • Alihfungsikan server lama yang sudah di-health check menjadi infrastruktur backup on-site.
  • Tambahkan salinan off-site di cloud storage atau colocation.
  • Salinan 1: cluster · Salinan 2: on-site · Salinan 3: off-site.

SecurityKeamanan

  • Firewall for traffic filtering and network segmentation.
  • Endpoint security against malware and ransomware.
  • Firewall untuk filtering trafik dan segmentasi jaringan.
  • Endpoint security terhadap malware dan ransomware.

Why it matters for hospitalsMengapa penting bagi rumah sakit

The health check and the new platform support the technical side of these regulations. Compliance itself also depends on the hospital's policies and procedures.Health check dan platform baru mendukung sisi teknis regulasi berikut. Kepatuhan itu sendiri juga bergantung pada kebijakan dan prosedur rumah sakit.

Permenkes 24/2022 · Pasal 20 (2) & 29
RequirementKetentuanElectronic medical record storage must guarantee security, integrity, confidentiality and availability.Penyimpanan rekam medis elektronik harus menjamin keamanan, keutuhan, kerahasiaan, dan ketersediaan.
How the work supports itDukungan pekerjaanHealth check: finds failing cooling, aged disks and unmonitored memory early. Platform: three copies of every block, HA for VMs and redundant switching keep records available and intact.Health check: menemukan pendinginan bermasalah, disk tua, dan memori tak terpantau sejak dini. Platform: tiga salinan setiap blok, HA untuk VM, dan switching redundan menjaga rekam medis tetap tersedia dan utuh.
Permenkes 24/2022 · Pasal 20 (4)–(5)
RequirementKetentuanA backup system is required, kept at a place different from the facility, performed periodically and documented in SOPs.Wajib memiliki cadangan data, diletakkan di tempat berbeda dari lokasi fasyankes, dilakukan periodik, dan dituangkan dalam SOP.
StatusStatusScheduled backups run on the cluster today. The recommended 3-2-1 plan, with an off-site copy, is what completes this requirement.Backup terjadwal sudah berjalan di cluster. Rencana 3-2-1 yang direkomendasikan, dengan salinan off-site, yang akan melengkapi ketentuan ini.
PP 71/2019 · Pasal 24 (1)
RequirementKetentuanElectronic system operators must have procedures and means to secure their systems against disruption, failure and loss.Penyelenggara sistem elektronik wajib memiliki prosedur dan sarana pengamanan untuk menghindari gangguan, kegagalan, dan kerugian.
How the work supports itDukungan pekerjaanA repeatable check, prioritized findings and a monitoring roadmap turn prevention into a procedure.Pengecekan yang bisa diulang, temuan berprioritas, dan peta jalan pemantauan menjadikan pencegahan sebuah prosedur.

Sources: official texts ofSumber: teks resmi Permenkes No. 24 Tahun 2022 anddan PP No. 71 Tahun 2019. Technical mapping, not legal advice.Pemetaan teknis, bukan nasihat hukum.

When did you last look inside your servers?Kapan terakhir kali server Anda diperiksa menyeluruh?

“Online” is not the same as “healthy”. In a free 30-minute consultation we discuss your servers, what a health check would cover, and whether a highly available platform makes sense for you.“Online” tidak sama dengan “sehat”. Dalam konsultasi gratis 30 menit, kita bahas server Anda, cakupan health check yang sesuai, dan apakah platform high availability tepat untuk Anda.

Book a free consultationJadwalkan konsultasi gratis Save ContactSimpan Kontak
M. Afiful Hashif
M. Afiful HashifTechnical Consultant · SiSTECH Kharisma · elhashif.net

Customer identity, hardware brands, hostnames and identifying details are withheld for confidentiality.Identitas pelanggan, merek hardware, hostname, dan detail yang dapat mengidentifikasi tidak dicantumkan demi kerahasiaan.