https://l1-ca0.github.io/tags/distributed-systems/ Recent content in Distributed-Systems on Li Cao's Blog Hugo -- 0.148.0 en-us Fri, 10 Apr 2026 12:00:00 +0000 https://l1-ca0.github.io/posts/distributed-systems-project-proposal/ Fri, 10 Apr 2026 12:00:00 +0000 https://l1-ca0.github.io/posts/distributed-systems-project-proposal/ <p>A high-performance inference gateway in C++ that routes client requests to a cluster of LLM serving replicas. The gateway provides KV-cache-aware routing via consistent hashing, weighted load balancing, fault tolerance with mid-stream failover and request hedging, circuit breaker for degraded replica detection, streaming token delivery, backpressure management, and zero-downtime rolling updates. Replicas participate in a SWIM gossip protocol for decentralized membership and failure detection.</p> <div style="position:relative;padding-bottom:75%;height:0;overflow:hidden;"> <iframe src="https://l1-ca0.github.io/ds_project_proposal.pdf" style="position:absolute;top:0;left:0;width:100%;height:100%;border:0;" title="Distributed Systems Project Proposal"></iframe> </div>