Here is learning concepts for distributed system through implementation.

1. The Messenger

1.1. Design a system where multiple services communicate reliably without shared memory. What primitives would you use and why?

• Shared memory: in same machine

• Message passing: different machines.

Answer: Message queues (Kafka, SQS) for async decoupling; gRPC/HTTP for synchronous RPCs. Each message must be self-contained. Discuss at-least-once vs at-most-once delivery, idempotency keys to handle retries, and correlation IDs for request tracing across services.

1.2. How does a service know which message it received is a response to a specific request it sent earlier?

Answer: Correlation/request IDs: the sender attaches a unique ID to each outgoing request. The receiver echoes this ID in its response. The sender maps incoming response IDs to pending callbacks. This is how HTTP/2 stream IDs, Kafka consumer group offsets, and Maelstrom msg_ids all work.

1. How to know the client

• Using request id.

1. How to know the offset of multiple packets

• Sequence number: the byte that send in the payload.

• Acknowledge number: the byte is expected next.

1.3. A service sends a request and never gets a response. How do you decide whether to retry?

• Use timeouts with exponential backoff and jitter. Retry only idempotent operations (GET, PUT with full replacement) or operations with idempotency keys.

• Use circuit breakers to stop retrying against a consistently failing downstream. Distinguish between 503 (retry) and 400/404 (do not retry).

1.4. What is the difference between a message broker and a service mesh? When would you choose each?

• A message broker (Kafka, RabbitMQ) provides async, durable message delivery with decoupled producers and consumers.

• A service mesh (Istio, Linkerd) handles synchronous service-to-service traffic with features like mTLS, retries, circuit breaking, and observability.

• Use a broker when you need temporal decoupling or fan-out; use a mesh for sync RPC with cross-cutting network concerns.

1.5. How would you debug a system where messages are being processed more than once?

• Check for duplicate delivery: is the queue at-least-once? Is the consumer crashing after processing but before acknowledging?

• Add idempotency: track processed message IDs in a store and skip duplicates.

• Use exactly-once semantics in Kafka (requires transactions + idempotent producers) for critical flows.

• Log message IDs at each processing step to trace duplicates.

1.6. Serialisation Message: JSON vs Protobuf vs MessagePack

Verdict: Start with JSON for correctness, migrate to Protobuf when payload size or parse latency becomes a bottleneck.

Example:

JSON:

{
  "id": 123,
  "name": "Alice",
  "active": true
}

Message Pack:

map(3)
string("id")
int(123)
string("name")
string("Alice")
string("active")
true

1.7. Transport: TCP or UDP

Verdict: TCP for reliability by default. UDP only when you implement your own reliability layer (like QUIC) or can tolerate loss (metrics, video).

1.8. Why Grpc sync call needed timeout:

1. Sends a message to another node

2. Blocks until a response is received (matched by in_reply_to)

3. Returns the response body

4. Times out after a configurable duration (default: 1 second)

1.7. Why server know what client request id

• Add request id in the header.

• Server response: what the client it reply to.

1.8. What is asynchronous RPC callback ?

1. Sends a message to a target node

2. Registers a callback function keyed by the outgoing msg_id

3. Returns immediately (non-blocking)

4. When a reply arrives (with matching in_reply_to), invokes the callback with the reply body

1.9. What is callback reaper to store message, when recipient crashes or the network drops ?

• DLQ for failure process consumer.

1.10. How server handle callback reaper ?

• Records the timestamp when each callback is registered

• Periodically scans for callbacks older than a threshold (default: 2 seconds)

• Removes expired callbacks and invokes them with a timeout error

• Reports how many callbacks were reaped

1.11. Using add jitter for thunderherd retry

• Add jitter for pods do not retry at the same time.

1.12. JSON Encode from and to Byte

• Merge from byte -> data structure: int, string, map.

• Convert it to JSON again.

1.13. Log for message tracing

• Add request, response and message_type.

1.14. Deduplication in network layer using LRU Cache

• Dedup by using (src, msg_id).

• But use LRU for dedup it -> but around message IDs (capacity: 1000)

1.15. Chaos Testing - Drop network packet loss

• When chaos mode is enabled, the node randomly drops a configurable percentage of outgoing messages (does not send them to stdout)

1.16. Benchmark Node Throughput and Latency

• Benchmark latency and throughput per node.

2. The Identifier