RocksDB is a high-performance embedded persistent key-value database originally developed by Facebook (Meta). It is optimized for high write throughput, low latency, and efficient SSD utilization.

1. Overview

Unlike MySQL or PostgreSQL, RocksDB is not a standalone database server. Instead, it is an embedded storage engine that runs inside your application process, similar to SQLite.

Official resources:

• GitHub: https://github.com/facebook/rocksdb
• Website: https://rocksdb.org/

Many large-scale distributed systems use RocksDB as their local storage engine, including:

• Apache Flink
• Kafka Streams
• CockroachDB
• TiKV
• Dgraph
• MyRocks (MySQL Storage Engine)

2. Why RocksDB?

Modern applications often need to store:

• Billions of key-value pairs
• Large application state
• High-frequency writes
• Fast recovery after crashes

Keeping everything in memory is expensive.

Traditional relational databases provide many features (SQL, joins, transactions), but those features also introduce overhead.

RocksDB focuses on one job:

Store and retrieve key-value data efficiently with persistence.

Example:

Key             Value
--------------------------
user:1001   →   JSON
user:1002   →   JSON
order:5001  →   Binary
session:88  →   Metadata

3. Embedded Database

Unlike MySQL:

Application
      │
      ▼
 MySQL Server
      │
      ▼
 Storage

RocksDB runs directly inside the application.

Application
      │
      ▼
   RocksDB
      │
      ▼
 SSD

There is:

• No database server
• No network communication
• No client/server protocol

Applications call RocksDB directly through its library.

Example:

db->Put(...);
db->Get(...);

This significantly reduces latency.

4. LSM Tree Architecture

RocksDB is built on an LSM Tree (Log-Structured Merge Tree) rather than a B+ Tree.

Instead of modifying data in place, writes are appended sequentially and merged later.

Write Request
      │
      ▼
 MemTable (Memory)
      │
      ▼
 Immutable MemTable
      │
      ▼
 Flush
      │
      ▼
 SSTable (Disk)

Benefits:

• Sequential writes
• High write throughput
• SSD-friendly storage

5. Write Path

When an application writes data:

Application
      │
      ▼
 Write Ahead Log (WAL)
      │
      ▼
 MemTable
      │
      ▼
 Flush
      │
      ▼
 SSTables

Step 1. Write Ahead Log (WAL)

The write is first appended to the WAL.

PUT user1001

This guarantees durability if the process crashes.

Step 2. MemTable

The data is inserted into an in-memory sorted structure.

Memory

A
B
C
D

Reads are served directly from the MemTable while the data remains in memory.

Step 3. Flush

When the MemTable becomes full, it becomes immutable.

MemTable

↓

Immutable

↓

Flush

The contents are written to disk as an SSTable.

Step 4. SSTable

Data is stored in immutable Sorted String Tables (SSTables).

Disk

SSTable 1

SSTable 2

SSTable 3

Since SSTables never change, writes remain fast.

6. Read Path

When reading data, RocksDB searches in order:

Read Request
      │
      ▼
 MemTable
      │
      ▼
 Immutable MemTable
      │
      ▼
 Block Cache
      │
      ▼
 SSTables

If the key exists in memory, RocksDB avoids disk access.

7. Compaction

Over time, many SSTables accumulate.

Level 0

SST1
SST2
SST3
SST4

RocksDB periodically merges them.

Compaction

↓

Level 1

Merged SST

Compaction:

• Removes deleted keys
• Removes overwritten values
• Reorders data
• Reduces read amplification

8. Multi-Level Storage

RocksDB organizes SSTables into multiple levels.

L0
 ├── SST
 ├── SST

L1
 ├── SST
 ├── SST

L2
 ├── SST
 ├── SST

As data moves down the levels:

• Files become larger
• Files overlap less
• Reads become more efficient

9. Why RocksDB is Fast

Several design decisions contribute to RocksDB’s performance.

Sequential Writes

Instead of random disk writes:

Random

A
      C
   B
         D

RocksDB writes sequentially:

A
B
C
D

This is significantly faster on SSDs.

Block Cache

Frequently accessed blocks are cached in memory.

Application

↓

Block Cache

↓

SSD

Repeated reads avoid disk I/O.

Bloom Filters

Before searching an SSTable, RocksDB checks a Bloom Filter.

Lookup

↓

Bloom Filter

↓

Not Found

↓

Skip Disk Read

This reduces unnecessary disk accesses.

10. Fault Tolerance

RocksDB persists data using the Write Ahead Log.

Application

↓

WAL

↓

Crash

↓

Recovery

↓

Replay WAL

After restarting, RocksDB replays the WAL to restore recent writes.

11. RocksDB in Apache Flink

One of the most common production uses of RocksDB is Apache Flink.

Flink operators maintain state.

Kafka

↓

Flink

↓

User State

Small state can remain entirely in memory.

Large state cannot.

Instead, Flink stores operator state in RocksDB.

Kafka

↓

Flink Operator

↓

RocksDB

↓

SSD

Benefits:

• Terabytes of state
• Fast checkpointing
• Fault recovery
• Scalable stream processing

This enables Flink to process very large streams without requiring all state to fit into RAM.

12. Common Use Cases

RocksDB is commonly used for:

• Stream processing state (Apache Flink)
• Kafka Streams state stores
• Metadata storage
• Caching
• Embedded databases
• Time-series storage
• Distributed databases

13. RocksDB vs Traditional Databases

14. Architecture Overview

                 Application
                      │
                      ▼
                  RocksDB API
                      │
          ┌───────────┴───────────┐
          ▼                       ▼
   Write Ahead Log          MemTable
          │                       │
          └───────────┬───────────┘
                      ▼
                 Flush to Disk
                      ▼
                 SSTables (L0)
                      ▼
                 Compaction
                      ▼
                SSTables (L1+)
                      │
                      ▼
                     SSD

15. Key Takeaways

• RocksDB is an embedded persistent key-value database developed by Facebook (Meta).
• It is optimized for high write throughput, low latency, and SSD-based storage.
• It uses an LSM Tree architecture instead of a B+ Tree.
• Data is written to a Write Ahead Log (WAL), stored in a MemTable, flushed to immutable SSTables, and periodically optimized through compaction.
• Features such as Bloom Filters, Block Cache, and multi-level compaction provide efficient reads while maintaining excellent write performance.
• Apache Flink uses RocksDB as its default state backend for managing large state that cannot fit entirely in memory.
• RocksDB serves as the storage engine for many large-scale distributed systems, making it a foundational component in modern data infrastructure.