Grab Engineering - Scaling Grab's Data Lake - Our journey to Apache Iceberg adoption

Here is my notes about the blog https://engineering.grab.com/our-journey-to-apache-iceberg-adoption

1. Concepts

Term Idea
S3 Stores the actual Parquet files.
Hive Traditional table format. Metadata points to partition folders.
Hive Metastore Stores table → partition → S3 location.
Iceberg Modern table format. Metadata points directly to data files.
Snapshot Current version of a table.
Manifest Metadata describing data files in a snapshot.
UnifiedSparkCatalog Routes Spark requests to Hive or Iceberg transparently.

2. Main Idea

Hive and Spark Architecture

Grab did not replace S3 or Parquet.

They replaced how metadata is managed.

2.1. Before Architecture

Spark
   │
Hive Metastore
   │
Parquet Files on S3

2.2. After Architecture

Spark
   │
UnifiedSparkCatalog
   │
Iceberg Metadata
   │
Parquet Files on S3

3. Why move from Hive to Iceberg?

3.1 Hive folder structure

Hive organizes data by directory partitions.

orders/

├── dt=2026-07-10/
│   ├── part-0001.parquet
│   ├── part-0002.parquet
│
├── dt=2026-07-11/
│   ├── part-0001.parquet
│
└── dt=2026-07-12/
    ├── part-0001.parquet
    ├── part-0002.parquet
    └── part-0003.parquet

Hive Metastore stores something like:

orders

↓

Partition:
dt=2026-07-12

↓

Location:
s3://orders/dt=2026-07-12/

It knows:

  • ✅ Partition name
  • ✅ Folder location

It does not know

  • which Parquet files exist
  • row count
  • column statistics
  • snapshots
  • transaction history

When Spark executes

SELECT *
FROM orders
WHERE dt='2026-07-12';

Workflow

Spark

↓

Hive Metastore

↓

Get partition folder

↓

S3 ListObjects()

↓

part-0001.parquet
part-0002.parquet
part-0003.parquet

↓

Read Parquet files

Spark has to discover the files by asking S3.

3.2 Iceberg folder structure

Iceberg no longer relies on partition folders.

orders/

├── metadata/
│   ├── v1.metadata.json
│   ├── snap-100.avro
│   ├── manifest-list.avro
│   └── manifest-001.avro
│
└── data/
    ├── 00001.parquet
    ├── 00002.parquet
    ├── 00003.parquet
    └── 00004.parquet

Notice:

There is no requirement for folders like:

dt=2026-07-12/

Instead, Iceberg metadata knows:

Snapshot

↓

Manifest

↓

00001.parquet
dt=2026-07-12
rows=2M

00002.parquet
dt=2026-07-12
rows=3M

00003.parquet
dt=2026-07-15
rows=2M

Same query

SELECT *
FROM orders
WHERE dt='2026-07-12';

Workflow

Spark

↓

Current Snapshot

↓

Manifest

↓

Read

00001.parquet

00002.parquet

No directory listing.

No searching S3.

Iceberg already knows which files belong to the query.

3.3 Key Difference

Hive

Metadata

↓

Partition

↓

Folder

↓

Discover files

Iceberg

Metadata

↓

Snapshot

↓

Manifest

↓

Data Files

Hive stores where a partition is located.

Iceberg stores which data files exist and metadata about each file.

3.4 Why is Iceberg faster?

The data files are still Parquet.

The storage is still S3.

The speed improvement comes from reducing metadata work.

Hive

Query

↓

Hive Metastore

↓

List S3 folder

↓

Find files

↓

Read data

Iceberg

Query

↓

Snapshot

↓

Manifest

↓

Read matching files

At small scale, the difference is small.

At Grab’s scale (petabytes of data and billions of S3 objects), avoiding S3 directory listings and leveraging rich metadata significantly reduces query planning time and S3 API costs.

4. Why introduce UnifiedSparkCatalog?

During migration, both Hive and Iceberg tables existed.

Table Format
orders Hive
payments Iceberg
drivers Iceberg

Without UnifiedSparkCatalog

SELECT * FROM hive.orders;

SELECT * FROM iceberg.payments;

Every application needs to know the table format.

With UnifiedSparkCatalog

             Spark

               │

    UnifiedSparkCatalog

       /             \

 Hive Catalog   Iceberg Catalog

Applications always execute

SELECT * FROM orders;

Internally

LoadTable("orders")

↓

Detect table format

↓

Hive Catalog
or
Iceberg Catalog

5. Learning Notes

  • Iceberg allow to describe metadata fields of item in manifest.

  • Hive only contains: partition -> s3 address.

  • For big data up to pentabytes -> distributed storage.

July 12, 2026