Here is my notes about the blog https://engineering.grab.com/one-click-data-ingestion-platform-with-apache-flink

1. Overview

• In legacy system, we have multiple services for data ingestion pipeline from datasource (MySQL, Postgres) -> Object Storage (S3).

• Introduce Flink CDC to save time for the process.

2. High-level Architecture

2.1. MySQL CDC -> S3

2.2. Kafka Connect -> S3

3. Dashboard

• Move onboarding data pipeline from days to minutes. Although we still have drop rates in onboarding process.

3.1. Drop rate Kafka Sink

3.2. Drop rate CDC Sink

4. Different from Service Mesh and Kafka

• Service mesh: synchronize grpc call -> pull, push.

• Kafka: asynchronize call

5. Different about arvo file and parquet files

Compress file with JSON, CSV -> save cost for big data storage in S3.

Feature	JSON	CSV	Avro	Parquet
Storage Layout	Text, hierarchical	Text, row-based	Binary, row-based	Binary, column-based

🔑 Core Difference

Avro = Row-based format Parquet = Column-based format

Avro

• Kafka (very common)
• Schema Registry (Confluent)
• Streaming systems

Parquet

• Spark, Flink, Presto, Trino
• Data lakes (Iceberg, Delta Lake, Hudi)

Mental Model

Avro = “Write fast, read full records” Parquet = “Read fast, scan columns

6. What is tool ?

• Spark: big data computing.

• Trino: Trino is a distributed SQL query engine designed to query data where it already lives, without moving it into a separate database.

            SQL Query
                │
                ▼
         +--------------+
         |    Trino     |
         | Query Engine |
         +--------------+
          /      |      \
         /       |       \
        ▼        ▼        ▼
  Parquet     MySQL     Iceberg
  (S3)      PostgreSQL  ksqlDB (allow to query in Kafka)
  

• Iceberg: storage big data with manifest

• Flink: straming flatform.

• Kafka Connect: connect to kafka

• Debezium CDC: capture change logs, allow to integration failure messages with Dead Letter Queue (DLQ) Kafka

• ksqlDB: allow query in Kafka

6. Learning notes

• Learning about integration: CDC Database and Kafka.

• Learning about query engine: Hive, Trino.

• Learning about distributed big data compute: Spark.

• Learning about distributed storage: Iceberge.