Design Web Crawler System - Follow-up questions - Intuition First, Need more How – Nguyễn Đức An – Software Engineer (Curious when learning, Intentional when doing)

Grill your weaknesses, be an ultra-learner
Ref: Ultra-learning, Scott H. Young

Coding Skills: Latter solution is better than previous solution.
System Design: Design high-level solution + verified each requirement and enhanced questions and facts (metrics, current system).

Telemetry First: Identify service dependencies and understand how services interact.
Explore Service Endpoints: Examine the HTTP and gRPC endpoints exposed by each service.
High-Level Thinking (Intuitive Curiosity): Understand the overall architecture and product vision from Figma designs and system design documents.
Explore the Flow: Trace the end-to-end request flow and identify the core functions within the codebase.
Testing: Run the application and test API endpoints to verify inputs, outputs, and expected behavior.
Coding: Optimize the implementation and perform thorough code reviews.
Monitoring: Add meaningful metrics, logs, and alerts to observe the health and performance of the feature.

Meta thinking first: Draw high-level idea map before doing anything.
Structured Thinking: Order the actions.
Implementation Skills: Understand full flow, not only tickets.
Retain long-term knowledge: store facts, feedback, concepts, active retrieve it in long-term knowledge base.
Present for impact (teach, convince, motivate): stimulate to explain your thinking for other people follow Feynman Technique, 3-sentences rules for other needs, use body language and focus to main point to grab user attention.
Improve quality of outreach, connections: enhance outreach to open communicate with strangers, value connections, and make service for community.

Fact is what happening. Proposal comes with trade-offs and reasons.
Needs are about survival. Wants are about desires.
How to learn new thing: See it (mostly 70%). Do it (for high-execution required skills). Feedback it (be corrective feedback with insights)

Design Web Crawler System - Follow-up questions - Intuition First, Need more How

Here is solutions for Design Web Crawler System - Follow-up questions.

Main topic:

Recursive level crawling per website: level 1, level 2 in a URL.
Parse HTML: to get a link.
Track URL Status: visualize content from a website.
Cronjob (Parse HTML, recursive push back link) to continue push back recursive link.

Web Crawler

July 26, 2026