Here is solutions for Design Notification Service - Follow-up questions.

1. High-level Design

2. Evaluate the solution

Nice design! You’ve created a well-structured notification service with good separation of concerns through channel-specific queues and proper worker patterns. Your approach to handling user preferences and preventing duplicate notifications shows solid distributed systems thinking, though there are some critical gaps around data consistency and failure handling that would need addressing before production deployment.

2.1. Positives

• Using separate SQS queues for each notification channel (iOS, Android, SMS, Email) provides excellent isolation and allows independent scaling of workers based on channel-specific load patterns

• The duplicate prevention strategy using notification ID lookups before sending prevents erroneous reprocessing, which is crucial for maintaining user trust in a notification system

• Including circuit breakers for third-party provider failures shows good defensive programming and will prevent cascading failures when external services experience issues

2.2. Potential Issues

• Race Condition in Duplicate Prevention

  • Your duplicate check reads the notification status from the database, then sends to the provider, then updates the status to success

  • Two workers processing the same notification simultaneously could both read “pending” status and proceed to send duplicate messages

  • This violates your requirement to avoid duplicate messages and could annoy users with multiple notifications

• Missing User Preference Lookup Caching

  • Every notification requires a database lookup for user preferences, creating unnecessary load on the User Database

  • With 10M notifications/day (116/second average, likely 500+ at peak), this creates significant database pressure

  • User preferences change infrequently, making them ideal candidates for caching to reduce database load

• No Retry Limit or Dead Letter Queue

  • Failed notifications are put back on the queue with an incremented attempt counter, but there’s no maximum retry limit mentioned

  • Without a retry limit and dead letter queue, permanently failing notifications will cycle indefinitely

  • This wastes resources and prevents identification of systemic issues with specific notification types

2.3. Follow-up Questions

1. How do you handle the scheduled notification feature mentioned in requirements? SQS delay is limited to 15 minutes, but users might schedule notifications days in advance

2. What happens when a user updates their notification preferences while messages are already in the queues?

3. How do you ensure the 1-second delivery SLA when third-party providers like APNS or FCM might have variable latency?

4. How does your system handle partial failures where some channels succeed but others fail for the same notification?

5. What’s your strategy for handling notification templates beyond email? Push notifications often need titles, bodies, and custom data

3. Data Flow

4. Follow-up questions

4.1. Current follow-up questions

4.2. How do you handle the scheduled notification feature mentioned in requirements? SQS delay is limited to 15 minutes, but users might schedule notifications days in advance

4.3. What happens when a user updates their notification preferences while messages are already in the queues?

4.4. How do you ensure the 1-second delivery SLA when third-party providers like APNS or FCM might have variable latency?

4.5. How does your system handle partial failures where some channels succeed but others fail for the same notification?

4.6. What’s your strategy for handling notification templates beyond email? Push notifications often need titles, bodies, and custom data

4.7. Other questions with same topic but in other threads