Design a notification system

Send email, SMS and push notifications at scale: fan-out through queues, user preferences, templates, idempotency keys so nothing is sent twice, retries with exponential backoff and a dead-letter list, then build a working notification service in Go with a live delivery log.

1. Requirements#

Functional: other services call one API with a user ID, a template name and data; the system renders the message per channel, honours the user's opt-outs, and sends through email (SES, SendGrid), SMS (Twilio) and push (APNs for iOS, FCM for Android). It records the status of every message.

Non-functional: the API answers in milliseconds even when providers are slow; no message is lost (at-least-once); a retried request does not send twice; urgent messages (password reset, one-time codes) are not stuck behind a marketing blast; providers' own rate limits are respected.

2. Estimates#

10 million daily users, about 5 notifications each per day: 50 million messages a day, about 600 per second on average and several thousand per second at peak (a marketing campaign, or a sports score pushed to everyone). Each status record is about 200 bytes, so 10 GB a day of delivery history, usually kept for 30–90 days.

The bottleneck is almost never your own servers. It is the providers: SMS costs money per message and has per-second limits, and push services throttle senders that misbehave.

3. High-level design#

Drawing diagram…

Notification API accepts the request, checks the idempotency key, and returns 202 Accepted right away.
Fan-out looks up the user's contact details and preferences, renders the template once per allowed channel, and puts one message per channel on that channel's queue.
Workers per channel pull messages and call the provider. Separate queues mean an SMS outage does not delay emails.
Retries: transient failures are retried with exponential backoff; messages that keep failing go to a dead-letter queue for inspection instead of retrying forever.
Status store records every attempt, which powers support tools ("did the user get the reset email?") and analytics.

4. Deep dives#

Idempotency. Callers retry when they time out, and the request may already have succeeded. The caller attaches a key that is the same on every retry (for example order-8812-shipped). The API stores accepted keys (in Redis with a TTL of a day or so) and ignores repeats.

Retries and backoff. Retry timeouts, 5xx responses and 429s; do not retry permanent errors such as an invalid number or an uninstalled app (APNs tells you the device token is no longer valid, so delete it). Wait longer after each failure: base × 2^attempt, capped, with random jitter so a crowd of failed messages does not retry in lockstep.

backoff.go

package main
 
import (
	"fmt"
	"time"
)
 
// backoff returns the wait before retry number attempt (1-based): base doubled each time, capped.
// In production, sleep a random duration between 0 and this value ("full jitter").
func backoff(attempt int, base, max time.Duration) time.Duration {
	d := base << (attempt - 1)
	if d > max || d <= 0 {
		return max
	}
	return d
}
 
func main() {
	var total time.Duration
	for a := 1; a <= 7; a++ {
		d := backoff(a, time.Second, 30*time.Second)
		total += d
		fmt.Printf("retry %d after %v (waited %v in total)\n", a, d, total)
	}
}

Output

retry 1 after 1s (waited 1s in total)
retry 2 after 2s (waited 3s in total)
retry 3 after 4s (waited 7s in total)
retry 4 after 8s (waited 15s in total)
retry 5 after 16s (waited 31s in total)
retry 6 after 30s (waited 1m1s in total)
retry 7 after 30s (waited 1m31s in total)

Priorities. Put one-time codes and security alerts on a separate high-priority queue with its own workers, so a 5-million-message campaign never delays a login code.

Per-user limits. Nobody wants 40 pushes in an hour. A per-user rate limit (the token bucket from the rate limiter lesson) drops or collapses low-priority messages into a digest.

Delivery guarantees. A worker can crash after the provider accepted the message but before it recorded "sent". When the message is redelivered, the user sees it twice. That is why these systems promise at-least-once, and why providers that accept an idempotency key of their own (many do) should be given one.

Build it: a notification service in Go#

You will build a notify package with templates per channel, user preferences, idempotency keys, a worker pool, retries with backoff and a dead-letter status, then an HTTP API and a page that shows deliveries as they happen. Providers are an interface, so tests use fakes and production would plug in SES, Twilio and FCM.

notifier/
├── go.mod
├── main.go                         HTTP API + demo providers
├── main_test.go
├── internal/notify/notify.go       fan-out, dedupe, workers, retries
├── internal/notify/notify_test.go
└── web/index.html                  send notifications, watch the delivery log

Step 1: module

Shell

mkdir notifier && cd notifier && go mod init notifier
mkdir -p internal/notify web

go.mod

module notifier
 
go 1.24

Step 2: the service

Submit does the fast part (dedupe, preferences, rendering) and puts one job per channel on a buffered channel. A pool of workers does the slow part: calling providers and retrying. Sleep is injected so tests check the backoff schedule without waiting.

internal/notify/notify.go

// Package notify delivers notifications through pluggable channels with
// user preferences, idempotency keys, retries and a dead-letter status.
package notify
 
import (
	"context"
	"errors"
	"fmt"
	"strings"
	"sync"
	"text/template"
	"time"
)
 
// Channel is a delivery route.
type Channel string
 
const (
	Email Channel = "email"
	Push  Channel = "push"
	SMS   Channel = "sms"
)
 
// channels is the fixed fan-out order, so results are deterministic.
var channels = []Channel{Email, Push, SMS}
 
// Request is what another service sends.
type Request struct {
	IdempotencyKey string            `json:"idempotency_key"`
	UserID         string            `json:"user_id"`
	Template       string            `json:"template"`
	Data           map[string]string `json:"data"`
}
 
// Message is one rendered notification for one channel.
type Message struct {
	ID      string
	Channel Channel
	To      string
	Body    string
}
 
// Provider sends messages through an outside service (SES, Twilio, APNs, FCM).
type Provider interface {
	Send(ctx context.Context, m Message) error
}
 
// ErrPermanent marks failures that retrying cannot fix, such as an invalid address.
var ErrPermanent = errors.New("permanent failure")
 
var (
	ErrDuplicate = errors.New("duplicate idempotency key")
	ErrClosed    = errors.New("service closed")
)
 
// User holds contact details and opt-outs.
type User struct {
	ID     string
	Email  string
	Phone  string
	Device string // push token
	OptOut map[Channel]bool
}
 
func (u User) address(c Channel) string {
	switch c {
	case Email:
		return u.Email
	case SMS:
		return u.Phone
	case Push:
		return u.Device
	}
	return ""
}
 
// Status of one message.
type Status string
 
const (
	Queued  Status = "queued"
	Sent    Status = "sent"
	Failed  Status = "failed" // gave up: the dead-letter list
	Skipped Status = "skipped"
)
 
// Record is the delivery history of one message.
type Record struct {
	ID       string  `json:"id"`
	Key      string  `json:"key"`
	UserID   string  `json:"user_id"`
	Channel  Channel `json:"channel"`
	Body     string  `json:"body"`
	Status   Status  `json:"status"`
	Attempts int     `json:"attempts"`
	Error    string  `json:"error,omitempty"`
}
 
// Config wires the service together.
type Config struct {
	Providers   map[Channel]Provider
	Users       map[string]User
	Templates   map[string]map[Channel]string // template name -> channel -> text/template source
	Workers     int
	QueueSize   int
	MaxAttempts int
	BaseBackoff time.Duration
	Sleep       func(time.Duration) // nil means time.Sleep
}
 
type job struct {
	rec *Record
	msg Message
}
 
// Service accepts requests and delivers them in the background.
type Service struct {
	cfg   Config
	tmpls map[string]map[Channel]*template.Template
	queue chan job
	wg    sync.WaitGroup
 
	qmu    sync.RWMutex // guards closed; held for reading while enqueueing
	closed bool
 
	mu   sync.Mutex // guards seen, recs and every Record's fields
	seen map[string]bool
	recs []*Record
}
 
// New parses all templates up front, so a typo fails at startup instead of at send time.
func New(cfg Config) (*Service, error) {
	if cfg.Workers <= 0 {
		cfg.Workers = 4
	}
	if cfg.QueueSize <= 0 {
		cfg.QueueSize = 1024
	}
	if cfg.MaxAttempts <= 0 {
		cfg.MaxAttempts = 3
	}
	if cfg.Sleep == nil {
		cfg.Sleep = time.Sleep
	}
	s := &Service{cfg: cfg, tmpls: map[string]map[Channel]*template.Template{}, queue: make(chan job, cfg.QueueSize), seen: map[string]bool{}}
	for name, byChannel := range cfg.Templates {
		s.tmpls[name] = map[Channel]*template.Template{}
		for c, src := range byChannel {
			t, err := template.New(name + "/" + string(c)).Option("missingkey=error").Parse(src)
			if err != nil {
				return nil, fmt.Errorf("template %s/%s: %w", name, c, err)
			}
			s.tmpls[name][c] = t
		}
	}
	return s, nil
}
 
// Start launches the worker pool.
func (s *Service) Start(ctx context.Context) {
	for range s.cfg.Workers {
		s.wg.Add(1)
		go func() {
			defer s.wg.Done()
			for j := range s.queue {
				s.deliver(ctx, j)
			}
		}()
	}
}
 
// Close stops accepting requests and waits until every queued message is handled.
func (s *Service) Close() {
	s.qmu.Lock()
	if !s.closed {
		s.closed = true
		close(s.queue)
	}
	s.qmu.Unlock()
	s.wg.Wait()
}
 
// Submit validates, dedupes, renders and enqueues one message per allowed channel.
func (s *Service) Submit(req Request) ([]Record, error) {
	if req.IdempotencyKey == "" {
		return nil, errors.New("idempotency_key is required")
	}
	u, ok := s.cfg.Users[req.UserID]
	if !ok {
		return nil, fmt.Errorf("unknown user %q", req.UserID)
	}
	byChannel, ok := s.tmpls[req.Template]
	if !ok {
		return nil, fmt.Errorf("unknown template %q", req.Template)
	}
 
	// Render before claiming the key, so a bad request can be fixed and resent with the same key.
	type planned struct {
		rec *Record
		msg Message
	}
	var plan []planned
	for _, c := range channels {
		t, ok := byChannel[c]
		if !ok {
			continue
		}
		var b strings.Builder
		if err := t.Execute(&b, req.Data); err != nil {
			return nil, fmt.Errorf("render %s: %w", c, err)
		}
		rec := &Record{Key: req.IdempotencyKey, UserID: u.ID, Channel: c, Body: b.String(), Status: Queued}
		switch {
		case u.OptOut[c]:
			rec.Status, rec.Error = Skipped, "user opted out"
		case u.address(c) == "":
			rec.Status, rec.Error = Skipped, "no address"
		case s.cfg.Providers[c] == nil:
			rec.Status, rec.Error = Skipped, "no provider"
		}
		plan = append(plan, planned{rec, Message{Channel: c, To: u.address(c), Body: rec.Body}})
	}
 
	s.qmu.RLock() // held until the jobs are enqueued, so Close cannot close the queue under us
	defer s.qmu.RUnlock()
	if s.closed {
		return nil, ErrClosed
	}
	s.mu.Lock()
	if s.seen[req.IdempotencyKey] {
		s.mu.Unlock()
		return nil, ErrDuplicate
	}
	s.seen[req.IdempotencyKey] = true
	out := make([]Record, 0, len(plan))
	for _, p := range plan {
		p.rec.ID = fmt.Sprintf("n%d", len(s.recs)+1)
		s.recs = append(s.recs, p.rec)
		out = append(out, *p.rec)
	}
	s.mu.Unlock()
 
	for _, p := range plan {
		if p.rec.Status == Queued {
			p.msg.ID = p.rec.ID
			s.queue <- job{p.rec, p.msg} // blocks when the queue is full: backpressure
		}
	}
	return out, nil
}
 
func (s *Service) deliver(ctx context.Context, j job) {
	p := s.cfg.Providers[j.msg.Channel]
	for attempt := 1; ; attempt++ {
		err := p.Send(ctx, j.msg)
		s.mu.Lock()
		j.rec.Attempts = attempt
		switch {
		case err == nil:
			j.rec.Status, j.rec.Error = Sent, ""
		case errors.Is(err, ErrPermanent) || attempt == s.cfg.MaxAttempts:
			j.rec.Status, j.rec.Error = Failed, err.Error()
		default:
			j.rec.Error = err.Error() // still queued: will retry
		}
		done := j.rec.Status != Queued
		s.mu.Unlock()
		if done {
			return
		}
		s.cfg.Sleep(s.cfg.BaseBackoff << (attempt - 1))
	}
}
 
// Records returns a snapshot of every message, oldest first.
func (s *Service) Records() []Record {
	s.mu.Lock()
	defer s.mu.Unlock()
	out := make([]Record, len(s.recs))
	for i, r := range s.recs {
		out[i] = *r
	}
	return out
}

Tests use a fake provider that can fail a given address a set number of times, or permanently, and a Sleep that records the waits instead of sleeping.

internal/notify/notify_test.go

package notify
 
import (
	"context"
	"errors"
	"fmt"
	"slices"
	"sync"
	"testing"
	"time"
)
 
type fakeProvider struct {
	mu    sync.Mutex
	sent  []Message
	fails map[string]int  // address -> transient failures left
	perm  map[string]bool // address -> always fails permanently
}
 
func (f *fakeProvider) Send(_ context.Context, m Message) error {
	f.mu.Lock()
	defer f.mu.Unlock()
	if f.perm[m.To] {
		return fmt.Errorf("invalid address %s: %w", m.To, ErrPermanent)
	}
	if f.fails[m.To] > 0 {
		f.fails[m.To]--
		return errors.New("provider timeout")
	}
	f.sent = append(f.sent, m)
	return nil
}
 
type sleepLog struct {
	mu sync.Mutex
	d  []time.Duration
}
 
func (s *sleepLog) Sleep(d time.Duration) {
	s.mu.Lock()
	defer s.mu.Unlock()
	s.d = append(s.d, d)
}
 
func newTest(t *testing.T, f *fakeProvider, sl *sleepLog) *Service {
	t.Helper()
	s, err := New(Config{
		Providers: map[Channel]Provider{Email: f, SMS: f, Push: f},
		Users: map[string]User{
			"alice": {ID: "alice", Email: "alice@example.com", Phone: "+15550001", Device: "tok-alice"},
			"bob":   {ID: "bob", Email: "bob@example.com", Phone: "+15550002", OptOut: map[Channel]bool{SMS: true}},
		},
		Templates: map[string]map[Channel]string{
			"shipped": {
				Email: "Hi {{.name}}, order {{.order}} has shipped.",
				SMS:   "Order {{.order}} shipped.",
				Push:  "Order {{.order}} is on its way",
			},
		},
		Workers:     1,
		MaxAttempts: 3,
		BaseBackoff: 100 * time.Millisecond,
		Sleep:       sl.Sleep,
	})
	if err != nil {
		t.Fatal(err)
	}
	s.Start(context.Background())
	return s
}
 
func shipped(key, user string) Request {
	return Request{IdempotencyKey: key, UserID: user, Template: "shipped", Data: map[string]string{"name": "Alice", "order": "8812"}}
}
 
func byChannel(recs []Record, c Channel) Record {
	for _, r := range recs {
		if r.Channel == c {
			return r
		}
	}
	return Record{}
}
 
func TestFanOutPreferencesAndDedupe(t *testing.T) {
	f := &fakeProvider{}
	s := newTest(t, f, &sleepLog{})
 
	recs, err := s.Submit(shipped("k1", "alice"))
	if err != nil || len(recs) != 3 {
		t.Fatalf("alice: %v, %d records", err, len(recs))
	}
	recs, err = s.Submit(shipped("k2", "bob"))
	if err != nil {
		t.Fatal(err)
	}
	if r := byChannel(recs, SMS); r.Status != Skipped || r.Error != "user opted out" {
		t.Errorf("bob sms: %+v", r)
	}
	if r := byChannel(recs, Push); r.Status != Skipped || r.Error != "no address" {
		t.Errorf("bob push: %+v", r)
	}
	if _, err := s.Submit(shipped("k1", "alice")); !errors.Is(err, ErrDuplicate) {
		t.Errorf("repeat of k1: err = %v, want ErrDuplicate", err)
	}
	s.Close()
 
	if len(f.sent) != 4 {
		t.Fatalf("sent %d messages, want 4 (alice x3, bob email)", len(f.sent))
	}
	if !slices.ContainsFunc(f.sent, func(m Message) bool { return m.Body == "Hi Alice, order 8812 has shipped." }) {
		t.Errorf("rendered email missing: %+v", f.sent)
	}
}
 
func TestRetryWithBackoff(t *testing.T) {
	f := &fakeProvider{fails: map[string]int{"alice@example.com": 2}}
	sl := &sleepLog{}
	s := newTest(t, f, sl)
	if _, err := s.Submit(shipped("k1", "alice")); err != nil {
		t.Fatal(err)
	}
	s.Close()
	r := byChannel(s.Records(), Email)
	if r.Status != Sent || r.Attempts != 3 {
		t.Fatalf("email: %+v, want sent after 3 attempts", r)
	}
	if want := []time.Duration{100 * time.Millisecond, 200 * time.Millisecond}; !slices.Equal(sl.d, want) {
		t.Errorf("backoff = %v, want %v", sl.d, want)
	}
}
 
func TestGiveUpAndPermanentFailure(t *testing.T) {
	f := &fakeProvider{fails: map[string]int{"alice@example.com": 10}, perm: map[string]bool{"+15550001": true}}
	s := newTest(t, f, &sleepLog{})
	if _, err := s.Submit(shipped("k1", "alice")); err != nil {
		t.Fatal(err)
	}
	s.Close()
	recs := s.Records()
	if r := byChannel(recs, Email); r.Status != Failed || r.Attempts != 3 || r.Error != "provider timeout" {
		t.Errorf("email: %+v, want failed after 3 attempts", r)
	}
	if r := byChannel(recs, SMS); r.Status != Failed || r.Attempts != 1 {
		t.Errorf("sms: %+v, want failed after 1 attempt (permanent)", r)
	}
	if r := byChannel(recs, Push); r.Status != Sent {
		t.Errorf("push: %+v", r)
	}
}
 
func TestBadRequestDoesNotBurnTheKey(t *testing.T) {
	s := newTest(t, &fakeProvider{}, &sleepLog{})
	defer s.Close()
	bad := shipped("k1", "alice")
	bad.Data = map[string]string{"name": "Alice"} // no order
	if _, err := s.Submit(bad); err == nil {
		t.Fatal("missing template data should fail")
	}
	if _, err := s.Submit(shipped("k1", "alice")); err != nil {
		t.Fatalf("fixed request with the same key: %v", err)
	}
	if _, err := s.Submit(Request{IdempotencyKey: "k2", UserID: "zed", Template: "shipped"}); err == nil {
		t.Error("unknown user should fail")
	}
}
 
func TestSubmitAfterClose(t *testing.T) {
	s := newTest(t, &fakeProvider{}, &sleepLog{})
	s.Close()
	if _, err := s.Submit(shipped("k1", "alice")); !errors.Is(err, ErrClosed) {
		t.Errorf("err = %v, want ErrClosed", err)
	}
}

Step 3: the API and demo providers

The demo providers stand in for SES, Twilio and FCM: each takes 150 ms, the SMS one times out on every third call (so the log shows retries), and any address ending in invalid fails permanently. Carol's phone number is one of those.

main.go

package main
 
import (
	"context"
	_ "embed"
	"encoding/json"
	"errors"
	"fmt"
	"log/slog"
	"net/http"
	"os"
	"strings"
	"sync"
	"time"
 
	"notifier/internal/notify"
)
 
//go:embed web/index.html
var indexHTML []byte
 
type demoProvider struct {
	name      string
	failEvery int // every nth call times out (0 = never)
	mu        sync.Mutex
	calls     int
}
 
func (p *demoProvider) Send(_ context.Context, m notify.Message) error {
	if strings.HasSuffix(m.To, "invalid") {
		return fmt.Errorf("%s rejected %q: %w", p.name, m.To, notify.ErrPermanent)
	}
	p.mu.Lock()
	p.calls++
	n := p.calls
	p.mu.Unlock()
	time.Sleep(150 * time.Millisecond) // a provider round trip
	if p.failEvery > 0 && n%p.failEvery == 0 {
		return fmt.Errorf("%s timeout", p.name)
	}
	slog.Info("sent", "via", p.name, "to", m.To, "body", m.Body)
	return nil
}
 
func demoConfig() notify.Config {
	return notify.Config{
		Providers: map[notify.Channel]notify.Provider{
			notify.Email: &demoProvider{name: "email"},
			notify.SMS:   &demoProvider{name: "sms", failEvery: 3},
			notify.Push:  &demoProvider{name: "push"},
		},
		Users: map[string]notify.User{
			"alice": {ID: "alice", Email: "alice@example.com", Phone: "+15550001", Device: "tok-alice"},
			"bob":   {ID: "bob", Email: "bob@example.com", Phone: "+15550002", OptOut: map[notify.Channel]bool{notify.SMS: true}},
			"carol": {ID: "carol", Email: "carol@example.com", Phone: "+1555invalid", Device: "tok-carol"},
		},
		Templates: map[string]map[notify.Channel]string{
			"shipped": {
				notify.Email: "Hi {{.name}}, order {{.order}} has shipped.",
				notify.SMS:   "Order {{.order}} shipped.",
				notify.Push:  "Order {{.order}} is on its way",
			},
			"reply": {
				notify.Email: "{{.from}} replied to your post: {{.text}}",
				notify.Push:  "{{.from}}: {{.text}}",
			},
			"login_code": {
				notify.SMS:   "Your login code is {{.code}}",
				notify.Email: "Your login code is {{.code}}. It expires in 10 minutes.",
			},
		},
		Workers:     4,
		MaxAttempts: 4,
		BaseBackoff: 500 * time.Millisecond,
	}
}
 
func writeJSON(w http.ResponseWriter, code int, v any) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(code)
	json.NewEncoder(w).Encode(v)
}
 
func newServer(svc *notify.Service) http.Handler {
	mux := http.NewServeMux()
	mux.HandleFunc("GET /{$}", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/html; charset=utf-8")
		w.Write(indexHTML)
	})
	mux.HandleFunc("POST /api/notify", func(w http.ResponseWriter, r *http.Request) {
		var req notify.Request
		if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<16)).Decode(&req); err != nil {
			writeJSON(w, http.StatusBadRequest, map[string]string{"error": "invalid JSON"})
			return
		}
		recs, err := svc.Submit(req)
		switch {
		case errors.Is(err, notify.ErrDuplicate):
			// Not an error for the caller: the first request was accepted, this retry is a no-op.
			writeJSON(w, http.StatusOK, map[string]any{"duplicate": true})
		case errors.Is(err, notify.ErrClosed):
			writeJSON(w, http.StatusServiceUnavailable, map[string]string{"error": err.Error()})
		case err != nil:
			writeJSON(w, http.StatusBadRequest, map[string]string{"error": err.Error()})
		default:
			writeJSON(w, http.StatusAccepted, map[string]any{"records": recs})
		}
	})
	mux.HandleFunc("GET /api/deliveries", func(w http.ResponseWriter, r *http.Request) {
		writeJSON(w, http.StatusOK, svc.Records())
	})
	return mux
}
 
func main() {
	svc, err := notify.New(demoConfig())
	if err != nil {
		slog.Error("config", "err", err)
		os.Exit(1)
	}
	svc.Start(context.Background())
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	slog.Info("listening", "port", port)
	srv := &http.Server{Addr: ":" + port, Handler: newServer(svc), ReadHeaderTimeout: 5 * time.Second}
	if err := srv.ListenAndServe(); err != nil {
		slog.Error("server", "err", err)
		os.Exit(1)
	}
}

main_test.go

package main
 
import (
	"context"
	"encoding/json"
	"net/http/httptest"
	"strings"
	"testing"
	"time"
 
	"notifier/internal/notify"
)
 
type okProvider struct{}
 
func (okProvider) Send(context.Context, notify.Message) error { return nil }
 
func TestAPI(t *testing.T) {
	cfg := demoConfig()
	cfg.Providers = map[notify.Channel]notify.Provider{notify.Email: okProvider{}, notify.SMS: okProvider{}, notify.Push: okProvider{}}
	cfg.Sleep = func(time.Duration) {}
	svc, err := notify.New(cfg)
	if err != nil {
		t.Fatal(err)
	}
	svc.Start(context.Background())
	h := newServer(svc)
 
	post := func(body string) *httptest.ResponseRecorder {
		rec := httptest.NewRecorder()
		h.ServeHTTP(rec, httptest.NewRequest("POST", "/api/notify", strings.NewReader(body)))
		return rec
	}
	ok := `{"idempotency_key":"order-8812","user_id":"alice","template":"shipped","data":{"name":"Alice","order":"8812"}}`
	if rec := post(ok); rec.Code != 202 {
		t.Fatalf("first: %d %s", rec.Code, rec.Body)
	}
	if rec := post(ok); rec.Code != 200 || !strings.Contains(rec.Body.String(), `"duplicate":true`) {
		t.Fatalf("retry: %d %s", rec.Code, rec.Body)
	}
	if rec := post(`{"idempotency_key":"x","user_id":"alice","template":"nope"}`); rec.Code != 400 {
		t.Errorf("unknown template: %d", rec.Code)
	}
	if rec := post(`not json`); rec.Code != 400 {
		t.Errorf("bad JSON: %d", rec.Code)
	}
	svc.Close()
 
	rec := httptest.NewRecorder()
	h.ServeHTTP(rec, httptest.NewRequest("GET", "/api/deliveries", nil))
	var recs []notify.Record
	if err := json.NewDecoder(rec.Body).Decode(&recs); err != nil {
		t.Fatal(err)
	}
	if len(recs) != 3 {
		t.Fatalf("%d deliveries, want 3", len(recs))
	}
	for _, r := range recs {
		if r.Status != notify.Sent {
			t.Errorf("%s: %s", r.Channel, r.Status)
		}
	}
}

Step 4: a page to watch deliveries

Pick a user and an event, then send it. The table polls the delivery log, so you can watch messages go from queued to sent, see SMS retries, and see Carol's SMS land in the dead-letter state. Send again (same key) shows idempotency at work: the server answers duplicate and nothing new is queued.

web/index.html

HTML

<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Notification service</title>
  <style>
    body { font-family: system-ui, sans-serif; max-width: 860px; margin: 3rem auto; padding: 0 1rem; }
    select, button { padding: .45rem .7rem; border-radius: 8px; border: 1px solid #999; background: #fff; font: inherit; }
    button { cursor: pointer; }
    #msg { min-height: 1.5em; color: #555; }
    table { width: 100%; border-collapse: collapse; font-size: 14px; }
    th, td { text-align: left; padding: .4rem .5rem; border-bottom: 1px solid #ddd; vertical-align: top; }
    .st { font: 12px monospace; padding: .1rem .4rem; border-radius: 4px; color: #fff; }
    .queued { background: #b57614; } .sent { background: #79740e; } .failed { background: #9d0006; } .skipped { background: #7c6f64; }
  </style>
</head>
<body>
  <h1>Notification service</h1>
  <p>
    <select id="user"><option>alice</option><option>bob</option><option>carol</option></select>
    <select id="tmpl"><option value="shipped">order shipped</option><option value="reply">new reply</option><option value="login_code">login code</option></select>
    <button id="send">Send</button>
    <button id="again">Send again (same key)</button>
  </p>
  <p id="msg"></p>
  <table>
    <thead><tr><th>id</th><th>user</th><th>channel</th><th>status</th><th>tries</th><th>message</th></tr></thead>
    <tbody id="rows"></tbody>
  </table>
  <script>
    const data = {
      shipped: { name: "there", order: String(1000 + Math.floor(Math.random() * 9000)) },
      reply: { from: "Sam", text: "Great write-up!" },
      login_code: { code: String(Math.floor(100000 + Math.random() * 900000)) },
    };
    let lastKey = null;
    async function send(key) {
      const body = { idempotency_key: key, user_id: user.value, template: tmpl.value, data: data[tmpl.value] };
      const res = await fetch("/api/notify", { method: "POST", body: JSON.stringify(body) });
      const out = await res.json();
      msg.textContent = `${res.status} ` + (out.duplicate ? "duplicate: already accepted, nothing sent" :
        out.error ? out.error : `accepted ${out.records.length} message(s), key ${key}`);
    }
    document.getElementById("send").onclick = () => send(lastKey = crypto.randomUUID());
    document.getElementById("again").onclick = () => lastKey ? send(lastKey) : (msg.textContent = "Send something first.");
    async function refresh() {
      const recs = await (await fetch("/api/deliveries")).json();
      rows.innerHTML = "";
      for (const r of recs.reverse()) {
        const tr = document.createElement("tr");
        for (const v of [r.id, r.user_id, r.channel, null, r.attempts, r.body + (r.error ? ` (${r.error})` : "")]) {
          const td = document.createElement("td");
          if (v === null) {
            const s = document.createElement("span");
            s.className = "st " + r.status;
            s.textContent = r.status;
            td.append(s);
          } else td.textContent = v;
          tr.append(td);
        }
        rows.append(tr);
      }
    }
    setInterval(refresh, 400);
    refresh();
  </script>
</body>
</html>

Step 5: run it