Go Guide

System design · Lesson 54 of 62 · about 90 min

Design a URL shortener

Design a service like bit.ly: short code generation with base62, redirects at scale with caching, analytics off the hot path, then build it end to end in Go with an API, a web page, tests and a Dockerfile.

RFC 9110: 301 and 302 redirects ↗Go: net/http routing patterns (Go 1.22) ↗
On this page · 11 sections
  1. 011. Requirements
  2. 022. Estimates
  3. 033. API
  4. 044. Data model
  5. 055. Generating codes
  6. 066. High-level design
  7. 077. Deep dives and scaling
  8. 08Build it: a URL shortener in Go
  9. 09Summary
  10. 10Check your understanding
  11. 11Practice in Go

1. Requirements#

Functional

  • Create a short link for a long URL; optionally choose a custom alias.
  • Visiting the short link redirects to the long URL.
  • See basic stats: click count and creation time.

Non-functional

  • Redirects are fast (p99 under ~50 ms at the server) and highly available: a broken redirect is a broken link on someone's website.
  • Codes are short (≤ 7 characters), unique, and not trivially guessable.
  • Read-heavy: about 100 redirects per link created.

Out of scope here: user accounts, link editing, QR codes.

2. Estimates#

From the fundamentals lesson: 100 million new links a month gives about 40 writes/s and 4,000 redirects/s on average (12,000 at peak), about 6 billion links in 5 years, roughly 3 TB of data, and a hot set of tens of gigabytes. Conclusions: one primary database handles the writes; reads need a cache; storage needs a plan for growth (partitioning by code later).

3. API#

Method and pathBody / resultNotes
POST /api/links{"url": "...", "alias": "optional"} → 201 {"code", "short_url"}validate the URL; 409 if the alias is taken
GET /{code}302 Location: <long url>404 if unknown
GET /api/links/{code}{"code", "url", "clicks", "created_at"}stats

4. Data model#

SQL
CREATE TABLE links (
    code        TEXT PRIMARY KEY,        -- the short code, also the partition key later
    url         TEXT NOT NULL,
    created_at  TIMESTAMPTZ NOT NULL DEFAULT now(),
    clicks      BIGINT NOT NULL DEFAULT 0
);

All reads are by primary key, so a key-value store would work equally well. Clicks are updated in batches by a background job, not on every redirect.

5. Generating codes#

ApproachHowProsCons
hash the URLfirst 7 base62 chars of SHA-256(url)same URL → same codecollisions must be detected and resolved
random code7 random base62 chars, retry on conflictunguessable, no coordinationa uniqueness check on every insert
counter + base62a unique numeric ID encoded in base62no collisions, very short codesguessable unless the ID is shuffled
key generation servicepre-generate unused codes, hand out batchesfast insertsanother service to run

A counter works well if the ID is shuffled before encoding (see the practice problems), and unique IDs across many servers are exactly what the Distributed ID generator case study builds. Base62 encoding, step by step:

6. High-level design#

Drawing diagram…

The redirect path is: look up the code in Redis; on a miss, read a replica and fill the cache; return 302; publish a click event without waiting.

301 or 302? A 301 Moved Permanently lets browsers cache the redirect forever, which saves load but hides later clicks. Most shorteners use 302 (or 307) to keep analytics and the ability to disable abusive links.

7. Deep dives and scaling#

  • Hot links: a viral link can get millions of hits a minute. The cache absorbs it; add a small in-process cache on each app server for the hottest keys.
  • Partitioning: shard the links table by code (hash partitioning) once one database is not enough.
  • Abuse: validate URLs, block known malicious domains, rate-limit link creation per IP or API key (see the rate limiter case study).
  • Expiry and cleanup: an expires_at column and a periodic job that deletes expired rows.

Build it: a URL shortener in Go#

You will build a working shortener: a JSON API, redirects, click counting, a small web page, and tests at every layer. Each step adds files; by the end the project looks like this:

shortener/
├── go.mod
├── main.go                 HTTP server: routes, handlers, wiring
├── main_test.go            API tests with httptest
├── internal/
│   ├── code/base62.go      encode/decode
│   ├── code/base62_test.go
│   ├── store/store.go      the Store interface + in-memory implementation
│   └── store/store_test.go
└── web/index.html          the frontend, embedded into the binary

Step 1: create the module

Shell
mkdir shortener && cd shortener
go mod init shortener
mkdir -p internal/code internal/store web
go.mod
module shortener
 
go 1.24

Step 2: base62 codes, with tests first

The domain logic lives in its own package so it can be tested without any HTTP. Write the test table first: it pins down the behavior, including the edge case 0 and invalid input.

internal/code/base62.go
Go
// Package code converts numeric IDs to short base62 strings and back.
package code
 
import (
	"errors"
	"strings"
)
 
const alphabet = "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ"
 
// ErrInvalid is returned by Decode for characters outside the alphabet.
var ErrInvalid = errors.New("invalid base62 code")
 
// Encode returns the base62 form of n.
func Encode(n uint64) string {
	if n == 0 {
		return "0"
	}
	var buf [11]byte // 62^11 > 2^64
	i := len(buf)
	for n > 0 {
		i--
		buf[i] = alphabet[n%62]
		n /= 62
	}
	return string(buf[i:])
}
 
// Decode is the inverse of Encode.
func Decode(s string) (uint64, error) {
	if s == "" {
		return 0, ErrInvalid
	}
	var n uint64
	for _, r := range s {
		d := strings.IndexRune(alphabet, r)
		if d < 0 {
			return 0, ErrInvalid
		}
		n = n*62 + uint64(d)
	}
	return n, nil
}
internal/code/base62_test.go
Go
package code
 
import "testing"
 
func TestRoundTrip(t *testing.T) {
	tests := []struct {
		n    uint64
		want string
	}{
		{0, "0"},
		{9, "9"},
		{10, "a"},
		{61, "Z"},
		{62, "10"},
		{125_000_000, "8sud2"},
	}
	for _, tc := range tests {
		got := Encode(tc.n)
		if got != tc.want {
			t.Errorf("Encode(%d) = %q, want %q", tc.n, got, tc.want)
		}
		back, err := Decode(got)
		if err != nil || back != tc.n {
			t.Errorf("Decode(%q) = %d, %v; want %d", got, back, err, tc.n)
		}
	}
}
 
func TestDecodeInvalid(t *testing.T) {
	for _, s := range []string{"", "ab-c", "hé"} {
		if _, err := Decode(s); err == nil {
			t.Errorf("Decode(%q): expected an error", s)
		}
	}
}

Run go test ./.... Seeing the tests pass before writing any HTTP code is the point of keeping logic in its own package.

Step 3: the store

Handlers will depend on an interface, not on a concrete database. The in-memory implementation is enough to build and test everything; step 7 swaps in PostgreSQL behind the same interface.

internal/store/store.go
Go
// Package store keeps links. Memory is a concurrency-safe in-memory implementation.
package store
 
import (
	"errors"
	"sync"
	"time"
 
	"shortener/internal/code"
)
 
var (
	ErrNotFound = errors.New("link not found")
	ErrConflict = errors.New("code already in use")
)
 
// Link is one shortened URL.
type Link struct {
	Code      string    `json:"code"`
	URL       string    `json:"url"`
	Clicks    int64     `json:"clicks"`
	CreatedAt time.Time `json:"created_at"`
}
 
// Store is what the HTTP layer needs. Any database can implement it.
type Store interface {
	Create(url, alias string) (Link, error)
	Resolve(code string) (Link, error) // also counts a click
	Get(code string) (Link, error)
}
 
// Memory implements Store with a map guarded by a mutex.
type Memory struct {
	mu     sync.Mutex
	links  map[string]*Link
	nextID uint64
	now    func() time.Time
}
 
func NewMemory() *Memory {
	return &Memory{links: map[string]*Link{}, nextID: 100_000, now: time.Now}
}
 
func (m *Memory) Create(url, alias string) (Link, error) {
	m.mu.Lock()
	defer m.mu.Unlock()
	c := alias
	if c == "" {
		for { // skip IDs whose code collides with a custom alias
			c = code.Encode(m.nextID)
			m.nextID++
			if _, taken := m.links[c]; !taken {
				break
			}
		}
	} else if _, taken := m.links[c]; taken {
		return Link{}, ErrConflict
	}
	l := &Link{Code: c, URL: url, CreatedAt: m.now().UTC()}
	m.links[c] = l
	return *l, nil
}
 
func (m *Memory) Resolve(c string) (Link, error) {
	m.mu.Lock()
	defer m.mu.Unlock()
	l, ok := m.links[c]
	if !ok {
		return Link{}, ErrNotFound
	}
	l.Clicks++
	return *l, nil
}
 
func (m *Memory) Get(c string) (Link, error) {
	m.mu.Lock()
	defer m.mu.Unlock()
	l, ok := m.links[c]
	if !ok {
		return Link{}, ErrNotFound
	}
	return *l, nil
}
internal/store/store_test.go
Go
package store
 
import (
	"errors"
	"sync"
	"testing"
)
 
func TestCreateAndResolve(t *testing.T) {
	s := NewMemory()
	l, err := s.Create("https://go.dev", "")
	if err != nil || l.Code == "" {
		t.Fatalf("Create: %+v, %v", l, err)
	}
	for range 3 {
		if _, err := s.Resolve(l.Code); err != nil {
			t.Fatal(err)
		}
	}
	got, _ := s.Get(l.Code)
	if got.Clicks != 3 || got.URL != "https://go.dev" {
		t.Errorf("Get = %+v, want 3 clicks", got)
	}
}
 
func TestAliasConflict(t *testing.T) {
	s := NewMemory()
	if _, err := s.Create("https://a.example", "docs"); err != nil {
		t.Fatal(err)
	}
	if _, err := s.Create("https://b.example", "docs"); !errors.Is(err, ErrConflict) {
		t.Errorf("want ErrConflict, got %v", err)
	}
	if _, err := s.Resolve("nope"); !errors.Is(err, ErrNotFound) {
		t.Errorf("want ErrNotFound, got %v", err)
	}
}
 
func TestConcurrentCreatesAreUnique(t *testing.T) {
	s := NewMemory()
	var wg sync.WaitGroup
	codes := make(chan string, 1000)
	for range 1000 {
		wg.Add(1)
		go func() {
			defer wg.Done()
			l, _ := s.Create("https://x.example", "")
			codes <- l.Code
		}()
	}
	wg.Wait()
	close(codes)
	seen := map[string]bool{}
	for c := range codes {
		if seen[c] {
			t.Fatalf("duplicate code %q", c)
		}
		seen[c] = true
	}
}

Run the tests with the race detector: go test -race ./.... The concurrency test would catch a missing lock.

Step 4: the HTTP API

newServer builds the router and returns an http.Handler, so tests can call it directly without opening a port. Go 1.22's patterns (POST /api/links, GET /{code}) do the routing and extract path values.

main.go
Go
package main
 
import (
	"context"
	_ "embed" // needed for //go:embed into a []byte
	"encoding/json"
	"errors"
	"log/slog"
	"net/http"
	"net/url"
	"os"
	"os/signal"
	"regexp"
	"syscall"
	"time"
 
	"shortener/internal/store"
)
 
//go:embed web/index.html
var indexHTML []byte
 
var aliasRE = regexp.MustCompile(`^[A-Za-z0-9_-]{3,32}$`)
 
func writeJSON(w http.ResponseWriter, status int, v any) {
	w.Header().Set("Content-Type", "application/json")
	w.WriteHeader(status)
	json.NewEncoder(w).Encode(v)
}
 
func writeError(w http.ResponseWriter, status int, msg string) {
	writeJSON(w, status, map[string]string{"error": msg})
}
 
func validURL(raw string) bool {
	u, err := url.ParseRequestURI(raw)
	return err == nil && (u.Scheme == "http" || u.Scheme == "https") && u.Host != ""
}
 
func newServer(st store.Store, baseURL string) http.Handler {
	mux := http.NewServeMux()
 
	mux.HandleFunc("GET /{$}", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/html; charset=utf-8")
		w.Write(indexHTML)
	})
 
	mux.HandleFunc("POST /api/links", func(w http.ResponseWriter, r *http.Request) {
		var req struct {
			URL   string `json:"url"`
			Alias string `json:"alias"`
		}
		r.Body = http.MaxBytesReader(w, r.Body, 4<<10) // refuse huge bodies
		if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
			writeError(w, http.StatusBadRequest, "body must be JSON: {\"url\": \"...\"}")
			return
		}
		if !validURL(req.URL) {
			writeError(w, http.StatusBadRequest, "url must be an absolute http(s) URL")
			return
		}
		if req.Alias != "" && !aliasRE.MatchString(req.Alias) {
			writeError(w, http.StatusBadRequest, "alias: 3-32 letters, digits, - or _")
			return
		}
		l, err := st.Create(req.URL, req.Alias)
		if errors.Is(err, store.ErrConflict) {
			writeError(w, http.StatusConflict, "alias already taken")
			return
		}
		if err != nil {
			writeError(w, http.StatusInternalServerError, "could not create link")
			return
		}
		writeJSON(w, http.StatusCreated, map[string]string{"code": l.Code, "short_url": baseURL + "/" + l.Code})
	})
 
	mux.HandleFunc("GET /api/links/{code}", func(w http.ResponseWriter, r *http.Request) {
		l, err := st.Get(r.PathValue("code"))
		if err != nil {
			writeError(w, http.StatusNotFound, "no such link")
			return
		}
		writeJSON(w, http.StatusOK, l)
	})
 
	mux.HandleFunc("GET /{code}", func(w http.ResponseWriter, r *http.Request) {
		l, err := st.Resolve(r.PathValue("code"))
		if err != nil {
			http.NotFound(w, r)
			return
		}
		http.Redirect(w, r, l.URL, http.StatusFound) // 302: keeps counting clicks
	})
 
	return mux
}
 
func main() {
	port := os.Getenv("PORT")
	if port == "" {
		port = "8080"
	}
	base := os.Getenv("BASE_URL")
	if base == "" {
		base = "http://localhost:" + port
	}
	srv := &http.Server{
		Addr:              ":" + port,
		Handler:           newServer(store.NewMemory(), base),
		ReadHeaderTimeout: 5 * time.Second,
		WriteTimeout:      10 * time.Second,
	}
	go func() {
		slog.Info("listening", "addr", srv.Addr)
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			slog.Error("server", "err", err)
			os.Exit(1)
		}
	}()
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()
	<-ctx.Done()
	shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()
	srv.Shutdown(shutdown)
}

The API tests drive the real handler with httptest. They cover the happy path and each error branch:

main_test.go
Go
package main
 
import (
	"encoding/json"
	"net/http"
	"net/http/httptest"
	"strings"
	"testing"
 
	"shortener/internal/store"
)
 
func do(t *testing.T, h http.Handler, method, path, body string) *httptest.ResponseRecorder {
	t.Helper()
	req := httptest.NewRequest(method, path, strings.NewReader(body))
	rec := httptest.NewRecorder()
	h.ServeHTTP(rec, req)
	return rec
}
 
func TestShortenRedirectStats(t *testing.T) {
	h := newServer(store.NewMemory(), "http://sho.rt")
 
	rec := do(t, h, "POST", "/api/links", `{"url":"https://go.dev/doc/"}`)
	if rec.Code != http.StatusCreated {
		t.Fatalf("create: status %d, body %s", rec.Code, rec.Body)
	}
	var created struct {
		Code     string `json:"code"`
		ShortURL string `json:"short_url"`
	}
	json.NewDecoder(rec.Body).Decode(&created)
	if !strings.HasPrefix(created.ShortURL, "http://sho.rt/") {
		t.Errorf("short_url = %q", created.ShortURL)
	}
 
	rec = do(t, h, "GET", "/"+created.Code, "")
	if rec.Code != http.StatusFound || rec.Header().Get("Location") != "https://go.dev/doc/" {
		t.Fatalf("redirect: %d %q", rec.Code, rec.Header().Get("Location"))
	}
 
	rec = do(t, h, "GET", "/api/links/"+created.Code, "")
	var stats store.Link
	json.NewDecoder(rec.Body).Decode(&stats)
	if stats.Clicks != 1 {
		t.Errorf("clicks = %d, want 1", stats.Clicks)
	}
}
 
func TestErrors(t *testing.T) {
	h := newServer(store.NewMemory(), "http://sho.rt")
	tests := []struct {
		name, method, path, body string
		want                     int
	}{
		{"not json", "POST", "/api/links", `nope`, http.StatusBadRequest},
		{"relative url", "POST", "/api/links", `{"url":"/just/a/path"}`, http.StatusBadRequest},
		{"bad scheme", "POST", "/api/links", `{"url":"javascript:alert(1)"}`, http.StatusBadRequest},
		{"bad alias", "POST", "/api/links", `{"url":"https://a.b","alias":"a b"}`, http.StatusBadRequest},
		{"unknown code", "GET", "/zzzzzz", "", http.StatusNotFound},
		{"unknown stats", "GET", "/api/links/zzzzzz", "", http.StatusNotFound},
	}
	for _, tc := range tests {
		t.Run(tc.name, func(t *testing.T) {
			if rec := do(t, h, tc.method, tc.path, tc.body); rec.Code != tc.want {
				t.Errorf("status %d, want %d (%s)", rec.Code, tc.want, rec.Body)
			}
		})
	}
	do(t, h, "POST", "/api/links", `{"url":"https://a.b","alias":"docs"}`)
	if rec := do(t, h, "POST", "/api/links", `{"url":"https://c.d","alias":"docs"}`); rec.Code != http.StatusConflict {
		t.Errorf("duplicate alias: status %d, want 409", rec.Code)
	}
}
 
func TestHomePage(t *testing.T) {
	rec := do(t, newServer(store.NewMemory(), ""), "GET", "/", "")
	if rec.Code != 200 || !strings.Contains(rec.Body.String(), "<form") {
		t.Errorf("home page: %d", rec.Code)
	}
}

The javascript: test matters: without the scheme check, your shortener would happily redirect visitors to script URLs, a classic security hole.

Step 5: the frontend

One HTML file with a form. JavaScript posts to the API, shows the short link with a copy button, and remembers recent links in localStorage. //go:embed web/index.html in main.go compiles it into the binary, so there is still only one file to deploy.

web/index.html
HTML
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Shortener</title>
  <style>
    body { font-family: system-ui, sans-serif; max-width: 640px; margin: 3rem auto; padding: 0 1rem; color: #1d2021; }
    form { display: flex; flex-wrap: wrap; gap: .5rem; }
    input { flex: 1 1 220px; padding: .6rem .8rem; border: 1px solid #bbb; border-radius: 8px; font-size: 1rem; }
    button { padding: .6rem 1rem; border: 0; border-radius: 8px; background: #076678; color: #fff; font-weight: 600; cursor: pointer; }
    .result { margin-top: 1rem; padding: .8rem 1rem; border-radius: 8px; background: #eef6f7; display: none; }
    .error { background: #fbe7e7; color: #9d0006; }
    li { margin: .3rem 0; }
  </style>
</head>
<body>
  <h1>Shorten a link</h1>
  <form id="f">
    <input id="url" type="url" placeholder="https://example.com/long/path" required>
    <input id="alias" placeholder="custom alias (optional)">
    <button>Shorten</button>
  </form>
  <div id="result" class="result"></div>
  <h2>Recent</h2>
  <ul id="recent"></ul>
  <script>
    const $ = (id) => document.getElementById(id);
    const recent = JSON.parse(localStorage.getItem("recent") || "[]");
    function render() {
      $("recent").innerHTML = "";
      for (const r of recent) {
        const li = document.createElement("li");
        li.innerHTML = `<a href="${r.short_url}">${r.short_url}</a> → ${r.url}`;
        $("recent").append(li);
      }
    }
    $("f").addEventListener("submit", async (e) => {
      e.preventDefault();
      const body = { url: $("url").value, alias: $("alias").value || undefined };
      const res = await fetch("/api/links", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body) });
      const data = await res.json();
      const box = $("result");
      box.style.display = "block";
      if (!res.ok) {
        box.className = "result error";
        box.textContent = data.error;
        return;
      }
      box.className = "result";
      box.innerHTML = `<a href="${data.short_url}">${data.short_url}</a> <button id="copy">Copy</button>`;
      $("copy").onclick = () => navigator.clipboard.writeText(data.short_url);
      recent.unshift({ short_url: data.short_url, url: body.url });
      recent.splice(10);
      localStorage.setItem("recent", JSON.stringify(recent));
      render();
    });
    render();
  </script>
</body>
</html>

Step 6: run it

Shell
go test -race ./...
go run .
Shell
curl -s -X POST localhost:8080/api/links -d '{"url":"https://go.dev/doc/"}'
# {"code":"q0U","short_url":"http://localhost:8080/q0U"}
curl -si localhost:8080/q0U | grep -E '^(HTTP|Location)'
# HTTP/1.1 302 Found
# Location: https://go.dev/doc/
curl -s localhost:8080/api/links/q0U
# {"code":"q0U","url":"https://go.dev/doc/","clicks":1,"created_at":"..."}

Open http://localhost:8080 for the web page.

Step 7: from prototype to production

  1. PostgreSQL. Create the links table from section 4, implement store.Store with database/sql (go get github.com/jackc/pgx/v5/stdlib), and allocate IDs with a sequence: INSERT INTO links (code, url) VALUES ($1, $2) after SELECT nextval('link_ids'). Run the same store tests against it.
  2. Redis cache for redirects: on GET /{code}, try GET link:<code> first; on a miss, read the database and SET link:<code> <url> EX 86400.
  3. Asynchronous clicks: instead of l.Clicks++ inside Resolve, send the code into a buffered channel; a background goroutine batches increments every second (UPDATE links SET clicks = clicks + $2 WHERE code = $1). At larger scale, publish to Kafka or a cloud queue.
  4. Rate limiting on POST /api/links with the token bucket from the next case study.
  5. Deploy with the Dockerfile from the fundamentals lesson. Set BASE_URL to your real domain.

Summary#

  • Requirements and estimates show the shape: few writes, many reads, a hot set that fits in memory.
  • Codes come from hashing, randomness, or a counter in base62; shuffle counters so codes are not guessable.
  • Redirects go through a cache; clicks are counted asynchronously; 302 keeps analytics possible.
  • The build keeps logic in internal/ packages behind interfaces, tests every layer, and embeds the frontend in one binary.

Check your understanding

Answer before scrolling back up. If you miss one, reread the matching section, then try again.

  1. Q1.Why does a shortener with 100:1 reads to writes put a cache in front of the database?

  2. Q2.How many distinct codes do 7 base62 characters give?

  3. Q3.What is the main downside of encoding a sequential counter as the short code?

  4. Q4.Why might a shortener use 302 instead of 301 redirects?

  5. Q5.Where should click analytics be recorded so redirects stay fast?

Practice in Go

Write a table test first, then the solution. On LeetCode, choose Go as the language. Open a hint only after 15 minutes stuck.

  • Encode and Decode TinyURL ↗medium
    Show hint

    A map plus a counter in base62 is enough for the problem; then think about guessability.

  • Add expiring linksbuild

    Add an optional expires_in (seconds) to POST /api/links. Expired codes return 410 Gone. Test it by injecting a fake clock into the store.

  • Unguessable codesbuild

    Replace the sequential code with id * 1_580_030_173 mod 62^7 (a bijection because the multiplier is coprime with 62^7). Prove with a test that 100,000 consecutive IDs give 100,000 distinct codes.

  • PostgreSQL storebuild

    Implement store.Store with database/sql and the pgx driver using the schema in step 7. Run the same store tests against both implementations.

Finished this lesson?

Mark it done to track your progress. It is saved in this browser only.