System Design: URL Shortener from Scratch

URL shorteners are deceptively simple systems. The core functionality just converts long URLs to short ones and redirects users when they visit the short URL. But building one that handles millions of users with low latency and high availability reveals interesting challenges.

This case study walks through designing a URL shortener like bit.ly or TinyURL.

Introduction

Functional Requirements

Users need to:

• Shorten a long URL into a compact link
• Access the short URL and get redirected to the original
• Optionally set expiration dates
• Optionally customize the short code
• View statistics on link usage

Non-Functional Requirements

The system must be:

• Fast: Redirects under 100ms
• Available: Handle service disruptions gracefully
• Scalable: Millions of links, billions of redirects
• Durable: No lost links

Capacity Estimation

Daily active users: 100 million URL creations Redirect ratio: 100:1 (one creation, one hundred redirects)

Storage needed over 5 years:

• 100M links/day 365 days 5 years = 182.5 billion links
• At 500 bytes per link: ~91 TB

Redirect QPS: 100M * 100 / 86400 = ~115,000 QPS

Core Components

Short Code Generation

The short code is the heart of the system. It needs to be:

• Unique
• Random enough to be unpredictable
• Short (6-8 characters typical)
• URL-safe (alphanumeric)

Hash Function Approaches

Three approaches generate short codes:

Approach 1: MD5/SHA hash of long URL + salt

import hashlib
import base62

def generate_short_code(url: str, salt: str = "mysalt") -> str:
    hash_input = f"{url}:{salt}"
    md5_hash = hashlib.md5(hash_input.encode()).hexdigest()
    # Take first 8 characters and encode in base62
    return base62.encode(int(md5_hash[:8], 16))[:8]

# Example
short = generate_short_code("https://example.com/very/long/url/path")
# Result: "xV2bP9qK"

Problem: Same URL always produces same hash, enabling URL enumeration attacks.

Approach 2: Hash + counter for uniqueness

import hashlib
import base62
import time

def generate_short_code(url: str, salt: str = "mysalt") -> str:
    # Combine URL with timestamp and random to ensure uniqueness
    combined = f"{url}:{time.time_ns()}:{random.randint(0, 999999)}"
    md5_hash = hashlib.md5(combined.encode()).hexdigest()
    return base62.encode(int(md5_hash[:8], 16))[:8]

Approach 3: Counter-based (KGS approach)

Use a Key Generation Service that pre-generates short codes:

class KeyGenerationService:
    def __init__(self, batch_size=1000):
        self.batch_size = batch_size
        self.available_keys = []

    def get_next_key(self) -> str:
        if not self.available_keys:
            self._refill_batch()
        return self.available_keys.pop()

    def _refill_batch(self):
        # Generate batch from counter
        start = self._get_current_counter()
        for i in range(start, start + self.batch_size):
            self.available_keys.append(base62.encode(i))
        self._increment_counter(start + self.batch_size)

The counter approach guarantees uniqueness and allows easy key management.

Base62 Encoding

Base62 uses characters 0-9, A-Z, a-z giving 62 characters per position:

Length	Possible Combinations	Equivalent URLs
6	62^6 = 56.8 billion	Enough for all links
7	62^7 = 3.5 trillion	Generous headroom
8	62^8 = 218 trillion	Future-proof

Data Model

Relational Schema

CREATE TABLE urls (
    id BIGSERIAL PRIMARY KEY,
    short_code VARCHAR(12) NOT NULL UNIQUE,
    original_url TEXT NOT NULL,
    created_at TIMESTAMP WITH TIME ZONE DEFAULT NOW(),
    expires_at TIMESTAMP WITH TIME ZONE,
    is_custom BOOLEAN DEFAULT FALSE,
    creator_id BIGINT,
    click_count BIGINT DEFAULT 0,
    is_active BOOLEAN DEFAULT TRUE,

    CONSTRAINT urls_short_code_idx UNIQUE (short_code)
);

CREATE INDEX idx_urls_short_code ON urls(short_code);
CREATE INDEX idx_urls_creator ON urls(creator_id);
CREATE INDEX idx_urls_expires ON urls(expires_at) WHERE expires_at IS NOT NULL;

NoSQL Alternative (DynamoDB)

{
  "TableName": "urls",
  "KeySchema": [{ "AttributeName": "short_code", "KeyType": "HASH" }],
  "AttributeDefinitions": [
    { "AttributeName": "short_code", "AttributeType": "S" },
    { "AttributeName": "creator_id", "AttributeType": "N" }
  ],
  "GlobalSecondaryIndexes": [
    {
      "IndexName": "creator-index",
      "KeySchema": [{ "AttributeName": "creator_id", "KeyType": "HASH" }],
      "Projection": { "ProjectionType": "ALL" },
      "ProvisionedThroughput": {
        "ReadCapacityUnits": 100,
        "WriteCapacityUnits": 50
      }
    }
  ],
  "ProvisionedThroughput": {
    "ReadCapacityUnits": 1000,
    "WriteCapacityUnits": 500
  }
}

Caching Strategy

Redirect latency is critical. Cache aggressively.

Cache Structure

# Redis cache key pattern
cache_key = f"url:{short_code}"

# Cache value
cache_value = {
    "original_url": "https://example.com/very/long/path",
    "expires_at": "2027-01-01T00:00:00Z",
    "is_active": True
}

Cache TTL Strategy

CACHE_TTL = {
    "frequently_accessed": 3600,    # 1 hour for popular links
    "recently_created": 300,        # 5 minutes for new links
    "custom": 86400,                # 24 hours for custom links
    "expired": 60                   # 1 minute for recently expired
}

Write-Through Cache

async def create_short_url(url: str, custom_code: str = None) -> str:
    short_code = custom_code or generate_short_code(url)

    # Write to database
    await db.urls.create({
        "short_code": short_code,
        "original_url": url,
        "is_custom": custom_code is not None
    })

    # Write to cache
    await cache.set(f"url:{short_code}", {
        "original_url": url,
        "expires_at": None,
        "is_active": True
    }, ttl=CACHE_TTL["recently_created"])

    return short_code

Cache Miss Handling

async def get_original_url(short_code: str) -> Optional[str]:
    # Check cache first
    cached = await cache.get(f"url:{short_code}")
    if cached:
        return cached["original_url"]

    # Cache miss - fetch from database
    url_record = await db.urls.get(short_code=short_code)

    if not url_record:
        return None

    # Populate cache
    await cache.set(f"url:{short_code}", {
        "original_url": url_record.original_url,
        "expires_at": url_record.expires_at,
        "is_active": url_record.is_active
    }, ttl=CACHE_TTL["recently_created"])

    return url_record.original_url

Redirect Logic

HTTP Redirect Types

Status	Use Case	Browser Behavior
301	Permanent move	Caches redirect
302	Temporary redirect	No cache
303	Post -> Get	Converts to GET
307	Temporary	Preserves method
308	Permanent	Preserves method

For URL shorteners, typically use 301 (permanent) for SEO or 302 (temporary) for analytics tracking.

Redirect Handler

from fastapi import FastAPI, HTTPException, status
from fastapi.responses import RedirectResponse

app = FastAPI()

@app.get("/{short_code}")
async def redirect_to_original(short_code: str):
    # Check for special paths
    if short_code in ["health", "metrics", "docs"]:
        raise HTTPException(status_code=404)

    # Validate short code format
    if not is_valid_short_code(short_code):
        raise HTTPException(status_code=400, detail="Invalid short code")

    # Get original URL
    original_url = await get_original_url(short_code)

    if not original_url:
        raise HTTPException(status_code=404, detail="URL not found")

    # Track click asynchronously
    asyncio.create_task(track_click(short_code))

    return RedirectResponse(
        url=original_url,
        status_code=status.HTTP_302_FOUND
    )

Rate Limiting

Prevent abuse with rate limits per IP:

from slowapi import Limiter
from slowapi.util import get_remote_address

limiter = Limiter(key_func=get_remote_address)

@app.post("/shorten")
@limiter.limit("10/minute")
async def create_short_url(request: Request, url: str = Body(...)):
    # Validate URL
    if not is_valid_url(url):
        raise HTTPException(status_code=400, detail="Invalid URL")

    # Check if already shortened by this user
    existing = await find_existing_mapping(url, request.client.host)
    if existing:
        return {"short_url": f"https://short.ly/{existing.short_code}"}

    short_code = await create_mapping(url)
    return {"short_url": f"https://short.ly/{short_code}"}

High Availability Design

Database High Availability

-- PostgreSQL synchronous replication
ALTER SYSTEM SET synchronous_commit = on;
ALTER SYSTEM SET synchronous_standby_names = '*';

-- Read replicas for redirects
CREATE PUBLICATION url_shares FOR TABLE urls;

-- On replica
CREATE SUBSCRIPTION url_sub CONNECTION 'host=primary port=5432 dbname=urlshort' PUBLICATION url_shares;

Multiple Redis Instances

from rediscluster import RedisCluster

# Redis Cluster configuration
rc = RedisCluster(
    startup_nodes=[
        {"host": "redis-1", "port": 6379},
        {"host": "redis-2", "port": 6379},
        {"host": "redis-3", "port": 6379}
    ],
    decode_responses=True
)

async def get_original_url(short_code: str) -> Optional[str]:
    # Consistent hashing handles failover automatically
    cached = await rc.get(f"url:{short_code}")
    return json.loads(cached)["original_url"] if cached else None

Geographic Distribution

Deploy redirector clusters in multiple regions:

# Route 53 latency routing
- Name: short.ly
  Type: A
  SetIdentifier: us-east-1
  Region: us-east-1
  AliasTarget:
    DNSName: dualstack.api-elb-us-east-1.amazonaws.com
    EvaluateTargetHealth: true

# EU redirector
- Name: short.ly
  Type: A
  SetIdentifier: eu-west-1
  Region: eu-west-1
  AliasTarget:
    DNSName: dualstack.api-elb-eu-west-1.amazonaws.com
    EvaluateTargetHealth: true

Users are routed to the nearest cluster based on latency.

Analytics Pipeline

Track clicks without slowing redirects:

async def track_click(short_code: str):
    # Fire and forget - don't await
    asyncio.ensure_future(
        kafka.send("clicks", {
            "short_code": short_code,
            "timestamp": datetime.utcnow().isoformat(),
            "user_agent": request.headers.get("user-agent"),
            "referer": request.headers.get("referer"),
            "ip_hash": hash_ip(request.client.host)
        })
    )

Click Analytics Consumer

async def process_clicks():
    consumer = KafkaConsumer("clicks", bootstrap_servers=["kafka:9092"])

    for message in consumer:
        event = json.loads(message.value)

        # Update click count in background
        await db.query("""
            UPDATE urls
            SET click_count = click_count + 1
            WHERE short_code = $1
        """, event["short_code"])

        # Update analytics warehouse
        await warehouse.insert("click_events", event)

Complete API Specification

Endpoints

Method	Endpoint	Description
POST	/api/v1/shorten	Create short URL
GET	/{short_code}	Redirect to original
GET	/api/v1/links/{short_code}	Get link info
GET	/api/v1/links/{short_code}/stats	Get click statistics
DELETE	/api/v1/links/{short_code}	Delete a link
PUT	/api/v1/links/{short_code}	Update link settings

Request/Response Examples

# Create short URL
curl -X POST https://short.ly/api/v1/shorten \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/very/long/path/that/needs/shortening"}'

# Response
{
  "short_code": "xV2bP9qK",
  "short_url": "https://short.ly/xV2bP9qK",
  "original_url": "https://example.com/very/long/path/that/needs/shortening",
  "created_at": "2026-03-22T10:30:00Z",
  "expires_at": null
}

---

Abuse Prevention and Security

Malicious URL Detection

URL shorteners are frequently abused for phishing, malware distribution, and spam. Implement safeguards:

class MaliciousURLDetector:
    """Detect potentially malicious URLs before shortening"""

    def __init__(self, threat_intel_client: ThreatIntelClient):
        self.threat_intel = threat_intel_client
        self.suspicious_tlds = {
            '.tk', '.ml', '.ga', '.cf', '.gq',  # Free tier often abused
            '.xyz', '.top', '.club'  # Often used in spam
        }

    async def check_url(self, url: str) -> ThreatAssessment:
        checks = await asyncio.gather(
            self._check_domain_reputation(url),
            self._check_url_pattern(url),
            self._check_content_scan(url),
            self._check_google_safe_browsing(url)
        )

        if any(check.threat for check in checks):
            return ThreatAssessment(
                threat=True,
                reason="URL flagged by security checks",
                severity="high"
            )

        return ThreatAssessment(threat=False)

    async def _check_url_pattern(self, url: str) -> CheckResult:
        parsed = urlparse(url)

        # Check for suspicious TLDs
        if any(parsed.netloc.endswith(tld) for tld in self.suspicious_tlds):
            return CheckResult(threat=True, reason="Suspicious TLD")

        # Check for IP address instead of domain
        if self._is_ip_address(parsed.netloc):
            return CheckResult(threat=True, reason="IP address used")

        # Check for excessive subdomains
        if parsed.netloc.count('.') > 4:
            return CheckResult(threat=True, reason="Excessive subdomains")

        return CheckResult(threat=False)

Rate Limiting Tiers

RATE_LIMITS = {
    "anonymous": {"shorten": "5/hour", "redirect": "100/hour"},
    "authenticated_free": {"shorten": "100/hour", "redirect": "1000/hour"},
    "authenticated_pro": {"shorten": "10000/hour", "redirect": "100000/hour"},
}

@app.middleware
async def rate_limit_middleware(request: Request, call_next):
    user_tier = get_user_tier(request)

    if user_tier == "anonymous":
        # Rate limit by IP
        client_ip = request.client.host
        if not await rate_limiter.check_limit(f"ip:{client_ip}", RATE_LIMITS["anonymous"]["shorten"]):
            raise HTTPException(status_code=429, detail="Rate limit exceeded")

    elif user_tier == "authenticated":
        user_id = get_user_id(request)
        if not await rate_limiter.check_limit(f"user:{user_id}", RATE_LIMITS[user_tier]["shorten"]):
            raise HTTPException(status_code=429, detail="Rate limit exceeded")

    return await call_next(request)

Spam Link Prevention

async def create_short_url(url: str, user_id: int = None) -> ShortUrl:
    # Validate URL format
    if not is_valid_url(url):
        raise HTTPException(status_code=400, detail="Invalid URL format")

    # Check for known spam domains
    if await spam_database.is_spam_domain(extract_domain(url)):
        raise HTTPException(status_code=403, detail="URL blocked")

    # Require authentication for custom codes
    if custom_code and not user_id:
        raise HTTPException(status_code=401, detail="Authentication required for custom codes")

    # Create short URL
    return await url_service.create(url, custom_code, user_id)

---

Production Failure Scenarios

Failure Scenario	Impact	Mitigation
Redis cache failure	All redirects hit DB, high latency	Fallback to direct DB reads; circuit breaker on cache
KGS (key gen) failure	Cannot create new short URLs	Use hash-based codes as fallback; KGS recovery priority
Database primary failure	Cannot create or redirect	Promote read replica; use eventual consistency for analytics
DNS resolution failure	short.ly domain unreachable	Multi-cloud DNS; anycast IP; aggressive caching
CDN failure for stats page	Stats load slowly	Static asset caching; local caching

Cache Failure Handling

async def get_original_url(short_code: str) -> Optional[str]:
    try:
        # Try cache first
        cached = await redis.get(f"url:{short_code}")
        if cached:
            return json.loads(cached)["original_url"]
    except RedisConnectionError:
        # Cache unavailable - fall through to DB
        pass

    # Fallback to database
    url_record = await db.urls.get(short_code=short_code)
    if not url_record:
        return None

    # Don't try to repopulate cache if Redis is down
    return url_record.original_url

---

Real-world Failure Scenarios

Scenario 1: Bitly Outage (2010)

What happened: Bitly suffered a major outage affecting all URL shortening services for several hours. The incident exposed critical infrastructure weaknesses in their failover mechanisms.

Root cause: Inadequate redundancy in DNS configuration combined with a cascading failure when the primary database cluster became unavailable. No automatic failover was configured.

Impact: All shortened links returned errors, affecting millions of redirected URLs across social media and marketing campaigns. Service was completely unavailable for ~4 hours.

Lesson learned: Design for failure at every layer. Implement multi-region failover, regular chaos testing, and ensure DNS failover is automatic and tested.

Scenario 2: AWS S3 Availability Zone Failure

What happened: A major cloud provider experienced a partial Availability Zone failure that knocked out access to S3 bucket data in one AZ. Services relying on single-AZ S3 configurations went dark.

Root cause: Some URL shortening services configured S3 as the primary storage backend without cross-AZ replication. When the AZ failed, all read/write operations failed.

Impact: Services storing shortened link metadata and redirect targets in a single AZ experienced complete data unavailability, even though other AZs remained healthy.

Lesson learned: Always use S3 cross-region replication for critical data. Design storage backends to tolerate AZ failures. Test failure scenarios regularly.

Scenario 3: Database Connection Pool Exhaustion

What happened: A popular URL shortener experienced a spike in traffic during a major sporting event, causing database connection pool exhaustion and complete service degradation.

Root cause: The connection pool size was configured based on normal traffic patterns. During the traffic spike, all available connections were consumed by read operations, blocking write operations needed for creating new shortened URLs.

Impact: Users could not create new shortened links even though existing redirects continued to work. The service appeared functional but was effectively in a degraded state for several hours.

Lesson learned: Implement connection pool monitoring, use read replicas to offload read traffic, and configure automatic scaling for database connection pools based on real-time demand metrics.

---

Common Pitfalls / Anti-Patterns

Pitfall 1: Using Sequential IDs as Short Codes

Problem: Sequential IDs (1, 2, 3…) allow URL enumeration - attackers can guess other short URLs.

Solution: Use cryptographically random codes (base62) with minimum 6 characters. Use KGS for guaranteed uniqueness without predictability.

Pitfall 2: Not Handling URL Expiration

Problem: Expired URLs still redirect until cache expires.

Solution: Check expiration on every redirect. Set aggressive cache TTL for URLs with near-term expiration.

Pitfall 3: Storing Only Short Code in Cache

Problem: Cache miss requires DB query for every redirect.

Solution: Cache generously. Use “recently accessed” eviction. Pre-populate cache for trending links.

---

Observability Checklist

Metrics to Capture

• url_redirects_total (counter) - By short_code prefix, status code
• url_shortens_total (counter) - By user_tier, custom vs auto
• redirect_latency_seconds (histogram) - P50, P95, P99
• cache_hit_ratio (gauge) - Cache efficiency
• malicious_url_attempts_total (counter) - Blocked attempts by type
• kgs_available_keys (gauge) - Key generation health

Logs to Emit

{
  "timestamp": "2026-03-22T10:30:00Z",
  "event": "redirect",
  "short_code": "xV2bP9qK",
  "status": 302,
  "latency_ms": 12,
  "cache_hit": true,
  "user_ip_hash": "abc123"
}

Alerts to Configure

Alert	Threshold	Severity
Redirect latency P99 > 200ms	200ms	Warning
Cache hit ratio < 50%	50%	Warning
KGS keys < 1000	1000	Critical
Malicious attempts spike	> 100/min	Warning
DB connection pool > 80%	80%	Warning

---

Security Checklist

• TLS 1.3 for all connections
• URL validation and sanitization
• Malicious URL scanning (Google Safe Browsing API)
• Rate limiting per IP and per user
• Custom code length and character validation
• Authentication required for custom short codes
• Audit logging of all URL creations
• GDPR compliance for analytics data
• Regular security audits

---

Trade-off Analysis

Design Decision	Option A	Option B	Recommendation
Short Code Generation	Hash-based (MD5/SHA + salt)	Counter-based KGS	KGS for production; hash-based acceptable for simple cases
Code Length	6 chars (56.8B combos)	7-8 chars (3.5T - 218T)	7-8 chars to avoid birthday paradox collisions
Redirect Status	301 Permanent	302 Temporary	302 to avoid SEO consolidation issues
Cache Strategy	Cache URL only	Cache full URL object with metadata	Cache full object to avoid second round-trip
Custom Codes	Require authentication	Allow anonymous	Require authentication + premium pricing
URL Deduplication	Per-user dedup	Allow duplicates	Per-user dedup for efficiency
Analytics Tracking	Synchronous write	Async fire-and-forget	Always async to keep redirect fast
Database	PostgreSQL	DynamoDB/Cassandra	PostgreSQL for <1B URLs; NoSQL for billions
KGS Recovery	Return orphaned codes after timeout	Fall back to hash-based codes	Both strategies needed

When to Choose Each Option

Hash vs KGS for Code Generation:

• Use hash-based when: simplicity is priority, collision handling is acceptable, moderate scale
• Use KGS when: predictability required, high availability critical, cannot tolerate collision checking latency

PostgreSQL vs NoSQL:

• Use PostgreSQL when: need ACID transactions, moderate scale, team expertise available
• Use DynamoDB/Cassandra when: billions of rows, global distribution, multi-region writes required

---

Interview Questions

---

Further Reading

• Caching Strategies — Deep dive into Redis caching patterns
• Load Balancing — Geographic distribution and routing
• Database Scaling — Read replicas and partitioning
• Rate Limiting — Anti-abuse patterns
• Distributed Caching — Multi-node cache architectures

For more system design patterns, see our Caching Strategies guide which covers caching patterns used here. The Load Balancing guide covers geographic distribution.

Conclusion

A URL shortener looks simple on paper. The complexity shows up at scale. The key decisions:

• Short code generation: Counter-based with KGS ensures uniqueness and predictability
• Caching: Aggressive caching with Redis handles redirect traffic
• Database: PostgreSQL with read replicas for availability
• Analytics: Asynchronous click tracking via Kafka

Quick Recap Checklist

Before deploying your URL shortener to production, verify these essentials:

Core Functionality

• Short code generation using KGS or hash-with-salt
• Base62 encoding with 6-8 character output
• URL validation and sanitization
• Custom code reservation for system paths
• Expiration handling on every redirect

Performance

• Redis caching with write-through on create
• Cache TTL differentiated by URL type (hot/cold/custom)
• Database indexes on short_code for fast lookups
• Read replicas for redirect traffic
• Async analytics (fire-and-forget click tracking)

Availability

• Circuit breaker for cache failures
• Database fallback when cache unavailable
• Multi-region deployment with latency-based routing
• KGS warm standby for high availability
• Connection pool monitoring and auto-scaling

Security

• Rate limiting per IP and per user tier
• Malicious URL scanning (Safe Browsing API)
• Authentication required for custom codes
• Reserved path blocking at creation time
• TLS 1.3 for all connections
• Audit logging for URL creations

Observability

• Redirect latency histograms (P50, P95, P99)
• Cache hit ratio monitoring
• KGS available keys gauge
• Alert thresholds configured for all critical metrics