Object Storage: S3, Blob Storage, and Scale of Data

Introduction

Object storage organizes data as objects in buckets. Each object has a key, data, and metadata. The key is a string, like a file path but without directory semantics. The data is arbitrary bytes. Metadata is key-value pairs describing the object.

import boto3

s3 = boto3.client('s3')

# Upload an object
s3.put_object(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg',
    Body=image_data,
    ContentType='image/jpeg',
    Metadata={'product-id': '12345', 'uploaded-by': 'jane'}
)

# Retrieve an object
response = s3.get_object(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg'
)
image_data = response['Body'].read()

The key looks like a path but the storage is flat. There are no directories, though tools often display keys as if they had folders. The slash in the key is just another character.

Data Protection & Compliance

Buckets are containers for objects. They have global names across regions. You choose the bucket name, and it must be unique across all S3 users worldwide.

# Create a bucket
s3.create_bucket(Bucket='my-unique-bucket-name')

# List buckets
response = s3.list_buckets()
for bucket in response['Buckets']:
    print(bucket['Name'])

Keys identify objects within a bucket. The combination of bucket name and key uniquely identifies every object. No two objects in the same bucket can have the same key.

Keys can contain slashes, which tools display as folder hierarchies. But the storage treats them as flat. You can list objects with a prefix filter to simulate directory browsing.

Object Metadata and Tags

Object metadata travels with the object. System metadata includes things like content type, size, and last-modified time. User metadata is custom key-value pairs you define.

# Retrieve metadata without downloading
head = s3.head_object(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg'
)
print(head['ContentType'])  # image/jpeg
print(head['ContentLength'])  # 245632
print(head['Metadata'])  # {'product-id': '12345', 'uploaded-by': 'jane'}

Tags are separate from metadata. They are key-value pairs used for filtering and billing. You can have up to 10 tags per object. Tags are indexed separately for efficient querying.

# Add tags to an existing object
s3.put_object_tagging(
    Bucket='my-bucket',
    Key='images/product/photo123.jpg',
    Tagging={
        'TagSet': [
            {'Key': 'department', 'Value': 'marketing'},
            {'Key': 'project', 'Value': 'spring-campaign'}
        ]
    }
)

Bucket and Key Fundamentals

Versioning

S3 versioning keeps multiple versions of an object. When you overwrite an object, the previous version is preserved. You can retrieve any historical version.

# Enable versioning on a bucket
s3.put_bucket_versioning(
    Bucket='my-bucket',
    VersioningConfiguration={'Status': 'Enabled'}
)

# List object versions
versions = s3.list_object_versions(Bucket='my-bucket')
for version in versions['Versions']:
    print(f"{version['Key']} - {version['VersionId']}")

Versioning costs more storage since every version is kept. But it protects against accidental overwrites and deletions. You can retrieve any previous state.

Versioning also enables point-in-time recovery. Combined with lifecycle policies, you can keep historical versions for compliance without manual intervention.

Storage Classes and Lifecycle Policies

S3 offers multiple storage classes with different cost and availability characteristics.

Standard is the default, most expensive tier. Infrequent Access stores data cheaper but charges more for access. Glacier is for archiving, with retrieval times of minutes to hours. Intelligent Tiering moves data automatically based on access patterns.

from datetime import datetime

# Define lifecycle rule to transition objects to IA after 30 days
s3.put_bucket_lifecycle_configuration(
    Bucket='my-bucket',
    LifecycleConfiguration={
        'Rules': [
            {
                'ID': 'MoveToIA',
                'Filter': {'Prefix': 'logs/'},
                'Status': 'Enabled',
                'Transitions': [
                    {'Days': 30, 'StorageClass': 'STANDARD_IA'},
                    {'Days': 90, 'StorageClass': 'GLACIER'}
                ]
            }
        ]
    }
)

Lifecycle policies automate data movement between tiers. Old logs might move to Infrequent Access after 30 days, then to Glacier after 90 days. You set it and forget it.

flowchart LR
    Upload[("Object<br/>Uploaded")] --> Standard[("S3 Standard<br/>Frequently accessed")]

    Standard -->|30 days<br/>no access| IA[("S3 Standard-IA<br/>Infrequent access")]

    IA -->|90 days<br/>no access| Glacier[("S3 Glacier<br/>Archived")]

    Glacier -->|180 days<br/>no access| DeepArchive[("S3 Deep Archive<br/>Long-term archive")]

    DeepArchive -->|365 days<br/>no access| Delete[("Lifecycle<br/>Expiration")]

    Upload -.->|versioning<br/>enabled| Versions[("Version<br/>Preserved")]

    Versions -.->|MFA delete<br/>required| Restore[("Restore<br/>before delete")]

Without versioning, overwrites and deletes are permanent. With versioning, previous versions persist until you explicitly delete them or the lifecycle policy purges old versions.

Storage classes across AWS S3:

Class	Durability	Availability	Access Latency	Min Storage Duration	Best For
Standard	11 nines	99.99%	Milliseconds	None	Frequently accessed data
Standard-IA	11 nines	99.9%	Milliseconds	30 days	Infrequently accessed data
Intelligent Tiering	11 nines	99.9%	Milliseconds	None	Unknown or changing access patterns
One Zone-IA	99.5%	99.5%	Milliseconds	30 days	Re-creatable non-critical data in one AZ
Glacier Instant Retrieval	11 nines	99.9%	Milliseconds	90 days	Rarely accessed but needs instant retrieval
Glacier Flexible Retrieval	11 nines	99.99%	1-12 hours	90 days	Long-term archives with occasional access
Glacier Deep Archive	11 nines	99.99%	12-48 hours	180 days	Longest-term retention, regulatory compliance

Most teams keep everything in Standard. They should not. Standard-IA is about half the price, and Glacier Deep Archive is roughly 95% cheaper. The tradeoff is retrieval cost and latency — match the class to how you actually access the data.

Durability and Availability

Object storage is designed for massive scale and high durability. S3 Standard offers 11 nines of durability. That means losing an object is extraordinarily unlikely.

The durability comes from storing multiple copies across availability zones. S3 automatically replicates your data. You do not need to configure RAID or backup software.

Availability guarantees differ from durability. S3 Standard guarantees 99.99% availability. That is about 52 minutes of downtime per year. Different storage classes offer different availability SLAs.

# Check if an object exists
try:
    s3.head_object(Bucket='my-bucket', Key='important-file.pdf')
    print("Object exists")
except ClientError as e:
    error_code = e.response['Error']['Code']
    if error_code == '404':
        print("Object not found")

Using Object Storage in Applications

Object storage replaces both file storage and some database use cases. Static assets like images and videos live in object storage. User uploads go there. Backup files. Data lakes.

The pattern is straightforward. Generate a unique key, upload with metadata, store the key in your database if needed. The object storage handles the rest.

import uuid

def handle_upload(file_data, content_type):
    # Generate unique key
    key = f"uploads/{uuid.uuid4()}/{file_data.filename}"

    # Upload to S3
    s3.put_object(
        Bucket='my-bucket',
        Key=key,
        Body=file_data.read(),
        ContentType=content_type
    )

    # Store key in database, not the file itself
    db.files.insert({'s3_key': key, 'original_name': file_data.filename})

    return key

CDN integration is common. CloudFront, Cloudflare, and others cache objects from S3. Users download from edge locations rather than origin servers. This reduces latency and origin load.

Access Control Patterns: Bucket Policies, IAM, and CORS

S3 offers layered access control: IAM policies for fine-grained user permissions, bucket policies for bucket-wide rules, and ACLs for legacy compatibility. Bucket policies are JSON documents that apply to entire buckets — they can grant or deny access based on source IP, requestor, object prefix, or operation type.

# Bucket policy denying HTTP traffic except from a specific IP range
bucket_policy = {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AllowReadFromVPN",
            "Effect": "Deny",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-bucket/*",
            "Condition": {
                "NotIpAddress": {
                    "aws:SourceIp": "203.0.113.0/24"
                }
            }
        },
        {
            "Sid": "AllowInternalReads",
            "Effect": "Allow",
            "Principal": "*",
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::my-bucket/internal/*"
        }
    ]
}

s3.put_bucket_policy(Bucket='my-bucket', Policy=json.dumps(bucket_policy))

IAM vs bucket policies:

Aspect	IAM Policies	Bucket Policies
Scope	User, group, or role level	Entire bucket or prefix
Use case	Cross-account access, detailed action control	Public/private boundaries, cross-account grants
Evaluation	Evaluated in combination with bucket policies	Evaluated independently as resource-based policy
Management	Per-user in IAM	Single JSON doc attached to bucket

CORS for browser access:

If your application serves objects to browsers directly from S3, you need CORS configuration:

# Configure CORS to allow browser-based uploads from your domain
cors_configuration = {
    'CORSRules': [
        {
            'AllowedHeaders': ['Authorization', 'Content-Type'],
            'AllowedMethods': ['GET', 'POST', 'PUT'],
            'AllowedOrigins': ['https://your-domain.com'],
            'ExposeHeaders': ['ETag'],
            'MaxAgeSeconds': 3600
        }
    ]
}

s3.put_bucket_cors(Bucket='my-bucket', CORSConfiguration=cors_configuration)

Public access block: At minimum, enable block public access at the account level before deploying any bucket:

# Block all public access to prevent accidental exposure
s3.put_public_access_block(
    Bucket='my-bucket',
    PublicAccessBlockConfiguration={
        'BlockPublicAcls': True,
        'IgnorePublicAcls': True,
        'BlockPublicPolicy': True,
        'RestrictPublicBuckets': True
    }
)

Multipart Upload: Parallelizing Large File Transfers

S3 splits large objects into parts for upload. Multipart upload enables parallel transfers, resumable uploads, and handling of files larger than 5GB (the single-put limit).

# Multipart upload example
import boto3
import math

s3 = boto3.client('s3')

def upload_large_file(bucket, key, file_path, part_size_mb=100):
    """Upload a large file using multipart upload."""
    file_size = file_path.stat().st_size
    part_size = part_size_mb * 1024 * 1024  # 100MB default
    num_parts = math.ceil(file_size / part_size)

    # Initiate multipart upload
    response = s3.create_multipart_upload(
        Bucket=bucket,
        Key=key,
        ContentType='application/octet-stream'
    )
    upload_id = response['UploadId']

    parts = []
    with open(file_path, 'rb') as f:
        for i in range(num_parts):
            data = f.read(part_size)
            part_num = i + 1

            # Upload each part
            result = s3.upload_part(
                Bucket=bucket,
                Key=key,
                UploadId=upload_id,
                PartNumber=part_num,
                Body=data
            )
            parts.append({
                'PartNumber': part_num,
                'ETag': result['ETag']
            })

    # Complete multipart upload
    s3.complete_multipart_upload(
        Bucket=bucket,
        Key=key,
        UploadId=upload_id,
        MultipartUpload={'Parts': parts}
    )
    return key

Part size considerations:

File Size	Recommended Part Size	Max Parts	Notes
< 100MB	Single PUT	1	Simpler, no multipart needed
100MB - 5GB	50-100MB	< 100	Default part size works well
5GB - 5TB	100-500MB	10,000	Must use multipart; 5GB is single PUT limit
> 5TB	Not supported	-	S3 maximum object size is 5TB

Performance tips: Upload parts in parallel to maximize throughput. Use byte-range fetches for parallel downloads. Part numbers must be 1-10000 and can be uploaded concurrently.

# Abort incomplete multipart upload to avoid stray charges
s3.abort_multipart_upload(
    Bucket='my-bucket',
    Key='large-file.zip',
    UploadId='upload-id-here'
)

Set lifecycle rules to abort incomplete multipart uploads after 7 days. This prevents accumulating storage costs from failed uploads.

Performance & Retrieval

S3 Select: Querying Data Without Downloading

S3 Select lets you query CSV, JSON, or Parquet objects directly. You retrieve only the data you need, reducing transfer costs and improving query performance.

# Query CSV data with S3 Select
response = s3.select_object_content(
    Bucket='my-bucket',
    Key='analytics/events.csv',
    Expression='SELECT * FROM s3object WHERE status = \'completed\' LIMIT 10',
    ExpressionType='SQL',
    InputSerialization={
        'CSV': {
            'FileHeaderInfo': 'USE',
            'RecordDelimiter': '\n',
            'FieldDelimiter': ','
        }
    },
    OutputSerialization={'CSV': {}}
)

# Process results
for event in response['Payload']:
    if 'Records' in event:
        print(event['Records']['Payload'].decode('utf-8'))

S3 Select use cases:

• Log analysis: Query specific error codes from large CloudWatch logs
• Time-series filtering: Extract metrics for a specific date range from Parquet files
• Partial file retrieval: Get only needed columns from wide CSV datasets

Cost benefits: You pay for data scanned (Query Request pricing) rather than data transferred. For filtered queries on large files, this can reduce costs by 80-90%.

Object Lock and Compliance: WORM Storage

S3 Object Lock provides WORM (Write Once, Read Many) protection. Objects cannot be deleted or overwritten for a specified retention period.

# Enable Object Lock on bucket creation
s3.create_bucket(
    Bucket='compliance-bucket',
    ObjectLockEnabledForBucket=True
)

# Put Object Lock retention on an object
from datetime import datetime, timedelta

retention_until = datetime.utcnow() + timedelta(days=365)
s3.put_object_retention(
    Bucket='compliance-bucket',
    Key='financial-records/2024.xlsx',
    Retention={
        'Mode': 'COMPLIANCE',  # Cannot be overridden by any user
        'RetainUntilDate': retention_until.isoformat()
    }
)

# Put legal hold (indefinite protection)
s3.put_object_legal_hold(
    Bucket='compliance-bucket',
    Key='financial-records/2024.xlsx',
    LegalHold={'Status': 'ON'}
)

Retention modes:

Mode	Override by Admin	Use Case
COMPLIANCE	No	Regulatory requirements, legal hold
GOVERANCE	Yes (with special permission)	Temporary protection, testing environments

Legal hold remains until explicitly removed. It survives IAM user permissions changes and account closures. Ideal for litigation hold or permanent records.

Cross-Region Replication: Disaster Recovery Patterns

CRR replicates objects to destination buckets in different regions. It provides lower latency, disaster recovery, and compliance requirements.

# Enable cross-region replication
import boto3

iam = boto3.client('iam')
s3 = boto3.client('s3')

# Create IAM role for replication
role_policy = {
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Principal": {"Service": "s3.amazonaws.com"},
        "Action": "sts:AssumeRole"
    }]
}

# Enable versioning first (required for CRR)
s3.put_bucket_versioning(
    Bucket='source-bucket',
    VersioningConfiguration={'Status': 'Enabled'}
)

# Create replication configuration
replication_config = {
    'Role': iam.get_role(RoleName='s3-replication-role')['Role']['Arn'],
    'Rules': [{
        'ID': 'replicate-to-us-west-2',
        'Status': 'Enabled',
        'Priority': 1,
        'DeleteMarkerReplication': {'Status': 'Enabled'},
        'Destination': {
            'Bucket': 'arn:aws:s3:::dest-bucket-us-west-2',
            'StorageClass': 'STANDARD_IA',
            'EncryptionConfiguration': {'ReplicaKmsKeyID': 'kms-key-id'}
        },
        'Filter': {'Prefix': 'production/'}
    }]
}

s3.put_bucket_replication(
    Bucket='source-bucket',
    ReplicationConfiguration=replication_config
)

Replication considerations:

• Cost: Data transfer between regions charges egress fees. Budget accordingly.
• Latency: Objects appear in destination within seconds typically, but async replication means RPO > 0.
• Filtered replication: Use prefix filters to replicate only production data, not test data.
• KMS encryption: Encrypted objects require KMS key access in destination region.

Multi-region active-active: For zero RPO, replicate synchronously to a read replica bucket. Route traffic with Route53 geolocation. This is complex and expensive but achieves true active-active.

Cost & Capacity Planning

Capacity Estimation: Cost Calculation for S3 Tiers

Object storage pricing varies significantly by storage class. Understanding tier costs enables cost-effective lifecycle management.

Monthly storage cost formula:

storage_cost_per_month = storage_gb × price_per_gb_per_month
total_storage_cost = sum(storage_tier_gb × tier_price_per_gb_per_month)

For a media platform with the following storage distribution:

Tier	Storage	Price/GB/mo	Monthly Cost
S3 Standard	10TB	$0.023	$230
S3 IA	50TB	$0.0125	$625
S3 Glacier	200TB	$0.004	$800
S3 Glacier Deep Archive	500TB	$0.00099	$495
Total	760TB		$2,150/mo

Request costs matter as much as storage:

request_cost_per_month = get_requests × price_per_1k_gets + put_requests × price_per_1k_puts

S3 Standard pricing: $0.023/GB storage, $0.0004 per 1,000 GET requests, $0.005 per 1,000 PUT requests. For a high-traffic application with 100M GET/day and 1M PUT/day: storage might be $500/mo but requests add $40 + $5 = $45/mo. For a low-traffic archive with 1M GET/day and 10K PUT/day: storage $500/mo, requests $0.40 + $0.05 = $0.45/mo. Storage dominates for cold data; requests dominate for hot data.

Early deletion fees: S3 Glacier and Glacier Deep Archive charge early deletion fees if data is deleted within 90 or 180 days respectively. Plan lifecycle transitions accordingly — do not move data to Glacier if you might need to delete it within 90 days.

Trade-off Analysis

When designing object storage solutions, you constantly trade off cost, performance, durability, and operational complexity.

Storage Class Trade-offs:

Consideration	Standard	Standard-IA	Glacier
Storage cost	Highest	~50% of Standard	~95% cheaper than Standard
Retrieval cost	None	Per-GB fees	Per-GB + retrieval tier fees
Access latency	Milliseconds	Milliseconds	Minutes to hours
Minimum duration	None	30 days	90-180 days
Best for	Hot, accessed data	Monthly access	Archival, rarely accessed

Versioning Trade-offs:

Factor	With Versioning	Without Versioning
Storage cost	2x-10x depending on churn	1x base storage
Protection	Accidental deletes/overwrites recoverable	Permanent loss risk
Operational complexity	More storage to manage, list versions	Simpler
MFA delete available	Yes	No protection against admin mistakes

Replication Trade-offs:

Strategy	Cost	RPO	RTO	Complexity
No replication	$0	N/A	Full loss possible	None
Same-region replication	Low	Minutes	Minutes-hours	Medium
Cross-region async	Medium (egress fees)	Seconds-minutes	Minutes	Medium
Cross-region sync	High	Near zero	Near zero	High

Key naming strategies:

Strategy	Pros	Cons
Descriptive paths (images/2024/product/photo.jpg)	Human readable, easy to navigate	May hit hot prefix limits
UUID-based (uploads/a1b2c3d4/photo.jpg)	No collision risk, evenly distributed	Hard to navigate manually
Hybrid (uploads/2024/a1b2c3d4.jpg)	Balanced	Slightly more complex

Encryption trade-offs:

Method	Key management	Performance	Compliance
SSE-S3 (AES-256)	AWS managed	Minimal overhead	Basic
SSE-KMS	Your keys in KMS	~3-5% overhead	Enhanced
CSE-KMS	Your keys, client-side	Higher CPU	Highest control

When to Use / When Not to Use

When to Use Object Storage:

• Storing unstructured files: images, videos, documents, audio files
• Backing up databases, logs, or system files
• Hosting static website assets (HTML, CSS, JS, images)
• Building data lakes for analytics workloads
• Distributing large files via CDN integration
• Storing user uploads that exceed database BLOB limits
• Archiving data for compliance with infrequent access patterns

When Not to Use Object Storage:

• You need filesystem semantics (directories, symlinks, permissions)
• Your workload requires sub-millisecond latency (use block storage or memory)
• You need to modify parts of files frequently (object storage is write-all-or-nothing)
• Your application requires strong consistency for concurrent modifications
• You need database-like queries across object metadata (use a database index)
• Real-time file system operations are required (use NFS, EBS, or similar)

Production Failure Scenarios

Failure	Impact	Mitigation
Accidental deletion	Permanent data loss without versioning	Enable versioning, implement soft-delete policies, use MFA delete
Bucket policy misconfiguration	Public exposure or access denial	Review policies with least privilege, use policy validation tools
Cost overrun from unconstrained uploads	Unexpectedly high storage costs	Set budget alerts, implement upload size limits, lifecycle policies
Cross-region replication delay	Stale data in DR region	Set realistic RPO, test replication lag, use synchronous replication if needed
Throttling from request rate limits	Upload/download failures under load	Implement retry with exponential backoff, request rate smoothing
Object key namespace collision	Data overwrites	Use UUIDs or timestamp-based keys, implement key validation
Bucket name conflicts	Deployment failures	Use globally unique naming conventions, automate naming
CDN cache invalidation delays	Stale content served after updates	Plan invalidation strategy, use versioned object keys

Common Pitfalls / Anti-Patterns

1. Using object storage as a database: Object storage has no query language. Storing millions of objects with no index makes retrieval extremely slow. Use a database for structured data with query needs.

2. Not planning for request rate limits: S3 has per-bucket and per-prefix limits. Popular content from a single prefix throttles. Spread across prefixes and use CloudFront.

3. Storing too many small objects: Each object has overhead. Millions of tiny objects waste money and slow listings. Consider archiving small files together.

4. Ignoring storage class optimization: Storing everything in Standard costs more than necessary. Use lifecycle policies to move rarely-accessed data to cheaper tiers.

5. Not using versioning for mutable objects: Overwriting without versioning destroys the previous version. Enable versioning for any object that changes.

6. Assuming strong consistency immediately: New objects and updates take time to propagate. Do not assume immediate consistency for read-after-write.

7. Leaking pre-signed URLs: Pre-signed URLs grant access to anyone with the URL. Do not log or expose them. Use short expiration times.

8. Not implementing cleanup policies: Upload failures, test runs, and temporary files accumulate. Implement lifecycle rules to auto-delete incomplete multipart uploads and old temp files.

Real-World Case Studies

Case Study 1: Netflix — Video Storage at Scale

Netflix stores billions of hours of video content in S3-compatible object storage. Each title is encoded at multiple resolutions and bitrates, resulting in hundreds of object versions per film or episode. Netflix uses:

• Lifecycle policies to automatically transition old or rarely viewed content from Standard to Glacier storage classes
• Multipart upload for reliable upload of very large video files
• Cross-Region Replication to replicate popular content to regions near users, reducing playback latency

Case Study 2: Airbnb — Photo Storage with S3

Airbnb stores millions of listing photos as objects in S3. Each photo is uploaded via a backend service that:

• Assigns a unique key (e.g., listings/{listing_id}/photos/{photo_id}.jpg)
• Generates multiple thumbnails stored as separate objects under the same listing prefix
• Applies storage classes to thumbnails based on age (older thumbnails move to Infrequent Access)

Case Study 3: GitHub — Git Object Storage

GitHub stores Git repository objects (blobs, trees, commits, tags) as content-addressable objects in S3. Each object is stored under its SHA-1 hash as the key, enabling efficient deduplication across millions of repositories.

---

Quick Recap Checklist

Use this checklist to validate your object storage setup:

• Bucket naming: Global uniqueness required; use lowercase, hyphens only, no periods
• Versioning: Enabled for any bucket storing mutable objects that may need recovery
• Lifecycle policies: Configured to transition rarely-accessed data to cheaper tiers (IA after 30 days, Glacier after 90 days)
• Multipart upload: Lifecycle rule set to abort incomplete uploads after 7 days
• Encryption: SSE-S3 (AWS-managed) applied by default, or SSE-KMS if compliance requires key audit trails
• Public access block: Enabled at account or bucket level before deployment
• Storage class assignment: New objects placed in appropriate tier based on expected access pattern
• Intelligent Tiering: Consider for workloads with unpredictable or changing access patterns
• CORS configuration: Configured if browser-based direct uploads/downloads are required
• Cost monitoring: Budget alerts configured; request vs. storage cost split understood
• Replication (if DR required): Cross-region replication configured with versioned destination bucket
• Access logging: Server access logging enabled, logs written to a separate versioned bucket
• Access Points: Consider for multi-team or multi-application bucket access segmentation
• MFA Delete: Enabled on versioned buckets requiring protection against admin-level mistakes

Interview Questions

Further Reading

Official AWS Documentation:

• Amazon S3 Documentation — The primary reference for all S3 API operations
• S3 Storage Classes — Covers when to use each tier
• S3 Multipart Upload Documentation — Part sizing, limits, and abort operations

AWS Re:Invent Sessions:

• Deep Dive: Amazon S3 — How S3 works internally
• S3 Performance Optimization — Hot prefix issues and request rate scaling

Tools and Utilities:

• S3 Browser — Windows GUI for browsing buckets
• rclone — Sync files to S3 from the command line
• s3-parallel-upload — Upload large files faster using multipart in parallel
• S3 Inventory — Generate daily or weekly CSV reports of bucket contents

Comparison Resources:

• MinIO — Open-source S3-compatible storage for on-prem or Kubernetes
• Google Cloud Storage — GCS equivalent features
• Azure Blob Storage — Azure’s object storage offering

For related reading, see Database Scaling to learn about scaling database storage, and NoSQL Databases to understand other data storage patterns.

Conclusion

Key takeaways from Object Storage:

• Object storage organises data as objects (data + metadata + unique ID) rather than files in a hierarchy
• S3 (and S3-compatible services like MinIO, GCS, Azure Blob) is the dominant interface for object storage
• Buckets are top-level containers; keys are the full object path within a bucket
• Storage classes (S3 Standard, IA, Glacier, etc.) enable cost optimisation based on access patterns
• Lifecycle policies automate object transitions between storage classes and expiration
• Versioning preserves previous versions of objects; MFA Delete prevents accidental or malicious deletion
• Multipart upload allows parallel upload of large objects in parts
• S3 Select enables querying objects directly with SQL-like filters — no full download needed
• Object Lock (WORM) enforces immutability for compliance and regulatory requirements
• Cross-Region Replication (CRR) provides disaster recovery and geo-low-latency access
• Object storage is not suitable for complex relational queries, ultra-low latency, or fine-grained ACID transactions

---