Cloud Sandbox System Design

Back to Portfolio

Scaling & Messaging Systems

Fault-tolerant cloud sandbox platform for isolated code execution, durable queueing, and scale-to-zero worker capacity.

Scale 0 -> N | Durable Queue | Readiness-Gated Wake-Up

Live DemoSource RepositoryRead Scale-to-Zero DesignRead Production Upgrade Log

Why This Project Matters

Shows SRE-first platform engineering: the control plane remains reachable while idle execution workers release capacity, then recover safely when queued work arrives.

Tech + Architecture Summary

  • Tech: Node.js, AWS ECS/Fargate, CloudWatch, EventBridge, Terraform (IaC), FinOps
  • Architecture: cloudsandbox.space -> always-available control API -> durable queue/DLQ -> queue-depth scaling policy -> 0-to-N ECS/Fargate worker pool -> readiness-gated execution -> result store.

Impact Metrics

  • Implemented queue-depth-driven scaling with a minimum execution-worker capacity of zero, removing idle worker allocation while keeping request intake available.
  • Designed the wake-up path so queued jobs remain durable while ECS/Fargate starts capacity and workers pass readiness checks before execution.
  • Added cooldown, hysteresis, bounded retries, and DLQ recovery to reduce scale thrashing and protect jobs across cold starts.
  • Engineered a self-healing queue ecosystem using Redis Dead Letter Queues (DLQ) and AWS EventBridge cron triggers, achieving 100% payload recovery during staged network partition drills.

Core Problem

Execute untrusted user code safely while controlling runtime limits, output size, and request-level isolation.

Build Notes

What I Owned

This project is where I separated always-available request intake from elastic execution capacity so idle workers can scale to zero without dropping accepted jobs.

Hard Lesson

The important lesson was that scale-to-zero is a queue and readiness problem: saving idle capacity is only safe when work remains durable and cold workers cannot receive traffic too early.

Next Enhancement

Next I would add a visible job timeline with queued, running, completed, failed, and DLQ states so reviewers can watch the lifecycle instead of only seeing the API response.

High-Level Architecture

mermaid
graph LR
  Client[Web Client]-->Control[Execution Control API]
  Control-->API[Execution API Endpoint]
  API-->Queue[Execution Queue]
  Queue-->Policy[Queue-Depth Scaling Policy]
  Policy-->Worker[Sandboxed Workers: 0 to N]
  Worker-->Result[Execution Result Store]
  Result-->API
  API-->Control

Production-Grade Capabilities

  • Asynchronous queue-worker execution model with bounded retries and durable result flow.
  • Queue-depth-driven scale-to-zero worker policy with readiness-gated scale-from-zero recovery.
  • Tenant-aware API boundary with safer sandbox and runtime guardrails.
  • Terraform-managed infrastructure topology with explicit control-plane and execution-plane separation.

Engineering Decisions

  • Strict sandbox limits improve safety but can reject edge-case workloads that need higher resource ceilings.
  • Queue-based execution improves throughput stability, but adds extra latency compared to direct synchronous execution.
  • Scaling execution workers to zero removes idle capacity, but the first job after an idle window pays a cold-start delay.
  • Splitting web and API deployments improves scalability isolation, but increases operational surface area.

Behavioral + Impact Signals

  • Designed around safe defaults for sandboxing and bounded retries.
  • Prioritized service isolation to protect user-facing workflows from backend spikes.
  • Added operational observability and auditability for execution lifecycle events.

Quality Guarantees

  • Every execution request runs with bounded CPU and memory limits.
  • Queued jobs remain durable while execution capacity is zero or warming.
  • Only ready workers can claim jobs after a scale-from-zero event.
  • Execution output is returned in a deterministic response format.
  • Failed runs do not block subsequent queue processing.

Recent Upgrades

  • Added scale-to-zero for eligible execution workers while preserving an available control plane and durable queued jobs.
  • Documented the scale-from-zero wake path, cooldown safeguards, and cold-start tradeoff for technical reviewers.
  • Cut over Cloud Sandbox reviewer access to cloudsandbox.space so the live execution path uses a dedicated production-style domain.
  • Introduced Terraform-governed dual-endpoint model for control plane and execution API traffic separation.
  • Expanded Cloud Sandbox scope into a mini Replit/Judge0-style platform with async queue-worker execution and tenant quota controls.
  • Added stronger sandbox controls: bounded runtime resources, idempotent job handling, and audit visibility.
  • Clarified the live execution proof path and documented queue/DLQ recovery in the production upgrade log.

Outcome Highlights

  • Upgraded deployment to separate web app and API endpoints for clearer platform architecture.
  • Shipped a public cloud API endpoint for real execution requests.
  • Designed for isolation-first execution behavior under backend constraints.
  • Implemented execution flow with queue-worker reliability patterns.