Skip to content
TheLearningHubOrgPublic

About

Complete guide to learn aws.

Resources

Code of conduct

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Latest commit

 

History

3 Commits

Folders and files

Repository files navigation

Learn AWS ☁️

License: MIT AWS Solutions Architect Tested with Node.js Production Ready

An exhaustive, production-grade masterclass from absolute zero to staff-level AWS Solutions Architect & Principal Cloud SRE.
Master AWS global infrastructure, IAM least-privilege security boundaries, multi-AZ VPC architecture, serverless compute (Lambda, Fargate, ECS/EKS), cloud databases (Aurora, DynamoDB, ElastiCache), event-driven messaging (SQS, SNS, EventBridge), KMS envelope encryption, and multi-region disaster recovery.

Getting Started • VPC Networking • Compute & Serverless • Storage & Databases • Event Messaging • Security & Governance • Solutions Architect Handbook


Buy Me A Coffee

🗺️ 7-Stage Pedagogical Roadmap

flowchart LR
    S1["Stage 1: Global Infra & IAM"] --> S2["Stage 2: Multi-AZ VPC"]
    S2 --> S3["Stage 3: Compute & Serverless"]
    S3 --> S4["Stage 4: Aurora & DynamoDB"]
    S4 --> S5["Stage 5: SQS, SNS & Events"]
    S5 --> S6["Stage 6: Security & Well-Arch"]
    S6 --> S7["Stage 7: Architect Handbook"]
Loading
Stage Focus Domain Core Concepts & Engineering Outcomes
Stage 1 Global Infra & IAM Security Regions, AZs, Edge Locations, IAM policies (Identity, Resource, Boundaries, SCPs), STS AssumeRole, AWS Organizations.
Stage 2 Networking & Multi-AZ VPC Subnet CIDR design, Internet Gateways, NAT Gateways, Security Groups vs NACLs, Transit Gateway, VPC Endpoints / PrivateLink.
Stage 3 Compute & Serverless EC2 launch templates, Auto Scaling Groups, ECS on Fargate, EKS, Lambda execution lifecycle, cold starts, API Gateway.
Stage 4 Storage & Cloud Databases S3 lifecycle & Object Lock, EBS gp3/io2, Aurora distributed storage engine, DynamoDB (GSI/LSI & Single-Table design).
Stage 5 Event-Driven Messaging SQS (Standard vs FIFO deduplication), SNS Pub/Sub fanout, EventBridge bus routing, Step Functions distributed state machines.
Stage 6 Security & Observability KMS envelope encryption, Secrets Manager vs SSM, WAF, CloudWatch Logs Insights, X-Ray tracing, 6 Well-Architected Pillars.
Stage 7 Staff Architect Interview Playbook Disaster recovery (Pilot Light vs Active-Active), Cost optimization, 25 Staff Solutions Architect Q&As, AWS CLI Cheat Sheet.

📋 Comprehensive Table of Contents

  1. Stage 1: Global Infrastructure & Identity Access Management (IAM)
  2. Stage 2: Networking & Virtual Private Cloud (VPC)
  3. Stage 3: Compute & Serverless Architectures
  4. Stage 4: Storage, Databases & Caching
  5. Stage 5: Messaging, Integration & Asynchronous Pipelines
  6. Stage 6: Security, Governance & Observability
  7. Stage 7: Staff Solutions Architect & Cloud SRE Interview Handbook

1. Stage 1: Global Infrastructure & Identity Access Management (IAM)

1.1 AWS Global Infrastructure: Regions, Availability Zones & Edge Locations

Amazon Web Services (AWS) powers the modern cloud using a globally distributed physical footprint:

flowchart TD
    Global["AWS Global Footprint"] --> Regions["Regions (e.g. us-east-1, eu-west-1)"]
    Global --> Edge["Edge Locations (450+ CloudFront CDN & Route 53 PoPs)"]
    Regions --> AZ1["Availability Zone A (Independent Datacenter / Power / Networking)"]
    Regions --> AZ2["Availability Zone B (Physically Isolated)"]
    Regions --> AZ3["Availability Zone C (Ultra-low Latency < 2ms Interconnect)"]
Loading
  • Region: A geographic location containing at least 3 physically isolated Availability Zones (AZs).
  • Availability Zone (AZ): One or more discrete data centers with independent redundant power, cooling, and physical security, interconnected via low-latency fiber (< 2ms).
  • Edge Locations: Global Points of Presence (PoP) running CloudFront (CDN), Route 53 (DNS), and AWS WAF at network perimeters closest to end users.

1.2 IAM Entity Hierarchy: Users, Groups, Roles, and Policies

IAM (Identity and Access Management) provides centralized identity governance across AWS resources:

flowchart TD
    Principal["Principal (User / EC2 Instance / Lambda / CI/CD)"] --> Role["IAM Role (AssumeRole API via STS)"]
    Role --> TempCreds["Temporary Credentials (AccessKey, SecretKey, SessionToken)"]
    TempCreds --> Policies["IAM Policies (Permissions Granted)"]
    Policies --> Resource["AWS Resource (S3 Bucket, DynamoDB Table)"]
Loading
  • IAM User: Permanent identity representing a person or service (discouraged in modern multi-account setups in favor of SSO / IAM Identity Center).
  • IAM Role: An identity with specific permissions that can be assumed dynamically by authorized principals (EC2 instances, Lambda functions, or external federated users via SAML 2.0 / OIDC). Roles use temporary, auto-expiring security credentials generated by AWS Security Token Service (STS).
  • IAM Policy: A JSON document that explicitly grants or denies permissions to actions on resources.

1.3 IAM Policy Evaluation Logic

When a principal attempts an action in AWS, the evaluation engine resolves permissions using a strict hierarchy:

       ┌──────────────────────────────────────────────┐
       │ Is there an EXPLICIT DENY?                   │
       └──────────────┬───────────────────────────────┘
                      │
         YES ─────────┴───────── NO
          │                      │
          ▼                      ▼
  [ACCESS DENIED]      ┌──────────────────────────────────────────────┐
                       │ Is there an EXPLICIT ALLOW?                  │
                       └──────────────┬───────────────────────────────┘
                                      │
                         YES ─────────┴───────── NO
                          │                      │
                          ▼                      ▼
                   [ACCESS GRANTED]        [DEFAULT DENY]
                                           (ACCESS DENIED)
  1. Default Deny: By default, all requests are implicitly denied.
  2. Explicit Deny Always Wins: If any policy (Identity policy, SCP, Resource policy) contains a Deny, the request is immediately rejected, overriding any Allow!
  3. Explicit Allow: Request must be explicitly permitted by an authorized policy with no active denies.

1.4 Cross-Account Delegation with AWS STS AssumeRole

In enterprise multi-account architectures (e.g. Dev, Staging, Production), engineers do NOT create permanent IAM users in production:

sequenceDiagram
    autonumber
    participant DevUser as Developer (Account A: Shared Services)
    participant STS as AWS STS (Security Token Service)
    participant ProdAccount as Production Account B (Target)

    DevUser->>STS: sts:AssumeRole(RoleArn="arn:aws:iam::ProdAccount:role/DeployerRole")
    STS->>DevUser: Returns Temporary Credentials (Valid for 1 Hour)
    DevUser->>ProdAccount: Execute Deployment using Temporary Credentials
    ProdAccount-->>DevUser: Verifies Session Token & Executes Action
Loading

1.5 Line-by-Line Breakdown: Enterprise Least-Privilege IAM Policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowListAndReadOrdersBucketOnly",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetObject"
      ],
      "Resource": [
        "arn:aws:s3:::enterprise-orders-prod",
        "arn:aws:s3:::enterprise-orders-prod/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "true"
        },
        "StringEquals": {
          "s3:ExistingObjectTag/classification": "confidential"
        }
      }
    },
    {
      "Sid": "ExplicitDenyUnencryptedPut",
      "Effect": "Deny",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::enterprise-orders-prod/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      }
    }
  ]
}
Policy Block Purpose Security & Compliance Guarantee
"Version": "2012-10-17" Specification Current standard IAM policy language version (do not use current calendar date!).
s3:ListBucket vs s3:GetObject Action Scoping ListBucket acts on the bucket ARN (...prod); GetObject acts on objects inside (...prod/*).
"aws:SecureTransport": "true" TLS Enforcement Strictly rejects any unencrypted HTTP requests; forces TLS 1.2+ for in-transit data.
"s3:ExistingObjectTag/classification" ABAC (Attribute) Restricts access to objects tagged specifically as confidential (Attribute-Based Access Control).
ExplicitDenyUnencryptedPut Encryption Guard Rejects uploads that fail to specify AWS KMS server-side encryption.


1.7 AWS Organizations & Multi-Account Landing Zone Architecture

Enterprises never run production and development workloads in a single AWS account:

flowchart TD
    Root["Root Organizational Unit (Management Account)"] --> CoreOU["Core / Infrastructure OU"]
    Root --> SecurityOU["Security OU"]
    Root --> WorkloadsOU["Workloads OU"]

    CoreOU --> NetworkAcc["Network Hub Account (Transit Gateway / Direct Connect)"]
    CoreOU --> SharedAcc["Shared Services Account (CI/CD / Artifact Registries)"]

    SecurityOU --> LogArchive["Log Archive Account (Immutable S3 Buckets)"]
    SecurityOU --> SecTooling["Security Tooling Account (GuardDuty / Security Hub / IAM Identity Center)"]

    WorkloadsOU --> ProdAcc["Production Workloads Account"]
    WorkloadsOU --> NonProdAcc["Non-Production Workloads Account"]
Loading

Production Service Control Policy (SCP): Enforce Approved Regions

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnapprovedRegions",
      "Effect": "Deny",
      "NotAction": [
        "iam:*",
        "organizations:*",
        "route53:*",
        "cloudfront:*",
        "support:*"
      ],
      "Resource": "*",
      "Condition": {
        "StringNotEquals": {
          "aws:RequestedRegion": [
            "us-east-1",
            "eu-west-1"
          ]
        }
      }
    }
  ]
}
SCP Rule Component Behavior Security Governance Impact
NotAction Whitelists Global Services Prevents locking out global AWS control planes (iam:*, route53:*, cloudfront:*).
Condition.StringNotEquals Region Lock Blocks provisioning resources in any unapproved region (e.g. ap-northeast-1).
Root User Scope Unconditional Override Applies strictly even to account root users across all member accounts in the OU.

2. Stage 2: Networking & Virtual Private Cloud (VPC)

2.1 Designing a Production VPC: CIDR Blocks & Subnet Topology

A Virtual Private Cloud (VPC) is your logically isolated software-defined network in the AWS cloud. Production architectures follow a 3-Tier Subnet Layout spread across at least 3 Availability Zones:

flowchart TD
    VPC["Production VPC (CIDR: 10.0.0.0/16 - 65,536 IPs)"] --> AZ1["AZ us-east-1a"]
    VPC --> AZ2["AZ us-east-1b"]

    subgraph AZ1["AZ us-east-1a"]
        Public1["Public Subnet (10.0.1.0/24)<br/>ALB / NAT Gateway"]
        Private1["Private Subnet (10.0.10.0/24)<br/>EKS / ECS / Lambda"]
        Isolated1["Isolated Subnet (10.0.20.0/24)<br/>Aurora PostgreSQL / Redis"]
    end

    subgraph AZ2["AZ us-east-1b"]
        Public2["Public Subnet (10.0.2.0/24)<br/>ALB / NAT Gateway"]
        Private2["Private Subnet (10.0.20.0/24)<br/>EKS / ECS / Lambda"]
        Isolated2["Isolated Subnet (10.0.40.0/24)<br/>Aurora PostgreSQL / Redis"]
    end
Loading
Subnet Tier Routing & Gateways Allowed Workloads
Tier 1: Public Subnets Direct route table path to Internet Gateway (0.0.0.0/0 -> igw-xxx). Public Application Load Balancers (ALB), NAT Gateways.
Tier 2: Private App Subnets Outbound route via NAT Gateway (0.0.0.0/0 -> nat-xxx). No inbound internet access! Microservices, API backends, Kubernetes worker nodes.
Tier 3: Isolated Database Subnets Zero default route to internet (no IGW, no NAT). Reachable only via private subnets! Amazon Aurora, RDS, DynamoDB endpoints, Redis.

2.2 Security Groups (Stateful) vs Network ACLs (Stateless)

flowchart LR
    Internet["Incoming Packet"] --> NACL["Network ACL (Stateless Subnet Boundary)"]
    NACL --> SG["Security Group (Stateful Instance Boundary)"]
    SG --> EC2["EC2 Instance / Container"]
Loading
Feature Security Groups Network ACLs (NACL)
Scope of Enforcement Operates at the Instance / ENI level. Operates at the Subnet boundary.
State Tracking Stateful: Return traffic is automatically allowed, regardless of inbound rules. Stateless: Return traffic must be explicitly allowed via ephemeral ports (1024-65535).
Rule Types Supports Allow rules only. Supports both Allow and Deny rules.
Evaluation Order Evaluates all rules collectively before deciding. Evaluates rules sequentially in numerical order (lowest number first).
Primary Use Case Host firewall protecting microservices. Blocking specific malicious IP addresses / CIDR ranges.

2.3 AWS PrivateLink & VPC Endpoints (Gateway vs Interface)

Accessing AWS services (like S3 or DynamoDB) from private subnets over the public internet exposes traffic to internet routing and incurs expensive NAT Gateway data processing fees ($0.045/GB). VPC Endpoints route traffic entirely over the private Amazon network backbone:

Endpoint Type Target Services Cost Architecture
Gateway Endpoint Amazon S3 and DynamoDB only. 100% Free Adds an entry to VPC Route Table (pl-63a5400a -> vpce-xxx).
Interface Endpoint (PrivateLink) 100+ AWS services (Secrets Manager, SQS, KMS, ECR). Hourly fee + per GB fee Deploys an Elastic Network Interface (ENI) with a private IP inside your subnet.


2.6 Production Multi-AZ VPC Infrastructure as Code (Terraform)

# main.tf - Enterprise 3-Tier Multi-AZ VPC
resource "aws_vpc" "main" {
  cidr_block           = "10.0.0.0/16"
  enable_dns_hostnames = true
  enable_dns_support   = true

  tags = {
    Name        = "production-vpc"
    Environment = "production"
  }
}

# Internet Gateway for Public Subnet Egress
resource "aws_internet_gateway" "gw" {
  vpc_id = aws_vpc.main.id
}

# Elastic IP for NAT Gateway
resource "aws_eip" "nat" {
  domain = "vpc"
}

# NAT Gateway deployed in Public Subnet
resource "aws_nat_gateway" "nat" {
  allocation_id = aws_eip.nat.id
  subnet_id     = aws_subnet.public[0].id
  depends_on    = [aws_internet_gateway.gw]
}

# Private Route Table routing outbound traffic through NAT Gateway
resource "aws_route_table" "private" {
  vpc_id = aws_vpc.main.id

  route {
    cidr_block     = "0.0.0.0/0"
    nat_gateway_id = aws_nat_gateway.nat.id
  }
}
Terraform Resource Architectural Purpose Cloud Engineering Guarantee
enable_dns_hostnames VPC DNS Resolution Allocates public and private Amazon DNS hostnames to instances inside VPC.
aws_nat_gateway Outbound Egress Allows private microservices to pull Docker images/updates without exposing ports to internet.
depends_on = [gw] Dependency Ordering Ensures Internet Gateway is attached before NAT Gateway allocates its public IP.
0.0.0.0/0 -> nat Private Route Table Default gateway route for private subnets directing all internet traffic through NAT.

3. Stage 3: Compute & Serverless Architectures

3.1 Amazon EC2: Purchasing Models & Auto Scaling

Amazon Elastic Compute Cloud (EC2) provides resizable compute capacity:

flowchart TD
    EC2["EC2 Instance Purchasing Models"] --> OnDemand["On-Demand: Pay by the second; zero commitment (Spiky workloads)"]
    EC2 --> Savings["Savings Plans / Reserved: Commit 1-3 years for up to 72% discount"]
    EC2 --> Spot["Spot Instances: Bid on spare AWS capacity for up to 90% discount!"]
Loading
  • Spot Instances: AWS can reclaim Spot instances with a 2-minute warning. Perfect for stateless containers (ECS Fargate Spot, EKS Spot nodes, batch processing, CI/CD runners).

3.2 Container Orchestration: AWS ECS vs EKS

  • AWS ECS (Elastic Container Service): AWS-native, highly integrated container orchestrator. Extremely fast startup, low operational overhead, native IAM role per task (TaskRoleArn).
  • AWS EKS (Elastic Kubernetes Service): Fully certified Kubernetes control plane managed by AWS. Ideal for multi-cloud parity, complex Helm chart ecosystems, and service meshes (Istio).
  • AWS Fargate: Serverless compute engine for both ECS and EKS. Eliminates EC2 server management entirely; you specify CPU/RAM, and AWS provisions and patches the underlying execution environment.

3.3 AWS Lambda Execution Lifecycle & Cold Start Optimization

AWS Lambda executes code in response to events with zero server provisioning:

flowchart LR
    ColdStart["Cold Start: Download Code -> Start Container -> Init Runtime"] --> Warm["Warm Execution: Invoke Handler"]
    Warm --> NextInvoke["Subsequent Invokes: Reuses Warm Container (Instant!)"]
Loading

Cold Start Mitigation Strategies

  1. Provisioned Concurrency: Keeps pre-initialized execution environments permanently warm, guaranteeing single-digit millisecond latency at all times.
  2. Minimize Package Size: Tree-shake dependencies; avoid heavy monolithic libraries.
  3. Initialize Outside the Handler: Database connection pools and AWS SDK clients should be instantiated outside the exports.handler function so they are preserved across warm invocations!


3.5 Production Serverless Lambda Function Handler (TypeScript)

import { APIGatewayProxyEvent, APIGatewayProxyResult } from 'aws-lambda';
import { DynamoDBClient } from '@aws-sdk/client-dynamodb';
import { DynamoDBDocumentClient, GetCommand } from '@aws-sdk/lib-dynamodb';

// Cold Start Optimization: Initialize DB client OUTSIDE the handler execution scope!
const rawClient = new DynamoDBClient({ region: process.env.AWS_REGION || 'us-east-1' });
const ddbDocClient = DynamoDBDocumentClient.from(rawClient);

export const handler = async (event: APIGatewayProxyEvent): Promise<APIGatewayProxyResult> => {
  const userId = event.pathParameters?.userId;

  if (!userId) {
    return {
      statusCode: 400,
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ error: 'Missing userId path parameter' }),
    };
  }

  try {
    const response = await ddbDocClient.send(
      new GetCommand({
        TableName: process.env.USERS_TABLE_NAME,
        Key: { PK: `USER#${userId}`, SK: 'PROFILE' },
      })
    );

    if (!response.Item) {
      return {
        statusCode: 404,
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ error: 'User profile not found' }),
      };
    }

    return {
      statusCode: 200,
      headers: {
        'Content-Type': 'application/json',
        'Cache-Control': 'max-age=60',
      },
      body: JSON.stringify(response.Item),
    };
  } catch (error) {
    console.error('DynamoDB lookup failed:', error);
    return {
      statusCode: 500,
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ error: 'Internal Server Error' }),
    };
  }
};
Implementation Detail Purpose Production Benefit
Client initialized outside handler Connection Reuse HTTP connections and TCP/TLS handshakes are preserved across warm container invocations.
DynamoDBDocumentClient Native Type Unmarshaling Automatically translates native JS types to/from DynamoDB AttributeValues.
Cache-Control: max-age=60 Edge Caching Informs CloudFront CDN to cache responses at edge PoPs, reducing Lambda invocations.

4. Stage 4: Storage, Databases & Caching

4.1 Amazon S3: Storage Classes & Lifecycle Management

Amazon S3 provides 99.999999999% (11 9s) of durability by automatically replicating objects across a minimum of 3 Availability Zones:

flowchart TD
    Upload["Object Uploaded"] --> S3Std["S3 Standard (Hot Frequent Access)"]
    S3Std -->|30 Days| S3IA["S3 Standard-Infrequent Access (Cheaper Storage, Retrieval Fee)"]
    S3IA -->|90 Days| S3Glacier["S3 Glacier Flexible (Cold Archive - Minutes to Retrieve)"]
    S3Glacier -->|365 Days| S3Deep["S3 Glacier Deep Archive (Lowest Cost - 12hr Retrieval)"]
Loading

4.2 Amazon Aurora: Storage Architecture & High Availability

Unlike traditional RDS (which runs a standard PostgreSQL/MySQL engine with EBS block replication), Amazon Aurora fundamentally redesigns database storage:

flowchart TD
    Primary["Aurora Writer Node (Compute Instance)"] --> StorageEngine["Aurora Distributed Log Storage Fleet"]
    subgraph StorageEngine["6-Way Replication Across 3 Availability Zones"]
        AZ1_A["AZ 1: Copy 1"]
        AZ1_B["AZ 1: Copy 2"]
        AZ2_A["AZ 2: Copy 3"]
        AZ2_B["AZ 2: Copy 4"]
        AZ3_A["AZ 3: Copy 5"]
        AZ3_B["AZ 3: Copy 6"]
    end
    Replica["Aurora Read Replica (Shares Same Storage Layer!)"] <--> StorageEngine
Loading
  • Quorum I/O: Writes require confirmation from only 4 of 6 storage copies ($4/6$ write quorum); reads require only 3 of 6 copies ($3/6$ read quorum). Tolerates loss of an entire AZ plus an additional drive without data loss!
  • Zero-Lag Read Replicas: Read replicas do NOT replicate data independently; they read directly from the shared distributed storage layer, achieving sub-10ms replication lag.

4.3 Amazon DynamoDB: Single-Table Architecture

DynamoDB is a fully managed NoSQL key-value and document database delivering single-digit millisecond performance at any scale:

Table: EnterpriseStore (Partition Key: PK, Sort Key: SK)
┌──────────────────────┬──────────────────────┬──────────────────────────────────────────┐
│ PK                   │ SK                   │ Attributes (Polymorphic Payload)        │
├──────────────────────┼──────────────────────┼──────────────────────────────────────────┤
│ USER#1042            │ PROFILE              │ Name: "Alex", Email: "alex@co.com"       │
│ USER#1042            │ ORDER#2026-09-07#001 │ Total: $149.50, Status: "SHIPPED"        │
│ USER#1042            │ ORDER#2026-09-07#002 │ Total: $89.00, Status: "PENDING"         │
│ PRODUCT#PROD_99      │ DETAILS              │ Title: "Mechanical Keyboard", Stock: 42  │
└──────────────────────┴──────────────────────┴──────────────────────────────────────────┘
  • Single-Table Design: Instead of creating multiple relational tables, store multiple heterogeneous entity types in a single table using generic partition (PK) and sort keys (SK). Retrieve a user and all their recent orders in a single atomic query!


4.6 DynamoDB Single-Table Querying with Global Secondary Indexes (GSI)

import { QueryCommand } from '@aws-sdk/lib-dynamodb';

// Query all orders for a customer placed in 2026 using Sort Key range conditions:
export async function getCustomerOrdersIn2026(docClient: any, customerId: string) {
  const command = new QueryCommand({
    TableName: 'EnterpriseStore',
    KeyConditionExpression: 'PK = :pk AND SK BETWEEN :start_date AND :end_date',
    ExpressionAttributeValues: {
      ':pk': `USER#${customerId}`,
      ':start_date': 'ORDER#2026-01-01',
      ':end_date': 'ORDER#2026-12-31',
    },
    // ScanIndexForward = false sorts results descending (newest orders first)
    ScanIndexForward: false,
    Limit: 25,
  });

  const result = await docClient.send(command);
  return result.Items;
}
Parameter Functionality Technical Behavior
KeyConditionExpression Partition + Sort Key Filter Operates directly on the B-Tree index in storage; ignores non-matching records in $O(\log N)$ time.
ScanIndexForward: false Chronological Reversal Traverses the B-Tree leaf blocks in reverse order; outputs newest orders with zero sorting compute.
Limit: 25 Pagination Slicing Returns first 25 matching records; returns LastEvaluatedKey if more records exist.

4.7 Production Hardened S3 Bucket Policy (JSON)

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "EnforceTLSRequestsOnly",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::enterprise-data-lake-prod",
        "arn:aws:s3:::enterprise-data-lake-prod/*"
      ],
      "Condition": {
        "Bool": {
          "aws:SecureTransport": "false"
        }
      }
    },
    {
      "Sid": "EnforceKMSEncryptionOnUpload",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::enterprise-data-lake-prod/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "aws:kms"
        }
      }
    }
  ]
}
Policy Statement Mechanism Security Standard Enforced
EnforceTLSRequestsOnly Explicit Deny on aws:SecureTransport = false Blocks unencrypted plain HTTP transport; enforces TLS 1.2+ for in-transit data.
EnforceKMSEncryptionOnUpload Explicit Deny on x-amz-server-side-encryption Mandates that every uploaded file is encrypted with a Customer Managed KMS key.

5. Stage 5: Messaging, Integration & Asynchronous Pipelines

5.1 Amazon SQS: Standard vs FIFO Queues

flowchart LR
    Producer["Producers"] --> SQS["Amazon SQS Queue"]
    SQS --> Consumer["Consumer Fleet (Auto-scaled by Queue Depth)"]
    SQS -.->|Failed 3x Retries| DLQ["Dead Letter Queue (DLQ for Inspection)"]
Loading
Dimension SQS Standard Queue SQS FIFO Queue (.fifo)
Ordering Best-effort ordering (out-of-order possible). Strict First-In-First-Out guaranteed.
Delivery Semantics At-Least-Once (duplicates possible). Exactly-Once (deduplication via MessageDeduplicationId).
Throughput Capacity Nearly unlimited transactions/sec. 3,000 msgs/sec with batching (300/s without batching).
Message Grouping None. MessageGroupId ensures strict ordering per user/partition.

5.2 Amazon SNS + SQS Pub/Sub Fanout Pattern

Allows a single event to be processed independently by multiple microservices:

flowchart TD
    OrderSvc["Order Service (Publisher)"] --> SNS["SNS Topic: order-created"]
    SNS --> SubQueue1["SQS Queue: Payment Processing Service"]
    SNS --> SubQueue2["SQS Queue: Inventory Warehouse Service"]
    SNS --> SubQueue3["SQS Queue: Email Customer Notification Service"]
Loading


5.6 Enterprise SQS Dead Letter Queue & Redrive Architecture

flowchart LR
    OrdersAPI["Orders API Service"] -->|Send Order Event| MainQueue["Amazon SQS (orders-queue)"]
    MainQueue -->|Polls Messages| LambdaWorker["Lambda Worker Function"]
    LambdaWorker -->|Processing Fails!| ErrorRetry["Automatic Retry with Exponential Backoff"]
    ErrorRetry -.->|Failed maxReceiveCount = 3 times| DLQ["Dead Letter Queue (orders-dlq)"]
    DLQ --> CloudWatchAlarm["CloudWatch Alarm (Trigger PagerDuty / SRE On-Call)"]
    DLQ -.->|Bug Fixed in Worker| RedriveTask["AWS SQS Dead Letter Queue Redrive Task"]
    RedriveTask -->|Reprocess Messages| MainQueue
Loading
DLQ Configuration Parameter Value Operational Guarantee
maxReceiveCount 3 If a message fails processing 3 consecutive times, SQS moves it to the DLQ automatically.
visibilityTimeout 6x Lambda Timeout Ensures consumer has sufficient time to complete processing before message becomes visible again.
RedriveTask Native AWS Feature Reprocesses all accumulated DLQ messages back into the main queue once bug is patched.

6. Stage 6: Security, Governance & Observability

6.1 AWS KMS: Envelope Encryption Architecture

To protect data at rest while maintaining high cryptographic performance, AWS employs Envelope Encryption:

flowchart TD
    KMS["AWS KMS (Stores Customer Master Key / KMS Key)"] -->|1. GenerateDataKey API| PlainDEK["Plaintext Data Encryption Key (DEK)"]
    KMS --> EncDEK["Encrypted Data Encryption Key (Ciphertext DEK)"]
    PlainDEK -->|2. Encrypts Massive 50GB File locally| EncData["Encrypted Data Payload"]
    PlainDEK -.->|3. Plaintext DEK destroyed from RAM!| Purged["Purged Memory"]
    EncDEK --> Packaged["Stored Together on Disk: [Encrypted DEK + Encrypted Data Payload]"]
Loading
  • Why Envelope Encryption? Encrypting gigabytes of data directly via KMS API exceeds network bandwidth limits. With envelope encryption, KMS only encrypts small 256-bit symmetric keys; data itself is encrypted locally with AES-256.

6.2 The AWS Well-Architected Framework: 6 Core Pillars

  1. Operational Excellence: Infrastructure as Code (Terraform), observing Golden Signals, testing failure injection (Chaos Engineering).
  2. Security: Apply least privilege (IAM), zero-trust networking, encrypt data at rest and in transit.
  3. Reliability: Multi-AZ and multi-region deployments, automated recovery from failure (ASGs, Aurora failover).
  4. Performance Efficiency: Right-sizing compute, caching at edge (CloudFront) and memory (ElastiCache).
  5. Cost Optimization: Decommissioning idle resources, using Spot instances, implementing S3 lifecycle policies.
  6. Sustainability: Selecting modern AWS Graviton (ARM-based) chips that provide up to 60% better energy efficiency per compute unit.


6.6 CloudWatch Logs Insights Production Query Library

Query 1: Find Top 10 API Paths Generating HTTP 500 Errors

fields @timestamp, requestPath, status, errorMessage
| filter status >= 500
| stats count(*) as errorCount by requestPath
| sort errorCount desc
| limit 10

Query 2: Identify Lambda Cold Starts and Memory Over-Allocation

filter @type == "REPORT"
| parse @message /Init Duration: (?<initDuration>.*?) ms/
| stats
    count(*) as totalInvocations,
    count(initDuration) as coldStarts,
    avg(initDuration) as avgColdStartDuration,
    max(@maxMemoryUsed / 1024 / 1024) as maxMemoryMB,
    max(@memorySize / 1024 / 1024) as allocatedMemoryMB
by @logStream
| filter coldStarts > 0

Query 3: Analyze Rejected Traffic from VPC Flow Logs

fields @timestamp, srcAddr, dstAddr, srcPort, dstPort, protocol
| filter action == "REJECT"
| stats count(*) as rejectCount by srcAddr, dstPort
| sort rejectCount desc
| limit 20

6.7 AWS KMS Envelope Encryption Implementation (TypeScript)

import { KMSClient, GenerateDataKeyCommand, DecryptCommand } from '@aws-sdk/client-kms';
import crypto from 'node:crypto';

const kms = new KMSClient({ region: 'us-east-1' });

export class EnvelopeEncryptionService {
  constructor(private kmsKeyId: string) {}

  async encryptPayload(plaintext: Buffer): Promise<{ encryptedData: Buffer; encryptedKey: Uint8Array }> {
    // 1. Ask AWS KMS to generate a 256-bit symmetric AES Data Encryption Key (DEK)
    const command = new GenerateDataKeyCommand({
      KeyId: this.kmsKeyId,
      KeySpec: 'AES_256',
    });
    const { Plaintext: plainDEK, CiphertextBlob: encryptedKey } = await kms.send(command);

    if (!plainDEK || !encryptedKey) throw new Error('KMS key generation failed');

    // 2. Encrypt the data payload locally using AES-256-GCM in RAM
    const iv = crypto.randomBytes(12);
    const cipher = crypto.createCipheriv('aes-256-gcm', plainDEK, iv);
    const encryptedData = Buffer.concat([iv, cipher.update(plaintext), cipher.final(), cipher.getAuthTag()]);

    // 3. ZERO OUT and destroy the plaintext DEK from memory immediately!
    plainDEK.fill(0);

    return { encryptedData, encryptedKey };
  }
}
Envelope Step Execution Location Performance & Security Impact
GenerateDataKey AWS KMS Cloud Hardware Small 256-bit key returned; fast API call ($&lt;15\text{ ms}$).
aes-256-gcm Local CPU RAM Encrypts gigabytes of data with hardware AES-NI instructions at $&gt;2\text{ GB/sec}$.
plainDEK.fill(0) Application Process Eliminates plaintext key from memory dumps and GC heap sweeps.

7. Stage 7: Staff Solutions Architect & Cloud SRE Interview Handbook

7.1 Disaster Recovery (DR) Strategies Compared

flowchart TD
    DR["Disaster Recovery Strategies"] --> DR1["1. Backup & Restore (RTO: Hours, RPO: 24h) - Lowest Cost"]
    DR --> DR2["2. Pilot Light (RTO: 10-30 min, RPO: Minutes) - Core DB synced, compute idle"]
    DR --> DR3["3. Warm Standby (RTO: Minutes, RPO: Seconds) - Scaled-down fleet running in Region B"]
    DR --> DR4["4. Multi-Region Active-Active (RTO: Real-time, RPO: ~0) - Highest Cost & Resilience"]
Loading
  • Recovery Time Objective (RTO): The maximum acceptable time to restore operations after a disaster.
  • Recovery Point Objective (RPO): The maximum acceptable data loss measured in time (e.g. 5 minutes of lost transactions).

7.2 25 Staff Solutions Architect & SRE Interview Q&As

Q1: What is the exact difference between an AWS Region, an Availability Zone (AZ), and a Local Zone?

Answer: An AWS Region is a physical geographical area containing multiple isolated Availability Zones. An Availability Zone (AZ) consists of one or more discrete data centers with redundant power, networking, and connectivity, interconnected by ultra-low-latency private fiber links (< 2ms). A Local Zone extends an AWS Region by placing compute, storage, and database services in major metropolitan centers closer to large population hubs to achieve single-digit millisecond latency for local end users (e.g. game streaming, real-time trading).

Q2: How do you design an active-active multi-region database architecture on AWS?

Answer: Use Amazon Aurora Global Database or Amazon DynamoDB Global Tables.

  • With DynamoDB Global Tables: Tables are replicated automatically across selected regions using multi-master replication. Writes to any region are asynchronously propagated to all other regions within seconds using internal DynamoDB Streams, with "last-write-wins" conflict resolution based on millisecond timestamps.
  • Route traffic using Amazon Route 53 with Latency-Based Routing paired with Route 53 Health Checks for instant regional failover.

Q3: Why should NAT Gateways be deployed per Availability Zone instead of sharing a single NAT Gateway?

Answer:

  1. High Availability: If the single AZ hosting the shared NAT Gateway experiences a power or network failure, all private subnets across all other AZs immediately lose outbound internet connectivity. Deploying an independent NAT Gateway in each AZ isolates failure domains.
  2. Cross-AZ Data Transfer Costs: Traffic traveling across AZ boundaries incurs inter-AZ data transfer fees ($0.01/GB). Routing private traffic through a local AZ NAT Gateway avoids cross-AZ network costs.

Q4: How does AWS IAM evaluate conflicting identity-based, resource-based, and Service Control Policies (SCPs)?

Answer:

  1. The request starts with an implicit default deny.
  2. The engine evaluates Service Control Policies (SCPs) at the AWS Organizations level. If the SCP denies the action, access is denied.
  3. The engine checks for an Explicit Deny in any attached policy. If found, access is immediately denied.
  4. The engine checks Permission Boundaries. If defined, the action must be permitted by the boundary.
  5. If the request is cross-account, both the identity policy in the calling account AND the resource-based policy (or trust policy) in the destination account must explicitly allow the action.
  6. If any check fails, access is denied.

Q5: What is the difference between Amazon SQS Visibility Timeout and Message Retention Period?

Answer:

  • Visibility Timeout (default 30 seconds): When a consumer receives a message via ReceiveMessage, the message remains in the queue but becomes invisible to other consumers. If the consumer fails to process and delete the message before the timeout expires, the message becomes visible again for another consumer to process.
  • Message Retention Period (default 4 days, max 14 days): The total amount of time a message can reside in the queue before being permanently deleted by AWS if unconsumed.

Q6: How do you prevent the Thundering Herd problem in Amazon DynamoDB?

Answer: DynamoDB partitions throughput evenly across physical partitions based on the hash of the Partition Key (PK). If millions of requests hit the exact same PK simultaneously (a "Hot Partition"):

  1. Write Sharding (Salted Keys): Append a random or calculated suffix to the partition key (ORDER#2026#001_N, where $N \in [1..10]$), distributing writes across 10 physical storage partitions.
  2. DAX (DynamoDB Accelerator): Place an in-memory DAX cluster in front of DynamoDB to cache reads in RAM with microsecond latency.

Q7: What are the security differences between AWS Secrets Manager and AWS Systems Manager Parameter Store?

Answer:

  • SSM Parameter Store: Free for standard parameters; supports SecureString encrypted via KMS. Ideal for static configuration, database URLs, and API keys that do not require automatic rotation.
  • Secrets Manager: Charges $0.40/secret/month. Supports native automated credential rotation via pre-built Lambda functions for Amazon RDS, Aurora, and Redshift. Features cross-account secret replication and fine-grained IAM resource policies.

Q8: Explain how Amazon S3 achieves 99.999999999% (11 9s) durability.

Answer: S3 automatically stores objects across a minimum of three geographically separated Availability Zones within an AWS Region. It uses erasure coding and continuous background data integrity verification. If physical disks fail or an entire data center is destroyed, the remaining storage nodes reconstruct missing data chunks automatically without downtime or loss.

Q9: What is the difference between AWS ALB, NLB, and GLB?

Answer:

  • Application Load Balancer (ALB): Layer 7 (HTTP/HTTPS/gRPC). Content-based routing (paths, hostnames, headers), TLS termination, integration with AWS WAF.
  • Network Load Balancer (NLB): Layer 4 (TCP/UDP/TLS). Ultra-low latency, capable of handling tens of millions of requests per second with static Elastic IP addresses.
  • Gateway Load Balancer (GLB): Layer 3/4. Designed for deploying and scaling third-party virtual security appliances (firewalls, IDS/IPS) using the GENEVE protocol.

Q10: How does AWS Lambda handle concurrency and scaling?

Answer: Concurrency is the number of execution environments handling requests simultaneously: $$\text{Concurrency} = \text{Invocations Per Second} \times \text{Average Execution Duration (seconds)}$$ By default, accounts have a regional pool of 1,000 concurrent executions.

  • Reserved Concurrency: Guarantees a dedicated slice of capacity for a critical function and caps it to prevent it from exhausting the regional pool.
  • Provisioned Concurrency: Keeps execution environments initialized and warm, eliminating cold starts.

Q11: What is the difference between S3 Transfer Acceleration and CloudFront?

Answer:

  • CloudFront: A full Content Delivery Network (CDN) that caches static and dynamic responses at edge locations globally.
  • S3 Transfer Acceleration: Does not cache files. It uses CloudFront edge locations to route uploads over the optimized, private AWS global fiber network directly to the target S3 bucket, bypassing congested public internet transit.

Q12: How do you achieve zero-downtime database upgrades on Amazon RDS?

Answer: Use Amazon RDS Blue/Green Deployments:

  1. AWS clones the production database environment (Blue) into an isolated staging environment (Green) with continuous replication.
  2. Apply minor or major engine upgrades, schema migrations, or OS patches to the Green environment.
  3. Run integration tests on Green.
  4. Execute switchover: RDS redirects traffic to Green in $&lt;1\text{ minute}$ with zero data loss and automated rollback if errors occur.

Q13: What is the purpose of an Amazon S3 Gateway Endpoint and why is it preferred over a NAT Gateway?

Answer: S3 Gateway Endpoints are completely free and provide a direct routing path from private subnets to S3 through the internal AWS network. Routing S3 data through a NAT Gateway incurs $0.045/GB data processing charges and exposes transfers to NAT throughput limits. Gateway Endpoints deliver higher throughput at zero cost.

Q14: How does AWS Shield Advanced differ from standard AWS Shield?

Answer:

  • Shield Standard: Free, automatic Layer 3/4 DDoS protection applied to all AWS customers (SYN floods, UDP reflection).
  • Shield Advanced: Paid subscription ($3,000/month). Provides customized Layer 7 DDoS mitigation, 24/7 access to the AWS Shield Response Team (SRT), and financial cost protection against bill spikes caused by DDoS attacks.

Q15: What is the difference between Amazon SQS Short Polling and Long Polling?

Answer:

  • Short Polling (WaitTimeSeconds = 0): SQS samples a subset of storage servers and returns immediately, even if no messages are found. Generates high empty response counts and increases AWS API costs.
  • Long Polling (WaitTimeSeconds = 1-20): SQS holds the connection open until a message arrives or the timeout expires, drastically reducing empty receives and lowering billable API calls.

Q16: How does AWS KMS implement Key Separation of Duties?

Answer: KMS separates the key administrator from the key user:

  • Key Administrators have IAM permissions to manage, rotate, enable, and delete the KMS Key, but have ZERO permissions to perform cryptographic operations (kms:Encrypt, kms:Decrypt).
  • Key Users (applications) have permissions to encrypt/decrypt data, but cannot delete or alter key policies.

Q17: What is an AWS Transit Gateway and when should it replace VPC Peering?

Answer: VPC Peering requires point-to-point connections ($O(N^2)$ connections for $N$ VPCs). Managing 50 VPCs requires 1,225 peering links! AWS Transit Gateway acts as a centralized cloud router (hub-and-spoke model). All VPCs, on-premises VPNs, and Direct Connect links attach to the Transit Gateway, simplifying network routing to $O(N)$ attachments.

Q18: What is AWS Step Functions and why shouldn't you orchestrate complex workflows in Lambda?

Answer: Orchestrating multi-step workflows inside a Lambda function by having Lambda sleep or poll downstream services forces you to pay for idle execution time. If the function times out at 15 minutes, state is lost. AWS Step Functions manages state machines natively, handles automatic retries with exponential backoff, visualizes execution flow, and can pause for days or weeks (e.g. human approval workflows) at zero idle cost.

Q19: Explain the difference between Amazon EBS gp3 and io2 Block Express.

Answer:

  • gp3: General Purpose SSD. Provides baseline 3,000 IOPS and 125 MB/s throughput independently of storage capacity, with ability to provision up to 16,000 IOPS. Ideal for virtual desktops and general workloads.
  • io2 Block Express: Extreme performance sub-millisecond SAN storage for mission-critical databases (Oracle, SAP HANA). Provides up to 256,000 IOPS and 4,000 MB/s throughput with 99.999% durability.

Q20: What are AWS Service Control Policies (SCPs) and where are they enforced?

Answer: SCPs are governance guardrails applied at the Organization or Organizational Unit (OU) level in AWS Organizations. They specify the maximum permissions that can be granted in member accounts. Crucially, SCPs restrict even the root user of a member account! For example, an SCP can deny anyone in the organization from disabling CloudTrail or launching resources outside approved regions (eu-west-1, us-east-1).

Q21: How do you achieve zero data loss (RPO = 0) with Amazon Aurora?

Answer: Enable Aurora Global Database with synchronous write replication across local AZs, paired with fast asynchronous replication (< 1 second) across regions. In the event of primary regional outage, promote the secondary region to primary via Managed Planned Failover.

Q22: What is Amazon CloudWatch Logs Insights and what is an example query?

Answer: CloudWatch Logs Insights is a purpose-built query engine that allows interactive analysis of log data using a SQL-like syntax:

fields @timestamp, @message
| filter @message like /ERROR/
| stats count(*) by bin(5m)
| sort @timestamp desc
| limit 20

Q23: What is the difference between AWS Graviton and x86 EC2 instances?

Answer: AWS Graviton processors are custom 64-bit ARM-based CPUs designed by Amazon (Annapurna Labs). Graviton instances (c7g, m7g, r7g) provide up to 40% better price-performance and consume up to 60% less energy compared to equivalent x86 (Intel/AMD) instances for containerized and database workloads.

Q24: How does AWS Auto Scaling Group (ASG) handle instance termination during scale-in?

Answer: By default, ASG termination policies:

  1. Determine which AZ has the most instances and at least one non-protected instance.
  2. Select the instance with the oldest launch configuration or launch template.
  3. Select the instance closest to the next billing hour.
  4. Developers can customize this using Scale-In Protection or custom Lambda lifecycle hooks to drain container traffic before terminating.

Q25: How do you perform a FinOps audit to reduce AWS cloud spending by 30%?

Answer:

  1. Identify and terminate unattached EBS volumes and outdated EBS snapshots.
  2. Analyze NAT Gateway usage and deploy free S3 Gateway Endpoints.
  3. Upgrade all gp2 volumes to gp3 (immediate 20% cost reduction).
  4. Run AWS Compute Optimizer to right-size over-provisioned EC2 and Lambda resources.
  5. Purchase 1-year or 3-year Compute Savings Plans for predictable baseline workloads.
  6. Migrate stateless batch workers to Spot Instances.

7.4 The Ultimate AWS CLI v2 Power Cheat Sheet

Identity & Authentication

# Verify current active IAM caller identity
aws sts get-caller-identity

# Assume cross-account role and capture credentials
aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/Deployer" --role-session-name "CLI-Session"

Compute & Networking Inspection

# List all running EC2 instances with private and public IPs
aws ec2 describe-instances \
  --filters "Name=instance-state-name,Values=running" \
  --query "Reservations[*].Instances[*].[InstanceId,PrivateIpAddress,PublicIpAddress,Tags[?Key=='Name'].Value|[0]]" \
  --output table

# Inspect all VPCs in current region
aws ec2 describe-vpcs --query "Vpcs[*].[VpcId,CidrBlock,IsDefault]" --output table

Storage & Serverless Operations

# High-speed parallel S3 sync
aws s3 sync ./dist s3://my-production-bucket --delete

# Invoke Lambda function synchronously and output payload
aws lambda invoke --function-name MyService --payload '{"action":"ping"}' response.json && cat response.json


7.5 Enterprise FinOps: The 8-Step AWS Cloud Cost Reduction Playbook

Phase Resource Category Action Item Expected Monthly Savings
1 EBS Volumes Delete unattached available EBS volumes and obsolete manual snapshots. 5% – 10%
2 Block Storage Migrate all legacy gp2 volumes to modern gp3 (provisioned IOPS decoupled). 20% flat reduction
3 VPC Gateways Deploy free S3 & DynamoDB Gateway Endpoints; eliminate NAT Gateway data processing. $200 – $2,000 / mo
4 Compute Purchase 1-Year or 3-Year No-Upfront Compute Savings Plans for baseline usage. 28% – 66% discount
5 Architecture Switch x86 EC2/ECS/Lambda instances to AWS Graviton3 (ARM64). 40% price-performance
6 Object Storage Enable S3 Intelligent-Tiering with archive status rules. 30% – 60% of S3 bill
7 Environments Automatically terminate non-production dev/staging instances outside business hours. 65% non-prod compute
8 Databases Right-size over-provisioned RDS instances using AWS Compute Optimizer insights. 15% – 35%

🤝 Community & Contributing

Contributions are welcome! Please review our CONTRIBUTING.md and CODE_OF_CONDUCT.md guidelines before opening issues or submitting pull requests.

📄 License

This project is open-source software licensed under the MIT License.

About

Complete guide to learn aws.

Resources

Code of conduct

Contributing

Stars

0 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages