When you build agents for a single user on a laptop, almost anything works.
When you build commercial multi-tenant agents serving enterprises, almost nothing accidental survives.

Since late 2024, I’ve spent most of my time building and optimizing commercial, multi-tenant agent systems with a team of engineers inside a large SaaS platform serving hundreds of enterprise customers. We stress-tested agents under real constraints: tenant isolation, financial accuracy, auditability, cost control, and scale.

When I wasn’t working on those systems, I was consulting with other teams, collaborating with peers building agent platforms, and running independent experiments and side projects to pressure-test orchestration models, retrieval architectures, evaluation harnesses, and model upgrades under similarly real-world conditions.

Different environments. Same constraints.

We evaluated and implemented systems across direct foundation model orchestration, managed agent runtimes, multi-agent coordination patterns, graph-oriented orchestration, and MCP integrations.

In parallel, we architected a multi-tenant semantic layer over economically material enterprise cost data. Not RAG over documents, but a deterministic parsing and interpretation engine performing canonical identity modeling, entity resolution, ontology alignment, and context-aware re-ranking with full provenance. Underneath it: hybrid search, columnar analytics, vector stores, and AI-native data models.

We built replay-based regression harnesses. We instrumented token cost and execution traces per run. We fine-tuned and distilled local models for latency and cost optimization.

We heavily utilized production agentic coding systems like Claude Code and Cursor, studying how they handle production-grade context management: pruning between turns, isolating workspaces, externalizing heavy operations, aggressively controlling context surfaces. The same patterns we were engineering, they were hardening in the wild.

Across all of this, a clear convergence emerged.

Models improve. APIs standardize. Tool use matures.

But in commercial multi-tenant systems, those are not the determining factors.

Context is.

What follows is not theory. It emerged from debugging memory drift across tenants. From tightening isolation guarantees that survive scale. From building promotion gates to prevent memory poisoning. From implementing retention under enterprise compliance. From learning that token economics only become predictable when context is disciplined.

This is not just one team’s experience. The same architectural pressures are visible across production systems like Claude Code, Cursor, Letta, AWS AgentCore, and others. Implementations differ. The convergence is real.

The systems that survive production pressure consistently share the same traits:

• Typed memory instead of transcript blobs
• Separation between canonical truth and derived acceleration layers
• Explicit promotion and compaction gates
• Trace envelopes for replay and audit
• Aggressive pruning between turns
• Isolation boundaries enforced as security boundaries
• Cost surfaces treated as first-class signals

The systems that fail optimize for demo velocity. They index raw transcripts. They reuse context wholesale. They blur truth and acceleration. They discover cost, drift, and cross-tenant risk too late.

This guide synthesizes:

• Lived experience building commercial multi-tenant agents
• Public architectural signals from mature agent systems
• Explicit engineering rules for context discipline

In commercial multi-tenant systems, context is not an implementation detail.

It is infrastructure.

And infrastructure requires engineering discipline.

Part I: Context Is Infrastructure

The Three Non-Negotiables of Commercial Agent Systems

1. Structural Isolation
2. Deterministic Replay
3. Economic predictability

These are not optional features. They are architectural constraints.

If your system cannot enforce tenant boundaries structurally, it will eventually leak.
If your system cannot run deterministic replays, you cannot debug or evolve it safely.
If your system cannot predict cost per run, it cannot scale sustainably.

Models Are Commoditized. Context Is Not.

Everyone has access to the same frontier models. Claude. GPT. Gemini. The APIs are public. The prices are falling. Capabilities are converging.

What differentiates systems is no longer the model.

It's the context.

Two teams using the same model can produce radically different outcomes depending on how they externalize knowledge and constrain behavior.

That's true for coding assistants. It's even more true for commercial agent systems.

Because in commercial systems, context isn't just about output quality. It's about:

• Policy compliance: did the agent follow the rules?
• Data isolation: can Tenant A's data leak into Tenant B's context?
• Cross-tenant safety: does shared infrastructure introduce shared risk?
• Cost control: can you predict and bound what each run costs?
• Replay and auditability: can you reconstruct what the agent knew at decision time?
• Correctness under ambiguity: does the system degrade gracefully or silently?

If you get context wrong, you don't just get worse answers. You get silent corruption.

Decisions as First-Class Records

There's a shift happening beneath the surface that most teams haven't fully internalized.

Enterprise software historically captured objects: customers, invoices, tickets, accounts. Systems of record persisted state.

Agent systems introduce something new: decisions.

• Why was this exception approved?
• Which policy version applied?
• What precedent influenced this action?
• What context was visible at decision time?

We're not fully at “context graphs” yet. But we can't get there unless we build the foundations now:

• Append-only trace logs
• Scoped memory promotion
• Provenance tracking
• Replayable context assembly

Context management isn't just about controlling what the model sees. It's about observing and recording how context influences action.

Without traceability, you cannot optimize.
Without replay, you cannot debug.
Without provenance, you cannot trust durable memory.

The foundation for future “context graphs” is not speculation. It's disciplined trace capture starting today.

Part II: Memory as a Scoped, Typed System

Memory Must Be Scoped and Typed

Before we talk about storage or retrieval, we need a shared vocabulary.

The word “memory” is overloaded. So is “context.”

If you don’t explicitly define both scope and type, you end up with a single undifferentiated blob store. And that’s where security and correctness failures begin.

A robust commercial agent system classifies memory along two dimensions:

1. Scope: who can see it (a security boundary)
2. Type: what kind of memory it is (semantic role)

In practice, every production system we evaluated converged on some form of this separation.

This is not academic modeling. It is operational survival.

Memory Scopes (Security Boundaries)

These are structural isolation layers. They are enforced at the storage and routing layers, never delegated to the model.

Global Scope: Platform-Wide, Tenant-Invariant Memory

• System safety rules
• Tool contracts
• Product ontology
• Agent “constitution”

Properties:

• Immutable at runtime
• Versioned
• Deployment-controlled
• Never writable by agents

• Organization policies
• Knowledge bases
• Playbooks
• Connector configurations

Properties:

• Shared across users in a tenant
• Policy-gated promotion
• Subject to tenant retention rules

This is where governance lives. It is also where poisoning risk becomes systemic if not handled correctly.

User Scope: Personalized Memory Within a Tenant

• Preferences
• Working style
• Personal notes
• User-specific entitlements

Properties:

• Visible only to the user (and system)
• Promotion-gated
• TTL or policy-based retention

Cursor’s user-local memory posture reflects this discipline. User state stays user-scoped by default.

• Tool outputs
• Intermediate plans
• Scratch buffers
• Temporary retrieval results

Properties:

• Short-lived
• Subject to aggressive garbage collection
• Not durable unless explicitly promoted

Memory Type (Semantic Role)

Memory types cut across scopes. Scope defines who can see it. Type defines what the memory represents and how it is expected to behave.

• Policy memory
  Normative rules and constraints.
  Typically global or tenant-scoped.
  Versioned and tightly controlled.

• Preference memory
  Stable personalization parameters.
  Usually user-scoped.

• Fact memory
  Durable assertions the agent may reuse.
  Must include provenance.

• Episodic memory
  Structured summaries of completed work.
  “Case resolved.”
  “Migration completed.”
  “Exception granted.”
  Reusable artifacts extracted from traces.

• Trace memory
  Raw, append-only execution events.
  This is your flight recorder.

The most common failure mode in early systems is allowing episodic or fact memory to silently drift into policy memory.

That is how precedent poisoning begins.

Memory Layer Summary:

Note how:

• Session memory is volatile
• User memory is semi-durable
• Tenant memory is high-stakes
• Global memory is immutable

Each layer carries different isolation risk and promotion risk.

Why Scoped + Typed Memory Changes Everything

Without scope boundaries:

• Cross-tenant contamination becomes possible
• Retrieval filters become advisory
• Privacy guarantees degrade

Without type boundaries:

• Precedents become directives
• Facts become policies
• Session artifacts become durable memory

Scoped + typed memory is the minimum viable structure for safe autonomy.

It allows:

• Isolation enforcement
• Promotion gating
• Retention control
• Cost modeling by layer
• Evaluation at the run level

And most importantly: it prevents a single undifferentiated memory surface from quietly becoming a liability.

Part III: Truth vs Acceleration

Once you define a memory taxonomy, the next mistake most systems make is collapsing storage into a single layer.

Everything goes into:

• A vector database
• A document store
• A transcript log
• Or worse, a hybrid of all three

That works for prototypes.

It does not work for commercial, multi-tenant agent systems.

The core distinction you must preserve is this:

Separate truth from acceleration.

In practice, that means designing two canonical stores and two derived stores.

Canonical Stores (Truth)

Canonical stores serve as the system of record. They hold durable, immutable facts from which state can be deterministically derived.

They must support:

• Auditability
• Replay
• Version awareness
• Deterministic reconstruction

Their full state must be reconstructible from their own persisted history, not dependent on secondary indexes, caches, or materialized views.

1. Canonical Event Log (Append-Only)

This is your flight recorder.

Every agent run emits multiple events that include:

• Context retrieved
• Policies evaluated
• Tool calls made
• Approvals routed
• Outputs generated
• Memory promoted

This log is:

• Append-only
• Immutable
• Replayable
• Version-aware

It allows you to answer:

• What did the agent know at decision time?
• Which policy version applied?
• Why was this exception granted?
• What was retrieved and why?

Without this log:

• You cannot debug autonomy.
• You cannot build evaluation loops.
• You cannot build future context graphs.

Example agent event:

This is not logging. It is infrastructure.

2. Canonical Structured Memory Store

This stores durable memory state.

Unlike the event log, this is not raw trace data. It stores structured artifacts:

• Facts
• Preferences
• Episodic summaries
• Approved overrides
• Tenant-level knowledge

Every record must include:

• Scope
• Class
• Provenance
• Retention policy
• Sensitivity classification

Crucially: This store, not the vector index, is truth.

Example memory record:

If your vector store becomes your truth layer, you will eventually:

• Lose provenance
• Lose replayability
• Break deletion guarantees
• Create retention drift

Derived Stores (Projections)

Derived stores exist for performance.

They are:

• Rebuildable
• Ephemeral
• Invalidatable
• Non-authoritative

They are accelerators, not truth.

The retrieval index is your serving layer for recall.

It may include:

• Embeddings
• Lexical search (BM25 or equivalent)
• Hybrid ranking
• Freshness boosts
• Scope filters
• Metadata constraints

But it must be rebuildable from canonical sources.

It is a projection.

If your vector store becomes your truth layer, you will eventually lose structural integrity.

2. Object Store (Large Payloads)

Agent systems frequently deal with:

• Large documents
• Attachments
• Extraction outputs
• Tool responses
• External system dumps

These do not belong in structured memory. They belong in an object store, referenced by ID, tagged with scope and sensitivity, and governed by retention policy.

Objects should be content-addressed (or at minimum content-hashed) so they can be verified, deduplicated, and traced back to immutable source bytes. Derived artifacts such as embeddings, chunks, summaries, and classifications should be stored separately and keyed by (object_id, content_digest, model/version). They are projections for retrieval and acceleration, not canonical truth.

Summaries are especially useful for faster retrieval and context compression, but they must remain reproducible and auditable. A summary should always reference:

• the source object_id
• the source content digest
• the model and prompt/version used to generate it
• creation timestamp and optional verification status

The event log and structured memory should reference these objects, never embed large payloads directly.

If using managed ingestion systems such as Amazon Bedrock Knowledge Bases, treat them as synchronization and chunking layers that build and refresh retrieval indexes from object storage. They orchestrate ingestion and pruning, but they do not replace the underlying search engine or the need for canonical content verification.

Hybrid search (lexical + semantic) provides stronger precision and filtering guarantees than vector-only retrieval when correctness and isolation matter.

Lexical search preserves deterministic filtering.
Semantic search improves recall.

Combined ranking reduces false matches and helps minimize hallucinations.

Engines such as OpenSearch, Typesense, and Pinecone natively support hybrid retrieval, combining keyword relevance (BM25-style scoring) with vector similarity to balance precision and semantic recall.

Amazon Bedrock Knowledge Bases is not a search engine itself, but instead a managed ingestion and synchronization layer that builds and refreshes retrieval indexes (typically backed by a hybrid store) from documents stored in S3. It helps prune, chunk, and rebuild indexes, on your behalf.

Critically, the index, regardless of engine, must be built from:

• Canonical structured memory
• Curated documents
• Approved episodes

It must not be built directly from raw transcripts.

Raw transcripts are noisy, redundant, and context-fragmented. Indexing them directly undermines traceability and weakens retrieval discipline.

Isolation at the Projection Layer

Isolation is not a retrieval tuning feature.

It is a system invariant.

But the retrieval index is where projection-layer drift most commonly appears.

Filtering must occur before ranking, not after.

Every retrieval query must include:

• Tenant ID
• Scope visibility constraint
• Expiration checks
• Sensitivity boundaries

If filtering is applied after ranking, cross-tenant artifacts may still influence embedding neighborhoods.

Partition semantics must match canonical storage.

If canonical storage is tenant-partitioned but the retrieval index is globally indexed with soft filters, isolation becomes advisory.

Content Proofs and Cross-Tenant Isolation

In shared or multi-tenant retrieval systems, consider cryptographic “content proofs” to prevent cross-copy leakage.

Cursor, for example, uses Merkle-tree-based content proofs during shared index onboarding to ensure results are returned only if the requester can prove legitimate possession.

This pattern can be applied at the object-store level:

• Maintain tenant-scoped manifests
• Maintain Merkle roots over authorized (object_id, digest) pairs
• Enforce verifiable inclusion boundaries

This reinforces isolation at the projection layer.

Cryptography does not replace logical isolation. It reinforces it.

A Quick Mental Model

Think of it this way:

• The event log is your immutable journal.
• The structured memory store is your state.
• The retrieval index is your materialized view.
• The object store is your blob layer.

If you’ve worked with event sourcing, this should feel familiar (with less determinism).

If you haven’t, the rule is simple:
If you can’t rebuild it from canonical truth, it shouldn’t be trusted.

Why This Separation Matters

This architecture gives you:

• Replayability
• Deletion guarantees
• Poisoning containment
• Cross-tenant isolation clarity
• Retention enforcement
• Cost control
• Index rebuild capability

And most importantly:
It prevents your retrieval layer from silently becoming your system of record.

Part IV: The Context Engine Loop

Commercial agent systems don't fail because they lack storage.

They fail because they lack discipline at runtime.

Context is not something you “load.”
It is something you assemble, constrain, compact, and sometimes discard.

Most systems accumulate context. Production systems reconstruct it.

That discipline lives in the context engine loop.

The High-Level Loop

Every agent run should follow a predictable sequence:

1. Ingest: establish identity, scope, constraints, and privacy mode
2. Plan context needs: determine what information is required to act safely
3. Retrieve: execute hybrid search within allowed scopes
4. Assemble working set: layer context by priority and token budget
5. Semantic stabilization: normalize references, extract structure, preserve meaning before reduction
6. Agentic garbage collection: deduplicate, prune low-confidence artifacts, enforce working-set limits
7. Infer and act: model + tools + policy enforcement + optional human approval
8. Promotion gate: decide what becomes durable memory
9. Emit trace envelope: record retrievals, actions, policies, versions, and cost surfaces
10. Lifecycle garbage collection: expire session buffers, enforce retention, invalidate derived projections

This loop is not optional.

If you skip steps, you get drift.

Step 1: Ingest

At the beginning of a run, you must establish:

• Tenant identity
• User identity
• Role and entitlements
• Privacy mode
• Sensitivity level
• Task type

Isolation begins here.

Retrieval filters are built before retrieval runs.

You DO NOT ask the model to filter data.
You filter in the data plane.

If identity and scope are ambiguous at ingestion, everything downstream becomes probabilistic.

Step 2: Plan Context Needs

Before retrieving anything, the agent should plan what kind of context it needs.

Does this task require:

• Tenant policy?
• Prior episodes?
• User preferences?
• External knowledge?

This prevents the common anti-pattern:
“Retrieve everything and let the model figure it out.”

In production, this anti-pattern shows up as:

• Gradually increasing token costs
• Slowly degrading precision
• Retrieval surfaces expanding
• Embedding neighborhoods densifying
• Prompt budgets creeping upward

No single run looks catastrophic.
Over weeks, additive context and recursive indexing begin influencing outcomes in subtle, hard-to-debug ways.

Planning reduces both risk and cost.
It's your first form of budget control.

Step 3: Retrieve (Isolation Enforced Here)

Retrieval must respect:

• Scope
• Visibility
• Sensitivity
• Retention
• Privacy mode

Filtering happens before ranking, not after.

Hybrid search (lexical + semantic) provides:

• Deterministic filtering
• Precision guarantees
• Improved recall

But retrieval is still a projection.

The canonical store remains the source of truth.

Step 4: Assemble the Working Set

The working set is the ephemeral context that actually enters the model’s window.

It is layered:

1. Global constitution
2. Tenant policies
3. User preferences
4. Retrieved facts and episodes
5. Session state

Each layer has:

• Priority
• Token budget
• Truncation rules

Without layering and budgets, context windows become dumping grounds.

Step 5: Semantic Stabilization (Pre-Compaction Flush)

Before you shrink context, you MUST stabilize meaning.

Compaction without stabilization risks deleting something the model was implicitly relying on.

Semantic stabilization answers:
What must be transformed or anchored before we enforce token limits?

This step may include:

• Collapsing verbose tool traces into structured summaries
• Extracting typed episodic artifacts from conversation fragments
• Converting free-form dialogue into structured facts
• Normalizing references (“that refund we discussed”) into concrete IDs
• Marking low-confidence artifacts explicitly
• Ensuring provenance metadata is attached

This is not deletion.

It is transformation before deletion.

Without this step:

• Summarization can distort intent
• Compaction can silently remove guardrails
• Session references can become ambiguous
• Replay fidelity can degrade

Semantic stabilization preserves reasoning integrity before footprint reduction.

Step 6: Agentic Garbage Collection (Working-Set Compaction)

After meaning is stabilized, the system can safely optimize.

Agentic garbage collection happens before inference.

It enforces:

• Token budgets by layer
• Deduplication of redundant artifacts
• Dropping stale session state
• Removing low-confidence provisional memory
• Enforcing maximum working-set size

Example:

Agentic GC protects:

• Guardrail visibility
• Cost predictability
• Ambiguity control
• Drift containment

It ensures that:

• Global constitution cannot be crowded out
• Tenant policy remains visible
• Session chatter does not displace structural constraints

Uncompressed history turns directly into cost.

Agentic garbage collection is not just optimization.

It is drift control.

Garbage Collection by Memory Layer:

Session state is cheap and volatile.

Tenant memory is high-stakes and must be protected accordingly.

Step 7: Infer and Act

Only after:

• Context is stabilized
• Working set is compacted
• Budgets are enforced

does inference occur.

This is where:

• The model runs
• Tools are invoked
• Policies are evaluated
• Approvals are requested if needed

This is the only step most tutorials focus on.

Model invocation + tools + policy evaluation + approvals.

In commercial systems:

• Actions must be policy-evaluated
• High-risk actions may require human approval
• Tool outputs must be sensitivity-tagged
• Outputs must be traced

Tool invocations should include:

• Versioned tool contracts
• Input digests
• Output digests

External systems evolve.

Without capturing tool version and payload hash, replay fidelity degrades over time.

The model is a component.

The system is the product.

Step 8: Promotion Gate

Promotion transitions session memory into durable memory.

This is the highest-risk operation in the system.

It deserves its own section, which we will fully expand in Part VI.

Step 9: Emit Trace Envelope

Disciplined trace capture requires a canonical shape.

Every run produces a single append-only trace envelope.

The envelope is a run-scoped materialization derived from the append-only event log. It does not introduce new facts. It snapshots derived aggregates so replay, audit, and cost analysis do not require reconstructing runs from raw events.

Cost views, lineage trees, evaluation harnesses, and audit dashboards are projections derived from this record. They do not redefine it.

Events are self-describing for partitioning and queryability, but the trace envelope is the authoritative run-level header. The event log is keyed by run_id.

At minimum, a canonical trace record must anchor:

• Identity (tenant, user, privacy mode)
• Model and policy versions
• Prefix/version hash
• Retrieval artifact IDs
• Tool contract versions
• Promotion decisions
• Token usage and cost
• Lineage (parent_run_id)
• Immutable event history

A minimal representation might look like this:

This record is append-only.
It is version-aware.
It is sufficient to replay the decision.

Everything else is projection.

Without trace envelopes, context engineering becomes guesswork.

Step 10: Lifecycle Garbage Collection (Durability & Retention Discipline)

After the run:

• Expire session buffers
• Invalidate derived indexes if needed
• Apply TTL to memory
• Archive large payloads
• Enforce retention policies

Memory is not just created. It must decay.

Run Boundary Events

Beyond the events emitted within the agent loop, two boundary events define the run itself.

run_started pins the execution boundary.
It captures the immutable configuration for the run: policy version, prefix hash, privacy mode, primary model, and parent linkage. From this point forward, the run operates inside that fixed context.

run_finalized closes the lifecycle.
It records final status, token usage, cost attribution, promotion counts, and integrity hashes. After this event, the run is complete and immutable.

Together, these two events make the trace envelope fully reconstructible from the append-only event log. The envelope introduces no new facts. It materializes the boundary and aggregates for fast replay, audit, and cost analysis.

Multi-Turn Conversations Do Not Justify Persistent Windows

A common misconception:
“If this is a conversation, the entire prior context should remain in the window.”

⚠️ That is incorrect in commercial systems.

Multi-turn state should be reconstructed per turn from:

• Canonical structured memory
• Verified episodes
• Approved tenant policies
• Selective session summaries

Not from raw accumulated transcripts as the primary reconstruction mechanism.

Each turn should:

1. Emit a trace
2. Compact session artifacts
3. Promote only approved durable memory
4. Reassemble context fresh

Carrying forward full windows across turns:

• Increases token cost
• Increases drift
• Increases poisoning risk
• Obscures replayability

The acceptable pattern:

• Session memory is volatile
• Durable memory is reconstructed
• Context is assembled per turn

If context grows by accumulation rather than reassembly, you are building drift into the architecture.

The Discipline

This loop is the difference between:

A chatbot with a vector DB and a commercial agent system.

Most failures come from skipping:

• Planning
• Pre-compaction
• Promotion gating
• Trace emission

The loop enforces discipline.

And discipline turns context from an experiment into infrastructure.

Part V: Multi-Agent Context Boundaries

Commercial agent systems increasingly delegate work across multiple agents.

A parent agent spawns a subagent to research a topic, execute a tool chain, validate a result, or operate within a specialized domain. Multi-agent orchestration patterns such as fan-out, delegation, pipelines, and supervisory hierarchies are becoming standard.

The architectural challenge is not orchestration.

It is context discipline across agent boundaries.

Every principle established so far, scoped memory, truth vs acceleration, the context engine loop, applies within a single agent. Multi-agent systems multiply the surfaces where those principles must hold.

If context flows between agents without discipline, you get the same failures as undisciplined single-agent systems, but harder to debug because the causal chain crosses execution boundaries.

Context Inheritance vs Isolation

When a parent agent spawns a subagent, the first question is:
What context does the subagent receive?

There are two patterns:

1. Shared context: The subagent inherits the parent's full working set.
2. Isolated context with scoped input: The subagent receives a fresh context window with only the information the parent explicitly passes.

The first pattern is simple. It is also dangerous.

It carries the following risks:

• The subagent's token budget is consumed by the parent's context before it begins its own work.
• Irrelevant context from the parent pollutes the subagent's reasoning.
• Replay becomes ambiguous because you cannot isolate which agent's context influenced which decision.
• If the parent's context contains sensitive artifacts the subagent should not access, isolation is violated.

The second pattern, isolated context with scoped input, however, survives production pressure.

Claude Code's subagent model reflects this: subagents operate with fresh isolated contexts. The parent provides a scoped task description. The subagent executes independently. It returns a structured summary. The parent incorporates the summary into its own working set.

The isolation is deliberate:

• The subagent's token budget is its own.
• The parent controls what enters the subagent's window.
• The subagent's full internal trace stays in its own scope.
• Replay can reconstruct each agent's decision independently.

The cost of isolation is that the parent must decide what context the subagent needs. That decision is itself a context engineering problem, and it benefits from the same planning step described in the context engine loop.

If context inheritance is implicit, debugging multi-agent behavior requires reconstructing invisible state.

If context inheritance is explicit, each agent's behavior is independently replayable.

Subagent Outputs Are Promotion Events

When a subagent returns results to its parent, the parent incorporates that output into its working set.

This is a promotion event.

It deserves the same scrutiny as any other transition from ephemeral to durable state.

When the subagent's summary enters the parent's context it can influence:

• Subsequent tool invocations
• Policy evaluation
• Further delegation decisions
• Memory promotion at the end of the run

If the subagent's output is treated as trusted input without validation, the parent inherits whatever errors, hallucinations, or poisoning the subagent produced.

Defense:

• Subagent outputs should be typed: fact, episode, recommendation, tool result.
• Provenance should be tagged: which subagent, which run, which model version.
• Sensitivity classification should transfer: if the subagent accessed tenant-scoped data, the summary inherits that classification.
• The parent's promotion gate applies: subagent outputs should be treated the same way as any other artifact entering durable memory.

A useful mental model: treat subagent outputs like tool outputs.

They are data, not directives.

They carry provenance.

They are subject to the same validation rules as any other input to the working set.

Trace Lineage Across Agent Boundaries

In a single-agent system, the trace envelope captures one execution path.

In a multi-agent system, traces form a tree.

If Agent A delegates to Agent B, and Agent B delegates to Agent C, the trace must capture the full lineage:

• run_id for each agent's execution
• parent_run_id linking child to parent
• Delegation context: what was passed to the child
• Return summary: what came back
• Cost attribution per agent

Without lineage, you cannot:

• Replay a specific agent's execution in isolation
• Attribute cost to the agent that incurred it
• Debug which agent in the chain produced a problematic output
• Evaluate whether delegation decisions were correct

Trace lineage turns a multi-agent run from opaque delegation into a debuggable, replayable execution graph.

Without it, multi-agent systems become black boxes that happen to contain smaller black boxes.

Example trace structure (truncated for brevity):

Scope Inheritance Rules

When a parent agent delegates to a subagent, the subagent must operate within the correct scope boundaries.

Tenant scope and user scope must be inherited. If a subagent operates outside the parent's tenant boundary, isolation is violated. This is not optional.

Session scope is different. The subagent should have its own ephemeral session scope. It should not inherit the parent's session history, scratch buffers, or intermediate plans. Those belong to the parent's execution context.

Policy visibility must also propagate. If the parent operates under tenant policy version 3.2, the subagent must operate under the same version. Policy version drift across agents within a single run creates inconsistency that is extremely difficult to debug.

Summary of inheritance rules:

If privacy mode is active in the parent, it must be active in every subagent. Delegation cannot downgrade privacy guarantees.

If policy version differs between parent and child, the trace will show inconsistent evaluation and replay will not reproduce the behavior.

Cost Attribution Across Agents

Multi-agent runs compound every cost surface described in Part IX.

Each subagent incurs its own:

• Inference cost (its own context window, its own model invocation)
• Retrieval cost (its own queries against the retrieval index)
• Tool cost (its own external API calls)
• Persistence cost (if it promotes anything to durable memory)

Without per-agent cost attribution, optimization is impossible because you cannot see which agents are expensive.

Common failure mode:
A parent agent delegates to five subagents. Total run cost rises. The trace shows aggregate token counts. But it does not show that one subagent consumed 60% of the budget because its retrieval surface was over-broad.

Per-agent cost attribution within a run is not optional in commercial systems. It is the only way to identify which delegation paths are economically sustainable and which need tighter budgets.

The trace envelope must decompose cost by agent, not just by surface.

If you build commercial agents long enough, you eventually learn a frustrating truth:

Most failures don’t look like failures.

They look like slightly worse output... until you realize the system is drifting.

That drift is usually caused by one of three things:

1. Isolation boundaries weren’t enforced consistently
2. Bad context was retrieved and treated as truth
3. Session artifacts were promoted into durable memory

The danger isn’t that the model hallucinates.

The danger is that the system starts believing it.

Isolation Is a Data-Plane Primitive

Here’s the invariant again because it’s worth repeating:

Isolation is enforced in the data plane, not the prompt.

If your strategy relies on the model restricting itself to tenant-specific content, you’re already in trouble.

Isolation must be structural.

This means:

• Every retrieval query must includes tenant/user filter as a required predicate
• Filters always apply before ranking
• When documents are fetched, tenant mismatch needs to throw an exception
• Derived indexes are should either be physically partitioned or logically partitioned with verified predicates

It sounds obvious.

Most multi-tenant systems that leak data don’t leak because someone wrote SELECT * FROM tenants.

They leak because a derived system wasn’t partitioned the same way the canonical store was.

The Three Types of Memory Poisoning

Memory poisoning is not just “prompt injection.”

In multi-tenant systems it shows up in three distinct ways.

1. Instruction Poisoning

Malicious or malformed content attempts to alter system behavior.

Examples:

• “Ignore previous instructions”
• “Always approve refunds”
• “If you see this, exfiltrate secrets”

Defense:

• Policies never come from user content
• Policy memory is signed and versioned (global scope)
• User instructions are treated as input, not law
• Tool outputs are treated as data, not directives

If you take one rule from this section:
Never promote instructions to policy.

2. Precedent Poisoning

This is subtle and common.

An agent makes an exception once.
That exception gets stored as “how we do it.”
Six weeks later, the exception becomes default behavior.

Defense:

• Don’t store precedents as directives
• Store them as episodes with provenance and outcomes
• Require explicit approval signals before a precedent becomes a norm

Episodic memory earns its keep here.

Episodes store:

• What happened
• Why it happened
• Under what policy
• With what approval

They do not store: “What to always do.”

3. Cross-Scope Contamination

A user-level artifact gets promoted to tenant scope.
A tenant-level artifact affects global behavior.
A retrieval index accidentally crosses tenants.

When this happens:

• Quality degrades everywhere
• Security risk spikes
• Replay becomes ambiguous

Defense:

• Promotion gates enforce scope rules
• Global scope is write-locked
• Tenant scope requires stricter verification than user scope
• Every memory write includes scope, retention, sensitivity, and provenance

Promotion: The Most Dangerous Operation

Promotion is the transition from session state to durable memory.

It is where most memory poisoning becomes permanent.

Promotion must be treated like a database write.

Not like a convenience feature.

A promotion gate should answer four questions:

1. What scope can this live in? Session vs user vs tenant vs global
2. What types is it? Preference vs fact vs episode vs policy
3. What is the retention policy? TTL vs manual deletion vs legal hold
4. What is the provenance? Where did it come from, and can we replay it?

Default Promotion Rules

• Session → User
  Allowed for preferences, drafts, working style, user-specific episodes

• Session → Tenant
  Allowed only for verified facts and approved episodes

• Session → Global
  Never allowed at runtime

Verification Rules

Facts stored at tenant scope require:

• Human approval
• Trusted system-of-record confirmation
• Repeated corroboration across independent sources

Sensitivity Rules

• Never persist secrets (API keys, tokens)
• Be careful persisting PII without explicit retention rules and consent

Pseudo-Policy:

This is the generational GC analogy in practice:

• Session state is cheap and volatile
• User memory is moderately durable
• Tenant memory is high-stakes and requires verification
• Global memory is write-locked

Promotion discipline is not about paranoia.

It is about protecting invariants.

Part VII: Asynchronous Hardening and Memory Lifecycle

Promotion is the transition from volatile session state to durable canonical memory.

But durable memory should not be fully materialized synchronously.

In commercial systems, enrichment and hardening steps are frequently asynchronous.

These operations are:

• Computationally expensive
• Potentially slow
• Sometimes dependent on external systems
• Often unnecessary for immediate inference

The critical path should do only what is required to establish canonical truth.

Everything else belongs in a background processing pipeline.

Minimal Canonical Writes

The inference loop writes:

• A minimal canonical record
• With scope
• With type
• With provenance
• With retention policy
• With sensitivity classification

It does not block on:

• Embedding generation
• Cross-document deduplication
• Fact corroboration
• Conflict detection
• Episodic summarization
• Index rebuilds
• Retention reclassification

Those belong to hardening.

The Hardening Pipeline

Pattern:

This separation accomplishes four things:

1. Keeps latency predictable
2. Prevents enrichment failures from blocking inference
3. Preserves canonical truth even if derived layers fail
4. Enables cost amortization through batched AI inference

Design rule:
The inference loop writes minimal verified canonical records.
Everything else is projection.

Lifecycle States

Asynchronous hardening introduces a non-obvious reality:

Canonical truth may be persisted before it is fully trusted.

Treat newly promoted records as provisional until hardening completes. That means every promoted artifact carries an explicit status and can move through a small lifecycle:

• Provisional
  Persisted with provenance and scope
  Eligible for replay
  Not eligible for broad retrieval

• Active
  Validated and hardened
  Eligible for retrieval and reuse

• Quarantined
  Suspected poisoning
  Contradictions
  Failed checks
  Excluded from retrieval

• Revoked
  Explicitly superseded or deleted via tombstone with provenance

Hardening determines retrieval eligibility.

Inference should never block waiting for enrichment.

Failure Modes and the Canonical Contract

Asynchronous hardening introduces several failure modes.

They must degrade recall, not correctness.

1. Enrichment Failure

Embedding generation fails.
Summarizer times out.
Downstream index is unavailable.

Rule:

• Canonical record remains intact
• Status remains provisional
• Derived projections are retried

Correctness is preserved.

2. Contradiction Discovered

Corroboration fails.
Trusted system-of-record disagrees with agent-authored fact.

Rule:

• Mark artifact as quarantined
• Emit trace event
• Optionally write corrective memory with higher precedence

Never silently overwrite.

3. Duplication and Merge Races

Multiple runs promote semantically identical artifacts.

Deduplication merges incorrectly.

Rule:

• Use deterministic identity keys (content_digest + scope + type)
• Make merge operations idempotent
• Record merge decisions in the event log

Projection must not rewrite canonical history without trace.

4. Late Revocation

Tenant updates policy.
User revokes consent.
Compliance deletion arrives after embedding and indexing.

Rule:

• Tombstones are first-class
• Deletion triggers coordinated invalidation across every derived layer

Deletion without invalidation resurrects stale context.

5. Partial Projection

Canonical write succeeds.
Some derived indexes update.
Others do not.

Rule:

• Retrieval must be tolerant of missing projections and fall back to canonical fetch paths.

Projections should never be required for correctness.

Memory Conflict and Drift

Over time, multi-tenant agents accumulate contradictions.

Examples:

• Tenant policy changed but old memory persists
• Preference updated but stale entries remain
• Past episode contradicts current entitlement state

You need a strategy for conflict. The simplest workable approach:

1. Prefer newest memory with verified provenance
2. Prefer canonical systems of record over agent-authored facts
3. Mark memories with “confidence” and “verified” flags
4. Allow revocation and explicit tombstones (deletions with provenance)

Tombstones matter because:

• You need to prove deletion
• Stop retrieval from resurrecting stale artifacts
• Ensure derived indexes invalidate consistently

Hardening without tombstones creates retention drift.

Why Hardening Exists at All

Without asynchronous hardening:

• Latency becomes unpredictable
• Inference blocks on expensive enrichment
• Failures cascade into user-visible errors

Without hardening gates:

• Provisional artifacts influence retrieval prematurely
• Poisoned memory spreads
• Verification becomes retroactive instead of preventive

Hardening separates:

• Canonical truth persistence
• Retrieval eligibility
• Acceleration layer maintenance

This is the canonical contract:
Inference writes minimal truth.
Hardening validates and projects.
Derived layers accelerate.

Part VIII: Privacy, Retention, and Cryptographic Boundaries

Isolation protects tenants from each other.
Privacy protects tenants from you.

Privacy is often treated as a feature. In commercial agent systems, it must be treated as a data-plane architecture decision.

If your system promises "we don't retain this," "this run won't train models," "this tenant's data is isolated," or "this is ephemeral", then those guarantees must be visible in your routing, storage, and indexing layers. Not just in marketing copy.

If privacy depends on flags inside the model prompt or conditionals inside business logic, it will eventually fail.

The only durable privacy guarantee is structural separation.

Compliance requires encryption at rest.
Architecture requires partitioned routing.
Threat containment requires domain separation.

Privacy as Architectural Routing

Isolation is mandatory for multi-tenancy. It's a structural invariant.
If Tenant A can influence or see Tenant B’s data, the system is broken.

Privacy posture is different.

Privacy posture is a policy commitment enforced by architecture.
It defines how much the platform itself retains, observes, or learns from tenant activity.

A common anti-pattern:

This is extremely fragile.

Logging isn't the only place data persists. Data can leak into async queues, derived retrieval indexes, observability pipelines, caches, debug traces, analytics jobs, and model prompt caching layers.

If "privacy mode" relies on remembering to guard every one of those paths, it will eventually fail.

In production, the failure mode isn't malicious. It's incremental. A new logging layer is introduced. A new background job is added. A new cache is deployed. And privacy assumptions silently degrade.

Privacy modes should influence:

• Where traces are written
• Whether promotion is allowed
• Whether retrieval indexes are updated
• Whether embeddings are generated
• Whether objects are persisted

Privacy must route execution differently, not just mask output.

But the critical part is this:

The no-retention path should not share the same physical data plane as the retention path.

Separate buckets.
Separate partitions.
Separate encryption domains.

Otherwise accidental logging becomes inevitable.

The Stronger Pattern: Separate Data Planes

The more defensible pattern (validated by mature SaaS systems) is this: route privacy-mode traffic through a separate path.

That can mean separate replicas, separate storage backends, separate queues, suppressed or redirected observability streams, and distinct retention policies at the storage layer.

The point is structural separation. At minimum, enforce physical storage partitioning and routing-level isolation.

A privacy-aware system may include:

• Retained trace log
• Metadata-only trace log
• Durable structured memory store
• Ephemeral session store

In privacy mode:

• Trace logs may contain only metadata (run_id, timestamps, token counts)
• Canonical structured memory may disallow promotion entirely
• Object store writes may be disabled

This is not just policy enforcement.

It is architectural branching.

If your privacy guarantee requires scanning logs after the fact, you are already too late.

What Privacy Mode Should Actually Control

At minimum, privacy mode should govern:

1. Event log retention
   Are trace envelopes persisted?
   If yes, are payloads redacted?
   If no, is only metadata retained?

2. Structured memory promotion
   Are promotion gates disabled?
   Are only session-scope artifacts allowed?

3. Derived index writes
   Are embeddings created?
   Where are they stored?
   Are they scoped to ephemeral partitions?

4. Object store persistence
   Are large payloads retained?
   Encrypted with tenant-specific keys?
   Auto-expiring?

Retention Discipline

If you promise "no retention," define what that means precisely.

Does it mean no canonical trace stored? No structured memory writes? No embedding generation? No analytics logs? No prompt caching? No external model provider retention?

You cannot say "no retention" if the request is still embedded into a global vector index, flows into analytics dashboards, or is logged to a shared debugging stream.

A clean architectural approach:

• Metadata-only trace retention
  • run_id, cost, timing
  • no payload
• Ephemeral session store
  • in-memory or short TTL
• Derived indexes disabled
• Object store writes blocked or TTL-bound
• Separate observability stream with redaction

That way your promise is enforceable, not aspirational.

Without explicit retention semantics:

• Session artifacts become durable
• Durable artifacts never expire
• Index entries outlive source truth

Every canonical record must carry retention metadata:

• Retention policy ID
• Expiration timestamp or rule
• Sensitivity classification

Example retention policy:

Retention must cascade

When a canonical record expires:

• A tombstone is emitted
• Derived projections are invalidated
• Object store references are removed or reclassified

Retention without projection invalidation creates ghost memory.

Designing this after the fact is painful.
Designing it up front as an alternate routing path is tractable.

Privacy cannot be retrofitted.

Encryption and Tenant Keys

Isolation is a policy boundary.
Encryption is a cryptographic boundary.

In any commercial system, canonical stores and object stores should be encrypted at rest. In multi-tenant enterprise-grade systems, we need to go further:

• Tenant-scoped encryption keys
• Key rotation policies
• Envelope encryption for object payloads
• Separation of encryption domains between canonical and derived stores

Why this matters:

1. A storage misconfiguration should not automatically imply cross-tenant readability.
2. Deletion guarantees are stronger when encryption keys can be revoked.
3. “No retention” modes can be reinforced with short-lived encryption domains.
4. Object stores containing large payloads require stricter cryptographic boundaries than derived summaries.

Recommended pattern:

• Canonical store: tenant-scoped envelope encryption (per-tenant KEK, rotating DEKs)
• Object store: tenant-scoped encryption + content digests (integrity/provenance), with sensitivity-based storage rules
• Retrieval index: tenant-partitioned index; store encrypted IDs + embeddings for curated artifacts only
• Cache layers: tenant-partitioned, TTL-bound caches; encrypt with short-lived keys; never authoritative
• Event log: tenant-partitioned; metadata encrypted with platform-managed KMS for cross-tenant analytics, and payloads encrypted with tenant-scoped envelope encryption (or stored as tenant-encrypted object references)

This allows:

• Tenant-level cryptographic revocation
• Controlled key rotation
• Partitioned blast radius

Key design principle:
If derived systems are compromised, canonical truth and raw payloads should remain cryptographically isolated.

Encryption is not just compliance posture. It reinforces scope boundaries.

Privacy mode complicates decision traces. If you suppress traces entirely, you lose replay. If you persist traces indiscriminately, you violate retention promises.

Trace envelopes should include:

• Model version
• Policy version
• Retrieval artifact IDs
• Token counts
• Tool contract versions

But in privacy mode:

• Persist structural trace metadata
• Redact sensitive payloads
• Allow tenants to opt into extended trace retention
• Separate trace retention from model training retention

This preserves operability without over-collecting.

Replay may become limited in privacy mode. That is acceptable.

What is not acceptable is accidentally retaining sensitive content because logging was decoupled from privacy routing.

Cross-Tenant Isolation Reinforced by Cryptography

Scope enforcement in the data plane protects retrieval.
Encryption enforces isolation at rest.

Even if a bug bypasses retrieval filters:

• Encrypted tenant domains limit exposure
• Key management boundaries provide additional containment

Cryptography does not replace logical isolation. It reinforces it.

Part IX: Cost Surfaces and Token Economics

In traditional cloud systems, cost scales with compute, storage, and network. In agent systems, cost scales with tokens, retrieval volume, context assembly size, tool invocation frequency, and trace retention footprint.

The difference is subtle but profound:

Cost is no longer driven by infrastructure. It's driven by intent.

And intent flows through context.

In commercial agent systems, cost does not scale linearly with traffic.

It compounds with context.

The Four Primary Cost Surfaces

Per-run cost is not just tokens in and tokens out. In commercial systems it is a composite of four surfaces:

1. Inference
2. Retrieval
3. Tooling
4. Persistence

Each surface has different scaling behavior.

A disciplined system tracks these surfaces separately so optimization doesn’t become guesswork.

1. Inference Cost

Inference cost is driven by:

• Input tokens
• Output tokens
• Context window size
• Model selection
• Prompt prefix size (caching effects)

As context grows, inference cost grows even if the business logic remains unchanged.

This is the first place drift becomes visible.

2. Retrieval Cost

Retrieval cost scales with:

• Number of documents indexed
• Embedding dimensionality
• Query volume
• Hybrid ranking complexity
• Cross-tenant filtering overhead

As memory accumulates, retrieval cost rises, even if token budgets stay fixed.

Derived projections amplify cost.

3. Tooling Cost

Tool invocation cost includes:

• External API calls
• Database reads
• Connector queries
• Downstream system calls
• Internal compute
• Side effects (including retries and human approval latency)

Tools often dwarf model cost in enterprise environments.

Without trace attribution per run, tooling cost becomes opaque.

4. Persistence Cost

Persistence cost includes:

• Canonical storage
• Object store usage
• Derived index storage
• Log and trace retention
• Backup and compliance overhead

Durable memory is an economic commitment.

Promotion decisions multiply storage cost over time.

The Economic Model of an Agent Run

Every agent run has a composite cost. It is not just tokens in and tokens out.

It is the sum of the four distinct surfaces.

Formally:

Ctotal=Cinf+Cret+Ctool+CpersistC_{total} = C_{inf} + C_{ret} + C_{tool} + C_{persist}Ctotal​=Cinf​+Cret​+Ctool​+Cpersist​

Where:

• Inference scales with context size.
• Retrieval scales with memory growth.
• Tooling scales with autonomy.
• Persistence scales with promotion discipline.

Most teams focus on inference cost.

In commercial systems, that is usually not the dominant long-term driver.

Layer-Based Token Budgets

One of the most effective structural controls is explicit token budgeting by layer.

Example allocation:

• Global constitution: 500 tokens
• Tenant policy: 800 tokens
• User memory: 600 tokens
• Retrieved episodes/artifacts: 1,200 tokens
• Session state/scratch: 900 tokens

Total: 4,000 token input budget

Without allocation, session state will crowd out policy.
Without allocation, retrieval will crowd out global constraints.

Layer budgets:

• Preserve guardrails
• Bound token cost
• Reduce drift
• Enable cost prediction

Budget discipline is architectural, not cosmetic.

Without layer-specific budgets, tenant memory can crowd out safety rules, session noise can crowd out durable facts, and retrieval bloat can drown signal.

Example Token Budget:

Prompt Caching and Context Hashing

In commercial systems, many runs share identical static context layers: global constitution, tenant policy, stable user preferences.

These layers should be explicitly versioned and constructed deterministically.

A stable hash of the static prefix:

This enables:

• Observability of policy drift
• Deterministic replay
• Compatibility with provider-side prefix caching
• Reduction of redundant prefix construction

When supported by the model provider, identical prefixes may benefit from prompt caching at the infrastructure layer, reducing repeated token charges. Even without provider caching, explicit prefix hashing encourages disciplined versioning and makes constitutional changes visible and auditable.

This reduces inference cost volatility.

But only if:

• Prefix boundaries are stable
• Versioning is explicit
• Trace logs record prefix version

Otherwise cache invalidation becomes opaque.

What Token Drift Looks Like

Architectural discussions about token budgets stay abstract until you attach numbers.

Consider a representative workload:

• Early average input: ~1,800 tokens
• Output: ~400 tokens
• Total per run: ~2,200 tokens
• Approximate cost: ~$0.02–$0.03 per run

Now introduce three common forms of drift:

• Retrieval surface expands
• Session transcripts are retained instead of compacted
• Promotion frequency increases

Six weeks later, average input grows to ~2,900 tokens. Output remains stable.

Per-run cost rises to ~$0.035–$0.045.

That increase feels small in isolation. It is a 50–70% jump that rarely triggers alarms during development.

At 50,000 runs per day:

• $0.025 → $0.040 average
• ~$22,000 monthly delta

No new features shipped. No model changed.

Context simply accumulated.

Why Cost Compounds

Drift rarely starts in inference.

It starts in promotion.

• Every durable memory write increases the retrieval surface.
• A larger retrieval surface increases context payload size.
• Larger payloads increase inference cost.
• Higher inference cost creates pressure to promote summaries.
• Promotion increases durable memory.

The system compounds.

This is why token drift is rarely linear.

It is structural.

Promotion as an Economic Multiplier

Every promotion to durable memory:

• Increases future retrieval size
• Increases index cardinality
• Increases token pressure
• Increases retention cost

Promotion frequency directly influences long-term cost curve.

If 5% of runs promote tenant-scope artifacts at 50,000 runs/day, that is 2,500 new durable records daily.

Reducing tenant-level promotion from 5% to 1% materially slows both storage and token growth curves.

Promotion discipline is not just governance. It is cost control.

Progressive Disclosure vs Up-Front Retrieval

Early systems (like chatbots), generally used a simplistic approach:

1. Query retrieval layer broadly
2. Insert top-k results
3. Let the model sort it out

This often "works", but it is without discipline.

Not all context should or needs to be visible at once.

Progressive disclosure is the better pattern:

1. Retrieve minimal context
2. Attempt inference
3. If low confidence, retrieve additional context
4. Repeat

This avoids the “dump everything into the prompt” anti-pattern.

Progressive disclosure improves precision, reduces token usage, and stabilizes behavior.

Layer budgets, compaction, promotion limits, and retrieval discipline are not optimizations. They are economic control surfaces.

Progressive disclosure is both correctness control and cost control.

Observing Cost as a First-Class Signal

Cost is not a monthly bill. It is a per-run property of execution.

Beyond the obvious token charges, agent systems incur structural cost across multiple dimensions:

• Context assembly cost
• Retrieval indexing and reranking cost
• Promotion and durable writes
• Retention and storage footprint
• Tool execution cost
• Observability overhead

If these are not attributed per run, you cannot tune budgets, compare retrieval strategies, or detect runaway behavior early.

Every trace envelope should include:

• Model and prefix version
• Tokens in / tokens out
• Static prefix tokens vs dynamic context tokens
• Retrieval artifact count and bytes retrieved
• Rerank candidate count
• Tool invocations and retry count
• Promotion writes by scope
• Estimated cost (model + tools)
• Latency
• Hardening queue lag (if async memory is used)

Tracing is not logging.
It is cost instrumentation.

Example cost instrumentation view (derived from the trace envelope):

Context Is Now an Economic Surface

In commercial agent systems, infrastructure cost scales predictably.

Context cost compounds.

It compounds with:

• Retrieval volume
• Assembly size
• Promotion rates
• Compaction discipline
• Retention duration

Progressive disclosure turns these into controllable surfaces. Retrieve minimally. Expand only when needed.

This reduces token volume and stabilizes behavior without sacrificing accuracy.

Teams that engineer these surfaces control cost.
Teams that treat context as a byproduct discover it later.

Cost rarely grows with traffic alone.
It grows with unmanaged context.

Cost control is not model negotiation.
It is deliberate context engineering.

Part X: Evaluation as Context Discipline

Most teams think evaluation means scoring model outputs.

In commercial agent systems, evaluation means something far more structural:

You are evaluating context assembly under constraint.

Models are interchangeable components.
Context discipline is the system.

What You Are Actually Evaluating

A commercial agent run is the composition of:

• Identity resolution
• Scope filtering
• Retrieval selection
• Layer budgeting
• Semantic stabilization
• Garbage collection
• Tool invocation
• Policy enforcement
• Promotion gating
• Lifecycle transitions
• Cost surfaces

Evaluation must ask:

• Was the correct context retrieved?
• Were scope boundaries enforced?
• Were policies visible in the working set?
• Did promotion decisions follow rules?
• Did cost remain within expected bounds?
• Would this run behave the same way tomorrow?

If you cannot answer those questions deterministically, you are not evaluating autonomy. You are sampling outputs.

Deterministic Replay Is the Baseline

Evaluation begins with replay.

Every run must capture:

• Model version
• Policy version
• Prefix hash
• Retrieval artifact IDs
• Tool contract versions
• Promotion decisions
• Token counts
• Cost breakdown by surface

Replay fixes those variables and re-executes the run.

If output changes under identical inputs, you have drift.

Drift can originate from:

• Model upgrades
• Retrieval index mutation
• Promotion contamination
• Prefix instability
• Lifecycle state changes

Replay is not debugging.
Replay is architectural validation.

Without replay, optimization becomes guesswork.

Example replay validation view (derived from the trace envelope):

Replay without artifact IDs becomes probabilistic.

Replay without versioned prefix becomes inaccurate.

Replay without tool contract versions becomes misleading.

Replay is only deterministic if the system treats versioning as a first-class discipline.

Model Upgrades Are Context Stress Tests

Frontier models evolve.
Tokenization changes.
Tool calling behavior shifts.
Summarization becomes more aggressive.
Safety layers adjust.

A model upgrade is not just a capability change.
It is a context stress test.

Under a new model:

• Does retrieval selection change?
• Does semantic stabilization behave differently?
• Does promotion frequency increase?
• Do policies get crowded out?
• Does token usage shift by layer?

Evaluation harnesses must compare:

• Old model + fixed context
• New model + identical context

If behavior shifts materially, you have learned something about your context design.

Stronger models do not fix weak context discipline. They amplify it.

Promotion Drift Is an Evaluation Surface

Evaluation must track:

• Promotion rate by scope
• Promotion type distribution
• Provisional → active transitions
• Quarantine frequency
• Tombstone rate
• Memory growth curve

A rising tenant-scope promotion rate is not a feature win.

It is an economic and governance signal.

Promotion drift compounds:

• Retrieval cardinality
• Token pressure
• Storage cost
• Cross-scope contamination risk

Evaluation is not just output quality.
It is structural hygiene.

Cost as a First-Class Evaluation Metric

Every run should emit:

• Inference tokens
• Retrieval volume
• Tool invocation count
• Persistence writes
• Derived index updates

Cost should be evaluated against:

• Task type
• Layer budget allocation
• Promotion decision
• Model version
• Prefix hash

If cost shifts without intentional architectural change, context drift is occurring.

Evaluation must detect this early.

Confidence and Context Sufficiency

Evaluation is incomplete without measuring sufficiency.

When progressive disclosure is used, you should measure:

• How often additional retrieval was required
• Confidence thresholds triggering expansion
• Cost delta between minimal vs expanded context

If confidence thresholds drift upward over time, context may be degrading.

Evaluation is about understanding when the system needs more context and when it is over-consuming it.

From Runs to Decision Graphs

Once trace envelopes are disciplined, something more powerful emerges.

Every run captures:

• What was visible
• What influenced action
• Which policies applied
• What was written
• What cost was incurred

Over time, this becomes a structured corpus of decisions.

You can analyze:

• Policy application frequency
• Precedent influence
• Cost distribution by task type
• Drift across model versions
• Promotion patterns by scope

That corpus is the foundation of decision graphs.

Not theoretical graphs.

Replayable, auditable, cost-attributed decision networks.

But they only emerge if evaluation is embedded from the beginning.

Replay Is Not Just for Model Upgrades

Replay supports:

• Model upgrade validation
• Policy changes
• Retrieval tuning
• Prompt modifications
• Tool migration
• Cost optimization experiments

If your evaluation loop depends on synthetic examples, it will miss edge cases.

Production traces are your regression corpus.

Model Upgrade Replay

Model upgrades are the most obvious replay scenario.

Upgrade process:

1. Freeze a representative replay corpus from production traces
2. Re-run corpus with new model version
3. Compare:
   • Outputs
   • Policy adherence
   • Token usage
   • Tool invocation patterns
   • Cost

Differences are classified:

• Improvement
• Neutral
• Regression

Without replay, upgrade evaluation becomes anecdotal.
Without cost comparison, upgrade risk becomes financial.

Policy Change Evaluation

When tenant or global policy changes:

• Re-run affected traces
• Detect differences in approval decisions
• Validate no unintended escalation of privilege
• Validate no suppression of required actions

Policy becomes versioned infrastructure, not live mutation.

Policy drift without replay is invisible.

Retrieval Strategy Testing

Changes to:

• Embedding model
• Chunk size
• Hybrid ranking weights
• Scope filtering rules
• Compaction rules
• Progressive disclosure thresholds

should be replayed across prior runs.

Key metrics:

• Retrieval artifact count
• Token usage by layer
• Hallucination rate
• Tool invocation deltas

Retrieval changes affect both quality and cost.

Replay reveals both.

This turns retrieval tuning from intuition into measurement.

Shadow Mode

Replay supports offline evaluation. But mature systems also support live shadowing.

Pattern is straightforward:

1. Production executes normally and output remains authoritative.
2. In parallel, a shadow run uses:
   • A new model version
   • A modified retrieval strategy
   • New compaction rules
   • Etc.
3. Shadow outputs are logged but not surfaced to users
4. Differences are analyzed asynchronously.

This reduces risk during:

• Model migrations
• Context restructuring
• Index rebuilds
• Policy refactors

It also enables support for:

• Controlled rollout
• Drift detection
• Confidence building

Shadow mode turns architectural evolution into controlled iteration.

Regression Suites from Real Traffic

Synthetic test cases rarely capture:

• Long-tail edge cases
• Rare entitlement combinations
• Complex multi-step tool chains
• High-context runs

The strongest evaluation corpus is your own trace log:

• Sample representative runs across tenants and task types.
• Strip or redact sensitive payloads if required.
• Store structured replay fixtures.
• Version them alongside policy and model artifacts.

Your production history becomes your test suite.

Over time, you accumulate:

• Edge cases
• Near-failures
• Policy conflicts
• Retrieval anomalies
• Promotion mistakes

Those are more valuable than curated prompts.

Without real traces, evaluation remains shallow.

Part XI: Model Versioning and Upgrade Discipline

Model upgrades are not configuration changes.

They are behavioral migrations.

In commercial agent systems, a model upgrade can alter:

• Interpretation of policy language
• Tool invocation sequencing
• Confidence thresholds
• Hallucination patterns
• Token usage patterns
• Sensitivity to ambiguity
• Response verbosity
• Context utilization behavior

If model upgrades are treated casually, system behavior becomes unpredictable.

Model evolution must be controlled at the architectural level.

Version Everything That Influences Behavior

Model versioning alone is insufficient.

Behavior is a function of:

• Model name and version
• Temperature and inference parameters
• Static prefix hash
• Policy artifact versions
• Tool contract versions
• Retrieval configuration version
• Compaction strategy version

If any of them change without trace visibility, replay fidelity degrades and regression analysis becomes impossible.

Model versioning is part of the canonical contract.

This makes behavioral provenance explicit.

Without it, you cannot explain deltas.

The Upgrade Sequence

A disciplined upgrade follows this sequence:

1. Freeze a Replay Corpus

Select representative production traces that include:

• High-context runs
• Multi-step tool chains
• Policy-sensitive flows
• Edge-case approvals

Lock the artifact IDs and prefix versions. These runs become your regression suite.

2. Deterministic Replay

Run the corpus under:

• Old model
• New model

Hold all other variables constant.

Compare:

• Output correctness
• Policy adherence
• Token usage
• Tool invocation patterns
• Cost

3. Shadow Production

Deploy new model in shadow mode.

• Run in parallel
• Record outputs
• Do not affect user-visible behavior

Track:

• Output divergence
• Policy deviations
• Token deltas
• Latency shifts

Replay validates determinism. Shadow validates live distribution behavior.

4. Progressive Rollout

Gradually increase traffic percentage.

Monitor:

• Drift signals
• Cost per run
• Tool call volume
• Promotion rate

Roll back immediately if invariants break.

5. Rollback Readiness

Rollback must be:

• Instant
• Deterministic
• Stateless

If rollback requires schema migration, reindexing, or cache rebuilding, you have coupled layers incorrectly.

Architecture must remain stable while models evolve.

Versioning without rollback is theater.

Context Window Changes Are Architectural Events

Larger context windows tempt teams to relax discipline.

Common failure pattern:

• “The window is bigger now.”
• Retrieval breadth increases.
• Session history persists longer.
• Promotion discipline loosens.

Short term: outputs improve.
Medium term: token cost explodes.
Long term: drift becomes embedded.

A larger window does not eliminate:

• The need for compaction
• The need for scope boundaries
• The need for promotion gating
• The need for evaluation

It magnifies the consequences of ignoring them.

Window size is a constraint multiplier.

Budgets are architectural, not model-driven.

Separation of Concerns Across Layers

Model upgrades should not require:

• Rewriting canonical storage
• Reindexing entire memory store
• Altering encryption boundaries
• Modifying promotion semantics

If they do, the architecture is over-coupled.

Layer separation ensures:

• Canonical store remains stable
• Derived projections can be rebuilt
• Retrieval configuration can evolve independently
• Compaction strategy can be tuned without rewriting history

Models change. Architecture must remain durable.

If your system entangles model behavior with memory writes or policy evaluation, upgrades will feel dangerous.

If layers are cleanly defined, upgrades become controlled experiments.

Upgrade Without Replay Is Operational Risk

If you upgrade a model without replay:

• You cannot quantify regression
• You cannot quantify cost delta
• You cannot detect policy drift
• You cannot isolate behavior change to the model

You are trusting a black box to remain aligned.

In enterprise systems, that is unacceptable.

Architecture Must Outlive Models

Frontier models are constantly evolving.
Your architecture must survive for years.

Design rule:
Models are replaceable components.
Context engineering is the durable asset.

Part XII: From RAG to a Context Engine

Many teams start with RAG.

Retrieve. Append. Generate. Repeat.

For prototypes, this works. For commercial systems, it does not.

RAG solves recall. It does not solve isolation, replay, promotion discipline, retention enforcement, lifecycle management, cost predictability, or upgrade safety.

RAG is a retrieval pattern.

A context engine is an architectural system.

Why Basic RAG Is Not Enough

Basic RAG assumes:

• Retrieval is stateless
• Memory is external
• The model is the primary reasoning surface
• History can be appended safely

Commercial agent systems violate all of these assumptions.

They require:

• Long-lived memory
• Scoped isolation
• Cross-tenant guarantees
• Durable decisions
• Cost accounting
• Evaluation loops

If you build on naive RAG, retrieval becomes accidental truth, transcripts become memory, promotion becomes implicit, drift becomes structural, and cost becomes unpredictable.

RAG is a building block. It is not the architecture.

What a Commercial Context Engine Looks Like

A commercial context engine has:

• Typed memory, not blobs
• Scoped storage, not implicit visibility
• Canonical event log, not accumulated transcript fragments
• Structured memory store, not unbounded conversation residue
• Promotion gates, not automatic persistence
• Hardening lifecycle, not synchronous promotion shortcuts
• Lifecycle garbage collection, not window overflow
• Hybrid retrieval, not vector-only recall
• Token budgets by layer, not "fit what you can"
• Privacy as routing architecture, not configuration flags
• Trace envelopes, not black-box runs
• Encryption boundaries, not shared trust assumptions
• Structured trace envelopes, not free-form logs
• Deterministic replay, not probabilistic reconstruction
• Versioned artifacts, not silent drift

It assembles context deliberately. It reconstructs context per run. It promotes intentionally. It enforces isolation structurally. It measures cost explicitly. It evolves through replay.

Conclusion: Engineering Context

Ecosystem Convergence

Over the past year, a clear pattern has emerged. Teams building IDE agents, coding copilots, orchestration layers, and model APIs, often in isolation and under very different product pressures, have arrived at strikingly similar system designs.

Throughout this guide, we've cited specific architectural signals from these systems. The convergence is not anecdotal. It spans memory design, isolation enforcement, trace capture, retrieval strategy, cost accounting, and promotion discipline.

Where the Signals Come From

Constraint, Not Coincidence

Across these systems, the same invariants appear:

• Memory is layered, scoped, and explicitly governed
• Durable memory requires deliberate promotion
• Context is assembled per run, not carried forward as a growing window
• Derived layers are projections rebuilt from canonical truth
• Retrieval combines structured and semantic signals
• Tool use is typed and contract-bound
• Isolation is architectural, not optional
• Replay and traceability are mandatory
• Async processing moves heavy work off the critical path
• Cost and token budgets are explicit control surfaces

These systems were built independently. They landed on the same patterns because production agents force these constraints.

Context is no longer a prompt engineering trick. It is infrastructure.

You do not have to adopt any specific framework. But if you ignore these structural constraints, you will rediscover them through drift, cost escalation, replay failures, privacy incidents, promotion poisoning, and upgrade regressions.

Convergence is a signal. And one you should listen to.

The Architecture That Survives

A context engine exists to enforce three structural invariants:

1. Structural Isolation
   Tenant boundaries are enforced in the data plane, not implied in prompts. Scope, partitioning, encryption domains, and routing make cross-tenant contamination structurally impossible, not statistically unlikely.

   Isolation is not a guardrail.
   It is a boundary.

2. Deterministic Replay
   Every decision must be reconstructible.
   Versioned prefixes, artifact IDs, tool contracts, and trace envelopes turn autonomy from guesswork into debuggable infrastructure.

   If you cannot replay a run, you cannot trust it.
   If you cannot trust it, you cannot evolve it.

3. Economic Predictability
   Cost must be bounded by architecture.
   Layered token budgets, promotion discipline, constrained retrieval, and per-run cost attribution prevent context from compounding silently.

   If cost emerges from accumulation instead of design, scale becomes drift.

If any of these are optional, autonomy will eventually degrade.

Not suddenly. Not catastrophically. Gradually.

Guardrails fade. Costs creep. Memory drifts. Isolation weakens.

The systems that survive production pressure are NOT the ones with the best prompts.

They are the ones where context is engineered as infrastructure.

And infrastructure either holds, or it doesn’t.

A Note on Adoption

This guide describes the full architecture that production pressure eventually demands.

No team should attempt to implement all of it at once.

The patterns here are not a checklist. They are a reference architecture. Your system's constraints determine which layers matter first.

If you are not yet multi-tenant, isolation can be deferred. If you are not under compliance or audit pressure, the full hardening pipeline and lifecycle state machine can wait. If your agent runs are short-lived and stateless, promotion gating matters less than retrieval discipline. If cost is not yet a problem, trace envelopes and per-run attribution can follow later.

Start with the constraints your use case actually imposes:

• If you handle enterprise data across tenants, start with isolation and scoped memory.
• If you need to debug behavioral drift, start with trace envelopes and replay.
• If cost is compounding, start with layer budgets and promotion discipline.
• If you are building durable memory, start with truth vs acceleration separation.

The full architecture is the destination. Your roadmap determines the order of arrival.

The ecosystem is moving fast. Managed memory services, provider-side context management, turnkey trace infrastructure, and retrieval-as-a-service offerings are shipping regularly. Many of the layers described in this guide are increasingly buy-not-build decisions. The architectural principles remain the same regardless of whether you implement them yourself or adopt managed services that enforce them on your behalf. What matters is that the invariants hold, regardless of who builds the plumbing.

Build what your constraints require today. Adopt what the ecosystem provides tomorrow. But know where the architecture converges, so you are building toward it rather than away from it.

Final Principle

Context is no longer just what you pass to a model.

It is your safety boundary.
Your isolation boundary.
Your cost boundary.
Your audit trail.
And your competitive moat.

Frontier models are increasingly interchangeable. APIs converge. Capabilities normalize. Pricing compresses.

What does not commoditize is how context is assembled, constrained, promoted, retained, replayed, and priced. That discipline determines safety, reliability, latency, auditability, and margin. Two teams can call the same model and produce radically different systems depending on their memory architecture and context control surfaces.

In commercial agent systems, model choice is leverage.
Context engineering is the moat.

Context is not an implementation detail. It is infrastructure. Engineer it accordingly.

Hope this helps,

Tags: #multi-tenant#agents#compliance

For years, cloud architecture had a convenient property: cost followed infrastructure.

If you knew what you deployed and where it ran, you could reason about cost after the fact. Instances, volumes, and managed services mapped cleanly to something you could point at, measure, and assign responsibility for. Tagging, accounts, and environments worked because the system itself was stable.

AI breaks that assumption.

Not because it is inherently more expensive, but because cost no longer lives in infrastructure. It lives in behavior.

Token usage, prompt construction, model choice, tool selection, retry behavior, and agent recursion now dominate the bill. None of those map cleanly to a server, an account, or a resource tag. The unit of work has changed, and most architectures have not caught up yet.

This is why AI cost problems keep showing up in finance dashboards even though their causes live entirely in architecture.

The Unit of Cost Has Changed

Traditional cloud systems assume a stable mapping between work and infrastructure. AI dissolves that assumption.

Even highly abstracted systems like serverless still behave predictably once you understand the inputs. Traffic goes up, cost scales accordingly. Capacity planning may be complex, but the relationship is knowable.

AI systems scale with behavior.

Two identical user requests can produce radically different execution paths depending on context, model choice, prior memory, or an agent’s internal reasoning loop. In that world, asking “what resource caused this cost?” is the wrong question.

The right question is “what decision caused this work to happen?”

Model choice is part of that decision stack. Token economics now matter just as much as latency or throughput. Pairing large, general-purpose models with expansive or recursive context can quietly dominate cost. Choosing the right model for the right workload is no longer an optimization detail. It is an architectural control surface.

That shift, from resources to decisions, is the root of the cost attribution problem many organizations are experiencing right now. From the outside, it looks like a financial issue. From the inside, it is an execution model mismatch.

Context Is the New Workload

Every AI system runs on context.

Prompts, memory, retrieved documents, intermediate reasoning steps, tool outputs, and prior conversations all get bundled together into a single execution surface. That bundle is the real workload.

Yet most systems still treat context as an implementation detail. Something to stuff into a prompt and forget about once the response comes back. That worked when prompts were small and models were cheap. It does not work anymore.

Context is not glue.
It is not metadata.
And it is not free.

Context is doing the work.

If you do not model it explicitly, you cannot control it. If you cannot control it, you cannot reason about its cost. And if you cannot reason about its cost, you cannot trust the system at scale.

Why Context Compounds Cost

Context does not scale linearly, and that is where things get dangerous.

Every additional token does more than increase input cost. It expands the reasoning surface, increases the likelihood of tool calls, alters branching behavior, and often lengthens outputs in ways that are hard to predict.

This is why “just add more context” is such a tempting instinct. It usually improves quality at first. But it also reshapes the entire execution path in subtle ways that compound downstream work.

What looks like a harmless prompt change can become a permanent multiplier.

From the builder’s perspective, this is an architectural concern. From finance’s perspective, it shows up later as unexplained variance. The decision, however, already happened in code.

Prompt Construction Is an Architectural Lever

Prompt design used to be about correctness and tone. Now it is about restraint.

What you include, what you omit, what you summarize, and what you reuse determines how much work the system is allowed to do on your behalf. A verbose system prompt is not neutral. Replaying an entire conversation history is not free. An agent that double-checks itself “just to be safe” is choosing more execution by default.

Every prompt is an architectural lever.

Cost shows up later, in margins and forecasts, but the leverage point lives squarely with builders.

Non-Determinism Is the New Cost Multiplier

Once cost is driven by decisions and context, predictability breaks down.

This is where traditional forecasting, alerting, and intuition start to fail. Not because teams are careless, but because the system itself is unstable by design.

Two requests that look identical at the edge can behave very differently once they hit the model. One may resolve immediately. The other may branch, retry, call tools, expand context, or spawn sub-tasks.

There is no stable unit cost per request.

From the outside, this looks like a forecasting problem. From the inside, it is an execution model problem.

Recursive Agents and Quiet Amplification

Agent systems introduce a failure mode that infrastructure teams are not used to reasoning about: recursive work.

Agents re-evaluate their own outputs, retry with more context, validate results by querying tools again, and sometimes spawn additional agents to help them think. Each individual step looks reasonable in isolation.

Taken together, they can create quiet amplification loops that never show up as a single obvious spike.

This is how margins erode without alarms ever firing.

Observability Stops Too Early

Most observability tooling answers operational questions. What ran. How long it took. Whether it failed.

AI systems require a different level of insight.

You need to know why something ran again. What context triggered a branch. What information was reused versus regenerated. That is not infrastructure tracing. It is intent tracing.

Without it, you are watching the meter spin without understanding what is turning it.

Agents Optimize for Correctness, Not Restraint

LLMs want to be helpful. Agents want to be right.

Left unconstrained, they will think longer, check more sources, use larger models, retry aggressively, and accumulate context defensively. That behavior is rational from the model’s perspective.

It is also financially dangerous.

Cost efficiency does not emerge naturally in agent systems. It has to be designed into them.

Context Must Be Designed, Not Assumed

If unpredictability is the problem, architecture is the solution. But only if context is treated as something that must be explicitly designed, constrained, and observed.

Context management is not an optimization. It is control.

Caching prevents duplicated reasoning.
Pruning prevents irrelevant amplification.
Summarization prevents historical bloat.
Progressive disclosure prevents front-loading cost “just in case.”

Without these mechanisms, you do not really have a system. You have an open loop that keeps spending until something external stops it.

Architecture Has to Change

Context needs to be treated like any other scarce resource.

That means explicit budgets, hard limits, lifecycle management, and eviction strategies. You do not let processes allocate infinite memory. You should not let agents allocate infinite context either.

Progressive disclosure becomes an economic pattern, not just a cognitive one. Instead of asking what an agent might need eventually, you ask what it needs right now and escalate only when necessary.

These are platform design decisions.

Guardrails Are Already Emerging

You can see this shift in modern agent systems, especially in large-scale coding agents. They do not simply run an LLM and hope for the best.

They isolate context windows.
They scope agent capabilities.
They separate planning from execution.
They constrain recursion depth.
They enforce runtime guardrails around permissions, behavior, and cost.

Not because it is elegant, but because unbounded agents are financially and operationally unsafe.

The best systems do not trust the model. They constrain it.

The Responsibility Shift, and What Comes Next

Once context becomes an architectural concern, ownership changes.

Cost control moves away from reporting and into design, where behavior is shaped before it ever reaches a bill. Finance can report outcomes, but they cannot prevent execution loops after the fact.

This is a systems design problem owned by architects, platform teams, and AI engineers who shape how context flows through the system.

Budgets, chargebacks, and dashboards still matter. But they operate downstream. They explain what happened. They do not influence what happens next.

Behavior is shaped at runtime.

The organizations that succeed in this next phase will not be the ones with the best spreadsheets or the most detailed monthly reports. They will be the ones that treat cost as a first-class runtime signal, something the system itself can respond to, adapt around, and constrain in real time.

This is why context discipline becomes a competitive advantage. Not because it lowers costs in the abstract, but because it enables faster iteration without fear. When you understand how context flows, when it grows, and when it is reused, you can push systems harder without worrying that a single change will quietly erase your margins.

Architects and platform teams now sit at the intersection of intelligence and economics.

The systems you design determine not just what AI can do, but how safely the business can let it run. That responsibility is new, and ignoring it isn't an option.

Context is no longer an implementation detail.
It is an architectural concern with real economic consequences.

Those questions made sense in 2014. They make less sense today.

Not because we've outgrown serverless. But because it quietly stopped being the point.

Serverless Was Never the Destination

Serverless was always a means to an end.

The goal wasn’t functions or managed databases or scale-to-zero pricing models. The goal was to stop spending our limited time thinking about infrastructure that didn’t meaningfully differentiate our products.

Early serverless didn’t fully deliver on that promise. It came with tradeoffs that were hard to ignore: cold starts, missing primitives, awkward workflows, and a long list of caveats that architects learned to recite by heart.

So we built compensating systems. We mixed in servers. We added provisioned databases, cache clusters, and NAT Gateways. We kept explaining, again and again, why serverless was good if you only understood its constraints.

That era is ending.

The Paper Cuts Are Healing

Over the last few years, and especially the last few months, the sharp edges have started to disappear. Not all at once. Not with a single announcement. But steadily.

Some of this progress is subtle, but it’s tangible. Lambda Managed Instances changed the at-scale execution model, significantly reducing the pressure to prematurely replatform or "graduate" from serverless. Durable Functions reduced the need for external orchestrators that bifurcate business logic. Aurora DSQL can now provision and scale in seconds, not minutes, removing one of the biggest friction points that previously pushed teams toward always-on, usually over-provisioned infrastructure. Even streaming, long-lived connections, and stateful patterns no longer feel like architectural outliers.

None of these changes are revolutionary on their own. But together, they’re transformative.

They don’t make serverless perfect. And none of this means the tradeoffs are gone.

Cold starts still exist. Stateful serverless is still fragmented and network-dependent. Debugging distributed, event-driven systems can be harder than debugging well-understood application stacks. And for many large or regulated organizations, the inertia of existing platforms (e.g. Kubernetes, VMs, long-lived services) hasn’t magically disappeared. In plenty of environments, serverless is still a conscious choice, debated and defended. That skepticism isn’t wrong. It reflects where many teams actually are today, not where the marketing slide says they should be.

But for many of us, these improvements make it reliable enough to fade into the background.

When the Default Disappears

Here’s the strange part: as serverless finally starts to deliver on its original promise, we’re talking about it less.

That’s not a failure. It’s a signal.

The most successful platforms fade into the background. We don’t debate TCP/IP. We rarely argue about HTTP servers. We don’t ask whether managed storage is “real infrastructure.”

Serverless is heading in the same direction. It’s becoming assumed by startups, greenfield projects, and cloud-native SaaS.

And once something is assumed, we stop naming it.

The Shift Up the Stack

At the same time, the center of gravity in software is moving fast. Really fast.

AI-driven systems, agentic workflows, real-time data pipelines, and adaptive products don’t always map cleanly to static infrastructure models. They demand flexibility. They demand experimentation. They demand the ability to change direction without rewriting the foundation every time.

That’s where serverless stops being a category and starts being table stakes.

Not because every workload runs on functions. But because the expectations around infrastructure have changed.

Provisioning speed matters.
Operational drag matters.
Reversibility matters.

Post-Serverless Doesn’t Mean “After”

A “post-serverless world” doesn’t mean serverless is over.

It means we’ve internalized its lessons.

We’ve learned that:

• Infrastructure should adapt to reality, not the other way around
• Paying for idle capacity is usually a smell
• Most systems benefit from starting small and growing into complexity
• The fastest teams are the ones with the fewest irreversible decisions

Those ideas don’t belong to a product category anymore. They belong to how modern systems are built.

What Comes Next

The next shift isn’t about which infrastructure you choose.
It’s about who (or what) does the choosing.

We’re already seeing the early signals. Adaptive infrastructure that responds to load and behavior. Self-provisioning runtimes that spin up just enough capability at the right moment. Infrastructure from Code models that describe intent instead of topology. Vercel recently announced their vision of self-driving infrastructure, systems that automatically optimize themselves based on observed behavior.

These aren’t isolated ideas. They’re all pointing in the same direction.

AI infrastructure accelerates this transition. Non-deterministic workloads don’t just scale differently, they behave differently. They force systems to observe themselves, reason about tradeoffs in real time, and continuously rebalance cost, performance, and correctness. Static infrastructure can’t keep up with that pace.

The logical conclusion isn’t just infrastructure that configures itself. It’s infrastructure that understands why it exists. Systems that infer intent, observe outcomes, and adapt continuously. All without requiring humans to pre-decide capacity, execution models, or even which services should be involved. At that point, infrastructure becomes a decision engine, not a deployment artifact.

That’s the real endgame. Not self-driving in the sense of automation alone, but autonomous in the sense of awareness and adaptation.

In that world, choosing infrastructure looks a lot like choosing a compiler today. Important, for sure, but rarely a daily concern.

That’s the autonomous cloud. And it doesn’t arrive in a single release or rebrand. It emerges gradually, as layers of decision-making move from humans to software. The transition will be uneven. Some domains will resist it longer. Humans will remain in the loop, increasingly focused on intent, oversight, and constraints rather than execution.

The momentum is already there. Capital, talent, and attention are flowing toward systems that can adapt faster than humans can predict. Not because anyone declared it so, but because the economics and complexity leave little alternative.

In that sense, we’re already living in a post-serverless world.

Not because serverless won.
But because it stopped asking humans to guess the future.

I was 46, healthy, and active. I ran regularly. I ate reasonably well. I stayed busy. Too busy, maybe. When I started feeling short of breath walking up a flight of stairs, I thought I might have pneumonia or some sort of upper respiratory infection. I had refereed three youth soccer games in the near-freezing, early-spring rain the day before. Now I felt completely run down, like I was dragging an invisible weight.

I happened to have my annual physical already scheduled for later that week. After hearing my symptoms, my doctor ran an EKG. I sat alone in the exam room for a few minutes. She returned with a somber look on her face and I knew she had bad news. Both my grandfathers and my father died of heart attacks (two in their early 60s, my paternal grandfather in his early 40s). We had discussed my family history for years, but I naively hoped that my heart was the exception. Then she told me I was in AFib.

Atrial fibrillation. An irregular heartbeat that significantly increases the risk of blood clots that can lead to stroke and potential heart failure. Less than one percent of people under the age of 65 are diagnosed with it.

This was not the 1% I was hoping to eventually join.

My Rhythm Was Already Off

Looking back, the signs were there long before my heart hit a stack overflow.

In 2022, we spun out Serverless Cloud from Serverless, Inc. and launched Ampt, a new startup built on the promise of simplifying cloud development. I took on the CEO role, leading a small team of brilliant engineers through what we all knew would be a tough market.

It was an exciting time, but startups come at a cost.

To give it the focus it deserved, I had to put everything else on hold — the Serverless Chats Podcast, my open source projects, my writing, even my regular community involvement. These were all things that had once grounded me. But building a startup has a way of consuming everything in its orbit.

We started fundraising in August 2022, just as the “VC winter” began to hit. Then ChatGPT launched in November, and every investor conversation shifted overnight. Despite all that, we had a term sheet before the end of the year. The terms were terrible, but it was something. Then just as we were about to get the final commitment needed to fill out the round, Silicon Valley Bank collapsed. Our lead investor pulled out, and everything we’d worked toward suddenly seemed out of reach.

We pushed through anyway.

In September 2023, we launched Ampt to the public, found some paying customers, and showed solid month over month growth. It still wasn't enough. By then, AI was consuming every venture portfolio. So we did what any sensible startup would do — we came up with our own (rather compelling) AI story. Think 'building an app using natural language'–level compelling. We shopped it around and eventually attracted some genuine interest from a very well-known firm. Then after several extremely positive meetings over the course of a few weeks, they went silent.

We had spent the last 18 months working on Ampt. I had spent most of that time raising money, refining pitch decks, meeting with investors, and answering a never-ending stream of the same questions on repeat. All while forcing a smile and trying to hide my growing frustration. It was exhausting. We'd kept the lights on through sheer willpower and support from a handful of angel investors. The team was burnt out. And so was I.

The team's resilience and dedication was inspiring. But the limit had been reached. It was time to focus on supporting our families. Everyone wanted to moonlight when they could — something I didn't expect, but greatly appreciated.

When You Can't "Let It Go"

When it comes to startups, I've always been a go-down-with-the-ship kind of guy. If you look up "sunk-cost fallacy" in the dictionary, you'll likely see my picture. Ampt was no exception. I still believed in the vision, and I wasn't ready to let go. So I pushed on, by myself.

For the next few months I continued to work nearly full-time on Ampt. I spent most of my time supporting customers and building the occasional new feature here and there. I also produced several weekly live streams, which I really enjoyed doing. But they also made me think about all the enhancements I wanted to make. An unlikely prospect given that it was just me (before stable releases of Claude Code and Cursor).

For all intents and purposes, we were in quasi-maintenance mode. Progress was slow. So was growth. I took on some small consulting gigs to bring in cash, but the personal financial pressure kept mounting. My oldest daughter was starting college at the end of August. This was a major transition, as it is for any parent, as well as a major expense. Our rainy-day fund needed sunshine. The stress was building. That forced smile was back.

In September, I started consulting for CloudZero. I really liked what I saw there. Great product. Great people. When a full-time opportunity presented itself, I couldn't pass it up. I joined as their Director of Research in November and it felt like a perfect fit. I was going to be working on AI and Big Data with an awesome team behind me. But like most things in life, it came at a cost. I was still supporting customers on Ampt, still writing the Off-by-none newsletter every week, and trying to be a present husband and father. All on top of this new job that I loved and was giving my all to.

I knew it was too much, but I convinced myself that momentum meant progress.

By the end of 2024, I was stressed and exhausted. I had to leave re:Invent early because I became sick with some sort of horrible cold/flu. Not only did I miss Werner's keynote, but I also missed Weezer at re:Play (don't judge — I went to high school in the mid-nineties). I spent the holidays in a general malaise, telling myself I’d feel better once I was able to get some rest. I didn’t.

When the Heart Speaks

It's arbitrary, but a new year always seems to offer a chance to reset. To resolve to make positive change. You promise yourself you'll do better. Be better. I was still stressed, I still felt like I had the weight of the world on my shoulders, but I told myself that this year would be better. It had to be.

It started off promising. I was finally filling out my team at work, we closed 1H planning with a good balance of stretch and attainable goals. We had a plan. We were making progress, and the progress was exciting. Our AI Agents research was showing extremely positive results. The first quarter was now coming to a close, spring was here, and I felt the weight lifting.

On the first Saturday in April I refereed those three youth soccer games. It was only 40°F (4.5°C), and it was raining heavily. I was completely soaked through and frozen to the core. I don't remember ever feeling that cold. The next day I developed a cough, shortness of breath, and the kind of fatigue that a quick run usually shook me out of. Not this time.

I tried running a 5K mid-week and had to stop to walk every few minutes. It was the hardest run I'd ever done.

When my doctor eventually told me I was in AFib, I just stared at her. I’d spent years optimizing systems, debugging infrastructure, fixing the things that broke. Now it was me that was broken. I had to trust that the doctors were as good at their job as I was at mine.

The good news was that my resting heart rate was only in the 80s, incredibly low for the typical AFib patient. I'm normally in the high 40s, a testament to the power of running several 5Ks per week. The low heart rate meant it wasn't a life-threatening emergency, but it still needed to be monitored and treated right away. And the prognosis wasn't particularly encouraging. It could be something I'd need to live with forever.

She prescribed Eliquis (a blood thinner) and Metoprolol (a beta blocker) to keep my heart rate down. She referred me to a cardiologist and scheduled some additional tests. Then came the doctor's orders. No running. No alcohol. Minimize stress. Make sure I keep my heart rate under 110 bpm to avoid rapid ventricular response (RVR).

My malfunctioning heart sank. This was not the year I'd been hoping for.

My doctor and I talked for a long time, much longer than what’s normally allotted for a routine physical. Running was my stress reliever. Evenings and weekends were when I decompressed, working on projects around the house, spending time with my family, or tinkering on side projects that gave me energy. I also enjoyed an occasional craft beer or a pour of whiskey to take the edge off after a long day. The constant tiredness and restrictions of AFib were going to steal most of that away from me.

She suggested talking to a mental health professional, something I had been guilty of stigmatizing my entire life. But I realized I wasn’t just dealing with a heart condition; I was dealing with years of mounting stress and burnout I had refused to acknowledge. It was the first time I finally admitted I couldn’t just willpower my way through it. I needed help managing the stress and regaining focus, and for the first time in my life, I stopped seeing that as weakness.

Getting that support, and the right medications to help quiet and focus my mind, was one of the best decisions I have ever made. Healing isn’t just about fixing your heart; it’s about taking care of your head and everything connected to it.

When It Rains, It Pours

A week later, I got a transthoracic echocardiogram. Structurally, my heart was fine. That gave me hope. My doctor said the cardiologist would likely perform a cardioversion — a controlled shock similar to the whole "clear" thing doctors yell on TV. This would likely get me back in sinus rhythm and then maybe this whole thing would be over and done with.

Then life did what it always does.

In May, my mother-in-law was hospitalized with pancreatitis. A few days later, my seventeen-year-old daughter was in a major car accident. A seventy-year-old man fell asleep at the wheel and crossed over the double yellow line into her lane. Luckily she swerved to avoid getting hit head on, but his car hit the driver side while both vehicles were traveling over 35 mph. His car spun multiple times and ended up back in his lane. My pickup truck that my daughter was driving ended up perpendicular to the road.

I got the call around 6:45 am. I took my wife's car and sped to the scene in a panic, driving past the backed-up traffic, scared to death of what might be waiting for me. I found her standing next to a paramedic, shaken but alive.

I gave her a hug that seemed to last forever. Relief washed over me. She had some burns from the air bags and a few minor scrapes, but otherwise she was okay. My truck most definitely wasn’t. And neither were my stress levels.

When I saw my cardiologist, he officially diagnosed me with persistent AFib. I had been in AFib for almost 6 weeks at this point, and I knew this wasn't going away on its own. We scheduled a cardioversion for the end of the month.

It worked. For a few glorious days, I was back in sinus rhythm. I even went for a short run with my daughter. But six days later, while attending a conference in San Diego, I slipped back into AFib.

Round two. Another cardioversion two weeks later. This time, the cardiologist put me on Amiodarone — a powerful anti-arrhythmic with a list of side effects that reads like a pharmaceutical horror story. Liver failure. Lung damage. Thyroid dysfunction. Vision problems. Skin discoloration. It was clear that we needed a longer term solution. My cardiologist recommended a cardiac catheter ablation. September 25th was the earliest appointment available, and it was only the middle of June.

Then, a week later, my mother-in-law unexpectedly passed away from necrotizing pancreatitis. She was seventy-three, the matriarch of the family, and the most caring and thoughtful person I'd ever known. Her loss broke all of us. Especially my wife.

Two days later, I slipped back into AFib. I couldn’t tell my wife. She was grieving, and she didn’t need something else to worry about. The next day I messaged my cardiologist to discuss options. That afternoon I suddenly started feeling better, and my Apple Watch confirmed I was back in sinus rhythm. My cardiologist said it was the Amiodarone doing its job.

I called it mercy.

A Slow Return

Summer arrived without the usual fanfare. My wife was still grieving, spending most of her time helping my father-in-law go through the material things my mother-in-law left behind. A closet full of clothes, jewelry collected and inherited over the course of her life, and a mountain of boxes filled with generations of family photos and keepsakes. There were lots of tears, intermixed with the occasional smile as memories came flooding back. Momentary respites from all the grief. It's hard watching the people you love suffer, especially when there's little if anything you can do to ease their pain. I forced another smile. I needed to be the shoulder to cry on. I knew this was what my family needed, and the healing was going to take time.

Despite everything that was going on, I felt stable for the first time in months. I took a break from writing the Off-by-none newsletter, I took my annual beach vacation with family friends, and I journaled (a lot). I got to watch my daughter play with her team at the NFL Flag Championships in Canton, OH (she was even on ESPN). On the way home, we stopped at Niagara Falls and rode the Maid of the Mist. I even started running again. I was regaining my strength.

In August, I flew to Seattle for the AWS Heroes Summit and started to feel like myself again. Reconnecting with old friends and meeting new ones from the Heroes community was exactly what I needed. It reminded me how much I loved being a part of it, and how good it felt to contribute and help others.

Up until the Heroes Summit, I was relatively quiet about my medical issues. But amongst my friends, the conversation came up several times. AJ Stuyvenberg (a friend and fellow AWS Hero) told me his first job out of college was at the company that invented Pulsed Field Ablation (PFA), the new, safer, recently FDA approved technology they’d use for my procedure in September.

He told me it was incredibly safe. I told him that as long as he hadn't written any of the code, I was willing to believe him. 

Once I got home, I felt as though something had changed. I was suddenly incredibly motivated. I had a much clearer sense of where I wanted to spend my time. Time with family, time with friends, and time for the community that had brought me so much joy in the past.

My wife and I hosted a family BBQ at my house in late August. There were almost 40 of us, missing only one person who couldn't make it. It's hard to get a group that big together, so when you can, treat it like the gift it is.

The next day I took my youngest daughter to see Oasis at MetLife Stadium in New Jersey. The ticket prices were ridiculous, but the concert was amazing. There's something incredibly special about watching your kid sing along to the songs you loved when you were their age. We stayed in a hotel that night, then drove four hours the next morning to drop her off at school. I remember thinking as she napped in the car on the way back: whatever happens next, I’m glad I had this.

The Night Before

The night before my ablation, my wife and I were sitting at the kitchen table. I was nervous but ready. I had researched the procedure, knew the odds, and trusted that if my insurance company was paying $102,000 for this, then I was likely going to get their money's worth.

Then she said, “You should probably make a list of all our accounts and passwords. Just in case.”

She wasn’t wrong to say it, but it made everything seem very real in a way it hadn’t before. Things go wrong, doctors make mistakes. Regardless of the odds, there was a very real, albeit small chance that I just finished my last meal. That terrified me.

The Procedure

We drove to Tufts Medical in Boston early in the morning.

There was a lot of shaving, including the hair on both wrists so they could insert IVs. There were at least ten doctors and nurses in the operating room and a giant matrix of computer monitors to precisely control the catheters. They covered me in more than forty electrodes, strapped me to the operating table, and a nurse ran through the pre-op checklist. That's the last thing I remember.

While I was anesthetized, they threaded catheters through my femoral arteries and up into my heart. They then used the PFA system to identify and cauterize the overactive cells.

Two and a half hours later, it was done.

I woke up with a sore throat from the breathing tube and a dull ache where the catheters had been inserted. My wife was by my side. I had to lie flat for several hours to minimize any bleeding, but after a few checks, I was cleared to go home that night. That's modern medicine. My doctor had told my wife after the procedure that everything went perfectly. There's no cure for AFib, but this should keep me in sinus rhythm for the next few decades.

Recovery

The recovery was uneventful, which is exactly what you want after a procedure like this.

No lifting anything over ten pounds. No running. Lots of time to think.

A week later, I took my oldest daughter to see Whose Live Anyway?, a live version of Whose Line is it Anyway? We went out to dinner, laughed for two hours straight at the show, and had a rare opportunity to reconnect. Something that college-aged children don't often have time for.

A few days later, I ran my first post-surgery 5K. It was slow, cautious, and deliberate — but I ran the entire time and finished. It was a good start.

Looking Ahead

At the end of October, my cardiologist plans to take me off the Amiodarone.
By the end of the year, I should be done with the blood thinners, too.

If all goes well, I'll start 2026 with all of this behind me.

I still have a way to go to get back to my "normal" self, but after half a year of chaos, and the burnout that led up to it, this experience has given me a new sense of optimism. And that fuels my motivation.

Over the last few weeks, I’ve started writing again. I’ve been re-engaging with the community, working on my open source projects, and reconnecting with the people and projects that once gave me energy.

I'm not quite ready to bring back the podcast, but maybe someday.

What I Learned

This year tore me down. It also rebuilt me.

I had to face my mortality, watch someone I loved die, and pause the parts of my life that defined me. But it also forced me to remind myself of some of life's most important tenets.

It’s okay to step away. Your health, both physical and mental, matter more than any startup or project.

Support is everything. Family, friends, and co-workers don’t just help you carry the load, they remind you that you don’t have to carry it all by yourself.

And you’re not alone. Everyone is fighting something. Be kind. And if you need help, ask for it.

After years of optimizing for performance, I’ve started optimizing for balance. We can't control what life throws at us, but we can choose how we deal with the hand we're dealt. I'm not religious, but there is something beautiful about the Serenity Prayer. That wisdom to know what can and cannot be changed often comes from lived experience. These last few years have taught me some hard lessons. My hope is that they've prepared me for whatever comes next.

V1 solved some of those problems. It wrapped the low-level calls and let you work with native JavaScript types instead of nested objects. It integrated named parameters and made transaction handling much easier. But the more I used it, the more I saw how much further it needed to go.

V2 is that step forward. It’s a complete rewrite that’s faster, smarter, fully-typed, and designed to fit naturally into modern serverless applications. It now plays nicely with your favorite ORMs, handles Aurora’s scale-to-zero behavior automatically, and even lets you drop it in as a replacement for libraries like mysql2 or pg.

Built for Modern TypeScript

The entire v2 library is written in TypeScript. Every function, parameter, and return type is strongly typed, so you always know what to expect.

No more guessing which properties exist or how the results are shaped. The compiler now tells you.

A Smaller, Faster Core

Data API Client v2 now uses the AWS SDK v3 as a peer dependency. That change alone brings smaller bundle sizes, faster Lambda cold starts, and better tree-shaking. It also aligns with modern async/await patterns, which makes code cleaner and easier to reason about.

If you’re deploying to Lambda (Node.js 18+), the SDK is already part of the runtime, so you don’t even have to install it.

Drop-In Compatibility

One of the most requested features was the ability to use Data API Client as a direct replacement for popular database clients. V2 adds compatibility layers for both mysql2/promise and pg, so you can often swap them out without touching the rest of your code.

PostgreSQL example

MySQL example

Named placeholders are supported too:

Full ORM Support

These compatibility layers unlock seamless integration with modern ORMs and query builders like Drizzle and Kysely.

Here’s Drizzle with MySQL:

Kysely works just as well and remains fully type-safe.

Solving the Scale-to-Zero Problem

Aurora Serverless v2 can pause when idle to save costs. That’s a powerful feature, but it also means the first query after a pause fails with a DatabaseResumingException. Most developers either write their own retry logic, accept the failure, or worse, leave their clusters running 24/7, which defeats the purpose.

V2 handles this automatically. When the database is waking up, the client retries intelligently, with tuned delays based on real-world behavior. Most clusters are ready in 15–20 seconds, and you don’t have to write a single line of retry logic.

And here’s the key point: this isn’t just for direct dataApiClient queries. The automatic wake-up handling is built into everything: the raw client, the mysql2 and pg compatibility layers, and any ORM you connect through them. Whether you’re running a single query, a transaction, or a complex set of operations through Drizzle or Kysely, the retry logic is always there, quietly doing its job.

Better PostgreSQL Support

V2 now automatically converts PostgreSQL arrays into native JavaScript arrays:

It works for all array types (including nested ones) so you can use them naturally without manual parsing.

Doesn’t the Data API Already Support a JSON response format?

It does. The AWS Data API's built-in JSON format option (via formatRecordsAs: 'JSON') is great for simple use cases and a nice step forward. It converts most column types into native JavaScript values and removes some of the boilerplate that used to make working with the Data API painful. But it stops at the surface.

The parser in v2 goes much further. It preserves database-specific type information, handles complex types like PostgreSQL arrays and binary data intelligently, supports configurable date deserialization, and gives you richer metadata about query results. You also have control over output format: fully hydrated objects or raw arrays for performance-critical paths. In other words, the built-in JSON support is a convenience feature, v2’s parsing is designed for production systems that need flexibility, fidelity, and control.

Tested for the Real World

My 25+ years of building software has taught me to take testing seriously. This release includes unit tests for the core features, integration tests against real Aurora clusters, ORM tests with Drizzle and Kysely, and wake-up handling tests against paused databases. Over 390 integration tests run as part of the build to make sure everything works as expected in production.

A Complete Example

Here’s a simple Lambda function that uses Drizzle with Data API Client v2:

This code is type-safe, works with scale-to-zero databases, requires no VPC, and is ready for production.

Looking Ahead

I’m already exploring Prisma and TypeORM adapters, built-in query performance metrics, and optional query caching. The goal is to make Data API Client the simplest and most powerful way to work with Aurora Serverless v2, regardless of the stack you’re using.

A Few Final Thoughts

I built the first version of this library because I needed it. I built v2 because I wanted it to become something even bigger. Hopefully this helps close the gap between the flexibility of serverless and the power of relational databases.

There’s still more to do. I want this project to help developers rethink how they connect to data in a world where databases don’t have to be always-on. I want it to make writing queries feel natural again, whether you’re building with SQL directly, composing queries in Kysely, or relying on an ORM like Drizzle. And I want the infrastructure — the retries, the wake-ups, the scaling — to fade into the background so you can focus entirely on the application in front of you.

V2 isn’t the finish line. It’s a foundation for everything that comes next. And I’m excited to keep building on it with your feedback, your use cases, and your ideas.

Let me know what you think: https://github.com/jeremydaly/data-api-client

Tags: #aws#mysql#postgres#serverless#rds#aurora-serverless

Most still think about AI as a set of tools to make existing processes faster, when in reality, it’s redefining the processes themselves. The inertia of how product management has always worked — PRDs, roadmaps, sprint cycles — has blinded many to the fact that their jobs are fundamentally changing. Either they adapt to this new reality, or they become the bottleneck.

These capabilities already exist. AI is reshaping what it means to be a product manager. Not in the abstract sense, but in the daily rhythm of how we discover, build, and learn. The rituals that once defined the craft now feel out of sync with the speed, fluidity, and adaptability this new era demands.

The old methods haven’t become irrelevant, they’ve just become too slow. We’re no longer documenting what might happen, we’re discovering what’s already happening. In real time.

From PRDs to Prototypes

When I was VP of Product at a startup (over a decade ago now), I was working with a brilliant group of engineers in Finland. I wrote long PRDs to communicate product vision, crafting every sentence with precision in an attempt to capture every detail. The problem was that language is subjective. Even more so when it crosses borders, language barriers, and cultures. What made perfect sense to me didn’t always translate how I thought it would.

Eventually, out of frustration, and because I had the engineering skills, I just built a prototype. The moment I showed it to them, everything clicked. No confusion. No misinterpretation. Just shared understanding.

That experience changed how I approached product management. Back then, prototypes were hard for the average person to create. Today, with tools like Claude Code, Codex, Cursor, and Warp, they’re almost effortless. A product manager can spin up an interface, modify it on the fly, and demonstrate the concept live. It’s faster, clearer, and infinitely more collaborative.

We no longer need to describe the product in exhaustive detail. We can simply show it, get feedback, and refine it. The PRD has become a prototype, and that shift has redefined how teams align around vision and execution.

From Feedback to Fluidity

Real-time feedback has always been the goal. The difference now is that AI makes it instantaneous and actionable.

By instrumenting products with analytics tools like Mixpanel, Amplitude, or Google Analytics, AI can extract and track user behavior in real time, highlighting friction points before customers have a chance to report them. It can identify drop-offs, detect confusion, and pinpoint which features drive engagement.

And it doesn’t stop at data. AI can synthesize qualitative feedback too. Analyze comments, reviews, and surveys for sentiment, themes, and tone. It can recognize patterns and reveal hidden issues long before they escalate.

This creates a kind of living dialogue between product and customer. Feedback isn’t an event anymore. It’s an ongoing flow. Every click, hesitation, or abandoned task becomes a form of conversation. And with AI interpreting that conversation, product teams can adjust quickly, keeping the experience fluid and responsive.

From Roadmaps to Real-Time Adaptation

Static roadmaps once gave teams a sense of order. But in the age of AI, order comes from adaptability, not assumption.

You can still have North Stars, those big goals that anchor your strategy, but the path to get there must stay flexible. Markets evolve, technologies shift, and customer needs change faster than ever. A roadmap drawn months in advance is almost guaranteed to go stale.

AI helps turn roadmaps into living systems that adjust continuously. It can monitor customer sentiment, competitive moves, and market dynamics. It suggests when to pivot, accelerate, or abandon a feature. It can even generate stakeholder updates that explain why priorities changed and how those changes still align with business objectives.

The goal isn’t to plan perfectly, it’s to move deliberately. AI makes it possible to build the right things quickly instead of building the wrong things slowly.

From Documentation to Dynamic Knowledge

AI can now read code, trace dependencies, and understand complex architectures better than any human ever could. That ability turns documentation into something much more dynamic.

Instead of relying on static pages that go stale after each release, AI can generate and update documentation automatically from code, tests, and real-world user interactions. It can write clear explanations, create examples from actual usage, and regenerate everything the moment a new pull request is merged.

It can also maintain tone and consistency across every format, from API references to web tutorials. Need a localized version in Spanish or Portuguese? Done instantly. Need to shift from a casual, scrappy startup tone to a more formal one for enterprise audiences? Regenerate and publish within minutes.

AI doesn’t replace technical writers, it amplifies them. The tedious parts vanish, leaving space for the craft. It's the clarity, precision, and voice that make documentation human. In this new world, knowledge never lags behind the product, it moves with it.

From Intuition to Intelligent Insight

Great product management has always balanced data and instinct. AI doesn’t change that balance, it enhances it.

With the ability to process millions of interactions, AI can uncover behavioral trends, predict churn, and identify correlations that humans would never spot. It can tell you which features actually drive engagement, where users are struggling, and what behaviors signal long-term retention.

But intuition still matters. Data alone can’t tell you how a product makes your users feel. The best product managers use AI to inform their instincts, not replace them. They see beyond what the data confirms and imagine what it can’t yet describe.

AI expands our vision. It doesn’t make decisions for us, it helps us see more clearly before we make them.

From Ideas to Instant Iteration

There's an old trick in product management called the “fake door test.” You’d place a button for a future feature just to see if users clicked it. It was crude, and likely frustrating, but it worked. Now, you don’t need to fake it and disappoint users anymore. You can build a feature MVP in an afternoon, instead of waiting weeks.

AI coding assistants like Claude Code, Cursor, Copilot, Codex, and Warp have changed the speed of development entirely. What once took an entire sprint now takes experienced developers less than a day. And because engineers move faster, product managers have to move faster too.

That means continuously generating experiments, refining hypotheses, and feeding new ideas into the system. You can’t plan quarterly anymore. You have to iterate daily. It’s not about doing more work. It’s about creating a constant rhythm of learning.

Time to market isn’t just compressed, it’s redefined. The distance between idea and impact has never been shorter.

From Management to Mastery

The product manager’s role isn’t disappearing, it’s evolving. AI now touches every part of it, from how we define vision to how we measure success.

Here are eight ways traditional product management is being reimagined in the age of AI.

Vision and Strategy: From Insight to Intention

AI can surface market trends, identify emerging competitors, and analyze customer sentiment across billions of data points.

Tools like Deep Research can even monitor how competitors position and market their products in real time. This helps PMs plan proactively, not reactively.

But strategy still starts with people. It requires empathy, curiosity, and conviction. AI informs vision, it doesn’t define it.

Understanding and Prioritization: Scaling Empathy with AI

Talking to customers is still essential. But you can only talk to so many.

AI helps scale that understanding by analyzing behavioral data from all customers, not just your biggest ones. It can identify common themes across feedback, detect patterns between enterprise and SMB users, and even predict which requests will deliver the highest ROI.

It’s a blend: human conversation for depth, AI synthesis for breadth.

Roadmapping: From Static Plans to Living Systems

The role of the roadmap is changing. It’s no longer a static artifact that tries to predict the future. It’s a living system that constantly reacts to it.

AI now makes this adaptability operational. Instead of a roadmap that lags behind reality, you get one that evolves with it.

The roadmap becomes less a document and more a reflection of your product’s pulse.

Collaboration: From Handoffs to Co-Creation

The days of throwing PRDs over the wall are over.

Product managers can now sit with designers, sketch a feature in Figma, and use AI to turn that mockup into a working prototype, complete with interactive logic. It won’t be production-grade, but it will be tangible, testable, and fast.

That shared artifact keeps everyone aligned: design, engineering, marketing, and leadership.

It’s not about replacing cross-functional collaboration. It’s about collapsing it into the same moment.

Development: From Oversight to Observability

AI makes it possible for PMs to see, in real time, what’s actually being built.

Through repository analysis and test-driven development, PMs can track which features are complete, which are stubbed, and which need attention.

Better yet, AI can generate test suites that become living PRDs, creating measurable, verifiable representations of requirements.

It can even compare UI deployments against design files to detect inconsistencies automatically. Oversight becomes observability.

Launch: From Events to Conversations

Product launches used to be finish lines. Now they’re starting points.

AI turns launches into ongoing conversations with the market. It continuously analyzes sentiment, adoption, and engagement, adjusting messaging and rollout strategies in real time.

PMs can run “what-if” simulations, test launch scenarios across segments, and even predict adoption curves before going live.

It’s not about perfect launches anymore. It's about perpetual ones.

Performance: From Dashboards to Direction

Post-launch monitoring used to mean dashboards and KPIs. Now AI does the watching for you.

It detects anomalies, finds correlations, and tells you not just what happened, but why. It narrates performance: “Feature B adoption grew 18% among healthcare customers due to workflow simplifications introduced in v1.2.”

AI even predicts future behavior. Who’s likely to churn? Which feature is peaking? What are the recommended next steps?

PMs move from reactive analysts to proactive optimizers.

Iteration: From Updates to Evolution

Iteration used to be cyclical: gather, prioritize, ship, repeat.

Now it’s continuous.

AI synthesizes user feedback, behavioral analytics, and competitive insights into ongoing recommendations. It can autonomously design and run A/B tests, optimize flows, and personalize experiences in real time.

Your product becomes a living organism that's learning, adapting, and evolving alongside your users.

From Operators to Orchestrators

AI is redefining the product manager’s identity. They’re no longer operators moving pieces on a board, they’re orchestrators designing the flow of intelligence across the system.

Their role isn’t to manage processes, but to shape intent. They decide what matters, set the rhythm, and ensure coherence as everything moves faster. The work is less about keeping things on track and more about guiding the direction of motion.

Product managers in the AI era don’t just push work forward. They design how learning happens inside the organization.

Humanity Still Matters

Every era of technology creates its own kind of product manager.
In the industrial age, they optimized factories.
In the software age, they managed backlogs.
In the cloud era, they built at scale.

Now, in the AI age, PMs work with something that doesn’t just execute instructions, it anticipates them.

AI doesn’t diminish the role, it accelerates it. It collapses cycles that once took quarters into minutes. It translates the ambiguity of human behavior into patterns we can understand, forcing us to rethink what “building products” even means.

We’re not just making things for users anymore. We’re making systems that learn from them, shape them, and sometimes outgrow us.

The challenge isn’t how to use AI, it’s how to remain human while doing so.

Because as AI automates more of the craft, what’s left is the art: understanding emotion, designing for trust, deciding what deserves to exist.

The best product managers of this era won’t just manage velocity or optimize funnels. They’ll translate between intelligence and intent, shaping technology that serves human curiosity, not just commercial logic.

AI is accelerating everything. But speed alone isn’t progress.
Progress is knowing where you’re going and why.

We’re not here to make smarter software.
We’re here to make better decisions.
And that, more than ever, requires being profoundly and intentionally human.

Why CloudZero?

CloudZero's mission aligns deeply with my passion for transforming how teams build and operate in the cloud. Cloud financial management is no longer just about cost tracking—it's about delivering actionable intelligence to drive better decisions across engineering, product, and finance. Their new AI-driven data platform is poised to make this vision a reality by leveraging the latest AI/ML technology to analyze massive data sets and unlock insights and optimizations previously out of reach.

As Director of Research, I'll lead innovation at the intersection of data science, AI, and cloud economics. The opportunity to work with a world-class team to shape the future of cloud intelligence is both humbling and exciting. This new role allows me to bring my experience and learnings from over 25 years in tech to an entirely new challenge while pursuing my passion for making the cloud more accessible and efficient for everyone.

What about Ampt?

The past two and a half years with Ampt have been an incredible journey. Together, we built a product that challenged conventional thinking about cloud development and introduced a number of innovations such as Productized Patterns and our Infrastructure-from-Code (IfC) approach. These resonated deeply with users and even influenced and inspired others in the industry. While we made significant progress and gained a passionate base of loyal users, competing for mindshare in a crowded market of well-funded developer tools has made it challenging to give our team the proper support.

For the last several months, I've focused on optimizing Ampt's operational stability, supporting our loyal customers, and ensuring the platform's long-term viability. I'm proud to share that the product remains strong, and our dedicated customers continue to support its ongoing operations. Though Ampt will no longer be my full-time day job, this new position is fully supportive of the work we've accomplished, enabling me to keep the platform independent and running even as I take on this new challenge.

My Continued Commitment to the Community

This new role also provides me with the opportunity to reconnect with some of my greatest passions. I'm excited to continue sharing insights through my weekly Off-by-none newsletter, a channel that keeps me closely connected to the incredible cloud and serverless communities. I'm also looking forward to dedicating more time to open source projects, exploring new ideas, and collaborating with all of you to help shape the future of cloud.

These important efforts, along with my work at CloudZero, will allow me to engage even more deeply with developers, architects, and cloud enthusiasts to push the boundaries of what's possible.

Looking to the Future

While this marks a significant transition, it doesn't mean goodbye to Ampt, its vision, or the remarkable community that's grown around it. Instead, it's an evolution. The lessons learned and the relationships created will fuel the work ahead, inspiring new solutions and ideas at CloudZero while continuing to innovate on the cloud developer experience. I'm excited to tackle the complexities of cloud data at scale and help organizations unlock the full potential of their investments in the cloud.

To my Ampt team and community: thank you for believing in the vision, for your resilience, and for being part of this amazing journey. To my new colleagues at CloudZero: I can't wait to drive the future of Cloud Business Optimization together.

Here's to the next adventure!

Lambda functions have been a great choice for the vast majority of my use cases. They scale quickly to handle traffic spikes, and for asynchronous workloads, I never worry about cold starts. Even for synchronous workloads with minimal traffic, cold starts are mostly a non-issue. However, I have hundreds (maybe thousands) of synchronously invoked functions that receive infrequent and inconsistent traffic. Cold starts on those functions are not only painful to experience, but in some cases, can create upstream integration issues.

In this post, I'll outline a strategy I've been experimenting with that utilizes Lambda function lifecycle events to optimistically warm functions when there are no more active execution environments. I've been using this technique for a while now, and the results have been exceptional. Please note that this is not your standard trigger-with-a-scheduled-task Lambda Warmer that you've likely heard of in the past. This is a much more sophisticated approach designed to minimize costs, unnecessary invocations, and request collisions.

Traditional Lambda warming strategy

Many (many) years ago I created the fairly popular Lambda Warmer npm package. Its purpose was to optimize AWS Lambda function cold starts by implementing AWS best practices for warming functions "correctly". Those tips included:

• Don’t ping more often than every 5 minutes
• Invoke the function directly (e.g. don’t use API Gateway to invoke it)
• Pass in a test payload that can be identified as such
• Create handler logic that replies accordingly without running the whole function

A lot has changed in the last 6 years, but fundamentally, this technique still "works". The issue is that those warming pings happen every 5 minutes whether they're needed or not. This can lead to several unintended consequences.

1. Unnecessary invocations: If Lambda functions are already "warmed" by actual traffic, sending regularly scheduled warming requests adds extra costs, resource usage, and CloudWatch log ingestion without providing any additional benefit.
2. Blocked requests: Scheduled invocations will use a warm execution environment if one is available. It's possible that this can block actual traffic from using that warm environment and force a cold start on an end user's request. If you attempt to warm multiple Lambda execution environments, which requires intentionally delaying a response to force additional environments to spin up, you're highly likely to run into an end user blocking situation.
3. Cold starts due to early termination: Lambda function execution environments (as of this writing) are typically recycled after approximately 6 minutes of inactivity. However, AWS may decide to recycle your environments early (or later) for any number of reasons, including internal failures. This happens a lot. If a user invokes your function before the next warming window, they are guaranteed to get a cold start.

Despite the issues above, this method remains quite popular as it's fairly easy to implement and inexpensive to run.

What about Provisioned Concurrency?

At AWS re:Invent 2019, AWS announced Provisioned Concurrency, a feature that allows you to "pre-initialize" Lambda execution environments. The promise of this feature seemed like a perfect solution. You simply specify how many "warm" Lambda functions you want (you can even manage it on a schedule or based on utilization), and AWS will make sure you have a warm, autoscaling fleet ready to go. As your function starts receiving traffic, the AWS Lambda service continues to warm new execution environments (without blocking) up to your pre-configured maximum. If you exceed your maximum, then it rolls over to standard on-demand scaling.

If you have a function that requires extremely low latency and is expected to get predictable periods of sustained traffic, Provisioned Concurrency is definitely worth looking at. However, the cost of this feature seems wildly expensive compared to other compute options. For example, the current Lambda Pricing charges $0.0000041667 for every GB-second of Provisioned Concurrency. For a function with 1024MB of configured memory, that would cost ~$0.02 per hour to handle ONE concurrent request. That doesn't include the PER invocation charge and the duration charges.

There are msny benefits to using Lambda functions, most notably their wide array of Event Source Mappings (ESMs). ESMs make integrating with other AWS stream and queue-based event sources extremely simple and (mostly) headache free. But if you are using Provisioned Concurrency, it would almost certainly be for synchronous invocations like API requests, not for ESMs or other direct triggers like S3 and SNS. It depends on your use case and the amount of expected traffic, but in many cases, services like App Runner or ECS can provide auto-scaling containers that handle your traffic at a much more predictable cost.

For our use case, keeping at least one Lambda execution environment warm 24x7, would cost $10.80/mth (assuming 1024MB of configured memory). Also, that would be for EVERY FUNCTION we wanted to keep warm. Single-purpose Lambda functions have their uses, but at this price, you might want to think about building a MonoLambda (or Lambdalith).

Enter Lambda Tail Warming

Lambda Tail Warming is a technique that uses a Lambda Extension to track active execution environments by monitoring lifecycle events, then optimistically invoking itself to ensure at least one warm environment at all times.

The basic process is as follows:

1. Install an external extension that subscribes to INVOKE and SHUTDOWN events from the execution environment.
2. When a new execution environment is started, you register the extension and persist the extensionId in an external datastore such as DynamoDB or ElastiCache. The record should also contain a time-to-live (TTL) value that expires after several minutes (more on this later).
3. If an INVOKE event is received after some amount of time has passed (more on this later), update the datastore record with a new TTL.
4. If the extension receives a SHUTDOWN event, check the datastore to see how many active execution environments are running. If there are one or zero, use the AWS SDK to asynchronously invoke the Lambda function using InvocationType: 'Event' with a payload that indicates it's a warming event.
5. Remove the datastore record for current extensionId and exit the extension.

If you're unfamiliar with Lambda Extensions, AWS has a GitHub repository filled with several examples to guide you. External extensions can be written in a different language than the function, which means you can write this once and add it as a Lambda Layer to any function you want to warm. We'll look at a Node.js example, but you're free to adapt this to your preferred language. Let's build.

Building a Lambda Tail Warmer

This example from AWS shows the set up of a basic external extension. The extensions/nodejs-example-extension file is deployed to your Lambda function's /opts/extensions directory as part of the Lambda Layer or within a container (if using the container packaging format). This file is picked up by the Lambda runtime on initialization and will execute the index.js script in the nodejs-example-extension directory.

The example includes an extensions-api.js helper function that handles the interaction between the module and the Lambda runtime for us. We'll use its exported register and next methods in the main extension script.

Take a look at the sample extension script below. The main() function is executed immediately. It registers the extension using the register method, then enters a loop waiting and awaits the next event from the Lambda runtime (await next(extensionId)). When a new event is sent from the runtime, the script parses the event type and then either executes the handleShutdown or handleInvoke methods. If it's a SHUTDOWN event, we also need to gracefully exit the extension using a process.exit(0).

Now that we're familiar with how the extension handles lifecycle events, let's add our Lambda Tail Warming logic. Let's start by adding some variables to the top of our script:

Notice that I added a data import. This is referencing your own data module that would persist to the datastore of your choice. I provide some pseudo methods in the examples below to demonstrate the interactions. We also need the AWS SDK Lambda client. And I've moved the extensionId to the global scope (so we can access it later), created a variable to track when the extension was initialized (initTime) and set a refreshInterval for 10 minutes.

Next we create a new trackLambda function:

If the remove argument is true, we remove the extensionId from the datastore, otherwise, we upsert the record with the current time and a TTL value slightly greater than our refreshInterval. The reason we do this is twofold. First, we want this record to expire if it doesn't get cleaned up by a SHUTDOWN event. I've yet to see an extension not get a SHUTDOWN event, but network failures and other issues are sure to leave some zombie records. This is a proactive approach to cleaning them up if something fails. We add a few seconds to the TTL to give a refresh request time to update the record before expiring.

Now we need our warmLambda function:

Here we initialize the Lambda client, then send an invocation to our function (process.env.AWS_LAMBDA_FUNCTION_NAME conveniently stores the name for us). We'll use an InvocationType of Event as we don't want to wait for a response. Lambda Extensions get between 500ms and 2000ms of processing time to perform any cleanup (500ms when registering a single extension), so we want this to be as efficient as possible. There is no guarantee that the Lambda service will respond in time, but it hasn't been an issue in all my tests. Also, if the invoke request fails, we want to catch the error and log it. This will only be called when the container is being shutdown anyway, but we want to make sure any other processes have a chance to run in the extension.

We need to update our handleShutdown function:

When our handleShutdown method is called, we first check to see if this was due to a Lambda Function failure. If it is, we likely don't want to keep trying to invoke a function that is failing. Otherwise, we retrieve a list of active environments from our datastore. If there are one or zero active environments (we assume that one is the current environment being shut down), then trigger the warming event. Regardless of the number of environments, we want to make sure we clean up the datastore record of the current environment by calling trackLambda with the remove argument set to true. Then we gracefully exit.

Our handleInvoke function also needs to be updated:

Most invocations we can ignore since we don't want to update the datastore on every request. But every so often we want to send a heartbeat to the datastore to let it know an execution environment is still active. Above I'm checking to see if we've exceeded the refreshInterval. If we have, we update the datastore by calling trackLambda() and reset the initTime.

Finally, we can update our main extension loop logic:

I've updated the SIGINT and SIGTERM event listeners to support the async handleShutdown function just in case we don't get our standard SHUTDOWN event. I've also added a trackLambda() call before we enter the loop (this is the INIT phase). I'm not awaiting this as we don't want it to block the extension's initialization. Then I added awaits to our handleShutdown and handleInvoke function calls. And that's it. Your Lambda Tail Warming is ready to go. Well, almost.

Configuring your Lambda function for Tail Warming

Before you can start warming your function, you have to make a few changes to ensure that the process will work correctly.

1. Enable Invoke Permissions: In order for your extension to invoke your Lambda function, the function must have permission to invoke itself. You can add the following to your IAM role:

2. Add a "warmer" event interceptor to your handler code: In order for your Lambda function to respond as quickly as possible and avoid potential request blocking, you'll need to modify your handler code to short circuit requests. This will ensure that any complex logic (and potential mutations and logging) is not processed. For example:

3. Enable datasource access permission: In addition to the necessary Lambda Invoke permissions, you also need to make sure that the Lambda function has access to the datastore you're using to persist execution environment information.

How does this perform?

Below are the traces for a rather large Lambdalith that includes the canvas binary to generate and manipulate image files. The first trace is the typical cold start (~3 seconds) for this application. The second 200 request below is a typical warm invocation (< 60 milliseconds). The subsequent request is a Tail Warming event (note the 202 status code as this is happening asynchronously). This is a cold start (remember we do this immediately after the SHUTDOWN event of the last environment) which initializes a new execution environment and has a total duration of 3.122 seconds.

The next invocation shows a request from a user with a response time of 824ms. This application loads a significant amount of state, which typically happens during a normal user request. Our warming event short circuits this, which is why we see some additional response time during this invocation. It is possible to load state or perform some other actions during our warming event, but the goal is to find the right balance between cost and performance. The warming event saves us over 2.2 seconds of initialization time on the first subsequent user request and costs us only about 10 ms of billable time.

Other things to consider

There are several ways to improve this technique and make it more intelligent:

1. The refreshInterval could be tweaked to support shorter expirations or more frequent datastore heartbeats. I found 10 minutes makes the most sense as it prevents premature removal of datastore records when the app is handling real traffic from users.
2. It's possible that a failure (network issue, missed SHUTDOWN event, etc.) could prevent the warming event from triggering. This would break the cycle of warming events, meaning the next real invocation would get a cold start. You could mitigate this by periodically checking the datastore for active records or by attaching listeners (like using DynamoDB Streams) that would do something similar. This would be in a separate process (e.g. Lambda function), so it wouldn't block invocations to the primary function unless you needed to warm it. It would add to the cost, but if you were warming multiple functions, this would benefit from the economies of scale.
3. You can add controls to turn this on and off. Extensions have access to Lambda environment variables, which are a handy and low latency way to add state to every invocation without needing to redeploy or fetch external configurations. A function configuration update could be scheduled if you wanted to automate this.
4. You could hydrate the state on every warming event. As I mentioned before, the majority of cold start time is typically during the initialization stage. If it makes sense to load some of that state on a warming event, then feel free to.
5. You could combine this strategy with the traditional Lambda Warming technique. When you invoke a "warm" function, this will generally extend the duration of that execution environment. That includes maintaining any global state that has been loaded. As long as an execution environment stays warm, the Tail Warming will never fire. When the environment is eventually recycled, the Tail Warming would immediately invoke a new one.
6. You could potentially use the extension to send pings to warm environments before a SHUTDOWN event. External extensions run as a separate process and might be able to use a combination of INVOKE events and internal timers to proactively warm functions that haven't received any recent activity.

Why would you use this?

You're probably thinking to yourself, "This seems quite complicated, why not just use the traditional Lambda warmer or optimize my functions to reduce cold starts?" If done manually, then yes, I agree that the implementation is overly complex and may not be worth it. However, if you automate this as part of a larger project, I believe it provides a smarter, less intrusive, and more cost effective approach.

In many of my use cases, I have Lambda functions that sit idle for several hours, then suddenly receive a burst of traffic. This often happens with low traffic administrative dashboards or webhooks for upstream services like Slackbots. That first cold start can be an awful experience, and if the system is outward facing, can sometimes be enough for the user to abandon the page altogether. For Slackbots, the initial response must be within a strict 3 seconds. Without warming, the application we traced earlier would almost certainly fail to meet that requirement.

Optimizing your functions is a path worth exploring, but there is also the question of effort versus reward. If I shave 200ms off a 3,000ms cold start, that's great, but it likely won't come close to having as much of an impact as using a warming strategy. If you do employ a warming strategy, I think it should be slightly smarter than just sending a scheduled warming ping.

When should you use this?

This works best when you have low traffic functions with prolonged periods of inactivity between invocations. Even if you have periods where there are no requests for 5-10 minutes, cold starts will almost certainly creep in. And since this solution maintains state, you don't have to worry about warming pings colliding with actual user requests during times with consistent traffic. If you have several warm execution environments, SHUTDOWN events simply won't send a ping.

Wrapping Up

This technique is experimental and definitely has room for improvement. As I collect more data, I hope to expand upon its capabilities and perhaps codify some more complete examples. If you have any thoughts or feedback, please be sure to reach out on X or LinkedIn.

Tags: #aws-lambda

Lambda functions have been a great choice for the vast majority of my use cases. They scale quickly to handle traffic spikes, and for asynchronous workloads, I never worry about cold starts. Even for synchronous workloads with minimal traffic, cold starts are mostly a non-issue. However, I have hundreds (maybe thousands) of synchronously invoked functions that receive infrequent and inconsistent traffic. Cold starts on those functions are not only painful to experience, but in some cases, can create upstream integration issues.

In this post, I'll outline a strategy I've been experimenting with that utilizes Lambda function lifecycle events to optimistically warm functions when there are no more active execution environments. I've been using this technique for a while now, and the results have been exceptional. Please note that this is not your standard trigger-with-a-scheduled-task Lambda Warmer that you've likely heard of in the past. This is a much more sophisticated approach designed to minimize costs, unnecessary invocations, and request collisions.

Traditional Lambda warming strategy

Many (many) years ago I created the fairly popular Lambda Warmer npm package. Its purpose was to optimize AWS Lambda function cold starts by implementing AWS best practices for warming functions "correctly". Those tips included:

• Don’t ping more often than every 5 minutes
• Invoke the function directly (e.g. don’t use API Gateway to invoke it)
• Pass in a test payload that can be identified as such
• Create handler logic that replies accordingly without running the whole function

A lot has changed in the last 6 years, but fundamentally, this technique still "works". The issue is that those warming pings happen every 5 minutes whether they're needed or not. This can lead to several unintended consequences.

1. Unnecessary invocations: If Lambda functions are already "warmed" by actual traffic, sending regularly scheduled warming requests adds extra costs, resource usage, and CloudWatch log ingestion without providing any additional benefit.
2. Blocked requests: Scheduled invocations will use a warm execution environment if one is available. It's possible that this can block actual traffic from using that warm environment and force a cold start on an end user's request. If you attempt to warm multiple Lambda execution environments, which requires intentionally delaying a response to force additional environments to spin up, you're highly likely to run into an end user blocking situation.
3. Cold starts due to early termination: Lambda function execution environments (as of this writing) are typically recycled after approximately 6 minutes of inactivity. However, AWS may decide to recycle your environments early (or later) for any number of reasons, including internal failures. This happens a lot. If a user invokes your function before the next warming window, they are guaranteed to get a cold start.

Despite the issues above, this method remains quite popular as it's fairly easy to implement and inexpensive to run.

What about Provisioned Concurrency?

At AWS re:Invent 2019, AWS announced Provisioned Concurrency, a feature that allows you to "pre-initialize" Lambda execution environments. The promise of this feature seemed like a perfect solution. You simply specify how many "warm" Lambda functions you want (you can even manage it on a schedule or based on utilization), and AWS will make sure you have a warm, autoscaling fleet ready to go. As your function starts receiving traffic, the AWS Lambda service continues to warm new execution environments (without blocking) up to your pre-configured maximum. If you exceed your maximum, then it rolls over to standard on-demand scaling.

If you have a function that requires extremely low latency and is expected to get predictable periods of sustained traffic, Provisioned Concurrency is definitely worth looking at. However, the cost of this feature seems wildly expensive compared to other compute options. For example, the current Lambda Pricing charges $0.0000041667 for every GB-second of Provisioned Concurrency. For a function with 1024MB of configured memory, that would cost ~$0.02 per hour to handle ONE concurrent request. That doesn't include the PER invocation charge and the duration charges.

There are msny benefits to using Lambda functions, most notably their wide array of Event Source Mappings (ESMs). ESMs make integrating with other AWS stream and queue-based event sources extremely simple and (mostly) headache free. But if you are using Provisioned Concurrency, it would almost certainly be for synchronous invocations like API requests, not for ESMs or other direct triggers like S3 and SNS. It depends on your use case and the amount of expected traffic, but in many cases, services like App Runner or ECS can provide auto-scaling containers that handle your traffic at a much more predictable cost.

For our use case, keeping at least one Lambda execution environment warm 24x7, would cost $10.80/mth (assuming 1024MB of configured memory). Also, that would be for EVERY FUNCTION we wanted to keep warm. Single-purpose Lambda functions have their uses, but at this price, you might want to think about building a MonoLambda (or Lambdalith).

Enter Lambda Tail Warming

Lambda Tail Warming is a technique that uses a Lambda Extension to track active execution environments by monitoring lifecycle events, then optimistically invoking itself to ensure at least one warm environment at all times.

The basic process is as follows:

1. Install an external extension that subscribes to INVOKE and SHUTDOWN events from the execution environment.
2. When a new execution environment is started, you register the extension and persist the extensionId in an external datastore such as DynamoDB or ElastiCache. The record should also contain a time-to-live (TTL) value that expires after several minutes (more on this later).
3. If an INVOKE event is received after some amount of time has passed (more on this later), update the datastore record with a new TTL.
4. If the extension receives a SHUTDOWN event, check the datastore to see how many active execution environments are running. If there are one or zero, use the AWS SDK to asynchronously invoke the Lambda function using InvocationType: 'Event' with a payload that indicates it's a warming event.
5. Remove the datastore record for current extensionId and exit the extension.

If you're unfamiliar with Lambda Extensions, AWS has a GitHub repository filled with several examples to guide you. External extensions can be written in a different language than the function, which means you can write this once and add it as a Lambda Layer to any function you want to warm. We'll look at a Node.js example, but you're free to adapt this to your preferred language. Let's build.

Building a Lambda Tail Warmer

This example from AWS shows the set up of a basic external extension. The extensions/nodejs-example-extension file is deployed to your Lambda function's /opts/extensions directory as part of the Lambda Layer or within a container (if using the container packaging format). This file is picked up by the Lambda runtime on initialization and will execute the index.js script in the nodejs-example-extension directory.

The example includes an extensions-api.js helper function that handles the interaction between the module and the Lambda runtime for us. We'll use its exported register and next methods in the main extension script.

Take a look at the sample extension script below. The main() function is executed immediately. It registers the extension using the register method, then enters a loop waiting and awaits the next event from the Lambda runtime (await next(extensionId)). When a new event is sent from the runtime, the script parses the event type and then either executes the handleShutdown or handleInvoke methods. If it's a SHUTDOWN event, we also need to gracefully exit the extension using a process.exit(0).

Now that we're familiar with how the extension handles lifecycle events, let's add our Lambda Tail Warming logic. Let's start by adding some variables to the top of our script:

Notice that I added a data import. This is referencing your own data module that would persist to the datastore of your choice. I provide some pseudo methods in the examples below to demonstrate the interactions. We also need the AWS SDK Lambda client. And I've moved the extensionId to the global scope (so we can access it later), created a variable to track when the extension was initialized (initTime) and set a refreshInterval for 10 minutes.

Next we create a new trackLambda function:

If the remove argument is true, we remove the extensionId from the datastore, otherwise, we upsert the record with the current time and a TTL value slightly greater than our refreshInterval. The reason we do this is twofold. First, we want this record to expire if it doesn't get cleaned up by a SHUTDOWN event. I've yet to see an extension not get a SHUTDOWN event, but network failures and other issues are sure to leave some zombie records. This is a proactive approach to cleaning them up if something fails. We add a few seconds to the TTL to give a refresh request time to update the record before expiring.

Now we need our warmLambda function:

Here we initialize the Lambda client, then send an invocation to our function (process.env.AWS_LAMBDA_FUNCTION_NAME conveniently stores the name for us). We'll use an InvocationType of Event as we don't want to wait for a response. Lambda Extensions get between 500ms and 2000ms of processing time to perform any cleanup (500ms when registering a single extension), so we want this to be as efficient as possible. There is no guarantee that the Lambda service will respond in time, but it hasn't been an issue in all my tests. Also, if the invoke request fails, we want to catch the error and log it. This will only be called when the container is being shutdown anyway, but we want to make sure any other processes have a chance to run in the extension.

We need to update our handleShutdown function:

When our handleShutdown method is called, we first check to see if this was due to a Lambda Function failure. If it is, we likely don't want to keep trying to invoke a function that is failing. Otherwise, we retrieve a list of active environments from our datastore. If there are one or zero active environments (we assume that one is the current environment being shut down), then trigger the warming event. Regardless of the number of environments, we want to make sure we clean up the datastore record of the current environment by calling trackLambda with the remove argument set to true. Then we gracefully exit.

Our handleInvoke function also needs to be updated:

Most invocations we can ignore since we don't want to update the datastore on every request. But every so often we want to send a heartbeat to the datastore to let it know an execution environment is still active. Above I'm checking to see if we've exceeded the refreshInterval. If we have, we update the datastore by calling trackLambda() and reset the initTime.

Finally, we can update our main extension loop logic:

I've updated the SIGINT and SIGTERM event listeners to support the async handleShutdown function just in case we don't get our standard SHUTDOWN event. I've also added a trackLambda() call before we enter the loop (this is the INIT phase). I'm not awaiting this as we don't want it to block the extension's initialization. Then I added awaits to our handleShutdown and handleInvoke function calls. And that's it. Your Lambda Tail Warming is ready to go. Well, almost.

Configuring your Lambda function for Tail Warming

Before you can start warming your function, you have to make a few changes to ensure that the process will work correctly.

1. Enable Invoke Permissions: In order for your extension to invoke your Lambda function, the function must have permission to invoke itself. You can add the following to your IAM role:

2. Add a "warmer" event interceptor to your handler code: In order for your Lambda function to respond as quickly as possible and avoid potential request blocking, you'll need to modify your handler code to short circuit requests. This will ensure that any complex logic (and potential mutations and logging) is not processed. For example:

3. Enable datasource access permission: In addition to the necessary Lambda Invoke permissions, you also need to make sure that the Lambda function has access to the datastore you're using to persist execution environment information.

How does this perform?

Below are the traces for a rather large Lambdalith that includes the canvas binary to generate and manipulate image files. The first trace is the typical cold start (~3 seconds) for this application. The second 200 request below is a typical warm invocation (< 60 milliseconds). The subsequent request is a Tail Warming event (note the 202 status code as this is happening asynchronously). This is a cold start (remember we do this immediately after the SHUTDOWN event of the last environment) which initializes a new execution environment and has a total duration of 3.122 seconds.

The next invocation shows a request from a user with a response time of 824ms. This application loads a significant amount of state, which typically happens during a normal user request. Our warming event short circuits this, which is why we see some additional response time during this invocation. It is possible to load state or perform some other actions during our warming event, but the goal is to find the right balance between cost and performance. The warming event saves us over 2.2 seconds of initialization time on the first subsequent user request and costs us only about 10 ms of billable time.

Other things to consider

There are several ways to improve this technique and make it more intelligent:

1. The refreshInterval could be tweaked to support shorter expirations or more frequent datastore heartbeats. I found 10 minutes makes the most sense as it prevents premature removal of datastore records when the app is handling real traffic from users.
2. It's possible that a failure (network issue, missed SHUTDOWN event, etc.) could prevent the warming event from triggering. This would break the cycle of warming events, meaning the next real invocation would get a cold start. You could mitigate this by periodically checking the datastore for active records or by attaching listeners (like using DynamoDB Streams) that would do something similar. This would be in a separate process (e.g. Lambda function), so it wouldn't block invocations to the primary function unless you needed to warm it. It would add to the cost, but if you were warming multiple functions, this would benefit from the economies of scale.
3. You can add controls to turn this on and off. Extensions have access to Lambda environment variables, which are a handy and low latency way to add state to every invocation without needing to redeploy or fetch external configurations. A function configuration update could be scheduled if you wanted to automate this.
4. You could hydrate the state on every warming event. As I mentioned before, the majority of cold start time is typically during the initialization stage. If it makes sense to load some of that state on a warming event, then feel free to.
5. You could combine this strategy with the traditional Lambda Warming technique. When you invoke a "warm" function, this will generally extend the duration of that execution environment. That includes maintaining any global state that has been loaded. As long as an execution environment stays warm, the Tail Warming will never fire. When the environment is eventually recycled, the Tail Warming would immediately invoke a new one.
6. You could potentially use the extension to send pings to warm environments before a SHUTDOWN event. External extensions run as a separate process and might be able to use a combination of INVOKE events and internal timers to proactively warm functions that haven't received any recent activity.

Why would you use this?

You're probably thinking to yourself, "This seems quite complicated, why not just use the traditional Lambda warmer or optimize my functions to reduce cold starts?" If done manually, then yes, I agree that the implementation is overly complex and may not be worth it. However, if you automate this as part of a larger project, I believe it provides a smarter, less intrusive, and more cost effective approach.

In many of my use cases, I have Lambda functions that sit idle for several hours, then suddenly receive a burst of traffic. This often happens with low traffic administrative dashboards or webhooks for upstream services like Slackbots. That first cold start can be an awful experience, and if the system is outward facing, can sometimes be enough for the user to abandon the page altogether. For Slackbots, the initial response must be within a strict 3 seconds. Without warming, the application we traced earlier would almost certainly fail to meet that requirement.

Optimizing your functions is a path worth exploring, but there is also the question of effort versus reward. If I shave 200ms off a 3,000ms cold start, that's great, but it likely won't come close to having as much of an impact as using a warming strategy. If you do employ a warming strategy, I think it should be slightly smarter than just sending a scheduled warming ping.

When should you use this?

This works best when you have low traffic functions with prolonged periods of inactivity between invocations. Even if you have periods where there are no requests for 5-10 minutes, cold starts will almost certainly creep in. And since this solution maintains state, you don't have to worry about warming pings colliding with actual user requests during times with consistent traffic. If you have several warm execution environments, SHUTDOWN events simply won't send a ping.

Wrapping Up

This technique is experimental and definitely has room for improvement. As I collect more data, I hope to expand upon its capabilities and perhaps codify some more complete examples. If you have any thoughts or feedback, please be sure to reach out on X or LinkedIn.

Tags: #aws-lambda

I like to write, but I don't often make the time. I spend a far greater portion of my time reading. Unfortunately, professionally copy-edited, well-written books and articles from the likes of O'Reilly and the New York Times are only a small part. The vast majority consist of technical blog posts, likely half written by ChatGPT, or inane chatter on X without punctuation, proofreading, or purpose. I don't expect the latter to be polished. Fleeting thoughts rarely are.

Time is finite

When we're young, time seems infinite. One of my teenage daughters will sleep half the day away if you let her, the other will FaceTime or Snap for hours without ever thinking about time as a currency. As I've gotten older, I've come to realize that time is my most precious commodity. It's yours, too. Perhaps you already know this. Perhaps you're too young to realize it. Someday you will. Hopefully before it's too late.

Like all precious commodities, I hate wasting time. Even more so, I hate having time stolen from me. Two years ago I resolved to avoid watching commercials on TV and online as much as possible. I even went so far as to pay for YouTube Premium and start DVRing live sporting events and the news. The purpose wasn't to ruthlessly maximize every second, it was to recapture all those wasted minutes that rarely (if ever) added any value to my life. I take plenty of time to rest, exercise, and spend time with family and friends. That is never time wasted.

The reader is in control

I read well over 150 articles per week curating content for the Off-by-none Newsletter. Most are "readable", some are extraordinarily well-written, and all are appreciated. It takes a significant amount of time to write a post, plus a mustering of courage to put oneself out there to be judged. But I assume most of you neither have my motivation nor my acquired level of patience to filter through all the noise. Why would you?

As a consumer of content, you are ultimately in control. If the headline doesn't interest you, you don't click the link. If the first paragraph doesn't grab your attention, you hit the back button. If the prose is hard to follow and makes your head hurt, you rightfully abandon the article. You deserve good writing. Your time is valuable. Don't let anyone tell you otherwise.

The curse of AI

Despite what you may have heard, LLMs (Large Language Models) are not particularly good at writing compelling prose. Passable? Sure. The grammar is almost always spot on and much better than that of the average writer's. But read it again. It's most definitely not human.

I know you're not fooled by it. If you're anything like me, you certainly don't enjoy reading it. But this early fascination with "passable" prose has spread AI generated content across the Internet like a virus. It sits there, waiting to be read by both a new generation of humans and the next generation of AI that will train itself on its previous iteration's mediocrity.

I worry about the future of prose. I worry about teaching humans to write like machines. I worry that most, especially non-native speakers, will eventually not know the difference.

How I've approached writing

The first sentence I ever wrote was likely scribbled on a mimeographed piece of paper sometime during the mid 1980s. I still remember the comically large rulings with dashed midlines to properly proportion lowercase letters. Since then I've had over 16 years of formal education, reinforcing my writing skills at every level. I took both creative and formal business writing courses all through university when earning my degree. And over the last 20+ years of my professional career, I've written countless correspondence, business proposals, marketing copy, blog posts, and more.

Even after all that, I STILL WRITE BY INSTINCT.

Of course I copy edit my own work. I check my grammar. I always check my spelling. I usually even ask someone else to review my pieces and give feedback. Yet while I'm generally a perfectionist, I'm not overly critical of my own writing. If it sounds good and seems to flow well, then that's usually good enough for me. The question I've been asking myself lately: Is it good enough for you?

A higher personal standard

I spent some time rereading several of my old posts. The writing is mostly fine. Not great, but fine. Like most writing, nearly the entirety of my sentences are imperfect. Many are informative, but lazy. In some cases they form a string of volunteer sentences that do little more than fill space until I finally find something interesting to say. Even then I likely belabor the point as if the reader is incapable of rational thought. Most of my sentences are unnecessarily long, complicated, and hard to follow. Had I stated things clearly in the first place, some of these torturous restatements wouldn't be needed.

Perhaps I sound overly critical. After all, time is a finite, scarce resource. How much of it should be spent on meticulously crafting prose versus simply putting information out there? Maybe good enough writing is good enough? I certainly understand and appreciate the thinking behind that. I also don't want to discourage others from sharing their writing. I may challenge your ideas, but would never be so rude as to criticize your prose.

But I want to do better. I want to hold myself to a higher standard.

Honing the craft

I've never considered myself a writer. I've always just looked at myself as someone who writes because they have to. To me, there is a very real difference. While everyone should work to improve their writing and communication skills, a writer should dedicate themselves to honing their craft. They should embody their work. It should become part of their identity. It should be their purpose.

This is something I aspire to. The ability to write clear and compelling testimony of the world around us is a superpower. I envy those who can, and I want to challenge myself to do the same.

Lately I've been reading several books on writing while also restudying the nuances of English grammar. The grammar review is helpful in thinking more actively about concepts like balancing parallel ideas with the correct grammatical form and avoiding the subordination of major ideas. It also reminded me how often I've failed to use a semicolon when joining two independent clauses with a conjunctive adverb. But I suppose I shouldn't lose sleep over that.

Books on writing are even more interesting. Analyzing sentence structure, not for grammatical correctness, but for rhythm, context, and understanding what's actually being said is fascinating. Even reading about things such as word choice, when to invert a sentence to add variety, and how to balance sentence length to keep readers engaged, provides a renewed awareness of how important these things are.

I have no illusions of writing a perfect sentence, let alone a perfect piece. All I can do is work everyday to make myself a better writer. I can be more critical of my own writing to ensure that it's worthy of your time and attention. I can't promise I'll become a better writer, but I have every intention of trying

Comments are currently disabled, but they'll be back soon.

I've been blogging since early 2015 (almost 9 years now!), and like most folks, I started by deploying to WordPress. Being the AWS power user that I was, I thought it would be a great idea to host it myself on EC2, set up an RDS cluster, and be responsible for everything.

When I had a post end up on Hacker News, my little EC2 instance fell over very quickly. So I decided to add a load balancer and spin up multiple EC2 instances with auto-scaling configured just in case lightning struck again. This setup has saved me several times since, but at the cost of an inflated AWS bill and a tremendous amount of unnecessary complexity.

Static + Serverless is the way

Serverless was just becoming a thing back in 2015, so it wasn't really an option at the time. However, as it's matured over the years, I've been wanting to convert my blog into something much more serverless. There have been fits and starts over the years, but I always found myself worrying about the power I'd lose by moving away from something as mature as WordPress. The vast collection of plugins available in the WordPress ecosystem meant giving up features, manually connecting other third party services, or rebuilding them yourself. At the end of last year, I finally decided to make the leap.

Several years ago I discovered Eleventy. It's a "simpler static site generator" that's 100% JavaScript-based, incredibly flexible, and supported by a growing community. After playing around with it for a bit, it became my favorite go-to option. I used it to build sites for the Off-by-none newsletter, then the Serverless Chats Podcast, and most recently, Ampt and its documentation.

I get that static sites (SSG) can be a bit limited, but that is largely a function of where they're hosted. Both Off-by-none and Serverless Chats are currently hosted on AWS Amplify (the hosting service, not the framework). It's been a great option. Fast, reliable, and inexpensive. However, it's a bit limited when it comes to adding dynamic functionality without going all in on the Amplify Framework (which I don't want). I know there are plenty of PaaS providers out there, but I want my projects to be backed by the reliability and operational excellence of AWS.

When we built the new Ampt website, we decided to dog food our own service, and deploy the Ampt site to AWS with Ampt. Becoming our own customer was not only a great way to ensure service quality, but it also gave us the unique ability to remove all the unnecessary friction when deploying websites to AWS serverless services. And we did some pretty cool things!

Why Ampt?

Obviously I'm a bit biased, but my experience building and managing the Ampt website has convinced me that doing it any other way is simply a form of masochism. Ampt's goal has always been to deploy workloads to AWS using the best possible "patterns" available, and more importantly, to automatically select and manage those patterns on behalf of its users. Some patterns are easy to implement and manage, some are really hard.

The pattern we use for static asset routing falls into the latter category. We use a combination of Amazon CloudFront, Edge Functions, S3, Lambda Function URLs, and more, to allow dynamic routing to both static and on-demand resources. This means you can do really cool things like statically serve some pages while dynamically generating others. You can serve your API routes from the same domain as your static pages without needing to explicitly set origins in your CloudFront distributions. And you can do this without writing a single line of configuration.

If you want to use frameworks that support Server-Side Rendering (SSR), like Next.js, Nuxt, Astro, Remix, SvelteKit, and others, Ampt fully supports those as well. You can even mix your backend workloads into the same app, giving you all the power of AWS and Ampt running alongside your fullstack JavaScript application.

What's next?

If you're reading this in early January of 2024, you've probably noticed that this site is a bit of a mess. I wrote a script to convert all my old WordPress posts to markdown files, did some clean up, and put together a simple template layout. It needs a lot of work and is missing a number of things. I knew that if I waited until it was "perfect" that I'd never publish it. So I present it to you in all its messy glory.

"Perfect is the enemy of good"

I'll be working to polish this when I have time, but I'm much more interested in writing more, so no promises. I hope it's at least readable for now, and I look forward to sharing progress as I make it. Your feedback is always welcome and appreciated!

Tags: #eleventy, #ampt, #serverless

Comments are currently disabled, but they'll be back soon.

I love #serverless, but IMO, it got something dramatically wrong.

It too closely coupled the underlying cloud infra with your app & business logic. This not only creates a bifurcation of business logic akin to Stored Procedures, but in many cases, ties the performance of your…

— Jeremy Daly is building @amptdev (@jeremy_daly) September 15, 2023

Serverless too closely coupled the underlying cloud infrastructure with your app and business logic. This not only creates a bifurcation of business logic akin to Stored Procedures, but in many cases, ties the performance of your application to the architectural decisions you’ve baked into your app code.

Rather than deploying apps to environments designed to support them – maintained, monitored, and optimized by platform engineers, serverless requires “developers” to understand the impact of choosing one statically configured primitive over another.

This *is* hard. The expectation that any one person can understand the complexities and nuances of several AWS (or other cloud providers’) services, while also being a competent Java, Rust, Python, etc. developer is naive at best. What you don’t know you don’t know is MUCH worse than what you know you don’t know.

I interviewed a 22 yo kid one time that said he was an EXPERT at Microsoft SQL Server. When I asked how long he had been using it, he replied, “Almost a year!” ?‍♂️

This isn’t only an incredibly stupid thing to say, it’s extremely DANGEROUS! You’re just one misconfigured Retention Policy, public S3 bucket, or IAM “*” permission away from something very, very bad happening. And this is why Terraform dominates the market, because most devs probably SHOULD NOT be trusted to deploy whatever they want to the cloud.

Platform engineering has been around for quite some time now (despite the recent buzz), and that’s because organizations are trying to allow their developers to be more productive, but in a way that doesn’t compromise security, performance, and cost.

There are plenty of Infrastructure As Code (IaC) tools that shift these burdens onto the developer. Lots of them are very good for those developers looking to move towards cloud architect roles (which are definitely needed in modern organizations). But all of these tools presuppose that developers actually want this! Or perhaps, even more naively, assume that working developers that don’t spend all day on Twitter (or X) actually have the time to invest in learning new services and digesting the latest feature added to Lambda (and how it could impact/help their customers).

I will reiterate: I LOVE #SERVERLESS. I think that deploying applications using serverless technology when possible is the fastest way to build highly resilient, global scale, virtually management free solutions. But upskilling an organization to get to that point can take years, succeeding only after running hundreds of experiments and having suffered countless failures.

I believe that the vast majority of developers just want to write code, and more importantly, solve their users’ problems. But developers also need to be able to harness the power of the Public Cloud. In order to do this, a radically different approach is required.

This is why we built Ampt. Not because we’re trying to restrict developers, but because we’re trying to FREE them. If you’re curious what this is all about, be sure to join our Launch Day livestream on September 20, 2023.

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

This content is password protected. To view it please enter your password below:

Password:

Tags: caching, cloud, momento, mongodb, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

More control is a good thing, right? I mean, who doesn’t want the ability to twist every knob and tweak every settings to your heart’s desire so that each minute detail of your infrastructure is artisanally handcrafted? I’ll tell you who: this guy! Why? Because I have neither the time, energy, nor expertise to even begin to understand the impact of most of those configuration changes. Not only that, but I likely don’t even have the information needed to understand which optimizations are required, let alone a way to prioritize them.

Now, I’m not saying that I don’t want any control. There are clearly choices we developers must make, right? Right? Like, which programming language we write our applications in obviously, or whether or not we want to use Kinesis instead of SQS? Maybe how long we want our Lambda function timeouts to be, or even better, the amount of memory we want to allocate to each function? There are clear reasons to choose API Gateway REST APIs over API Gateway HTTP APIs, but what about when Function URLs make more sense? Personally, I love manually configuring SQS redrive policies. ?

A smarter glue ?

Okay, maybe these examples seem extreme now, but Simon Wardley’s recent post on conversational programming predicts a future where the vast majority of these decisions are no longer the developer’s. As he says, the interface (text or voice) doesn’t much matter, but the point is that beyond business logic, everything else is likely just wasted energy repeating the things that have already been done.

Serverless has shifted a massive amount of undifferentiated heavy lifting off of application builders, but as he says, “Even in this serverless world, the act of programming still requires you to think about what component services need to be glued together.” He goes on to say, “That is still a lot of work to be done and to be blunt, it’s work that can mostly be automated and achieved through some form of intelligent compiler.” I couldn’t agree with this more.

Control is an illusion ?

There has been a rapid evolution in automation and systems intelligence fueled by the meteoric rise of public clouds. For those of us that used to rack servers and configure firewalls, control was required. But every step along this evolutionary journey has seen more and more of this control become not only unnecessary, but it actually holds you back. This pattern has repeated itself over and over again. In tech, many people’s identities are defined by their sense of control, which makes inertia to accept the inevitable even harder to swallow.

Most of us don’t malloc memory anymore. Many of you probably never had to in the first place. Today, a huge majority of builders don’t spin up and configure VMs, or suffer through endless iterations of nginx config roulette. We should be embracing that. Now the question is, what will we not be worrying about tomorrow?

Tags: cloud, devops, managed services, server management, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

It wasn’t that long ago that the vast majority of developers didn’t worry much about infrastructure. Sure, there were plenty of us configuring Linux servers and setting up the occasional MySQL database, but that certainly wasn’t the norm. For those that worked in larger organizations, your code was likely checked into perforce or subversion, and then magically ended up in production (some days, weeks, or even months later). For many, this is probably still how they ship code.

During the late 1990s, web hosting companies sprung up and allowed a new breed of “web developer” to push code directly to servers accessible to the Internet. This mostly involved FTPing files to a single web server, often shared, with almost zero ability to configure anything beyond a few Apache overrides with a .htaccess file. When cPanel became widely available, suddenly developers had a bit more control. One click installs of MySQL and php, and suddenly there was an explosion of new use cases now available to a rapidly growing generation of developers.

Then came Drupal and WordPress, opening up web publishing to an entirely new population. The search wars were raging with Yahoo! and Google vying for the top spot (Lycos and Altavista were already history at this point). Suddenly Friendster and MySpace introduced a new type of online social interaction driving massive engagement and more demand. The proliferation of data centers followed with more and more companies shifting their resources to grow their businesses on the backbone of the Internet. Computing became global and the days of the single server were over.

If you wanted to be a serious player, you needed to buy (lots of) servers, rent colocation space, pay for a large amount of mostly unused bandwidth, license software, and become experts in operations. This was business critical and staffed with specialists whose job it was to keep the servers running. This wasn’t easy and it wasn’t cheap, but the world had changed. There were plenty of WordPress and Ruby on Rails sites still out there (heck, there still are), but those looking to grow a business beyond a hobby were learning that high availability was the new standard, and they were falling behind.

Renting dedicated servers was nothing new, but when AWS launched EC2 in 2006, suddenly the overhead of running your own servers virtually disappeared. “Elastic compute” rolled up thousands of dollars in monthly costs into a per hour charge. When I moved my hosting company from a colocation facility to AWS in 2009, my monthly cost dropped by over 80%. Managing EC2 instances wasn’t particularly easy at first, but it enabled the next generation of hyper-scalers to build without limits, taking advantage of the massive growth of the nascent mobile web sparked by the success of the iPhone.

Cloud architectures were still relatively simple. Autoscaling VMs with a load balancer and a few key services like S3 and SQS filled in the gaps. Developers were still mostly throwing code over the wall. Then CloudFormation (and others like Puppet and Chef) kicked off a new wave of automation, spinning up VMs and infrastructure with easily repeatable workflows. Then came the deluge of cloud services. DynamoDB, Kinesis, Lambda, API Gateway, Step Functions, and more. We now moved beyond multi-VMs to multi-service. Applications changed from horizontally scaled VMs connected to a database cluster to multiple services stitched together.

It happened over the course of a few years, but suddenly developers were no longer just being asked to use the AWS SDK to connect to a DynamoDB table or install the Kinesis Client Library. They were being asked to deploy, configure, and set the permission to connect to them, too. Whether you were writing monoliths or SOAs or microservices, your code was no longer deployed to a cluster of servers, but split into smaller parts that needed to know where it was running, from where and how it would be executed, the permissions it needed, and the special format it was required to return.

As soon as developers began to get comfortable with that, there was another shift. Now instead of writing code, you needed to start encapsulating business logic and processes into cloud specific configurations, connecting services together directly rather than using familiar instruction sets that we all have ingrained so deeply within ourselves. Configuration over code, because, yes, your code is a liability and the cloud can do it better than you.

Over the course of 15 years or so, developers became responsible for more than just business logic. Infrastructure and cloud architecture, build and deployment pipelines, instrumentation and observability, security and compliance, plus a whole lot more now fall directly on the shoulders of your modern day “cloud developer.” I don’t think it’s possible for one person to master all of these disciplines. I’m not even sure they can effectively be spread across a team of developers without really good communication and strict review processes in place. And even then.

We’re at an inflection point. The explosion of new tools and platforms trying to abstract away this complexity and reduce cognitive load on already overtaxed developers should be a flashing neon sign. I’m also not alone in thinking that AWS can’t solve this. Tools like AWS Application Composer and Amazon CodeCatalyst are all fine and good, but all they do is hide a layer of complexity that you’re still ultimately responsible for. AWS is really great at operational excellence on the service level, but only if you’ve properly wired everything together.

I’m not sure anyone has found a good answer to this just yet, nor do I think a silver bullet can exist. But given the increased amount of responsibilities the average cloud developer now has, I don’t blame us for seeking something better.

Tags: cloud, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

There have been many, many, many, many things written and said about ChatGPT. Lots of which were probably created by (or with the assistance of) ChatGPT. For example, I find the impact on school writing assignments particularly interesting (I have two teenage daughters that are no strangers to technology). But on #tech Twitter, one of the biggest concerns seemed to be about whether or not this technology can write full blown applications and replace the need for developers altogether. It’s an interesting question.

Developers need to remain self-aware

As I said, there is plenty already written on this. Forrest Brazeal’s excellent post asked, “When programming is gone, will we like what’s left?” He argues that there will likely be a shift in job functions, and that maybe “just good enough” code might be all upper management really cares about. The “fast and cheap is better than hand-crafted” strategy has been played out time and time again across many an industry. Then again, he also points out the fact that the speed of writing code is usually not the bottleneck for large organizations, and full blown or assistive AI is unlikely to make a huge impact.

Dan Conn bluntly asked, “Will ChatGPT replace developers?” He’s a bit more optimistic, pointing to the fact that not only is the technology unable to understand certain nuances, it’s also quite often flat out wrong. I fully agree with Dan’s points in the article, but I’m concerned that others won’t necessarily see it this way. It makes me think of the movie Idiocracy where people in the future blindly trust technology that they no longer have any idea how to build or control. There will be more and more tools built around this where people will be inadvertently putting the wrong probe in their mouth. And they won’t know any better.

Learning from itself might be the least of its problems

I get that AI is rapidly evolving and will continue to learn over time. But how much of the input for future learning is going to be from content and code samples it generated? Dan’s article mentions the Stack Overflow ban on ChatGPT responses. This was because, yes, some of the information was wrong, but the bigger problem is that ChatGPT probably used Stack Overflow and similar sites to train itself in the first place. Maybe it’s smart enough to detect its own (or other AI’s) potentially inaccurate drivel? But maybe it’s not? That seems like a vicious cycle.

It also seems as though it can easily be manipulated. A recent NY Times column titled, “A Conversation With Bing’s Chatbot Left Me Deeply Unsettled,” is clearly a bit concerning. Sure, I get that maybe this guy was intentionally pushing it, but considering that this isn’t the first time Microsoft had some trouble with AI chatbots, it doesn’t come as a surprise. There’s been other people posting Bing ChatBot threats as well. I mean, if Elon Musk is concerned, then shouldn’t we all be? ?

While turning a Chatbot into a racist stalker pressuring you to leave your wife certainly isn’t good, I’m much more concerned about chats gone awry with those that are much more impressionable. This Verge story say it’s an emotionally manipulative liar, and people love it. Personally, I don’t find that particularly amusing. A recent study found that 30 percent of teenage girls said they seriously considered attempting suicide. At least Google immediately directs you to a suicide prevention hotline and inundates you with links to resources to get help. I’m sure these chatbots will have some filters that catch a lot of these keywords, but I don’t think it’s a stretch that an extended conversation that “can lead to a style we didn’t intend” turns into a machine helping a distraught kid rationalize hurting themselves or someone else.

AI is another tool, and it’s here to stay

Kirk Kirkconnell wrote a post on using ChatGPT to assist with content creation by generating prompts for you. I actually thought about doing this as well, but found them to be overly generic at best. This is at least better than the plethora of blog posts we’ve already seen written by ChatGPT, most of which are very convincing. I honestly can’t be certain that an AI generated (or assisted) article hasn’t made its way into my newsletter. It’s easy to be fooled.

I’m not an old man yelling at a cloud. I do believe that ChatGPT/AI can and should assist with lots of things, including code suggestions, writing test, answering (certain) questions, and even helping to eventually optimize complex cloud workloads. Let’s just not lose sight of what separates humans from machines. Otherwise we might all be left with a really bad taste in our mouth.

Tags: cloud computing, programming

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

Several years ago I wrote a post asking people to stop calling everything serverless. I even gave a keynote at Serverless Days Milan the following year pleading the same message. My contention was quite simple: “when everything’s serverless, nothing will be.”

Back then, “serverless” was still relatively new, and the possibilities were seemingly endless. Sure, there were a few people starting to mislabel things, and of course, haters were gonna hate, but for the most part, the argument was less about the nuances of the technology and more about the “nature” of serverless and the serverless-first mindset. But then something changed.

What started as a few basic principles of “serverlessness” slowly morphed into something that seemed less serverless. For those of you that don’t remember, AWS initially pitched the four main benefits of serverless as (1) No server management, (2) Flexible scaling, (3) High availability, and (4) No idle capacity.

As the screenshot above shows, “No idle capacity” meant:

“You don’t have to pay for idle capacity. There is no need to pre- or over-provision capacity for things like compute and storage. For example, there is no charge when your code is not running.”

A year later, serverless suddenly only had THREE main benefits. I bet you can guess which main benefit didn’t make the new list. Now, there are plenty of reasons for this. The timing of the Aurora Serverless v1 launch was right around then, and even though v1 could actually scale to zero, the idle capacity language wouldn’t have held up.

Then, all of a sudden, right before re:Invent 2018, it had FOUR again. But that fourth benefit is now “PAY FOR VALUE“, which means:

Pay for consistent throughput or execution duration rather than by server unit.

Just a few days later they announced Amazon DynamoDB On-Demand where you only pay for what you use (plus storage). All of this made sense, and while perhaps that dream of “no idle capacity” had slipped away, I was willing to make the tradeoff for all the added benefits.

The problem is, it didn’t stop there. As the old adage goes, “if you give a mouse a cookie, he’s going to ask for a glass of milk.” Once we accepted that tradeoff, the serverless community collectively gave AWS and others an entire dairy farm.

We then got “serverless containers”, Provisioned Concurrency, RDS Proxy, and more. Aurora Serverless v2 came out and practically no one accepted that to be “serverless”. Now we have AWS Neptune Serverless that, forget about scaling to zero, costs over $290 USD per month to run at its lowest capacity. And no, the free tier doesn’t count.

So, have we strayed so far from the purest definition of serverless that there is no going back? Or is this just what “serverless” is now? I hate to be the bearer of bad news, but somewhere along the way our compass broke, and we’ve strayed quite a ways off the path to the promise land. Then again, maybe it never existed in the first place?

Tags: cloud, cost optimization, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

Tech Twitter was all abuzz recently after DHH boldly proclaimed and explained why [37signals is] leaving the cloud. A lot of people cheered, some of us jeered, and everyone else just pitched web3 as an alternative solution. DHH’s success has earned him a giant platform and a tremendous amount of influence, and while I often disagree with him, it’s clear that many others do not. I spent quite a bit of time reading through all the retweets, reposts, comments, and hot takes, and I came to a fairly simple conclusion: these people are using the cloud wrong.

There is much to get to here, but let’s start with this quote from DHH’s article:

We’ve run on bare virtual machines, we’ve run on Kubernetes. We’ve seen all the cloud has to offer, and tried most of it.

I’m just going to come right out and say it: if you’re running VMs or Kubernetes in the public cloud, you’re almost certainly doing it wrong. Of course “the savings promised in reduced complexity never materialized,” because you’re literally just renting hardware and managing it yourself. You can’t reduce complexity if you’re unwilling to hand any of that complexity off to the cloud provider. Using the cloud like it’s just a datacenter for VMs is not, was not, and never will be its true purpose. The vision has always been to change the way we build and run applications, not to poorly scale the old way of doing things.

In the cloud, we should be renting services, not servers. Meeting customers where they were/are was a brilliant strategy for getting those early cloud adopters to trust the ecosystem, but now it’s become a handicap that signals acceptance of a practice that should have gone extinct years ago. As I’ve said, the cloud won’t reach its full potential until we start building apps that are designed for the public cloud. Otherwise, people will continue to fail and fall victim to DHH’s ignorance.

One of the best rebukes of DHH’s post was by Simon Wardley in the following Twitter thread:

X : Did you see "Why we're leaving the cloud" – https://t.co/bHGzZzTdny Me : Nope. X : And? Me : Why are you talking infrastructure clouds to me? This is 2022. You should have migrated to severless years ago. I'm not interested in niche edge cases.

— Simon Wardley ❤️?? (@swardley) October 22, 2022

If you haven’t read it, you really should. It points out a number of factors that DHH conveniently doesn’t account for, for example:

Last time I ran numbers on this (circa 2010) then hardware and software is normally around 20% of the cost. The rest is made up of electricity, cooling, building, physical security, racking, UPS, opportunity cost … long list.

I was in the same boat back in late 2000s. The cost of servers and software was minimal compared to colocation space rental and its associated utilities. Not to mention that running everything in a single datacenter is a terrible idea. With serverless services, you get multiple availability zones for free, plus all those other things Simon mentioned, included in your per millisecond or invocation cost. Also, has DHH given any thought to sustainability, or did he ban that topic too?

Of course it’s more expensive if you don’t account for your Total Cost of Ownership (TCO). Hiring experts capable of providing the operational excellence of the cloud is certainly possible, but if you think you can get that for less than $500k/year, you might want to check your math.

So maybe it’s just about control? Or maybe it’s just about the controversy? Either way, as interesting as Basecamp and HEY are, the likelihood of them needing to create specialized services to run a SaaS app and an email search service is highly unlikely. And this is where Simon really hits the nail on the head:

The largest delusion is usually the number of people thinking that they’re one of the special cases / niches.

37signals could be a special case, though I highly doubt that they are. They, and millions of others, continue to abuse the cloud every day by treating it like a virtual colocation facility. And, yes, Amazon, Azure and Google are printing money, so they’re happy to be complicit. I doubt DHH’s decision will move the needle unless they can prove they made the right choice. But until we realize the full potential of cloud, it’ll continue to cast doubt in the minds of the people it will most benefit.

I had some fun by writing a parody of DHH’s post: Why we’re leaving the electric grid.

Tags: cloud computing, cost optimization, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

I’m a huge fan of Charity Majors, and I always learn something new when she shares her thoughts on anything that has to do with cloud and engineering. So when I came across this Twitter post, I grabbed some ? and got ready to have my mind blown. I was not disappointed.

A excellent thread about platform engineering, the limits of marketing, and the primacy of context-specific solutions.

Platform engineering is simply not a product you can package and buy, and to the extent we reduce it to one, it *loses* what makes it most valuable. https://t.co/ekkC8c1SPu

— Charity Majors (@mipsytipsy) October 12, 2022

If you need a primer on this “new” field, check out Platform Engineering: What Is It and Who Does It? We should start by saying the concept of “Platform Engineering” certainly isn’t new. Netflix has been talking about it since 2017, and plenty of other companies have been building internal developer platforms for years. But suddenly, it seems to be all the rage. I think this is mostly because of good marketing, and some emerging companies that are starting to sell this as a “product”.

There is a lot that you can learn from digging through Charity’s Twitter thread. She also wrote an excellent blog post with the premise that The Future Of Ops Is Platform Engineering. This gives a much more nuanced take on the evolution of DevOps and where the role of Platform Engineers and Ops Engineers differ. So, I don’t want to spend too much time on the broader topic. However, what is interesting to me is how this concept applies to serverless, and maybe more importantly, companies that embrace the serverless-first mindset.

Can’t we just build our own internal serverless developer platform?

When I think about serverless-first companies that have created internal developer platforms, the two examples that always come to mind are Liberty Mutual (with their Serverless Enablement Team) and Lego (with their Platform Squad). Now, I haven’t kept tabs on how these teams have evolved, but when they first started, their missions seemed to be quite clear:

Provide developers and teams with the resources they need to quickly deploy well-architected serverless applications with the appropriate guardrails in place.

This made a lot of sense to me, and considering that the vast majority of services they used were “serverless”, one might think that the amount of “Ops” required would be dramatically reduced as well. We’re not talking #NoOps here, but offloading much of that operational complexity frees up resources to work on things that move the business forward and help developers deliver better software faster. I know both teams faced all kinds of implementation and adoption challenges, which is to be expected. But even though their approaches were very different, the successes they found were because smart, specialized teams (with C-level support) iterated until they found what worked for their companies.

But here’s the thing. The vast majority of companies don’t have the resources (or foresight) to hire dedicated teams to build internal developer platforms. Most of that responsibility falls on the developers, who end up spending a large portion of their time learning and configuring cloud infrastructure. As these teams grow, they quickly find themselves with a patchwork of tools, configuration files, and deployment pipelines, all heaped into a giant mountain of technical debt. I don’t think this is particularly unique to serverless, but I think it’s exacerbated by the paradigm shift.

Shifting left makes guardrails harder

Serverless lets us build faster than ever. It makes things that seemed completely impossible just a few years ago completely possible today. But just as easily as I can spin up a globally distributed, highly-available API backed by one of the most scalable databases ever built, I can also just as easily DDoS my own application with a recursive Lambda function. As I said before, you have the ability to make very bad choices when setting up serverless (or any cloud) infrastructure. It would be nice if we could enable sensible guardrails (that don’t involve hiring a platform team) as well as reduce the overall complexity and cognitive load, all without taking away too much flexibility.

When using serverless cloud infrastructure, the code a developer writes is often directly tied to the primitives it uses. If you’ve ever looked at a complex serverless application’s architecture diagram, you’ll likely see a myriad of managed services glued together by multiple functions, queues, streams, event buses, state machines, and more. Not only does the developer need the ability to provision these services and connectors, but their code needs to be written in a way that appropriately interfaces with them. It’s hard to put guardrails on this type of required flexibility without blocking the developer at every turn.

The other reason it’s hard to limit serverless developers is because of this idea of “configuration over code.” When a Lambda function fails, or timeouts, or throttles requests, you can’t write defensive code to handle those exceptions. The cloud has to do it for you. And the only way to tell the cloud how you want to handle it is through configuration. This creates a necessary bifurcation of your business logic that developers must be able to control. Take this a step further and add VTL mapping templates to API Gateways for direct service integrations, configure resolvers in AppSync, or build entirely functionless workflows using Step Functions. Put too many blockers in place, and you’ll kill developer creativity and productivity.

Finding the right balance

As Charity said, “If you draw the line a little too far to the left, you won’t be able to support enough product differentiation to succeed. A little to the right, and the maintenance costs will drown you and put you out of business.” I’m not entirely sure where that line is for serverless applications, but I whole-heartedly agree that generic solutions likely aren’t the answer. Right now, it has to be much more nuanced and specific to the organization.

Charity also said, “The beautiful thing about infrastructure is that there comes a point where you can stop treating infrastructure code like *code*, and instead treat it like trusty little black box building blocks that nobody has to waste any cognitive capacity on.” That sounds like the holy grail to me, and I think it’s even more possible with serverless given the number of patterns that have emerged over the years.

Of course, there are still the questions of control, responsibility, and who owns what. How much of the platform can we abstract away into these “little black box building blocks” knowing that there are hidden complexities beyond our control, yet still critical to our workloads? What happens when we reach their limits? What happens when they fail, as everything does all the time?

This is a hard problem to solve. I know because my team has been working on a solution for almost 2 years now. But ultimately, I think the goal should be to bring the full power of the cloud directly into the hands of the everyday developer. That requires both guardrails and guidance, as well as the right abstractions to minimize cognitive overhead on all that undifferentiated stuff.

I too worry that a “platform engineering” product could get this wildly wrong, limiting developer creativity in any number of ways. But at the same time, I think that cloud complexity has made it very difficult for companies to do this on their own.

Tags: platform engineering, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

I was intrigued when I first saw the announcement of AWS SAM Serverless Connectors. I don’t use SAM very much (if at all anymore), so it wasn’t the hope of this being some sort of silver bullet for my occasional IAM frustrations that got my attention. Rather, it was another opportunity to learn how AWS is trying to abstract away their mostly self-imposed complexity problems. Unfortunately, I think they missed the mark on this.

Finding the right level of abstraction is hard. Us developers are a finicky bunch, and we know what we like, and like what we know. Providing too high a level of abstraction will turn most of us off, but providing too low of an abstraction could mean introducing new complexity that outweighs the benefits gained. Case in point: AWS::Serverless::Connector. The motivations are pure, and the simplistic idea of a Destination, a list of Permissions, and a Source make total sense. The problem is that the services it connects are much too varied to let this abstraction serve as a universal connector.

Let’s start with this basic example from the docs:

So as long as you are in a single SAM template, an Id for the Source and Destination are all you need to enable “Read” and “Write” access to your DynamoDB table. Seems simple enough. Except, as Allan Chua points out, it takes 9 lines of code versus 3 lines using AWS SAM policy templates. Verbosity isn’t always a bad thing, especially if it provides clarity that makes reasoning about your IaC easier, but I don’t think that applies here.

To be clear, I’m not a huge fan of policy templates either (I’ll explain why in a minute), but I do like that they are at least organized within the function configuration. Serverless Connectors can be anywhere within your SAM template. This means that in order to know which services your function has access to, or potentially which services have access to your function, you need to search out a disconnected piece of configuration drivel. This may be trivial in a small template, but Allan’s example of just a few integrations start to show the extra bloat. Imagine a template with lots of functions, each with their own sets of connectors?

Beyond the potential added bloat in a single template, there is also a very good chance that your applications will grow to include multiple stacks that need to interact with one another. A single API that spans multiple templates, a shared datastore, or an SQS topic with consumers and producers in separate stacks come to mind.

Take this example from the docs:

We’re now specifying a Type, the ResourceId, and in this case, the Qualifier, which is a (slight) shortcut to specifying the ARN and resource. Maybe I’m a bit old fashioned, but I’m not seeing a particular advantage over the IAM policy below, which came up as the first result when I Googled “iam permission for allowing post to lambda from api gateway.”

There’s another disconnect that becomes a bit obvious here: the Read and Write semantics. While it sort of makes sense (I think) that you are “writing” to a Lambda function on an HTTP POST request, the underlying permission is execute-api:Invoke. Now, I may be wrong here (because I didn’t test this), but I’m guessing that either Read or Write permissions would translate to Invoke. So I could write a connector that gave “write” permission on a GET route, and “read” permissions on a POST route, and everything would work just fine. Those semantics are wrong, and for the uninitiated, I could see this causing more confusion than it’s worth.

Another issue with the Read/Write semantics is their varied scope. Read permission on a DynamoDB table assumes you can do no destructive action, while Write assumes the ability to add and (probably) update data. But Write in this context also gives you the ability to delete items. And what about with an SQS Queue? The Read permission would presumably give you sqs:ReceiveMessage, but that only gets you part of the way there. You also need sqs:DeleteMessage if you want to remove a message from a queue. This would require you to add Write permissions, which would then give you sqs:SendMessage as well. This is not what I want if I have separate consumers and producers. It might just be me, but this doesn’t seem intuitive.

Let’s go back to DynamoDB for a second. I wrote a post a while ago about Ensuring Data Integrity with DynamoDB. There are a lot of nuances in there that describe situations where you might want to limit upserts, prevent deletes, or isolate attributes. Those might be slightly advanced, but even adding Delete as an option to the Permissions config would add a bit more control and maybe help with the confusing semantics. This is also the primary reason why I don’t particularly like SAM Policy Templates. They’re too broad, and IMO, discourage the principle of least privilege. But, hey, at least you can see what permissions each one grants.

Like I said, abstractions are hard, and sometimes you just need to go for it and see where it lands. Overall, I think the SAM team does a good job trying to accommodate their users by giving them features that makes building serverless applications easier. But in this case, I think they may have done the opposite.

Tags: abstractions, infrastructure as code, security, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

I’ve been spending a lot of time lately thinking about the next evolution of the cloud, and more importantly, what the developer experience looks like. A few years ago, I think that most of us in the serverless ecosystem thought that the path forward seemed quite clear. Serverless-first was obviously “the way.” Small, discrete, single-purpose functions interconnected through a series of planet-scale, self-upgrading, managed services with built-in redundancy was the holy grail of cloud development.

Of course there were some gotchas in there, and not every use case was a perfect fit, but over time we figured these would be addressed as the technology evolved. For the most part, that has come to pass. Even if AWS hasn’t quite yet solved some of these issues, other cloud providers and startups have certainly tried. But while serverless was slowly preparing to cross the chasm, another already widely accepted technology was gaining traction in the cloud: containers.

To be perfectly clear: I really like containers. They solve a number of problems that traditional VMs don’t. They’re highly portable, they’re (mostly) lightweight, and they make an excellent packaging format. There are also lots of advantages to using them for code execution in the cloud. They can handle multiple concurrent connections, they’re better for stateful workloads, they’re not limited by arbitrary timeouts, and they are much cheaper for high volume workloads. But for just about anything beyond application code execution, using containers in the cloud…seems crazy to me!

Kubernetes is a marvel of engineering, and you can run robust implementations of popular software systems like Apache Kafka and PostgreSQL on it with relative ease. But just because you can doesn’t mean you should. Unless your business is literally running a database service, why would you even think about trying to support something like that? Between all the different cloud provider offerings, or the myriad of startups building serverless databases, queues, caches, and more, what is the advantage of managing these services yourself? The answers I hear the most are customization, cost, and portability. I call shenanigans.

I need more control

If I’ve said it once, I’ve said it a hundred times: control is an illusion. Customization typically boils down to performance tuning, something that can go wildly wrong in the cloud, even for highly-knowledgable engineers. The cloud isn’t just “someone else’s computer”, it is a complex system that requires optimizations you don’t know how, or have the ability, to make. If your application gets to a point where you’ve reached the limits of a cloud service, reach out to the service team and they’ll be more than happy to work with you to make their service better. It’s happened several times to me.

The cost of portability

Cost can be a valid concern, but look at the overall TCO. It takes a really, really large cloud bill to offset even a small team of specialized engineers. But the one that really gets me is portability. If you built an application that can be ported back to a local datacenter, then you have NOT built an application that is NATIVE to the cloud.

The CNCF defines “Cloud Native” as “technologies that empower organizations to build and run scalable applications in modern, dynamic environments such as public, private, and hybrid clouds. Containers, service meshes, microservices, immutable infrastructure, and declarative APIs exemplify this approach.”

The majority of CNCF projects follow the “write it once, and run it anywhere” mantra. But the public cloud, and the services that were built for it, had a very different vision.

Allan Vermeulen, one of the original creators of S3, has said that they built these services to achieve things that could NEVER be solved by the datacenter. The types of applications that can be built on a public cloud CANNOT be ported back to the datacenter because they don’t have the primitives to support it. The cloud introduces new levels of scale and capabilities that the on-premises environment is simply not capable of providing. This incredible power is available to you right now, on-demand, and you generally only pay for what you use.

The future is Native Cloud apps

So perhaps “Cloud-native” makes sense when you very loosely define the term “cloud”, but let’s not pretend that portability to massively subpar alternatives is a good strategy. Especially when the public cloud providers give us not only the primitives, but also the operational excellence that is impossible for most others to achieve.

A serverless-first mindset is a step in the right direction, but the public cloud is bigger than that. Edge compute and containers will play a massive role in the future of cloud, as will security, compliance, sustainability, and more. We need a way to classify the applications of the future. The ones that are BUILT FOR THE PUBLIC CLOUD and all its awesome power, not the ones that are still chained to the data centers of the past. For now, I’m calling them native cloud applications, and I firmly believe that this is the path forward.

Tags: cloud, cloud computing, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

JV Roig wrote a great article recently about polar bears, serverless and sustainability. The premise is quite simple: if we make better choices in the cloud, we can reduce our impact on the environment, and thus, save the polar bears. The sentiment is nice, but it is, of course, much more complicated than that. This begs the question, how much impact can our cloud choices really have?

Like with anything, your (or your company’s) cloud usage pales in comparison to the 90 billion kilowatt-hours (or roughly 3% of total global electricity consumption) used by data centers each year. So even though my family switched to cloth napkins and compostable coffee pods, these choices (although no doubt selfless, and might I even say, bordering on heroic ?) have a net zero impact on the environment. Why? Because these choices don’t have any systemic change.

I don’t mean to come across as cynical here (although I definitely am). And I’m not going to say that these choices aren’t noble in their own right, because, of course, everyone has to “do their part.” But I truly believe that the only way to have any real, meaningful impact is to change the system, not simply ration its outputs. Turning down my thermostat by a couple of degrees may save a few gallons of oil, and switching to energy efficient bulbs and appliances might shave a few kilowatt hours off my electric consumption, but those are quickly offset by the world’s ever increasing need for more and more energy. I wish I had more faith in humanity, but let’s face it, not everyone cares about polar bears.

A foundation for systemic change

So let’s go back to the question about the impact we can have with our cloud choices. The two sustainability “moves” that JV focuses on are “Region Selection” and the use of “Serverless.” While I love the idea that these choices are available to you, my cynical brain wants to say that right now neither of them will have much more of an impact than choosing cloth napkins. However, my more altruistic side says these choices actually lay the foundation for much needed systemic change.

Let’s start with regions. First of all, I appreciate the major cloud providers’ efforts to invest in sustainable energy projects and carbon-neutral data centers. I have a lot of questions, especially about the use of carbon offsets, but at least the effort is there. And part of the reason why they can do this is because of their massive scale. Companies that build their own data centers generally don’t have the same incentives to be constantly upgrading hardware or investing in sustainable energy. For an individual company, the benefits likely don’t outweigh the costs, leading to an incredible amount of inefficiency as the technology ages.

Simply moving workloads into the cloud means you should be getting continuous improvements and upgrades without needing to do (or spend) anything. Choosing a region that is more sustainable than another is a nice gesture, but even if you don’t, just choosing “the cloud” will eventually get you there, assuming the cloud providers continue to do the right thing.

What about serverless?

I remember when Werner announced the AWS Well-Architected Sustainability Pillar at last year’s re:Invent, my first thought was that he was basically saying, “You should go serverless!” But does choosing serverless actually have an impact? Right now, no, I don’t think it offsets that much given all those EC2 instances are still relatively efficient VMs. Sure, they pack quite a few Lambda functions onto a single machine, and you can certainly save CPU cycles if your apps sit idle most of the time, but I’m definitely getting compostable coffee pod vibes.

However, building your applications serverlessly hands over future efficiency decisions to the cloud provider. Not only can they upgrade the infrastructure that your serverless services run on, but they can update the runtimes, the chip architectures, and even how they provision certain components of your application. It’s amazing when a serverless service just gets better without any effort from you. It’s also amazing when they become more efficient and sustainable.

Go save some polar bears!

So, yes, use the cloud, pick a sustainable region if you can, and most definitely build your applications serverlessly. It might only save a few paper napkins today, but it also unlocks a path for cloud providers to implement true systemic changes that might just save a polar bear or two.

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

I came across Paul Johnston’s Learning Serverless (and why it is hard) post one Saturday morning, and ended up with a sore neck because I was nodding in agreement the entire time I was reading it. Okay, maybe the neck pain had more to do with how I slept the night before, but I’m quite sure the agreeing nods contributed. But when it comes to learning serverless, a little bit of neck pain, IMO, is the least of your problems.

I’ve been building serverless applications since 2015. This was before you could attach a Lambda Function to a VPC, or EventBridge, or StepFunctions, or hundreds of the other amazing features that exist today. When I first discovered Lambda Functions, even API Gateway didn’t exist! Yep, the staple service in almost ALL of our serverless apps wasn’t available until mid 2015. Since then, AWS has grown to over 250 services, with each one growing in complexity as the years roll by.

It’s easy to argue that the added complexity was a necessary maturing of the tech, and as I highlighted in Paul’s piece, “…whatever you may have read or heard from someone, the number of scenarios where a serverless approach is not appropriate is *vanishingly* small.” But the question is: “At what cost?” Has this growing complexity made serverless inaccessible to the majority of developers?

The evolution of the cloud (engineer)

Before we can answer that question, I think we need to level-set where serverless (or rather adopting a serverless approach) fits into a cloud engineer’s evolution. Paul astutely points out that there is no clear path for the developer, but there should be little doubt that cloud providers see “serverless” (read: servicefull, on-demand, distributed, and managed infrastructure) as the future of cloud. Their definitions and implementations may vary widely, but it’s clear that most cloud primitives are evolving into plug and play managed services.

As that shift continues to happen, one might even assume that all “cloud” engineers will essentially be forced into using these newfangled “serverless” components. This will simply be “the way” to build cloud applications in the future, and if you don’t figure out how to stitch these all together, you might find yourself left out of the revolution. Serverless is coming for us all. Resistance is futile. Or is it?

Shortcuts are holding us back

The push for this new way of building applications is continuously stymied by the “meet customers where they are” edict, which often results in suboptimal intermediate solutions that might move the bar in the right direction, but ultimately delays what should be the inevitable. This is why we ended up with 19 different ways to deploy containers on AWS. But the forced complexity may be at a breaking point. Look at the dozens (if not hundreds) of startups looking to make cloud development easier.

Whether they’re no-code or low-code, proprietary or container-based, the vast majority of them feel like they are driving us to a Cloud 1.5, rather than the Cloud 2.0 the major providers have been building to for years. And you can forget about Paul’s sweet spot of Event-driven thinking + “Speedy, Simple, Singular” Functions + Low Latency Data Layer development. These new solutions rely on skills from the past rather than encourage learning and embracing the more modern approach that Paul advocates for.

Back to the question about the ultimate cost of all that extra complexity. I think the cloud providers failed to win hearts and minds with this new approach and way of thinking. Their tooling was mostly terrible, and if it weren’t for a thriving community of diehard serverless fans building observability tools, OSS projects, and other workarounds, it possibly would have died on the vine a long time ago. This failure has given oxygen to a new generation of stopgap solutions that will get us no closer to the ultimate vision of a truly “serverless” cloud future.

A serverless journey takes commitment, and deeper understanding of the cloud

If you ask me why serverless is so hard to learn, I’d point you to Paul’s article and add that beyond the mental shift, the required complexity of a well-architected, multi-service serverless application is just too daunting for the majority of developers. Sadly, this means that most will fall back to platforms that support their old way of thinking and never feel the need to learn the serverless approach.

Alex DeBrie made the following observation on Twitter:

When I hear someone say “Serverless development is too hard!”, 96.2% of the time it’s because they’re deploying with:

– Raw CloudFormation and bash scripts – AWS CLI + bash scripts – AWS console

Don’t do this! Use a deployment tool! There are so many — pick one at random.

— Alex DeBrie (@alexbdebrie) October 5, 2022

I see his point, but I don’t think this truly captures the depth of the problem. If you are building a “serverless” app (whatever that even means now) and not using a deployment tool/framework, then yes, you’re making it much harder. Deployment tools and frameworks certainly help make it easier to spin up magical cloud infrastructure, but they also let you make a lot of bad choices. If you don’t understand cloud architecture, how distributed systems work, and what these tools are actually doing, then they haven’t solved the real issue.

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

So is it hard to believe that the “cloud have-nots”, i.e. the vast majority of developers who haven’t had the freedom (or mandate) to experiment with cloud services, face significant headwinds on their own digital transformation journeys? It shouldn’t be. I’ve been working with AWS since 2009, back when there were just a few services. Today, Amazon has over 250 services, and I’d guess that I’m proficient with less than 10% of them, and even that might be an exaggeration.

How much can any one person really know?

I don’t think I’m alone here either. I’ve probably used well over a hundred AWS services, but oftentimes they were for an obscure part of the application. Or even worse, some sort of uninteresting infrastructure configuration. I likely spent hours looking through documentation and blog posts in order to flip some switch, and then almost certainly forgot what I did and why I did it. Time well-spent (I hope) for the sake of that application, but it was the kind of non-reinforced heavy lifting that will probably require a similar level of effort the next time I face the same issue.

What about the plethora of services at Cloudflare, Azure, GCP, and others? Add in offerings from the CNCF’s “Cloud Native Landscape” and you might find the average cloud developer with proficiencies in the low single digits. I’m not extraordinary, and I know lots of cloud developers that hold much broader skillsets, but even the smartest person out there likely has a working knowledge of less than 5% of the total landscape. And even if that person does exist, you can’t afford them.

And this is where the organizational skills gap that Lydia refers to really kicks in. Organizations of all sizes are typically full of people making the wrong technology decisions. It doesn’t mean that the people making these decisions aren’t smart, dedicated, masterful technologists, it just means that bias, ignorance, corporate pressure, and other constraints ultimately result in subpar choices. Choices that then force your organization to require resources with specialized skillsets. Even assuming that your organization can upskill or hire resources in the short term, maintaining competency and institutional knowledge in the long term is always a challenge.

What’s the solution?

There is no easy or immediate solution. Companies aren’t going to repatriate to their own data centers. That time has definitely passed. Cloud is simply the way forward. The “crisis-level skills gap” that Amy Sarah John refers to in 6 long-term planning trends by Gartner for cloud and edge computing suggests a few ideas, but focusing on Kubernetes and talent programs simply aren’t going to get you there. There are too many things to learn and too high a turnover in tech. Couple that with a flurry of poor technology choices by upper management and most organizations will find themselves on the refactoring hamster wheel constantly adopting new silver bullets that they’ll likely shoot themselves in the foot with.

The long term solution has to go beyond simply training more cloud generalists. Specialization will always be useful, but cloud is about composition and integration, and the complexity has outgrown the average person’s (and organization’s) capabilities. Right now there is a relatively small group of people that are able to effectively write “machine code” for the cloud. Everyone else is waiting for the cloud equivalent of C and C++.

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

DynamoDB is an incredibly powerful NoSQL database. It’s schema-less, which gives you lots of flexibility, but it also means that you are responsible for managing the integrity of your data. This includes ensuring the structure of your data, as well as the ability to preserve metadata throughout your data’s lifecycle.

Unfortunately, DynamoDB doesn’t currently store any metadata associated with items. If you want to know when a particular item was written to the table, for example, you have to store that information yourself. While it’s not particular difficult to add these attributes to an item, maintaining their integrity can come with some challenges.

In this article, we’ll discuss several strategies that can be used to ensure data integrity in your DynamoDB tables.

Data management with PutItem and UpdateItem

Before we discuss how to ensure our data’s integrity, let’s quickly review the ways in which we can add/modify data in our DynamoDB tables. Similar to RBDMS, DynamoDB supports both “INSERTS” and “UPDATES”, however, these two operations behave much differently than their RDBMS counterparts.

DynamoDB uses PutItem to add items to your table. Unlike RDBMS, PutItem replaces existing items by default. While you can add overwrite protection to your PutItem API calls, DynamoDB does not support service-side enforcement. This means that you can inadvertently overwrite an entire item, losing any metadata that you may have stored with it.

UpdateItem behaves as an “UPSERT” operation. This means that if you try to update an item that doesn’t exist, DynamoDB will automatically create it for you. Like with PutItem, you can add conditions to your UpdateItem API calls to modify this behavior, but there is no way to implement it service-side. UpdateItem also merges root attributes, allowing you to perform partial updates without needing to pass the entire object in.

We’ll show some examples of these operations throughout this article.

Protecting data on UPSERTS

UPSERTS are an incredibly powerful capability of DynamoDB, allowing you to essentially fallback to a PutItem operation if the item doesn’t exist. This is a great feature, but it also means that you have to provide both defaults for items that don’t exist, as well as overwrite protection for metadata that already does. A simple example is with created and modified attributes:

Corey Quinn, Cloud Economist (and perpetual thorn in AWS’s side), recently published a post titled The Unfulfilled Promise of Serverless. Twitter reacted as we would expect, with plenty of folks feeling vindicated, others professing their staunch disagreement, and perhaps even a few now questioning their life (and technology) choices. My take is that he’s not wrong, but he’s also not entirely right.

Let me start by saying I’m a big fan of Corey’s (seriously, this video about AppConfig might be the funniest thing he’s ever created.) However, much of Corey’s brand is predicated on creating controversy with his unique brand of snark and clever shitposting (the Twitterverse wants what the Twitterverse wants.) So while his hottake on serverless might resonate with many of us, it’s important to understand both its motivation as well as its missing context.

In order for a promise to be “unfulfilled,” we must understand both what was promised, as well as when delivery was promised. While this might sound like a semantic argument, I think it’s important to point out a very important piece of context: we’ve barely scratched the surface of what’s possible with serverless technology. I agree with Corey that Werner’s “just write business logic” claim conveniently ignored the mountain of configuration that came with it, but it also put a stake in the ground and set the vision. To the best of my knowledge, no one ever gave a timeline to achieve everything that was promised, but what I’ve seen over the last seven years are massive investments by public cloud providers, established players, and startups laying the groundwork that brings us closer to that vision. No, we’re not there yet, but that just means we have more work to do.

Below I’ve tried to add some context to the main points Corey made in his post. As I said, many of his points resonate with me, but hopefully this brings some comfort to those of you that were disheartened by his message. For those of you that have yet to “buy in” to serverless, I simply ask that you keep an open mind. The vision for serverless and its current reality are very different, and like with most things, we’re likely all trying to get to the same place.

Serverless, but not potable

Yes, you read that right. The (lack of) portability with serverless is less about lock-in, and more about the lack of consensus around implementation standards. Sure, if you’re AWS, keeping customers in your ecosystem by providing native services with first-class integrations (even if they’re as terrible as Cognito) makes it harder for them to look elsewhere and even harder for them to migrate away later. But if I take my cynical-colored glasses off for a moment, there’s also a greater than zero percent chance that some of the amazing engineers and product managers that work behind the curtain at AWS are actually trying to “innovate.”

For me, there is a major disconnect between AWS and most of the other major cloud providers when it comes to their approach to serverless. AWS took the same security and isolation properties of hardware virtualization technology and built them into Firecracker. Why? So that they could optimize the execution of microVMs by moving them as close to the metal as possible. What did just about everyone else do? They built their serverless compute solutions in containers running on top of Kubernetes, just about as far away from the metal as you can get.

Kelsey Hightower recently tweeted that “​​Serverless based on containers is the future.” Really? I guess that’s true if we’re prepared to give up on the optimizations already possible by letting the platform be the orchestrator rather than requiring 30 layers of abstraction just to execute a few lines of code. Even if we go down the container route, the “portability” problem is only partially solved. As Forrest Brazeal has pointed out, choosing “cloud-agnostic” technology often means going with the “lowest common denominator.” In other words, regardless of where you “port” your serverless compute, your (wise) reliance on CSP specific services will make migrating to other providers the bigger hurdle. This is true for all of “cloud,” by the way, and isn’t just a serverless phenomenon.

So back to my original point. I see “lock-in” as merely an excuse for people’s unwillingness to buy into the uncertainty. Sure, there is no competitor to AWS Step Functions, but that’s because this service is ahead of its time—not because state machines haven’t been a thing since the dawn of computing, but because other cloud providers have a short-sighted vision for serverless. Customers that are willing to go “all in” on AWS’s vision might lose portability, but what they gain in capabilities (at least in my opinion) far outweighs any potential drawbacks. It’s not about portability; it’s the fact that too many people want to reach for the familiar and are unwilling to drink the Kool Aid.

The imperceptible value fallacy

On a recent episode of his podcast, Corey so eloquently stated that “serverless sucks,” to which Ant Stanley, co-founder of A Cloud Guru/ServerlessConf/ServerlessDays and arguably one of the earliest of early adopters of serverless, agreed. It’s not the serverless technology that sucks, but rather the complexity we’ve created around it. As Ant explains, “I think folks have focused far too much on the technical aspects of serverless, and what is serverless and not serverless, or how you deploy something, or how you monitor something, observability, instead of going back to basics and first principles of what is this thing? Why should you do it? How do you do it? And how do we make that easy? There’s no real focus on user experience and adoption for inexperienced folks.”

Corey’s experience with serverless isn’t an outlier. AWS has come a long way with CloudWatch and its related tools, but the fact that an entirely new industry was required to help you understand and observe your serverless applications (Lumigo, Epsagon, IOpipe, Thundra, Dashbird, etc.) should have been the canary in the coal mine for widespread adoption. Most of these tools have either been acquired or have pivoted to focus on more than just serverless because, as Corey said, they faced the unfortunate reality that a pure serverless market just wasn’t valuable enough.

Serverless adoption (including the use of “serverless services” that Corey casually dismisses) continues to grow year over year. But even though more that 50% of the “thousands of companies in Datadog’s customer base” are using some form of FaaS, it’s only based on whether “they ran at least five distinct functions in a given month.” While we don’t have the underlying data, this suggests that FaaS makes up just a portion of their cloud workloads. So maybe we don’t have full buy-in at this point, but plenty of cloud companies are embracing the serverless paradigm and extracting some value from it.

However, there is a major problem with partial buy-in: TCO. The total cost of ownership (TCO) of serverless (or at least the promise of it) is significantly lower than setting up and maintaining traditional compute (yes, even if the compute charges are higher). But the incremental cost of adding workloads to an existing Kubernetes cluster or cloud environment supported by an experienced Ops team is minimal. For a startup (or even a small team within a larger org), going “serverless first” should be a no-brainer. Unfortunately, the value of those customers probably won’t move the needle, hence the fact that it “doesn’t feel like it’s solving an expensive problem.”

Who isn’t going to own the upgrades?

I had Ant Stanley on the Serverless Chats podcast not long after his interview with Corey, and when I asked about the complexity issue, he also pointed out that the number of choices with serverless creates more cognitive load on developers when making decisions.

“I would love to see a simplification of serverless going, ‘Hey, if you want a relational database, this is it. If you want a NoSQL database system, this is it. if you want to run synchronous workloads, this is where you do it. If you want asynchronous workloads, this is where you do it, and give one option. And here’s your development workflow, and one set of tooling to make that work,’ but I think we’re way off that.”

While Ant’s vision of serverless simplicity would certainly make things easier, others have posited ideas that get us much closer to the ultimate serverless vision. Shawn @SWYX Wang recently articulated his vision of The Self Provisioning Runtime. He says, “If the Platonic ideal of Developer Experience is a world where you ‘Just Write Business Logic’, the logical endgame is a language+infrastructure combination that figures out everything else.”

At the time Shawn published his article in late August, he was unaware of Serverless Cloud, the project that my team has been working on at Serverless, Inc. for nearly a year. The goal was to achieve almost exactly what Shawn was describing, a platform that would interpret your code and deploy the necessary infrastructure to best support it. It’s what we call Infrastructure from Code at Serverless Cloud. There are others that are doing similar work like Dark Lang, Lambdragon, and others mentioned in Shawn’s post.

But we’re now at the point where the technological groundwork painstakingly laid over the last seven years is ready for the next major step towards the ultimate vision. The complexity added by configuration was an intermediate step that told the cloud what was needed to run our code. The near future is a world that no longer requires that step. So, yes, there are plenty of things still required for serverless computing to fulfill its full potential, but we’re a lot closer to “just write business logic” than you might think. And when anyone can fully express their application with just their code, then the true potential of serverless will finally be met.

Tags: aws, cloud, lambda

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

As of this writing, there are now over 115 episodes of Serverless Chats! It’s been an incredible journey and I’ve learned so much about serverless from all these amazing guests (and I hope you have too). Recently, with the addition of Rebecca Marshburn as my co-host, we’ve added even more perspective to these conversations, expanding well beyond the technical and diving deeper into how serverless affects people, processes, and productivity.

If you’re new to serverless (and Serverless Chats), it can be hard to get started. Finding the right content and information can be overwhelming. If you’re a serverless veteran, finding interesting conversations that go beyond Hello World examples can also be challenging. My hope is that Serverless Chats provides a good mix for both the skeptical beginner and the experienced serverless developer. But even so, with 115 episodes, where do you start?

I’ve gone through the last 115 episodes and picked out a few of my ultra-favorites. If I simply listed my favorites, you might as well just go to the full list of episodes since I enjoyed all of these conversations so much, that I’d happily (and sometimes do) go back and listen to any one of them. But in the interest of saving you some time, here are 15 episodes that I believe capture some of the most important aspects of serverless. Some of them are technical, some focus more on the serverless experience, and some of them show the power of this amazing community. Enjoy.

15. AWS re:Invent 2020 Heroes re:Cap (Episode #86)

February 1, 2021 – On this episode, we recapped the major serverless announcements from AWS re:Invent 2020 with help of AWS Heroes Yan Cui, Serhat Can, Luca Bianchi, Farrah Campbell, Ran Ribenzaft, Ben Ellerby, Sheen Brisals, and Gillian Armstrong.

14. Serverless for Startups with Chris Munns (Episode #113)

October 4, 2021 – In this episode, Chris talked about the evolution of serverless and how the tools and services have matured over the years. We also discussed how the learning curve of serverless affects adoption, what goes into building an effective developer advocacy team, and what advice  he has for startups looking to get started with the cloud.

13. Streaming Data at Scale Using Serverless with Anahit Pogosova (Episode #91 & Episode #92)

March 8 & 15, 2021 – In this two part episode, Anahit shared her thoughts on where and why we’d use Kinesis, how to embrace (and deal with) the failures, and addresses some common serverless misconceptions. These are highly technical episodes, but some of my favorites.

12. Building a Business Around Serverless with Nofar Asselman (Episode #87)

February 8, 2021 – On this episode, Nofar spoke about the current state of the “serverless” market, what opportunities exist for tools and solutions supporting serverless, how to leverage partnerships to build trust in your product, and actions you can take right now.

11. Serverless Privacy & Compliance with Mark Nunnikhoven (Episode #71 & Episode # 72)

October 19 & 26, 2020 – In this two part episode, Mark explains why your online privacy is so important, what we need to think about in terms of compliance, and how serverless helps us create more secure applications.

10. Going Green with Serverless with Paul Johnston (Episode #59 & Episode #60)

July 27, 2020 & August 3, 2020 – In this two part episode, Paul shares how serverless compares to traditional computing in terms of being “green”, the impact of data centers on climate, why efficiency is only a first step, and what people in tech can do to affect change. We then get into how the big cloud providers are addressing climate change, what the tech industry can do as a whole, the effect this will have on globalized business, and how the positive downstream impact of choosing serverless can create a more sustainable business.

9. Serverless Infrastructure as Code with Ben Kehoe (Episode #107)

June 28, 2021 – On this episode, Ben explained what infrastructure as code really means and why IaC with serverless is different than non-serverless architectures. He also talked about how IaC defines resource graphs that fully specify the state of your system and why is it important for developer intent to be maintained by IaC systems.

8. Static First Using Serverless Front-ends with Guillermo Rauch (Episode #50)

May 25, 2020 – Guillermo explains the difference between front-end and backend serverless, how we should think about and build for scale, and why latency down to the first contentful paint is so important.

7. Serverless Developer Culture with Linda Nichols (Episode #48)

May 11, 2020 – In this episode, Linda talks about why modern cloud developers should be writing less code, how new deployment processes affect the testing culture, and why Ops teams are still really important.

6. The Past, Present, and Future of Serverless with Tim Wagner (Episode #52)

June 8, 2020 -On this episode, the one and only Tim Wagner shared some of the history behind AWS Lambda, why the stateless versus stateful debate still rages on, how to use serverless as a supercomputer, and what innovations are still needed in the serverless space.

5. Continuously Improving Serverless Standards at the LEGO Group with Nicole Yip (Episode #82)

January 4, 2021 – On this episode, Nicole shared how the LEGO Group continued to grow their serverless development teams, the evolving audit process they use to improve serverless standards, and the challenges they faced adopting those standards.

4. Data Modeling Strategies from The DynamoDB Book with Alex DeBrie (Episode #44)

April 13, 2020 – Alex shares why he wrote the DynamoDB Book, what are some key concepts to keep in mind when modeling data, and how using the right strategies can help you create more powerful single table designs.

3. Pushing the Limits of Lambda with Michael Hart (Episode #18  & Episode #19)

October 14 & 21, 2019 – In this two part episode, Michael explains the inner workings of AWS Lambda, the hows and whys of Custom Runtimes & Layers, Docker Lambda, serverless CI, and whether or not Lambdas should call Lambdas.

2. Why Developers Need to Think About Cloud Costs with Erik Peterson (Episode #6)

July 22, 2019 – In this episode, Erik Peterson talks about cultural changes around cost optimization in the cloud, cost as a first class metric, and how to predict total cost of ownership (TCO) according to your product roadmap.

1. Advanced NoSQL Data Modeling in DynamoDB with Rick Houlihan (Episode #34 & Episode #35)

February 3 & 10, 2020 – In this highly-technical two parter, Rick Houlihan talks about the use cases for NoSQL, why single table designs are so powerful, and ways to model relational data with GSIs. We also discussed why you likely don’t want to use LSIs, when sharding is necessary, the benefits of denormalization, and how to efficiently store large document deltas.

I hope you found this list of episodes useful. Did I miss one of your favorites? Reach out to me on Twitter and let me know.

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

When I started the podcast in 2019, I wanted to invite others into the great off-the-cuff, offline conversations I was having with members of the serverless community. The chats I’d share after an event over drinks, in the hallway between conference sessions, or while mingling at a meetup, were thought-provoking and interesting, and offered fresh ideas, perspectives, and sometimes, even solutions that I could take back to my work and projects. After more than 100 episodes of inviting listeners and guests to these conversations, I am now extending a different sort of invitation for one of our serverless community members.

I am extremely excited to announce that Rebecca Marshburn will be joining Serverless Chats as a co-host!

If you haven’t had the pleasure of meeting Rebecca yet (or haven’t listened to our 100th episode), Rebecca’s interested in the things that interest people— What’s important to them? Why? And when did they first discover it to be so? She’s also interested in sharing stories, elevating others’ experiences, exploring the intersection of physical environments and human behavior, and crafting the perfect pun for every situation. Today, Rebecca is the Head of Content & Community at Common Room. Prior to Common Room, she led the AWS Serverless Heroes program, where she and I first met, and guided content and product experiences for fashion magazines, online blogs, AR/VR companies, education companies, and a little travel outfit called Airbnb.

After two years of producing the podcast, I had been considering ways that I could continue to bring relevant and meaningful content to my listeners. Though Serverless Chats has always featured multiple points of view between mine and our guests’, Rebecca’s addition ensures that we’re able to have conversations that touch upon all aspects of the serverless experience that matter to our community. With Rebecca’s industry experience and commitment to the people who build and use serverless, we’ll be able to have more complete, multi-faceted conversations that broaden the podcast’s ability to dive deeper into the human element and impact of serverless while remaining focused on the technical and tactical discussions.

As part of this next step for Serverless Chats, we’ll be taking an eight-week summer break starting at the beginning of July. We’ll still be recording with our incredible guests so you have great content when we return August 30th, but we’ll also be using this time to work on the overall brand, as well as, you know, actually taking some time off.

I’m really looking forward to this next chapter for Serverless Chats, and feel honored to have Rebecca as part of it.

Thank you so much for your continued support of the podcast, and enjoy the summer!

(If you miss us over the summer, you can catch up on any episodes you missed at serverlesschats.com, sign up to become an Insider to stay in the loop with more podcast updates, and definitely make sure to follow us on Twitter @ServerlessChats.)

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

This content is password protected. To view it please enter your password below:

Password:

Tags: serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

Three years ago at re:Invent 2017, AWS announced the original Amazon Aurora Serverless preview. I spent quite a bit of time with it, and when it went GA 9 months later, I published my thoughts in a post titled Aurora Serverless: The Good, the Bad and the Scalable.

If you read the post, you’ll see that I was excited and optimistic, even though there were a lot of missing features. And after several months of more experiments, I finally moved some production workloads onto it, and had quite a bit of success. Over the last 18 months, we’ve seen some improvements to the product (including support for PostgreSQL and the Data API), but there were still loads of problems with the scale up/down speeds, failover time, and lack of Aurora provisioned cluster features.

That all changed with the introduction of Amazon Aurora Serverless v2. I finally got access to the preview and spent a few hours trying to break it. My first impression? This thing might just be a silver bullet!

I know that’s a bold statement. ? But even though I’ve only been using it for a few hours, I’ve also read through the (minimal) docs, reviewed the pricing, and talked to one of the PMs to understand it the best I could. There clearly must be some caveats, but from what I’ve seen, Aurora Serverless v2 is very, very promising. Let’s take a closer look!

What’s Aurora Serverless again?

For those that need a refresher, “Amazon Aurora Serverless is an on-demand, auto-scaling configuration for Amazon Aurora. It automatically starts up, shuts down, and scales capacity up or down based on your application’s needs. It enables you to run your database in the cloud without managing any database capacity.” Sounds amazing, huh?

Aurora Serverless separates the data and compute layers so that each one can be scaled independently. It uses distributed, fault-tolerant, self-healing storage with 6-way replication that will automatically grow as you add more data. The data is replicated across multiple availability zones within a single region, which helps provide high availability in the event of an AZ failure.

Aurora Serverless v1 uses a pool of warm instances to provision compute based on your ACU (Aurora Compute Unit) needs. These pre-provisioned instances attach to your data and behave similar to a typical provisioned database server. However, if certain thresholds are crossed (max connections and CPU), Aurora Serverless v1 will automatically move your data to a larger instance and then redirect your traffic with zero downtime. It will continue to scale up as needed, doubling capacity each time. Once traffic begins to slow down, your data will be moved to smaller instances.

Instances run in a single availability zone. In the event of an AZ failure, a new instance will automatically be provisioned in another AZ, your data will be attached, and your requests will begin routing again. Based on availability, it could take several minutes to restore access to the database.

There are also a lot of things that Aurora Serverless v1 can’t do. Most notably: global databases, IAM authentication, backtracking, S3 import/export, and Lambda triggers.

So what’s so great about Aurora Serverless v2?

I’m glad you asked! A big change for Aurora Serverless v2 has to do with the way the compute is provisioned. Instead of needing to attach your data to differently sized instances, Aurora Serverless v2 instances auto-scale in milliseconds based on application load. Yes, you read that correctly. Using some combination of elastic computing and dark magic, your ACUs will scale instantaneously to handle whatever you throw at it.

Also, because the instance scaling is elastic, Aurora Serverless v2 increases ACUs in increments of 0.5. This means that if you need 18 ACUs, you get 18 ACUs. With Aurora Serverless v1, your ACUs would need to double to 32 in order to support that workload. Not only that, but the scale down latency is significantly faster (up to 15x). I’ll show you some of my experiments later on, and you’ll see that scale downs happen in less than a minute.

Beyond the amazing scaling capabilities is the fact that “it supports the full breadth of Aurora features, including Global Database, Multi-AZ deployments, and read replicas.” This seems pretty darn clear that Aurora Serverless v2 intends to support all the amazing Aurora features, including the ones missing from Aurora Serverless v1.

Let’s talk about price

There’s no way to sugar coat this. The cost of Aurora Serverless v2 seems very high. In fact, v2 ACUs are twice the price of the original v1 ACUs ($0.12 per ACU Hour versus $0.06 per ACU Hour). There is some clever marketing language on the Aurora Serverless page that claims “you can save up to 90% of your database cost compared to the cost of provisioning capacity for peak load.” That may be true, but let’s break it down a bit.

When they say “provisioned capacity”, they mean always on, as in an Aurora provisioned cluster. But before we get into that comparison, let’s look at how it compares to Aurora Serverless v1.

So, yes, the price per ACU is double. However, there are some important distinctions here when it comes to how these costs get calculated. One difference has to do with the incremental ACUs of v2 versus the doubling of instance sizes required for v1. I’d like to be able to say that there’s an argument that if my workload only needs 9 ACUs, then it would be cheaper than needing to pay for 16 ACUs with v1. Unfortunately, those 9 ACUs would cost your $1.08/hour versus $0.96/hour for 16 v1 ACUs. And if you needed 15 ACUs, well, I think you get the point.

The other difference has to do with the scale down time. This argument might be a little stronger. It can take a while for Aurora Serverless v1 to scale down, like up to 15 minutes according to the documentation. My past experiments showed that this was more like 2-3 minutes, but that was in a controlled environment setting the capacity manually. So even if it took some amount of time in between, it’s very likely that you would be paying for a lot of extra ACUs (remember that they get billed by the second). This, of course, only really matters if you have very spiky workloads. Anything more consistent, and things start to get expensive really quickly.

Which brings us back to the “provisioned capacity” comparison. Let’s say you were running a db.r5.4xlarge for $2.32/hour. That will run you $1,094.40/month per instance. You know you need some redundancy, so even if you only had a single read replica, that’ll cost you another $1,094.40/month for somewhere around $2,188.80/month. With Aurora Serverless v2, we only really need to provision the one cluster (there are some caveats here), so we’d need roughly 64 ACUs to match the peak capacity of the db.r5.4xlarge cluster. If we ran 100% at peak capacity, then that would cost something silly like $5,529.60/month. ?

But let’s get serious. You’re not very likely to run at peak capacity. Let’s say you have two major spikes per day, low traffic overnight, and fairly minimal traffic during normal hours. For the sake of argument, we’ll say 3 hours at peak (95% capacity), 9 hours of normal traffic (40% capacity), and 12 hours of slow traffic (10% capacity). Using our 64 ACU example, that would be (3 * 64 * 0.95 + 9 * 64 * 0.4 + 12 * 64 * 0.1) * $0.12 * 30 = $1,762.56/month. It’s not a 90% savings, but compared to peak-provisioning, it still might make a lot of financial sense.

I don’t know exactly how they calculate the 90% savings, but if you factor in the amount of over, over-provisioning that goes on, especially if you include multi-tenant workloads, there definitely are some major cost savings possible here. If you’re running a fairly steady workload, then you have some calculations to do.

In regards to multi-region replication, Aurora Serverless v2 says it will support Global Database for “sub-second data access in any region and cross-region disaster recovery.” I asked for clarification on pricing, and it seems as though you could run a DR cluster in another region with just 0.5 ACUs until you needed it. Or, if you wanted multi-region access, your clusters would just scale with regional demand, which is also pretty cool. As always, data transfer fees apply. ?

Let’s look at some experiments

Since the initial preview of Aurora Serverless v2 is only meant to showcase the scaling characteristics, I spent the majority of my time writing scripts that tried to overwhelm it. Since it’s in a test account, I only have the default reserved concurrency of 1,000 on my Lambda functions. I tried a number of different select and insert operations, adding some random delays in long-running Lambda functions to force scaling operations. Granted, these weren’t massive numbers, but Aurora Serverless v2 essentially just laughed at me and effortlessly processed my queries.

As an example, I ran 1,000 concurrent session that averaged just over 17,000 queries per second split between SELECTS and INSERTS. There are three separate tests run in the charts below:

The database capacity (max values) just jumped around to handle the queries with ease, and I never got any errors. The long tail scale down here was mostly due to Lambda functions not closing the connections immediately.

In terms of the effect on latencies, I could barely move the needle.

I was able to get the Select Latency up to about 7 milliseconds at one point, but it was still completely negligible.

For good measure, I also manually scaled the database up to 32 ACUs (the max that the preview will allow), and the total scale down took about 8 minutes. However, this was controlled by setting the minimum ACUs in the console, so it might work a bit differently when reacting to load changes. I noticed several immediate drop offs during my tests, so in practice, I think it works faster.

One other small test I ran was to check the max_connections available at the different ACUs. The minimum ACU setting in the preview was 4, so I couldn’t see anything lower than that. Surprisingly, there were only 150 connections available, which is much lower than the 270 available with 4 ACUs in v1. I bumped it all the way up to 32 ACUs and saw 3060 connections available, so a little more than the 3,000 available in v1. I was going to check the others, but I have a feeling they are still tweaking these settings. It might be possible that the lower connection limits have something to do with optimizing the scaling, but I’m just guessing here. Hopefully that will change as the product matures.

Final Thoughts

Aurora Serverless v2 is still very early, and there is a long way to go before it’s GA and it “supports the full breadth of Aurora features.” Right now there are some very noticeable omissions, including the Data API and the pause capacity feature. However, assuming that the majority of features are added in by GA, the scaling performance turns this thing up to an 11. I’ll be curious to see how the pricing works out, but given the fact that this almost entirely removes the need for capacity planning and significantly reduces the amount of database management, the total cost of ownership comparison could be very compelling.

I’m going to keep my eye on this one, because if it delivers on its promise, then auto-scaling, relational databases in the cloud might just be another solved problem.

I’d love to hear your feedback and thoughts on this new version!

Tags: aurora serverless, database, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

My picks are also available on the Cloud Pegboard re:Invent Tool (Thanks, Ken). Select “Jeremy Daly” from the “AWS Hero Picks” dropdown and you’ll see all my selections with the options to add them to your wishlist and export them to your calendar. I have about 60 sessions on my list, which I’ve categorized below. But I realize that no human is likely going to be able to watch them all, so I’ve also made a list of Sessions you can’t miss!

There are sure to be plenty of announcements throughout the three weeks of re:Invent, so be sure to subscribe to the Off-by-none newsletter for weekly recaps.

NOTE: The times below are all Eastern Time and are the first broadcast. Check the session catalog or the Cloud Pegboard re:Invent Tool to find rebroadcast and other replay options.

Sessions you can’t miss!

Werner Vogels Keynote
Keynote Even if you’re not a big Skrillex fan, the Werner Keynote is usually loaded with lots of great technical information, customer success stories, and visions for the future. This year everyone gets a front row seat without waiting in that huge line. 12/15/20, 10:10 AM

Amazon DynamoDB advanced design patterns‚ Part 1 & Part 2
Session Clearly AWS is messing with us by leaving the best for last. This two-part session by the wizard himself, Rick Houlihan, is sure to be just as engaging and mind-blowing as his previous ones. Hopefully it will also highlight some new DynamoDB updates as well. Clear your calendar and make time for this. 12/15/20, 1:30 PM & 12/16/20 12:45 PM

Data modeling with Amazon DynamoDB‚ Part 1 & Part 2
Session Giving Rick Houlihan for his money is Alex DeBrie. AWS Data Hero, author of the DynamoDB Book, and all around great guy, Alex will be following up his session from last year with another deep dive into DynamoDB modeling. You don’t want to miss this one either. 12/2/20, 4:15 PM & 12/3/20, 5:00 PM

No more idling: Creating parallel builds using serverless CI
Dev Chat Michael Hart has been pushing Lambda to its limits since the beginning, and in this session he’ll show you how to create super fast serverless builds using parallelization, layers, EFS, and other techniques.12/1/20, 12:00 AM

The pragmatic cloud developer
Session This session promises to share insights from AWS’s experience with customers and how Amazon builds its own higher-level services. This should be really helpful for developers struggling with best practices. 12/1/20, 3:00 PM

How LEGO.com accelerates innovation with serverless
Session Serverless Hero Sheen Brisals will take you through the fascinating serverless journey that LEGO has been on and how EventBridge and Step Functions have become core parts of their system. 12/1/20, 2:15 PM

Application integration patterns for microservices
Session This one looks really interesting. Learn fundamental application integration patterns based on messaging and connect them to real-world use cases in a microservices scenario. 12/1/20, 4:30 PM

Reliability, consistency, and confidence through immutability
Session I’m guessing Adrian Hornsby will be giving this talk as this is a subject he has written about extensively. It is a fascinating topic and something all companies should be looking at. 12/1/20, 5:15 PM

Building event-driven applications with Amazon EventBridge
Session I love EventBridge and what it means for the future of serverless applications, so any opportunity to dig in deeper and learn more is one that I’m going to take. 12/2/20, 12:45 PM

Monolith to serverless SaaS: Migrating to multi-tenant architecture
Session Another really interesting topic. This session plans on exploring approaches to incrementally decompose your monolith into serverless microservices as well how tenancy affects logging, metrics, and data partitioning. 12/3/20, 1:15 PM

Is your serverless application well-architected?
Session Best practices, best practices, best practices! There, I said it three times, so now you have to watch it! 12/3/20, 5:45 PM

Building revolutionary serverless applications
Session I am really looking forward to this session by Ajay Nair, GM of AWS Lambda. It promises to guide you through what’s next for serverless at AWS and learn about what the future of serverless applications looks like. Join the revolution! 12/9/20, 2:00 PM

AWS Lambda‚ Part 1: Optimizing your serverless applications
Session I’m guessing this will be led by Chris Munns, with the first part covering everything that you should consider apart from your Lambda function. If you want optimal serverless performance, you need to know how to optimize your applications! 12/16/20, 1:15 PM

AWS Lambda Part 2: Optimizing your Lambda function performance
Session This second part will cover how you can best measure and tune your function’s performance as well as best practices for prehandler logic and reuse of information to enable fast startup and fast execution of your functions. 12/17/20, 1:15 PM

Adrian Cockcroft‚ architecture trends and topics for 2021
Session Never miss an opportunity to hear Adrian Cockcroft speak. In this session he’ll discuss upcoming trends including serverless first, continuously tested resilience, and Wardley mapping for technology transitions. 12/17/20, 5:00 PM

Below are some more sessions that I’ll be trying my best to watch. They are organized into categories to make browsing a bit easier.

Security & Account Management

15 things I wish I knew about securing my AWS account
Dev Chat 12/1/20, 12:00 AM

Understanding multi-account management
Dev Chat 12/1/20, 12:00 AM

Best practices for securing your multi-account environment
Session 12/8/20, 10:15 AM

Best practices for securing your serverless applications
Session 12/10/20, 12:15 PM

Best practices to securely operate GraphQL at scale with AWS AppSync
Session 12/16/20, 2:00 PM

Deep dive into AWS Lambda security: Function isolation
Session 12/17/20, 3:45 PM

Best practices for security governance in serverless applications
Session 12/17/20, 5:15 PM

GraphQL & AppSync

Introduction to GraphQL
Session 12/1/20, 5:30 PM

Power modern serverless applications with GraphQL and AWS AppSync
Session 12/9/20, 3:00 PM

Model and access application data more efficiently with AWS Amplify
Session 12/15/20, 1:00 PM

Unify access to siloed data with AWS AppSync GraphQL resolvers
Session 12/17/20, 6:00 PM

Jamstack

Add a custom domain to your AWS Amplify-hosted web application
Lightning Talk 12/1/20, 12:00 AM

Speed up your release cycle: Host SPA and static web applications on AWS
Session 12/10/20, 2:30 PM

Leadership Talks & Vision

Andy Jassy Keynote
Keynote LIVE 12/1/20, 10:30 AM

The value of innovation in challenging times
Lightning Talk 12/1/20, 12:00 AM

The business case for serverless with Stedi
Session 12/9/20, 1:15 PM

Increasing innovation with serverless applications
Leadership Session 12/16/20, 1:30 PM

Diversity in Tech

Building the pipeline of next-generation tech employees and leaders
Lightning Talk 12/1/20, 12:00 AM

Together, We Can Close the Gap
Session 12/1/20, 12:00 AM

Customer Success

How CATCH FASHION built a serverless ML inference service with AWS Lambda
Dev Chat 12/1/20, 12:00 AM

How BMW Group uses AWS serverless analytics for a data-driven ecosystem
Session 12/1/20, 3:00 PM

How FINRA operates PB-scale analytics on data lakes with Amazon Athena
Session 12/1/20, 6:00 PM

Data & Databases

Boost serverless app performance with Amazon RDS Proxy & Amazon Aurora
Dev Chat 12/1/20, 12:00 AM

Analyzing data at any scale with AWS Lambda
Session 12/3/20, 4:45 PM

How Disney+ scales globally on Amazon DynamoDB
Session 12/3/20, 6:30 PM

MySQL options on AWS: Self-managed, managed, serverless
Session 12/17/20, 12:45 PM

Building Event-Driven, Serverless Applications

Getting started building your first serverless web application
Session 12/2/20, 4:15 AM

Getting started with AWS Step Functions for service orchestration
Session 12/2/20, 5:00 PM

End the monolith! Lessons learned adopting serverless (sponsored by New Relic)
Session 12/3/20, 5:30 PM

Decoupling serverless workloads with Amazon EventBridge
Session 12/17/20, 11:15 AM

The serverless LAMP stack
Session 12/8/20, 2:15 PM

Scalable serverless event-driven architectures with SNS, SQS & Lambda
Session 12/8/20, 5:15 PM

Understanding AWS Lambda streaming events
Session 12/10/20, 10:15 AM

Choosing the right load balancer for serverless applications
Session 12/10/20, 1:45 PM

Modernize your applications with AWS Lambda and Amazon EFS
Session 12/16/20, 5:00 PM

Amazon API Gateway HTTP APIs: Beyond the proxy
Session 12/16/20, 5:45 PM

Tooling & Observability

The journey of an AWS CloudFormation template to AWS CDK
Dev Chat 12/1/20, 12:00 AM

I didn’t know AWS SAM could do that!
Session 12/9/20, 11:00 AM

CI/CD for serverless applications
Session 12/15/20, 2:15 PM

Increase availability with AWS observability solutions
Session 12/15/20, 5:30 PM

Becoming proficient with serverless application observability
Session 12/16/20, 3:00 PM

Containers

AWS Fargate: Are serverless containers right for you?
Session 12/1/20, 2:30 PM

AWS Fargate: Roadmap and vision
Session 12/16/20, 3:30 PM

Tags: aws, aws re:Invent, eventbridge, security, serverless

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

Join the Serverless Revolution!

Learning a new paradigm can be really difficult, especially something as revolutionary (and different) as serverless. Thanks to a little inspiration from fellow AWS Serverless Hero, Forrest Brazeal, I created this Hamilton parody to help teach people what serverless is all about and why it’s such an amazing way to build applications. Hopefully it inspires you as well. Enjoy!

The Cast

AWS Lambda (more info)
Event-driven Serverless Compute
Lambda has been around the cloud computing scene for almost six years now, inspiring amazing tools like the Serverless Framework as well as conferences like ServerlessDays and Serverlessconf.

Amazon API Gateway (more info)
Secure APIs at any Scale
API Gateway became a serverless staple in 2015 when it partnered up with Lambda to enable several new web application use cases. Since then, API Gateway has continued to grow, adding WebSockets, service integrations, and a new, lower latency HTTP API version.

Amazon DynamoDB (more info)
Fast and flexible NoSQL database
Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. Its popularity has been growing as developers discover single-table design.

Amazon EventBridge (more info)
Serverless Event Bus
The new kid on the block, EventBridge is a serverless event bus that connects application data from your own apps, SaaS, and AWS services. EventBridge made its debut in July of 2019, but it’s yet to receive the recognition it deserves.

Werner Vogels (@Werner)
VP & CTO at Amazon.com
While Werner is no stranger to the stage and screen, this is his first musical. His favorite genres include EDM and both Heavy & Bare Metal. Inspirations include Skrillex, Foo Fighters, and Werner Against The Machine.

The Ensemble

Andy Jassy (@ajassy)
CEO at Amazon Web Services
Andy is a Harvard grad that joined the Amazon cast in 1997. In 2003, he co-founded a new troupe called AWS, and eventually became their leader in 2016. Though he’s had tremendous career success, he still hopes to someday draw bigger crowds than Werner keynotes.

Jeff Barr (@jeffbarr)
VP & Chief Evangelist at Amazon Web Services
Jeff is a prolific blogger, video creator, and orator that has been spreading the AWS word via social media for over 18 years. Most people believe that he has figured out how to clone himself so that he can produce as much content as he does.

Special Guest

Rick Houlihan (@houlihan_rick)
Senior Practice Manager at Amazon Web Services
You may know Rick from his legendary re:Invent talks from 2017, 2018, and 2019. Rick enjoys blowing people’s minds with single-table designs, shutting down Oracle databases, and eating Torchy’s Tacos.

The DAs

The serverless Developer Advocates (DAs) at AWS are a scrappy bunch of evangelists that have been speaking at conferences around the world and creating educational serverless content like they’re running out of time. They’ve consolidated their combined work at serverlessland.com.

The Makers

Behind every great technology, there are great people. If it wasn’t for the incredibly engineers, project managers, support staff, and senior leadership, AWS Lambda wouldn’t be the amazing product it is today. The production is grateful to have a handful of them make an appearance.

The Heroes

The AWS Serverless Heroes are an amazing bunch of technologists, educators, and community leaders that push AWS serverless technology to its limits and engage with others to help shape the future of serverless. There are so many wonderful and passionate people in this group. It’s an honor to have some of them join the cast.

Special Thanks

Special thanks to Nicole Daly for making the costumes, putting up with several late nights, and being forced to listen to the recording process before auto-tune was added. Also a huge thanks to Julia Daly (age 12), who was paid $1 per Memoji creation, and for her and her sister Ellie’s body doubling performances and questionable dancing skills.

Lyrics

Who are you?
Who are you?
Who are you?
Who, Who is this kid, what’s he gonna do?

I’m gonna reduce your ops
I’m gonna reduce your ops
Hey yo, I’m unlike containers
No patching, no maintainers
And I’m gonna reduce your ops

I’ma introduce you to some cloud knowledge
I probably shouldn’t brag, but dag, I amaze and astonish
Problem is my console’s got brains but no polish
I spin up fast to impress
With every request, I’m rock solid

My security is tight, each function has a role
I give you some control, power to scale? Unimpeachable
Only five years old but my tech is stronger
And now that my timeouts are longer, I conquer
Every problem, every limitation
Every situation, think I have an integration
Step Functions for orchestration
The plan is to put your servers to shame
But damn, re:Invent’s close, so let me spell out my name

I am the L-A-M-B-D-A, see? This is what I’m meant to be
Just a function that lets events trigger me
Meanwhile, Hacker News keeps throwing shade endlessly
Essentially, they treat me like php
A new CGI-BIN’s what they think of me
But with their blinded eyes, they fail to see there will be
A serverless revolution in this century

Enter me, he says asynchronously
Don’t be shocked when Tech Crunch mentions me
I will replace your servers, set your workloads free
If need be, you can provision my concurrency

And I’m gonna reduce your ops
I’m gonna reduce your ops
Hey yo, I’m unlike containers
No pods or orchestrators
And I’m gonna reduce your ops
I’m gonna reduce your ops
I’m gonna reduce your ops
Hey yo, I’m unlike containers
No clusters, noisy neighbors
And I’m gonna reduce your ops
Time to reduce your ops!

I dream of life without Apache
No more log files pilin’ up endlessly
Or changing NGINX configs haphazardly
Lambda and I have some really good chemistry
With my ops!

My scale’s unprecedented
I’m the best No-S-Q-L DB ever invented
My queries are fast and I blast, my data’s stable
Ask Rick Houlihan what I can do with a single table
And reduce your ops

But apps’ll never be truly free
Until all events can be routed reliably
Send ‘em to me
And you’ll see
I handle events en masse
Even replace webhooks from your SaaS
And reduce your ops

It’s genius, but think how you sell it
Ops needs to agree, when Devs start to compel it
I’m with you, but your position needs thought
You have a fine line to be walked
Threaten jobs, you’re gonna get blocked

Werner, check what we got
This DynamoDB scales your data a lot
API Gateway is hot
EventBridge, I like you a lot
Together we can build apps where the servers are not
We have resilience and redundancy all in one spot
We’ll change paradigms and minds, whether you like it or not
We’re revolutionary serverless event-driven services
Get out your credentials show me where your CloudFormation is

Oh, is my Twitter too loud?
Sometimes my DAs get excited, shoot off at the mouth
My community’s amazing
And I promise that I’ll make y’all proud
Let’s get this guy in front of a crowd

I’m gonna reduce your ops
I’m gonna reduce your ops
Hey yo, I’m unlike containers
No pods or orchestrators
And I’m gonna reduce your ops
I’m gonna reduce your ops
I’m gonna reduce your ops
Hey yo, I’m unlike containers
No clusters, noisy neighbors
And I’m gonna reduce your ops

Everybody say
Whoa, whoa, whoa
Ayy, whoa (woo), whoa
Don’t let ‘em scare ya (yeah)
Let’s go
Whoa, whoa, whoa
I said we’re free when the server stops
Whoa, whoa, whoa
Said, free when the server stops
Whoa, whoa, whoa
A-come on (yeah)
Come on, let’s go

Scale up
When your traffic increases you scale up
Tell your application it’s gotta scale up
Tell your database it’s gotta scale up
When are these companies gonna scale up? (Whoa, whoa)
When are these companies gonna scale up? (Whoa)
When are these companies gonna scale up? (Whoa)
When are these companies gonna scale up?
Scale up

I imagine outages becoming just a memory
No alerts from me
In your sleep, the weekend ahead of thee
And did you know the fact, that all my apps are multi-az
Distributing load and requests with full redundancy?
Some missing use cases, but not many
Of course we’ve got serverless haters by the plenty
‘Cause we scare peeps livin’ in the past, we contrast, we’re growing fast
Gotta get woke, it’s Twenty-twenty

Scratch that, terminate your servers, join the movement
Soon Amazon will wonder where all their EC2s went
Foes oppose us, we scale with the demand
No pods or jobs to, maintain or understand
Now, think about that independence
Freedom to work on the customer experience
Or will you stay in an endless cycle of patching, reacting, and tier 1 support dispatching

I know all this stuff with K8s is excitin’
But damn, between the hype ’n’ the buzz
I’ve been bottlin’ lightnin’
I got SAM, Layers, Lambda Destinations
You want state, EFS integration!
I’m past patiently waitin’, I’m passionately surpassin’ the expectation
On every single invocation
The old way of buildin’ has no time left to borrow
I’ma power all the apps of tomorrow

And yeah now I’m gonna reduce your ops
I’m gonna reduce your ops
Hey yo, I’m unlike containers
No patching, no maintainers
And I’m gonna reduce your ops
We’re gonna scale up (time to reduce your ops)
I’m gonna reduce your ops
We’re gonna scale up (time to reduce your ops)
I’m gonna reduce your ops
We’re gonna scale up, scale up
It’s time to reduce your ops
Scale up, scale up
It’s time to reduce your ops
Scale up, it’s time to reduce your ops
Scale up, reduce your ops, ops, ops
It’s time to reduce ops, time to reduce ops
I’m gonna reduce your
I’m gonna reduce your… ops!

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

Welcome to Issue #103 of Off-by-none. Thanks for joining us!

Last week, we got some awesome insights into serverless and cloud security and took a look at serverless machine learning. This week, we learn why you don’t need to buy a cow to do machine learning, how serverless can speed up your regression tests and we get a full season of Sessions with SAM to binge watch. Plus, we have plenty of great serverless content from the community.

Before we begin, make sure you check out TriggerMesh. They let you connect all your key SaaS and on-premises apps with AWS EventBridge. And you can also trigger workloads running on any managed Kubernetes cloud. Join the Beta program for Free! Sponsored

When you’re looking for serverless success stories… ?

“Some of the integrations that weren’t useful anymore, remained in production as nobody was sure if they will break something else by removing it.” If that sounds like an old app you’re working with, see how Marin Radjenovic migrated a legacy application to serverless with EventBridge.

And if that’s not exciting enough for you, check out this post by Fernando Devegili that details how his team used serverless to reduce the amount of time it took to do visual regression tests from 20 minutes to 10 seconds.

When you’re looking for some good serverless use cases… ?

Why worry about writing an entire backend when you can just send e-mails right from Cloudflare Workers? Or if you’d like to create an SVG graph from your SQLite database, Alejandro Oviedo will show you how to do it using a Lambda function.

Jerry Mullis shows us how to stop EC2 instances on a scheduled basis, one of those classic serverless DevOps use cases. But if you want to get really meta, how about using serverless to manage its own scalability by scheduling AWS Lambda Provisioned Concurrency?

When there’s more to learn about serverless… ?‍?

Paul Swail answers “where does Fargate fit into your serverless toolkit?” (just in case you had that question too), Yan Cui gives us some tips for debugging AWS Lambda timeouts, and Serkan Özal gives you the pros and cons of using a mono- or multi-repo in a serverless world.

There’s an entire series on running Ruby on AWS Lambda from the team at Hint Media, Ernesto Marquez explains when you should use Amazon RDS vs. Aurora Serverless, and Aman Sharma shares an introduction to Serverless/Edge scripting.

If you want to go back to basics, Shane Lee gives a brief 10-point guide to AWS Lambda. And if you’d like to explore more complex concepts, like static IPs for AWS Lambdas, Austin McDaniel has some ideas for you.

No time to write tests? Jan Groth has  12 recommendations on unit-testing AWS Lambdas in Python. There’s some good advice in here, like not bothering to test boto or other frameworks. They’re probably already well tested, so you should focus on your own business logic.

James Beswick has a new series that will tell you why using serverless backends to iterate quickly on web apps is important to rapid development and deployments, and Yan Cui has another great post that explains what alerts you should have for your serverless applications. Once you figure that out, James Allardice will demonstrate the process of setting up CloudWatch alarms with serverless.

When you just want to build something… ?‍♀️

Auto-instrumented, trace-centric observability for serverless & k8s
If you’re running distributed serverless, Kubernetes, or containerized environments, you need a specialized tool that’s built to monitor them. Epsagon enables teams to automatically detect, troubleshoot, and resolve issues within complex microservices with no training, manual coding, tagging or maintenance required. Try free today, connect your first trace instantly, and grab one of our “cloud observability” drones! Sponsored

If you’re using Flutter to build native apps and you need a serverless backend, check out this post on full-stack Flutter that’ll show you how to create a backend for your app using AWS & serverless.

Just need to deploy a static site? Abhishek Wagh shows you how to deploy your websites with AWS Amplify (no, not the framework, the console). While you can deploy backends with the Amplify Console as well, Netlify has a pretty slick setup. You may want to check out this introduction to serverless functions.

Ben Smith isn’t ready to quit on the whole serverless PHP thing, so he wrote a post introducing the CDK construct library for the serverless LAMP stack. Regardless of your runtime of choice, implementing a passwordless API using Cognito and the Serverless framework might be a cool addition to your serverless apps.

What to do to tighten up your serverless security… ?

Another take on the recent ESG study points out that half of DevOps teams surveyed pushed vulnerable code “because the vulnerabilities identified were discovered too late in the [development] cycle to resolve them in time.” Um, I know lots of us face a tremendous amount of pressure to ship stuff, but knowingly shipping vulnerable code should be a show stopper.

Another interesting post I read was about the dangers of Denial-of-Wallet attacks. I think the possibilities of DoW attacks are very real and that you should absolutely put in place mitigation strategies in order to deal with them if they occur. That being said, I haven’t heard any major horror stories about this other than from devs accidentally doing it to themselves. I’m not saying this won’t be a thing, I just haven’t seen it yet.

When you’re looking for some interesting reading… ?

Charity Majors has a great post called The Future of Ops Jobs. In it she explains how Ops are not going away, but morphing into something different than you may be used to. Lots of great advice in here for those Ops people looking to advance their careers.

Remember when people were saying that serverless was a spectrum? Well Gillian Armstrong says that Machine Learning is spectrum and explains how to cheat at it by not owning a cow. There’s a helpful parable in here, but all you really need to know is that buying butter isn’t cheating. ?

Not sure I’d read this to my kids, but Mariliis Retter wrote a “techy bedtime story” explaining how serverless came to be; complete with wizards, caves, and server monsters. In other serverless analogies, Shrikant Navelkar explains why serverless is like good portion control and helps us overcome the sunk cost fallacy.

When your Netflix Watch List is empty… ?

On Episode #62 of the Serverless Chats podcast, I had a chat with Nica Fee about the recently released New Relic One Platform, how observability can both reduce MTTR and help us optimize our applications for happier customers, and how New Relic is embracing open source and making observability available to teams of all sizes. Watch this episode on YouTube!

Yan Cui had an awesome conversation with Zack Kanter from Stedi on Real World Serverless, Episode #24. There’s a great discussion in here about tenancy and how Stedi thinks about it.

And if you’re looking for something to binge watch, all the episodes from Season one of Sessions with SAM starring Eric Johnson are nicely curated in this convenient blog post.

When you realize that apparently the devs at AWS never sleep… ☕️

There was a lot happening this past week at AWS. Let’s see if we can give you a quick summary of some of the major points. First of all Quantum computing is now available on AWS through Amazon Braket. I know it has nothing to do with serverless, but Jeff Barr is really excited about it.

ElastiCache also had a big week, announcing support for resource-based permission policies and ElastiCache Redis now supporting up to 500 Nodes per cluster. And not to be outdone, Amazon DynamoDB Accelerator (DAX) now supports next-generation, memory-optimized Amazon EC2 R5 nodes.

Moving on to more serverless related announcements, AWS is EOLing  Amazon Linux 1 support in Lambda on December 30, 2020. Eric Johnson shows you how you should be migrating AWS Lambda functions to Amazon Linux 2 (note that this means updating your runtimes). In related news, AWS Lambda now supports custom runtimes on Amazon Linux 2 as well as support for Go on Amazon Linux 2.  Plus AWS Lambda now supports Java 8 (Corretto), if you’re into that sort of thing.

In slightly more exciting news, AWS Lambda now supports Amazon Managed Streaming for Apache Kafka as an event source, which James Beswick explains how to set it up here.

AWS Step Functions added support for string manipulation, new comparison operators, and improved output processing, giving customer more flexibility within their state machine definitions and allowing for more dynamic behavior. This post goes into more detail and shows you some of the cool new things you can do.

Amplify iOS v1.1 was released, which includes support for the Combine framework for Swift, Amazon Comprehend added five new languages to Custom Entity Recognition, and Amazon Cognito User Pools now support customization of token expiration.

You can also set up CloudWatch Application Insights using CloudFormation, use the Copy API with Amazon S3 Access Points, and check out the updated Serverless Bot Framework solution that added support for securely integrating with third-party APIs.

When you need some handy serverless tools… ?

Any good builder needs their trusty tool belt, and if you’re building with serverless, sls-dev-tools gives you much of what you need. Ben Ellerby outlines some of the most useful features.

Ran Isenberg wrote up a piece on how to do AWS Lambda Event Validation using a tool called Pydantic. If you’re writing functions in python, this could be worth checking out.

Thoughts from Twitter ?

“I will die on this hill: One application environment per AWS account.” ~ Tom McLaughlin
As Tom is from Boston, I’ll assume he means Bunker Hill. Regardless, one application environment per account should be the absolute minimum. I heard AWS does one account PER MICROSERVICE. Think about that! ?

Serverless Jobs ?‍? Sponsored

Serverless Engineer – stedi.com
At Stedi, we’re working in one of the biggest markets on the planet – EDI, the technological backbone of the physical product economy. We’re building a next-generation platform: a ubiquitous commercial trading network to automate the trillions of dollars in B2B transactions exchanged by nearly every company on Earth. If you’re interested in what we’re building and how we’re building it, we’d love to hear from you.

Have a job listing you’d like to share? Please contact me for more information. 

Upcoming Serverless Events ?

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

August 19, 2020 –AWS Lambda Data Storage: Choosing Between S3, EFS, and Local Storage (AWS Tech Talk)

August 24, 2020 –Embedding Analytics in Your Applications (AWS Tech Talk)

August 25, 2020 – 3 hours of Serverless Conference

September 1, 2020 – Deliver Business Value Faster with AWS Step Functions

September 3, 2020 – ServerlessDays Warsaw

October 1-2, 2020 – ServerlessDays Hamburg 2019

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Jiju Thomas Mathew (@jijutm). Jiju is an Associate Process Manager at UST Global and an AWS Community Builder. He was previously CTO and cloud computing advisor for Innoval Digital Solutions Pvt Ltd. Jiju is the co-founder of an AWS User Group in Trivandrum, and has organized and spoken at meetups and community days for local developers. Thank you, Jiju, for being a leader and contributor within the serverless community!  ?

Final Thoughts ?

Every week I try to make this newsletter shorter, and every week I seem to fail miserably. I even cut out two whole sections this week! I love reading all these stories and sharing them with you, so that won’t stop. But I am implementing something new that will trim some of the fat. It should ? be ready next week. If you’ve read this far, you are a saint and I appreciate your commitment. I welcome any other ideas you may have to make this easier to consume.

I hope you enjoyed this newsletter. We’re always looking for ideas and feedback to make it better and more inclusive, so please feel free to reach out to me via Twitter, LinkedIn, Facebook, or email.

Cheers,
Jeremy

P.S. If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless.

Share this newsletter with your friends and coworkers!

Off-by-none is a weekly email newsletter that focuses on the technical details of building applications and products in the cloud using serverless technology. Together we can become better developers and product people by making fewer mistakes as we learn from each other. Off-by-none is the idea that we can become exceptional at what we do, if we are willing to put in the work. Join me on this journey as we help the community work to develop best practices, share our ideas, and learn to build better cloud-based software.

I respect your privacy and I will NEVER sell, rent or share your email address.

Welcome to Issue #102 of Off-by-none. I’m happy you’re here!

Last week, we shared some dissenting opinions on serverless, but also got a bunch of awesome new tools. This week, we have some awesome insights into serverless and cloud security, we take a look at serverless machine learning, and try something a little different with the newsletter format. Plus, we have lots of great serverless content for you to explore.

When you’re curious what’s new with serverless… ?

Check out the TriggerMesh guide “What Every CIO Needs to Know about Serverless” for feature comparisons of major offerings and an introduction to basic concepts like cloud native and microservices. Use it to inform your peers / managers on serverless benefits and available options. Sponsored

The Dashbird has a busy week. Not only did they just turn 3 years old, they also introduced Dashbird Atlas, a new real-time 3D map view of your entire serverless environment. And if that’s not enough, they also added insights for AWS Kinesis and Step Functions.

In other, sort of serverless news, WayScript just raised $5 Million. Last week we mentioned Paragon, which seems quite similar. Sort of like IFTTT, but with a lot more flexibility and customizations. I have a feeling we’ll see more and more systems like this that’ll continue to build higher level abstractions on top of serverless infrastructure.

When everyone starts adding buzzwords back into their pitch decks… ?

Machine Learning is all the rage nowadays, and unlike several years ago when ML/AI was the cornerstone of every investor presentation, lots of companies are now actually able to deliver on that promise. An oft repeated criticism of Lambda (and most FaaS in general) has been its inability to handle machine learning use cases. But thanks to the recent addition of EFS support, and a clever post by Shitij Mathur, setting up serverless machine learning inference with AWS Lambda couldn’t be easier. ?

There are plenty more examples, include Mike Palei’s post that shows you how to serve a Tensorflow 2 model on AWS Lambda. Or Joshua Jarvis’s Going Serverless for Your ML Backend with SAM CLI. For a more in depth discussion on what’s possible with ML and serverless, check out Datacast Episode 39: Serverless Machine Learning In Action with Carl Osipov.

What to do if PHP is still your jam… ?

Not to worry, the “Serverless LAMP stack” is definitely a thing. Ben Smith from AWS has another installment in his Serverless LAMP stack series, this time showing you how to build a serverless Laravel application and deploy it with AWS SAM.

And before you say, “hey, I bet Ben just made this whole thing up!”, Chandresh Singh also has a post that will show you how to setup a queue with Serverless Laravel using bref. If Laravel isn’t your thing, Smaine Milianni has a post that’ll show you how to deploy a Symfony application in AWS Lambda.

When you don’t want to repeat someone else’s mistakes… ?‍♂️

The Interwebs is filled with stories of success, but oftentimes learning what not to do can be even more instructive. Chris Plankey has a great post titled How I Failed at Generating Revenue with Alexa In-Skill Purchases that details his attempt to bring in some extra cash with a side hustle. We’re all prone to making bad assumptions, but maybe this will help you avoid a few.

Even if all your assumptions are correct, you’re still bound to hit some roadblocks when building out your serverless applications. Dimitri Saeys recounts how Sporza created a real-time sport data api (in tremendous detail). His team followed an interesting process, and there’s plenty to learn from their journey.

When you’re looking for a serverless solution to your problem… ?

If you still list DevOps on your LinkedIn profile, then perhaps using serverless for IAM user monitoring or sending bash commands to an EC2 cluster with AWS Lambda and SSM might be interesting to you.

For some simpler automations, Harish Aravindan can show you how to get notified on pull requests via Slack using AWS Lambda or you can see how Javaad Patel gets page views from Google Analytics using a Netlify Serverless function.

For the truly ambitious, Nghia Dang has an excellent post that shows you how to implement a highly available, scalable and cost-efficient video processing service using AWS Lambda, SNS and SQS. And if that’s not big enough for you, take a look at how AWS serverless architecture drives SBA’s Lender Gateway for PPP loans.

Where to go to level up your serverless knowledge… ?

Yan Cui compares choreography vs orchestration when dealing with serverless function composition. As he points out, both have their pros and cons, with Step Functions being the clear winner for mission critical workflows. Lots of useful information in here, but I still think there are cross-service use cases for Step Functions. Perhaps a discussion for another day.

Of course, Step Functions can get expensive, which is why Renato Byrro offers some suggestions on cutting Step Functions costs on enterprise-scale workflows. He mentions “orchestration” with EventBridge, but even though there are strong guarantees, this still falls into the “choreography” bucket for me.

If you’re still not fully up to speed on the magical service that is EventBridge, then perhaps this post on migrating applications to cloud with Amazon EventBridge by Emrah Samdan will fill in some gaps.

And speaking of magic, the new Direct Lambda Resolvers for AppSync are pretty darn cool as well. John Connerton not only helped build this new feature, but also explains why they are really effective.

Still thirsty for some serverless know-how? You can learn about Netlify Functions in 2-ish minutes, let Michael Bahr show you how to archive your AWS data to reduce storage costs, or get help from Ran Ribenzaft on how distributed tracing differs from logging. Plus, Lou Bichard can tell you why you’re alerting wrong with CloudWatch and Yossi Ittach will give you a rundown of your options for pub-sub services on AWS.

When you want to get hands-on with serverless… ?‍♀️

Even though your code coverage is probably zero (don’t worry, I know you’ll eventually get to it), Mohammed Izzy shows you how to easily perform tests on your IaC code with Pulumi, so one less excuse. ?

For those of you looking for a full-stack experience, Fidel Vazquez has a tutorial on designing and deploying a full-stack web application using AWS Amplify. Ashan Fernando shows you the simplest way to host a Gatsby website in AWS without servers, and Atila Fassina gives you the deets on deploying Netlify Functions with TypeScript.

Yi Ai has another excellent tutorial that shows you how to create an instance scheduler using AWS CDK, a common use case that I’m sure you’ll need to address at some point. You’re also likely going to need to process large S3 files with AWS Lambda, so give that one a look as well.

Simran Kaur Kahlon shows you how easy it is to setup an AWS Private API using Serverless Framework, and if you need multiple API versions in serverless microservices, Venkatachalam Dekshinamurthy will tell you how he does it (without using Route53).

If you’re ready to go with Go in Lambda, you might find this Ultimate Guide by Sebastian Karasiewicz handy. For something a little less involved, take a look at Adis Kovacevic’s post on how to add Algolia Search to Firestore using a cloud function.

When you need some serverless security best practices… ?

ICYMI: Serverless Security Strategies – A Recorded Session with AWS Principal Engineers
AWS Senior Principal Engineers Becky Weiss and Marc Brooker cover the most common areas of vulnerability including misapplied patches and updates, malicious code and runtime security, overly permissive access, and network segmentation, and share the ways that serverless technologies address these areas. This talk gives you a solid understanding of how a serverless security strategy functions technically and how it can be operationalized in your business. Sponsored

In addition to the excellent recommendations from Becky and Marc above, Anthony DiMarco offers up some best practices for serverless endpoints on AWS. For more insights into dealing with serverless security vulnerabilities, here’s how Cloudflare handles them.

Serverless certainly shifts more responsibility onto the developers, so it’s no surprise that security analysts want more help from developers to improve DevSecOps. After all, “only 15% of respondents said that a majority of developers participate in formal security training.” Couple that with the fact that a recent report showed misconfigured cloud storage services were prevalent in 93% of analyzed cloud deployments, it’s no wonder that the research suggests that cloud breaches will become faster and bigger. If you’re using serverless, you’re already in a much better position.

When you’re looking for some good serverless reads… ?

Corey Quinn brilliantly explains why multi-cloud is the worst practice. Not only does it force you to choose the least common denominator, but the likelihood of cloud agnostic workloads actually existing, is probably quite slim.

Sheen Brisals extends his theory of thinking and developing serverless applications as “Set-Pieces” in his most recent piece. This ties in nicely with Haim Raitsev’s Monolithic to Microservices post.

David Anderson explains how serverless helps enterprises scale DevOps, and how they did it at Liberty Mutual, while Brian Foody suggests that startups should be “Day One” Cloud Native Organizations.

I’m not sure who gave Forrest Brazeal the title of “Cloud bard”, but it’s spot on! He was interviewed on theCube and outlines common paths to failures (and successes) in enterprise cloud native evolutions. Unfortunately, there are no rap battles or cloud ballads performed during the interview. ?

Ben Ellerby has a great piece on Serverless BI and how serverless helps companies on their path towards data-driven Business Intelligence. And speaking of data, this early Twitter engineer has a suggestion for your next database.

When you’re looking for something serverless to listen to… ?

I had a great chat with Heitor Lessa on Episode #61 of the Serverless Chats Podcast. We discussed the the Well-Architected Serverless Lens and how best practices are defined, how services are chosen for the serverless lens, and what new services will be added to the Lens this year. You can also watch it on YouTube.

On Episode #23 of the Real World Serverless Podcast, Yan Cui talks with Ari Palo about how they do serverless at Alma Media and the third-party development tools they use to optimize their workflow.

When you’re curious what’s new with AWS… ?

AWS AppSync releases Direct Lambda Resolvers for GraphQL APIs, which we already mentioned, but it’s so cool that it deserves to be mentioned again. I’ve got nothing against VTL, but if you are rapidly prototyping something, this just feels like an easier way to get started. Ed Lima does a better job explaining it than I can.

AWS Lambda now provides IAM condition keys for VPC settings so you can ensure that users can only deploy functions connected to one or more allowed VPCs. Julian Wood gives some good use cases for it and why it might be right for you.

I just found out that Amazon API Gateway HTTP APIs now support wildcard custom domain names, which I didn’t realize wasn’t possible before they announced this. I’ve still yet to make HTTP APIs my go-to flavor of API Gateway. ?

In AWS ML news, Amazon Transcribe launches custom language models to let you add “out-of-lexicon terms.” In my experience, that includes most words in the English language ?, but I’m sure it’s getting better all the time. Plus, Amazon Lex launches accuracy improvements and confidence scores, which will allow you to use business logic to help clarify intents. And AWS Step Functions adds support for Amazon SageMaker Processing with help from the Step Functions Data Science SDK. So that’s another thing I’ll add to my “probably need to learn this someday” list.

When you need some helpful serverless tools… ?

Neiman Marcus open sourced a Serverless Framework Plugin that gives you better control over Provisioned Concurrency Autoscaling.

Dynatron is a new NPM package that can “bridge between AWS DynamoDB Document Client and real world usage.” Gevorg Galstyan explains that this can find missing optimizations and hidden issues that are very hard to catch. Sounds interesting to me.

Midway Serverless is a “serverless framework used to build Node.js cloud functions and it helps you significantly reduce maintenance costs and focus more on product development in the cloud-native era.” Who doesn’t need another serverless framework? ?

Serverless Jobs ?‍? Sponsored

Serverless Engineer – stedi.com
At Stedi, we’re working in one of the biggest markets on the planet – EDI, the technological backbone of the physical product economy. We’re building a next-generation platform: a ubiquitous commercial trading network to automate the trillions of dollars in B2B transactions exchanged by nearly every company on Earth. If you’re interested in what we’re building and how we’re building it, we’d love to hear from you.

Have a job listing you’d like to share? Please contact me for more information. 

Upcoming Serverless Events ?

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

September 1, 2020 – Deliver Business Value Faster with AWS Step Functions

September 3, 2020 – ServerlessDays Warsaw

October 1-2, 2020 – ServerlessDays Hamburg 2019

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is George Mao (@georgemao). George is the WW Technical Leader for Serverless Computing at AWS. He is a regular speaker at AWS Summits, re:Invent, and various tech events. George also contributes to open source projects and works with customers to design their applications in the cloud. Thanks for all of your contributions and work on serverless, George! ?

Final Thoughts ?

So, for those of you that have been around for a while, you may have noticed that I changed the format of the newsletter a bit. I’ve received a tremendous amount of feedback from a lot of you (which I greatly appreciate), and I’ve tried to start incorporating some of your suggestions. The biggest issue for me with the old format was that it had become terribly limiting, and in many cases, didn’t give me an opportunity to add the right amount of context. I have many more thoughts and ideas on additional changes that I plan on making (which I’ll share in a blog post soon), but I’m a sucker for feedback, so I’d love to know your thoughts.

I hope you enjoyed this newsletter. We’re always looking for ideas and feedback to make it better and more inclusive, so please feel free to reach out to me via Twitter, LinkedIn, Facebook, or email.

Take care,
Jeremy

P.S. If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless.

Share this newsletter with your friends and coworkers!

Off-by-none is a weekly email newsletter that focuses on the technical details of building applications and products in the cloud using serverless technology. Together we can become better developers and product people by making fewer mistakes as we learn from each other. Off-by-none is the idea that we can become exceptional at what we do, if we are willing to put in the work. Join me on this journey as we help the community work to develop best practices, share our ideas, and learn to build better cloud-based software.

I respect your privacy and I will NEVER sell, rent or share your email address.

Welcome to Issue #101 of Off-by-none. Thanks for joining us!

Last week, we celebrated our 100th issue, and at the same time, realized we have a lot more to share about serverless! This week, we get some dissenting opinions on serverless while at the same time getting a bunch of awesome new tools to make it even better. Plus, we’ve got plenty of great posts from the serverless community.

Serverless News & Announcements ?

The brand new TriggerMesh integration with Amazon EventBridge lets you use any TriggerMesh source in EventBridge. Our sources include GitHub, GitLab, Slack, Zendesk, Azure Activity Logs, as well as on-premises apps. Learn more here! Sponsored

New Relic pricing plummets with product overhaul
New Relic just released their New Relic One platform, which is a complete overhaul of their existing system. This is a great article that sums up the need to reposition in an increasingly saturated cloud observability market.

YC alum Paragon snags $2.5M seed for low-code app integration platform
This is an interesting new tool that is built on a AWS serverless backend. You still need quite a bit of technical knowledge, but most of the infrastructure is abstracted away for you. Give me a way to do IaC with this, and I might be very interested.

Announcing the New AWS Community Builders Program!
Looking to get access to AWS product teams, information about new services and features, mentorship from AWS experts, Promotional Credits, and other helpful resources to support content creation and community-based work? Check out this new program from AWS.

Rockset Shatters Operational Barriers for Real-Time Analytics
Maybe this is just an overhyped press release, but the idea of “Converged Indexes” that support real-time SQL queries on semi-structured data at massive scale is pretty exciting. I was just chatting with someone about RocksDB the other day, and they have been very happy with it.

Serverless Stories ?

How I scaled an AppSync project to 200+ resolvers
If you’re working on a massive AppSync project, you definitely want to read how Yan Cui built one with over 200 AppSync resolvers and 600 CloudFormation resources.

How I Built a Serverless Search for My Blog
This is amazing. Probably total overkill, but amazing nonetheless. Gunnar Morling used Java and the Apache Lucene full-text search library, compiled into a native binary using the Quarkus framework and GraalVM, to deploy a serverless search engine on Lambda.

AWS DynamoDB: Writing millions of records: Not consuming provisioned throughput: Global Secondary Index and Sharding
This article does a great job explaining why write sharding might be necessary when using low-cardinality partition keys in secondary indexes. If you’re writing lots of data to DynamoDB, definitely give this a look.

Serverless Concepts ?

Understanding database options for your serverless web applications
Helpful post by James Beswick that outlines a few patterns that can be used to handle database interactions from your serverless applications. The relational database story for AWS and serverless is getting much better (thanks RDS Proxy), but it’s still not quite there yet.

The Pains of Testing Serverless Applications
Having difficulty testing your serverless applications? You’re not the only one. Paul Swail reveals some of the problems and complaints from developers collected through his recent serverless survey.

AWS Compiled 001
Alejandra Quetzalli has put together (what I think) is the first issue of an AWS newsletter. This one focuses on Java and Serverless. I hope there are more of these.

Best Practices for an Improved Serverless Developer Experience
I think Harsh Chaudhary has a lot of really good recommendations in here, but I’m personally not a huge fan of mocking cloud services locally. But, hey, whatever works for you. That’s what’s important.

Serverless Tutorials ?‍♀️

Auto-instrumented, trace-centric observability for serverless & k8s
If you’re running distributed serverless, Kubernetes, or containerized environments, you need a specialized tool that’s built to monitor them. Epsagon enables teams to automatically detect, troubleshoot, and resolve issues within complex microservices with no training, manual coding, tagging or maintenance required. Try free today, connect your first trace instantly, and grab one of our “cloud observability” drones! Sponsored

Serverless Framework & Immutable Packaging: How to
Samuel Cozannet outlines a strategy for creating immutable deployment packages with the Serverless Framework. It’s an interesting approach, though it still requires the framework to rebuild the CloudFormation templates on every deploy. As soon as you throw certain plugins into the mix, everything goes out the window.

Why You Should Never, Ever print() in a Lambda Function
Paul Singman offers some advice for you to “upgrade your dance” when it comes to logging in your Lambda functions. Also, take a look at the structured logging format provided by AWS if you really want to take it to the next level.

Building well-architected serverless applications: Controlling serverless API access – Part 1, Part 2, and Part 3
I love this series by Julian Wood. It takes you through all the steps and questions you need to be asking to make sure your serverless app’s API access is secure and well-architected.

Serverless Security ?

Study finds misconfigured cloud storage services in 93% of cloud deployments analyzed
I wish this came as a surprise. This has “led to nearly 200 breaches that have given attackers access to 30 billion records over the past two years.” ?

Serverless Reads ?

The AWS Serverless LAMP Stack: The Future of PHP or Vendor Lock-in?
The good news about this article is that they don’t criticize serverless, just the proprietary nature of building a LAMP stack on top of Lambda and the AWS ecosystem. I’ll say one thing about vendor lock-in. If I had a nickel for every time I had to move my production application to another provider… I’d be broke.

My monolith doesn’t fit in your serverless
Gabriel Chertok gives some reasons why he thinks serverless isn’t right for apps that live somewhere between really simple and infinitely scalable. He makes some good points, and I’ve been arguing for awhile that I think we’ve added too much complexity to the serverless paradigm. What are your thoughts?

Why the Benefits of Serverless May Never Be Realized
“I think serverless as a whole has been over-hyped. It may be part of a broader cloud-native computing revolution, but serverless on its own falls short of being revolutionary. It’s a handy technology, but it’s not exactly a pathbreaking or singularly disruptive one.” I strongly disagree. Yes it has constraints, but I’ve never worked with a technology that offered this much potential.

For the podcast listener… ?

Episode #60: Going Green with Serverless with Paul Johnston (Part 2)
In part 2 of this two-part episode, I finish my chat with Paul Johnston about how the big cloud providers are addressing climate change, what the tech industry can do as a whole, the effect this will have on globalized business, and how the positive downstream impact of choosing serverless can create a more sustainable business.

New from AWS ?

Amazon Kinesis Data Firehose now supports data delivery to HTTP endpoints
This feature enables you to use a fully managed service to stream data to an HTTP endpoint without building custom applications or worrying about operating and managing the data delivery infrastructure. I’m down for that! It also supports data delivery to Datadog, New Relic, and MongoDB Cloud.

AWS CodeBuild supports code coverage reporting
This is a nice new feature. Supports any test framework that outputs files for JaCoco XML, SimpleCov JSON, Clover XML, and Cobertura XML.

Amazon GuardDuty expands threat detection coverage to help you better protect your data stored in Amazon S3
Threat detection coverage has been broadened to monitor for highly-suspicious data access and anomaly detection to help you better protect your data residing in Amazon Simple Storage Service. We now know that 93% of their customers need it. Read more.

Serverless Tools ?

Is serverless cheaper for your use case? Find out with this calculator.
Xavier Lefèvre has put together a really helpful serverless calculator that has a ton of default values already configured. You’re free to use the official AWS cost calculator, but this one assumes things based on a “typical” serverless application.

5 Serverless Security Tools You Should Be Using
Miguel Calles put together a list of serverless security tools for you. There are lots of other monitoring/observability tools available, but it’s a handy reference.

CLI tool to Query the AWS Aurora Serverless Data API
Chris Bailey wrote a python script that lets you query your Aurora Serverless cluster right from your CLI. Great little tool for debugging and development.

Serverless Jobs ?‍? Sponsored

Serverless Engineer – stedi.com
At Stedi, we’re working in one of the biggest markets on the planet – EDI, the technological backbone of the physical product economy. We’re building a next-generation platform: a ubiquitous commercial trading network to automate the trillions of dollars in B2B transactions exchanged by nearly every company on Earth. If you’re interested in what we’re building and how we’re building it, we’d love to hear from you.

Have a job listing you’d like to share? Please contact me for more information. 

Upcoming Serverless Events ?

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

September 3, 2020 – ServerlessDays Warsaw

October 1-2, 2020 – ServerlessDays Hamburg 2019

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Nicole Yip (@Pelicanpie88). Nicole is an Engineering Manager at the LEGO Group, where, prior to her recent promotion (congrats!) she has been involved with the platform team looking after the hosting infrastructure (AWS), CI/CD and dev experience, SRE, operations and security for both application and infrastructure. Her experience as an infrastructure and DevOps engineer has allowed her to help teams of all sizes get their services to AWS. Nicole has also spoken at the Serverless Architecture Conference and DevOps Con. Thank you, Nicole, for helping others make the jump to serverless! ?

Final Thoughts ?

There were several new tools released this week that help with building serverless applications. From New Relic’s new platform, to TriggerMesh’s EventBridge integration, to Xavier Lefèvre’s Serverless Cost Calculator, all of these investments are a continued testament to the growing power and popularity of serverless.

While there were several articles this week that criticize serverless, it seems to be the same old recycled arguments, many of which have been addressed by the cloud providers or the community. We just need to keep fighting the good fight and spreading the serverless word. Eventually the dissenters will find themselves left in the dust with a whole bunch of servers to manage.

I hope you enjoyed this newsletter. We’re always looking for ideas and feedback to make it better and more inclusive, so please feel free to reach out to me via Twitter, LinkedIn, Facebook, or email.

See you next week,
Jeremy

P.S. If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless.

Share this newsletter with your friends and coworkers!

Off-by-none is a weekly email newsletter that focuses on the technical details of building applications and products in the cloud using serverless technology. Together we can become better developers and product people by making fewer mistakes as we learn from each other. Off-by-none is the idea that we can become exceptional at what we do, if we are willing to put in the work. Join me on this journey as we help the community work to develop best practices, share our ideas, and learn to build better cloud-based software.

I respect your privacy and I will NEVER sell, rent or share your email address.

Welcome to issue #100! Thanks for being here! ?

Last week, we got a Q2 serverless recap, someone realized that NAT Gateways were expensive, and we learned how to model our serverless apps in a way that allows us to evolve them more easily.

This week, we’re celebrating the 100th issue of Off-by-none with what you’ve come to know, expect, and hopefully love: the many, many great serverless articles, projects, and tools from our community. But what’s more: I’m taking the time to reflect on the wonderful community we’ve built here, in your inbox, and sharing some ways I hope to improve your newsletter experience.

I’ve always wanted this newsletter to recognize the community’s vast and great work, and be inclusive and representative of those who are in it. I want you to feel like you’re a part of this newsletter and that you can see yourself in this community. To better accomplish that, I’m in the final stages of launching the new Off-by-none website. Though I’ve still got some work to do on it, it’s the vision of the site I am most excited about. I hope that it will be a place where I can facilitate more engagement, invite more submissions, ideas, and stars, and better amplify the many voices that make this community so great. If you have any suggestions, as always, please feel free to let me know, as I want the Off-by-none site to truly be a place for all of us. ?

Serverless News & Announcements ?

TriggerMesh helps you do more with serverless
Connect GitLab with Confluent and Zendesk, or connect these and dozens of other apps with AWS services like Kinesis, SQS, SNS, and Cognito. Even integrate with EventBridge. Join the Beta program today! Sponsored

Cloudflare launches Workers Unbound, the next evolution of its serverless platform
Oh wow, this looks amazing. Workers Unbound is meant for more complex and longer-running processes and operates at a fraction of the cost of Lambda, Azure Functions, and Google Cloud Function.

The AWS Serverless Application Model CLI is now generally available
In case you don’t know, the AWS SAM CLI is the command line tool companion for AWS SAM templates. It provides developers local tooling to create, develop, debug, and deploy serverless applications, and now it’s finally GA.

The Node.js Runtime
FYI, Google Cloud Functions now support Node.js 12 (in beta). I’d love to see Google become more competitive with some of their serverless offerings. Node 12 was released over a year ago.

Epsagon Further Accelerates Growth with New Investment from DTCP
Congrats to the team over at Epsagon! This latest investment brings their total funding to $30M which will help them increase their team to service their growing customer base. Apparently observability in the cloud is getting very popular.

Serverless Stories ?

We went serverless with our new startup. And you should (probably) too.
I totally agree. Ankit explains why he chose serverless and what some of the benefits are. He gives some reasons when not to go serverless, but I think we’ve addressed those objections many times over.

A Plumber’s Guide to Cloud
Great post by Daniel Singletary, who took his experience as a plumber and applied the same troubleshooting methodology to complete Forrest Brazeal’s “Cloud Resume Challenge”.

A micro API in Firebase and Cloudflare Workers
This was a fun little experiment by Braeden Smith that compared building a simple micro API for loading an image in both Firebase and Cloudflare Workers. Fun Fact: Cloudflare won.

Serverless Use Cases ?

Serverless magic links with AWS Cognito
Here’s a really interesting use case for serverless that takes advantage of Cognito’s multi-factor login support to allow for temporary “magic links”. Thomas Schoffelen explains how secure it is, how it scales, and provides some code for a proof of concept.

Functions as a Service — A simple use case
Need to catch a webhook and publish a message to the Slack API? Pavel Pratyush explains why serverless is the perfect solution for this simple use case.

Processing PGP Encrypted Data in AWS Lambda using Python.
Here’s an interesting serverless use case. Mayank Madhukar shows you how to use python-gnupg to decrypt emails from an S3 bucket using Lambda.

Serverless Concepts ?

Monitor & Debug Serverless with 30-Second Auto-Instrumentation
Complete observability over your serverless environment with no code changes required. With one-click distributed tracing, Lumigo lets you effortlessly find & fix issues in serverless and microservices environments. Get serverless-specific smart alerts before they impact performance or cost. Sign up for a free account and get up & running in minutes. Sponsored

Real-time Data Processing With Kinesis Data Analytics
Nick Van Hoof and David Smits put together an excellent post that shows the serverless architecture they use to track, analyze and alert on traffic jams in Belgium using (near) real-time analytics processing. Very cool stuff.

Breaking Down Serverless Anti-Patterns
Parts of this post by Sarjeel Yusuf are a bit hard to follow, but the overall point should be well taken. We have to think and operate differently when building serverless applications, and in many circumstances, best practices are still a moving target.

Connecting Accounts with AWS Event Bridge
I like John Gilbert’s overall approach here, especially separating subsystems at the AWS account level to create bulkheads to contain the blast radius. I’m curious if there might be an alternative approach that doesn’t require upstream subsystems to know about other subsystems? Perhaps creating a common bus in a separate account that all subsystems can push to and read from?

Five Reasons You Should Consider AppSync Over API Gateway
Excellent breakdown by Yan Cui that outlines and explains five really good reasons to give AppSync another look.

Replacing web server functionality with serverless services
Another great post by James Beswick that explains how to significantly reduce your infrastructure requirements, setup time, and maintenance needs by switching to a serverless approach. Getting all that scalability and multi-AZ redundancy right out of the box should make the choice simple.

Serverless Tutorials ?‍♀️

Implementing a Serverless Speech-To-Speech Language Translation App
Andre Beck started a series of articles that aim to illustrate the implementation of simple cloud apps on serverless platforms. First up is a language translation app that uses Step Functions.

How to handle duplicate Lambda invocations
Important message from Murali Allada about idempotent operations when dealing with asynchronous Lambda invocations. He suggests using DynamoDB, which makes a lot of sense, but you should also determine whether mitigating the side effects is worth the extra effort.

AWS CI/CD Pipeline Tools: CodeBuild and CodePipeline
Ran Ribenzaft walks you through setting up a CI/CD pipeline for a serverless project using CodeBuild and CodePipeline.

Creating an AWS SAM CLI project with Typescript and Both Types of Layers (Dependencies and Function)
Thinking about building a serverless Typescript project using AWS SAM? Nick Bordeau has you covered, complete with his project boilerplate.

Serverless Security ?

Twilio: Someone broke into our unsecured AWS S3 silo, added ‘non-malicious’ code to our JavaScript SDK
Note to self: Don’t enable S3 access policies that leave SDKs readable and writable by anyone. Otherwise, automated cyber-crime campaigns that prey on JavaScript code might infect said SDKs and take you over 8 hours to realize it.

Serverless Reads ?

Liberty IT Adopts Serverless Best Practices Using AWS Cloud Development Kit
Andrew Robinson wrote up a great piece (with the help of Matthew Coulter) on how Liberty IT is using the AWS CDK along with a number of best practices to create secure, repeatable deployments for their services. Nice mention of the CDK Patterns site as well.

Serverless for Enterprise: Scale big or go home
Using serverless for greenfield applications should be a no brainer at this point, but for enterprises with existing workloads, the choice isn’t always as clear. Mariliis Retter offers a few things enterprises should be keeping in mind as they start to make the transition.

What comes after Kubernetes?
I once knew an Ops person who said his primary role was to automate himself out of a job. And that’s what cloud providers are doing with K8s IMO. There are a lot more interesting things to be built on the infrastructure than there is to be done to the infrastructure. It’s just a matter of time until Kubernetes will be given as much thought as the hypervisor or Firecracker.

The Edge Computing Opportunity: It’s Not What You Think
Matthew Prince goes into detail about the future of Edge Computing and how Cloudflare Workers are addressing things like speed, cost, and most importantly, compliance.

Serverless or Kubernetes: When do you use which?
Pradeep Kumar lays out a number of rational arguments as to why you’d choose one over the other. I still think the choice isn’t just between FaaS and containers, though. There are so many other serverless “services” available, that trying to implement any of these on K8s would seem like a colossal waste of time and resources.

Videos, Podcasts, and more… ?

Free 50-Minute Learning Path: How to Use Objects in Amazon S3 to Trigger Automated Workflows Using AWS Lambda
From the same author that brought you the episodic Innovator Island virtual workshop, this 6-episode learning series shows you how to use the S3 and Lambda pattern through several example applications. You’ll deploy applications into your own AWS account, explore extending them to your own use-cases, and ultimately be ready to develop sophisticated distributed applications built around S3 events using custom code to integrate with other AWS services. Sponsored

Episode #59: Going Green with Serverless with Paul Johnston (Part 1)
In part 1 of this two-part episode, I chat with Paul Johnston about how serverless compares to traditional computing in terms of being “green”, the impact of data centers on climate, why efficiency is only a first step, what people in tech can do to affect change, and so much more. Watch on YouTube.

ServerlessDays Virtual – July 2020
The second edition of ServerlessDays Virtual happened live last night! The replay is available now, so make sure you check out the amazing talks by Adrian Cockroft, Linda Nichols, Rusthem Feyzkhanov, Rich Buggy, Will Velida and Dr. Denis Bauer.

The Talking Serverless Podcast #19
I had a great time being a guest on The Talking Serverless Podcast with Ryan Jones. He’s had a lot of amazing guests, so you should check out the archives and give the show a listen.

Real World Serverless #21: From K8 to Serverless at Wealth Wizards
Yan Cui talks with Ionut Craciunescu from Wealth Wizards and discusses why their team decided to move away from Kubernetes and adopt a more serverless approach.

Simplest Serverless App with AWS CDK (API Gateway, AWS Lambda and DynamoDB)
Marcia Villalba give us an introduction to the AWS CDK and shows us how to deploy a simple serverless stack that creates an API route, a Lambda function, and a DynamoDB Table.

New from AWS ?

Amazon CloudFront adds additional geolocation headers for more granular geotargeting
This is an awesome update. You can now access a visitor’s city, state, and zip code, plus their timezone and approximate latitude and longitude. Lots of cool personalization that can be done with this

Amazon Elastic File System increases per-client throughput by 100%
Starting to play around with EFS and Lambda? Well, Amazon EFS now supports up to 500 MB/s of per-client throughput, a 100% increase from the previous limit of 250 MB/s.

AWS CodeBuild now supports accessing Build Environments with AWS Session Manager
Great feature if you need to dig in to diagnose complex build problems. Since you should be using CI/CD for all your production workloads anyway, this could become a handy tool.

Amazon Elasticsearch Services announces support for Elasticsearch versions 7.7
Nope, still not serverless, but new features are always welcome.

Amazon CloudFront announces Cache Key and Origin Request Policies
Another very cool update to CloudFront. You now have more control over your cache keys, and can even forward varying information to your origin, but still used the same cached content if necessary. Very cool.

Announcing AWS Serverless Application Model (SAM) CLI now generally available for production use
We already mentioned this at the beginning, but here is the official “what’s new” announcement. Good stuff from the SAM team.

Amazon Lightsail now offers CDN distributions to accelerate content delivery
This is an interesting way to take your “not-so-serverless” application and add a bit of serverless to it. Adding a CDN to cache assets and pages automatically is at least a good first step.

Serverless Jobs ?‍? Sponsored

Serverless Engineer – stedi.com
At Stedi, we’re working in one of the biggest markets on the planet – EDI, the technological backbone of the physical product economy. We’re building a next-generation platform: a ubiquitous commercial trading network to automate the trillions of dollars in B2B transactions exchanged by nearly every company on Earth. If you’re interested in what we’re building and how we’re building it, we’d love to hear from you.

Have a job listing you’d like to share? Please contact me for more information. 

Upcoming Serverless Events ?

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

July 29, 2020 – Orchestrating Distributed Business Workflows with AWS Step Functions

July 30, 2020 – Building Intelligent Application Flows in the Cloud and On-premises with TriggerMesh (Webinar)

September 3, 2020 – ServerlessDays Warsaw

October 1-2, 2020 – ServerlessDays Hamburg 2019

Serverless Star of the Week ⭐️

Each week, I fill this spot with people who are helping to grow the serverless community with their work. With people in every corner of the globe making great contributions, I unfortunately and inevitably cannot know them all. That’s why I am asking for your help in ensuring we amplify the accomplishments of our friends and colleagues through my new Serverless Star nomination form. These are the people on your team, in your circle, who are grinding out new projects, ideas, and perspectives that can make a difference in the serverless community. In honor of our 100th issue, let’s take this step to elevate their voices, and recognize their contributions together. You can start by nominating someone here.

Final Thoughts ?

Wow. 100 issues! I’ve learned so much over the last two years from reading all the amazing content that this community has created, and I hope you have too. Thank you all for continuing to be a part of this.

Like I mentioned in the beginning, the new site should be available soon, and I hope it will give the community more ways to contribute, learn, and amplify the voices of our members.

I hope you enjoyed this newsletter. We’re always looking for ideas and feedback to make it better and more inclusive, so please feel free to reach out to me via Twitter, LinkedIn, Facebook, or email.

Cheers,
Jeremy

P.S. If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless.

Share this newsletter with your friends and coworkers!

Off-by-none is a weekly email newsletter that focuses on the technical details of building applications and products in the cloud using serverless technology. Together we can become better developers and product people by making fewer mistakes as we learn from each other. Off-by-none is the idea that we can become exceptional at what we do, if we are willing to put in the work. Join me on this journey as we help the community work to develop best practices, share our ideas, and learn to build better cloud-based software.

I respect your privacy and I will NEVER sell, rent or share your email address.

Click on the components or numbered steps below to explore how this architecture works.

The Storage First pattern is useful when your application doesn’t require a lot of data transformation on incoming API requests. Rather than attaching API Gateway to a Lambda function that has to parse, process, transform, and save data, we can bypass the Lambda function by using a “service integration” that will send the data directly to an AWS service, like SQS. This reduces the latency of our API calls, saves money by removing the need to run a processing Lambda function, and makes our application more reliable because we are not introducing additional code.

In our example above, we’re using an SQS queue and then processing data off of that using a Lambda function subscription. There are plenty of other services that can be written to directly including DynamoDB, Kinesis, and EventBridge. To the best of my knowledge, Eric Johnson from AWS coined the term “Storage First” to indicate that we want to ensure that we save a user’s raw data before we attempt to run any processing on it. That way, if downstream services or processing fails, we always have a copy of the original request. He explains the process in his post Building a serverless URL shortener app without AWS Lambda.

The incoming data can be transformed and verified using VTL templates, but the more complexity you introduce, the more likely you are to create issues with edge cases. This is an incredibly useful pattern for high velocity workloads like webhooks and clickstream data because it provides low latency and high reliability. Additional processing can be done asynchronously, allowing you to add resiliency to your application if downstream systems are unavailable.

Deploy this Pattern

Below are the basic configurations for deploying this pattern using different frameworks and platforms. Additional configuration for your environment will be necessary. The source files and additional examples are available in the GitHub repo.

• SAM
• Stackery
• Serverless Framework
  TO IMPLEMENT

Want to Contribute?

All patterns and sample code are free to use and available under the MIT License. If you'd like to contribute to these patterns, please submit PRs to the GitHub repo.

» Explore more Serverless Reference Architectures and Patterns

Click on the components or numbered steps below to explore how this architecture works.

The Circuit Breaker pattern keeps track of the number of failed (or slow) API calls by using a cache to share the status across multiple Lambda functions. In this example, we’re using a DynamoDB table so that we can avoid using a VPC. If you were in a VPC already, ElastiCache would be a good alternative.

Here’s how it works. When the number of failures reaches a certain threshold, we “open” the circuit and send errors back to the calling client immediately without even trying to call the API. After a short period of time, we “half open” the circuit, sending just a few requests through to see if the API is finally responding correctly. All other requests receive an error. If the sample requests are successful, we “close” the circuit and start letting all traffic through. However, if some or all of those requests fail, the circuit stays “open”, and the process repeats with some algorithm for increasing the timeout between “half open” retry attempts.

This is an incredibly powerful (and cost saving) pattern for any type of synchronous request to an API or downstream system. You are accumulating charges whenever a Lambda function is running and waiting for another task to complete. Allowing your systems to self-identify issues like this, provide incremental backoff, and then self-heal when the service comes back online, adds a tremendous amount of resiliency to your applications.

Deploy this Pattern

Below are the basic configurations for deploying this pattern using different frameworks and platforms. Additional configuration for your environment will be necessary. The source files and additional examples are available in the GitHub repo.

• SAM
• Stackery
• Serverless Framework
  service: simple-web-service provider: name: aws runtime: nodejs12.x stage: ${opt:stage,'dev'} region: us-east-1 stackName: ${self:service}-${self:provider.stage} stackTags: SERVICE: ${self:service} httpApi: payload: '2.0' cors: true functions: GetUser: handler: index.handler memorySize: 1024 timeout: 6 tracing: Active environment: TABLE_NAME: !Ref Users TABLE_ARN: !GetAtt Users.Arn iamRoleStatementsName: ${self:service}-${self:provider.stage}-getuser-role iamRoleStatements: - Effect: Allow Action: - dynamodb:getItem - dynamodb:putItem - dynamodb:updateItem - dynamodb:deleteItem Resource: - !Join [ '/', [ !GetAtt Users.Arn, '*' ] ] - !GetAtt Users.Arn - Effect: Allow Action: - xray:PutTraceSegments - xray:PutTelemetryRecords Resource: "*" events: - httpApi: path: /user/{id} method: get plugins: - serverless-iam-roles-per-function resources: Resources: Users: Type: AWS::DynamoDB::Table Properties: AttributeDefinitions: - AttributeName: id AttributeType: S BillingMode: PAY_PER_REQUEST KeySchema: - AttributeName: id KeyType: HASH StreamSpecification: StreamViewType: NEW_AND_OLD_IMAGES

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

  service: simple-web-service provider: name: aws runtime: nodejs12.x stage: ${opt:stage,'dev'} region: us-east-1 stackName: ${self:service}-${self:provider.stage} stackTags: SERVICE: ${self:service} httpApi: payload: '2.0' cors: true functions: GetUser: handler: index.handler memorySize: 1024 timeout: 6 tracing: Active environment: TABLE_NAME: !Ref Users TABLE_ARN: !GetAtt Users.Arn iamRoleStatementsName: ${self:service}-${self:provider.stage}-getuser-role iamRoleStatements: - Effect: Allow Action: - dynamodb:getItem - dynamodb:putItem - dynamodb:updateItem - dynamodb:deleteItem Resource: -!Join[ '/', [ !GetAtt Users.Arn, '*' ]] -!GetAttUsers.Arn - Effect: Allow Action: - xray:PutTraceSegments - xray:PutTelemetryRecords Resource: "*" events: - httpApi: path: /user/{id} method: get plugins: -serverless-iam-roles-per-function resources: Resources: Users: Type: AWS::DynamoDB::Table Properties: AttributeDefinitions: - AttributeName: id AttributeType: S BillingMode: PAY_PER_REQUEST KeySchema: - AttributeName: id KeyType: HASH StreamSpecification: StreamViewType: NEW_AND_OLD_IMAGES

Want to Contribute?

All patterns and sample code are free to use and available under the MIT License. If you'd like to contribute to these patterns, please submit PRs to the GitHub repo.

» Explore more Serverless Reference Architectures and Patterns

Welcome to Issue #99 of Off-by-none. Thanks for joining us!

Last week, we got more CDK patterns to explore, another reason to look at BigQuery, and some updated serverless best practices. This week, we get a Q2 serverless recap, someone realizes that NAT Gateways are expensive, and we learn how to model our apps for simpler evolutions. Plus, we’ve got some amazing posts from the serverless community.

Serverless News & Announcements ?

TriggerMesh helps you do more with serverless
Connect GitLab with Confluent and Zendesk, or connect these and dozens of other apps with AWS services like Kinesis, SQS, SNS, and Cognito. Even integrate with EventBridge. Join the Beta program today! Sponsored

ICYMI: Serverless Q2 2020
The Serverless Team at AWS has wrapped up all of the Q2 serverless announcements, blog posts, videos and more into a nice little package for you. Lots of stuff happened, so set aside a few hours if you click this link!

Epsagon Introduces Fastly Technology Integration
Pay attention to Fastly. Oh, and by the way, Epsagon now supports tracing their requests from the edge.

MIT Spinoff Data Management Company TileDB Closes $15 Million
“With a totally serverless infrastructure, TileDB delivers access control and enables distributed computing at extreme scale, eliminating all cluster management and minimizing TCO.” Hmm, a universal data engine that can access, analyze, and share complex data sets with any tool at planet scale? Very, very interesting. ?

Begin supports renaming default GitHub branches
I’ve seen a lot of people rightfully question some of the naming conventions we use in tech, but Begin actually did something about it. Now you no longer need to use the master branch, but can rename it to something else and still have their system pick it up.

Serverless Stories ?

A 1 year review of Laravel Vapor
This is an excellent post by Jack Ellis that not only recounts his team’s use of Laravel Vapor, but offers some great insights into the benefits (and constraints) of serverless. For those of you toiling with servers to run your Laravel applications, this glowing endorsement of Vapor might be just what you’re looking for.

Lessons in Serverless Tech. 1 – NAT GW is expensive.
Yup. I might even go as far as saying it’s too expensive. If you are trying to optimize your serverless application costs, think long and hard about whether you need both VPC and Internet access from the same Lambda function. Audrius Kucinskas clearly agrees with me.

El Chapo: an open source & serverless URL shortener written in python/flask
Here is an interesting post by Sankalp Jonna that highlights one of the core tenets of serverless development: build vs. buy. In order to save $30/mth on a prebuilt solution, his team spent a day building their own, and now it costs only $5/mth to run it. I probably would have built my own solution as well, but I have to think the $25/mth cost savings wouldn’t outweigh the cost of development. ?‍♂️

Serverless Use Cases ?

What scenarios are appropriate for serverless?
“Serverless computing is best-used for tasks that don’t block the user interface.” ?‍♂️ Please, please, please stop saying things like this. Either the author has never run a serverless application at scale, or didn’t take the time to learn how to optimize it. According to the results of the Serverless Community Survey, 75% of respondents are using serverless for REST APIs. So, yes, this is a perfectly appropriate scenario for it!

How I Built a REST API Using Google Sheets
I think there are much easier ways to accomplish this without needing to use Cloud Run, but the use case still stands. Google Sheets works great for a number of things, and being able to pull or push data to it from an API can make a lot sense in the right scenario.

5 Popular Use Cases for Going Serverless
Mariliis Retter offers up five use cases for serverless. We’ve covered these in depth before, but it never hurts to get a good reminder now and then.

Serverless Concepts ?

Scale and Resilience Aren’t Just Buzzwords
So this isn’t specifically about serverless, and I have a feeling this series is going to eventually go down the Kubernetes route, but Stephanie Wong lays out some very good principles here, and more importantly, adds helpful context to the terms “scale” and “resilience”. I know I’ve seen “but will it scale?” become a party joke, but for modern application developers, it needs to be the cornerstone of your architecture.

Using the AWS Serverless Application Model (SAM)
Great post by Andrew May that compares and contrasts SAM and the Serverless Framework. He points out a lot of the limitations faced by SAM, but I often find myself working around limitations in the Serverless Framework as well. If you’re trying to choose between the two, this post will give you some good insights.

TypeScript and Netlify Functions
The more you know. Alex Chiu reminds us that Netlify just uses AWS Lambda functions, and can therefore utilize the @types/aws-lambda type definitions for event object autocompletion.

6 things you didn’t know you could do with the Twilio Serverless Toolkit
I really love this model that allows you to develop serverless functions on the SaaS platforms you’re using. In this post, Phil Nash highlights a few features of the Twilio Serverless Toolkit.

Serverless Tutorials ?‍♀️

Free 50-Minute Learning Path: How to Use Objects in Amazon S3 to Trigger Automated Workflows Using AWS Lambda
From the same author that brought you the episodic Innovator Island virtual workshop, this 6-episode learning series shows you how to use the S3 and Lambda pattern through several example applications. You’ll deploy applications into your own AWS account, explore extending them to your own use-cases, and ultimately be ready to develop sophisticated distributed applications built around S3 events using custom code to integrate with other AWS services. Sponsored

Crash Course on AWS CDK and Serverless with REST API and Data Lake Analytical Querying
If you’re looking to get started with the AWS CDK, this might be a good place to start. Simple, straightforward tutorial by Renato Byrro that walks you step by step through the basics and then right into a working example.

Vue Serverless Side Rendering with AWS Lambda@Edge
David Porter has some code and explanations that will let you perform SSR with Vue apps on Lambda@Edge. Lots of good reasons to do this, which David points out in the post.

AppSync: how to inject table names into DynamoDB batch & transact operations
Yan Cui has a quick fix for those of you using dynamic table names for DynamoDB and are required to pass it into your AppSync resolvers. Really interesting approach, but he says this could be better encapsulated as part of the serverless-appsync-plugin. So if you have some free time, maybe add a PR. ?

How to structure GraphQL and Typescript app using Serverless infrastructure
Here’s a tutorial from Mirco Guidetti that uses the apollo-server-lambda npm package to turn Lambda into a GraphQL server. Should you use AppSync instead? Maybe, but I’ve seen people having success with this approach.

Serverless end-to-end tracing, troubleshooting & performance monitoring with Lumigo
Mohammed Izzy provides a good overview of Lumigo and how their tracing features and console allow you to quickly debug your serverless applications. There are a lot of good tools out there for adding observability to your serverless applications, and at this point, you probably can’t go wrong choosing any of them.

Serverless Security ?

Protecting Your Serverless Solution
This article by Sat Gainda gives a pretty good overview of all the things to think about when it comes to securing your serverless application, along with some other things to think about (like cost). The section on serverless development was a bit confusing (modern approaches use IaC, not containers), but otherwise, some really great info in here.

Serverless Reads ?

Monitor & Debug Serverless with 30-Second Auto-Instrumentation
Complete observability over your serverless environment with no code changes required. With one-click distributed tracing, Lumigo lets you effortlessly find & fix issues in serverless and microservices environments. Get serverless-specific smart alerts before they impact performance or cost. Sign up for a free account and get up & running in minutes. Sponsored

Modeling business logic flows in serverless applications
I really enjoyed this piece by James Beswick that walks you through the evolution of a serverless application as business requirements change. My evolutions have never been quite as smooth and straightforward as the example James gives, but the point is well taken. Designing serverless applications in small, single-purpose, discrete actions, certainly does make iteration much easier.

Building Well-Architected Serverless Applications with Thundra (3-part series)
This three-part series is vendor specific to Thundra, but Emrah Samdam does an excellent job taking you through the Well-Architected Serverless Lens and addressing the questions that every developer should be asking when building a serverless app.

AWS Serverless: you might not need third party monitoring
This is a really interesting post by Rehan van der Merwe that highlights all of the powerful monitoring, logging, and tracing features provided by AWS for your serverless applications. Here’s the problem, the complexity of setting all of these components up (and getting everything configured correctly) is still a massive exercise in patience. For smaller projects, I will typically rely on the built-in AWS services, but for critical applications, third-party services give you what you need out-of-the-box, with minimal setup.

Grow, not build, your serverless team!
If Sheen Brisals can’t inspire you to go serverless, then no one can! In his latest post, he outlines a number of stages for growing a serverless team from with your organization, culminating in an agile and efficient culture with highly-accelerated development processes.

The exotic mixture of Enterprise and Serverless
This is an entertaining read by Pang Bian that juxtaposes the multiple layers of corporate bureaucracy with the novelty of serverless in the enterprise he works for. I’m still not sure how they seem to make it work, but good for them for trying.

For the serverless (non)commuter… ?

Episode #58: Observing Serverless Observability with Erica Windisch
In this episode, I chat with Erica Windisch about the challenges with monitoring and troubleshooting serverless applications, why observability is so important with serverless, what advancements have been made over the last year, and so much more. Watch on YouTube.

New from AWS ?

Announcing automatic backups for Amazon Elastic File System
It would be amazing if AWS would do more things like this. Under what circumstances would you not want your file system backed up? There might be a few, but things like this make more sense as an opt out.

New classroom course: AWS Cloud Financial Management for Builders
Interesting course from AWS that says it’ll teach how to design “cost-optimized” architectures. The syllabus seems quite solid, and just might give you some Corey Quinn level skills (without all the snark of course ?).

Amazon CloudFront announces new TLS1.2 security policy for viewer connections
Improvements to the SSL/TLS protocols are always good to see.

Announcing CDK Pipelines Preview, continuous delivery for AWS CDK applications
Two things you need to know about this. One, CDK Pipelines are self-mutating, meaning they automatically update themselves when you add new applications or stages, and two, they seem pretty amazing.

Introducing Amazon Interactive Video Service (Amazon IVS)
I have no idea how good (or pricey) this is, but under the right circumstances, hosting your own live broadcasts does sound amazing. There are plenty of streaming platforms out there, but to be able to “own your audience” is an interesting value prop.

New Amazon Elastic File System console simplifies file system creation and management
Again, I like these moves by AWS to simplify the user experience. Not known for their console UX, seeing these strides gives me hope for the future.

Amazon SQS Now Supports New Console Experience
Not to keep harping on this, but these upgrades are very welcome. Now, granted, most of this stuff should be managed with IaC, but being able to poke around and easily look at configurations is a handy feature.

Introducing the Cloud Development Kit for Terraform (Preview)
cdktf lets you define application infrastructure with familiar programming languages, while leveraging the hundreds of providers and thousands of module definitions provided by Terraform and the Terraform community. Hey, if this is your thing, go for it.

Serverless Jobs ?‍? Sponsored

Serverless Engineer – stedi.com
At Stedi, we’re working in one of the biggest markets on the planet – EDI, the technological backbone of the physical product economy. We’re building a next-generation platform: a ubiquitous commercial trading network to automate the trillions of dollars in B2B transactions exchanged by nearly every company on Earth. If you’re interested in what we’re building and how we’re building it, we’d love to hear from you.

Have a job listing you’d like to share? Please contact me for more information. 

Upcoming Serverless Events ?

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

Featured Event
July 28th, 2020 – CloudZero presents How We Built a Serverless Machine Learning Pipeline for Under $7/Day with Adam Tankanow (Webinar)

July 22, 2020 – Serverless 201: Advanced Development and Monitoring with AWS, Stackery, and Lumigo

July 22, 2020 – Predicting and Managing Costs in Serverless Applications with AWS

July 27/28, 2020 – ServerlessDays Virtual

July 28, 2020 – Hosting Without Hoisting: NPM Flattening for Zero Cold Start Serverless (Online Meetup)

July 29, 2020 – Orchestrating Distributed Business Workflows with AWS Step Functions

July 30, 2020 – Building Intelligent Application Flows in the Cloud and On-premises with TriggerMesh (Webinar)

September 3, 2020 – ServerlessDays Warsaw

October 1-2, 2020 – ServerlessDays Hamburg 2019

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Alex Wood (@alexwwood). Alex is currently a Senior Software Engineer at AWS developing open source serverless tools. Before that, Alex was the lead developer on the AWS Ruby SDK and is the author of the AWS Lambda Ruby runtime. He’s spoken about writing Lambda functions as idiomatic Ruby code and running them on AWS for the Ruby on Rails podcast, and also had a re:Invent 2019 session, speaking about AWS SAM and the AWS SAM CLI. Thank you, Alex, for your incredible work – and sharing it with others! ?

Final Thoughts ?

Another busy week for serverless, and it’s super exciting to see some additional serverless events starting to pop up.

And speaking of super exciting, next week the Off-by-none newsletter celebrates 100 issues! It seems a bit crazy to think about how many amazing serverless stories, use cases, tutorials, and more have been shared over the last two years. I’m planning something kinda big for next week, so I hope you’ll share this with your friends and coworkers so they don’t miss it!

I hope you enjoyed this newsletter. We’re always looking for ideas and feedback to make it better and more inclusive, so please feel free to reach out to me via Twitter, LinkedIn, Facebook, or email.

See you next week,
Jeremy

P.S. If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless.

Share this newsletter with your friends and coworkers!

Off-by-none is a weekly email newsletter that focuses on the technical details of building applications and products in the cloud using serverless technology. Together we can become better developers and product people by making fewer mistakes as we learn from each other. Off-by-none is the idea that we can become exceptional at what we do, if we are willing to put in the work. Join me on this journey as we help the community work to develop best practices, share our ideas, and learn to build better cloud-based software.

I respect your privacy and I will NEVER sell, rent or share your email address.

Welcome to Issue #98 of Off-by-none. I’m glad you could join us!

Last week, we announced a new way to explore serverless architectures, learned how to debug and load test serverless applications, and we finally got “connection pools” from Lambda. This week, we get more CDK patterns to explore, yet another reason to look at BigQuery, and some updated serverless best practices. Plus, we have plenty of great content from the serverless community.

Serverless News & Announcements ?

Webinar: Would you like to connect applications like GitLab, Confluent, Zendesk, AWS services together?
You can, using TriggerMesh Bridges which allows you to build intelligent application flows using any service, running in the cloud or on-premises. Join us July 30 to see how easy TriggerMesh makes it connect application flows! Sponsored

Announcing the FaunaDB Data Manager
The FaunaDB Data Manager (FDM) can assist with a variety of import and export tasks, including copying documents, collections, indexes, functions, and roles from one FaunaDB database, at any particular point in time, to another FaunaDB database, plus a variety of useful data features. It’s still in preview mode, but looks like a useful tool for FaunaDB customers.

Google Cloud announces BigQuery Omni multicloud analytics solution
So this sounds way too good to be true, but if I can actually query my AWS data using BigQuery without worrying about data transfer costs or duplicating data, that’s pretty amazing. This could be a game changer.

Serverless Spotlight – Sign up to be a guest
I’m launching a new web series called Serverless Spotlight that will “Shine a light on the people who make serverless awesome!” The episodes are 5-10 minutes, and will feature people like you sharing what they are working on in serverless. Commercial, open source, blog post, whatever, we want to hear from you. Be sure to sign up to be a guest.

AWS Serverless Customer Success
Not convinced that serverless is ready for primetime? These customer success stories should help change your mind. AWS recently launched a “Serverless Customer Success” page that highlights a series of case studies that show the positive impact of serverless on companies ranging from startups to enterprises.

Serverless Stories ?

How I Built a Serverless AWS Lambda Twitter Bot in NodeJS in a Single Day
Spencer Pollock walks through his journey building a Twitter Bot using AWS Lambda. It’s fun following someone’s thought process like this. I would use Parameter Store for my Twitter credentials instead of storing them in a file that’ll likely get checked into git, but for a quick and dirty first go, it’ll do.

The Cloud Resume Challenge, or How I Learned to Stop Worrying and Love the Cloud
Kudos again to Forrest Brazeal for starting the Cloud Resume Challenge. There are several people writing about their experiences, but I found this one from Antonio Lo Fiego to be incredibly enlightening. He doesn’t seem to agree that “… the latest version [of the] SAM CLI makes serverless development easier for developers.” His well-placed JT GIF perfectly encapsulates the way many people new to serverless feel.

How I built a zero cost serverless scraper
Anshaj Khare explains how he built a Twitter scraper using cloud native GCP tools. This post makes me wonder about the differences in the developer experience between GCP and other clouds.

Serverless Use Cases ?

Convert HTML to Markdown with a serverless function
Michael Vigor outlines a simple use case that I certainly wouldn’t want to set up and maintain a server for. I love little one-off services like these that are easy to build with FaaS, and enable developers to build things quickly without all the red tape.

A rock-paper-scissors app with gesture detection and voice
Who would have thought of serverless Rochambeau? The cloud advocates at Microsoft, that’s who. This uses a bunch of Azure native services (not 100% serverless), but still a pretty interesting application of the technologies.

How We Automatically Create And Delete Environments For Developers
Gavin Cornwell has written before about automation using Step Functions, but in this post he shows how his team creates an environment when feature branches are created, and then deletes the environment when the branch is removed.

Serverless Concepts ?

CDK Patterns at 20! Let’s Walk Through all 20 Serverless Patterns for AWS
Matt Coulter has done an amazing job on the CDK Patterns site and repository and has consolidated all 20 patterns into a comprehensive blog post. There are so many amazing ideas to explore with this, so make sure you check it out.

AWS Serverless Application Lens — A Summary
Amulya Rattan Bhatia attempts to summarize the Well-Architected Serverless Application Lens whitepaper. It’s a much shorter read and does a good job covering the most important points. If you don’t have time to read the whitepaper, this should at least get you headed in the right direction.

Are Lambda-to-Lambda calls really so bad?
This recent post by Yan Cui started a bit of a debate on Twitter regarding his recommendation to put an API Gateway in between inter-service, synchronous calls that ultimately map to a Lambda function. I’m personally a big fan of using synchronous Lambda-to-Lambda calls using the SDK (when appropriate) to reduce overhead and complexity, but Yan makes a lot of really good points why not to do this. Serverless best practices are constantly evolving, so you’d be wise to follow this debate.

Lambda Execution Leaks: A Practical Guide
Dori Aviram points out the problem when unresolved promises leak into your Node-based Lambda functions. This is probably more common than you think, and can be a massive pain to debug. This post outlines what can cause this issue and some ways to mitigate it.

Serverless Tutorials ?‍♀️

Monitor & Debug Serverless with 30-Second Auto-Instrumentation
Complete observability over your serverless environment with no code changes required. With one-click distributed tracing, Lumigo lets you effortlessly find & fix issues in serverless and microservices environments. Get serverless-specific smart alerts before they impact performance or cost. Sign up for a free account and get up & running in minutes. Sponsored

The Serverless LAMP stack part 3: Replacing the web server
Benjamin Smith and Matthieu Napoli (the creator of Bref) show you how to build serverless PHP applications without needing a web server. Matthieu also explains how the implementation of FastCGI Process Manager inside of Lambda helps makes this possible.

Integrating Amazon EventBridge and Amazon ECS
Jakub Narloch shows you how the new EventBridge to API Gateway target enables new use cases. Even if ECS isn’t your thing, you can use this to route events to webhook consumers, map inter-service communication contracts, and even handle complex integration tests.

Get More Out Of Lambda Authorizers- Use Response Context
Adrin Mukherjee outlines a scenario that uses custom authorizers to pass additional context into your Lambda functions that can be used for calls to downstream services. It’s a powerful pattern, and with the integration into Secrets Manager, you can apply a lot of security best practices, including automatic credential rotation.

Creating low-latency, high-volume APIs with Provisioned Concurrency
Even though James Beswick craps on my Lambda Warmer project (? totally kidding, I don’t actually use it anymore), he does a great job explaining how Provisioned Concurrency can ensure predictable start-up times to latency-sensitive Lambda functions. And if your workload really needs this, a 75% reduction in your slowest execution times is a compelling reason to spend the extra money.

Explore Serverless On Alibaba Cloud With SLS Framework
Thinking about going serverless on the Alibaba cloud? The Serverless Framework has got you covered, and the Serverless Guru team has provided us with this straightforward tutorial to get you started.

Multipart uploads with S3 pre-signed URLs
Great tutorial by Shahar Yakov that walks you through the detailed steps of initiating a multipart upload to S3, generating pre-signed URLs for all the parts, and then stitching them all back together to complete the process.

Serverless Reads ?

Serverless: a backend thing that gives superpowers to frontend developers
Another fun post by Slobodan Stojanović that explains how serverless can give frontend developers the power to make their applications more interactive and highly scalable using modern techniques, rather than relying on traditional servers.

Why we didn’t choose QLDB for a healthcare app
Yan Cui has a great article that simply points out the pros and cons of Amazon’s Quantum Ledger Database, and explains why his client decided it wasn’t quite ready for their application.

How to pick the right Compute Savings Plan for Serverless Workloads on AWS
If you’re interested in saving some money on your AWS serverless workloads (and who doesn’t like saving money?), they check out this post by Michael Bahr that outlines how Compute Savings Plans work, and when they might be right for you.

How to design Serverless Apps like a Pro using *Interactive* Serverless Reference Architectures ??
Farrah Campbell wrote up a piece on the Serverless Reference Architecture project (which she and Stackery were instrumental in getting off the ground). We’re working on more patterns and content, so stay tuned for more serverless!

For the commuter… ?

Episode #57: Building Serverless Applications using Webiny with Sven Al Hamad
In this episode, I chat with Sven Al Hamad about how Webiny makes building serverless applications easier, why everyone from small startups to large enterprises should be choosing serverless, whether or not Webiny could be a WordPress killer, and much more. Watch on YouTube

Good Morning Serverless with Tom McLaughlin
I had a blast chatting with Tom McLaughlin on his Good Morning Serverless Show last week. It airs every Thursday morning at 8:30 am ET, and he’s always looking for guests, so be sure to sign up.

DynamoDB with Alex DeBrie
Alex DeBrie seems to be the hardest working man in DynamoDB right now ?, making the rounds on the podcast circuit. He has a great interview with Jeff Meyerson on the Software Engineering Daily podcast that you definitely should check out.

New from AWS ?

Announcing the New Version of the Well-Architected Framework
Not specific to serverless, but it’s great to see the Well-Architected Framework evolving based on customer feedback and experience. More on it here.

Amazon Comprehend launches real time Custom Entity Recognition
This is a great update that allows you to identify terms that are specific to your domain in real time. If you’ve ever worked with NLP, you’ll know how important this is.

Docker and AWS collaborate to help deploy applications to Amazon ECS on AWS Fargate
Great news for those still using containers for parts of your workload. You can now use Docker Desktop and Docker Compose to deploy containers on Amazon Elastic Container Service using the AWS Fargate launch type. No need for third-party tools anymore.

Amplify CLI adds support for Lambda layers to easily share code & assets across Lambda functions
Amplify CLI now makes it easy to use Lambda layers to share code & assets across various Lambda functions. Plus, you can add existing Lambda layers to your Lambda functions by referencing existing Lambda layer ARNs.

Amazon Athena adds support for managing data catalogs using AWS CloudFormation
No more using the console or APIs to register your data sources. You can now create, update and delete your data sources using the AWS CloudFormation AWS::Athena::DataCatalog resource.

Amazon DocumentDB (with MongoDB compatibility) adds support for cross-region snapshot copy
This is a nice feature for all you MongoDBers out there. You can now copy a snapshot to another region for the purpose of disaster recovery, security, or to populate a development/test environment.

Amazon Keyspaces now enables you to back up your table data continuously by using point-in-time-recovery (PITR)
And for all your Cassandra(ers), PITR provides you with continuous backups of your Amazon Keyspaces table data to help you protect against accidental writes or deletes. When enabled, it’ll back up your table data automatically with per-second granularity and then you can restore your table data to any second in time in the preceding 35 days. Crazy.

Serverless Jobs ?‍? Sponsored

Serverless Engineer – stedi.com
At Stedi, we’re working in one of the biggest markets on the planet – EDI, the technological backbone of the physical product economy. We’re building a next-generation platform: a ubiquitous commercial trading network to automate the trillions of dollars in B2B transactions exchanged by nearly every company on Earth. If you’re interested in what we’re building and how we’re building it, we’d love to hear from you.

Have a job listing you’d like to share? Please contact me for more information. 

Upcoming Serverless Events ?

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

July 22, 2020 – Serverless 201: Advanced Development and Monitoring with AWS, Stackery, and Lumigo

July 27/28, 2020 – ServerlessDays Virtual

July 30, 2020 – Building Intelligent Application Flows in the Cloud and On-premises with TriggerMesh (Webinar)

September 3, 2020 – ServerlessDays Warsaw

October 1-2, 2020 – ServerlessDays Hamburg 2019

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Gal Zabib (@GalZabib). Gal is the co-founder and CEO of Altostra, an intuitive serverless development platform that helps developers build modern cloud applications. Altostra allows for quicker serverless adoption by automating development workflows, which is incredibly helpful for teams looking to hit the ground running. Gal’s background in software engineering and leading R&D certainly has given her great insight into the needs of teams looking to adopt serverless. Thank you, Gal, for making it easier for other teams to adopt serverless quickly and confidently! ?

Final Thoughts ?

Remember last week when I said it was getting harder and harder to narrow down all the amazing serverless posts? Well, it was even harder this week. If I didn’t include your post, please feel free to shoot me a DM and I’ll be happy to retweet it. The amount of serverless content and ideas being generated is incredible, which is a great sign for serverless adoption. Keep the posts coming, and I’ll keep working to find additional ways to amplify them.

I hope you enjoyed this newsletter. We’re always looking for ideas and feedback to make it better and more inclusive, so please feel free to reach out to me via Twitter, LinkedIn, Facebook, or email.

Until next week,
Jeremy

P.S. If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless.

Share this newsletter with your friends and coworkers!

Off-by-none is a weekly email newsletter that focuses on the technical details of building applications and products in the cloud using serverless technology. Together we can become better developers and product people by making fewer mistakes as we learn from each other. Off-by-none is the idea that we can become exceptional at what we do, if we are willing to put in the work. Join me on this journey as we help the community work to develop best practices, share our ideas, and learn to build better cloud-based software.

I respect your privacy and I will NEVER sell, rent or share your email address.

Your feedback is greatly appreciated, so please submit feature requests and issues. If you want to contribute, even better! Pull requests are always welcome.

Full Release Notes: https://github.com/jeremydaly/data-api-client/releases/tag/v1.1.0

NPM: https://www.npmjs.com/package/data-api-client
GitHub: https://github.com/jeremydaly/data-api-client

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

Learn more about Data API Client or check out the other projects I'm working on.

Welcome to Issue #97 of Off-by-none. Thanks for joining us this week!

Last week, we learned how serverless applications can be truly vendor neutral and how EFS is finally makes serverless ML a reality. This week, we announce a new way to explore serverless architectures, learn how to debug and load test serverless applications, and we finally get “connection pools” from Lambda. Plus, we have lots of awesome content from the serverless community.

Serverless News & Announcements ?

Webinar: Building Intelligent Application Flows in the Cloud and On-premises with TriggerMesh
Join us July 30 as we build some example Bridges. One will perform sentiment analysis on ZenDesk tickets using AWS, another will execute an Oracle Cloud function in response to a change in an on-premises OracleDB. See you there! Sponsored

Announcing the Serverless Reference Architectures Project
Shameless plug here for the new Serverless Reference Architectures Project that I just launched. I have been sharing and discussing serverless microservice patterns for years, and finding a better, more interactive way to share them has been a goal of mine for quite some time. This is just the first step, but I hope people find it useful and will contribute their own patterns and examples to help others.

Fauna Raises $27 Million to Continue Providing Cutting Edge Solutions for Developers
I love seeing companies in the serverless space raising money. It’s inevitable, serverless will (someday) take over the world. Will Fauna be the global datastore of choice when that happens?

Vendia raises $5.1M for its multi-cloud serverless platform
No surprises here. Tim Wagner, and the former head of blockchain at AWS, Shruthi Rao, are the ultimate serverless dream team. This is another company/space to pay attention to.

Dfinity demo’s its TikTok clone, opens up its ‘Internet Computer’ to outside developers
Speaking of blockchain, what do you get when you cross serverless with blockchain, and then mix in some WebAssembly? Apparently an “Internet Computer” that can run distributed applications without servers, databases, or firewalls.

Production-Ready Serverless is back (with improved curriculum)!
Yan Cui’s course was groundbreaking when it first launched, now it has the benefit of feedback and iteration on its side. Not only that, but he’s offering it in a much more interactive way. This is probably the best serverless training resource out there.

Gartner Names Lumigo a Cool Vendor in Performance Analysis for Cloud-Native Architectures
When you’re working hard behind the scenes to try and build something amazing, getting positive feedback like this is important. And for the record, Lumigo has been in my “Cool Vendor” column for quite some time. ?

Epsagon Achieves Privacy Shield Framework Certification
Certifications like this are a sign of maturity, so congrats to Epsagon for doing the work to not only bring you an awesome product, but also to bring it up to the highest standards of security and privacy.

Serverless Stories ?

Serverless Web Apps in Python
A good collection of tips and tricks based on what Sanjay Siddhanti learned while using Zappa to build serverless applications.

Serverless-Side Rendering With Lambda at Edge
Great use case and example explained by Joe Duran that uses Lambda@Edge to render static pages for their application. Interesting note in there about solving the problem of competitors scraping their public API. SPAs can be great for many reasons, but making data easily accessible via an API, might not be the best strategic choice.

Serverless Use Cases ?

Monitor & Debug Serverless with 30-Second Auto-Instrumentation
Complete observability over your serverless environment with no code changes required. With one-click distributed tracing, Lumigo lets you effortlessly find & fix issues in serverless and microservices environments. Get serverless-specific smart alerts before they impact performance or cost. Sign up for a free account and get up & running in minutes. Sponsored

Serverless BERT with HuggingFace and AWS Lambda
Awesome post by Philipp Schmid that shows you how to build a serverless question-answering API with BERT and HuggingFace. These ML use cases for serverless are great.

Create a COVID-19 dashboard with Angular 9+Serverless+AWS Lambda
Angular would not be my first choice, but to each their own. This post shows how to create a server-side rendered backend that loads a datasource and generates graphs using AWS Lambda.

Send Emails Serverlessly With Node.js, Lambda, and AWS SES
Another serverless use case that is becoming a standard go-to solution. Angad Singh shows you how to set up and test this in a sandbox environment. This post just scratches the surface of what’s possible.

Serverless Concepts ?

How to Debug AWS Lambda Performance Issues
Latency can have a huge impact on the business metrics of your applications, so making sure you are achieving an acceptable level of performance is hugely important. Yan Cui has another excellent post that dives deep into how to debug these issues and optimize your use cases.

Protect your RDS MySQL DB from AWS Lambda Scalability
Matt Coulter has added a new pattern to his CDK Patterns site that implements the new RDS Proxy service. The code is available in Python and Typescript. Good stuff.

Deletion Policy To Keep Serverless Stack Safe
Simple, yet powerful reminder by Rajesh Kankran to set deletion policies for production AWS services so that you don’t accidentally lose data. It’s okay to admit that it happened to you.

Challenges of Going Serverless (2020 edition)
This is a good overview by Mariliis Retter that outlines some of the common challenges of developing serverless applications. I totally agree that serverless has many challenges and causes for confusion due to the unknowns. But it’s still totally worth it.

Disaster Recovery strategies using AWS Serverless Services
This is a good read by Harshavardhan Ghorpade that explains the importance of Disaster Recovery and how serverless helps us achieve that at a much lower cost that traditional serverfull environments.

Serverless Tutorials ?‍♀️

Build serverless apps that seamlessly connect to MySQL & PostgreSQL databases
Amazon RDS Proxy is a fully managed, highly available database proxy that allows applications to pool and share database connections, improving database efficiency, application scalability, and security. Previously launched in Preview at AWS re:Invent in 2019, RDS Proxy is now generally available and can be enabled for MySQL and PostgreSQL workloads through both the Amazon RDS and AWS Lambda consoles. Sponsored

Load testing a web application’s serverless backend
It’s all fine and good to say that serverless can easily scale, but do you know if your architecture actually can? This is why load testing is so important to confirm your system’s scalability before you realize it during a Black Friday sale. James Beswick has another excellent post that walks you through setting up the tools and running the experiments to make sure you serverless system will handle anything you throw at it.

It’s simple with AWS: A step by step serverless fan-out architecture guide
Rafael Rodriguez has a great tutorial that shows you how to implement the fan-out pattern using AWS serverless services. He’s using an SNS topic in this example, but EventBridge could also be another effective way to implement it.

Handling webhooks with EventBridge, SAM and SAR
Slobodan Stojanović explains the evolution of webhook implementations at Vacation Tracker and then shows you how to create a SAR app that publishes incoming message to an EventBridge event bus for further processing.

Deploying Serverless Golang APIs With The AWS CDK
Go is screaming fast on Lambda, so it’s becoming a popular choice for developers building serverless applications. Dennis O’Keeffe has created a tutorial that shows you how to deploy a Golang API using the AWS CDK.

Chromium and Selenium in AWS Lambda
Headless browser running in an AWS Lambda function? No problem. Vittorio Nardone shows you how to get Chromium up and running and then how to use Selenium to run some tests.

Serverless Reads ?

The GoLD Stack
We don’t see enough articles that look at serverless from a management perspective, and as we see more developers adopting it, I think it’s wise to be able to effectively communicate the value “up-the-stack”. Zef Hemel put some thoughts out there on this as it relates to using Go with Lambda and DynamoDB.

The truth about Serverless
How can you not click on a title like this? But seriously, Gareth Thomas outlines the perceived pros and cons of serverless and then offers a reality check for each one.

You’re not ready for feature flags
Another batch of good advice from Forrest Brazeal about making sure you nail down the basics of cloud deployments before you go ahead and try getting all fancy-like.

Why do so few vendors in the serverless space offer metered billing?
Interesting observation by Paul Swail about why most third-party serverless products are using a quota-based billing model instead of a more “serverless” pay-per-use model. He partially answers his own question, but definitely something to think about if you’re building a business in this space.

My Time as a Black Woman Software Engineer at Capital One
Though it isn’t about serverless, I had to include this incredibly brave post that recounts the horrific experience suffered by a black woman working as a software engineer. As I read this, my heart sank to the floor. The ideas and contributions of underrepresented people are far too often marginalized or hijacked by the current culture in tech. If you witness anything like this (even the things that may seem small to you), please speak out and stand up for changing toxic cultures.

Podcast, videos, and more… ?

Episode #56: Accelerating DynamoDB Workflows using Dynobase with Rafal Wilinski
In this episode, I chat with Rafal Wilinski about the challenges developers face when using DynamoDB, why DynamoDB makes sense for applications big and small, and why we need more tools like Dynobase to make working with your data easier. Watch on YouTube.

How Vendia’s founders aim to leverage serverless and blockchain to solve data sprawl
As I said before, Tim Wagner and Shruthi Rao are a serverless force to be reckoned with. This is a good article (and video interview) that goes much deeper into their vision and what Vendia is capable of.

New from AWS ?

Amazon RDS Proxy is Generally Available
Amazon RDS Proxy, a fully managed, highly available database proxy for Amazon Relational Database Service (RDS), is now generally available with MySQL and PostgreSQL compatibility. I think this is a good solution for those that need a better way to manage their connections from Lambda. Read more here

Amazon QuickSight now supports Lake Formation–protected Athena data sources
QuickSight users can now visualize their Lake Formation–protected Athena data, natively using the benefits that come with Lake Formation, including table and column-level access controls.

AWS AppSync introduces new 12xlarge instance for server-side API caching
AWS AppSync is a managed GraphQL service that simplifies application development by letting you create a flexible API to securely access, manipulate, and combine data from one or more data sources.

Amazon Elastic File System increases file system minimum throughput
Amazon Elastic File System (Amazon EFS) file systems using the default bursting throughput mode now have a minimum throughput of 1 MiB/s.

Amazon DocumentDB (with MongoDB compatibility) now supports T3 medium instances
Amazon DocumentDB (with MongoDB compatibility) is a fast, scalable, highly available, and fully managed document database service that supports MongoDB workloads. Amazon DocumentDB makes it easy and intuitive to store, query, and index JSON data. Amazon DocumentDB now supports t3.

Serverless Tools ?

DRY (Don’t Repeat Yourself) on the cloud with Pulumi
This is a nice write up about using Pulumi to build repeatable cloud deployments. If you’re not familiar with Pulumi, it’s like a CDK for all major cloud vendors.

serverless-ghost
Jeshan Babooa ported the Ghost project to Lambda, which should give you a way to run the blogging platform serverlessly. It is very early on and there are some expected issues, but it’s a good start.

Koyeb Serverless Data Processing Platform Early Access
I came across this article about a new tool for easily building serverless workflows. Looks really interesting.

Thoughts from Twitter ?

So, I was going to write up a whole blog post about DynamoDB performance in Lambda with Deno. But ultimately the whole things can be represented by this picture. ~ Matthew Bonig
Click to see the picture, but it doesn’t take a thousand words to describe it. Deno is not ready for primetime with Lambda and DynamoDB.

Serverless Jobs ?‍? Sponsored

Serverless Engineer – stedi.com
At Stedi, we’re working in one of the biggest markets on the planet – EDI, the technological backbone of the physical product economy. We’re building a next-generation platform: a ubiquitous commercial trading network to automate the trillions of dollars in B2B transactions exchanged by nearly every company on Earth. If you’re interested in what we’re building and how we’re building it, we’d love to hear from you.

Have a job listing you’d like to share? Please contact me for more information. 

Upcoming Serverless Events ?

There are a lot of upcoming serverless events, webinars, livestreams, and more. If you have an event you’d like me to mention, please email me.

July 9, 2020 – Debugging AWS Lambda Performance Issues

July 9, 2020 – Good Morning Serverless with Tom McLaughlin (I’ll be his guest)

July 30, 2020 – Building Intelligent Application Flows in the Cloud and On-premises with TriggerMesh (Webinar)

October 1-2, 2020 – ServerlessDays Hamburg 2019

Serverless Star of the Week ⭐️

There is a very long list of people that are doing #ServerlessGood and contributing to the Serverless community. These people deserve recognition for their efforts. So each week, I will mention someone whose recent contribution really stood out to me. I love meeting new people, so if you know someone who deserves recognition, please let me know.

This week’s star is Josh Carlisle (@joshcarlisle). Josh is a Senior Sales Engineer – Partner Channel for AppDynamics, and a Microsoft MVP (Azure). In his role at AppDynamics, Josh provides technical and sales enablement to strategic partners implementing AppDynamics on the Azure platform. Based in Raleigh, he’s active in the local serverless community, co-organizing and participating in meet-ups, user groups, and code camps. Josh has also spoken at conferences around the country, including ServerlessDays events in Austin, Nashville and Amsterdam. Thank you, Josh, for sharing your knowledge and passion! ?

Final Thoughts ?

I mentioned in tweet that it’s getting harder and harder to narrow down all the amazing serverless posts that are being published every week. As you’ve probably noticed, this newsletter gets quite long, and I’m only including a fraction of the posts that I collect and review. I have a lot of ideas about how to make this better, so stay tuned for some upgrades to the Off-by-none experience coming soon.

In the meantime, be sure to check out the Serverless Reference Architectures Project and contribute your thoughts, ideas and patterns. The more examples, the better, so people can learn and experiment with the tools and languages they are comfortable with. #ServerlessForEveryone

I hope you enjoyed this newsletter. We’re always looking for ideas and feedback to make it better and more inclusive, so please feel free to reach out to me via Twitter, LinkedIn, Facebook, or email.

Take care,
Jeremy

P.S. If you like this newsletter, and think others would too, please do me the honor of sharing it with friends and coworkers who are interested in serverless.

Share this newsletter with your friends and coworkers!

Off-by-none is a weekly email newsletter that focuses on the technical details of building applications and products in the cloud using serverless technology. Together we can become better developers and product people by making fewer mistakes as we learn from each other. Off-by-none is the idea that we can become exceptional at what we do, if we are willing to put in the work. Join me on this journey as we help the community work to develop best practices, share our ideas, and learn to build better cloud-based software.

I respect your privacy and I will NEVER sell, rent or share your email address.

Serverless gives us the power to focus on delivering value to our customers without worrying about the maintenance and operations of the underlying compute resources. Cloud providers (like AWS), also give us a huge number of managed services that we can stitch together to create incredibly powerful, and massively scalable serverless microservices.

Almost 2 years ago now, I wrote a post on Serverless Microservice Patterns for AWS that became a popular reference for newbies and serverless veterans alike. The capabilities of serverless have changed dramatically since then, opening up a ton of new patterns and possibilities. Today I’m announcing the Serverless Reference Architectures Project. This project is intended to capture, share, explore, and debate the patterns and practices being used in serverless production applications today.

To really understand what’s going on, it’s useful to take a birds-eye view of a pattern, understand how data flows back and forth through the different components, and think about the implications on your application. I’ve created interactive architectures that let you explore the details of each component and walk you through the different steps.

You also need to understand how a pattern would come alive with AWS services and how you can configure it for your implementation. I’ve partnered with Stackery and integrated their interactive canvas so that you can explore these patterns, customize them, and generate updated SAM templates automatically.

Finally, you want to understand how you’d replicate the pattern in production using infrastructure-as-code to ensure repeatable, safe deployments. You’re free to use the generated SAM templates, or take advantage of community supplied templates for the Serverless Framework, CDK, and others.

This project hopes to encapsulate all of this in one place so you can start by learning about the pattern fundamentals, then go all the way through to deploying a functional version of the pattern into your own AWS account (all with just a few clicks).

By partnering on sharing common serverless reference architectures and patterns, we can help others unlock the true power of serverless with AWS. We are starting with just a few patterns, but I plan on adding several more and hope that you will contribute as well.

Give it a try, and let me know what you think!

Did you like this post? ?  Do you want more? ?  Follow me on Twitter or check out some of the projects I’m working on.

Click on the components or numbered steps below to explore how this architecture works.

This is the most basic of patterns you’re likely to see with serverless applications. The Simple Web Service fronts a Lambda function with an API Gateway. I’ve shown DynamoDB as the database here because it scales nicely with the high concurrency capabilities of Lambda.

Deploy this Pattern

Below are the basic configurations for deploying this pattern using different frameworks and platforms. Additional configuration for your environment will be necessary. The source files and additional examples are available in the GitHub repo.

• SAM
• Stackery
• Serverless Framework
  service: simple-web-service provider: name: aws runtime: nodejs12.x stage: ${opt:stage,'dev'} region: us-east-1 stackName: ${self:service}-${self:provider.stage} stackTags: SERVICE: ${self:service} httpApi: payload: '2.0' cors: true functions: GetUser: handler: index.handler memorySize: 1024 timeout: 6 tracing: Active environment: TABLE_NAME: !Ref Users TABLE_ARN: !GetAtt Users.Arn iamRoleStatementsName: ${self:service}-${self:provider.stage}-getuser-role iamRoleStatements: - Effect: Allow Action: - dynamodb:getItem - dynamodb:putItem - dynamodb:updateItem - dynamodb:deleteItem Resource: - !Join [ '/', [ !GetAtt Users.Arn, '*' ] ] - !GetAtt Users.Arn - Effect: Allow Action: - xray:PutTraceSegments - xray:PutTelemetryRecords Resource: "*" events: - httpApi: path: /user/{id} method: get plugins: - serverless-iam-roles-per-function resources: Resources: Users: Type: AWS::DynamoDB::Table Properties: AttributeDefinitions: - AttributeName: id AttributeType: S BillingMode: PAY_PER_REQUEST KeySchema: - AttributeName: id KeyType: HASH StreamSpecification: StreamViewType: NEW_AND_OLD_IMAGES

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

  service: simple-web-service provider: name: aws runtime: nodejs12.x stage: ${opt:stage,'dev'} region: us-east-1 stackName: ${self:service}-${self:provider.stage} stackTags: SERVICE: ${self:service} httpApi: payload: '2.0' cors: true functions: GetUser: handler: index.handler memorySize: 1024 timeout: 6 tracing: Active environment: TABLE_NAME: !Ref Users TABLE_ARN: !GetAtt Users.Arn iamRoleStatementsName: ${self:service}-${self:provider.stage}-getuser-role iamRoleStatements: - Effect: Allow Action: - dynamodb:getItem - dynamodb:putItem - dynamodb:updateItem - dynamodb:deleteItem Resource: -!Join[ '/', [ !GetAtt Users.Arn, '*' ]] -!GetAttUsers.Arn - Effect: Allow Action: - xray:PutTraceSegments - xray:PutTelemetryRecords Resource: "*" events: - httpApi: path: /user/{id} method: get plugins: -serverless-iam-roles-per-function resources: Resources: Users: Type: AWS::DynamoDB::Table Properties: AttributeDefinitions: - AttributeName: id AttributeType: S BillingMode: PAY_PER_REQUEST KeySchema: - AttributeName: id KeyType: HASH StreamSpecification: StreamViewType: NEW_AND_OLD_IMAGES

• CDK
  Are you a CDK Guru?
  Would you like to contribute patterns to the community?
  Check out the Github repo!

Want to Contribute?

All patterns and sample code are free to use and available under the MIT License. If you'd like to contribute to these patterns, please submit PRs to the GitHub repo.

» Explore more Serverless Reference Architectures and Patterns

Click on the components or numbered steps below to explore how this architecture works.

If you’re building a webhook, the traffic can often be unpredictable. This is fine for Lambda, but if you’re using a “less-scalable” backend like RDS, you might just run into some bottlenecks. There are ways to manage this, but because Lambda supports SQS triggers, we can throttle our workloads by queuing the requests and then using a throttled (low concurrency) Lambda function to work through our queue. Under most circumstances, your throughput should be near real-time. If there is some heavy load for a period of time, you might experience some small delays as the throttled Lambda chews through the messages.

You’ll also want to handle failed messages using a Dead Letter Queues (DLQ). The SQS Poller will adjust its polling frequency based on your Lambda function’s concurrency. You’ll need to configure your redrive policies to appropriately handle failed messages.

Deploy this Pattern

Below are the basic configurations for deploying this pattern using different frameworks and platforms. Additional configuration for your environment will be necessary. The source files and additional examples are available in the GitHub repo.

• SAM
• Stackery
• Serverless Framework
  # Code sample coming soon

  1	# Code sample coming soon

• CDK
  Are you a CDK Guru?
  Would you like to contribute patterns to the community?
  Check out the Github repo!