“If you think cryptography will solve your problem, then you don’t understand cryptography and you don’t understand your problem.” — Roger Needham

A few colleagues and I recently revisited a familiar debate. In most data systems, identifier generation is treated as undifferentiated infrastructure—one library, one default, applied everywhere. The implicit assumption is that a sufficiently random identifier is a sufficient answer to whatever question the system happens to ask of it; that the differences between identifier classes—surrogate keys, content fingerprints, anonymized handles, span identifiers, request correlations—are differences of usage rather than differences of construction. That assumption is convenient. It survives precisely until the system encounters constraints that aren’t engineering constraints.

For the past several years, I’ve designed and currently operate a clinical research data pipeline serving multiple concurrent Institutional Review Board (opens in a new window) (IRB)-approved studies against Electronic Health Record (opens in a new window) (EHR)-sourced data at production scale. Though, implementation specifics are abstracted from this post, I want to highlight the architectural patterns. In human-subjects research, identifier construction stops being a performance choice and becomes a regulatory and ethical primitive.

Why? Because the constraints are different from those that arise in commercial systems. Re-identification resistance is enforceable under the Health Insurance Portability and Accountability Act (opens in a new window) (HIPAA) and the Common Rule (opens in a new window) . Per-study isolation is an IRB requirement. Provenance MUST (opens in a new window) survive transformation but MUST NOT enable cross-study linkage. Workflows that depend on the operator’s inability to reverse a transformation can’t be retrofitted onto a single global identifier scheme; they have to be built into the construction itself. None of these properties emerge from a generic randomness primitive applied uniformly to every identifier the pipeline produces.

So let me state the pattern plainly. A real human-research pipeline needs at least three distinct identifier constructions. Each answers a different question. Each lives at a different boundary. Treating identifier generation as a single problem with a single answer breaks one architectural pillar in service of another. The failure mode isn’t that any single construction is wrong, but that no single construction is right for every boundary the pipeline must respect.

Three Pillars, Three Constructions

A human-subjects research pipeline must answer three questions of every row that flows through it. The questions are independent and every row is interrogated by the following pillars:

1. Fidelity. Is this row the same as before, or has it changed? A pipeline that can’t answer this can’t deduplicate, can’t detect drift, and can’t reason about whether a transformation preserved meaning. Fidelity is the substrate on which idempotency, change detection, and lineage are built. Without it, every other property the pipeline claims to provide is conjectural.

2. Governance. Can this row be re-linked to a person, or to the same person across studies? In a research context, the answer must be no, and the answer must be enforceable rather than promised. Policy is necessary but not sufficient; if the construction permits re-linkage, then policy is the only thing preventing it, and policy can be circumvented, misconfigured, or socially engineered. Governance asks whether the construction itself denies the operations that policy forbids.

3. Provenance. Which pipeline run produced this row, from which source, on which day, by which transformation? Provenance is the operational substrate of accountability. It allows any finding to be traced back to its derivation, any anomaly to be localized to a stage and a run, any audit to reconstruct what happened without depending on the cooperation of the system that produced the result.

Each question is answered by a different cryptographic construction at a different boundary:

• Content-addressed SHA-256 (opens in a new window) hashing answers fidelity at the ingest boundary.
• Per-tenant keyed HMAC (opens in a new window) hashing answers governance at the publication boundary.
• CSPRNG (opens in a new window) -backed identifier generation answers provenance at the operational layer.

The constructions are not interchangeable. Using any one of them outside its intended boundary either fails the question its boundary is supposed to answer or undermines a pillar that some other boundary was responsible for. Let me walk each in turn, identify the constraint it answers, and explain why the substitution of a generic identifier scheme is an architectural mistake rather than a minor inefficiency.

Fidelity at the Ingest Boundary

At the ingest boundary, the pipeline receives raw data from a source system that doesn’t coordinate with the pipeline about identity. Files arrive on a schedule, are written to a lake in raw form, and become the substrate from which everything downstream is derived. The question fidelity asks of every row at this boundary is whether it’s the same row that arrived yesterday, last week, or in any prior incremental load. If the pipeline can’t answer that question cheaply and unambiguously, then:

• Deduplication becomes guesswork.
• Idempotency becomes a hope.
• The lake’s role as the recovery anchor for the entire pipeline is compromised.

The construction that answers this question is a SHA-256 hash computed over the row’s sorted business columns at the moment of lake ingestion. Simply put, “business columns” are all columns except for metadata columns that were added by the pipeline itself. The hash is written into a metadata column that is then carried unchanged through every downstream stage. It MUST NOT be recomputed at the warehouse boundary, must not be recomputed after foreign-key re-keying at segmentation, and must not be recomputed under any condition the pipeline operator may otherwise consider reasonable. The construction is content-addressed and immutable.

The non-randomness of this identifier is the point. Two rows with identical business-column content produce identical hashes. Two rows with any meaningful difference produce different hashes with overwhelming probability. The hash is unique by virtue of its content, not by virtue of its randomness, and that property is what makes change detection tractable across overlapping daily exports without requiring the source system to participate in a coordination protocol it was never designed to support. A randomized identifier would be wrong here in the strongest sense: it would foreclose exactly the property the boundary needs.

The same property that enables change detection also serves as a duplicate pre-filter. Rows that share identical business-column values are duplicates by definition, and the hash exposes that fact at the moment of ingestion rather than deferring it to a downstream reconciliation step. Duplicate detection becomes a comparison of fixed-width hashes rather than a comparison of arbitrarily wide row contents, and the cost of identifying duplicates collapses to the cost of comparing two values. The work the pipeline would otherwise have to do further downstream is done once, at the boundary, by the construction itself.

The hash also serves as the first link in the lineage chain that fidelity, governance, and provenance all eventually depend upon. Any row anywhere in the pipeline can be traced back to its source by following the hash. Any divergence between a stage’s input and its output can be diagnosed by comparing hashes. Any claim that two rows in different marts derive from the same source row is verifiable without consulting either source or mart, because the hash is the same in both places. The hash is the substrate; everything else is built on top of it.

A subtlety worth naming is that the hash construction itself sits on a hot path. At ingest scale, tens of millions of rows pass through the pipeline on every full load, and the hash is computed for every one of them. The disciplines that apply to identifier generation under sustained concurrency apply equally here: allocation behavior MUST be controlled, primitive selection MUST avoid implicit coordination, and the cost of the construction MUST NOT become the dominant cost of ingestion. The arguments I developed in my earlier post on NanoID hot-path optimization generalize directly to hash construction in pipelines of this kind. The construction is different; the discipline is the same.

Governance at the Publication Boundary

The boundary between the warehouse and the data marts is where the pipeline transitions from internal computation to external publication. Inside the warehouse, identity is managed by the pipeline operator, and the SHA-256 hash from the ingest boundary serves every internal purpose well. Outside the warehouse, identity becomes a research artifact, and a fundamentally different requirement asserts itself: the identifier that reaches the mart MUST NOT enable re-identification of the underlying subject, and it MUST NOT (opens in a new window) enable cross-mart linkage of the same subject across separate research projects.

Both requirements are properties of the construction, not properties of the data. A SHA-256 hash carried untouched into a mart would satisfy neither. Why? The hash is deterministic; identical inputs produce identical outputs; an attacker with sufficient knowledge of the source schema could attempt to reconstruct the input space and brute-force the mapping. More immediately, the same warehouse row would carry the same hash into every mart, and any researcher with access to two marts could link rows trivially by comparing hashes. The construction permits exactly the operations that governance forbids.

Stay with me—the answer here is a small but architecturally decisive change. The construction that answers governance at this boundary is HMAC-SHA256, keyed with a per-mart secret stored in a Hardware Security Module (opens in a new window) (HSM)-backed secrets manager and never exposed to the pipeline operator. At the segmentation stage that produces each mart, the warehouse-side SHA-256 hash is replaced by HMAC-SHA256(warehouse-hash, per-mart-secret). The result is a new identifier that is:

• Deterministic with respect to the same inputs.
• Cryptographically opaque without the secret.
• Distinct across marts, because the secret is distinct across marts.

This construction has a property that’s easy to overlook and architecturally decisive: the HMAC operates on the SHA-256 hash, not on the raw business columns. An attacker who somehow obtained a per-mart secret would not face the relatively constrained problem of inverting the HMAC over a known schema of patient identifiers; they would face the problem of inverting it over the full output space of SHA-256, which is 2^256 possible inputs. A dictionary attack against patient identifiers, plausible in principle against a single-layer keyed hash, is eliminated by construction. The two-layer design—content hash first, keyed hash of the hash second—is what makes the cryptographic claim survive contact with a realistic threat model.

The architectural consequence is that the publication boundary becomes a place where identity is not merely transformed but severed. The original SHA-256 hash never reaches the mart. The mart receives the HMAC, and only the HMAC, as its identifier for that row. The pipeline operator, who has access to the warehouse and to the segmentation code, can’t reverse the transformation, because the operator doesn’t have the per-mart secret. The secret lives in an HSM-backed store that the operator can use but can’t extract. The same warehouse row, written into three different marts, carries three different identifiers, and no party—including the pipeline operator—can establish that the three identifiers refer to the same subject without simultaneous access to all three secrets and the cooperation of the secrets manager.

This is what makes the honest-broker workflow cryptographically tenable rather than merely operationally promised. In human-subjects research, there are workflows where the operator’s inability to reverse the transformation is structurally required: the team that operates the pipeline MUST be able to publish the data, but MUST NOT be able to re-identify the subjects whose data it published. Under a single-layer construction, the operator can always be compelled, suborned, or compromised into reversing the mapping. Under the two-layer construction described here, the operator simply doesn’t have the capability to reverse, and therefore can’t be compelled to exercise it. The construction enforces what policy could only request.

The bridge to my earlier work on AES-CTR-DRBG is direct. In that post I argued that deterministic cryptographic constructions belong in environments where reproducibility, auditability, and explicit state evolution are required. The HMAC-at-publication pattern is the same disposition applied to identity rather than to byte streams. The construction is deterministic with respect to its inputs; the state that determines its output evolves only when the per-mart secret is rotated; the auditor can reason about what the construction produces without having to reverse-engineer the library that implements it. Configuration is a contract. Boundaries are explicit. The cryptographic posture doesn’t change mid-flight.

A randomized identifier would be wrong here in a different way than at the ingest boundary, but no less wrong. A randomized identifier is non-deterministic; the same row processed twice would produce different identifiers; the mart’s referential integrity would collapse on the first re-run. What the boundary needs is deterministic-but-cryptographically-opaque, and the only construction that delivers both properties is a keyed hash.

Provenance at the Operational Layer

The constructions described so far handle the headline identity story—the identity of the rows themselves. They’re computed once per row and carried thereafter. They’re also, by a substantial margin, not the identifiers the pipeline generates most often.

A pipeline of any operational seriousness produces identifiers continuously for things that aren’t rows. Pipeline runs need identifiers. Stage executions need identifiers. Batch tasks under a thread pool need identifiers. Observability spans, if the pipeline exports traces, need identifiers. Synthetic sentinel rows injected at segmentation need identifiers distinguishable from real source identifiers. Project allocations need identifiers that survive across systems. Honest-broker requests need identifiers that the requesting system, the secrets manager, and the provisioning workflow can all reference unambiguously. Quality diagnostic runs need identifiers for the audit trail that links them to the pipeline run that triggered them.

These identifiers have a constraint regime distinct from anything the row-level constructions answer:

1. They are allocated at high concurrency, often from many worker threads or pods simultaneously, with no central allocator available.
2. They MUST be collision-resistant by virtue of their construction, because no coordinator exists to detect and resolve a collision after the fact.
3. They MUST be URL-safe and human-tractable, because they will appear in logs, dashboards, audit trails, and the bodies of cross-system requests.
4. They MUST be allocation-cheap, because the pipeline generates them at orders of magnitude higher volume than it generates SHA-256 row hashes or per-mart HMACs.
5. They MUST NOT leak temporal patterns or sequence structure that an observer could exploit to infer pipeline behavior or correlate identifiers from independent runs.

In raw call volume, this class of identifier dominates the SHA-256 and HMAC constructions by orders of magnitude. The hash is computed once per source row at ingest. The HMAC is computed once per row per mart at segmentation. The operational identifiers are generated continuously: on every notebook, on every span, on every parallel task, on every cross-system request. If the construction here is wrong, the cost is paid everywhere the pipeline does any work at all.

This is the design space my NanoID library (opens in a new window) was built for, and the disposition that produced it is the disposition that earlier governed its randomness backends. The construction at the call site is identical regardless of the workload—a single function call returning a compact, URL-safe identifier—but the randomness source behind it is configurable, and the choice of source is made per identifier class rather than per call site. The architectural payoff is that the cryptographic posture of the identifier is determined at construction time and survives every downstream invocation without requiring the call site to be aware of it.

For the highest-volume operational identifiers—span identifiers, batch identifiers, stage execution identifiers, the thousands of small handles that a pipeline produces in the course of doing its work—the appropriate backend is the ChaCha20-based PRNG I developed in an earlier post. It’s designed for predictable behavior under sustained concurrency, with allocation discipline on the hot path and no shared coordination between worker threads. It doesn’t target any particular compliance regime, because none applies at this layer; what applies is the constraint that identifier generation MUST remain invisible in profiling output even at the volumes the pipeline reaches.

For the identifiers that participate in audit, IRB, or compliance trails—project allocations, honest-broker request identifiers, anything that ends up in a Key Vault audit log or a regulatory submission—the appropriate backend is the AES-CTR-DRBG construction I developed in the post before that. It targets FIPS 140-2 (opens in a new window) -aligned cryptographic randomness with deterministic state evolution, allowing the identifiers it produces to be traceable, auditable, and defensible under scrutiny in ways that a general-purpose PRNG is not. The construction is the same NanoID library, configured with a different randomness source via the same WithRandReader option, called identically at the call site.

The two backends are not competing; they answer two different constraints within the same pipeline, on the same library surface, decided once at construction time. The same disposition that governs SHA-256 and HMAC at their boundaries—the construction is chosen for the boundary, not for the codebase—governs NanoID at the operational layer.

One Library, Three Backends

The library design that makes this tractable isn’t a convenience feature. It’s an architectural commitment that allows per-boundary construction choice to coexist with a single call-site surface. NanoID exposes a WithRandReader option that accepts any io.Reader as its randomness source. The choice of source is made once, at generator construction time, and survives every subsequent identifier the generator produces. The call site doesn’t change; the cryptographic posture does.

This mirrors the architectural move SHA-256 and HMAC are making at their respective boundaries. Both belong to the SHA-2 family. Both compose with the same underlying hash primitive. The difference between them is whether a per-tenant key is involved, and whether the construction can therefore enforce the cryptographic break that publication requires. The primitive is shared; the construction is chosen for what the boundary MUST guarantee. Same disposition, same shape, different layer.

The negation is instructive. A pipeline that uses uuid_generate_v4() everywhere—or NanoID with a single hardcoded randomness source applied to every identifier it produces—collapses distinctions that the architecture is doing real work to preserve. Surrogate keys for fact tables receive the same construction as project identifiers feeding IRB submissions, which receive the same construction as observability spans. Each is correct for some workload and wrong for at least one other. The pipeline that adopts a single global construction hasn’t simplified its identifier story; it has merely declined to make decisions that its boundaries will eventually force it to revisit.

The pipeline that adopts the layered model described here makes those decisions once, at the boundary where each one belongs, and then stops thinking about them.

• The hash is content-addressed at ingest.
• The HMAC is per-tenant keyed at publication.
• The NanoID is randomly allocated at the operational layer, with the randomness backend chosen for the identifier class.

Each construction is correct for its boundary. Each construction is composable with the others, because the boundaries are explicit and the constructions don’t interfere. The architecture isn’t more complex than the alternative; it’s more honest about the complexity that was already present.

Wrapping it all up…

The earlier articles in this sequence each followed a single constraint to its conclusion. My 2024 article on NanoID followed the hot-path discipline that emerges when identifier generation moves from incidental utility to infrastructure. My 2025 article on AES-CTR-DRBG followed the requirements of compliance, auditability, and explicit state evolution. My article on the ChaCha20-based PRNG followed the constraints of portability, concurrency, and operational simplicity. In each case, a single set of requirements produced a single construction.

This article inverts that move. Given a domain—human-subjects research—that imposes multiple constraint regimes simultaneously, the architectural response isn’t a single construction but a layered one. Fidelity, governance, and provenance aren’t competing concerns to be balanced against one another. They’re independent questions answered by independent constructions at independent boundaries, and the pipeline that respects the independence of the questions can satisfy all three without trading any of them off against the others.

Make no mistake; the constructions themselves are unremarkable. SHA-256 has been a standard primitive for two decades. HMAC has been a standard primitive for longer. CSPRNG-backed identifier generation is a solved problem with well-understood implementations. The architectural work isn’t in the choice of primitives. It’s in the recognition that each primitive is correct only for the boundary that actually needs the property it provides, and in the discipline of refusing to substitute one for another where the substitution would be easier but wrong.

Good architecture emerges when constraints are REQUIRED (opens in a new window) and followed to their conclusions, even when those conclusions lead to different constructions under different constraints. The earlier articles argued this for randomness. The pipeline architecture described here argues it for identity. The answer is the same in both cases: make the boundaries explicit, choose the construction the boundary needs, and let the disposition do the rest.

My next post will take a step back and explore the data analytics and insights pipeline from an architectural perspective.

Implementation: https://github.com/sixafter/nanoid (opens in a new window)

“A system is not determined by the algorithms it uses, but by the assumptions it makes.”
— Leslie Lamport

Randomness appears deceptively uniform in most software systems. A call is made, bytes are returned, and execution proceeds. The details of how those bytes were produced rarely matter—until they do. As with many foundational components, the assumptions baked into a randomness source tend to surface only when systems scale, diversify, or are subjected to constraints they were not originally designed to accommodate.

In the previous article , I explored deterministic cryptographic randomness under compliance and auditability constraints, where explicit state evolution and reproducibility were not optional. That work followed inevitably from environments shaped by regulation and forensic accountability. This article explores a different design space: one where the constraints are imposed not by certification regimes, but by software portability, concurrency, and operational simplicity.

The result is a ChaCha20-based (opens in a new window) pseudorandom number generator (PRNG) designed for predictable behavior in real systems, rather than alignment with formal validation programs.

What distinguishes this space from compliance-driven designs is not weaker security assumptions, but different priorities. Software systems are often judged not by formal proofs or certifications, but by how they behave under sustained load, across diverse platforms, and in the presence of concurrency. In those environments, simplicity is not aesthetic—it is operational. Every implicit dependency, hidden allocation, or global coordination point becomes a liability over time.

Constraints Shape Constructions

Cryptographic primitives are often discussed as interchangeable building blocks. In practice, the surrounding constraints determine which properties matter and which can be safely ignored. A construction optimized for auditability and deterministic replay will look very different from one optimized for simplicity, portability, and high-throughput concurrent use.

In software-heavy environments—particularly those spanning multiple platforms or deployed as libraries rather than systems—certain requirements surface repeatedly. The randomness source must behave predictably under concurrency. It must avoid surprising allocation behavior. It must scale without coordination bottlenecks. And it must do so without pulling in complex dependencies or imposing policy decisions on its callers.

These are not compliance constraints. They are engineering constraints.

Engineering constraints tend to accumulate quietly. A library that is easy to use in isolation may behave very differently when embedded in larger systems, reused across services, or exercised under high concurrency. Over time, small assumptions compound: about allocation behavior, about shared state, about initialization cost. What begins as a convenient abstraction can become a systemic bottleneck. Designs that survive this environment are rarely the most general ones; they are the ones that make their tradeoffs explicit early.

ChaCha20 fits naturally into this space. As a stream cipher designed for efficient software implementation, it offers strong cryptographic properties without reliance on specialized hardware instructions or platform-specific optimizations. That makes it a practical foundation for a general-purpose PRNG intended to behave consistently across environments.

State, Not Entropy, Does the Work

As with any PRNG, the core of the design is not entropy acquisition but state evolution. Entropy enters the system at initialization. From that point forward, output is a function of internal state advancing according to a defined algorithm.

This distinction matters. By treating entropy acquisition as a boundary rather than a continuous process, the generator remains honest about its dependencies. The operating system provides unpredictability at startup. The PRNG provides expansion, isolation, and repeatable behavior thereafter.

ChaCha20’s design reinforces this model. Once keyed and initialized, the cipher produces a stream of pseudorandom bytes derived entirely from its internal state. There are no conditional branches based on environmental input, no opportunistic refreshes, and no hidden coordination with external sources. State advances monotonically. Output follows deterministically.

Thinking in terms of state rather than entropy also clarifies responsibility. Once initialization completes, the generator no longer depends on the operating system or the environment for correctness. The burden shifts entirely to the implementation and its callers. That boundary is important: it allows failures, assumptions, and limitations to be localized rather than diffused across layers. When something goes wrong, there is no ambiguity about where to look.

This makes the generator easier to reason about, test, and integrate. The absence of implicit behavior is not a limitation; it is a deliberate choice that keeps responsibility visible.

Concurrency Without Contention

Concurrency tends to expose the weaknesses of otherwise elegant designs. Shared mutable state invites contention. Global locks introduce unpredictable latency. Per-call initialization imposes overhead that scales poorly as systems grow.

The ChaCha20 PRNG approaches concurrency as a distribution problem rather than a synchronization problem. Independent generator instances maintain independent state. A pooling strategy amortizes initialization costs while avoiding shared counters or global coordination. Each request draws from a generator whose behavior is local, predictable, and isolated from others.

This approach mirrors patterns commonly used in distributed systems: partition state, avoid global serialization, and accept modest duplication in exchange for clarity and scalability.

In practice, this means resisting the temptation to centralize randomness behind a single shared construct. While such designs appear simpler at first, they introduce hidden coupling that surfaces under load. Contention becomes observable latency. Synchronization becomes an availability concern. By allowing each generator instance to evolve independently, the system preserves the same semantic behavior regardless of how many consumers are active.

Runtime Semantics and Predictable Behavior

Allocation behavior is treated as a semantic concern rather than a micro-optimization. A PRNG that allocates unpredictably during output generation cedes part of its execution model to the runtime.

The effects of these choices are observable in practice. Small reads primarily reflect fixed overhead. Larger reads scale linearly with the amount of data processed, as expected from a stream cipher advancing its internal counter. Under concurrent workloads, latency remains stable because requests are serviced independently rather than serialized behind shared state.

What matters is not the absolute numbers, but their predictability. The behavior matches the design model, rather than fluctuating with runtime conditions.

This predictability has secondary effects that are easy to underestimate. Monitoring becomes more meaningful when latency distributions are stable. Capacity planning becomes tractable when throughput degrades linearly rather than catastrophically. Perhaps most importantly, anomalies become visible. When behavior deviates from expectation, it signals a real change in conditions rather than noise introduced by the runtime.

Failure Modes and Explicit Boundaries

Boundaries are explicit. Initialization depends on a trusted entropy source. Once initialized, the generator’s behavior is deterministic and bounded by construction. Errors are surfaced rather than hidden.

Surfacing errors is not merely a defensive choice; it is an architectural one. Silent degradation may preserve uptime in the short term, but it undermines trust in the system over time. By making failure explicit, the generator avoids the pretense of correctness under conditions it was not designed to handle. This makes integration more demanding, but also more honest.

This approach does not attempt to cover every possible threat model. It does not claim resistance to state compromise beyond what the underlying primitive provides. It does not attempt to retrofit compliance semantics onto a software-oriented design.

Closing Thoughts

This ChaCha20-based PRNG (opens in a new window) is used as the randomness source in other systems, including my NanoID (opens in a new window) implementation, where predictable behavior, low overhead, and concurrency safety are practical requirements rather than abstract goals.

Randomness is not a single problem. Different environments impose different constraints, and those constraints shape the constructions that make sense within them.

This ChaCha20-based PRNG exists because certain systems demand a randomness source that behaves predictably under concurrency, avoids unnecessary complexity, and makes its assumptions explicit. It treats randomness as evolving state rather than as an opaque service call.

The tension between generality and specificity is unavoidable in systems design. Libraries that attempt to serve every use case often end up serving none particularly well. By contrast, designs that accept their constraints openly tend to age more gracefully. They are easier to reason about, easier to integrate, and harder to misuse.

Implementation: https://github.com/sixafter/prng-chacha (opens in a new window)

“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger W. Dijkstra

Randomness occupies an unusual position in software systems. We rely on it for security, for fairness, for simulation, and yet we rarely scrutinize its behavior with the same rigor we apply to storage engines or replication protocols. We accept entropy as something that simply happens, mediated by an operating system call or a library abstraction, and move on.

That approach works—until it doesn’t.

When I began working on systems that required both cryptographic correctness and operational predictability, the implicit assumptions around randomness became a liability. Latency mattered. Allocation behavior mattered. Reproducibility mattered. And in certain environments, auditability mattered most of all. The standard tools did not fail outright, but they failed quietly, by obscuring behavior that ought to be explicit.

At its core, the problem is not randomness itself, but how cryptographic state evolves over time—and whether that evolution remains visible, bounded, and predictable.

This article is about addressing that gap.

Determinism Is Not the Enemy

In distributed systems, we accept a fundamental axiom: there is no way to guarantee complete knowledge of global state at any given moment. We design for tolerance, convergence, and recovery, not for omniscience; ref Parallel and Distributed System Design ). Randomness, however, often escapes this discipline. It is treated as an oracle—invoked, trusted, and forgotten.

Cryptographic systems take a different view. They do not reject determinism; they constrain it. Given a fixed internal state and a fixed construction, output must follow. This is not a weakness but a requirement. It is precisely what allows us to reason about forward security, backtracking resistance, and compromise scenarios.

AES-CTR-DRBG sits squarely in this space. It provides a deterministic construction based on a well-understood primitive, AES, and defines explicit rules for how internal state evolves over time. Once you accept that premise, several consequences follow naturally. Hidden entropy sources become suspect. Implicit reseeding becomes problematic. Mutable configuration becomes dangerous.

The design space narrows considerably.

Deterministic Constructions and Cryptographic Honesty

Cryptography is often discussed in terms of strength, but strength alone is not sufficient. A system can employ strong primitives and still behave in ways that are difficult to reason about. When that happens, security becomes an emergent property rather than an explicit one.

Deterministic constructions reject that ambiguity.

An AES-CTR-DRBG (opens in a new window) does not attempt to surprise the reader. Given a key, a counter, and a defined update function, the output follows directly. There is no hidden mixing, no opportunistic entropy acquisition, and no conditional behavior based on environmental state. This predictability is not merely convenient; it is foundational. It allows one to reason about compromise without first reverse-engineering library behavior.

Once determinism is accepted as a requirement, architectural freedoms disappear. You can no longer “improve” security by quietly pulling in entropy. You can no longer treat reseeding as an implementation detail. You must decide, explicitly, when state changes and why.

This is not a limitation of AES-CTR-DRBG. It is a discipline imposed by cryptography itself.

These constraints do not arise in isolation. They emerge most clearly in regulated, multi-tenant, and distributed systems, where randomness is no longer an incidental utility but a component subject to scrutiny. In such environments, deterministic and auditable cryptographic randomness becomes a requirement rather than a preference. Reproducibility matters for forensic analysis. Isolation matters for per-tenant security. Predictable resource behavior matters when concurrency and throughput are no longer theoretical concerns.

The standard random primitives in Go do not address this space. They make no guarantees around deterministic output, explicit key management, or resource predictability under concurrency, nor do they target alignment with NIST SP 800-90A (opens in a new window) or FIPS 140-2 (opens in a new window) requirements. Third-party alternatives often complicate matters further, introducing additional dependencies or obscuring allocation behavior and latency characteristics behind opaque abstractions.

At scale, these gaps become operational risks. System PRNGs (opens in a new window) do not ensure reproducibility. Key lifecycle management and tenant isolation are left to the caller. Failure modes relevant to compliance and auditability remain implicit, even as entropy sources degrade or cryptographic assumptions are violated. What is missing is not another source of randomness, but a generator whose behavior remains explicit, bounded, and inspectable over time.

These constraints define the space for a deterministic random bit generator based on AES (opens in a new window) in counter mode ( CTR (opens in a new window) ), as specified in NIST SP 800-90A and aligned with FIPS 140-2. The implementation discussed here relies solely on Go’s standard library cryptography and targets allocation-free output and low, predictable latency, making it suitable for high-throughput and regulated environments where correctness and observability are inseparable.

Entropy, State, and the Shape of Time

Many practical failures in cryptographic systems arise not from broken primitives, but from blurred boundaries. Entropy acquisition is a boundary. Deterministic generation is another. When the two are conflated, responsibility becomes unclear.

Operating systems are good at collecting entropy. Applications are good at applying policy. A DRBG sits between the two, which is why many implementations quietly collapse the distinction.

I chose not to.

In this design, entropy enters the system at a clearly defined point and nowhere else. The generator consumes seed material and expands it deterministically. If reseeding occurs, it occurs because the caller has decided that new entropy is available and appropriate. This makes the system honest about its dependencies. It also makes it auditable. One can trace exactly when and how entropy influences output, which is a prerequisite for meaningful analysis.

This mirrors a lesson learned repeatedly in distributed systems: implicit coordination is coordination nonetheless. If you depend on entropy, say so.

It is helpful to visualize the generator not as a black box, but as a state machine evolving over time.

Imagine a simple diagram with three persistent elements: Key, Counter (V), and Limits. At initialization, these are set from seed material and configuration. Each request for output advances the counter deterministically, producing a block of output via AES encryption. The key remains fixed until a reseed event occurs. Limits monotonically decrease as output is generated.

Time, in this model, does not pass in seconds. It passes in blocks generated.

A reseed event resets the key and counter and restores limits. Absent a reseed, state evolution is linear, irreversible, and predictable. There are no side paths, no hidden refreshes, and no conditional jumps based on runtime conditions. The diagram contains no feedback loops other than the explicit reseed boundary.

This mental model is intentionally simple. It allows one to reason about compromise, exhaustion, and recovery without hand-waving.

Reseeding is often presented as a binary feature: enabled or disabled. In practice, it is a temporal question. How long may a generator safely operate on a given internal state? How much output may be derived before assumptions begin to erode?

FIPS-aligned guidance does not answer these questions abstractly; it answers them in terms of limits. Maximum bytes per key. Maximum requests between reseeds. Defined update functions. These limits exist not because AES is fragile, but because systems operate over time, and time introduces exposure.

In a deterministic generator, time is measured not in seconds but in output. Each block generated advances the internal counter. Each advancement is irreversible. By enforcing explicit limits, the generator treats time as a first-class dimension rather than an afterthought.

This perspective aligns closely with how we reason about versioning and state evolution in distributed systems. State does not merely exist; it progresses. If you do not bound that progression, you eventually lose the ability to reason about it.

Prediction resistance is frequently discussed as an absolute good. In reality, it is a trade. Enabling it increases resilience against certain compromise scenarios, but it also introduces new operational dependencies. Reseeding requires entropy. Entropy requires coordination with the environment. Coordination introduces latency and failure modes.

Rather than treating prediction resistance as a default, this design treats it as a conscious decision. When enabled, reseeding behavior is strict and enforced. When disabled, the generator behaves as a pure deterministic expander. Neither mode is inherently superior; each corresponds to a different threat model.

Cryptographic systems fail when they attempt to serve all threat models simultaneously. By forcing the caller to choose, the system avoids pretending that one size fits all.

Runtime Semantics and Predictability

Garbage collection is often discussed in terms of performance, but in latency-sensitive systems it also becomes a semantic concern. Allocation introduces nondeterminism: not in output, but in timing. When a generator allocates during its hot path, it cedes control of latency to the runtime.

The effects of these choices are observable in practice. Under serial workloads, small reads complete in tens of nanoseconds, largely reflecting fixed overhead rather than per-byte cost. Larger reads exhibit higher absolute latency, not because of inefficiency, but because AES-CTR scales linearly with the number of blocks processed. Encrypting a few blocks versus a few hundred necessarily occupies different regions of the CPU pipeline, even when the underlying primitive is highly optimized. What matters is that this behavior is predictable and stable under load, matching expectations derived from the algorithm and the memory hierarchy rather than fluctuating with runtime conditions.

Achieving zero allocations in the output path was, therefore, non-negotiable; read “ MUST (opens in a new window) ”.

What matters here is not raw speed, but control. A system that allocates implicitly hands part of its execution semantics to the runtime. In many contexts that is acceptable; in some, it is not. When randomness participates in protocol boundaries, scheduling decisions, or security-sensitive operations, latency variance becomes observable behavior. Treating allocation as a semantic choice forces the design to acknowledge that reality. Once allocation is removed from the generation path, timing becomes a property of the algorithm itself rather than a side effect of memory management. That shift restores a degree of predictability that is otherwise difficult to recover.

The decision to make this DRBG allocation-free during generation was not an optimization exercise; it was a correctness constraint. Once generation begins, no heap allocations occur. The caller owns the buffers. The internal state resides in fixed-size structures. The runtime remains uninvolved. Latency variance collapses. Throughput becomes predictable. The generator behaves the same under load as it does in isolation.

Concurrency often exposes design shortcuts. Shared mutable state invites contention; global locks invite collapse. In the context of deterministic randomness, concurrency introduces an additional risk: accidental nondeterminism caused by interleaving.

The solution follows a familiar distributed-systems pattern: partition the state. Each shard maintains its own independent DRBG state. Shards do not share counters, keys, or reseed thresholds. They share only configuration. This trades a small amount of memory for clarity and scalability. Concurrency improves throughput, not entropy.

Determinism remains intact.

Configuration is where many security failures quietly begin. Defaults accrete. Options multiply. Behavior changes without callers noticing. Over time, the configuration surface becomes an implicit API, understood only by convention.

Here, configuration is a contract.

All configuration occurs at construction time. Options are explicit. Once initialized, the generator does not change its behavior. There is no hidden reseeding, no dynamic mutation, no silent fallback. If prediction resistance is enabled, it is enabled because the caller said so.

A generator whose security posture can change mid-flight violates basic architectural discipline. A useful way to evaluate a DRBG is to ask a simple question: if an attacker learns the internal state at time t, what can they infer? With a deterministic construction, the answer is precise. Past output remains secure if the construction provides backtracking resistance. Future output becomes predictable unless new entropy is introduced. These are not surprising conclusions; they are the expected consequences of the model.

What matters is that the model makes these consequences explicit. By avoiding hidden entropy and implicit reseeding, the generator ensures that compromise scenarios are analyzable rather than speculative. An auditor does not need to guess whether the library refreshed itself “recently.” The code tells the story plainly.

In cryptography, clarity is often mistaken for weakness. In practice, it is the opposite.

Closing Thoughts

This (opens in a new window) AES-CTR-DRBG implementation exists because certain systems demand more from randomness than opacity and convenience. They demand predictability, auditability, and explicit control. They demand that design choices surface in code rather than hide behind defaults.

If you need a general-purpose source of entropy, the standard library remains an excellent choice.

If you need deterministic cryptographic randomness whose behavior you can reason about—whose latency you can predict and whose state evolution you can audit—then the design presented here follows inevitably from that requirement. Once randomness is treated as state evolving over time, rather than as an oracle invoked on demand, many familiar shortcuts stop making sense. What remains is a system whose behavior is explicit, bounded, and knowable. Good architectures do not happen by accident. They emerge when constraints are taken seriously and followed to their logical conclusions—even when those conclusions lead to different constructions under different constraints.

This design is not theoretical; the AES-CTR-DRBG implementation described here is used as the underlying randomness source in my NanoID (opens in a new window) library, where deterministic behavior and predictable performance are equally important.

Implementation: https://github.com/sixafter/aes-ctr-drbg (opens in a new window)

“All problems in computer science can be solved by another level of indirection.” — David J. Wheeler

Identifier generation is one of those concerns that quietly disappears into the background of a system. At small scale, it is effectively free. A call to a random number generator, a string conversion, and the work is done. There is little reason to revisit it, and even less reason to question its design. This holds largely because identifier generation tends to be invisible: it sits at the edges of requests, happens quickly, and rarely shows up in profiling output. When it does, it is usually dismissed as noise.

That assumption holds only as long as the surrounding system remains small. As concurrency increases and identifier generation moves onto hot paths, what was once incidental becomes part of the system’s observable behavior. Identifiers are created everywhere: at request boundaries, inside storage layers, across services that never coordinate with one another. They are infrastructure in the most literal sense—pervasive, unavoidable, and relied upon precisely because they are assumed to be cheap. When they stop being cheap, the cost is paid everywhere.

NanoID (opens in a new window) fits neatly into this mental model. It produces compact, URL-safe identifiers with reasonable entropy and a simple interface, and in most environments it behaves exactly as expected. It is easy to adopt and difficult to misuse, which reinforces the idea that identifier generation is a solved problem. The difficulty is not with NanoID itself, but with the way its cost profile changes when it is exercised continuously and concurrently.

In Go, the most direct implementation of NanoID draws randomness from crypto/rand.Reader. From a correctness standpoint, this is beyond reproach. The operating system provides high-quality entropy, the guarantees are well understood, and the resulting identifiers are unpredictable in exactly the ways they should be. Under light use, the cost of doing so is effectively invisible, which further entrenches the assumption that there is nothing here worth examining.

Under sustained concurrency, that assumption no longer holds. Each call to crypto/rand crosses into the kernel, sources entropy, fills a buffer, and returns. Temporary buffers are allocated and discarded. None of this is problematic in isolation, but taken together and repeated at scale, it becomes the dominant cost of identifier generation. At that point, “generating an ID” is no longer a small operation. It is a collection of system calls, allocations, and coordination that happens to terminate in a string.

This is not a criticism of crypto/rand. It is doing exactly what it is designed to do. The issue is one of proximity. Entropy acquisition and identifier construction are tightly coupled, even though they serve different purposes. Correctness requires the former. Performance becomes sensitive when it occupies the hottest path of the system.

Concurrency has a way of making these relationships visible. Throughput flattens earlier than expected. Latency becomes uneven. Profilers begin to attribute meaningful time to what was assumed to be trivial. In practice, the profiler does not report time spent “generating IDs”; it reports time spent in the kernel, allocation paths, and garbage collection. Identifier generation becomes visible not because it is conceptually complex, but because it is structurally misplaced.

Separating Entropy from Generation

At that point, there are two broad directions a system can move. One is to accept the cost and design capacity around it. The other is to separate concerns that were previously implicit. The former tends to entrench friction around an otherwise unremarkable operation. The latter introduces an architectural boundary.

NanoID does not require fresh entropy for every identifier. What it requires is unpredictability. Those two properties are related, but they are not equivalent. Entropy must originate from a trusted source, but once acquired, it can be expanded safely using a cryptographically sound construction. There is no requirement to consult the operating system on every invocation, and doing so ties correctness to cost in a way that becomes increasingly visible under load.

This observation reshapes the implementation. Instead of treating identifier generation as a single indivisible operation, it becomes possible to draw a boundary between entropy acquisition and identifier construction. In practical terms, this means seeding a generator once from crypto/rand and then producing random bytes locally. The operating system remains the root of trust, but it no longer sits in the inner loop.

The Shape of the Hot Path

A stream cipher such as ChaCha20 fits this role naturally. It is designed to generate large volumes of pseudorandom output from a fixed key and nonce, and its properties are well understood. Once initialized, it produces random bytes without further system interaction. Used this way, it does not replace the operating system’s entropy source; it amortizes it.

Each generator instance is seeded once using crypto/rand and then used to satisfy multiple NanoID generations. The generator itself is not shared across callers. Sharing would introduce contention and obscure the very boundary the design is trying to establish. Instead, generator instances are treated as independent state machines that can be reused opportunistically.

This is where sync.Pool becomes useful, not as a micro-optimization, but as a way to preserve locality without introducing coordination. Generator instances are pooled so that goroutines can borrow them briefly, generate the required bytes, and return them. There is no shared mutable state and no locking around generation itself. The pool exists solely to reduce repeated setup cost when reuse is inexpensive.

Once entropy is decoupled, other sources of variability become apparent. NanoID generation requires temporary byte buffers—first for random data, then for mapping those bytes into an alphabet. These buffers are uniform in size and short-lived. Allocating and discarding them repeatedly introduces allocation pressure that is unrelated to the semantics of identifier generation. Pooling these buffers follows the same reasoning as pooling generator state: reuse when convenient, allow reclamation when not.

The same discipline applies to configuration. NanoID relies on derived values such as alphabet size, bit masks, and the number of random bytes required to generate an identifier of a given length. These values do not change per call, yet they are often computed as part of the generation path. Computing them once and fixing them at construction time pushes variability outward and keeps the inner loop small and predictable. The generation path becomes a straight-line transformation from random bytes to characters, without branching or recomputation.

At this stage, the hot path is deliberately unremarkable. There are no system calls, no locks, and no dynamic allocation beyond the final string itself. The work performed is exactly the work required to produce an identifier, and nothing else. What remains visible in benchmarks is not overhead, but the irreducible cost of string construction in Go.

The effect of these changes is straightforward to measure. Allocation counts collapse to a single allocation per identifier. Latency stabilizes. Throughput scales with available CPU until saturation. More importantly, behavior becomes predictable. Identifier generation stops appearing as a source of variance and resumes its role as infrastructure.

An implementation of the approach described here is available as an open-source Go module, along with a small command-line tool built on top of it:

• https://github.com/sixafter/nanoid (opens in a new window)
• https://github.com/sixafter/nanoid-cli (opens in a new window)

What does not change is the security model. Entropy still originates from the operating system. Generator state is seeded from a cryptographically secure source. Unpredictability is preserved. There is no attempt to weaken guarantees in exchange for speed. The improvement comes from respecting boundaries, not from relaxing them.

Identifier generation stops being trivial when it becomes infrastructure. At that point, it deserves the same architectural treatment as any other component that occupies a hot path. Once those boundaries are made explicit, the problem largely resolves itself.

At that point, the choice of entropy source becomes an explicit design decision rather than an incidental one. The NanoID implementation described here does not assume a single generator, nor does it require that entropy be sourced directly from the operating system on every invocation. Instead, it is structured to accept a well-defined pseudorandom generator that is seeded from a trusted source and then exercised locally.

For environments where throughput and steady behavior under concurrency are the primary concerns, a ChaCha20-based generator provides a practical balance. Seeded once from crypto/rand, it offers high-quality pseudorandom output with predictable performance characteristics, making it well suited for systems where identifier generation sits on a hot path and must remain invisible.

In environments where regulatory or compliance requirements apply, particularly those that mandate FIPS 140-2 (opens in a new window) validation, an AES-CTR-DRBG construction becomes the appropriate choice. In that context, the same architectural separation applies: entropy is sourced from the operating system, expanded via a standards-aligned deterministic generator, and consumed locally. The difference is not architectural, but contractual—the guarantees are defined externally rather than operationally.

The important point is that these choices do not alter the shape of the system. Whether the generator is ChaCha20-based or an AES-CTR-DRBG, the boundary between entropy acquisition and identifier construction remains intact. The cost model is stable, the hot path is predictable, and the guarantees are explicit rather than implicit.

Implementation: https://github.com/sixafter/nanoid (opens in a new window)

The map is not the territory. – Alfred Korzybsky in Science and Sanity, 1933

This is the second post in a series where I’ll discuss various aspects of parallel and distributed system design. I’d like to provide a brief introduction to taxonomy (or classification of data) and eventual consistency as these building blocks are fundamental to parallel and distributed system design.

Extending my thoughts from a post I made some time ago , in computer science and information modeling, an ontology formally represents knowledge as a set of concepts within a domain, and the relationships among those concepts. It can be used to reason about the entities within that domain and may be used to describe the domain. Why do we need to know this? Because all parallel and distributed systems require data to operate and, in turn, operate on data.

Semantic Ontologies

A semantic ontology, or ontology, is a formal explicit description of concepts in a domain of discourse, properties of each concept describing various features and attributes of the concept, and restrictions on properties. An ontology together with a set of individual instances of classes [or objects] constitutes a knowledge base. In reality, there is a very fine line where the ontology ends and the knowledge base begins but let’s put that aside for a moment.

An implementation of conceptual ontologies is the Semantic Web [ URI (opens in a new window) ]. Simply put, the Semantic Web is the representation of data on the web in which information is given well-defined meaning—“to be a universal medium for the exchange of data”.

The principal technologies of the Semantic Web fit into a set of layered specifications called the Resource Description Framework (opens in a new window) (RDF). The current components of that framework are the RDF Core Model, the RDF Vocabulary Description Language and the Web Ontology Language ( OWL (opens in a new window) ). OWL is a descriptive layer built on top of RDF used to model classes, properties, and objects. These languages all build on the foundation of URIs, XML, and XML namespaces.

Classes are the focus of most semantic ontologies. Outlined in most ontology research are the following observations:

• There is no one correct way to model a domain— there are always viable alternatives. The best solution almost always depends on the application that you have in mind and the extensions that you anticipate.
• Ontology development is necessarily an iterative process.
• Concepts in the ontology should be close to objects (physical or logical) and relationships in your domain of interest. These are most likely to be nouns (objects) or verbs (relationships) in sentences that describe your domain.

Solution domains require, at times, different classifications. Regardless of which classification the solution domain requires, the following rules and considerations have proven effective in designing an efficient information model:

• Effectively understand the information.
• Describe it unambiguously.
• Enforce structure and style guidelines.
• Allow for efficient storage and retrieval of information.
• Keep network communication to a minimum. Don’t over engineer.

I cannot overstate the need to understand your information model but always with an understanding of your delivery time frames. All too many times, we are caught up in the perfect model only in the end to take shortcuts to satisfy deadlines. The solution is acceptance and understanding that change is an integral part of any information model. Temporal and transient taxonomies may be an integral part of your solution domain. Semantic ontologies may be created at runtime to address a special-purpose need where the need itself is transient in nature. For example, in order to fulfill a set of calculations with a high time and computational complexity rating a transient taxonomy might need to be created to provide global access to intermediate calculation results. This is all too common in financial simulation.

Not surprising is that classes are the focus of most ontologies. So with that, let’s move on to a more usual and enjoyable example of a semantic ontology: wine. Wine is a potable liquid produced by at least one maker of type winery, and is made from at least one type of grape. For example, a class of Wine represents all wines. Specific wines are instances of this class. The Bordeaux wine in the glass in front of you while you read this post is an instance of the class of Bordeaux wine. A class can have subclasses that represent concepts that are more specific than the superclass. For example, we can divide the class of all wines into red, white, and rose wines. Alternatively, we can divide a class of all wines into sparkling and non-sparkling wines.

We can further describe properties of classes and instances: Chateau Lafite Rothschild Pauillac wine has a full body; it is produced by the Chateau Lafite Rothschild winery. We can have two properties describing the wine, the property body with the value full and the property maker with the value Chateau Lafite Rothschild winery. At the class level, we can say that instances of the class Wine will have properties describing their flavor, body, sugar level, the maker of the wine and so on.

All instances of the class Wine, and its subclass Pauillac, have a property maker the value of which is an instance of the class Winery. All instances of the class Winery have a property produces that refers to all the wines (instances of the class Wine and its subclasses) that the winery produces. The entire example OWL ontology derived for Wine can be found here (opens in a new window) . I am of course a huge fan.

The strength of your application requires you to understand the data on which it operates—unambiguosly. Though perfection is not key, it is second to paramount. Once you fully understand your information model, then you are able to understand eventual consistency for your application. And with this in mind, let’s discuss eventual consistency.

Eventual Consistency

Eventual consistency is one of the consistency models used in the domain of parallel programming. It means that given a sufficiently long period of time over which no changes are sent, all updates can be expected to propagate eventually through the system and all the replicas will be consistent. I will remind you from a previous blog entry there is no way to guarantee complete knowledge of the current or future state of a distributed system, because knowledge of state changes must be propagated and propagation takes time, during which more state changes may occur. This is an axiom of distributed computing. So how can we begin to reconcile eventual consistency with this axiom? Well, we don’t really [ URI (opens in a new window) ]. We design for loose coupling. We design for tolerance. We design for symmetry of algorithms.

At a high level, most mission- and safety-critical distributed applications SHOULD (opens in a new window) strive to achieve these high-level technical requirements:

• Provide high availability read and write access to information.
• Minimize network communication.
• Provide a highly available and fault tolerant system that can support an annual uptime guarantee of 99.999 percent , which is equal to 5.256 minutes of annual unscheduled downtime.
• Provide instrumentation to aid implementers and customers in their hardware sizing estimates. Scalability MUST (opens in a new window) be measurable.
• Devise a software architecture that scales up by a factor of 10^6. That is, an application’s storage and processing capacity can automatically grow by a factor of a million, doing jobs faster 10^6x speed up or doing 10^6 larger jobs in the same time 10^6 scale up, just by adding more resources.
• Provide for conflict resolution. Common implementations in this area leverage Lamport timestamps (opens in a new window) and vector clocks (opens in a new window) .
• Provide for transient node failure.

While the goal is to devise a software architecture that scales up without limits, there has to be some kind of limit: billions of dollars, or giga watts, or just space. So, the more realistic goal is to be able to scale from one node to a million nodes all working on the same problem or same set of problems.

While fundamentally a Distributed Hash Table (DHT) (opens in a new window) is generally well suited for specific classes of decentralized distributed systems, it is certainly not well suited for all. For example, a DHT is well suited for problems where a highly reliable network between nodes is usual. But a DHT is not necessarily well suited for problem domains where frequent node joins and leaves are the norm. That said, DHTs have been used for routing in many Peer-to-Peer (P2P) implementations [ URI ]. Even some of the more well known NoSQL implementations leverage DHTs.

But as I mentioned, DHTs are not necessarily well suited for all implementations. When we designed the core replication algorithms for the Microsoft Active Directory (AD) service, a DHT was not chosen. Not only did we want to account for Byzantine (opens in a new window) failures but we also needed to account for replica failures that lasted for long periods of time (30+ days). AD subscribes to eventual consistency. Roughly speaking, the replication model of AD is multi-master loose consistency with convergence (eventual consistency). In this model, the directory can have many replicas; a replication system propagates changes made at any given replica to all other replicas. The replicas are not guaranteed to be consistent with each other at any particular time (“loose consistency”), because changes can be applied to any replica at any time (“multi-master”). If the system is allowed to reach a steady state, in which no new updates are occurring and all previous updates have been completely replicated, all replicas are guaranteed to converge on the same set of values (“convergence or eventual consistency”).

In addition to the technical requirements listed earlier in this post, we imposed the following requirements on the design of AD replication:

• To adapt to customer networks, provide flexibility in replication topology including choice of transports.
• Provide a fully asynchronous and highly scalable architecture.
• Minimize network communication in terms of size and round-trips, support encryption of data, and support transitive (store/forward) transportation of data.
• Work well over high-latency communication links.
• Operate correctly when encountering changes to the distinguished name (DN) of an object and to discriminate between a deleted object and a new object with the same DN. In other words, we replicate information based on object’s universally unique identifier ( UUID (opens in a new window) ) and not based on DN.
• Automatic system topology generation. The system automatically creates (and destroys) “short cut” replication links between nodes in order to maintain a specific number of hops between any two points.

Active Directory uses a state-based approach to replication. This approach is easier to appreciate in contrast with an alternative, log-based replication. In a typical log-based system each master keeps a log of the updates that it originated. The goal of each master is to communicate its log to every other replica. Once a log arrives at a replica, the replica applies the log, bringing its state more up to date.

In state-based replication each master applies updates, both originating and replicated, to its replica as they arrive. As such, replication is not driven from logs stored with the source replica but from current state of the source replica. This state includes information for resolving conflicts (also needed in the log-based approach) and information to avoid sending the full replica on each replication cycle (inherent in the log-based approach.) A state-based approach uses a single mechanism for incremental and full sync, and performs fewer database updates since repeated or conflicting updates to an attribute are collapsed into a single state. The resulting design has several stability-enhancing properties, such as:

• No matter how long two replication partners have been out of communication, they can always build upon any previous replication they have accomplished. They never have to start over from scratch because somebody’s log “wrapped around.”
• If a replica has a “hot spot” attribute that’s being updated frequently for some reason (perhaps a bug in an application or an implementation similar to a performance counter), only the value that’s current at the time of replication is sent to a replication partner. Some designs send the full history of all intermediate values thereby multiplying the problem.
• The very same mechanism is used to initialize a new replica as is used for bringing a replica up to date with recent changes. Some other designs require two mechanisms.
• AD has a simple and robust solution to the “time went backwards due to restore from backup” problem (also called the “back sync” problem) that plagues some other systems.
• In a replication relationship the destination (i.e. the replica being updated) takes all responsibility for keeping track of how up to date it is. The source (i.e. the replica supplying the updates) takes no responsibility. This design choice avoids a lot of complexity and potential sources of failure; some other systems don’t work this way.
• In a replication relationship the destination always “pulls” changes from the source. The source may notify the destination “now would be a good time to pull,” but if the notification is lost (e.g. because the destination is overloaded or down) the result is longer replication latency, not incorrectness.
• When two replicas establish a new replication relationship, that relationship is established in an incremental fashion, so not much work is lost should one replica go down before the relationship becomes complete.

Impact on Distributed Applications

A multi-master distributed system induces several problems on applications.

• Version Skew. Version skew occurs when applications read the same object(s) from different replicas before a change has replicated. Applications reading the remote replica see the unchanged object. Version skew is an issue when a given application or set of applications use the information in the directory to interoperate.
• Partial Updates. Partial Update occurs when applications read the same set of objects from different replicas while replication is in progress. Applications at the remote replica see some of the changes but not all. Note that there is a small window in which partial update can affect an application: the application must start reading objects while inbound replication is in progress, after one or more of the related, changed objects have been received but before all have been received. The time between the updates at the source replica directly affects the size of this window—updates that occur close together in time will be replicated close together in time. Partial update is an issue when an application uses a related set of objects.
• Collisions. Collisions occur when the same properties of two or more replicas of a given object are changed during the same replication interval. The replication process reconciles the collision; because of reconciliation a user or application may “see” a value other than the one they wrote. A simple example is user address information—if a user changes their mailing address at replica R-a and an administrator changes the same mailing address at replica R-b, the value ultimately propagated to (R-a, R-b), and all other replicas will be the value selected by the collision reconciliation mechanism. Collision resolution is an issue for applications that make assumptions about the internal consistency of objects or sets of objects.

Though not in scope of this post, applications must accommodate replication latency. The best way to accommodate replication latency is to design applications to minimize the effects—to tolerate latency. The ideal distributed application is of course unaffected by replication latency induced state. Other applications must adopt an avoidance or detection strategy as a mechanism to tolerate replication latency.

Many NoSQL systems, such as MongoDB, employ a master/slave topology and so writes are accepted on a single node. Even the newer replica sets in MongoDB adhere to this model. Sharding (or partitioning), of course, is a key consideration in these types of systems. Sharding enables horizontal scaling across multiple nodes. A sharded MongoDB cluster performs automated leader election for the primary node. Leader election in a ring is another topic in and of itself. So let’s wrap this up…

Wrapping it all up…

Regardless of the underlying durable storage technology, it is of paramount responsibility that you properly classify your information model, understand your consistency model, and develop your application to be highly resilient to failures. It is this latter point—being highly resilient to failures—that is not an easy task and that is precisely why I’ll discuss fault tolerance and recovery in my next post.

…since my intention is to write something useful for anyone who understands it, it seemed more suitable to me to search after the effectual truth of the matter, rather than its imagined one. – Niccolo Machiavelli in The Prince, 1532

While conducting a bit of research on scalability and efficiently storing large amounts (opens in a new window) of information, I ran across an abstract (opens in a new window) in the Communications of the ACM written by a friend and former colleague Pat Helland. Pat and I worked at Microsoft together during the late 90s. Our paths naturally crossed given his core research in high performance transaction systems ( HPTS (opens in a new window) ) and my core work in parallel and distributed systems. Pat recently moved back to San Francisco after 15 years with Microsoft [ URI (opens in a new window) ]. For the two years prior to leaving, Pat worked on Cosmos (opens in a new window) , some of the plumbing for Bing (opens in a new window) . It stores hundreds of petabytes of data on tens of thousands of computers.

The paper (opens in a new window) , “If You Have Too Much Data, then ‘Good Enough’ Is Good Enough”, is brilliantly written. Pat has always been a wonderful and capable writer with an ability to reduce complex problems into a simple presentation. Similar to his clear and concise description of application models using concepts known as “fiefdoms” and “emissaries” in 2002 [ URI (opens in a new window) ] [ URI (opens in a new window) ], Pat identifies a world where many of our classic SQL database principles are being eroded by combining too much data each with disparate characteristics.

The non-relational storage technology (a.k.a. “NoSQL”) movement has of course given rise to the likes of MongoDB (opens in a new window) , CouchDB (opens in a new window) , and Redis (opens in a new window) . Each of these NoSQL technologies have their own unique advantage so judicious evaluation is a MUST (opens in a new window) . The vast amounts of information we process each day in turn has forced us to consider alternate ways to process this information else risk not being able to translate this data into actionable information. Applied technologies such as Hadoop (opens in a new window) and MapReduce (opens in a new window) are fundamental to processing “big data”. We often employ practices and technologies to enable timely dissemination, such as data analytics, business performance management, data warehousing, dashboards and key performance indicators (KPIs). Regardless of what we choose, the fundamentals of how we react to this vast amount of information must be swift, decisive, and–more importantly–accurate in today’s global economy. Being capable of pivoting is essential.

Mobile + Cloud

Mobile and cloud technologies are a winning distributed system solution for many problem domains. But let’s not forget this combination is a distributed system. When designing and implementing distributed systems and the data models on which a distributed system operates, organization is indeed paramount; and so is synchronization. For safety- and mission-critical applications, high availability has always been a paramount concern and recent experience with large Internet sites has underscored the need for availability in that domain as well. Traditional approaches to the problem have made three implicit assumptions:

1. Failure rates of hardware and software are low and improving.
2. Systems can be modeled for reliability analysis and their failure modes can be predicted.
3. Human error during maintenance is not a major source of failures.

The result is an emphasis on failure avoidance as the path to high availability. These assumptions are in many cases based on incorrect perceptions of today’s environment, and that renewed emphasis should be given to failure recovery. Even the most highly tested systems occasionally exhibit “ Heisenbugs (opens in a new window) ” and suffer from transient or permanent hardware failure and software aging, and human error has empirically been found to account for a nontrivial fraction of catastrophic failures. The most successful systems have been those that can recover from these unexpected errors because they were designed for recovery.

If a problem has no solution, it may not be a problem, but a fact—not to be solved, but to be coped with over time. – Shimon Peres, 1923

There is no way to guarantee complete knowledge of the current or future state of a distributed system, because knowledge of state changes must be propagated and propagation takes time, during which more state changes may occur. This is an axiom of distributed computing.

Tightly coupled systems deal with uncertainty by attempting to eliminate it. This is done through constraints on updates, requiring all nodes or some majority of nodes to be available before updates can be performed; using distributed locking schemes or single mastering for critical resources, constraining all nodes to be well connected, or some combination of these techniques. “Majority” often involves weighted voting schemes; so “big” nodes can be more influential than “small” nodes. The more tightly coupled the computing nodes in a distributed system are, the lower the scaling limit.

Loosely coupled systems deal with uncertainty by tolerating it. A loosely coupled system allows participating nodes to have differing views of the overall system state and provides algorithms for resolving conflicts.

My choice for parallel and distributed system design is loosely coupled because:

1. Customers require a highly distributed solution in which parts of the infrastructure can be spread across the internal and public networks and administered locally.
2. Large customers need to grow in capacity to many millions of transactions per day or to hundreds or thousands of nodes, or both.
3. Many networks provide only intermittent connectivity to some locations, for example remote oil drilling platforms and ships at sea, so the system MUST (opens in a new window) be tolerant of partly connected or disconnected operation.

Tightly coupled solutions are unsuitable for parallel and distributed system design because of the requirements for scalability to a very large numbers of nodes and disconnected operation. The loosely coupled model satisfies all of the above requirements.

I’d like to quickly point out an observation regarding tolerance as it relates to most Internet-aware software. As the World Wide Web Consortium ( W3C (opens in a new window) ) points out with respect to tolerance, the principle of tolerance does not blunt the need for a perfectly clear specification which draws a precise distinction between conformance and non-conformance. The principle of tolerance is no excuse for a product which contravenes a standard. Naturally, the aphorism “any problem in computer science can be solved with another level of indirection” still rings true but to a varying degree. The quantification of that degree is a difficult balance to achieve as it involves many variables with the most notable, in my opinion, being scalability, flexibility, and reliability.

I plan to discuss the merits of loose coupling in the larger fabric of parallel and distributed system design more concretely over the next several posts.

” It has been said that art is a collaboration between God and the artist, which works best when the artist contributes as little as possible; so too with designing distributed systems ” –André Gide

Try to imagine the strategic direction of distributed computing frameworks. Just for a moment transition into a state where you think of nothing and yet listen to everything. What key technologies (disruptive or otherwise) have we experienced over recent years?

Notion of Autonomous Computing

Arguably, one of the most well known papers on the subject of autonomic computing (opens in a new window) to be recently released is titled “Autonomic Computing: IBM’s Perspective on the State of Information Technology” [ URI (opens in a new window) ]; authored by Paul Horn (opens in a new window) , Senior Vice President of IBM Research (opens in a new window) . As outlined by Dr. Horn’s manifest, there are many challenges we face over the next decade within Information Technology.

Within the autonomic landscape, each system node must satisfy the following criteria:

• self-identification, self-knowing
• self-(re)configuration
• self-recovery (from perturbations)
• self-protection (security)
• self-learning (including from errors)
• self-regulating (to open standards)
• self-resource-allocation

Notion of Grid Computing

Wouldn’t it be nice to economically provide a highly available and fault tolerant system that can support an annual uptime guarantee of 99.999 percent—which equates to 5.256 minutes of annual unscheduled downtime—and that scales up by a factor of 106? That is, an application’s storage and processing capacity can automatically grow by a factor of a million, doing jobs faster (106x speed up) or doing 106 larger jobs in the same time (106x scale up), just by adding more resources. Stay focused.

The concept of Grid (or Utility) computing was coined in the mid 1990s and is best defined—to quote The Anatomy of the Grid (opens in a new window) —the real and specific problem that underlies the Grid concept is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations.

The Grid Global Forum (GGF (opens in a new window) ) has made great headway; so has OGSA (Open Grid Services Architecture). Message Passing Interface (MPI) (opens in a new window) is widely accepted. Microsoft entered the HPC (opens in a new window) game with the Microsoft Compute Cluster Server (opens in a new window) , which I installed many months ago and is now consuming most of my free time.

Notion of Adaptive Autonomous Agents

An agent is a system that tries to fulfill a set of goals of goals in a complex, dynamic environment. An agent it situated in the environment; it can sense the environment through its sensors and act upon the environment using its actuators. An agent’s goal can take many different forms: they can be “end goals”, or particular states the agent tries to achieve; they can be selective reinforcement or reward that the agent attempts to maximize; they can be internal needs or motivations that the agent has to keep within certain viability zones and so on. An agent is called autonomous if it operates completely autonomously, i.e. if it decides itself how to relate its sensor data to motor commands in such a way that its goals are attended to successfully. An agent it said to be adaptive, if it is able to improve over time, i.e. if the agent becomes better at achieving its goals with experience.

The study of Adaptive Autonomous Agents is grounded in two important insights, which serve as “guiding principles” for most of the current research performed:

• Looking at complete systems changes the problems often in a favorable way.
• Interaction dynamics can lead to emergent complexity.

Essentially, an agent is viewed as a set of competence modules (often called behaviors). These modules are responsible for a particular small task-oriented competence. Each of the modules is directly connected to its relevant sensors and actuators. Modules interface to one another via extremely simple messages rather than a common representation of beliefs, and so on. The communication between modules is almost never of a “broadcast” nature, but happens rather on a point-to-point (or one-to-one) bases. Typically, the messages consist of activation energy, or simple suppression and inhibition signals, or simple tokens in a restricted language. In addition to communication via simple messages, modules also communicate “via the environment”. One module may change some aspect of the environment, which will trigger another module, etc.

Notion of the Semantic Web

Simply put, the Semantic Web (opens in a new window) is the representation of data on the web in which information is given well-defined meaning—“to be a universal medium for the exchange of data”.

The principal technologies of the Semantic Web fit into a set of layered specifications called the Resource Description Framework (RDF) (opens in a new window) . The current components of that framework are the RDF Core Model, the RDF Vocabulary Description Language and the Web Ontology Language, which all build on the foundation of URIs, XML, and XML namespaces.

The most interesting of these languages is the Web Ontology Language (OWL) (opens in a new window) , which is a descriptive layer built on top of RDF used to model classes, properties, and objects.

Ontology is also a term borrowed from philosophy that refers to the science of describing the kinds of entities in the world and how they are related. Stated another way, an ontology defines the terms used to describe and represent an area of knowledge.

Notion of Service-Oriented Architecture

I think everyone has been beaten of the head with the “SOA stick”, which is why you won’t feel a thing; keep reading.

In computing, the term Service-Oriented Architecture (SOA) expresses a software architectural concept that defines the use of services to support the requirements of software users. In a SOA environment, nodes on a network make resources available to other participants in the network as independent services that the participants access in a standardized way. Most definitions of SOA identify the use of Web services (i.e. using SOAP or REST) in its implementation. Here (opens in a new window) and here (opens in a new window) are two links worth reading. However, one can implement SOA using any service-based technology.

The WS* specifications have gained wide adoption but other advances are needed.

One last point as related to service orientation; I caution everyone to not forget about the elegance in design of class libraries as they are the underpinnings of every SOA. Design for the in-process consumer first with an eye to areas of the API surface that might benefit from hosting within an SOA.

Other advances…

There are many others such as advances in peer-to-peer algorithms, discrete-event simulation and the event horizon, social networking, recovery-oriented systems, storage and machine virtualization, bioinformatics (opens in a new window) , biotechnology (opens in a new window) , and quantum computing (opens in a new window) to name but a few. In my mind, the two most important questions to answer are which of these variables are of significant weighting in the equation of strategic direction and which are “noise”? I certainly have my opinion…but then again, you know what those are like.

Most readers and programmers have little patience to read discussions of this length. However, the length (at least to me) seems disproportionate to the importance of the topic. There is so much more to say and even more to ponder. But, before you run off to save the world let me leave you with one additional thought:

“The map is not the territory.” Alfred Korzybsky Science and Sanity, 1933