“If you think cryptography will solve your problem, then you don’t understand cryptography and you don’t understand your problem.” — Roger Needham

A few colleagues and I recently revisited a familiar debate. In most data systems, identifier generation is treated as undifferentiated infrastructure—one library, one default, applied everywhere. The implicit assumption is that a sufficiently random identifier is a sufficient answer to whatever question the system happens to ask of it; that the differences between identifier classes—surrogate keys, content fingerprints, anonymized handles, span identifiers, request correlations—are differences of usage rather than differences of construction. That assumption is convenient. It survives precisely until the system encounters constraints that aren’t engineering constraints.

For the past several years, I’ve designed and currently operate a clinical research data pipeline serving multiple concurrent Institutional Review Board (opens in a new window) (IRB)-approved studies against Electronic Health Record (opens in a new window) (EHR)-sourced data at production scale. Though, implementation specifics are abstracted from this post, I want to highlight the architectural patterns. In human-subjects research, identifier construction stops being a performance choice and becomes a regulatory and ethical primitive.

Why? Because the constraints are different from those that arise in commercial systems. Re-identification resistance is enforceable under the Health Insurance Portability and Accountability Act (opens in a new window) (HIPAA) and the Common Rule (opens in a new window) . Per-study isolation is an IRB requirement. Provenance MUST (opens in a new window) survive transformation but MUST NOT enable cross-study linkage. Workflows that depend on the operator’s inability to reverse a transformation can’t be retrofitted onto a single global identifier scheme; they have to be built into the construction itself. None of these properties emerge from a generic randomness primitive applied uniformly to every identifier the pipeline produces.

So let me state the pattern plainly. A real human-research pipeline needs at least three distinct identifier constructions. Each answers a different question. Each lives at a different boundary. Treating identifier generation as a single problem with a single answer breaks one architectural pillar in service of another. The failure mode isn’t that any single construction is wrong, but that no single construction is right for every boundary the pipeline must respect.

Three Pillars, Three Constructions

A human-subjects research pipeline must answer three questions of every row that flows through it. The questions are independent and every row is interrogated by the following pillars:

1. Fidelity. Is this row the same as before, or has it changed? A pipeline that can’t answer this can’t deduplicate, can’t detect drift, and can’t reason about whether a transformation preserved meaning. Fidelity is the substrate on which idempotency, change detection, and lineage are built. Without it, every other property the pipeline claims to provide is conjectural.

2. Governance. Can this row be re-linked to a person, or to the same person across studies? In a research context, the answer must be no, and the answer must be enforceable rather than promised. Policy is necessary but not sufficient; if the construction permits re-linkage, then policy is the only thing preventing it, and policy can be circumvented, misconfigured, or socially engineered. Governance asks whether the construction itself denies the operations that policy forbids.

3. Provenance. Which pipeline run produced this row, from which source, on which day, by which transformation? Provenance is the operational substrate of accountability. It allows any finding to be traced back to its derivation, any anomaly to be localized to a stage and a run, any audit to reconstruct what happened without depending on the cooperation of the system that produced the result.

Each question is answered by a different cryptographic construction at a different boundary:

• Content-addressed SHA-256 (opens in a new window) hashing answers fidelity at the ingest boundary.
• Per-tenant keyed HMAC (opens in a new window) hashing answers governance at the publication boundary.
• CSPRNG (opens in a new window) -backed identifier generation answers provenance at the operational layer.

The constructions are not interchangeable. Using any one of them outside its intended boundary either fails the question its boundary is supposed to answer or undermines a pillar that some other boundary was responsible for. Let me walk each in turn, identify the constraint it answers, and explain why the substitution of a generic identifier scheme is an architectural mistake rather than a minor inefficiency.

Fidelity at the Ingest Boundary

At the ingest boundary, the pipeline receives raw data from a source system that doesn’t coordinate with the pipeline about identity. Files arrive on a schedule, are written to a lake in raw form, and become the substrate from which everything downstream is derived. The question fidelity asks of every row at this boundary is whether it’s the same row that arrived yesterday, last week, or in any prior incremental load. If the pipeline can’t answer that question cheaply and unambiguously, then:

• Deduplication becomes guesswork.
• Idempotency becomes a hope.
• The lake’s role as the recovery anchor for the entire pipeline is compromised.

The construction that answers this question is a SHA-256 hash computed over the row’s sorted business columns at the moment of lake ingestion. Simply put, “business columns” are all columns except for metadata columns that were added by the pipeline itself. The hash is written into a metadata column that is then carried unchanged through every downstream stage. It MUST NOT be recomputed at the warehouse boundary, must not be recomputed after foreign-key re-keying at segmentation, and must not be recomputed under any condition the pipeline operator may otherwise consider reasonable. The construction is content-addressed and immutable.

The non-randomness of this identifier is the point. Two rows with identical business-column content produce identical hashes. Two rows with any meaningful difference produce different hashes with overwhelming probability. The hash is unique by virtue of its content, not by virtue of its randomness, and that property is what makes change detection tractable across overlapping daily exports without requiring the source system to participate in a coordination protocol it was never designed to support. A randomized identifier would be wrong here in the strongest sense: it would foreclose exactly the property the boundary needs.

The same property that enables change detection also serves as a duplicate pre-filter. Rows that share identical business-column values are duplicates by definition, and the hash exposes that fact at the moment of ingestion rather than deferring it to a downstream reconciliation step. Duplicate detection becomes a comparison of fixed-width hashes rather than a comparison of arbitrarily wide row contents, and the cost of identifying duplicates collapses to the cost of comparing two values. The work the pipeline would otherwise have to do further downstream is done once, at the boundary, by the construction itself.

The hash also serves as the first link in the lineage chain that fidelity, governance, and provenance all eventually depend upon. Any row anywhere in the pipeline can be traced back to its source by following the hash. Any divergence between a stage’s input and its output can be diagnosed by comparing hashes. Any claim that two rows in different marts derive from the same source row is verifiable without consulting either source or mart, because the hash is the same in both places. The hash is the substrate; everything else is built on top of it.

A subtlety worth naming is that the hash construction itself sits on a hot path. At ingest scale, tens of millions of rows pass through the pipeline on every full load, and the hash is computed for every one of them. The disciplines that apply to identifier generation under sustained concurrency apply equally here: allocation behavior MUST be controlled, primitive selection MUST avoid implicit coordination, and the cost of the construction MUST NOT become the dominant cost of ingestion. The arguments I developed in my earlier post on NanoID hot-path optimization generalize directly to hash construction in pipelines of this kind. The construction is different; the discipline is the same.

Governance at the Publication Boundary

The boundary between the warehouse and the data marts is where the pipeline transitions from internal computation to external publication. Inside the warehouse, identity is managed by the pipeline operator, and the SHA-256 hash from the ingest boundary serves every internal purpose well. Outside the warehouse, identity becomes a research artifact, and a fundamentally different requirement asserts itself: the identifier that reaches the mart MUST NOT enable re-identification of the underlying subject, and it MUST NOT (opens in a new window) enable cross-mart linkage of the same subject across separate research projects.

Both requirements are properties of the construction, not properties of the data. A SHA-256 hash carried untouched into a mart would satisfy neither. Why? The hash is deterministic; identical inputs produce identical outputs; an attacker with sufficient knowledge of the source schema could attempt to reconstruct the input space and brute-force the mapping. More immediately, the same warehouse row would carry the same hash into every mart, and any researcher with access to two marts could link rows trivially by comparing hashes. The construction permits exactly the operations that governance forbids.

Stay with me—the answer here is a small but architecturally decisive change. The construction that answers governance at this boundary is HMAC-SHA256, keyed with a per-mart secret stored in a Hardware Security Module (opens in a new window) (HSM)-backed secrets manager and never exposed to the pipeline operator. At the segmentation stage that produces each mart, the warehouse-side SHA-256 hash is replaced by HMAC-SHA256(warehouse-hash, per-mart-secret). The result is a new identifier that is:

• Deterministic with respect to the same inputs.
• Cryptographically opaque without the secret.
• Distinct across marts, because the secret is distinct across marts.

This construction has a property that’s easy to overlook and architecturally decisive: the HMAC operates on the SHA-256 hash, not on the raw business columns. An attacker who somehow obtained a per-mart secret would not face the relatively constrained problem of inverting the HMAC over a known schema of patient identifiers; they would face the problem of inverting it over the full output space of SHA-256, which is 2^256 possible inputs. A dictionary attack against patient identifiers, plausible in principle against a single-layer keyed hash, is eliminated by construction. The two-layer design—content hash first, keyed hash of the hash second—is what makes the cryptographic claim survive contact with a realistic threat model.

The architectural consequence is that the publication boundary becomes a place where identity is not merely transformed but severed. The original SHA-256 hash never reaches the mart. The mart receives the HMAC, and only the HMAC, as its identifier for that row. The pipeline operator, who has access to the warehouse and to the segmentation code, can’t reverse the transformation, because the operator doesn’t have the per-mart secret. The secret lives in an HSM-backed store that the operator can use but can’t extract. The same warehouse row, written into three different marts, carries three different identifiers, and no party—including the pipeline operator—can establish that the three identifiers refer to the same subject without simultaneous access to all three secrets and the cooperation of the secrets manager.

This is what makes the honest-broker workflow cryptographically tenable rather than merely operationally promised. In human-subjects research, there are workflows where the operator’s inability to reverse the transformation is structurally required: the team that operates the pipeline MUST be able to publish the data, but MUST NOT be able to re-identify the subjects whose data it published. Under a single-layer construction, the operator can always be compelled, suborned, or compromised into reversing the mapping. Under the two-layer construction described here, the operator simply doesn’t have the capability to reverse, and therefore can’t be compelled to exercise it. The construction enforces what policy could only request.

The bridge to my earlier work on AES-CTR-DRBG is direct. In that post I argued that deterministic cryptographic constructions belong in environments where reproducibility, auditability, and explicit state evolution are required. The HMAC-at-publication pattern is the same disposition applied to identity rather than to byte streams. The construction is deterministic with respect to its inputs; the state that determines its output evolves only when the per-mart secret is rotated; the auditor can reason about what the construction produces without having to reverse-engineer the library that implements it. Configuration is a contract. Boundaries are explicit. The cryptographic posture doesn’t change mid-flight.

A randomized identifier would be wrong here in a different way than at the ingest boundary, but no less wrong. A randomized identifier is non-deterministic; the same row processed twice would produce different identifiers; the mart’s referential integrity would collapse on the first re-run. What the boundary needs is deterministic-but-cryptographically-opaque, and the only construction that delivers both properties is a keyed hash.

Provenance at the Operational Layer

The constructions described so far handle the headline identity story—the identity of the rows themselves. They’re computed once per row and carried thereafter. They’re also, by a substantial margin, not the identifiers the pipeline generates most often.

A pipeline of any operational seriousness produces identifiers continuously for things that aren’t rows. Pipeline runs need identifiers. Stage executions need identifiers. Batch tasks under a thread pool need identifiers. Observability spans, if the pipeline exports traces, need identifiers. Synthetic sentinel rows injected at segmentation need identifiers distinguishable from real source identifiers. Project allocations need identifiers that survive across systems. Honest-broker requests need identifiers that the requesting system, the secrets manager, and the provisioning workflow can all reference unambiguously. Quality diagnostic runs need identifiers for the audit trail that links them to the pipeline run that triggered them.

These identifiers have a constraint regime distinct from anything the row-level constructions answer:

1. They are allocated at high concurrency, often from many worker threads or pods simultaneously, with no central allocator available.
2. They MUST be collision-resistant by virtue of their construction, because no coordinator exists to detect and resolve a collision after the fact.
3. They MUST be URL-safe and human-tractable, because they will appear in logs, dashboards, audit trails, and the bodies of cross-system requests.
4. They MUST be allocation-cheap, because the pipeline generates them at orders of magnitude higher volume than it generates SHA-256 row hashes or per-mart HMACs.
5. They MUST NOT leak temporal patterns or sequence structure that an observer could exploit to infer pipeline behavior or correlate identifiers from independent runs.

In raw call volume, this class of identifier dominates the SHA-256 and HMAC constructions by orders of magnitude. The hash is computed once per source row at ingest. The HMAC is computed once per row per mart at segmentation. The operational identifiers are generated continuously: on every notebook, on every span, on every parallel task, on every cross-system request. If the construction here is wrong, the cost is paid everywhere the pipeline does any work at all.

This is the design space my NanoID library (opens in a new window) was built for, and the disposition that produced it is the disposition that earlier governed its randomness backends. The construction at the call site is identical regardless of the workload—a single function call returning a compact, URL-safe identifier—but the randomness source behind it is configurable, and the choice of source is made per identifier class rather than per call site. The architectural payoff is that the cryptographic posture of the identifier is determined at construction time and survives every downstream invocation without requiring the call site to be aware of it.

For the highest-volume operational identifiers—span identifiers, batch identifiers, stage execution identifiers, the thousands of small handles that a pipeline produces in the course of doing its work—the appropriate backend is the ChaCha20-based PRNG I developed in an earlier post. It’s designed for predictable behavior under sustained concurrency, with allocation discipline on the hot path and no shared coordination between worker threads. It doesn’t target any particular compliance regime, because none applies at this layer; what applies is the constraint that identifier generation MUST remain invisible in profiling output even at the volumes the pipeline reaches.

For the identifiers that participate in audit, IRB, or compliance trails—project allocations, honest-broker request identifiers, anything that ends up in a Key Vault audit log or a regulatory submission—the appropriate backend is the AES-CTR-DRBG construction I developed in the post before that. It targets FIPS 140-2 (opens in a new window) -aligned cryptographic randomness with deterministic state evolution, allowing the identifiers it produces to be traceable, auditable, and defensible under scrutiny in ways that a general-purpose PRNG is not. The construction is the same NanoID library, configured with a different randomness source via the same WithRandReader option, called identically at the call site.

The two backends are not competing; they answer two different constraints within the same pipeline, on the same library surface, decided once at construction time. The same disposition that governs SHA-256 and HMAC at their boundaries—the construction is chosen for the boundary, not for the codebase—governs NanoID at the operational layer.

One Library, Three Backends

The library design that makes this tractable isn’t a convenience feature. It’s an architectural commitment that allows per-boundary construction choice to coexist with a single call-site surface. NanoID exposes a WithRandReader option that accepts any io.Reader as its randomness source. The choice of source is made once, at generator construction time, and survives every subsequent identifier the generator produces. The call site doesn’t change; the cryptographic posture does.

This mirrors the architectural move SHA-256 and HMAC are making at their respective boundaries. Both belong to the SHA-2 family. Both compose with the same underlying hash primitive. The difference between them is whether a per-tenant key is involved, and whether the construction can therefore enforce the cryptographic break that publication requires. The primitive is shared; the construction is chosen for what the boundary MUST guarantee. Same disposition, same shape, different layer.

The negation is instructive. A pipeline that uses uuid_generate_v4() everywhere—or NanoID with a single hardcoded randomness source applied to every identifier it produces—collapses distinctions that the architecture is doing real work to preserve. Surrogate keys for fact tables receive the same construction as project identifiers feeding IRB submissions, which receive the same construction as observability spans. Each is correct for some workload and wrong for at least one other. The pipeline that adopts a single global construction hasn’t simplified its identifier story; it has merely declined to make decisions that its boundaries will eventually force it to revisit.

The pipeline that adopts the layered model described here makes those decisions once, at the boundary where each one belongs, and then stops thinking about them.

• The hash is content-addressed at ingest.
• The HMAC is per-tenant keyed at publication.
• The NanoID is randomly allocated at the operational layer, with the randomness backend chosen for the identifier class.

Each construction is correct for its boundary. Each construction is composable with the others, because the boundaries are explicit and the constructions don’t interfere. The architecture isn’t more complex than the alternative; it’s more honest about the complexity that was already present.

Wrapping it all up…

The earlier articles in this sequence each followed a single constraint to its conclusion. My 2024 article on NanoID followed the hot-path discipline that emerges when identifier generation moves from incidental utility to infrastructure. My 2025 article on AES-CTR-DRBG followed the requirements of compliance, auditability, and explicit state evolution. My article on the ChaCha20-based PRNG followed the constraints of portability, concurrency, and operational simplicity. In each case, a single set of requirements produced a single construction.

This article inverts that move. Given a domain—human-subjects research—that imposes multiple constraint regimes simultaneously, the architectural response isn’t a single construction but a layered one. Fidelity, governance, and provenance aren’t competing concerns to be balanced against one another. They’re independent questions answered by independent constructions at independent boundaries, and the pipeline that respects the independence of the questions can satisfy all three without trading any of them off against the others.

Make no mistake; the constructions themselves are unremarkable. SHA-256 has been a standard primitive for two decades. HMAC has been a standard primitive for longer. CSPRNG-backed identifier generation is a solved problem with well-understood implementations. The architectural work isn’t in the choice of primitives. It’s in the recognition that each primitive is correct only for the boundary that actually needs the property it provides, and in the discipline of refusing to substitute one for another where the substitution would be easier but wrong.

Good architecture emerges when constraints are REQUIRED (opens in a new window) and followed to their conclusions, even when those conclusions lead to different constructions under different constraints. The earlier articles argued this for randomness. The pipeline architecture described here argues it for identity. The answer is the same in both cases: make the boundaries explicit, choose the construction the boundary needs, and let the disposition do the rest.

My next post will take a step back and explore the data analytics and insights pipeline from an architectural perspective.

Implementation: https://github.com/sixafter/nanoid (opens in a new window)

“A system is not determined by the algorithms it uses, but by the assumptions it makes.”
— Leslie Lamport

Randomness appears deceptively uniform in most software systems. A call is made, bytes are returned, and execution proceeds. The details of how those bytes were produced rarely matter—until they do. As with many foundational components, the assumptions baked into a randomness source tend to surface only when systems scale, diversify, or are subjected to constraints they were not originally designed to accommodate.

In the previous article , I explored deterministic cryptographic randomness under compliance and auditability constraints, where explicit state evolution and reproducibility were not optional. That work followed inevitably from environments shaped by regulation and forensic accountability. This article explores a different design space: one where the constraints are imposed not by certification regimes, but by software portability, concurrency, and operational simplicity.

The result is a ChaCha20-based (opens in a new window) pseudorandom number generator (PRNG) designed for predictable behavior in real systems, rather than alignment with formal validation programs.

What distinguishes this space from compliance-driven designs is not weaker security assumptions, but different priorities. Software systems are often judged not by formal proofs or certifications, but by how they behave under sustained load, across diverse platforms, and in the presence of concurrency. In those environments, simplicity is not aesthetic—it is operational. Every implicit dependency, hidden allocation, or global coordination point becomes a liability over time.

Constraints Shape Constructions

Cryptographic primitives are often discussed as interchangeable building blocks. In practice, the surrounding constraints determine which properties matter and which can be safely ignored. A construction optimized for auditability and deterministic replay will look very different from one optimized for simplicity, portability, and high-throughput concurrent use.

In software-heavy environments—particularly those spanning multiple platforms or deployed as libraries rather than systems—certain requirements surface repeatedly. The randomness source must behave predictably under concurrency. It must avoid surprising allocation behavior. It must scale without coordination bottlenecks. And it must do so without pulling in complex dependencies or imposing policy decisions on its callers.

These are not compliance constraints. They are engineering constraints.

Engineering constraints tend to accumulate quietly. A library that is easy to use in isolation may behave very differently when embedded in larger systems, reused across services, or exercised under high concurrency. Over time, small assumptions compound: about allocation behavior, about shared state, about initialization cost. What begins as a convenient abstraction can become a systemic bottleneck. Designs that survive this environment are rarely the most general ones; they are the ones that make their tradeoffs explicit early.

ChaCha20 fits naturally into this space. As a stream cipher designed for efficient software implementation, it offers strong cryptographic properties without reliance on specialized hardware instructions or platform-specific optimizations. That makes it a practical foundation for a general-purpose PRNG intended to behave consistently across environments.

State, Not Entropy, Does the Work

As with any PRNG, the core of the design is not entropy acquisition but state evolution. Entropy enters the system at initialization. From that point forward, output is a function of internal state advancing according to a defined algorithm.

This distinction matters. By treating entropy acquisition as a boundary rather than a continuous process, the generator remains honest about its dependencies. The operating system provides unpredictability at startup. The PRNG provides expansion, isolation, and repeatable behavior thereafter.

ChaCha20’s design reinforces this model. Once keyed and initialized, the cipher produces a stream of pseudorandom bytes derived entirely from its internal state. There are no conditional branches based on environmental input, no opportunistic refreshes, and no hidden coordination with external sources. State advances monotonically. Output follows deterministically.

Thinking in terms of state rather than entropy also clarifies responsibility. Once initialization completes, the generator no longer depends on the operating system or the environment for correctness. The burden shifts entirely to the implementation and its callers. That boundary is important: it allows failures, assumptions, and limitations to be localized rather than diffused across layers. When something goes wrong, there is no ambiguity about where to look.

This makes the generator easier to reason about, test, and integrate. The absence of implicit behavior is not a limitation; it is a deliberate choice that keeps responsibility visible.

Concurrency Without Contention

Concurrency tends to expose the weaknesses of otherwise elegant designs. Shared mutable state invites contention. Global locks introduce unpredictable latency. Per-call initialization imposes overhead that scales poorly as systems grow.

The ChaCha20 PRNG approaches concurrency as a distribution problem rather than a synchronization problem. Independent generator instances maintain independent state. A pooling strategy amortizes initialization costs while avoiding shared counters or global coordination. Each request draws from a generator whose behavior is local, predictable, and isolated from others.

This approach mirrors patterns commonly used in distributed systems: partition state, avoid global serialization, and accept modest duplication in exchange for clarity and scalability.

In practice, this means resisting the temptation to centralize randomness behind a single shared construct. While such designs appear simpler at first, they introduce hidden coupling that surfaces under load. Contention becomes observable latency. Synchronization becomes an availability concern. By allowing each generator instance to evolve independently, the system preserves the same semantic behavior regardless of how many consumers are active.

Runtime Semantics and Predictable Behavior

Allocation behavior is treated as a semantic concern rather than a micro-optimization. A PRNG that allocates unpredictably during output generation cedes part of its execution model to the runtime.

The effects of these choices are observable in practice. Small reads primarily reflect fixed overhead. Larger reads scale linearly with the amount of data processed, as expected from a stream cipher advancing its internal counter. Under concurrent workloads, latency remains stable because requests are serviced independently rather than serialized behind shared state.

What matters is not the absolute numbers, but their predictability. The behavior matches the design model, rather than fluctuating with runtime conditions.

This predictability has secondary effects that are easy to underestimate. Monitoring becomes more meaningful when latency distributions are stable. Capacity planning becomes tractable when throughput degrades linearly rather than catastrophically. Perhaps most importantly, anomalies become visible. When behavior deviates from expectation, it signals a real change in conditions rather than noise introduced by the runtime.

Failure Modes and Explicit Boundaries

Boundaries are explicit. Initialization depends on a trusted entropy source. Once initialized, the generator’s behavior is deterministic and bounded by construction. Errors are surfaced rather than hidden.

Surfacing errors is not merely a defensive choice; it is an architectural one. Silent degradation may preserve uptime in the short term, but it undermines trust in the system over time. By making failure explicit, the generator avoids the pretense of correctness under conditions it was not designed to handle. This makes integration more demanding, but also more honest.

This approach does not attempt to cover every possible threat model. It does not claim resistance to state compromise beyond what the underlying primitive provides. It does not attempt to retrofit compliance semantics onto a software-oriented design.

Closing Thoughts

This ChaCha20-based PRNG (opens in a new window) is used as the randomness source in other systems, including my NanoID (opens in a new window) implementation, where predictable behavior, low overhead, and concurrency safety are practical requirements rather than abstract goals.

Randomness is not a single problem. Different environments impose different constraints, and those constraints shape the constructions that make sense within them.

This ChaCha20-based PRNG exists because certain systems demand a randomness source that behaves predictably under concurrency, avoids unnecessary complexity, and makes its assumptions explicit. It treats randomness as evolving state rather than as an opaque service call.

The tension between generality and specificity is unavoidable in systems design. Libraries that attempt to serve every use case often end up serving none particularly well. By contrast, designs that accept their constraints openly tend to age more gracefully. They are easier to reason about, easier to integrate, and harder to misuse.

Implementation: https://github.com/sixafter/prng-chacha (opens in a new window)

“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger W. Dijkstra

Randomness occupies an unusual position in software systems. We rely on it for security, for fairness, for simulation, and yet we rarely scrutinize its behavior with the same rigor we apply to storage engines or replication protocols. We accept entropy as something that simply happens, mediated by an operating system call or a library abstraction, and move on.

That approach works—until it doesn’t.

When I began working on systems that required both cryptographic correctness and operational predictability, the implicit assumptions around randomness became a liability. Latency mattered. Allocation behavior mattered. Reproducibility mattered. And in certain environments, auditability mattered most of all. The standard tools did not fail outright, but they failed quietly, by obscuring behavior that ought to be explicit.

At its core, the problem is not randomness itself, but how cryptographic state evolves over time—and whether that evolution remains visible, bounded, and predictable.

This article is about addressing that gap.

Determinism Is Not the Enemy

In distributed systems, we accept a fundamental axiom: there is no way to guarantee complete knowledge of global state at any given moment. We design for tolerance, convergence, and recovery, not for omniscience; ref Parallel and Distributed System Design ). Randomness, however, often escapes this discipline. It is treated as an oracle—invoked, trusted, and forgotten.

Cryptographic systems take a different view. They do not reject determinism; they constrain it. Given a fixed internal state and a fixed construction, output must follow. This is not a weakness but a requirement. It is precisely what allows us to reason about forward security, backtracking resistance, and compromise scenarios.

AES-CTR-DRBG sits squarely in this space. It provides a deterministic construction based on a well-understood primitive, AES, and defines explicit rules for how internal state evolves over time. Once you accept that premise, several consequences follow naturally. Hidden entropy sources become suspect. Implicit reseeding becomes problematic. Mutable configuration becomes dangerous.

The design space narrows considerably.

Deterministic Constructions and Cryptographic Honesty

Cryptography is often discussed in terms of strength, but strength alone is not sufficient. A system can employ strong primitives and still behave in ways that are difficult to reason about. When that happens, security becomes an emergent property rather than an explicit one.

Deterministic constructions reject that ambiguity.

An AES-CTR-DRBG (opens in a new window) does not attempt to surprise the reader. Given a key, a counter, and a defined update function, the output follows directly. There is no hidden mixing, no opportunistic entropy acquisition, and no conditional behavior based on environmental state. This predictability is not merely convenient; it is foundational. It allows one to reason about compromise without first reverse-engineering library behavior.

Once determinism is accepted as a requirement, architectural freedoms disappear. You can no longer “improve” security by quietly pulling in entropy. You can no longer treat reseeding as an implementation detail. You must decide, explicitly, when state changes and why.

This is not a limitation of AES-CTR-DRBG. It is a discipline imposed by cryptography itself.

These constraints do not arise in isolation. They emerge most clearly in regulated, multi-tenant, and distributed systems, where randomness is no longer an incidental utility but a component subject to scrutiny. In such environments, deterministic and auditable cryptographic randomness becomes a requirement rather than a preference. Reproducibility matters for forensic analysis. Isolation matters for per-tenant security. Predictable resource behavior matters when concurrency and throughput are no longer theoretical concerns.

The standard random primitives in Go do not address this space. They make no guarantees around deterministic output, explicit key management, or resource predictability under concurrency, nor do they target alignment with NIST SP 800-90A (opens in a new window) or FIPS 140-2 (opens in a new window) requirements. Third-party alternatives often complicate matters further, introducing additional dependencies or obscuring allocation behavior and latency characteristics behind opaque abstractions.

At scale, these gaps become operational risks. System PRNGs (opens in a new window) do not ensure reproducibility. Key lifecycle management and tenant isolation are left to the caller. Failure modes relevant to compliance and auditability remain implicit, even as entropy sources degrade or cryptographic assumptions are violated. What is missing is not another source of randomness, but a generator whose behavior remains explicit, bounded, and inspectable over time.

These constraints define the space for a deterministic random bit generator based on AES (opens in a new window) in counter mode ( CTR (opens in a new window) ), as specified in NIST SP 800-90A and aligned with FIPS 140-2. The implementation discussed here relies solely on Go’s standard library cryptography and targets allocation-free output and low, predictable latency, making it suitable for high-throughput and regulated environments where correctness and observability are inseparable.

Entropy, State, and the Shape of Time

Many practical failures in cryptographic systems arise not from broken primitives, but from blurred boundaries. Entropy acquisition is a boundary. Deterministic generation is another. When the two are conflated, responsibility becomes unclear.

Operating systems are good at collecting entropy. Applications are good at applying policy. A DRBG sits between the two, which is why many implementations quietly collapse the distinction.

I chose not to.

In this design, entropy enters the system at a clearly defined point and nowhere else. The generator consumes seed material and expands it deterministically. If reseeding occurs, it occurs because the caller has decided that new entropy is available and appropriate. This makes the system honest about its dependencies. It also makes it auditable. One can trace exactly when and how entropy influences output, which is a prerequisite for meaningful analysis.

This mirrors a lesson learned repeatedly in distributed systems: implicit coordination is coordination nonetheless. If you depend on entropy, say so.

It is helpful to visualize the generator not as a black box, but as a state machine evolving over time.

Imagine a simple diagram with three persistent elements: Key, Counter (V), and Limits. At initialization, these are set from seed material and configuration. Each request for output advances the counter deterministically, producing a block of output via AES encryption. The key remains fixed until a reseed event occurs. Limits monotonically decrease as output is generated.

Time, in this model, does not pass in seconds. It passes in blocks generated.

A reseed event resets the key and counter and restores limits. Absent a reseed, state evolution is linear, irreversible, and predictable. There are no side paths, no hidden refreshes, and no conditional jumps based on runtime conditions. The diagram contains no feedback loops other than the explicit reseed boundary.

This mental model is intentionally simple. It allows one to reason about compromise, exhaustion, and recovery without hand-waving.

Reseeding is often presented as a binary feature: enabled or disabled. In practice, it is a temporal question. How long may a generator safely operate on a given internal state? How much output may be derived before assumptions begin to erode?

FIPS-aligned guidance does not answer these questions abstractly; it answers them in terms of limits. Maximum bytes per key. Maximum requests between reseeds. Defined update functions. These limits exist not because AES is fragile, but because systems operate over time, and time introduces exposure.

In a deterministic generator, time is measured not in seconds but in output. Each block generated advances the internal counter. Each advancement is irreversible. By enforcing explicit limits, the generator treats time as a first-class dimension rather than an afterthought.

This perspective aligns closely with how we reason about versioning and state evolution in distributed systems. State does not merely exist; it progresses. If you do not bound that progression, you eventually lose the ability to reason about it.

Prediction resistance is frequently discussed as an absolute good. In reality, it is a trade. Enabling it increases resilience against certain compromise scenarios, but it also introduces new operational dependencies. Reseeding requires entropy. Entropy requires coordination with the environment. Coordination introduces latency and failure modes.

Rather than treating prediction resistance as a default, this design treats it as a conscious decision. When enabled, reseeding behavior is strict and enforced. When disabled, the generator behaves as a pure deterministic expander. Neither mode is inherently superior; each corresponds to a different threat model.

Cryptographic systems fail when they attempt to serve all threat models simultaneously. By forcing the caller to choose, the system avoids pretending that one size fits all.

Runtime Semantics and Predictability

Garbage collection is often discussed in terms of performance, but in latency-sensitive systems it also becomes a semantic concern. Allocation introduces nondeterminism: not in output, but in timing. When a generator allocates during its hot path, it cedes control of latency to the runtime.

The effects of these choices are observable in practice. Under serial workloads, small reads complete in tens of nanoseconds, largely reflecting fixed overhead rather than per-byte cost. Larger reads exhibit higher absolute latency, not because of inefficiency, but because AES-CTR scales linearly with the number of blocks processed. Encrypting a few blocks versus a few hundred necessarily occupies different regions of the CPU pipeline, even when the underlying primitive is highly optimized. What matters is that this behavior is predictable and stable under load, matching expectations derived from the algorithm and the memory hierarchy rather than fluctuating with runtime conditions.

Achieving zero allocations in the output path was, therefore, non-negotiable; read “ MUST (opens in a new window) ”.

What matters here is not raw speed, but control. A system that allocates implicitly hands part of its execution semantics to the runtime. In many contexts that is acceptable; in some, it is not. When randomness participates in protocol boundaries, scheduling decisions, or security-sensitive operations, latency variance becomes observable behavior. Treating allocation as a semantic choice forces the design to acknowledge that reality. Once allocation is removed from the generation path, timing becomes a property of the algorithm itself rather than a side effect of memory management. That shift restores a degree of predictability that is otherwise difficult to recover.

The decision to make this DRBG allocation-free during generation was not an optimization exercise; it was a correctness constraint. Once generation begins, no heap allocations occur. The caller owns the buffers. The internal state resides in fixed-size structures. The runtime remains uninvolved. Latency variance collapses. Throughput becomes predictable. The generator behaves the same under load as it does in isolation.

Concurrency often exposes design shortcuts. Shared mutable state invites contention; global locks invite collapse. In the context of deterministic randomness, concurrency introduces an additional risk: accidental nondeterminism caused by interleaving.

The solution follows a familiar distributed-systems pattern: partition the state. Each shard maintains its own independent DRBG state. Shards do not share counters, keys, or reseed thresholds. They share only configuration. This trades a small amount of memory for clarity and scalability. Concurrency improves throughput, not entropy.

Determinism remains intact.

Configuration is where many security failures quietly begin. Defaults accrete. Options multiply. Behavior changes without callers noticing. Over time, the configuration surface becomes an implicit API, understood only by convention.

Here, configuration is a contract.

All configuration occurs at construction time. Options are explicit. Once initialized, the generator does not change its behavior. There is no hidden reseeding, no dynamic mutation, no silent fallback. If prediction resistance is enabled, it is enabled because the caller said so.

A generator whose security posture can change mid-flight violates basic architectural discipline. A useful way to evaluate a DRBG is to ask a simple question: if an attacker learns the internal state at time t, what can they infer? With a deterministic construction, the answer is precise. Past output remains secure if the construction provides backtracking resistance. Future output becomes predictable unless new entropy is introduced. These are not surprising conclusions; they are the expected consequences of the model.

What matters is that the model makes these consequences explicit. By avoiding hidden entropy and implicit reseeding, the generator ensures that compromise scenarios are analyzable rather than speculative. An auditor does not need to guess whether the library refreshed itself “recently.” The code tells the story plainly.

In cryptography, clarity is often mistaken for weakness. In practice, it is the opposite.

Closing Thoughts

This (opens in a new window) AES-CTR-DRBG implementation exists because certain systems demand more from randomness than opacity and convenience. They demand predictability, auditability, and explicit control. They demand that design choices surface in code rather than hide behind defaults.

If you need a general-purpose source of entropy, the standard library remains an excellent choice.

If you need deterministic cryptographic randomness whose behavior you can reason about—whose latency you can predict and whose state evolution you can audit—then the design presented here follows inevitably from that requirement. Once randomness is treated as state evolving over time, rather than as an oracle invoked on demand, many familiar shortcuts stop making sense. What remains is a system whose behavior is explicit, bounded, and knowable. Good architectures do not happen by accident. They emerge when constraints are taken seriously and followed to their logical conclusions—even when those conclusions lead to different constructions under different constraints.

This design is not theoretical; the AES-CTR-DRBG implementation described here is used as the underlying randomness source in my NanoID (opens in a new window) library, where deterministic behavior and predictable performance are equally important.

Implementation: https://github.com/sixafter/aes-ctr-drbg (opens in a new window)

“All problems in computer science can be solved by another level of indirection.” — David J. Wheeler

Identifier generation is one of those concerns that quietly disappears into the background of a system. At small scale, it is effectively free. A call to a random number generator, a string conversion, and the work is done. There is little reason to revisit it, and even less reason to question its design. This holds largely because identifier generation tends to be invisible: it sits at the edges of requests, happens quickly, and rarely shows up in profiling output. When it does, it is usually dismissed as noise.

That assumption holds only as long as the surrounding system remains small. As concurrency increases and identifier generation moves onto hot paths, what was once incidental becomes part of the system’s observable behavior. Identifiers are created everywhere: at request boundaries, inside storage layers, across services that never coordinate with one another. They are infrastructure in the most literal sense—pervasive, unavoidable, and relied upon precisely because they are assumed to be cheap. When they stop being cheap, the cost is paid everywhere.

NanoID (opens in a new window) fits neatly into this mental model. It produces compact, URL-safe identifiers with reasonable entropy and a simple interface, and in most environments it behaves exactly as expected. It is easy to adopt and difficult to misuse, which reinforces the idea that identifier generation is a solved problem. The difficulty is not with NanoID itself, but with the way its cost profile changes when it is exercised continuously and concurrently.

In Go, the most direct implementation of NanoID draws randomness from crypto/rand.Reader. From a correctness standpoint, this is beyond reproach. The operating system provides high-quality entropy, the guarantees are well understood, and the resulting identifiers are unpredictable in exactly the ways they should be. Under light use, the cost of doing so is effectively invisible, which further entrenches the assumption that there is nothing here worth examining.

Under sustained concurrency, that assumption no longer holds. Each call to crypto/rand crosses into the kernel, sources entropy, fills a buffer, and returns. Temporary buffers are allocated and discarded. None of this is problematic in isolation, but taken together and repeated at scale, it becomes the dominant cost of identifier generation. At that point, “generating an ID” is no longer a small operation. It is a collection of system calls, allocations, and coordination that happens to terminate in a string.

This is not a criticism of crypto/rand. It is doing exactly what it is designed to do. The issue is one of proximity. Entropy acquisition and identifier construction are tightly coupled, even though they serve different purposes. Correctness requires the former. Performance becomes sensitive when it occupies the hottest path of the system.

Concurrency has a way of making these relationships visible. Throughput flattens earlier than expected. Latency becomes uneven. Profilers begin to attribute meaningful time to what was assumed to be trivial. In practice, the profiler does not report time spent “generating IDs”; it reports time spent in the kernel, allocation paths, and garbage collection. Identifier generation becomes visible not because it is conceptually complex, but because it is structurally misplaced.

Separating Entropy from Generation

At that point, there are two broad directions a system can move. One is to accept the cost and design capacity around it. The other is to separate concerns that were previously implicit. The former tends to entrench friction around an otherwise unremarkable operation. The latter introduces an architectural boundary.

NanoID does not require fresh entropy for every identifier. What it requires is unpredictability. Those two properties are related, but they are not equivalent. Entropy must originate from a trusted source, but once acquired, it can be expanded safely using a cryptographically sound construction. There is no requirement to consult the operating system on every invocation, and doing so ties correctness to cost in a way that becomes increasingly visible under load.

This observation reshapes the implementation. Instead of treating identifier generation as a single indivisible operation, it becomes possible to draw a boundary between entropy acquisition and identifier construction. In practical terms, this means seeding a generator once from crypto/rand and then producing random bytes locally. The operating system remains the root of trust, but it no longer sits in the inner loop.

The Shape of the Hot Path

A stream cipher such as ChaCha20 fits this role naturally. It is designed to generate large volumes of pseudorandom output from a fixed key and nonce, and its properties are well understood. Once initialized, it produces random bytes without further system interaction. Used this way, it does not replace the operating system’s entropy source; it amortizes it.

Each generator instance is seeded once using crypto/rand and then used to satisfy multiple NanoID generations. The generator itself is not shared across callers. Sharing would introduce contention and obscure the very boundary the design is trying to establish. Instead, generator instances are treated as independent state machines that can be reused opportunistically.

This is where sync.Pool becomes useful, not as a micro-optimization, but as a way to preserve locality without introducing coordination. Generator instances are pooled so that goroutines can borrow them briefly, generate the required bytes, and return them. There is no shared mutable state and no locking around generation itself. The pool exists solely to reduce repeated setup cost when reuse is inexpensive.

Once entropy is decoupled, other sources of variability become apparent. NanoID generation requires temporary byte buffers—first for random data, then for mapping those bytes into an alphabet. These buffers are uniform in size and short-lived. Allocating and discarding them repeatedly introduces allocation pressure that is unrelated to the semantics of identifier generation. Pooling these buffers follows the same reasoning as pooling generator state: reuse when convenient, allow reclamation when not.

The same discipline applies to configuration. NanoID relies on derived values such as alphabet size, bit masks, and the number of random bytes required to generate an identifier of a given length. These values do not change per call, yet they are often computed as part of the generation path. Computing them once and fixing them at construction time pushes variability outward and keeps the inner loop small and predictable. The generation path becomes a straight-line transformation from random bytes to characters, without branching or recomputation.

At this stage, the hot path is deliberately unremarkable. There are no system calls, no locks, and no dynamic allocation beyond the final string itself. The work performed is exactly the work required to produce an identifier, and nothing else. What remains visible in benchmarks is not overhead, but the irreducible cost of string construction in Go.

The effect of these changes is straightforward to measure. Allocation counts collapse to a single allocation per identifier. Latency stabilizes. Throughput scales with available CPU until saturation. More importantly, behavior becomes predictable. Identifier generation stops appearing as a source of variance and resumes its role as infrastructure.

An implementation of the approach described here is available as an open-source Go module, along with a small command-line tool built on top of it:

• https://github.com/sixafter/nanoid (opens in a new window)
• https://github.com/sixafter/nanoid-cli (opens in a new window)

What does not change is the security model. Entropy still originates from the operating system. Generator state is seeded from a cryptographically secure source. Unpredictability is preserved. There is no attempt to weaken guarantees in exchange for speed. The improvement comes from respecting boundaries, not from relaxing them.

Identifier generation stops being trivial when it becomes infrastructure. At that point, it deserves the same architectural treatment as any other component that occupies a hot path. Once those boundaries are made explicit, the problem largely resolves itself.

At that point, the choice of entropy source becomes an explicit design decision rather than an incidental one. The NanoID implementation described here does not assume a single generator, nor does it require that entropy be sourced directly from the operating system on every invocation. Instead, it is structured to accept a well-defined pseudorandom generator that is seeded from a trusted source and then exercised locally.

For environments where throughput and steady behavior under concurrency are the primary concerns, a ChaCha20-based generator provides a practical balance. Seeded once from crypto/rand, it offers high-quality pseudorandom output with predictable performance characteristics, making it well suited for systems where identifier generation sits on a hot path and must remain invisible.

In environments where regulatory or compliance requirements apply, particularly those that mandate FIPS 140-2 (opens in a new window) validation, an AES-CTR-DRBG construction becomes the appropriate choice. In that context, the same architectural separation applies: entropy is sourced from the operating system, expanded via a standards-aligned deterministic generator, and consumed locally. The difference is not architectural, but contractual—the guarantees are defined externally rather than operationally.

The important point is that these choices do not alter the shape of the system. Whether the generator is ChaCha20-based or an AES-CTR-DRBG, the boundary between entropy acquisition and identifier construction remains intact. The cost model is stable, the hot path is predictable, and the guarantees are explicit rather than implicit.

Implementation: https://github.com/sixafter/nanoid (opens in a new window)

Originally posted on BUSRide Magazine (opens in a new window) and subsequently referenced by Vix Technology (opens in a new window) , here are my responses featured alongside other industry commentators. It shouldn’t be a surprise to anyone that DevOps and software engineering displine are fundamental when designing and constructing parallel and distributed systems at a quality level commensurate with that of commercial software.

In what ways does your platform enable a reliable, dependable revenue stream throughout its useable life?

Building on three generations of advanced automated fare collection (AFC) and payments solutions, the Vix solution provides an advanced platform supporting account-based ticketing, card-based ticketing, open payments and closed loop features in a single, highly configurable product. Sophisticated financial management tools, including transit clearing house capabilities, are also core capabilities, all of which can be deployed “on premise” within customer data centers or “as a Service” in a public cloud.

Our solution enables a reliable, dependable revenue stream throughout its usable life because, for Vix, the most important aspect of public transportation is a better journey for our customers. Our relentless focus on building a flexible, scalable, and reliable AFC and payments platform is fundamental to delivering this vision for our customers and for ensuring dependable revenue streams for agencies.

What best practices can agencies use to decrease fare system deployment times, to maximize benefits and minimize revenue disruption?

At Vix, there are many lenses through which we measure how we deliver value to our customers: we recognize the need to reduce build, deployment, operating, and maintenance costs along one dimension whilst ensuring a repeatable and secure quality level commensurate with that of a world-class commercial solution along another dimension.

Full automation for forming, deploying, and upgrading all operational environments and the Vix solution is critically important in reducing overall costs, reducing the risk of human error, increasing operational efficiency, ensuring consistent quality and enable measurable compliance with security practices and governance. To this end, the Vix solution focuses on the best practice of DevOps automation as a core engineering practice across three phases to align with design, build, operations, and maintenance in support of continuous improvement.

Our investment in this area has resulted in vastly reduced build and deployment times. Our relentless focus in this area allows us to minimize costs for systems operations, security operations and disaster recovery scenarios while greatly reducing the timelines for our solution engagement schedules, resulting in a direct savings for our customers without the need to compromise their revenue protection and business continuity objectives.

How does a flexible, open architecture lend itself to increased revenue over time?

An open architecture lends itself to increased revenue over time because the inherent standards-based design patterns and technologies fosters a wide array of integration scenarios within the surrounding and transit payments ecosystem. The APIs that come with an open architecture allow for a range of integrations to be supported as part of a fare collection system, allowing agencies to, for example, partner with coffee shops or other retail outlets to offer deals for transit riders. This provides another revenue stream for the transit agencies, as well as the partner retailers, etc.

However, simply supporting an open architecture isn’t enough. Software and hardware engineering teams must possess the discipline and rigor required to produce quality, scalable, and flexible architectural patterns to support not only revenue streams that we can think of but, equally as important, with forward looking attention to market trends.

How can agencies utilize improved data collection and analysis to leverage new revenue streams?

To begin to utilize improved data collection and analysis to leverage new revenue streams, agencies must first make it a priority. Analytics and the field of data science are now foundational to nearly all industries - so much so that data itself has become an important strategic and competitive asset. This data represents a rather rich taxonomy for enabling a variety of revenue opportunities for agencies to improve a rider’s experience.

At Vix, our focus on a better customer journey shapes how we support agencies in the areas of analytics. The areas of analytics that are important to us, and thus to our solution, are predictive journey analytics, predictive maintenance analytics, fraud prevention, real-time passenger information, capacity and pricing optimization, customer analytics and loyalty marketing, intelligent transportation system (ITS) optimization and the co-occurrence of information from each of these areas to model how to improve the overall transit experience.

Originally posted on BUSRide Magazine (opens in a new window) .