“If you think cryptography will solve your problem, then you don’t understand cryptography and you don’t understand your problem.” — Roger Needham

A few colleagues and I recently revisited a familiar debate. In most data systems, identifier generation is treated as undifferentiated infrastructure—one library, one default, applied everywhere. The implicit assumption is that a sufficiently random identifier is a sufficient answer to whatever question the system happens to ask of it; that the differences between identifier classes—surrogate keys, content fingerprints, anonymized handles, span identifiers, request correlations—are differences of usage rather than differences of construction. That assumption is convenient. It survives precisely until the system encounters constraints that aren’t engineering constraints.

For the past several years, I’ve designed and currently operate a clinical research data pipeline serving multiple concurrent Institutional Review Board (opens in a new window) (IRB)-approved studies against Electronic Health Record (opens in a new window) (EHR)-sourced data at production scale. Though, implementation specifics are abstracted from this post, I want to highlight the architectural patterns. In human-subjects research, identifier construction stops being a performance choice and becomes a regulatory and ethical primitive.

Why? Because the constraints are different from those that arise in commercial systems. Re-identification resistance is enforceable under the Health Insurance Portability and Accountability Act (opens in a new window) (HIPAA) and the Common Rule (opens in a new window) . Per-study isolation is an IRB requirement. Provenance MUST (opens in a new window) survive transformation but MUST NOT enable cross-study linkage. Workflows that depend on the operator’s inability to reverse a transformation can’t be retrofitted onto a single global identifier scheme; they have to be built into the construction itself. None of these properties emerge from a generic randomness primitive applied uniformly to every identifier the pipeline produces.

So let me state the pattern plainly. A real human-research pipeline needs at least three distinct identifier constructions. Each answers a different question. Each lives at a different boundary. Treating identifier generation as a single problem with a single answer breaks one architectural pillar in service of another. The failure mode isn’t that any single construction is wrong, but that no single construction is right for every boundary the pipeline must respect.

Three Pillars, Three Constructions

A human-subjects research pipeline must answer three questions of every row that flows through it. The questions are independent and every row is interrogated by the following pillars:

1. Fidelity. Is this row the same as before, or has it changed? A pipeline that can’t answer this can’t deduplicate, can’t detect drift, and can’t reason about whether a transformation preserved meaning. Fidelity is the substrate on which idempotency, change detection, and lineage are built. Without it, every other property the pipeline claims to provide is conjectural.

2. Governance. Can this row be re-linked to a person, or to the same person across studies? In a research context, the answer must be no, and the answer must be enforceable rather than promised. Policy is necessary but not sufficient; if the construction permits re-linkage, then policy is the only thing preventing it, and policy can be circumvented, misconfigured, or socially engineered. Governance asks whether the construction itself denies the operations that policy forbids.

3. Provenance. Which pipeline run produced this row, from which source, on which day, by which transformation? Provenance is the operational substrate of accountability. It allows any finding to be traced back to its derivation, any anomaly to be localized to a stage and a run, any audit to reconstruct what happened without depending on the cooperation of the system that produced the result.

Each question is answered by a different cryptographic construction at a different boundary:

• Content-addressed SHA-256 (opens in a new window) hashing answers fidelity at the ingest boundary.
• Per-tenant keyed HMAC (opens in a new window) hashing answers governance at the publication boundary.
• CSPRNG (opens in a new window) -backed identifier generation answers provenance at the operational layer.

The constructions are not interchangeable. Using any one of them outside its intended boundary either fails the question its boundary is supposed to answer or undermines a pillar that some other boundary was responsible for. Let me walk each in turn, identify the constraint it answers, and explain why the substitution of a generic identifier scheme is an architectural mistake rather than a minor inefficiency.

Fidelity at the Ingest Boundary

At the ingest boundary, the pipeline receives raw data from a source system that doesn’t coordinate with the pipeline about identity. Files arrive on a schedule, are written to a lake in raw form, and become the substrate from which everything downstream is derived. The question fidelity asks of every row at this boundary is whether it’s the same row that arrived yesterday, last week, or in any prior incremental load. If the pipeline can’t answer that question cheaply and unambiguously, then:

• Deduplication becomes guesswork.
• Idempotency becomes a hope.
• The lake’s role as the recovery anchor for the entire pipeline is compromised.

The construction that answers this question is a SHA-256 hash computed over the row’s sorted business columns at the moment of lake ingestion. Simply put, “business columns” are all columns except for metadata columns that were added by the pipeline itself. The hash is written into a metadata column that is then carried unchanged through every downstream stage. It MUST NOT be recomputed at the warehouse boundary, must not be recomputed after foreign-key re-keying at segmentation, and must not be recomputed under any condition the pipeline operator may otherwise consider reasonable. The construction is content-addressed and immutable.

The non-randomness of this identifier is the point. Two rows with identical business-column content produce identical hashes. Two rows with any meaningful difference produce different hashes with overwhelming probability. The hash is unique by virtue of its content, not by virtue of its randomness, and that property is what makes change detection tractable across overlapping daily exports without requiring the source system to participate in a coordination protocol it was never designed to support. A randomized identifier would be wrong here in the strongest sense: it would foreclose exactly the property the boundary needs.

The same property that enables change detection also serves as a duplicate pre-filter. Rows that share identical business-column values are duplicates by definition, and the hash exposes that fact at the moment of ingestion rather than deferring it to a downstream reconciliation step. Duplicate detection becomes a comparison of fixed-width hashes rather than a comparison of arbitrarily wide row contents, and the cost of identifying duplicates collapses to the cost of comparing two values. The work the pipeline would otherwise have to do further downstream is done once, at the boundary, by the construction itself.

The hash also serves as the first link in the lineage chain that fidelity, governance, and provenance all eventually depend upon. Any row anywhere in the pipeline can be traced back to its source by following the hash. Any divergence between a stage’s input and its output can be diagnosed by comparing hashes. Any claim that two rows in different marts derive from the same source row is verifiable without consulting either source or mart, because the hash is the same in both places. The hash is the substrate; everything else is built on top of it.

A subtlety worth naming is that the hash construction itself sits on a hot path. At ingest scale, tens of millions of rows pass through the pipeline on every full load, and the hash is computed for every one of them. The disciplines that apply to identifier generation under sustained concurrency apply equally here: allocation behavior MUST be controlled, primitive selection MUST avoid implicit coordination, and the cost of the construction MUST NOT become the dominant cost of ingestion. The arguments I developed in my earlier post on NanoID hot-path optimization generalize directly to hash construction in pipelines of this kind. The construction is different; the discipline is the same.

Governance at the Publication Boundary

The boundary between the warehouse and the data marts is where the pipeline transitions from internal computation to external publication. Inside the warehouse, identity is managed by the pipeline operator, and the SHA-256 hash from the ingest boundary serves every internal purpose well. Outside the warehouse, identity becomes a research artifact, and a fundamentally different requirement asserts itself: the identifier that reaches the mart MUST NOT enable re-identification of the underlying subject, and it MUST NOT (opens in a new window) enable cross-mart linkage of the same subject across separate research projects.

Both requirements are properties of the construction, not properties of the data. A SHA-256 hash carried untouched into a mart would satisfy neither. Why? The hash is deterministic; identical inputs produce identical outputs; an attacker with sufficient knowledge of the source schema could attempt to reconstruct the input space and brute-force the mapping. More immediately, the same warehouse row would carry the same hash into every mart, and any researcher with access to two marts could link rows trivially by comparing hashes. The construction permits exactly the operations that governance forbids.

Stay with me—the answer here is a small but architecturally decisive change. The construction that answers governance at this boundary is HMAC-SHA256, keyed with a per-mart secret stored in a Hardware Security Module (opens in a new window) (HSM)-backed secrets manager and never exposed to the pipeline operator. At the segmentation stage that produces each mart, the warehouse-side SHA-256 hash is replaced by HMAC-SHA256(warehouse-hash, per-mart-secret). The result is a new identifier that is:

• Deterministic with respect to the same inputs.
• Cryptographically opaque without the secret.
• Distinct across marts, because the secret is distinct across marts.

This construction has a property that’s easy to overlook and architecturally decisive: the HMAC operates on the SHA-256 hash, not on the raw business columns. An attacker who somehow obtained a per-mart secret would not face the relatively constrained problem of inverting the HMAC over a known schema of patient identifiers; they would face the problem of inverting it over the full output space of SHA-256, which is 2^256 possible inputs. A dictionary attack against patient identifiers, plausible in principle against a single-layer keyed hash, is eliminated by construction. The two-layer design—content hash first, keyed hash of the hash second—is what makes the cryptographic claim survive contact with a realistic threat model.

The architectural consequence is that the publication boundary becomes a place where identity is not merely transformed but severed. The original SHA-256 hash never reaches the mart. The mart receives the HMAC, and only the HMAC, as its identifier for that row. The pipeline operator, who has access to the warehouse and to the segmentation code, can’t reverse the transformation, because the operator doesn’t have the per-mart secret. The secret lives in an HSM-backed store that the operator can use but can’t extract. The same warehouse row, written into three different marts, carries three different identifiers, and no party—including the pipeline operator—can establish that the three identifiers refer to the same subject without simultaneous access to all three secrets and the cooperation of the secrets manager.

This is what makes the honest-broker workflow cryptographically tenable rather than merely operationally promised. In human-subjects research, there are workflows where the operator’s inability to reverse the transformation is structurally required: the team that operates the pipeline MUST be able to publish the data, but MUST NOT be able to re-identify the subjects whose data it published. Under a single-layer construction, the operator can always be compelled, suborned, or compromised into reversing the mapping. Under the two-layer construction described here, the operator simply doesn’t have the capability to reverse, and therefore can’t be compelled to exercise it. The construction enforces what policy could only request.

The bridge to my earlier work on AES-CTR-DRBG is direct. In that post I argued that deterministic cryptographic constructions belong in environments where reproducibility, auditability, and explicit state evolution are required. The HMAC-at-publication pattern is the same disposition applied to identity rather than to byte streams. The construction is deterministic with respect to its inputs; the state that determines its output evolves only when the per-mart secret is rotated; the auditor can reason about what the construction produces without having to reverse-engineer the library that implements it. Configuration is a contract. Boundaries are explicit. The cryptographic posture doesn’t change mid-flight.

A randomized identifier would be wrong here in a different way than at the ingest boundary, but no less wrong. A randomized identifier is non-deterministic; the same row processed twice would produce different identifiers; the mart’s referential integrity would collapse on the first re-run. What the boundary needs is deterministic-but-cryptographically-opaque, and the only construction that delivers both properties is a keyed hash.

Provenance at the Operational Layer

The constructions described so far handle the headline identity story—the identity of the rows themselves. They’re computed once per row and carried thereafter. They’re also, by a substantial margin, not the identifiers the pipeline generates most often.

A pipeline of any operational seriousness produces identifiers continuously for things that aren’t rows. Pipeline runs need identifiers. Stage executions need identifiers. Batch tasks under a thread pool need identifiers. Observability spans, if the pipeline exports traces, need identifiers. Synthetic sentinel rows injected at segmentation need identifiers distinguishable from real source identifiers. Project allocations need identifiers that survive across systems. Honest-broker requests need identifiers that the requesting system, the secrets manager, and the provisioning workflow can all reference unambiguously. Quality diagnostic runs need identifiers for the audit trail that links them to the pipeline run that triggered them.

These identifiers have a constraint regime distinct from anything the row-level constructions answer:

1. They are allocated at high concurrency, often from many worker threads or pods simultaneously, with no central allocator available.
2. They MUST be collision-resistant by virtue of their construction, because no coordinator exists to detect and resolve a collision after the fact.
3. They MUST be URL-safe and human-tractable, because they will appear in logs, dashboards, audit trails, and the bodies of cross-system requests.
4. They MUST be allocation-cheap, because the pipeline generates them at orders of magnitude higher volume than it generates SHA-256 row hashes or per-mart HMACs.
5. They MUST NOT leak temporal patterns or sequence structure that an observer could exploit to infer pipeline behavior or correlate identifiers from independent runs.

In raw call volume, this class of identifier dominates the SHA-256 and HMAC constructions by orders of magnitude. The hash is computed once per source row at ingest. The HMAC is computed once per row per mart at segmentation. The operational identifiers are generated continuously: on every notebook, on every span, on every parallel task, on every cross-system request. If the construction here is wrong, the cost is paid everywhere the pipeline does any work at all.

This is the design space my NanoID library (opens in a new window) was built for, and the disposition that produced it is the disposition that earlier governed its randomness backends. The construction at the call site is identical regardless of the workload—a single function call returning a compact, URL-safe identifier—but the randomness source behind it is configurable, and the choice of source is made per identifier class rather than per call site. The architectural payoff is that the cryptographic posture of the identifier is determined at construction time and survives every downstream invocation without requiring the call site to be aware of it.

For the highest-volume operational identifiers—span identifiers, batch identifiers, stage execution identifiers, the thousands of small handles that a pipeline produces in the course of doing its work—the appropriate backend is the ChaCha20-based PRNG I developed in an earlier post. It’s designed for predictable behavior under sustained concurrency, with allocation discipline on the hot path and no shared coordination between worker threads. It doesn’t target any particular compliance regime, because none applies at this layer; what applies is the constraint that identifier generation MUST remain invisible in profiling output even at the volumes the pipeline reaches.

For the identifiers that participate in audit, IRB, or compliance trails—project allocations, honest-broker request identifiers, anything that ends up in a Key Vault audit log or a regulatory submission—the appropriate backend is the AES-CTR-DRBG construction I developed in the post before that. It targets FIPS 140-2 (opens in a new window) -aligned cryptographic randomness with deterministic state evolution, allowing the identifiers it produces to be traceable, auditable, and defensible under scrutiny in ways that a general-purpose PRNG is not. The construction is the same NanoID library, configured with a different randomness source via the same WithRandReader option, called identically at the call site.

The two backends are not competing; they answer two different constraints within the same pipeline, on the same library surface, decided once at construction time. The same disposition that governs SHA-256 and HMAC at their boundaries—the construction is chosen for the boundary, not for the codebase—governs NanoID at the operational layer.

One Library, Three Backends

The library design that makes this tractable isn’t a convenience feature. It’s an architectural commitment that allows per-boundary construction choice to coexist with a single call-site surface. NanoID exposes a WithRandReader option that accepts any io.Reader as its randomness source. The choice of source is made once, at generator construction time, and survives every subsequent identifier the generator produces. The call site doesn’t change; the cryptographic posture does.

This mirrors the architectural move SHA-256 and HMAC are making at their respective boundaries. Both belong to the SHA-2 family. Both compose with the same underlying hash primitive. The difference between them is whether a per-tenant key is involved, and whether the construction can therefore enforce the cryptographic break that publication requires. The primitive is shared; the construction is chosen for what the boundary MUST guarantee. Same disposition, same shape, different layer.

The negation is instructive. A pipeline that uses uuid_generate_v4() everywhere—or NanoID with a single hardcoded randomness source applied to every identifier it produces—collapses distinctions that the architecture is doing real work to preserve. Surrogate keys for fact tables receive the same construction as project identifiers feeding IRB submissions, which receive the same construction as observability spans. Each is correct for some workload and wrong for at least one other. The pipeline that adopts a single global construction hasn’t simplified its identifier story; it has merely declined to make decisions that its boundaries will eventually force it to revisit.

The pipeline that adopts the layered model described here makes those decisions once, at the boundary where each one belongs, and then stops thinking about them.

• The hash is content-addressed at ingest.
• The HMAC is per-tenant keyed at publication.
• The NanoID is randomly allocated at the operational layer, with the randomness backend chosen for the identifier class.

Each construction is correct for its boundary. Each construction is composable with the others, because the boundaries are explicit and the constructions don’t interfere. The architecture isn’t more complex than the alternative; it’s more honest about the complexity that was already present.

Wrapping it all up…

The earlier articles in this sequence each followed a single constraint to its conclusion. My 2024 article on NanoID followed the hot-path discipline that emerges when identifier generation moves from incidental utility to infrastructure. My 2025 article on AES-CTR-DRBG followed the requirements of compliance, auditability, and explicit state evolution. My article on the ChaCha20-based PRNG followed the constraints of portability, concurrency, and operational simplicity. In each case, a single set of requirements produced a single construction.

This article inverts that move. Given a domain—human-subjects research—that imposes multiple constraint regimes simultaneously, the architectural response isn’t a single construction but a layered one. Fidelity, governance, and provenance aren’t competing concerns to be balanced against one another. They’re independent questions answered by independent constructions at independent boundaries, and the pipeline that respects the independence of the questions can satisfy all three without trading any of them off against the others.

Make no mistake; the constructions themselves are unremarkable. SHA-256 has been a standard primitive for two decades. HMAC has been a standard primitive for longer. CSPRNG-backed identifier generation is a solved problem with well-understood implementations. The architectural work isn’t in the choice of primitives. It’s in the recognition that each primitive is correct only for the boundary that actually needs the property it provides, and in the discipline of refusing to substitute one for another where the substitution would be easier but wrong.

Good architecture emerges when constraints are REQUIRED (opens in a new window) and followed to their conclusions, even when those conclusions lead to different constructions under different constraints. The earlier articles argued this for randomness. The pipeline architecture described here argues it for identity. The answer is the same in both cases: make the boundaries explicit, choose the construction the boundary needs, and let the disposition do the rest.

My next post will take a step back and explore the data analytics and insights pipeline from an architectural perspective.

Implementation: https://github.com/sixafter/nanoid (opens in a new window)

“A system is not determined by the algorithms it uses, but by the assumptions it makes.”
— Leslie Lamport

Randomness appears deceptively uniform in most software systems. A call is made, bytes are returned, and execution proceeds. The details of how those bytes were produced rarely matter—until they do. As with many foundational components, the assumptions baked into a randomness source tend to surface only when systems scale, diversify, or are subjected to constraints they were not originally designed to accommodate.

In the previous article , I explored deterministic cryptographic randomness under compliance and auditability constraints, where explicit state evolution and reproducibility were not optional. That work followed inevitably from environments shaped by regulation and forensic accountability. This article explores a different design space: one where the constraints are imposed not by certification regimes, but by software portability, concurrency, and operational simplicity.

The result is a ChaCha20-based (opens in a new window) pseudorandom number generator (PRNG) designed for predictable behavior in real systems, rather than alignment with formal validation programs.

What distinguishes this space from compliance-driven designs is not weaker security assumptions, but different priorities. Software systems are often judged not by formal proofs or certifications, but by how they behave under sustained load, across diverse platforms, and in the presence of concurrency. In those environments, simplicity is not aesthetic—it is operational. Every implicit dependency, hidden allocation, or global coordination point becomes a liability over time.

Constraints Shape Constructions

Cryptographic primitives are often discussed as interchangeable building blocks. In practice, the surrounding constraints determine which properties matter and which can be safely ignored. A construction optimized for auditability and deterministic replay will look very different from one optimized for simplicity, portability, and high-throughput concurrent use.

In software-heavy environments—particularly those spanning multiple platforms or deployed as libraries rather than systems—certain requirements surface repeatedly. The randomness source must behave predictably under concurrency. It must avoid surprising allocation behavior. It must scale without coordination bottlenecks. And it must do so without pulling in complex dependencies or imposing policy decisions on its callers.

These are not compliance constraints. They are engineering constraints.

Engineering constraints tend to accumulate quietly. A library that is easy to use in isolation may behave very differently when embedded in larger systems, reused across services, or exercised under high concurrency. Over time, small assumptions compound: about allocation behavior, about shared state, about initialization cost. What begins as a convenient abstraction can become a systemic bottleneck. Designs that survive this environment are rarely the most general ones; they are the ones that make their tradeoffs explicit early.

ChaCha20 fits naturally into this space. As a stream cipher designed for efficient software implementation, it offers strong cryptographic properties without reliance on specialized hardware instructions or platform-specific optimizations. That makes it a practical foundation for a general-purpose PRNG intended to behave consistently across environments.

State, Not Entropy, Does the Work

As with any PRNG, the core of the design is not entropy acquisition but state evolution. Entropy enters the system at initialization. From that point forward, output is a function of internal state advancing according to a defined algorithm.

This distinction matters. By treating entropy acquisition as a boundary rather than a continuous process, the generator remains honest about its dependencies. The operating system provides unpredictability at startup. The PRNG provides expansion, isolation, and repeatable behavior thereafter.

ChaCha20’s design reinforces this model. Once keyed and initialized, the cipher produces a stream of pseudorandom bytes derived entirely from its internal state. There are no conditional branches based on environmental input, no opportunistic refreshes, and no hidden coordination with external sources. State advances monotonically. Output follows deterministically.

Thinking in terms of state rather than entropy also clarifies responsibility. Once initialization completes, the generator no longer depends on the operating system or the environment for correctness. The burden shifts entirely to the implementation and its callers. That boundary is important: it allows failures, assumptions, and limitations to be localized rather than diffused across layers. When something goes wrong, there is no ambiguity about where to look.

This makes the generator easier to reason about, test, and integrate. The absence of implicit behavior is not a limitation; it is a deliberate choice that keeps responsibility visible.

Concurrency Without Contention

Concurrency tends to expose the weaknesses of otherwise elegant designs. Shared mutable state invites contention. Global locks introduce unpredictable latency. Per-call initialization imposes overhead that scales poorly as systems grow.

The ChaCha20 PRNG approaches concurrency as a distribution problem rather than a synchronization problem. Independent generator instances maintain independent state. A pooling strategy amortizes initialization costs while avoiding shared counters or global coordination. Each request draws from a generator whose behavior is local, predictable, and isolated from others.

This approach mirrors patterns commonly used in distributed systems: partition state, avoid global serialization, and accept modest duplication in exchange for clarity and scalability.

In practice, this means resisting the temptation to centralize randomness behind a single shared construct. While such designs appear simpler at first, they introduce hidden coupling that surfaces under load. Contention becomes observable latency. Synchronization becomes an availability concern. By allowing each generator instance to evolve independently, the system preserves the same semantic behavior regardless of how many consumers are active.

Runtime Semantics and Predictable Behavior

Allocation behavior is treated as a semantic concern rather than a micro-optimization. A PRNG that allocates unpredictably during output generation cedes part of its execution model to the runtime.

The effects of these choices are observable in practice. Small reads primarily reflect fixed overhead. Larger reads scale linearly with the amount of data processed, as expected from a stream cipher advancing its internal counter. Under concurrent workloads, latency remains stable because requests are serviced independently rather than serialized behind shared state.

What matters is not the absolute numbers, but their predictability. The behavior matches the design model, rather than fluctuating with runtime conditions.

This predictability has secondary effects that are easy to underestimate. Monitoring becomes more meaningful when latency distributions are stable. Capacity planning becomes tractable when throughput degrades linearly rather than catastrophically. Perhaps most importantly, anomalies become visible. When behavior deviates from expectation, it signals a real change in conditions rather than noise introduced by the runtime.

Failure Modes and Explicit Boundaries

Boundaries are explicit. Initialization depends on a trusted entropy source. Once initialized, the generator’s behavior is deterministic and bounded by construction. Errors are surfaced rather than hidden.

Surfacing errors is not merely a defensive choice; it is an architectural one. Silent degradation may preserve uptime in the short term, but it undermines trust in the system over time. By making failure explicit, the generator avoids the pretense of correctness under conditions it was not designed to handle. This makes integration more demanding, but also more honest.

This approach does not attempt to cover every possible threat model. It does not claim resistance to state compromise beyond what the underlying primitive provides. It does not attempt to retrofit compliance semantics onto a software-oriented design.

Closing Thoughts

This ChaCha20-based PRNG (opens in a new window) is used as the randomness source in other systems, including my NanoID (opens in a new window) implementation, where predictable behavior, low overhead, and concurrency safety are practical requirements rather than abstract goals.

Randomness is not a single problem. Different environments impose different constraints, and those constraints shape the constructions that make sense within them.

This ChaCha20-based PRNG exists because certain systems demand a randomness source that behaves predictably under concurrency, avoids unnecessary complexity, and makes its assumptions explicit. It treats randomness as evolving state rather than as an opaque service call.

The tension between generality and specificity is unavoidable in systems design. Libraries that attempt to serve every use case often end up serving none particularly well. By contrast, designs that accept their constraints openly tend to age more gracefully. They are easier to reason about, easier to integrate, and harder to misuse.

Implementation: https://github.com/sixafter/prng-chacha (opens in a new window)

“The purpose of abstraction is not to be vague, but to create a new semantic level in which one can be absolutely precise.” — Edsger W. Dijkstra

Randomness occupies an unusual position in software systems. We rely on it for security, for fairness, for simulation, and yet we rarely scrutinize its behavior with the same rigor we apply to storage engines or replication protocols. We accept entropy as something that simply happens, mediated by an operating system call or a library abstraction, and move on.

That approach works—until it doesn’t.

When I began working on systems that required both cryptographic correctness and operational predictability, the implicit assumptions around randomness became a liability. Latency mattered. Allocation behavior mattered. Reproducibility mattered. And in certain environments, auditability mattered most of all. The standard tools did not fail outright, but they failed quietly, by obscuring behavior that ought to be explicit.

At its core, the problem is not randomness itself, but how cryptographic state evolves over time—and whether that evolution remains visible, bounded, and predictable.

This article is about addressing that gap.

Determinism Is Not the Enemy

In distributed systems, we accept a fundamental axiom: there is no way to guarantee complete knowledge of global state at any given moment. We design for tolerance, convergence, and recovery, not for omniscience; ref Parallel and Distributed System Design ). Randomness, however, often escapes this discipline. It is treated as an oracle—invoked, trusted, and forgotten.

Cryptographic systems take a different view. They do not reject determinism; they constrain it. Given a fixed internal state and a fixed construction, output must follow. This is not a weakness but a requirement. It is precisely what allows us to reason about forward security, backtracking resistance, and compromise scenarios.

AES-CTR-DRBG sits squarely in this space. It provides a deterministic construction based on a well-understood primitive, AES, and defines explicit rules for how internal state evolves over time. Once you accept that premise, several consequences follow naturally. Hidden entropy sources become suspect. Implicit reseeding becomes problematic. Mutable configuration becomes dangerous.

The design space narrows considerably.

Deterministic Constructions and Cryptographic Honesty

Cryptography is often discussed in terms of strength, but strength alone is not sufficient. A system can employ strong primitives and still behave in ways that are difficult to reason about. When that happens, security becomes an emergent property rather than an explicit one.

Deterministic constructions reject that ambiguity.

An AES-CTR-DRBG (opens in a new window) does not attempt to surprise the reader. Given a key, a counter, and a defined update function, the output follows directly. There is no hidden mixing, no opportunistic entropy acquisition, and no conditional behavior based on environmental state. This predictability is not merely convenient; it is foundational. It allows one to reason about compromise without first reverse-engineering library behavior.

Once determinism is accepted as a requirement, architectural freedoms disappear. You can no longer “improve” security by quietly pulling in entropy. You can no longer treat reseeding as an implementation detail. You must decide, explicitly, when state changes and why.

This is not a limitation of AES-CTR-DRBG. It is a discipline imposed by cryptography itself.

These constraints do not arise in isolation. They emerge most clearly in regulated, multi-tenant, and distributed systems, where randomness is no longer an incidental utility but a component subject to scrutiny. In such environments, deterministic and auditable cryptographic randomness becomes a requirement rather than a preference. Reproducibility matters for forensic analysis. Isolation matters for per-tenant security. Predictable resource behavior matters when concurrency and throughput are no longer theoretical concerns.

The standard random primitives in Go do not address this space. They make no guarantees around deterministic output, explicit key management, or resource predictability under concurrency, nor do they target alignment with NIST SP 800-90A (opens in a new window) or FIPS 140-2 (opens in a new window) requirements. Third-party alternatives often complicate matters further, introducing additional dependencies or obscuring allocation behavior and latency characteristics behind opaque abstractions.

At scale, these gaps become operational risks. System PRNGs (opens in a new window) do not ensure reproducibility. Key lifecycle management and tenant isolation are left to the caller. Failure modes relevant to compliance and auditability remain implicit, even as entropy sources degrade or cryptographic assumptions are violated. What is missing is not another source of randomness, but a generator whose behavior remains explicit, bounded, and inspectable over time.

These constraints define the space for a deterministic random bit generator based on AES (opens in a new window) in counter mode ( CTR (opens in a new window) ), as specified in NIST SP 800-90A and aligned with FIPS 140-2. The implementation discussed here relies solely on Go’s standard library cryptography and targets allocation-free output and low, predictable latency, making it suitable for high-throughput and regulated environments where correctness and observability are inseparable.

Entropy, State, and the Shape of Time

Many practical failures in cryptographic systems arise not from broken primitives, but from blurred boundaries. Entropy acquisition is a boundary. Deterministic generation is another. When the two are conflated, responsibility becomes unclear.

Operating systems are good at collecting entropy. Applications are good at applying policy. A DRBG sits between the two, which is why many implementations quietly collapse the distinction.

I chose not to.

In this design, entropy enters the system at a clearly defined point and nowhere else. The generator consumes seed material and expands it deterministically. If reseeding occurs, it occurs because the caller has decided that new entropy is available and appropriate. This makes the system honest about its dependencies. It also makes it auditable. One can trace exactly when and how entropy influences output, which is a prerequisite for meaningful analysis.

This mirrors a lesson learned repeatedly in distributed systems: implicit coordination is coordination nonetheless. If you depend on entropy, say so.

It is helpful to visualize the generator not as a black box, but as a state machine evolving over time.

Imagine a simple diagram with three persistent elements: Key, Counter (V), and Limits. At initialization, these are set from seed material and configuration. Each request for output advances the counter deterministically, producing a block of output via AES encryption. The key remains fixed until a reseed event occurs. Limits monotonically decrease as output is generated.

Time, in this model, does not pass in seconds. It passes in blocks generated.

A reseed event resets the key and counter and restores limits. Absent a reseed, state evolution is linear, irreversible, and predictable. There are no side paths, no hidden refreshes, and no conditional jumps based on runtime conditions. The diagram contains no feedback loops other than the explicit reseed boundary.

This mental model is intentionally simple. It allows one to reason about compromise, exhaustion, and recovery without hand-waving.

Reseeding is often presented as a binary feature: enabled or disabled. In practice, it is a temporal question. How long may a generator safely operate on a given internal state? How much output may be derived before assumptions begin to erode?

FIPS-aligned guidance does not answer these questions abstractly; it answers them in terms of limits. Maximum bytes per key. Maximum requests between reseeds. Defined update functions. These limits exist not because AES is fragile, but because systems operate over time, and time introduces exposure.

In a deterministic generator, time is measured not in seconds but in output. Each block generated advances the internal counter. Each advancement is irreversible. By enforcing explicit limits, the generator treats time as a first-class dimension rather than an afterthought.

This perspective aligns closely with how we reason about versioning and state evolution in distributed systems. State does not merely exist; it progresses. If you do not bound that progression, you eventually lose the ability to reason about it.

Prediction resistance is frequently discussed as an absolute good. In reality, it is a trade. Enabling it increases resilience against certain compromise scenarios, but it also introduces new operational dependencies. Reseeding requires entropy. Entropy requires coordination with the environment. Coordination introduces latency and failure modes.

Rather than treating prediction resistance as a default, this design treats it as a conscious decision. When enabled, reseeding behavior is strict and enforced. When disabled, the generator behaves as a pure deterministic expander. Neither mode is inherently superior; each corresponds to a different threat model.

Cryptographic systems fail when they attempt to serve all threat models simultaneously. By forcing the caller to choose, the system avoids pretending that one size fits all.

Runtime Semantics and Predictability

Garbage collection is often discussed in terms of performance, but in latency-sensitive systems it also becomes a semantic concern. Allocation introduces nondeterminism: not in output, but in timing. When a generator allocates during its hot path, it cedes control of latency to the runtime.

The effects of these choices are observable in practice. Under serial workloads, small reads complete in tens of nanoseconds, largely reflecting fixed overhead rather than per-byte cost. Larger reads exhibit higher absolute latency, not because of inefficiency, but because AES-CTR scales linearly with the number of blocks processed. Encrypting a few blocks versus a few hundred necessarily occupies different regions of the CPU pipeline, even when the underlying primitive is highly optimized. What matters is that this behavior is predictable and stable under load, matching expectations derived from the algorithm and the memory hierarchy rather than fluctuating with runtime conditions.

Achieving zero allocations in the output path was, therefore, non-negotiable; read “ MUST (opens in a new window) ”.

What matters here is not raw speed, but control. A system that allocates implicitly hands part of its execution semantics to the runtime. In many contexts that is acceptable; in some, it is not. When randomness participates in protocol boundaries, scheduling decisions, or security-sensitive operations, latency variance becomes observable behavior. Treating allocation as a semantic choice forces the design to acknowledge that reality. Once allocation is removed from the generation path, timing becomes a property of the algorithm itself rather than a side effect of memory management. That shift restores a degree of predictability that is otherwise difficult to recover.

The decision to make this DRBG allocation-free during generation was not an optimization exercise; it was a correctness constraint. Once generation begins, no heap allocations occur. The caller owns the buffers. The internal state resides in fixed-size structures. The runtime remains uninvolved. Latency variance collapses. Throughput becomes predictable. The generator behaves the same under load as it does in isolation.

Concurrency often exposes design shortcuts. Shared mutable state invites contention; global locks invite collapse. In the context of deterministic randomness, concurrency introduces an additional risk: accidental nondeterminism caused by interleaving.

The solution follows a familiar distributed-systems pattern: partition the state. Each shard maintains its own independent DRBG state. Shards do not share counters, keys, or reseed thresholds. They share only configuration. This trades a small amount of memory for clarity and scalability. Concurrency improves throughput, not entropy.

Determinism remains intact.

Configuration is where many security failures quietly begin. Defaults accrete. Options multiply. Behavior changes without callers noticing. Over time, the configuration surface becomes an implicit API, understood only by convention.

Here, configuration is a contract.

All configuration occurs at construction time. Options are explicit. Once initialized, the generator does not change its behavior. There is no hidden reseeding, no dynamic mutation, no silent fallback. If prediction resistance is enabled, it is enabled because the caller said so.

A generator whose security posture can change mid-flight violates basic architectural discipline. A useful way to evaluate a DRBG is to ask a simple question: if an attacker learns the internal state at time t, what can they infer? With a deterministic construction, the answer is precise. Past output remains secure if the construction provides backtracking resistance. Future output becomes predictable unless new entropy is introduced. These are not surprising conclusions; they are the expected consequences of the model.

What matters is that the model makes these consequences explicit. By avoiding hidden entropy and implicit reseeding, the generator ensures that compromise scenarios are analyzable rather than speculative. An auditor does not need to guess whether the library refreshed itself “recently.” The code tells the story plainly.

In cryptography, clarity is often mistaken for weakness. In practice, it is the opposite.

Closing Thoughts

This (opens in a new window) AES-CTR-DRBG implementation exists because certain systems demand more from randomness than opacity and convenience. They demand predictability, auditability, and explicit control. They demand that design choices surface in code rather than hide behind defaults.

If you need a general-purpose source of entropy, the standard library remains an excellent choice.

If you need deterministic cryptographic randomness whose behavior you can reason about—whose latency you can predict and whose state evolution you can audit—then the design presented here follows inevitably from that requirement. Once randomness is treated as state evolving over time, rather than as an oracle invoked on demand, many familiar shortcuts stop making sense. What remains is a system whose behavior is explicit, bounded, and knowable. Good architectures do not happen by accident. They emerge when constraints are taken seriously and followed to their logical conclusions—even when those conclusions lead to different constructions under different constraints.

This design is not theoretical; the AES-CTR-DRBG implementation described here is used as the underlying randomness source in my NanoID (opens in a new window) library, where deterministic behavior and predictable performance are equally important.

Implementation: https://github.com/sixafter/aes-ctr-drbg (opens in a new window)

“All problems in computer science can be solved by another level of indirection.” — David J. Wheeler

Identifier generation is one of those concerns that quietly disappears into the background of a system. At small scale, it is effectively free. A call to a random number generator, a string conversion, and the work is done. There is little reason to revisit it, and even less reason to question its design. This holds largely because identifier generation tends to be invisible: it sits at the edges of requests, happens quickly, and rarely shows up in profiling output. When it does, it is usually dismissed as noise.

That assumption holds only as long as the surrounding system remains small. As concurrency increases and identifier generation moves onto hot paths, what was once incidental becomes part of the system’s observable behavior. Identifiers are created everywhere: at request boundaries, inside storage layers, across services that never coordinate with one another. They are infrastructure in the most literal sense—pervasive, unavoidable, and relied upon precisely because they are assumed to be cheap. When they stop being cheap, the cost is paid everywhere.

NanoID (opens in a new window) fits neatly into this mental model. It produces compact, URL-safe identifiers with reasonable entropy and a simple interface, and in most environments it behaves exactly as expected. It is easy to adopt and difficult to misuse, which reinforces the idea that identifier generation is a solved problem. The difficulty is not with NanoID itself, but with the way its cost profile changes when it is exercised continuously and concurrently.

In Go, the most direct implementation of NanoID draws randomness from crypto/rand.Reader. From a correctness standpoint, this is beyond reproach. The operating system provides high-quality entropy, the guarantees are well understood, and the resulting identifiers are unpredictable in exactly the ways they should be. Under light use, the cost of doing so is effectively invisible, which further entrenches the assumption that there is nothing here worth examining.

Under sustained concurrency, that assumption no longer holds. Each call to crypto/rand crosses into the kernel, sources entropy, fills a buffer, and returns. Temporary buffers are allocated and discarded. None of this is problematic in isolation, but taken together and repeated at scale, it becomes the dominant cost of identifier generation. At that point, “generating an ID” is no longer a small operation. It is a collection of system calls, allocations, and coordination that happens to terminate in a string.

This is not a criticism of crypto/rand. It is doing exactly what it is designed to do. The issue is one of proximity. Entropy acquisition and identifier construction are tightly coupled, even though they serve different purposes. Correctness requires the former. Performance becomes sensitive when it occupies the hottest path of the system.

Concurrency has a way of making these relationships visible. Throughput flattens earlier than expected. Latency becomes uneven. Profilers begin to attribute meaningful time to what was assumed to be trivial. In practice, the profiler does not report time spent “generating IDs”; it reports time spent in the kernel, allocation paths, and garbage collection. Identifier generation becomes visible not because it is conceptually complex, but because it is structurally misplaced.

Separating Entropy from Generation

At that point, there are two broad directions a system can move. One is to accept the cost and design capacity around it. The other is to separate concerns that were previously implicit. The former tends to entrench friction around an otherwise unremarkable operation. The latter introduces an architectural boundary.

NanoID does not require fresh entropy for every identifier. What it requires is unpredictability. Those two properties are related, but they are not equivalent. Entropy must originate from a trusted source, but once acquired, it can be expanded safely using a cryptographically sound construction. There is no requirement to consult the operating system on every invocation, and doing so ties correctness to cost in a way that becomes increasingly visible under load.

This observation reshapes the implementation. Instead of treating identifier generation as a single indivisible operation, it becomes possible to draw a boundary between entropy acquisition and identifier construction. In practical terms, this means seeding a generator once from crypto/rand and then producing random bytes locally. The operating system remains the root of trust, but it no longer sits in the inner loop.

The Shape of the Hot Path

A stream cipher such as ChaCha20 fits this role naturally. It is designed to generate large volumes of pseudorandom output from a fixed key and nonce, and its properties are well understood. Once initialized, it produces random bytes without further system interaction. Used this way, it does not replace the operating system’s entropy source; it amortizes it.

Each generator instance is seeded once using crypto/rand and then used to satisfy multiple NanoID generations. The generator itself is not shared across callers. Sharing would introduce contention and obscure the very boundary the design is trying to establish. Instead, generator instances are treated as independent state machines that can be reused opportunistically.

This is where sync.Pool becomes useful, not as a micro-optimization, but as a way to preserve locality without introducing coordination. Generator instances are pooled so that goroutines can borrow them briefly, generate the required bytes, and return them. There is no shared mutable state and no locking around generation itself. The pool exists solely to reduce repeated setup cost when reuse is inexpensive.

Once entropy is decoupled, other sources of variability become apparent. NanoID generation requires temporary byte buffers—first for random data, then for mapping those bytes into an alphabet. These buffers are uniform in size and short-lived. Allocating and discarding them repeatedly introduces allocation pressure that is unrelated to the semantics of identifier generation. Pooling these buffers follows the same reasoning as pooling generator state: reuse when convenient, allow reclamation when not.

The same discipline applies to configuration. NanoID relies on derived values such as alphabet size, bit masks, and the number of random bytes required to generate an identifier of a given length. These values do not change per call, yet they are often computed as part of the generation path. Computing them once and fixing them at construction time pushes variability outward and keeps the inner loop small and predictable. The generation path becomes a straight-line transformation from random bytes to characters, without branching or recomputation.

At this stage, the hot path is deliberately unremarkable. There are no system calls, no locks, and no dynamic allocation beyond the final string itself. The work performed is exactly the work required to produce an identifier, and nothing else. What remains visible in benchmarks is not overhead, but the irreducible cost of string construction in Go.

The effect of these changes is straightforward to measure. Allocation counts collapse to a single allocation per identifier. Latency stabilizes. Throughput scales with available CPU until saturation. More importantly, behavior becomes predictable. Identifier generation stops appearing as a source of variance and resumes its role as infrastructure.

An implementation of the approach described here is available as an open-source Go module, along with a small command-line tool built on top of it:

• https://github.com/sixafter/nanoid (opens in a new window)
• https://github.com/sixafter/nanoid-cli (opens in a new window)

What does not change is the security model. Entropy still originates from the operating system. Generator state is seeded from a cryptographically secure source. Unpredictability is preserved. There is no attempt to weaken guarantees in exchange for speed. The improvement comes from respecting boundaries, not from relaxing them.

Identifier generation stops being trivial when it becomes infrastructure. At that point, it deserves the same architectural treatment as any other component that occupies a hot path. Once those boundaries are made explicit, the problem largely resolves itself.

At that point, the choice of entropy source becomes an explicit design decision rather than an incidental one. The NanoID implementation described here does not assume a single generator, nor does it require that entropy be sourced directly from the operating system on every invocation. Instead, it is structured to accept a well-defined pseudorandom generator that is seeded from a trusted source and then exercised locally.

For environments where throughput and steady behavior under concurrency are the primary concerns, a ChaCha20-based generator provides a practical balance. Seeded once from crypto/rand, it offers high-quality pseudorandom output with predictable performance characteristics, making it well suited for systems where identifier generation sits on a hot path and must remain invisible.

In environments where regulatory or compliance requirements apply, particularly those that mandate FIPS 140-2 (opens in a new window) validation, an AES-CTR-DRBG construction becomes the appropriate choice. In that context, the same architectural separation applies: entropy is sourced from the operating system, expanded via a standards-aligned deterministic generator, and consumed locally. The difference is not architectural, but contractual—the guarantees are defined externally rather than operationally.

The important point is that these choices do not alter the shape of the system. Whether the generator is ChaCha20-based or an AES-CTR-DRBG, the boundary between entropy acquisition and identifier construction remains intact. The cost model is stable, the hot path is predictable, and the guarantees are explicit rather than implicit.

Implementation: https://github.com/sixafter/nanoid (opens in a new window)

Originally posted on BUSRide Magazine (opens in a new window) and subsequently referenced by Vix Technology (opens in a new window) , here are my responses featured alongside other industry commentators. It shouldn’t be a surprise to anyone that DevOps and software engineering displine are fundamental when designing and constructing parallel and distributed systems at a quality level commensurate with that of commercial software.

In what ways does your platform enable a reliable, dependable revenue stream throughout its useable life?

Building on three generations of advanced automated fare collection (AFC) and payments solutions, the Vix solution provides an advanced platform supporting account-based ticketing, card-based ticketing, open payments and closed loop features in a single, highly configurable product. Sophisticated financial management tools, including transit clearing house capabilities, are also core capabilities, all of which can be deployed “on premise” within customer data centers or “as a Service” in a public cloud.

Our solution enables a reliable, dependable revenue stream throughout its usable life because, for Vix, the most important aspect of public transportation is a better journey for our customers. Our relentless focus on building a flexible, scalable, and reliable AFC and payments platform is fundamental to delivering this vision for our customers and for ensuring dependable revenue streams for agencies.

What best practices can agencies use to decrease fare system deployment times, to maximize benefits and minimize revenue disruption?

At Vix, there are many lenses through which we measure how we deliver value to our customers: we recognize the need to reduce build, deployment, operating, and maintenance costs along one dimension whilst ensuring a repeatable and secure quality level commensurate with that of a world-class commercial solution along another dimension.

Full automation for forming, deploying, and upgrading all operational environments and the Vix solution is critically important in reducing overall costs, reducing the risk of human error, increasing operational efficiency, ensuring consistent quality and enable measurable compliance with security practices and governance. To this end, the Vix solution focuses on the best practice of DevOps automation as a core engineering practice across three phases to align with design, build, operations, and maintenance in support of continuous improvement.

Our investment in this area has resulted in vastly reduced build and deployment times. Our relentless focus in this area allows us to minimize costs for systems operations, security operations and disaster recovery scenarios while greatly reducing the timelines for our solution engagement schedules, resulting in a direct savings for our customers without the need to compromise their revenue protection and business continuity objectives.

How does a flexible, open architecture lend itself to increased revenue over time?

An open architecture lends itself to increased revenue over time because the inherent standards-based design patterns and technologies fosters a wide array of integration scenarios within the surrounding and transit payments ecosystem. The APIs that come with an open architecture allow for a range of integrations to be supported as part of a fare collection system, allowing agencies to, for example, partner with coffee shops or other retail outlets to offer deals for transit riders. This provides another revenue stream for the transit agencies, as well as the partner retailers, etc.

However, simply supporting an open architecture isn’t enough. Software and hardware engineering teams must possess the discipline and rigor required to produce quality, scalable, and flexible architectural patterns to support not only revenue streams that we can think of but, equally as important, with forward looking attention to market trends.

How can agencies utilize improved data collection and analysis to leverage new revenue streams?

To begin to utilize improved data collection and analysis to leverage new revenue streams, agencies must first make it a priority. Analytics and the field of data science are now foundational to nearly all industries - so much so that data itself has become an important strategic and competitive asset. This data represents a rather rich taxonomy for enabling a variety of revenue opportunities for agencies to improve a rider’s experience.

At Vix, our focus on a better customer journey shapes how we support agencies in the areas of analytics. The areas of analytics that are important to us, and thus to our solution, are predictive journey analytics, predictive maintenance analytics, fraud prevention, real-time passenger information, capacity and pricing optimization, customer analytics and loyalty marketing, intelligent transportation system (ITS) optimization and the co-occurrence of information from each of these areas to model how to improve the overall transit experience.

Originally posted on BUSRide Magazine (opens in a new window) .

Dummy objects are passed around but never actually used. Usually, they’re simply used to fill parameter lists.

Fake objects actually have working implementations, but usually take some shortcut which makes them not suitable for production (an in memory database is a good example).

Stubs provide answers to calls made during the test, usually not responding to anything outside what’s programmed for the test. Stubs may also record information about calls, such as an email gateway stub that remembers the messages it ‘sent’ or only how many messages it ‘sent’.

Mocks are what we’re talking about here. Mocks are objects pre-programmed with expectations forming a specification of the calls they expect to receive.

But as with all things, first a bit of context.

The Agile Development Methodology

Agile is a software development methodology that borrows from Scrum (opens in a new window) project management practices, Extreme Programming (opens in a new window) (XP), and Test Driven Development (opens in a new window) (TDD).

It’s not my intention to write about Scrum, XP, and TDD in excruciating detail since there are many excellent books on these subjects. I would, however, like to share my view on why TDD is foundational to any software engineering effort.

Scrum

As most are aware, Scrum is an iterative, incremental framework for managing complex work and though intended for management of software development projects, it can be used as a general program management approach. Contrary to popular belief, however, Scrum is not an acronym. First used to describe hyper-productive development in 1987 by Ikujiro Nonaka (opens in a new window) and Hirotaka Takeuchi (opens in a new window) , Scrum actually refers to the mechanism used in rugby for getting an out-of-play ball back into play.

Many software engineering teams still practice the Waterfall development methodology and for some projects – in particular when requirements are static and rarely change – it’s an appropriate practice. The building of a jet propulsion engine is a good example of a project that likely benefits from the Waterfall development methodology. In contrast, Scrum’s two pillars are team empowerment and adaptability:

1. Team empowerment: Once teams are given work to do, they are responsible for figuring out how to do it. The team does the best it can during each iteration. While a team works, their only interaction with management is to tell management what is getting in their way and needs to be removed to improve their productivity.

2. Adaptability: Scrum uses “punctuated equilibrium”. The team maintains an equilibrium during each iteration, insulated from outside disturbance. Iteration are generally punctuated every thirty days so that the team and management can evaluate what should be done during the next iteration; this decision is based on what the team has accomplished and what the environment dictates is the next most important thing to do.

Velocity

It’s difficult to discuss the importance of writing tests without first discussing the notion of velocity and while I could write a dissertation just on the notion of velocity, it is simply defined as how many story points of effort a team can complete in a single iteration. Once established, velocity can be used to plan projects and forecast release and product completion dates. A team’s velocity is an important key performance indicator.

eXtreme Programming

From XP, we borrow pair programming and continuous integration. Code reviews are good so let’s do them all the time; this is pair programming. Measuring code quality and ensuring a source level change didn’t cause any regressions is good so let’s do it as often as possible; continuous integration.

It should be no surprise that writing automated tests and measuring code coverage go hand-in-hand with continuous integration.

Test-Driven Development

For any particular software engineering effort, there are three axes to control: quality, schedule, and scope. Quality remains locked at “high” and so the engineering team must choose how to balance the remaining two.

Clean code that works…now. This is the seeming contradiction that lies behind much of the pain of programming. Test driven development replies to this contradiction with a paradox – test the program before you write it.

• Test Driven Development: By Example, 2002

Regardless of your development methodology, writing tests are important. Automated testing underpins many Agile practices, most significantly as part of Continuous Integration (opens in a new window) . TDD is a process by which the tests are written before the code itself. Through a rapid cycle of adding new tests, making them pass, and then refactoring to clean code, the software design evolves through the tests.

Red, Green, Refactor

When a test is first written, it most likely won’t even compile - the implementation hasn’t been written. Modern IDEs allow a developer to rapidly create code and then refactor it into shape. After adding a test, a developer strives to reach a “green bar” (all tests passing) as quickly as possible, and then refactors backed up by the safety of the tests. This quickly becomes a cycle of “red, green, refactor”, adding a kind of rhythm to the development process. When a new feature is required, tests are added, code is written to make the tests pass, and finally refactoring removes any duplication.

Benefits of Writing Tests

Whether your project is open source, commercial, or corporate, there are clear and important advantages to writing automated tests.

1. Courage. Writing tests allows one to move from a state of uncertainty to that of courage. When you write tests, you know the state of the your code. Intimately. You know if your code works or if it doesn’t. You know if your old code, and the code of your colleagues works as expected.
2. Communication. Tests can also be used for communication. This works well to communicate what the software does. We all may not know how to speak the same language but as a member of a particular software engineering team, we all know how to communicate using a specific programming language. We all can read and write code with good clarity.
3. Documentation. Beyond shore-to-shore communication, tests also make a good candidate to use as examples in documentation and in comments.
4. Refactoring. Setting your minimum allowable test coverage percentage to a respectable number (90 percent and greater) allows you to refactor your source code with confidence. Use cases pivot. You refactor the code. Then simply make all failing tests pass again.
5. Design. Writing tests first leads to a better design.
   1. It forces you to think about the shape of your API up front; to really understand how your consumer uses the API before you spend valuable time writing the implementation.
   2. It allows you to understand any possible frustrations and complexities with the shape of your API before your consumer.
   3. It saves time by ensuring the API satisfies your design goals and nothing more.

All of these points are critical to a team’s velocity.

Rules of Engagement

The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD, SHOULD NOT, RECOMMENDED, MAY and OPTIONAL, when they appear in this document, are to be interpreted as described in RFC-2119 (opens in a new window) .

In the end, automated tests are just code. Even with best intentions, they can be written incorrectly and contain bugs. Here are a few rules to consider.

1. Keep unit tests small and fast. Ideally the entire test suite SHOULD be executed before every code check in. Keeping the tests fast reduce the development turnaround time.
2. Tests MUST be fully automated and non-interactive.
3. Keep tests independent. To ensure testing robustness and simplify maintenance, tests SHALL NOT rely on other tests and SHALL NOT depend on the order in which other tests are executed.
4. A test MUST leave the environment exactly as it was before execution began. This is an extremely important point since test execution order is never guaranteed and so environmental changes may cause sporadic test failures, which can waste precious time.
5. A test MUST contain at least one assertion.
6. A test MUST contain 3 logical sections: arrange, act, and assert.
7. A test MUST cover boundary cases.
8. A test fixture MUST provide negative tests. Negative tests intentionally misuse the code and verify robustness and appropriate error handling.
9. Write tests to reproduce bugs. When a bug is reported, write a test to reproduce the bug (i.e. a failing test) and use this test as success criteria when fixing the code.
10. Conduct peer code reviews (opens in a new window) on all tests.

Regarding bullet 6, William C. Wake wrote about (opens in a new window) using the “arrange, act, assert” pattern in 2003. Here’s a summary:

• Arrange: What is being set up and initialized is contained in the arrange section.
• Act: What method is being executed is contained in the act section
• Assert: What determines the outcome of the test is in the the assert section.

I’ll write about dummies, mocks, stubs, and fakes in a future post.

Conclusion

With test-driven development, robust code is delivered along with a comprehensive suite of tests, allowing developers to make future changes with confidence. I firmly believe that unless you’ve been a contributing member of a software team responsible for a complex code base that it may be difficult to appreciate the critical importance of having a suite of automated unit, integration, and story tests.

Move away from a position of uncertainty to one of confidence. Write tests.

Last year was very busy for me professionally. I decided very early that in order to become healthier, I needed to design my workout schedule and nutritional plan with efficiency in mind. The fundamental areas of physical health are nutrition and exercise. The purpose of this post is simply to share my thoughts on what I’ve learned over the past year with respect to nutrition.

Nutrition

You are what you eat. It’s that simple. The cornerstone of becoming healthier is consuming nutrients that aid in a strong immune system, muscle recovery, vital organ functioning, and a reduction of gynoid and android (visceral) fat. The production of visceral fat is your body’s survival strategy as a result of the internal organs insulating themselves from toxins by encapsulating the toxins in fat.

I’ll first provide a bit of basic background information on foods that I’ve decided to incorporate into my daily routine and end with two of my favorite health smoothie recipes.

Orange Juice:

Usually, orange juice and almond milk are the base liquids of my smoothie. Orange juice increases HDL significantly (opens in a new window) . I use Simply Orange (opens in a new window) , which you can purchase from most any grocer.

Greek Yogurt:

I add yogurt to my smoothies for protein, flavor, and the probiotics (opens in a new window) . I used to use regular yogurt but switched to Greek yogurt because the amount of protein in Greek yogurt is much higher.

Almond Milk:

Almond milk is especially beneficial to our health. For starters, it’s filled with vitamin D, which is known to improve cell function and immunity, and this has also been proven that it can help decrease the risk of Alzheimer’s.

Sources: The Science of Eating: Benefits of Almond Milk URL (opens in a new window)

Flaxseed Oil:

Flaxseed oil contains alpha-linolenic acid ( ALA (opens in a new window) ), which is converted by the body into EPA and DHA, the same omega-3 fatty acids found in fish oil. I do also take a fish oil pill once daily before bed but I like getting the additional fatty acids in the smoothie. I use Spectrum Organic Flaxseed Oil (opens in a new window) from Whole Foods.

Protein Powder:

The types of protein used in protein powders can be divided into two categories: animal source proteins and vegetable source proteins. Animal source proteins include milk protein derivatives like whey and casein, goat’s milk and egg white protein. Vegetable source proteins include soy, rice, pea and hemp proteins. Nutritionally animal proteins are superior to vegetable proteins. Of the animal protein types, the most popular is whey protein. Of the vegetable protein types, soy is the most popular. Most people using vegetable protein powders do so as part of a vegetarian or vegan lifestyle, although many people use soy protein primarily for its heart-health and/or hormone-balancing benefits.

Whey protein is derived from milk. The protein portion of whole milk consists of 20% whey protein and 80% casein protein. Whey is by far the most popular type of protein used in protein powders. For most people, it’s the best all-around choice in terms of taste (it’s one of the best-tasting), quality (it’s the highest) and cost (it’s the most economical). Another unique benefit of whey protein, and one that is often overlooked, is that it enhances the immune system in several ways.

Whey protein comes in two varieties; whey concentrate and whey isolate. The advantages of each are:

• Whey Concentrate. Whey concentrate is more economical per gram of protein. It has a low lactose level that is well tolerated by most lactose-sensitive people. It has trivial amounts of fat and carbs relative to your overall nutrient intake.
• Whey Isolate. Whey isolate is virtually fat-free for those wishing to eliminate as much fat from their diet as possible. It is typically lactose free for those few individuals who are very sensitive to the low-lactose levels found in whey concentrate. Whey isolate tends to taste slightly better than whey concentrate, yet its consistency is a little thinner, without the fat.

My choice in protein is Jay Robb whey protein (opens in a new window) , which is a non-GMO (opens in a new window) whey protein isolate powder that delivers 25 grams of first class protein, 0-fat, 0-cholesterol, 0-sugars, and only 1 gram of carbohydrate per 30 gram serving.

Green Tea:

Green tea has special antioxidants called catechins which appear to have many health benefits (opens in a new window) . Rather than sip endless cups of green tea throughout the day, I add a single teaspoon of green tea powder called matcha (opens in a new window) . I use organic ceremonial matcha (opens in a new window) from DoMatcha.

Kale:

Kale’s health benefits (opens in a new window) are incredible.

Raw Honey:

Raw honey is honey that has not been heated, pasteurized or processed in any manner. The differences between raw and pasteurized honey are substantial. Raw honey has anti-viral, anti-bacterial, and anti-fungal properties. It promotes body and digestive health, is a powerful antioxidant, and strengthens the immune system.

Cacao Nibs:

Cacao nibs are cacao beans that have been roasted, separated from their husks, and broken into smaller pieces. Think of cacao nibs as having the benefits of chocolate but without all the sugar and fat and other non-beneficial disadvantages of chocolate.

Sources:

• Live Super Foods. URI (opens in a new window) .
• Natural News URI (opens in a new window) .
• Huffington Post. URI (opens in a new window) .

Vietnamese Cinnamon:

The health benefits are quite impressive (opens in a new window) . The most health beneficial harvest is organic Vietnamese “Saigon” Cassia Cinnamon, which I order from The Spice House (opens in a new window) .

Coconut Oil:

Nearly 50 percent of the fat in coconut oil is of a type rarely found in nature called lauric acid, a “miracle” compound because of its unique health promoting properties. Your body converts lauric acid into monolaurin, which has anti-viral, anti-bacterial and anti-protozoa properties. More excellent information can be found here (opens in a new window) .

I prefer Nature’s Way Organic Coconut Oil (opens in a new window) .

Sunflower Lecithin:

Sunflower lecithin is a type of phospholipid abundant in sunflower seeds that supports healthy brain function and nervous system.

Source: Real Raw Food URI (opens in a new window)

A Few Healthy Smoothies…

After a bit of refinement of smoothie recipes over the past year, I’ve settled on two of my favorite ones that I have for breakfast and post-workout recovery.

Breakfast Smoothie:

• 250 ml organic, unsweetended almond milk
• 1 Tbsp Flaxseed oil
• 1 Tbsp raw organic honey
• 1/2 cup organic, non-fat greek yogurt
• 1 scoop Jay Robb whey protein
• 1 tsp organic DoMacha (opens in a new window)
• 1 to 2 cups of mixed berries. Strawberries, blackberries, blueberries, etc.
• 3 cups fresh organic Kale
• 1 cup fresh organic Spinach
• Approximately 10 to 16 ice cubes

Add the ingredients in the order listed and blend well. This recipe makes approximately 32 oz. of smoothie.

Post-Workout Smoothie:

• 250 ml organic, unsweetended almond milk
• 1 Tbsp Flaxseed oil
• 1 Tbsp raw organic honey
• 1/2 cup organic, non-fat greek yogurt
• 1 scoop Jay Robb whey protein
• 1/2 cup Cacao nibs, organic.
• 1 tsp Vietnamese “Saigon” Cassia Cinnamon, organic.
• 1 frozen Banana, organic
• 1 Tbsp Coconut oil, organic.
• 1 Tbsp fresh ground almond butter, organic
• Approximately 10 to 12 ice cubes

Add the ingredients in the order listed and blend well. This recipe makes approximately 24 oz. of smoothie.

Paleolithic

On a related note, I began to focus on a Paleolithic nutritional plan, which is a nutritional plan based on the presumed diet of Paleolithic humans. It is based on the premise that human genetics have scarcely changed since the dawn of agriculture, which marked the end of the Paleolithic era, around 15,000 years ago, and that modern humans are adapted to the Paleolithic diet.

The Paleolithic diet consists mainly of fish, grass-fed pasture raised meats, eggs, vegetables, fruit, fungi, roots, and nuts, and excludes grains, legumes, dairy products, potatoes, refined salt, refined sugar, and processed oils.

Though my next post is focused on exercise, I’ll certainly touch on my experiences with a Paleolithic nutritional plan and how I think it’s worked with my my overall CrossFit routine and corresponding recovery cycle at The Foundry (opens in a new window) .

There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. – C.A.R. Hoare

Mobile, cloud, social, and non-relational storage technologies (big data) represent agility for many businesses. In particular, the convergence of these technologies equip businesses with efficient methods to attribute key performance indicators, optimize direction, and, equally as important, allocate changes in business processes; all in real-time and for a vast array of incoming data formats. Individually, each of these technology areas are immense and yet are enablers for each other; and so in early 2008, I began to focus my attention, application of algorithms, and research to the convergence of these technologies.

The technology shifts created by mobile, cloud, social, and big data have changed and are drastically changing the business landscape. The convergence of these technologies began years ago and has matured considerably since 2008. Social media has contributed to this significantly. Big Table (opens in a new window) and, subsequently, HBase (opens in a new window) was born out of need within Google and the open source community, respectively. The Apache Cassandra (opens in a new window) project was born out of need within Facebook. We’re able to learn interesting statistics like the half-life of links shared on Twitter is approximately 2.8 hours, sports stories spread the fastest of any topics, and there’s a material difference between what people share and what they click. From a sales and marketing perspective, we can use big data to better leverage social channels to rapidly detect changes in buyer activities and preferences, identify market shifts, monitor competitive activity, and identify opportunities for your business. Social media is analyzed to understand behavior, although it’s important to remember as with all population sampling, it only provides a specialized subset of opinions about a particular topic.

Big Data and Analytics

Big Data is primarily classified by the use of non-relational durable storage solutions that embrace the following characteristics of information technology.

• Volume. As of 2012, about 2.5 exabytes of data are created each day, and that number is doubling every 40 months or so. More data cross the internet every second than were stored in the entire internet just 20 years ago. For example, it is estimated that Wal Mart collects more than 2.5 petabytes (opens in a new window) of data every hour from its customer transactions.
• Velocity. For many applications, the speed of data creation is more important than the volume. Real-time or nearly real time information makes it possible for a company to be much more agile than its competitors. Just consider the speed at which information is collected from the sheer number of active mobile devices each and every day.
• Variety. Information now flows from a plethora of sources in the form of messages, updates, and images from social networks; electronic communication; instrumented machinery; readings from sensors; GPS signals from mobile phones; point of sale terminals; and so on. Structured databases are not well suited to storing and processing information from these disparate sources.

…and it’s only becoming even more challenging. Each and every day, three times per second, we produce the equivalent of the amount of data that the Library of Congress has in its entire print collection. Most of the data produced is irrelevant noise. Clearly, the challenge is in filtering this noise and distilling the remaining information into meaningful results. This vast amount of information is essentially useless without analytics. Timely analytics. Accurate analytics. Technical solutions in this space must be able to traverse multiple data centers, the cloud, and geographical zones. Accurate, real-time analysis affords us an opportunity to answer crucial business questions by those wanting to confidently pivot in response to key performance indicators (KPI).

Before you delve into the non-relational storage arena, it’s important to understand the CAP theorem (opens in a new window) . The theorem began as a conjecture made by University of California, Berkeley computer scientist Eric Brewer (opens in a new window) at the 2000 Symposium on Principles of Distributed Computing (PODC). In 2002, Seth Gilbert and Nancy Lynch of MIT (opens in a new window) published a formal proof of Brewer’s conjecture, rendering it a theorem.

Formally, in theoretical computer science, the CAP theorem states that it is impossible for a distributed computer system to simultaneously satisfy all three of the following characteristics:

1. Consistency. All nodes possess the same data at the same time.
2. Availability. A guarantee the system is available at all times to fully operate.
3. Partition tolerance. The system continues to operate despite arbitrary message loss or failure of part of the system.

In May 2012, Brewer clarified some of his positions (opens in a new window) on why the oft-used “two out of three” concept can be misleading or misapplied.

Broadly speaking, durable storage technologies are divided into two innate categories: relational and non-relational. Relational databases have been widely used for the past 40 years and are still quite widely used today. Microsoft SQL Server, IBM DB2, Oracle, and MySQL are all examples of relational database implementations. Suffice it to say most non-relational database system architectures favor partition tolerance and availability over strong consistency. Other implementations such as Cassandra (opens in a new window) offer per-operation control over which two characteristics of the CAP theorem you would like to favor. My favorite distribution of Cassandra is DataStax (opens in a new window) , which combines Cassandra, Hadoop (opens in a new window) , and SOLR (opens in a new window) into a single packaged distribution. Their enterprise edition (opens in a new window) is free to developers, which is convenient.

With respect to non-relational databases, one must consider a few orientations.

• Object. Object databases work to void object-relational impedance mismatch that occurs when trying to use a relational database under an application written in an object-oriented programming language. These databases store information in terms of the object themselves and not in terms of columns and rows.
• XML. XML databases are optimized specifically for working with XML and are, essentially, a special form of a document-oriented database. XML databases have one core function: to allow for efficient storage and query of XML documents. Examples of XML databases are Tamino (opens in a new window) (Software AG), eXist (opens in a new window) , Berkeley XML DB (opens in a new window) (Oracle), MarkLogic (opens in a new window) Server, and Sedna (opens in a new window) .
• Document. The basic unit or storage for a document database is a complete document. CouchDB (opens in a new window) and MongoDB (opens in a new window) are examples of document databases. This storage model offers several key advantages over a relational model:
  • A document can store any number of fields of any length, and each field can store multiple values. A relational model requires all fields to be present for every record.
  • An empty (or null) value (and the related field) does not need to be stored, which can save space. On a related note, many relational databases currently offer support for sparse tables.
  • Full text search capabilities are typically an intrinsic feature.
  • Security can be assigned at an individual document level.
  • A predefined schema is not required whereas a relational database does require schema to be defined in advance of performing any storage or retrieval operation.
• Key-Value. A key/value database stores values based on keys and are classically represented as a Distributed Hashtable (DHT). In a relational model, we tend to first consider the tables that our domain requires, then think of how we can normalize the tables to avoid duplicate data. In a key-value store, however, typically we don’t define a schema. Examples of key-value databases are MongoDB (opens in a new window) , Redis (opens in a new window) , Amazon Dynamo (opens in a new window) , and Project Voldemort (opens in a new window) .
• Graph. Graph databases differ from other non-relational database offerings (such as key-value stores) in that they represent the edge intrinsically. Instead of tables and columns, a graph database uses three basic constructs to represent data: nodes, edges, and properties. A node is a standalone, independent object. An edge is an object that depends on the existence of two nodes. Properties represent attributes of a node. Examples of graph databases are FlockDB (opens in a new window) (open sourced by Twitter in April 2010) and Neo4J (opens in a new window) .
• Columnar. A columnar database organizes data around columns instead of rows. This subtle difference optimizes some workloads—in particular, data warehouse and analytics applications that require computing aggregate values over very large sets of data—for certain kinds of problems. Databases that follow this orientation represent distributed hash tables ( DHTs ). Examples of columnar databases are Cassandra (opens in a new window) , Google Bigtable (opens in a new window) , HBase (opens in a new window) , and Hypertable (opens in a new window) .

It’s worth noting, a few databases exist that represent hybrid orientation implementations; for example, Riak (opens in a new window) is a hybrid implementation based on Amazon’s Dynamo in that it is both a document-oriented and key-value implementation.

Though, just as with relational databases, planning is key when thinking about launching a non-relational database project. Non-relational database projects require a slightly different software engineering approach. In a relational model, we tend to consider our data model first and then we build queries to satisfy application requirements. Even though a database may be schema-less, in a non-relational model we MUST (opens in a new window) first understand an application’s queries as a starting point since that influences database and table layout.

Even companies with a large investment in relational-based analytics tools can leverage non-relational storage technologies but even so, the definition of a taxonomy (or multiple taxonomies) for analytics processing is of paramount importance. Typically, companies that are heavily invested in relational technology as their primary systems of record employ extract transform and load (ETL) processes to periodically restructure information based on defined taxonomies into non-relational storage products so they are able to take advantage of more efficient and real-time processing using massively parallel analytics technologies such as Hadoop and Map Reduce.

Capturing data without analytics in mind from a business’ point of view is generally moot. Consider the goals, and what key data sets are needed to reach those goals. More important is to consider the set of business questions you are attempting to answer and, to that point, I strongly recommend you bring in a data scientist to perform an audit of an existing data architecture.

Mobile

Business-to-consumer (B2C) mobile applications impose different requirements than that of business-to-business (B2B) and corporate applications not only in terms of marketing but in terms of the requirement for an extremely efficient and aesthetically pleasant user experience (UX) and more stringent software engineering practices. It’s very difficult to recover from poor reviews and customer feedback.

In the mobile space, three companies represent the majority globally: Apple, Google, and Microsoft. Apple and Google make up 96% of the entire mobile space. Microsoft is less than 1% with the remaining (just over 3%) still on others such as RIM, Symbian, etc. If we add tablets to the equation (so now not just mobile phones) then as of April 2013 Apple iOS represents 59.04%, Google Android represents 26.02%, and Microsoft Windows Phone represents 1.14%; see Net Marketshare (opens in a new window) .

Across all mobile operating systems, if you observe how people use smartphones and look beyond telephone calls, email, and texting then you’ll see that native applications dominate. Users spend on average, 82% of their mobile minutes with native applications and 18% with web browsers. They download approximately 50 to 75 applications to their phones (out of more than a million available) but regularly use about 10.

As another data point, mobile applications broadly fall into six categories:

1. Games and Entertainment
2. Social Networks ( FaceBook (opens in a new window) , Twitter (opens in a new window) , Pinterest (opens in a new window) , Path (opens in a new window) , Tumblr (opens in a new window) , etc)
3. Utilities (maps, clocks, calendars, cameras, email, etc.)
4. Discovery ( Yelp (opens in a new window) , GrubHub (opens in a new window) , etc.)
5. Education ( AccelaStudy (opens in a new window) , Contig (opens in a new window) , Flashcard Champion (opens in a new window) , etc.)
6. Brands ( Nike (opens in a new window) , Red Bull (opens in a new window) , etc.)

Regardless of your application’s category, native mobile applications offer the absolute best user experience in many aspects with frame rates, offline support, and access to the device’s hardware to name but a few. Contrary to what some may espouse, mobile devices are not well connected; even within major United States and European cities where connectivity is commonly sporadic and network latency is commonly high. Consider many scenarios whereby consumers pass in and out of good network connectivity while on mass transit systems; even mobile applications that “think” they are connected to a hi-fidelity wireless network may really be connected to a personal hotspot that is in turn connected to the internet over a cellular channel. It is for these reasons I strongly discourage the creation of hybrid and HTML 5 mobile applications. Granted, hybrid and HTML 5 mobile applications may prove satisfactory in some corporate settings but if you want your application to be among the 10 regularly used ones then write native applications; it’s that simple. And for the record, I am not a fan of “ 4GL (opens in a new window) ” Integrated Development Environments (IDE) that generate code for multiple mobile platforms. Perhaps I am a purist but I prefer to have direct access to and control over the core mobile platform rather than having to interact with process and technical abstractions. Your consumers do appreciate this level of attention to detail and UX refinement.

If you do write applications in the consumer space and if you’ve purchased a smartphone or tablet recently, and if it doesn’t have an Apple logo on it then it’s likely you already know of Android. Android is an open sourced Linux-based operating system (OS) launched approximately 5 years ago as of the time of this writing. The Android operating system dates back to 2003 when Rich Milner, Nick Sears, Chris White, and Andy Rubin began working on a mobile operating system, which Google would eventually purchase in 2005.

For years, the dominant OS was Symbian, which has its roots in Psion’s EPOC (opens in a new window) OS. By 2007, Symbian accounted for over 63% of the worldwide smartphone market. However, by 2008, when it was purchased by Nokia, that number dropped to 52%. At that time, the closest competitors were Research in Motion’s Blackberry OS (16.6%), Microsoft Windows Mobile (11.8%), and the still very new iOS (8.2%). With Nokia’s OS was poorly equipped to compete with iOS, RIM not wanting to leave its business focus comfort zone and Microsoft’s mobile attempts clearly failing, it was Google’s purchase of Android Inc in 2005 that would eventually provide competition. The Linux-based, open-source mobile operating system would eventually find its way to the market with the release of the T-Mobile G1 in 2008. The G1 sold well and that elevated Android to reach a 3.9% share in the smartphone market in 2009. At this point, iOS had doubled its share to 14.4%, RIM grew to just under 20%, Windows Mobile had dropped to 8.7%, and Symbian correspondingly dropped to 46.9%. Today, Android is used in 61% of the smartphones (excluding tablets) sold worldwide, which is primarily attributed to the fact it can be found in both high-end handsets (targeting the Apple iPhone) and in lower budget smartphones (where Symbian once dominated). Apple iOS commands 20.5% of the worldwide smartphone marketshare, which is impressive considering it’s only included in Apple’s own high-end phones. RIM is now down to less than 1.5%.

In general, it’s easier to get an application released onto Android, but from technical engineering and business points of view, Android does have significant disadvantages over Apple’s iOS:

• Google Play is not nearly as mature as Apple’s AppStore
• People are reluctant to pay for Android applications
• Piracy of applications is widespread
• Security concerns and malware
• Little to no enterprise penetration
• OS upgrades are cumbersome, slow, and generally require intervention by the carrier

However, one can’t ignore Android these days; and in particular for the consumer space. Corporate and business applications are a different matter altogether and I’ll save that discussion for a later point in time time.

Cloud

So, where does cloud technology fit into all of this? Everywhere. Cloud is the cohesive technology between mobile, social, and big data. Over the past nearly 6 years, and after nearly 500 mobile applications shipped, I can think of exactly one mobile application I’ve written that did not involve hosted cloud services; and cloud providers are only getting better. Cloud providers such as Amazon AWS (opens in a new window) and Microsoft Windows Azure (opens in a new window) provide full featured platform-as-a-service (PaaS) and infrastructure-as-a-service (IaaS) offerings that complement solutions spanning the mobile, social, and non-relational database solution domains. There are other providers such as Linode (opens in a new window) that offer competitive IaaS offerings and with a very approachable cost model. Discussing the technical offering of each cloud provider is not in scope of this post; perhaps at a later point in time.

The use of cloud computing technologies offer companies an opportunity to recognize cost savings that would otherwise not be possible with on-premise hardware. For example, it’s not uncommon to develop scripts that create an environment starting with mere credentials to a fully functioning application stack in a matter of minutes only to tear it down once the day is over. Patterns such as this are quite attractive to stakeholders.

Conclusion

For organizations seeking a fast, easy and cost-effective way to manage business operations, the convergence of mobile, cloud, social, and non-relational storage technologies equips parallel and distributed system designers and business stakeholders with a unique opportunity and insight into new patterns that would otherwise require a much higher degree of cooperation and effort among disperate teams. an architecture that seeks signals, models them for their impact, and then adapts to the business process of the organization. These technologies also facilitate pattern-based strategy architectures, which are attractive to both technical architects and business stakeholders. A pattern-based strategy identifies signals, models them for their impact, and then adapts to the business process of the organization instead of current strategies defined by making hypotheses, performing analyses, and reaching conclusions that are often wrong given the time to implement the change and pivot occurs at a slower pace than the tracked market trends.

Good architectures don’t just happen, they are designed.

References:

• Andrew McAfee and Erik Brynjolfsson. “Big Data: The Management Revolution”. Harvard Business Review, October 2012.

I constantly reason through and research, among other things, many aspects of health and fitness. Categorically speaking, the two cornerstone aspects of heath and fitness are diet and exercise. As a former athelete, I am quite familiar with elements and mechanics of muscle development, diet, and exercise and the implications of each on one’s health and wellness. I’ve been actively involved in sports since very early in life; throughout middle school, high school and college . From my experience, the set of baseline measurements to gather are our metabolic rates, body fat percentages, and blood chemistry. Once we have these baselines, then we simply employ the scientific method (opens in a new window) .

BMR and RMR:

Muscle burns more calories than fat and so understanding how many calories one burns at rest over a specific duration is important and particularly in the context of your body fat percentage, which I discuss later in this post.

BMR and RMR are estimates of how many calories you would burn if you were to do nothing but rest for 24 hours. Together, they represent the minimum amount of energy required to keep your body functioning, including your heart beating, lungs breathing, and body temperature normal.

• BMR stands for Basal Metabolic Rate, and is synonymous with Basal Energy Expenditure or BEE. BMR measurements are typically taken in a darkened room upon waking after 8 hours of sleep; 12 hours of fasting to ensure that the digestive system is inactive; and resting in a reclining position.
• RMR stands for Resting Metabolic Rate, and is synonymous with Resting Energy Expenditure or REE. RMR measurements are typically taken under less restricted conditions than BMR, and do not require that the subject spend the night sleeping in the test facility prior to testing.

All this said, however, BMR and RMR does not represent the minimum amount of calories one burns in a day. Why? Because many of us don’t normally sit around all day in a horizontal position with our eyes closed. We perform tasks. Therefore, for a more accurate estimate of how many calories one burns in a day you may consider looking into an “activity calculator”. If you do find one, then do not add your BMR or RMR to those results. Use one method or the other.

Oh, and please keep in mind that “calories burned” calculations are based on your weight. As you lose weight and activity becomes easier, you’ll burn fewer calories performing the same activities. Therefore as you lose weight, you will need to periodically recalculate the number of calories you burn in a day.

The apple, the pear, and the egg…

There are essentially three prolific body shapes represented in the different obese classifications for people: the apple, the pear, and the egg. With these shapes in mind, let’s discuss the notion of android and gynoid fat.

Android fat is the fat stored in the midsection of the body predominantly in the abdomen, but also the fat stored in the chest and upper arms. This is a common fat-storage location among men and is associated with an apple shape and is consistently associated with increased risk of heart disease and diabetes, hormonal imbalances, some cancers, sleep apnea, and more. In this type, the fat is mainly accumulated within the abdomen and around the vital organs (visceral fat).

Gynoid fat storage represents the “pear” shape and is more common among women. This fat is stored primarily around the buttox, hips and thigh areas. Overweight pear-shaped people tend to suffer from mechanical problems such as hip, knee and other joint issues due to an excess of fat (weight stress) on the lower half of their body.

The Ovoid, or egg, shape represents a unisex body type wherein a generalized coverage of body fat is exhibited.

It’s important that both one’s subcutaneous fat (the fat under your skin) and visceral fat (fat around your internal organs) be specifically measured. Determining the ratio of android to gynoid fat (the A/G ratio) and monitoring that ratio is critical important because it directly correlates to the prevalence of visceral fat. The production of viceral fat is your body’s survival strategy as a result of the internal organs insulating themselves from toxins by encapsulating the toxins in fat. Unfortunately, this may impede organ function and, as a result of this diminished function, increase the risk for disease. Ideally, your android fat should be lower than your gynoid fat; so a value below 1.0 is in the ideal range of the threshold metrics.

I scheduled an appointment with DexaFit (opens in a new window) to have them conduct a DexaFit DXA Total Body Composition Scan. In short, this scan is very accurate and produces an exact measurement of one’s A/G ratio along with other vitally important infomation; and best of all, the entire process takes approximately 7 minutes. I certainly encourage anyone interested in their body fat analysis to strongly consider DexaFit.

Incidentally, this is why body fat analysis methods that use the notion of simple subcutanous clamps or water immersion are vastly inferior to the DexaFit method of measurement since those methods do not measure visceral fat and so the overall body fat percentage calcualation is, therefore, inaccurate.

Blood Chemistry Analysis:

A blood chemistry analysis provides an important set of baseline measurements to serve as key indicators for a person’s overall picture of health and wellness. For a male over the age of 40, a comprehensive blood chemistry analysis is recommended on an annual basis. I use DirectLabs (opens in a new window) . DirectLabs provides direct access laboratory testing for those who want to take charge of their own health and personally monitor their own wellness. In the course of time, they assist in the prevention or early detection of disease by providing discounted, high quality online blood and laboratory testing services directly and confidentially to consumers.

There are of course many tests offered by DirectLabs but the ones I chose are panel-selected as the most important battery of tests for males over the age of 40:

Comprehensive Wellness Profile. This profile measures…

• Lipids: This is a group of simple blood tests that reveal important information about the types, amount and distribution of the various types of fats (lipids) in the bloodstream. Includes Total Cholesterol, HDL (good) Cholesterol, LDL (bad) Cholesterol, Risk Ratio (good to total), and Triglycerides.
• Complete Blood Count (CBC’s): Used as a broad screening test to check for such disorders as anemia, infection, and many other diseases. It is actually a panel of tests that examines different parts of the blood.
• Fluids and Electrolytes: Includes Chloride, Potassium, Sodium, and Carbon Dioxide.
• Thyroid w/TSH: Includes T-3 Uptake, Total T4, (Free thyroxine index)T7, and TSH.
• Liver: Includes Albumin, Alkaline Phosphatase, Alanine Transaminase (ALT or SGPT), Aspartate Transaminase (AST or SGOT), Total Bilirubin, Total Protein, LDH, Total Globulin, Albumin/Globulin Ratio, and GGT.
• Kidney: Includes Blood Urea Nitrogen (BUN), Creatinine, BUN/Creatinine Ratio, eGFR, and Uric Acid.
• Glucose (Diabetes);
• Mineral and Bone: Total Iron, Calcium, and Phosphorus.

Fibrinogen, Quantitive: Fibrinogen is used to detect suspected bleeding disorders or abnormal blood clotting. Fibrinogen is often significantly increased in conditions involving tissue damage, infection, or inflammation. Increased levels may be seen in smokers, during pregnancy, and in women taking oral contraceptives. Fibrinogen levels can be diminished in advanced liver disease.

C-Reactive Protein, hs (CRP, hs): CRP, hs is a critical component of the immune system and can be predictive of future risk of heart attack, stroke, sudden cardiac death, and the development of peripheral arterial disease. Individuals with elevated levels of CRP have a risk about 2 to 3 times higher than the risk of those with low levels.

Homocysteine: Homocysteine is an amino acid that plays a role in destroying the lining of your artery walls, promoting the formation of blood clots, and also accelerates the buildup of scar tissue. High levels may increase the chance of heart disease and stroke, especially if you have other risk factors such as diabetes, high blood pressure, obesity, smoking, or family history.

Hemoglobin A1c: This non-fasting test, also known as A1c, HbA1c, Glycohemoglobin, or Glycated hemoglobin, indicates how well you have controlled your diabetes over the last few months. Even though you may have some very high or very low blood glucose values, Hemoglobin A1C will give you a picture of the average amount of glucose in your blood over that time period. While the Hemoglobin A1C is the standard tool to determine blood sugar control for patients with diabetes, it is not a substitute for daily, routine blood glucose testing.

DHEA-s: DHEA-S serves as a building block for making the male sex hormone testosterone and the female sex hormone estrogen. DHEA-s concentrations peak after puberty and then the levels tend to decline with age. In women, too much DHEA-s can lead to excessive hair growth or male body characteristics as well as adrenal tumors, cancers, and adrenal hyperplasia. Under production can be an indication of Addison’s disease or adrenal hypoplasia.

Total Testosterone: Testosterone is a hormone that causes male characteristics. The blood level is used by men to investigate abnormal sexual development and sexual dysfunction. Small amounts are produced in women’s ovaries and levels are tested to evaluate virilization.

Testosterone, Free: Testosterone, Free is the free flowing testosterone that is not bound within the body and only accounts for about 2% of all testosterone. The free flowing testosterone is unencumbered in the charging of testosterone related systems, the brain, muscles, blood and sex drive, while two-thirds of bound testosterone is connected the SHGB (Sex Hormone Binding Globulin) and the other third to albumin. Low free testosterone means testosterone related systems are not getting the necessary charge. When testosterone free decreases the SHGB increases, this can inhibit the way in which testosterone is distributed to bodily systems.

Estradiol: Estradiol, also known as E2, is the most active of the estrogens. For women, it is important to look at the relationship between estradiol and progesterone in evaluating menopausal symptoms such as hot flashes, mood disorders, and aging skin. It is also used for monitoring pregnancy.

In both men and women, low levels of estradiol can be associated with osteoporosis.

PSA: The prostate specific antigen (PSA) is a protein made only in the prostate gland. PSA is produced by normal, abnormal and cancerous prostatic tissue. The PSA blood test is an accurate measure of this amount. The theory is that cancer causes more of the protein to be made and leaked into the blood than normal prostate tissue, so PSA is now used for assisting in the diagnosis and monitoring of prostatic carcinoma.

Best of all, the cost of tests are covered by my insurance company since my policy does provide for preventative health care.

Diet and Exercise:

As for exercise, my routine is heavily focused on Escalation Density Training (EDT) and I am only now starting to incorporate Yoga into my schedule. I plan to dedicate an entire post to EDT in the very near future so will not discuss it now.

The map is not the territory. – Alfred Korzybsky in Science and Sanity, 1933

This is the second post in a series where I’ll discuss various aspects of parallel and distributed system design. I’d like to provide a brief introduction to taxonomy (or classification of data) and eventual consistency as these building blocks are fundamental to parallel and distributed system design.

Extending my thoughts from a post I made some time ago , in computer science and information modeling, an ontology formally represents knowledge as a set of concepts within a domain, and the relationships among those concepts. It can be used to reason about the entities within that domain and may be used to describe the domain. Why do we need to know this? Because all parallel and distributed systems require data to operate and, in turn, operate on data.

Semantic Ontologies

A semantic ontology, or ontology, is a formal explicit description of concepts in a domain of discourse, properties of each concept describing various features and attributes of the concept, and restrictions on properties. An ontology together with a set of individual instances of classes [or objects] constitutes a knowledge base. In reality, there is a very fine line where the ontology ends and the knowledge base begins but let’s put that aside for a moment.

An implementation of conceptual ontologies is the Semantic Web [ URI (opens in a new window) ]. Simply put, the Semantic Web is the representation of data on the web in which information is given well-defined meaning—“to be a universal medium for the exchange of data”.

The principal technologies of the Semantic Web fit into a set of layered specifications called the Resource Description Framework (opens in a new window) (RDF). The current components of that framework are the RDF Core Model, the RDF Vocabulary Description Language and the Web Ontology Language ( OWL (opens in a new window) ). OWL is a descriptive layer built on top of RDF used to model classes, properties, and objects. These languages all build on the foundation of URIs, XML, and XML namespaces.

Classes are the focus of most semantic ontologies. Outlined in most ontology research are the following observations:

• There is no one correct way to model a domain— there are always viable alternatives. The best solution almost always depends on the application that you have in mind and the extensions that you anticipate.
• Ontology development is necessarily an iterative process.
• Concepts in the ontology should be close to objects (physical or logical) and relationships in your domain of interest. These are most likely to be nouns (objects) or verbs (relationships) in sentences that describe your domain.

Solution domains require, at times, different classifications. Regardless of which classification the solution domain requires, the following rules and considerations have proven effective in designing an efficient information model:

• Effectively understand the information.
• Describe it unambiguously.
• Enforce structure and style guidelines.
• Allow for efficient storage and retrieval of information.
• Keep network communication to a minimum. Don’t over engineer.

I cannot overstate the need to understand your information model but always with an understanding of your delivery time frames. All too many times, we are caught up in the perfect model only in the end to take shortcuts to satisfy deadlines. The solution is acceptance and understanding that change is an integral part of any information model. Temporal and transient taxonomies may be an integral part of your solution domain. Semantic ontologies may be created at runtime to address a special-purpose need where the need itself is transient in nature. For example, in order to fulfill a set of calculations with a high time and computational complexity rating a transient taxonomy might need to be created to provide global access to intermediate calculation results. This is all too common in financial simulation.

Not surprising is that classes are the focus of most ontologies. So with that, let’s move on to a more usual and enjoyable example of a semantic ontology: wine. Wine is a potable liquid produced by at least one maker of type winery, and is made from at least one type of grape. For example, a class of Wine represents all wines. Specific wines are instances of this class. The Bordeaux wine in the glass in front of you while you read this post is an instance of the class of Bordeaux wine. A class can have subclasses that represent concepts that are more specific than the superclass. For example, we can divide the class of all wines into red, white, and rose wines. Alternatively, we can divide a class of all wines into sparkling and non-sparkling wines.

We can further describe properties of classes and instances: Chateau Lafite Rothschild Pauillac wine has a full body; it is produced by the Chateau Lafite Rothschild winery. We can have two properties describing the wine, the property body with the value full and the property maker with the value Chateau Lafite Rothschild winery. At the class level, we can say that instances of the class Wine will have properties describing their flavor, body, sugar level, the maker of the wine and so on.

All instances of the class Wine, and its subclass Pauillac, have a property maker the value of which is an instance of the class Winery. All instances of the class Winery have a property produces that refers to all the wines (instances of the class Wine and its subclasses) that the winery produces. The entire example OWL ontology derived for Wine can be found here (opens in a new window) . I am of course a huge fan.

The strength of your application requires you to understand the data on which it operates—unambiguosly. Though perfection is not key, it is second to paramount. Once you fully understand your information model, then you are able to understand eventual consistency for your application. And with this in mind, let’s discuss eventual consistency.

Eventual Consistency

Eventual consistency is one of the consistency models used in the domain of parallel programming. It means that given a sufficiently long period of time over which no changes are sent, all updates can be expected to propagate eventually through the system and all the replicas will be consistent. I will remind you from a previous blog entry there is no way to guarantee complete knowledge of the current or future state of a distributed system, because knowledge of state changes must be propagated and propagation takes time, during which more state changes may occur. This is an axiom of distributed computing. So how can we begin to reconcile eventual consistency with this axiom? Well, we don’t really [ URI (opens in a new window) ]. We design for loose coupling. We design for tolerance. We design for symmetry of algorithms.

At a high level, most mission- and safety-critical distributed applications SHOULD (opens in a new window) strive to achieve these high-level technical requirements:

• Provide high availability read and write access to information.
• Minimize network communication.
• Provide a highly available and fault tolerant system that can support an annual uptime guarantee of 99.999 percent , which is equal to 5.256 minutes of annual unscheduled downtime.
• Provide instrumentation to aid implementers and customers in their hardware sizing estimates. Scalability MUST (opens in a new window) be measurable.
• Devise a software architecture that scales up by a factor of 10^6. That is, an application’s storage and processing capacity can automatically grow by a factor of a million, doing jobs faster 10^6x speed up or doing 10^6 larger jobs in the same time 10^6 scale up, just by adding more resources.
• Provide for conflict resolution. Common implementations in this area leverage Lamport timestamps (opens in a new window) and vector clocks (opens in a new window) .
• Provide for transient node failure.

While the goal is to devise a software architecture that scales up without limits, there has to be some kind of limit: billions of dollars, or giga watts, or just space. So, the more realistic goal is to be able to scale from one node to a million nodes all working on the same problem or same set of problems.

While fundamentally a Distributed Hash Table (DHT) (opens in a new window) is generally well suited for specific classes of decentralized distributed systems, it is certainly not well suited for all. For example, a DHT is well suited for problems where a highly reliable network between nodes is usual. But a DHT is not necessarily well suited for problem domains where frequent node joins and leaves are the norm. That said, DHTs have been used for routing in many Peer-to-Peer (P2P) implementations [ URI ]. Even some of the more well known NoSQL implementations leverage DHTs.

But as I mentioned, DHTs are not necessarily well suited for all implementations. When we designed the core replication algorithms for the Microsoft Active Directory (AD) service, a DHT was not chosen. Not only did we want to account for Byzantine (opens in a new window) failures but we also needed to account for replica failures that lasted for long periods of time (30+ days). AD subscribes to eventual consistency. Roughly speaking, the replication model of AD is multi-master loose consistency with convergence (eventual consistency). In this model, the directory can have many replicas; a replication system propagates changes made at any given replica to all other replicas. The replicas are not guaranteed to be consistent with each other at any particular time (“loose consistency”), because changes can be applied to any replica at any time (“multi-master”). If the system is allowed to reach a steady state, in which no new updates are occurring and all previous updates have been completely replicated, all replicas are guaranteed to converge on the same set of values (“convergence or eventual consistency”).

In addition to the technical requirements listed earlier in this post, we imposed the following requirements on the design of AD replication:

• To adapt to customer networks, provide flexibility in replication topology including choice of transports.
• Provide a fully asynchronous and highly scalable architecture.
• Minimize network communication in terms of size and round-trips, support encryption of data, and support transitive (store/forward) transportation of data.
• Work well over high-latency communication links.
• Operate correctly when encountering changes to the distinguished name (DN) of an object and to discriminate between a deleted object and a new object with the same DN. In other words, we replicate information based on object’s universally unique identifier ( UUID (opens in a new window) ) and not based on DN.
• Automatic system topology generation. The system automatically creates (and destroys) “short cut” replication links between nodes in order to maintain a specific number of hops between any two points.

Active Directory uses a state-based approach to replication. This approach is easier to appreciate in contrast with an alternative, log-based replication. In a typical log-based system each master keeps a log of the updates that it originated. The goal of each master is to communicate its log to every other replica. Once a log arrives at a replica, the replica applies the log, bringing its state more up to date.

In state-based replication each master applies updates, both originating and replicated, to its replica as they arrive. As such, replication is not driven from logs stored with the source replica but from current state of the source replica. This state includes information for resolving conflicts (also needed in the log-based approach) and information to avoid sending the full replica on each replication cycle (inherent in the log-based approach.) A state-based approach uses a single mechanism for incremental and full sync, and performs fewer database updates since repeated or conflicting updates to an attribute are collapsed into a single state. The resulting design has several stability-enhancing properties, such as:

• No matter how long two replication partners have been out of communication, they can always build upon any previous replication they have accomplished. They never have to start over from scratch because somebody’s log “wrapped around.”
• If a replica has a “hot spot” attribute that’s being updated frequently for some reason (perhaps a bug in an application or an implementation similar to a performance counter), only the value that’s current at the time of replication is sent to a replication partner. Some designs send the full history of all intermediate values thereby multiplying the problem.
• The very same mechanism is used to initialize a new replica as is used for bringing a replica up to date with recent changes. Some other designs require two mechanisms.
• AD has a simple and robust solution to the “time went backwards due to restore from backup” problem (also called the “back sync” problem) that plagues some other systems.
• In a replication relationship the destination (i.e. the replica being updated) takes all responsibility for keeping track of how up to date it is. The source (i.e. the replica supplying the updates) takes no responsibility. This design choice avoids a lot of complexity and potential sources of failure; some other systems don’t work this way.
• In a replication relationship the destination always “pulls” changes from the source. The source may notify the destination “now would be a good time to pull,” but if the notification is lost (e.g. because the destination is overloaded or down) the result is longer replication latency, not incorrectness.
• When two replicas establish a new replication relationship, that relationship is established in an incremental fashion, so not much work is lost should one replica go down before the relationship becomes complete.

Impact on Distributed Applications

A multi-master distributed system induces several problems on applications.

• Version Skew. Version skew occurs when applications read the same object(s) from different replicas before a change has replicated. Applications reading the remote replica see the unchanged object. Version skew is an issue when a given application or set of applications use the information in the directory to interoperate.
• Partial Updates. Partial Update occurs when applications read the same set of objects from different replicas while replication is in progress. Applications at the remote replica see some of the changes but not all. Note that there is a small window in which partial update can affect an application: the application must start reading objects while inbound replication is in progress, after one or more of the related, changed objects have been received but before all have been received. The time between the updates at the source replica directly affects the size of this window—updates that occur close together in time will be replicated close together in time. Partial update is an issue when an application uses a related set of objects.
• Collisions. Collisions occur when the same properties of two or more replicas of a given object are changed during the same replication interval. The replication process reconciles the collision; because of reconciliation a user or application may “see” a value other than the one they wrote. A simple example is user address information—if a user changes their mailing address at replica R-a and an administrator changes the same mailing address at replica R-b, the value ultimately propagated to (R-a, R-b), and all other replicas will be the value selected by the collision reconciliation mechanism. Collision resolution is an issue for applications that make assumptions about the internal consistency of objects or sets of objects.

Though not in scope of this post, applications must accommodate replication latency. The best way to accommodate replication latency is to design applications to minimize the effects—to tolerate latency. The ideal distributed application is of course unaffected by replication latency induced state. Other applications must adopt an avoidance or detection strategy as a mechanism to tolerate replication latency.

Many NoSQL systems, such as MongoDB, employ a master/slave topology and so writes are accepted on a single node. Even the newer replica sets in MongoDB adhere to this model. Sharding (or partitioning), of course, is a key consideration in these types of systems. Sharding enables horizontal scaling across multiple nodes. A sharded MongoDB cluster performs automated leader election for the primary node. Leader election in a ring is another topic in and of itself. So let’s wrap this up…

Wrapping it all up…

Regardless of the underlying durable storage technology, it is of paramount responsibility that you properly classify your information model, understand your consistency model, and develop your application to be highly resilient to failures. It is this latter point—being highly resilient to failures—that is not an easy task and that is precisely why I’ll discuss fault tolerance and recovery in my next post.

…since my intention is to write something useful for anyone who understands it, it seemed more suitable to me to search after the effectual truth of the matter, rather than its imagined one. – Niccolo Machiavelli in The Prince, 1532

While conducting a bit of research on scalability and efficiently storing large amounts (opens in a new window) of information, I ran across an abstract (opens in a new window) in the Communications of the ACM written by a friend and former colleague Pat Helland. Pat and I worked at Microsoft together during the late 90s. Our paths naturally crossed given his core research in high performance transaction systems ( HPTS (opens in a new window) ) and my core work in parallel and distributed systems. Pat recently moved back to San Francisco after 15 years with Microsoft [ URI (opens in a new window) ]. For the two years prior to leaving, Pat worked on Cosmos (opens in a new window) , some of the plumbing for Bing (opens in a new window) . It stores hundreds of petabytes of data on tens of thousands of computers.

The paper (opens in a new window) , “If You Have Too Much Data, then ‘Good Enough’ Is Good Enough”, is brilliantly written. Pat has always been a wonderful and capable writer with an ability to reduce complex problems into a simple presentation. Similar to his clear and concise description of application models using concepts known as “fiefdoms” and “emissaries” in 2002 [ URI (opens in a new window) ] [ URI (opens in a new window) ], Pat identifies a world where many of our classic SQL database principles are being eroded by combining too much data each with disparate characteristics.

The non-relational storage technology (a.k.a. “NoSQL”) movement has of course given rise to the likes of MongoDB (opens in a new window) , CouchDB (opens in a new window) , and Redis (opens in a new window) . Each of these NoSQL technologies have their own unique advantage so judicious evaluation is a MUST (opens in a new window) . The vast amounts of information we process each day in turn has forced us to consider alternate ways to process this information else risk not being able to translate this data into actionable information. Applied technologies such as Hadoop (opens in a new window) and MapReduce (opens in a new window) are fundamental to processing “big data”. We often employ practices and technologies to enable timely dissemination, such as data analytics, business performance management, data warehousing, dashboards and key performance indicators (KPIs). Regardless of what we choose, the fundamentals of how we react to this vast amount of information must be swift, decisive, and–more importantly–accurate in today’s global economy. Being capable of pivoting is essential.

Mobile + Cloud

Mobile and cloud technologies are a winning distributed system solution for many problem domains. But let’s not forget this combination is a distributed system. When designing and implementing distributed systems and the data models on which a distributed system operates, organization is indeed paramount; and so is synchronization. For safety- and mission-critical applications, high availability has always been a paramount concern and recent experience with large Internet sites has underscored the need for availability in that domain as well. Traditional approaches to the problem have made three implicit assumptions:

1. Failure rates of hardware and software are low and improving.
2. Systems can be modeled for reliability analysis and their failure modes can be predicted.
3. Human error during maintenance is not a major source of failures.

The result is an emphasis on failure avoidance as the path to high availability. These assumptions are in many cases based on incorrect perceptions of today’s environment, and that renewed emphasis should be given to failure recovery. Even the most highly tested systems occasionally exhibit “ Heisenbugs (opens in a new window) ” and suffer from transient or permanent hardware failure and software aging, and human error has empirically been found to account for a nontrivial fraction of catastrophic failures. The most successful systems have been those that can recover from these unexpected errors because they were designed for recovery.

If a problem has no solution, it may not be a problem, but a fact—not to be solved, but to be coped with over time. – Shimon Peres, 1923

There is no way to guarantee complete knowledge of the current or future state of a distributed system, because knowledge of state changes must be propagated and propagation takes time, during which more state changes may occur. This is an axiom of distributed computing.

Tightly coupled systems deal with uncertainty by attempting to eliminate it. This is done through constraints on updates, requiring all nodes or some majority of nodes to be available before updates can be performed; using distributed locking schemes or single mastering for critical resources, constraining all nodes to be well connected, or some combination of these techniques. “Majority” often involves weighted voting schemes; so “big” nodes can be more influential than “small” nodes. The more tightly coupled the computing nodes in a distributed system are, the lower the scaling limit.

Loosely coupled systems deal with uncertainty by tolerating it. A loosely coupled system allows participating nodes to have differing views of the overall system state and provides algorithms for resolving conflicts.

My choice for parallel and distributed system design is loosely coupled because:

1. Customers require a highly distributed solution in which parts of the infrastructure can be spread across the internal and public networks and administered locally.
2. Large customers need to grow in capacity to many millions of transactions per day or to hundreds or thousands of nodes, or both.
3. Many networks provide only intermittent connectivity to some locations, for example remote oil drilling platforms and ships at sea, so the system MUST (opens in a new window) be tolerant of partly connected or disconnected operation.

Tightly coupled solutions are unsuitable for parallel and distributed system design because of the requirements for scalability to a very large numbers of nodes and disconnected operation. The loosely coupled model satisfies all of the above requirements.

I’d like to quickly point out an observation regarding tolerance as it relates to most Internet-aware software. As the World Wide Web Consortium ( W3C (opens in a new window) ) points out with respect to tolerance, the principle of tolerance does not blunt the need for a perfectly clear specification which draws a precise distinction between conformance and non-conformance. The principle of tolerance is no excuse for a product which contravenes a standard. Naturally, the aphorism “any problem in computer science can be solved with another level of indirection” still rings true but to a varying degree. The quantification of that degree is a difficult balance to achieve as it involves many variables with the most notable, in my opinion, being scalability, flexibility, and reliability.

I plan to discuss the merits of loose coupling in the larger fabric of parallel and distributed system design more concretely over the next several posts.

Additionally, iOS 5 introduces a new compile-time memory management feature known as Automatic Reference Counting (ARC) (opens in a new window) . The abridged version is you will never need to type retain or release again, which dramatically simplifies the development process while reducing crashes and memory leaks. The compiler has a complete understanding of your objects, and releases each object the instant it is no longer used, so apps execute much faster, with predictable, smooth performance. However, it’ll take the community some time to refactor various open source libraries to support ARC. Therefore, we have a much larger need today than ever before to move these source files into static libraries that are compiled with ARC disabled so we can compile our primary applications with ARC enabled.

Alternatively, if you are including files that don’t yet support ARC in a project that is ARC enabled, you can set the -fno-objc-arc compiler flag for each of these files. To do this in Xcode, go to your active target and select the “Build Phases” tab. In the “Compiler Flags” column, set -fno-objc-arc for each of the source files. Of course, this can be a very laborious process.

Apple defines a framework (opens in a new window) as a bundle (a structured directory) that contains a dynamic shared library along with associated resources, such as nib files, image files, and header files. When you develop an application, your project links to one or more frameworks. For example, iOS application projects link by default to the Foundation, UIKit, and Core Graphics frameworks. Your code accesses the capabilities of a framework through the application programming interface (API), which is published by the framework through its header files. Because the library is dynamically shared, multiple applications can access the framework code and resources simultaneously. The system loads the code and resources of a framework into memory, as needed, and shares the one copy of a resource among all applications. A universal (or multi-architecture) file is nothing more than an application bundle.

As I mentioned above, iOS does not support dynamic shared libraries but using lipo (opens in a new window) (a tool which comes with iOS SDK (opens in a new window) ) we are able to merge several static libraries into a single static library. By using lipo, we are able to create a static library that supports all architectures (armv6, armv7, and i386) packaged as a universal framework. Here’s how.

Within Xcode, either create a new iOS project or open an existing iOS project. If you do decide to create a new project then feel free to choose any application template you prefer. Which one you choose is irrelevant. Add a “Cocoa Bundle” target (not Cocoa Touch Bundle). Select the newly created bundle target in the left navigation panel, select the “Build Settings” tab and modify the following settings (opens in a new window) as required:

• Architectures (ARCHS): Standard (armv6 armv7 armv7s). In Xcode 4.2, use the value of $(ARCHS_STANDARD_32_BIT). If you wish to compile for older devices then add a new line with a value of armv6. Please see this (opens in a new window) Apple Developer forum post for more details.
• Base SDK (SDKROOT). Latest iOS (iOS X.X).
• Build Active Architecture Only (ONLY_ACTIVE_ARCH). No. This allows us to compile for armv6 and armv7.
• Valid Architecture (VALID_ARCHS): Standard (armv6 armv7 armv7s). In Xcode 4.2 and greater, use the value of $(ARCHS_STANDARD_32_BIT). If you wish to compile for older devices then add a new line with a value of armv6.
• Dead Code Stripping (DEAD_CODE_STRIPPING): No
• Link With Standard Libraries (LINK_WITH_STANDARD_LIBRARIES): No
• Mach-O Type (MACH_O_TYPE): Relocatable Object File. This is the most important change. Here, we instruct the compiler to treat the Bundle as a relocatable file, by doing this, we can turn it into a framework with the wrapper setting.
• Wrapper Extension (WRAPPER_EXTENSION). framework. Here we change the Bundle to a Framework. To Xcode, frameworks is just a folder with the extension .framework, which has inside one or more compiled binary sources, resources and some folders, a folder, usually called Headers, contains all the public headers.
• Generate Debug Symbols (GCC_GENERATE_DEBUGGING_SYMBOLS): No.
• Generate Position-Dependent Code (GCC_DYNAMIC_NO_PIC): No
• Targeted Device Family (TARGETED_DEVICE_FAMILY). iPhone/iPad.

Here are the steps…

• With the bundle target still in focus, select the “Info” tab and ensure the “Bundle OS Type code” is equal to FMWK.
• Add any source code and resources to the bundle. With the bundle target selected, click the “Build Phase” tab. At the bottom, press the “Add Phase” button and then “Add Copy Headers“.
• Open the newly created “Copy Headers” section and separate your public headers from private or project headers.
• Open the “Compile Source” section and add any .m, .c, .mm, .cpp and any other compilable source file. If your framework contains any non-compilable files such as images, sounds, and other resources, you can add them to the “Copy Bundle Resources” section.

You can access any non-compilable resources by using NSBundle.

[[NSBundle mainBundle] pathForResource:@"MyFramework.framework/Resources/FileName" ofType:@"fileExtension"];

Next we’ll create a new Aggregate Target. As mentioned earlier, to join both architectures products into one, we must to use lipo.

• Add a new target by pressing the “Add Target” button.
• Under the Cocoa Touch section, select the “Other” subcategory and then choose the “Aggregate” target. The “Product Name” is arbitrary.
• Add a new “Run Script” phase under this newly created target. Copy and paste the following script into the “Run Script” phase.

# The framework name and version
X_FRAMEWORK_NAME=CoreFramework
X_FRAMEWORK_VERSION=A

# This folder contains the final output of the framework.
X_INSTALL_DIR=${SRCROOT}/Products/${X_FRAMEWORK_NAME}.framework

# This working directory will be deleted after completion.
X_WORKING_DIR=build
X_DEVICE_DIR=${X_WORKING_DIR}/${CONFIGURATION}-iphoneos/${X_FRAMEWORK_NAME}.framework
X_SIMULATOR_DIR=${X_WORKING_DIR}/${CONFIGURATION}-iphonesimulator/${X_FRAMEWORK_NAME}.framework

echo "******************************************************"
echo "X_DEVICE_DIR = ${X_DEVICE_DIR}"
echo "X_SIMULATOR_DIR = ${X_SIMULATOR_DIR}"
echo "******************************************************"
echo "SYMROOT = ${SYMROOT}"
echo "OBJROOT = ${OBJROOT}"
echo "PROJECT_DIR = ${PROJECT_DIR}"
echo "CONFIGURATION_BUILD_DIR = ${CONFIGURATION_BUILD_DIR}"
echo "CONFIGURATION = ${CONFIGURATION}"
echo "CONFIGURATION_TEMP_DIR = ${CONFIGURATION_TEMP_DIR}"
echo "DERIVED_FILE_DIR = ${DERIVED_FILE_DIR}"
echo "BUILD_PRODUCTS_DIR = ${BUILT_PRODUCTS_DIR}"
echo "BUILD_DIR = ${BUILD_DIR}"
echo "TARGET_TEMP_DIR = ${TARGET_TEMP_DIR}"
echo "PROJECT_TEMP_DIR = ${PROJECT_TEMP_DIR}"
echo "PRODUCT_NAME = ${PRODUCT_NAME}"
echo "******************************************************"

# Build both simulator and device architectures.
xcodebuild clean
xcodebuild -configuration ${CONFIGURATION} -target "${X_FRAMEWORK_NAME}" -sdk iphoneos -SYMROOT=${SYMROOT} -OBJROOT=${OBJROOT}
xcodebuild -configuration ${CONFIGURATION} -target "${X_FRAMEWORK_NAME}" -sdk iphonesimulator -SYMROOT=${SYMROOT} -OBJROOT=${OBJROOT}

# Clean the oldest.
if [ -d "${X_INSTALL_DIR}" ]
then
rm -rf "${X_INSTALL_DIR}"
fi

# Recreate the folder structure for the final product binaries.
mkdir -p "${X_INSTALL_DIR}"
mkdir -p "${X_INSTALL_DIR}/Versions"
mkdir -p "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}"
mkdir -p "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}/Resources"
mkdir -p "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}/Headers"

# Create the required symbolic links. Please note the paths MUST relative, 
# otherwise the symbolic links will be invalid when the folder is copied/moved.
ln -s "${X_FRAMEWORK_VERSION}" "${X_INSTALL_DIR}/Versions/Current"
ln -s "Versions/Current/Headers" "${X_INSTALL_DIR}/Headers"
ln -s "Versions/Current/Resources" "${X_INSTALL_DIR}/Resources"
ln -s "Versions/Current/${X_FRAMEWORK_NAME}" "${X_INSTALL_DIR}/${X_FRAMEWORK_NAME}"

# Copy the headers and resources files to the final product folder.
cp -R "${X_DEVICE_DIR}/Headers/" "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}/Headers/"
cp -R "${X_DEVICE_DIR}/" "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}/Resources/"

# Remove artifacts from the resources folder.
rm -r "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}/Resources/Headers" "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}/Resources/${X_FRAMEWORK_NAME}"

# Use lipo to merge both binary files (i386 + armv6/armv7) into one universal files.
lipo -create "${X_DEVICE_DIR}/${X_FRAMEWORK_NAME}" "${X_SIMULATOR_DIR}/${X_FRAMEWORK_NAME}" -output "${X_INSTALL_DIR}/Versions/${X_FRAMEWORK_VERSION}/${X_FRAMEWORK_NAME}"

# Remove the working directory
rm -r "${X_WORKING_DIR}"

You’ll need to modify the first non-comment line in the script to be equal to the product name of your framework bundle target.

Now build the Aggregate target. It does not matter which architecture you select to build (iOS Device or Simulator) since the script creates a working folder, compiles the framework target twice (once for the iOS device and once for the iOS Simulator) and generates the output to a folder named “Products” located in the $(SRCROOT) root folder

If you previously upgraded from Xcode 3.x to 4.2, then your existing (SYMROOT) value will very likely cause the above script to fail. The failure is caused because the script expects your (SYMROOT) value to be set to the new Xcode 4 value of “build”, which refers to a relative path to $(SRCROOT). Unfortunately, you are not able to modify this setting in Xcode 4 due to an already reported bug. To work around this bug, close Xcode and simply delete the existing SYMROOT key and value from your com.apple.dt.Xcode.plist file, which is located in ~/Library/Preferences/.

Isn’t it about time for the WWDC to be here again?

I suggest reading Apple’s Thread Programming Guide (opens in a new window) as a prerequisite, which also covers some of the basic concepts behind Blocks and Grand Central Dispatch (GCD) (opens in a new window) .

Singletons

For purposes of this discussion, a singleton is a class that only allows a single instance of itself to be created within an operating system process.

Apple’s documentation (opens in a new window) on creating a singleton recommends a pattern that works for most iOS applications but is not thread-safe. Beginning with iOS 2.0 and prior to iOS 4.1, one of the most widely used methods to introduce a thread-safe singleton was through the use of the @synchronized (opens in a new window) directive.

+ (MyClass*) instance {
    static MyClass *gInstance = nil;
    @synchronized(self) {
        if(nil == gInstance) {
            gInstance = [[MyClass alloc] init];
        }
    }
    return (gInstance);
}

There are a few inefficiencies with this approach because not only generates a recursive mutex lock but also introduces an exception handler. Specifically, Apple’s documentation indicates that as a precautionary measure, the @synchronized block implicitly adds an exception handler to the protected code. This handler automatically releases the mutex in the event that an exception is thrown. This means that in order to use the @synchronized directive, you must also enable Objective-C exception handling in your code. If you do not want the additional overhead caused by the implicit exception handler, you should consider using the lock classes.

At the time of this writing, the @synchronized directive turns into this basic pseudo-code:

id _eval_once = ;
objc_sync_enter( _eval_once );
@try {
    /* code goes here */
}
@finally {
    objc_sync_exit( _eval_once );
}

For implementation details see: http://www.opensource.apple.com/source/objc4/objc4-437.1/runtime/objc-sync.m (opens in a new window)

Much of Apple’s sample code prior to iOS 4.0 use this approach. There is a technically more efficient and lock-free approach for applications that target operating systems earlier than Mac OS X version 10.5 or iOS 4.0.

+ (MyClass*) gInstance {
    static void * volatile gInstance = nil;
    while (!gInstance) {
        MyClass *temp = [MyClass [alloc] init];
        if(!OSAtomicCompareAndSwapPtrBarrier(0x0, temp, &gInstance)) {
            [temp release];
        }
    }
    return (gInstance);
}

You can read more about preferred versions of the atomic and synchronization operations here (opens in a new window) .

For applications that target operating systems equal to or greater than Mac OS X 10.6 or iOS 4.0, the recommended singleton pattern uses dispatch_once, which relies on components of Grand Central Dispatch (GCD).

+ (MyClass *) instance {
    static MyClass* gInstance = nil;
    static dispatch_once_t pred;
 
    dispatch_once(&pred, ^{
        gInstance = [[MyClass alloc] init];
    });
    return (gInstance);
}

It should go without saying but I’ll do so for completeness. All instance-level methods of classes that implement a singleton pattern MUST (opens in a new window) be thread-safe.

NSCache

I suspect this class remains relatively unused by most iOS developers. NSCache (opens in a new window) is essentially a container that it stores key-value pairs but unfortunately does so without an O(1) time complexity (like that of NSMutableDictionary (opens in a new window) ) but does automatically evicts objects from its store when the ‘cost’ (a heuristic that of course involves memory pressure) of the cache rises above a configurable threshold.

Per Apple’s documentation, NSCache objects differ from other mutable collections in a few ways:

• The NSCache class incorporates various auto-removal policies, which ensure that it does not use too much of the system’s memory. The system automatically carries out these policies if memory is needed by other applications. When invoked, these policies remove some items from the cache, minimizing its memory footprint.
• You can add, remove, and query items in the cache from different threads without having to lock the cache yourself.
• Retrieving something from an NSCache object returns an autoreleased result.
• Unlike an NSMutableDictionary object, a cache does not copy the key objects that are put into it.These features are necessary for the NSCache class, as the cache may decide to automatically mutate itself asynchronously behind the scenes if it is called to free up memory.

Though not required, NSCache works in conjunction with objects that implement the NSDiscardableContent (opens in a new window) protocol in that the discardContentIfPossible (opens in a new window) method is called when an object is removed.

The two most widely used caching strategies are proactive and reactive loading.

1. Proactive Cache Loading. A proactive cache loading strategy attempts to retrieve all required state when a process or application starts and cache it for the lifetime of the process or application. A decision to use a proactive cache loading strategy is generally done so in conjunction with either asynchronous pull loading based on expected not actual usage of the information being cached or notification-based loading whereby an application’s services are notified when cached state changes.
2. Reactive Cache Loading. A reactive cache loading strategy retrieves data as it is requested by the application and caches it for future requests. A decision to use a reactive cache loading strategy is generally done so in conjunction with synchronous pull loading since the pattern is relatively easy to test and implement.

Regardless of the selected caching strategy, determining a cache expiration policy is is key for most applications. There are a plethora of expiration policy patterns but the majority of them fall into the following general categories:

• Time-based. Cached information is invalidated based on relative (sliding window) or absolute time periods.
• Notification-based. Cached information is invalidated based on instructions from a source.
• Counter-based. Cached information is invalidated based on a reference counts.

Regardless of the expiration policy, scavenging based on memory pressure and other heuristics are key to a proper cache design and symmetry of that design is a key component in any distributed algorithm.

For most applications, a reactive cache loading strategy with a synchronous pull model is the most appropriate option. NSCache does evict objects based on memory pressure and so I strongly recommend you implement the NSDiscardableContent protocol for any objects you plan to store in NSCache. NSDiscardableContent is a simple counter-based pattern. For information stored in NSCache that represents NSData objects, Apple provides the NSPurgeableData (opens in a new window) class. NSPurgeable data inherits from NSMutableData and is available in iOS 4.0 and Mac OS X 10.6 and later. A description of how to use this class can be found be reading Caching and Purgeable Memory (opens in a new window) .

I’d like to make one last point. Please be careful not to over-engineer your caching strategy but be elegant about your design.

Here are a few very interesting statistics at the time of this writing:

• There are over 225 million iOS devices worldwide.
• Apple represents a 48% share of the tablet and mobile market.
• 25 million iPads have been sold to date.
• There are 425,000 applications in the App Store with 90,000 of those applications specifically designed to run on the iPad.
• The App Store has enjoyed over 14 billion downloads
• Apple has 225 million credit card accounts.
• Apple has shipped 54 million Macs and growing, which is a 28% growth over last year. To contrast this, PC growth is at a negative 1% growth over the same year.
• 73% of the Macs shipped today are notebooks

These statistics are astounding. I am of course not at liberty to discuss any information not made public by Apple regarding the upcoming technologies but it is sufficed to say Mac OS X Lion + iCloud (opens in a new window) + iOS 5 (opens in a new window) represents another leap in market share for Apple and a set of very useful features for consumers.

Make no mistake, software is art. Some artists prefer oil on canvas as their medium. Others choose water color. As software design engineers (SDEs), we have chosen 1s and 0s to communicate our creative visions. Elegance must exist at each step in the engineering process and not only as an afterthought in the development phase. Elegance is the responsibility of each and every participant with accountability functioning as the enforcement point. Without accountability, chaos reigns. Accountability starts within.

Many software engineers find themselves searching for or memorizing patterns that “work” to shortly find themselves bulimically spewing the results onto their canvas. What happened to the elegance of originality? It’s far too easy to become a GP (“Google programmer”). Why? Because creativity and originality aren’t as convenient as letting others do the work, which leads to chaos and hard-to-maintain code; especially, when you’re not the original developer.

Stay with me; we’re only scratching the surface. Do you remember the last time you really evaluated the usability of your consumer API surface? Did your evaluation begin in the design phase? Start with a high-level view of the scenarios. Then oscillate between architectural design and the scenario-based consumer API surface. Do not think about technical design until you’re satisfied with the consumer API surface and the architectural design. Keep in mind; it’s certainly acceptable to refactor your architectural design and consumer API based on the technical design.

An apparent and even larger obstacle I’ve noticed lately is the inability for an SDE to make the jump from procedure (function)-oriented design concepts to that of object-oriented design while keeping an eye to efficiency in terms computational complexity and network round-trips. The devil is in the details. In my opinion, one can’t be an effective architect without being an even more effective SDE; period. The implementation details do change the architecture.

In general, people tend to migrate toward what makes them feel good. Rework doesn’t feel good; no one enjoys it. But, if that was true—if you truly don’t enjoy rework—then why would you not elegantly architect, design, and develop your code initially? Why do you make more work for yourself? Elegance as a foundational element is difficult. More often than not, SDEs take the easy way out because they equate efficiency to speed of completion. I’m not advocating “analysis paralysis” but instead a balanced design. Deadlines must be met. Deadlines should not be met at the expense of elegance.

It’s your canvas.

” It has been said that art is a collaboration between God and the artist, which works best when the artist contributes as little as possible; so too with designing distributed systems ” –André Gide

Try to imagine the strategic direction of distributed computing frameworks. Just for a moment transition into a state where you think of nothing and yet listen to everything. What key technologies (disruptive or otherwise) have we experienced over recent years?

Notion of Autonomous Computing

Arguably, one of the most well known papers on the subject of autonomic computing (opens in a new window) to be recently released is titled “Autonomic Computing: IBM’s Perspective on the State of Information Technology” [ URI (opens in a new window) ]; authored by Paul Horn (opens in a new window) , Senior Vice President of IBM Research (opens in a new window) . As outlined by Dr. Horn’s manifest, there are many challenges we face over the next decade within Information Technology.

Within the autonomic landscape, each system node must satisfy the following criteria:

• self-identification, self-knowing
• self-(re)configuration
• self-recovery (from perturbations)
• self-protection (security)
• self-learning (including from errors)
• self-regulating (to open standards)
• self-resource-allocation

Notion of Grid Computing

Wouldn’t it be nice to economically provide a highly available and fault tolerant system that can support an annual uptime guarantee of 99.999 percent—which equates to 5.256 minutes of annual unscheduled downtime—and that scales up by a factor of 106? That is, an application’s storage and processing capacity can automatically grow by a factor of a million, doing jobs faster (106x speed up) or doing 106 larger jobs in the same time (106x scale up), just by adding more resources. Stay focused.

The concept of Grid (or Utility) computing was coined in the mid 1990s and is best defined—to quote The Anatomy of the Grid (opens in a new window) —the real and specific problem that underlies the Grid concept is coordinated resource sharing and problem solving in dynamic, multi-institutional virtual organizations.

The Grid Global Forum (GGF (opens in a new window) ) has made great headway; so has OGSA (Open Grid Services Architecture). Message Passing Interface (MPI) (opens in a new window) is widely accepted. Microsoft entered the HPC (opens in a new window) game with the Microsoft Compute Cluster Server (opens in a new window) , which I installed many months ago and is now consuming most of my free time.

Notion of Adaptive Autonomous Agents

An agent is a system that tries to fulfill a set of goals of goals in a complex, dynamic environment. An agent it situated in the environment; it can sense the environment through its sensors and act upon the environment using its actuators. An agent’s goal can take many different forms: they can be “end goals”, or particular states the agent tries to achieve; they can be selective reinforcement or reward that the agent attempts to maximize; they can be internal needs or motivations that the agent has to keep within certain viability zones and so on. An agent is called autonomous if it operates completely autonomously, i.e. if it decides itself how to relate its sensor data to motor commands in such a way that its goals are attended to successfully. An agent it said to be adaptive, if it is able to improve over time, i.e. if the agent becomes better at achieving its goals with experience.

The study of Adaptive Autonomous Agents is grounded in two important insights, which serve as “guiding principles” for most of the current research performed:

• Looking at complete systems changes the problems often in a favorable way.
• Interaction dynamics can lead to emergent complexity.

Essentially, an agent is viewed as a set of competence modules (often called behaviors). These modules are responsible for a particular small task-oriented competence. Each of the modules is directly connected to its relevant sensors and actuators. Modules interface to one another via extremely simple messages rather than a common representation of beliefs, and so on. The communication between modules is almost never of a “broadcast” nature, but happens rather on a point-to-point (or one-to-one) bases. Typically, the messages consist of activation energy, or simple suppression and inhibition signals, or simple tokens in a restricted language. In addition to communication via simple messages, modules also communicate “via the environment”. One module may change some aspect of the environment, which will trigger another module, etc.

Notion of the Semantic Web

Simply put, the Semantic Web (opens in a new window) is the representation of data on the web in which information is given well-defined meaning—“to be a universal medium for the exchange of data”.

The principal technologies of the Semantic Web fit into a set of layered specifications called the Resource Description Framework (RDF) (opens in a new window) . The current components of that framework are the RDF Core Model, the RDF Vocabulary Description Language and the Web Ontology Language, which all build on the foundation of URIs, XML, and XML namespaces.

The most interesting of these languages is the Web Ontology Language (OWL) (opens in a new window) , which is a descriptive layer built on top of RDF used to model classes, properties, and objects.

Ontology is also a term borrowed from philosophy that refers to the science of describing the kinds of entities in the world and how they are related. Stated another way, an ontology defines the terms used to describe and represent an area of knowledge.

Notion of Service-Oriented Architecture

I think everyone has been beaten of the head with the “SOA stick”, which is why you won’t feel a thing; keep reading.

In computing, the term Service-Oriented Architecture (SOA) expresses a software architectural concept that defines the use of services to support the requirements of software users. In a SOA environment, nodes on a network make resources available to other participants in the network as independent services that the participants access in a standardized way. Most definitions of SOA identify the use of Web services (i.e. using SOAP or REST) in its implementation. Here (opens in a new window) and here (opens in a new window) are two links worth reading. However, one can implement SOA using any service-based technology.

The WS* specifications have gained wide adoption but other advances are needed.

One last point as related to service orientation; I caution everyone to not forget about the elegance in design of class libraries as they are the underpinnings of every SOA. Design for the in-process consumer first with an eye to areas of the API surface that might benefit from hosting within an SOA.

Other advances…

There are many others such as advances in peer-to-peer algorithms, discrete-event simulation and the event horizon, social networking, recovery-oriented systems, storage and machine virtualization, bioinformatics (opens in a new window) , biotechnology (opens in a new window) , and quantum computing (opens in a new window) to name but a few. In my mind, the two most important questions to answer are which of these variables are of significant weighting in the equation of strategic direction and which are “noise”? I certainly have my opinion…but then again, you know what those are like.

Most readers and programmers have little patience to read discussions of this length. However, the length (at least to me) seems disproportionate to the importance of the topic. There is so much more to say and even more to ponder. But, before you run off to save the world let me leave you with one additional thought:

“The map is not the territory.” Alfred Korzybsky Science and Sanity, 1933

Let’s begin with formal documentation. The MethodImplOptions.Synchronized option “specifies that the method can be executed by only one thread at a time. Static methods lock on the type, while instance methods lock on the instance. Only one thread can execute in any of the instance functions and only one thread can execute in any of a class’s static functions”.

I’d like to focus on the statement “static methods lock on the type”. Microsoft strongly discourages locking on any public types, or on instances you do not control. This means the common constructs lock(this), lock(typeof(SomeType)), and lock(“myLock”) violate this guideline. If you’re not yet familiar with the reason behind this guideline then please read Joe Duffy’s (opens in a new window) blog entry titled “ Rude unloads and orphaned locks (opens in a new window) “. To provide additional support for Joe’s comments, the formal documentation for MethodImplOptions states “Locking on the instance or on the type, as with the Synchronized flag, is not recommended for public types because code other than your own can take locks on public types and instances. This might cause deadlocks or other synchronization problems.”

To be explicit, the lock(typeof(MyClass)) semantic is exactly what MethodImplOptions.Synchronized implements when adorned on static methods . The lock in this case is across all AppDomains (opens in a new window) in the same operating system process. Consider the following code, which if executed from multiple AppDomains would result in a conflict.

ManualResetEvent e1= new ManualResetEvent(false);
lock (typeof(object)) {
    AppDomain domain = AppDomain.CreateDomain("MyDomain");
    domain.DoCallBack(delegate {
        ThreadPool.QueueUserWorkItem(delegate {
            ManualResetEvent e2= new ManualResetEvent(false);
            lock (typeof(object)) {
                e2.Set();
            }
        });
    });
 
    e1.WaitOne();
}

As a result, you should shy away from using MethodImplOptions.Synchronized for the same reasons you wouldn’t use lock(this) or lock(typeof(SomeClass)).

Additionally, the MethodImplOptions.Synchronized option favors neither readers nor writers. Use of a low-locking technique such as a spinlock (opens in a new window) or the intrinsic ReaderWriterLockSlim (opens in a new window) and Monitor (opens in a new window) classes will perform and scale much better on both single-core and multi-core (CPU) hardware.

The above points are important for several reasons:

1. The scope of a lock is very relevant if a goal of your code is reuse. As a framework writer is it your responsibility to ensure consumers writing applications with differing scalability thresholds and concurrency rates are equally favored; the solution should either work for all consumers or offer options for each category of consumer.
2. If you’re writing code that is to be used in systems where efficient use of system resources (CPU and memory) is paramount then it is your responsibility to provide an implementation that meets this fundamental goal.
3. If you’re executing under the Microsoft SQL Server CLR host and your code takes too long to execute then SQL will force an AppDomain unload, which means (as a specific example), that a finally block wishing to execute Monitor.Exit won’t even be run. As Joe indicates, until you’ve created 4,294,967,295 threads such that the Thread IDs wrap around and the old ID gets assigned to a new thread, and that thread spuriously decides to Exit the Monitor without acquiring it first, your system is going to be locked up for a bit. In other words, deadlocked.
4. The MethodImplAttribute can only be used to adorn (you guessed it) methods.

So how much of a difference can a low-locking technique really make? That obviously depends on the technique. I wrote a parallelized program (using the Microsoft Parallel Extensions for .NET 3.5 (opens in a new window) ) that tests synchronization using MethodImplOptions.Synchronized, Monitor, Interlocked, ReaderWriterLockSlim, and a spin-lock. For those interested, I’ve included my spin-lock implementation to the bottom of this article.

Result Set 1: Write-only access:

• Monitor executed 10,000,000 iterations in 13,372,540 (ticks), which is 747 iterations per tick.
• Interlocked executed 10,000,000 iterations in 18,767,746 (ticks), which is 532 iterations per tick.
• ReaderWriterLockSlim executed 10,000,000 iterations in 18,908,171 (ticks), which is 528 iterations per tick.
• MethodImpl executed 10,000,000 iterations in 22,724,176 (ticks), which is 440 iterations per tick.
• SpinLock executed 10,000,000 iterations in 17,713,122 (ticks), which is 564 iterations per tick.

Result Set 2: Read/write access with an equal read and write frequency:

• Monitor executed 10,000,000 iterations in 23,701,864 (ticks), which is 421 executions iterations per tick.
• Interlocked executed 10,000,000 iterations in 19,864,150 (ticks), which is 503 iterations per tick.
• ReaderWriterLockSlim executed 10,000,000 iterations in 33,751,562 (ticks), which is 296 iterations per tick.
• MethodImpl executed 10,000,000 iterations in 43,442,379 (ticks), which is 230 iterations per tick.
• SpinLock executed 10,000,000 iterations in 20,713,905 (ticks), which is 482 iterations per tick.

Result Sets 3: Read-only access:

• Monitor executed 10,000,000 iterations in 13,810,171 (ticks), which is 724 iterations per tick.
• Interlocked executed 10,000,000 iterations in 17,628,526 (ticks), which is 567 iterations per tick.
• ReaderWriterLockSlim executed 10,000,000 iterations in 20,326,949 (ticks), which is 491 iterations per tick.
• MethodImpl executed 10,000,000 iterations in 34,891,919 (ticks), which is 286 iterations per tick.
• SpinLock executed 10,000,000 iterations in 7,046,085 (ticks), which is 1,419 iterations per tick.

The results are not surprising. Use of the MethodImplOptions.Synchronized option on static methods resulted in a lower concurrency rate. If you require thread synchronization for static methods then I strongly recommend you consider an alternate synchronization technique.

The full source code can be found here .

For this discussion, I’d like to focus on two of the variables: testability and efficient use of system resources.

Garbage Collection

As a prerequisite, I recommend you read this article (opens in a new window) on how garbage collection works at a high-level in the Microsoft .NET Framework. If you don’t have time (opens in a new window) to read the article then do take time to read the next paragraph.

The following is a good idea what sorts of things we should try to avoid to get the best performance out of the garbage collector:

• Too many allocations.
• Too-large allocation.
• Too many pointers.
• Too many roots.
• Too many object writes.
• Too many Almost-Long-Life objects.
• If you implement IDisposable then suppress finalization to reduce costs.

Dependency Injection (DI)

A foundational pattern in xUnit testing is Dependency Injection (DI) (opens in a new window) , which is a form of Inversion of Control (IoC) (opens in a new window) . Specifically, DI is a programming technique where the implementation of one class is actually performed partially by another. Inversion of Control is where a program gives up control of its own execution and simply responds to requests made of it. In the same way, a class using dependency injection gives up control over some of its implementation and lets the injected class do the work.

Three types of DI exist; constructor injection, property injection, and method call injection. Constructor injection is a pattern whereby dependencies are “injected” into an object during construction. Whereas, property injection uses set operations [on properties] to inject dependencies, which obviously occurs after construction. Method call injection simply uses method calls to inject dependencies.

Several libraries exist in each category. For example, the Castle Project (opens in a new window) and ObjectBuilder (opens in a new window) both provide supports for constructor and property injection. As a side note, the Microsoft Enterprise Library (opens in a new window) uses ObjectBuilder internally as its DI container framework. Unity (opens in a new window) is Microsoft’s latest offering from the Pattern’s & Practices team. Unity offers property, constructor, and method call injection.

Static Classes

A static class (opens in a new window) does not have any instance-level members (including constructors) and is defined by the static modifier (opens in a new window) . The compiler in turn marks a static class as sealed and automatically creates a private constructor.

The main features of a static class are:

• They only contain static members.
• They cannot be instantiated.
• They are sealed.
• They cannot contain Instance Constructors (opens in a new window) .

Inherently in the context of unit (and integration) testing, static classes are not recommended due to the difficulty in implementing a thread-safe property injector DI pattern and being unable to use constructor DI. However, the advantage of a properly written static class is there’s only one object instance per AppDomain (opens in a new window) , which can be very efficient on the GC (see the Too Many Object Writes section of the previously referenced article (opens in a new window) ).

This is a very important and annoying impedance that exists between the use of a constructor DI pattern and efficient use (and preservation) of system resources (in this case CPU and memory). We want to be testable and mindful of system resources.

Singletons

For purposes of this discussion, a singleton is a class that only allows a single instance of itself to be created within an AppDomain. The following are recommended singleton patterns.

Sidebar: Understand the Impact of Low-Lock Techniques in Multithreaded Apps (opens in a new window)

Pattern 1: Niladic Constructor

public class MySingleton {
    private static readonly MySingleton _instance = new MySingleton();
 
    private MySingleton() {
    }
 
    public static MySingleton Instance {
        get { return _instance; }
    }
}

Pattern 2: Niladic Constructor (Full Lazy Initialization)

public class MySingleton {
    class Nested {
        //
        // Explicit static constructor to tell C# compiler not to mark
        // type as BeforeFieldInit
        //
        static Nested() {
        }
 
        internal static readonly MySingleton Instance = new MySingleton();
    }
 
    private MySingleton() {
    }
 
    public static MySingleton Instance {
        get { return Nested.Instance; }
    }
}

Note, the following alternate method results in identical MSIL…

public class MySingleton {
    static class Nested {
        internal static readonly MySingleton Instance = new MySingleton();
    }
 
    private MySingleton() {
    }
 
    public static MySingleton Instance {
        get { return Nested.Instance; }
    }
}

A rather dated but still applicable discussion regarding the use of the double-check lock pattern written by Vance Morrison (opens in a new window) can be found here (opens in a new window) and provides insight into subtle nuances of the Microsoft .NET Framework memory model implementation. I’ve attached a PDF version of the discussion to this entry (at the bottom) in case the archive disappears.

Singleton

So back to the question at hand, which is how do I ensure my production code is testable and uses system resources efficiently as possible? I offer the Singleton class, which is defined as…

public sealed class Singleton where T : new() {
    private static readonly T _instance = new T();
 
    private Singleton() {
    }
 
    public static T Instance {
        get { return _instance; }
    }
}

With the Singleton class, we’re now able to use the following syntax in production code…

public class CustomerManager : BusinessLogicComponent {
    private ICustomerRepository _repository;
 
    //
    // This constructor is required by the Singleton generic class.
    //
    [EditorBrowsable(EditorBrowsableState.Never)]
    public CustomerManager()
        : this(Singleton.Instance) {
    }
 
    //
    //  This constructor is to be used only by test projects.
    //
    [EditorBrowsable(EditorBrowsableState.Never)]
    public CustomerProcessor(ICustomerRepository repository) {
        _repository = repository;
    }
 
    public void Add(Customer customer) {
        _repository.Add(customer);
    }
}

…and the following syntax from test projects.

[Test]
public void Customer_Add_Succeeds() {
    //
    // Create a mock customer repository and inject it into the business
    // logic component (manager).
    //
    ICustomerRepository repository = new MockCustomerRepository();
    CustomerManager manager = new CustomerManager(repository)
 
    //
    // Perform remaining test operations.
    //
}

Consumers of the CustomerManager business logic class in production simply use the Singleton class just as the CustomerManager does with the CustomerRepository class in its niladic constructor.

A final note on thread-safety. The singleton pattern does require all instance level methods to be thread-safe.

Conclusion

The Singleton class provides an implementation pattern to balance between two important variables in any design; that of testability and efficient use of system resources. I encourage you to experiment.

Garbage Collector Basics and Performance Hints.zip

On May 31 2008 a tribute will be held at UC Berkeley to honor Jim Gray (opens in a new window) , who went missing (opens in a new window) during a solo sailing trip in January of this year. Although Jim is listed as missing, and will be until 2011, a Tribute be held to honor him before too much time has passed. There are two parts to this event:

The morning event, which will be in a very large hall, is open and public; the technical session, which is in a smaller hall, and for which you need to register. More details at the Tribute website (opens in a new window) .

I expect to attend.

“The fiefdom syndrome stems from the inclination of managers and employees to become fixated on their own activities, their own careers, their own territory or turf to the detriment of those around them.

People who create fiefdoms can become dangerously insular, losing perspective on what is happening in the world outside their own control. They also lose their ability to act consistently on behalf of the greater good, or in a way that enhances the effectiveness of the larger organization. They often resist new situations and change.

People who create fiefdoms tend to hoard resources. They are determined to do things their own way, often duplicating or complicating what should be streamlined throughout the company, leading to runaway costs, increased bureaucracy and slower response times.

Organizations infected with fiefdoms tend to kill off or stifle individual creativity, leading to what I call the ‘freeze factor’: when organizations become frozen or stuck in place, letting competitors pass them by.”

“In a forest a tree will fade; from a forest a tree is made.” –Unknown

Over the past several years I’ve focused on application of P2P algorithms for the commercial space. Anyway, I thought I’d share some of my thoughts and tertiary research.

First, what is Peer-To-Peer (P2P)? P2P [ URI (opens in a new window) ] is a decentralized, fault tolerant, self-organizing system architecture comprised of many unreliable and heterogeneous nodes operating in a functionally symmetric manner—frequent joins and leaves are the norm. P2P is not a client / server architecture. P2P is a paradigm shift from coordination to cooperation, from centralization to decentralization, and from control to incentives.

Algorithms

To my knowledge, the Plaxton mesh (1997) was the earliest known proposal for a scalable P2P network. One major problem with a Plaxton mesh is its static—there is no concept of node arrival, departure, or failure. Essentially, nodes act as routers, clients, and servers simultaneously. A Plaxton mesh routes a message to the nodes whose name is numerically closest to the destination.

From 2001 through 2003, P2P algorithms such as Chord (opens in a new window) , CAN, Pastry (opens in a new window) , Tapestry, Viceroy, P-Grid, Kademlia (opens in a new window) , Koorde, SkipGraph, and SkipNet appeared.

Applications

Napster (opens in a new window) was one of the first P2P applications that appeared in 1999—though it’s not really a P2P network. With Napster, a single central server kept the directory of nodes and objects, which made is essentially a large file catalog—the discovery and routing algorithms are not P2P.

Gnutella (opens in a new window) first appeared in 2000 and then again in 2002. Though a pure P2P application, Gnutella generated far too much search traffic with a [broadcast-based] O(n) complexity—not very scalable.

eDonkey (opens in a new window) appeared in 2001 and presented a slight architectural improvement over Napster in that no single centralized server held the file catalog. Nevertheless, eDonkey was not a pure P2P application since it relied on status node structures.

Kazaa (opens in a new window) also appeared in late 2001 but I haven’t studied its architecture at all. I listed it here for completeness.

Gnutella-2 (opens in a new window) and BitTorrent (opens in a new window) were the P2P applications of 2002. In 2003, Skype (opens in a new window) , Steam, and PS3 (opens in a new window) appeared.

Architecture

Common characteristics of P2P architectures are:

• No central servers.
• High level of scalability. Discovery and routing is O(log(n)). Routing table size is also O(log(n)).
• Highly resilient.
• Dynamic adaptability to node arrival, departure and failure.

Because of these characteristics, P2P architectures offer challenges in many areas. The most visible of the challenges facing designers of P2P routing algorithms are:

• Scalability. Scalability is a measure of how a system performs when the number of nodes and/or number of messages on the network grows.
• Computational Complexity. Computational complexity is the measure of the order of steps required for a packet to travel from one host to another in a worst case scenario.
• Anonymity. Anonymity is not a requirement of most P2P networks, however if a network is to be designed to provide anonymity then this is a problem that must be solved at the routing level.

Topology

A P2P infrastructure is best described as an overlay network—usually on top of an IP network. In many ways, you can think of P2P as an application-level router providing services that underlying layers do not (i.e. IP multicast). Generally speaking, P2P architectures employ one of the following topological structural designs:

• Centralized (Napster)
• Decentralized
  • Unstructured (Gnutella)
  • Structured (Chord)
• Hierarchical (Multicast Backbone (MBone))
• Hybrid (eDonkey)

Different P2P algorithms employ different topological structures. For example, Chord employs a ring topology whereas Kademlia employs a tree and CAN a hypercube.

Routing

Different topologies require different algorithm tactics and strategies. Routing within P2P architectures are very problematic.

Within P2P architectures, algorithms such as flooding, replication and caching, random walkers and probabilistic algorithms, super-peers, Time to Live (TTL), epidemic and gossip protocols, propagation, and dampening algorithms are widely used. These algorithms work well depending on the size of the P2P network. For example, flooding (opens in a new window) is more suited for a small to medium size networks but it has been shown that the cost of searching on a Gnutella style network increases super-linearly as the number of nodes increases.

Distributed Hash Tables (DHT)

DHTs (opens in a new window) are a class of decentralized distributed systems that partition ownership of a set of keys among participating nodes, and can efficiently route messages to the unique owner of any given key. A hash function (such as SHA-1) accepts a variable length string of bytes and returns a one-way hash. The physical nodes are the hash buckets. Chord is a good example of a DHT algorithm.

Incidentally, in general DHT algorithms will always beat flooding algorithms in the area of routing.

One advantage of the Chord DHT algorithm is that it guarantees receipt of a reply with log(n) time, which is a far better guarantee in terms of computational and time complexity than that offered by flooding techniques. That said, the Kademlia DHT algorithm does offer advantages over Chord depending on application requirements.

However, DHTs are not without problem:

• Routing state maintenance algorithms generate overhead.
• They do not work well when lots of nodes join and leave frequently (churn).
• Their structured topology and routing algorithms make them frail and vulnerable to security attacks.
• Network locality problem: Nodes numerically-close are not topologically-close.

Strategies do exist to address these problems. Regardless, structured DHT algorithms provide the most interesting P2P infrastructures.

Semantic Routing

This is a relatively new area of routing as related to P2P infrastructures. Semantic Routing is a method of routing which is more focused on the nature of the query to be routed than the network topology. Essentially, semantic routing improves on traditional routing by prioritizing nodes which have been previously good at providing information about the types of content referred to by the query.

Semantic routing differs fundamentally from other routing techniques because prospective nodes are selected because of another node’s confidence in their ability to respond correctly to a given query irrespective of their position within the network. Neurogrid (opens in a new window) and Remindin (opens in a new window) are of the most interesting semantic routing systems with the Remindin system incorporating a semantic routing algorithm developed with the intention of mimicking social networks.

Summary

Overall, the most interesting of the DHT algorithms are Chord (opens in a new window) , Kademlia (opens in a new window) , and XRing (opens in a new window) . I hope to release .NET implementations of all three of these algorithms (and a few variations) in the near future. You’ll find them here.

Generally speaking, the life of any distributed system directly relates to its level of entropy. Furthermore, the level of entropy in a system directly relates to the system’s computational complexity. When designing applications to solve business problems, information storage and retrieval is one of the more important foundational design points. If the information schema is designed without efficiency, scalability, and flexibility in mind then not only will the system perform poorly but it will not meet the requirements imposed by future business demands.

At either end of the spectrum, an information schema is categorized as either concrete or abstract—though there most certainly are varying degrees in between. Each design has advantages and disadvantages. A concrete design is easier to conceptualize and requires less complex algorithmic considerations. In most cases, a concrete design is far less flexible than that of an abstract design. A properly designed abstract information schema remains invariant in the face of change that would otherwise impose a relatively large amount of downtime for an equivalent concrete model.

Arguably, however, many abstract systems are indeed designed with concreteness in mind. By this I mean the system is designed to contain representations of concrete constructs. Why? It’s how humans think in every day life. If I ask you to describe yourself then you’re statistically more likely to forgo attributes such as “person” or “human” and instead provide attributes such as height and weight. The former attributes are more abstract (relatively speaking) than the latter attributes. However, both indeed describe a concrete construct: you.

I tend to favor abstract information models but then again, I spend the majority of my time in abstract problem domains. However, regardless of which classification the solution domain requires, the following rules and considerations have proven effective in designing an efficient information model:

1. Effectively understand the information.
2. Describe it unambiguously.
3. Enforce structure and style guidelines.
4. Allow for efficient storage and retrieval of information.
5. Keep network communication to a minimum. Don’t over engineer.

The last bullet (5) is delivered with a caveat. Over engineering is a very subjective quantification that takes time to perfect and is directly related to your complete understanding of the information (bullet 1). If bullet 1 suffers then so does bullet 5.

Usually, the questions are quick. It’s the answers that take the time.

To provide a bit of context, we were considering a solution space that incremented the USN value for each write operation for a specific set of protected data. The USN is 64 bits, and is advanced for each update on a given server. At 100 writes per second, the USN will roll over in 58,494,241,735 years (approximately).

On a very busy server performing 10,000 writes per second it will roll over much sooner, in 584,942,417 years. Either way, the sun will burn out or go nova first, making USN’s moot (for this astronomical neighborhood anyway).

Just using 128 bits for USNs would give us a few more millennia breathing room. Now for the good news. In point of fact, since USN’s from different servers are never compared, all that is required is detecting rollover of the USN on a given server, for which algorithms are available. See RFC 1982 (opens in a new window) for a discussion.