Scale data pathways before code complexity

Why senior architecture decisions often begin with movement, storage, backpressure, and access patterns rather than another abstraction layer.

Systems usually feel scale through their data pathways before they feel it through the elegance of their class hierarchy.

Find where the system actually feels pressure

When a system slows down, the visible symptom often appears in application code. The underlying pressure may be elsewhere: a synchronous network dependency, an unbounded queue, a repeated scan, a hot partition, a large payload, a missing cache boundary, or an analytical query competing with an operational write path.

Before adding framework machinery, map how data enters, moves, waits, multiplies, changes ownership, and leaves. That map usually exposes the first useful architecture decision.

System design: name the pathway before naming the technology

A drone detection event and a high-definition video stream do not need the same transport policy. A real-time bid response and an attribution query cannot share the same latency contract. A user-facing transaction and a model-evaluation export should not fight for the same database resources.

The design becomes simpler when each path has one clear promise, and technology choices follow the promise instead of leading it.

Latency-critical

• A bounded dependency graph — no synchronous call to a system with its own tail latency
• Preallocated resources over dynamic allocation under load
• Benchmarked and load-tested independently of the rest of the platform

Durable asynchronous

• Producers and consumers scale independently of each other
• At-least-once delivery with idempotent consumers on the receiving end
• Backpressure visible as a queue-depth metric, not a silent slowdown

Analytical

• Isolated from the transactional write path, never sharing its connection pool
• Optimized for scan and aggregation, not point lookups
• Freshness measured in minutes or hours, and stated explicitly to consumers

Archival

• Cheap storage, rare access, and a clear retention policy
• A retrieval path allowed to be slow, because it is rarely exercised

Best-effort

• An explicit drop policy under overload instead of an implicit one
• No pathway is allowed to silently borrow another pathway's resources during a spike

Add abstractions after stable repetition

Architecture does need abstractions, but the timing matters. A platform primitive should capture repetition that has become stable enough to name. Inventing it before teams have experienced the pattern often creates a paved road to the wrong destination.

My preference is to observe two or three real implementations, record what repeats, preserve the meaningful escape hatches, and only then make the common path easy. The result is less impressive in an early diagram and far more useful to an engineering organization.

Capacity is a product decision

Throughput, freshness, retention, latency, and cost are not purely technical numbers. They encode what the product promises. If the business cannot state which data must be immediate, which can be delayed, and which can be dropped under stress, the architecture cannot make an honest capacity plan.

The architect's job is to turn those trade-offs into a decision that product, engineering, and operations can all see—and later revisit with evidence.

From production: what the pathway split actually bought

AgentBid is the clearest example of this in the portfolio. The Rust bidder answers an auction under a strict latency budget and never waits on anything else in the system — the architecture decision was to keep the deterministic bidding path small and native, with no LLM anywhere on the hot path. Everything downstream — billing, attribution, fraud review, and the agent that explains a spend anomaly to an operator — reads from an event stream and a ClickHouse store that can lag by seconds without breaking anything, because that path's promise was never "immediate." Collapsing that separation, even to make one more analytics query convenient, would put the auction path at the mercy of a database it has no reason to know exists.

CGC Sentinel drew the same line in the opposite domain. MQTT event traffic — small, structured, latency-sensitive — was kept separate from WebRTC video, which is large and only needed on demand as evidence rather than a permanent broadcast. That split is most of the reason the field deployment measured roughly 80% lower bandwidth use and 60–70% lower decision latency: the architecture never asked a best-effort payload to travel on a latency-critical promise, and it never asked a latency-critical decision to wait on a best-effort transport.

The Bedrock knowledge-base platform shows the same discipline applied to a cache instead of a network link. Repeated questions hit a semantic cache before the slower hybrid retrieval and generation path runs at all — but that cache is deliberately treated as its own pathway with its own freshness contract, not as a free optimization layered onto retrieval. When that contract was respected, latency and cost dropped for repeated enterprise queries. When it briefly wasn't, stale cached answers for one tenant were the direct result — a reminder that a pathway's promise has to be enforced, not just declared.

What to carry forward

• Map data movement before adding abstractions
• Give different pathways different promises
• Design overload behavior explicitly
• Let stable repetition earn a platform primitive
• Enforce a pathway's promise with a metric, not a comment

Applied in the portfolio

• CGC Sentinel — Edge computer vision for drone operations, moving decisions closer to the camera and reducing network pressure.
• AgentBid — An open architecture for real-time bidding operations, combining a high-throughput data plane with explainable AI assistance.
• StreetSenseAI — An open-data intelligence system for understanding roadworks, disruption, and operational risk across UK streets.
• Enterprise Knowledge Base Multi-Agents on AWS Bedrock — A governed AWS Bedrock knowledge platform for multi-source ingestion, multi-agent reasoning, semantic retrieval, observability, and FinOps.