[{"content":"I used to spend most of my time on direction. As a Principal Engineer, the work was talking to other Principals, Sr. Principals and Senior SDEs, shaping how problem got framed, influecing upward on what we\u0026rsquo;d build next quarter. The keyboard sat further away each year with me only coming back for proof of concepts and prototypes to help unblock ideas. That was the accepted arc - seniority meant moving away from production work.\nThat arc has inverted.\nI\u0026rsquo;m back at the keyboard, working on 4-5 streams in parallel with teams of agents, still doing the people work alongside it. Some of this can be attributed to my new job. Most of it isn\u0026rsquo;t. The agent-native way of working has made being in the work cheaper than it used to be, and the leverage of pure direction has dropped. Picking up the keyboard isn\u0026rsquo;t a regression.\nBut the keyboard isn\u0026rsquo;t what it used to be either. It operates fleets now. And the work it produces - multiplied by every other engineer doing the same - has created a problem none of us has completely figured out.\nThey keyboard came back, but it operates fleets When I last say in the role years ago, deep focus blocks were the unit of work. A few uninterrupted hours to read a system, write a spec, talk to people, leave the office having moved on one thing meaningfully. The new shape is different. My day is 4-5 streams running in parallel - one agent investigating a live customer issue, another iterating on a spec for a new feature or project, next one reviewing a PR for me to look at, another helping me rebase my feature branch against the massive influx of changes from the team - and in between I\u0026rsquo;m in 1:1s, reading partner team\u0026rsquo;s design docs, preparing for a leadership review.\nThe context-switching is constant. The window between switches are short. Productivity suffers when I\u0026rsquo;m sloppy about which stream I attend to. The tradeoff I\u0026rsquo;m making is real: they keyboard returned, but the people work didn\u0026rsquo;t leave. Both are mine now, and the day has to absorb both.\nI don\u0026rsquo;t think this is just a new-job artifact. The AI-native way of working seems to actively require this shape. When you have a smart team of agents to collaborate with, the friction of building drops far enough that staying out of work means leaving leverage on the table. It also means that old social structures - designated implementers, designated reviewers, designated coordinators - start to look strange. We\u0026rsquo;re moving past the titles whether we acknowledge it or not. The poeple who thrived through the recent rounds of org compression were the ones who already moved across the role boundaries, or who learned to adapt.\nThe bottleneck moved The old story about engineering was that bandwidth was the expensive part. Every process we layered on - waterfall, then agile, then design docs, then RFCs - existed to allocate scarce engineering time. Fiona Fung, who leads engineering for Claude Code, put it crisply in her recent Code w/ Claude talk 1: writing code, writing tests, and refactoring rarely slow her team down anymore.\nWhat slows them down - and what slows us down on my team - is what came next.\nThe bottlenecks didn\u0026rsquo;t go away when agentic coding took away the actual need to type code. Verification, code review, and security took their place.\nThis mental model reframes most of what\u0026rsquo;s happening in engineering orgs right now. The process we layered were never critical because of typing time. They were critical because the gap between intent and output - gap that humans naturally fill with judgement, review cycles, design discussions, second opinions. Closing the typing gap didn\u0026rsquo;t close the judgement gap. It exposed it.\nAnd the closing of one gap creates pressure downstream. Adam Bender\u0026rsquo;s Google I/O 2026 talk 2 made this explicit through a systems-thinking lens: take any system, crank the output of one node way up, and the rest struggles to keep pace. The weakest links get exposed. The trade-offs an organization built up over decades get rebalanced under productivity multipliers some teams are now measuring in orders of magnitude.\nHis concrete example sticks with me. A 10x increase in code volume doesn\u0026rsquo;t trigger a 10x increase in test runs. Because of how dependency graphs amplify, it can trigger and 1000x increase. Compute budget vanishes before the feature ships. Review pipelines turn into a permanent jam. CI gets overwhelmed. The output-side wins do not stay isolatedl they cascade through veery downstream node that was sized for the old throughput.\nSo the bottleneck doesn\u0026rsquo;t move neatly. It creates ripple effect.\nReviewing one PR at a time is losing The bottleneck I\u0026rsquo;m currently working on, in my seat, is review.\nMost of my team\u0026rsquo;s PR are agent generated. Some engineers don\u0026rsquo;t review their own agent output before sending it. A team\u0026rsquo;s system has 100+ open PRs on a typical day. I do a lot of review myself. The day I noticed I had become the bottleneck was the day I had to do something different.\nThere\u0026rsquo;s a temptation to push back on the velocity - slow down, review more carefully. I tried that. It doesn\u0026rsquo;t scale. The throughput multiplier is real, and trying to throttle it back into a shape that a single-PR human review can handle just creates a different bottleneck (engineers waiting on reviewers) without solving the underlying one. The right move isn\u0026rsquo;t to slow production. The right move is to change what review is.\nThe human is not the reviewer The naive next step is the one most teams will reach for: delegate the review to agents too. Have one agent produce the PR and another agent review it. That works for tactical correctness but raises a question that took me a while to articulate clearly: if agents produce and agents review, where am I in this?\nThe honest answer is: not in either of those places. I\u0026rsquo;m a layer up.\nThe work I\u0026rsquo;m doing isn\u0026rsquo;t reviewing PRs anymore. It\u0026rsquo;s specificying what review means - what we look for, what we\u0026rsquo;ll never tolerate, which patterns are red flags, which trade-offs we\u0026rsquo;ve already settled, which invariants are non-negotiable in this codebase. I started encoding that into a review agent that runs against the PR before I see it. The agent applies my first principles. I look only at what it flags as interesting, novel, or beyond its current understanding.\nMental model authorship is the differentiated work which we do as humans.\nThis is different craft from coding. Different from directing. Different even from reviewing. It\u0026rsquo;s authorship - the articulation of judgement with enough precision that the articulation can be applied at scale, by something that isn\u0026rsquo;t me.\nThe Principal Egnineer\u0026rsquo;s seat hasn\u0026rsquo;t disappeared. It moved up one layer. From person who reviews to person who specifies what review means. From individual judgement exercised on each artifact to judgement encoded once and applied to many.\n\u0026ldquo;Trust but verify\u0026rdquo; is the concrete phrase which helps me. Trust the agents to handle volumel verify with criteria you\u0026rsquo;ve authored. The verification isn\u0026rsquo;t human\u0026rsquo;s job anymore. The criteria are.\nTime spent on intent compounds There\u0026rsquo;s a second surface, and its the one I spend more time pushing on with the team.\nBefore the agent runs the implementation, somebody has to say waht the implementation should do. That\u0026rsquo;s the spec. The intent. The goal. Under the old model, specs were often vague because humans naturally filled the gaps with judgement as they implemented. A Sr. SDE reading \u0026ldquo;build a service that handles retries cleanly\u0026rdquo; would fill in a dozen reasonable assumptions and produce something workable. Agents don\u0026rsquo;t fill gaps the same way. They either ask, or they invent something that probably isn\u0026rsquo;t what you wanted.\nThe time engineers used to spend on implementation has to be reallocated. Not to validation downstream, though some of it goes there - but to intent upstream. The dominant industry framing right now is AI made implementation fast, so make validation fast too. I\u0026rsquo;d push against that. The bigger leverage is: AI made implementation fast, so invest the saved time in being precise about what you mean. It intent is captured completely, validation has something concrete to check against. If it isn\u0026rsquo;t, you iterate the implementation, then iterate the validation, and burn the supposed productivity gain on rework.\nThroughput is one metric poeople gravitate towards however the real metreic is whether the thing which you were actually trying to solve got solved. Precisely defining what does success look like?. A clean PR that ships the wrong feature is still a failure. Without precise intent, you have no way to tell.\nThe push toward ideation is the harder cultural move on my team. It\u0026rsquo;s easier to start typing - even when \u0026ldquo;typing\u0026rdquo; means prompting an agent - than to stop and articulate what success actually looks like. The discipline I\u0026rsquo;m trying to build is: do the spec first, treat it as the contract, and resist the pull to start generating code before the contract is clear.\nA validation framework Once intent is captured, the vlaidation work has a shape. The framework I\u0026rsquo;ve been converging on:\nOriginal intent must be captured and verifiable. - Not a single0line description - the complete goal you settled on: what done looks like, what invariants hold, what edge cases matter. Without completeness here, validation has nothing concrete to check against. Throughput against a vague goal is throughput against the wrong thing.\nValidation runs in a different session from implementation. - The implemeneter agent has all the context that biased the implementation - the path it took, the assumptions it folded in, the dead ends it abandoned. A fresh session reveiwing against intent has none of that bias. This operationalizes \u0026ldquo;trust but verify\u0026rdquo; structurally: separation of session is separation of context, which is the precondition for honest verification.\nThe knowledge base records what you pay attention to. - This is where mental model authorship lives as an artifact. Invariants. Things hard to refactor later. Patterns that have bittent he team before. The knowledge base as a skill is versioned, diffable, reviewable. When my judgement changes, the files changes. When the team\u0026rsquo;s collective judgement changes, the shared knowledge base of the team change. The mental model is no longer in my head; its in a file other people and other agents can read.\nValidation is a workflow that produces structured findings. - Each finding has severity, source references, and a proposed mitigation. Free form review output is hard to triage and impossible to track. Structured findings can be sorted, filtered, batched, and learned from. The workflow looks at the intent, inspects the change, and emits a structured result - not a long paragraph.\nEach step can escalate to a human when no encoded principle applies. - This is the rsolution of the recursion problem. The agent doesn\u0026rsquo;t decide everything. It decides what it has been taught to decide. When it hits something its mental model doesn\u0026rsquo;t cover, it surfaces the case. The workflow encodes where human reviews are essential and matter rather than asking humans to review everything by default.\nThe primary steps are review, document findings, run tests, static analysis, and rebase. - The shape is convering across paractitioners. Kun Chen\u0026rsquo;s no-mistakes 3 pipeline runs intent -\u0026gt; rebase -\u0026gt; review -\u0026gt; test -\u0026gt; docs -\u0026gt; lint -\u0026gt; push -\u0026gt; PR -\u0026gt; CI. The overlap with what I\u0026rsquo;m stating is not accidental. Engineers building this kind of validation arrive at similar steps because the same pressures shape the design.\nEach escalation feeds back into the knowledge base. - When I weigh in on a flagged case, the resolution updates the knowledge base. The next time something similar appears, the agent already knows. This is where the work compounds. The knowledge base isn\u0026rsquo;t a static prompt - its a living artifact that absorbs every novel case it encounters. Some of the principles I write a personal - taste, idiosyncractic preferences for code I\u0026rsquo;ll maintain. Others are generic to the team - invariants we all share. The classification matters: personal skills stay local; team skills get checked into shared paths so leverage compounds across the org.\nThe seven points are not a recipe. They\u0026rsquo;re a shape. The substance lives in the knowledge base themselves - the actual principles, in the actual files. The framework is the container; the mental model is the content.\nDeterministic structure, non deterministic judgement The most useful insight I\u0026rsquo;ve had on this so far is about where determinism lives in the loop.\nThe goal is determinism on outcomes. I want the tests to run. I want the lint to pass. I want certain invariants checked everytime, not when somebody remembers. The intuitive solution si to make the agent more deterministic. That\u0026rsquo;s the wrong move. Agents are useful because their judgement is flexible. Forcing them into a deterministic mode eliminates the very thing that makes them useful.\nThe right answer is to put determinism in the structure, not the judgement. Git push hooks deterministically run after the push. The orchestration script deterministically invokes each workflow step. The skill.md file deterministically loads context into each agent call. The agent\u0026rsquo;s judgement happens inside this container - flexible, situational, capable of nuance - but the container around it is rigid.\nThat\u0026rsquo;s how you get determinism on outcomes without sacrificing the value of judgement. The hooks make sure the right tests run. The script makes sure each step happens in the right order. The skills make sure the agent sees what it needs to see. What the agent decides, within those rails, is allowed to be context-dependent.\nThis is alos the architectural answer to Adam\u0026rsquo;s amplifier warning. AI amplifies whatever your pipeline already does. If your pipelien is held by social convention - engineers will remember to run tests, reviewers will catch the obvious bugs - then a 10x througput increase amplifies the holes in social convention. If the pipeline is held together by deterministic structure - hooks always run, scripts always orchestrate, skills always load - then the 10x amplifies a pipeline that doens\u0026rsquo;t have those holes.\nWhere the human sits So if the agent produces, the agent reviews, and a script orchestrates the loop, where is the human in this?\nAt the two ends of the loop, doing the authorship work that brackets everything the agent does in between.\nThe first end is intent. I articulate the goal, the success criteria, and the invariants the implementation has to respect. Most of the typing time we\u0026rsquo;ve saved should go here. What gets defined at the start shapes everything downstream; vagueness at this end becomes iteration further out.\nThe second end is review. The shape of the work is the framework above. The substance is authorship — writing principles that capture what good looks like across everything we build, not just for the feature in front of me. The invariants. The patterns we never want to see. The trade-offs already settled. These principles live in skills, get applied to every PR, and outlive any single feature. Time spent here compounds harder than time spent on intent: a principle written once runs forever, on every change, without me.\nThe work at both ends isn\u0026rsquo;t only accumulation. The model underneath this loop keeps getting smarter, and as it does, some of what I encoded becomes redundant. Principles I wrote months ago to correct for behaviors the model used to produce are now noise — the model doesn\u0026rsquo;t make those mistakes anymore. The mental model corpus isn\u0026rsquo;t a one-way ledger. It needs pruning, especially as capability moves underneath it. Part of the seat is going back through the skills and deciding what to delete because the model has caught up. A principle that mattered last quarter may be friction this quarter, and a stale principle adds drag without payoff.\nI\u0026rsquo;ve also been exploring feeding the review criteria back into the spec stage. If I know what validation will look for at the end, the implementer agent should know that shape going in. The spec stops being an isolated artifact and becomes informed by the principles that will judge the result. What I look for in review is what the spec should make explicit upfront. I\u0026rsquo;m still working through what this looks like in practice — the direction feels right, the mechanics aren\u0026rsquo;t fully figured out. The more the agent knows about the criteria, the less it produces things that need to be flagged later.\nBoth bookends, together with the work of curating and connecting them, add up to a different definition of the senior IC seat than the one I trained into. Fiona talks about hiring less for raw throughput and more for creative builders and deep systems experts. The seat she\u0026rsquo;s describing — the one I\u0026rsquo;m trying to occupy — is the one that prioritizes judgment encoded at scale over judgment applied case-by-case. The keyboard came back, but what the keyboard is for changed.\nOne bottleneck at a time I keep coming back to a question Adam Bender posed to his audience: do you know what would break first? Not as a guess. As a map.\nThe honest answer, for most of us, is no. We have a sense of where the pressure is right now. The next bottleneck — the one that will surface after this one resolves — is a guess, sometimes an informed one, often not.\nFiona named several of the bottlenecks that have already moved on her team: review, security, planning rituals, context-gathering, team composition. Adam mapped the cascade structure that explains why fixing one moves pressure to the next. What I\u0026rsquo;m doing — what a lot of us are doing right now — is taking the bottleneck in front of us and trying to dissolve it before it breaks something irrecoverable.\nI\u0026rsquo;ve come to think this is what the senior IC seat is in the agent-native era. Not a fixed set of responsibilities. Not \u0026ldquo;review code\u0026rdquo; or \u0026ldquo;design systems\u0026rdquo; or \u0026ldquo;mentor engineers.\u0026rdquo; Those are activities. The seat itself is a discipline: paying attention to which constraint is binding now, naming it precisely, and being willing to invent the craft that resolves it. Then doing the same thing for the next one.\nThe discipline has a name in process engineering: debottlenecking. Continuous removal of binding constraints. It\u0026rsquo;s a mature practice in physical systems — refineries have been thinking this way for decades. We\u0026rsquo;re learning to think this way about software organizations because the productivity multipliers are now large enough that bottlenecks cascade visibly within months, not years.\nI don\u0026rsquo;t have the full map. Nobody I\u0026rsquo;ve talked to does. What I have is the bottleneck in front of me and a set of moves for working through it: invest in intent, encode judgment into skills, run validation in fresh sessions, structure determinism around the agent\u0026rsquo;s flexibility, sit at the escalation point.\nI\u0026rsquo;m making my way through it one bottleneck at a time, trying to dissolve what I\u0026rsquo;m noticing. Some of the moves will turn out to be right. Some won\u0026rsquo;t. The map will keep redrawing.\nThe underlying shape of the seat, though, is consistent across the moves. I\u0026rsquo;m not the reviewer. I\u0026rsquo;m not the implementer. I\u0026rsquo;m the author of the principles that get applied to both, by agents, at scales I couldn\u0026rsquo;t touch one PR at a time. The Principal Engineer\u0026rsquo;s seat moved up a layer. The keyboard came back not because there was more typing to do, but because the keyboard now operates the fleet that operates on my judgment. That\u0026rsquo;s where the leverage lives. That\u0026rsquo;s the work.\nReferences Fiona Fung, \u0026ldquo;Running an AI-native engineering org,\u0026rdquo; Code w/ Claude SF 2026. (running-an-ai-native-engineering-org)\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nAdam Bender, (\u0026ldquo;Software engineering at the tipping point),\u0026rdquo; Google I/O 2026.\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nKun Chen, no-mistakes. (no-mistakes)\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://blog.girishshekhar.com/posts/pe-role-inversion/","summary":"The Principal Engineer is back at the keyboard, but the keyboard now operates fleets. Review has become the binding constraint, and the work has moved up a layer — from reviewing artifacts to authoring the criteria against which artifacts get reviewed. Notes on what that looks like in practice, and on the longer discipline that I think now defines the seat.","title":"The Principal Engineer's Inversion"},{"content":"Why this post exists For about two years, the dominant way to talk about agents has been to talk about building them. You build an agent. You give it a role. You compose agents into crews, chains, graphs. The verb is always build, and the noun is always agent — a thing you instantiate, configure, and assemble like any other object in your system.\nThat language shaped the tooling, and the tooling shaped how a generation of engineers thinks about the problem. My perspective has evolved over time and I sincerely believe this is the wrong abstraction. Not saying that the earlier tools were useless — they were a reasonable response to the conditions of their moment — but wrong in a way that is now actively costing teams time, reliability, and the ability to reason about their own systems.\nThe reframe is simple to state and surprisingly hard to fully absorb:\nThe agent is not the artifact you build. The harness is. The agent is what the harness does when you run it.\nThe mental model most of us are still carrying The naive model is An agent is a thing. The framework helps you build the thing. The model is the brain of the thing. Under this model, the agent is the unit of work: you reach for an agent the way you\u0026rsquo;d reach for a class, you compose agents the way you\u0026rsquo;d compose objects, and when something goes wrong you ask \u0026ldquo;what\u0026rsquo;s wrong with my agent?\u0026rdquo;\nThis framing came from somewhere reasonable. The word harness itself comes from the evaluation world — lm-evaluation-harness and the SWE-bench harness used it to mean the runtime scaffolding that makes a model executable against a task. The loop shape came from the ReAct pattern 1 — reason, act, observe, repeat. It interleaved reasoning traces with tool actions so a model could plan, act, and incorporate observations in a single loop. And the frameworks that followed — the ones that gave us Agent, Task, and Crew as first-class abstractions — were a sensible bet given what models could do at the time. When the model is weak, the orchestration around it is the hard part, and abstracting that orchestration adds real value. A framework that handled the loop, the tool wiring, and the conversation state for you was doing the genuinely difficult work so you didn\u0026rsquo;t have to.\nSo the agent-as-object model was a correct response to a world where the model was the unreliable component and the scaffolding was where the engineering lived. That world has evolved and the abstraction built for it is the wrong-shape.\nThe harness is the artifact Strip an agent down to what is actually running and you find four things: a model, a loop that calls the model repeatedly, a set of tools the model can invoke, and a context manager that decides what the model sees on each turn. Layer a permission model on top — what the thing is allowed to do in the world — and you have described every production coding agent in existence. Claude Code, Cursor\u0026rsquo;s agent mode, Aider, Cline, Codex - all have the same shape.\nThree of those four things — the loop, the tools, the context discipline — are code you write and own. The fourth, the model, is a component you call. The \u0026ldquo;agent\u0026rdquo; is not any one of these. The agent is the behavior you observe when you run the loop. It is emergent, not constructed. You don\u0026rsquo;t build an agent any more than you build a process — you build the thing that produces it when executed.\nAnthropic\u0026rsquo;s own guidance points the same direction. In Building Effective Agents2, they describe the basic building block of an agentic system as an LLM augmented with tools, retrieval, and memory. Whats interesting is that they draw the line between workflows (where the paths are orchestrated through predefined code) — and agents (where the model dynamically directs its own process). The advice is to reach for that machinery only when the task demands it, rather than starting from a heavy framework.\nMental Model : the augmented loop is the artifact, and the agent is what it does at runtime.\nComparing to traditional system, this is the kernel, the process, and the program. You write a program and hand it to the system; the kernel schedules it, allocates its memory, and mediates its syscalls; the CPU executes the instructions; and the process — the running thing, with its live state and behavior — is what emerges when all of that turns over. You do not construct the process as a first-class object and compose it. You write the program and build (or rely on) the kernel; the process is the runtime consequence. Map it across: the task you hand the agent is the program, the harness is the kernel, the model is the CPU, and the agent — the running behavior — is the process.\nLLM System Traditional System Task Program Harness Kernel Model CPU Agent Process The engineering lives in the kernel: the scheduler, the syscall surface, the memory model, the isolation boundaries — not in some Process class you instantiate. A \u0026ldquo;process framework\u0026rdquo; that hid the kernel and handed you composable process objects would be abstracting exactly the wrong layer.\nAgent-as-object model Harness-as-artifact model (what people say) (what is actually running) ┌──────────────────┐ ┌──────────────────────────┐ │ Agent │ │ Harness │ │ (the artifact) │ │ (the artifact) │ │ │ │ ┌────────────────────┐ │ │ ┌──────────┐ │ │ │ loop (your code) │ │ │ │ model │ │ │ │ tools (your code) │ │ │ │ (brain) │ │ │ │ context (yours) │ │ │ └──────────┘ │ │ │ permissions │ │ │ │ │ └─────────┬──────────┘ │ └──────────────────┘ │ │ calls │ | ▼ | │ ┌────────────┐ │ │ │ model │ │ │ │(component) │ │ │ └────────────┘ │ └──────────────────────────┘ │ when run ▼ \u0026#34;the agent\u0026#34; = behavior The takeaway here is the relocation of the model. In the left model, the model is the core and the agent is the box around it. In the right model, the model is one swappable component inside the artifact, and the artifact is the loop. The agent is not in the picture at all, because the agent is what comes out when you press play.\nThe clearest recent signal that this is the real artifact: harnesses are starting to factor out of the products that built them. When Cline extracted its core agent harness3 into a standalone, open-source runtime — packaging the loop, tool orchestration, and session handling as something other products are built on rather than logic buried inside one app — that is the abstraction stabilizing in public. The loop, not the agent, became the thing worth shipping on its own.\nConclusion 1: What is means to \u0026ldquo;own\u0026rdquo; the loop The first noticeable change is control. While the abstraction provided an illusion that the frameworks \u0026ldquo;hid\u0026rdquo; what the agent was doing. Anyone who has printed a LangChain trace or opened LangSmith (LangChain\u0026rsquo;s tracing and observability tool) knows the calls were right there. The frameworks gave us observability. We could watch every model call, every tool invocation, every context mutation.\nWhat they did not give us was control over the loop at the point where we need it. The distinction is between watching and owning.\nIn a framework, the loop runs inside the framework\u0026rsquo;s control flow. We observe it from the outside through callbacks and traces. When we want to change what happens : a. inject a step between the model call and the tool execution b. rewrite a tool result before the model sees it c. alter how context is compacted under pressure d. change the retry logic for one specific failure mode\nWe aren\u0026rsquo;t editing the loop and instead we are configuring someone else\u0026rsquo;s loop through whatever extension points its authors anticipated. The control is inverted: the framework calls our code; we do not call the loop.\nThe structural difference is worth drawing, because it is not a difference of features (both have the same parts), but a difference of who holds the loop:\nAGENT FRAMEWORK HARNESS (inversion of control) (we hold the loop) ┌─────────────────────────┐ our code: │ Framework runtime │ ┌─────────────────────────┐ │ owns the loop │ │ while not done: │ │ │ │ msg = call_model() │ │ ┌─────────────────┐ │ │ if needs_tool(msg):│ │ │ internal loop │ │ │ ┌──\u0026gt; result = run(tool)│ │ │ ─ call model │ │ │ │ ┌─ # edit HERE │ │ │ ─ route tool │ │ │ │ │ ctx.append(...) │ │ │ ─ update ctx │ │ │ │ │ done = check() │ │ └───────┬─────────┘ │ │ └───┴──────────────────┘ │ │ fires │ └─────────────────────────┘ │ ▼ │ the loop IS our code; │ ┌─────────────────┐ │ every line is an edit point │ │ our callback │ │ │ │ (on_tool_end, │ │ the model, tools, and context │ │ on_step, ...) │ │ are functions we call — │ └─────────────────┘ │ not events we react to │ fires only where the │ │ framework decided to │ └─────────────────────────┘ We react to the loop. We write the loop. Control points: where the Control points: every line. framework exposes a hook. In the framework, control flows outward from the runtime to our callbacks, and we can only intervene where the framework chose to fire one. In the harness, there is no outward flow — the loop is our code, so every line is a point we can change. While both let us watch, only one lets us reach in anywhere.\nIn a harness, the loop is a while statement we wrote. Changing what happens between the model call and the tool execution means editing the line between them.\nWatch the difference play out in the one scenario every agent developer eventually hits — the agent does something dumb at step four of a seven-step task:\nIn a framework: 1. Open the trace. Confirm the bad behavior is at step 4. (observability works) 2. Decide we want to intercept the tool result at step 4 and correct it. 3. Look for a callback that fires at the right point. on_tool_end? on_agent_action? 4. Read the framework source to learn what state is available when it fires. 5. Discover the hook fires after the result is already committed to context. 6. Subclass the executor. Or monkeypatch. Or fork. Or file an issue and wait. In a harness: 1. Open the loop. Find the line where the tool result returns to context. 2. Add a conditional. Edit it. Run it. The leverage was the larger problem and not the visibility. The framework let us stand outside the glass and watch the machine; the harness handed us the machine. And the moment this bites hardest is precisely the moment that matters most — when something fails in a way the framework\u0026rsquo;s authors did not anticipate, which in a fast-moving field is most of the time.\nThis is why \u0026ldquo;own the loop\u0026rdquo; makes debugging tractable in a way that \u0026ldquo;configure the loop\u0026rdquo; never quite does. We are no longer reasoning about the framework\u0026rsquo;s internal state machine as a prerequisite to reasoning about our own bug.\nConclusion 2: the highest-leverage work moves elsewhere The second conclusion follows directly highlighting where we should be spending most of our time.\nOnce we accept that the harness is the artifact and the model is a component inside it, a question quietly dies: which agent should I use? It is replaced by a better one: what should my loop\u0026rsquo;s surface area be? The work moves from selection of agents to designing the two concrete surfaces the model interacts with.\nThe first surface is the tool set. A model in a loop is only as capable as the tools we expose and how legibly we expose them. A tool that returns three thousand lines of unstructured log output is a tool that poisons the next turn\u0026rsquo;s context. A tool that returns a tight, structured summary with the relevant fields named is a tool that compounds. Most of the difference between an agent that finishes a task and one that thrashes is not the model — it is whether the tools were designed for a model to use or merely exposed because they existed in our codebase. This is design work, and it is now our highest-leverage activity.\nThe second surface is context discipline: what the model sees on each turn (which includes tool discovery and outputs from the first surface), what gets compacted, what gets dropped, what gets pulled in fresh. The loop\u0026rsquo;s quality is largely determined here. A harness that lets context grow unboundedly degrades; a harness with deliberate compaction and retrieval stays sharp across long tasks. None of this is the model\u0026rsquo;s job. It is the harness\u0026rsquo;s job, which means it is ours.\nThere is a third conclusion which emerges. When the model is a component, swapping it becomes a bounded experiment rather than a rewrite. A well-built harness treats the model as a parameter. We can run the same loop, the same tools, the same context policy against a different model and get a clean comparison for our specific workload — because everything except the model is held constant. \u0026ldquo;Which model is better - the chat one or the code based reasoning one\u0026rdquo; starts being a measurement we can actually take. The agent stays the same while the substrate does.\nSo should we still build agents? If the harness is the real abstraction, the obvious question is whether frameworks are now obsolete and everyone should be hand-writing loops. The answer as always is it depends. Both abstractions persist. The interesting question is how to decide which one a given application is suitable for, and the deciding axis is where our differentiation lives.\nThe cleanest way to see it is an analogy every engineer already trusts. We use a managed framework for the same reason we use a managed database: until the layer it abstracts becomes the layer where your quality is determined, the abstraction is pure leverage. Most applications should never run their own Postgres. The handful where the storage engine is the product — where replication, sharding, and consistency control are the differentiation — eventually outgrow the managed layer and take ownership of it. Frameworks are managed agents. Harnesses are self-hosted ones. The decision is the one engineers have always made: own the layer where your differentiation lives, rent everything else.\nThis is not a fringe view. OpenAI\u0026rsquo;s guide to building agents 4 lays out a spectrum from\ndeclarative graph frameworks, where we predefine every node and edge upfront, through a code-first approach where we express the loop in ordinary code, to building directly from scratch It recommends starting with\na single agent that loops through tool calls until an exit condition is met escalating to heavier machinery only when complexity demands it. Since the framework vendors themselves describe a spectrum, and that the choice between declarative and code-first is the evidence that there is no single right abstraction. There is a right one for where our quality is determined.\nThat gives a usable test:\nReach for a framework when… Reach for a harness when… The loop is generic; standard reason-act-observe with no exotic control flow The loop is the product; how we sequence tools, manage context, and recover from failure is where our quality lives We are prototyping and want defaults that move we fast We are in production and need to debug, version, and tune behavior at specific points in the loop Our differentiation is in the data, domain, or surrounding UI — not how the agent thinks Reliability requires owning every decision in the loop rather than inheriting the framework\u0026rsquo;s The abstraction\u0026rsquo;s cost (less control) is cheaper than building the loop The abstraction now costs more than it saves — every feature is a fight with the framework\u0026rsquo;s assumptions Note: This is not a migration where harnesses \u0026ldquo;win\u0026rdquo; and frameworks die. It is that the layer of differentiation moved. When models were weak, orchestration was the hard part, so abstracting it added value and the framework was the right call for almost everyone. Now that models are strong, the loop, the tools, and the context discipline are where quality is won or lost — so for applications whose quality depends on those things, owning them matters more than the convenience of renting them. If model capability shifts again, the conclusions shifts again as well.\nThe key takeway here is : own the layer where your quality is determined; abstract the rest.\nIf you are using a framework, and notice that every nontrivial thing you want to do is now a workaround — a subclass, a monkeypatch, a careful study of the framework\u0026rsquo;s internals before you can change your own behavior. That friction is the abstraction telling you it no longer fits. The loop has become your product, and you are still renting it.\nWhat we carry forward The reframe is the whole point, so I\u0026rsquo;m restating it again - we do not build agents. We build harnesses, and the agent is what the harness does when we run it. The model is a component we call, not the core we wrap. The loop, the tools, and the context discipline are the artifact — the thing we version, the thing we debug, the thing where our engineering actually lives. Frameworks abstracted that artifact when abstracting it was the right trade. For a growing class of applications, it no longer is, and the tell is the moment we find ourself fighting the framework to change behavior we should simply own.\nThere is a question this opens : A harness is one kernel running one agent. Real enterprises do not want one agent — they want many, delegating to and falling back on each other, bounded so they cannot run away. That is a network of kernels, and a network of kernels needs something a single harness was never meant to provide: discovery, a protocol between agents, and a policy layer that enforces what each is allowed to do regardless of what it decides. The harness solved the single agent cleanly. The layer above it — how bounded ecosystems of agents interoperate without going off the rails — is where the next hard problem is, and it is not solved by making the harness bigger. That seems like a topic for a different post :) .\nReferences ReAct - Yao et al. in 2022 (arXiv:2210.03629)\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nBuilding Effective Agents\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nCline-SDK, Rebuilt Harness\u0026#160;\u0026#x21a9;\u0026#xfe0e;\nA Practical Guide to Building Agents\u0026#160;\u0026#x21a9;\u0026#xfe0e;\n","permalink":"https://blog.girishshekhar.com/posts/harness-is-the-agents/","summary":"For two years we have talked about building agents as if an agent were a thing you construct. It isn\u0026rsquo;t. The harness — the loop, the tools, the context discipline — is the artifact you actually build, version, and debug. This post reframes the agent as emergent behavior, explains why agent frameworks now feel wrong-shaped in production, and offers a way to decide when to own the loop and when to rent it.","title":"The Harness Is the Agent"},{"content":"The Argument So Far Part 1 covered how work gets decomposed and coordinated: direct delegation, handoff, cascaded delegation, scatter-gather, and chaining. [Part 2] covered how agents self-correct, find each other, and handle failure: iterative refinement, progressive discovery, fallback, and speculative execution.\nEvery pattern in both posts is, at root, an answer to the same question: who receives what, and what do they know when they act on it?\nThe answer to that question is context — the shared state that an agent carries into every decision, every tool call, every handoff. And context in an agent system has properties that have no clean analog in classical distributed systems: it is expensive to replicate, lossy when compressed, and actively harmful when it grows too large.\nThis post is about context. It is the part of the engineering problem that the patterns from Parts 1 and 2 do not fully solve — and the part that becomes the dominant constraint when you try to coordinate agents asynchronously across disconnected environments.\nThe Naive Mental Model The naive mental model of multi-agent context sharing is additive: more context passed to a subagent means better results, because the subagent has more information to work with. Larger context windows enable this, and the frontier models have made those windows very large. The limit appears to be mainly cost.\nThis is wrong in two ways that matter.\nThe first is positional. The \u0026ldquo;lost in the middle\u0026rdquo; finding (Liu et al., published in Transactions of the Association for Computational Linguistics, 2024) showed that LLM performance follows a U-shaped curve over context position. Information at the beginning and end of a prompt receives disproportionate attention. Information in the middle — regardless of its relevance or importance — is systematically underweighted. This holds even for models explicitly designed for long-context operation.\nA model handed a 180k-token context with the critical instruction at position 90k is not operating on full information. It is operating on a degraded version of that information, biased toward what appeared first and last. Passing more context does not always produce better results. Sometimes it produces worse ones.\nThe second is economic. Multi-agent architectures using frontier models at every layer are expensive. The practical response the industry has converged on is model heterogeneity: use a large frontier model for reasoning and synthesis as the orchestrator, and smaller, faster, cheaper models for scoped subtasks as the specialist. OpenClaw\u0026rsquo;s routing architecture does this explicitly, routing across Claude, GPT, and Gemini based on task type. Google\u0026rsquo;s Gemini CLI does it via intelligent routing between Flash-Lite, Flash, and Pro. OpenAI\u0026rsquo;s GPT-5.1 Auto does it between Instant and Thinking variants.\nThe consequence of model heterogeneity for context is significant. When you spawn a specialist subagent on a smaller model, the question of what context to pass is not just a cost question — it is a capability question. A smaller model handed a 180k-token context may not be able to reason over it as effectively as the orchestrator that produced it. The right context for a large model and the right context for a small model may be different in kind, not just in size.\nThe Real Problem The real problem is not how to share context. It is how to decide what to share, when to summarize versus pass in full, and how to make that decision without the decision itself becoming the expensive part.\nThis maps cleanly to a classical distributed systems tradeoff. In synchronous RPC, you pass the full request payload and block for the response — full fidelity, high latency, high coupling. In a message queue, you pass a structured payload and move on — lower fidelity by design, lower coupling, async. In shared state (a database, a distributed cache), you don\u0026rsquo;t pass anything (beyond referential identifiers) — all parties read from the same source of truth, with the coordination costs pushed into the consistency model.\nAgents face the same three options:\nPass in full. The subagent gets the full conversation history or the full document. High fidelity, but the lost-in-the-middle degradation applies, and the token cost is linear in context size. Summarize before passing. The orchestrator compresses the relevant context into a scoped prompt before delegation. Lower cost, but the summarization is lossy. The specifics that the subagent needed most are the ones most likely to be compressed away. Pull on demand. The subagent receives a minimal prompt and is given tools to retrieve additional context when it needs it — reading from a file, querying a database, calling back to the orchestrator. Higher round-trip cost, but the subagent pulls exactly what it needs rather than receiving everything the orchestrator thought it might need. e.g. The right answer depends on the task.\nVerification tasks — where the subagent is checking work against a known standard — sometimes need full context, positioned carefully to avoid the middle-of-context degradation. Implementation tasks — where the subagent needs precise specifications — benefit from summarized context that front-loads the constraints. Exploration tasks — where the subagent\u0026rsquo;s job is to survey a space — benefit from minimal initial context and pull-on-demand retrieval. The wrong strategy applied to the wrong task is not benign. A verification subagent given a summarized prompt when it needed full context will make confident calls based on the summary. The edge case that was compressed away — \u0026ldquo;preserve existing session token behavior\u0026rdquo; from the chaining example in Post 1 — is exactly the kind of specific constraint that summarization discards. The subagent finishes, reports success, and the failure surfaces only when the result is tested or used. No exception was raised. No retry was triggered. The context strategy was wrong and the system had no way to know it.\nAnthropic\u0026rsquo;s Claude Code makes this concrete. The default for Explore subagents is minimal context — fresh start, scoped task, summarized results returned to the orchestrator. The exception is fork mode, where the subagent inherits the full parent context because the task genuinely requires awareness of everything that came before. Fork mode is not the default because the cost and the positional degradation are both real. It is the explicit override for tasks where full context is genuinely necessary.\nContext Decision Framework Task Type Context Strategy Reasoning ────────────── ────────────────── ───────────────────────────────── Exploration Minimal + pull-on-demand Subagent should discover, not confirm Implementation Summarized + front-loaded Constraints must be precise and early constraints in context to avoid positional loss Verification Full context, carefully Needs ground truth; position matters — structured put the standard at the top Long-horizon Compress across context Context windows fill; compression must coordination windows preserve decisions, not conversation The compounding problem in long-horizon tasks is context window exhaustion. An agent working on a complex task across many turns will eventually fill its context window. The options are to compress (losing detail), to truncate (losing history), or to save state externally and start a new context window with a summary. Claude Code\u0026rsquo;s architecture handles this with a three-layer compression strategy: recent turns in full, older turns summarized, key decisions and constraints extracted to a persistent file that anchors each new context window. The file is the memory. The context window is the working set.\n10. Asynchronous Collaboration The patterns in Parts 1 and 2 are predominantly synchronous: the caller blocks until the callee responds, and the interaction happens within a single session or context window. Asynchronous collaboration is the pattern where agents submit tasks and retrieve results without blocking — and where agents may coordinate across sessions, across time, and across organizational boundaries.\nIn classical distributed systems, async collaboration is the message queue: post a task, receive an acknowledgment and an ID, poll or receive a callback when results are ready. This decouples producers from consumers, enables work to span time boundaries, and allows the system to absorb load spikes without cascading them to all participants.\nMCP\u0026rsquo;s Agents Working Group is shipping async support as SEP-1391. The current MCP protocol is synchronous — when a tool is called, the caller blocks until it returns. The async extension lets a server kick off long-running work, return immediately with a task handle, and let the client retrieve results when they are ready. The framing is \u0026ldquo;call-now, fetch-later\u0026rdquo; — submit a task, continue doing other work, retrieve when ready.\nA2A includes async support at the protocol level by design: task management with defined lifecycle states (pending, working, completed), real-time status notifications via server- sent events, and streaming support for long-running tasks. This is designed explicitly for the cross-vendor, cross-cloud scenario — an orchestrator in one environment delegating to a specialist in a completely different one, without holding a connection open for minutes or hours.\nThe harder version of async collaboration is coordination across fully disconnected environments. An orchestrator running on Google infrastructure needs to delegate to a specialist running on AWS, operated by a different organization, possibly on a different model. A2A provides the protocol for the delegation. The unsolved part is context: how does the orchestrator pass enough context for the specialist to do useful work, across an organizational boundary, to a system it has no visibility into?\nThe naive answer is to pass everything. Consider what that means in practice: the orchestrator serializes its full context — 80k tokens of conversation history, tool call results, intermediate reasoning — and posts it to the specialist\u0026rsquo;s A2A endpoint. The specialist is running on a smaller model with a 32k context window. The payload exceeds the window. The specialist truncates, or errors, or processes a degraded version of the context and returns a confident but wrong result. The orchestrator receives it, synthesizes it into a final response, and the user sees the output. The cross-organizational delegation \u0026ldquo;succeeded.\u0026rdquo; The result was wrong from the moment the context was truncated.\nThe emerging answer is reference-based context: instead of embedding context inline, the orchestrator passes a structured task description and references to the resources the specialist can retrieve on demand. The specialist pulls what it needs, when it needs it, from sources it can access. This is the pull-on-demand model applied to cross- organizational delegation. It requires that the resources be accessible to the specialist — which requires trust, access control, and a shared understanding of what the references point to.\nThe Decentralized Future Every pattern in this series assumes some form of coordination authority — an orchestrator/a supervisor/a root agent that owns the trace and synthesizes the result. The decentralized version removes that authority: agents discover each other dynamically, coordinate peer-to-peer, and reach outcomes through emergent collaboration rather than top-down direction.\nThis is where the research is, not yet where production is.\nThe Google and MIT scaling paper (Kim et al., December 2025) is the most relevant quantitative evidence. In their evaluation of decentralized architectures — peer-to-peer coordination with no central orchestrator — independent agents amplified errors up to 17 times across a task, because mistakes propagated unchecked from one agent to the next with no centralized validation step. Centralized orchestration limited error propagation to approximately four times, because the orchestrator validated outputs before passing them along.\nMore agents, without coordination, does not mean better outcomes. The Google/MIT paper\u0026rsquo;s finding is precise: independent agents amplified errors 17x. Centralized orchestration held error propagation to 4.4x. The coordination layer is not overhead — it is the error-correction mechanism.\nThis is not an argument against decentralization. It is an argument that decentralization requires explicit error-checking at every hop, which is expensive, and that the cost of getting this wrong is not a degraded result — it is a compounding failure that reaches the user having been validated by no one.\nThe gossip protocol research (Khan et al., December 2025) frames this as an open engineering problem: how do you enable emergent coordination across large numbers of agents without a central planner, while maintaining coherence and resisting error cascade? Gossip mechanisms — the same epidemic protocols distributed systems use for state dissemination across large clusters — are proposed as a complement to structured protocols. Instead of direct agent-to-agent messaging for every coordination need, agents propagate state updates probabilistically through the network, allowing the system to converge on shared context without requiring every agent to communicate with every other agent directly.\nThe practical state today: the ecosystem is fragmented by vendor. Companies building on Google ADK, Microsoft\u0026rsquo;s Agent Framework, and OpenAI\u0026rsquo;s Agents SDK are building in distinct environments with different state models, different context representations, and different coordination primitives. A2A is the bet that these environments will need to interoperate. That bet is reasonable. The interoperability problem is real. But A2A solves the protocol problem, not the context problem. Two agents can now speak the same language. They still need to agree on what to say.\nDesign Principles Three posts of patterns lead to a small number of principles. They are not new. They are the distributed systems principles applied to a new medium, with the adjustments the medium requires.\nPartition along independence boundaries. Before reaching for a multi-agent architecture, establish that the subtasks are genuinely independent. If step B requires the output of step A, parallelization adds overhead without adding throughput. The Google/MIT paper is precise: multi-agent coordination improves performance on parallelizable tasks by up to 81% and degrades it by up to 70% on sequential ones. The question is not \u0026ldquo;can I use multiple agents?\u0026rdquo; — it is \u0026ldquo;are the subtasks actually independent?\u0026rdquo; Minimize what crosses the boundary. Every handoff is a compression opportunity and a compression risk. Pass what the receiver needs to do its job, not everything you have. The Claude Code architecture demonstrates this working at scale: Explore subagents start fresh, return summaries, and the noise from their traversal never contaminates the main context. The fork is the explicit override, not the default. Position context intentionally. When you do pass context, where it lands in the prompt matters. Constraints, task specifications, and ground truth belong at the top — never in the middle. The lost-in-the-middle degradation is not a bug to be patched; it is a property of the architecture to be designed around. Make failures explicit and specific. The iterative refinement patterns work because the failure signal is deterministic — a failed test, a linter error, an exception. Vague failure signals produce vague reflections produce vague corrections. If you are building a loop that catches and corrects errors, the first design decision is the error signal, not the retry logic. Use centralized coordination until you have a reason to decentralize. The error amplification numbers from the scaling paper are not theoretical. Four times error propagation with a supervisor is a tractable engineering problem. Seventeen times without one is not. Start with a supervisor. Move toward decentralization only when throughput requires it and only with explicit error-checking at every hop. What Remains Unsolved The coordination patterns are converging. The protocol layer is mostly settled. The context problem is not.\nThe specific open question is how to make the context routing decision — full pass, summary, or pull-on-demand — reliably, cheaply, and at scale. If you need a model to evaluate what context is relevant before you can pass it, you have added a step. If that step is cheap and accurate, it pays for itself. If it is expensive or unreliable, you have made the problem worse.\nThe MCP roadmap\u0026rsquo;s progression toward reference-based results — where tool calls return pointers to large payloads rather than embedding them inline — is the right direction. A2A\u0026rsquo;s design for stateless cross-organizational delegation, where specialists pull context on demand rather than receiving it upfront, is the right direction. Anthropic\u0026rsquo;s Claude Code\u0026rsquo;s three-layer compression strategy for long-horizon tasks — full recent context, summarized older context, key decisions in a persistent file — is the right direction.\nNone of these is a complete solution. They are engineering toward a cleaner separation between the reasoning layer (what the agent does) and the memory layer (what the agent knows and how it retrieves it). When that separation is clean, the patterns in this series compose well. When it is not, the patterns work against each other — scatter-gather produces results too large to synthesize, chaining compresses specifics into noise, delegation hands off context that the receiver cannot use effectively.\nThe mental model from Part 1 holds all the way through: agent communication is message passing where the message is a lossy, positionally biased, token-expensive representation of shared state. Every design decision is a decision about how much of that shared state to pass, to whom, and in what form. Getting that decision right — systematically, at scale, across heterogeneous models and organizational boundaries — is the engineering problem the next generation of agent infrastructure is being built to solve.\nReferences Liu, N.F. et al., \u0026ldquo;Lost in the Middle: How Language Models Use Long Contexts,\u0026rdquo; TACL 2024 — https://arxiv.org/abs/2307.03172 Kim, Y. et al., \u0026ldquo;Towards a Science of Scaling Agent Systems,\u0026rdquo; arXiv December 2025 — https://arxiv.org/abs/2512.08296 Google Research blog on the scaling paper — https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/ Khan, N.I. et al., \u0026ldquo;A Gossip-Enhanced Communication Substrate for Agentic AI,\u0026rdquo; December 2025 — https://arxiv.org/pdf/2512.03285 MCP Official Roadmap (2026) — https://modelcontextprotocol.io/development/roadmap MCP async support (SEP-1391) — https://modelcontextprotocol.info/blog/mcp-next-version-update/ A2A Protocol documentation — https://a2a-protocol.org/latest/ LMCache, \u0026ldquo;Context Engineering \u0026amp; Reuse Pattern Under the Hood of Claude Code,\u0026rdquo; December 2025 — https://blog.lmcache.ai/en/2025/12/23/context-engineering-reuse-pattern-under-the-hood-of-claude-code/ MCP tool description quality paper, February 2026 — https://arxiv.org/html/2602.14878v1 ","permalink":"https://blog.girishshekhar.com/posts/agent-coordination-patterns/3-context-engineering-is-the-real-problem/","summary":"The coordination patterns are well understood. The protocol layer is converging. The unsolved problem is context: how do you decide what to pass between agents, when to summarize versus delegate in full, and how to do this without the routing decision itself becoming the expensive part. This post lands the thesis — and examines what happens when agents coordinate asynchronously across disconnected environments.","title":"Context Is the Real Engineering Problem"},{"content":"Picking Up Where Part 1 Left Off Part 1 covered the synchronous coordination patterns: direct delegation, handoff, cascaded delegation, scatter-gather, and chaining. Those patterns answer one question — how do you decompose work across agents and coordinate the results?\nThis post covers a different set of concerns.\nWhat happens when an agent produces a wrong answer and needs to correct it? What happens when an agent doesn\u0026rsquo;t know which specialist exists for a given task? What happens when the model or service it depends on is unavailable? And how do you reduce the latency cost of the classification and routing decisions that sit in front of every pattern from Part 1? These are the reliability and discovery problems. In classical distributed systems, they have well-established names: control loops, service discovery, circuit breakers, and speculative execution. In the agent world, each pattern takes a specific form shaped by the medium — natural language context rather than structured messages — but the underlying engineering logic is the same.\n6. Iterative Refinement An agent produces an output, evaluates it against a success criterion, generates a reflection on what went wrong, and uses that reflection as additional context for the next attempt. The loop continues until the criterion is met or an iteration cap is reached.\nThis is not multi-agent verification. There is no second agent reviewing the first agent\u0026rsquo;s work. The same agent is acting as its own evaluator and critic — producing, assessing, and correcting in a closed loop.\nThe naive implementation of this is a retry loop: if the agent fails, run it again with the same prompt. This is not self-correction. It is repetition. The same model, given the same input, will produce the same output distribution. What makes iterative refinement work is that the failure is converted into a specific, informative signal — and that signal becomes part of the next attempt\u0026rsquo;s input. The loop is not \u0026ldquo;try again.\u0026rdquo; It is \u0026ldquo;try again with this specific information about why the last attempt was wrong.\u0026rdquo;\nIn classical distributed systems, this is the control loop: observe current state, compute error against desired state, apply correction, repeat. The PID controller is the canonical example. The agent version replaces the error signal with a verbal reflection and the correction with a new attempt conditioned on that reflection.\nThe formal version is the Reflexion framework (Shinn et al., NeurIPS 2023). The agent maintains two memory components: short-term memory holding the most recent trajectory, and long-term memory accumulating reflections across trials. On each failure, the agent converts the failure signal into verbal feedback — a specific, actionable description of what went wrong and how to approach it differently. The next attempt is conditioned on the accumulated reflection history. Reflexion showed an 11% improvement on HumanEval coding benchmarks over the GPT-4 baseline, and a 22% improvement on sequential decision-making tasks within 12 trials.\nControl Loop (Distributed Systems) Iterative Refinement (Agent World) ────────────────────────────────── ─────────────────────────────────── Desired State Task + Success Criterion \u0026lt;───── | | | v v | Controller ──► Actuator ──► Plant Agent (Actor) | ^ | | | | v v | Error Signal ◄── Sensor ◄── Output Evaluator checks output | | | v | Fail: Self-Reflection Model | generates verbal feedback | | | v | Feedback stored in memory | | | v | Next attempt conditioned ──────── on accumulated reflections The production instantiation is the Ralph Wiggum technique — named after the persistently optimistic Simpsons character. In its purest form, it is a bash while loop: run Claude Code on a task, intercept the exit signal, feed the same prompt back, repeat until a completion promise is emitted. The sophistication is entirely in the prompt: the success criterion must be deterministic (all tests green, linter clean), and the failure output must be fed back as context for the next iteration. Failures from tests and linters become the signal. The loop handles retry logic. The operator\u0026rsquo;s job is to make success detectable and failure informative.\nThe known failure mode of single-agent iterative refinement, documented in the Multi-Agent Reflexion paper (December 2025), is degeneration of thought. When one model acts as its own generator and critic, it reproduces the same flawed reasoning structure across iterations — the same structural mistake reappears with different surface wording. The fix is to introduce a second agent with a deliberately different perspective as the critic. At that point, iterative refinement becomes multi-agent verification, and the pattern changes character: it is no longer a control loop but a debate.\nTip: use single-agent iterative refinement when the success criterion is deterministic and the failure signal is specific. Use multi-agent verification when the evaluation requires judgment that the actor is unlikely to apply accurately to its own work.\n7. Progressive Discovery An agent does not know at startup what other agents or capabilities are available to it. It discovers them at runtime, lazily, as it needs them — rather than loading a complete registry into context upfront.\nIn classical distributed systems, this is service discovery: instead of hardcoding the address of every service you might call, you query a registry (Consul, Eureka, DNS-SD) at runtime for services matching your needs. The registry maintains the catalog; callers query on demand.\nProgressive discovery applies this pattern to capabilities rather than endpoints.\nThe MCP version of this problem starts with tool catalog bloat. When an agent connects to an MCP server, it receives a list of available tools. If the server exposes many tools, all tool descriptions land in the agent\u0026rsquo;s context on every turn. A February 2026 paper studying MCP deployments in production found that tool metadata repeatedly injected into a model\u0026rsquo;s context during a typical interaction is inflating token usage and increasing execution cost. Two mechanisms are emerging to address this. Agent Skills exposes only tool names, descriptions, and paths at startup, loading full instructions only when a skill is selected. Tool Search lets the model query for a tool on demand rather than loading the full catalog upfront.\nThe 2026 MCP roadmap adds MCP Server Cards: a .well-known/agent.json endpoint that any registry or crawler can read without establishing a live connection. This makes capability discovery offline and non-blocking — the agent card is available before a session begins, the same way a DNS record is available before a TCP connection is opened.\nA2A takes the same approach at the agent level. Every A2A-compliant agent exposes an Agent Card advertising its capabilities in a standardized format. A client agent can read this card without establishing a live session. Registries can index Agent Cards and present them to orchestrators as a searchable catalog.\nEager Loading (simple \u0026amp; naive) Progressive Discovery (correct) ─────────────────────────────── ─────────────────────────────── Agent starts Agent starts | | v v Load ALL tool descriptions Load tool names + short descriptions into context (200 tools = large into context (lightweight) context hit on every turn) | | v v Task arrives Task arrives | | v Agent reasons over bloated context Agent queries: \u0026#34;which tool handles X?\u0026#34; | | v v Relevant tool buried in noise Retrieve full tool definition on demand | v Context stays clean The gotcha is that the quality of discovery depends entirely on the quality of the metadata. A poorly described Agent Card or tool description produces the same result as a misconfigured service registry: the agent routes to the wrong specialist, or fails to route at all, because the capability description didn\u0026rsquo;t match the query. In distributed systems you validate service registry entries with health checks and schema contracts. In agent systems, the equivalent discipline — writing precise, unambiguous capability descriptions — is mostly informal today.\n8. Fallback and Circuit Breaking When a primary agent or model fails — due to unavailability, capability mismatch, or quality threshold not met — the system routes to an alternative without surfacing the failure upstream.\nIn classical distributed systems, this is the circuit breaker pattern (Nygard, \u0026ldquo;Release It!\u0026rdquo;): when a downstream service starts failing, the circuit opens and subsequent calls fail fast rather than waiting for timeouts. When the downstream recovers, the circuit closes. The purpose is to prevent one failing service from cascading its failure through the entire dependency graph.\nIn agent systems, fallback operates at two levels.\nAt the provider level, it is model redundancy: if the primary model provider degrades, route to an alternative. OpenClaw, the open-source autonomous agent runtime, implements this explicitly — routing across Claude, GPT, and Gemini with configurable fallback chains and exponential backoff. The agent continues operating across a provider outage by transparently routing to the next available option. Lets consider what happens without it. An orchestrator spawns three parallel Explore subagents. Halfway through the task, the primary provider degrades. All three subagent calls begin timing out. The orchestrator waits. The timeouts cascade. The task fails silently — not with an exception, but with an empty or partial result that the orchestrator has no way to distinguish from a completed one. The user sees a wrong answer. The system saw nothing unusual.\nAt the capability level, it is quality-based escalation: if a lightweight model produces output below a quality threshold, escalate to a more capable one. Amazon Bedrock\u0026rsquo;s supervisor with routing mode implements this: when a simple query is routed to a specialist and that specialist cannot handle it — ambiguous intent, out-of-scope request — the system escalates to the full supervisor rather than returning an error. The gotcha is determining what constitutes a failure worth routing around. A model that produces a low-quality response is not obviously distinguishable from one that produces a high-quality response to a subtly wrong question. Deterministic failure signals — a failed test, an exception, an empty result set — make circuit breaking tractable. Soft failures — \u0026ldquo;this answer seems off\u0026rdquo; — require either a quality evaluator in the loop, which adds cost, or a human in the loop, which adds latency. The Reflection loop and the fallback pattern are often combined: the agent tries, the loop detects failure via a deterministic signal, the fallback routes to a more capable model for the next attempt.\n9. Speculative Execution A system begins processing the most likely version of a request before it has confirmed which version is correct, then commits to that execution or discards it once the actual request is confirmed.\nIn classical distributed systems, speculative execution appears in CPU branch prediction (execute both paths of a conditional, discard the wrong one when the branch resolves), read-repair in Cassandra (speculatively repair stale replicas in the background while serving the read), and optimistic concurrency control (execute a transaction assuming no conflict, validate on commit, retry on conflict).\nAt the inference layer, speculative decoding is production-confirmed. A small draft model proposes multiple tokens ahead; a large verify model validates them in a single parallel forward pass. Tokens where the models agree are accepted; tokens where they diverge are rejected and replaced. The output is identical to what the verify model would have produced alone — the speedup comes from exploiting GPU capacity that single-token autoregressive generation leaves idle. Production deployments achieve two to three times throughput improvement with no quality loss.\nThe architectural extension to the agent coordination layer is the obvious next step: pre-execute the most likely branch of a request while classification is still in progress, then commit if the branch was correct and discard if not. If the classification accuracy is high enough and the pre-execution is cheap enough relative to the cost of waiting, the expected latency savings outweigh the waste on incorrect branches. This is the same tradeoff CPU architects have been making for decades.\nThe Reliability Stack The four patterns in this post form a reliability stack. Progressive discovery determines whether the right agent or capability can be found at all. Iterative refinement catches quality failures within a single agent\u0026rsquo;s work. Fallback and circuit breaking handle infrastructure and model failures. Speculative execution reduces the latency cost of the routing and classification decisions that sit in front of everything else.\nIn a well-designed system, these are not independent choices — they compose. A scatter-gather orchestrator uses progressive discovery to find its specialists, iterative refinement if a specialist\u0026rsquo;s output fails validation, fallback if a specialist is unavailable, and speculative execution to reduce the latency of the routing decision that started the whole thing.\nEvery pattern in this post works better when it knows precisely what \u0026ldquo;wrong\u0026rdquo; looks like. Deterministic signals — failed tests, exceptions, empty responses, quality scores below a threshold — make these patterns tractable. Vague signals — \u0026ldquo;this seems off\u0026rdquo; — make them expensive. Designing for failure detection is not an afterthought in agent systems. It is the prerequisite for everything in this post.\nThere is a deeper assumption embedded in all of this: that the context being passed between agents is well-formed — correctly scoped, correctly positioned, correctly sized for the receiving model. The reliability patterns here assume that assumption holds. Post 3 is about what happens when it doesn\u0026rsquo;t — and why getting the context decision right is harder than any individual coordination or reliability pattern makes it appear.\nReferences Shinn, N. et al., \u0026ldquo;Reflexion: Language Agents with Verbal Reinforcement Learning,\u0026rdquo; NeurIPS 2023 — https://arxiv.org/abs/2303.11366 Multi-Agent Reflexion (MAR), December 2025 — https://arxiv.org/html/2512.20845 Ralph Wiggum loop technique — https://awesomeclaude.ai/ralph-wiggum MCP Official Roadmap (2026) — https://modelcontextprotocol.io/development/roadmap MCP tool description quality paper, February 2026 — https://arxiv.org/html/2602.14878v1 A2A Protocol documentation — https://a2a-protocol.org/latest/ Amazon Bedrock multi-agent collaboration GA, March 2025 — https://aws.amazon.com/blogs/aws/introducing-multi-agent-collaboration-capability-for-amazon-bedrock/ Speculative decoding production guide — https://introl.com/blog/speculative-decoding-llm-inference-speedup-guide-2025 ","permalink":"https://blog.girishshekhar.com/posts/agent-coordination-patterns/2-feedback-discovery-fault-tolerance/","summary":"The synchronous coordination patterns tell you how to decompose work. This post covers what happens when that work goes wrong — or when agents don\u0026rsquo;t yet know who can help. Iterative refinement, progressive discovery, fallback, and speculative execution are the patterns that govern self-correction, capability negotiation, and graceful degradation in production agent systems.","title":"Feedback Loops, Discovery, and Fault Tolerance"},{"content":"Why This Post Exists In fall 2023, I was in a room with a Distinguished Engineer at Amazon, working through what would become Alexa+\u0026rsquo;s agent architecture. The question on the table was simple: when a generalist agent needs help from a specialist, what does that communication look like? Do you delegate and wait? Hand the customer off entirely? Broadcast and see who responds?\nI wrote a one-pager based on this conversation, that gave structure to the problem before it was fully understood. It defined terms: operations, APIs, tools, agents, orchestrators. It sketched two coordination patterns — delegate (the generalist stays in the loop) and handoff (the specialist takes over). It asked three questions it couldn\u0026rsquo;t yet answer: when do you create a new agent versus a tool, how many agents will there be, and how does the ecosystem discover who can do what.\nThat was late 2023. In the two and a half years since, the industry produced a wave of frameworks, protocols, and papers trying to answer those questions. MCP standardized agent-to-tool communication. A2A standardized agent-to-agent communication. Amazon Bedrock shipped multi-agent collaboration as a managed service. Google and MIT published the first quantitative scaling principles for agent systems. Anthropic\u0026rsquo;s Claude Code became one of the most studied real-world agent implementations. Every hyperscaler converged their agent frameworks toward production.\nWhat emerged is not a new science. It is distributed systems with a different payload.\nEvery pattern in the agent communication landscape — delegation, handoff, scatter-gather, chaining, fallback, speculative execution — has a direct analog in how distributed systems engineers have coordinated processes for decades. The orchestration problems are the same. The failure modes are recognizable. The tradeoffs are familiar.\nWhat differentiates them is that you are passing natural language and semantic context instead of structured payloads over a wire. And that has cascading effects on how you reason about correctness, cost, and failure.\nThis is the first of three posts mapping those patterns. This post covers the synchronous coordination patterns — the ones most teams encounter first. Post 2 covers feedback loops, discovery, and fault tolerance. Post 3 lands the thesis: why context is the real engineering problem across all of them, and where the decentralized future creates challenges the current patterns don\u0026rsquo;t fully solve.\nThe Mental Model The instinct when approaching multi-agent systems is to treat natural language context like a well-typed API payload (not following the strict semantics, but fundamentally the same kind of thing); a message with a sender, a receiver, and a contract. It is not.\nIn a classical distributed system, processes communicate by passing messages with schemas — structured contracts that define what the sender asserts and what the receiver does with it. When the contract is violated, something deterministic fails: a type error, a serialization exception, a protocol rejection.\nIn an agent system, the messages are natural language. The contracts are implicit. When they are violated, you do not get an exception. You get a subtly wrong response, a misunderstood task, a subagent that made a reasonable inference from bad context and propagated the error forward. The failure mode is qualitative degradation, not a stack trace.\nEverything else — the coordination patterns, the discovery mechanisms, the context-sharing strategies — is engineering around this one fact.\nMental model: Agent communication is message passing where the message is a lossy, positionally biased, token-expensive representation of shared state. Every design decision in a multi-agent system is, at root, a decision about how much of that shared state to pass, to whom, and in what form.\nDistributed systems engineers reason about this constantly under different vocabulary — partitioning, coupling, shared mutable state, message ordering, idempotency. The underlying question is identical: how do you coordinate independent processes without creating the coupling that defeats the purpose of independence?\nThe patterns that follow are all answers to that question. None of them are new. What is new is that getting the answer wrong is harder to detect, because the failure looks like a slightly worse result rather than a crashed process.\nThe Protocol Layer Before the patterns, it helps to understand what infrastructure the industry has converged on. Two protocols define the current stack.\nMCP (Model Context Protocol, Anthropic, November 2024) standardizes how agents connect to tools, data sources, and external services. Think of it as the plugin system: a single interface through which an agent discovers and invokes capabilities without bespoke integration code. Every major platform supports it. OpenAI abandoned their proprietary Assistants API in favor of MCP. The 2026 roadmap focuses on two problems: scaling the transport layer for production deployments, and progressive discovery — letting agents learn what a server can do without loading the full tool catalog into context upfront.\nA2A (Agent2Agent Protocol, Google, April 2025), albeit very nascent, standardizes how agents communicate with other agents. Where MCP is the plugin system, A2A is the networking layer. Agents advertise capabilities via an Agent Card, accept tasks from client agents, and return results — without exposing their internal state, memory, or tooling to the caller. Microsoft\u0026rsquo;s Agent Framework, Google\u0026rsquo;s ADK, and CrewAI all support it natively.\nThe framing Google uses is worth keeping: MCP connects agents to tools; A2A connects agents to agents. You use MCP to give an agent access to a database. You use A2A to let that agent delegate to a specialist in another organization\u0026rsquo;s infrastructure.\nThese two protocols are the substrate. The patterns below operate on top of them.\nThe Patterns 1. Direct Delegation The orchestrator receives a request, determines it requires a specialist, and delegates a scoped subtask. The specialist completes it and returns a result. The orchestrator synthesizes the final response. Control never transfers — the orchestrator stays in the loop throughout.\nThis is the pattern we settled on in 2023 and it remains the most common in production today. The reason is observability. When the orchestrator retains control, you always have the trace: what was sent, what came back, where the chain stands. In a world where LLM behavior is probabilistic and debugging is hard, that trace is the difference between diagnosable failures and mysterious ones.\nIn classical distributed systems, this maps to a synchronous RPC call: service A invokes service B with a well-defined request, blocks until it gets a response, and integrates that response into its own output. The contract is explicit. The caller owns the result.\nThe implementation detail that matters most in the agent version is what the orchestrator passes to the specialist. Anthropic\u0026rsquo;s Claude Code, whose internals have been studied in unusual detail by the LMCache team, reveals the right default. When the main agent spawns an Explore subagent, it does not pass its full conversation history. The subagent starts fresh, with only the delegated task in its context. When the subagent finishes, only its summary returns to the main context — not the intermediate file reads, search results, or exploratory tool calls. The noise stays inside the subagent\u0026rsquo;s context window and never touches the main conversation.\nClaude Code\u0026rsquo;s Explore→Plan→Implement subagent chain achieves a stunning 92% prompt prefix reuse rate across phases. This is an intentional architecture choice, not an optimization — each phase inherits a lean, scoped prompt rather than the accumulated noise of everything that came before.\nThe alternative — passing full context to every subagent — is expensive, noisy, and runs directly into the lost-in-the-middle problem (more on this in Post 3). A subagent handed a 180k-token conversation history with the relevant task buried in the middle will not perform as well as one handed a clean, scoped prompt.\nNaive (full context) Direct Delegation (scoped) ──────────────────── ────────────────────────── Orchestrator Orchestrator │ [180k tokens: full history] │ [Scoped task prompt only] │ │ ▼ ▼ Specialist Specialist │ reasons over bloated context │ reasons over clean, scoped task │ relevant task buried in middle │ ▼ ▼ Result (degraded) Summary only ──► Orchestrator synthesizes final response The key asymmetry: in the naive version, noise travels in both directions. In direct delegation, the orchestrator sends a scoped task and receives only a summary — the specialist\u0026rsquo;s intermediate work never contaminates the main context.\nThe gotcha in direct delegation is when the summary that crosses the boundary is lossy in a way that matters. The compressed handoff captures the shape of prior context; it loses the specifics. For tasks where specifics are everything — a particular middleware behavior, an edge case surfaced three hours ago — the subagent makes a confident call on incomplete information. The failure looks correct until it isn\u0026rsquo;t.\n2. Handoff The orchestrator transfers control entirely to a specialist. The specialist responds directly to the user or downstream system. The originating agent exits the loop.\nIt maps to the call center transfer in the real world: the generalist agent determines it cannot or should not handle the request, identifies the right specialist, and passes the conversation along. The specialist now owns the interaction.\nIn classical distributed systems, handoff appears in load balancer sticky sessions, TCP connection migration, and process handoff in operating system scheduling. The key property is the same: once the transfer completes, the originating party is no longer in the communication path.\nThe model routing systems that every major AI product shipped in 2025 are handoff patterns. When a ChatGPT query is classified as requiring image generation, the request is handed to the image generation model, which responds directly. The classification layer made a routing decision and exited. When a query is classified as requiring deep reasoning, it is handed to the Thinking variant. The router does not synthesize — it transfers.\nThe critical distinction from direct delegation: in handoff, the originating agent has no visibility into what happens after the transfer. There is no trace that connects the original request to the specialist\u0026rsquo;s handling. In TCP connection migration — the cleanest distributed systems analog — the originating party truly exits the path. It has no way to observe what the new endpoint does with the connection. The agent handoff is identical: once the transfer completes, the router is out of the conversation.\nThis matters because when the specialist handles the request poorly, you see the outcome from the user\u0026rsquo;s side but cannot diagnose it from the system\u0026rsquo;s side. The failure is observable; the cause is not. In delegation, the trace connects the symptom to the source. In handoff, you have only the symptom.\nDirect Delegation Handoff ───────────────── ─────── User ──► Orchestrator User ──► Router │ │ │ [scoped task] │ [transfers control] ▼ ▼ Specialist Specialist │ │ │ [result] │ [responds directly] ▼ ▼ Orchestrator ──► User User │ [full trace: request, result, synthesis] [no trace after transfer] Failure: orchestrator sees it, Failure: user sees it, can diagnose it. system cannot. This is why most production agent frameworks have converged on delegation over handoff as the default. Handoff is appropriate when the specialist is authoritative and the originating agent genuinely has nothing to add to the synthesis. It is not appropriate when you need the trace.\n3. Cascaded Delegation Direct delegation applied recursively. Each agent in the chain makes a routing or decomposition decision before passing work along. The result travels back through each layer, with each agent synthesizing before returning upstream.\nIn classical distributed systems, this is the API gateway pattern: a request arrives at the gateway, which routes to a service, which calls a downstream dependency, which queries a database. Each layer adds context, transforms the response, and returns it to the layer above. The caller at the top owns the final composition.\nThe production agent example that best illustrates this is Amazon Bedrock\u0026rsquo;s supervisor mode. The supervisor agent receives a complex request, decomposes it, and delegates subtasks to specialist subagents. Each specialist executes and returns its result to the supervisor. The supervisor synthesizes across all results and returns the final response. This is two levels of delegation: user → supervisor → specialist, with results traveling back up the chain.\nThe depth of the cascade in Bedrock\u0026rsquo;s routing mode is determined dynamically. Simple queries skip the supervisor and route directly to the appropriate specialist — one level of delegation. Ambiguous or complex queries escalate to the supervisor, which coordinates across multiple specialists — two levels. The system adjusts the cascade depth based on query complexity.\nSimple query (1 level) Complex query (2 levels) ────────────────────── ──────────────────────── User User │ │ ▼ ▼ Router ──────────────────► Router │ [direct] │ ▼ │ [escalate] Specialist A ▼ │ Supervisor │ [result] │ │ ▼ ▼ ▼ Router ──────────────────► Specialist A Specialist B │ [result] │ │ ▼ └──── ▼ ──────┘ User Supervisor │ [synthesized result] ▼ User Results travel back up through every layer they descended. The gotcha specific to cascaded delegation is error compounding. A misclassification or a bad decomposition decision at the first level propagates to every downstream agent. The classifier or supervisor, being a chokepoint by design, is also the place where subtle errors have the largest blast radius. A wrong routing decision at level one means every specialist below it is working on the wrong subproblem.\nThe distributed systems mitigation is validation at each layer boundary — the equivalent of schema validation or a contract test between services. In agent systems, this means the supervisor should verify that the specialist\u0026rsquo;s output is coherent with the original task before synthesizing. This adds latency. The tradeoff is explicit.\n4. Scatter-Gather The orchestrator fans out a task to multiple specialists simultaneously, waits for all to complete, and synthesizes their results into a single response. In distributed systems this is the fan-out/fan-in pattern — familiar from search engines, distributed query processing, and MapReduce.\nAnthropic\u0026rsquo;s Claude Code uses scatter-gather as its default exploration strategy. When a task arrives, the main agent spawns three parallel Explore subagents with different exploration goals: one traces a specific implementation, one investigates the relevant API surface, one checks related utilities. All three run simultaneously, return summaries to the main agent, and the main agent synthesizes before spawning the Plan subagent with only those summaries as input.\nThe Google and MIT paper \u0026ldquo;Towards a Science of Scaling Agent Systems\u0026rdquo; (Kim et al., December 2025) quantifies when scatter-gather earns its cost. Multi-agent coordination improved performance on parallelizable tasks by up to 81% over a single-agent baseline — but degraded it by 39 to 70% on sequential tasks. The key variable is whether subtasks are genuinely independent. If step B requires the output of step A, parallelization does not help. The coordination overhead displaces reasoning budget that would have been better spent on the task itself.\nScatter-Gather (Agent World) Fan-Out / Fan-In (Distributed Systems) ───────────────────────────── ────────────────────────────────────── Orchestrator Query Coordinator / | \\ / | \\ Explore Explore Explore Shard 0 Shard 1 Shard 2 Agent 1 Agent 2 Agent 3 (search) (search) (search) \\ | / \\ | / Plan Agent Result Merger | | Final Output Ranked Results The paper introduces a \u0026ldquo;45% rule\u0026rdquo;: if a single agent already solves a task with greater than 45% accuracy, adding more agents yields diminishing or negative returns. Token costs are also non-linear — hybrid scatter-gather architectures consume approximately five to six times more tokens than equivalent single-agent runs.\nThe distributed systems engineer\u0026rsquo;s instinct here is the right one: partition along independence boundaries, and only parallelize when subtasks share no world state. The agent version adds one wrinkle — the \u0026ldquo;state\u0026rdquo; being shared is often context, which does not have a well-defined ownership boundary the way a database row does. Two subagents exploring the same codebase are not mutually exclusive, but they may reach contradictory conclusions that the orchestrator then has to reconcile.\n5. Chaining Each agent in the chain receives the output of the previous agent as its input, transforms or extends it, and passes the result forward. The chain is a pipeline: input flows through a series of specialized stages, each adding value, each depending on what came before.\nIn classical distributed systems, chaining is the Unix pipe, the Kafka topic-to-topic processor, the ETL pipeline. Each stage reads from a well-defined input, produces a well-defined output, and has no knowledge of what comes before or after it in the chain.\nLangGraph\u0026rsquo;s state machine model is the canonical production implementation for agents. Each node in the graph is an agent or function; edges define the flow; state is a typed schema that nodes read from and write to. This forces precision about what each stage consumes and produces. CrewAI\u0026rsquo;s sequential process is the social-metaphor version of the same thing — agents with roles pass task outputs to the next agent in line — with less upfront schema work and correspondingly less control.\nThe failure mode of chaining is context pollution, also called information degradation. In a long chain, the original task specification gets pushed progressively further from the active attention window by the accumulated output of intermediate stages. Agent C receives a summary of Agent B\u0026rsquo;s work, which was itself a summary of Agent A\u0026rsquo;s. The original constraints are three compressions deep. No individual agent made an error; the pipeline architecture degraded the signal.\nOriginal Task Specification: \u0026#34;Refactor auth module to support OAuth2, preserving existing session token behavior\u0026#34; | v Agent A (Analysis) Output: \u0026#34;Auth module uses JWT. Three token types: access, refresh, session.\u0026#34; | v Agent B (Design) Input: Agent A\u0026#39;s output (original constraint about session tokens: attenuated) Output: \u0026#34;Proposed OAuth2 flow with access and refresh tokens.\u0026#34; | v Agent C (Implementation) Input: Agent B\u0026#39;s output (session token constraint: absent) Output: Code that breaks existing session token behavior. The mitigation in distributed systems is explicit contracts at each stage boundary — schema validation, checksums, assertions. In agent chains, the equivalent is either explicit state schemas (LangGraph\u0026rsquo;s approach) or structured handoff prompts that restate the original task constraints at each stage. The cost of the former is upfront schema work. The cost of the latter is prompt size. Both are preferable to discovering the degradation at the end of the chain.\nWhat the Patterns Have in Common Each of the five patterns above is a different answer to the same question: who controls the result, and what do they know when they synthesize it?\nIn direct delegation, the orchestrator controls the result and knows the full task context. In handoff, the specialist controls the result and knows only what the router passed. In cascaded delegation, each layer synthesizes with progressively less context than the layer above it. In scatter-gather, the orchestrator synthesizes across parallel results and must reconcile contradictions without having been in any of the subagent conversations. In chaining, each stage synthesizes with only the output of its immediate predecessor.\nThe distributed systems lens makes the tradeoffs legible. More control = more observability = more overhead. Less control = faster execution = harder to debug. This tradeoff is not new. What is new is that the payload is natural language, so the \u0026ldquo;schema violation\u0026rdquo; that signals something going wrong is not a type error — it is a subtly wrong answer that a user has to notice, in a system where no exception was raised and no trace flagged the failure.\nThat is the fundamental difference. And it is why the patterns in this post converge on the same default: keep the orchestrator in the loop, minimize what crosses the boundary, and make the trace explicit. The distributed systems instincts are right. The cost of ignoring them is just harder to see.\nPost 2 covers what happens when the coordination is right but the execution still goes wrong: the feedback loops, discovery mechanisms, and fault tolerance patterns that govern how agent systems self-correct and recover.\nReferences LMCache, \u0026ldquo;Context Engineering \u0026amp; Reuse Pattern Under the Hood of Claude Code,\u0026rdquo; December 2025 — https://blog.lmcache.ai/en/2025/12/23/context-engineering-reuse-pattern-under-the-hood-of-claude-code/ Kim, Y. et al., \u0026ldquo;Towards a Science of Scaling Agent Systems,\u0026rdquo; arXiv December 2025 — https://arxiv.org/abs/2512.08296 Google Research blog on the scaling paper — https://research.google/blog/towards-a-science-of-scaling-agent-systems-when-and-why-agent-systems-work/ MCP Official Roadmap (2026) — https://modelcontextprotocol.io/development/roadmap Google A2A launch post, April 2025 — https://developers.googleblog.com/en/a2a-a-new-era-of-agent-interoperability/ Amazon Bedrock multi-agent collaboration GA, March 2025 — https://aws.amazon.com/blogs/aws/introducing-multi-agent-collaboration-capability-for-amazon-bedrock/ Claude Code subagent documentation — https://code.claude.com/docs/en/sub-agents ","permalink":"https://blog.girishshekhar.com/posts/agent-coordination-patterns/1-synchornous-patterns/","summary":"Multi-agent systems are not a new class of engineering problem. They are distributed systems with a different payload. This post maps the synchronous coordination patterns emerging in agent ecosystems — delegation, handoff, scatter-gather, chaining — to the primitives distributed systems engineers have used for decades, and explains why the medium changes the failure modes in ways that matter.","title":"The Distributed Systems Playbook in a New Medium"},{"content":"Purpose If you are a Principal or Senior Staff Engineer, your most important resource is not your technical depth. It is your attention. Where you spend it, for how long, and in what mode — these decisions compound across every team you touch. Get them right and you multiply your impact. Get them wrong and you become a bottleneck, a permanent Catcher, or a well-intentioned presence who never quite transfers what they know.\nTwo frameworks have helped me think about this precisely.\nThe first is the PE Roles framework, developed by Mai Lan Tomsen Bukovec at Amazon S3. The second is a concentric circle engagement model I encountered through my mentor, Dhan, which developed independently but addresses related territory. Neither framework is sufficient on its own. Together, they answer two questions that most role frameworks address separately: what mode of engagement does a situation require? and how do you know when to change it?\nThis post walks through both frameworks, maps their relationship, and then focuses on what neither fully addresses: the mechanics of intentional transition — how you move from the center of the circle outward, what you install before you leave, and why the conversations you have with your manager about capacity are not soft skills but engineering decisions.\nTwo Frameworks, One Problem The PE Roles Framework The PE Roles framework, developed in 2016 for Amazon S3 and later refined and shared more broadly, starts from a practical problem: at the scale of a complex service like S3, there are more hard technical decisions to make than any single Principal Engineer can cover. You need a model that lets you leverage broader senior engineering talent, helps PEs focus on the highest-impact work, and gives the organization a shared vocabulary for how technical leadership actually operates.\nThe framework defines six roles:\nSponsor — The project or program lead across multiple teams. A Sponsor\u0026rsquo;s job is not necessarily to make every decision, but to ensure decisions get made. That means driving the forums where debate happens, closing the loop, and escalating when something goes wrong. This is a high-bandwidth role. As a mental model: you should not be actively Sponsoring more than two projects at once, and that number drops to one if you are simultaneously Guiding something else. Guide — Deeply involved in the architecture of a project, but not \u0026ldquo;The Architect.\u0026rdquo; A Guide works through others to produce designs, and produces exemplary artifacts — design docs, bodies of code — that set a pattern the team can carry forward. A Guide influences teams, not just individuals. If you find yourself primarily in 1:1s instead of influencing group decisions, you have drifted from Guide to mentor. Like Sponsoring, Guiding more than two projects simultaneously is a sign something is wrong with how you are allocating your attention. Catalyst — Takes an idea that is \u0026ldquo;kicking around\u0026rdquo; the organization and turns it into a resourced program. Catalysts write the docs and prototypes that drive buy-in with senior decision makers. Crucially, this is a time-bounded role: once a project is resourced and has a team, the Catalyst moves on. They may take on a Guide or Sponsor role on the same project, but that is a separate decision. You cannot effectively Catalyze more than one or two things at once. Tie-Breaker — Makes a decision after a debate and formally closes it with a doc or email to the group. This is a \u0026ldquo;moment in time\u0026rdquo; role. A team stuck in consensus-seeking mode needs a Tie-Breaker; once the decision is made and communicated, the role ends. The risk here is a Tie-Breaker who starts resolving every decision regardless of size — at that point they have transitioned from Tie-Breaker to bottleneck. Catcher — Gets a project back on track, typically from a technical perspective, under tight deadlines. A Catcher does their own rapid diagnosis, formulates a pragmatic path forward, and drives execution until the project is stable. Like the Tie-Breaker, this role has a natural endpoint: once caught, the project no longer needs the Catcher. The anti-pattern is a succession of Catcher engagements with no time for anything else. Participant — Works on something without an explicitly assigned leadership role. Active participation (oncall, picking up a coding task, contributing substantively to design discussions) is valuable. Passive participation (a few points in a meeting, then moving on) should be time-boxed deliberately. The risk is becoming a Participant in too many things — it will consume your daylight hours with no leverage. The Concentric Circle Model The concentric circle model organizes similar territory with a different emphasis. Where the PE Roles framework names what you are doing, the concentric circle model emphasizes how deep your engagement is and, critically, what the expected direction of travel looks like.\n┌──────────────────────────────────────────┐ │ Participant │ │ ┌────────────────────────────────┐ │ │ │ Active Consultant │ │ │ │ ┌────────────────────────┐ │ │ │ │ │ Supervisor │ │ │ │ │ │ ┌──────────────────┐ │ │ │ │ │ │ │ Owner │ │ │ │ │ │ │ └──────────────────┘ │ │ │ │ │ └────────────────────────┘ │ │ │ └────────────────────────────────┘ │ └──────────────────────────────────────────┘ Starting point: center (Owner / Sponsor) Direction of travel over time: outward Velocity: set by team readiness, not by PE comfort At the center is the Owner: single-team scope, deep day-to-day involvement, personally driving the most technically complex and ambiguous work. Moving outward, the Supervisor maintains technical accountability but operates through mechanisms — tracking, guiding, reviewing — rather than direct execution. The Active Consultant engages through the same channels as the broader engineering leadership, attending quarterly reviews and design sessions, but is not monitoring day-to-day progress. The outermost ring is passive participation: available when called, but not actively tracking. The critical design principle of this model is that the center is not the destination. It is the starting condition. A PE entering a high-touch engagement at the Owner level is not choosing a permanent operating mode — they are beginning a time-bounded intervention whose success is measured by how cleanly they can move outward without the project degrading.\nWhere They Fit Together The two frameworks are complementary, not competing. The PE Roles framework gives you the vocabulary for what you are doing in a given engagement: Sponsoring, Guiding, Catalyzing, Catching. The concentric circle model gives you the spatial metaphor for how intensively you are engaged and the direction you should be moving.\nA Catalyst/Sponsor at the start of a complex initiative is operating near the center — high-touch, high-bandwidth. As the team develops its vocabulary and senior engineers internalize the direction, that same Sponsor should be moving outward to a Guide, converting daily involvement into a Supervisory mode and eventually into consultation as a Tie Breaker. A Guide who never exits the center has not guided anyone — they have created a dependency. Dependending on the audit mechanisms installed, you may be required to go back in. A special role which allows you to operate at border but not go back to the center is a Catcher.\nPE Roles framework → What role am I playing? Concentric model → How intensively, and where am I headed? Together → Am I in the right mode, and am I moving in the right direction at the right pace? The failure mode that neither framework addresses well in isolation is this: engineers who understand their current role clearly but have no explicit plan for how or when to change it. They are accurate about what they are doing. They are not deliberate and intentional about when they should stop.\nThe Naive Mental Model: Roles as Static Descriptions The most common way engineers engage with frameworks like these is as inventory tools. You examine your calendar, assign time blocks to different roles, and figure out where your attention is actually going. That diagnostic is valuable. It tells you where you are.\nWhat it does not tell you is where you should be in three months, or what you need to put in place now to get there. This is the naive version of role thinking: roles as stable descriptions of current engagement, rather than as temporary states in a deliberate trajectory.\nEngineers operating with this mental model do good work within their current mode. But they do not build the conditions that let them move. They stay at the center of the circle — or in the Sponsor/Guide mode — long after the team no longer needs them there.\nThe consequence is not just personal. A Principal Engineer who remains in Owner or Sponsor mode past the point of necessity is, by definition, preventing the development of the senior engineers who could be taking that space. The team does not build vocabulary. It does not build the judgment to argue tenets independently. It becomes better at deferring to the Principal, which looks like alignment but is actually stagnation.\nThe Better Mental Model: Roles Are Phases, Not Positions The concentric circle model makes the necessary reframe visible in a way a flat list of role names does not. When you visualize engagement as rings, the implicit direction becomes clear: the goal is to move outward. High-touch engagement at the center is appropriate for a bounded period — long enough to establish vocabulary, build mental models in the team, and identify the senior engineers who will carry the work forward. Then you move out, deliberately.\nThe center is not the goal. It is the starting condition. And the measure of success is not how well you performed at the center — it is how quickly and cleanly you were able to move out, without the project degrading.\nThis reframe changes what \u0026ldquo;doing well\u0026rdquo; looks like for a senior engineer. In the naive model, doing well means executing your current role effectively. In this model, doing well means executing your current role while simultaneously building the conditions that make your role unnecessary.\nCapacity Is Not a Soft Constraint Before discussing how transitions work, it is worth being direct about something that role frameworks often leave implicit: the number of high-touch engagements a Principal Engineer can hold simultaneously is small. Not small as a matter of preference. Small as a matter of arithmetic.\nThe PE Roles framework is explicit about this. You should not be Sponsoring or Guiding more than two projects at any one time — and that ceiling drops further if you are doing both simultaneously. The reason is not that Sponsoring and Guiding are particularly exhausting. It is that doing them well requires sustained, focused attention. The moment you spread that attention across three or four engagements, you have converted every one of them from Guide to Participant, whether or not your title on the project has changed.\nThis is where a specific kind of conversational discipline becomes necessary.\nThe default behavior for senior engineers in fast-moving organizations is to accumulate. A new initiative spins up and someone says your name. A team gets stuck and leadership routes the escalation to you. A promising engineer asks you to review their design. Each individual ask is reasonable. The aggregate is not.\nWhat works — and this is less intuitive than it sounds — is treating capacity as a zero-sum constraint that you make visible to your manager explicitly and consistently. Not as a complaint. Not as a limit you are imposing on the organization. As a prioritization question that only they can answer.\nThe mechanism is simple: every time something new is proposed, you name what you are currently carrying and ask what should move. \u0026ldquo;I\u0026rsquo;m actively Guiding Project A and Sponsoring Project B. I can take this on, but one of those needs to go. Which one matters less right now?\u0026rdquo;\nThat framing does two things. It prevents accumulation from happening invisibly, which is how most overcommitment happens. And it converts the conversation from a refusal into a trade-off that the manager is participating in, which is where they should be. Eisenhowever Matrix is an effective tool here.\nThis is not about protecting your time. It is about maintaining the quality of your engagement on the things that actually matter. A Principal Engineer nominally attached to five projects and actively driving none of them is not adding five units of value. They are adding close to zero — and consuming a disproportionate amount of organizational attention in the process.\nThe corollary is equally important and contentious : when something small comes in — a one-off consultation, an ad-hoc review, a request to weigh in on a decision that \u0026ldquo;should only take an hour\u0026rdquo; — the default answer can often be no or not yet, without a full prioritization conversation. If nobody in the team can tell you exactly what needs to be built or what your role would be, the right response is to wait until there is clarity, not to spend time investigating a vacuum. The signal to engage is a well-formed question, not an open-ended pull on your attention.\nWhy Engineers Don\u0026rsquo;t Move Outward There is a failure mode worth naming directly because it feels virtuous from the inside: the Principal Engineer who stays at the center because they care about the project.\nThey stay because the stakes are high. They stay because they know the edge cases better than anyone. They stay because transitioning away feels like abandoning something incomplete. And in some cases, they stay because the team genuinely has not reached the point where they could hold the work without that support.\nBut here is the thing: if a team cannot hold the work after months of close engagement from a Principal Engineer, that is not an argument for staying longer. It is a signal that something went wrong in how the engagement was structured. The question to ask is not \u0026ldquo;am I still needed?\u0026rdquo; — that question will always return an affirmative answer if you have not built for your own departure. The question is: \u0026ldquo;what would have to be true for this team to operate without me at the center, and what am I doing today to make that true?\u0026rdquo;\nThis is a systems design problem. You are designing the conditions under which you become less necessary. That requires being deliberate about what you are installing in the team — not just what you are delivering to the project.\nWhat to Install Before You Move If the goal is to move outward without the project degrading, then what you do during the high-touch phase becomes a design problem with specific, verifiable outputs.\nThe most important thing to install is not a process. It is a vocabulary. Specifically: the vocabulary for arguing about the right things. The tenets, mental models, and failure-mode awareness that let a team evaluate trade-offs without needing a Principal Engineer in the room. When that vocabulary is working, you see specific signals. Engineers argue with each other using the right frame, not just the most convenient one. Code reviews surface architectural concerns, not just style issues. Debates about approach reference the team\u0026rsquo;s tenets rather than individual preferences. Escalations to you become smaller in number and sharper in quality — they are about genuine edge cases, not uncertainty about basic direction. When those signals are absent, the vocabulary has not landed yet. Moving out at that point does not scale — it creates an accountability vacuum that someone else fills, usually in a way that undoes what you built at the center. The next thing to install is explicit accountability ownership. You need to identify — not implicitly, but explicitly — the two or three senior engineers who will hold the technical direction after you move. Once identified, you start routing decisions through them, not around them. You attend reviews to observe, not to answer. You let the escalation path go to them first, and come to you only when they hit a genuine limit. That practice — routing through rather than around — is what makes the transition real rather than nominal. The last thing is to make the transition visible to your manager before it happens. A Principal Engineer who moves from Owner to Supervisor without announcing it is just becoming less available. That is not scaling; that is withdrawal. The move only works if the manager understands what is changing, why, and what the senior engineers taking over accountability are expected to do with it. That conversation is not optional. It is part of the engineering work. Setting Expectations Before You Start The cleanest version of this — and the version that avoids the most confusion — is to set the trajectory at the start of the engagement, not partway through.\nWhen you take on a high-touch role, make the expected arc explicit. Something like: \u0026ldquo;I\u0026rsquo;m going to be deeply involved for the next eight to twelve weeks. In that time, I need to establish the core architecture direction, build the vocabulary in the team for evaluating trade-offs, and identify who will own this technically after I step back. After that, I\u0026rsquo;ll move into a Supervisory mode — design reviews, key decision points, but not day-to-day. I\u0026rsquo;d like to agree on who those senior engineers are before we start.\u0026rdquo;\nThat framing does several things at once. It signals to the team that deep involvement is time-bounded, so they do not mistake presence for permanence. It identifies the accountability transfer as a goal of the engagement, not an afterthought. And it gives the manager a clear criterion for evaluating whether the engagement is working — not just whether the project is moving, but whether the team is developing the capacity to move without you at the center.\nThe first impression you set in a new organization or on a new team matters more than most engineers expect, and it is harder to reset than it looks. If you enter a new role and signal that you will take on anything that comes your way, the organization will calibrate to that. The workload will grow to fill the available space, and the expectation that you will absorb new asks indefinitely will harden quickly. Setting the expectation that your engagement is structured and bounded from the start is not a limit you are placing on the organization. It is an accurate description of how high-quality Principal Engineering actually works.\nThe Catcher Problem One of the clearest diagnostics the role framework provides is around the Catcher role, and it is worth dwelling on because the failure mode is both common and self-reinforcing.\nCatching is important. Every organization running complex systems needs engineers who can step into a project that has gone sideways, rapidly build a diagnosis from scratch, and drive a pragmatic path forward under pressure. It teaches real skills and it keeps ambitious initiatives from dying quietly.\nBut catching is not owning, and a Principal Engineer who is always catching never finds time to build the conditions that prevent the next catch from being necessary. They are managing entropy, not reducing it. The more an organization depends on a specific PE to catch, the less pressure there is to build the engineering practices that don\u0026rsquo;t require it. The system feeds on its own demand.\nThe healthier pattern treats catching as a role that transfers, like any other. When a Principal steps into a Catcher role, they are simultaneously identifying the senior engineers who can develop that capacity. The catch is a training scenario, not just a rescue operation. When the project is stable, the lessons from the catch get deliberately propagated — so the team builds the ability to avoid or self-recover from similar situations going forward.\nIf you are a leader mapping PE engagement across your portfolio and you find that most of your PEs are in sustained Catcher mode, that is not a sign of strong engagement. It is a sign that something is wrong upstream — in how projects are scoped, how technical decisions are made, or how senior the engineering talent below the PE level actually is.\nThe Organizational Read Mapping what roles your PEs are playing across projects gives you signal that pure output metrics don\u0026rsquo;t. The question to ask for each PE is not just \u0026ldquo;what are they doing?\u0026rdquo; It is \u0026ldquo;what is the trajectory?\u0026rdquo; Are they positioned to move outward in the next quarter? Do the senior engineers beneath them have the mental models and accountability structure to hold the work when that happens?\nIf most PEs are at the center for extended periods — in Owner, Catcher, or heavy Sponsor modes — without a visible plan to move out, that is a staffing and structure problem as much as it is an individual one. PEs who cannot move outward are usually being held at the center by one of two forces: the team has not developed the capacity to hold the work, or the organization has not created the conditions for the transition to happen cleanly. Both are worth diagnosing.\nThe goal of the engagement model, at the organizational level, is not to maximize PE presence. It is to maximize PE leverage — which means getting the most impact per unit of focused PE attention. That happens when PEs are moving at the right velocity through roles, not when they are embedded in the same high-touch mode indefin\nIf There\u0026rsquo;s One Takeaway The PE Roles framework and the concentric circle model together answer the two questions that matter for senior technical leaders: what mode of engagement does this situation require, and when should it change?\nThe vocabulary from the PE Roles framework — Sponsor, Guide, Catalyst, Tie-Breaker, Catcher, Participant — is precise and useful. But vocabulary alone is not enough. The spatial logic of the concentric model adds the missing piece: direction. The center is not where you aim to stay. It is where you begin a well-designed intervention, and the design of that intervention includes a plan for your own exit.\nMoving outward is the job. Everything at the center is preparation for that move.\n","permalink":"https://blog.girishshekhar.com/posts/pe-roles-intentionality/","summary":"Most senior engineers think about what role they play on a team. Fewer think about how long they should play it — and what happens when they don\u0026rsquo;t move on. This post examines two complementary frameworks for PE role clarity, and why intentional role transitions — not just role awareness — are what actually let Principal Engineers scale their impact across an organization.","title":"Principal Engineers Don't Scale by Doing More. They Scale by Moving Out."},{"content":"Why This Post Exists The first post in this series established that scaling as a Principal Engineer is a step-function change, not a gradient, and that your voice carries weight you need to actively manage. The second post argued that your primary lever at scale isn\u0026rsquo;t technical authority exercised directly—it\u0026rsquo;s mechanisms that carry your judgment into rooms you\u0026rsquo;ll never enter.\nThis post is about the most accessible, most underused, and most compounding of those mechanisms: writing.\nNot writing in the sense of documentation. Not writing in the sense of filling out templates or updating wiki pages. Writing as infrastructure—the deliberate practice of encoding your technical judgment, your pattern recognition, and your hard-won lessons into artifacts that operate independently of your presence, accumulate value over time, and reach people you\u0026rsquo;ll never meet.\nMost Principal Engineers including myself and the Senior PE I talked to arrived at it through burnout. And the path from burnout to one of the most effective scaling practices I\u0026rsquo;ve encountered is worth tracing carefully, because the obstacles we hit are the same ones that keep most PEs from ever starting.\nThe Misconception: Writing Is a Secondary Activity If you ask Principal Engineers how they create impact, you\u0026rsquo;ll hear about design reviews, architecture decisions, mentoring, technical strategy, cross-team coordination. Writing rarely makes the list. And when it does, it\u0026rsquo;s framed as a byproduct—\u0026ldquo;I also write up my decisions\u0026rdquo; or \u0026ldquo;I document things when I have time\u0026rdquo;—not as a primary lever.\nThis framing reveals the misconception: most PEs think of writing as recording. You do the real work—the thinking, the designing, the deciding—and then, if time permits, you write some of it down. The writing is downstream of the impact. It\u0026rsquo;s a nice-to-have. It\u0026rsquo;s what diligent people do after the important work is finished.\nThis is backwards. And understanding why it\u0026rsquo;s backwards requires looking at what happens when you don\u0026rsquo;t write.\nWhen your mentorship is purely conversational, it has the same scaling ceiling as every other form of direct leverage we discussed in the previous post. You say something insightful in a one-on-one, and one person benefits. You explain a mental model in a design review, and the five people in that room hear it. You share a hard-won lesson during a team meeting, and maybe a dozen people absorb it. Then the next person asks you the same question, and you explain it again. And again. And again.\nThe Senior PE hit this wall in a way that\u0026rsquo;s worth describing precisely, because it illustrates the failure mode that writing solves.\nBefore joining his current organization, he had accumulated over forty active mentoring relationships. Not forty people he\u0026rsquo;d mentored at some point in the past—forty concurrent 1:1s with people who were actively seeking his guidance. He never said no. He didn\u0026rsquo;t have the skills to turn people away when they asked for help. And the demand kept growing because he was good at it. People benefited, told others, and those others showed up asking for the same thing.\nForty concurrent mentoring relationships is not sustainable for anyone. But here\u0026rsquo;s the part that matters: a huge fraction of those conversations covered the same ground. The same questions about navigating ambiguity at the staff plus level. The same confusion about how to influence without authority. The same patterns of getting stuck on technical depth when the real problem was organizational. He was delivering essentially the same insights, customized slightly for each person\u0026rsquo;s context, forty times over.\nThat\u0026rsquo;s not mentorship at scale. That\u0026rsquo;s a single-threaded server handling the same request repeatedly, with no caching layer.\nMental model: Writing is the caching layer for your judgment. A conversation is a synchronous, single-use transfer of knowledge. An artifact—a blog post, a technical guide, an architecture document, a lessons-learned writeup—is an asynchronous, multi-use transfer that compounds over time. Every person who reads it gets the benefit without consuming your time. Every new reader adds to the impact without adding to your load. And unlike a conversation, the artifact doesn\u0026rsquo;t degrade as it passes through intermediaries. The tenth person to read your post gets the same fidelity as the first.\nThe Pivot: When Writing Becomes Infrastructure When the Senior PE transitioned to his current organization, he recognized the pattern forming again. New colleagues who had never worked with a Senior Principal Engineer started asking for mentoring. \u0026ldquo;Will you mentor me?\u0026rdquo; became a regular request. And he thought, \u0026ldquo;I\u0026rsquo;ve seen this before. I\u0026rsquo;m going to get myself into the same unsustainable situation. How do I avoid it?\u0026rdquo;\nHis answer was to start blogging internally. Every time he found himself giving the same advice in multiple 1:1s, he\u0026rsquo;d write it down as a post. Not polished thought leadership — practical, specific guidance addressing the questions people were actually asking. When the next person came to him with a familiar question, he could point them to the post and use their one-on-one time for the parts that were genuinely unique to their situation.\nThis is the shift from writing-as-recording to writing-as-infrastructure. He wasn\u0026rsquo;t documenting what he\u0026rsquo;d already done. He was building a system that would handle a class of requests without his direct involvement, freeing him to focus on the problems that actually required his presence.\nThe math here is worth making explicit:\nMENTORSHIP WITHOUT WRITING: ┌──────────────┐ │ Same insight │──▶ Person 1 (1 hour) │ delivered │──▶ Person 2 (1 hour) │ 40 times │──▶ Person 3 (1 hour) │ │──▶ ... │ │──▶ Person 40 (1 hour) └──────────────┘ Total time: 40 hours for 40 people Ongoing cost: every new person = another hour Fidelity: degrades with fatigue and context-switching MENTORSHIP WITH WRITING: ┌──────────────┐ │ Same insight │──▶ Written once (3-4 hours) │ written once │ │ │ │ ├──▶ Person 1 reads it (0 hours from you) │ │ ├──▶ Person 2 reads it (0 hours from you) │ │ ├──▶ ... │ │ ├──▶ Person 400 reads it (0 hours from you) │ │ └──▶ Person 401 reads it next year (still 0 hours) └──────────────┘ Total time: 3-4 hours for unlimited people Ongoing cost: zero marginal cost per reader Fidelity: consistent, reviewable, improvable over time This isn\u0026rsquo;t a subtle difference. It\u0026rsquo;s a structural change in the economics of knowledge transfer. And it\u0026rsquo;s available to every PE who\u0026rsquo;s willing to invest the upfront cost of writing things down.\nThe Power of Imperfect Artifacts If the scaling argument for writing is so clear, why don\u0026rsquo;t more PEs do it? The Senior PE\u0026rsquo;s experience points to two obstacles, and the first one is the more insidious: the belief that an artifact needs to be comprehensive and correct before it\u0026rsquo;s worth sharing.\nHe told a story that illustrates why this belief is exactly wrong.\nWhen a new senior leader joined the organization and asked to be walked through the architecture of their core platform, the Senior PE realized there was no current, accurate architecture document. The only thing he could find was a 2 year old diagram created by a Distinguished Engineer — a good starting point, but so much had changed in the actual implementation that it no longer reflected reality.\nThen he found something unexpected. A senior engineering manager in a different part of the organization had created a simplified version of the architecture on an internal website, intended to help application developers understand the builder experience platform they were building. It wasn\u0026rsquo;t fully accurate—there were gaps and some outright errors. But it existed. It was tangible. It was something you could point at and react to.\nThe Senior PE made a deliberate decision: instead of starting from scratch and trying to create the definitive architecture document (which would have taken weeks and might never have been finished), he took the imperfect artifact, improved what he could, and shared it broadly. His reasoning drew on Cunningham\u0026rsquo;s Law — the observation that the best way to get the right answer on the internet is not to ask a question but to post a wrong answer. People are far more motivated to correct errors than to respond to open-ended requests for input.\nAnd that\u0026rsquo;s exactly what happened. Engineers responsible for different parts of the system came out of the woodwork to correct inaccuracies, fill in gaps, and add context. The artifact became a living document—not because anyone was assigned to maintain it, but because the imperfect first version created a gravitational pull that drew in contributions from people who cared about accuracy.\nThe before-and-after here is stark:\nBEFORE: - No current architecture artifact exists - Knowledge lives in individuals\u0026#39; heads - New leaders/engineers spend weeks piecing together the system through conversations - Each person reconstructs a partial, possibly incorrect mental model independently AFTER: - Imperfect artifact published and shared - Domain experts correct and extend it organically - New leaders/engineers have a starting point that\u0026#39;s 80% right on day one - Corrections improve the artifact for everyone, not just the person who asked The Senior PE admitted he didn\u0026rsquo;t have the foresight to see how valuable that artifact would become. He wasn\u0026rsquo;t executing a strategy to position himself as the architecture authority. He was solving an immediate problem—his new boss needed to understand the system—in a way that happened to create lasting value. But the lesson generalizes: an imperfect artifact that exists is worth more than a perfect artifact that doesn\u0026rsquo;t. The gap between \u0026ldquo;something\u0026rdquo; and \u0026ldquo;nothing\u0026rdquo; is always larger than the gap between \u0026ldquo;something\u0026rdquo; and \u0026ldquo;something better.\u0026rdquo;\nThis matters because perfectionism is the primary killer of PE writing output. The engineer\u0026rsquo;s instinct—make it correct, make it complete, make it thorough—actively works against the goal of getting artifacts into the world where they can create value. A blog post that\u0026rsquo;s 80% right and published today will help people. A blog post that\u0026rsquo;s 100% right and published never will help no one.\nThe Unexpected Consequence: Writing Builds Connective Tissue Here\u0026rsquo;s where the argument goes beyond the obvious \u0026ldquo;writing scales your knowledge\u0026rdquo; point. Writing creates a second-order effect we don\u0026rsquo;t anticipate: For the Senior PE - it fundamentally changed the quality and reach of his professional relationships.\nThe trajectory started internally. He kept the blog inside the company because he was—his words—\u0026ldquo;terrified to post on the internet and potentially get negative feedback from strangers.\u0026rdquo; That\u0026rsquo;s worth pausing on, because it\u0026rsquo;s the second obstacle that prevents PEs from writing. The first is perfectionism. The second is exposure. Writing publicly means making your thinking visible to people who might disagree, might criticize, might find errors. For people who built their careers on being technically rigorous and correct, that vulnerability is genuinely uncomfortable.\nWhat changed for him was circumstantial. Someone connected him with a communications coach who pushed him to post externally as part of a structured program. He did—reluctantly—and the response was immediate. Former colleagues who had left the company engaged with his posts. People he\u0026rsquo;d never met reached out. His first public post got hundreds of thousands of impressions, which was terrifying in its own way. \u0026ldquo;Oh shoot,\u0026rdquo; he thought, \u0026ldquo;now I\u0026rsquo;m committed. Now I better do this regularly.\u0026rdquo;\nBut the interesting part isn\u0026rsquo;t the audience size. It\u0026rsquo;s what happened to his relationships.\nThrough writing publicly, he connected with other senior technical people who also write about their work. These are people across different organizations who share the practice of thinking through problems in writing and publishing their conclusions. The Senior PE described this as finding a \u0026ldquo;tribe\u0026rdquo;—people who speak the same \u0026ldquo;mother tongue\u0026rdquo; of thinking in public.\nAnd that shared practice created a connective tissue that made cold outreach warm. He gave a specific example: \u0026ldquo;It\u0026rsquo;s easier for me to reach out to someone and say, \u0026lsquo;Hey, you\u0026rsquo;re an expert in this area—in the world of AI, how does testing change?\u0026rsquo; Normally a cold Slack message like that would get ignored. But because of that extra connection, because we speak the same mother tongue, there\u0026rsquo;s an openness to engage.\u0026rdquo;\nThis is not networking in the traditional, transactional sense. It\u0026rsquo;s the natural consequence of making your thinking visible. When you write about what you\u0026rsquo;re learning, what you\u0026rsquo;re struggling with, and what you\u0026rsquo;ve figured out, you become findable by people who care about the same problems. Those people are often exactly the ones you need relationships with to be effective at scale—the senior engineers in adjacent organizations, the technical leaders working on parallel challenges, the people whose expertise complements yours.\nThe relationship-building isn\u0026rsquo;t the goal of the writing. But it\u0026rsquo;s a compounding side effect that makes every other aspect of the PE role easier. Cross-organizational influence, which the first post identified as essential at Senior PE scope, depends on relationships with people you don\u0026rsquo;t work with daily. Writing creates and maintains those relationships at a scale that no meeting cadence or offsite schedule can match.\nFrom Artifacts to Authority (Whether You Wanted It or Not) There\u0026rsquo;s a final dimension to this that the Senior PE was visibly uncomfortable discussing: writing creates visibility, and visibility creates a form of authority that compounds whether you seek it or not.\nAfter publishing the architecture artifact and sharing lessons learned from his organization\u0026rsquo;s work on agentic AI, the Senior PE found himself invited to present at the company\u0026rsquo;s major technical conference. That visibility led to people asking him to represent his organization in cross-company technical discussions. He started noticing that teams in other parts of the company were months behind where his organization was in the agentic AI journey—making the same mistakes his team had already made and learned from.\nSo he did something that felt natural but turned out to be strategically important: he started writing and presenting specifically about lessons learned. Not \u0026ldquo;here\u0026rsquo;s how great our system is\u0026rdquo; but \u0026ldquo;here\u0026rsquo;s what we tried that didn\u0026rsquo;t work,\u0026rdquo; \u0026ldquo;here\u0026rsquo;s what we learned the hard way,\u0026rdquo; and \u0026ldquo;here\u0026rsquo;s what you should avoid.\u0026rdquo; The framing was deliberately generous—not claiming credit for successes but offering the failures and hard-won insights so that others could benefit.\nThis is a form of leadership that requires no positional authority. You\u0026rsquo;re not telling people what to do. You\u0026rsquo;re not making decisions for them. You\u0026rsquo;re sharing knowledge so openly that people can\u0026rsquo;t help but benefit from it. And when you do this consistently, something happens that you can\u0026rsquo;t manufacture through any other means: your name becomes associated with the domain. Not because you campaigned for it, but because you were the person who made your knowledge available when others kept theirs locked in meetings and Slack threads.\nThe Senior PE was candid about his discomfort with this: \u0026ldquo;There are so many people who deserve more credit than me. The people building the next version—they\u0026rsquo;re doing the hard work. But if I can take one for the team, I\u0026rsquo;ll do it. I feel bad running toward the spotlight. I\u0026rsquo;m the opposite of that. I run away from it.\u0026rdquo;\nThat discomfort is worth naming because it\u0026rsquo;s common among PEs. Most Principal Engineers built their careers by going deep, not by being visible. The spotlight feels unearned, or self-promotional, or like it takes credit away from the team. These feelings are understandable, and to some degree they reflect good instincts—you should be generous with credit and cautious about self-promotion.\nBut there\u0026rsquo;s a reframe that the Senior PE arrived at: the visibility isn\u0026rsquo;t for you. It\u0026rsquo;s for the knowledge. When you write publicly about lessons learned, the visibility that accrues to you personally is a side effect of making important knowledge accessible. If you hold that knowledge back because the visibility feels uncomfortable, you\u0026rsquo;re prioritizing your own comfort over the benefit that others would get from your experience. The discomfort is real. But the calculus favors sharing.\nWhat This Looks Like in Practice If you\u0026rsquo;re convinced that writing is infrastructure rather than documentation, the practical question is: where do you start?\nStart with what you\u0026rsquo;re already repeating. If you\u0026rsquo;re giving the same advice in multiple 1:1s, that\u0026rsquo;s your first post. If you\u0026rsquo;re explaining the same architectural concept in every design review, that\u0026rsquo;s your second. The signal for what to write is the repetition in your conversations—anywhere you\u0026rsquo;re acting as a single-threaded server handling the same request, there\u0026rsquo;s an artifact waiting to be created.\nStart internal, start imperfect. The Senior PE began with internal blog posts, not public ones. The audience was smaller, the stakes were lower, and the feedback was from people he knew. More importantly, he didn\u0026rsquo;t wait until his posts were perfect. He wrote them well enough to be useful and published them. Corrections came. Refinements followed. The artifacts improved over time, which is exactly how infrastructure works—you iterate on it, you don\u0026rsquo;t wait until it\u0026rsquo;s perfect to deploy it.\nLet Cunningham\u0026rsquo;s Law work for you. If you\u0026rsquo;re creating an artifact that maps complex territory—an architecture document, a technical decision history, a guide to how your organization\u0026rsquo;s systems interconnect—don\u0026rsquo;t let the fear of inaccuracy stop you. Publish the 80% version. The people who know the remaining 20% will find you, and they\u0026rsquo;ll be more motivated to contribute corrections than they ever would have been to create the document from scratch.\nPay attention to what compounds. Not all writing has the same leverage. A post about a specific production incident is valuable but time-bound—it loses relevance as the system changes. A post about the pattern of failures that led to that incident, and the mental model for preventing that class of failure, is durable. It will still be useful a year from now, to engineers working on systems you haven\u0026rsquo;t built yet. Prioritize the writing that encodes transferable judgment over writing that documents specific events.\nAccept that visibility is part of the mechanism. If you write but nobody reads it, the artifact has no leverage. Distribution matters. Sharing your writing in relevant forums, referencing it in reviews and discussions, and yes, posting it where a broader audience can find it—these aren\u0026rsquo;t self-promotion. They\u0026rsquo;re the equivalent of deploying your code to production. An artifact that sits unread in a wiki is like a service that\u0026rsquo;s never deployed. It technically exists, but it creates no value.\nThe Real Shift Writing is the most accessible mechanism a Principal Engineer has. It requires no organizational permission. It has no marginal cost per reader. It compounds over time. It builds relationships you can\u0026rsquo;t build through meetings. And it creates a form of authority that\u0026rsquo;s grounded in generosity rather than hierarchy.\nBut calling it \u0026ldquo;accessible\u0026rdquo; doesn\u0026rsquo;t mean it\u0026rsquo;s easy. Writing well takes time. Writing publicly takes courage. Writing imperfectly—which you must, or you\u0026rsquo;ll never publish—takes a willingness to be wrong in front of people whose respect you value. Every obstacle the Senior PE described is real: the perfectionism, the fear of exposure, the discomfort with visibility, the feeling that writing is somehow less legitimate than \u0026ldquo;real\u0026rdquo; technical work.\nThe reframe that resolves all of these obstacles is the one this post opened with: writing is not documentation. It is not a secondary activity that happens after the important work. Writing is the work, at the scale you\u0026rsquo;re now operating at. When you write down the insight you\u0026rsquo;ve been giving in 1:1s, you\u0026rsquo;ve just multiplied it from reaching one person to reaching hundreds. When you publish an imperfect architecture artifact, you\u0026rsquo;ve just created a nucleus for collaborative knowledge that no meeting could have produced. When you share lessons learned from your team\u0026rsquo;s failures, you\u0026rsquo;ve just prevented other teams from spending months rediscovering what you already know.\nIf there\u0026rsquo;s one takeaway from this series, it\u0026rsquo;s this: the Principal Engineers who scale their impact aren\u0026rsquo;t the ones who become better individual contributors in a larger room. They\u0026rsquo;re the ones who build mechanisms—processes, frameworks, forums, and above all, written artifacts—that carry their judgment into conversations they\u0026rsquo;ll never be part of, for engineers they\u0026rsquo;ll never meet, on timelines that extend far beyond their own tenure. Writing is the simplest, most durable, and most underused of those mechanisms. The only thing standing between you and that leverage is the decision to start.\n","permalink":"https://blog.girishshekhar.com/posts/scaling-as-a-pe/3-writing-is-the-most-under-used-leverage/","summary":"Most Principal Engineers treat writing as documentation—a secondary activity that happens after the real work. But writing is the highest-leverage mechanism available at staff-plus scope. It scales mentorship, compounds over time, creates artifacts that outlive individual conversations, and builds the connective tissue for relationships you can\u0026rsquo;t maintain through meetings alone. This post covers why PEs underinvest in writing, what changes when they stop, and how imperfect artifacts often create more value than perfect silence.","title":"Writing Is the Most Underused Leverage a Principal Engineer Has"},{"content":"Why This Post Exists The first post in this series established two foundational ideas: scaling as a Principal Engineer is a step-function change (not a gradient), and your voice carries outsized weight that you need to actively manage. But understanding that the scope changed doesn\u0026rsquo;t tell you how to operate at the new scope. Knowing you need 10x leverage doesn\u0026rsquo;t tell you where to find it.\nThis post is about the operational side of that shift. Specifically, it\u0026rsquo;s about a pattern the Senior PE I spoke with described as \u0026ldquo;borrowing superpowers\u0026rdquo; — observing how adjacent roles create leverage, and adapting those patterns to the PE context. The insight isn\u0026rsquo;t that PEs should become managers or TPMs. It\u0026rsquo;s that managers and TPMs have solved a problem that most PEs haven\u0026rsquo;t even recognized they have: how to create impact at scale without doing the work yourself.\nIf the first post was about changing your mental model, this one is about changing your operating model.\nThe Misconception: Technical Authority Is Your Primary Lever Ask most Principal Engineers what their leverage is, and the answer comes quickly: technical expertise. Deep knowledge. The ability to see around corners architecturally. Being the person who can look at a system design and spot the flaw that would have caused a high severity production incident six months from now.\nAnd that\u0026rsquo;s real. That expertise is what earned you the role. It\u0026rsquo;s why people seek you out for design reviews. It\u0026rsquo;s why your opinion carries weight in architecture discussions. It\u0026rsquo;s why, as we discussed in the first post, your voice has a megaphone attached to it.\nBut here\u0026rsquo;s the problem: technical authority is a direct lever. It works when you\u0026rsquo;re in the room. It works when you\u0026rsquo;re reviewing the document. It works when you\u0026rsquo;re asking the question that nobody else thought to ask. The moment you leave the room, the lever stops working. The next design review happens without you. The next architectural decision gets made by a team you\u0026rsquo;ve never met. The next production system gets built by engineers who don\u0026rsquo;t know your opinion on the tradeoffs involved.\nWhen your scope was 50 engineers, direct leverage was sufficient. You could be in enough rooms, review enough designs, and have enough one-on-ones to personally influence most of the consequential technical decisions. At 500 to 5,000 engineers, that model collapses. You can\u0026rsquo;t be in every room. You can\u0026rsquo;t review every design. And if your impact depends on your physical or virtual presence in a conversation, you\u0026rsquo;ve already hit the ceiling.\nThis is not a time management problem. It is a leverage problem. And the solution doesn\u0026rsquo;t come from getting better at being a technical authority. It comes from building systems that carry your technical judgment into contexts where you\u0026rsquo;ll never be personally present.\nMental model: The Principal Engineers who scale effectively don\u0026rsquo;t just have good judgment. They encode good judgment into mechanisms—processes, frameworks, forums, artifacts—that operate independently of their presence. The shift isn\u0026rsquo;t from doing technical work to doing non-technical work. It\u0026rsquo;s from direct influence to systemic influence.\nThe Superpowers You\u0026rsquo;re Not Using Here\u0026rsquo;s something that might surprise you: the two roles in engineering organizations that have most thoroughly solved the \u0026ldquo;influence without presence\u0026rdquo; problem are managers and Technical Program Managers (TPMs). And most PEs have never seriously studied how either group operates.\nThis isn\u0026rsquo;t about career envy or role confusion. It\u0026rsquo;s about recognizing that these roles have developed specific capabilities that PEs need at scale—and that borrowing those capabilities doesn\u0026rsquo;t make you less of an engineer. It makes you a more effective one.\nWhat Managers Know That You Don\u0026rsquo;t Managers are, by the nature of their job, multipliers. They\u0026rsquo;re expected to create impact through others. And they have a lever that PEs don\u0026rsquo;t: positional authority. When a manager says \u0026ldquo;we need to prioritize this,\u0026rdquo; there\u0026rsquo;s organizational weight behind that statement that comes from the reporting structure. People listen partly because the manager has good judgment, but also because the manager has the organizational authority to set direction.\nAs Principal Engineers, we don\u0026rsquo;t have that lever. Nobody reports to us. We can\u0026rsquo;t direct someone\u0026rsquo;s work, set their priorities, or make organizational decisions about how teams are structured. And this is by design—the PE role is explicitly an individual contributor track.\nBut positional authority is only one of the tools managers use. Watch how effective engineering managers actually operate, and you\u0026rsquo;ll notice something: the best ones don\u0026rsquo;t rely heavily on positional authority either. They create alignment systems. They build shared understanding of priorities so that teams make good decisions autonomously. They participate in talent reviews and organizational planning forums that give them visibility into what\u0026rsquo;s happening across the organization. They create feedback loops that surface problems early. They establish norms and expectations that persist even when they\u0026rsquo;re not watching.\nThese are the patterns worth borrowing. Not the positional authority—you don\u0026rsquo;t have it and you don\u0026rsquo;t need it. But the systems thinking about how to create alignment at scale. The understanding of how organizational forums work and how to use them. The recognition that being included in talent reviews and technical assessments—which many PEs are, even as individual contributors—isn\u0026rsquo;t a bureaucratic obligation. It\u0026rsquo;s a strategic lever that gives you visibility and influence far beyond your immediate scope.\nThe Senior PE I spoke with made this point directly: \u0026ldquo;We benefit from the respect that the Principal Engineering community has earned. Our titles do reflect organizational weight—we\u0026rsquo;re included in talent reviews and tech assessments even though we\u0026rsquo;re individual contributors. That inclusion matters. It gives us a seat at tables where decisions are made.\u0026rdquo; Most PEs treat these forums as overhead. The ones who scale treat them as infrastructure.\nWhat TPMs Know That You Don\u0026rsquo;t The Senior PE mentioned that he gravitates toward TPM-like work—not because he particularly enjoys it, but because it happens to be something he\u0026rsquo;s good at, and because TPMs have a superpower that\u0026rsquo;s directly applicable to how PEs should think about scaling.\nThat superpower: TPMs create and manage mechanisms to get things done without positional authority. They don\u0026rsquo;t have direct reports. They don\u0026rsquo;t have organizational leverage in the traditional sense. But effective TPMs are extraordinarily good at creating structured processes and systems that enable work to happen across organizational boundaries.\nThink about what TPMs actually build: project tracking systems, cross-team collaboration forums, decision-making frameworks, escalation paths, communication cadences. If you\u0026rsquo;ve ever dismissed these as \u0026ldquo;just process\u0026rdquo; or \u0026ldquo;bureaucratic overhead,\u0026rdquo; you\u0026rsquo;ve missed the point. These are leverage tools. They\u0026rsquo;re mechanisms that allow coordination to happen at scale without requiring any single person to be present in every conversation.\nAnd they work because people trust that the mechanisms serve a real purpose. Not because the TPM has authority over anyone, but because the mechanism itself has earned credibility through consistent usefulness.\nAs PEs, we can build the same kind of mechanisms, tuned to our domain - Architecture review processes; Technical decision frameworks; Documentation standards; Cross-team knowledge-sharing forums; Design principle repositories. These aren\u0026rsquo;t administrative tasks. They\u0026rsquo;re the infrastructure through which your technical judgment reaches the 500 or 5,000 engineers you\u0026rsquo;ll never personally interact with.\nHere\u0026rsquo;s the contrast that makes this concrete:\nDIRECT LEVERAGE (doesn\u0026#39;t scale): ┌───────────────┐ ┌───────────┐ │ PE reviews │────▶│ 1 design │ │ a design doc │ │ improves │ └───────────────┘ └───────────┘ Time cost: 2-4 hours Impact radius: 1 team, 1 decision SYSTEMIC LEVERAGE (scales): ┌───────────────────┐ ┌──────────────────────────────┐ │ PE creates a │────▶│ Every design doc across the │ │ decision-record │ │ org includes alternatives │ │ template with │ │ considered, tradeoffs made, │ │ required fields │ │ and failure modes analyzed │ └───────────────────┘ └──────────────────────────────┘ Time cost: 8-16 hours (once) Impact radius: all teams, all decisions, ongoing The PE in the first row is doing important work. The PE in the second row is doing leveraged work. Both matter, but only the second one scales to Senior PE scope.\nBuilding Mechanisms Without Asking Permission Understanding that mechanisms are your primary lever is one thing. Actually building them is where most PEs get stuck. And they get stuck for a predictable reason: they wait for permission.\nThis instinct makes sense if you think about it. Engineers are trained to operate within defined systems. We work within architectures, follow processes, use tools that someone else set up. When we want to change something systemic—a review process, a communication forum, a decision framework—our instinct is to propose it, get buy-in, navigate the organizational approval chain, and then implement it once everyone agrees.\nBy the time that process completes, the window of opportunity has often closed. Or the mechanism has been committee\u0026rsquo;d into something so watered down that it doesn\u0026rsquo;t actually solve the problem it was designed to solve.\nThe Senior PE shared a story that illustrates the alternative approach. When his organization went through a major restructuring—splitting into two separate orgs—the existing PE community was at risk of getting diluted. The PE community in the new structure needed its own identity, its own coordination mechanisms, and its own forums for making technical decisions. Without that, consequential architectural choices would be made inconsistently across teams, and the PEs who should have been coordinating would be isolated in their respective corners of the org.\nSo the Senior PE started a council. A small group of Principal Engineers who had the technical standing and organizational context to make certain cross-cutting decisions. He created a regular forum where these PEs would discuss architectural direction, coordinate on shared technical problems, and establish standards that would apply across the organization.\nHere\u0026rsquo;s the important part: he didn\u0026rsquo;t have permission to do any of this. He didn\u0026rsquo;t submit a proposal. He didn\u0026rsquo;t ask his leadership chain whether it was okay to create a PE council. He didn\u0026rsquo;t wait for someone else to notice the coordination gap and assign someone to fix it. He just built the thing and waited for someone to object.\nNo one did. What happened instead was that people started thanking him. Engineers and managers were grateful that someone had created a forum where decisions could be made efficiently. Where the PE community could coordinate rather than working in isolation. Where technical direction could be set with input from the people who had the deepest understanding of the systems involved.\nAt that point, he thought, \u0026ldquo;Okay, let\u0026rsquo;s keep doing it.\u0026rdquo; And the council became an established mechanism that people now rely on.\nWhy This Works (And When It Doesn\u0026rsquo;t) This approach works because of a dynamic that most PEs underestimate: in large organizations, there is usually more demand for coordination mechanisms than there is supply of people willing to create them. Most coordination problems persist not because anyone opposes solving them, but because nobody takes the initiative to build the solution. The friction isn\u0026rsquo;t resistance—it\u0026rsquo;s inertia.\nWhen you build a mechanism that clearly solves a real problem, adoption tends to happen naturally. People use it because it makes their lives easier, not because someone told them to. And leadership tends to support it because it reduces chaos and improves decision quality. The Senior PE\u0026rsquo;s council succeeded not because he had authority to mandate it, but because it filled a genuine vacuum.\nThat said, this approach has boundaries. There are situations where building without permission can backfire:\nWhen the mechanism steps on someone\u0026rsquo;s existing authority. If there\u0026rsquo;s already a team or a leader who considers coordination in that space to be their responsibility, creating a parallel mechanism can feel like an encroachment rather than a contribution. The fix isn\u0026rsquo;t to avoid building—it\u0026rsquo;s to build with those people, not around them.\nWhen the mechanism adds burden without clear value. If people experience your mechanism as \u0026ldquo;yet another meeting\u0026rdquo; or \u0026ldquo;yet another document to fill out,\u0026rdquo; it will die from neglect or breed resentment. The test is simple: does this mechanism make it easier for people to do the right thing, or does it add friction? If it adds friction, it\u0026rsquo;s not a mechanism—it\u0026rsquo;s bureaucracy.\nWhen you\u0026rsquo;re solving a problem that doesn\u0026rsquo;t actually exist. Sometimes the coordination gap you\u0026rsquo;ve identified isn\u0026rsquo;t felt by anyone else. Before building, validate that others experience the problem. You don\u0026rsquo;t need a formal survey—a few conversations with the engineers and managers who would use the mechanism will tell you whether the need is real.\nThe rule of thumb the Senior PE offered: start small, build something that clearly adds value, and be willing to shut it down if it doesn\u0026rsquo;t work. The organizational cost of a failed lightweight experiment is negligible. The organizational cost of a coordination vacuum that persists for months because nobody was willing to act without permission is substantial.\nYour Unique Leverage: What Managers and TPMs Can\u0026rsquo;t Do Borrowing from managers and TPMs is essential, but it\u0026rsquo;s also important to understand what makes PE leverage distinct. You\u0026rsquo;re not a manager without direct reports. You\u0026rsquo;re not a TPM who happens to know how to code. You have a form of influence that neither role can replicate.\nYour leverage is trust built through demonstrated competence. When you speak about technical matters in your domain, people listen because they know you\u0026rsquo;ve built systems, debugged production incidents at 3 AM, made hard tradeoffs under real constraints, and lived with the consequences. That credibility isn\u0026rsquo;t granted by a title or an org chart. It\u0026rsquo;s earned through a track record of being right about things that are hard to be right about.\nThis means your mechanisms carry a different kind of authority than a manager\u0026rsquo;s process or a TPM\u0026rsquo;s coordination structure. When a PE creates a technical decision framework, it\u0026rsquo;s not just a process artifact—it\u0026rsquo;s an encoding of how someone with deep systems experience thinks about tradeoffs. When a PE establishes architecture review standards, those standards carry the implicit endorsement of someone who knows what good architecture looks like because they\u0026rsquo;ve built it and maintained it and debugged it when it failed.\nThis is also why the Three-Position Framework from the first post matters so much in the context of mechanisms. The mechanisms you create should be grounded in your Position One expertise—the areas where you genuinely have deep, earned authority. A decision framework for distributed systems architecture, created by a PE who has spent a decade building distributed systems, carries inherent credibility. The same PE creating a mechanism for, say, product prioritization or team staffing would be operating in Position Three territory, and the mechanism would lack the earned authority that makes it trustworthy.\nThe combination is what makes PEs uniquely effective at scale: systemic leverage (borrowed from managers and TPMs) combined with earned technical credibility (your own). Neither alone is sufficient. Technical credibility without systemic leverage means you\u0026rsquo;re brilliant but limited to the rooms you\u0026rsquo;re in. Systemic leverage without technical credibility means you\u0026rsquo;re creating processes that people follow grudgingly rather than trust deeply.\n┌─────────────────────────────────┐ │ Scalable PE Influence │ │ │ │ = Systemic Leverage │ │ (mechanisms, forums, │ │ artifacts, processes) │ │ │ │ × Technical Credibility │ │ (earned through competence, │ │ grounded in Position One │ │ expertise) │ │ │ │ This is multiplicative, not │ │ additive. Zero on either side │ │ means zero scaled impact. │ └─────────────────────────────────┘ Putting It Together: From Instinct to Operating Model Let me make this concrete with a before-and-after that captures the shift this post is arguing for.\nThe PE operating on instinct gets pulled into a design review. The design has a subtle problem with how it handles failure modes in a distributed workflow. The PE catches it, explains the issue, suggests a better approach. The team fixes the design. Good outcome. Time cost: three hours for the review plus follow-up. Impact: one team, one design, one decision.\nTwo weeks later, a different team in a different part of the org ships a design with the exact same failure mode. The PE never saw that design. Nobody asked for a review. The problem goes to production and causes an incident.\nThe PE operating on mechanisms has the same instinct and the same expertise. But instead of stopping at the individual review, they recognize the pattern: this failure mode keeps showing up because teams don\u0026rsquo;t have a shared framework for thinking about failure handling in distributed workflows. So the PE writes a technical decision document—not a long architectural treatise, but a concise, opinionated guide to the three or four failure-handling patterns that apply in their domain, with clear guidance on when to use each one. They publish it to the PE community. They reference it in the next architecture review. They ask other PEs to reference it in their reviews.\nSix months later, new designs in that organization routinely consider failure modes that teams used to miss. Not because the PE reviewed every design, but because the mechanism carried their judgment into dozens of design conversations they were never part of.\nThat\u0026rsquo;s the difference between direct leverage and systemic leverage. Same expertise. Same good judgment. Radically different impact radius.\nThe Real Shift The Principal Engineers who successfully scale beyond their initial scope share a common trait: they stop thinking of themselves primarily as technical experts who happen to influence people, and start thinking of themselves as systems builders who happen to have deep technical expertise.\nThis is not a demotion of technical skill. It\u0026rsquo;s a recontextualization of it. Your technical expertise doesn\u0026rsquo;t become less important—it becomes the input to mechanisms rather than the output of your daily work. You still need to be deeply technical. You still need to review designs, debug hard problems, and make consequential architectural calls. But those activities shift from being the primary way you create impact to being the way you stay credible enough that your mechanisms carry weight.\nThe leverage equation for a Principal Engineer at scale is multiplicative: systemic leverage times technical credibility. If either term is zero, the product is zero. A PE who builds great mechanisms but has lost touch with the technical realities of the systems they influence will create processes that teams route around. A PE who maintains deep technical expertise but never builds mechanisms will be individually brilliant but organizationally limited.\nIf there\u0026rsquo;s one takeaway from this post, it\u0026rsquo;s this: the question to ask yourself isn\u0026rsquo;t \u0026ldquo;how can I apply my technical expertise to more things?\u0026rdquo; It\u0026rsquo;s \u0026ldquo;what mechanisms can I build that will apply my technical judgment to things I\u0026rsquo;ll never personally touch?\u0026rdquo; That\u0026rsquo;s the shift from direct leverage to systemic leverage. That\u0026rsquo;s how you operate at 10x to 100x scope without working 10x to 100x hours.\nThe next post in this series covers a specific and powerful form of mechanism: the artifacts, writing, and knowledge-sharing practices that scale your mentorship and establish your technical authority across organizational boundaries.\n","permalink":"https://blog.girishshekhar.com/posts/scaling-as-a-pe/2-multiply-impact-through-mechanisms/","summary":"Principal Engineers often default to technical authority as their primary lever. But at staff-plus scope, the PEs who scale effectively are the ones who build mechanisms—borrowed from managers and TPMs—that carry their judgment into rooms they\u0026rsquo;ll never enter. This post covers how to identify, build, and deploy those mechanisms without waiting for permission.","title":"The Leverage Playbook: How Principal Engineers Multiply Impact Through Mechanisms"},{"content":"Why This Post Exists There\u0026rsquo;s a transition point in a Principal Engineer\u0026rsquo;s career that almost nobody talks about clearly. Not the promotion itself—there\u0026rsquo;s plenty written about how to get to PE. What\u0026rsquo;s missing is an honest account of what changes after you arrive, and why the instincts that earned you the role become the very things that prevent you from growing beyond it.\nThis post is based on a conversation with a Senior Principal Engineer Luu Tran who has mentored dozens of PEs through this transition. He\u0026rsquo;s lived through the mistakes, watched others make the same ones, and developed frameworks that actually help. The insights here aren\u0026rsquo;t theoretical career advice. They\u0026rsquo;re lessons extracted from real missteps and hard-won clarity about how influence actually works at the staff-plus level.\nIf you\u0026rsquo;re a Principal Engineer wondering what \u0026ldquo;scaling your impact\u0026rdquo; actually means in practice—or if you\u0026rsquo;re an engineering leader trying to understand why some PEs plateau while others multiply their influence—this is for you.\nThe Misconception: Scaling Means Doing More When Principal Engineers hear \u0026ldquo;you need to scale your impact,\u0026rdquo; most translate that into an obvious plan: take on more projects, review more designs, mentor more people, attend more meetings. It feels logical. If your impact at the current level came from deep technical contribution and hands-on involvement, then more involvement should produce more impact.\nThis is not scaling. It is linear accumulation. And it has a hard ceiling.\nConsider the math. A typical Principal Engineer (a.k.a. Sr. Staff) has a sphere of influence covering roughly 50 engineers. That\u0026rsquo;s maybe two to three teams, a set of systems you know deeply, and a group of people who see you regularly enough that your judgment carries weight through direct interaction.\nWhen you move to Senior Principal Engineer (a.k.a. Architect), that sphere expands to somewhere between 500 and 5,000 engineers. This is not a 2x increase. It is a 10x to 100x expansion. There are not enough hours in any week to individually influence that many people through code reviews, design consultations, or one-on-one mentoring.\nThe Senior PE I spoke with put it directly: \u0026ldquo;What got you here won\u0026rsquo;t get you there.\u0026rdquo; The transition from Sr. Staff to PE isn\u0026rsquo;t a gradual progression where you get incrementally better at the same activities. It\u0026rsquo;s a step-function change. The nature of the work—technical leadership, architectural thinking, raising the bar—doesn\u0026rsquo;t change dramatically. You don\u0026rsquo;t have to become a fundamentally different person. But the scale at which you operate changes completely, and that demands a fundamentally different approach to how you spend your time and where you look for leverage.\nMental model: Think of it as the difference between being a gifted individual musician and being a conductor. The skills aren\u0026rsquo;t unrelated—you need deep musical understanding for both. But a conductor who tries to play every instrument simultaneously doesn\u0026rsquo;t produce an orchestra. They produce noise and exhaustion.\nThe Megaphone Effect: Your Voice Changed Before You Noticed Before we talk about frameworks for scaling, there\u0026rsquo;s something more urgent to understand. It\u0026rsquo;s perhaps the most important lesson the Senior PE shared, and it came from a mistake he still regrets.\nThe moment you\u0026rsquo;re promoted to Principal Engineer, your voice gets an invisible megaphone attached to it. You may not feel different. You may still think of yourself as \u0026ldquo;just an engineer with opinions.\u0026rdquo; But the organization doesn\u0026rsquo;t see you that way anymore. People will take action based on your words, even when you don\u0026rsquo;t intend them to.\nHere\u0026rsquo;s what happened to him. Early in his PE career, he looked at a feature and thought, \u0026ldquo;That\u0026rsquo;s not so hard.\u0026rdquo; In a meeting, he said something like, \u0026ldquo;I think we can do that in six months with the people we have.\u0026rdquo; It felt like a casual observation—maybe even helpful context.\nBut here\u0026rsquo;s what actually followed: product teams made commitments to goals based on that statement. Engineering managers committed their teams to timelines. Engineers started executing against those timelines. And when it turned out to take far longer than six months, the managers—the people who were accountable to their teams and their leadership—took the hit for his mistake. Personally I have lived this experience so many times as I continued growing professionally.\nThe estimate wasn\u0026rsquo;t just wrong in the usual \u0026ldquo;estimates are hard\u0026rdquo; sense. It was wrong because he hadn\u0026rsquo;t considered things that weren\u0026rsquo;t in his domain to estimate. He didn\u0026rsquo;t know the team\u0026rsquo;s capability, not just their capacity. He didn\u0026rsquo;t know their competing priorities. He didn\u0026rsquo;t account for the risk of attrition if people were pushed too hard on an aggressive timeline. These are the nuances that engineering managers who are close to their teams understand deeply. And his casual statement—amplified by the megaphone of his title—overrode all of that context in a single sentence.\nThis matters because the damage isn\u0026rsquo;t abstract. Real managers had to explain to their teams why commitments were being missed. Real engineers felt the pressure of an unrealistic timeline that originated from someone who wouldn\u0026rsquo;t be doing the implementation. The people closest to the work had their judgment implicitly overruled by someone who was speaking from surface-level familiarity, not deep understanding.\nThe lesson isn\u0026rsquo;t \u0026ldquo;never estimate\u0026rdquo; or \u0026ldquo;never share opinions.\u0026rdquo; The lesson is that the cost of a casual statement from a Principal Engineer is dramatically higher than the cost of the same statement from a senior engineer. Your words have organizational weight whether you intend them to or not. And the worst version of this failure is when you\u0026rsquo;re making pronouncements in areas where you don\u0026rsquo;t have the depth to back them up.\nThe 3-Position Framework So how do you navigate this? How do you know when to speak with authority, when to offer a perspective, and when to stay quiet? The Senior PE learned a framework—originally attributed to Reid Hoffman—that has become central to how he operates. It\u0026rsquo;s deceptively simple, but it changes how people receive and act on what you say.\nThe idea: every time you speak from a position of influence, you\u0026rsquo;re implicitly operating from one of three positions. The problem is that most PEs never make this explicit, and the default assumption from listeners is always that you\u0026rsquo;re speaking from Position 1.\nPosition 1: The Expert Voice This is the position where you\u0026rsquo;re saying, \u0026ldquo;I\u0026rsquo;m the expert here. I\u0026rsquo;ve thought through this deeply. This is my considered technical judgment, and I\u0026rsquo;m putting my credibility behind it.\u0026rdquo;\nFor Principal Engineers, the default assumption when you open your mouth is that you\u0026rsquo;re coming from Position 1. People hear your title, they hear your statement, and they treat it as authoritative guidance. That\u0026rsquo;s the megaphone effect in action.\nYou should actually be speaking from Position 1 about 5% of the time. Maybe less. This position is reserved for situations where you have genuine, deep expertise in the area. Where you\u0026rsquo;ve thought through the alternatives. Where the decision has been vetted through structured mechanisms—architecture reviews, technical decision documents, formal debates with peers. Where the decision is consequential enough and within your domain enough that it warrants the full weight of your authority.\nAn example: structured technical decisions that have gone through proper review and debate within the PE community. These aren\u0026rsquo;t casual opinions—they\u0026rsquo;re genuine technical positions that have been vetted, challenged, and refined. When those decisions are communicated, they carry Position 1 authority because the process has earned it.\nIf you\u0026rsquo;re using Position 1 for everything you say, you\u0026rsquo;re overstepping your expertise, making people afraid to push back, and setting yourself up for exactly the kind of six-month-estimate failure described above.\nPosition 2: The Experienced Advisor This is the middle ground, roughly 10% of your communication. In Position 2, you\u0026rsquo;re saying, \u0026ldquo;I\u0026rsquo;ve seen situations like this before. I have a perspective worth considering. But you\u0026rsquo;re closer to this problem than I am, you\u0026rsquo;ve spent more time with the details, and if you disagree, I genuinely want to hear why.\u0026rdquo;\nPosition 2 opens the door for dialogue. It makes it safe for people to push back. It adds value through experience and pattern recognition without shutting down the people who are actually doing the work.\nThe Senior PE gave an example of this working well. He initially pushed back on a technical decision from Position 2—he was skeptical and wanted the team to think it through more carefully. After discussion and debate, the core decision turned out to be sound. People disagreed with some edge cases around it, but the foundation held. When he asked, \u0026ldquo;Are you telling me I should reverse this decision?\u0026rdquo; the answer was no. The decision stood, but the conversation refined it.\nThat\u0026rsquo;s Position 2 working as intended. You\u0026rsquo;re contributing genuine value from your experience. You\u0026rsquo;re pressure-testing the team\u0026rsquo;s thinking. But you\u0026rsquo;re not creating a dynamic where your word is final simply because you\u0026rsquo;re the most senior person in the room.\nPosition 3: The Layperson\u0026rsquo;s Perspective This is where you spend the vast majority of your time—roughly 85% percent. In Position 3, you\u0026rsquo;re saying, \u0026ldquo;I don\u0026rsquo;t have deep expertise here. As a layperson, or maybe as a potential customer of this feature, here\u0026rsquo;s a thought. It\u0026rsquo;s worth about two cents. It\u0026rsquo;s just another data point.\u0026rdquo;\nAnd honestly, much of the time when you\u0026rsquo;re in Position 3, the best thing to do is stay quiet. The team has people who actually know what they\u0026rsquo;re doing. They have data. They have context you don\u0026rsquo;t have. Another casual opinion from a senior person isn\u0026rsquo;t going to help—and it might actively hurt if the megaphone effect turns your two cents into a mandate.\nThe Senior PE shared a story that illustrates how to handle Position 3 well. A senior leader asked him point-blank what he really thought about the AI models their organization was using. His initial response was exactly right: \u0026ldquo;It\u0026rsquo;s not my place to say. I\u0026rsquo;m not a scientist. I can\u0026rsquo;t tell you whether I agree or disagree with the approach, but I\u0026rsquo;m going to commit to making it work for all of us.\u0026rdquo;\nThe leader pressed. Hard. And eventually, the Senior PE shared his concerns—the accuracy issues, the instruction-following problems, all the things that worried him. But he made sure to say: \u0026ldquo;Please don\u0026rsquo;t make decisions based on this alone.\u0026rdquo; And the leader didn\u0026rsquo;t. He triangulated with distinguished scientists and engineers, seeking diverse feedback before making consequential decisions. The PE\u0026rsquo;s input was one data point among many, weighted appropriately because both parties understood it was a Position 3 perspective.\nWhy Making This Explicit Changes Everything Here\u0026rsquo;s the transformation this framework enables:\ngraph TD subgraph Without_Framework [WITHOUT THE FRAMEWORK] A1[PE says something casual about timeline/approach] --\u0026gt; B1[\u0026#34;Team assumes Position 1 \u0026lt;br\u0026gt;(expert authority)\u0026#34;] B1 --\u0026gt; C1[Commitments made based on casual observation] C1 --\u0026gt; D1[\u0026#34;Things go wrong. Managers take the hit. \u0026lt;br\u0026gt;Trust erodes.\u0026#34;] style A1 fill:#f96,stroke:#333 style D1 fill:#ff9999,stroke:#b22222 end graph TD subgraph With_Framework [WITH THE FRAMEWORK] A2[\u0026#34;PE says: \u0026#39;Position 3 thought here—\u0026lt;br\u0026gt;take it with a grain of salt...\u0026#39;\u0026#34;] --\u0026gt; B2[Team treats it as one data point among many] B2 --\u0026gt; C2[Decision stays with people closest to the problem] C2 --\u0026gt; D2[\u0026#34;Better decisions. Trust preserved. \u0026lt;br\u0026gt;PE stays credible.\u0026#34;] style A2 fill:#69f,stroke:#333 style D2 fill:#99ff99,stroke:#228b22 end The problem isn\u0026rsquo;t that PEs have opinions outside their expertise — everyone does. The problem occurs when you speak from Position 3 but people assume you\u0026rsquo;re speaking from Position 1. That\u0026rsquo;s when teams make commitments based on your casual observations. That\u0026rsquo;s when estimates get baked into roadmaps even though you were spitballing. That\u0026rsquo;s when managers who are accountable for the outcome get hurt by a statement you forgot you even made.\nBy explicitly calling out which position you\u0026rsquo;re speaking from, you prevent that misunderstanding. You make it safe for people to weight your input appropriately. And perhaps most importantly, you protect your own credibility. A PE who is known for caveating their input honestly—who says \u0026ldquo;I don\u0026rsquo;t have the depth here to give you a reliable answer\u0026rdquo; when that\u0026rsquo;s true—earns more trust over time, not less. The PE who speaks with authority about everything eventually gets things wrong enough that people stop trusting anything they say.\nWhat This Mental Model Demands of You If you accept that scaling is a step-function change and that your voice carries outsized weight, several practical consequences follow.\nFirst, you need to audit how you\u0026rsquo;re spending your time. If most of your week is spent on activities that scale linearly with your effort—individual code reviews, one-off design consultations, debugging specific production issues—you\u0026rsquo;re operating at the wrong level of leverage. These activities made you a great PE. They won\u0026rsquo;t make you an effective Senior PE. You need to find 5x and 10x opportunities: mechanisms that multiply your impact across hundreds of engineers rather than benefiting one team at a time. (This is the subject of the next post in this series.)\nSecond, you need to develop the habit of position-labeling. Before you offer an opinion in any meeting, mentally ask yourself: is this Position 1, 2, or 3? If it\u0026rsquo;s Position 3, either stay quiet or explicitly caveat it. If it\u0026rsquo;s Position 2, invite disagreement. Reserve Position 1 for the 5% of situations where you\u0026rsquo;ve genuinely earned the authority to be definitive. This is a practice, not a one-time decision. It requires conscious effort until it becomes instinct.\nThird, you need to build leverage through relationships, mechanisms, and artifacts—not just through direct technical contribution. The Senior PE I spoke with described borrowing \u0026ldquo;superpowers\u0026rdquo; from managers (who multiply impact through organizational systems) and Technical Program Managers (who create mechanisms to coordinate work without positional authority). As a PE, you have neither direct reports nor organizational leverage in the traditional sense. What you have is technical trust and the ability to create systems—review processes, decision frameworks, knowledge-sharing forums—that carry your judgment into contexts you\u0026rsquo;ll never personally be in. We briefly touched on this in our Building Influence post and we\u0026rsquo;ll explore this in depth in the next post.\nFourth, accept that this transition is uncomfortable. The Senior PE was remarkably candid about the discomfort. He didn\u0026rsquo;t seek the spotlight. He described himself as someone who runs away from visibility, not toward it. But the role requires a kind of visibility and influence that doesn\u0026rsquo;t come naturally to most people who built their careers on deep technical work. The math of the scope change doesn\u0026rsquo;t leave room for staying comfortable. The question isn\u0026rsquo;t whether you\u0026rsquo;re comfortable with it. The question is whether you\u0026rsquo;re willing to be uncomfortable in service of the impact you can have.\nThe Real Shift Scaling as a Principal Engineer is not about doing more. It is about changing the nature of what you do.\nThe step-function mental model matters because it prevents you from optimizing the wrong thing. If you think the path to Senior PE is \u0026ldquo;be a better PE,\u0026rdquo; you\u0026rsquo;ll work harder, take on more, and eventually burn out without meaningfully expanding your impact. The engineers in your immediate orbit will benefit, but the 500 or 5,000 engineers you need to influence will never feel your presence.\nThe 3-Position Framework matters because it solves the most dangerous failure mode at this level: the gap between the weight your words carry and the depth behind them. The megaphone doesn\u0026rsquo;t care whether you intended to be authoritative. It amplifies everything equally. The only defense is to explicitly calibrate how people should receive what you\u0026rsquo;re saying—and to have the discipline to stay quiet when the honest answer is \u0026ldquo;I don\u0026rsquo;t have the depth here.\u0026rdquo;\nIf there\u0026rsquo;s one takeaway from this post, it\u0026rsquo;s this: the PE who scales effectively is not the one who becomes the best individual contributor in a larger room. It\u0026rsquo;s the one who recognizes that the room changed, the rules changed, and the leverage points changed—and rebuilds their operating model accordingly.\nThe next post in this series covers the operational side of this shift: how to borrow leverage from other disciplines, create mechanisms that carry your influence without your presence, and scale your knowledge through writing and artifacts.\n","permalink":"https://blog.girishshekhar.com/posts/scaling-as-a-pe/1-step-function-change/","summary":"Most Principal Engineers try to scale by doing more of what already works. That approach breaks down when your scope jumps from 50 engineers to 5,000. This post introduces the step-function mental model for PE growth and the Three-Position Framework for managing the outsized influence your voice carries at this level.","title":"Scaling as a Principal Engineer Is a Step-Function Change, Not a Gradient"},{"content":"Why This Post Exists A large part of my role involves mentoring senior technologists in staff and senior staff roles. During my conversations with them, I have been asked what is the best way to increase influence. There is a transition point in every engineer\u0026rsquo;s career where the thing that got you here stops being the thing that gets you further. You\u0026rsquo;ve built the technical depth. You can ramp up on new systems, debug production incidents, and design architectures that hold. And then you walk into a room full of Sr. Managers, Directors and C-suite executives and you don\u0026rsquo;t speak up — even when you know the answer.\nThis is not a confidence problem. It is a model problem. The mental model most senior engineers carry about how influence works, how mentorship works, and what \u0026ldquo;growth\u0026rdquo; looks like is incomplete. And that incomplete model creates real friction: missed opportunities to shape decisions, stalled relationships with stakeholders, and the quiet frustration of knowing you belong in the room but not feeling like you can operate in it.\nThis post is about replacing that model. It draws from real mentorship conversations and hard-won patterns about what actually works when you are trying to grow from a strong individual technologist into a trusted leader — where impact is measured less by what you build and more by what you enable others to build. While you have been set up with a support system to increase your influence like mentors, operating across different roles in the organization, you have not been told on how to effectively leverage this system.\nThe Misconception: Mentorship Is a One-Person Relationship The default mental model for mentorship looks like this: find a senior person, meet regularly, ask questions, get advice. One mentor, one mentee, one relationship.\nThis is not mentorship. It is a dependency.\nThe real shift is understanding that effective mentorship is a village, not a single relationship. You don\u0026rsquo;t need one person who can answer every question. You need a network of people across distinct job families, each contributing a different capability to your growth.\nHere is what that village actually looks like:\nA peer might be teaching you something they do better than you — maybe operational rigor, maybe system design intuition. A Technical Program Manager (TPM) might be showing you how they manage stakeholders across a large footprint, because TPMs are extraordinarily good at influencing without authority. An engineering manager (SDM) might be demonstrating how they delegate and execute — they are, in many ways, execution machines, and there are patterns worth studying in how they break down and distribute work leading through others. A product manager might be helping you understand how they define problems and prioritize, which changes how you frame technical decisions.\nNone of these relationships need to be formal. They don\u0026rsquo;t need standing agendas or structured formats. What they need is intentionality — you know what you are trying to learn from each person and why that specific job family has something to teach you that your own does not.\nMental model: Mentorship is not a ladder you climb with one guide. It is an umbrella you build from diverse relationships, each covering a different gap. The individuals who have an accelerated growth are the ones who build the widest umbrella, not the ones who find the single \u0026ldquo;best\u0026rdquo; mentor.\nThe Cap You Hit: When Technical Strength Stops Being the Bottleneck Here is something most senior technologists don\u0026rsquo;t want to hear: you are starting to reach the cap of what technical strength alone can do for you.\nThis is not a criticism. It is a structural observation. At the senior level, you\u0026rsquo;ve proven you can solve hard problems, ramp up on new domains, learn and be curious, and deliver technically sound work. Those are table stakes. They got you where you are today, and they will always matter. But they are no longer the differentiator.\nThe next phase of growth lives under what I call the people umbrella. It includes influence without authority, stakeholder management, building trust across organizational boundaries, and the soft skills that most engineers dismiss as \u0026ldquo;not real work\u0026rdquo; until they realize those skills are exactly what separates a strong staff from a sr. staff or a sr. staff from a principal.\nThis matters because the problems you are asked to solve now are not harder technically — they are harder organizationally. They span multiple teams, multiple managers, multiple priorities. The technical solution is often the easy part. Getting six teams to align on it, getting a skeptical SDM to allocate resources, getting a product leader to prioritize your use case — that is the actual work. And none of it responds to technical depth alone.\nSpeaking Up: Why Qualifiers Are More Powerful Than Confidence Let\u0026rsquo;s talk about a specific symptom of this transition: the hesitation to speak up in rooms with senior people you don\u0026rsquo;t know well.\nThe naive model says this is a confidence problem, and the solution is to \u0026ldquo;just be more confident.\u0026rdquo; That advice is useless. It is like telling someone who is lost to \u0026ldquo;just know where you are.\u0026rdquo;\nThe real issue is risk perception. When you are in a room with people whose work style you don\u0026rsquo;t know, whose expectations you can\u0026rsquo;t predict, and whose opinion of you matters for your career, speaking up feels like an unhedged bet. What if I\u0026rsquo;m wrong? What if I leave the wrong impression? What if these people decide I don\u0026rsquo;t know what I\u0026rsquo;m talking about?\nThe tool that actually works here is the qualifier. A qualifier is a framing statement that reduces the perceived risk of contributing, both for you and for the room. It sounds like this:\n\u0026ldquo;I don\u0026rsquo;t think I\u0026rsquo;m an expert at this, but with the experience I\u0026rsquo;ve had on this side, I believe it should be done this way.\u0026rdquo;\nOr: \u0026ldquo;I have a strong opinion here based on the post mortem my team has written, but I\u0026rsquo;d love to pressure-test it.\u0026rdquo;\nOr even: \u0026ldquo;This might be a dumb question, but what would happen if we did X?\u0026rdquo;\nThe qualification removes the hesitation because it explicitly scopes your claim. You are not asserting authority you don\u0026rsquo;t feel you have. You are contributing a perspective while acknowledging its boundaries. And here is the counterintuitive part: qualifiers don\u0026rsquo;t make you look weaker. In rooms filled with executives, they make you look more credible, because they signal that you understand the limits of your own knowledge — which is exactly the kind of judgment those rooms value.\nThis is not about false modesty. It is about giving yourself a framework to contribute when the alternative is silence. Silence in a room where you have the answer is not humility. It is a missed opportunity to build the reputation that earns you influence later.\nInfluence Without Authority: The Actual Mechanics Influence without authority is the phrase everyone uses and almost nobody teaches concretely. So let me be concrete.\nStart From Customer Problems, Not Technology When you are trying to influence a team you don\u0026rsquo;t manage, resist the instinct to lead with your technical solution. Lead with the customer problem.\nThis is not a platitude. It is a tactical choice. When you walk into a room and say \u0026ldquo;I need your team to integrate with my API,\u0026rdquo; you have framed the conversation as a resource request. The other team\u0026rsquo;s SDM is now doing mental math on capacity, priorities, and whether your request is worth disrupting their roadmap. You have made yourself an obstacle to their goals.\nWhen you walk into the same room and say \u0026ldquo;We have a customer problem where X is happening, and I\u0026rsquo;d love to brainstorm how we might solve it together,\u0026rdquo; you have framed the conversation as a shared challenge. The technology will come naturally — it always does for engineers. But grounding on the customer problem first means you\u0026rsquo;re not asking for something. You\u0026rsquo;re inviting collaboration. And that reframe changes everything about how the conversation unfolds.\nListen Before You Solve Here is a concrete example of what this looks like in practice. An SDM once rejected a proposal outright in a cross org group review with senior management. His team was underwater — they had launched a major product on a monolithic architecture, operational metrics were degrading, and they were in the middle of migrating to a new framework. The proposal to build on their old codebase felt like someone dumping more work on a sinking ship.\nThe instinct is to push harder, escalate, or find a workaround. What actually worked was a back-channel conversation: \u0026ldquo;Tell me more about the problem. What\u0026rsquo;s really going on?\u0026rdquo;\nThat conversation revealed the full picture — the scaling challenges, the operational mess, the migration timeline. And it opened up a different approach entirely: \u0026ldquo;What if I do the work on both the old and new systems? I\u0026rsquo;ll even activate some features on your new framework that give you a win too.\u0026rdquo;\nThe SDM went from blocking the work to actively supporting it. Not because the technical proposal changed — but because the relationship changed. He went from being a stakeholder who felt imposed upon to a collaborator who felt heard.\nThe pattern here is worth naming explicitly:\nWhat most engineers do: → Present solution → Ask for support → Get blocked → Escalate What actually works: → Ask about their problems → Listen → Find mutual wins → Earn support Map the Stakeholders, Find the Decision Makers Not everyone in a group of stakeholders has equal decision-making power. Everyone wants visibility, but only a few people will actually make the real decisions. Your job as a senior technologist is to identify who those people are.\nSometimes you\u0026rsquo;ll get lucky and connect with the decision maker directly. Sometimes you won\u0026rsquo;t. When direct influence fails, the move is lateral: find the trusted individual that the decision maker trusts. Can you build a relationship with their team\u0026rsquo;s senior technologist or trusted advisor? Can you offer to mentor someone on their team? Can you find a peer IC (individual contributor) in that organization and grab coffee?\nThis is not manipulation. It is how organizational influence actually works. As an IC, it is far easier to influence another IC than it is to influence a leader two levels above you. Peer-to-peer relationships are the natural channel for IC influence. Use them.\nAdapt Your Communication Style Here is a story that still stings. I once spent a month and a half failing to make progress with an senior leader in another organization. I brought well-structured documents. I brought clarity. I brought everything that had always worked before. Nothing landed.\nIt met with a senior engineer on his team who finally told me: \u0026ldquo;You\u0026rsquo;re doing it wrong. He doesn\u0026rsquo;t like docs. He likes open-ended conversations.\u0026rdquo;\nI had been optimizing for the wrong communication medium. Every interaction was friction because I was forcing my preferred format onto someone whose brain worked differently. Once I switched to unstructured, conversational whiteboard sessions, everything unlocked.\nMental model: Influence without authority is not a single skill. It is a system of skills — reading stakeholders, adapting communication, finding mutual wins, building trust through peer relationships, and knowing when to push and when to listen. And like any system, it only works when the components are connected.\nWhat This Means for Your Growth If you are a senior technologist reading this and recognizing yourself — strong technically, hesitant in unfamiliar rooms, unclear on how influence actually works — here is how to think about your next phase of growth.\nFirst, build your village. Identify gaps in your growth that a single mentor cannot fill. Find a TPM who can teach you stakeholder management. Find an SDM who can show you how they delegate. Find a peer who does something better than you and learn from them. The diversification of your mentorship network is not optional — it is structural to how you grow at this level.\nSecond, start using qualifiers. They are not a crutch. They are a tool that lets you contribute in rooms where silence is currently your default. Practice them until they feel natural, and notice how they change the dynamic of your participation.\nThird, invest in the people\u0026rsquo;s umbrella. This is not soft, fluffy, nice-to-have work. It is the actual work of operating at the sr staff or principal position. The technical depth you\u0026rsquo;ve built is the foundation. The relationships, the influence patterns, the communication adaptability — that is the structure you build on top of it.\nIf there\u0026rsquo;s one takeaway, it\u0026rsquo;s this: growth from one to another level is not a deeper version of the same game. It is a different game entirely. The engineers who make that transition successfully are not the ones who get even better at technology. They are the ones who build a village, learn to influence without authority, and understand that the hardest system to design is the one made of people.\n","permalink":"https://blog.girishshekhar.com/posts/building-influence-with-a-village/","summary":"Most engineers treat mentorship as a one-person relationship and influence as a byproduct of technical skill. Both assumptions break down at this level. This post introduces the village model for mentorship, practical tactics for influence without authority, and the mental shift from closing conversations to opening relationships.","title":"The Village Model: How To Actually Build Influence"},{"content":"Purpose Up to this point, this series has focused on:\nWhat changed with LLMs How they work Why they fail How systems compensate What skills engineers need to adapt This final post answers the most practical question of all:\n“How does this actually show up in day-to-day engineering work?”\nNot hypotheticals.\nNot demos.\nJust the places where LLMs genuinely help — and the places where experienced engineers deliberately keep them at arm’s length.\nLLMs Are Multipliers, Not Replacements The most consistent pattern across teams is this:\nLLMs don’t replace engineering judgment.\nThey amplify whatever judgment already exists.\nFor experienced engineers, that often means:\nFaster iteration Better exploration Less cognitive overhead on low-leverage tasks For inexperienced use, it can mean:\nConfidently shipping the wrong thing Masking misunderstanding Accumulating hidden technical debt This distinction matters more than any specific use case.\nCode Generation: Useful, But Narrowly Scoped LLMs are surprisingly effective at generating code — within constraints.\nThey work best when:\nThe problem is well-scoped The interfaces are known The patterns are common The output is reviewed, not trusted Examples where engineers use them productively:\nWriting boilerplate Translating between languages or frameworks Generating test scaffolding Exploring unfamiliar APIs graph TD A[Well-Scoped Problem] --\u0026gt; B[LLM Generation] B --\u0026gt; C{Human Review} C --\u0026gt;|Syntax Error / Hallucination| D[Refinement Prompt] D --\u0026gt; B C --\u0026gt;|Logic Correct| E[Integration \u0026amp; Testing] E --\u0026gt; F[Production Code] style B fill:#f9f,stroke:#333 style C fill:#bbf,stroke:#333,stroke-width:4px style F fill:#00ff00,stroke:#333 Where engineers stay cautious:\nCore business logic Performance-critical paths Security-sensitive code Subtle concurrency or correctness issues Mental model:\nTreat generated code like a first draft from a junior engineer — helpful, but never authoritative.\nExplaining Existing Code (This Is the Quiet Superpower) One of the most consistently valuable uses of LLMs is explanation.\nEngineers use LLMs to:\nSummarize unfamiliar codebases Explain legacy logic Walk through control flow Translate dense code into plain language This is especially powerful when:\nOnboarding to a new team Debugging unfamiliar services Revisiting old systems with no original context graph LR Code[Dense Legacy Code] --\u0026gt; LLM{LLM Processor} LLM --\u0026gt; Sum[Executive Summary] LLM --\u0026gt; Flow[Control Flow Breakdown] LLM --\u0026gt; Risks[Potential Edge Cases] Sum --\u0026gt; Engineer((Deep Understanding)) Flow --\u0026gt; Engineer Risks --\u0026gt; Engineer style Code fill:#f9f,stroke:#333 style LLM fill:#bbf,stroke:#333 style Engineer fill:#dfd,stroke:#333,stroke-width:2px Unlike generation, explanation reduces risk:\nYou still control the final interpretation Errors are easier to spot Understanding compounds over time Research, Comparison, and Option Mapping Another strong use case is structured exploration.\nEngineers use LLMs to:\nCompare libraries and tools Summarize tradeoffs Generate decision matrices Identify unknown unknowns The key is that the model is used to surface options, not make decisions.\nPattern:\nLLMs expand the search space.\nHumans collapse it.\nThis is particularly useful early in design phases, where breadth matters more than precision.\nTest Generation and Edge Case Discovery LLMs are effective at generating:\nUnit test skeletons Input variations Boundary cases Failure scenarios They often surface:\nAssumptions baked into code Missing validation Unhandled edge cases graph TD Code[Source Code] --\u0026gt; LLM{LLM \u0026#39;Breaker\u0026#39;} subgraph \u0026#34;Inspiration Phase\u0026#34; LLM --\u0026gt; B1[Empty Inputs] LLM --\u0026gt; B2[Latency/Timeout Scenarios] LLM --\u0026gt; B3[Type Mismatches] end B1 \u0026amp; B2 \u0026amp; B3 --\u0026gt; Engineer{Curator} Engineer --\u0026gt;|Discard| Junk[Irrelevant Noise] Engineer --\u0026gt;|Accept| Suite[Hardened Test Suite] style LLM fill:#ff4d4d,stroke:#333,color:#fff style Engineer fill:#bbf,stroke:#333,stroke-width:4px That said, engineers still:\nCurate the tests Decide what matters Remove irrelevant cases The value is in coverage inspiration, not correctness guarantees.\nData Analysis and One-Off Queries For exploratory analysis, LLMs shine.\nEngineers and data practitioners use them to:\nTranslate questions into SQL Explain unfamiliar schemas Generate quick summaries Prototype transformations This is especially helpful when:\nThe cost of being wrong is low Results are manually inspected The goal is insight, not automation LLMs reduce friction — not responsibility.\nWhere Engineers Avoid Using LLMs Equally important are the places LLMs are not used.\nExperienced engineers are cautious about:\nSilent automation Decision-making without oversight Long-lived logic embedded in prompts Systems with unclear failure modes If a mistake would:\nBreach trust Violate policy Create cascading failures then the system usually includes:\nHuman review Hard constraints Or no LLM at all This restraint is a feature, not a limitation.\nThe Meta-Shift: How Engineers Think Changes The biggest change isn’t any single workflow.\nIt’s how engineers allocate attention.\nLLMs shift effort away from:\nMechanical translation Repetitive scaffolding Recall-heavy tasks And toward:\nProblem framing System boundaries Failure analysis Tradeoff communication In that sense, LLMs don’t make engineering easier — they make judgment more central.\nClosing Thoughts Large Language Models are not magic.\nThey are:\nProbabilistic Fallible Context-dependent But when placed thoughtfully into engineering workflows, they reduce friction in places that used to drain energy — without removing the need to think.\nThe teams that succeed are not the ones who automate the most.\nThey are the ones who remain clear about where automation ends and responsibility begins.\nThat clarity, more than any model choice, is what will matter long-term.\nEnd of Series This concludes “LLMs and How Our Work Is Changing.”\nIf there’s one idea to carry forward, it’s this:\nLLMs don’t change what good engineering is.\nThey change where it shows up.\n","permalink":"https://blog.girishshekhar.com/posts/llms-and-how-our-work-is-changing/7-real-world-application/","summary":"After all the theory and architecture, this post looks at how experienced engineers actually use Large Language Models in their daily work — where they help, where they hurt, and how to integrate them without outsourcing thinking.","title":"LLMs in Day-to-Day Engineering Work"},{"content":"Purpose By now, we’ve talked about:\nWhy LLMs behave differently from prior ML systems Why they fail in predictable ways Why architectural patterns like RAG keep emerging At this point, a common reaction is:\n“Okay, but what do engineers actually need to learn?”\nThe answer is not “prompt engineering” in the narrow sense.\nLLMs don’t require a new programming language.\nThey require new engineering instincts.\nThis post focuses on the skills that actually transfer to production systems — not tricks, templates, or vendor-specific features.\nPrompting Is Interface Design, Not Spellcasting A useful mental shift is to stop thinking about prompts as magic incantations.\nA prompt is an interface.\nIt defines:\nWhat context the model receives What constraints it should respect What form the output should take What assumptions are left implicit Poor prompts fail for the same reason poor APIs fail:\nAmbiguous contracts Hidden dependencies Overloaded responsibilities graph LR subgraph \u0026#34;The User (Intent)\u0026#34; A[Vague Idea] --\u0026gt; B[Structured Prompt] end subgraph \u0026#34;The Interface (The Contract)\u0026#34; B --\u0026gt; C{Rules \u0026amp; Constraints} B --\u0026gt; D{Context \u0026amp; Grounding} B --\u0026gt; E{Output Format} end subgraph \u0026#34;The Model (Execution)\u0026#34; C --\u0026gt; F[Token Prediction] D --\u0026gt; F E --\u0026gt; F F --\u0026gt; G[Result] end style B fill:#f96,stroke:#333,stroke-width:2px style F fill:#bbf,stroke:#333 Key reframing:\nPrompt quality is less about clever wording and more about clarity of intent.\nEngineers who are good at API design often adapt quickly here.\nReasoning About Probabilistic Systems Traditional software rewards determinism.\nLLMs don’t.\nYou are no longer asking:\n“Is this code correct?”\nYou are asking:\n“Is this behavior acceptable across a distribution of outcomes?”\nThis requires new habits:\nTesting multiple runs, not one Evaluating failure patterns, not single failures Designing guardrails instead of guarantees graph TD In((Input)) --\u0026gt; P{The Prompt} P --\u0026gt;|Path A| R1[Outcome 1: Ideal] P --\u0026gt;|Path B| R2[Outcome 2: Verbose] P --\u0026gt;|Path C| R3[Outcome 3: Hallucination] P --\u0026gt;|Path D| R4[Outcome 4: Refusal] style R1 fill:#d4edda,stroke:#28a745 style R3 fill:#f8d7da,stroke:#dc3545 style R4 fill:#fff3cd,stroke:#ffc107 This is uncomfortable at first — especially for engineers trained to eliminate nondeterminism.\nBut it’s essential.\nDesigning for Failure Up Front One of the biggest mistakes teams make is treating LLM failures as edge cases.\nThey are not.\nFailures are the default state unless you design around them.\nThat design usually includes:\nValidation of outputs Confidence thresholds Fallback behavior Human review paths Analogy:\nYou wouldn’t deploy a distributed system without retries and timeouts.\nLLM-based systems deserve the same treatment.\nThis is reliability engineering, not AI research.\nSeparating Knowledge from Reasoning LLMs are good at reasoning over text.\nThey are bad at being the source of truth.\nStrong systems separate:\nKnowledge (databases, documents, policies) Reasoning (summarization, explanation, transformation) RAG is one manifestation of this principle, but the idea is broader.\ngraph LR subgraph \u0026#34;External State (Knowledge)\u0026#34; DB[(Vector Database)] Docs[PDFs/Logs] API[Live APIs] end subgraph \u0026#34;The Reasoning Engine\u0026#34; LLM{LLM} end DB -.-\u0026gt;|Inject Facts| LLM Docs -.-\u0026gt;|Inject Facts| LLM API -.-\u0026gt;|Inject Facts| LLM LLM --\u0026gt;|Transform/Reason| Output[Final Answer] style LLM fill:#bbf,stroke:#333,stroke-width:4px style DB fill:#f9f,stroke:#333 Engineers who already think in terms of:\nStateless services Externalized state Clear ownership boundaries will find this pattern familiar.\nThinking in Systems, Not Models A recurring theme in this series is that model choice is rarely the hardest decision.\nWhat matters more:\nHow context is assembled How outputs are validated How failures propagate How users recover from errors This pushes engineers toward:\nPipeline thinking Observability beyond logs Feedback loops with users Important:\nMost “LLM problems” turn out to be systems problems wearing new clothes.\nSecurity and Trust Boundaries Change LLMs collapse boundaries between:\nInput and instruction Data and control User intent and system behavior That has security implications.\nEngineers now need to think about:\nPrompt injection Data exfiltration via generation Overly broad context exposure Indirect instruction following graph TD U[User Input] --\u0026gt; G1{Guardrail LLM} G1 --\u0026gt;|Clean| P[Primary LLM] G1 --\u0026gt;|Malicious| R[Block/Reset] P --\u0026gt; G2{Output Scrubber} G2 --\u0026gt;|Safe| Final[Display to User] G2 --\u0026gt;|Sensitive Data Found| Mask[Redact/Block] style G1 fill:#ffcccb,stroke:#333 style G2 fill:#ffcccb,stroke:#333 style P fill:#bbf,stroke:#333 This isn’t fundamentally new — it resembles SQL injection and XSS — but the surface area is wider and less explicit.\nBoundary Traditional Software LLM Systems Input vs. Instruction eval() is a known danger. Everything is eval(). Data vs. Control SQL Parameters separate data. Data can contain \u0026ldquo;\u0026ldquo;System\u0026rdquo;\u0026rdquo; commands. Trust Boundary Hard-coded logic defines limits. Probability determines behavior. Evaluation Becomes a First-Class Skill You cannot improve what you cannot measure.\nBut evaluating LLM systems is harder than measuring:\nLatency Error rates Throughput Evaluation often involves:\nHuman judgment Task-specific criteria Qualitative feedback Teams that succeed tend to:\nDefine “good enough” explicitly Track regressions over time Invest in evaluation infrastructure early This is closer to product design than pure engineering — and that’s not accidental.\nWhat This Means for Engineering Careers None of this replaces core engineering skills.\nIt amplifies them.\nEngineers who thrive in LLM-heavy environments tend to be strong at:\nSystems thinking API and interface design Debugging ambiguous behavior Communicating tradeoffs to non-engineers The models will keep changing.\nThese skills will not.\nWhat’s Next In Part 7, we’ll ground all of this in day-to-day engineering work:\nWhere LLMs actually help Where they slow teams down How experienced engineers use them without over-relying on them This is where theory meets lived experience.\n","permalink":"https://blog.girishshekhar.com/posts/llms-and-how-our-work-is-changing/6-leveling-up/","summary":"Large Language Models don’t just introduce new tools — they demand new skills. This post explains what actually changes for engineers and technical leaders when LLMs become part of production systems, and where traditional instincts need to be updated.","title":"The Skills Engineers Need to Level Up for LLM-Based Systems"},{"content":"Purpose Up to now, this series has focused on understanding what LLMs are and why they behave the way they do.\nIf you’ve read the earlier posts, one conclusion should already feel obvious:\nLeft on their own, LLMs are powerful but unreliable.\nThey generate fluent text, but they do not:\nKnow what is true right now Know what is true for your organization Know which source should be trusted Retrieval-Augmented Generation (RAG) exists to address exactly that gap.\nThis post explains:\nWhy RAG emerged What problem it actually solves How it changes system design decisions Where teams commonly get it wrong The Core Problem: Models Don’t Know Your World LLMs are trained on large, static datasets.\nNo matter how large the model:\nIt does not know your internal documentation It does not know your database schemas It does not know last week’s policy change It does not know which source you consider authoritative When you ask a question that depends on private, recent, or domain-specific knowledge, the model must guess.\nAnd as we saw in Part 4, it will guess confidently.\nThis is not a failure of intelligence.\nIt is a mismatch between where knowledge lives and how models generate text.\nWhat RAG Actually Does Retrieval-Augmented Generation changes where the model gets its context.\nInstead of relying solely on its internal parameters a.k.a. parametric knowledge, the system:\nRetrieves relevant external information Injects that information into the prompt Asks the model to generate an answer grounded in that context The model is still doing what it always does — predicting the next token.\nBut now it’s doing so with explicit, curated grounding.\ngraph LR User([User Query]) --\u0026gt; Search[1. Search Engine / Vector DB] Search --\u0026gt; Context[2. Relevant Documents Found] subgraph \u0026#34;The Prompt Sandwich\u0026#34; Context --\u0026gt; Prompt[\u0026#34;[Grounding Data] + [User Query]\u0026#34;] end Prompt --\u0026gt; LLM[3. LLM Processing] LLM --\u0026gt; Output[4. Grounded Answer] style Search fill:#f9f,stroke:#333 style Context fill:#bbf,stroke:#333 style Prompt fill:#dfd,stroke:#333 style LLM fill:#fff,stroke:#333,stroke-width:4px Mental model:\nRAG doesn’t make the model smarter.\nIt gives the model better evidence.\nWhy Retrieval Works Better Than Fine-Tuning (Most of the Time) A common early instinct is to fine-tune the model on proprietary data.\nSometimes that’s appropriate. Often, it’s not.\nFine-tuning:\nBakes knowledge into model weights Is expensive to repeat Is hard to audit Is slow to update Retrieval:\nKeeps knowledge external Allows updates without retraining Preserves source attribution Makes failures easier to debug This separation of concerns turns out to be critical.\nAnalogy:\nFine-tuning is like rewriting your application binary every time data changes.\nRetrieval is like querying a database at runtime without the need for redeployment\nMost production systems prefer the latter.\nA Concrete Walkthrough Consider a simple internal Q\u0026amp;A system.\nA user asks:\n“What is our data retention policy for customer logs?”\nA RAG-based system might:\nConvert the question into an embedding Search a vector index of internal documents Retrieve the most relevant policy excerpts Inject those excerpts into the prompt Ask the model to answer using only that context The model is no longer inventing policy language.\nIt is summarizing and explaining content you already trust.\nRetrieval Is an Architectural Choice, Not a Feature One mistake teams make is treating RAG as a bolt-on enhancement.\nIn practice, retrieval affects:\nData pipelines Indexing strategies Latency budgets Failure modes Security boundaries You are no longer just building a “prompt.”\nYou are building an information system.\nKey design questions emerge immediately:\nWhat data is indexed? How often is it refreshed? How is access controlled? How do we know which source won? These are not model questions.\nThey are classic systems and data engineering questions. These have existed in Search Systems long before Machine Learning was democratized.\nRetrieval Quality Matters More Than Model Choice A surprisingly consistent observation in production systems:\nBetter retrieval beats a bigger model.\nIf retrieved context is:\nIrrelevant Outdated Too verbose Contradictory then even the best model will struggle.\nConversely, a smaller model with clean, focused context often outperforms a larger one operating blind.\nThis is why teams spend more time tuning:\nChunking strategies Embedding models Ranking heuristics than swapping LLM vendors.\nCommon Failure Modes in RAG Systems RAG does not magically solve reliability. It changes where failures live.\nCommon issues include:\nRetrieving too much context, overwhelming the model Retrieving too little, forcing hallucination Indexing documents without structure Treating embeddings as ground truth rather than heuristics Important:\nRetrieval is probabilistic too.\nYou are stacking uncertainty unless you design carefully.\nWhy RAG Is Becoming the Default Pattern RAG keeps showing up because it aligns with how organizations already operate.\nKnowledge lives outside models Data changes frequently Trust varies by source Auditing matters RAG lets models stay general while systems stay specific.\nThat separation is what makes LLMs usable at scale.\nWhat’s Next Now that we’ve covered grounding and retrieval, the next post shifts focus to people and practice.\nIn Part 6, we’ll talk about:\nSkills engineers actually need to develop Prompting as an interface design problem Reliability, security, and failure containment Why “prompt engineering” is a misleading term This is where LLMs stop being a novelty and start becoming part of everyday engineering work.\n","permalink":"https://blog.girishshekhar.com/posts/llms-and-how-our-work-is-changing/5-retrieval-augmented-generation/","summary":"Retrieval-Augmented Generation (RAG) is one of the most important architectural patterns to emerge alongside Large Language Models. This post explains why RAG exists, how it works, and what problems it actually solves — without turning it into a tooling tutorial.","title":"Retrieval-Augmented Generation (RAG): Grounding LLMs in Reality"},{"content":"Purpose By this point in the series, we’ve covered:\nWhy LLMs changed the game How they work internally (at a usable level of detail) How different model families trade off capability, cost, and control And yet, for many teams, the lived experience still looks like this:\n“It worked perfectly yesterday.”\n“Why did it answer confidently but completely wrong?”\n“Why does changing one sentence break the whole thing?”\nThis post is about that gap.\nThe goal here is not to catalog edge cases or failure anecdotes.\nIt’s to explain why these failures are expected, and why treating LLMs like deterministic software almost always leads to frustration.\nThese Systems Are Not Executing Logic Traditional software executes instructions.\nEven most classical machine learning systems — classifiers, recommenders, regressors — produce outputs that feel mechanical. Given the same input, you reliably get the same output.\nLLMs do not work that way.\nAn LLM is sampling from a probability distribution over tokens. Even when temperature is set low, the system is still choosing plausible continuations, not correct ones.\nMental model:\nAn LLM is not retrieving facts or running rules.\nIt is producing the most statistically likely continuation of text given the context provided.\nOnce you internalize that, many “weird” behaviors stop being surprising.\nConfident Nonsense Is a Feature, Not a Bug One of the most jarring behaviors for new users is how confidently wrong LLMs can be.\nThey:\nUse authoritative tone Cite plausible-sounding details Follow formatting perfectly Remain calm even when fabricating This is not deception. It’s alignment between style and training objective.\nFeature The Goal: Plausibility The Reality: Accuracy Training Reward \u0026ldquo;\u0026ldquo;Does this look like a smart person wrote it?\u0026rdquo;\u0026rdquo; \u0026ldquo;\u0026ldquo;Is this statement true in the real world?\u0026rdquo;\u0026rdquo; Mechanism Pattern matching and style replication. Fact-checking against a live database. Outcome High Fluency: Perfect grammar, authoritative tone. Variable Truth: Correct only if the pattern holds. During training, models are rewarded for producing text that looks like high-quality human writing — not for checking facts against an external source.\ngraph TD subgraph \u0026#34;Training Objective\u0026#34; A[Text Quality] --\u0026gt; B[Fluent Syntax] A --\u0026gt; C[Authoritative Tone] A --\u0026gt; D[Logical Formatting] end subgraph \u0026#34;The \u0026#39;Hallucination\u0026#39; Loop\u0026#34; E[No Fact Found in Weights] --\u0026gt; F[Predict \u0026#39;Most Likely\u0026#39; Next Token] C --\u0026gt;|Conditioning| F F --\u0026gt; G[Confident Nonsense] end style G fill:#ff4d4d,stroke:#333,color:#fff style A fill:#bbf,stroke:#333 If the training data contains:\nConfident explanations Structured answers Well-formed arguments Then confidence becomes part of the learned pattern.\nWhy this matters architecturally You cannot rely on tone or fluency as a proxy for correctness.\nSystems must verify outputs outside the model.\nThis is why retrieval, validation, and constraints show up again and again in production designs.\nPrompt Sensitivity Is Structural Small prompt changes can cause disproportionately large output changes.\nThis often gets dismissed as “prompt engineering immaturity,” but the deeper reason is simpler:\nPrompts are not instructions. They are conditioning signals.\nA prompt does not tell the model what to do.\nIt shifts the probability distribution over what comes next.\nChange the wording, and you change:\nToken boundaries Attention patterns Which prior examples the model implicitly activates Analogy:\nThink of prompts less like function calls and more like nudging a crowd’s attention.\ngraph TD A[Raw Model State] --\u0026gt;|Neutral| B(Equal Probability for All Tokens) subgraph Conditioning C[Prompt: \u0026#39;Explain like a scientist\u0026#39;] D[Prompt: \u0026#39;Explain like a pirate\u0026#39;] end B --\u0026gt; C B --\u0026gt; D C --\u0026gt;|High Prob| E[Technical Jargon / Passive Voice] D --\u0026gt;|High Prob| F[Slang / Nautical Metaphors] E --\u0026gt; G((Result A)) F --\u0026gt; H((Result B)) style C fill:#d1e7ff,stroke:#004a99 style D fill:#ffe5d1,stroke:#994f00 style G fill:#00ff00,stroke:#000 style H fill:#00ff00,stroke:#000 This is why:\nLong prompts degrade unpredictably Over-specified instructions sometimes perform worse Examples (“few-shot”) often outperform explicit rules Long Context Feels Like Memory — But Isn’t Large context windows give the illusion of memory.\nThe model can reference things you mentioned earlier, sometimes dozens of pages back. But that doesn’t mean it remembers them in the human sense.\nContext is:\nA sliding window Reprocessed on every token Not stored or retrieved later As context grows:\nCosts increase Attention becomes diffuse Important details compete with irrelevant ones graph TD subgraph \u0026#34;The \u0026#39;Sliding\u0026#39; Window\u0026#34; A[Earliest Input] --- B[Middle Context] B --- C[Recent Instructions] C --- D[Current Query] end D --\u0026gt; E{Attention Mechanism} subgraph \u0026#34;Dilution Effect\u0026#34; E -.-\u0026gt;|Weak Focus| A E -.-\u0026gt;|Moderate Focus| B E ==\u0026gt;|High Focus| C E ==\u0026gt;|Priority| D end style A fill:#f9f,stroke:#333,stroke-dasharray: 5 5 style D fill:#00ff00,stroke:#333 style E fill:#fff,stroke:#333,stroke-width:4px Key misconception:\nMore context does not always mean better answers.\nThis is why summarization, chunking, and retrieval are architectural necessities — not optimizations.\nGenerality Hides Unspoken Assumptions LLMs feel flexible because they’ve seen a little bit of everything.\nBut that breadth also means they carry:\nImplicit cultural assumptions Averaged conventions Conflicting norms across domains When you ask a vague question, the model fills in the blanks using statistical priors — not your intent.\nThis shows up most clearly when:\nDomain-specific language overlaps with general terms Organizational context is missing Edge cases contradict common patterns Why These Failures Keep Reappearing None of the failure modes above are solved by:\nBigger models alone Better prompts alone More training data alone They are system-level problems.\nLLMs are powerful components, but fragile foundations.\nTreating them like:\nAPIs Rule engines Deterministic services almost always leads to surprises.\nThe Architectural Takeaway If there’s one thing to internalize from this post, it’s this:\nLLMs must be surrounded by structure.\nThat structure typically includes:\nExternal data sources Validation layers Feedback loops Human-in-the-loop checkpoints Not because models are bad — but because they are probabilistic by design.\nWhat’s Next In the next post, we’ll look at Retrieval-Augmented Generation (RAG).\nNot as a buzzword, but as an architectural pattern that:\nCompensates for model weaknesses Shifts reliability boundaries Changes where complexity actually lives That’s where theory turns into systems engineering.\n","permalink":"https://blog.girishshekhar.com/posts/llms-and-how-our-work-is-changing/4-why-llms-fail/","summary":"Large Language Models often fail in ways that feel inconsistent or irrational. This post explains why those failures are not bugs, but natural consequences of probabilistic systems trained on language — and what that means for real product and system design.","title":"Why LLMs Fail in Surprising Ways (And Why That’s Expected)"},{"content":"Why Model Choice Is Harder Than It Looks After teams get past the initial excitement of “LLMs can do this now,” they quickly hit a practical question:\nWhich model should we actually use?\nThis is where many discussions go off the rails.\nModel choice is often framed as a popularity contest — which model is bigger, newer, or ranks higher on a benchmark. In practice, choosing an LLM is an architectural decision, not a feature comparison. The wrong choice won’t just cost more money; it will surface as reliability issues, latency spikes, and systems that are difficult to reason about or control.\nThe key mistake is assuming that LLMs differ primarily in intelligence. They don’t. They differ in tradeoffs.\nA More Useful Way to Think About LLMs At a high level, most modern LLMs share the same underlying architecture and training paradigm (as covered in Part 2). What changes is not what they are, but what they are optimized for.\nThe most important dimensions tend to be:\nCapability – how well the model performs across a range of tasks Cost – both training and inference, usually tied to token usage Latency – how quickly responses are generated at scale Controllability – how predictable and steerable outputs are Domain alignment – how well the model matches your problem space Every model sits somewhere different along these axes. There is no universally “best” model — only models that are better or worse aligned with your system’s constraints.\nGeneral-Purpose Models: Breadth First General-purpose models are trained on massive, diverse corpus and optimized to perform reasonably well across many domains.\nThey shine when:\nTasks are open-ended Requirements are still evolving You need strong zero-shot or few-shot behavior This is why they’re often the first models, teams reach for. They feel powerful, flexible, and surprisingly capable.\nThe tradeoff is variability. These models are harder to constrain, more expensive to run, and more likely to produce outputs that look correct while subtly missing context. As systems mature, that variability becomes a liability.\nGeneral-purpose models are excellent for exploration. They are not always the best choice for stable production systems.\nDomain-Specific Models: Precision Over Reach Domain-specific models narrow the problem space deliberately. They are trained or fine-tuned on data that reflects a particular domain, task, or style of output.\nWhen aligned well, they offer:\nHigher precision Lower inference cost More predictable behavior Lower latency This predictability matters. In production systems, consistency is often more valuable than peak capability.\nThe downside is obvious: these models are less flexible. They don’t generalize as broadly, and they require domain data and maintenance. But for many real systems, that is an acceptable — even desirable — tradeoff.\nEncoder vs. Decoder Models in Practice One source of confusion in model selection is architectural class.\nEncoder-based models are optimized for understanding: classification, search, clustering. Decoder-based models are optimized for generation: text, code, structured output. Using a generative model for pure understanding tasks is often overkill. It increases cost and complexity without improving outcomes. Conversely, forcing encoder models into generative workflows leads to brittle systems and awkward pipelines.\nGood architecture starts with matching the model class to the job.\nConcrete LLM families BERT [Encoders] (2018) BERT is a bidirectional encoder representation from transformers. It is a natural language processing model that was introduced in 2018 by Google AI. Its bidirectional nature helps consider the context of a word by looking at the words that come before and after it. This is in contrast to unidirectional models, which can only consider the words that come before a given word. BERT is also a transformer model, which means that it uses attention to learn the relationships between words.\nBERT was pre-trained on a massive dataset of text and code. It was then fine-tuned on a variety of natural language processing tasks, including question answering, natural language inference, and sentiment analysis. BERT has achieved state-of-the-art results on many of these tasks.\nBERT is a powerful language processing model that has been used in a variety of applications including but not limited to Google Search (to improve the accuracy of search results), Google Translate (to improve the accuracy of translations), chatbots, machine translation, and text summarization.\nBERT was open sourced in 2018 and led to many flavors including RoBERTa (Facebook), DistilBERT (Hugging Face), TinyBERT, etc.\nGPT-J [Decoders] (2021) GPT-J is an open source LLM developed by EleutherAI. It is a 175 billion parameter model, making it one of the largest language models ever created. Trained massive dataset of text and code, it can be used for a variety of tasks, including text generation, translation, and question answering. In a paper published in 2021, researchers showed that GPT-J was able to achieve state-of-the-art results on a number of natural language processing tasks, including question answering (on diverse topics and can be used for answering homework questions, customer service, etc), generating texts (poems, code, scripts, musical pieces, email, letters, etc.), summarization, and translation.\nFLAN Unified Language Learning (UL2) [Encoder-Decoder] (2022) Flan Collection is a newer and more extensive publicly available collection of tasks, templates, and methods for instruction tuning to advance the community’s ability to analyze and improve instruction-tuning methods. This collection was first used in Flan-T5 and Flan-PaLM, for which the latter achieved significant improvements over PaLM.\nUnified Language Learner (UL2) is a framework improves the performance of language models universally across datasets and setups. UL2 frames different objective functions for training language models as denoising tasks, where the model has to recover missing sub-sequences of a given input. During pre-training it uses a novel mixture-of-denoisers that samples from a varied set of such objectives, each with different configurations. This was open sourced in Q2 2022.\n​ Combining both methods, Flan UL2 (20B checkpoint - A snapshot in time where the model converged) makes it more usable for few-shot in-context learning and removes the need for mode token switching. It outperforms several LLMs with large parameters (PALM ~540 B). Its based on T5 architecture and was open sourced in Feb 2023. More details available in Flan References.\nAlpaca [Decoders] (2023) Alpaca is fine-tuned from Meta’s LLaMA 7B model. It was created by the research community and Stanford who were unable to study the closed text-davinci-003 model from Open AI. Alpaca shows many behaviors similar to OpenAI’s text-davinci-003, but is also surprisingly small and easy/cheap to reproduce.\nThe model was released with training recipe, data, and model weights. Alpaca is intended only for academic research and any commercial use is prohibited inheriting its licenses from Llama and Open AI instruction data.\nWhy Benchmarks Rarely Settle the Question Benchmarks are useful, but incomplete.\nThey measure performance on static datasets under controlled conditions. Production systems care about different questions:\nHow does the model fail? How predictable are those failures? How expensive is failure at scale? How easily can we observe and correct behavior? These questions don’t show up on leaderboards, but they dominate real-world outcomes.\nMy recommendation is to build your own benchmark for your production system and evaluate models against these instead of relying on public benchmarks.\nLooking Ahead Once models meet real users, limitations surface quickly. In the next post, we’ll look at the failure modes engineers consistently encounter in production, and why many of them are structural rather than prompt-related.\n","permalink":"https://blog.girishshekhar.com/posts/llms-and-how-our-work-is-changing/3-the-llm-landscape/","summary":"Once you understand how LLMs work, the next real question is model choice. This post explains how to reason about LLM tradeoffs — capability, cost, controllability, and domain fit — without relying on benchmarks alone.","title":"The LLM Landscape: Models, Capabilities, and Tradeoffs"},{"content":"Purpose In Part 1, we focused on what changed and why Large Language Models (LLMs) feel fundamentally different from earlier machine learning systems.\nThis post goes one level deeper.\nThe goal here is not to turn you into an ML researcher. Instead, this post explains just enough of how LLMs work so you can reason about:\nWhy models behave the way they do Why prompts sometimes fail unexpectedly Why cost, latency, and accuracy trade off against each other Why certain architectural patterns keep showing up Reader note:\nThis is a depth article. If you’re a product manager or engineering leader, feel free to skim the deeper sections and focus on the summaries and callouts. The structure is intentionally designed to support that.\nTokens, Revisited (With Practical Implications) In Part 1, we introduced tokens as the basic unit models operate on. Let’s make that more concrete.\nWhat Is a Tokenizer? A tokenizer converts raw text into tokens — numeric IDs that the model can work with.\nINPUT TEXT: \u0026#34;The stratosphere\u0026#34; | V +-----------+ +-------+-------+----------+ | TOKENIZER | ----\u0026gt; | The | stra | tosphere | (Tokens) +-----------+ +-------+-------+----------+ | | | | V V V V OUTPUT IDs: [ 464, 8134, 15234 ] Unlike humans, models don’t see words. They see:\nCommon words as single tokens Rare words split into multiple tokens Punctuation and symbols as tokens This process is learned from data and optimized for compression and coverage, not readability.\nWhy Tokenization Matters in Practice Tokenization directly affects:\nCost: pricing is usually per token Latency: more tokens = more computation Context limits: long inputs can silently overflow Prompt reliability: small phrasing changes can alter token boundaries Example:\nTwo prompts that look similar to a human may tokenize very differently — leading to different model behavior.\nThis is one reason prompt engineering often feels unintuitive at first.\nHow This Differs from Classic Text Models Older text-based ML systems typically relied on:\nBag-of-words N-grams Manually engineered features Those approaches required humans to decide what mattered. Tokenization plus large-scale learning shifts that burden to the model — at the cost of opacity.\nTransformers: The Architecture That Made LLMs Practical Most modern LLMs are built on transformer architectures. You don’t need to understand the math to understand why they matter.\nThe Core Idea: Attention At a high level, transformers use attention to decide:\nWhich parts of the input matter most How strongly different tokens relate to each other Instead of processing text one word at a time, transformers look at entire sequences at once and assign importance dynamically.\nMental model:\nAttention lets the model ask, “Which earlier parts of this text should I focus on right now?”\nWhy Transformers Scaled When Others Didn’t Before transformers, language models often used:\nRecurrent Neural Networks (RNNs) — processed text sequentially RNN Step 1: [The] -\u0026gt; (Memory A) | Step 2: [cat] + (Memory A) -\u0026gt; (Memory B) | Step 3: [sat] + (Memory B) -\u0026gt; (Memory C) Convolutional Neural Networks (CNNs) — While they are primarily used for image tasks, they use a \u0026ldquo;sliding window\u0026rdquo; (kernel) to look at 2, 3, or 5 words at a time to find local patterns (like phrases). CNN Sentence: [ The ] [ cat ] [ sat ] [ silently ] [ in ] \\_______|_______/ | | Filter 1 (sees \u0026#34;The cat sat\u0026#34;) | \\___________|____________/ Filter 2 (sees \u0026#34;cat sat silently\u0026#34;) These approaches worked but struggled with:\nLong-range context Parallelization Training efficiency at scale (required supervision) Transformers removed these bottlenecks, which is why LLMs became feasible only in the last few years.\nTransformers --------------------------------------------------------- | The [cat] sat silently in the corner of the room. | | ^ | | | | | | (Strong Attention Weight) | | | | | | [Many sentences later...] | | | | | | | | ------------------------------- | | | | | Even though the lights were off, [it] still saw. | --------------------------------------------------------- | [Self-Attention Mechanism] +---\u0026gt; \u0026#34;It\u0026#34; = \u0026#34;Cat\u0026#34; Feature RNN (The Conveyor) CNN (The Scanner) Transformers (The Crowd) Speed Slow (One by one) Fast (Parallel chunks) Fastest (Entire text at once) Memory Forgets long distance Only sees local \u0026ldquo;\u0026ldquo;windows\u0026rdquo;\u0026rdquo; Infinite (Relates every word) Best For Short, fluid sequences Detecting specific phrases Everything (LLMs) Encoder, Decoder, and Encoder–Decoder Models (With Use Cases) Transformers show up in different configurations depending on the task.\nEncoder Models: Understanding Encoder models read text and produce representations that capture meaning.\nCommon uses:\nSearch and retrieval Semantic similarity Classification Embeddings for analytics, clustering, and RAG systems If your goal is understanding text rather than generating new text, encoders are often cheaper and more predictable.\nDecoder Models: Generation Decoder models generate text one token at a time, using previous tokens as context.\nThis pattern powers:\nChat interfaces Code generation Autocomplete Content creation Most general-purpose LLMs in production today are decoder-only models.\nEncoder–Decoder Models: Transformation Encoder–decoder models first encode an input, then generate an output conditioned on it.\nThey are well-suited for:\nTranslation Summarization Structured text transformation Why this matters:\nUnderstanding these patterns helps avoid overusing large, general-purpose models when simpler architectures are sufficient.\nHow LLMs Are Trained (Narrative View) Training LLMs typically happens in stages. The details vary by model, but the high-level flow is consistent.\n1. Pre-Training: Learning Language In pre-training, models learn general language structure by predicting missing or next tokens across massive datasets.\nThis phase:\nUses self-supervised learning Requires no manual labeling Produces a foundation model Most general capability comes from this stage.\n2. Fine-Tuning: Shaping Behavior Fine-tuning adapts a foundation model for specific tasks or domains.\nThis may involve:\nTask-specific examples Domain-relevant data Partial or full parameter updates Fine-tuning improves usefulness but increases cost and specialization.\n3. Reinforcement Learning from Human Feedback (RLHF) RLHF aligns models with human preferences.\nAt a high level:\nThe model generates multiple responses Humans rank or select preferred outputs A reward model is trained Reinforcement learning optimizes the model against that reward This stage is a major reason modern LLMs feel usable in real products.\nWhy These Details Matter (Even If You Skipped Them) You don’t need to remember every concept here. But understanding that they exist helps explain:\nWhy prompts behave probabilistically Why long context windows are expensive Why models sometimes sound confident but are wrong Why architectural patterns like RAG keep emerging These mechanics directly shape system design decisions.\nWhat’s Next Now that we have a working mental model, the next post will zoom out and look at the model landscape:\nGeneral-purpose vs domain-specific models Capability vs cost tradeoffs Why model choice is an architectural decision, not just a vendor decision That’s where theory starts turning into concrete choices.\n","permalink":"https://blog.girishshekhar.com/posts/llms-and-how-our-work-is-changing/2-how-llms-work/","summary":"A deeper look at how Large Language Models actually work — focusing on tokens, transformers, and training stages — explained in a way that helps engineers, product managers, and technical leaders reason about behavior and tradeoffs.","title":"How LLMs Work (Only What You Need to Know)"},{"content":"Purpose Over the last few years, generative AI — and Large Language Models (LLMs) in particular — have started to show up everywhere: in developer tools, customer support, analytics workflows, and product experiences.\nThis isn’t just another machine learning trend.\nLLMs change how we think about software design. Instead of hard-coding rules or building narrowly scoped models, we now interact with systems that can reason over language, code, and data in a much more general way.\nThis post lays the foundation. We’ll keep the concepts intentionally high-level and accessible, so engineers, product managers, and data practitioners all start from the same mental model.\nWhat Is a Language Model? At its simplest, a language model predicts what comes next.\nGiven some text, the model estimates the probability of the next piece of text. There is no rules engine or symbolic reasoning system underneath. Everything the model produces comes from repeatedly answering one question:\n“Given everything I’ve seen so far, what is most likely to come next?”\nWhen trained at sufficient scale, this surprisingly simple objective leads to behavior that feels intelligent.\nTokens: How Models Actually See Text One detail that’s easy to miss: models don’t operate on words.\nBefore text reaches a model, it is broken into tokens. A token might be:\nA whole word Part of a word Punctuation or symbols For example, the word “unbelievable” may be split into multiple tokens depending on the tokenizer.\nSentence: \u0026#34;Subterranean depths are cool !\u0026#34; WORDS: [Subterranean] [depths] [are] [cool] [!] | | | | | TOKENS: [Sub] [terr] [anean] [depths] [are] [cool] [!] Why this matters:\nTokenization affects cost, latency, and accuracy It explains why long or oddly formatted inputs behave differently It’s a key difference from older text models that relied on hand-crafted features Mental model:\nIf words are how humans read text, tokens are how machines read text.\nWe’ll revisit tokenization in more detail later in the series.\nHow LLMs Differ from Classic Machine Learning Models If you’ve worked with machine learning before, LLMs may feel unfamiliar. That’s because they differ in a few fundamental ways.\n1. The Data Is Broad, Not Narrow Traditional ML models are usually trained on task-specific datasets:\nFraud detection Recommendations Classification Forecasting LLMs are trained on large, diverse collections of text, including:\nDocumentation Articles and books Code Forums and Q\u0026amp;A Structured and semi-structured text This breadth allows LLMs to generalize across tasks instead of being good at just one thing.\nFeature Classic Machine Learning Large Language Models (LLMs) Data Source Specialized (Medical logs; Stock prices) General (Common Crawl; Books; GitHub) Size Megabytes to Gigabytes Terabytes to Petabytes Variety Single domain / Structured Multi-domain / Unstructured Labeling Human-labeled (Supervised) Self-supervised (Predict next word) Classic ML (Laser Focus) LLM (The Library of Babel) [ Narrow ] [ Broad ] [ Curated] [ Mixed ] [ Data ] [ Corpus ] | | v v Specific Tasks General Reasoning (e.g. Fraud Detection) (e.g. Writing Poetry) 2. Learning Is Self-Supervised A key enabler of LLMs is self-supervised learning.\nInstead of relying on large, manually labeled datasets, LLMs learn by using the data itself as the training signal. The model is shown text and trained to predict missing or next tokens — effectively generating its own labels.\nWhy this matters:\nIt scales to massive datasets It removes the bottleneck of human annotation It allows models to learn general language structure before being adapted to specific tasks We’ll dive deeper into how self-supervision works in a later post, but it’s one of the main reasons LLMs became feasible at this scale.\n3. Features Are Learned, Not Engineered Classic models often depend on humans to decide:\nWhat features matter How text should be represented LLMs learn internal representations directly from data. As models grow larger, these representations become reusable across many tasks.\nWhy this matters:\nThis is why a single LLM can summarize text, explain code, and generate SQL — without being explicitly trained for each task.\n4. Scale Unlocks New Behavior As models grow in size and see more data, new capabilities appear:\nFollowing instructions from examples Adapting behavior based on prompts Handling tasks they were never directly trained on These capabilities are often described as emergent behavior and are one of the defining characteristics of modern LLMs.\nA Simple Tour of Model Architectures Most modern LLMs are built using a family of models called transformers. You don’t need to understand the math to understand the roles these models play.\nAt a high level, there are three common patterns.\nEncoder Models: Understanding Text Encoder models focus on understanding input.\nThey take text in and produce representations that capture meaning. These models are commonly used for:\nSearch and retrieval Semantic similarity Classification Embeddings for analytics or clustering Decoder Models: Generating Text Decoder models focus on generation.\nThey produce text one token at a time, using previously generated tokens as context. This is the pattern behind:\nChatbots Code generation Autocomplete systems Most general-purpose LLMs used today fall into this category.\nArchitecture How it works Primary Use Cases Encoder-Only Reads the whole input at once to understand context. Sentiment analysis, Named Entity Recognition (NER), Classification. Decoder-Only Predicts the next token one by one (auto-regressive). Text generation (ChatGPT, Claude), Creative writing, Coding help. Encoder-Decoder Maps an input sequence to a specific output sequence. Translation (English -\u0026gt; French), Summarization, Paraphrasing. ENCODER DECODER ENCODER-DECODER [Input] [Prompt] [Input] -\u0026gt; [Hidden State] | | | [Understanding] [Generation] [Translation] | | | [Label/Class] [Response] [Output] Encoder–Decoder Models: Transforming Text Encoder–decoder models combine both ideas and is used with the input representation is different than what we want to produce on the output.\nThey:\nEncode an input Generate an output conditioned on that input These models are commonly used for:\nTranslation Summarization Structured text transformation Why this distinction matters:\nUnderstanding these patterns helps you choose the right tool — and avoid using a general-purpose LLM when a simpler, cheaper model would suffice.\nWhat About Older Approaches Like RNNs and CNNs? Before transformers, language models were often built using:\nRecurrent Neural Networks (RNNs), which process text sequentially Convolutional Neural Networks (CNNs), adapted from image processing These approaches worked but struggled with:\nLong-range context Parallelization Scaling to very large datasets Transformers addressed these limitations by allowing models to look at entire sequences at once, which is a major reason LLMs scale as well as they do today.\nWhy This Changes How We Build Software LLMs blur boundaries that used to be clear:\nLogic vs data Code vs configuration Input vs interface Instead of encoding every rule, we increasingly design prompts, context, and feedback loops. Instead of rigid APIs, we design systems that can adapt based on examples.\nThat shift has real architectural consequences — and that’s what the rest of this series will explore.\nWhat’s Next In the next post, we’ll look more closely at how text is represented inside models:\nTokenization Text encoding Why these choices matter in real systems ","permalink":"https://blog.girishshekhar.com/posts/llms-and-how-our-work-is-changing/1-why-llms-changed-the-game/","summary":"Large Language Models feel different from earlier machine learning systems. This post explains why — focusing on tokens, training data, self-supervision, and the architectural patterns that changed how we build software.","title":"Why Large Language Models Changed the Game"},{"content":"Over the years, I’ve found myself being asked the same kinds of questions again and again:\nCan we clone Girish? How do you scale beyond 1:1 mentoring? How do you scale beyond group mentoring? How do you spread your mental models so decisions can be made in your absence? How do you write more consistently? Individually, these questions are reasonable. Collectively, they point to a deeper issue:\nImpact doesn’t scale if it only exists in conversations.\nAt first, I treated these as one-off conversations — hallway chats, coffee catch-ups, Slack threads, mentoring sessions. Over time, it became clear that this approach doesn’t scale—neither for me nor for the people I’m trying to help.\nWhat became increasingly clear was this:\nthe bottleneck wasn’t intent or effort — it was reach.\nWriting is a better multiplier.\nWhen ideas are written down, they:\npersist, can be referenced, can be challenged, and can be reused without my presence, scale thinking, preserve context, create durable artifacts others can build on Discussing with my manager, I decided to start writing more deliberately—not as an obligation, but as a way to scale learning and thinking beyond the room I’m in.\nThe Goal This blog is an attempt to externalize how I think.\nI want to:\nShare mental models I use when making engineering and leadership decisions Write about system design, trade-offs, and architecture at scale Capture lessons learned the hard way so others don’t have to Create a space where ideas can evolve over time instead of disappearing into chat history This isn’t meant to be a polished authority blog. It’s a working notebook—public, imperfect, and iterative.\nIf even one post helps someone make a better decision when I’m not around, it’s worth it.\nWhat to Expect You can expect writing around:\nDistributed systems and scalability Engineering leadership and decision-making Design trade-offs in real systems Career growth and mentorship at scale And occasionally, thoughts that are still forming.\n","permalink":"https://blog.girishshekhar.com/posts/why-now/","summary":"Why I decided to start writing — and why now felt like the right moment.","title":"Why Now?"}]