Enduring Systems

Working across several companies of varying sizes, goals, and scopes, I’ve had to contend with a lot of what proponents call “canonical procedures” or “standards” and opponents call “[bureaucratic] red tape.”

I’m generally a fan of documented procedures. If there’s a consistent, reliable way to do something, chances are someone is going to need guidance to get through the process the first time around. This leads to two requirements:

Your solution must function in the most naive, even adversarial state.
Your solution must be able to adapt without violating its foundational guarantees.

This is very tricky! And it’s easy to “get wrong” the first time. The key is to have a foundational set of axioms that strongly promote collaborative problem-solving as the “only” mechanism for building reliable changes in the system.

This may sound like an overly simplistic solution but the mechanism has maximum leverage for building a coherent and enduring approach to work.

Rule of Law

In order to for the solution to succeed, you have to make the rules (and the outcomes from those rules) binding for everyone.

In many ways, this framework is very similar to “the rule of law” in that “no one is above the law” – an executive or director or manager is just as bound to the rules and their consequences as the individual contributor (IC). In particular, an executive cannot “just choose” to override the rules – they go through the same process as everyone else to address changes.

This may rankle those with positions of authority, but that’s kind of the point: this is a ulysses pact designed to ensure the entire organization/group aligns on a common means of action and enforcement. This pact ensures that the whole of the organization supports the rules, because they are the ones adapting and enforcing them. If the rules need to change, the whole of the organization will support that too (because it follows the rules they defined).

Violations of the rule

Okay, what if a VP or a Principal Engineer says “screw the rules, I make them” and acts on their own, outside the agreement?

There need to be two possible consequences:

The rules need to be modified to allow for that specific action (or a generalized version of it).
The act needs to be undone, the project replaced (with a conforming one), and/or the executor needs to be removed from the project / banned from operating in that environment again.

Now, that seems kind of extreme if an SVP says “we’re doing this” and the engineering organization collectively says “nuh uh,” but… that’s where this leads.

The engineering rules need to be flexible enough that an SVP can work within them to effect change while also ensuring that SVP can’t give directives that would ruin the cohesion of the team. This is the part where companies (in my experience) have allowed rule-violating outcomes to be “overidden” by a SVP/VP when they stake their name and reputation on it; if the SVP “owns” the outcome, then they can break the rules (at least temporarily) for the sake of their objectives.

Crisis Violations

We have addressed the deliberate violators of rules, but what happens when everything is broken?

When a company is in crisis, the only goal is “survival” through whatever situation caused the crisis. This does mean that, in those very specific cases, all of the rules can be broken.

What happens after the crisis is most important: a review of what went wrong, why it went wrong, and whether or not the rules need amending. Look at why the rules had to be broken in the first place; was it due to not having time to get approvals? Or was it due to restrictions being too broad/narrow in the first place?

These are the kinds of questions that allow for crises to be useful to evolving and conforming the rule-based practice of engineering.

Adversarial Evolution

Generally speaking, having to constantly justify oneself to another is seen as “defensive” and even a pathology if taken to an extreme. The difficulty is that corporations and working groups are not people; they are built of and from people.

As the working group (WG) comes up with rules and processes, no one person “owns” the rules. It is not the product of a singular individual or their ego; it has to be the product of everyone, and (as the saying goes) is the product of no one.

Justification

Per above, the company owns the documents, the code, the work product. Ownership being vested in the company means it’s not yours. The code, or the document, or the process can have your hand in it, can have your authorship and contributing tag, but ultimately it’s not something you-the-individual can have ownership over any more.

I repeat this point because we also need to turn our criticism toward the current process and documents; where do they have gaps? Where doesn’t it work? What do we do when Jimothy gets into the process and attempts to break things? A set of standards or guidelines or rules must constantly justify itself to its users, its operators.

If an organization has a regular practice of evaluating all rules, foundational assertions and axioms, and the consequences of those rules, then you have room to reasonably critique and/or reinforce them. If the assertions no longer hold, you have even more opportunity to change the rules to fit.

Each rule must be justifiable on its own, and in concert with the others. If it is, then individuals can use those same rules to justify their own positions; they become self-reinforcing the more they are used/accepted, to build better outcomes.

Ownership

As folks work on their particular deliverables, they will often have a sense of ownership about their work. Very rarely does anyone “just do the task” and not invest something of themselves in “doing good work.” (If someone doesn’t have any form of ownership or investment in their work, that’s a well-documented risk for burnout/departure/disengagement.)

Ownership in a moderate form is great! The individual or team will be self-motivated to Do A Good Job and produce quality output to the best of their ability. Having a feeilng of ownership can lead to strong and sometimes forceful advocacy for a particular decision or standard, and when folks care about something, they’re often invested in its success. This goes wrong when someone has “too much” ownership on the resulting artifact – just because you worked hard on a thing doesn’t mean it’s flawless. This applies to everything, including the durable processes and standards.

As part of the standards, one needs to lean towards global optimization, rather than local maxima. What is best for the group/company as a whole is often compromised compared to what can be done at the local level. “It takes ten minutes to ship code through the central pipeline, but I can make this change in 30 seconds” has led to more outages and divergent infrastructure decisions than the inverse. Getting advocates to disconnect from their position and view it from the frame of “is this best for everyone across the organization” is key.

By making decisions that promote the benefit of the whole organization, everyone can build better outcomes. If centralization is too much for an organization with a majority of individual ownership, consider a federated model to address local nuance while maintaining a central set of tenets and justification. (At the very least, federation can be a step towards full centralization.) Once an organization and its sub-teams can start taking collective ownership of outcomes, the rules become self-reinforcing down to the level of individuals.

Regular Reviews Start Here

Ensuring your rules are published, their reasoning documented (in text!), attacked and defended (without relying on Authority as a defense), and discoverable by the rest of the organization will ensure that you can build your team/division/org around a culture of open, unambiguous decisions in a self-reinforcing way.

While standards are often unique to the organization, starting from somewhere often makes for better ideas than a blank slate. To that end, I have a (small, limited) engineering standards repo available here. If you end up using it, drop me a note (per the Author page on this site).