Feature flags need to be treated as short-lived and experimental otherwise they end up getting abused for everything and make it very difficult to reason about your application.
If it's config/customization, it should be in code. If it's experimental it can be a flag until it solidifies, and then it needs to get moved to code.
When I was at Shopify a couple of years ago they mandated that feature flags had to be short-lived (Like 2-4w lifetime tops, some had exceptions) because they would end up getting left in code and never cleaned up, or for extended periods of time like months. Hard to tell if it's genuinely a "feature flag" or actually just a normal part of the system at that point.
Feature flags being flipped in prod was also a major source of incidents, in part because people didn't treat them as experimental and with the associated risk profile of something experimental.
The only exception where having long-lived flags was useful and required was for operational killswitches (E.g. disable Apple Pay because it's having issues), but that is explicitly not application config.
This is the kind of design wisdom that’s both true and difficult to win an argument over.
It reminds me of arguments related to over-engineering and complexity. The principles are super important to having a codebase that scales and continues to be efficient to work in as the team grows, but they are hard to objectively measure.
Locally or in isolation something may sound like a great idea. Being able to step back and see the greater ripple effects require some experience and intuition that can’t always be used to convince people otherwise.
I disagree with just about everything you said being a problem except the process of cleaning up is absolutely required.
Notably feature flags triggering incidents is expected and desired vs the alternative of shipping the code and having to roll a release back because there is no other way to remove the feature from prod.
In a company the size of Shopify people flipping their feature flags would very often impact *other teams*, and like I said feature flags got abused with even seemingly innocuous changes being put behind them or being left long periods of time before being fully used.
When someone else flips a flag that impacts your team and they have no idea they even caused a problem, it becomes very difficult to resolve the issue. Usually you can check for recent deploys, instead you have to go and guess at which feature flag which was recently flipped could possibly be affecting your code. I experienced this several times.
Also, it was actually more desirable for most of these things to go straight to production. Test it properly before shipping, then when you ship it soaks on a 5% traffic canary at which point you can monitor and cancel the deploy if you see errors. That is generally safer than a feature flag rollout unless you are doing something very high impact/risk, in large part because it gives any other team affected by your rollout the ability to respond and be able to easily find the source of errors.
In my org it was a fairly common failure mode to ship something and accidentally cause an issue for another team. Usually it was other teams/orgs shipping things that impacted us.
Runtime evaluated feature flags can always be used for control plane levers and emergency handbrakes.
You just have to label them as such and prevent other teams from fiddling with them.
This is not an antipattern, it's just semantic hand-wringing.
My team managed critical systems in the online flow of billions of dollars of daily payment volume. We also wrote the feature flag system that the rest of the company used. Not only were we completely fine with feature flags as long-lived control plane levers, we heavily used the system that way ourselves.
You just have to clearly distinguish between ephemeral rollout flags (and clean them up or expire them) and the permanent control plane levers.
It's the exact same functionality for both sets of tools. Just different practices around the two usages.
I completely agree with your distinction and that is exactly what they mandated :)
I don't think that is what most people colloquially mean by "feature flags" though. Even most teams in Shopify abused "ephemeral" flags for long periods of time.
When they rolled out the mandate it was very annoying for my team because we had a lot of operational flags like you're describing that we needed to get exemptions for.
The fact you are trying to use Copilot as an example here shows you don't understand how Copilot's previous billing worked.
Previously they used "premium requests" which would allow you to make a request to one of the more expensive models. People abused the shit out of this because a request was disconnected from tokens.
You could make one request which used tens of dollars worth of tokens, obviously not the intended usage pattern and obviously unsustainable.
Tokens for a given intelligence level are becoming much cheaper very quickly, but everyone wants to use the smartest frontier models so tokens are not dirt cheap. Even frontier models are a bit cheaper in absolute terms than they previously were, and much cheaper in terms of intelligence.
> shows you don't understand how Copilot's previous billing worked
Having used it for > 4 years and having paid for it for > 2.5 years, I think I know full well how it's previous billing worked.
> You could make one request which used tens of dollars worth of tokens, obviously not the intended usage pattern and obviously unsustainable.
Gee, thanks Mr. Obvious! It never occurred to me this was the reason Microsoft recently removed Opus 4.6 and added a 15x multiplier in front of the inferior, but less token-intensive Opus 4.7!
No other provider works like Copilot did with "premium requests". Usage limits (Codex/Claude Code), which are inherently linked to tokens, are the most common. Some providers like Amp charge you per-token like Copilot is moving to.
Microsoft's previous model was not linked to tokens at all. Complete anomaly among coding agent providers. It's not representative of token economics at large. Claude Code recently announced increased limits. Codex does regular limit refreshes.
Tokens are pretty damn abundant even though they're not bargain basement cheap yet.
Blizzard still brings World of Warcraft down every Tuesday for maintenance. It's down right now to apply a new content patch, which they estimated would take 8 hours.
People like to make this point, but traditional engineering has the opposite problem: insanely overwrought processes and box-checking that exists for no reason and slows everything down to a snail's pace. Yes there are safety-critical parts, but they surrounded by a ton of bullshit.
It's also absurd to think that there is no company which does genuine software "engineering". If you break ads at Google/Meta, streaming at Netflix, etc there are massive consequences. They are heavily incentivized to properly engineer their systems.
The main thing that governs whether time is spent to well-engineer something is if there is incentive to do it. In traditional engineering that incentive is the law (Getting council approval, not getting sued, etc). In software engineering that incentive is revenue.
That's quite the take. Throughout human history there were lots of instances of vibe-engineering and vibe-architecting, in the physical world.
Since the failings of not doing proper engineering is far more evident, the reasons for the "insanely overwrought processes and box-checking that exists for no reason and slows everything down to a snail's pace" go back to the earliest written law, AKA the Code of Hammurabi, circa 1754 BC! These rules are part of the core of our functional civilization.
Examples:
- Law 229 (Death of Owner): If a house collapses and kills the owner, the builder is put to death.
- Law 230 (Death of Owner’s Son): If the collapse kills the owner's son, the builder's son is put to death.
- Law 232 (Property Damage): The builder must replace any destroyed property and rebuild the collapsed house at their own expense.
- Law 233 (Structural Defects): If a wall "shifts" or is not built properly before completion, the builder must strengthen or repair it using their own silver/means.
No point in discussing with someone who is arguing in bad faith. I already agreed that some parts of the engineering process are safety critical. If you think there is no bullshit in the process you don't have enough knowledge about the requirements imposed by e.g. building regulations.
Material is not just about quality, but rarity or uniqueness. For example, japanese denim can get very expensive in part because it's very low volume. For dress pants, it might be a particularly interesting fabric.
A lot of more expensive pants also have interesting designs or proportions that are very unique or hard to find elsewhere. There is a lot of cool stuff you can get for under $500 USD though, that is still pretty expensive.
Because it's not his job. He should elevate someone else into that IC role instead of holding it for himself. The way he describes it, there is no one else in the company who can do the IC work he is doing, which is long-term bad.
Coding IC work takes a lot of focus and context that someone who is operating at the company-level should not really be in sole possession of.
To me, the whole point of these positions is to take the hit on random bullshit, planning, people management, etc and give your ICs space to do the kind of work he is taking on.
That doesn't mean you have no technical context or involvement in the development process, but it does mean you should probably be at least one step removed from it.
I wouldn't really call it "demand". It's more like one-shotting humans with a product which maximally stimulates them through what is basically a psychological hack.
We were not built with the capacity to handle the sheer amount of stimulation the modern world has. You have to put in a lot of effort to not succumb to natural desires that would have been adaptive behaviours until recent history.
Succumbing to constant distraction, even if a natural desire, would never have been a successful evolutionary strategy for an individual organism. Spending large amounts of time absorbing and repeating bullshit has proven to be a pretty successful group survival strategy throughout human history, though.
Lets call it a next great man-made filter. Weak personalities will take a hit and have a lesser life compared to their potential, the ones more mentally resilient or with good parents (or both) gain a clear advantage in basically all aspects of life. Waiting around for state regulations to cover our asses has always been a bad move, and its same now. They will come but too little too late, one has to fight for oneself and closest ones in true capitalist spirit, and this is indeed distilled capitalism at work. Its jungle out there, and servants of the biggest predators form like 50% of this very forum (go ahead and downvote some meaningless number in DB, but take a good look in the mirror and ask yourself how good human being you truly are).
I can't bring myself to feel much sympathy for the ones that fully realize this, and yet go full speed to their addictions, even push it to their kids since good parenting always take a lot more continuous effort. We keep discussing this mind cancer for a decade here, its not something shocking on any level for anybody who gives a fraction of a f*k about their quality of life or mental health. The rest has bread and games for the poor, version 2025.
If it's config/customization, it should be in code. If it's experimental it can be a flag until it solidifies, and then it needs to get moved to code.
When I was at Shopify a couple of years ago they mandated that feature flags had to be short-lived (Like 2-4w lifetime tops, some had exceptions) because they would end up getting left in code and never cleaned up, or for extended periods of time like months. Hard to tell if it's genuinely a "feature flag" or actually just a normal part of the system at that point.
Feature flags being flipped in prod was also a major source of incidents, in part because people didn't treat them as experimental and with the associated risk profile of something experimental.
The only exception where having long-lived flags was useful and required was for operational killswitches (E.g. disable Apple Pay because it's having issues), but that is explicitly not application config.
reply