> Modularity, indirection and abstraction are not panaceas for this problem. In most cases they reduce local complexity at the cost of global complexity. This is a decision that should be consciously weighed in each case rather than assumed to be an unquestionable win.

This is a very important point, and one that a lot of developers miss, or even believe the opposite of. It's common to approach problems by 1. identifying the abstractions that you'd want to have to solve the problem, 2. implement those abstractions, and 3. use the abstractions to solve the problem.

This is how (by and large) we're taught to program, and it basically attacks the problem in a backwards way. Start specific, implement the whole thing with fairly minimal abstraction, then if the local complexity warrants abstraction, move to something more abstract.

Why? So that you can put off "the point past which you can no longer collectively understand the system". Local complexity is better than global complexity, and defaulting to a globally complex system is a terrible choice.

> 1. identifying the abstractions that you'd want to have to solve the problem, 2. implement those abstractions, and 3. use the abstractions to solve the problem.

Also the reason to do it bottom-up is because you never understand the problem perfectly, especially if it has anything to do with real-world things. This means that whatever global domain model you created is wrong, and by the time you discover where exactly, you'll be so invested that it's too late to change it. It's easier to adjust the model when going bottom-up - you can change or discard just the top layers, as soon as you realize your new abstraction level doesn't match the problem domain properly.

I watched the Structure and Interpretation of Computer Programs lecture recordings, and one lesson I got out of them very strongly: create a toolkit to solve problems in the space of your initial conception of the problem. Your initial instinct of what the problem is, is typically wrong, and the demands on your program typically evolve anyway.

I hesitate to use the phrase "domain specific language" because it brings baggage, but that's the platonic ideal of a toolkit.

The core skill of building toolkits is picking flexible ways to decompose problems. Top-down planning and decomposition has a way of biting you in the ass at the worst times, when you're almost done except for one corner case. Bottom-up construction tends to be more flexible if dirtier.

I actually liked the way they referred to those toolkits as "languages". Maybe let's not call them Domain Specific Languages because of the baggage, but it's a good way of thinking to see layers of abstraction as defining programming languages for layers above them.

Familiarity with Lisp helps understand this concept, because creating a new "DSL" there is so easy that you treat it just like creating a new class or interface, and not as some kind of great endeavour that most people mean when thinking about DSLs today.

I like Casey Muratori's idea of removing the word 'abstraction' from your vocabulary in favor of the word 'compression'. You start by writing code absolutely directly and then if the same patterns keep cropping up, you compress them into a named pattern. It then becomes clearly a bad idea to try to 'compress' something that you only use once in an attempt to make it simpler.

http://mollyrocket.com/casey/stream_0019.html

As a sort-of counterexample, it can still be useful to put code that you only use once in its own function: functions have very definite entry and exit points, and a defined interface for communicating with the rest of the code (via arguments, captured variables and return values).

There are several tradeoffs here. Yes, this has a readability benefit, and helps ease developers into the module, especially if they're not familiar with it to start with.

A good argument in favor of not breaking them into separate functions is available here[0], basically boiling down to the fact that splitting something into a separate function has the undesirable side effect of hiding the full scope (not in the programming sense) of what it does.

If this function is pure, this isn't a problem, but if it manipulates -- or even just accesses -- global state, you might call it when it's in a state that it doesn't expect, leading to subtle bugs. You also might move the call to it to a new location, not realizing the full scope of what it does (this is doubly true for removing calls).

[0] https://web.archive.org/web/20150926045718/http://number-non... (sad to see this site slip away ...)

I agree, all bets are off when manipulating global state. (Unless your type system tracks such manipulations in detail.)

The analogy to math always lingers in my mind: before we had efficient, well-understood formulas, we would always rely on lookup tables to solve real problems. True of construction in ancient Egypt - also true of many programming problems today.

I got into an argument the other day about the idea of using fewer names for things in code - I wasn't doing a stellar job of articulating the point since the obvious rebuttal is "if you aren't using a name what are you going to use instead?"

But it happens so often that what you are - or should be - implementing is not a business rule, but supporting computation - and in the latter, naming is relatively unimportant. And when you do finally reach business logic, it turns out to flow more smoothly if you allow a procedure to be a complex "train station" or "Manhattan Island" and don't carve it up unnecessarily, because more is done with fewer variables or more obviously short-lived and temporary variables. Fewer things get passed around, thus fewer things need names. And the code is generously imperative but can be understood because a top-to-bottom reading is possible.

Using more table-driven code just happens to be one particularly effective way of streamlining logic and named fields out - it's more effective than most uses of OO abstraction because it is a concrete specification of possibilities, yet it doesn't preclude extension in any dimension.