"Framework" for some has the connotation of being dogmatically prescriptive: a monolithic culture of libraries where you must buy into the whole system to use any of it. For example, a classical C++ framework might come with its own string class, its own container classes, its own smart pointers, its own way of binding to other languages, etc. Escaping the cultural ecosystem to integrate something external can be quite a burden.

A middle ground is where you try to factor out useful parts but do so in a way that keeps in mind that users might not want to also use associated packages X, Y, and Z. This means backing off a bit from the maximally elegant integration you might do when you have a giant framework ecosystem and imagine people wanting to use all of it in its glory.

Doing this requires resisting the inclination to build a code empire -- a megalomania that anyone ambitious among us has felt as your excitement at being able to build more complex things grows. You only learn the opposite instinct after being burned suitably by frameworks, or looking upon your own empire and despairing.