That is not true, it is not the 'only way' to formally deal with them. A better way to think of them is as the vector space dual of functions (/forms) under the pairing given by integration. No measures required. The measure theoretic explanation is very much "fitting delta functions into our existing machinery" rather than any sort of inherent requirement.

Actually an even better way to think of delta functions is just as a geometric object, a point (or line/plane/etc). Which is somewhat related to the measure theoretic version, but much more simple to think about.