Unfortunately there's some RCE vulnerabilities and other nasty bugs between 18 and 23 that can be exploited remotely. I wouldn't feel safe running an unpatched version if it can be helped.
I don't advocate staying on an older version, but that depends on the services you have running. If you don't have any external service, you should be OK.
I tend to minimize internal services as well. I don't use OpenWRT as a general purpose server.
Still, an LTS release from time to time would be good (the effort would be the same, as there would still be only two maintained releases: the current stable and the LTS).
https://openwrt.org/docs/guide-developer/security