Caching or CDN does not help in squeezing out JS performance (actual JS execution and not n/w performance)

You think that a page that loads jQuery but doesn't use most of it incurs a performance penalty? I'll grant there are a few initializations and dom checks that jQuery does, but it probably blocks for all of 20ms.

...which is significant.

20ms is significant?

Think of it as a fifth of your time budget spent doing nothing.

Most modern browsers are able to paint while loading script files, especially if you put your script files at the bottom of the page, where they belong. So, no, 20ms of parsing, compiling, and a small amount of execution is not a lot of time. Period.

caching won't help for libraries that want to have a particular version of jquery baked in ...