I have not a small number of plugins on fish and I've never noticed a problem with startup time. I do wish there was a better way of managing or installing plugins. It just feels so fragile.