If it feels alright when you don't have any plugin, then try to see which is / are causing the slow downs.
Vim shouldn't be slow without any plugin.