Training is an overkill at this point imo. I have seen agents work quite well with a feedback loop, some tools and prompt optimisation. Are you doing fine-tuning on the models when you say training?

Nope - just use memory layer with model routing system.

https://github.com/rush86999/atom/blob/main/docs/EPISODIC_ME...

Memory is usually slow and haven't seen many voice agents atleast leverage it. Are you building in text modality or audio as well?