Might want to just try a high quality conference speakerphone. They have advanced audio processing, directional microphones, noise cancellation, etc. built in - since they are designed for this kind of audio from the ground up.
You can find the Konftel Ego for $80 at Provantage.com (around $120 everywhere else). Konftel makes amazing speakerphones that rival polycomm, for a lot less. Indeed, I actually prefer their sound to polycomm.
The Ego will be a dramatic upgrade from whatever is built into your other gear. If you still aren't happy with it, then you can start going down the USB/XLR microphone rathole if you still prefer. But it's probably unnecessary.
You can find the Konftel Ego for $80 at Provantage.com (around $120 everywhere else). Konftel makes amazing speakerphones that rival polycomm, for a lot less. Indeed, I actually prefer their sound to polycomm.
The Ego will be a dramatic upgrade from whatever is built into your other gear. If you still aren't happy with it, then you can start going down the USB/XLR microphone rathole if you still prefer. But it's probably unnecessary.