In the ideal case the IPC mechanism is just a block of memory shared between processes running simultaneously on separate cores, so there is no need to context-switch. You still want some sort of kernel-based blocking semaphore to avoid busy-waiting when the IPC queue is empty, but since it's only used when the queue is empty it isn't part of the performance-critical path.
And replacing it with expensive userspace<--->userspace switching.
But because the kernel is still doing the context switch between the processes, now you're doing userspace<-->kernel<--->userspace switching.