The GPU can matter as well. On the Nexus 10, for example, the fill rate is pretty low compared to the screen size, so applications have to be very very careful not to exceed the limit or they'll end up adding frames of latency (which looks like touch or interaction latency).

But in general you're right, those are the components that contribute the large majority touch latency to a device.