That's what I thought too, that there is just a byte stream being tunneled between SSH server and client and not a TCP packet stream.

Btw I use unix sockets on client and server to avoid the local connection TCP overhead.