That threat model essentially prohibits "tmux attach", which allows an attacker running as your user to connect to your terminal session, so I don't think it's a particularly useful threat model here. That's basically exactly what we signed up for by using tmux.
Running on a server doesn't change anything, you'd need to be running on a server where you routinely give people who shouldn't have root access, access to an account with sudo privileges with a password. And be relying on your attacker to not say, simply put aliases into your shell, replace your shell, modify your path, add an LD_PRELOAD, ptrace your processes, etc.
