I worked through large parts of PQP ([1]) with only the prerequisites that you state, as part of a Uni course taught by Aleks Kissinger. I got quite comfortable with the formalism and could solve most problems without too much issue, but at the end I still knew nothing at all about quantum physics. YMMV