After some experimentation I found out that pygame is even better them pyglet.

I had multiple issues with pyglet with high monitor refresh rate, HDR and refresh rate mysteriously slowing down after I dragged the window around. While pygame works flawlessly.