Actually, the codecs themselves usually aren't responsible for displaying to the screen via DirectX, OpenGL, etc. In the case of <video>, the browser needs to be able to mix the video into the web page, which means the browser needs the video data in its own memory, or as a pixmap/texture in video card memory.
But, most codecs are going to be written in C and assembly language for speed, which brings the potential for buffer overflows and other low-level exploits. Plus, that video data does eventually make it to the kernel and then the video hardware (often via a separate overlay interface like DirectShow, Xv, or VDPAU, though that is probably not the case with web browsers), so a vulnerability at any point along the chain is a serious issue.
But, most codecs are going to be written in C and assembly language for speed, which brings the potential for buffer overflows and other low-level exploits. Plus, that video data does eventually make it to the kernel and then the video hardware (often via a separate overlay interface like DirectShow, Xv, or VDPAU, though that is probably not the case with web browsers), so a vulnerability at any point along the chain is a serious issue.