Don't know about SerenityOS' but Dillo (predating even NetSurf) is pretty nice minimal browser with independent layout engine and there's Abaco (for Plan 9) which supports most of HTML4. That said from your list you miss KHTML (WebKit's parent; have diverged significantly since), Trident, EdgeHTML, Presto, and of course, Servo. All without doubt advanced engines. It's disappointing that both Microsoft and Opera didn't open sourced their own after retiring them.

Edit: Oops, didn't see the "open source" clause. So Trident, EdgeHTML, and Presto don't count.