So does every other language, where it matters is with objects. Until I can mark a whole object as a value type and fix it in memory relative to other objects then you'll pay a 10-50x slowdown for cache misses and the like.
Typed arrays are close, but still not good enough.
On the Java side I've used ProtoBuf for similar approach(using byte[] which does have memory placement semantics). While it helps, the bounds checking and accessors that you need still have a non-trivial cost both in branching and cache thrashing(len is usually stored for an array at a different location or at the head so data items near the mid/tail will still take a hit).
Realistically you can get within about 1/5th the speed of native using cache aware data structures and techniques. As always it matters if your use case needs that speed but choosing a JS/managed based tech-stack will always limit you from getting that last 5x perf improvement.
so aside from startup speed you get the same perf