Most compression algorithms have a limited data reference window (dictionary size) - tens to hundreds of MB being typical upper limits. If duplicated data is further apart, it gets compressed again.

If the data isn't pre-compressed, or doesn't otherwise significantly vary, then using the right tools should de-duplicate the data. Things like lrzip, rsync batch mode, bdiff should work well. If large blocks are identical, many filesystem archivers/backup solutions should do good deduplication, and squashfs can be kind of nice.

That MIGHT be happening, honestly not sure.

There are possibly challenges in that however, my understanding of most compression algos (which may be inaccurate or incorrect) is that you'd run into limitations either with size of data overall or the memory requirements.

Of course, I'm sure there's other ways around that (And I have no clue if they are in use). One option would be to send over the assets in a master 'assets' package and then duplicate/assemble the data as intended during installation.

That's not how compression would work. Think about if they are adding a new level. The algorithm would have nothing to compare the files against except what is actually being sent. If the texture appears in each level, and each level comes in a separate update, there's no way for the algorithm to reference or incorporate the files that are already in your computer, so it has to compress it like it's a brand new file. That is unless you have a costume installer which duplicates the file once it runs, but that would probably be a ton more work for the developers.

You have this completely wrong. "there's no way"? It's extremely easy to make a patch file that references existing data. This is a problem that has been solved many times.

And that's not relevant to initial install anyway, which can/should be a single download.

But thats not compression. That's downloading and running an executable that duplicates files.

Having a reference dictionary is a type of compression.

You need a program that decompresses no matter what, and the ability to reference existing files barely changes it. It's not a post-processing stage where files get copied around. It's the ability to reference arbitrary chunks of data inter-file just like it can reference arbitrary chunks of data intra-file.

At the most basic level it's like taking a compressed file and chopping it in half. The user already has the first half, then they download the second half and "resume" decompressing.

What's your definition or compression? A .tar.gz will de-duplicate files that will be re-duplicated on decompression.

I think AAA games do this now anyway to allow this like smaller patches and playing while the game downloads.