I like to use Google search as an example since I was involved in building a competing search engine roughly at the same time as Google, so I happen to know a thing or two about the difficulty of that task.

Everyone and their dog thinks Google won solely because of PageRank, but anyone who has ever worked on web scale search engine can tell you that long before you can start worrying about ranking, you have to solve a bunch of problems that are really hard. Like copying a sufficiently large portion of the web onto your servers and processing that copy. Or even removing all the duplicates you will get (which is certainly much harder than I had imagined when I was tasked with this problem).

Google did all of these things as well as, or better than the competition. And on top of that they put together a cocktail of ranking methods that they were able to apply to information amounts for which there was effectively no practically oriented literature to turn to for advice. (Sure you could find lots of neat algorithms for this and that, but very few people had any experience with these at web scale).

Oh, and Google were not alone in having PageRank, or algorithms similar to PageRank. All web search engines I have any inside knowledge of (2-3) had this within six months of Google popularizing the idea.

Ask any one of the dozen or so web search engine companies that did web search if any of this was "easy".

What Google did in the late 90s was indeed heavy lifting. What pointcast did was relatively trivial and easy to copy in a few months.

edit: fixed typo

And FWIW, Google's real strength -- I guess I'm not everyone and their dog? -- was that they made a business case for the individual searcher. In the collapse of the .COM bubble an individual searching was seen as a close to worthless commodity, which is why every once-vigorous search engine had largely abandoned engineering advances. Google at the time first saw being the premiere search engine as a way of selling search appliances to business, and then with their text ads reinvigorating web ads after, again, the .COM collapse.

Alas, herein we see the magic of the survivorship bias: Google won, therefore Google must have done everything right. Anyone who lost must have done everything wrong. This is cargo cult thinking that does nobody any good. How you so casually write off the engineering challenges of Pointcast (an incredibly popular, multimedia-rich product, including pooled and queued subscription for millions of people, when bandwidth and computing power were very limited. A whole industry of Pointcast caching appliances appeared).

It's also interesting how you create the caricature of executives decrying the Google interface, as if it were the vindicated underdog after all of those years. Yet in those early years, the simplistic interface of Google was overwhelmingly the number 1 lauded feature of the site. Even before it was featured for good and quick search results, it was heralded for its simplicity.

Yes.

Look at who is successful, note some random characteristics of these individuals or businesses and then come up with your own theory of how success is deliberately attained. The share it over the web.

What's even more amusing is when someone sings Google's praises as a company that "did things differently" and then proceeds to dismiss any future innovators (who choose to do things differently) in the markets that Google is in.

I have often thought Google is a grossly misunderstood company, despite many facts being readily available. I'm still waiting for someone to show me otherwise.

Some years back I watched a presentation where a representative of one of the internet's most respected infrastructure companies claimed that the secret of Google's success was diversification and not relying on one product (search). He suggested the winning strategy was to release numerous products and cited non-search Google's products as examples. However, every example was a product released by another company (who, go figure, focused exclusively on one product) that Google had acquired. Some folks do know the facts about Google, but I see an inordinately large number of folks who you think would know the facts, but sadly do not. Instead they have their theories and are eager to share.

Well, I'd say otherwise. It was hard and there were big differences in how efficient the various platforms were. And sadly, not all of them could keep up with the development pace and the growth of the web.

(I used to work for two of Google's competitors in the late 90s and early 2000s, I had some knowledge of the internals of a third competitor and I eventually ended up working at Google for a few years).

> Alas, herein we see the magic of the survivorship bias: Google > won, therefore Google must have done everything right.

That was not what I said. Please do not pretend that I said that. I said that what Google did represented heavy lifting and what Pointcast did wasn't. I didn't touch on why Google succeeded at all. In fact, Google's success is entirely irrelevant.

At this point I have absolutely no idea what it is you are trying to say, then, as you apparently want to have your argument work in any way way that you think has some sort of lesson.

Google is successful today because they made business bets after the .COM crash that no one else was making. They brought a new approach that earned them attention and customers, and the rest is history. They did good engineering, but so did countless companies that failed or continue in obscurity. I suppose that is the "heavy lifting"?

Pointcast obviously was doing good engineering (they had an engineering scale that was pretty much unprecedented at the time), but their business case completely fell apart and the organization was dissolved. There were literally zero issues with the engineering of Pointcast, nor were they replaced by competitors (despite that apparently being your angle?). It simply wasn't a viable business at the time, and they couldn't find a way to make money.

Heck, even Twitter and Facebook led to a fundamental change of view of the internet, namely comms and social networking.

Compare this to pointcast or snapchat. What is the fundamental shift in point of view that happened due to pointcast of snapchat?

That is the very premise of the WWW. There was nothing new in that.

There is a too common "the victors write the history" sort of way of looking at things. And FWIW, the notion of counting, weighting and sorting on links is hardly a novel premise, but the schism that Google really brought was that they had a business case to spend far more computer resources on any given search, and on the content that served those results, the results being a much better product. When they in-housed some custom ad serving, the ascent of Google began.

Every product along the way -- whether it is long gone or a thriving success -- has an influence on technologies that follow. Pointcast is absolute the ancestor of RSS and other so-called push technologies, which themselves could be considered the ancestor to things like Twitter. To write it off, rewriting history based upon some sort of survivorship bias, just clouds the topic.

Yet their competitors insisted on treating each document as standalone until after Google popularised this idea.

At the time it very much was seen as a new idea, and the idea that this was good enough to deprecate lots of advanced search operators was so alien that it took a lot of us a long time to stop trying to construct complicated searches of the type encouraged by e.g. Altavista.

That's not entirely true, but this is the simplified view that the press popularized because it made for a story that was easier to report.

When you say "insisted" that seems to imply that other search engines were opposed to the idea (whether that is what you meant is another matter). Back in the 90s there were a lot of things we wanted to do, but that we either didn't have the manpower/time/money/talent to do or that we just hadn't figured out how to implement efficiently.

Ideas are cheap, but implementation is what counts.

I can remember that we discussed various link-aware ranking factors long before we even had built a proper search engine, but we needed to solve a lot of more pressing problems before that was even on the agenda (like build a crawler that didn't break the internet :)).

Once on the agenda the question was how to implement it at scale. Remember that this was before AWS, before Hadoop, and before a lot of other things that the average developer has easy access to today. We had a finite set of machines and a finite amount of money to buy machines. And even if we could have had all the hardware in the world we still had to figure out how to turn a lot of algorithms implemented for single machine processing into distributed systems.

That being said, the fellow I shared an office with was able to crank out a sufficiently scalable PageRank-like implementation in a few weeks.

Not that this made much of a difference PR-wise (and initially, it didn't have quite the impact on the quality of search results we had hoped for). In the view of the public only Google used PageRank and PageRank was the thing that made them stand out. End of story. The fact that Google did dozens of other things much better than their competition was...well, too complicated for journalists to report.

As a side note: I think we reinvented, and implemented, various subsets of MapReduce at least a dozen times during 1998-2000. A lot of our processing systems were based around sort and linear disk scans. Two things we knew how to do fast. Google (at some point) had the good sense to recognize that this could be turned into an infrastructure, which allowed them to spend less time building systems that could be expressed using these primitives. They also took the time to solve things like reliably storing data without getting too hung up on traditional filesystem semantics. (When data-sizes grow to the point where you have to maintain large population of disks, you can't really trust a single disk to be there tomorrow. We did some testing with RAID systems, but were unable to find a solution that was both cheap enough and reliable enough. Not having something akin to GFS was, in retrospect, a major impediment)