But they are making a year-on-year relative analysis, so even for a non-representative sample, you can identify a strong trend.

The trend is still just searches. Do their historical search numbers closely represent actual figures for people coming and going?

Maybe it's a demographic shift between people who don't use their platform and those who do.

Or that people moving into the city/state are paying moving services while those leaving are DIY moving.

Not to be cynical, but this strikes me as more of a ad - or click bait. Search numbers on a website that I, at least, have never heard of is pretty poor evidence for a trend.

The traditional signal is when U-Haul starts having logistics problems getting trucks back to the City.

UHaul publishes top growth cities and states based on their data. Here's the list of top growth states for 2019: https://www.uhaul.com/Articles/About/19965/U-Haul-Names-Top-...

There is nothing wrong with that sample size. That should give less than a 3% margin of error at 95% confidence.

If it were uniform.

Some stats from https://www.surveysystem.com/sscalc.htm Plugged in 800,000 as pop size, and 99% as confidence level and 4 as confidence interval. The number it spits out is 1039 It is a pretty good sample size.

This would be true if the website drew a random sample of residents of San Francisco and asked them. They didn't. Rather, they drew a census of their website. No one is doubting that the website has correctly confirmed that 90% of the SF-related searches on their website were people looking to migrate outbound (so the "sampling error" here is effectively 0, unless we choose to model the census as a sample from a meta-population, in which case the classical MOE would be +-1.7% without a finite sample adjustment);

The issue here is whether we can draw a reasonable inference from that sample to the target population (all people in San Francisco). My guess is it does not. The title is not justified by the data presented.

The claims we're able to make are narrow:

A combination of annual secular growth, advertising spend, and COVID that we cannot disentangle suggests 30% increase in move requests to this moving site in the area; Separately, the sample of people using this website are more interested than outbound moves than last year.

If we want to combine these, we have "Website Shows 90% growth in raw outbound move searches among SF visitors"

That only holds true if you assume this sample is representative of the overall population of SF.

The outbound relocation effect went from 57% to 90%. I suspect that 1200 samples is more than enough to demonstrate strong statistical confidence intervals. (Though I haven't run the numbers.)

They are if they're a representative sample. Margin of error is bounded by 1/sqrt(n) in that case.

That MOE would be extremely conservative given that the proportion of interest is far from p=0.5; the classical MOE would be +- 1.96 * sqrt(0.9 * 0.1 / 1200) = +- 1.697% -- but of course it's not a representative sample as you mention.

came hear to say the same thing. the submitter should not have used a over exaggerated title. 1200 people is not an "Unprecedented Rate".

90% is the rate. 1200 is the sample size.