it gives historical data for how big the revisions have been. Looks like historically, the revisions have been at least an order of magnitude or two less than 250,000. In fact to get revisions anywhere near that large, you have to go back to April of 2020. Which was when the whole world was going bananas because of COVID.
Unless I'm reading the data wrong (always a possibility) it really does seem like a 250,000 downward revision, to like 17,000, a two-order-of magnitude miss, does seem to be anomalous.
The revisions were -133k for June and -120k for May, and yes the are unusually large.
Generally, it seems that estimation post-covid became less accurate, with five (if I'm counting correctly) revisions of 100k or more since 2022, excluding the latest two.
I imagine this is because covid messed up the numbers for the seasonal adjustment.
Do note that the reported standard errors are about 83k/70k for first/second estimates (tables 3/4 at https://www.bls.gov/web/empsit/cesvarae.htm), meaning that deviations of up to ~133k/112k in either direction from the true value are expected at a confidence of 90%.
Overall, my conclusion is that these revisions are higher than normal, but not unexpected if you dig into the numbers.
hmmm... did it get harder to estimate, or was the estimator just not up to the task? It certainly is different after COVID--did they keep updating their methodologies, or were they just relying on the same-old same-old?
sigh What do I know. My point is that probably any president would be displeased if the numbers they had been bragging about for a few quarters came in orders of magnitude worse. And any president would probably ask the question of whether the right person had the job.
Trump massively politicized the issue, in his typical very nasty and mean-spirited way. Shockingly indecorous. But we can't just reflexively attribute malice to every thing Trump does. We will lose all credibility of we overreach and overstate our case.
> did it get harder to estimate, or was the estimator just not up to the task?
It got harder to estimate, because less people reply to the survey (see sibling comment). They are aware of this and they could improve their analysis and data collection methods with more funding and staff, but have been facing challenges for years and the actions of the current administration certainly did not help:
"The government recently disbanded two outside advisory committees that used to consult on the numbers, offering suggestions on ways to improve the reliability of the government data."
March 11th, https://www.npr.org/2025/03/11/nx-s1-5323155/economic-data-r...
> any president would be displeased if the numbers they had been bragging about for a few quarters
These numbers came with uncertainty estimates that covered the possibility of such a large revision. But this was not taken into account. Again, basic numeracy required to interpret poll results.
> But we can't just reflexively attribute malice to every thing Trump does.
Malice or incompetence I agree, which is why we dug deep and found:
1. Numbers came with uncertainty estimates that were ignored;
2. Data from the last few years was less informative than usual because fewer people respond to polls;
3. Recent cuts to budget and personnel made it even more difficult to improve data collection and analysis.
Well, thanks for digging deep, at least this is a conversation and not an ad hom party.
W.R.T. #1: I put it to you that if the margin of error is two or three orders of magnitude greater than the value reported as being measured, its not really a measurement, or even an estimate. If I told my foreman, "yeah, I need either 3 more feet of lumber or 300 more feet of lumber to finish framing this house" have given him a measurement, or even an estimation?
It was her job to deliver a measurement (we'll come around to whether she was given enough resources, etc I promise!) But that was her job. Looking at the July numbers, the reported number is 73k jobs, and the error bars are between -63k to 209k. An order of magnitude larger than the reported number? Did she really do her job? Did she really give us any information at all?
W.R.T #2: Sure, people are not responding to polls so much. You could even make a case that this is partly because of Trump's policies: they probably don't want ICE busting into the workplace and chunking their new employees into a concentration camp--best just not to report hiring anybody at all.
But as times change, the means and methods must also change. If the polls are no longer accurate enough, a new method of measurement must be found. Why not, say, scrape social media to see how many people talk about getting a new job or getting laid off. Or just buy the data.
It's scary how much social media companies know about us. I'll bet Zuckerberg has a pretty durn accurate number of how many jobs were created in July.
W.R.T. #3: Yes, again, Trump's policies have made her job way harder. But tell me, is there any CEO or business owner who isn't faced with the problem of trying to do more with less? Intel just fired Pat Gelsinger (one of the few executives who is so talented that I would blindly follow him into the Gates of Hell), because he couldn't do enough more with less. Personally, I think the board was wrong, but there's no reason to think the board is corrupt or evil.
The whole history of humanity is being pressured to do more with less--which is another way of saying that we have continually challenged ourselves to become more and more productive so that we can have better and better lives.
Now, Trump is a very bad manager, and he very well have "misunderestimated" the amount of resources she needed. Please understand, I'm not saying that Trump's incompetence has nothing to do with why he fired her. Perhaps he did set her up to fail, but it could very well have been done because of incompetence and not malfeasence.
What I am saying is that comments like the one I originally replied to, which claim that this is transparently a case of government corruption, etc etc, are not necessarily so. Trump has spun this as a case of government corruption, certainly for the purposes of covering his own backside on this, but what I'm saying is perhaps we shouldn't just take Trump's word for it as to why this happened. He said what he said to get BOTH left AND right to jump to the conclusion he wanted us to.
We certainly shouldn't take the bait which Trump has offered us, to just further drive a wedge between us, get us to reflexively hate everything the other side does, immediately just assume the worst of each other, etc etc. As long as they can keep us divided, they can keep us conquered.
Statistical significance is not connected to real-world relevance.
With enough resources one could know the exact number without any margin of error. This number could however be the result of random fluctuations rather than systematic effects such as policies or natural phenomena. Only the latter are interesting, because you do not want to take decisions based on random noise. While some sources of noise can be eliminated, for example by polling more people, other types of noise are just there and are inherent of the process you are measuring. Wide confidence intervals also convey this kind of information, and more advanced statistical analyses can separate these two fundamental sources of noise. Those who care no doubt are informed.
Speaking of real-world relevance, there are about 159M people employed, so a confidence interval of 270k means that you can statistically detect changes larger than 0.17% within a month, and a CI of 70k after three months reveals changes of 0.04% or more. Since policy changes need time to propagate around the economy, at least one and perhaps two quarters, I would imagine that data for the immediately preceding month is not critical. Given the budget challenges in the last years, one could speculate that decision makers in the government were generally satisfied with this level of accuracy. And if they are not, sacking people does not change how statistics works.
Speaking of Trump: actions speak louder than words, and history remains a good teacher. I won't go any further since I am not American, I can only hope you guys get your shit together.
Survey response rates have decreased significantly since Covid.
"LS surveys firms in the payrolls survey over the course of three months, gaining a more complete picture as more businesses respond. But a smaller share of firms are responding to the first poll. Initial collection rates have repeatedly slid below 60% in recent months — down from the roughly 70% or more that was the norm before the pandemic."[0]
Is that an excuse for bad numbers though? I mean, a revision of a quarter of a million jobs is abnormally large. Yes, because of COVID, etc, etc, times have changed, but if savers are not accurate enough, another method of measurement must be found.
One idea would be to scrape data off of social media sites, or just buy the data from Palantir or something. I'll bet Zuckerberg has a very accurate number of how many jobs were created last quarter.
I'm sure any president from any party would not be happy with employment numbers which were essentially useless. Trump said what he said to get BOTH left and right sides to jump to opposite conclusions. Lets not take the bate.
https://www.bls.gov/web/empsit/cesnaicsrev.htm
it gives historical data for how big the revisions have been. Looks like historically, the revisions have been at least an order of magnitude or two less than 250,000. In fact to get revisions anywhere near that large, you have to go back to April of 2020. Which was when the whole world was going bananas because of COVID.
Unless I'm reading the data wrong (always a possibility) it really does seem like a 250,000 downward revision, to like 17,000, a two-order-of magnitude miss, does seem to be anomalous.