Hacker Newsnew | past | comments | ask | show | jobs | submit | userbee's commentslogin

"Big data" also checks model assumptions, if only if by monitoring whether or not acting on the information moves a business metric.

Statistics involves inference over prediction, but either one when done right validates assumptions.

By the way big data will sit on your face for days.


I meant checking assumptions not just to see whether the use of the big data moved a business metric, but also that the model makes sense from a statistical perspective.

A lot of statistics in business does not bother to check modeling assumptions. Models are chosen based on whether they've been used in the past and what the team is familiar with.

I don't doubt that big data (as we call it now) will one day rule. Ronald Fisher would keel over if he saw the size of datasets we work with nonchalantly on a daily basis. 50 data points (the size of the Iris data) is laughable these days.

My reservation with big data is that the technologies are often unnecessary for the size of the tasks being done. Other than a few data scientists working on truly large projects, most of the big data talk I hear comes from people who aren't fighting in the trenches (execs, marketing, journalists).


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: