Like most here, I am very good at web scraping and automated form fills. I keep trying to figure out a profitable side project or business idea to make out of it and keep coming up with nothing that works.
You can do it as a service, but that is highly competitive and basically trading time for money. Best ways are to productize it:
- build a on-demand data api for a specific type of data and charge a premium for it. Good example is https://serpapi.com/ who do Google data, charge ~10X markup on proxy costs
- proxy solutions make good money. To scrape at scale you need proxies, and lots of users pay $1-5k per month. Lots of proxy solutions doing +$100k per month.
- build a tool that uses web scraped data, analyses/filters it and displays it to users. Lots of the biggest web scrapers are doing this, ex. doing product monitoring products for e-commerce companies, etc. Lots of competition there, but you can do it in new markets, like NFTs, etc.
- hedge funds will pay huge money for web data, if you have 5 years of continuous data so they can backtest it.
> build a tool that uses web scraped data, analyses/filters it and displays it to users. Lots of the biggest web scrapers are doing this, ex. doing product monitoring products for e-commerce companies, etc. Lots of competition there, but you can do it in new markets, like NFTs, etc.
Do you have any examples of such sites?
> hedge funds will pay huge money for web data, if you have 5 years of continuous data so they can backtest it.
what kind of web data would they be interested in?
For a while I had a hobby project that would scrape real estate websites listing properties in my city. Goal was to try and figure out trends, pricing data, find good deals. Eventually the site added those features itself (heatmaps based on prices, for example)
With all that data you can do stuff like make heatmaps from pricing data, figure out the most attractive areas for certain profiles (singles, families, ...). You could then mash up that data to produce things like a "Walkscore" or let people indicate what's important for them (green areas, bars & restaurants, time & distance to other destinations, even crime levels) and then show real estate that meets their criteria.
Some sites in the US already show this but in other countries that's not the case, while the data's all there just to grab.
Most likely it wouldn't be legal and certainly not if you made money from it. But it's incredibly fun and hugely useful. Maybe that could get you started on some ideas!
It would be relatively easy to solve this problem if the original supplier wanted the problem to be solved. Instead of releasing a batch of inventory at a certain time, run a raffle over a week or two and then randomly select folks to allow to purchase the item.
That assumption doesn't seem like it'd hold. You'd just replace bot services with package forwarding services that can generate unique PO Box numbers or whatever.
> But what other than artificial scarcity drives people to spend hundreds of dollars on bots to snipe sneakers?!
There's a whole sneaker collecting subculture. Some buy and wear while others just collect. The big names in sneakers do release limited production models or limited runs of certain color combinations.
What economic value (other than hoping for the value to increase, aka tulip mania) can I derive out of ultra-rare sneakers? What usage that goes above "it looks cooler than an unbranded, otherwise identical sneaker"?
For me, this kind of product is part of the "bullshit economy" - similar to "bullshit jobs", this kind of product has no reason to exist other than vanity, as almost all of these "collectibles" won't ever be used. We are using up valuable, finite resources to create and distribute this kind of useless "bullshit product", we are using up valuable human time and IT resources on developing websites capable to resist (D)DoS attacks and on developing snipers to bypass the anti-bot technologies employed by the shops, and we are creating a lot of demand for all kinds of sneaker-related crime - and there's a lot of that: theft and robberies from stores, theft and robberies in the supply chain, ebay/classifieds scams, credit card fraud, robberies on broad daylight [1].
Seriously, fuck all that shit. No one needs hundreds of dollars worth of sneakers that only incentivize crime and bullshit.
Any good ideas?