Does AWS actually allow you to crawl like this? I've been interested in a similar project but the cloud providers I typically use seem to ban it in their terms of service
Those hours that with gentle work did frame
The lovely gaze where every eye doth dwell,
Will play the tyrants to the very same
And that unfair which fairly doth excel:
It's because their target audience are enterprise customers who want to use their cloud hosted models, not local AI enthusiasts. Making the model larger is an easy way to scale intelligence.
