Came here to suggest 80legs too, happy to see you're on it. I would add that if the project was being done for actual work and not just curiosity, then 80legs has the added advantage of not spending all that dev time building and testing it.

Wouldn't the cost of transferring the crawled data (compressed/archived) to AWS/EC2 partially offset the cost saving?

Edit: AWS still has free imbound BW.. my bad

No - inbound data transfer cost to AWS is free.

nice thing about running crawlers from EC2 - all inbound bandwidth is free

Like running crawlers what can be some other use cases where free inbound is a blessing?

Backup/archive, where uploading data is frequent and downloading it is only very occasionally required.

I notice the welcome message and action buttons on the home page are all jpeg images rather than text, which makes it harder for search engines to make a judgement about the content. (The text appears fuzzy as well.) As a developer of a crawler, is there a reason you found this to be beneficial?

(Completely aside: Your front page's colors look strange on my Mac. https://skitch.com/joshua/ecykp/80legs-custom-web-crawlers-p... -- see the blue border?)

Not just your Mac - my Firefox & Chrome on Linux. Seems the color palettes in http://80legs.com/_images/bg-home.jpg vs http://80legs.com/_images/the-most-powerful-web-crawler-ever... don't match up. Looks great on all browsers on Windows, so I'm guessing the jpgs have a color profile attached.

Looks like http://80legs.com/_images/the-most-powerful-web-crawler-ever... uses slightly different gradient stops than http://80legs.com/_images/bg-home.jpg