The goal is not to scrape sites en-masse, but to allow people to automate their existing workflows and actions that they perform already via a browser. I understand the concerns around this being unethical, and it's something I spent a lot of time thinking about when I worked on automations previously. I've written a decent amount about how I don't think that sneaker bots or ticket bots are ethical. I don't support mass scraping websites/making the web more inaccessible for others.
I do have to push back on the ToS comments though. Automation is used daily by nearly all companies. RPA is a billion dollar industry. Browserbase raised at 300M valuation. Is using puppeteer to automate a form submission a violation of ToS? If so then why is using a screen reader not? Is it the intention? Why is hitting network requests directly different? I personally don't think that automation is unethical (as long as it is not affecting server capacity). I don't think the answer to the ethical problems in scraping is just not to automate at all. Open to disagreement here though.
> Is using puppeteer to automate a form submission a violation of ToS? If so then why is using a screen reader not?
Without taking a position on the ethics of automation, surely this isn't a serious question? Things that the ToS prohibits you from doing are ToS violations, and other things aren't.
For instance, from AirBnb's terms of service: "Do not use bots, crawlers, scrapers, or other automated means to access or collect data or other content from or otherwise interact with the Airbnb Platform."
There is no similar prohibition against using screen readers.
My broader point is that these ToS clauses are often so broad and vague that they're essentially unenforceable and not meaningful in practice. For example, "Do not use bots" covers a pretty substantial amount of ground, and intention isn't exactly something you can screen for. Is an autofill chrome extension a bot? If so what separates that autofill from accessibility extensions? Is someone using Whispr flow to fill forms considered a bot? AirBNB doesn't block Google's crawler. Why? A company can enforce its TOS as it wishes. My general point is that the waters are murky, and that automation is a sort of sliding scale.
> For instance, from AirBnb's terms of service: "Do not use bots, crawlers, scrapers, or other automated means to access or collect data or other content from or otherwise interact with the Airbnb Platform."
> There is no similar prohibition against using screen readers.
A screen reader uses automated means to access or collect data or other content from or otherwise interact with a platform.
Under that ToS would a screen reader not be considered “other automated means” of “interacting with” the platform? It is automatically walking an accessibility tree.
Ah yes, AirBnB, the company that famously hacked Craigslist to achieve viral growth by using a bot, crawler, scraper and definitely automated means to access and collect Craiglist's data and other content from and otherwise interact with the Craigslist platform.
FTX also raise lots of money. so did Terra/Luna. large valuation does not mean you cannot end up in life inprisonment for fraud.
> Why is hitting network requests directly different?
again. business publishes their official API. prohibit automatic and tool use in their ToC.
you go pass that and access what they do not release publicly.
this is borderline Copyright infringement, Trade Secrets theft, and violation of ToS.
> others do it
no they don't. "assisting screen reader" in ios. does not intercept MITM network traffic, attempts de-obfsucate, decrypt, bypass TLS and CAs with fake CA roots with intent to "udnerstand inner workings".
this is very far from the "assisting user". you do not need that level of breaking into internals.
and besides. if app really wants to assit user there are legitimate ways to do so, like accessibility labels in ios, exposed by app itself to OS. crucial point here - app already exposes it itself.
I mean, if you get legal permission from business owner to run this tool and expose their API. sure, run whatever you want (as long as your ad-hoc contract allows). and if you cannot get this contract, you are clearly breaching use and going directly agains what business intended.
you are not assisting any user here. you are trying to directly steal trade secrets to replicate their business.
I do have to push back on the ToS comments though. Automation is used daily by nearly all companies. RPA is a billion dollar industry. Browserbase raised at 300M valuation. Is using puppeteer to automate a form submission a violation of ToS? If so then why is using a screen reader not? Is it the intention? Why is hitting network requests directly different? I personally don't think that automation is unethical (as long as it is not affecting server capacity). I don't think the answer to the ethical problems in scraping is just not to automate at all. Open to disagreement here though.