It wasn't "catastrophic" for the largest of the 3 US credit reporting agencies when their entire dataset was breached. The company is 100% IP and the only value they have was completely copied. Their largest value is to verify identities by the things Americans know (KDB) and after that "single factor of identity" was 100% compromised, the company only got bigger and more contracts.
When there are only 4 competitors in the large scale foundation model business and they all throw caution to the wind because they are racing to own the "$30 trillion TAM" they are all going to make critical security, RBAC, and segregation mistakes.
Both ChatGPT and Claude threads marked for sharing have been indexed in Google at large scale. This is incredibly easy to tell Google crawlers via robots.txt not to crawl those URLs, but nobody at either of these uber unicorns could be bothered to add that one pattern to the one file.
And all of the skepticism here is about verifiability. The foundation model companies are liable for potentially more the companies are worth if found to be violating copyrights of content used for training. They aren't going to make it easier for lawsuits against them by detailing their data ingestion into training pipeline.
The credit reporting agencies lost the data of normal people, they didn’t lose their customers proprietary internal data. The credit agencies didn’t loose or misplace their customers data, so obviously their customers don’t really care that much, and the credit agencies weren’t sued into oblivion.
But I can guarantee you that if a companies internal data got leaked or misused, then every single enterprise customer of that lab would turn around and start suing them. As an enterprise customer you would be foolish not to, if only if figure out via discovery just how badly you got screwed.
You want to see how nasty that can get. Just go and look at what Apple is doing to OpenAI at the moment. Do you really think Apple wouldn’t find a way to sue a lab into oblivion if they discovered a lab had secretly started training on their data?
You can structure it so that it becomes accidental.
1. Ensure security barriers are weak or honor based.
2. Put individual researchers under a lot of pressure.
3. If you get caught, blame the weak barriers, or the individual researcher.
Basically setup the incentive structure to incentivize researchers sticking their mittens in the private cookie jar while putting the cookie jar in a dark unmonitored/unsecured room with a sign on the door saying please don't enter.
Yeah, the steps follow exactly what happened at VW with DieselGate. The diesel emissions lies were found out because some enterprising person set up an emissions testing system and drove the car in real world scenarios with it to verify the claimed emissions.
There's no reliable way to verify a foundation model has been trained on a particular piece of proprietary data. If an API key is ingested, hopefully the foundation model is wrapped in enough moderation that the raw API key oberserved during training is not recited verbatim in the output.
Not trying to be rude, but do you work in tech? I can't imagine presenting this as a plan of record in a design review. And the world runs on good faith. If you call a pharmacy, claim to be some doctor, leave a voice mail, and give their (public) NPI number, there is no validation.
Why aren't people calling in prescriptions for themselves? I guess it just kinda runs on trust me bro and the threat of being put in prison.
It wasn't "catastrophic" for the largest of the 3 US credit reporting agencies when their entire dataset was breached. The company is 100% IP and the only value they have was completely copied. Their largest value is to verify identities by the things Americans know (KDB) and after that "single factor of identity" was 100% compromised, the company only got bigger and more contracts.
When there are only 4 competitors in the large scale foundation model business and they all throw caution to the wind because they are racing to own the "$30 trillion TAM" they are all going to make critical security, RBAC, and segregation mistakes.
Both ChatGPT and Claude threads marked for sharing have been indexed in Google at large scale. This is incredibly easy to tell Google crawlers via robots.txt not to crawl those URLs, but nobody at either of these uber unicorns could be bothered to add that one pattern to the one file.
And all of the skepticism here is about verifiability. The foundation model companies are liable for potentially more the companies are worth if found to be violating copyrights of content used for training. They aren't going to make it easier for lawsuits against them by detailing their data ingestion into training pipeline.