Why? Attach seasoned product teams from Google to the research org and have experienced PMs paired with experienced EMs launch products.
Trying to transition a research org into a product org is going to be needlessly painful, especially since the research org needs to be firing on all cylinders in this hyper-competitive space.
Yep. I joined Google X Robotics (which became Everyday Robots, which got canceled) just as the org was winding down a big R&D push and moving to product development. Engineers were palpably hurt by their various projects being canceled, and in my opinion they never had a viable product strategy beyond “let’s see if we can find a consumer use for this robot”. This strategy ultimately failed and they canceled the project, let go a bunch of the people, and now the robots are being used for AI research. So the whole shift from R&D to product development was a failure. They could have saved a lot of grief and money if they just continued as an R&D org, and in my opinion they would have left open some important doors which would have really helped with AI research.
I'm afraid this is going to be another Everyday Robots or Boston Dynamics. Google is ruled by managers, not visionaries. Strategy is 'try and see if it works in 4 years. if not cancel'. They followed it in many cases. So, DeepMind's cancellation is long overdue. A couple of years back one of Google's top managers talking about AGI said it's most likely to happen in DM. But current LLM boom happened elsewhere. Likely managers are disappointed in DM.
Just because a company is 99% one thing, doesn’t mean that the 1% remaining don’t have moments of real genius. What it does mean however is that when those real genius moments happen, the company isn’t in a good position to capitalize on it (ala Xerox or Kodak mentioned previously, but there are so many more).
Because that's the hallmark of business-minded people being at the helm and not engineers, which has been an issue for Google for a long time: the greatest inventions ever seen will be squandered in terms of the org itself because management can't see beyond the next few quarters and won't invest properly in it.
??? Xerox and Kodak are immortal household names that dominated their niches for decades, what level of "success" would satisfy you? what are you lamenting, that they didn't completely enshittify their products while they were on top and torch their brands to the ground for a little more revenue?
Well, strictly speaking modern LLMs are transformers (not counting state space models like Mamba, for the moment). Not sure what hidden Markov chains have to do with modern LLMs.
Well, LLMs are not AGI. They have serious limitations [1] and honestly my fear is that it’s too soon. I agree Google is ruled by managers (that’s why I hated it), and my fear is that the managers have FOMO and want to push to productize asap even tho the tech isn’t ready yet.
Look what happened when Google tried to throw an LLM in to search. Absolute shitshow. That’s not ready to become any kind of product!
If they kill R&D now to focus on productizing something that is half baked, they will fail to develop those new inventions which might get us to AGI. When I worked at Google X Robotics I was hired on to the remnants of the last research team, which was dissolved six months after I started (I was moved to hardware test engineer). Our subteam really wanted to research multi-finger grippers but we got overruled, so the robot had to do everything with a two finger pinch gripper. Which is fine for research but absolutely unsuitable for real world tasks. It couldn’t even operate a spray bottle without special attachments and they thought it was going to clean people’s homes!
[1] I am sharing this one a lot lately but I’m very moved by Yann LeCun’s arguments about the limits of autoregressive approaches here. As a robotics engineer I have been dismayed at all the attention LLMs are getting despite serious limitations that make them generally unsuitable to solve some of the most important problems in robotics. https://youtu.be/1lHFUR-yD6I
> my fear is that the managers have FOMO and want to push to productize asap even tho the tech isn’t ready yet
This is exactly it. With the limitations ChatGPT is encountering around safety and hallucination, Google probably should've just said "we're working on something awesome - hold on" and kept plugging away before releasing, instead of ex-Product CEO making them release something now, even if half of the demo video is fake.
I'm using chatGPT to Google something for around a year (or whenever bing browsing became available), and I'm yet to set a single hallucination based on web search results. May be Google is just not very good at this.
Neither is Yann (who has since proposed a different architecture / vision which is yet to take off), but my comment was meant to highlight a recent claim from another accomplished researcher in the field.
Right but I’m not talking about claims, I’m talking about arguments. As a robotics engineer running a farming robot project, I am all too aware of the serious differences in computational challenges and dataset availability between text and image data on the web and the kinds of problems that once faces in robotics. LeCun doesn’t just claim LLMs aren’t up to the task, he provides a detailed list of provable shortcomings which I feel overall make for a compelling argument. I’m hopeful that his JEPA may shed some light on possible solutions, but it’s also a fact that one can find issues with proposed engineering solutions even if they don’t have their own better solution. It you say you have a faster than light space engine designed, one wouldn’t need to have their own functional design to show why yours didn’t work (although such expertise is always helpful). And, well, he did invent convolutional neural networks, tho as I say such expertise is not required to raise valid arguments against some proposed solution.
fwiw, I agree with Yann & you (and many others!) on the shortcomings of LLMs (not from a position of authority, but as a matter of opinion).
I meant to counter-balance OP's point in that there are other equally accomplished individuals who aren't swayed by Yann's (and others accelerationists like Andrew Ng) arguments or claims.
Same experience at EDR. It was a great research platform, but nowhere near ready for the real world. Compute, power, functional safety... The "real" working robots of today are completely different. A successful product from EDR would have looked completely different.
God, yea. I've flipped between R&D and product development a few times in my career, sometimes at the same company, and it's a really rough transition even for an individual experienced in both. I can't imagine trying to flip a whole research team to make products is going to go well, especially when the products are getting a ton of well-deserved bad press and a lot of those researchers were coming from academia rather than elsewhere in industry in the first place
I tend to agree. I'm curious whether this is Deepmind saying "I think we could do things better, let's do this ourselves" or the leadership of Alphabet saying "Get these ivy league intellectuals to prioritize productionizing products!"
Would seem far more sensible to allow Deepmind to continue to release hit after hit in the ML research world, and simply embed "fly on the wall" PM's into their org that can independently productionize any golden nuggets they happen to create.
I feel it's more of leadership of Alphabet saying, "Our people literally invented transformers, so how come we're at the bottom of AI race instead of at the top? A random non-profit took out research and run with it, and they're the hottest company in the world now. This is unacceptable![0]".
Bad idea. People good or lucky enough to land in R&D like doing R&D. Force them to be product people, I expect most of them will leave.
I also think people and society also give themselves way too much credit for their successes. There's plenty of smart hardworking people out there who continue to contribute but never stumble upon a unicorn.
To a large extent its luck, a much larger contributor than people realize.
All you can do is to play the game, consistently contribute and work hard on R&D and products and you improve your odds of stumbling upon success. But it's never guaranteed.
Yes, but that doesn't explain how OpenAI able to walk and chew bubble gum at the same time, unless you think there is some extra spark that Google is lacking.
I think OpenAI was chilled out, and mostly lucky. Chilled in the sense that Google seems to be too full of itself, and their practice of publishing groundbreaking research without actually publishing anything other people can use is, frankly, annoying. OpenAI managed to leapfrog them by slapping a chat interface on a tuned GPT-3 and putting it on the Internet.
The chat service bit is them being chill, but the real spark was the model. I say they were lucky, because AFAIK back then no one expected LLMs to show so many and so advanced general capabilities. This took everyone by surprise, and since people could already play with it, ChatGPT took off on its own - it had so much real, transformative value, that it spread out with zero marketing. That's a rare, bona fide case of "word of mouth", it was just that useful. But that wasn't a strategy, that was luck.
To their credit though, OpenAI turned this early win into an opportunity and is excellent at exploiting it. Being small helps.
> it had so much real, transformative value, that it spread out with zero marketing
IMHO, in retrospect, the failure gradient of early LLMs is underappreciated in driving adoption.
Windows 95 failure: blue screen with inscrutable error code. Everyone noticed that.
LLM failure: run-around non-answer (user shrugs and tries again) or confident and plausible incorrect answer (user doesn't recognize this without research).
Essentially, the ways in which LLMs didn't work were the most hidden and hardest to discover failure mode.
Which was perfectly tuned for the "I'm going to try this thing for 5 minutes and be amazed" first impression.
Which allowed subsequent generations to backfill the capability gaps.
Tl;dr - We shouldn't underappreciate quiet-failing as a product adoption driver.
>We shouldn't underappreciate quiet-failing as a product adoption driver.
I'm certain it drives a lot of early user retention in the short term, but I feel strongly that this is ultimately a very myopic view which will prove catastrophic in the long term in much the same way that swallowing exceptions at runtime builds compounding technical debt you'll have to reckon with sooner or later
more broadly, there is just so much handwaving away all the black box parts of deep neural networks that are completely opaque and there seems to be very little interest in building the tooling to properly visualize, explore, and DEBUG latent space; until those priorities change this whole thing is a huge time bomb.
imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!", and not a single engineer at Microsoft had a complete understanding of why the BSOD happened in the first place; sometimes it just does that! whoops!
> imagine if instead of coming with full memory dumps and diagnostic codes, BSODs just said "sorry, your computer had an oopsie!"
So, MacOS? ;)
In all seriousness, I wasn't opining on the usefulness of opaque/hidden errors, but rather the effectiveness of them.
In an alternate reality where the first LLMs instead spit back an error reference instead of English, I don't think we would have seen nearly as rapid mass market adoption.
And, not to put too fine a point on it, early conversational LLMs and image diffusion models were literally trained so their junk output is as plausible as possible.
GPT-3 was out for years already for Googlw to see, but with what OpenAI saw of it they began training an expensive GPT-4, before chatgpt success, maybe started before RLHF.
Agreed. As an example of that, Meta's kept its research org, FAIR, still doing fundamental research. Research orgs are great at demos, but actual productionalization takes a different mindset.
Referrals are vetted by global teams who are just checking boxes - it's very common to have to reach out to the recruiter for the role directly to resurrect a rejected referral.
Definitely the people working on the model. It ultimately doesn’t matter what the users want because you can’t arbitrarily deliver an experience. You can only deliver what it’s possible to extract from the model, so growing the possible things the model can do well is most important.
In my mind you could not be proving that we need product people more. They'd never say "It ultimately doesn’t matter what the users want" - they'd say "let's find a way to build what users want" not "let's grow the possible things a model can do well".
I have experience in both R&D and product. Both need different approaches to work. The goals of people working on the model will be different from product people. As mentioned by the other user, a product team can look at things produced from research and see how it can bring it to users.
If engineering/research is all mattered we would have maybe two order of magnitude more successful products or companies. Because product-market fit is a thing we don't have any successful research turning to successful product.
Must transfer value, and the guy in charge of the company is not good at allocating the company's resources to do that with an eye on long-term results.
Trying to transition a research org into a product org is going to be needlessly painful, especially since the research org needs to be firing on all cylinders in this hyper-competitive space.