you2idea@video:~$ watch E6E49TLZiiA [14:28]
// transcript — 196 segments
0:01 what most differ in Strawberry from other conversational AI is the ability
0:05 to think before responding rather than immediately answering a query so the two
0:07 people who have tested the model so two people hopefully open the eye that
0:12 thinking stage usually lasts 10 to 20 seconds they said but there are other
0:15 key differences for one thing the initial version will only be able to
0:19 take in and produce text and not images which means it isn't multimodal the way
0:23 other open a models are as most large language models released today are
0:26 multimodal this seems to be noticeable shortcoming the decision to release it
0:30 as text only could reflect the pressure opening ey is feeling to release
0:33 products as it faces more competition now they say this is going to come out
0:37 in two weeks which would align with opening ey Dev day that I was trying to
0:43 attend are you not going no I had I had a person uh put an application for me
0:47 who worked inside and they they said no we can't we can't do it there's not
0:49 there's not enough space and I was like I get it if only I had a million Subs um
0:56 so so then they mentioned there's pricing strawberry is likely to be
0:58 priced differently in opening ice chatbot which is free and subscription
1:01 pricing TI we're not sure exactly how strawberry will be priced but likely
1:05 have rate limits restricting users to some maximum number of messages per hour
1:09 the potential for a higher price tier that's faster to respond according to
1:13 another be like Sora they'll just do a really great demo and then never never
1:16 see it again exactly what we be looking at y'all is it could be a thirst trap again they
1:22 could say hey strawberry Deb day oh yeah you're not going to see this until
1:25 another 3 or four years now and also GPT for you're not going to get the voice
1:30 voice six years from now do we have a mug that says AI thirst strap on it Joe
1:35 that's I'm WR I'm writing that down you should probably have one that just has a
1:39 big strawberry on it and then underneath it says AI thirst trap AI thirst trap
1:44 that's a on the back of it you could list all of the product announcements
1:48 that never really came true yeah you have Gemini on the top of the list oh
1:51 it's going to be so powerful we're have to change the game it's G it's going to
1:55 be terrible oh you know I think Sora is at the sora's on there sora's on there
2:00 so let's go to the other thirst trap which is reflection 70b so reflection 70b
2:08 so they have this guy Nam mat Matt Schumer he said I'm excited to announce
2:12 reflection 70b the world's most top open source model trained using reflection
2:16 tuning a technique develop to enable nlms to fix their own mistake the 405
2:21 billion version is coming next week uh we expect it to be the best model in the
2:25 world and so I I remember when that when that happened and he said that I was
2:32 like uh wait a minute like who is this guy he doesn't have a whole data center
2:37 behind him um he's not working for any the top Labs let's really quickly and I went and
2:44 looked at his background and was like okay CEO and co-founder other siiz AI
2:51 hyper writer were hiring um and then so he is a Syracuse University doing
2:56 entrepreneurship and then founder of a couple companies but not in the tech
2:59 side and then all of a sudden he's coming out saying we're coming out with
3:03 a model that's better than most of these top tier model creators and it's going to
3:11 change everything so I kind of maybe like scratched my head did did you have
3:14 that initial impression when that happened Joe when you saw this come out
3:19 well I saw the the announcements and bit a self-promoting and my reaction was if
3:30 the model's really that great you're not going to have to promote it you're just
3:33 going to put it up on the leaderboard and people are going to lose their minds
3:37 and sort of the way open AI does you could just give it a mystery name and
3:40 people will be talking about what is this model but what I didn't realize was
3:46 the reaction wasn't so positive because people got to try the model and saw that
3:51 it was fantastic it was positive because he announced it with a bunch of uh
3:56 Benchmark results that were really fantastic so people were going on the
4:00 benchmarks and saying my God this thing looks great for its size and then they
4:04 wanted to try it and they were overwhelming the the servers at his
4:07 company where he where he was hosting the model and so then his company was in
4:11 this position of trying to add more servers more capacity so more people
4:16 could try the model and I think for quite a while it was just a Feeding
4:20 Frenzy no one was actually able to use it right and so that was you know that
4:25 worked very well for the promotion so I don't know if that was on purpose but it
4:30 it certainly fed into it right exactly and you have almost like a tulip B Mania
4:35 basically like I gotta get in I gotta get yeah there was there was like a lot
4:40 of fomo a lot of excitement yeah and he was also teasing that you know they were
4:43 working on a larger version of the open source model and if the 70 billion one
4:50 was this good the 450 billion one must be even more spectacular so it was
4:54 further sort of promoting things that were coming what I didn't realize to go
5:00 back to the to the start of the this he I didn't realize that those Benchmark
5:05 numbers were only numbers that he had posted I thought that since the model
5:08 was available on a hugging face that they would do their own evals right that
5:13 these were verified by third party you know it's like when I want to try well
5:16 yeah it's like an elementary school kid gets all FS instead just turns them A's
5:20 hey look look what I've done here Mom all A's um but you're also talking about
5:27 top tier folks who got uh got taken on by us Boris power head of Applied
5:30 research Open the Eyes said I got fooled by the reflection 70b announcement too
5:34 long didn't read the model performs very badly and then this gentleman here
5:39 Stannis love Fort Stanford PhD and AI in Cambridge physics says the 99.2 test
5:45 accuracy on the GSM AK alone was extremely suspicious gbd4 Plus Code
5:49 interpreter gets only 96% so um then this guy ended up making
5:55 the rounds and Matthew Burman had him on the show like immediately so we're gonna
6:00 watch a Quick Clip here let's let's push to theune itself yeah I mean I've had this idea
6:06 like I mentioned for for many months now and different forms of it um so it was
6:11 sort of like how do you enable that for everyone without them needing to figure
6:14 it out themselves but even if they wanted to figure it out themselves
6:16 without a fine tune it doesn't work super well you know prompting like like
6:20 sah said can get you a marginal gain like you definitely will see better
6:22 performance but it's not going to be the performance jump we saw I mean we saw 10
6:27 plus percent uh jumps in many benchmarks I mean it was it was taking llama 70b
6:32 and bring it past 45b and much further beyond that I mean we're expecting to
6:35 see with 45b is going to be hopefully insane um so yeah it's really that this
6:40 can't be done with prompting alone if you really want to get the full force of
6:44 it um this is more close to this is closer to how people think I think there
6:48 are many more Gams we can make there and it was just one of those stupid obvious
6:53 ideas that no one really thought to do right I think a lot of the major Labs
6:56 they're they're way too in the weeds in many ways they're thinking too
6:59 technically and too intelligently about this where sometimes the simple thing is
7:03 what actually is going to work I love that you know we they're
7:07 spending too much time reading the research papers and actually training
7:12 these models and working reality but us random Outsiders could see things
7:16 differently and come through with a breakthrough what happened with cold
7:23 fusion you know people made the announcements everyone got excited but
7:26 then no one could duplicate the results including the original researchers uhoh
7:32 and so now we like well did they did they never have the results or did they
7:36 somehow have a fluke or did they honestly misread their results which is
7:40 possible with a complex you know yeah system so so everyone was like okay how
7:45 did we get here how did how did they think they had such great results such
7:49 that they could publish them and people reviewed their paper and no one thought
7:54 to you know step byep verify what they were saying and we're sort of in the
8:00 same boat here like it's basically fog of War we don't really know if they ever
8:05 got great results if they just made it up or if they uh maybe misread it they
8:10 they tricked themselves and thought they were getting great results you know like
8:14 there's all kinds of possibilities but we still don't know exactly it reminds
8:21 me um about the story about some new space engine they were making and this
8:25 research made made it through we saying oh we we've discovered this new way of
8:29 doing propulsion and everyone was super excited about it but when they did their
8:33 calculations they didn't uh adequately I think measure for gra some aspect of
8:36 gravity which gave them like a false positive but that story went everywhere
8:40 and people were things were just blowing up completely um but for those
8:44 researchers it was a honest mistake compared to what it looks like a Matt's
8:47 doing everything which is grift that's going on here you know or like that one
8:51 lady who was doing Parkinson's research and came out and basically was
8:55 falsifying her statistics and everything purposely so and the sad part is he
9:00 might actually have something really interesting just a way of fine-tuning
9:05 the model that gets you a few percentage points of improvement across a broad
9:10 array of benchmarks if that's true that's fantastic we should take whatever
9:14 approach he came up with and add it to what we already do and make improvements
9:18 across the board for all the models that's sort of the spirit of this sort
9:22 of research and if if it's not this can be applied and consistently demonstrated
9:27 then maybe he made a mistake and his he thought he had a gain and he didn't
9:31 that's also possible but the way it's going now is the claims were so
9:37 aggressive so over the top that if it turns out that that nobody can verify it including him he's
9:46 just going to get crucified exactly and he's people are going to assume bad
9:50 faith right and he has been silent on Twitter for the last two days so then
9:54 people assume him being silent is him knowing he's been caught in a lie or
9:57 something when you're in those type of situations you would immediately come
10:01 out and be like hey actually we screwed up here I'm sorry for the hype this is
10:05 what we learned from the process but all hey I want to do a quick update so
10:15 Matt Schumer the one who was talking about this new model uh just tweeted at
10:20 2:31 p.m saying I got ahead of myself when I announced this project and I'm
10:24 sorry that that was not my intention I made a decision to ship this new
10:28 approach based on the information that we had at the moment I know that many of
10:31 you are excited about the potential for this and are now skeptical nobody is
10:35 more excited about the potential for this approach than I am for the moment
10:38 we have a team working tirelessly to understand what happened and we'll
10:41 determine how to proceed once we get to the bottom of it once we have all the
10:45 facts we will continue to be transparent with the community about what happened
10:50 and next steps um it just doesn't look good you you think that if they would
10:55 come out with a technique that's vastly better than what these other um weal
11:01 funed organizations are producing they would at least have some type of third-
11:05 party verification before doing a splash everywhere in the media
11:10 um uh Maddie Lane says impulsive and unchecked product delivery is a mistake
11:14 in leadership it should be done it should be once you have all the facts
11:18 then you can be more transparent and don't add we to a u situation take
11:24 ownership you know mistakes are made I think it's fair but I still think that
11:28 third party verification is so important and him seeing all you know when Google
11:34 decided to look at their Gemini model and uh takes IND independent um prompts of to the Gemini
11:42 model what's going on this picture and then getting a response and then
11:45 stringing it together to make it look like it was one seamless uh Gemini was
11:51 watching just natural video and coming together with thoughts Google got scour
11:55 uh got attacked for that rightfully so so you would think that Matt and crew
11:58 would be like hey we don't want to be fall that same scenario and be raked
12:04 over the coals um and then uh sahill the person who was working with him on this
12:08 project um did a longer response saying I want to address the confusion and
12:11 valid criticisms that this has caused in the community I'm currently
12:14 investigating what happened that led to this and we'll share a transparent
12:17 summary as soon as possible there are two areas I like to address which I'm
12:20 investigating first I want to be clear that at no point was I running any
12:24 models from other providers as the API that was being served my compute I'm
12:27 working on providing evidence to this and understanding why people Model
12:31 Behavior such as using a different tokenizer or completely skipping words
12:35 like Claude second the benchmark scores I shared with Matt haven't been
12:39 reproducible so far I'm working to understand why this is and if the
12:42 original scores of reported were accurate or result of contamination or
12:47 misconfiguration so again third party independent verification is critical in
12:51 this because then they would be able to say hey we also um showed it to so and
12:56 so and they ran the same test and they came back the same results and so
12:59 were're in good shape that's what a lot of these um researchers do um going back
13:05 to the tweet I have a lot of work to do on both of these and I'm working on a
13:07 full post morm that will share with the community sorry for the confusion that's
13:10 caused and I know that I've let the community down and lost trust I still
13:14 believe in the potential the approach my focus is on rebuilding trust through
13:17 increased transparency I'll have more of Shar soon and then Tim Tru said no I'm
13:22 not buying this Matt was telling people to do their model benchmarks against
13:26 your private API and it scored amazingly yet you have no model anywhere near that
13:29 level the only explain ation is routing it to Claude and then other providers
13:34 apparently tired of lies um so yeah super interesting but just
13:40 again this goes to show how hungry people are for improved um language
13:46 models so uh anyways thank you for watching and uh tomorrow is our bonus
13:51 episode we are going to be talking about um some re a recent model that Google
13:57 released which is using uh which is could be used to um help
14:02 um create simulations for proteins um that can work for Medical compounds to
14:05 help to help find the right protein and delivery mechanism for new drug then
14:09 we're going to go into a llm that was trained on weather data then we're going
14:12 to go on some tips on just how to network as a tech professional uh if
14:17 You' like to see that then um become a supporter by going to patreon.com svic
14:21 or hitting the join button on YouTube and become a channel supporter um also
14:24 here's two other videos you can check out um hope you enjoy talk to you later
$

Matthew Berman Hypes Matt Shumer's Fake AI Model & OpenAI Strawberry Launch Rumors

@MattShumer 14:28 8 chapters
[AI agents and automation][content creation and YouTube][community building and audience growth]
// chapters
// description

Patreon.com/svic for access to our private discord and Ad Free Hour-Long bonus episodes. Follow us: Threads: www.threads.net/@svicpodcast Twitter: https://x.com/svicpodcast LinkedIn: https://www.linkedin.com/company/svicpodcast Tiktok: https://www.tiktok.com/@svicpodcast Instagram: https://www.instagram.com/svicpodcast Spotify: https://podcasters.spotify.com/pod/show/svicpodcast Need consulting help on M&A or do you have questions about AI? Reach out to Jordan on intro - intro.co/JordanThibode

now: 0:00
// tags
[AI agents and automation][content creation and YouTube][community building and audience growth]