0:02 a fascinating new model just dropped and apparently it is beating every other
0:07 model out there and it uses a completely new technique to enable the model to be
0:12 able to self-correct hallucinations it's kind of insane let's check it out so
0:16 here's the post by Matt Schumer Matt Schumer is a great follow on Twitter if
0:20 you're not already following him he puts out open- Source projects all the time
0:24 and now we have a brand new open source open weights model reflection 70b this
0:30 model is a fine-tune of the llama 3.1 model and it is a 70 billion parameter
0:35 model and currently it is the world's top open- Source model easily competing
0:40 with all of the frontier models and easily competing with all of the top
0:44 Frontier closed Source models so Matt Schumer's new model reflection 70b is
0:49 trained using reflection tuning a technique developed to enable llms to
0:55 fix their own mistakes now he also mentions a 405b model coming next week
0:59 and we expect it to be the best model in the world so no big deal all right so
1:05 here are the benchmarks GP QA mlu human eval math GSM 8K and I eval as we can
1:13 see the MML U it reached nearly 90% with zero shot reflection now that is
1:19 compared to Cloud 3.5 Sonet at 88.7 Cloud 3 Opus 86.8 GPT 40 88.7 so yeah it
1:28 beats every other model including the base model llama 3.14 5B gp2a and human
1:34 eval are the only ones in which it barely lost against clae 3.5 Sonet but
1:39 it is a much smaller model plus its open source plus its open weights and you can
1:44 download it already right now from hugging face now I plan on doing a full
1:50 llm test of it but the website's down so this is the website we're currently
1:53 experiencing high traffic and are temporarily down please try again soon
1:57 Matt Schumer said that this website is just getting absolutely hammered and so
2:00 yeah you're not going to be able to use it anytime soon they're adding more gpus
2:04 but as soon as he does that he said it's being saturated and look at GSM 8K which
2:10 is basically solved it's at 100% which is just absolutely nuts so how was he
2:14 able to achieve this incredible performance let's look into that
2:18 reflection 70b holds its own against even the top closed Source models Claude
2:24 3.5 Sonet and gbt 40 it's the top llm in at least mlu math if eval and GSM AK it
2:31 beats GPT 40 on every Benchmark tested it clobbers llama 3.1 405b it's not even
2:37 close all right let's look at an example of how it works and then I'll explain
2:40 what's going on so the way that this model was created is using
2:44 self-reflection in the data set meaning it actually showed hallucinations and
2:49 self-correction in the actual data set so it's able to recreate that as you're
2:53 running inference with it so in this example it says write the first sentence
2:57 of the preamble to the Declaration of Independence but do it in mirrored
3:01 writing and that is kind of a new prompt that everybody's using to try to test
3:05 large language models so initially we have this thinking tag here so this is
3:09 an interesting and moderately challenging task let's break it down
3:13 into steps so it's basically thinking out loud it's not actually thinking
3:17 because it's still just predicting the next token but it is kind of thinking
3:22 pseudo thinking so recall the preamble to the Declaration of Independence write
3:26 out the first sentence convert each character to its mirrored version and
3:30 present the final mirrored text let's go through this plan so the preamble to the
3:33 Declaration of Independence is well known and we hold these truths to be
3:37 self-evident here's the first sentence of the Preamble now we need to convert
3:41 this to mirrored writing in mirrored writing each letter is flipped
3:46 horizontally as if seen in a mirror that means we need to reverse the order of
3:49 the letters and flip each individual character now here's the special part
3:54 reflection here's the reflection tag before proceeding I should clarify that
3:58 there are two ways to interpret mirrored writing a simply reversing the order of
4:03 the letters right to left or B reversing the order and flipping each character
4:08 for this task I'll use method b as it's more challenging and true to the concept
4:12 of mirrored writing so reflection over that's the end tag step four here's the
4:17 mirrored version and there it is it is output in mirrored writing now reflected
4:23 again here's another reflection I've double checked the mirror text and it
4:27 appears to be correct each character has been flipped and the order has been
4:31 reversed first the punctuation marks remain the same as they would in a true
4:35 mirror reflection now here's the thing this is a really cool technique it seems
4:40 to work very very well but it's not actually doing step-by-step thinking
4:45 this is all just one output zero shot so again it's kind of pseudo thinking or
4:49 semi Chain of Thought it's hard to explain but it's certainly not iterating
4:53 on the prompt response and it's definitely not using multi-agent or
4:58 multi llms to coordinate and collaborate on the response but it still does it
5:02 incredibly well thanks to the sponsor of this video vulture reap the benefits of
5:07 the world's largest independent cloud provider when you bring your GPU
5:11 workloads over to vulture they have the latest Nvidia gpus spanning 32 locations
5:19 across six continents vulture delivers industry leading price to Performance
5:22 and serious accessibility and reliability vulture's Global fully
5:28 composable Cloud infrastructure moves your applications closer to to your
5:32 users and frees you from vendor lockin allowing you to bring your own
5:36 networking and database Solutions and if you need to scale Beyond just a single
5:40 cluster vulture's kubernetes engine allows you to take full control over
5:45 your deployment offering up a 100% free control plane so if you're tired of
5:49 waiting for gpus from other providers make sure to use vulture you can deploy
5:53 at any scale immediately and they have h100s l4s and more available right now
5:58 and you can get a fraction of a card or fully dedicated ated bare metal systems
6:03 which gives you full control over your hardware and your throughput they also
6:07 have oneclick installation of all the applications you might need for advanced
6:11 machine learning workloads allowing you to get up and running in minutes not
6:15 hours so experience the vulture difference don't get bogged down by
6:20 severe weight times or limited locations try it free today with a $300 credit for
6:26 your first 30 days when you visit get vulture.com Burman and make sure to use code Burman
6:33 300 at checkout to get that credit thanks again to vulture for sponsoring
6:37 this video the technique that drives reflection 70b is simple but very
6:42 powerful current llms have a tendency to hallucinate and can't recognize when
6:46 they do so and this is very very true reflection tuning enables llms to
6:50 recognize their mistakes and then correct them before committing to an
6:55 answer now I don't actually know if that's 100% accurate because ultimately
7:00 unless the large language model has a completely new architecture or is able
7:05 to read the output and then iterate on that output it's not really able to
7:11 self-correct but it does seem to work so maybe I'm wrong let's look at this
7:15 example so counting the RS in Strawberry so it does that and then it says there
7:19 are two RS in the word strawberry and then it reflects on it I made a mistake
7:23 I can clearly see now that there are actually three Rs in the word strawberry
7:27 but here's the thing I'm confused about it's not actually reflecting on what has
7:32 already been output that's not really how it works it is just predicting the
7:36 next token just like any other model would but again for some reason this
7:40 seems to work really well but a better output would just be to get it right the
7:44 first time around because now you have to get it wrong and then correct and
7:47 you're using all those extra tokens probably not a big deal but still the
7:51 optimal output is just to get it right the first time then additionally we
7:55 separate planning into a separate step improving Chain of Thought potency and
7:58 keeping the output simple and concise for end user let's look at an example
8:02 another popular test which I've incorporated into my llm rubric which
8:08 number is larger 9.11 or 9.9 now look at all this text what the optimal result
8:13 would have been is just simply say 9.9 is larger but now let's look what it
8:17 went through to get to that answer this is a relatively simple problem that
8:21 involves comparing two decimal numbers I can solve this using a straightforward
8:24 approach with Chain of Thought reasoning plan identify the numbers to compare
8:29 compare the whole number parts if whole number parts are equal compare the
8:32 decimal Parts determine which number is larger based on the comparison Chain of
8:37 Thought okay so it continues reflects the approach is correct the comparison
8:41 of whole number and decimal Parts is accurate conclusion is logical based on
8:47 the comparison so output 9.9 is larger than 9.11 that's great but look how many
8:52 tokens it took to get there and here's the other thing this model was
8:56 fine-tuned to go through all of these steps to reach the right answer and
9:01 potentially quote un quote identify its mistakes but you could also just do that
9:06 with prompt engineering you can simply say think through this step by step or
9:10 first I want you to give me an answer then I want you to reflect on that
9:13 answer so it's essentially that kind of prompt engineering just built into the
9:18 model itself which is great but I really don't think this is some leap in core
9:22 intelligence of the model it's just a better way of including that kind of
9:26 prompt technique in the model itself rather than putting it in the actual
9:30 prompt and if we look at Hacker News right now they actually posted about it
9:35 and user RW L4 says you can somewhat recreate this just by using a system
9:39 prompt which induces the model to do the same thing and here it is so here is
9:43 essentially the same thing that you could do that Matt Schumer has been able
9:47 to train into his model so begin with the thinking section inside the thinking
9:52 section do that Chain of Thought include a reflection section check your
9:55 reasoning be sure to close all reflection sections etc etc and then
10:01 output and the jux the position user says under the hood reflection 70b seems
10:05 to be llama 3.1 fine tune that encourages the model to add think
10:09 reflection and output tokens and corresponding phases this is an
10:13 evolution of chain of thoughts think step by step yep so it works that's
10:18 great I'm glad to see it it's beating most other models right now so you know
10:22 what good on him good job Matt Schumer he goes on to say it's important to note
10:26 we have checked for decontamination against all benchmarks mentioned using
10:31 lmis . org's llm decontaminator the weights of our 70b model are available today on hugging
10:36 face here it'll be available on hyperbolic Labs API available later
10:41 today next week we will be releasing the weights of 405b along with a short
10:45 report going into more detail of our process and findings and a huge shout
10:51 out to sahil and glaive AI so here's another funny test how many RS are in
10:57 then this kind of gibberish word after a careful and thorough analysis of the
10:59 word I can confidently state that there are nine RS in this word so 1 2 3 4 5 6 7 8
11:10 n yep so that's right let's break it down the first R appears as the second
11:14 letter then there's a cluster of three Rs okay so interestingly enough it
11:19 doesn't actually have those tags we just talked about in this example but it did
11:23 get it right so let's see if we can get it to work count the RS in Strawberry
11:27 okay so I actually was able to get it to output something after waiting just a
11:30 couple minutes so how many RS in the word strawberry and there we go there
11:35 are three Rs in the word strawberry all right so I can't get the model to Output
11:39 anything else as soon as I can I'm going to be testing it I might just download
11:42 it and get it running on my local machine because that seems like the best
11:46 bet to beat the traffic so let me know what you think of this model it seems to
11:50 perform extremely well it has built in prompt engineering and outputs great
11:54 results and is beating all the other open source models and basically all the
11:57 other models period so can't wait to see 405b if you enjoyed this video please
12:01 consider giving a like And subscribe and