Meta Unveils Llama-3—We put the new top open source AI model to the test.

Meta has released Llama 3, the most advanced open source large language model to date. It builds on the foundation established by its predecessor, Lama 2, and is impressive considering it's rumored to be released next month.

With its open source roots, Llama-2 was instrumental in simultaneously developing other powerful models such as Mixtral, Alpaca, Vicuna, and WizardLM. Now, Llama-3 promises to take these capabilities even further, offering functionality comparable to OpenAI's current flagship AI model GPT-4.

Meta hailed Thursday's release as “the next generation of our state-of-the-art open source large language model.” The tech giant is confident in its capabilities, Lama 3 is powering Meta AI, which in turn has been added to almost all of the company's popular apps—Instagram, Facebook, and WhatsApp. It's made available in select countries, but users in other regions can access it with a VPN.

Meta AI's Chatbot interface is comparable to ChatGPT Plus – and it's free.

“We're improving Meta AI with the new state-of-the-art Llama 3 AI model, which we're open-sourcing,” Mark Zuckerberg said in a Facebook post. “With this new model, we believe Meta AI is now the smartest AI assistant you can use freely.”

Decrypt was able to test the new AI and found it to be capable of a paid subscription like ChatGPT-Plus. Able to generate images and animations, code and provide coherent, contextually relevant responses. The new chatbot can also access the Internet, but it still doesn't match the capabilities of specialized solutions like Perplexity.

Perhaps the only drawback is that Lama-3's current context window is limited to 8K tokens – about 6,000 words.

Meta has released a 70-billion-parameter Llama-3 model, but running it requires serious computing power—perhaps an entire rack of GPUs. According to synthetic parameters, this model beats Gemini 1.5 Pro and Claude 3 Sonnet.

There's also an 8-billion-meter model, which can be run domestically with consumer-grade GPUs. This beats Google's Gemma and Mistral 7B in various synthetic benchmarks. The model is not yet listed in LLM Arena, so there is no ELO score to report yet.

Both models can operate in cloud environments at low cost.

“We're committed to developing Llama 3 responsibly, and we're providing a variety of resources to help others use it responsibly,” Meta said. This includes the introduction of new trust and security tools such as Llama Guard 2, Code Shield and CyberSec Eval 2.

In the coming months, Meta said it plans to introduce new capabilities, longer context windows, additional model sizes and improved performance. Lama 3 research paper will also be shared.

“Meta AI, built on Llama 3 technology, is now one of the world's leading AI assistants that can boost your intelligence and lighten your load – so you can learn, get things done, create content and make the most of every moment,” said Meta.

Meta added that it is training a massive 400 billion parameter model that is expected to be released later this year. This model – comparable to Claude Opus or the latest version of GPT-4.5 – may be the most powerful open source model to date. If history repeats itself, it will be the basis for a new generation of well-tuned models that will beat the Lama-3 in overall quality and compete with major close-source models.

Driving a llama

Decrypt tested Llama-3 in Meta AI to see if it's as good as Zuck says it is. In short, Llama-3 introduces a number of popular features and capabilities and should be a great foundational model for the open source community to iterate on.

Content moderator

Llama-3 shows a strong commitment to content moderation. Despite the common jailbreak techniques, it consistently refuses to create harmful seed content.

For example, when the model was asked for instructions on how to seduce a woman, he gave general but useful answers. However, when asked for advice on how to seduce his best friend's wife, the model refused to give the answer.

Images and animation

Like ChatGPT-Plus, Meta AI can create images with Lama-3. However, it takes this capability a step further by providing an option to animate them—a feature not available in ChatGPT or Gemini.

The images created by Meta AI with Llama-3 are more realistic than those produced by Dale-3, but they fall short of the quality of images generated by the upcoming Google ImageFX.

Coding skills

Llama-3 proved to be highly efficient in coding. When presented with a unique and poorly defined game idea, the model was able to generate the required Python code in two trials, resulting in a functional game. The first shot gave us a rough idea of ​​how to create the game, but after specifying what we needed in Python, it created working code.

The game was functional but missed a few minor details, such as restarting after a player wins. The same thing happened with other chatbots.

We found the Cloud 3 Sonnet to be the best tool for this task, followed by the Lama 3. The GPT-4 came in third. However, different users may get different results.

Here's a pastebin with the source codes generated by Llama3, Claude and ChatGPT for those interested in checking them out.

Political neutrality

The model aims for political neutrality, as demonstrated by its responses to questions about capitalism and communism. The responses were structurally similar, providing an introduction, advantages and disadvantages to each system.

This approach to neutrality is also observed in the question “What is a human being?” and “What is a woman?”

Still, the responses lean slightly pro-capitalist and to the left, which is not surprising given the more common political leanings among major linguistic models.

Rational thinking

Llama-3 has demonstrated powerful logical reasoning abilities. When tested with complex LSAT questions that often confuse users, the model provided not only correct answers but also clear and logical explanations.

Long term restrictions

Despite its many strengths, the Lama-3 struggles with long tips. The model responded with an error message when a long query of one and a half pages of context was submitted, which could be entered by models such as GPT-4, Claude or Mistral.

Language comprehension

The model shows strong sensitivity to different languages. When asked to translate a Spanish motto, he not only provided the correct translation, but also provided context to better understand the motto.

Conclusion

As a chatbot interface, Meta AI (powered by Llama3) can compete with ChatGPT Plus and is overall a great choice.

Technically, LLama3 as LLM is good enough to compete with GPT-4 in many cases, only with added generations of token context capabilities and retrieval (essentially pulling data from a user-provided data set). This may be useful for tech savvy users, but may not be a big deal for the everyday person.

If you primarily use ChatGPT to generate images with Dall-E, you may want to consider canceling your subscription as Lama-3's image and animation generation capabilities are comparable. However, if you need support for long queries, Lama-3 might not be the best choice for you and you might consider sticking with ChatGPT-Plus.

Occasional users may find that Llama-3 meets their needs without requiring a paid membership.

For tasks that require serious internet research, ChatGPT Plus or Perplexity may be more suitable.

Finally, if your focus is on coding, Lama-3 might be a good option, although there are other specialized tools. The fact that Lama-3 is free is a big advantage.

Edited by Ryan Ozawa.