Meta launches biggest AI model ever

Back in April, Meta revealed it was building an industry-first AI model that would be the biggest open-source model the industry had ever seen, and would rival industry favorites, like OpenAI’s GPT and Anthrropics Claude, in performance. That day has come: Llama 3.1 is Meta’s biggest AI model and the largest open-source model in the industry, to date. 

What can Llama 3.1 do? 

Although Llama 3.1 isn’t multimodal, meaning it can only handle text, not images, audio or video (although it can analyze PDFs and spreadsheets), it can complete coding tasks, answer math questions, summarize documents in 8 languages (English, German, French, Italian, Portuguese, Hindi, Spanish and Thai), and integrate with search engine APIs to “retrieve information from the internet based on a complex query” (something which Meta itself has done, by integrating Llama 3.1 into its AI assistant (a ChatGPT-like chatbot which resides on Facebook, Instagram, and WhatsApp).

Llama 3.1 training 

Llama 3.1 has been trained on the same datasets as Meta’s previous models, but Meta has applied “more rigorous” quality assurance and data filtering approaches to finetune and improve the model. The dataset comprises a mix of non-English data (to improve multi-lingual performance), mathematical and code data (to improve its mathematical and code reasoning skills), and recent web data (to improve its knowledge of current events).

Meta hasn’t confirmed exactly where it got its training data from, only establishing it used a mixture of synthetic data (data generated by other AI models) and public web pages and web files, claiming their training data was “the secret recipe and the sauce that goes into building these models”, in a bid to give itself—like many other AI firms—a competitive advantage.

Although, critics believe that the real reason behind the secrecy is to try and avoid the impending copyright lawsuits that will inevitably come their way, especially after recent reports that it uses Facebook and Instagram posts, photos, and captions and makes it hard for users to opt-out.

Llama 3.1 performance 

Llama 3.1 was built using NVIDIA chips and leverages new training and development methods which Meta says makes it perform better than other, similar-sized models, like ChatGPT-4o and Claude 3.5 Sonnet, across a range of tasks including general knowledge, math, tool use, and multilingual translation. 

(Plus, Meta reckons it will cost about half of what it costs to run GPT-4o).

And with over 405B parameters (ChatGPT-4o has over 200B) and a larger context window (which means it can summarize longer pieces of text and is less likely to ‘forget’ previous conversations), it should perform better, overall, especially when it comes to problem-solving.

However, in early benchmark tests, it achieved “mixed results”, compared to GPT-4o and Claude 3.5 Sonnet: While it was better at coding and generating plots, it was weaker in programming and general reasoning.