The headline-generating decision by Elon Musk’s xAI earlier this week to release the weights and architecture of its Grok-1 large language model (LLM) put a spotlight on the ongoing debate within the AI community about the open sourcing and transparency around AI models and created another chapter in his growing feud with OpenAI.
Musk, who launched xAI in early 2023, wrote in a March 11 post on X (formerly Twitter) that the company would “open source Grok” and the announcement was made six days later. In the brief post, xAI wrote that it was releasing the base model weights and network architecture of Grok-1, the new company’s answer to OpenAI’s ChatGPT and Google’s Gemini chatbots.
Key differences include a more sarcastic attitude than rivals and greater availability to the massive amounts of data collected on X. Grok, the AI built on the foundation of the LLM, was introduced in a slow-roll release in November 2023.
Grok-1 is a 324 billion-parameter Mixture of Experts LLM that xAI said was trained from scratch on a large amount of text data using a custom training stack sitting atop of the JAX machine learning framework and Rust programming language.
The company noted that what it released is the “raw base model checkpoint from the Grok-1 pre-training phase” that ended in October, which means that “the model is not fine-tuned for any specific application, such as dialogue.” The weights and architecture are being released under the Apache 2.0 license and can be found on GitHub.
Open Source and AI a Sticky Issue
The question of open sourcing AI platforms and other software is a thorny one, given the huge amounts of data that these LLMs are trained and on the need to ensure the security of the data. At the same time, advocates for transparency – in the AI technologies and how they work – argue that people should know how the systems make decisions given the broad effect those decisions can have.
Rob Enderle, principal analyst with The Enderle Group, noted that there are at lease eight LLMs that are open source and that open source has been relatively common across the ecosystem since 2009, adding that most of the people building LLMs come from a time when open source was a more common practice.
Enderle pointed to the announcement by Microsoft – a big-time investor in OpenAI, to the tune of more than $10 billion over multiple years – in February to license Mistral AI’s platform, which is open source.
He added that Nvidia, the technology leader for AI, is an advocate for open source, saying the platforms can advance more quickly and that it creates a more cooperative ecosystem. Nvidia co-founder and CEO Jensen Huang has said an open approach will help the world get to artificial general intelligence – or AGI, the theoretical point when AI can perform and think as well as a human – by 2029 rather than having to wait until the 2040s.
“This approach helps buyers deploy more quickly because it potentially reduces ramp time for LLMs and SLMs [small-language models] significantly,” Enderle told Techstrong.ai. “Some LLMs, like ChatGPT, are closed, but the reason is that they used training data which has to be protected – in this example medical data – so it would appear that if the training data isn’t protected – and training data from a social media platform isn’t protected data, at least not now – then the open source approach is the most common.”
That said, he said Musk’s decision to release Grok-1 “is likely more because it isn’t ready and Musk wants to make an impression. While Musk talks transparency, he really isn’t very open in practice.”
Musk vs. OpenAI
The issue of openness also is at the heart of Musk’s ongoing dispute with OpenAI – which accelerated the generative AI industry with its release of ChatGPT in November 2022 – and a lawsuit filed by Musk against the vendor. Musk, an early supporter and source of funds for OpenAI since its launch in 2015, is accusing OpenAI and executives like co-founder and CEO Sam Altman of violating a Founding Agreement that said the company would remain a nonprofit and be open about its technologies.
Its 2020 licensing agreement with Microsoft angered Musk, as did the company’s decision to keep the internal design of its GPT-4 LLM secret, contrary what OpenAI did with GPT-3.
Musk also frets that the future of AGI – which he believes, along with others, poses an existential risk to human survival and that GPT-4 represents a significant in that direction – is now in the hands of Microsoft, a for-profit company.