Now, taking GPT-4 as an example, LLM development relies on excessive costs and equipment, and some so-called "we just invented general intelligence" hype is a bit crazy. Now, it's the turn of the open-source community to do what they do best: provide free competitive software with less money and equipment.
OpenAI has raised over $11 billion in funding, and it is estimated that each training run of GPT-3.5 costs $5-6 million. We know very little about GPT-4 because OpenAI has not disclosed much information, but it is conservatively estimated that its cost will not be less than GPT-3.5. LLM startups have obtained over $100 million in Series A financing with huge valuations without owning any LLM intellectual property. The LLM frenzy is in full swing, and funds are flowing.
It seems that only well-funded companies like Microsoft/OpenAI, Amazon, and Google can afford to train models with trillions of parameters. People think that bigger models are better. Is GPT-3 having issues? Just wait for a bigger version! Small companies that want to compete must raise more funds, otherwise, they can only build commodity integrations in the ChatGPT market. The academic community with even more limited budgets is also marginalized.
Fortunately, some smart people see this as a challenge rather than a limitation. Researchers at Stanford University have released Alpaca, a 7 billion parameter model that performs similarly to GPT-3.5's 175 billion parameter model. Since they don't have enough resources to build a training dataset on the scale of OpenAI, they cleverly chose to use a pre-trained open-source LLM called LLaMA and fine-tuned it on a range of GPT-3.5 prompts and outputs. Essentially, the model learns how GPT-3.5 works and proves to be a very effective strategy for replicating its behavior.
The code and data of Alpaca are only allowed for non-commercial use because it uses an open-source non-commercial LLaMA model, and OpenAI explicitly prohibits using its API to create competing products. This creates an enticing prospect of fine-tuning another open-source LLM on Alpaca's prompts and outputs, creating a new GPT-3.5-like model with different licensing options.
There is another layer of irony here: all major LLMs are trained on copyrighted texts and images available on the internet, and they don't pay a penny to the copyright owners. These companies claim to have a "fair use" exemption under the U.S. copyright law, arguing that their use is "transformative." However, when it comes to the outputs of the models they built with free data, they don't want others to do the same. I expect this situation to change as copyright owners become aware and it may eventually end up in court.
This is different from the perspective of restrictive open-source authors who oppose the use of their code for training in products like CoPilot, citing non-compliance with licenses. For individual open-source authors, the issue is proving that their code has indeed been copied and that they have suffered losses as a result. It is difficult to prove this because the model makes it hard to link the output code to the input code (i.e., the author's source code), and there is no economic loss (it should be free). This is unlike commercial creators (e.g., photographers) whose entire business model revolves around licensing/selling their works and can show substantial replication through image repositories like Getty Images.
Another interesting thing about LLaMA is that it comes from Meta. It was initially released only to researchers and then leaked to the world through BitTorrent. Meta's business is completely different from that of OpenAI, Microsoft, Google, and Amazon. It does not sell cloud services or software, so it has different commercial motivations and incentives. Meta has previously opened up its compute designs (OpenCompute) and has seen improvements from the community - it understands the value of open-source.
Meta is expected to become one of the most important contributors to open-source artificial intelligence. It not only has abundant resources but can also benefit from popularizing generative AI technology: there will be more content on social media to generate profits. Meta has released three other open-source AI models: ImageBind (multidimensional data indexing), DINOv2 (computer vision), and Segment Anything. The latter can identify unique objects in images and is released under the highly permissive Apache license.
Finally, we have a leaked Google internal document titled "We Have No Moat, Neither Does OpenAI," which takes a pessimistic view of closed models and community innovation, which has produced smaller, cheaper models that perform as well as or better than proprietary models. While there is no concrete evidence to prove that this document originated from within Google, it does contain this striking chart:
[caption id="attachment_51768" align="alignnone" width="740"] The vertical axis represents the rating of GPT-4 on LLM outputs[/caption]
Stable Diffusion is another example that demonstrates how open-source generative AI can develop faster than proprietary models. The latest version of this project (ControlNet) surpasses the capabilities of Dall-E2. This is the result of numerous attempts and experiments from around the world, progress that is hard for any institution to match.
So we come back to the beginning. Not having too much money and too many resources has sparked clever innovations among ordinary people. This is the era of AI developers.
Source: https://www.kdnuggets.com/2023/05/llm-apocalypse-revenge-open-source-clones.html
Alphabet Q1 paid subscribers reached 350M, fueled by YouTube & Google One. YouTube ad revenue slightly missed at $9.88B amid Premium growth. Total Q1 revenue hit $109.9B, beating forecasts.
Anthropic releases Claude connectors for Adobe, Blender, and Ableton, enabling AI workflow automation and app control. It also sponsors the Blender Development Foundation.
Apple introduces an App Store model splitting 12-month plans into monthly installments. On iOS 26.4+, it lowers upfront costs without reducing total price, adding billing tracking.
Google launches Aadhaar digital IDs in Wallet for India, enabling secure QR/NFC verification. A phased partner rollout targets key sectors, while passport-based credentials expand globally.
YouTube tests "Ask YouTube," an AI tool blending text and video for complex queries. Initially for US Premium users, it supports interactive follow-ups, expanding Google's AI search.
Google grants US DoD classified AI access, joining OpenAI and xAI in defense deals amid Anthropic's dispute. Contractual safeguards against surveillance and lethal weapons remain unenforceable.
Mistral AI launches Workflows, a Temporal-powered orchestration platform for secure, Python-driven enterprise AI automation featuring human-in-the-loop approvals.