"Message claims GPT-5 training to cost $2.5 billion and start next year"

It is reported that the training of GPT-5 will require a budget of $2 billion to $2.5 billion. This training will last for 90 days and will require the use of 500,000 H100s Tensor Core GPUs or other configurations. The training plan is scheduled to start next year.

OpenAI is actively improving GPT-4 by adding some features such as embodiment, agency, Socratic reasoning, knowledge graph, world model, multimodality, planning, semantic interpretability, collective intelligence, control and boundedness, as well as some smaller high-value tasks.

There have been questions about the production scale of H100/A100. Will so many GPUs be enough to support such a massive task? It is estimated that about 1 million H100s will be produced by the end of this year, and 5 million are expected to be shipped next year.

Regarding the cost issue, there is a valid point about the GPUs. Including the cost of these GPUs in the training expenses may be misleading because they will not become obsolete after the training process is completed. The cost of these GPUs alone could be as high as $20 billion.

It is worth noting that the chip manufacturing company Sustainable Metal Cloud (SMC) is currently capable of producing up to 15,000 H100s per month, but they have increased their capacity to about 50,000 units per month.

In terms of power costs, they only account for a small portion of the total computing costs. For example, 6 million kilowatt-hours is approximately equivalent to $1 million.

Acquiring 500,000 H100s by next year seems to be a challenging task, even with Microsoft's support. Additionally, if the training process is indeed as computationally intensive as implied, the cost of the inference process is also a concern.

In terms of Nvidia's market performance in 2023, there are reports that it has achieved a growth of over 100%, surpassing $1 trillion. This growth is mainly attributed to the widespread adoption of Nvidia chips in artificial intelligence applications. However, the restrictions on the sale of high-end AI chips in the Chinese market due to US export restrictions may impact manufacturing and training costs.

According to Tae Kim, a senior writer at Barron's, Nvidia can achieve a profit margin of nearly a thousand percent for each H100 GPU accelerator sold. This means that the market price for each high-performance computing accelerator from Nvidia is approximately $25,000 to $30,000, including a cost of around $3,320 for each chip and peripheral component. The cost analysis is not yet clear, but it is believed to be purely a manufacturing cost issue. Additionally, the research and development costs of Nvidia need to be taken into account, as developing chips like H100 requires thousands of hours of specialized work. However, Nvidia's AI acceleration products have already been sold through 2024, and the value of the AI accelerator market is expected to reach approximately $150 billion by 2027.

The company is benefiting from its infrastructure and product portfolio, but budget and opportunity costs may limit investments in other areas or reduce risks in research and development.