Large models with nearly one trillion parameters can now run on desktop devices, and Apple and NVIDIA have jointly upended the existing landscape of the AI PC sector.
When we talk about personal AI, on-device models are an unavoidable topic. However, current on-device models can barely meet the AI operation requirements for relatively complex tasks, and the reason is simple: the scale and capability of the models are still limited by hardware constraints.
Nowadays, the parameter scale of cutting-edge large models has moved towards the trillion-level. For example, the publicly announced parameter count of Kimi K3 is as high as 2.8 trillion. Even with 4-bit quantization (which sacrifices part of the accuracy to reduce performance requirements), the model weights alone require about 1.4TB of storage space, and additional memory overhead is needed during actual operation.
Most mainstream consumer-grade graphics cards only have 8/12GB of video memory, and a small number of high-end graphics cards have no more than 16/24/32GB of video memory. As for professional graphics cards with higher video memory, the deployment price of the entire system usually starts at six figures, which is enough to make people "deterred at the sight of it".
Therefore, for a long time, Lei Technology has held the view that there are no large models on the device side, but this view is now outdated. Because NVIDIA launched RTX Spark earlier and Apple launched Mac Studio later, the two companies simultaneously challenged the goal of "on-device large models".
The real AI PC is here
Some time ago, Meta officially released Muse Spark Open, an open-weight model with 816 billion parameters, and Apple used the MLX framework to make this model run full inference on a cluster composed of M5 Ultra version Mac Studio. In terms of parameter count alone, Muse Spark Open is in the first echelon of large models with publicly available parameter counts at present.
Image source: Apple
Although the top-spec Mac Studio is not cheap, it can at least be placed on your desk, and you can deploy a model with nearly a trillion parameters on it. This point alone is quite magical. In comparison, NVIDIA's approach is not that exaggerated, and their RTX Spark laptop only integrates a 120 billion-parameter model into the computer.
In the past, everyone was thinking about how to "shrink" the model to fit into the video memory; now, the giants directly break the old rules and move the "AI supercomputer" to the office desk. From the perspective of Lei Technology, this basically announces that personal AI PCs have officially entered the "era of hundred-billion parameters".
Some readers may ask at this point: "Didn't the 816 billion-parameter model have to run on an H100 server cluster before? How can it be fit into a desktop Mac Studio?"
The core lies in the combination of the MoE architecture and high-bandwidth unified memory. What does that mean? Simply put, although Muse Spark Open has 816 billion parameters and 256 expert networks, it only dynamically activates about 42 billion (42B) parameters each time a Token is generated.
Image source: NVIDIA
This is like a super hospital with 256 specialist doctors. Each time you see a doctor, only 8 corresponding experts are needed for consultation. The single calculation amount is close to the 42B-level model, but it can take advantage of the huge parameter scale of the 816B model.
However, MoE has a fatal threshold: all the weights of the model still need to be stored in the memory. It can save the calculation amount, but it cannot save the memory capacity. Therefore, Apple and NVIDIA have targeted the same solution: integrate the CPU and GPU into one chip, and then use super-large unified memory to accommodate the model weights.
Apple uses 2 M5 Ultra version Mac Studios equipped with 512GB of unified memory, connected via Thunderbolt 5 to form a distributed inference cluster, and uses the total 1TB unified memory capacity of the two devices to hold the 816B model. Moreover, on this basis, parallel inference can be performed through four Mac Studios. In the test announced by Apple, the generation speed of four machines exceeds 36 Token/second, which is sufficient for many daily development scenarios.
Although it cannot compare with the speed of some cloud models, local deployment allows sensitive data to stay on your own device as much as possible, and it can also be used without an internet connection, which is very attractive for practitioners in fields such as finance. As for ordinary developers, to be honest, cloud computing power is cheaper, because deploying an M5 Ultra version Mac Studio with 512GB of memory is expected to cost more than 100,000 yuan, and the specific price will be announced by Apple at the end of the month.
Many people may feel a pinch on their wallets after reading this. What if you don't buy a Mac Studio, but buy an RTX Spark laptop instead? It is much cheaper, but the 128GB version still costs 40,000 to 50,000 yuan. Although the current memory price is too high, this premium is indeed quite staggering.
What is the point of spending hundreds of thousands to deploy a large AI model?
So here comes the question: Since the cost of deploying large models on the device side is so high, why should we promote it? What is the use for individual users?
To be honest, if you only do daily tasks such as translation and copywriting, cloud APIs are indeed the optimal solution. However, when your demand for AI goes deeper, cloud AI may not be able to meet all your requirements. Take programming as an example. As the scale of code increases, the Token consumption generated by AI Agent during the task will also increase sharply, a large part of which comes from context review and testing.
These additional inference costs are not low, especially when you also need to carry out continuous testing and monitoring. But if a 120B or even 816B model is deployed on the device side, the situation will be completely different. Developers can split these trivial inference tasks to the on-device model, and only assign complex tasks to the cloud. In this way, the marginal inference cost can be greatly reduced, and there is no need to pay API fees for each call.
Image source: Apple
In addition, for tasks related to local computer and data security, such as home smart device management, local data analysis, and data sorting with high confidentiality requirements, deploying on-device models can reduce the risk of data leakage.
For lawyers, financial analysts, and some programmers, these investments may be completely worthwhile. After all, once data is leaked, the loss is often far more than the cost of the equipment.
In addition, in the field of AI video creation, on-device models can save more cloud generation costs. Creators can deploy the video model locally, then generate videos repeatedly at a low marginal cost. After getting a satisfactory video clip, they can upload it to the cloud AI to generate a complete video with more details and better effects, which greatly reduces the early "trial and error" cost.
For ordinary users, there is really no need to bother with large-parameter on-device models, but for users who use AI as a productivity tool, this is just an investment.
NVIDIA pursues portability, while Apple goes all out for computing power
From the perspective of Lei Technology, NVIDIA actually wants to take the route of "all-around mobility + sinking of CUDA ecosystem". NVIDIA integrates a custom Arm CPU and Blackwell GPU into a single-chip SoC on RTX Spark, and puts it into thin and light bodies such as ASUS ProArt PX16 and Lenovo YOGA Pro 15.
At the same time, it cooperated with Microsoft to provide the native PyTorch-CUDA version for Windows on Arm on the first day of launch (supporting 97% of popular Python packages), and completed multi-scenario adaptation including local AI Agent (such as the hit OpenClaw "AI Lobster"), ComfyUI RTX Video super-resolution frame interpolation, and DLSS 5 games.
With the support of NVIDIA PAIR technology, the laptop can also be turned into a local area network "AI inference server" at home for other computers to call. These capabilities directly meet the needs of mobile developers. In short, RTX Spark is prepared for developers who do not develop in a fixed place most of the time and have high requirements for data security.
Image source: ASUS
In NVIDIA's vision, RTX Spark PC can be used as a portable assistant for developers to run on-device large models. At the same time, its energy efficiency performance is better than many traditional high-performance laptops, it has good battery life when used on the move, and it can also take into account the needs of creation, games and other scenarios, that is, universality.
Apple is different. It actually takes the heavy-asset route of "extreme energy efficiency ratio + desktop cluster expansion". Apple does not expect you to carry this device that can hold the 816B model out, but hopes that it can become an alternative to traditional enterprise-level servers in some local AI inference scenarios.
In fact, for small enterprises and individual studios, this set of equipment is quite attractive. After all, compared with expensive GPU servers and supporting maintenance costs, a Mac Studio cluster that costs two to three hundred thousand yuan is not that unreachable.
Whether it is RTX Spark or Mac Studio, they both prove one thing: "Super-large unified memory" is becoming an important direction for the evolution of AI PCs. Although at the current stage, due to the rise in memory prices and chip costs, the price of such products remains high, based on past experience, as the entire industry continues to promote unified memory technology, the cost of related products is expected to gradually decrease.
However, at present, these "real AI PCs" are still "toys" for professional developers, heavy creators, and wealthy geeks. Among them, Apple's multi-machine parallel solution is definitely exclusive for "high-spending players", and the 512GB version will not be available until the end of the month. Although NVIDIA RTX Spark starts at $2599, the 128GB full-powered version that can smoothly run the 120B model generally costs about 40,000 yuan in China.
At present, the first batch of models such as ASUS ProArt PX16 RTX Spark and Lenovo YOGA Pro 15 RTX Spark have been launched on the market. The pricing of Apple's 512GB M5 Ultra version will also be announced at the end of the month. If there is an opportunity later, Lei Technology may get an ASUS ProArt PX16 RTX Spark back and do a review for everyone. Stay tuned.
This article is from the WeChat official account "Lei Technology", written by Lei Technology, and published with authorization by 36Kr.