Dig into Tencent Cloud's AI storage products
Over the past year, the whole society has paid more attention to storage than ever before.
Driven by the AI wave, storage demand has surged, and prices have also risen sharply.
Previously, for the development of AI, people mainly focused on computing power, algorithms, chips and large models. Nowadays, with the continuous implementation of AI, data and the storage base behind data are constantly highlighting their value.
Not long ago, the Tencent Global Digital Ecosystem Summit was held in Shenzhen. At the conference, industry leaders have pointed out that the development of AI has entered a brand-new stage, and the important symbol of this stage is that Agent is becoming a new form of productivity.
Building Agent infrastructure with high performance and extremely high cost-effectiveness is an important foundation to support this development trend.
Large models and Agents are accelerating their transition to productivity. As a key part of the underlying infrastructure, the entire storage system must launch a system-level reconstruction covering media, hierarchy, architecture and software stack.
The underlying logic of storage technology has changed completely. It is no longer a simple competition between capacity and speed, but has evolved into an in-depth game centered on data lifecycle, access patterns and total cost of ownership.
Next, based on the views of industry leaders and combined with my own understanding, I will talk about this transformation and reconstruction.
When traditional storage meets AI, a revolution is inevitable
The theoretical foundation of modern computers is the von Neumann architecture. In this architecture, computing and storage are the two most core parts.
Von Neumann Architecture
Modern storage technology started in the 1950s. After more than half a century of evolution, from magnetic tapes to optical discs, from mechanical hard disks to solid-state drives, from single-machine storage to distributed storage, storage media and forms have been iterated continuously.
Behind all iterations, the core mission of storage has never changed — to store data and read it out when needed.
To ensure that data can be saved and read in time, storage systems have always been pursuing two core indicators: performance and capacity. Performance mainly refers to the speed of data reading and writing.
Essentially, storage is a well-designed "data hierarchical circulation system". From registers, caches, memory, to SSDs, HDDs, and then to offline media such as magnetic tapes and optical discs, each layer uses different speeds, capacities and costs to carry data of different popularity. The higher the performance of a layer, the higher its cost and the smaller its capacity.
Storage Hierarchy System
In the era before the arrival of AI, this hierarchical system ran very smoothly — all layers could basically meet application requirements, the data popularity model was relatively stable, and the cost was also under control.
However, the arrival of AI has completely broken this balance.
Large model training and inference tasks put forward completely different requirements for storage from traditional Internet scenarios.
Large model training requires continuous reading of a large number of samples, which puts higher requirements on storage throughput and data supply efficiency; HBM on GPU undertakes high-speed data access during model calculation. Inference scenarios not only need memory to support context and cache, but also need storage systems to carry model files, knowledge bases and session data. Different links have different requirements for capacity, bandwidth and latency.
Agent applications, which have been extremely popular in the past two years, also pose new challenges to storage. When an Agent executes tasks, it will continuously accumulate files, memories and intermediate results. The execution environment can be started and recycled on demand, but these working data need to be retained independently for subsequent tasks to continue using. At the same time, data of different users and Agents need to be isolated, and can be shared according to permissions during collaboration, so that work results can be saved safely and reused continuously.
In short, in the AI era, the upgrading of algorithms and computing power requires the collaboration of more powerful storage systems. The in-depth development of AI requires the construction of large-scale and high-quality data supply.
The popularity distribution of data no longer follows the slow attenuation rule of the past Internet era, but presents complex features of high frequency, burst and cross-layer. The traditional logic of "simple hot and cold layering" begins to fail, and the storage system must rethink a series of fundamental issues: where the data should be placed, in what way, and when to migrate.
How to balance the relationship between capacity, performance and cost, and how to build storage products that truly meet the needs of scenarios, has become a difficult problem facing the entire storage industry. The solution of these problems will directly affect the data preparation time, computing power utilization and operating cost of AI tasks, and then affect the efficiency of enterprises' large-scale application of AI.
Breaking the storage dilemma, what exactly does Tencent Cloud plan to do?
When dealing with the above challenges, Tencent Cloud's answer is "Reconstruction + Innovation" —
Instead of simply pursuing the ultimate performance of a certain layer, they redesign the form and layout of storage products around the flow law of data in the whole lifecycle. For different AI scenarios such as training, inference, data aggregation, Agent and embodied intelligence, on the one hand, they reconstruct on the basis of existing traditional products to further improve product performance, on the other hand, they have launched more innovative products to make up for the shortcomings in the existing product matrix.
Through the improved product system, each type of data can be efficiently accessed by appropriate computing power at an appropriate cost on a suitable media. In other words, it is not only to "store enough", but also to "supply fast", "retrieve accurately" and "afford to use".
Next, let's take a detailed look at what upgrades Tencent Cloud has made to traditional products and what innovations it has launched.
Storage Product System Hierarchy
From the perspective of layering, Tencent Cloud has carried out in-depth refinement on the basis of traditional simple layering, which is divided into extremely cold layer, cold layer, online warm layer, online hot layer and acceleration layer.
● Cold layer, online warm layer, online hot layer (COS tiered storage)
COS (Cloud Object Storage) is the general name of Tencent Cloud's object storage service. It is the most core unified base for massive unstructured data, adopting Tencent Cloud's self-developed Yotta architecture. The storage scale of a single cluster can reach hundreds of EB, the bandwidth can reach Tbps level, and the number of objects in a single bucket can reach trillion level.
On the basis of COS, it is further divided into COS Standard Storage, COS Express-Lite, COS Infrequent Access/Archive/Deep Archive, corresponding to the online hot layer, online warm layer and cold layer respectively.
It is worth noting that COS Express-Lite is a product newly launched by Tencent Cloud not long ago, targeting massive warm data that needs to stay online for a long time.
It uses QLC (a type of NAND Flash semiconductor storage technology) as the underlying media, does not occupy the scarce HDD production capacity. The unit price of per GB storage is about 1/4 of that of high-performance storage, and there is no retrieval fee threshold, which can provide online capacity with sufficient performance and controllable cost.
In terms of compatibility, COS Express-Lite supports S3 compatible access, which can reuse existing data pipelines and tool chains to reduce access transformation costs.
● Extremely cold layer (tape storage)
In the extremely cold layer, Tencent Cloud adopts tape storage. Although this form of storage looks very old, it is a very low-cost and reliable method for extremely cold data (such as logs, historical camera data).
Tencent Cloud's tape storage adopts the latest self-developed tape engine technology, leading the industry in writing speed, with a total storage of more than 2000PB.
Under the background of rising storage prices and tight media supply, tape storage can take over part of the space vacated by HDD and SSD, effectively reducing the overall TCO (Total Cost of Ownership).
● Acceleration layer (GooseFS)
Let's look at the very critical acceleration layer. At this layer, the corresponding solution presented by Tencent Cloud is GooseFS (Data Accelerator Goose File System).
GooseFS is located between the computing framework and the underlying persistent storage (COS/CHDFS/HDFS, etc.). Through caching, reading and writing acceleration, it collaborates with the underlying persistent storage to improve data access efficiency, solving the pain points of COS in big data and AI training scenarios such as slow metadata, small files, random IO and repeated pulling.
For example, during the large model training and inference process, datasets, validation sets, pre-trained weights and vector libraries are repeatedly read by multiple GPUs. There are many small files and very dense random IO (Input/Output) operations. With GooseFS, the cache hit rate can be improved, the waiting time of GPU can be reduced, and the bandwidth pressure at the COS layer can be reduced.
GooseFS also has some "variants". For example, GooseFS-Cache can reuse the local NVMe of GPU to create a global cache. Another example is GooseFS-X (fully managed high-performance parallel file), which can provide distributed metadata acceleration, achieving hundreds of GB throughput per second, millions of IOPS, and sub-millisecond latency.
GooseFS provides high-throughput data supply and on-demand reading and writing, while COS provides massive persistence and elastic expansion. The collaboration of the two can well cope with the data access challenges in AI training and inference scenarios, achieving a balance between performance and cost. That is to say, different types of data can find a suitable location to be stored.
● COS Rapid Bucket
Facing a large number of GPUs fetching data in parallel, Tencent Cloud plans to launch a new product — COS Rapid Bucket (high-performance storage bucket, in internal test).
COS is the entire object storage service, while Bucket (storage bucket) is the container under COS for storing objects.
COS Rapid Bucket is oriented to AI training, inference and high-performance computing scenarios, and can provide high-concurrency and low-latency data reading and writing capabilities. In other words, its task is to ensure that data can be sent to the computing end in time.
COS Rapid Bucket uses all-SSD media, and deploys computing and storage in the same availability zone nearby to shorten the data access path. According to the public product information of Tencent Cloud, COS Rapid Bucket has million-level IOPS, millisecond-level latency and TB/s-level throughput capabilities, which can be used for training sample reading, model checkpoint writing, and model weight loading when inference services are scaled out.
● CBS and CFS (CFS Turbo)
In terms of product form, Tencent Cloud also has CBS and CFS.
CBS is Cloud Block Storage, which provides block-level storage to carry data for computing environments such as cloud servers and containers.
CFS is File Storage, which supports NFS, SMB and POSIX protocols, and allows multiple machines to share and read/write files.
It is particularly worth mentioning that the CFS Turbo version has been deeply optimized for AI training scenarios, positioned as high-performance parallel file storage, and supports all-flash parallel data plane.
The newly launched TurboS3 by Tencent Cloud enables the parallel file system to access the S3 ecosystem, realizing that one copy of data supports three access methods (POSIX, HDFS, S3), unifying namespace and distributed metadata, fully releasing the performance of parallel storage.
Storage Technology Innovation for Agent Scenarios
For the underlying technical framework system we mentioned earlier, what we see is mainly the upgrade and enhancement of traditional storage. For new AI scenarios, relying only on upgrade is not enough, more depends on product innovation, that is, launching some customized new products to meet the needs of scenarios.
The most typical one is of course the currently most popular AI Agent scenario.
Agent (Intelligent Agent) is one of the most important tracks for artificial intelligence to move towards industrial implementation. Throughout 2026, from the "Crayfish (OpenClaw)" that became popular at the beginning of the year to the now widely popular WorkBuddy, the penetration speed of Agents is beyond imagination.
More and more employees use Agents like WorkBuddy for daily work, including professional fields such as business analysis, business operation, market research, marketing planning and execution. Enterprise managers also use it to complete more accurate decision-making and faster execution, improving market competitiveness. Agents have become productivity tools that can be used conveniently and truly improve efficiency.
As a powerful digital assistant, the core features of Agent are autonomous planning, tool calling, memory and context management. These features put forward completely different requirements for underlying storage: it is necessary to support high-frequency metadata reading and writing, and also ensure the consistency and low-latency access of context data in multiple rounds of interactions.
Tencent Cloud has launched a number of products such as Agent Bucket, Vector Bucket, MetaInsight (multimodal data management engine), AgentCFS and AgentCBS, covering the full link of Agent implementation, and laying a solid foundation for turning AI into productivity.
Agent AI Storage Infrastructure
We can take a detailed look at what features these products have and how they support the larger-scale popularization of Agents.
● Agent Bucket
Agent Bucket is a new type of storage bucket launched for the AI Agent era, targeting the storage requirements of user profiles, task intermediate results and generated outputs in Agent applications.
Agent Bucket introduces an independent Space under the COS storage bucket, supports configuring permissions, capacity quotas and speed limits according to users, agent instances or projects, and provides capabilities such as directory management, file sharing, deduplication and instant transmission, and recycle bin.
It provides platform-level isolation to prevent unauthorized access, which not only