Elon Musk Rarely Backs Down: 𝕏 to Open-Source Recommendation Algorithm, Self-Mocks It as “Terrible” but Promises Monthly Updates

Right now, you can fully view Elon Musk's open - sourced 𝕏 Recommendation Algorithm System on GitHub.

The open - source files clearly state that this is an algorithm system almost entirely driven by AI models.

We removed all manually designed features and most heuristic rules.

As soon as the news came out, the entire community immediately boiled over. The top - voted comment was a flurry of praise:

Incredible! No other platform can be so transparent.

Elon Musk himself quickly retweeted the original post from the 𝕏 engineering team. However, the usually high - profile Musk said modestly this time:

We know this algorithm is dumb and needs significant improvement, but at least you can see in real - time and transparently our efforts to improve it.

No other social media company has done this.

Even before acquiring 𝕏 (formerly Twitter) in 2022, Musk had criticized the platform for being too closed - off on multiple occasions.

Since the acquisition, he has fulfilled his promise and made the core Twitter recommendation algorithm public several times. This time, he is still staying true to his original intention.

So this is how a purely AI - driven recommendation system works!

Without further ado, let's dig into the operating mechanism of the entire system.

In a nutshell, this system can be described as follows:

Built on the same Transformer architecture as Grok - 1, it can determine what content to recommend to you by learning your historical interaction behavior (what you've liked, replied to, or retweeted).

When a user opens the "For You" page, the client sends a request to the server, triggering the entire algorithm process.

Then the system will first figure out who you are, what you've been doing recently, and what kind of content you usually respond to.

To achieve this goal, the system will pull two types of user information:

Action Sequence: This type represents the most direct and strongest interest signals, such as what you've recently liked, replied to, retweeted, clicked into, or spent time on.

Features: This type represents long - term attributes, such as your follow list, declared interest topics, geographical location, and the device you use.

The goal of this step is not to manually construct features, but to build a "real - time user profile" as realistically as possible.

In the past, engineers might assume that "certain attributes are important" and then manually write rules or formulas to calculate a "user interest score".

But this is essentially the engineers' conjecture, not a reflection of the users' real state.

So Musk's algorithm decides not to make any preset assumptions. Instead, it collects as much raw and real user behavior data as possible and feeds this data directly to the subsequent models, allowing the models to learn and discover patterns from the raw data by themselves (i.e., "de - humanization" and "end - to - end").

After obtaining the real - time user profile, the system will split into two paths to quickly filter out thousands of "potentially relevant" tweets from the massive number of tweets on the entire platform.

One path is through your acquaintance circle. That is, directly fetch the latest tweets from all the accounts you follow through the Thunder module.

The other path is through external sources. Use the Phoenix Retrieval, the core retrieval module, to fetch tweets that you might be interested in but are from accounts you don't follow.

The above two types of information from different sources will be treated equally in the subsequent stages.

It should be noted that at this point, what has been filtered out are just tweet IDs.

So the system will use the Hydration module to complete the information of each candidate tweet, including the full text of the tweet, author details, images/videos, and historical interaction data, for subsequent in - depth evaluation.

Moreover, before the formal calculation begins, the system will further eliminate obviously unwanted content through the Filtering module, such as:

Duplicate or expired posts

Content posted by the user themselves

Posts from blocked or muted accounts

Content containing keywords blocked by the user

Posts that have been viewed or displayed in the current session

Subscription content that the user has no permission to access

Remember, this step only does one thing: answer whether a piece of content "can appear, rather than whether it is worth recommending".

After all the preparation, the remaining content will be sent one by one to the Phoenix sorting model for scoring.

This model is a Transformer - based model that will simultaneously receive:

The user's action sequence and attribute information

The content and author information of a single candidate post

Then the model will predict the probability of the user performing various actions on a tweet and combine these probabilities with preset weights (for example, positive actions like liking add points, while negative actions like blocking subtract points) to form a final sorting score.

Based on this, the system will also make some minor engineering - level adjustments:

For example, control the diversity of authors to prevent a single account from occupying too high a proportion in the news feed (to prevent a single influencer from flooding the feed).

It should also be noted that to ensure that each post is scored independently, the system specifically sets the rule that "candidate posts are not allowed to'see' each other" (there is no cross - attention mechanism between tweets).

All candidate posts are sorted by their final scores, and the system selects the Top - K posts as the recommendation results for this request.

Moreover, before returning the results to the client, the system will conduct a final round of verification to ensure that the content complies with the platform's security regulations:

For example, remove any tweets that have been deleted, marked as spam, or contain illegal content such as violence and gore.

Finally, after going through multiple rounds of screening, the information will be displayed to the client users in descending order of scores.

In summary, the five key factors for the successful operation of this system (as highlighted by the official) are:

(1) Purely data - driven, rejecting manual rules.

Completely abandon the complex manual rules for defining "what content is good" and let the AI model learn directly from the raw user data.

(2) Adopt a candidate isolation mechanism for independent scoring.

When the AI model scores the content, each piece of content "cannot see" other candidate content and can only see the user information. This ensures that the score of each post will not change due to other posts in the same batch, and the scores are consistent and can be efficiently cached and reused.

(3) Hash embedding for efficient retrieval.

Both retrieval and sorting use multiple hash functions for vector embedding lookup to improve efficiency.

(4) Predict multiple behaviors instead of a single score.

The AI model does not directly output a vague "recommendation value" but predicts multiple user behaviors simultaneously.

(5) Modular pipeline to support rapid iteration.

The entire recommendation system uses a modular design, and each component can be independently developed, tested, and replaced.

"Yes, this algorithm is terrible."

However, although people have expressed their appreciation for Musk's open - source gesture, this algorithm still has some "flaws".

Some netizens complained after the 𝕏 recommendation algorithm was open - sourced:

Due to limited and costly API access, the practice of using block lists is now rare, but it was very common in the past.

The algorithm must make older block lists gradually fade over time so that these older block lists will not be maliciously exploited.

What they mean is that the algorithm code shows that "being blocked by a large number of users" is a strong negative signal, which will directly lead to an account being "down - weighted", meaning its content is less likely to be recommended. However, there is no clear time - decay mechanism for the "block" signal in the code.

This means that historical block records may still be affecting an account's recommendation score to this day.

This statement also attracted Musk himself to comment in the comment section:

Yes, this algorithm is terrible.

Anyway, Musk's attitude towards change is clear:

Not only has he been open - sourcing in the past and is doing so now, but he will also continue to do so in the future. The open - source update will be repeated every 4 weeks in the future.

Open - source repository: https://github.com/xai-org/x-algorithm

Reference links: [1]https://x.com/elonmusk/status/2013482798884233622[2]https://x.com/elonmusk/status/2013496642851279270

This article is from the WeChat official account "QbitAI". Author: Focus on cutting - edge technology. Republished by 36Kr with permission.