The 15-year Evolutionary History of AI Assistants from Siri to Muse

In 2011, Apple integrated Siri into the iPhone, and for the first time people realized that apart from making phone calls, a mobile phone could accommodate another small tool that talks to you, chats with you and answers your questions.

Fifteen years later, AI assistants are no longer satisfied with just answering questions. They start reading emails, creating documents, writing programs, operating computers, and even continuing to perform tasks after the user leaves.

Over the past fifteen years, five epoch-making products of AI assistants have emerged, which divide the development history of AI assistants into five stages:

Siri enabled machines to understand human instructions, Alexa brought AI into users' homes, Copilot made large models participate in knowledge work, Codex brought AI into production processes...

The recently launched Muse, on the other hand, tries to become an AI assistant that understands you in the long run.

Looking back at these products, we find that each generation of products has solved the key problem left by the previous generation, but also created new problems.

It seems that no one can create a perfect AI assistant, they are just targeted solutions for the needs of each era.

However, one obvious trend we can observe is that AI assistants are evolving from question-answering type to companion type.

And when AI assistants shift from question-answering type to companion type, the value of AI assistant products is no longer determined only by the model capability, but also by what information they can obtain, what tools they can call, and whether humans are willing to grant more permissions to them.

01 The launch of Siri: The sudden rise of voice assistants

On October 4, 2011, Apple held the iPhone 4S launch event with the theme "Let's talk iPhone".

The word "talk" here is a pun, because Apple was about to launch a new voice assistant called Siri at this event.

At the launch event, this small tool answered a series of seemingly simple questions from Scott Forstall, then Apple's Senior Vice President:

Do I need to bring an umbrella today? What restaurants are nearby? What time should I be woken up tomorrow morning...

One of the demos was: "Remind me to call my wife when I leave the company."

In comparison, traditional mobile software requires users to set a specific time, but Siri tries to understand that "leaving the company" is a location-related state, and the mobile phone triggers the reminder after detecting that the user has left a certain area.

This is the first time that an AI assistant has implemented the reminder function through geofencing and context association, and it is also the first time that an AI assistant has entered the mass market in the form of a complete product.

It showed ordinary consumers that computers can not only execute clicks, but also understand the intentions behind language.

It is worth mentioning that Steve Jobs had already resigned as CEO in August of the same year due to illness (Tim Cook took over), and passed away the day after the launch event. It is unknown whether Jobs was satisfied with the debut performance of Siri.

As a matter of fact, Siri was not Apple's original sudden idea. It was derived from the CALO project funded by the Defense Advanced Research Projects Agency (DARPA) of the United States. The SRI team participating in the project later founded Siri Inc., hoping to turn the cognitive assistant in the laboratory into a consumer product.

In 2010, Apple acquired Siri for 200 million US dollars. A year later, it was deeply integrated into the iPhone 4S.

Among them, the key step completed by Apple was to grant the voice assistant the permission to call the operating system.

Before that, speech recognition software only converted voice into text, but Siri could further call the phone, SMS, calendar, reminder, map and search services, thus forming a complete link through a single voice command:

The voice is transcribed into text, the system judges the user's intention, extracts time, location and contacts, and then calls the corresponding function.

This voice interaction technology later also set off a wave of productization in China. At that time, there were roughly two typical productization routes for voice interaction technology in China.

The first one is the technology service route represented by iFlytek.

In March 2012, iFlytek released iFly Input, which was targeted at Siri. At that time, Siri did not have Chinese voice function yet. Seizing this opportunity, iFly Input launched this "Chinese version of Siri". At the launch event, iFlytek's official also demonstrated functions such as making voice calls, sending text messages, launching applications, setting reminders and navigating through voice.

The problems iFlytek faced were not exactly the same as those of Apple.

From a technical perspective, there are a large number of homophones in Chinese. In addition to Mandarin, there are complex accents and dialects, and users often use voice in noisy environments such as streets, cars and stations. Apple tried to build product experience through the integration of software and hardware, while iFlytek paid more attention to underlying speech recognition, speech synthesis and Chinese processing capabilities.

From a market perspective, iFlytek did not have a top-level hardware product system like Apple at that time. Therefore, in 2012, iFlytek joined hands with Huawei, Lenovo, Changhong, Skyworth, SAIC, the three major operators and other enterprises to establish a voice industry alliance, hoping to become the voice technology platform behind mobile phones, automobiles, TVs, home appliances and operators through this alliance path.

It is worth mentioning that China Mobile, which also believed that voice technology might become a new entry point for the mobile Internet at that time, subscribed for 15% of iFlytek's shares for 1.363 billion yuan after watching the demo of iFly Input, and officially became a major shareholder of iFlytek.

China Mobile then launched its own Lingxi voice assistant at the end of 2012 based on the relevant technology of iFly Input.

Baidu, on the other hand, entered this competition from the other end of search.

Search engines have long operated based on the basic logic of "users ask questions, machines find answers". However, Baidu, which also did not master the operating system and terminal hardware, had data of search, maps, encyclopedias, Zhidao, music, application distribution and local life services.

Therefore, at that time, Baidu hoped to move these capabilities from the search box to voice dialogue, so that users can get results without entering keywords.

This story has many similarities with the second search engine war set off by ChatGPT later, except that the starting point of competition was pushed ten years earlier.

In September 2012, at the Baidu World Conference's special session on Baidu voice and image technologies, Baidu intensively demonstrated its own speech recognition, image recognition and other technologies, which were also the first two fields where artificial intelligence was widely applied.

On December 25, 2012, the Android version of "Baidu Voice Assistant" was officially launched. At that time, Baidu gave this product a very Internet-style slogan — "Speak it, make it done".

Compared with assistants that can only execute local commands, the biggest difference of Baidu Voice Assistant comes from search. It is connected to Baidu's years of accumulated data of web pages, encyclopedias, Zhidao, maps and life services, and can directly answer questions about mathematics, stocks, weather and geography.

Because of this, this product also brought another bug. At that time, when using Baidu Voice Assistant, some netizens found that although the product could recognize the voice correctly, it often only opened the Baidu search results, rather than directly understanding the question and organizing the answer.

Such user experience made many netizens think it was just a semi-finished product.

However, Baidu bet on the right technology for this product. The technical selling point Baidu chose for the voice assistant was to apply deep neural network to Chinese speech recognition.

According to the data officially released by Baidu at that time, after adopting the deep neural network technology, the error rate of speech recognition of Baidu Voice Assistant decreased by 25%.

Regarding the deep neural network, it is worth mentioning that Baidu participated in the secret auction of DNNresearch founded by Geoffrey Hinton and his students in 2013. In the secret auction participated by Microsoft, Google, DeepMind and Baidu, although Baidu offered a sky-high price of 44 million US dollars, Hinton finally chose Google due to various concerns.

After this auction, Baidu became more convinced of the importance of artificial intelligence technology, and established the Institute of Deep Learning in 2014, with Robin Li personally serving as the first dean, establishing Baidu's AI strategy.

As for the intelligent voice assistant, around 2014, Baidu recruited Jing Kun, who had participated in the R&D of Microsoft Xiaoice, to join the search product team, and the search team incubated a new type of intelligent voice assistant, which is later known as Duer.

However, after Siri brought voice assistants into the public view, due to the early technical bottlenecks, intelligent voice technology eventually failed to become an entry-level technology in the first ten years of the mobile Internet era. It mostly brought a technical vision:

After the machine can understand human instructions, what else can we do next?

02 Alexa and the "Smart Speaker War": The entry and outlet of smart homes

In November 2014, Amazon released a new device with a strange shape — a black cylinder about the size of a potato chip can.

It has no screen and no complicated buttons, but it can be placed in the living room or kitchen, waiting for the owner to call its name all day long. Its name is Alexa.

Many years later, Amazon founder Jeff Bezos recalled in his letter to shareholders, "No consumer asked us to make Echo. If we asked users in 2013 whether they would like to put a black cylinder that they can chat with in the kitchen, most people would probably refuse..."

Echo was not a user demand discovered through market research, but Amazon's attempt to recreate the computer in Star Trek. Users do not need to take out the device or learn to operate the interface, as long as they speak directly in the room, the computer can understand and respond.

The most representative scenario Amazon found for Echo is the kitchen. A user whose hands are covered with flour originally wanted to open the mobile phone to set an 8-minute timer, but with Echo, she does not need to unlock the mobile phone, just shout "Alexa, set an 8-minute timer for me".

In addition, music, weather and news can also be called through Alexa.

This makes Alexa distinctly different from Siri:

At that time, Siri was more of a function in the mobile phone, while Alexa tried to become a part of the room. It is not a software waiting for users to open, but a computing entry that always exists in the home environment.

As the computing entry of smart home, the concept of smart speaker began to take shape.

In order to make this product smoothly enter ordinary users' homes, Amazon first had to solve the problem of far-field speech recognition.

Echo needs to recognize the wake word from the sounds of music, TV and conversations several meters away, then judge the sound direction through microphone array, beamforming and noise reduction technology, send the subsequent instructions to the cloud, where speech recognition, intention judgment and service invocation are completed, and finally the device gives the answer.

As for the accuracy of speech recognition, Bloomberg reported in April 2019 that Amazon employed thousands of employees and outsourced personnel around the world to listen to the conversations between users and Alexa, manually transcribe, proofread and complete data annotation.