Can a factory still remain operational when it is hit by cyber extortion?

Recently, several factories have suffered hacker attacks and been extorted. Cybersecurity is no longer a distant event in news reports, but an operational risk that manufacturing enterprises need to face seriously.

In the past, when enterprises talked about digitalization, they mainly focused on how much efficiency could be improved and how much labor could be reduced by systems such as ERP, MES, WMS and QMS. However, when these systems suddenly become inaccessible, a more realistic problem arises:

If the enterprise's business system suddenly crashes tomorrow, how long can the factory still operate normally?

Production plans cannot be queried, inventory quantities cannot be confirmed, equipment programs and process documents cannot be accessed, inspection records cannot be traced, and customer orders cannot be delivered. At this point, what the enterprise is facing may not just be an IT failure, but the disruption of production, quality, supply chain and even the entire business activities.

Why can a single cyberattack bring the entire enterprise to a halt?

Imagine an enterprise that is suddenly hit by a ransomware attack. Some servers and business systems become inaccessible, order, inventory, finance and business data are affected, and many daily operations are forced to be interrupted.

The day before, the enterprise was still producing and delivering normally; the next day, the management found that many operations that they used to take for granted could no longer be completed.

There are materials in the warehouse, but it is impossible to confirm which ones have passed the inspection; there are work-in-progress products in the workshop, but it is impossible to query which processes have been completed; finished products have been packed, but the necessary inspection and traceability records cannot be obtained.

Although the enterprise still has employees, equipment and materials, it is difficult to maintain the original business rhythm due to the inability to obtain reliable information.

The management has to face a series of problems: which businesses must be stopped? Which ones can run temporarily? How to arrange customer delivery? If the system cannot be restored in the short term, how long can the enterprise persist?

What hackers lock is not only the enterprise's data, but also the management order that the enterprise relies on for normal operation.

The system is restored, but why may the business not necessarily be restored?

When an enterprise is hit by a cyberattack, the first thing it thinks of is often to contact IT personnel, find security experts, and restore servers. However, restoring the system is not the same thing as restoring the business.

For example, if ERP is restored but MES is not, the connection between production plans and actual production records may fail; if MES is restored but equipment programs, process parameters or inspection records have not been confirmed, there are still risks in production; if the warehouse management system is restored but the inbound and outbound records during the downtime have not been checked, the system inventory may be inconsistent with the physical inventory.

Therefore, what the enterprise needs to restore is not just a set of software, but a group of interrelated business capabilities.

The management should identify key businesses and their system dependencies in advance, clarify which activities must be restored first, which can wait, and what conditions need to be met after restoration to resume operation.

The IT department can restore the system, but business departments such as production, quality and logistics must jointly confirm whether the restored system can truly support normal business operations.

We have data backups every day, why may we still fail to restore the system?

Many enterprises believe that as long as they perform automatic backup every day, they don't need to worry too much about ransomware.

However, if the backup server uses the same management permissions as the business system, or is connected to the same network for a long time, the backup data may also be attacked.

Even if the backup files still exist, it does not mean that the business can be resumed immediately. The enterprise also needs to confirm whether the backup is complete, whether the version is applicable, whether the recovery environment is secure, and whether relevant applications and interfaces can operate normally.

For example, if the order database is restored but the corresponding drawings, process documents or equipment programs are not restored, production still cannot be carried out normally. If the system is restored to the state of several days ago, but purchase receipts, production reporting and customer delivery have taken place during this period, there may be discrepancies between the data and the actual business.

Therefore, backup management should not only focus on backup frequency, but also consider isolation protection, access permissions, historical versions and recovery tests. Enterprises also need to clarify how long interruption the key business can withstand, and what scope of data loss is acceptable.

A truly reliable backup does not prove that the data has ever been copied, but proves that the enterprise can resume business with trusted data.

The system is unavailable, can the factory switch to a temporary management mode?

After the system goes down, some enterprises may propose: "We will first record with Excel or paper forms, and then re-enter the data after the system is restored."

But for manufacturing enterprises, this is not a complete emergency plan.

For example, when the quality system is inaccessible, which version of the inspection specification should the inspectors use? Where are the inspection results recorded? How to maintain the traceability of product batches? Who has the authority to approve release?

When the warehouse management system is unavailable, how to prevent repeated outbound operations? How to confirm material batches? How to keep accounts consistent with actual inventory? Who will check and re-enter the data after the system is restored?

If these problems are not clarified in advance, new quality risks may arise during temporary operation.

Therefore, enterprises need to determine in advance which work can be operated temporarily under controlled conditions and which must be suspended according to business risks, and clarify the responsibilities, records, approval requirements and check requirements after restoration.

For links that cannot meet the requirements of safety, quality and traceability, forced operation cannot be carried out just to maintain production output.

Emergency operation can change the operation mode, but cannot arbitrarily reduce the key quality and safety control requirements.

The system is back online, why can't production and delivery resume immediately?

The risks caused by cyberattacks do not necessarily end when the servers go back online.

Enterprises also need to confirm whether the restored data is accurate, whether key files have been modified, whether the equipment programs are valid versions, and whether the system records are consistent with the actual on-site status.

For example, if the system shows that a batch of products has completed inspection, but the inspectors cannot obtain the corresponding original records, the products cannot be simply deemed qualified based on the system status.

Similarly, if the restored process documents are inconsistent with the programs actually used by the equipment, or the temporary records during the downtime have not been checked, the process may get out of control.

Therefore, before resuming production, in addition to having professionals confirm the system security, relevant business departments also need to confirm key contents such as drawings, BOM, process specifications, control plans, inspection records, inventory and product traceability.

For products whose quality status cannot be confirmed, measures such as isolation and assessment should be taken according to risks, and data missing cannot be covered up by retroactively signing records.

The fact that the server can be logged in again only means that progress has been made in technical recovery; whether products can continue to be produced and delivered needs to meet corresponding quality release conditions.

Why can't cyberattacks be handled only by the IT department?

Cybersecurity experts can figure out how to isolate attacked devices, protect evidence and restore systems, but it is difficult for them to decide on their own which orders should be prioritized, which production activities must be suspended, and how to explain delivery delays to customers.

These decisions involve production, quality, procurement, logistics, sales and finance, and require unified coordination by the management.

In the absence of unified command, the production department hopes to resume operation as soon as possible, the quality department requires supplementary verification, the sales department has promised customers the delivery date, while the IT department still believes that relevant systems do not meet the conditions for safe launch, which will easily cause new chaos.

Therefore, enterprises should establish an emergency decision-making mechanism in advance, clarify who is responsible for initiating the emergency response, who is responsible for technical disposal, who is responsible for business recovery, who communicates with customers, and who makes decisions when major disagreements occur.

Cyberattacks may start as technical problems, but returning to normal operation requires the participation of the entire organization.

The most important thing enterprises should do is to organize a real emergency drill

Many enterprises have established information security systems and emergency plans, but have never verified whether these plans are truly feasible after the system crashes.

You can set up a scenario: at 8 a.m. on Monday, ERP, MES and warehouse management systems are all inaccessible, there are urgent orders to be delivered on that day, and there are a large number of work-in-progress products in the workshop.

The management, production, quality, logistics, sales and IT departments discuss together: who initiates the emergency response? Which businesses must be stopped? How to confirm the product quality status? How to establish temporary records? Who approves the resumption of production and delivery? How to check the data after the system is restored?

These drills do not need to actually interrupt the production system. You can first carry out desktop deduction, and then professionals conduct recovery tests in an isolated environment.

Drills often expose some problems that cannot be seen normally: emergency contacts are no longer available, backup files are too old, different departments have inconsistent recovery priorities, or temporary records cannot be associated with system data.

If these problems are discovered in daily time, they are opportunities for management improvement; if they are discovered when the enterprise is actually attacked, they will increase the difficulties in the recovery process.

The value of emergency drills is not to prove that the enterprise is ready, but to find out in advance which parts are not yet ready.

The deeper digitalization goes, the more we need to consider what to do after the system fails

The cyber extortion suffered by these factories makes us rethink an important issue in the construction of enterprise digitalization: the system should not only consider how much efficiency can be improved during normal operation, but also consider whether the enterprise can control risks and resume operation after major exceptions occur.

As systems such as ERP, MES, WMS and QMS are more and more deeply involved in production organization, quality control and business decision-making, cybersecurity, data integrity, business continuity and quality management can no longer be separated from each other.

Enterprises need not only professional cybersecurity protection, but also a clear cross-departmental emergency mechanism; they need not only reliable data backup, but also verified business recovery capabilities.

A truly mature digital management not only enables the enterprise to run faster when everything is normal, but also enables the enterprise to know how to control risks, make decisions, and gradually resume normal operation even when it is hit by major impacts.

Because for a manufacturing enterprise, the most important thing is not only whether the system can be restored, but whether the enterprise still has the ability to control the business after the system fails.

This article is from the WeChat Official Account "TPP Management Consulting", author: TPP Management Consulting, published with authorization from 36Kr.