How to secure industrial automation

IT infrastructure, including software that supports the business processes of industrial environments (e.g., production machinery, pumps, railroad equipment), generally referred to as OT (Operational Technology) systems, were originally designed to operate 24/7 in “isolated" networks. Today, they are being integrated with IT systems, remote maintenance, cloud services and third-party vendor solutions, dramatically changing the risk profile.

Developing appropriate procedures and a practical approach to the security and business continuity of OT systems in an organization should become an area of systemic action, just as it has previously happened in IT environments.

Most concerns about industrial automation vulnerabilities are related to the risk of external interference. However, our experience with OT network cyber security deployment projects, including those based on the National Cyber Security System Act, indicates that a significant portion of incidents are internal in origin: failures, unintentional changes, configuration errors, operator inattention. It should be assumed that the expected amendment to the KSC Law will reflect these observations.

Already the first NIS directive implemented in Poland through the KSC Act placed great emphasis on risk analysis and incident response in this regard. So, it is important that preparations for the implementation of the new regulations, inventories, risk analysis, response plans and the OT environment monitoring tools that are being implemented allow parallel management of incidents and detection of operational anomalies resulting from failures or human error (business continuity).

When talking to decision makers (including at the board level) about security planning, emphasize the importance of both groups of threat sources and the need to monitor and prevent them.

While there is a growing conviction among decision makers that OT networks need to implement security systems, these projects often raise legitimate concerns. In many companies, the OT infrastructure is outdated and integrating it with modern IT systems can involve significant risks. However, this is an increasingly desirable measure. Not only for security reasons, but also because of the drive toward ever more automation and remote access to OT systems needed to reduce infrastructure maintenance costs.

Another concern has to do with whether IT professionals who are tasked with supporting their automation colleagues in this task, which is new to them, understand the world of automation.

Change awaits both sides. In IT, there are certainly major learning needs and lessons to be learned about what they will face as they enter the OT world. OT teams need to be aware of the need to adapt to IT requirements. Additionally, in companies burdened with the requirements to comply with a growing number of increasingly restrictive regulations at the EU and national levels, it is necessary to clearly communicate support for change from the board level.

It is worth noting that OT environments require a different kind of security measures, often of a lower scale of complexity compared to modern IT security technologies. This makes mainly passive incident detection measures applicable, while in IT the foundation is active interference with network traffic. The very static nature of OT architectures and the highly repetitive content of network traffic affect the simpler nature of incidents, which we will also detect as a direct consequence of what risks we identify.

We outline the key differences between IT and industrial automation solutions that determine the security architecture for these systems.

• Priority: IT protects data and service availability, while in OT networks, the risks are to the health and lives of plant personnel, the quality of the process affecting the quality of the product, and therefore often consequently the health and lives of customers;
• Network traffic: in IT it is sometimes chaotic; in OT it is stable and predictable – consequently, it enables anomaly detection (statistical analysis/ML);
• Maintenance and interruptions: in IT, maintenance windows are scheduled; in OT, they are sometimes infrequent or difficult to negotiate;
• Patching: IT needs to be updated; OT often runs old systems without patching (including through process dependencies);
• Protection measures: in IT we react actively (locks, automations), in OT we mainly use passive monitoring and procedural measures;
• Performance: in IT, high throughput is required, but some latency is accepted; in OT, throughput may be lower, but no latency is accepted (real-time operation);
• Maintenance: in IT, it is easy to change infrastructure maintenance providers; in OT, it is difficult, often the provider has a monopoly, limiting access to infrastructure;
• Theme owner: IT maintenance and development are usually the responsibility of the CFO (as business owner of the ERP system) and CIO, industrial automation is usually the domain of the COO;
• Lifespan: in IT, the lifespan of infrastructure is typically 3-5 years; in OT, 10 or even 15 years. Interestingly, many plants are still building new OT segments based on very old but proven network standards and protocols.

/en/offer/implementation-of-the-nis-2-directive/EU, national or industry regulations(NIS2, KSC, ISO 27001/22301 standards, TISAX) will increase pressure on the implementation of the risk management and incident response project in an OT environment. From the perspective of practical experience, we can say that the biggest, often difficult to estimate potential delays may arise from formal and contractual barriers (e.g., maintenance service monopoly, lack of vendor approval for additional monitoring) and technical barriers (original OT network architecture prevents monitoring).

Waiting for more regulations to come into force while failing to plan for adjustment activities is also unreasonable because some of them require a long time to implement changes. It is worthwhile to get down to removing obstacles and making those adjustments that will be necessary anyway, and often turn out not to generate significant costs. It is worth noting that, regardless of the details of the new Polish law on KSC, it can be assumed that the foundations of the obligations imposed on the new group of companies were already present in the regulations previously in force for other industries: the need to perform an inventory of resources, analysis of vulnerabilities and risks, preparation of an incident response plan.

Practical experience from similar projects allows us to formulate conclusions and recommendations.

• Get sponsorship at the level of plant/production directors, the company’s board of directors (taking into account who feels ownership of a particular group of risks);
• Renegotiate maintenance contracts: require rights to access infrastructure, make copies of traffic, use network probes;
• Be prepared to handle incidents. Separate responsibilities: business continuity on the automation side, cyber threats – IT/security team or external support (outtasking for forensics analysis, for example).

During the preparation stage of an industrial automation security project, we usually encounter similar obstacles, regardless of industries or company size. The most important of these are:

• Unmanageable switches with no free ports – no ability to make copies of traffic or report on the contents of the MIB, netflow;
• Lack of documentation or outdated documentation of topologies and protocol modifications;
• Contractual limitations of automation suppliers and its maintenance service (e.g., inability of third parties to implement the project);
• Reluctance to interfere – fear among automakers that monitoring will interfere with production (hence the requirement for full passivity and galvanic separation of monitoring means).

An example from the project: the documentation from OT does not reflect the real architecture of the network, the devices present in it – the devices in the copy of the traffic (pcap) can be seen more than “on paper". Conclusion: always verify documentation by performing network traffic analysis.

Based on our experience, we have prepared a methodology for conducting such projects, in which the following steps can be separated at the stage of building the foundation for future compliance with new regulations.

Step 1. Inventory of assets (IT platforms and network elements: PLCs, inverters, operator panels, sensors, server applications, diagnostic and monitoring devices), performed on the basis of traffic recording (PCAP/flow).

Step 2. Identify vulnerabilities and limitations (without scanners). Use vendor documentation and domain knowledge. Rarely (and preferably in a lab setting) actively scan for vulnerabilities.

Step 3. Mapping infrastructure to processes and assessing their level of criticality – essential for risk assessment.

Step 4. Risk analysis, incident response plan (along with selection of incidents you want to detect). This determines the choice of tools (sometimes open-source is enough; to detect anomalies in OT protocols you need probes that understand those protocols).

Step 5. Monitoring architecture design and implementation (selection of monitoring methods as a consequence of incident selection).

Step 6. Incident handling. Implement basic infrastructure monitoring with an incident response plan. Determine incident recipients to handle: business continuity (OT) alerts, cybersecurity incidents (IT/SOC/external partner).

Step 7. Education. Infrastructure and incident insights allow teams to build new requirements, expand monitoring, prepare the company for new regulations, but also look for benefits in other areas – such as predictive maintenance.

We have previously written that implementing OT infrastructure monitoring often does not require large budgets. A good example of this is the relationship between the incident sought and the method of event detection.

For detecting simple type events:

• A new device has appeared in the OT segment,
• A new unprecedented session between endpoints was initiated,
• The loss of movement between devices,

netflow collectors based on open source solutions are often sufficient.

Complex events such as:

• Anomalies in industrial protocols (e.g., non-standard commands, unusual frequency),
• changes in PLC registers (detected through passive monitoring – in a copy of the movement),
• Traffic profiles that deviate from the “stable" baseline,

require the use of more advanced tools (like a commercial OT probe with machine learning mechanisms).

Already during the implementation of the first step mentioned above, i.e., inventorying the equipment and looking into the traffic copy, it is possible to discover architectural errors, the analysis of which results in a recommendation to make an immediate change.

Example from the project: in the OT segment, a CCTV recorder was detected setting up an encrypted session to China. Recommendation: correct the rules on the firewall (block unwanted outbound connections).

• Netflow collectors: the simplest and cheapest to implement method for continuous network inventory and incident detection (e.g., unknown device, unknown session, network traffic outage, etc.);
• Surveillance system: Often also open source-based monitoring of the contents of MIB registers and SNMP tracer collector; the ability to detect infrastructure failures, as well as more detailed inventory (LLDP or SNMP gives, for example, the ability to record the software version of devices);
• Passive probes/IDS for OT. Monitoring while maintaining galvanic separation from the monitored network; support for legacy physical interfaces, modified protocols; traceability of register states;
• Network traffic recorder: open source can be used here as well. Enables very detailed network and traffic inventory, and post-event analysis capability with a copy of the traffic. Recording of RDP and SSH sessions makes it possible to track administrators’ activities;
• Firewalls, separation, DMZ. Analysis of copies of network traffic often ends with a recommendation to change the configuration of firewalls;
• Switches: managed switches enable traffic copies, collect data in MIB registers, send netlow reports, etc. Replacing old switches with manageable ones is one of the most cost-effective investments in the implementation of OT network monitoring projects.

Project practice suggests that the decision of which measures from the list presented above to implement is best preceded by an analysis of network traffic, an inventory of the infrastructure and a selection of the incidents you want to detect. This helps significantly reduce investment costs by avoiding spending money on technical measures that will not be used properly.

It is worth noting that the same technical solutions we use to detect cyber security incidents in OT networks are used in the business continuity management process: we identify events preceded by human error or failure. Thus, the project of industrial automation monitoring is worth justifying not only the need to ensure compliance with regulations, and the mere removal of technical and formal obstacles, although sometimes time-consuming, does not carry a large investment expenditure.

A well-prepared inventory will reduce costs. The choice of the type of incidents you want to manage determines the choice of tools. Passive monitoring does not pose risks to the industrial automation infrastructure. It is also worth remembering that detected incidents should be managed, so it is necessary to develop an incident response strategy.

• Don’t actively scan OT networks with IT tools, use passive means
• Start with the data: PCAP, netflow, MIB content
• Revise maintenance contracts (agree to access copies of traffic)
• Start building a response plan before investment – you’ll save on investment and prepare your team for additional responsibilities
• Build competence both ways: IT learns OT’s limitations; OT learns detection tools and their value in maintenance practice