Real-Time Wireless Control Networks

Team

Faculty: Chenyang Lu

PhD Students: Yehan Ma

Past Members: Humberto Gonzalez, Yixin Chen, Rahav Dor, Octav Chipara, Abusayeed Saifullah, Chengjie Wu, You Xu, Bo Li, Dolvara Gunatilaka

---

WirelessHART Network Architecture (Credit: HART Communication Foundation)

Wireless sensor-actuator networks represent a new generation of communication technology for industrial process monitoring and control. With the adoption of WirelessHART, an open wireless sensor-actuator network standard, recent years have seen successful real-world deployment of wireless control in process industries. Industrial control systems impose stringent real-time and reliability requirements on wireless control networks. We aim to develop real-time protocols and analysis of wireless sensor-actuator networks (WSANs), particularly those used in control applications. The project goals are three-fold: (1) end-to-end delay analysis for real-time flows sharing a WSAN, (2) network protocols and algorithms for real-time communication over WSANs, and (3) implementation and experimentation of real-time WSANs on a physical wireless testbed. Our research addresses both practical problems in current industrial wireless sensor-actuator networks (WSANs) and fundamental challenges faced by future wireless control networks.

This work is sponsored by NSF through grant CNS-1320921 (NeTS) and 1646579 (CPS).

Real-time Scheduling Theory for WirelessHART

We established a novel real-time scheduling framework for wireless control networks based on WirelessHART by bridging real-time scheduling theory and wireless networking.

Dynamic transmission scheduling: We devised both optimal and near optimal policies for dynamic priority scheduling of transmissions for real-time flows between sensors and actuators. We observed that transmission conflict (due to half-duplex radio of the nodes) plays a major role in communication delays, making the traditional real-time scheduling policies less effective for transmission scheduling in WirelessHART networks. Using this key observation, we designed an optimal algorithm based on branch and bound, and a heuristic called Conflict-aware Least Laxity First (C-LLF) for dynamic priority scheduling. C-LLF integrates the degree of conflicts associated with a transmission into LLF, and outperforms traditional real-time scheduling policies. This work was presented at [RTSS'10]. We also provided a schedulability analysis for earliest deadline first (EDF), a common dynamic priority scheduling in WirelessHART networks. This work was presented at [IWQoS'14].

Fixed-priority transmission scheduling: For wireless control with firm requirements on network latency, a delay analysis is required to quickly assess the schedulability of the real-tme flows, specially for online admission control and workload adjustment in response to network dynamics. We provided an efficient analysis of the worst-case communication delays of periodic real-time flows that are scheduled based on fixed priorities in a WirelessHART network. The key insight in the analysis is to map real-time multi-channel transmission scheduling to real-time multiprocessor scheduling. This approach allows us to build on existing real-time scheduling theory while focusing on incorporating unqiue features of wireless communication in the analysis. Our analysis establishes safe upper bounds on end-to-end delays, thereby enabling effective schedulability tests for WirelessHART networks. We also proposed optimal and near-optimal priority assignment algorithms based on local search and heuristic search, respectively. Our search approach leverages the lower and upper delay bounds provided by our delay analysis to reduce the search space. The delay analyses for single independent routes and for graph routes were presented at [RTAS'11] and [RTSS'15], respectively, and priority assignments was presented at [ECRTS'11].

Cyber-Physical Co-Design for Wireless Control Systems

Rate Selection for Wireless Control Systems: In a wireless control system, the choice of sampling rates of the feedback control loops must balance control performance and communication delays. A low sampling rate usually degrades the control performance while a high one may cause excessive communication delays also leading to degraded control performance. We addressed the scheduling-control co-design problem of sampling rate selection to optimize the overall control cost in a WirelessHART network. The resulting constrained optimization is challenging since it is non-differentiable, non-linear, and not in closed-form. We proposed and evaluated five optimization methods including greedy heuristic, subgradient method, simulated annealing based penalty method, and gradient descent method and interior point method upon a differentiable convex relaxation. Our result in this work has drawn some interesting conclusions towards co-design. In particular, we have shown the interior point method and the simulated annealing based adaptive penalty method as the two most effective approaches for rate selection. They represent the opposite ends of the tradeoff between control cost and execution time, while the interior method is likely the most effective approach in practice due to its run time efficiency. This work shows the promise of cyber-physical co-design, where tailoring real-time scheduling analysis allows for an elegant and efficient optimization approach for wireless control systems. This work was nominated for Best Paper Award at [RTAS'12].

Incorporating Emergency Alarms in Reliable Wireless Process Control: Many real world process control systems must handle various emergency alarms under stringent timing constraints in addition to regular control loops. However, despite considerable theoretical results on wireless control, the problem of incorporating emergency alarms in wireless control has received little attention. We develop the first systematic approach to incorporate emergency alarms into wireless process control. The challenge in emergency communication lies in the fact that emergencies occur occasionally, but must be delivered within their deadlines when they occur. We propose efficient real-time emergency communication protocols based on slot stealing and event-based communication. We build an open-source WirelessHART protocol stack in the Wireless Cyber-Physical Simulator (WCPS) for holistic simulations of wireless control systems, and conduct systematic studies on a coupled water tank system controlled over a 6-hop 21-node WSAN. Our results demonstrate our real-time emergency communication approach enables timely emergency handling, while allowing regular feedback control loops to effectively share resources in WSANs during normal operations. This work was presented at [ICCPS'15].

Wireless Networked Control System, Closed-Loop Architecture

Wireless Routing and Control: Wireless sensor-actuator networks (WSANs) are being adopted in process industries because of their advantages in lowering deployment and maintenance costs. While there has been significant theoretical advancement in networked control design, only limited empirical results that combine control design with realistic WSAN standards exist. We conduct a cyber-physical case study on a wireless process control system that integrates state-of-the-art network control design and a WSAN based on the WirelessHART standard. The case study systematically explores the interactions between wireless routing and control design in the process control plant. The network supports alternative routing strategies, including single-path source routing and multi-path graph routing. To mitigate the effect of data loss in the WSAN, the control design integrates an observer based on an Extended Kalman Filter with a model predictive controller and an actuator buffer of recent control inputs. We observe that sensing and actuation can have different levels of resilience to packet loss under this network control design. We then propose a flexible routing approach where the routing strategy for sensing and actuation can be configured separately. The proposed asymmetric routing configuration with different routing strategies for sensing and actuation can effectively improve control performance under significant packet loss. Our results highlight the importance of co-joining the design of wireless network protocols and control in wireless control systems. This work was presented at [ICCPS'16].

Holistic Control Framework.

Holistic Cyber-Physical Management for Wireless Control Systems: In traditional wireless control systems, the plant controller and the network manager operate in isolation, which ignore the significant influence of network reliability on plant control performance. To enhance the dependability of industrial wireless control, we proposed a holistic cyber-physical adaptation framework that employs run-time coordination between the plant control and network management. Our design includes a holistic controller that generates actuation signals to physical plants and reconfigures the WSAN to maintain desired control performance while saving wireless resources. As a concrete example of holistic control, we design a holistic manager that dynamically reconfigures the number of transmissions in the WSAN based on online observations of physical and cyber variables. We have implemented the holistic management framework in the Wireless Cyber-Physical Simulator (WCPS). A systematic case study has been presented based on two 5-state plants sharing a 16-node WSAN. Simulation results show that the holistic adaptation significantly enhanced the dependability of the system against both wireless interferences and physical disturbances, while effectively reducing the number of wireless transmissions. [TCPS'18].

Holistic Control Framework

Efficient Holistic Control over Industrial WSAN: Holistic control adopts a cyber-physical system approach to overcome the challenges by orchestrating network reconfiguration and process control at run time. We explore efficient holistic control designs to maintain control performance while reducing the communication cost. The contributions of this work are four-fold: (1) We introduce a holistic control architecture that integrates low-power wireless bus (LWB) and two control strategies, rate adaptation and self-triggered control, specifically proposed to reduce communication cost; (2) We design novel wireless network mechanisms to support rate adaptation and self-triggered control, respectively, in a multi-hop WSAN; (3) We build a real-time network-in-the-loop simulator that integrates MATLAB/Simulink and a three-floor WSAN testbed to evaluate wireless control systems; (4) We empirically explore the tradeoff between communication cost and control performance under alternative holistic control approaches. Our case studies show that rate adaptation and self-triggered control offer advantages in control performance and energy efficiency, respectively, in normal operating conditions. The advantage in energy efficiency of self-triggered control, however, may diminish under harsh physical and wireless conditions due to the cost of recovering from data loss and physical disturbances. This work was presented at [ICII'18] and [TCPS'20].

Implementation and Empirical Studies of Industrial WSAN Protocols

WSAN Testbed Implementation: Network Manager Software and Protocol Stack

WSAN Testbed: Network Topology

Wireless sensor-actuator networks (WSANs) offer an appealing communication technology for process automation applications. While industrial WSANs have received attention in the research community, most published results to date focused on the theoretical aspects and were evaluated based on simulations. There is a critical need for experimental research on this important class of WSANs. We developed an experimental testbed by implementing several key network protocols of WirelessHART, an open standard for WSANs widely adopted in the process industries, including Time Slotted Channel Hopping (TSCH) at the MAC layer and reliable graph routing supporting path redundancy. We performed a comparative study of the two alternative routing approaches adopted by WirelessHART, namely source routing and graph routing. Our study shows that graph routing leads to significant improvement over source routing in term of worst-case reliability, at the cost of longer latency and higher energy consumption. It is therefore important to employ graph routing algorithms specifically designed to optimize latency and energy efficiency. Our studies also suggest that channel hopping can mitigate the burstiness of transmission failures; a larger channel distance can reduce consecutive transmission failures over links sharing a common receiver. This work was published at [EWSN'15] and [IoT-J'17]

Enhancements of Industrial WSAN Protocols

Impact of Channel Selection on Industrial WSANs: To meet stringent reliability requirements of industrial applications, industrial standards such as WirelessHART adopt Time Slotted Channel Hopping (TSCH) as its MAC protocol. Since every link hops through all the channels used in TSCH, a straightforward policy to ensure reliability is to retain a link in the network topology only if it is reliable in all channels used. However, this policy has surprising side effects. While using more channels may enhance reliability due to channel diversity, more channels may also reduce the number of links and route diversity in the network topology. We empirically analyze the impact of channel selection on network topology, routing, and scheduling on a 52-node WSAN testbed. We observe inherent tradeoff between channel diversity and route diversity in channel selection, where using an excessive number of channels may negatively impact routing and scheduling. We propose novel channel and link selection strategies to improve route diversity and network schedulability. Experimental results on two different testbeds show that our algorithms can drastically improve routing and scheduling of industrial WSANs. This work was presented at [INFOCOM'17]

Real-Time Routing for Industrial WSANs: With the emergence of the Industrial Internet of Things (IIoT), process industries have started to adopt wireless sensor-actuator networks (WSANs) for control applications. It is crucial to achieve real-time communication in this emerging class of networks in which routing has significant impacts on end-to-end communication delays. However, despite considerable research on real-time transmission scheduling and delay analysis for such networks, real-time routing remains an open question for WSANs. We developed a conflict-aware real-time routing approach for WSANs. This approach leverages a key observation that conflicts among transmissions involving a common device contribute significantly to communication delays in industrial WSANs (e.g., WirelessHART networks). By incorporating conflict delays into the routing decisions, conflict-aware real-time routing algorithms allow a WSAN to accommodate more real-time flows while meeting their deadlines. This work was presented at [IoTDI'18]

Conservative Channel Reuse for Industrial IoT: Industrial applications impose stringent requirements in both reliability and real-time performance on WSANs. To enhance reliability, industrial standards such as WirelessHART, embrace Time Slotted Channel Hopping (TSCH) that integrates channel hopping and TDMA at the MAC layer. Within a network governed by a same gateway, WirelessHART prohibits channel reuse, i.e., concurrent transmissions in a same channel, to avoid interference between concurrent transmissions. Preventing channel reuse however negatively affects real-time performance. To meet the demand for both reliability and real-time performance by industrial applications, we proposed a conservative channel reuse approach designed to enhance the real-time performance while limiting its impact on reliability in WSANs. In contrast to traditional channel reuse designed to optimize performance at the cost of reliability, our conservative approach introduces channel reuse only when needed to meet the timing constraints of flows. Furthermore, we designed an algorithm to detect reliability degradation caused by channel reuse so that channels can be reassigned to further improve reliability. This work was presented at [ICDCS'18]

CapNet: Real-time Wireless Management Network for Data Center Power Capping

Data center management (DCM) is increasingly a significant challenge for enterprises hosting large scale online and cloud services. Machines need to be monitored, and the scale of operations mandates an automated management with high reliability and real-time performance. Existing wired networking solutions for DCM come with high cost. Wireless sensor networks provide a cost-effective networking solution for DCM while satisfying the reliability and latency performance requirements of DCM. We have developed CapNet, a real-time wireless sensor network for power capping, a time-critical DCM function for power management in a cluster of servers. CapNet employs an efficient event-driven protocol that triggers data collection only upon the detection of a potential power capping event. We deploy and evaluate CapNet in a data center. Using server power traces, our experimental results on a cluster of 480 servers inside the data center show that CapNet can meet the real-time requirements of power capping. CapNet demonstrates the feasibility and efficacy of wireless sensor networks for time-critical DCM applications. This work was reported at [RTSS'14].

Publications

• Y. Ma, C. Lu and Y. Wang, Efficient Holistic Control: Self-Awareness across Controllers and Wireless Networks, ACM Transactions on Cyber-Physical Systems, Special Issue on Self-Awareness in Resource Constrained Cyber-Physical Systems, accepted. PDF

• H. Li, C. Lu and C. Gill, Predicting Latency Distributions of Aperiodic Time-Critical Services. IEEE Real-Time Systems Symposium (RTSS'19), December 2019. PDF

• A. Saifullah, S. Sankar, J. Liu, C. Lu, R. Chandra, and B. Priyantha, Capnet: Exploiting wireless sensor networks for data center power capping, ACM Transactions on Sensor Networks, 15(1), Article No. 6, January 2019. PDF

• Y. Ma and C. Lu, Efficient Holistic Control over Industrial Wireless Sensor-Actuator Networks, IEEE International Conference on Industrial Internet (ICII'18), October 2018. PDF

• Y. Ma, D. Gunatilaka, B. Li, H. Gonzalez and C. Lu, Holistic Cyber-Physical Management for Dependable Wireless Control Systems, ACM Transactions on Cyber-Physical Systems, Special Issue on Dependability in Cyber Physical Systems and Applications, 3(1), Article No. 3, August 2018. PDF

• A. Saifullah, M. Rahman, D. Ismail, C. Lu, J. Liu and R. Chandra, Low-Power Wide-Area Network over White Spaces, IEEE/ACM Transactions on Networking, 28(4): 1893-1906, August 2018. PDF

• D. Gunatilaka and C. Lu, Conservative Channel Reuse in Real-Time Industrial Wireless Sensor-Actuator Networks, IEEE International Conference on Distributed Computing Systems (ICDCS'18), July 2018. PDF

• P. Park, S.C. Ergen, C. Fischione, C. Lu and K.H. Johansson, Wireless network design for control systems: A survey. IEEE Communications Surveys & Tutorials, 20(2):978–1013, second quarter 2018. PDF

• C. Wu, D. Gunatilaka, M. Sha and C. Lu, Real-Time Wireless Routing for Industrial Internet of Things, ACM/IEEE International Conference on Internet of Things Design and Implementation (IoTDI'18), April 2018.  PDF

• A. Saifullah, M. Rahman, D. Ismail, C. Lu, J. Liu and R. Chandra, Enabling Reliable, Asynchronous, and Bidirectional Communication in Sensor Networks over White Spaces, ACM Conference on Embedded Networked Sensor Systems (SenSys'17), November 2017. PDF

• M. Sha, D. Gunatilaka, C. Wu and C. Lu, Empirical Study and Enhancements of Industrial Wireless Sensor-Actuator Network Protocols, IEEE Internet of Things Journal, 4(3): 696-704, June 2017. PDF

• D. Gunatilaka, M. Sha and C. Lu, Impacts of Channel Selection on Industrial Wireless Sensor-Actuator Networks, IEEE International Conference on Computer Communications (INFOCOM'17), May 2017 PDF

• C. Lu, A. Saifullah, B. Li, M. Sha, H. Gonzalez, D. Gunatilaka, C. Wu, L. Nie and Y. Chen, Real-Time Wireless Sensor-Actuator Networks for Industrial Cyber-Physical Systems, Special Issue on Industrial Cyber-Physical Systems, Proceedings of the IEEE, 104(5): 1013-1024, May 2016. PDF

• B. Li, Y. Ma, T. Westenbroek, C. Wu, H. Gonzalez and C. Lu, Wireless Routing and Control: a Cyber-Physical Case Study, ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS'16), April 2016 PDF

• C. Wu, D. Gunatilaka, A. Saifullah, M. Sha, P.B. Tiwari, C. Lu and Y. Chen, Maximizing Network Lifetime of WirelessHART Networks under Graph Routing, IEEE International Conference on Internet-of-Things Design and Implementation (IoTDI'16), April 2016. PDF

• C. Lu, A. Saifullah, B. Li, M. Sha, H. Gonzalez, D. Gunatilaka, C. Wu, L. Nie and Y. Chen, Real-Time Wireless Sensor-Actuator Networks for Industrial Cyber-Physical Systems, to appear in Proceedings of the IEEE, 2016. PDF

• A. Saifullah, D. Gunatilaka, P. Tiwari, M. Sha, C. Lu, B. Li , C. Wu, and Y. Chen, “Schedulability analysis under graph routing for WirelessHART networks”, IEEE Real-Time Systems Symposium (RTSS'15), December 2015. PDF

• B. Li, L. Nie, C. Wu, H. Gonzalez and C. Lu, Incorporating Emergency Alarms in Reliable Wireless Process Control, ACM/IEEE International Conference on Cyber-Physical Systems (ICCPS'15), April 2015. PDF

• M. Sha, D. Gunatilaka, C. Wu and C. Lu, Implementation and Experimentation of Industrial Wireless Sensor-Actuator Network Protocols, European Conference on Wireless Sensor Networks (EWSN'15), February 2015. PDF

• A. Saifullah, Y. Xu, C. Lu and Y. Chen, End-to-End Communication Delay Analysis in Industrial Wireless Networks, IEEE Transactions on Computers, accepted. PDF

• A. Saifullah, S. Sankar, J. Liu, C. Lu, R. Chandra and B. Priyantha, CapNet: a Real-Time Wireless Management Network for Data Center Power Capping, IEEE Real-Time Systems Symposium (RTSS'14), December 2014. PDF

• C. Wu, M. Sha, D. Gunatilaka, A. Saifullah, C. Lu and Y. Chen; Analysis of EDF Scheduling for Wireless Sensor-Actuator Networks, ACM/IEEE International Symposium on Quality of Service (IWQoS'14), May 2014. PDF

• A. Saifullah, C. Wu, P. Tiwari, Y. Xu, Y. Fu, C. Lu and Y. Chen, Near Optimal Rate Selection for Wireless Control Systems, ACM Transactions on Embedded Computing Systems, Special Issue on Real-Time and Embedded Technology and Applications, 13(4s), Article 128, April 2014. PDF

• O. Chipara, C. Lu and G.-C. Roman, Real-time Query Scheduling for Wireless Sensor Networks, IEEE Transactions on Computers, 62(9): 1850-1865, September 2013. PDF

• A. Saifullah, C. Wu, P. Tiwari, Y. Xu, Y. Fu, C. Lu and Y. Chen; Near Optimal Rate Selection for Wireless Control Systems, IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS'12), April 2012. (Best Paper Nominee) PDF

• A. Saifullah, Y. Xu, C. Lu and Y. Chen; Priority Assignment for Real-time Flows in WirelessHART Networks, Euromicro Conference on Real-Time Systems (ECRTS'11), July 2011. PDF

• O. Chipara, C. Wu, C. Lu and W.G. Griswold, Interference-Aware Real-Time Flow Scheduling for Wireless Sensor Networks, Euromicro Conference on Real-Time Systems (ECRTS'11), July 2011. PDF

• A. Saifullah, Y. Xu, C. Lu and Y. Chen; End-to-End Delay Analysis for Fixed Priority Scheduling in WirelessHART Networks, IEEE Real-Time and Embedded Technology and Applications Symposium (RTAS '11), April 2011. PDF

• A. Saifullah, Y. Xu, C. Lu, and Y. Chen; Real-time Scheduling for WirelessHART Networks; IEEE Real-Time Systems Symposium (RTSS '10), December 2010. PDF

• Bibtex file of papers supported by NSF CPS grant 1646579 File:Cps1646579bib.bib

Selected Talks

• Dependable Industrial Internet of Things, Keynote, Cyber-Physical Systems Week, April 2018. PDF

• Dependable Internet of Things, University of Notre Dame, November 2017. PDF

• Real-Time Internet of Things, Keynote, IEEE International Symposium on Real-Time Computing (ISORC'17), May 2017. PDF

• Towards Real-Time Cloud Computing, Uppsala University, February 2017. PDF

• Wireless Control: Opportunities and Challenges, Plenary Panel, ACM Embedded Systems Week (ESWeek), October 2016. PDF

• Dependable Wireless Control through Cyber-Physical Co-Design, Keynote, International Conference on Embedded Wireless Systems and Networks (EWSN), February 2016. PDF

• Real-Time Wireless Control Networks for Cyber-Physical Systems, University College Cork, Ireland, July 2014. PDF

• Challenges in Wireless Control Networks for Cyber-Physical Systems, Panel on Networking Challenges for Cyber-Physical Systems, INFOCOM, May 2014. PDF

• Real-Time Wireless Control Networks for Cyber-Physical Systems, Keynote, International Conference on Pervasive and Embedded Computing and Communication Systems (PECCS'14), January 2014. PDF

• Real-Time Wireless Control Networks for Cyber-Physical Systems, Keynote, IEEE International Symposium on Industrial Embedded Systems (SIES'13), June 2013. PDF

• Real-Time Wireless Control Networks: Challenges and Directions, NITRD National Workshop on the New Clockwork for Time-Critical Systems, October 2012. PDF

• Real-Time Wireless Sensor Networks, Royal Institute of Technology (KTH), Stockholm, Sweden, April 2010. PDF