White Paper: Distribution and Federation with IoT
Paul Butterworth | January 2016
EXECUTIVE SUMMARY
The IoT (Internet of Things) is primarily being used to collect and analyze data (Big Data) to keep track of the status of “things.” To accomplish this, information is collected by sensors on machines (e.g., a jet engine), wearable devices (e.g., an oxygen monitor), or specialized devices (e.g., a water meter). Once sensor data is collected from a jet engine it is analyzed to predict failures and schedule maintenance. As beneficial as this is, it is just the beginning of the IoT.
The next phase of the IoT will allow for changes to be made to systems in real-time. Physical sensors provide streaming data, software analyzes it in real-time and automates activities or notifies appropriate personnel when attention is required. For example, IoT systems will optimize the performance of operations involved with warehousing (products, robots, gantries, distribution vehicles) or, in the longer run, the overall scheduling of traffic control resources (traffic lights, lane directions, speed limits) on the highway grid. This is IoT automation.
IoT automation will be naturally distributed and federated. Logic will be located where it is most appropriate. Machine control systems will typically be near the machines; computer systems interacting with large groups of people will be near the activities; analysis of Big Data will be in the Cloud. This locality of function approach addresses concerns such as responsiveness, availability, security, and machine resource requirements. Management and operations reporting will be in the most logical location, Cloud or computers located in the field (at the edge).
The provisioning, or distribution of logic, capabilities in Vantiq are extremely comprehensive and rich in functionality. Various system topologies (star, hierarchical, or peer-to-peer) are supported. The provisioning and management of these networks is made automatic and easy to manage by intelligent features built into the Vantiq platform. Application components can be dynamically changed anywhere in the distributed environment for one or thousands of systems while the system is running. Organizations can incrementally distribute functionality to all or a subset of the IoT system for controlled rollout of functionality. New computers can be dynamically added. Consequently, businesses can respond quickly to what are inevitably highly dynamic requirements.
The Vantiq platform is designed and architected unlike any product to enable the development of IoT applications which are inherently complex. Vantiq’s design goal is to let the software take care of as much as possible so that the development of IoT systems can focus on the business logic and not the underlying technology.
INTRODUCTION
As our capacity to automate nearly any function that can be sensed and controlled electronically becomes ubiquitous, the IoT is taking center stage as the primary infrastructure on which these automation systems are constructed. Much has been written about the tens or hundreds of billions of devices that will comprise the IoT and the major advances being made in communications, data analytics and artificial intelligence required to exploit the IoT. However, less has been written about the truly distributed nature of many IoT systems.
This paper describes the distributed nature of IoT systems and the reasons why automation must be distributed among the distributed tiers as well as the best approach to developing responsive, reliable automation applications in such environments.
BACKGROUND
IoT Systems
In this paper a fairly expansive definition of an IoT system is used to cover the wide array of modern automation applications. Such applications have the general flow of
- Input is received from a number of sensors perhaps over an extended period of time.
- The sensor data is processed to produce the information or context within which automation and recommendation decisions are made.
- An automation system evaluates the context in real-time to determine the actions that need to be taken.
- Actions are transmitted to the responsible systems or parties for implementation.
Data Acquisition
The notion of a sensor can be very broad including:
- Mobile devices hosting a wealth of sensor data including location, acceleration, audio, video, add-on sensors and derived behavioral patterns.
- Wearable devices such as watches, activity trackers, health monitors, audio and video headsets.
- Machines including industrial machines, land and airborne transportation, home appliances and any mechanical or electronic equipment that can be sensed and/or controlled. For example, imagine a robot’s manipulators instrumented with pressure sensors to vary the pressure applied to objects that may have different crush points.
- Stand-alone sensors deployed in great numbers. For example, moisture sensors distributed across the fields of a farm to minimize water consumption while maximizing growth rates for the crops.
- Video and audio feeds that produce high volumes of what can be considered sensor data. The key to this data is the recognition software used to determine what the video represents in order to translate the video into more discrete events on which automation decisions can depend.
The sensors can be connected directly to the internet with their own IP communications stack or they can be indirectly connected to the internet via a gateway. In the latter case, the sensors themselves may communicate over more specialized protocols such as Modbus or ZigBee with the gateway providing protocol conversion so that the sensors appear as virtual nodes participating in the IoT.
Analysis and Automation
Once the data has been acquired, the IoT system is responsible for making use of the information.
To date, much of the focus has been on Big Data analytics. Large amounts of data are collected and subjected to sophisticated analysis to identify trends represented by the data. Such a scheme can be used to predict machine failures by identifying sensor patterns that historically precede a failure or predict buying patterns among customers by determining products similar customers purchase under similar conditions.
A growing trend is analyzing sensor data in real-time in order to automate responses to the current state of the machine or customer and/or notify the appropriate operations personnel to what action was taken or what action to take. This is what we call automation and represents a large opportunity to both improve business operations or offer new opportunities:
- Providing more relevant responses to consumers based on their current situation (items on sale, facility map, emergency response recommendations).
- Respond more intelligently to exceptional conditions (close a valve, turn on sprinklers, stop a malfunctioning robot).
- Proactively alert personnel to opportunities/problems based on the current situation (extra delivery trucks available, shortage in part of the supply chain).
- Optimize the user or business resources to improve productivity and/or customer satisfaction (speed up an assembly line, advise sports attendees on shortest path to their car).
Reporting and Notification
Once situations have been identified and actions taken, it is important to notify interested parties of these actions and the new state of the system. Notifications can be delivered to other automated systems so that they can independently respond to the situation or delivered to responsible staff via desktops, mobile devices and wearable devices. Notifications can also include recommended actions and situational awareness of pending problems.
Distributed IoT Systems
Many IoT systems are naturally distributed. In manufacturing environments digital computers called PLCs (Programmable Logic Controllers) communicate with area controllers and gateways that forward the data to more centralized IT systems. In consumer environments data may be collected from a large number of position sensors, processed locally into logical locations on which immediate automation decisions are made and forwarded to remote systems that optimize the experience for the consumer. However, such a wide variety of distributed systems requires us to carefully classify their distributed topologies. The distributed topologies can range from devices directly reporting to a central site to hierarchically structured automation systems to federated peers collaborating to improve a collection of businesses.
The simplest architectures are sensors reporting to a central site. Many examples of such systems exist today. Systems that collect sensor data from a mobile phone and report that data to a cloud service represent a common example of a centralized architecture. These simple star architectures represent the bulk of the existing IoT systems as they are the easiest to understand and build.
More sophisticated architectures contain additional levels of processing and connectivity. Hierarchical systems are more complex and mimic many existing physical and organizational structures. For example, an industrial IoT system that consists of sensors reporting to local controllers that report to plant-wide controllers that report to divisional headquarters that report to corporate headquarters represents a tree topology. These systems provide for both centralized monitoring and control as well as decentralized monitoring and control. Such systems are more responsive in a real-time or near real-time situation. For example, it would be impractical to control factory equipment in real-time by collecting the data, shipping it to corporate HQ and having corporate HQ systems determine the next action for the machine. It is far more effective to do such an analysis on the local controller and simply report the situation and the action taken to the plant-wide controllers and, subsequently, to regional and corporate HQ. Faster response times, improved availability and local control make the distribution of the situational evaluation, decision making and response across the hierarchical topology more efficient than moving everything to HQ and making all decisions in a centralized fashion.
Another classic example of hierarchical IoT systems is the use of edge gateways to act as local processors for a collection of sensors and control points with the gateways then interacting with more centralized systems. The gateways can perform two functions:
- Gateways act as mediators for connected devices (sensors and controllers) that cannot be connected directly to the internet. For example, devices that run protocols like Modbus or ZigBee are incompatible with IP protocols but the gateway can make them accessible by mediating requests to and from such devices with its own IP protocol stack.
- Gateways perform local processing in real-time by eliminating the need to communicate with another system that performs the processing. Such communication over the internet will incur substantial delays and subject the overall process to lower levels of availability since the system is now dependent on the gateway, the internet and the remote service for every request. For example, you would not want to control an elevator system from a distance.
The most sophisticated distributed IoT systems are peer-to-peer systems where peers are managed by separate organizations. For example, in an electrical demand-response system, the overall IoT system consists of sensors managed by power utilities and sensors managed by utility customers while control of the system is distributed across the utility and its customers. In order to provide real-time demand-response, the utility IoT system and the customer IoT systems must collaborate. This is accomplished by each system making local decisions and transmitting both the local situation and the local decisions to the other party and then agreeing to modify their real-time behavior based on feedback from each other.
IoT Automation
IoT automation, as defined earlier, involves acquiring data, evaluating the real-time situation represented by that data and responding to that identified situation with the appropriate action or notification. Such a system consists of data definitions for storing the sensor data and the situational state, automation logic for detecting and responding to situations, and data transport to move the data to the appropriate location for evaluation and storage.
In real-time or near real-time environments, processing is always better performed locally to the device under control, improving response time and reliability. For example, in an industrial setting, managing the position of a materials handling system requires near real-time responses within a few hundred milliseconds. Such response times cannot be guaranteed by a remote decision making system that may be delayed by 1,000s of milliseconds if there is a network failure.
With this introduction to IoT systems in hand, the remainder of this paper focuses on best practices for developing real-time distributed IoT automation systems as implemented in the Vantiq system.