Controls Design, Checkout, and Troubleshooting

Controls Design

I have found that up front scoping of controls requirements is the key to a successful control system project. Scope means what needs to be controlled, the level of automation and the HMI required. With so many options for control hardware from plant wide DCS, to SCADA, to controls employing PLCs, Industrial PCs, and custom processor boards, it is important to have an idea of what the control system will look like at start of the project. It is a good idea to try to use hardware and software that the plant maintenance and technical staff is familiar with. However, that should not be an excuse to use the same controls implementation for everything. One should be aware that many vendors and consultants try to maximize their profit by recycling the same designs over and over to different customers, rather than taking the customers needs and capabilities into account.

What ever controls architecture is decided upon, modular design should be utilized to provide systems that are reliable and maintainable. All except the simplest controls employ digital processors. Programming techniques make use of RAD methodology in order to speed implementation and simplify checkout.

Simulation and Analysis Software

There are several control analysis and simulation tools, some specific to particular particular control systems and others more general academeic type programs (e.g. bode plot programs). I have installed MatLAB for an optimization application, and also included the Control Systems toolbox. I haven't had a chance to even open up the tool box but I have utilized the data analysis to plot system responses and perform frequency anlysis on data. The learning crurve seems to be very steep, but the results so far show promise.

Controls Checkout

Checkout is the one area where modern control systems are not immediately superior to the systems they replace. Checkout of old hardwired systems basically involved schematic checking to catch wiring errors and simple exercising of functions to catch logic errors such as race conditions. Attempts to morph this methodology to try to "exercise every branch of code" was not very good at detecting programming bugs, or transposed I/O.

Modular design with well defined process boundaries is the key to successfully checkout and startup. Each sub module is tested with all reasonable combinations of inputs. Analog quantities are tested at the boundary conditions. Logic is verified as well as error trapping to provide for graceful system failures. (A miswired or failed input should generate an error indication, not a system crash.) Then modules and finally subsystem and system tests are similarly performed. Note that with RAD methodology test procedures are developed as modules are coded.

Troubleshooting

Ease of troubleshooting should be a major factor in the design of any control system. However, it is one of many and often falls below cost and /or performance criteria. At some point a system will begin failing (usually easy to diagnose - if not intermittent) or behaving unexpectedly (harder to diagnose). Troubleshooting techniques are mainly common sense (did you change anything right before the system started acting up?). However, experience with systems of all kinds is probably the greatest aid when it comes to troubleshooting.

Control Engineer's Toolbox

Laptop:

I think I am fortunate that my employer has provided me with a high end laptop running some fairly expensive software. The only drawback has been the weight. I have experimented (on my own dime) with netbooks because the small size and long battery life are attractive when you have to carry the machine through a plant or prop it up in a cramped space. The results were entirely unsatisfactory. Some software (e.g. Eclipse IDE) wouldn't run on some machines, and the machines were really slow. The last such purchase I made was after reading a review of the Ubuntu netbook remix software. I tried to get one if the netbook models mentioned in the article but it was no longer available. I got the model that replace it and after fiddling with the bios to get it to boot off a USB device, and going through the install, the kernel crashed every time I tried to start X. (In comparison, if I boot a memory stick Ubuntu version on my regular laptop, everything works - except the wireless card. If I run VM player and start up my CentOS appliance I can even access the wireless card from Linux.)

For my particular work I need good display resolution (1440X900) in order to run RDP and VNC sessions on remote computers. My computer is also provided with a serial communications port (for direct interface to serial devices) and a PC card slot that I keep an ASE SCADA communications test set card plugged in all the time.

Some engineers, either by preference or fiat, keep a separate laptop for interfacing to control systems and another computer used for general office work and Internet communications. The idea is to reduce the risk of infecting a laptop that gets connected to a controls network. Due to the amount of time I spend on the road it is not practical for me to drag two laptops around. Also, it is hard to keep a computer patched and all the various applications (including Firewall and A/V) up to date if you don't connect to the Internet.

General Software:

For A/V, you will probably have to rely on whatever corporate IT decides on, although if you can put on your own you should consider checking reviews for the latest hot freeware that is available. (I use Avast on my home machines.) You should also use a separate third party firewall (I use Comodo) to get a handle on what software may be going out on the internet or listening on ports. Occasional nmap scans is probably a good idea as well. I use a light weight freeware PDF reader rather than Acrobat Reader to avoid exploits that target that ubiquitous product.

I like Open Office for home use, but MS Office is needed for work, mainly because of Excel (with various plug ins for analytical calculations) and Access (good for handling large data sets or linking to external databases via ODBC). Similarly IE is needed because some specialized functions (such as the VPN client I need to use to provide remote support) only work with it. However, for general web searches I use an installed instance of Firefox Portable with the security features all set up high. (The portable version doesn't even normally write to disk.)

Other general software I use includes Acronis backup software, Filezilla Client for uploading/downloading files, SSH PowerTerm for general remote terminal login, Putty for setting up SSH pipes as well as terminal services, Wdiff32, file compression software of your choice, some of the SysInternals utilities, Textpad for using macros to process large text files, a PDF file creator, a DXF reader for viewing AutoCAD files, and some soft of graphics software (I use Visio but don't like it much). I use a freeware version of the GANTTPROJECT package for generating simple project schedules.

For general automation of searches, text management I have gone through the effort to learn Perl which can make more complicated searches, replaces and formating compared to what can be done using Textpad. However, since I don't use it often I usually have to review syntax especially for regular expressions, especially for complex filtering. (Perl Cheat Sheet)

Communications Software:

As mentioned above, you need a real serial comm port to troubleshoot serial communications. A USB to serial converter will not always work depending on the hardware level RS232 handshaking. Similarly, we have found that sometimes you have to try several different brands or models of Ethernet port servers to find one that will work with a particular serial device. For general serial communications work, I use Realterm as it gives a much better picture of the states of a comm line than you could ever get with Hyperterminal. For SCADA RTU communications, ASE test set software is the industry standard. (Although I myself don't use it, we have KEMA test set software for ICCP communications testing.) For general Ethernet communications troubleshooting Wireshark is easier to use than the dedicated Fluke Optiview we also have in the office (which we usually forget to take on the road anyway).

Control System Support Software:

For a while I had a complete iFix installation running on the laptop. (It would shutdown after two ours since it wasn't licensed.) I also used a product called FactorySoft OPC explorer for working with OPC servers. However, they took up a lot of space and were fairly slow to run. Also, security has been tightened on PCs so it really isn't possible to browse OPC servers on other machines or easily connect to remote iFix servers. As my work took me away from small PLC control systems, and more towards SCADA/EMS I uninstalled these applications.

Some programs I have loaded to support recent or future work in include Configpro GE RTU configuration software, annunciator configuration software and a complete demo AREVA eterraControl SCADA FEP implementation.

Linux/Unix Support Software:

Where you are working with control systems or servers that are Unix based, it is sometimes better to boot into Linux where you get built in services (e.g. SSH, X, RCP, etc) support right from the get go. Options here are to set up dual boot (easily done right from most Linux iso installs) or run VMware player and a Linux appliance that you can download from various sources. My laptop was originally set up with dual boot (Windows/Redhat) but I switched to VMware player and a CentOS appliance when our corporate IT decreed that all laptop disks be encrypted with Safeboot. Safeboot did not support dual boot. Although forced on me, I have to admit the VMware player solution was a lot easier to implement.

To provide a remote desktop into Linux workstations we use one of the versions of VNC which is set up via a secure tunnel to localhost. You have to log in via SSH first and set up a tunnel from the VNC ports (5900, 5901, etc) to your own localhost to initiate a session.

Software Development Tools:

It used to be that I did all my PC programs in C and C++. C++ applications could be linked so that all libraries were included in the final executable. (This made the exe files huge, but they always worked - unless windows was upgraded!) A lot of that code was eventually ported to SCADA servers where most of the other code was C (if not FORTRAN) anyway. Initially, OPC software libraries required interfaces be written in C++. For developing test and application programs on the PC I have shifted away from C and C++ programming to VB and VB.net to the extent that I haven't written a new Windows application in C++ in years. It used to be that VB was not robust enough for real world use. Applications written in VB would not work when transferred to another PC or would stop working when another piece of software was installed because of an overwritten DLL. That doesn't seem to be the case today, and with VB supporting class tools it has more than enough power for the type of programming I do. In addition, many SCADA systems make use of either VB like scripting, VBA interfaces written for Excel, or actual VB coding, including for OPC.

Of course, a major advantage of VB.net is that code that is developed stand alone can be ported to an IIS web application without a massive rewrite of code.. (Note that latter versions of IIS are required for the latest VB.net code to work. Since servers don't get replaced very often that means I have two versions of Visual Studio on my laptop -Visual Studio.net (2003) and Visual Studio 2005.) I also have Eclipse installed to provide context sensitive editor for php programming of web pages. For general html, xhtml and xml file creation/generation I have Dreamweaver installed as well.

We have applications written for both Apache and IIS, so I have both installed on the laptop. The IIS was installed with the Visual Studio.net and the Apache was installed as part of a free WAMP distribution. They are both set up for manual start only since they slow down the laptop, and the Apache config file was change so it listens on port 8080 instead of 80. With this configuration they can even work at the same time.

Database Tools:

Most of our databases are presently on Oracle, so I use Toad for controlling and modifying the databases. However, I am investigating porting our applications to mySQL and have installed Database Fishing Tool to see if that provides a better interface than phpMyAdmin. For handling data I usually link tables in Access via ODBC to provide easy to use query, reporting and import/export capability.

Your Mileage May Vary:

Obviously, the above set up is not the optimum for every controls engineer. I wish that I had time to learn some of the more esoteric programming/engineering/simulation software like Matlab, Labview or SimuLink. Depending on the job at hand, I would also supplement the above with what ever support software would be provided by a controls system vendor (e.g. the xxxLINX software packages provided by A-B). If you don't plan on providing web interfaces, and you have IT staff to take care of your databases then you don't need half of the software mentioned. However, the above may still be useful to you by making you aware of the tools used to perform those functions.