A Control Design reader writes: As an automation engineer at a large discrete part manufacturer, I need to create a factory-acceptance-test (FAT) plan for a $2 million high-speed, automated system. It has multiple stations with several robots, motion control, vision inspections and many fieldbus digital and analog I/O. The question is what should I include in a FAT? I don't know where to start. Also, are there diagnostic tools or software to make performing the defined tests easier to complete? What do you suggest?
[javascriptSnippet]
ANSWER
FAT—from start to finish
The factory acceptance test (FAT) is a document, whether maintained in electronic format or printed on paper, designed to enumerate performance requirements and record the achievement of those requirements. Completion of the FAT provides the system integrator clearance to deliver the control system to the customer site—thus, delivery from the factory to the customer. This document explains the need for the FAT and its minimal requirements for contractual obligation.
Automation system integrators, PLC/PAC programmers, robotics designers, process engineers and custom coders all face a challenge when creating. The programmer writes code with the purpose of fulfilling a need. Those needs may have been scratched on a napkin; they may have been delivered in a 300-page governmental boilerplate specification; or most likely they’re conveyed somewhere in between. The programmer’s task is to create action out of need; the code fulfills a requirement that marries sensors, actuators, databases, images and reports. Eventually, the programmer will step back and admire the work, only to wonder if it has fully satisfied the specifications. Not only does it need to answer the question, “Is this what you wanted?” It must also provide proof to the same one who handed the specification in the first place. Perhaps fulfilling the specification also means significant fulfillment of contractual obligation, leading to remuneration.
The above scenarios have played out time after time in varying degrees ever since the advent of the electrical switch. So what is the manner in which one proves contractual satisfaction? This is where the factory acceptance test figures prominently in the integrator’s deliverables.
What is the factory acceptance test, and what is it not? The FAT should be a document serving as a platform of proof for moving from the programmer’s shop—the factory—to the end user’s site. In its simplest form, the FAT should be a document that is a near-perfect recount of the specification given the programmer. While this may seem overly simplistic, this is how the programmer conveys their understanding of what was expected. If there is any misunderstanding between programmer and customer, it will come out in the words of the FAT.
It should also be noted that sometimes the manufacturing systems are so large they can and must be broken down into subsystems. If there is a way to subdivide a manufacturing line to smaller standalone portions, then this could warrant the generation of a FAT for each portion. Robotics stations, standalone skids, subassemblies from other manufacturers: All can be described and tested in their own FATs. The decision to subdivide FATs is at the discretion of the system integrator, in agreement with the customer.
The FAT is not the first document of proof. It should never be the first document. Instead, the FAT is usually second-to-last in a long line of deliverables. To put it in perspective, many system integrators will employ a near facsimile of the following list of deliverables:
- functional requirement specification—system integrator expresses the understanding of the system desired by the customer, expressed in general terms
- scope of work—system integrator expresses their understanding of who does what during the execution of the contract
- proposal—system integrator then produces a price associated with fulfilling that functional requirement specification, within the guidelines of the scope of work
- detailed design specification—system integrator then creates documentation to support the work required, including equipment drawings, such as architectural drawings, schematics, enclosures, pipe schedules, I/O lists, HMI screen prints and report formats, and guiding the customer and subcontractors toward complete installation
- factory acceptance test—test of the programmed system prior to delivery
- site acceptance test—FAT performed after delivery, at the end user’s location.
Granted, there are deviations from this, based on the industry supported. Pharmaceuticals, food and chemical industries all follow different testing standards, per the U.S. Code of Federal Regulations (CFR), but the underlying framework still exists.
An important factor of each of the documents above is that they are described as deliverables. The system integrator creates them, while the customer should see them and should have the ability to influence them—yes, even the proposal. Often the system integrator will insist on customer signature on each document before moving on to the next. A customer signature means the customer is in agreement, and the system integrator is free to move forward.
Signatures generally imply the document is printed on paper and signed on paper. Other electronic means are becoming the norm, as well, as long as all parties have access.
What should be seen in a FAT? If the system integrator is following the above generic path of documents, the FAT will include many sections and subsections for examination. Their FAT will most likely “carry along” the details provided in the previous specifications. Below are some typical headings for FAT inclusion. Within each heading are questions the system integrator should try to explain.
Automatic operations: What is the system’s expected behavior? Is it a continuous system or a batching system? What are the major components? This could be paragraphs of information spelling out the behavior of the system.
- I/O Testing—If there is processor I/O, those points should be listed. This could also include inspection of the enclosure(s), wiring practices and quality of workmanship. This section only exists if the I/O is to be terminated to the field devices as part of the factory assembly. If the system can only be wired at the site, this section is omitted. If additional diagnostic tools are needed, this should also be noted.
- Normal sequence of operations—If the system is a batching operation or if it does behave in a regulated series of steps, those steps would be included in this section. Normal robotics actions would also be listed in this section.
- Interlocks—Often the operation of the system means defining conditions in which equipment should not run, or perhaps withdraw to a safe state. Interlocks are defined to protect the equipment from self-destruction (think overtravel, physical barriers, pre-existing conditions). List all interlocks for all equipment that can be energized.
- Safety—This is a topic that addresses personnel safety. If there is the presence of moving equipment (robotics, gears, belts) or exposure to released energy (steam, stored electricity, potential energy), safety equipment is necessary. This section would address those testing requirements. Not all systems would have the safety measures in place, dependent on the extent of system testing.
- Contingencies—When something in the process goes awry, there needs to be a way to walk back the system to a safe state.
- Manual operations—The system may provide for operator manual overrides. Such would be explained in this section.
Graphical user interface (GUI) operations: This is a section dedicated to the depiction of the system for the user. This could be something as simple as a single LED, a cloud-based status report or an HMI page, all the way up to FDA-regulated SCADA screens.
- Screen standards—What are the expected colors, fonts, behaviors of the GUI reports?
- Pushbuttons/function keys—If there are keystrokes that provide common functions throughout the pages, what are they?
- Screen descriptions—If the system contains multiple screens/pages, they should be listed, each page given its own subsection with details about what is depicted.
- Other functions—Is there user management? Are passwords required to limit access? Does an annunciator panel make noise? Will vision inspections need further definition? What are the go/no-go requirements?
- Reports—Perhaps the system generates periodic reports, or even user defined reports. What fields are provided in those reports?
Every single item that can be listed above can be tested. Keep in mind, if it can be disputed by the customer, it is worthy of testing. To extend this even further, if it is worthy of testing, it should have its own check box. That check box becomes the moment-by-moment target of achievement as the programmer walks through the FAT.
Every scenario can be played out differently. Some sections and subsections can be quite elaborate while some won’t even exist.
A copy of the FAT should be transmitted to the customer for review, ensuring all are working toward the same goal.
The system integrator will usually execute the FAT prior to inviting the customer to visit and witness. This way, the system integrator can work out the kinks before the customer arrives.
On the day(s) of FAT execution, all parties should agree on one copy being used for signatures. In general, if the document has been agreed upon prior, project communications have been open and honest prior, and the programmer has already executed the FAT prior, the day of FAT witness testing should be quite smooth.
Successful completion of the FAT should be evidenced in the signatures provided by both the system integrator and the customer. If there are discrepancies, changes or simple oversights, they should be addressed accordingly:
- Quick fixes should be addressed on the spot, streamlining the execution.
- Discrepancies or changes may require further work, thus necessitating a future date visit. If this is the case, appendices of changes should accompany the FAT document.
The need for testing can often require simulation. This could include simulated I/O (large switchboards with toggle switches, analog simulators, safety mats, light curtains), interconnection with other systems (fieldbus connections, interposing relays), polling drivers (Modbus RTU/TCP, BACnet for building automation, OPC servers/clients or others) or simply simulation efforts (forced I/O points or GUI inputs). If there are signals that must be checked, appropriate instrumentation should also be applied (serial breakout boxes, oscilloscopes, Wireshark captures, voltmeters). The need for diagnostic tools should be addressed in the FAT as a portion of the I/O testing, commonly where they’re applied.
Successful completion of the FAT means the system can be delivered. If the system is then to be wired in the field, the next step would be the site acceptance test (SAT), with real conditions and real-life equipment. A smooth FAT can lead to an equally smooth SAT.
Well-written specifications are crucial. This fact stands true for both sides of the contract. If the customer provides an incomplete or nebulous specification, or if the system integrator produces equally poor documentation, the GIGO (garbage in, garbage out) principle will promise both parties contentious phone calls and tense meetings. The production of good documents can be just as indispensable as the actual code writing. In light of this, if either party feels there is room for ambiguity, then the moment for discussion is now.
