Incident at the Paper Mill

Preface

In the first installment of our stories from the partners section, comes a tale from the 80s that is both entertaining and informative.

When I was much younger, I was sent to a paper mill in the north of England to investigate and fix their automated tractor. Upon arriving I was staggered at the size of this monster machine, designed to carry around 30-ton rolls of paper and load them into lines.

In this industrial environment, "Lines" refers to a series of machines that take a massive roll of paper, and then cut and shape it into A4, envelopes, notebooks, book paper, etc, and there were 8 lines in this factory. The image to the right is unfortunately not of the factory, but it's an illustration.

The tractor picks up rolls of paper from two bays and then transports it to the required line, and loads it into the spooler (being the thing that holds the roll and feeds it into the line).

The issue

This tractor takes simple instructions via radio, in the form of from bay to bay, and then heads to the pick-up bay, receives its roll, heads to the line and unloads its roll.

There's no fancy lidar, radar, or cameras, instead white and yellow lines on the ground allow the tractor to follow a path and various stripes indicate the location. With such simplicity, what could go wrong?

It just stops.

Then needing two forklift loaders to be driven in, take the paper roll, return that to the loading bay and finally power cycle the tractor and allow it to find its way back to the home position. A time consuming and dangerous process that the company wants to avoid.

The Investigation

The person who put this thing together had left the company, but I was handed a thick folder with circuit diagrams and two floppy disks with the software on. I took the weekend to trawl through this, and try and form a theory as to what was wrong. Unfortunately, I couldn't see anything immediately that could cause the issue in the hardware, so it was time to look at the software.

Written in GNU C, the code was surprisingly basic for the functionality. We had a main loop that ran through a series of functions to read status information, check reflective sensors, check for a radio buffer, update some lights (that didn't physically exist), and send commands to the tractor. Without a PIO, input and output was via a series of address mapped latches picked up in hardware with 74 series logic and moved in and out of the data bus, which sounds archaic but was fairly common at the time.

Why?

The first step was to identify what was causing it to 'stop'. Is it hardware, software or both, and can we not recover from this position without all the hassle?

A logic analyser would be a great tool, but unfortunately at the time they were excessively expensive, so all I had was a scope and a logic probe. Taking my extensive toolset to the tractor, which had nowhere to sit, I rode around for about 4 hours until it came to a halt, at which point I confirmed the processor was running, address/data lines were flitting (is that the right word?) and I could confirm that it was no longer responsive to any external sensor, so its a software crash right?

Well, not necessarily, so I swapped out the 8080, and the 4k of RAM and gave it another run. After a day it was still running fine and a sense of optimism was beginning to take hold when - its stopped again.

The Code

Looking at the source code in more detail I could see a few places where it could potentially get 'lost' simply because of the way it was written. Input was read and written into ram address space, and then other functions checked these values and made decisions by setting other addresses, which was then read by other functions to determine what command should be sent to where. That's not an unreasonable way to code it, but it can lead to ambiguous conditions because nothing was interrupt driven. For sensors that read the reflectivity of floor tape, for example, it reads either zero or one from the actual sensor which is latched into address space, together with a timer value that is incremented on each loop through. This provided something like sensor1 == 1, 100ms. When we read this again and it was no longer 1, then we'd push the 1 and the timer into a buffer which gave us something like; 1 100ms, 1 50ms, 0 50ms, 1 50ms, 0 100ms, so that's 1 long, 1 short, 0 short, 1 short, 0 long, and this would indicate that we're passing bay 5. The long 1 and 0 indicate the start and end of the location, the 3 bits in the middle are bay number.

This would have been better/easier to do in hardware, pushing the bits into a shift register with a 555 timer clocking it, then shifting the register into address space when full, but for whatever reason it was done in software.

Another issue was all this extra code that seems to have been a work in progress but is never called, that's not unusual either but it can make it hard to read. So, I removed all the stuff that's never called, and I removed the code that updates the lights, because we haven't got any lights, and added a bunch of comments in what was left, to help me work through it.

Now, why is it crashing? The only possible hole in the code is 'what if' we get a location that is not in sequence. That is, when moving down the factory we expect the bays to increase, bay 1, bay 2, bay 3 etc, and likewise if we're reversing back up the factory, then we'd expect to see this decreasing bay 3, bay 2, bay 1 etc. There is a check that makes sure for FORWARD, location should be > the last location, and for REVERSE, location should be < last location. If this test fails, the code sets location to null and since the forward signal depends on there being a location (so it won't move unless it knows where it is) then it will stop with other functions desperately cycling looking for more floor codes to read.

With a working theory, it was time to recompile the code, burn it onto a EEPROM (Yes I know) and then replace that in the tractor. At the same time I noticed that the supply voltage was low, the tractor was designed to work on 24v, yet we're only seeing about 18v supplied to the electronics, which then regulates it down to 5V. This was found to be a missing battery, which was in this vehicle 4 x 6v with only 3 connected. In discussion with the on-site maintenance we replaced the missing battery and measured the correct 24v.

The Test

With the new firmware, which really hadn't changed much, and the fixed batteries, the tractor was powered up and it lurched back until it found its stop line. We eagerly issued it a command to collect from bay 5, and deliver to bay 10.

With a renewed vigor the tractor made its way down the machine hall reaching the turn for bay 5, it stopped, rotated 90 degrees and headed in. After a little alignment it received 1 30ton roll of paper, and backed out until it reached the main path. Another -90 degree rotation and a little alignment and it took off towards bay 10. It should be noted that with 24v it was considerably more spritely, which was nice.

The Incident

About half way between bay 5 and bay 10 the tractor stopped, and not a slow stop but a sudden and immediate stop, which in itself isn't a problem, but anyone with even an even basic understanding of physics can appreciate that a 30 ton roll of paper is going to be less willing to come to an immediate stop, and therefore it jumped from the forks holding it and after hitting the factory floor with a thump that could be felt several buildings away, it started its slow but deliberate journey down the hall.

The staff, who seem surprised by nothing, moved slowly out of its path, pulling the odd bin out of the way and we all watched its progress. It is worth saying that the trip from where it was 'released' to the end of the machine hall was about 30 metres, and yet our paper roll carried on regardless in a slow but deliberate progression.

At this point, my attention, and that of those around me turned to the wall that it was approaching, slowly but deliberately, and there was a moment of panic when some people thought that something should be done. Nothing could be done and we collectively watched our beloved paper roll exit the machine hall through a newly created doorway, about the size of said paper roll.

At this time of course there was a fair degree of panic, and we rushed to the front of the building to see where it had stopped, only to find that it hadn't yet stopped and had passed through the training room, the locker room, the gents toilets and was now heading across the car park.

Whilst the factory floor was very flat and level, the car park was not, having a clear slope away from the building, giving our runaway some extra momentum as it remodelled a Jaguar and a Granada before heading out of the company through a wall and into wasteland.

The Aftermath

Now I'm sure many of you are now fairly sure this way my fault, and technically it was, but there were mitigating factors. Firstly, the circuit diagrams were outdated and not current, the code was even more outdated and a much later version clearly existed 'somewhere', but I can only work with what I'm given.

Secondly, the reason we only had 3 batteries connected, it transpired, was to solve the problem of sudden stops because with only 18V there wasn't enough to (a) apply the park brake, and (b) travel as fast, and rather than fix the code, it appears this was the 'fix' for escaping paper rolls.

Response

Was I asked to leave or escorted from the premises? No, they'd seen it all before and I was given the later code which was apparently on site all along, together with some sketches of newer circuit diagrams and changes.

I decided to simply start again, not having any confidence in the code or electronics, but the electrical hydraulic systems were fine. I used a Z80 development board, a Z80 PIO and some Veroboards with logic on them to reproduce the functionality. I kept the floor markings, because they worked but added some IR sensors to all sides to detect people, and I built logic to read in the floor codes, and handle them through the PIO from the shift registers.

The software was of course written in C, and took about 2 weeks to complete, after which the tractor had the added ability of handling multiple requests at once; e.g. Bay 5 to Bay 10, then Bay 5 to Bay 4, etc, which pleased everyone. I also had a granular control on the power, so it would slowly accelerate/decelerate and I used interrupts in the way they should be used to handle when it lost sight of one of the side, front or rear markers.

The system was handed over, with complete circuit diagrams and commented code, and my time was up. By this time of course the walls were repaired and the gents didn't have to take it outside any more - it was winter.

Summary

Its worth saying that this company was decades ahead of the competition, with an automated system that loaded these rolls into the machines, and those machines ran 24/7 producing paper products, with only a few staff that were there to monitor and to offload the resultant stock to storage.

It is still one of the most interesting jobs I undertook, and my first foray into robotics, and on a massive scale. Since then I've been involved in quite a few automation/robots projects but nothing quite as hands on as this, and I sincerely appreciated the assignment.

Blog Index   PermaLink   Comment           12 Votes  

Comments (2)