Return to Continuing Maintenance

Carl Claunch's
"Status of the 1401 Data Center"
2018-2019

For about a year now (Nov 2019) Carl had been
issuing comprehensive reports about
problems and maintenance activities.
Of course many others have discussed individual incidents.

Newest at the top
- Nov 20, 2019
- Nov 13, 2019
- Nov 6, 2019
- Oct 31, 2019
- Oct 24, 2019
- Oct 9, 2019
- Oct 6, 2019
- Sep 19, 2019
- Apr 17, 2019
- Mar 27, 2019
- Mar 09, 2019
- Feb 20, 2019
- Jan 03, 2019
- Dec 26, 2018
- Dec 05, 2018
- Nov 14, 2018

On Nov 20, 2019, Carl Claunch wrote:
German 1401 System

The 1401 processor, 1402 reader, 1403 printer and tape drives 1 and 3 are working perfectly.

We suspect that we have a bad tape on drive 2. Using the tape exercise program we see the tape doing large numbers of retries for every record successfully written/read, for the first inch to 1 1/2 inch of tape from the supply reel, then settles down to work flawlessly for the remainder of the reel. This is repeatable week after week as well as in multiple runs per day. We expect to verify this by using a different tape on the drive.

Frank, Alan, Glenn, Marc and Rick worked on the 1402 card punch all day. After straightening the plate which we observed had been bent, we tried various washer heights as shims but never got the alignment knives at the proper height so that a card would pass over but not jamb against the base of the knife.

At this point we noticed that the rectangular block onto which many parts are attached was cocked to make the front where the knives sit open too much and the rear where cards would exit into the punch dies was closed blocking any card movement.

This rectangular solid is attached with two screws going into each of the two ends, through the sideplates that hold the entire 1402 punch area together. It could be rotated along the long axis as well as shifted forward, back, up or down since the screw holes were wider than the screw bodies.

The maintenance manual has no adjustments listed for this bar, nor are there any markings to use as a gauge. Frank bent cards over to use the doubled thickness to check fit and clearance, while I loosened the screws and twisted the rectangular solid bar to various positions.

We found settings that gave us uniform back and front clearance, with no shims at all under the front plate. Single cards moved easily through the mechanism, never bumping into the front of the aligner knives. The knives properly moved the card forward to its proper position to punch the first row of the card.

All now appears good, so the plan for next week is to put the punch head, punch die, Geneva drive and all the belts back together. Once we get all the pulleys and gears in time with each other, we can do some test punching.

Connecticut 1401 system

The 1401 processor, 1402 reader, 1402 punch, 1403 printer and the first three 729 tape drives are working perfectly.

Michael and Marc worked on the fourth tape drive, whose take-up reel mechanism had inadequate clutch strength due to lost magnetic powder. Michael found some reference material to pick the right powder consistency and amount to refill the clutch, then assembled it all. Marc and he placed the parts back on the drive, including replacement of a broken clutch brush.

Initial testing did not go well. The brake wasn't strong enough on that reel and the drive wouldn't load or unload tape. At this point the demo was about to start so we defer debugging until next week.

Other equipment

The 001 and all three 026 keypunches work well, as does the card sorter.

Carl


Ignacio Menendez commented as follows
Carl, Mike, Marc, as a bystander, let me offer you some things to try.....
  1. for the suspect tape with lots of errors up front, remove the amount of tape where you see the errors happen, cut that piece, add a new load point sticker. (Tape stripping)
    Ed comments: Yes, computer operators did that to "work tapes", somewhat monthly - cut off about 30 feet of the beginning of the tape, then put on a new "beginning of tape" foil. It was part of the folk lore of keeping a site running smoothly. For many/most jobs, only the first part of a work tape was used, many times a day. Over a period of years, the work tapes got inconveniently short, and were used for other things.

  2. for the drive with the clutch problem...
    - determine for sure at what phase of the load/unload this is happening...
    - There are two phases, stop clutch control (front clutches worm gear driven), and vacuum control, which over when tape in column is detected on BOTH columns.... there is when the fwd and rev clutches move the shaft, under control of the two other vacuum switches in each column.
Once you determine that in fact you are seeing the failure of either of the two reel clutches, we now have to determine if the failure is due to an electrical or mechanical cause....

I understand that you went through this already, hence the replacement of the powder in the clutch...

At this point, perhaps we might question if it was the powder, the clutch itself somehow, or maybe, yikes, it is an electrical drive problem, that we had discarded.

It is horrible to think, but perhaps you may try replacing that clutch that you changed the powder on, with another one from stock (I seem to remember that Glen showed us more spare clutches)

You might consider also that the woodruff key may not be providing a positive solid grab of the clutch to the shaft.

If replacing that clutch does not fix the problem, then perhaps it is time to revisit the electrical signal driving the clutch, again.

I hope that this helps, very elementary, and perhaps you already have thought and done it, just my two cents worth.

Iggy

On Nov 13, 2019 10:18 pm, Carl Claunch wrote:
Today we focused almost exclusively on the restoration of the 1402 Punch on the German machine. Glenn, Frank, Alan, Ken and I worked to understand why cards were jamming under the left aligner knife instead of riding up over the top of the knife.

Alan suggested that the bottom plate upon which the cards slide, might be bent. This was after many false steps investigating possible adjustments and time spent seeking enlightenment from the maintenance manual and the parts catalog drawings.

We found that it was indeed bent, with the left and right ends that sit over the aligner knifes bent downward and the entire plate was bowed. The knives were at the correct height but the plate was too low due in part to the bend.

We removed most of the bend but found that the plate itself was too low even when flat across. This plate had recesses and screw openings that appeared to have been amateurishly executed; not at all like the other 1402 nor was it typical of IBM workmanship.

We put two washers between the plate and its mounting bracket, raising it by about 15 mm. This was much closer to correct, but the left side was still a bit low. We removed the plate as our last act of the day and will bend it further and/or use fatter washers next week.

Michael Marineau was the victim of a car parked across his driveway in Berkeley, aggravated by a police response to his complaint that took all morning and a tow truck that still hadn't arrived by noon. Thus he was not able to attend today to work on the clutch for the take-up reel of the number 4 tape drive on the CT machine.

All other units of both the DE and CT machines were working properly. The 001 manual punch, the card sorter and two of the 026 keypunches also worked properly. The 026 nearest the door had cards jamming as they were feeding down from the hopper, but with cards manually inserted it was otherwise fully operational.

Carl

On Nov 6, 2019 4:55 pm Carl Claunch wrote:
German system

1401 processor, 1402 card reader and 1403 printer working well, although the printer is still leaking hydraulic fluid.

Frank King, Alan Futterman, Glenn Lea and I worked on the 1402 punch, beginning reassembly. It was a long and often frustrating effort because we hadn't tried all N factorial ways to assemble the N parts so it is still not together properly.

We went through many iterations of assembling things then discovering the next part had to fit below all the rest. Couple that with several trips to the parts room to look for screws and the day gets long quickly. We think we now know the proper order of assembly and will do that once we take apart everything next Wednesday.

Michael Marineau and Iggy (on a special visit from Ajijic) put the new bearing into the 729 drive vacuum pump and finished reassembly. They put it back into drive 3. We now have all three 729 drives on the system working perfectly.

Connecticut system

The 1401 processor, 1402 reader, 1402 punch and 1403 printer are all working properly.

The first three 729 tape drives are working fine. Michael and Iggy removed the take-up reel hub to repair the bad clutch. This required a trip to a hardware store to buy special long bolts used to slight a subassembly back for service.

The two were joined by Frank and Glenn who located magnetic powder and helped them install a good supply in the failed clutch. We found some very fine powder in the clutch but the properly working one had coarser powder. They mixed up our stock to get a good consistence and packed it into the clutch.
A comment by Michael Marineau
    "One correction on the 729 clutch, we were just guessing on which powder and how much and it was either too course and/or too much because now it doesn't spin freely. So we will need to retry filling it next week."

We should reinstall the clutch into the number 4 drive next week and hopefully return it to full service shortly thereafter.

Other gear

The 001 manual keypunch, 083 card sorter, and all three 026 keypunches are working properly. The 029 keypunch in the workroom is ready to be handed over to the Education department.

The 552 Interpreter in the workroom still needs work to get it to print the correct characters. We had no time to work on it this week.

Carl

On Thu, Oct 31, 2019 8:19 am Carl Claunch wrote:
DE machine

Frank, Alan, and Glenn continued the rebuild of the punch side of the 1402.

The third 729 drive is inoperative due to a defective vacuum pump. Michael and Marc disassembled the pump in the workroom to determine the problem. This pump is an odd looking stack of alternating driven fans and free rotating fans that feed the air to the next of the seven stages.

After study, thought and assistance from a remote support expert, they determined the problem is loose nuts that allow the driven fans to slip, thus not producing adequate flow. It is expected that it can be reassembled with a properly tightened nut, although the vacuum pump expert, Iggy, is flying in to help do this next week.

The 1401 processor, 1402 reader side, 1403 printer and other two 729 drives are working properly.

CT 1401 system

The fourth 729 drive is still inoperative. We didn't work on it this week.

The 1401 processor, 1402 reader side, 1402 punch side, 1403 printer and first three 729 tape drives are working properly.

Keypunches

The middle keypunch was repaired. A solenoid coil on the right side of the machinery is used to block cards feeding down when using the Reg key, when Autofeed is off during a Release, or when attempting a feed while two cards are in place at the punch station.

The coil side had been scraped each time the cover of the keypunch was removed and replace, to the point that the coil windings were broken. Glenn found a replacement coil and he, along with Alexey, installed the coil and adjusted the keypunch. It now works properly.

We believe the first and third keypunches are working properly, although we have had a report of DUP operations on the 026 furthest from the door occasionally adding a 12 punch. We are not sure how that happened and are not able to reproduce the problem.

Other equipment

The 001 manual keypunch and 083 card sorter are fully operational. The 029 keypunch in the Liebert room is fully operational and the 029 in the workshop works perfectly except that program drum operation is imperfect.

Team members working on equipment this week

Frank King, Alan Parsons, Glenn Lea, Alexey Toptygin, Michael Marineau, Ken Shirriff, Marc Verdiell, Robert Garner and Carl Claunch

Carl

On Thur, Oct 24, 2019 at 9:27 AM Carl Claunch wrote:
- Michael Albaugh's comments are in italics
DE machine

The 1401 processor, 1403 printer, 1402 reader, and tape drives 1 and 2 are working fine.

We are continuing, under Frank King's leadership with assistance from Alan, Marc, Glen and others, to rebuild the punch side of the 1402 system. This may take a few more weeks as the reassembly and then readjustment are complicated.

Tape drive 3 has been slow to respond to Load buttons and other actions, which appear to be due to problems with the vacuum pump. This is the drive that smoked and smelled a number of weeks ago which we attributed to a stuck pump motor.

Michael Marineau removed the pump and is disassembling it to sort out the problems. We can hear parts rotating inside when the shaft is stopped and they make a grating sound, plus metal filings are being produced. In the process of pulling one bearint to permit disassembly, the puller tool snapped. Clearly we need a bigger puller. we must wait until next week to continue this, so the drive is unavailable at this time.

CT system

The 1401 processor, 1402 reader, 1402 punch, 1403 printer and tape drives 1, 2 and 3 are all working fine.

It took a little encouragement to get Fortran to boot off tape. First try just kept moving tape when booted, even though the normal first block is pretty short. We had to resort to using the TAU debug switches to stop it (Reseting the 1401, or hitting RESET on the 729 didn't seem to work). After that, we did several tries and it seemed to work, but my memory is not exactly clear.
Tape drive 4 is still inoperable. We did not get to it on Wednesday.

Keypunches

The keypunch near the door is assembled but still sporadically misfeeds. What is happening is the failure of a card to release full from the punch (first) station. It doesn't slide far enough to be under the pinch roller that will register the card at the dup (second) station.

This leaves the card partially under the punch block where the newly registered overlaps it. That causes the original card to remain in place as additional cards are fed in and registered atop it.

We are trying to sort out the root cause of this. It could be issues with the new card stock, but more diagnosis is needed. This keypunch is usable but the operator needs to watch and stop feeding if any card fails to release all the way to under the dup station pinch roller.

A critical spring had been lost a week ago by me. We searched in vain but this week Alexey took up the search once again and finally found the errant part. It is now installed in the machine.

The middle keypunch was working early in the day but suddenly refused to register cards. Frank, Alexey and Glen traced this to the failure of a solenoid coil that permits cards to feed. Glen found a replacement solenoid but we ran out of time before the 3PM demo so the machine is currently inoperative pending the replacement next week.

The keypunch furthest from the door seems to be adding 12-punches when DUPing. I first noticed due to a validity check (and STORAGE error on 1401 console), but failed to notice the actual mis-punch. Thinking it was an alignment error, I DUPed the source card again, and again got Validity (and STORAGE) Ken noticed the added 12-zones, so I put back the original in the deck (I had duped it because it was the wrong color and I will probably be sending this deck off to the guy who set us on that poetry quest). All went well except for an error in the compile. You guessed it, an added 12-punch that changed a '2' to a 'B'.
Card Sorter and 001 keypunch

These are both working well.

Tape drive tuneups

Michael Marineau has been carefully adjusting the tape drives to the proper signal levels and fault-free operation. He has completed this on all drives except for CT drive 4 due to its failed status. This \has much such a difference that we can run Fortran compiles from the tape with virtually every try working perfectly.

In the past we would have multiple process checks and other errors running with the same tape,

Note that I use the Low Density tape (200 CPI) these days.
often taking many tries before one small program could be compiled.
I do not recall whether it was on CT or DE that I noticed that flakey compiles could often be "fixed" by using the PARAM card to tell the compiler it was on a 12K machine, rather than 16K. Of course, if you really need 16K, this is not an option. This was discussed way back when we were originally messing with Fortran.
Michael Albaugh used the compiler to run the Computer Poetry program, generated pages of rhymes.
Nit-pick. The original author does not call it a Poetry program, but a "Line Generator". The lines don't (necessarily) rhyme. :-)

(And still seeing the funny form-feeds: Page (56 lines), Skip to Channel 1, one line, skip to channel 1, normal-ish page (does not have enough lines to ask for another form-feed.

Note: in all this that I think the performance of these machines is great, and I'm very grateful to have access to them.

Thanks,
-Mike

Carl

On Wed, Oct 9, 2019 at 4:40 PM Carl Claunch wrote:
Today we made progress on several fronts as well as encountering a few setbacks. Ken, Marc, Michael, Robert and Bhushan joined me in addressing the remaining problem areas.

German 1401 system:

The 1402 punch is still out of commission, waiting on Frank and others to finish the timing and setup adjustments. The 1402 reader is working well, as is the 1401 processor, the 1403 printer and the tape drives.

Michael is doing a more complete and precise skew adjustment on drive 2, using the Master Skew tape. Currently the check bit track is 3 degrees behind the bit 1 track with no delays, which means that we should rotate the head slightly to further minimize that skew. Then he will use the delay circuits for all tracks until all signals are coincident.
Iggy is likely to comment on this.

Connecticut 1401 system:

The 1402 punch is still working well, as are the first three tape drives, the 1401 processor and the 1403 printer.

The last tape drive remains offline since we need to rebuild the clutch on the takeup reel. Because it was not working, it had been powered down, but that removed power from the terminator that sits on the drive at the end of the tape cable chain.

In addition, we found that the tape emulator box was applying pull down voltages but not pullup voltages to the lines when it was plugged in as the terminator. That produced tape signals which swung down from zero but never to positive ranges. A power cycle of the tape emulator box restored the pullup.

We decided to remove the tape emulator from the chain unless we are actively using the box. Further, we chose to uncable the fourth drive, power it down, and put the terminator for the chain on drive three.

Bhushan suggested that we clean and lubricate the clutch on the 1402 reader. It never gave us problems today, like a watched pot refusing to boil, but Bhushan recommended that step based on his experiences in the field.

The clutch was filthy - jammed with old grease and dirt. He cleaned it on the workbench, applied new lubrication and reinstalled the clutch, pulleys, belts etc into the reader. We tested it and all works well now, with less chance of erratic problems in the future. The clutch is fitted with a grease coupling, thus normal practice would push new grease in atop the old grease and dirt.

Michael and Marc began to adjust the tape drive signal levels, quickly finding the terminator issues above before continuing to set good levels. They worked on one of the drives, achieving great signal levels but that leads to skew errors. Next week they will use the Master Skew tape to adjust the drive, then repeat the level setting and skew fixes for the other two tapes.

Bhushan also spotted the same triple card feed problem on the keypunch nearest the door. He did some detailed diagnosis and found two independent problems, both of which are leading to the issue.

Sporadically, the keypunch clutch will not release and cause the machine to take a spurious feed cycle. Inside the clutch, a nonconductive separator sits between the clutch coil and the armature, but with time these wear down. If the armature begins to physically touch the metal core of the coil, residual magnetism can hold the armature closed even though power is removed from the circuit.

The check for that is to stick a sliver of punched card; with the sliver, we had no more double clutch cycles. This means we need to open up the clutch and check the insulating part or adjust the gap.

The second problem is that sporadically cards will be released from the punch station but won't move all the way out from under the punch dies. We need to open the punch cover, clean it and examine the drive wheel for possible flat spots or misadjustment.

We worked on the remaining problem with the 029 keypunch in the workroom, erratic operation with the program drum.

Bhushan rotated the drum so that the star wheels are correctly placed down in the holes at each card column. The keypunch became much better, duplicating or skipping long stretches of columns, but still won't work for a full card.

We have some remaining connectivity issue to address, thus the keypunch is not yet ready to be handed over to the education department.

The other two 026 keypunches in the datacenter and the 001 manual punch, as well as the card sorter, are working properly.

Carl

Sun, Oct 06, 2019 8:02 pm Carl wrote
This last week we had a visit from Bhushan Mohan who is an experienced FE supporting a wide range of IBM systems in India. He joined us in attacking a number of problems we were facing with the systems.

The sporadic problems of the keypunch nearest the door, where it would sometimes feed three or more cards, had been hard to recreate and troubleshoot. We spent time checking and adjusting the CBs and other electrical components, but once the machine began to fail, Bhushan quickly zeroed in on the real problem, inadequate torque from the motor.

It was failing to move the card that had just finished being punched, thus it was not making it under the pinch roller to register in the dup station. With the card still under the punch and new cards arriving from a feed, we were experiencing jams of multiple cards. Once it was adjusted we should not have the problems any more.

The tape drives on the DE machine are all in service now. The burning smell came from the vacuum pump which was stalled temporarily. Marc opened it up and cleaned out a lot of dust, after which it rotated smoothly and ran without problems.

The card punch on DE is still being adjusted - Bhushan and Frank spent quite a bit of time working to get it properly timed. What has been causing problems is the transition from smooth continuous card motion to the start/stop movement between the 12 card row positions during punching.

The rollers must firmly move the card to the proper point, then pivot out of the way as the Geneva drive provides the stop and start movements. If they pivot too early, we get card skewing causing mispunching, while a late pivot leads to bent card edges.

The card punch on the CT system is working fine. The card reader on CT did not exhibit the bad behavior where it fails to feed on a Load or Start, although Bhushan was eager to trace the problem but it perversely refused to fail all day.

We worked on the star wheels (program drum) on the 029 keypunch in the workroom, but even with cleaned contacts we are still experiencing erratic contact operation. Will continue to work on this next week.

The far right tape drive on CT was failing to take up tape fast enough as the drive did reads or forward moves. The other three drives are working fine. We removed, tested and cleaned the vacuum switches in the columns. We used the scope to verify the current flowing to the clutch magnets, comparing the bad drive to one that works well.

We found that all three clutches on the takeup reel (stop, up and down) were receiving the proper current, somewhere in the range of 270 to 300 ma, but the up clutch wasn't providing a firm hold.

Sadly, this confirms that the problem is inside the clutch. These clutches are filled with a supply of iron powder and have a common failure where powder is slowly lost until there is insufficient supply in the hub to work properly. If it occurs quickly there is a gray powder visible on inspection, but this seems to have been a long and gradual loss.

IBM engineering, in their wisdom, did not put the refill port for the clutch at the top, therefore we have to disassemble and remove the entire shaft and three clutches in order to refill the errant one. This will take a few weeks, during which we will only have three working tape drives on each system.

Long term we want to rebuild a hydraulic unit for a 1403 printer, then when it is in good shape, swap it into the 1403 printer to stop the steady leaks. Some of the work can occur while the printer continues in operation, but once it is time to swap it in and adjust it, the printer will be out of operation for perhaps a week.

Carl

On Sep 19, 2019, Carl Claunch wrote:
CT SYSTEM

The 1401 processor is working properly

All four tape drives are working properly

The 1403 is working properly.

The 1402 punch side is now working properly.
It would not NPRO, raising a punch stop immediately without spinning the motor. We guessed that it believed there was a jam that couldn't be properly ejected with a feed, although there is no indicator light to show this situation. We manually cranked the punch, ejecting two cards which we believed were too close to each other. The punch was tested both with NPRO and by punching output.

The 1402 reader side continues to have intermittent problems but is in service.
The problem manifests itself by a failure to take feed cycles - the load button is pressed, the motor spins but no cards are fed. This also occurs with a read op code in the processor and Start pressed on the reader.

This problem occurs sporadically, lasts for a bit of time and then maddeningly goes away as we attempt to trace the issue to its root cause. By the end of yesterday, it appeared that the problem was a fault in the Interlock Stop latch which should be reset by the Load key or Start key but remains on. The fault disappeared before Ken and I could determine why the latch is erroneously on.

Often a power cycle of the 1401 system will make the problem go away.

DE SYSTEM

The 1401 processor is working properly.

The first and third tape drive are working properly.

The middle tape drive requires adjustment of skew delays between tracks to properly read and write tapes and is currently out of service.
Michael has been adjusting all the tape preamp and amp levels to the target of approximately 8V, up from seriously low levels on all three drives. Once we restored the signal levels, we encountered skew errors.

These mean that the time from when the first 1 bit arrives from some track and the arrival of the last 1 bit from any other channel has exceeded some maximum acceptable delay. This requires us to adjust all tracks to cause their bits to arrive as close to simultaneously as we can.

Adjustment is done by selecting one of a number of taps on a delay card - choosing among various predefined delays. All seven tracks have a delay card. We would watch the bit arrivals from the tape head for groups of tracks. to identify which were the furthest apart and begin changing those targeted tracks to bring the bit arrivals closer together.

Marc discovered some "Skew Test Master" tapes in the workroom which should give us known good patterns for this work. We would have tracks that were written as close to simultaneously as possible, with all tracks having a 1 bit recorded.

The 1403 printer is working properly.

The 1402 reader side is working properly.

The 1402 punch side is still being adjusted and is currently out of service.
Frank, Dale, Alan and others are going through the 1402 maintenance manual step by step, adjusting the timings of the punch. The initial punch checks we were receiving were a result of mistiming in the mechanism that allowed cards to slip and skew.

The punch moves cards two ways - continuously and with stop-start motion - which is what makes it mechanically complicated. The normal smooth motion takes place as cards are fed into position to punch the first (row 12) holes, and again when the finished card is moved though the read station and then into the stackers. Stop-start motion occurs to freeze the card while a row of holes is punched, the step the card forward to the position for the next row.

The stop-start mechanism is built around a Geneva drive. The pinch rollers that cause smooth card motion are moved out of contact by cams to cause smooth motion to cease, then dropped back in place to restart smooth motion.

This must be well synchronized since we want cards to roll up to the point where we begin punching holes for row 12, the stop-start mechanism to hold the card still and the cams to pull the pinch rollers out of the way stopping the smooth motion.

If the smooth motion continues when the stop-start is holding the card still, we fold up the card into a jam. If the smooth motion stops at the wrong time, the card may be mispositioned for punching row 12, causing a miscompare when the card is read after the punching is done.

The adjustment consists of lots of hand cranking of the mechanism, then adjustment of various screw settings, clamps and belt positions. The team is partway through the adjustment sequence. Once complete and after verifying punch operation, we can return this unit to service.

CARD SORTER

The sorter is operating correctly.

KEYPUNCHES

The two keypunches further from the entry door are working correctly.

The keypunch near the door is out of service while the keyboard is repaired.
This keypunch would lock up from time to time, with the problem becoming more frequent. We opened the keyboard to determine the root cause and found a worn part for the Space bar mechanism.

The Space bar pushes down, rotating a lever to pull a long bar forwards towards the keypunch operator. This bar will trip a vertical bar in the rear of the keyboard which falls down. That swivels a number of horizontal bars in the permutation unit attached to microswitches. For some of the keys is also activates an individual microswitch under the vertical bar.

The wiring of the microswitches in the permutation unit will convert a single pressed key into activation of some number of solenoids in the punch unit. The 'A' key will be converted to a 12 and a 1 punch, for example. Dual keys (e.g. U key is also the number 1. When in Alpha mode, this fires the 0 and 4 solenoids but in Numeric mode it fires the 1 solenoid instead.

The long bar moving forward pushes apart small ball bearings in a horizontal chamber. These are packed together tightly enough that once a single key's bar as entered, there is no room left to let any other keys move. This ensures that one and only one key is pressed at a time.

The forward movement of the long horizontal bar and the resulting downward movement of the vertical bar locks the key in place. Once the keypunch has responded to that keystroke by punching holes in a card, a solenoid under the keyboard is activated and it moves the restoring bail, a bar that pushes all the vertical bars up, all the long horizontal bars forward and the keycap back into its full up position.

The interaction of the long horizontal bar and the vertical bar depends on a notch in the horizontal bar. This holds the key down until the restoring bail is activated. On the bar for the Space key, the notch is worn into a sloped ramp not a sharp edge.

The result is that pushing Space will trip the vertical bar but latching the horizontal bar in place is temporary. Because of the slop, it easily slips off and unlatches. The restoring bail will fail to push the vertical bar back up if the horizontal bar isn't latched, thus the keyboard remains stuck.

We have a spare 026 in the workroom whose keyboard we removed to supply replacement parts. The long horizontal bars engage with the vertical keystems under the keycap. The keystem is inserted into the top, fits around a coil spring and then emerges through the bottom plate. Nylon fishing wire is laced through all the keystem bottoms to keep them held in the keyboard.

We had to remove the fishing wire, pull all the keys out, extracting the coil springs then opened up the upper and lower plates to get to the long horizontal bars. It was easy at this point to replace the worn space bar with the good one from the donor keyboard.

At this point Alexei has put the plates together but has to tediously inserted each keystem, holding the spring in place with tweezers, then thread the nylon line through its bottom. He has done part of a row but will continue on the weekend trying to finish rebuilding the keyboard.

You might ask why we didn't just swap the entire spare keyboard for the worn one? The wires to all the microswitches on the keyboard are soldered in place. This bundle runs down and is laced together with many other wires from the rest of the keypunch, ultimately being soldered to the strips inside the base where the tube logic resides. We judged the unwiring and rewiring process to be more difficult and time consuming than all the work detailed above.

MANUAL KEYPUNCH (001)

The manual keypunch at the railing in the computer room is working correctly.

029 KEYPUNCH IN LIEBERT ROOM

The 029 is working properly

NEW 029 KEYPUNCH IN WORKROOM

This 029 has multiple errors and needs restoration.

It is likely that the problems stem from oxide on the CB (cam actuated switch) and relay contacts, but we need to diagnose the conditions to be sure. This is a low priority task that will only be worked on when everything else is running well or a team member is particularly bored.

552 INTERPRETER IN WORKROOM

The interpreter is in the midst of restoration, not yet ready for service

This interpreter will take in punched cards, read the characters in the columns and print 60 of them on either of two lines across the top of the card. We intend this device to be used to print cards that were punched on the 001 keypunch by visitors. It uses a plugboard which we will wire to print the first 60 columns of a card.

At this point the basic mechanism is working and it reads cards, but the printing is a bit off. We suspect this is a matter of adjustment, since many of the characters are printed properly.

The device works with sixty long vertical typebars. These are pushed up to their top position at the beginning of a card cycle and then allowed to drop, being stopped by solenoid activated levers that hold a bar from moving further downward.

To add to the complication, the typebars have four vertical zones. These are for columns with a 12 row punched, an 11 row punched, a 0 row punched, or none of the above. Thus the characters A-I are in the topmost zone, J through R are in the second zone, /, number 0 and S through Z are in the third zone, and the digits 1 to 9 are in the fourth zone. Oddly, 0 is not part of the numeric (fourth) zone.

Once the bar is held in the 12, 11, 0 or numeric zone, further timing lets it drop so the solenoid can now select which of the nine values in that zone are printed. The proper zone is always selected but often the value within that zone is off by one. That is, a card column having A will select the 12 zone but the printed character is B not A.

We don't have an adjustment manual for the device which complicates the task of figuring out how it should work and why it is misbehaving the way it has. This is a low priority restoration meant for spare time or bored team members.

Carl

Wed, Apr 17, 2019 6:22 pm Carl wrote
Today Michael Marineau, Frank King, Ken Shirriff and Carl Claunch were joined by a new team member, Alan. Alan is a hardware engineer who worked for Fairchild and a plug compatible add-on memory vendor who sold memory upgrades on S/360 and S/370 machines.

Frank and Alan worked on one of the keypunches which had been reported to have triple fed cards from time to time. They were not able to reproduce the problem after extensive checking, but made adjustments to the 026 just in case it was marginal.

Ken and Michael continued their work hunting down the problems with the third tape drive on the CT machine. It sporadically drops ready. One challenge with finding the problems stems from the frustrating behavior of the drive - generally the problem disappears in the early afternoon and won't come back until the next day.

We continued to trace the failure backward using the new four channel scope donated by Marc. We found that a signal "tape in left column" was dropping out producing the loss of ready state.

As we put the scope on the air switch on the left tape column, the drive chose to stop its failures for the day. We did incur one failure soon after wiring to the switch, but it then stubbornly refused to fail any further.

The switch is vacuum operated and will connect the signal line to ground when there is a vacuum in the column, i.e. the tape is loaded in that column. If not actuated, the line is pulled to its -6V level by the MH card that first processes its signal.

We verified that the line is -6V when no vacuum exists and it jumps up towards ground when activated. However, we noticed that it wavered and dipped down to about -0.5 V, unlike the other similar switches which stay nailed at 0V.

The logic level going into the MH card is clipped SDTDTL whose voltage specs require that a high level be no lower than -0.5V to be valid. A low level has to be no more than -5.8V. Any value between -0.5 and -5.8V is the exclusion zone, the indeterminate level where the behavior of logic gates is not guaranteed.

Even though we couldn't catch the drive failing to prove that the sagging level is causing the problem, we know that this is not a good behavior. The vacuum switch has two sets of paralleled contacts, thus the only way to have the signal pull away from ground is if the resistance through BOTH contacts is high enough to yield the observed voltage drop.

Michael removed the vacuum switch, brought it to the workroom and carefully cleaned and tested both sets of contacts. Once they were operating with almost zero resistance, he reinstalled the switch in drive three.

While we can't definitively test this until the next work session when the drive will again be amenable to showing the failure, we did observe the switch to have no waver, no sag of voltage levels and reliably sit at ground level.

The conclusion is that, subject to verification next week, we have found and resolved the flaw causing the tape drive to drop ready.

Frank King brought in a repaired wood card joggling tray which is attached to the top of the 552 Interpreter cover. We believe we have successfully re-inked the ribbon for this device; Frank is giving it a final treatment to even out the ink level across the span of the ribbon, which is a bit wider than a punched card and many feet long.

Robert Garner did some helpful reorganization of the workroom, which he will continue in the following weeks until the workroom is at peak usability.

Carl

Wed, Mar 27, 2019 7:23 pm pm Carl wrote
Today we had a good turnout of volunteers. Frank King, Dale Jelsema, Bill Flora, Alexey Toptygin, Michael Marineau, Ken Shirriff and me. Rumor has it that Robert Garner's dog didn't have a suitable raincoat, so we didn't have his company today.

We ruined two red docent shirts inking the ribbon for the 552 Interpreter. Frank brought some ink refills for stamp pads, we diluted it in Isopropyl Alcohol and saturated the ribbon with it. The ribbon is hanging over some ethernet cable strung across our light fixture at the workbench, drying before we can reinstall it in the interpreter.

All keypunches were working fine, both CPUs, both 1402s and both 1403s. One of the tape drives, number 4 on the CT machine, had been misbehaving and got much, much worse while we worked on it.

The drive takeup reel didn't move strongly enough to keep up with the tape dumping into the right column during move forward, write or read operations. We examined everything and saw no obvious problem. The vacuum switch on the column works fine, the belts are newly replaced and the graphite brushes appear to have plenty of life left in them.

The drive then acted up, refusing to release the stop brake on the right reel so we could load tape. Oddly, the left (supply) reel is not locked even when it should be. Voltage on the right stop clutch was about 7V, definitely wrong.

We had been viewing the waveforms of the clutches on this drive, and others for comparison, but reached no conclusion in the early hours before the new problem arose.

The release button works properly, it just doesn't change the behavior of the two stop clutches. We tried to use the current measuring panel built into the drives; this puts each clutch through a 1ohm resistor and brings the two ends out to + and - banana jacks.

Using a voltmeter, we could verify the current, which was the voltage reading since IR with a resistor of 1 ohm makes V and I the same. The current reading on the bad drive was 0 for both left and right stop clutches, but as we mentioned the left was free and the right activated.

Each clutch has two paths for current - one used for full braking force and the other for partial braking force. Essentially, one mode is during Load and Unload, while the other is for normal tape operations. The current measuring panel only looks at one of the two paths to each clutch.

We searched through the schematics for any parts that were asymmetric - that could give different results on the left and right stop clutch. We were starting to check capacitors and other components and begin detailed voltage measurements when the clock hands approached 3PM. We surrendered for the day and will work on this again next week.

The demo team reported that the tapes weren't accessible by either program or TAU functions, although we only had the last drive switched off because of its problems. It may be that we had disrupted the chain or some more involved failure in drive 4 was jamming the tape bus even with power removed.

The demo team mentioned receiving some read checks on cards they fed into the CT 1402, although we had experienced no failures in our tests earlier in the day. We don't know if that was a problem with the deck or the reader, so we will watch it more carefully next week.

Tape drive number 2 on the DE machine had a bad door latch, something we wrestled with last week. This week, the team (mainly Frank, Bill and Dale) did some adjustments, grinding of excess metal and reshaping to get the latch to work properly and easily.

There is a cover inside the door that needs some trimming as it interferes with the newly adjusted latch plate - that will be done at home by Frank or Dale.

Carl

Sat, Mar 09, 2019 5:32 pm Carl wrote
Frank King, Michael Marineau, Ken Shirriff, Alexey Toptygin, Robert Garner and I were at the museum on Wednesday.

The number 4 tape drive on the CT machine had broken belts that were replaced the prior week, but the reconnection of all the cables and other parts was completed on Wednesday. The drive works well now.

The CT 1402 reader was jammed from a previous demonstration. We removed the bits of cards and verified that the reader was back in service.

Work continued on the 552 Interpreter, which had its print unit removed leaving the bottom bar with the 60 friction rods dangling in the machine. We further disassembled the unit to get the bottom bar out.

The friction rods were removed, soaked to clean and reinstalled in the bar with fresh grease. Other areas that had stale grease were cleaned as well. Next week we will insert all 60 friction spring rods into the print head, then reinstall the newly reunited parts into the interpreter.

We have the print ribbon removed but it is dry with barely any ink to transfer to cards. We are going to investigate methods of reinking this, probably using stamp pad ink refills diluted with appropriate solvents that will soak into the ribbon.

The workroom was further straightened and many objects were put onto the new shelving.

We swapped the formica covered desk from the Liebert room 029 with the donor machine that we are restoring in the workroom. This unit is now in very good condition and almost ready to hand over to the education department for use by the public.

We have dollies that we are going to be used to move this keypunch around. Frank has the appropriate spade bits to drill a hole for the four keypunch feet, which we will attempt next week.

We tested the tape drives on both systems writing and reading back patterns, as well as movement tests. They work well. There is a bad spot on the tape on CT drive 2 which results in a write check withing a dozen or two writes from the load point, but that is not a tape drive fault.

We did notice that demonstrators are setting all tape drives to the same number to drive them all from the TAU, which works okay for Move Forward commands but wont' work for other commands such as Rewind.

This can lead the demonstrators to report the drives as malfunctioning when one of them runs away or fails to rewind. Remember that multiple drives on the same number violates the design of the system; the fact that you got lucky in the past is no guarantee that it will work the next time.

Carl

Wed, Feb 20, 2019 6:36 pm Carl wrote
Hi All

Today we had Michael Marineau, Ken Shirriff, Frank King, Marc Verdiell, Robert Garner and our newest team member Jorge (sorry, don't remember his last name nor email address) joining me.

We had reports of failure to advance cards on the 026 closest to the door. Ken and I cleaned out the punch area with a card saw and it appears to be working well now.

Pat Buder had reported some BigPrint runs where visitor names were printed with numbers replacing certain letters, due to ignoring the 12 punch in those columns. He saved one card where this happened and gave it to us.

It is a bit puzzling how this could be misread without causing an error stop. We should verify that the console switch is set to stop on error. The reader was able to transfer the entire program and have it run correctly, plus it failed on only certain cards.

The card was in perfect registration and Pat reported that a rerun would usually result in a proper print of the name. Frank King checked the timing of the card reader pulses on the CT 1402 and they were right on the money. We will look into this further next week but the failure mechanism remains a mystery.

Frank King did some adjustments to the card sorter to make it operate better.

We continued to work on the 'new' 029 keypunch brought to the museum by Frank King. It started the day with problems with its Clear function. We used the opportunity to work with the two newest members as a training opportunity, showing them how to work through diagnosing the problems and understand the schematics. Mike and Jorge did identify problems with the PCC microswitches at the base of the program drum unit and removed the oxidation. Ken, Frank and I provided guidance but mainly sat back. Mike also repaired the latch so that the card feed cover could be mounted on the machine.

The print support SMS card for the keypunch was successfully tested today. This card was missing from the machine and we had no spares on hand. I had taken a blank SMS card and built up the circuit - mostly diodes to quench the reverse EMF of the print solenoids, but also an RC debounce filter. The first attempt at this card made use of diodes we had on hand, but we tried to use small diodes which couldn't handle the reverse EMF and immediately committed suicide. After I bought and brought diodes with high enough current and PIV ratings, I fixed it up and we can now use it with this keypunch.

We have been watching the tape drives on CT for some time, as they are exhibiting intermittent problems particularly drive 3 (and 4 to a lesser degree). Ken and I put them through their paces but they were perversely well behaved when we were watching.

Finally, Robert, Ken and I assembled the three new storage racks and put them in position inside the workroom.

Carl

Thu, Jan 03, 2019 1:35 pm Carl wrote
We came in to work on two intermittent problems today. 1 - The 1402 reader on CT will occasionally spin the motor on a LOAD operation without attempting to clutch and feed cards. 2 - CT tape 3 would drop ready during operations.

Present were Frank King, Alexey Toptygin, Ken Shirriff, Robert Garner and me. Robert was busy taping an oral history through the morning.

Alas, we could not get either of the problems to happen today. The tape worked flawlessly as did the card reader.

Frank, Ken and I finished building an SMS card for the 029 keypunch. This card provided nine diode suppression circuits for the print solenoids plus an RC circuit. We did not have any spares of this card but built it from a card blank, the schematic and spare components on hand.

The 029 was powered up and the cleaning and adjustments began. It appears that the relay as well as many microswitches have corroded contacts.

Robert is ordering some more cabinet racks and further cleaning and arranging the workroom to improve its usability.

While testing everything in advance of the demo, I discovered that tape drive 4 on CT is now failing. It selects but fails to respond to commands. In one case,, I issued a rewind/unload from the TAU, the drive turned off ready and sat there for a couple of minutes and then did the unload.

We didn't have enough time to chase this new tape bug. Perhaps next week at least one of the tape bugs (drive 3 or 4) will be present to allow us to fix it. Iggy, your tape drives miss you! At a minimum we can make progress on restoring the 029 keypunch.

Carl

Wed, Dec 26, 2018 4:30 pm pm Carl wrote
Alexey and I were the only team members present today.

We replaced the lamp for the CT machine bit 1 in the op code register, which was burned out. Thanks to the demonstrators who spotted this.

We ran various tests to check the machines. We spotted two problems that must be attended to, however we ran out of time before Alexey had to return to work.

Problem #1

The third tape drive on the CT machine will drop out of READY after operations. Holding the READY button down allows it to operate properly. This occurs, with move forward, read and write commands.

When running with multiple drives set to the same address, the problems with the third drive produce various symptoms. That drive might ignore commands that are processed by the other three. Sometimes that drive responds to a command, e.g. rewind, while the other three do not.

Our working hypothesis is that the drive will drop out of ready for any command and should restore ready status at the end, but is intermittently failing to do so. No TAU errors are recorded when the drive drops ready, but it always occurs when a command is being issued.

Problem #2

The CT machine card reader will intermittently fail to load, not even picking cards. It just runs the motor. Other times it works fine. Since we had a bad relay in the 1402 that we had replaced the prior week, it is the most likely suspect.

Safety rules forbid a single person from working on the energized machine if there are no other team members present. I therefore left after lunch.

Iggy - any ideas?

Carl

Wed, Dec 05, 2018 5:19 pm Carl wrote
Frank and Alexei worked on keypunch #1, where the friction clutch was set much stronger than spec. They adjusted it to the correct value.

Marc and Mike worked on tape drive #2 on the CT system. It had been tripping the +6V circuit breaker right after power up and was inducing errors in tape operations on other drives as long as it was in the string.

They isolated the drive and pulled groups of cards in a binary search pattern until the located the AR card that was tripping the breaker. On examination we found a 6.2 uf filter capacitor had failed in a short circuit from +6V to ground.

A replacement capacitor was installed on the card, it was placed back in the drive and we verified that it could be returned to service.

Ken finally discovered a hairline crack in a trace on the WW card, the second sense amplifier card in the 1403. This was causing intermittent connectivity, leading to the sync checks and other printer errors on the DE printer.

Last week, while working with the as-yet undiagnosed card, we smelled smoke from the printer. Testing today showed that four hammers were not firing - columns 2, 9, 13 and 85.

The fuses on the four hammer driver cards were blown. We believe the intermittent operation of the amplifier card let the 1401 set the hammer drivers on but fail to reset them in time because the subsequent pulse never arrived.

The four fuses were replaced and the printer now prints on all 132 columns. We do see, however, that column 85 is printing way too far to the left. This may be a damaged coil and the source of the unknown smoke from last week. Next week we will investigate and either adjust or replace and adjust the coil for column 85.

The DE 1403 had some light bulbs that were burned out, such as the Print Ready status. Ken located the bad bulbs and we looked for replacement lamps. We knew that these operated on 10-12 volts at a draw of 40 ma, so we set up a current limited power supply in the workship and sorted through our lamp stock until we found the appropriate bulbs. The new bulbs are in place.

We do have a defect on the CT machine that needs to be researched, but it is not causing any problems right now. When the 1402 reader has no cards in the hopper, the Reader error light on the 1401 panel will turn on. It goes off when cards are placed in the hopper. While it is on, there is no error condition lit up on the 1402 panel. This is not what happens on the DE system.

Carl

Wed, Nov 14, 2018 7:09 pm Carl wrote
CT would not power up, similar to the problem we had weeks ago with a oxidized contact, but this time on relay 4 rather than on relay 1. Sanded and system back in operation.

DE has printer sync checks. Discovered near demo time

Tape 1 on CT worked fine but reflective tape rather far in, dumping probably due to insufficient winding while loading.

Tape 2 blows +6V breaker. Breaker tested to 4A without trip. Two wires run from breaker - one to main logic panel and one to preamp board. Removed both and breaker still trips!!

Ran out of time to look further due to demo

Carl