Return to Continuing Maintenance
Carl Claunch'sFor about a year now (Nov 2019) Carl had been
"Status of the 1401 Data Center"
issuing comprehensive reports about
problems and maintenance activities.
Of course many others have discussed individual incidents.
The 1401 Data Center is closed "for the duration".
Newest at the top
- July 29, 2020
- Jun 10, 2020
Wuhan virus break
- Mar 11, 2020
- Mar 4, 2020
- Feb 26, 2020
- Feb 12, 2020
- Feb 5, 2020
- Jan 29, 2020
- Jan 22, 2020
- Jan 15, 2020
- Jan 8, 2020
- Dec 19, 2019
- Dec 11, 2019
- Dec 6, 2019
- Nov 27, 2019
- Nov 20, 2019
- Nov 13, 2019
- Nov 6, 2019
- Oct 31, 2019
- Oct 24, 2019
- Oct 9, 2019
- Oct 6, 2019
- Sep 19, 2019
- Apr 17, 2019
- Mar 27, 2019
- Mar 09, 2019
- Feb 20, 2019
- Jan 03, 2019
- Dec 26, 2018
- Dec 05, 2018
- Nov 14, 2018
July 29, 2020 - Carl Claunch wrote:
Visiting a lonely data center
Today, July 29, Robert Garner and I met at the CHM to power up the data center and check the condition of the equipment. A few boxes were initially balky but by the end of the session, everything was working fine except for the DE 1402 punch.
The leftmost tape drive on DE had dumped tape at loading the last time Robert and I visited. This time, it balked a bit at loading but after a bit of warming up it loaded. It ran perfectly during a long tape exerciser session, but gave a bit of trouble as we tried to unload it before leaving. It was not responsive to the buttons nor to the TAU reset and rew-unload, but after I commanded it to move forward for a few seconds, it successfully obeyed the unload request.
The 1403 printer on DE had a runaway carriage control tape when we began. I adjusted the tape, examined the brushes, and finally unclutched the printer to let the carriage do its thing. It spun for a couple of seconds and then successfully stopped. From that point forward we clutched it in and ran a print job to test the printer.
After Robert cleaned up the sea of hydraulic fluid on the floor under the DE 1403, he couldn't stop there. He 'detailed' the legs and bottom of the printer - it looks very presentable!
The 001 manual and three 026 electric keypunches performed perfectly. The card sorter works but still has its 'off by one' issue where it sorts cards one bucket over from the actual code in the column.
The 1402 reader on DE was initially refusing to load cards. I had some punch check and punch stop conditions that wouldn't clear, but after I did a short NPRO of the punch side and cleared the status, the reader side took off, loading cards just fine.
The CT machine had no balkiness at all. The four tape drives worked well, the 1402 reader loaded decks correctly, an 80-80 Dup proved that the 1402 punch side produced correct copies of the cards it read, and the 1403 printer did a great job printing the page for our virtual visitor, Elon Musk. The date on the Bigprint, for some reason, said July Forever, 2020
Robert and I left the machines under power for an hour, then shut it all down and closed the door for another lonely month for our forlorn vintage computing equipment. Not sure the machinery even knew who was visiting, given our masks and gloves.
Jun 10, 2020 - Carl Claunch wrote:
Today, Robert Garner and I visited the 1401 data center under special permission to make sure that the equipment still came up and worked properly. We also performed some PM work, cleaning up the lake of oil under the German 1403 and vacuuming out the filters of the power converter in the Liebert room.
German 1401 System
This system powered up fine but initially wouldn't take feed cycles when attempting to load card decks. After a few minutes stabilizing its temperature, it began working. Also, tape drive number 1 on the system spilled tape in both columns in a load failure. Reset didn't stop the tape from feeding but fortunately the Unload key stopped the activity. Again, after some temperature stabilization, this drive worked properly.
Frank's 1403 printer on this system has now failed hard. We were taking sporadic failures to see the channel 1 hole on the carriage control tape and skipping pages, but today it never saw the hole, attempting to spew pages whenever engaged.
The 1401 processor, 1402 reader and all three 729 tape drives work. The 1402 punch is still not working as we hadn't completed the restoration when our visits to the museum were terminated. The 1403 needs service.
Connecticut 1401 System
The 1401 processor, 1402 reader, 1403 printer and all four 729 tape drives worked properly. We didn't attempt a punch job.
We didn't attempt the card sorter because it has a known issue of sorting cards into the N+1 column. We are waiting for Alexei to make the adjustments to correct this issue.
The 001 manual card punch and all three 026 key punches worked properly.
Besides various power of 2 and pi print jobs, tape exercise runs and the like, we ran bigprint celebrating the visit of Anthony Fauci to the CHM on June 10, 2020. If a computer says it happened, who are we to disagree?
Robert Garner adds. ;-)
(We wore masks to help prevent the 1401 from getting its first virus :|
Here are some pictures
Spread no virus
A BIG number
Mar 11, 2020 - Carl Claunch wrote:
Connecticut 1401 System
We did not power up nor test the CT system. With no demonstrations to be conducted, there was no pressing need.
German 1401 System
The 1401 processor, 1402 reader, 1403 printer and all three 729 tape drives appeared to be working fine. Marc, Glenn, Robert and I worked on the 1402 card punch.
The symptom we faced was that a program, even one single stepped to a punch instruction, would repeat punching the pattern until cards ran out, without advancing to the next program instruction. The cards being punched were perfect.
We spent a few hours wandering through the CPU logic to determine why the punch instruction never ended. Early on we focused on a signal from the 1402 - After 9 Cam - that indicates that a punched card has moved beyond row 9.
When that was clearly working properly, we looked from the other end at the signals that end the punch instruction. We backed up to a latch circuit which was never turned off even though the After 9 Cam signal arrived.
Ultimately, we traced this to another signal arriving from the 1402 punch - CP CLD - that is driven by a chain of relays and CB cam operated switches. However, the signal never arrived into the CPU to turn off the punch operation.
We saw that some of the PL bank CBs were involved and set up to measure the timing. Soon, however, we noticed that several wires were loose that should have been plugged into the CBs. It appears they were pulled to make timing adjustments and never reinstalled.
We put all the wires back in place, hoping that our problems were solved. However, the CP CLD signal was still not getting to the CPU, but we had added a Punch Stop check of unknown origin. At this point we closed up, saving up some of the fun for others to work on next week.
The three 026 keypunches and the manual model 001 were working fine.
Alexei worked on the 083 card sorter. It had reported symptoms of cards being placed into the N+1 hopper when sorted with a hole in row N.
In prior weeks, Frank and Alexei had adjusted the brush to its specified length and position per the maintenance manual. Alexei first returned the brush to the old pre-adjustment position but the problem persisted. He re-aligned the brush to spec and moved on.
He then watched the solenoids firing and chute blade operation, which proved that the cards were going into the pocket that was being commanded by the sorter.
As a last measure, he checked the picker knife timing, since the implied row when the brush makes contact depends on when the card begins moving. The spec states that the knife should be snug at 216 degrees, but the machine is mis-adjusted to 209. This is a plausible case of the behavior we are experiencing. Next session, Alexei will adjust the picker knife timing to spec and continue diagnosing if needed.
Mar 4, 2020 - Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 drives were working properly.
The 1401 processor, 1402 reader, 1403 printer and all three 729 drives were working properly.
The 001 and three 026 keypunches were working fine. The 083 Sorter still has the issue that cards will sort into pocket N+1 when selecting a hole in row N. We are waiting for Alexei, Dale or Frank to work on it, since it is not an issue for conducting public demos.
The 1402 punch regressed, damaging cards and jamming. We disassembled to verify the settings and soon discovered that a locking screw for the adjustment of the first intermittent roller as loose, allowing the setting to vibrate loose. This allowed cards to skew.
With the mechanical settings adjusted once again, and having swapped the correct spring from the second stepped roller up front to the first stepped roller, we had relocated the incorrect spring to a less critical area. The result was that card movement through the path was again flawless, no skewing or nicking of edges.
We ran a 80-80 deck reproducing program, which produced perfectly registered cards with the same holes as were on the first data card. Unfortunately, it never read another data card, instead churning out the same data on card after card.
We hand stepped the program and determined that some signal is not reaching the processor or has gone bad inside the 1401, since the processor keeps refetching the same 4 op code and punching the card repetitively. There is no branch, it just refetches the same instruction over and over.
At this point we closed up, but we now seem to have a punch that moves cards and punches the data pattern from memory into the card flawlessly. If we can sort out this issue where the processor doesn't realize it has completed the punch op code, we should be FINALLY done with the restoration.
Restoration crew present
Mike Marineau, Marc Verdiell, Ken Shirriff, Stan Paddock, Robert Garner and I were the only team members that came in this week.
Feb 26, 2020 - Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 tape drives are working properly.
German 1401 System
The 1401 processor, 1402 reader and all three 729 tape drives are working properly.
When I arrived the 1402 reader would not turn on the reader motor. Mike and I traced it to a wire that had broken off the Reader On/Off rocker switch, probably due to our swinging the panel open every week to work on the punch side. We resoldered the wire; the reader now works properly.
The 1403 printer continues to miss channel 1 punches in the carriage control tape, thus a Carriage Restore operation may eject several pages before it stops, or it might work properly. We did not have time to investigate this problem but it does not stop the system from use, it simply wastes some paper.
We continued to work on the 1402 card punch, which had been scraping the sides of cards as well as punching off alignment and throwing up some Punch Stop and Punch Check conditions that appeared to be unwarranted.
During last week, Mike, Glenn and I had spotted a problem with the "first stepped feed roller". The adjustment specs give a rotational position where the rollers should grip a card tightly and another position where it should be completely free.
We found that one side of the roller, closer to the front of the machine, allowed the card to be tugged out, thus was not as tight as the rear side roller. There are no adjustments for this, other than rotational position. Marc, Mike and I puzzled over this for a while.
We finally spotted a discrepancy between the springs on each side. These pull the roller down so that it grips the card. The near side spring was noticeably easier to deflect than the rear spring. Further, the rear spring and all others inside the punch unit were constructed with six turns, while the deficient spring had seven turns.
Ken checked the parts catalog which verified that the front and rear springs are the same part number, thus should be identical. To test our theory, we swapped the springs front for rear. The problem then moved to the rear roller.
Mike devised a clever workaround by moving the mounting point for the spring further way and using a bit of twist tie to suspend the spring such that it gave approximately the same resistance as the correct part.
This resolved the problem and we had cards moving under power without any scraping of the sides of the card deck. We made some adjustments and the punches produced on cards are very close to ideal.
We believe we have some adjustments to make to the CBs that time impulses for the punch unit, which would fine tune the punch positions. That will take place next week.
We are still receiving Punch Stop conditions after punching each card, which appears to the 80-80 reproduce program card to be an error so the problem keeps repeating the first card. Some times we also received Punch Check conditions.
It is our suspicion that we have to go over all the CBs to get the timing correct. This too will be done next week, with the hope that we are closing in at last on the finish line to a fully working punch.
The three 026 keypunches in the center are working correctly. We believe that the 026 in the babbage closet and the two 029 keypunches are also working properly, but we didn't test them. The 001 is working fine.
The 083 card sorter has been reported to mis-sort cards by placing them in pocket N+1 when the hole exists in row N. Alexei is training on the 083 under Frank. Neither was here this week but we decided to leave the sorter until they return as this sounds like an interesting problem for Alexei to debug.
Feb 12, 2020
Comments from Frank King & Stan Paddock & Pat Buder & Michael Albaugh
from Frank King
The CT 1401 powered down today. We didn?t Have time to check the cause of the -12 power supply tripping it?s over current circuit breaker (cause of the power down). I just reset it and powered up again. We should check the load it is carrying and compare it to the German machine.
The group made some headway with the Punch Unit on the German machine. We. had trouble with the punch check light coming on Willie Nillie but after an apple restart we were able to punch a deck of cards without marking them. However, we got a validity check when we tried to load the deck we just punched. We ran out of time so we will have a little more mystery to solve next week.
The Sorter and Keypunches ran fine early in the day.
from Stan Paddock
I worked on the IBM PC to IBM 029 keypunch system todeay.
I checked out the wiring from a 24 page POWER POINT document stored on the PC.
The first column punched in a card works fine.
If the first character is an A, it will punch AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA after the first A.
If the computer tries to punch another letter, the punch will jam up with the A and other characters.
The power to the punch is controlled by a small relay.
I believe this relay has gotten stuck.
I will fix this next week and we can start punching cards again.
from Pat Buder
We had problems with the 026 farthest from the door. When feeding a card it would get stuck in the rightmost area of the punch station and then not register. We were using some of the new card stock. The cards in this particular box had a pronounced lateral concave bow. Even manually bending the cards to reduce the bow didn't help. We request that the restoration wizards take a look at this problem.
Speaking of keypunch health, please be advised that there are four upcoming spring Field Trip Days on these dates: 2/25, 3/3, 4/14, and 5/5. We request that you ensure that all three 026's in the lab and the 026 in the Babbage closet be fully operational on those days. Each student punches a card for BigPrint, and they come to the lab in large groups at a time.
from Michael Albaugh
Besides the bow, which I noticed when I duplicated a deck, I found another issue with those cards ("new 5081") from the box in the AV closet.
Wanting to be able to do some checking of the duplicated deck, I tried to print it with Van Snyder's 80-80 List program, but immediately noticed that they seem to be wider than normal. They were a snug fit between the sides of the CT 1402 reader. Snug enough that I had to gently push the deck down to get them ready to read, rather than them just falling into place. And, not unexpected, they wouldn't feed.
from Frank King re: ... four upcoming spring Field Trip Days on these dates: 2/25, 3/3, 4/14, and 5/5.
I think we should make the 029 available also. I will try to check out both the 026 in the Babbage closet and the 029 in the shop.
Feb 5, 2020 - Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 tape drives were working fine.
German 1401 System
The 1401 processor, 1402 reader, 1403 printer and all three 729 tape drives were working fine.
We continued to restore the 1402 punch, taking up after Mike and others had completed setting all 16 of the clutched cam operated switches (called PL Circuit Breakers by IBM).
The symptom at the start of the day was that when performing a Non-Process Runout of the punch (NPRO), the machine should stop with the clutch engaged so that the clutched portion of the mechanism is at its rest position, but instead it would stop at random points in the cycle.
Mike examined the schematics and deduced that it was caused by incorrect timing of one or more of three specific CBs. One of them (PL5) had been correctly adjusted and was verified to be good. The other two are on the PC bank of CBs which run continuously with the punch motor.
We checked the two and found them to be about 15 degrees out of time. Mike, Glenn, Alan and I set in to adjust the PC bank. By the end of the abbreviated work day, we had properly set CBs 1 thru 5. CBs PC3 and PC4 were the switches suspected by Mike based on the schematic.
We did verify that the NPRO now stops as it should with the clutched mechanisms at their rest position. We will continue to set these CBs and once done, move on to the third and last bank, the PA CBs.
This will stretch on more weeks but eventually we will have adjusted all of the more than 100 settings to restore this punch to full and proper operation.
The reason our session was abbreviated was a lunch at Michael's restaurant nearby, where we celebrated the life of George Ahearn with his widow Sandra and the CHM volunteers who had worked with him all these years. Kate McGregor put in a surprise visit with her 8 week old baby!
The three 026 keypunches, the manual 001 keypunch and the card sorter were working fine. The 029 keypunch in the back room was giving us trouble, refusing to allow Ken to auto-punch a deck of cards for a visitor who had written his own bootstrap program for the 1401.
Ken, Frank, Glenn and Stan worked away at the keypunch and the PC-based system to punch files to card. They oiled a dry bearing in the keypunch and made various adjustments to ensure that it was working properly. As of the end of the session, the PC based mechanism was unable to drive the keypunch correctly, something Stan and others will diagnose in coming weeks.
The other 029 is still sitting in the workroom and is presumed to still be fully functional.
We didn't put any time in on the 552 card interpreter in the workroom.
for Jan 29, 2020 - Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 tape drives were working properly.
German 1401 System
The 1401 processor, 1402 reader and all three 729 tape drives were working properly.
The 1403 printer is sporadically missing the column 1 hole on the carriage control tape and spewing out extra pages. We also found a wavy print line at the top of form when printing the PI day program output. Frank to investigate. We are also beginning to work on the replacement hydraulic unit as the printer currently leaks oil all over the floor.
The 1402 punch is still being adjusted as part of the major repair we undertook. Mike and Glenn adjusted more CBs to their target timing. Not yet working but we are getting closer.
THe 001 manual keypunch and all three 026 keypunches were working properly.
The card sorter is working properly.
The 552 interpreter in the workroom is still in need of restoration work, but nothing was attempted this week.
Jan 22, 2020 - Carl is experimenting with transportation.
This comment is from Frank King.
We fixed the problem with CT (Punch unit would not run) The would not run problem was the Feed Interlock. The hopper was left not locked down.
Maybe me, when I was checking CT for comparison with the German machine. the previous week.
Also,we noticed the cards seemed to select stacker 2 about 1/3. of the time rather than just going into stacker 1.
After attempts to adjust the chute blade magnets to no avail, I formed the chute blade upward a little. that seemed to solve the problem.
We punched a few cards with no punch checks or incorrect stacker selection.
PS. the 4 things that will prevent the punch unit from running are the DIE interlocks (2) the hopper feed interlock and the crank Interlock).
Be aware that the cover safety interlocks have been crippled on both 1402s; So remind all the new folks and us old guys to continue to be careful.
We can revisit that decision if anyone does not agree.
Jan 15, 2020, Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and 729 drives 2, 3 and 4 all worked perfectly.
At the start of the morning, 729 drive 1 was pulling the tape up out of the left column, dropping Ready status, and once dropped the tape to the bottom of the column. After Alexei cleaned the bottoms of the column, it started working fine. We saw many flakes of tape in both columns, which could be a poor quality tape.
German 1401 System
The 1401 processor, 1402 card reader and all three 729 tape drives were working perfectly.
The 1403 printer was skipping multiple pages on Carriage Restore when I first checked the system. Frank and I both adjusted the tape, checked the brushes and generally fussed over the printer. Franks' special skills gradually convinced the printer to skip more reliably. Skipped pages are very infrequent now.
We continued to hammer away at the card punch restoration. Last week Marc and team adjusted all the mechanical aspects of the punch so that it moves cards properly, at full speed. However, the electrical side of the machine is not working right.
The machine has a continual Punch Stop condition which doesn't clear with the Check Reset button. It also triggered Punch Check conditions without actually punching data on any cards.
These are triggered by combinations of relays and Circuit Breakers, IBM terminology for cam operated electrical switches. Given that every belt, roller and adjustable part of the mechanism needed to be readjusted, it was very likely that the CB timing also needs adjustment.
The punch side has three different banks of CBs - A, C and L - with up to 16 CBs in each bank. One turns continuously with the punch motor, whether or not the cards are moving. One turns with the clutched mechanism, once per card movement. The last one moves with the stop-start movement of the card under the punch component.
We checked one of the L side CBs and it was off by five degrees from the spec, which states a tolerance of 2 degrees or less. The obvious next steps of the restoration are the proper timing of all the CBs. There are two adjustments for a CB, duration and timing. We need a CB to make at one target value and break at the other target, with the two separated by the 'duration' number of degrees.
We moved the switch closer or further from the cam surface, until we saw that the switch is closed or open for the specified number of degrees. Once it is the correct duration, the cam is rotated until the make or break point is at the target degree point.
Glenn, Mike and I began at CB 16 of the L bank and worked towards the other end. This is a slow process, thus we only got through six CBs by the time of the public demo when work had to stop. It is going to take several more sessions before we have all the CBs set
The 001 manual punch and three 026 keypunches all work fine. Frank and Dale replaced the brush in the card sorter, since the old one was getting a bit scraggly.
Jan 8, 2020
Carl Claunch reported that he had "a nasty cold" so ...
Here is the Docent Report from Saturday 1/4/2020
Subject: 1401 Demo Saturday 1/4/2020
From: Jack Ghiselli
Date: Sat, Jan 04, 2020 11:30 pm
Paul Laughton and I gave our first 1401 demo of the new decade. Dave Hoyt was doing Revolution tours, and brought quite a number of visitors to the 1401. We had 48 visitors for the main demo, and after that was over, we had 38 more people stream in for most of the afternoon, so we did gallery interpretation for them.
We had visitors from Australia, Germany, India, Iran, Russia and the good old USA. Paul is always good at asking visitors where they?re from. We had an unusually large number of technical questions. When an Australian asked how fast the 1403 Printer chain travelled, I told him in inches per second, whereupon he had to remind me that they think in metric units. My bad! Have to remember to convert for our foreign visitors.
I had a post-demo discussion with one older visitor from Russia who described using Russian-built knockoffs of the IBM 029 keypunch years ago.
We ran on CT, which ran perfectly, as did the 029 Keypunches and the 083 Sorter. Thanks again to the Restoration Team?s hard work.
Thanks also to Paul Laughton, who found us another vial of cores, so our demo cart is back to 3 vials (we?d been down to 2).
Jack Ghiselli, Consultant
On Dec 19, 2019, Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 tape drives are working well.
German 1401 System
The 1401 processor, 1402 reader, 1403 printer and all three 729 tape drives are working well.
We (Frank, Alan, Alexey, Marc and Ken) continued our work repairing the 1402 punch. We made good progress today, having adjusted the first intermittent roller, second stepped roller, second intermittent roller, the punch unit and the CBs so that all are in time.
Cards seemed to feed perfectly when we manually cranked the punch. Under power, they moved through without any jams even using cards with downward bowing that had previous folded over.
However . . . we saw some markings on the sides of the cards after feeding through under power which suggests that we still have a bit of adjusting to complete before putting this back into service. We removed the punch unit and CBs to give us access again for the adjustments.
I suspect this involves a timing issue where cards are under roller pressure when they are being aligned and patted sideways. We will continue the work in a few weeks when we have the next sessions.
We also had a Punch Stop indication even when able to successfully NPRO cards through the transport. We need to diagnose this to ensure we don't have other faults remaining.
The 001 manual keypunch, card sorter and middle 026 keypunches were working well when we arrived. The 026 furthest from the door was waiting for a new woodruff key and reassembly after we had replaced the worn drive belts, necessitating quite a bit of disassembly. The 026 nearest the door was misfeeding cards from the hopper, with the card ending up skewed and jammed to the right rather than down in position to be registered.
Alexey and Frank installed a replacement woodruff key, finished assembly and placed the 026 (furthest from door) back in service. They also cleaned the rollers on the one near the door, since the failure to feed properly comes and goes and they didn't have hard evidence of the ultimate cause. This machine is back in service as well.
Restoration Team off for an extended period
Next Wednesday is Christmas day and the following Wednesday is New Years Day. Therefore, the team will not be on premises until Wednesday, January 8th.
On Dec 11, 2019, Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 tape drives were working well today.
German 1401 System
The 1401 processor, 1402 reader, 1403 printer and all three 729 tape drives were working well today.
Frank King, Michael Marineau, Alexey Toptygin, Glenn Lea, Stan Paddock, Ed Thelen and I worked on the 1402 punch. Michael figured out how to get the first stepped roller timed properly and moved us forward to dealing with the first intermittent roller timing. Progress is good.
Ed Thelen, claiming to be Dilbert's pointy haired manager, tried to stupervise ;-)
The 001 manual keypunch, the card sorter and the two 026 keypunches nearest the door were working well.
The furthest 026 from the door had been out of service due to a worn motor belt. Replacing it involves disassembling quite a bit of the center mechanism of the punch.
We were stymied for an hour by a woodruff key in a shaft that would not budge. It was only after we applied strong enough persuasion with screwdriver blade an hammer that we extracted it.
The belts are replaced but the keypunch is still out of service while we purchase a replacement woodruff key. We expect to put the keypunch back in service next Wednesday.
Stan was handing out Ding Dongs and appeared to be in a good holiday mood.
Ed Thelen here, something about "bells were heard ringing through Munchkin land".
On Dec 6, 2019, Carl Claunch wrote:
Connecticut 1401 System
The 1401 processor, 1402 reader, 1403 printer and two of the three 729 tape drives are working well.
The middle tape drive continues to have errors until the restoration team is nearing whatever is causing the problems, after which the drive spontaneously behaves for the rest of the day. One of the head signals is being distorted, but what causes this is the mystery.
Marc, Mike, and Alexei wrestled with the drive, swapping cards and moving probes until the errors stopped. We hope to sneak up on the problem next week.
Frank, Glenn, Alan and I battled the 1402 punch for another session, inching along towards re-timing everything. To date, we have set up the main punch clutch with the timer wheel indicating the proper trip point of 315. We verified the timing of a new card from the hopper reaching the first feed rollers at 185.
The next parts of the mechanism to set in time are the stepper rollers, aligner knives and side patter arm. A card is rolled into the punch area by the feed roller, under the stepped rollers. Those rollers move the card forward to a predefined spot then loosen their hold. This is due to an eccentric shape to the rollers.
The card that is now loose is grabbed by the two aligner knives and pushed forward and the side arm pats the card to push it towards the rear edge of the mechanism. As the card reaches its ultimate position, the eccentric stepper rollers grab hold and keep it from moving.
As we left the punch unit on Wednesday, the card was not moved far enough forward for the aligner knives to grab the rear. There are no alignment marks on the rollers nor much good advice for how to sync it with the main clutch. We hope to figure this out next week.
Note that even with the stepper rollers, aligner knives and side patter put in time, there will be more adjustments to come. We have the Geneva drive in time with the main clutch already. That drive converts the continuous rotary motion of the punch drive belt into stop-start movements that advance the card to the 12 row positions under the punch dies.
German 1401 System
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 tape drives are working well. Marc and Michael spent a bit of time tweaking the fourth drive.
The first 026 keypunch was sporadically failing to eject cards from the read station to the stacker - some would stop part way and not move up into the stacker properly. Alexei , Glenn and Frank fought a valiant battle, adjusting things here and tweaking things there. Alas, the keypunch is out of service until next week.
The 001 manual keypunch, the card sorter and the other 026 keypunches are in service.
On Nov 27, 2019, Carl Claunch wrote:
German 1401 System
The 1401 processor, 1402 reader, 1403 printer and all three 729 tape drives are working well. The repeated retries we were experiencing on drive 2 disappeared late in the day as Mike was finalizing adjustments. They may reappear but for now it appears the drive works fine.
The 1402 punch repair, week 99, was the main work today for Frank, Ken, Alexey and me. We reinstalled the Geneva drive, the last success of the day for our team. The Geneva drive converts the continuous rotary motion of the other punch parts into a start-stop stepping motion that moves the card forward to each of the 12 row positions.
With that in place, we began the long slow process of synchronizing all the moving parts. A clutch connects the continuous motor drive to the rest of the mechanism, taking one rotation for each card to be fed and punched. There are quite a few belts placed over many ribbed pulleys, thus each section must be synced to the main rotation.
The first two settings are the clutch engagement point and the point where a card feeding in from the hopper first touches the 'first feed shaft'. An index wheel, sometimes called a timer wheel, marks the rotation in degrees.
The index wheel mark where the card touches the first feed shaft, as well as the mark where the clutch engaged, were wrong. We then read the maintenance procedure for the setting. It told us to loosen the clamp on the index feed pulley.
No diagram anywhere in the manuals showed the index feed pulley. No pulley anywhere near this point had a clamp on it. We spent more than half an hour hunting for something that matched the instructions.
Finally, we chose to loosen the two set screws in a pulley on a shaft, since that would let us rotate the index wheel relative to the other mechanisms. Now that we had the method, we need to set the wheel to the clutch engagement and first feed shaft points, 315 and 185 degrees respectively.
Immediately we ran into complications. The clutch has four positions around the shaft where it can engage, but only one is the correct setting. The IBM manual warns that we must use the 'synchronizing tooth' as the reference or we could be 90, 180 or 270 degrees off the proper point.
They describe the synchronizing tooth in terms of which driving face is across from the centerline that passes through a collet to a hole on the other side. This collet has three holes, but four driving teeth.
Ken and I each read the description differently. IBM chose not to illustrate this point in any manual, but fortunately we had a working punch on the CT system. That disambiguated the description.
Further, while IBM stressed that the synchronizing tooth was the reference but then gave no explanation of how to use it. Is that tooth up, down, left, right? We had to reference the CT system to learn which way that tooth has to point.
This final discovery came right at 2PM when Frank and others had to leave. We closed up and stored all the tools and parts it the back room, to be used next week when we resume the restoration.
Connecticut 1401 system
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and all four 729 tape drives are working well. Mike, Marc and Ken completed the reassembly and checkout of the fourth drive, the one where they had to reload one clutch with magnetic powder to restore sufficient strength for the take-up reel operation.
The 001 manual punch, card sorter and three 026 keypunches are working. Alexey observed some misfeeding of cards from the hopper on the 026 near the door, but the problem was intermittent. Next week we will clean up the two feed rollers which are probably the cause of the skewing of cards during the feed.
Long awaited return of Stan to the team - thwarted
We were all waiting for Stan to rejoin us now that conditions are right for his participation. Frank and I both tried calling Stan but to no avail. This evening he called with a legitimate excuse - he is in Colorado for the holidays visiting family. Maybe next week.
Ed comments - Yes, Stan has a daughter and grandchild in Colorado :-)
On Nov 20, 2019, Carl Claunch wrote:
German 1401 System
The 1401 processor, 1402 reader, 1403 printer and tape drives 1 and 3 are working perfectly.
We suspect that we have a bad tape on drive 2. Using the tape exercise program we see the tape doing large numbers of retries for every record successfully written/read, for the first inch to 1 1/2 inch of tape from the supply reel, then settles down to work flawlessly for the remainder of the reel. This is repeatable week after week as well as in multiple runs per day. We expect to verify this by using a different tape on the drive.
Frank, Alan, Glenn, Marc and Rick worked on the 1402 card punch all day. After straightening the plate which we observed had been bent, we tried various washer heights as shims but never got the alignment knives at the proper height so that a card would pass over but not jamb against the base of the knife.
At this point we noticed that the rectangular block onto which many parts are attached was cocked to make the front where the knives sit open too much and the rear where cards would exit into the punch dies was closed blocking any card movement.
This rectangular solid is attached with two screws going into each of the two ends, through the sideplates that hold the entire 1402 punch area together. It could be rotated along the long axis as well as shifted forward, back, up or down since the screw holes were wider than the screw bodies.
The maintenance manual has no adjustments listed for this bar, nor are there any markings to use as a gauge. Frank bent cards over to use the doubled thickness to check fit and clearance, while I loosened the screws and twisted the rectangular solid bar to various positions.
We found settings that gave us uniform back and front clearance, with no shims at all under the front plate. Single cards moved easily through the mechanism, never bumping into the front of the aligner knives. The knives properly moved the card forward to its proper position to punch the first row of the card.
All now appears good, so the plan for next week is to put the punch head, punch die, Geneva drive and all the belts back together. Once we get all the pulleys and gears in time with each other, we can do some test punching.
Connecticut 1401 system
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and the first three 729 tape drives are working perfectly.
Michael and Marc worked on the fourth tape drive, whose take-up reel mechanism had inadequate clutch strength due to lost magnetic powder. Michael found some reference material to pick the right powder consistency and amount to refill the clutch, then assembled it all. Marc and he placed the parts back on the drive, including replacement of a broken clutch brush.
Initial testing did not go well. The brake wasn't strong enough on that reel and the drive wouldn't load or unload tape. At this point the demo was about to start so we defer debugging until next week.
The 001 and all three 026 keypunches work well, as does the card sorter.
Ignacio Menendez commented as follows
Carl, Mike, Marc, as a bystander, let me offer you some things to try.....
Once you determine that in fact you are seeing the failure of either of the two reel clutches, we now have to determine if the failure is due to an electrical or mechanical cause....
- for the suspect tape with lots of errors up front, remove the amount of tape where you see the errors happen, cut that piece, add a new load point sticker. (Tape stripping)
Ed comments: Yes, computer operators did that to "work tapes", somewhat monthly - cut off about 30 feet of the beginning of the tape, then put on a new "beginning of tape" foil. It was part of the folk lore of keeping a site running smoothly. For many/most jobs, only the first part of a work tape was used, many times a day. Over a period of years, the work tapes got inconveniently short, and were used for other things.
- for the drive with the clutch problem...
- determine for sure at what phase of the load/unload this is happening...
- There are two phases, stop clutch control (front clutches worm gear driven), and vacuum control, which over when tape in column is detected on BOTH columns.... there is when the fwd and rev clutches move the shaft, under control of the two other vacuum switches in each column.
I understand that you went through this already, hence the replacement of the powder in the clutch...
At this point, perhaps we might question if it was the powder, the clutch itself somehow, or maybe, yikes, it is an electrical drive problem, that we had discarded.
It is horrible to think, but perhaps you may try replacing that clutch that you changed the powder on, with another one from stock (I seem to remember that Glen showed us more spare clutches)
You might consider also that the woodruff key may not be providing a positive solid grab of the clutch to the shaft.
If replacing that clutch does not fix the problem, then perhaps it is time to revisit the electrical signal driving the clutch, again.
I hope that this helps, very elementary, and perhaps you already have thought and done it, just my two cents worth.
On Nov 13, 2019 10:18 pm, Carl Claunch wrote:
Today we focused almost exclusively on the restoration of the 1402 Punch on the German machine. Glenn, Frank, Alan, Ken and I worked to understand why cards were jamming under the left aligner knife instead of riding up over the top of the knife.
Alan suggested that the bottom plate upon which the cards slide, might be bent. This was after many false steps investigating possible adjustments and time spent seeking enlightenment from the maintenance manual and the parts catalog drawings.
We found that it was indeed bent, with the left and right ends that sit over the aligner knifes bent downward and the entire plate was bowed. The knives were at the correct height but the plate was too low due in part to the bend.
We removed most of the bend but found that the plate itself was too low even when flat across. This plate had recesses and screw openings that appeared to have been amateurishly executed; not at all like the other 1402 nor was it typical of IBM workmanship.
We put two washers between the plate and its mounting bracket, raising it by about 15 mm. This was much closer to correct, but the left side was still a bit low. We removed the plate as our last act of the day and will bend it further and/or use fatter washers next week.
Michael Marineau was the victim of a car parked across his driveway in Berkeley, aggravated by a police response to his complaint that took all morning and a tow truck that still hadn't arrived by noon. Thus he was not able to attend today to work on the clutch for the take-up reel of the number 4 tape drive on the CT machine.
All other units of both the DE and CT machines were working properly. The 001 manual punch, the card sorter and two of the 026 keypunches also worked properly. The 026 nearest the door had cards jamming as they were feeding down from the hopper, but with cards manually inserted it was otherwise fully operational.
On Nov 6, 2019 4:55 pm Carl Claunch wrote:
1401 processor, 1402 card reader and 1403 printer working well, although the printer is still leaking hydraulic fluid.
Frank King, Alan Futterman, Glenn Lea and I worked on the 1402 punch, beginning reassembly. It was a long and often frustrating effort because we hadn't tried all N factorial ways to assemble the N parts so it is still not together properly.
We went through many iterations of assembling things then discovering the next part had to fit below all the rest. Couple that with several trips to the parts room to look for screws and the day gets long quickly. We think we now know the proper order of assembly and will do that once we take apart everything next Wednesday.
Michael Marineau and Iggy (on a special visit from Ajijic) put the new bearing into the 729 drive vacuum pump and finished reassembly. They put it back into drive 3. We now have all three 729 drives on the system working perfectly.
The 1401 processor, 1402 reader, 1402 punch and 1403 printer are all working properly.
The first three 729 tape drives are working fine. Michael and Iggy removed the take-up reel hub to repair the bad clutch. This required a trip to a hardware store to buy special long bolts used to slight a subassembly back for service.
The two were joined by Frank and Glenn who located magnetic powder and helped them install a good supply in the failed clutch. We found some very fine powder in the clutch but the properly working one had coarser powder. They mixed up our stock to get a good consistence and packed it into the clutch.
A comment by Michael Marineau
"One correction on the 729 clutch, we were just guessing on which powder and how much and it was either too course and/or too much because now it doesn't spin freely. So we will need to retry filling it next week."
We should reinstall the clutch into the number 4 drive next week and hopefully return it to full service shortly thereafter.
The 001 manual keypunch, 083 card sorter, and all three 026 keypunches are working properly. The 029 keypunch in the workroom is ready to be handed over to the Education department.
The 552 Interpreter in the workroom still needs work to get it to print the correct characters. We had no time to work on it this week.
On Thu, Oct 31, 2019 8:19 am Carl Claunch wrote:
Frank, Alan, and Glenn continued the rebuild of the punch side of the 1402.
The third 729 drive is inoperative due to a defective vacuum pump. Michael and Marc disassembled the pump in the workroom to determine the problem. This pump is an odd looking stack of alternating driven fans and free rotating fans that feed the air to the next of the seven stages.
After study, thought and assistance from a remote support expert, they determined the problem is loose nuts that allow the driven fans to slip, thus not producing adequate flow. It is expected that it can be reassembled with a properly tightened nut, although the vacuum pump expert, Iggy, is flying in to help do this next week.
The 1401 processor, 1402 reader side, 1403 printer and other two 729 drives are working properly.
CT 1401 system
The fourth 729 drive is still inoperative. We didn't work on it this week.
The 1401 processor, 1402 reader side, 1402 punch side, 1403 printer and first three 729 tape drives are working properly.
The middle keypunch was repaired. A solenoid coil on the right side of the machinery is used to block cards feeding down when using the Reg key, when Autofeed is off during a Release, or when attempting a feed while two cards are in place at the punch station.
The coil side had been scraped each time the cover of the keypunch was removed and replace, to the point that the coil windings were broken. Glenn found a replacement coil and he, along with Alexey, installed the coil and adjusted the keypunch. It now works properly.
We believe the first and third keypunches are working properly, although we have had a report of DUP operations on the 026 furthest from the door occasionally adding a 12 punch. We are not sure how that happened and are not able to reproduce the problem.
The 001 manual keypunch and 083 card sorter are fully operational. The 029 keypunch in the Liebert room is fully operational and the 029 in the workshop works perfectly except that program drum operation is imperfect.
Team members working on equipment this week
Frank King, Alan Parsons, Glenn Lea, Alexey Toptygin, Michael Marineau, Ken Shirriff, Marc Verdiell, Robert Garner and Carl Claunch
On Thur, Oct 24, 2019 at 9:27 AM Carl Claunch
- Michael Albaugh's comments are in italics
The 1401 processor, 1403 printer, 1402 reader, and tape drives 1 and 2 are working fine.
We are continuing, under Frank King's leadership with assistance from Alan, Marc, Glen and others, to rebuild the punch side of the 1402 system. This may take a few more weeks as the reassembly and then readjustment are complicated.
Tape drive 3 has been slow to respond to Load buttons and other actions, which appear to be due to problems with the vacuum pump. This is the drive that smoked and smelled a number of weeks ago which we attributed to a stuck pump motor.
Michael Marineau removed the pump and is disassembling it to sort out the problems. We can hear parts rotating inside when the shaft is stopped and they make a grating sound, plus metal filings are being produced. In the process of pulling one bearint to permit disassembly, the puller tool snapped. Clearly we need a bigger puller. we must wait until next week to continue this, so the drive is unavailable at this time.
The 1401 processor, 1402 reader, 1402 punch, 1403 printer and tape drives 1, 2 and 3 are all working fine.
It took a little encouragement to get Fortran to boot off tape. First try just kept moving tape when booted, even though the normal first block is pretty short. We had to resort to using the TAU debug switches to stop it (Reseting the 1401, or hitting RESET on the 729 didn't seem to work). After that, we did several tries and it seemed to work, but my memory is not exactly clear.Tape drive 4 is still inoperable. We did not get to it on Wednesday.
The keypunch near the door is assembled but still sporadically misfeeds. What is happening is the failure of a card to release full from the punch (first) station. It doesn't slide far enough to be under the pinch roller that will register the card at the dup (second) station.
This leaves the card partially under the punch block where the newly registered overlaps it. That causes the original card to remain in place as additional cards are fed in and registered atop it.
We are trying to sort out the root cause of this. It could be issues with the new card stock, but more diagnosis is needed. This keypunch is usable but the operator needs to watch and stop feeding if any card fails to release all the way to under the dup station pinch roller.
A critical spring had been lost a week ago by me. We searched in vain but this week Alexey took up the search once again and finally found the errant part. It is now installed in the machine.
The middle keypunch was working early in the day but suddenly refused to register cards. Frank, Alexey and Glen traced this to the failure of a solenoid coil that permits cards to feed. Glen found a replacement solenoid but we ran out of time before the 3PM demo so the machine is currently inoperative pending the replacement next week.
The keypunch furthest from the door seems to be adding 12-punches when DUPing. I first noticed due to a validity check (and STORAGE error on 1401 console), but failed to notice the actual mis-punch. Thinking it was an alignment error, I DUPed the source card again, and again got Validity (and STORAGE) Ken noticed the added 12-zones, so I put back the original in the deck (I had duped it because it was the wrong color and I will probably be sending this deck off to the guy who set us on that poetry quest). All went well except for an error in the compile. You guessed it, an added 12-punch that changed a '2' to a 'B'.Card Sorter and 001 keypunch
These are both working well.
Tape drive tuneups
Michael Marineau has been carefully adjusting the tape drives to the proper signal levels and fault-free operation. He has completed this on all drives except for CT drive 4 due to its failed status. This \has much such a difference that we can run Fortran compiles from the tape with virtually every try working perfectly.
In the past we would have multiple process checks and other errors running with the same tape,Note that I use the Low Density tape (200 CPI) these days.often taking many tries before one small program could be compiled.I do not recall whether it was on CT or DE that I noticed that flakey compiles could often be "fixed" by using the PARAM card to tell the compiler it was on a 12K machine, rather than 16K. Of course, if you really need 16K, this is not an option. This was discussed way back when we were originally messing with Fortran.Michael Albaugh used the compiler to run the Computer Poetry program, generated pages of rhymes.Nit-pick. The original author does not call it a Poetry program, but a "Line Generator". The lines don't (necessarily) rhyme. :-)
(And still seeing the funny form-feeds: Page (56 lines), Skip to Channel 1, one line, skip to channel 1, normal-ish page (does not have enough lines to ask for another form-feed.
Note: in all this that I think the performance of these machines is great, and I'm very grateful to have access to them.
On Wed, Oct 9, 2019 at 4:40 PM Carl Claunch
Today we made progress on several fronts as well as encountering a few setbacks. Ken, Marc, Michael, Robert and Bhushan joined me in addressing the remaining problem areas.
German 1401 system:
The 1402 punch is still out of commission, waiting on Frank and others to finish the timing and setup adjustments. The 1402 reader is working well, as is the 1401 processor, the 1403 printer and the tape drives.
Michael is doing a more complete and precise skew adjustment on drive 2, using the Master Skew tape. Currently the check bit track is 3 degrees behind the bit 1 track with no delays, which means that we should rotate the head slightly to further minimize that skew. Then he will use the delay circuits for all tracks until all signals are coincident.
Iggy is likely to comment on this.
Connecticut 1401 system:
The 1402 punch is still working well, as are the first three tape drives, the 1401 processor and the 1403 printer.
The last tape drive remains offline since we need to rebuild the clutch on the takeup reel. Because it was not working, it had been powered down, but that removed power from the terminator that sits on the drive at the end of the tape cable chain.
In addition, we found that the tape emulator box was applying pull down voltages but not pullup voltages to the lines when it was plugged in as the terminator. That produced tape signals which swung down from zero but never to positive ranges. A power cycle of the tape emulator box restored the pullup.
We decided to remove the tape emulator from the chain unless we are actively using the box. Further, we chose to uncable the fourth drive, power it down, and put the terminator for the chain on drive three.
Bhushan suggested that we clean and lubricate the clutch on the 1402 reader. It never gave us problems today, like a watched pot refusing to boil, but Bhushan recommended that step based on his experiences in the field.
The clutch was filthy - jammed with old grease and dirt. He cleaned it on the workbench, applied new lubrication and reinstalled the clutch, pulleys, belts etc into the reader. We tested it and all works well now, with less chance of erratic problems in the future. The clutch is fitted with a grease coupling, thus normal practice would push new grease in atop the old grease and dirt.
Michael and Marc began to adjust the tape drive signal levels, quickly finding the terminator issues above before continuing to set good levels. They worked on one of the drives, achieving great signal levels but that leads to skew errors. Next week they will use the Master Skew tape to adjust the drive, then repeat the level setting and skew fixes for the other two tapes.
Bhushan also spotted the same triple card feed problem on the keypunch nearest the door. He did some detailed diagnosis and found two independent problems, both of which are leading to the issue.
Sporadically, the keypunch clutch will not release and cause the machine to take a spurious feed cycle. Inside the clutch, a nonconductive separator sits between the clutch coil and the armature, but with time these wear down. If the armature begins to physically touch the metal core of the coil, residual magnetism can hold the armature closed even though power is removed from the circuit.
The check for that is to stick a sliver of punched card; with the sliver, we had no more double clutch cycles. This means we need to open up the clutch and check the insulating part or adjust the gap.
The second problem is that sporadically cards will be released from the punch station but won't move all the way out from under the punch dies. We need to open the punch cover, clean it and examine the drive wheel for possible flat spots or misadjustment.
We worked on the remaining problem with the 029 keypunch in the workroom, erratic operation with the program drum.
Bhushan rotated the drum so that the star wheels are correctly placed down in the holes at each card column. The keypunch became much better, duplicating or skipping long stretches of columns, but still won't work for a full card.
We have some remaining connectivity issue to address, thus the keypunch is not yet ready to be handed over to the education department.
The other two 026 keypunches in the datacenter and the 001 manual punch, as well as the card sorter, are working properly.
Sun, Oct 06, 2019 8:02 pm Carl wrote
This last week we had a visit from Bhushan Mohan who is an experienced FE supporting a wide range of IBM systems in India. He joined us in attacking a number of problems we were facing with the systems.
The sporadic problems of the keypunch nearest the door, where it would sometimes feed three or more cards, had been hard to recreate and troubleshoot. We spent time checking and adjusting the CBs and other electrical components, but once the machine began to fail, Bhushan quickly zeroed in on the real problem, inadequate torque from the motor.
It was failing to move the card that had just finished being punched, thus it was not making it under the pinch roller to register in the dup station. With the card still under the punch and new cards arriving from a feed, we were experiencing jams of multiple cards. Once it was adjusted we should not have the problems any more.
The tape drives on the DE machine are all in service now. The burning smell came from the vacuum pump which was stalled temporarily. Marc opened it up and cleaned out a lot of dust, after which it rotated smoothly and ran without problems.
The card punch on DE is still being adjusted - Bhushan and Frank spent quite a bit of time working to get it properly timed. What has been causing problems is the transition from smooth continuous card motion to the start/stop movement between the 12 card row positions during punching.
The rollers must firmly move the card to the proper point, then pivot out of the way as the Geneva drive provides the stop and start movements. If they pivot too early, we get card skewing causing mispunching, while a late pivot leads to bent card edges.
The card punch on the CT system is working fine. The card reader on CT did not exhibit the bad behavior where it fails to feed on a Load or Start, although Bhushan was eager to trace the problem but it perversely refused to fail all day.
We worked on the star wheels (program drum) on the 029 keypunch in the workroom, but even with cleaned contacts we are still experiencing erratic contact operation. Will continue to work on this next week.
The far right tape drive on CT was failing to take up tape fast enough as the drive did reads or forward moves. The other three drives are working fine. We removed, tested and cleaned the vacuum switches in the columns. We used the scope to verify the current flowing to the clutch magnets, comparing the bad drive to one that works well.
We found that all three clutches on the takeup reel (stop, up and down) were receiving the proper current, somewhere in the range of 270 to 300 ma, but the up clutch wasn't providing a firm hold.
Sadly, this confirms that the problem is inside the clutch. These clutches are filled with a supply of iron powder and have a common failure where powder is slowly lost until there is insufficient supply in the hub to work properly. If it occurs quickly there is a gray powder visible on inspection, but this seems to have been a long and gradual loss.
IBM engineering, in their wisdom, did not put the refill port for the clutch at the top, therefore we have to disassemble and remove the entire shaft and three clutches in order to refill the errant one. This will take a few weeks, during which we will only have three working tape drives on each system.
Long term we want to rebuild a hydraulic unit for a 1403 printer, then when it is in good shape, swap it into the 1403 printer to stop the steady leaks. Some of the work can occur while the printer continues in operation, but once it is time to swap it in and adjust it, the printer will be out of operation for perhaps a week.
On Sep 19, 2019, Carl Claunch wrote:
CT SYSTEMDE SYSTEM
The 1401 processor is working properly
All four tape drives are working properly
The 1403 is working properly.
The 1402 punch side is now working properly.
It would not NPRO, raising a punch stop immediately without spinning the motor. We guessed that it believed there was a jam that couldn't be properly ejected with a feed, although there is no indicator light to show this situation. We manually cranked the punch, ejecting two cards which we believed were too close to each other. The punch was tested both with NPRO and by punching output.
The 1402 reader side continues to have intermittent problems but is in service.
The problem manifests itself by a failure to take feed cycles - the load button is pressed, the motor spins but no cards are fed. This also occurs with a read op code in the processor and Start pressed on the reader.
This problem occurs sporadically, lasts for a bit of time and then maddeningly goes away as we attempt to trace the issue to its root cause. By the end of yesterday, it appeared that the problem was a fault in the Interlock Stop latch which should be reset by the Load key or Start key but remains on. The fault disappeared before Ken and I could determine why the latch is erroneously on.
Often a power cycle of the 1401 system will make the problem go away.
The 1401 processor is working properly.
The first and third tape drive are working properly.
The middle tape drive requires adjustment of skew delays between tracks to properly read and write tapes and is currently out of service.
Michael has been adjusting all the tape preamp and amp levels to the target of approximately 8V, up from seriously low levels on all three drives. Once we restored the signal levels, we encountered skew errors.
These mean that the time from when the first 1 bit arrives from some track and the arrival of the last 1 bit from any other channel has exceeded some maximum acceptable delay. This requires us to adjust all tracks to cause their bits to arrive as close to simultaneously as we can.
Adjustment is done by selecting one of a number of taps on a delay card - choosing among various predefined delays. All seven tracks have a delay card. We would watch the bit arrivals from the tape head for groups of tracks. to identify which were the furthest apart and begin changing those targeted tracks to bring the bit arrivals closer together.
Marc discovered some "Skew Test Master" tapes in the workroom which should give us known good patterns for this work. We would have tracks that were written as close to simultaneously as possible, with all tracks having a 1 bit recorded.
The 1403 printer is working properly.
The 1402 reader side is working properly.
The 1402 punch side is still being adjusted and is currently out of service.
Frank, Dale, Alan and others are going through the 1402 maintenance manual step by step, adjusting the timings of the punch. The initial punch checks we were receiving were a result of mistiming in the mechanism that allowed cards to slip and skew.
The punch moves cards two ways - continuously and with stop-start motion - which is what makes it mechanically complicated. The normal smooth motion takes place as cards are fed into position to punch the first (row 12) holes, and again when the finished card is moved though the read station and then into the stackers. Stop-start motion occurs to freeze the card while a row of holes is punched, the step the card forward to the position for the next row.
The stop-start mechanism is built around a Geneva drive. The pinch rollers that cause smooth card motion are moved out of contact by cams to cause smooth motion to cease, then dropped back in place to restart smooth motion.
This must be well synchronized since we want cards to roll up to the point where we begin punching holes for row 12, the stop-start mechanism to hold the card still and the cams to pull the pinch rollers out of the way stopping the smooth motion.
If the smooth motion continues when the stop-start is holding the card still, we fold up the card into a jam. If the smooth motion stops at the wrong time, the card may be mispositioned for punching row 12, causing a miscompare when the card is read after the punching is done.
The adjustment consists of lots of hand cranking of the mechanism, then adjustment of various screw settings, clamps and belt positions. The team is partway through the adjustment sequence. Once complete and after verifying punch operation, we can return this unit to service.
The sorter is operating correctly.
MANUAL KEYPUNCH (001)
The two keypunches further from the entry door are working correctly.
The keypunch near the door is out of service while the keyboard is repaired.
This keypunch would lock up from time to time, with the problem becoming more frequent. We opened the keyboard to determine the root cause and found a worn part for the Space bar mechanism.
The Space bar pushes down, rotating a lever to pull a long bar forwards towards the keypunch operator. This bar will trip a vertical bar in the rear of the keyboard which falls down. That swivels a number of horizontal bars in the permutation unit attached to microswitches. For some of the keys is also activates an individual microswitch under the vertical bar.
The wiring of the microswitches in the permutation unit will convert a single pressed key into activation of some number of solenoids in the punch unit. The 'A' key will be converted to a 12 and a 1 punch, for example. Dual keys (e.g. U key is also the number 1. When in Alpha mode, this fires the 0 and 4 solenoids but in Numeric mode it fires the 1 solenoid instead.
The long bar moving forward pushes apart small ball bearings in a horizontal chamber. These are packed together tightly enough that once a single key's bar as entered, there is no room left to let any other keys move. This ensures that one and only one key is pressed at a time.
The forward movement of the long horizontal bar and the resulting downward movement of the vertical bar locks the key in place. Once the keypunch has responded to that keystroke by punching holes in a card, a solenoid under the keyboard is activated and it moves the restoring bail, a bar that pushes all the vertical bars up, all the long horizontal bars forward and the keycap back into its full up position.
The interaction of the long horizontal bar and the vertical bar depends on a notch in the horizontal bar. This holds the key down until the restoring bail is activated. On the bar for the Space key, the notch is worn into a sloped ramp not a sharp edge.
The result is that pushing Space will trip the vertical bar but latching the horizontal bar in place is temporary. Because of the slop, it easily slips off and unlatches. The restoring bail will fail to push the vertical bar back up if the horizontal bar isn't latched, thus the keyboard remains stuck.
We have a spare 026 in the workroom whose keyboard we removed to supply replacement parts. The long horizontal bars engage with the vertical keystems under the keycap. The keystem is inserted into the top, fits around a coil spring and then emerges through the bottom plate. Nylon fishing wire is laced through all the keystem bottoms to keep them held in the keyboard.
We had to remove the fishing wire, pull all the keys out, extracting the coil springs then opened up the upper and lower plates to get to the long horizontal bars. It was easy at this point to replace the worn space bar with the good one from the donor keyboard.
At this point Alexei has put the plates together but has to tediously inserted each keystem, holding the spring in place with tweezers, then thread the nylon line through its bottom. He has done part of a row but will continue on the weekend trying to finish rebuilding the keyboard.
You might ask why we didn't just swap the entire spare keyboard for the worn one? The wires to all the microswitches on the keyboard are soldered in place. This bundle runs down and is laced together with many other wires from the rest of the keypunch, ultimately being soldered to the strips inside the base where the tube logic resides. We judged the unwiring and rewiring process to be more difficult and time consuming than all the work detailed above.
029 KEYPUNCH IN LIEBERT ROOM
The manual keypunch at the railing in the computer room is working correctly.
NEW 029 KEYPUNCH IN WORKROOM
The 029 is working properly
552 INTERPRETER IN WORKROOM
This 029 has multiple errors and needs restoration.
It is likely that the problems stem from oxide on the CB (cam actuated switch) and relay contacts, but we need to diagnose the conditions to be sure. This is a low priority task that will only be worked on when everything else is running well or a team member is particularly bored.
The interpreter is in the midst of restoration, not yet ready for service
This interpreter will take in punched cards, read the characters in the columns and print 60 of them on either of two lines across the top of the card. We intend this device to be used to print cards that were punched on the 001 keypunch by visitors. It uses a plugboard which we will wire to print the first 60 columns of a card.
At this point the basic mechanism is working and it reads cards, but the printing is a bit off. We suspect this is a matter of adjustment, since many of the characters are printed properly.
The device works with sixty long vertical typebars. These are pushed up to their top position at the beginning of a card cycle and then allowed to drop, being stopped by solenoid activated levers that hold a bar from moving further downward.
To add to the complication, the typebars have four vertical zones. These are for columns with a 12 row punched, an 11 row punched, a 0 row punched, or none of the above. Thus the characters A-I are in the topmost zone, J through R are in the second zone, /, number 0 and S through Z are in the third zone, and the digits 1 to 9 are in the fourth zone. Oddly, 0 is not part of the numeric (fourth) zone.
Once the bar is held in the 12, 11, 0 or numeric zone, further timing lets it drop so the solenoid can now select which of the nine values in that zone are printed. The proper zone is always selected but often the value within that zone is off by one. That is, a card column having A will select the 12 zone but the printed character is B not A.
We don't have an adjustment manual for the device which complicates the task of figuring out how it should work and why it is misbehaving the way it has. This is a low priority restoration meant for spare time or bored team members.
Wed, Apr 17, 2019 6:22 pm Carl wrote
Today Michael Marineau, Frank King, Ken Shirriff and Carl Claunch were joined by a new team member, Alan. Alan is a hardware engineer who worked for Fairchild and a plug compatible add-on memory vendor who sold memory upgrades on S/360 and S/370 machines.
Frank and Alan worked on one of the keypunches which had been reported to have triple fed cards from time to time. They were not able to reproduce the problem after extensive checking, but made adjustments to the 026 just in case it was marginal.
Ken and Michael continued their work hunting down the problems with the third tape drive on the CT machine. It sporadically drops ready. One challenge with finding the problems stems from the frustrating behavior of the drive - generally the problem disappears in the early afternoon and won't come back until the next day.
We continued to trace the failure backward using the new four channel scope donated by Marc. We found that a signal "tape in left column" was dropping out producing the loss of ready state.
As we put the scope on the air switch on the left tape column, the drive chose to stop its failures for the day. We did incur one failure soon after wiring to the switch, but it then stubbornly refused to fail any further.
The switch is vacuum operated and will connect the signal line to ground when there is a vacuum in the column, i.e. the tape is loaded in that column. If not actuated, the line is pulled to its -6V level by the MH card that first processes its signal.
We verified that the line is -6V when no vacuum exists and it jumps up towards ground when activated. However, we noticed that it wavered and dipped down to about -0.5 V, unlike the other similar switches which stay nailed at 0V.
The logic level going into the MH card is clipped SDTDTL whose voltage specs require that a high level be no lower than -0.5V to be valid. A low level has to be no more than -5.8V. Any value between -0.5 and -5.8V is the exclusion zone, the indeterminate level where the behavior of logic gates is not guaranteed.
Even though we couldn't catch the drive failing to prove that the sagging level is causing the problem, we know that this is not a good behavior. The vacuum switch has two sets of paralleled contacts, thus the only way to have the signal pull away from ground is if the resistance through BOTH contacts is high enough to yield the observed voltage drop.
Michael removed the vacuum switch, brought it to the workroom and carefully cleaned and tested both sets of contacts. Once they were operating with almost zero resistance, he reinstalled the switch in drive three.
While we can't definitively test this until the next work session when the drive will again be amenable to showing the failure, we did observe the switch to have no waver, no sag of voltage levels and reliably sit at ground level.
The conclusion is that, subject to verification next week, we have found and resolved the flaw causing the tape drive to drop ready.
Frank King brought in a repaired wood card joggling tray which is attached to the top of the 552 Interpreter cover. We believe we have successfully re-inked the ribbon for this device; Frank is giving it a final treatment to even out the ink level across the span of the ribbon, which is a bit wider than a punched card and many feet long.
Robert Garner did some helpful reorganization of the workroom, which he will continue in the following weeks until the workroom is at peak usability.
Wed, Mar 27, 2019 7:23 pm pm Carl wrote
Today we had a good turnout of volunteers. Frank King, Dale Jelsema, Bill Flora, Alexey Toptygin, Michael Marineau, Ken Shirriff and me. Rumor has it that Robert Garner's dog didn't have a suitable raincoat, so we didn't have his company today.
We ruined two red docent shirts inking the ribbon for the 552 Interpreter. Frank brought some ink refills for stamp pads, we diluted it in Isopropyl Alcohol and saturated the ribbon with it. The ribbon is hanging over some ethernet cable strung across our light fixture at the workbench, drying before we can reinstall it in the interpreter.
All keypunches were working fine, both CPUs, both 1402s and both 1403s. One of the tape drives, number 4 on the CT machine, had been misbehaving and got much, much worse while we worked on it.
The drive takeup reel didn't move strongly enough to keep up with the tape dumping into the right column during move forward, write or read operations. We examined everything and saw no obvious problem. The vacuum switch on the column works fine, the belts are newly replaced and the graphite brushes appear to have plenty of life left in them.
The drive then acted up, refusing to release the stop brake on the right reel so we could load tape. Oddly, the left (supply) reel is not locked even when it should be. Voltage on the right stop clutch was about 7V, definitely wrong.
We had been viewing the waveforms of the clutches on this drive, and others for comparison, but reached no conclusion in the early hours before the new problem arose.
The release button works properly, it just doesn't change the behavior of the two stop clutches. We tried to use the current measuring panel built into the drives; this puts each clutch through a 1ohm resistor and brings the two ends out to + and - banana jacks.
Using a voltmeter, we could verify the current, which was the voltage reading since IR with a resistor of 1 ohm makes V and I the same. The current reading on the bad drive was 0 for both left and right stop clutches, but as we mentioned the left was free and the right activated.
Each clutch has two paths for current - one used for full braking force and the other for partial braking force. Essentially, one mode is during Load and Unload, while the other is for normal tape operations. The current measuring panel only looks at one of the two paths to each clutch.
We searched through the schematics for any parts that were asymmetric - that could give different results on the left and right stop clutch. We were starting to check capacitors and other components and begin detailed voltage measurements when the clock hands approached 3PM. We surrendered for the day and will work on this again next week.
The demo team reported that the tapes weren't accessible by either program or TAU functions, although we only had the last drive switched off because of its problems. It may be that we had disrupted the chain or some more involved failure in drive 4 was jamming the tape bus even with power removed.
The demo team mentioned receiving some read checks on cards they fed into the CT 1402, although we had experienced no failures in our tests earlier in the day. We don't know if that was a problem with the deck or the reader, so we will watch it more carefully next week.
Tape drive number 2 on the DE machine had a bad door latch, something we wrestled with last week. This week, the team (mainly Frank, Bill and Dale) did some adjustments, grinding of excess metal and reshaping to get the latch to work properly and easily.
There is a cover inside the door that needs some trimming as it interferes with the newly adjusted latch plate - that will be done at home by Frank or Dale.
Sat, Mar 09, 2019 5:32 pm Carl wrote
Frank King, Michael Marineau, Ken Shirriff, Alexey Toptygin, Robert Garner and I were at the museum on Wednesday.
The number 4 tape drive on the CT machine had broken belts that were replaced the prior week, but the reconnection of all the cables and other parts was completed on Wednesday. The drive works well now.
The CT 1402 reader was jammed from a previous demonstration. We removed the bits of cards and verified that the reader was back in service.
Work continued on the 552 Interpreter, which had its print unit removed leaving the bottom bar with the 60 friction rods dangling in the machine. We further disassembled the unit to get the bottom bar out.
The friction rods were removed, soaked to clean and reinstalled in the bar with fresh grease. Other areas that had stale grease were cleaned as well. Next week we will insert all 60 friction spring rods into the print head, then reinstall the newly reunited parts into the interpreter.
We have the print ribbon removed but it is dry with barely any ink to transfer to cards. We are going to investigate methods of reinking this, probably using stamp pad ink refills diluted with appropriate solvents that will soak into the ribbon.
The workroom was further straightened and many objects were put onto the new shelving.
We swapped the formica covered desk from the Liebert room 029 with the donor machine that we are restoring in the workroom. This unit is now in very good condition and almost ready to hand over to the education department for use by the public.
We have dollies that we are going to be used to move this keypunch around. Frank has the appropriate spade bits to drill a hole for the four keypunch feet, which we will attempt next week.
We tested the tape drives on both systems writing and reading back patterns, as well as movement tests. They work well. There is a bad spot on the tape on CT drive 2 which results in a write check withing a dozen or two writes from the load point, but that is not a tape drive fault.
We did notice that demonstrators are setting all tape drives to the same number to drive them all from the TAU, which works okay for Move Forward commands but wont' work for other commands such as Rewind.
This can lead the demonstrators to report the drives as malfunctioning when one of them runs away or fails to rewind. Remember that multiple drives on the same number violates the design of the system; the fact that you got lucky in the past is no guarantee that it will work the next time.
Wed, Feb 20, 2019 6:36 pm Carl wrote
Today we had Michael Marineau, Ken Shirriff, Frank King, Marc Verdiell, Robert Garner and our newest team member Jorge (sorry, don't remember his last name nor email address) joining me.
We had reports of failure to advance cards on the 026 closest to the door. Ken and I cleaned out the punch area with a card saw and it appears to be working well now.
Pat Buder had reported some BigPrint runs where visitor names were printed with numbers replacing certain letters, due to ignoring the 12 punch in those columns. He saved one card where this happened and gave it to us.
It is a bit puzzling how this could be misread without causing an error stop. We should verify that the console switch is set to stop on error. The reader was able to transfer the entire program and have it run correctly, plus it failed on only certain cards.
The card was in perfect registration and Pat reported that a rerun would usually result in a proper print of the name. Frank King checked the timing of the card reader pulses on the CT 1402 and they were right on the money. We will look into this further next week but the failure mechanism remains a mystery.
Frank King did some adjustments to the card sorter to make it operate better.
We continued to work on the 'new' 029 keypunch brought to the museum by Frank King. It started the day with problems with its Clear function. We used the opportunity to work with the two newest members as a training opportunity, showing them how to work through diagnosing the problems and understand the schematics. Mike and Jorge did identify problems with the PCC microswitches at the base of the program drum unit and removed the oxidation. Ken, Frank and I provided guidance but mainly sat back. Mike also repaired the latch so that the card feed cover could be mounted on the machine.
The print support SMS card for the keypunch was successfully tested today. This card was missing from the machine and we had no spares on hand. I had taken a blank SMS card and built up the circuit - mostly diodes to quench the reverse EMF of the print solenoids, but also an RC debounce filter. The first attempt at this card made use of diodes we had on hand, but we tried to use small diodes which couldn't handle the reverse EMF and immediately committed suicide. After I bought and brought diodes with high enough current and PIV ratings, I fixed it up and we can now use it with this keypunch.
We have been watching the tape drives on CT for some time, as they are exhibiting intermittent problems particularly drive 3 (and 4 to a lesser degree). Ken and I put them through their paces but they were perversely well behaved when we were watching.
Finally, Robert, Ken and I assembled the three new storage racks and put them in position inside the workroom.
Thu, Jan 03, 2019 1:35 pm Carl wrote
We came in to work on two intermittent problems today. 1 - The 1402 reader on CT will occasionally spin the motor on a LOAD operation without attempting to clutch and feed cards. 2 - CT tape 3 would drop ready during operations.
Present were Frank King, Alexey Toptygin, Ken Shirriff, Robert Garner and me. Robert was busy taping an oral history through the morning.
Alas, we could not get either of the problems to happen today. The tape worked flawlessly as did the card reader.
Frank, Ken and I finished building an SMS card for the 029 keypunch. This card provided nine diode suppression circuits for the print solenoids plus an RC circuit. We did not have any spares of this card but built it from a card blank, the schematic and spare components on hand.
The 029 was powered up and the cleaning and adjustments began. It appears that the relay as well as many microswitches have corroded contacts.
Robert is ordering some more cabinet racks and further cleaning and arranging the workroom to improve its usability.
While testing everything in advance of the demo, I discovered that tape drive 4 on CT is now failing. It selects but fails to respond to commands. In one case,, I issued a rewind/unload from the TAU, the drive turned off ready and sat there for a couple of minutes and then did the unload.
We didn't have enough time to chase this new tape bug. Perhaps next week at least one of the tape bugs (drive 3 or 4) will be present to allow us to fix it. Iggy, your tape drives miss you! At a minimum we can make progress on restoring the 029 keypunch.
Wed, Dec 26, 2018 4:30 pm pm Carl wrote
Alexey and I were the only team members present today.
We replaced the lamp for the CT machine bit 1 in the op code register, which was burned out. Thanks to the demonstrators who spotted this.
We ran various tests to check the machines. We spotted two problems that must be attended to, however we ran out of time before Alexey had to return to work.
The third tape drive on the CT machine will drop out of READY after operations. Holding the READY button down allows it to operate properly. This occurs, with move forward, read and write commands.
When running with multiple drives set to the same address, the problems with the third drive produce various symptoms. That drive might ignore commands that are processed by the other three. Sometimes that drive responds to a command, e.g. rewind, while the other three do not.
Our working hypothesis is that the drive will drop out of ready for any command and should restore ready status at the end, but is intermittently failing to do so. No TAU errors are recorded when the drive drops ready, but it always occurs when a command is being issued.
The CT machine card reader will intermittently fail to load, not even picking cards. It just runs the motor. Other times it works fine. Since we had a bad relay in the 1402 that we had replaced the prior week, it is the most likely suspect.
Safety rules forbid a single person from working on the energized machine if there are no other team members present. I therefore left after lunch.
Iggy - any ideas?
Wed, Dec 05, 2018 5:19 pm Carl wrote
Frank and Alexei worked on keypunch #1, where the friction clutch was set much stronger than spec. They adjusted it to the correct value.
Marc and Mike worked on tape drive #2 on the CT system. It had been tripping the +6V circuit breaker right after power up and was inducing errors in tape operations on other drives as long as it was in the string.
They isolated the drive and pulled groups of cards in a binary search pattern until the located the AR card that was tripping the breaker. On examination we found a 6.2 uf filter capacitor had failed in a short circuit from +6V to ground.
A replacement capacitor was installed on the card, it was placed back in the drive and we verified that it could be returned to service.
Ken finally discovered a hairline crack in a trace on the WW card, the second sense amplifier card in the 1403. This was causing intermittent connectivity, leading to the sync checks and other printer errors on the DE printer.
Last week, while working with the as-yet undiagnosed card, we smelled smoke from the printer. Testing today showed that four hammers were not firing - columns 2, 9, 13 and 85.
The fuses on the four hammer driver cards were blown. We believe the intermittent operation of the amplifier card let the 1401 set the hammer drivers on but fail to reset them in time because the subsequent pulse never arrived.
The four fuses were replaced and the printer now prints on all 132 columns. We do see, however, that column 85 is printing way too far to the left. This may be a damaged coil and the source of the unknown smoke from last week. Next week we will investigate and either adjust or replace and adjust the coil for column 85.
The DE 1403 had some light bulbs that were burned out, such as the Print Ready status. Ken located the bad bulbs and we looked for replacement lamps. We knew that these operated on 10-12 volts at a draw of 40 ma, so we set up a current limited power supply in the workship and sorted through our lamp stock until we found the appropriate bulbs. The new bulbs are in place.
We do have a defect on the CT machine that needs to be researched, but it is not causing any problems right now. When the 1402 reader has no cards in the hopper, the Reader error light on the 1401 panel will turn on. It goes off when cards are placed in the hopper. While it is on, there is no error condition lit up on the 1402 panel. This is not what happens on the DE system.
Wed, Nov 14, 2018 7:09 pm Carl wrote
CT would not power up, similar to the problem we had weeks ago with a oxidized contact, but this time on relay 4 rather than on relay 1. Sanded and system back in operation.
DE has printer sync checks. Discovered near demo time
Tape 1 on CT worked fine but reflective tape rather far in, dumping probably due to insufficient winding while loading.
Tape 2 blows +6V breaker. Breaker tested to 4A without trip. Two wires run from breaker - one to main logic panel and one to preamp board. Removed both and breaker still trips!!
Ran out of time to look further due to demo