Return to main menu

Unit Record (and Sequential Processing) Data Flow

by Ed Thelen - updated April 17, 2017
the following should be checked by a real data processing expert ;-))

This is a developing web page intended to provide an overview of how data used to flow through a data processing department in the 1900 through 1990 to handle a variety of problems, such as generating pay checks, bills, bank statements, and other common situations.
I will call these "traditional functions".

Several points are intended:

  1. There is no mystery to performing the "traditional functions", just plain straight forward, relatively simple logic.
  2. Oddly, the data flows follow similar paths in the above (and many other) situations.
  3. The data flow of the "traditional functions" is also shown through a stored program computer (such as a 1401 system) using serial (sorted) records.
        The actual data flow is substantially identical in a computer even though several "unit record" machines are replaced by the computer, .

Warning, for "simplicity", some interesting, useful details are left out.
For a first pass, the following is complex enough already yet!
    This letter discusses long card sorts.

"Data Flow"
There is the concept that business data "flows" through various stages of data processing, from data entry, sorting, through to the finished result - such as pay checks, bills, inventory, bank statements, ... There two basic kinds of data:
  1. Short term data, such as current electric meter reading, hours worked this week, ...
    We shall call this "Current Detail Data".
  2. Long term data such as customer number, customer name, customer postal address, pay per hour or cost per KWHour or ... Some data accumulations such as tax totals, total pay this year, ...
    This data is needed each billing, payroll, ... cycle.
    We shall call this "Master Data".

Customer or employee number
To get these finished results, you can use sequential processing (on sorted data records). By sorted I mean that say each customer has a unique customer number - this sounds heartless, but helps solve the question of how to uniquely handle three "John K. Smith" customers or employees.

Sequential Processing
I have long wished to find a "vehicle" so show that the data flow, from data entry and verification, sorting, through to final result
- be it payroll, billing, inventory, bank account processing, ...
in unit record and also magnetic tape based data processing (such as IBM 1401 system) is remarkably similar.

New data, say the electric meter reading, is processed against a master file (containing say the customer name and address and the previous meter reading and ) to print electric bills for thousands of customers. Other new data, say the records of payment, can also be processed against the same master file to print customer account information.

Also the maintenance of the sorted master files. The customer may have moved locally need address change, and also handle new and dropped customers. The methods used in maintenance is similar to the methods shown in Figures 1 and 2. As is reconciliation of errors and mistakes.

If you know one system above, you know the basis for all :-)). The data names change, but the techniques remain the same.

This book, 17 megabytes, (offered by Stan Paddock) at least has a ( too detailed? ) data flow diagram for "Accounts Payable" in chapter 6 "Case Study", pages 82 and 83.

Here is a Data Flow of an Electric Company Billing Operation

Figure 1, Data Flow of "Traditional Functions" though Unit Record equipment.

Oddly, the same general data flow is used for much of sequential business data processing, including the above mentioned
- payroll, billing, inventory, bank account processing, ...
only the types of data are different.

Details in sequential order

After replacement of much unit record equipment
- IBM 083 Sorter(s) (maybe keep one for utility use)
- IBM 077 Collator(s)
- IBM 602 Calculator(s)
- IBM 402 Accounting Machine(s)
- racks and racks of plug board programs for the above 077, 602, 402 units
with an IBM 1401 w tapes system, the data flow is remarkably similar, but the 1401 system replaces much equipment, and card handling, especially handling during sorting.

Figure 2, Data Flow of "Traditional Functions" using a stored program computer with magnetic tapes.

Added Details in sequential order

Section Describing various functions and details

Source Documents - "Electric Meter Readings" - -
These are the "source documents" for this particular "run".
In this case they are the scrawlings (actually careful hand printing) made by the meter reader who visits (or used to visit) your electric meter each month. (Automation, with electric meters which broadcast you meter reading, is replacing the meter reader who used to excite your dog each month.)

The meter reader hand prints the current electric meter reading onto forms which already contain:
- The customer's account number and street/building address and hints where to find the electric meter.
- The customer number is for later use by the accounting department.

IBM 026 Keypunch

The keypunch operator was always a woman - except for when I keypunched for a troubled insurance company during a college Christmas Break. The company was that desperate to get a particular "job" done.

She is seated at a leased IBM 026 keypunch, most likely one of many. IBM leased (did not sell) equipment. (For $15 less a month, a company/government could lease an IBM 024 keypunch, but that did not have a printer to print the meaning of the holes.)

There is an account number field, a detail value field (in this case watt_hours).
Each keypunch had a format card to help the operator punch into the required columns (called fields).
These women were fast and accurate - at least 3 times faster than when I got paid to do it.

IBM 056 Verifier

Another "keypunch girl", sitting at an IBM 056 Verifier, reads the same document and "keypunches" the data into the same detail card as the "girl" at the IBM 026.

Actually, the 056 compared the holes punched into the card with the keys being press by the 056 operator. If the holes in the card matched the key strokes by the operator, everything matched and the 056 cut a little semicircular notch in the 84th column of the card, at about the row 3 position. Since the 84th column was about 1/2 not present, the notch was on the trailing edge of the card, easily visible in a deck of cards.

If there was a mis-match ???

Any card in a deck that does not have the "verification notch" was clearly visible.

This procedure helped assure that two sets of eyes and hands agreed about what was on the source document.

An error on the source document or the detail card usually causes customer ill will, company/government embarrassment, and expensive labor to correct. "Garbage in, garbage out" is to be avoided !!!

Detail Cards
The newly punched "Detail Cards" are gathered together to be sorted. Each card contains at least two "fields" (groups of columns) of data
  1. A "sort key", most usually the customer/employee number. In later processing this helps gather the customer's detail information to be adjacent to the customer's other data, such as billing rate and address, some historical data such as past due bills, totals such at energy used year_to_date, ...
  2. the detail data, in this case the current electric meter reading

IBM 083 Sorter

An excellent article by Ken Shirriff,

The IBM 083 sorter (other potential sorters are 082 and 084) sort the cards by reading a single column and placing the card in the pocket identified by that column, 0 through 9, two "zone" rows ( 11 & 12, used for alphabetics and special characters), and a reject pocket for blanks or rejects or errors.

There is a definite ritual to sorting in these machines.

A bad joke - How was T.J. Watson buried? Face down, 9 edge in.
And that is how you insert cards into most hoppers.

Lets assume the customer/employee id is all numeric. (Easier for everyone that way)
You start sorting with the least significant digit, and in further passes sort on more significant digits. If the cards are taken from the output hoppers in the correct order and properly handled, you eventually get a deck of cards sorted from lowest customer to highest.

Sorted Detail Cards (or magnetic tape records)
Alright! You have completed gathering, keypunching, verifying, and sorting your input (detail) information. Lets hope no garbage got in so that the confusion and labor of "reconciliation" is avoided!!)

Sorted Master Cards (or magnetic tape records)
By "Master" data we mean that data that is long term, hopefully your customers are relatively long term. This includes name, address, historical data such as previous meter reading, electric rate, special discounts, ... as well as the customer to match up with the detail information.

IBM 077 Collator

This fascinating machine can compare the values of fields from two card decks. It can do electro-mechanical compares of two fields for greater_than, equal_to, and less_than, and make decisions, based on its plug_board wiring of what to do.

If the plug_board is properly wired for this, it can merge two sorted decks together. It can place the customer master card(s) first, then the customer detail card(s), into the output deck, so that the all the customer information is together and can be processed by some machine further down the line, such as an IBM 402 accounting machine.

It can also be wired so that it can detect if there is a customer detail card, but no customer master card. This situation is to be avoided !!

Merged Cards
The deck is sorted by customer number, with in this case, the master card(s) first followed by the detail card(s). The IBM 077 was programmed (by plug_board) to eliminate detail cards with no master cards.

IBM 602 Calculator

None of the previous machines, nor following machines, can multiply or divide. In our situation we need to multiply the watt_hours by the billing rate.

A plug_board program can read the billing rate from a master card and multiply that by the watt_hours from a detail card, and punch the product into the same detail card.

Calculated Merged Cards
Here we are, everything ready to be processed by the accounting machine.

IBM 402 Accounting Machine
IBM 402 & Trip Report, July 2010

This machine could read cards, accumulate sums, print data from cards (such as names and addresses) at up to 100 lines per minute, and print from the internal accumulators.

"1950's tax preparation: plugboard programming with an IBM 403 Accounting Machine" by Ken Shirriff

Ken (above) uses the very interesting descriptive sentence:
" Another important operation is to compare two cards to see if they have the same id (and should be counted together) or if they have different ids (so a subtotal should be printed and the counters reset)."

Data processing people use the phrase "control break" for this function.

The IBM 403 is similar in function and operation to the IBM 402. with a little handier multiline print optional control.

Printed Output - "Electric Bills"
Special preprinted forms could be used in the printer, such as for bills and pay checks.

Another machine was available called a "burster" to separate the continuous forms into individual documents. Yet another machine was available to remove carbon paper for "multi-part" documents such as an "original" and two "copies".

Old Cards
You could store these in long term storage (the mountain) say for seven years to satisfy government requirements.

New Sorted Master Cards (or magnetic tape records)
You may need to retain updated customer information such as electric usage year_to_date, pay year_to_date, FICA (Social Security contributions), ...
This can be punched (not shown) in a "Summary Punch" ( an IBM 514 will do ) under control of the IBM 402 accounting machine. These can be merged (not shown) back into the master cards.

In the case of magnetic tape records, no special equipment or passes are required. The program just writes a new master tape.

IBM 1401 - Sort (using magnetic tape records)
C24-3317-1_sort7spec.pdf - Sort7 Spec & Operating Procedures , in BitSavers

Sort 7 does not copy cards to tape. To do this, another program (not shown in Fig 2.) is used. This program is likely considered trivial, other than tape write error recovery. Standard I/O program, such as IOCS, can be used to handle the tape writing and error recovery.

(A quick (non-essential) word about placing more than one record (card) onto tape for operating efficiency. If you place only one card image per tape record or "block", you are wasting tape and running time. The key word is "record block" where maybe 8 or 10 card images are placed in one contiguous record on tape. See Block Layout in Wikipedia, page 12 of C24-3317-1_sort7spec.pdf - Sort7 Spec & Operating Procedures above, also see Magnetic Tape Capacity and the Effect of "Tape Blocking", and also from Stan Paddock, 10/26/2009.)

Sorting data records (cards) using a magnetic tape system is MUCH faster, and much less labor intensive compared to using the physical cards and a card sorter, such as an IBM 083.

IBM 1401 - Run (using magnetic tape records)
You may notice that the 1401 tape system replaces a number of (much slower) unit record devices. Since the 1401 system is so much faster than the replaced devices, multiples of each device can be replaced in medium and large size installations.
  1. Replacing the IBM 077 Collator(s), the merge operation using is almost trivially performed by the 1401 software as part of inputting Detail Data and Master Data. Exception situations (no corresponding master records) can be punched using the 1401 system's 1402 into a selected pocket.
  2. Replacing the IBM 602(s), any necessary arithmetic, adding, subtracting, multiplying, dividing, can be easily performed quickly in the 1401. If the optional multiply and divide features are not present, existing subroutines can be used.
  3. Replacing the IBM 402(s) accumulators, the 1401 easily handles any totalizing.
  4. Replacing the IBM 402(s) 150 line/minute printer, the IBM 1403 printer can print 600 lines/minute
  5. Replacing the IBM 514(s), the IBM 1401 write a New Sorted Master - to be used next time (an Electric Billing operation (run) is performed

Are you familiar with unit record data processing ??
   If so, could I use you as a consultant/author ??
(I was more a sidewalk superintendent, never seriously wired a plug board,
   and the only time I sorted cards was to straighten up a dropped deck.)

How it is done with the current relational data bases, I have no clue -
For all I know the traditional tasks are still done sequentially ?????????

Also I wish to expose folks to the joys of sorting
   - on unit record equipment such as an IBM 083
   - with a computer and magnetic tape storage

This letter discusses long card sorts.
While at the 2018 Vintage Computer Festival, in CHM, Henry Strickland asked about BIG card sorts.
IBM 1401 Docent Bill Worthington responds:
Hi, Henry.

Punched card sorting was a long, tedious, and very important part of data processing (DP).

We had sorters at the bank that worked for as a 1401 programmer for four years before joining IBM for forty years. It sounds as though your mother and I did very similar work for IBM. The bank's trust department work was a punched card (unit record) operation for the four years I worked for the bank. They had 3,000 card file trays that were wheeled around when it came time to do any of the batch accounting for the trust dept. As part of the processing, a lot of sorting. The sorters they used had an extra card holder that was mounted just behind the sorter itself. Here is a picture of an IBM 083 sorter with the extra card stackers;
Image result for ibm 087 sorter

We have a sorter set up like this in Revolution which you can see next time you visit the museum. There is another that we use during the 1401 demonstrations on Wednesday afternoons and Saturday mornings. I'll be leading the 1401 demo this Wednesday at 3:00. Can you come see the demo?

An adroit operator would be able to keep an eye on how full the sorter's stackers were and he would reach into the stacker and pull out the cards that had fallen into a pocket. He would then put the cards into the rack behind the sorter. (I'm sure there times when the extended stacker filled up and the whole sort process would have been stopped until space was found for the cards. I never saw this happen.)

Now, to your question: sorter operators would leave a card on the top of the sorter threatening all sorts of unusual punishment if anyone disturbed the in process sort. When it got close to the end of a shift, the operator would try very hard to find a stopping point where the cards could be taken from the sorter and put into the extra card frame to be continued. If the operator needed a bio-break,, he would just stop the sorter and resume when he got back.

Depending on the workload, an operator would hand off to the next shift operator. It would be very straight forward since the new operator would be told which card column was being sorted and just be handed the baton for the next shift. The "run book" told what to do next. It was very systematic.

The key to remember is that the sorter sorts only one card column at a time -- a, then b, then c, then... a is the ones digit, b is the tens digit, c is the hundreds digit, ... It isn't difficult. The key being to not drop the cards and have to restart the sort. So, to sort by Zip-code would mean five passes of the card file. If we're doing payroll, we might have nine passes for the Social Security number. Hope this helps.

Aside: Note that I said "he" as the sorter operator. A case of cards (five boxes of 2,000 cards) weighed about twenty-five pounds) Ladies weren't supposed to deal with heavy things. So they were key-punch operators and did the data entry for the DP department.


My e-mail address is :-))

Updated April 17, 2017