TOPS-20: Making more space in the public structure

The Panda distribution is for sure the easiest way to make contact with TOPS-20, and in a previous entry I explained how to set it up to run in a Raspberry Pi (or similar) computer. One of the challenges of using those ancient operating systems is the lack of support and/or documentation. Fortunately, there is a whole treasure of information in the internet, and a quite active newsgroup devoted to the PDP10 class computers, where some of the actual designers of those machines and those operating systems is happy to answer a beginners question about the work of their life.

On the other hand, one of the fun things one can do is to dig around and find the answers from himself!

The problem: lack of disk space in the public structure

The "public structure" is what in other environments we would call the "system disk" or the "root filesystem". The Panda Distribution comes with a public structure defined in a RP07 class disk. Those disks held about 476 Megabytes, and as far as I know the "regular" TOPS-20 did not support those devices for the public (boot) structure. The late Mark Crispin modified heavily the montitor (what we could call "kernel" nowadays) to support those big (at the time) disks. Just after the first boot, there is plenty of available space in Panda's TOPS20 structure:

@info disk

 TOPS20:<OPERATOR>

 3 Pages assigned

 500 Working pages, 500 Permanent pages allowed

 82831 Pages free on TOPS20:, 133545 pages used.

That is about 160 MBytes of space (remember that the PDP10 was a 36 bit machine, so our eight-bit bytes are not really the best unit in that environment). For a casual use it is obviously more than enough. But if you start installing stuff, testing things and messing around you can find yourself with a few thousands of pages left. In that case, you can do several things:

• Purge the old versions of the log files to free up space. Those files, called LOGFILE.LOG, accumulate themselves in the SYSTEM: directory (that is a logical which points to TOPS20:<SYSTEM> and can take a big chunk of space.
• Create another structure and mout it as a USER structure, using it to do your stuff.
• Add another volume to the TOPS20: structure. This is the best long-term solution and it will be explained in this blog entry.

Preparation

We must do two things before we start adding space: preparing a DUMPER tape and actually dumping (backing up) the TOPS20: structure.

Creating a bootable DUMPER tape.

The first thing we must do is to make a bootable TAPE which must contain the DUMPER utility. The DUMPER utility is the TOPS-20 backup program. Since we will create a new, pristine structure we need to be able to boot directly into DUMPER from tape to restore our current information.

The good news is that it is really easy to create a dumper tape:

• First, we will create a virtual tape, escaping to the KLH10 console (pressing CTRL-\) and typing:

@[HALTED: FE interrupt]

KLH10> devmount mta0 dumper.tap create

Mount requested: "dumper.tap"

KLH10> [mta0: Tape offline]

KLH10> cont

Continuing KN10 at loc 01063335...

@

• Now we have to copy the monitor program and DUMPER itself into the tape. We will use Mark Crispin's monitor, which supports big structures:

@copy system:monitr.exe mta0:

 <SYSTEM>MONITR.EXE.1 => MTA0:MONITR [OK]

@copy tops20:<subsys>dumper.exe mta0:

 <SUBSYS>DUMPER.EXE.1 => MTA0:DUMPER [OK]

@rewind mta0:

@unload mta0:

@

• And we are done. We just have to detach the virtual tape from the emulator, and keep the file safe

@[HALTED: FE interrupt]

KLH10> devunmount mta0

Unmount requested

[mta0: Tape offline]

KLH10> 

Backing up the system

Now we need to get a backup copy of our running system on tape. To do this we will shut down the system, and then we will use DUMPER to create the backup. These are the steps:

• Enable privileges and shut down the system typing CTRL-E followed by CEASE NOW

@enable

$^Ecease now

 TOPS20 Will be shut down IMMEDIATELY 

[Confirm]

12-Feb-2015 12:54:20 ACJ: System shutdown set by job 9, user OPERATOR, program C

EASE, TTY5

$

[Timesharing is over]

OPERATOR - Wait for the message "Shutdown complete" before

entering commands to PARSER.

[ni20_cmdchk: Illop=0 wd=4,,0 qe=1443101]

[dpni20-W: "tap0" multicast table ignored - interface not dedicated]

[ni20_cmdchk: Illop=0 wd=4,,0 qe=164061]

SJ  0: OPR>Killed Job 1, User OPERATOR, TTY13, at 12-Feb-2015 12:54:21

SJ  0:  Used 0:00:00 in 0:17:24

[dpni20-W: "tap0" multicast table ignored - interface not dedicated]

[ni20_cmdchk: Illop=0 wd=4,,0 qe=164061]

12-Feb-2015 12:54:24 HSYS: Shutdown complete

• Create a virtual tape to hold the backup. We do this as we did before, pressing CTRL-\ to access the console:

[HALTED: FE interrupt]

KLH10> devmount mta0 backup-tape.tap create

Mount requested: "backup-tape.tap"

[mta0: Tape online]

KLH10> cont

Continuing KN10 at loc 01003136...

$

• Run DUMPER and take the backup. This step can take some time. You must be careful to not forget to issue the CREATE command before the SAVE. The CREATE command allows you to do the full restore you'll need afterwards, adding to the backup tape the image of the directory structure.

$r dumper

[Using MTA-DUMPER:]

DUMPER>tape mta0:

DUMPER>create

DUMPER>save tops20:

 DUMPER tape #1, Thu 12-Feb-2015 1257. 

 TOPS20:<ALGOL>

 TOPS20:<BBOARD>

 TOPS20:<BLISS>

.

.

.

 TOPS20:<UNSUPPORTED>

 TOPS20:<UTF9>

 TOPS20:<UTILITIES>

 TOPS20:<WINDOW>

 Total files dumped:   7403

 Total pages dumped: 104812

 Directories dumped:    141

 CPU time, seconds:  570.00

• Detach the backup tape and shut down the simulator. Remember the system was already shut down, so we can quit KLH10 without damaging anything.

Now it is a good practice to make a copy of the virtual disk we are about to replace, just in case we are unable to restore the tape we have just made. Since we are going to create the disks from scratch, you can simply rename the file to avoid it to be overwritten. That is one of the nice things of emulation: we can do things we couldn't dream about if we were using real devices!

Creating the new system disks

Now we are going to create the new disk structure. The KLH10 distribution included in panda contains a configuration file which is appropriate to install a system from scratch. That file is inst-klt20.ini. We will modify this file to acomplish what we want to  do. The part of the file which contains the device definitions is:

devdef dte0 200   dte   master

devdef rh0  540   rh20

devdef rh1  544   rh20

devdef dsk0 rh0.0 rp    type=rp06 format=dbd9

devdef mta0 rh1.0 tm03  type=tu45

We will change the disk type to rp07 add another disk definition, so it will become:

devdef dte0 200   dte   master

devdef rh0  540   rh20

devdef rh1  544   rh20

devdef dsk0 rh0.0 rp    type=rp07 format=dbd9

devdef dsk1 rh0.1 rp    type=rp07 format=dbd9

devdef mta0 rh1.0 tm03  type=tu45

Now we will start the simulator. Be sure you have the mtboot.save file in the current directory and start kn10-kl using the modified configuration file:

$ ./kn10-kl inst-klt20.ini 

KLH10 V2.0H (MyKL) built Feb 12 2015 00:50:35

    Copyright ? 2002 Kenneth L. Harrenstien -- All Rights Reserved.

This program comes "AS IS" with ABSOLUTELY NO WARRANTY.

Compiled for LINUX on ARM with word model USEINT

Emulated config:

CPU: KL10-extend   SYS: T20   Pager: KL  APRID: 3600

Memory: 8192 pages of 512 words  (SHARED)

Time interval: SYNCH   Base: OSGET

Interval default: 60Hz

Internal clock: COUNT

Other: MCA25 JPC DEBUG PCCACHE EVHINT

Devices: DTE RH20 RPXX(DP) TM03(DP) NI20(DP)

[MEM: Allocating 8192 pages shared memory, clearing...done]

KLH10# ; Sample KLH10.INI for initial installation

.

.

.

KLH10# ; Load tape bootstrap directly

KLH10# load mtboot.sav

Using word format "c36"...

Loaded "mtboot.sav":

Format: DEC-CSAV

Data: 4067, Symwds: 0, Low: 040000, High: 054641, Startaddress: 040000

Entvec: JRST (120 ST: 0, 124 RE: 0, 137 VR: 0,,0)

KLH10# 

KLH10# ; Now ready to GO

KLH10# [EOF on inst-klt20.ini]

KLH10# 

Now we must attach the bootable dumper tape we build before, and start the virtual frontend:

KLH10# devmount mta0 dumper.tap

Mount requested: "dumper.tap"

[mta0: Tape online]

KLH10# go

Starting KN10 at loc 040000...

BOOT V11.0(315)

MTBOOT>

To load the monitor from the mounted tape, we will issue a '/L' command (without the quotes) and after that we will invoke the disk formatting procedure issuing a '/G143':

KLH10# go

Starting KN10 at loc 040000...

BOOT V11.0(315)

MTBOOT>/L

[BOOT: Loading] [OK]

MTBOOT>/G143

[For additional information type "?" to any of the following questions.]

Do you want to replace the file system on the system structure? 

The whole disk and structure formatting is as follows:

Do you want to replace the file system on the system structure? YES

Do you want to define the system structure? YES

How many packs are in this structure: 2

On which "CHANNEL,CONTROLLER,UNIT" is logical pack # 0 mounted? 0,-1,0

On which "CHANNEL,CONTROLLER,UNIT" is logical pack # 1 mounted? 0,-1,1

Do you want the default swapping space? Y

Do you want the default size front end file system? Y

Do you want the default size bootstrap area? Y

Do you want to enable password encryption for the system structure? Y

What is the name of this structure? TOPS20

[Structure "TOPS20" successfully defined]

[TOPS20 mounted]

?TOPS20 unit 0 has no BAT blocks.

Do you want to write a set of prototype BAT blocks? Y

?TOPS20 unit 1 has no BAT blocks.

Do you want to write a set of prototype BAT blocks? Y

PANDA PROGRAMMING, PANDA TOPS-20 Monitor 7.1(21733)-4 Internet: Loading host names

%%No SETSPD.

Be aware of the numbering of the controller (it is -1), and take almost all the defaults, excepting for the structure name, where you must specify TOPS20 explicitly. After that, our monitor will complain about not being able to load the internet address list and other stuff we don't need now. Eventually, the messages will stop. The next step will be to restore the backup we made previously.

Restoring our system

To begin the process we will press CTRL-C and will get a MX> prompt. Following from the last screen capture:

%%No SETSPD.

System restarting, wait...

Date and time is: Thursday, 12-February-2015 9:17PM

Why reload? 12-Feb-2015 21:17:48 ***BUGINF NOADDR*** Failed to find SYSTEM:INTERNET.ADDRESS file  Job: 0, User: OPERATOR

12-Feb-2015 21:17:48 ***BUGINF NOHSTN*** Failed to find host name file SYSTEM:HOSTS.TXT  Job: 0, User: OPERATOR

12-Feb-2015 21:17:48 ***BUGCHK NOACTF*** No account database - account validation disabled  Job: 0, User: OPERATOR

Why reload? OTHER - Question timeout

 DDMP: Started

No SYSJOB

12-Feb-2015 21:18:49 ***BUGCHK KNICFF*** PHYKNI - Cannot reload the NIA20  Job: 0, User: OPERATOR, Data: 600104

NO EXEC

MX>

Now we will load DUMPER into memory, following these steps:

MX>GET FILE mta0:

?

MX>GET FILE mta0:

MX>sTART

DUMPER>

Yes, the first command gives out and error, and the second one works. Notice you just have to type the first letter of the commands. A 'G' for GET FILE and an 'S' for START.

Now we must drop to the KLH10 console and mount the virtual tape which contains our backup:

DUMPER>[HALTED: FE interrupt]

KLH10> devmount mta0 backup-tape.tap 

Mount requested: "backup-tape.tap"

[mta0: Tape online]

KLH10> c

Continuing KN10 at loc 01053253...

DUMPER>

We are almost there. To restore the contents of the tape we will use the CREATE and RESTORE commands. Again, do not forget to issue the CREATE before the RESTORE. Ignore the error messages and be ready to wait for a while.

DUMPER>TAPE MTA0:

DUMPER>CREATE

DUMPER>RESTORE TOPS20: TOPS20:

UNNAMED saveset 12-Feb-2015 2057

%Failed to create TOPS20:<ROOT-DIRECTORY>Cannot find error message file - RETRYING

12-Feb-2015 21:24:52 ***BUGINF RTCRAT*** JSYSF: ROOT-DIRECTORY CRDIR attempted  Job: 1, User: OPERATOR, Data: 777576170026, 60000000

0000

12-Feb-2015 21:24:52 ***BUGINF RTCRAT*** JSYSF: ROOT-DIRECTORY CRDIR attempted  Job: 1, User: OPERATOR, Data: 777576170026, 60000000

0000

?Directory TOPS20:<ROOT-DIRECTORY> not created - Cannot find error message file

TOPS20:<ACCOUNTS> created

TOPS20:<ALGOL> created

TOPS20:<BBOARD> created

TOPS20:<BLISS> created

TOPS20:<BLISS-SOURCES> created

TOPS20:<CHIVES> created

TOPS20:<CHIVES.V1> created

TOPS20:<CHIVES.V1.CONFIG> created

TOPS20:<CHIVES.V1.DOCUMENTATION> created

TOPS20:<CHIVES.V1.SOURCE> created

TOPS20:<CLISP> created

TOPS20:<DECNET-SOURCES> created

.

.

.

TOPS20:<UTILITIES> created

TOPS20:<WINDOW> created

 Loading files into TOPS20:<ALGOL>

 Loading files into TOPS20:<BBOARD>

 Loading files into TOPS20:<BLISS>

.

.

.

Loading files into TOPS20:<TSU>

 Loading files into TOPS20:<UNSUPPORTED>

 Loading files into TOPS20:<UTF9>

 Loading files into TOPS20:<UTILITIES>

 Loading files into TOPS20:<WINDOW>

 End of Tape.

 Total files restored:   7402

 Total pages restored: 104811

 Directories created:   140

 Files skipped:     1

DUMPER>

And we are almost done! Now we just go back to the KLH10 console and quit the simulator. If we check the working directory we'll found two disk images (and possibily the old, renamed image). Now we have to modify the working configuration to add the newly created disk (the original name is klt20.ini) and start the simulator. If everthing has been done properly, we will be able to enjoy our new, super-sized public structure!

@info disk

 TOPS20:<OPERATOR>

 3 Pages assigned

 500 Working pages, 500 Permanent pages allowed

 297809 Pages free on TOPS20:, 134943 pages used.

Remember you can get the panda distribution, which includes both the PANDA TOPS-20 image and the KLH10 emulator (compiled for x86 linux) here. Enjoy!

Networking a virtual KS-10

SIMH is in continuous evolution, adding new enhancements practically each month. The last oficially released version is still 3.9, but the new 4.0 version can be downloaded (as source) from the development git repository, and it is usually stable enough to be used (with all the caveats about using beta software).

The communication devices for several of the emulated DEC machines have received some attention in the last months. Right now it is posible to configure synchronous communication devices for the PDP-10, PDP-11 and VAX simulators, and those emulated devices have reached a point so useful things can be done with them. Specifically, it is now posible to give DECNET connectivity to TOPS-10 running over the pdp10 simulator. That simulator impersonates a KS based DECSYSTEM-2020, which had no ethernet devices supported in DECNET, so the only chance to network it is to use the serial devices. Right now simh includes support for the KDP/DUP combo and for the DMR controller. The DMR controller is still a little bit green and does not work properly, but the KDP/DUP is good enough to set up a decnet link. This article will tell you how to do it.

Communication devices in simh 4.0

Most communication devices emulated in simh share a common framework, so they have similar setup commands. They can be wired to real communication ports in the host computer, so actual wires and communication lines can be used to hook virtual machines. Nevertheless, we are going to use "virtual" wires based on TCP/IP connections. For each virtual line, we have to configure both communication endpoints:

• We must attach our endpoint to a local port (TCP or UDP)
• We must peer with the IP address and port number of the remote endpoint

This is the relevant stanza of my vax780.ini configuration setting un two lines of a DMC-11 device:

set dmc enable

set dmc lines=4

set dmc0 peer=192.168.0.12:21080

att dmc0 33061,UDP

set dmc1 peer=192.168.0.10:21911

att dmc1 34061,UDP                                                              

The values are self-describing. Line 0 is attached to the local UDP port 33061, while it is paired to the remote UDP port 21080 in the machine with address 192.168.0.12. Of course if you are running all the machines in the same host you would unse 127.0.0.1 here.

Now, the DUP device is an exception, since it does not share the same underworks as the DMC, DZ, VH and other serial stuff. The config stanza in the pdp10.ini file is:

set kdp enable

set dup enable

ATTACH DUP0 21080,Connect=192.168.0.8:33061,UDP

                                                   

Instead of using two separated statements, the attaching and pairing is done in just one line. Just check the "wires" are correctly connected and it will be ok. The choice between UDP and TCP (to use TCP just don't specify UDP) is basically up to you. If you plan to use the link between an instance running in a laptop and your home system you will probably want to use TCP, since it makes it easier to set up the port forwarding and NAT stuff.

Generating a TOPS-10 monitor with KDP/DUP and DECNET

I am assuming you have a TOPS-10 version 7.04 up and running. If not, refer to this article. Once you have it up, you will need to know the name and address you want to assign to your future new TOPS-10 DECNET node. You will want also to get:

• The DECNET-10 4.0 for TOPS-10 7.04 tape image.
• The DECNET-10 manuals. 

Be aware those manuals are for the 7.03 version of the monitor, and there are important differences in the install process. They can be just a guideline. 

To generate a monitor with the necessary components, just follow the standard MONGEN procedure described here, with some differences. This is the section with the most important changes. The non-default responses are in red.

.

.

.

IPCF (YES,NO) :

ENQ/DEQ (YES,NO) :

Disk sets (ALL) :

Configure non-autoconfigured hardware (NO,YES) : YES

Number of RX211s (0,0-2) : 0

Number of KMC/DUP Lines (0,0-2) : 1

Type of line for KDP0 (ANF10,DECNET,USER,IBM) : DECNET

Number of DMR11 Lines (0,0-8) : 0

Number of PTYs (20,1-476) :

Network software (YES,NO) : YES

Node name : BTES11

Number of remote TTYs (456,0-456) : 16

ANF-10 software (YES,NO) : YES

  Node name (BTES11) :

  Node number of central site (1,1-77) : 11

  Remote terminals (YES,NO) :

  Virtual terminals (YES,NO) :

  Remote card readers (YES,NO) :

  Remote line printers (YES,NO) :

  Remote paper tape punches (NO,YES) :

  Remote paper tape readers (NO,YES) :

  Remote plotters (NO,YES) :

  Remote DN8x DDCMP devices (YES,NO) : no

  Remote data entry terminals (YES,NO) :

  Remote task-to-task (YES,NO) :

  Number of connects (256,1-512) : 16

DECnet software (YES,NO) : YES

  Node name (BTES11) :

  Area number of central site (1,1-63) : 7

  Node number of central site (1,1-1023) : 911

  Router type (ROUTING,NONROUTING) : ROUTING

  Transmit password (DECNET20) :

  Remote terminals (YES,NO) :

Decimal "symbol,value"    

.

.

.

It is very important to not take the default to the "Number of remote TTYs" and "Number of connects" questions. If you take the default the resulting monitor will not fit in the KS-10 memory space and it will not boot. Use your own node name and address. By the way, you must configure ANF10, or else the link step will fail with undefined symbols. It is also important to set up the machine as a ROUTING node; NONROUTING works only for ethernet connected nodes.

I use the following MIC file to automate the compilation and linking of the monitor:

. compile/compile f,s        
. compile/compile devprm,netprm,d36par        
. compile/compile syscnf+<common,comdev,commod>        

. r link newsys/save/noinitial/hash:13k = /locals -        common,comdev,commod,tops10[10,7,mon,ks]/search-
/patch:200/counters/go        
^Z        
. r pip        
new:newsys.exe=newsys.exe        
^Z

This procedure leaves the new monitor at NEW:NEWSYS.EXE, so it can be booted answering [1,5]NEWSYS to the BOOT> prompt.

Configuring DECNET-10

The monitor will start up the required components (NML, MX) automatically, but you have to add the commands to start up the circuit and the FAL listener to SYS:SYSTEM.CMD:

NCP SET CIRCUIT KDP-0-0 STATE ON

SET FAL-STREAM 0 NETWORK DECNET

SET FAL-STREAM 1 NETWORK DECNET

START FAL-STREAM 0

START FAL-STREAM 1                   

And while you are in it, make sure you have got also the following line:

CONFIG ADD MEMORY 512K 1024K

This one should go just before the NCP line shown above. This configuration allows for two simultaneous FAL operations (that is, you will be able to access your TOPS-10 files from two DECNET nodes at the same time). You can add more FAL streams, or leave just one, as you wish.

Booting the new monitor and testing the network

Now you can boot your newly created monitor. First shut down the system using SET KSYS NOW at the OPR> prompt, then stop the simulator using CTRL-E and boot it again.

sim> b rp

BOOT V4(76)

BOOT>[1,5]newsys

[Loading from DSKB:NEWSYS.EXE[1,5]]

BTES11 30-Jan-14

Why reload: new

Date:

Time:

Startup option: go

[Rebuilding the system search list from the HOM blocks]

[Rebuilding the active swapping list from the HOM blocks]

[Rebuilding the system dump list from the HOM blocks]

BTES11 15:05:25 CTY system 4096                           

.

.

.

Hopefully the machine will boot up and you will get the OPR> prompt. Now you can use NCP commands to check the network status (this is the result in my "production" machine):

OPR>enter ncp

NCP>show exec char

NCP>

14:07:10        NCP

Request # 3; Show Executor Node Characteristics Completed

Executor Node = 7.80 (BITXT1)

  Identification = DECnet-10 Version 4.0

  Management Version = 4.0.0

  Loop Count = 1

  Loop Length = 127

  Loop With = Mixed

  Incoming Timer = 30

  Outgoing Timer = 60

  NSP Version = 4.0.0

  Maximum Links = 65535

  Delay Factor = 48

  Delay Weight = 10

  Inactivity Timer = 120

  Retransmit Factor = 10

  Routing Version = 2.0.0

  Type = Routing IV

  Routing Timer = 600

  Broadcast Routing Timer = 40

  Maximum Address = 1023

  Maximum Circuits = 20

  Maximum Cost = 100

  Maximum Hops = 16

  Maximum Visits = 20

  Maximum Broadcast Nonrouters = 64

  Maximum Broadcast Routers = 32

  Maximum Buffers = 80

  Buffer Size = 576

  Segment Buffer Size = 576

NCP>show known circuits

NCP>

14:07:27        NCP

Request # 4; Show Known Circuits Summary Completed

Circuit     State                  Loopback    Adjacent

                                     Name        Node

KDP-0-0     On                                 7.61 (BITXOO)              

If the wiring of the KDP/DUP and the DMC interfaces is right and the VAX has the corresponding circuit (DMC-0) defined and up , you should get an adjacency up in your VAX machine (configured as router) and your network will be up and running:

.network

[ANF10 network: connected to BITXT1(10), located at BITXT1(10), 1 node]

Node    BITXT1  (10)    BITXT1  24-Jan-14

[DECnet network: local node BITXT1, 12 reachable nodes in area 7]

BITXOM  BITXOO  BITXOR  BITXOT  BITXOU  BITXOV  BITXOW  BITXOX  BITXOZ  BITXT0

BITXT1  MBSERV          

.r nft

*dir bitxov::

BITXOV::DUA1:[FAL$SERVER]BASIC-PLUS-2-RSX-V2-5.ZIP.1

BITXOV::DUA1:[FAL$SERVER]BASIC-PLUS-2-RSX-V2-7.ZIP.1

BITXOV::DUA1:[FAL$SERVER]CRDRV.MAC.1

BITXOV::DUA1:[FAL$SERVER]F77-V5-4.ZIP.1

BITXOV::DUA1:[FAL$SERVER]INFO.TXT.15

BITXOV::DUA1:[FAL$SERVER]NETSERVER.LOG.111

BITXOV::DUA1:[FAL$SERVER]TOPS10KLB.ZIP.1

BITXOV::DUA1:[FAL$SERVER]TOPS20PI.ZIP.1

BITXOV::DUA1:[FAL$SERVER]VAX-ULTRIX-4-5-INSTALL.ZIP.1

Total of 9 files            

The performance is good enough for SET HOST, but it is a bit slow for file transfers. You can get "Insufficient resources at remote node" trying to pull files from a VMS node. I have been poking around to fix that, but the lack of knowledge and available documentation has stopped me from progressing on that. The error seems to appear randomly, so it is not a big problem anyway.

And that is all! If you try to do this recipe and come into trouble, feel free to drop a comment and I will do my best to help you to fix it.

Ten over Pi

In a previous post I did a small incursion into the business oriented data processing. The system I used to illustrate an example of a CODASYL database was a simulated PDP-10 running the TOPS-10 operating system hosted in a Raspberry Pi computer.

TOPS-10 was quite a nice operating system for its time. It has some "modern" capabilities associated to mainframe-class computing, including a somehow advanced accounting system, job protection, resource allocation based on user and group, priority based scheduling, operator interface and batch system. Its command language is, anyway, quite basic and its help system is limited.

The successor to TOPS-10 for the PDP-10 architecture was named, in a show of originality, TOPS-20. It was itself based on a previous operating system developed by a private company named TENEX. TOPS-20 supported the execution of TOPS-10 binaries, but was an enormous improvement over its predecessors...And we could say it was also a big improvement over its successors too. Its command processor has an awesome feature, similar to the command completion we find in "modern" shells as bash or ksh. The difference is that the TOPS-20 command processor "knows" about the syntax and options of whatever you are typing and offers you a choice of suitable completions. And that facility can be used from user programs too... Incidentally, those of you who are openVMS users or hobbyists can experience that kind of help in the MULTINET command. Press '?' and you will get a list of options. Press 'ESC' and your current typing will be completed, as long as there are just one possible completion.

In this post we will see how we can set up a simulated TOPS-20 system. And, specifically, we will set it up on a Raspberry Pi computer. So we will have a mainframe class computer within a credit-card sized gizmo...

Simulating a KL10 system

Our good friend simh is not a good option to run TOPS-20. The reason is simh emulates a Decsystem-10, based on the KS-10 CPU. The KS-10 was a scaled down version of the "big" PDP-10 machines. It is posible to run TOPS-20 in a KS-10, but it is limited to the 4.something version. The last TOPS-20 version, the one we will want to run, is 7.1. Nevertheless, the real reason we will not want to use simh is the lack of the NI20 networking interface, necessary to plug the simulated machine into the network, either internet, decnet or both. Fortunately, there is another simulator we can use: Ken Harrenstien's KLH10. 

KLH10 has not been enhanced for some time. The newest version you can get, as far as I know, comes with a full image of TOPS-20, ready to be used. This is the work of Mark Crispin, and can be found also here. Unfortunately, the package contains binaries just for Linux over intel processors. Since we want to use a Raspberry, which is ARM-based, we will have to compile KLH from sources. 

The directory structure we get after expaning the tarball is:

panda-dist -- klh10-2.0h --- doc

                         +-- contrib

                         +-- run

                         +-- src

                         +-- bld  --- fbppc

                                  +-- fbx86

                                  +-- lnx86

                                  +-- lnxarm

                                  +-- lnxppc

                                  +-- nbaxp

                                  +-- nbx86

                                  +-- osfaxp

                                  +-- solsparc

We must start the compilation from the appropiate bld subdirectory. For the pi we could use the lnxarm one, but in my experience the setup in that directory does not work very well. What I did was to create another subdirectory (which I called lnxarmpi) and to copy everything from lnx86. Then, open the Makefile in your editor of choice and edit the stanza:

CENVFLAGS = -DCENV_CPU_I386=1 -DCENV_SYS_LINUX=1 \
               -DKLH10_DEV_LITES=1 \
               -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 

So it reads:

CENVFLAGS = -DCENV_CPU_ARM=1 -DCENV_SYS_LINUX=1 \
            -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 

That is, change CENV_CPU_I386 to CENV_CPU_ARM and remove the KLJ10_DEV_LITES definition (there is no parallel port in the Pi).

Before trying to compile KLH10 we must do another change. Move to the src directory and search for the base-kl rule. You can now create a new rule with a different name or simply edit the current one, changing these lines:

     -DKLH10_ITIME_INTRP=1   \
     -DKLH10_CTYIO_INT=1     \

To:

     -DKLH10_ITIME_SYNC=1   \
     -DKLH10_CTYIO_INT=0    \ 

This magic spell comes from this alt.sys.pdp10 post.The reason is "high performance" computers wreck the asynchronous mechanisms of KLH10. Of course, KLH10 was written during the late 90's so almost every current computer is a "high performance" platform. Even the humble Rapsberry Pi. Oh, and aparently there is no fix for that problem (I guess the whole simulator would have to be reengineered). I tried changing some timer related constants, just as a wild try, and did not succeed.

Once you have done this, change to the bld/lnxarmpi directory and issue the "make base-kl" command (unless you created a new rule; in that case use that new rule name). And then go for a coffee. Or two. The Raspberry Pi is not a fast machine!

Once the compilation is over, just follow the instructions you will find in the panda distribution package to do the initial run and setup of TOPS-20. If everything goes well you will have a telnet-accessible TOPS-20 system, connected to the internet (or, at least, to your local network). The system has a bunch of goodies: FORTRAN, COBOL, PASCAL and other compilers. Several games (text based, of course), including the original ADVENTURE (type "run advent" to play), colosal cave, a VT version of STARTREK and some more.

To learn about the system itself, nothing better than the original user manual. You can find a webified copy at http://tilt.twenex.org/ The original manuals are also available in scanned format at http://bitsavers.trailing-edge.com/pdf/dec/pdp10/TOPS20/. If you want to get deeper into TOPS-20 you will need at least the administration manuals.

Finishing touch

Having an internet connected TOPS-20 emulated system is cool, but it is even cooler to enable the "native" networking of those machines: DECNET. To do it just add the following lines at the end of the SYSTEM:7-1-CONFIG.CMD file:

DECNET MAXIMUM-ADDRESS 1023

NODE NODNAM A.NNNN

DECNET ROUTER-ENDNODE

ETHERNET 0 DECNET

Use your own nodename instead of NODNAM and your area and address instead of A.NNNN. Reboot your system and DECNET should come up. Remember to set up adequately the network interface (check the KLH10 documentation about setting up a DECNET capable configuration). Now, to populate your DECNET databases, you must use the SETNOD program:

@RUN SETNOD

SETNOD> SET NODE N.MMM NAME NODE1

SETNOD> SET NODE X.YYYY NAME NODE2

(...)

SETNOD>SAVE FILENAME

SETNOD>INSERT

The SAVE command saves the entered nodes into a binary file. The INSERT command updates the live tables with the defined nodes (something similar to the SET KNOWN NODES ALL command in VMS NCP). To reload the node database at startup, add the following lines to SYSTEM:7-1-SYSJOB.RUN, just after the ^ESET LOGIN ANY:

RUN SYS:SETNOD

GET FILENAME

INSERT

EXIT

Replace FILENAME by the name you used when you saved the node database from SETNOD.

You must also uncomment three lines from the file SYSTEM:SYSTEM:CMD


WAIT 5
NCP SET CIRCUIT NI-0-0 SERVICE ENABLED
NCP SET CIRCUIT NI-0-0 STATE ON       

And this is all for now. As always, if you want to try any of these recipes and have any kind of trouble, just drop a commend and I will try to help..

Post-scriptum.

This post has been in draft status for a while. I begun to wrote it just after the one about TOPS-10. When I was about to publish it, Mr. Mark Crispin, the man behind the PANDA distribution and a lot of other things, passed away. I did not have the pleasure to know him, neither personally nor thru electronic communication, but I have read his comments in the PDP-10 mailing lists and other places. It would be presumptuous to pretend to pay him an homage in this humble blog post, so all what I will say is a very big THANK YOU. We human beings all pass away, but for some admirable people, their work will live forever.

Update

I have just run into this post: http://victor.se/bjorn/its/raspi.php, which explains clearly and succintly the steps to do to run KLH10 in the Raspberry Pi, including a patch for KLH which makes it much easier to network. To apply the tap patch, just download it into the main panda distribution unzip directory and run:

$ patch -p0 < tuntap.patch

And then rebuild KLH10.

Data processing... 70's style

noSQL is the new trend... Seriously?

It is really amazing to see that in a purposedly scientific/engineering discipline as is (or should be) data processing we find some trends that we could name perfectly as "fashion" phenomena. Things that come in full force and dissapear whithout a trace. What is the hot development platform today will be forgotten (and badmouthed) tomorrow. I don't think I need to name an example of any of those. Specially in the field of "modern" java frameworks and environments. And meanwhile, in datacenters split al over the world, we can find lots and lots of lovely handcrafted COBOL and PLI code doing its job silently and faithfully, while a big part of the world economy relies in software build without all this modern fuss about patterns, architectures and... fancy trends...

Well, I guess the previous rant puts me definitely in the Grumpy Old Fart team. Not that I'm really so old (I am "just" 48 years old), but, to say it shortly, I'm really p*ss*d off about the younglings that try to teach me my job, day after day. Damn! I was crunching files when they were wearing diapers! :) Most of those guys are unable to code a master-transaction process program without using 2 gigabytes of framework code, a lot of XML descriptors, a fancy GUI and a Nespresso machine! Well, forget about the Nespresso, I use one of those myself :). And, to be honest, I amb not against the modern frameworks which save a lot of time and a lot of boilerplate code... I can write in java and I consider myself quite good at it, and I do know about some of those modern frameworks and I actually find some of those really briliant and smart. But, some times, the old fart in me has to rebel and yell out. Enter noSql, the last trend in database management.

Aparently some smart guys have noticed that the SQL layer adds a (sometimes) unnecessary overhead to a data processing application. And that the integrity safeguards imposed by the SQL databases (not a good choice of words... relational databases would be more precise) impair the scalability when an application reaches the multi-terabyte level. Then those smart guys have the really briliant thought of getting rid not only of the SQL layer, of but the relational model itself, and go to a pure, non-constrained key-value pair model...

At that time my semi-obsolete neurons start to send signals to my visual cortex, and I almost can see, in green over black letters the words ORGANIZATION IS INDEXED, ACCESS IS RANDOM. It looks like the smart java guys have rediscovered the indexed files! Of course, they are not going to call their amazing discovey "indexed file". It sounds too mainframish. They will use fancy words and fancy names to call that old, purposedly concept of tying a record... Ooops... I mean... an OBJECT to a key.

Now I will try a prescience exercise. I predict some of those brilliant guys will realize sometime in the next years that all those nice objects described by key-value parts have... oh, I'm feeling myself smart now... RELATIONSHIPS between them. And that those relationships can be modeled and integrated into the data storage. Now that is a revolutionary concept :) The only problem is that this revolution happened 40 years ago. Before the Codd model went mainstream, there were at least another two database models. And, what is even more revolutionary, those database models are still widely used today...

Pre-relational database models: hyerarchies and networks

You could build any information system just using plain old sequential files... but you don't want to do it. You could also do it using pure indexed files, and for very simple data models perhaps it could be a good choice. But if your data model gets complex (let's say... three or more files) your data management code will explode in a burst of tedious file and record management routines. The Database Management Sytems (DBMS) were designed to help with this complexity. At a very basic level, a DBMS will help the programer to:

• Coordinate the changes between several data sets (record types), ensuring the consitence between them.
• Isolate the programmer from the physical design of the data storage. 
• Protect the information, providing utilities to back up and restore the information to a consistent state.
• Providing the developers with views of the data restricted to their needs.

The first available DBMS models responded to two models:

• A hyerarchical model, in which the different record types are related via "parent-child" relationships (for instance, a machine to its parts).
• A network model, in which the different record types can be related via arbitrary relationships, not necessarily hyerarchical.

The network model can be seen as a generalization of the hyerarchical model, and the hyerarchical model can be seen as a restriction of the network one.

Take my data to the moon

The most known implementation of the hyerarchical model is  IBM IMS/DB. IMS is not just a database system, is also a transaction processing monitor. It runs in IBM zSeries mainframes, and it is widely used today in the banking and insurance industries, as well as in government related information management. It is insanely powerful, and eats heavy transactions like a kid devours candy. I would love to introduce my readers to IMS, but unfortunately there is no legal way to run IMS at home unless you have an indecent amount of cash. And, of course, IMS design and programming is quite ugly... That does not mean it is not fun. Actually, the restrictions of the hyerarchical model forces the designer to think twice (and trice!) about the design, and forces the programmer to get a deep" knowledge about the problem domain he is working in. Oh, and it is noSQL :). The basic method to access IMS data is to issue a 'GU' call, which returns the record ("segment" in IMS tongue) which satisfies a "SSA" (Segment Search Argument), which is basically a key search expression. So IMS gives you values associated to keys. Key-value parts... with the add-on of chains of "child" segments physically attached to the main (or "root") segments. Oh, by the way, the origin of IMS was the need to keep track of the parts of one of the most complex machines built by the mankind: the Apollo spacecraft. So, each time you use an ATM to get some cash you are probably using a piece of technology born to help to put the astronauts in the moon!

Untangle the network

The network model allows more flexibility than the hyerarchical one. Actually, it could look similar to the more used and known relational model. The main difference is in the relational mode there are (mostly) no physical relationships between record types (relations or tables); the relationships are built at run time using joins based on foreign keys. In a network database, the relationships do exist in the database, usually as physical pointers which relate the different record types. 

The network model databases were standarized at the late sixties and the first seventies by the CODASYL comitee. The network databases are also known as CODASYL databases, and that is the name we are going to use from now.

A CODASYL database is described by a SCHEMA definition. The schema definition contains:

• The physical characteristics of the database, like the devices it uses, the areas (or parts) it has, the size of those areas in pages, and the size of the pages in bytes or words.
• The different record types present in the database. For each record type the designer specifies the details about its physical placement, the retrieval and location methods (direct, by physical address, or using hashed keys) and the record structure (the composition of each record in fields). In contrast to the more formal relational model, a CODASYL record can contain arrays and "unstructured" byte bags.
• The relationships between records, named "sets" in the CODASYL nomenclature. For each set, the designer specifies the "owner" and the "members". The owner and the members are joined using pointers, and the designer has some degree of control about which pointers to use. The most basic structure has a pointer in the "owner" to the first "member", and pointers relating each member to the one following it in a linked list. That linked list can be enhanced using backwards pointers (making it a double linked list) and pointers to the owner.
• One or more "subschemas", which are subsets of the whole schema that can be used by the application programs. The designer/administrator has some (weak) tools to restrict the subschemas visible to the programmers; this allows him to hide the salary data of a personnel database to the guys doing work not related to payroll. I think you get the idea.

The best known CODASYL database is probably IDMS, owned by Computer Associates. It is still used nowadays. We cannot use IDMS legally (as far as I know), but we can use one of its derivatives. DEC licensed IDMS for use in its PDP-10 mainframes, and sold it as DBMS-10 and DBMS-20. And., guess what? We can run those under SIMH!

A little bit of time travel

We will need some material to set up our retro-database management experiment:

• A copy of simh, specifically of the pdp10 simulator.
• The TOPS-10 installation tapes. You will need (at least) these tapes:
• BB-X138C-BB (Boot tape)
• BB-X128C-SB (CUSP 1/2)
• BB-FP63B-SB (CUSP 2/2)
• BB-FP64B-SB (TOOLS)
• BB-X130C-SB (Cust Supported)
• OR, if you don't want to experience the fun of building your own TOPS-10 system:
• A pre-built TOPS-10 system.
• The COBOL compiler tape COBOL 74 V12C (unless you have used the pre-built system)
• The DBMS-10 tape DBMS V5A

If you are like me and decided to do the full install by yourself you will find how to do it here. You will not find how to install COBOL and DBMS there, but you can look at the BASIC and FORTRAN instructions. The TOPS-10 installations are basically manual: you restore a save set to a working directory and then move the files by hand to the SYS: and HLP: directories. Piece of cake! Oh, I recommend to install EDT unless you want to add the need to learn an editor to your pdp10 adventure. You will find EDT and other goodies in this tape.

Once you have the TOPS-10 system up and running with a working COBOL compiler and an installed DBMS-10 database you will need to do some magic to add DBMS support to COBOL. Just follow the docs here and here. It is not really complex. Basically, restore the full DBMS tape into a working directory, copy C74LIB.REL and C74SHR.REL to this directory and submit a file to rebuild DBMS. When it is done, copy back both files AND C74O12.EXE to SYS: and you will be ready. If you got stuck, feel free to post a comment and I will try to help.

The real stuff: managing your LP (not CDs please) 70's style.

I have written some code as an example of what could be a classic data management application. You can find and download the code from this github repositotry. The files you will find are:

• The schema file, RECRDS.DDL, which defines a database with two record types and one set:
• LP-RECORD holds information about LPs (yes, those big circular black pieces of vinyl)
• TRACK-RECORD holds information about the tracks of a record.
• LP-SET relates a set of tracks to a record
• REC001.CBL, a COBOL program to load the database from a flat, sequential file
• REC002.CBL, a COBOL program to mantain the database using transactions from a sequential file
• REC003.CBL, a COBOL program to empty the database.
• LP.CBL, TRACK.CBL and TRANS.CBL: "copybook" files with the input records.
• COMP.MIC, a comand procedure to compile and link the above programs.
• RECA01.DAT, a sample input file for REC001.
• RECA02.DAT, a sample input file for REC002.

By the way, the user executing these code must have the ENQ-DEQ privilege. You must use REACT to add it, or just use the OPERATOR account (which, in a real production system would be anathema, of course).

Oh, remember this is COBOL-74. That means there are no scoped statements. No END-IF. No END-PERFORM. No inline PERFORM. You get the idea...

Last details

If you want to try to build the programs you will need to create a COBOL text library using LP.CBL, TRACK.CBL and TRANS.CBL. This is the recipe:

. R LIBARY

*LIBARY=LIBARY

*INSERT LP,LP.CBL

*INSERT TRACK,TRACK.CBL

*INSERT TRANS,TRANS.CBL

*END

*^C

You will probably want to read the docs:

• COBOL-74 manual
• DBMS-10 Administrator manual
• DBMS-10 Programming manual

And this is all by now. This is a different post, leaving the system level discussion for a while. Enjoy data processing, oldies style!