Ten over Pi

In a previous post I did a small incursion into the business oriented data processing. The system I used to illustrate an example of a CODASYL database was a simulated PDP-10 running the TOPS-10 operating system hosted in a Raspberry Pi computer.

TOPS-10 was quite a nice operating system for its time. It has some "modern" capabilities associated to mainframe-class computing, including a somehow advanced accounting system, job protection, resource allocation based on user and group, priority based scheduling, operator interface and batch system. Its command language is, anyway, quite basic and its help system is limited.

The successor to TOPS-10 for the PDP-10 architecture was named, in a show of originality, TOPS-20. It was itself based on a previous operating system developed by a private company named TENEX. TOPS-20 supported the execution of TOPS-10 binaries, but was an enormous improvement over its predecessors...And we could say it was also a big improvement over its successors too. Its command processor has an awesome feature, similar to the command completion we find in "modern" shells as bash or ksh. The difference is that the TOPS-20 command processor "knows" about the syntax and options of whatever you are typing and offers you a choice of suitable completions. And that facility can be used from user programs too... Incidentally, those of you who are openVMS users or hobbyists can experience that kind of help in the MULTINET command. Press '?' and you will get a list of options. Press 'ESC' and your current typing will be completed, as long as there are just one possible completion.

In this post we will see how we can set up a simulated TOPS-20 system. And, specifically, we will set it up on a Raspberry Pi computer. So we will have a mainframe class computer within a credit-card sized gizmo...

Simulating a KL10 system

Our good friend simh is not a good option to run TOPS-20. The reason is simh emulates a Decsystem-10, based on the KS-10 CPU. The KS-10 was a scaled down version of the "big" PDP-10 machines. It is posible to run TOPS-20 in a KS-10, but it is limited to the 4.something version. The last TOPS-20 version, the one we will want to run, is 7.1. Nevertheless, the real reason we will not want to use simh is the lack of the NI20 networking interface, necessary to plug the simulated machine into the network, either internet, decnet or both. Fortunately, there is another simulator we can use: Ken Harrenstien's KLH10. 

KLH10 has not been enhanced for some time. The newest version you can get, as far as I know, comes with a full image of TOPS-20, ready to be used. This is the work of Mark Crispin, and can be found also here. Unfortunately, the package contains binaries just for Linux over intel processors. Since we want to use a Raspberry, which is ARM-based, we will have to compile KLH from sources. 

The directory structure we get after expaning the tarball is:

panda-dist -- klh10-2.0h --- doc

                         +-- contrib

                         +-- run

                         +-- src

                         +-- bld  --- fbppc

                                  +-- fbx86

                                  +-- lnx86

                                  +-- lnxarm

                                  +-- lnxppc

                                  +-- nbaxp

                                  +-- nbx86

                                  +-- osfaxp

                                  +-- solsparc

We must start the compilation from the appropiate bld subdirectory. For the pi we could use the lnxarm one, but in my experience the setup in that directory does not work very well. What I did was to create another subdirectory (which I called lnxarmpi) and to copy everything from lnx86. Then, open the Makefile in your editor of choice and edit the stanza:

CENVFLAGS = -DCENV_CPU_I386=1 -DCENV_SYS_LINUX=1 \
               -DKLH10_DEV_LITES=1 \
               -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 

So it reads:

CENVFLAGS = -DCENV_CPU_ARM=1 -DCENV_SYS_LINUX=1 \
            -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE 

That is, change CENV_CPU_I386 to CENV_CPU_ARM and remove the KLJ10_DEV_LITES definition (there is no parallel port in the Pi).

Before trying to compile KLH10 we must do another change. Move to the src directory and search for the base-kl rule. You can now create a new rule with a different name or simply edit the current one, changing these lines:

     -DKLH10_ITIME_INTRP=1   \
     -DKLH10_CTYIO_INT=1     \

To:

     -DKLH10_ITIME_SYNC=1   \
     -DKLH10_CTYIO_INT=0    \ 

This magic spell comes from this alt.sys.pdp10 post.The reason is "high performance" computers wreck the asynchronous mechanisms of KLH10. Of course, KLH10 was written during the late 90's so almost every current computer is a "high performance" platform. Even the humble Rapsberry Pi. Oh, and aparently there is no fix for that problem (I guess the whole simulator would have to be reengineered). I tried changing some timer related constants, just as a wild try, and did not succeed.

Once you have done this, change to the bld/lnxarmpi directory and issue the "make base-kl" command (unless you created a new rule; in that case use that new rule name). And then go for a coffee. Or two. The Raspberry Pi is not a fast machine!

Once the compilation is over, just follow the instructions you will find in the panda distribution package to do the initial run and setup of TOPS-20. If everything goes well you will have a telnet-accessible TOPS-20 system, connected to the internet (or, at least, to your local network). The system has a bunch of goodies: FORTRAN, COBOL, PASCAL and other compilers. Several games (text based, of course), including the original ADVENTURE (type "run advent" to play), colosal cave, a VT version of STARTREK and some more.

To learn about the system itself, nothing better than the original user manual. You can find a webified copy at http://tilt.twenex.org/ The original manuals are also available in scanned format at http://bitsavers.trailing-edge.com/pdf/dec/pdp10/TOPS20/. If you want to get deeper into TOPS-20 you will need at least the administration manuals.

Finishing touch

Having an internet connected TOPS-20 emulated system is cool, but it is even cooler to enable the "native" networking of those machines: DECNET. To do it just add the following lines at the end of the SYSTEM:7-1-CONFIG.CMD file:

DECNET MAXIMUM-ADDRESS 1023

NODE NODNAM A.NNNN

DECNET ROUTER-ENDNODE

ETHERNET 0 DECNET

Use your own nodename instead of NODNAM and your area and address instead of A.NNNN. Reboot your system and DECNET should come up. Remember to set up adequately the network interface (check the KLH10 documentation about setting up a DECNET capable configuration). Now, to populate your DECNET databases, you must use the SETNOD program:

@RUN SETNOD

SETNOD> SET NODE N.MMM NAME NODE1

SETNOD> SET NODE X.YYYY NAME NODE2

(...)

SETNOD>SAVE FILENAME

SETNOD>INSERT

The SAVE command saves the entered nodes into a binary file. The INSERT command updates the live tables with the defined nodes (something similar to the SET KNOWN NODES ALL command in VMS NCP). To reload the node database at startup, add the following lines to SYSTEM:7-1-SYSJOB.RUN, just after the ^ESET LOGIN ANY:

RUN SYS:SETNOD

GET FILENAME

INSERT

EXIT

Replace FILENAME by the name you used when you saved the node database from SETNOD.

You must also uncomment three lines from the file SYSTEM:SYSTEM:CMD


WAIT 5
NCP SET CIRCUIT NI-0-0 SERVICE ENABLED
NCP SET CIRCUIT NI-0-0 STATE ON       

And this is all for now. As always, if you want to try any of these recipes and have any kind of trouble, just drop a commend and I will try to help..

Post-scriptum.

This post has been in draft status for a while. I begun to wrote it just after the one about TOPS-10. When I was about to publish it, Mr. Mark Crispin, the man behind the PANDA distribution and a lot of other things, passed away. I did not have the pleasure to know him, neither personally nor thru electronic communication, but I have read his comments in the PDP-10 mailing lists and other places. It would be presumptuous to pretend to pay him an homage in this humble blog post, so all what I will say is a very big THANK YOU. We human beings all pass away, but for some admirable people, their work will live forever.

Update

I have just run into this post: http://victor.se/bjorn/its/raspi.php, which explains clearly and succintly the steps to do to run KLH10 in the Raspberry Pi, including a patch for KLH which makes it much easier to network. To apply the tap patch, just download it into the main panda distribution unzip directory and run:

$ patch -p0 < tuntap.patch

And then rebuild KLH10.

Blinkenlights: Not so difficult as I thought

I have to make a confession: I am a very, very clumsy person. That is one of the reasons I work in the software side of Computer Science. It is guaranteed I will break anything I try to repair or to fix. It is quite a miracle my VAXen  are still alive. I even had to replace a fan of my MicroVAX 3300. I made a... patchwork, and it works.

So, when I begun my little blinkenlights project I did not have a lot of faith about my manual abilities. Fortunately, electronics prototyping has advanced a lot and now it is posible to build circuit prototypes without having to struggle with a soldering iron. I will eventually try to do some soldering, but right now it is all about solderless breadboards.

And I been able to get some blinking leds :). I won't post again a video... just check the previous entry!

Goals and basic design

My goals are quite modest: I wanted just to have the equivalent to the display data lights one can find in a "classic" PDP-11 console, like the one in the PDP-11/70. That console has more lights than those: a 22 lights address display and several status and error reporting indicators. My "console" will have just 18 LEDS: 16 leds for the 16 bits of a PDP-11 word, plus a parity indicator and a generic error light.

The data to be displayed will be selected depositing values into three "registers" in SIMH, belonging to a new device which I have called "BLNK":

• DATASEL will represent the "Data select" key in the real console. The posible values are:
• 0 for Register
• 1 for Data Path
• 2 for Bus Register
• 3 for micro Address
• DISPSEL will represent the "Display select" key. The values are:
• 0 for Console Physical
• 1 for User - Instruction
• 2 for User - Data
• 3 for Super - Instruction
• 4 for Super - Data
• 5 for Kernel - Instruction
• 6 for Kernel - Data
• 7 for Program Physical
• DISPADD will hold the displayed value.

The default values will be zero for all three registers, so the "console" will show the value for R0. The light chasers implemented in several PDP-11 operating systems rotate and shift bit patterns in R0 which are shown in the lights, so it seems like a decent default setting.

Functional design and implementation

I have not done any simh hacking before, so one of my goals was to go the easy way. I implemented the console lights as a new device, so I didn't had to patch neither the scp (the control program for all the simh emulators) nor the core pdp-11 implementation. All the simh part functionality is in two new files (one of them being a header) and a small, controlled modification to pdp11_sys.c just to add the new device. This way it will be easy to merge my patches with future versions of simh. If you want to check it you can download from https://github.com/jguillaumes/retroutils/blob/master/simh-patches/pdp11-blinken.patch. This patch is for the current development version (4.0-beta), but it should apply to 3.9 too (I have not tried that).

If you want to build simh with this patch you will need too the code which actually implements the blinking lights. My implementation has two parts:

• A server part, which holds the real (or virtual) lights to be shown. This server gets the bits to represent via UDP packets or via serial connection. 
• A client library, which sends the information to display to the server, either via UDP or serial connection.

I have written three different servers. Two of them are real, and the other one is virtual. 

• The virtual one is written in java, and can be found at https://github.com/jguillaumes/retroutils/tree/master/JBlinkenServer. This is a netbeans project, but it should build without the IDE, just running ant. 

• I wrote also a Raspberry Pi version which gets the data listening to UDP packets. 
• And finally, I made an Arduino controlled version, which gets the data via serial port.

The rasperry and the arduino versions use basically the same hardware; the only difference is in the resistors used to limit the current going thru the leds. The Arduino operates at 5V, while the Raspberry uses 3.3V (it has also a 5V current source pin, but it is not wise to mix both tension levels). The resistors for the Rasp-pi version are of 330 Ohms, while for the Arduino I use 470 ohms. Oh, the circuit is basically the one described in this tutorial, with a minor modification.

The code for the client library and the servers is also accessible here https://github.com/jguillaumes/retroutils/tree/master/BlinkenServer. The arduino sketch is also there. To build simh with the client code you have to build the client library and install it (the makefile installs it in /usr/local, edit it if you want to use a different location). Then build the pdp11 simulator using the patched makefile.

Usage

Once you have built the pdp11 simulator, you will be able to use the new BLNK device (which is disabled by default). Enable it with:

sim> SET BLNK ENABLE

And attach it to a running udp server using:

sim> ATTACH BLNK udp:ip-address:port

If you are running the java application, it should be:

sim> ATTACH BLNK udp:localhost:11696

If you build the arduino circuit, then the attach command is:

sim> ATTACH BLNK tty:/dev/name-of-your-arduino-serial-usb

The only modifier is RATE. The command

sim> SET BLNK RATE=50

will try to establish an update refresh rate of 50 Hz (the leds will be updated 50 times at a second). The default value is 100. Don't push it too hard if you are using an Arduino, since the poor thing will not be able to cope with the load and will hang up...

If you attach BLNK and boot RSX-11M+ you will enjoy a nice light show. 

As usual, your comments and suggestions will be very welcome!

UPDATE

I have made some corrections to the simh part and to the arduino side. I have created a simh repositoy at github, accessible via https://github.com/jguillaumes/simh. The master branch of that repository will follow the "official" simh tree (https://github.com/simh/simh). The branch "BlinkenLights" will contain my modified version (https://github.com/jguillaumes/simh/tree/BlinkenLights). I will try to keep it in sync with the master, so it should build without too much trouble. I don't plan to ask Mark to merge my changes, since there is the possibility of developing an "official" blinkenlights support into simh (which would make mine obsolete).

The current version works more closely to the original hardware. So to watch a CPU register you have to load into the address register the correct "pseudoaddress" (R0 is at 017777700, R1 at 017777701 and so on). The "bus" and "micro" settings do not work (they just lite the red LED). But I was able to use the blinky thing to debug some work I did in the CR (card driver) emulation in simh :).

Let there be... blinkenlight!

This is a short post. I have been playing a little bit in the hardware side. Any serious computer MUST have blinkenlights, so I am adding some to my simh-simulated PDP-11s:

I have to work out the timing and the sampling rate, and I have not finished yet the integration with simh. But it works (somehow), and the lights blink (a litle bit). It is a distributed "blinkenlight server" which gets the data to show via UDP packets, although I plan to develop also a "local" version for instances of the simulator running in the same Ras-Pi. And at this moment, the address of the "server" and the item to use to make the lights move are hardcoded into a simh device. :)

I will publish the code (of both the server and the client side) and the schematics of the electronic part when they are presentable. But I'm very excited with the result and I wanted to share what I have got ;)

Oh, by the way, happy  new year to everyone!

Data processing... 70's style

noSQL is the new trend... Seriously?

It is really amazing to see that in a purposedly scientific/engineering discipline as is (or should be) data processing we find some trends that we could name perfectly as "fashion" phenomena. Things that come in full force and dissapear whithout a trace. What is the hot development platform today will be forgotten (and badmouthed) tomorrow. I don't think I need to name an example of any of those. Specially in the field of "modern" java frameworks and environments. And meanwhile, in datacenters split al over the world, we can find lots and lots of lovely handcrafted COBOL and PLI code doing its job silently and faithfully, while a big part of the world economy relies in software build without all this modern fuss about patterns, architectures and... fancy trends...

Well, I guess the previous rant puts me definitely in the Grumpy Old Fart team. Not that I'm really so old (I am "just" 48 years old), but, to say it shortly, I'm really p*ss*d off about the younglings that try to teach me my job, day after day. Damn! I was crunching files when they were wearing diapers! :) Most of those guys are unable to code a master-transaction process program without using 2 gigabytes of framework code, a lot of XML descriptors, a fancy GUI and a Nespresso machine! Well, forget about the Nespresso, I use one of those myself :). And, to be honest, I amb not against the modern frameworks which save a lot of time and a lot of boilerplate code... I can write in java and I consider myself quite good at it, and I do know about some of those modern frameworks and I actually find some of those really briliant and smart. But, some times, the old fart in me has to rebel and yell out. Enter noSql, the last trend in database management.

Aparently some smart guys have noticed that the SQL layer adds a (sometimes) unnecessary overhead to a data processing application. And that the integrity safeguards imposed by the SQL databases (not a good choice of words... relational databases would be more precise) impair the scalability when an application reaches the multi-terabyte level. Then those smart guys have the really briliant thought of getting rid not only of the SQL layer, of but the relational model itself, and go to a pure, non-constrained key-value pair model...

At that time my semi-obsolete neurons start to send signals to my visual cortex, and I almost can see, in green over black letters the words ORGANIZATION IS INDEXED, ACCESS IS RANDOM. It looks like the smart java guys have rediscovered the indexed files! Of course, they are not going to call their amazing discovey "indexed file". It sounds too mainframish. They will use fancy words and fancy names to call that old, purposedly concept of tying a record... Ooops... I mean... an OBJECT to a key.

Now I will try a prescience exercise. I predict some of those brilliant guys will realize sometime in the next years that all those nice objects described by key-value parts have... oh, I'm feeling myself smart now... RELATIONSHIPS between them. And that those relationships can be modeled and integrated into the data storage. Now that is a revolutionary concept :) The only problem is that this revolution happened 40 years ago. Before the Codd model went mainstream, there were at least another two database models. And, what is even more revolutionary, those database models are still widely used today...

Pre-relational database models: hyerarchies and networks

You could build any information system just using plain old sequential files... but you don't want to do it. You could also do it using pure indexed files, and for very simple data models perhaps it could be a good choice. But if your data model gets complex (let's say... three or more files) your data management code will explode in a burst of tedious file and record management routines. The Database Management Sytems (DBMS) were designed to help with this complexity. At a very basic level, a DBMS will help the programer to:

• Coordinate the changes between several data sets (record types), ensuring the consitence between them.
• Isolate the programmer from the physical design of the data storage. 
• Protect the information, providing utilities to back up and restore the information to a consistent state.
• Providing the developers with views of the data restricted to their needs.

The first available DBMS models responded to two models:

• A hyerarchical model, in which the different record types are related via "parent-child" relationships (for instance, a machine to its parts).
• A network model, in which the different record types can be related via arbitrary relationships, not necessarily hyerarchical.

The network model can be seen as a generalization of the hyerarchical model, and the hyerarchical model can be seen as a restriction of the network one.

Take my data to the moon

The most known implementation of the hyerarchical model is  IBM IMS/DB. IMS is not just a database system, is also a transaction processing monitor. It runs in IBM zSeries mainframes, and it is widely used today in the banking and insurance industries, as well as in government related information management. It is insanely powerful, and eats heavy transactions like a kid devours candy. I would love to introduce my readers to IMS, but unfortunately there is no legal way to run IMS at home unless you have an indecent amount of cash. And, of course, IMS design and programming is quite ugly... That does not mean it is not fun. Actually, the restrictions of the hyerarchical model forces the designer to think twice (and trice!) about the design, and forces the programmer to get a deep" knowledge about the problem domain he is working in. Oh, and it is noSQL :). The basic method to access IMS data is to issue a 'GU' call, which returns the record ("segment" in IMS tongue) which satisfies a "SSA" (Segment Search Argument), which is basically a key search expression. So IMS gives you values associated to keys. Key-value parts... with the add-on of chains of "child" segments physically attached to the main (or "root") segments. Oh, by the way, the origin of IMS was the need to keep track of the parts of one of the most complex machines built by the mankind: the Apollo spacecraft. So, each time you use an ATM to get some cash you are probably using a piece of technology born to help to put the astronauts in the moon!

Untangle the network

The network model allows more flexibility than the hyerarchical one. Actually, it could look similar to the more used and known relational model. The main difference is in the relational mode there are (mostly) no physical relationships between record types (relations or tables); the relationships are built at run time using joins based on foreign keys. In a network database, the relationships do exist in the database, usually as physical pointers which relate the different record types. 

The network model databases were standarized at the late sixties and the first seventies by the CODASYL comitee. The network databases are also known as CODASYL databases, and that is the name we are going to use from now.

A CODASYL database is described by a SCHEMA definition. The schema definition contains:

• The physical characteristics of the database, like the devices it uses, the areas (or parts) it has, the size of those areas in pages, and the size of the pages in bytes or words.
• The different record types present in the database. For each record type the designer specifies the details about its physical placement, the retrieval and location methods (direct, by physical address, or using hashed keys) and the record structure (the composition of each record in fields). In contrast to the more formal relational model, a CODASYL record can contain arrays and "unstructured" byte bags.
• The relationships between records, named "sets" in the CODASYL nomenclature. For each set, the designer specifies the "owner" and the "members". The owner and the members are joined using pointers, and the designer has some degree of control about which pointers to use. The most basic structure has a pointer in the "owner" to the first "member", and pointers relating each member to the one following it in a linked list. That linked list can be enhanced using backwards pointers (making it a double linked list) and pointers to the owner.
• One or more "subschemas", which are subsets of the whole schema that can be used by the application programs. The designer/administrator has some (weak) tools to restrict the subschemas visible to the programmers; this allows him to hide the salary data of a personnel database to the guys doing work not related to payroll. I think you get the idea.

The best known CODASYL database is probably IDMS, owned by Computer Associates. It is still used nowadays. We cannot use IDMS legally (as far as I know), but we can use one of its derivatives. DEC licensed IDMS for use in its PDP-10 mainframes, and sold it as DBMS-10 and DBMS-20. And., guess what? We can run those under SIMH!

A little bit of time travel

We will need some material to set up our retro-database management experiment:

• A copy of simh, specifically of the pdp10 simulator.
• The TOPS-10 installation tapes. You will need (at least) these tapes:
• BB-X138C-BB (Boot tape)
• BB-X128C-SB (CUSP 1/2)
• BB-FP63B-SB (CUSP 2/2)
• BB-FP64B-SB (TOOLS)
• BB-X130C-SB (Cust Supported)
• OR, if you don't want to experience the fun of building your own TOPS-10 system:
• A pre-built TOPS-10 system.
• The COBOL compiler tape COBOL 74 V12C (unless you have used the pre-built system)
• The DBMS-10 tape DBMS V5A

If you are like me and decided to do the full install by yourself you will find how to do it here. You will not find how to install COBOL and DBMS there, but you can look at the BASIC and FORTRAN instructions. The TOPS-10 installations are basically manual: you restore a save set to a working directory and then move the files by hand to the SYS: and HLP: directories. Piece of cake! Oh, I recommend to install EDT unless you want to add the need to learn an editor to your pdp10 adventure. You will find EDT and other goodies in this tape.

Once you have the TOPS-10 system up and running with a working COBOL compiler and an installed DBMS-10 database you will need to do some magic to add DBMS support to COBOL. Just follow the docs here and here. It is not really complex. Basically, restore the full DBMS tape into a working directory, copy C74LIB.REL and C74SHR.REL to this directory and submit a file to rebuild DBMS. When it is done, copy back both files AND C74O12.EXE to SYS: and you will be ready. If you got stuck, feel free to post a comment and I will try to help.

The real stuff: managing your LP (not CDs please) 70's style.

I have written some code as an example of what could be a classic data management application. You can find and download the code from this github repositotry. The files you will find are:

• The schema file, RECRDS.DDL, which defines a database with two record types and one set:
• LP-RECORD holds information about LPs (yes, those big circular black pieces of vinyl)
• TRACK-RECORD holds information about the tracks of a record.
• LP-SET relates a set of tracks to a record
• REC001.CBL, a COBOL program to load the database from a flat, sequential file
• REC002.CBL, a COBOL program to mantain the database using transactions from a sequential file
• REC003.CBL, a COBOL program to empty the database.
• LP.CBL, TRACK.CBL and TRANS.CBL: "copybook" files with the input records.
• COMP.MIC, a comand procedure to compile and link the above programs.
• RECA01.DAT, a sample input file for REC001.
• RECA02.DAT, a sample input file for REC002.

By the way, the user executing these code must have the ENQ-DEQ privilege. You must use REACT to add it, or just use the OPERATOR account (which, in a real production system would be anathema, of course).

Oh, remember this is COBOL-74. That means there are no scoped statements. No END-IF. No END-PERFORM. No inline PERFORM. You get the idea...

Last details

If you want to try to build the programs you will need to create a COBOL text library using LP.CBL, TRACK.CBL and TRANS.CBL. This is the recipe:

. R LIBARY

*LIBARY=LIBARY

*INSERT LP,LP.CBL

*INSERT TRACK,TRACK.CBL

*INSERT TRANS,TRANS.CBL

*END

*^C

You will probably want to read the docs:

• COBOL-74 manual
• DBMS-10 Administrator manual
• DBMS-10 Programming manual

And this is all by now. This is a different post, leaving the system level discussion for a while. Enjoy data processing, oldies style!

Status report

I have experienced a close relative loss in the last days, after some weeks of hospitalization, so I have not worked a lot in my pet projects. So this is going to be a very short post, just to keep this blog alive. I don't like to make online promises, but I plan to restart my work in MUXX and perhaps to post an entry about layered products installation under RSX-11M.

As for MUXX, I have stopped the work after achieving milestone 2. I plan to begin working in milestone 3 in the next days/weeks and the commits to github will resume their usual pace. So no new news about this.

On the simulators front, I have moved one of the emulated VAXen out of the Raspberry Pi. It is the uVAX-3900 simulation running OpenVMS 7.3. The Pi is too cramped to move that simulated machine with an acceptable speed, so it lives again in my old MacBook. Right now I have 5 simulations running in the Raspberry:

• pdp11 running RSX-11M
• pdp11 running RSX-11M+
• pdp11 running RSTS/E
• pdp10 running TOPS-10
• vax780 running VMS 4.7 (and acting as level 2 router for my HECnet area)

There are two simulations running in the macbook:

• pdp10 running TOPS-20 (Panda distribution)
• vax running OpenVMS 7.3

The TOPS-20 system does not run directly inside the macbook (which, in turn, runs Ubuntu 10.04 LTS). I run it inside a VirtualBOX VM which runs a super-light Linux distro. The reason is I was not able to share the ethernet adapter between the host system and the pdp10. Adding a thin layer of virtualization fixed that. Now I have both TCP/IP and DECNET in the pdp10, so life is good.

Oh, by the way. The forthcoming version of simh will come with some nice surprises for us DEC-nostalgics. The current repository adds some "new" VAX machines:

• uVAX I
• uVAX II
• VAX 730
• VAX 750
• Industrial uVAX

This, and a revamped I/O subsystem which will allow the simulators to use asynchronous I/O. Mark and the rest of contributors are doing, as always, a very good job. Thanks a lot, guys!

MUXX: Milestone 2 reached

My toy kernel slowly evolves...

This morning I have considered my little toy PDP-11 kernel has reached the second milestone. This means that right now my creature has the following super-advanced features:

• It can open and close channels to devices (currently just the paper tape puncher and reader and the line printer).
• It can read and write using those channels, in synchronous mode (no context switch when a task blocks for a read... it spins faithfully until the I/O operation is complete).
• It can load tasks from a channel and execute those tasks in a separate, memory protected address space. 

The next goal is to pull some functionality out of the kernel space and move it to different processes. At that point I will be able to say without being ashamed of myself that MUXX has a microkernel architecture... :) And I really must do it, because the current version is just at the limit of the 24KB I defined as kernel code space. So, the next steps will be:

• Define a framework of inter-processes message passing and replying (I will probably inspire myself on minix to do this).
• Decide which kernel functionality I will move to the auxiliary processes without breaking it all. It will be probably the memory management and the error/message handling.

I have swapped milestones 3 and 4. My original plan was to get interrupt-driven asynchronous I/O first, but right now I can't stuff anything else in the kernel space. On the other hand, the asynchcronous I/O implementation could and will use separated processes and inter-task messaging, so it makes much more sense to do it in this order.

About process creation and task loading

As I wrote when I presented this project, my goal is to learn about operating system implementation as well as to known better the PDP-11 architecture. I have to say I am progressing in both goals. A lot.

Lets put an example. The load() function. This funcion loads a task image from a channel and puts it into memory. When I started I planned it to be a syscall. It IS a syscall in some operating systems (in particular, MVS or z/OS, as you want to call that thing). In UNIX there is not a load() function, but a family of exec() functions which do (aproximately) the same function.

The UNIX operating system uses a somehow curious way to create a new task. To create a new task (or process) you must use the fork() system call, which basically duplicates the executing task. Then this function returns a different result for the "parent" task (in this case it returns the ID of the created process) and the "child" or "new" task (it returns NULL). Then the parent task can go on with its own business, while the "new" one can call the exec() function to load a new executable and transfer control to it. 

In contrast, MUXX has a different model, inspired in VMS. In that wonderful operating system the parent task uses the CREPRC syscall to build a new address space and load an image into it, just in one step. There is no "forking": the new process does not begin its existance as a copy of the parent. In MUXX the CREPRC system call does create a new address space and does set up its corresponding memory pages, but it does not load an image into it. It assumes the new process will execute some code already present in the kernel image (so the new task is hard-linked into the kernel). The reason to do it this way is really simple: during the development of MUXX I needed to be able to create new tasks before I could think about loading different, separated images. So CREPRC just creates and branches, but does not load.

So here comes LOADPRC, a second syscall which does load a new image. And now comes the problem: the MUXX kernel is not preemptable at this moment. That means that the kernel code (so the syscalls) executes with the interrupts disabled. And to do a potentially long (in terms of elapsed time) operation like an  image load with interrupts (and hence task switching) disabled is really an ugly solution. That means, basically, that the actual loading of the new image has to be performed in user mode, out of the kernel space.

This brings us into another problem. We are loading an image into a new task, what means different memory configuration (different MMU setup)... from user mode code and from (probably) a non-privileged task. I could probably have written some kludge to do the MMU config switching during the LOAD, but I felt a little bit lazy about it, so I decided to use another aproximation. Enters rshell.

rshell comes from resident shell. It will evolve to be the "resident" part of the system shell (the part which will be present in memory at all the time), and is hard-linked with the kernel code, so it has to live in the low 24KB of memory. Right now rshell is really simple. It does just one thing: it looks at the task memory base (at 060000 octal) for a device name (in the future it will be a file specification) and calls load() to read that task into memory (beginning at 060000!). load() is a library function, not a syscall, and executes mostly in user mode (except for the actual I/O). When/if the load is completed, rshell jumps to the address 060000, where the linker has located the crt0() function, which in turn initializes the runtime and calls the main() function for the task we are loading. 

The nice part of this is rshell executes in the new task address space. So when a program invokes LOADPRC it does the following:

• It prepares a new task address space, just like CREPRC does.
• It copies the name of the image to load at the position 060000 of the new address space
• It prepares the new task to begin its execution at the rshell() entry point

And that's it. When the scheduler selects the new task to execute it will run rshell, which will load the new image and transfer control to it... Of course, right now MUXX can use just the paper tape reader, so there are concurrency problems (if we try to run 3 tasks they will try to load at the same time...), but those are workable.

About the MMU setup and MUXX memory model

When I started loading tasks at their own address (beginning at 060000) I begun experiencing weird and hard to explain crashes. They came in all flavours: Illegal instructions, odd memory accesses, MMU exceptions... I really had no idea about what was happening, but they just occurred when I started the tasks using LOADPRC (if I linked the same tasks into the kernel and ran them using CREPRC they did fine). After some investigation, I realized the code resident at the page 6 (060000-080000) was being overwritten, so when the execution got transferred to the corrupted position all kinds of Bad Things happened. Observing the overwritten areas they seemed to be clobbered by printf()... so I spent a pair of days trying to find some bug there.

I obtained printf() from the 2.11BSD libc, with some minor changes to adapt the BSD code to MUXX ABI. I had previously found some stupid bugs in my minor changes, but I thought it was now correct. After some hours of work, I decided printf() was doing fine, so the problem had to be somewhere else. And that somewhere else happened to be precisely here. Let's take a closer look at that.

MUXX task memory map

The muxx_memsvc_svc.c module contains the basic memory management code. If you look at the linked version you'll see it is incomplete (right now the memory deallocation is not implemented yet).One of the things that module does is to set up the address space for a new task. That is what muxx_setup_taskmem() does.

A MUXX task address space has three (or four) different memory regions:

• 000000 to 057777 is the kernel space, shared by all the tasks. It is write-protected from user mode code (except for the tasks with operprv privilege and the system tasks). 
• 060000 to 137777 is the user space. CREPRC and LOADPRC allows to select between three task sizes (SMALL, MED and LARGE), which correspond to real user spaces up to 077777, 117777 or 137777 respectively.
• 0150000 to 157777 is the stack space, furtherly divided at 156000 between user mode stack and kernel mode stack

The space between 137777 and 150000 is not allocated at this point. It will be used when I implement different stack sizes.

The fourth region, between 160000 and 177777 is the mapped IO area, and it is not mapped into the user mode address space unless the task has the ioprv privilege.

PDP-11 memory management

The PDP-11 program-accessible address space is, then, a 65566 position array of bytes, or a 32768 array of words. The physical memory addressing capability of the PDP11 depends on the model and goes up to 2048 Kbytes. The model I am targeting, the 11-60, has a 256KB addressable space. The MMU allows the PDP to map those 256KB so they cant be seen through the 64KB address space. When the MMU is enabled, the addresses the programs work with are virtual addresses which must be relocated to real, physical addresses. Using the MMU we can provide different relocations for different tasks, so each task gets its own 64KB address space, with or without sharing memory with other tasks.

A 16-bit virtual address is formed by two parts:

• The highest 3 bits form a number from 0 to 7 which tells us what "page register" will we use to relocate the address.
• The lowest 13 bits form a displacement inside a 8192 bytes "page".

The MMU has a bunch of registers to configure the memory relocation. These registers are grouped into two or three sets corresponding to the processor execution modes (user, kernel and, in some models, supervisor). The processor models with separate I and D address spaces double the number of registers. Our PDP-11/60 has just two sets for user mode and kernel mode. Each set contains eigth pairs of registers, each of those pairs corresponding to one "page" between 0 and 7. And each pair is composed by a page address register (PAR) and a page descriptor register (PDR). Each PAR contains a number of physical memory block, being each block formed by 64 bytes. The PDR contain control information, like the access permissions for the page they describe, the size of the block and some other information, particulary the growth direction, which tells the MMU if the accesses will be done upwards (like in normal code or data arrays) or downwards (like in a stack).

Lets say our program refers to the virtual address 060042. That address breaks into:

• A page number in bits 13-15, which is 03.
• A displacement in bits 0-12, which is 42.

That means the MMU will use the PAR and PDR for the page 03 of the current mode. We will ignore the PDR at this moment. Let's say the PAR for the page 03 contains the value 01200. Then, the physical direction will be formed this way:

• Physical address base: 01200 * 0100 = 0120000 (remember we are using octal figures)
• Displacement: 042
• Physical address: 0120042

That will be the real address the CPU will reference. Changing the values of the PAR we can assign the same virtual address to different physical addresses, efectively isolating the tasks from each other. Manipulating the PDR we can establish protection for the memory pages, so the user mode code can't write into the kernel space or can't even see the IO mapped addresses.

The MMU can also detect when we try to access a memory address which is out of the mapped range. The PDR contains a field with the length (in blocks) of the current page. So, if  we confugure it with a 4KB memory page and we reference a position at that page with a displacement grearter than 4KB, the MMU forces a memory trap and invokes a handler using the usual PDP-11 trap sequence; this allows an operating system to do something about that, being it just killing the offending task, invoking some paging mechanism or panicking if the trap has occurred in kernel space.

Now you can tell me a liar. I've omitted that pesky "growth direction" bit in the PDR. If that bit is set then we are telling the MMU that the page grows downward, and then the size check works just the opposite way. That means in our exemple of a 4KB page it would trigger a trap if we referred to a virtual memory with a displacement less than 4KB! In MUXX the stack resides in a 4KB page, so if we grow the stack below that 4KB line, it will trigger the trap and will abort the task (currently, it panics the system, since MUXX does not know how to kill a task yet)

Misunderstanding the MMU and learning it the hard way

Let's look at the stack preparing code in muxx_setup_taskmem:

  /*
  ** Task stack space
  ** The stack size options should be evaluated and applied here
  ** Since the stack grows downward, the PAR contains Addr - Size 
  ** so the physical addresses are properly calculated without
  ** overlapping.
  **
  ** Example: Base virtual address: 0140000
  **          Top of stack:         0157777 (0120000 + 020000 - 1)
  **          Bottom of stack:      0150000 (0120000 + 010000 )
  **          PAR value:            0500
  **          Size:                 0100 (4K)
  **          Physical range:       060000:067777   
  */
  mcb = muxx_mem_getblock(task, 0100, MMCB_FLG_STK, 6);
  if (mcb != NULL) {
    task->mmuState.upar[6] = mcb->blockAddr-0100;
    task->mmuState.updr[6] = PDR_ACC_RW | PDR_SIZ_4K | PDR_DIR_DN;
    task->mmuState.kpar[6] = mcb->blockAddr-0100;
    task->mmuState.kpdr[6] = PDR_ACC_RW | PDR_SIZ_4K | PDR_DIR_DN;
  } else {
    return (ENOMEM);
  }

Don't you find anything unusual? Let's check the previous, bugged code:

  /*
  ** Task stack space
  ** The stack size options should be evaluated and applied here
  */
  mcb = muxx_mem_getblock(task, 0100, MMCB_FLG_STK, 6);
  if (mcb != NULL) {
    task->mmuState.upar[6] = mcb->blockAddr;
    task->mmuState.updr[6] = PDR_ACC_RW | PDR_SIZ_4K | PDR_DIR_DN;
    task->mmuState.kpar[6] = mcb->blockAddr;
    task->mmuState.kpdr[6] = PDR_ACC_RW | PDR_SIZ_4K | PDR_DIR_DN;
  } else {
    return (ENOMEM);
  }

Do you see the difference? In the corrected code I'm substracting 0100 from the PAR value for the stack page. So if I'm allocating the addresses 140000-150000 to the stack, I'm not writing 1400 in the PAR, but 1300. Why? 

The top of the stack for the kernel mode code is at 157777. That is, page 6, displacement 17777. If I put 1400 in the PAR this translates to 140000 + 17777 = 157777. Since the stack PDR is configure with the "grow downwards" bit, the MMU will not complain about this relocation and when MUXX writes something in the stack, it will clobber the 157777 physical memory address.

Of course, our stack is 4KB big, so we have just allocated 0100 blocks for it. And we have allocated the next available blocks to the next created task (since the stack is the last page we set up). So the next page will have probably the blocks 1500 to 1700 (for a 8K so 200 blocks page). Now we can see the problem. The stack of this task is sharing the same memory blocks of some page of the next created task. And that is precisely what we see... the 03 page of the next task (the one which will be allocated first!) is clobbed by our current task writing into its stack! 

The solution is to pull back the content we write in the PAR by 0100, corresponding to the 4KB size we are not allocating to the stack. Redoing the numbers, the 157777 virtual address would relocate into block 1300 plus displacement 17777 = 147777, which is just in the 1400-1477 range. If we go down 140000 the MMU will detect it and will trigger a trap, so we are safe. The weird thing is that in our memory management tables we will register the stack uses the pages  1400-1477, while the PAR will contain 1300. Weird, but correct.

Going on, and small machines...

And that is all for today. As unrelated stuff, my raspberry holds now 6 simulations: 2 VAXen, 3 PDP-11s (RSXM, RSXM+ and RSTS) and PDP-10... without a lot of load, but they work. For those of you who are part of HECNET, remember those machines are in area 7 and most of them have GUEST accounts, You will be welcome at my humble simulated, rapsberryzed digital home.

New toy...

This is going to be a short post... I've just got a little nice toy, and I've moved 3 of my SIMH machines to it. 

Yes, it is a Raspberry PI. And right now it is hosting three simulated machines: a VAX 780 running VMS 4.7 (and working as area router for my HecNet link), a microVAX 3900 running openVMS 7.3 and a PDP-11 running RSX11-MPlus 4.6. Of course, the simulated machines are idle most of the time, otherwise I guess they would overwhelm the poor raspberry...

To make it run, I have cloned the SIMH git repository from http://github.com/simh/simh, and before compiling it I've installed the packages for libpcap and vde2 (if you are going to do this, don't forget you have to install the -dev packages). SIMH compiled without any complaint, and once I got the simulated machines copied to the USB pendrive I've plugged to the thingy, they booted without a hassle.

After that, I just edited /etc/networking/interfaces so the vde magic gets configured at boot time:

auto lo

iface lo inet loopback

#iface eth0 inet dhcp

auto eth0
     iface eth0 inet static

auto tap0
     iface tap0 inet manual
     vde2-switch -t tap0 -n 16 -s /tmp/vde.ctl -M /tmp/vde.mgmt -m 666 --mgmtmode 666 

auto br0
     iface br0 inet static
     address 192.168.0.8
     network 192.168.0.0
     netmask 255.255.255.0
     broadcast 192.168.0.255
     gateway 192.168.0.128
     bridge_ports eth0 tap0             

And that is basically all (I'm using a static IP address bound to br0... the default configuration for the debian-based distribution I'm using is to use DHCP).

Not bad for a 35€ little computer...