vanalboom dot org

RIP Charles "Longstaff" McCluski

Lennert — Sun, 15 Nov 2009 08:22:19 +0000

RIP Charles "Longstaff" McCluski

On July 14th 2009, Charles McCluski, owner of Westfield Scientific Computing, internet Linux and OS/2 guru, cooking fanatic, avid Sam Adams Cream Stout lover and most of all a dear friend, passed away.

Without ever complaining and while keeping up his joyful spirit and love for long and deep conversations on life, the universe and everything, he battled a stomach cancer that caused him a tremendous lot of pain. Only a few people knew this, and no one really knew how bad a condition he was in until he suddenly went silent. Only 59 years old and a lot, lot younger in spirit still, he had lost the fight.

We miss you, Charles - we miss your dry wit, your extraordinary beef stews, your cheeky joking about "the girlies" and the "gronk" that could mean just about anything, but we always knew what.

Rest in peace.

To keep his memory alive, I've taken the liberty to keep a mirror of his obituary around. He may be forgotten in Westfield, but he won't be forgotten by us.

July 26, 2009
I remember Charles "Longstaff" McCluski as being fully dedicated in everything he did. We had some discussions (some might say debates) about many things, be it technologies or simply... life itself, and he always had very interesting points. We'll be missing you.
Wherever you are, take care "old branch" and... "See ya later".
Michel Goyette, Québec

July 18, 2009
I met Charles in 1998 in #os/2warp on the IRC... and we've stayed in touch since. It's a huge blow to find out he has passed away... he was a kind and caring man. I'll always remember our trip into Manhattan while I was in New York on business. :)
Goodbye, my friend... we'll meet again some day.
Chris Cureau, Covington, Louisiana

July 17, 2009
Charles was an amazing man. Always kind, always looking for the bright side of life, and helping other see it too. I too met him on the Internet through IRC (Internet Relay Chat). We had many deep discussion about life, and I also appreciated his honest opinions.
This is a great loss, to many who knew him.
May the Lord Bless and Keep you Charles. You will always be my friend.
Eric Werny, St. George, Utah

July 17, 2009
Nearly two decades ago, I ventured onto this new thing called the Internet and ended up in one of it's corners which was an IRC channel. There I met Charles, or Longstaff to many of his friends.
Over the years he was a mentor and good friend that always was willing to talk. I am very saddened to hear of his passing. Never once did I ever hear him say a unkind word to anyone. He was always accepting and genuinely interested in everyone. Truly, I don't think that I would be the engineer that I am today without the privilege of knowing him and being able to gain from his wisdom. Charles, you will be truly missed. I believe in a life beyond this one, so we'll have to chat again after what amounts to a ethereal netsplit, rejoins.
Matthew "ShadwWulf" Werny, North Ogden, Utah

July 17, 2009
Longstaff was friendly and very smart. When hs saw a major wrong going on he was first to help do something about it. He help me get #os/2warp going. He was my very close friend and adversary, I going to miss him very very much. A major loss to the os/2 and linux communities.
May God rest his spirit in heaven with his family.
Steven Taylor, Parkersburg, West Virginia

July 17, 2009
Almost seven years ago, I encountered him for the first time. I was impressed by his kindness and always enjoyed the talk I had with him. I usually asked him about New York City where he used to work for a long time. There was no doubt that I would come visit him if I ever got to visit NYC.
I am very sad not to be able to do so anymore.
Axel Meiss, Hamburg

July 17, 2009
Goodbye Charles, it's hard to really find the words to express how I feel right now, only finding out about your passing this morning. You were always there over the years to talk to me, you helped me through some hard times, and we also had a lot of laughs and good times together.
Things will never be the same without you "Longstaff".
I miss you terribly my friend.
Jeremy Workman, Deltona, Florida

July 17, 2009
While I never had the pleasure to meet with Charles in person, I have been corresponding with him for many years. His calm, respectful conversation on the most various topics, his many years of knowledge and expertise in information technology and the sudden wit with which he could bring laughter in the most dry matter are things I will always remember fondly.
Since before meeting him, I have made crazy plans about the road trip across the USA I wanted to do one day. The one thing there that was always beyond questioning was that I would be passing through Westfield, NJ to visit Charles, the house he never stopped working on, and to check out those extraordinary delicious-sounding cooking recipes he came up with all the time. I don't think I'll ever be able to visit the USA without the idea something's missing, without that sad feeling of having missed a chance to meet up with that distant friend in the town he loved so much.
I'll miss you, Charles. Bigtime.
Lennert Van Alboom, Belgium

Lennert Sun, 15/11/2009 - 10:22

Clustering OpenVMS on SIMH

Lennert — Fri, 27 Jan 2012 19:49:09 +0000

Clustering OpenVMS on SIMH

After following the article published earlier, you ended up with two OpenVMS 7.3 systems running on SIMH VAX emulator instances. Networking was configured, both over DECnet and TCP/IP. Next up: clustering those two systems.

Creating the basic cluster

A two-node cluster is vulnerable: the only way to get a proper quorum working in such a config, is by using a quorum disk. Since we don't have shared storage to do this (yes, we *could* try to do this on the same host, by sharing a disk-file - but what if we want to cluster a node that's not local?) we'll opt for a less sexy approach: one node will be 'master', the other 'slave'. If the 'master' goes down, the 'slave' will have to wait until the 'master' is back up again. This guarantees cluster data integrity.

The cluster configuration utility should be ran on both nodes; first on the master, then on the slave.

$ @SYS$MANAGER:CLUSTER_CONFIG_LAN.COM

Cluster Configuration Procedure
Executing on a VAX System

DECnet Phase IV is installed on this node.

The LAN, not DECnet, will be used for MOP downline loading.

To ensure that this procedure is executing with the required
privileges, invoke it from the system manager's account.

Enter a "?" for help at any prompt.  If you are familiar with
the execution of this procedure, you may want to mute extra notes
and explanations by invoking it with "@CLUSTER_CONFIG_LAN BRIEF".

This VAX node is not currently a cluster member.

A few questions will be asked. Yes, you want to add your node to a cluster, or add a new one if one doesn't exist yet. Yes, the LAN will be used for cluster communications.

Cluster number and password are things you should write down: they will be needed to authorize new nodes. To keep the organization clean and simple, we'll take "cluster number" == "DECnet area". So, in our case: 1. Our nodes won't be serving as boot servers (unless you really want to - go ahead and try it), but will serve disks, but no RFxx disks.

And thus we arrive at the dreaded ALLOCLASS parameter. Without going into too much detail: this parameter points at the allocation class value of the machine, which is used to identify devices across the cluster. If WHISKY and COGNAC both have a DUA1 (and they have) we need this identifier to specify *which* DUA1 we are talking about. Again, to keep things clean and simple, I'm keeping the ALLOCLASS value the same as the DECnet node number, so WHISKY gets ALLOCLASS=131 and COGNAC 132. The devices would thus be $131$DUA1 and $132$DUA1.

MAIN MENU

1. ADD WHISKY to existing cluster, or form a new cluster.
2. MAKE a directory structure for a new root on a system disk.
3. DELETE a root from a system disk.
4. EXIT from this procedure.

Enter choice [1]:
Will the LAN be used for cluster communications (Y/N)? Y
Enter this cluster's group number: 1
Enter this cluster's password:
Re-enter this cluster's password for verification:
Will WHISKY be a boot server [Y]? N
Will WHISKY be a disk server (Y/N)? Y
Will WHISKY serve RFxx disks [N]?
Enter a value for WHISKY's ALLOCLASS parameter [0]: 131
Does this cluster contain a quorum disk [N]?

WARNING: WHISKY will be a voting cluster member. EXPECTED_VOTES for
this and every other cluster member should be adjusted at
a convenient time before a reboot. For complete instructions,
check the section on configuring a cluster in the "OpenVMS
Cluster Systems" manual.

This cluster configuration sets a bunch of parameters. We don't want to keep all of them, so once the cluster config is done, we tell the system NOT to run autogen yet.

Execute AUTOGEN to compute the SYSGEN parameters for your configuration
and reboot WHISKY with the new parameters. This is necessary before
WHISKY can become a cluster member.

Do you want to run AUTOGEN now [Y]? N

Add or change the following parameters on the master node:

$ EDIT SYS$SYSTEM:MODPARAMS.DAT

VOTES=1
EXPECTED_VOTES=1
SHADOWING=2
SHADOW_MAX_COPY=128

On node COGNAC, make sure to set VOTES=0. The "VOTES" and "EXPECTED_VOTES" do what they say: in this cluster, WHISKY has the only vote. Shadowing is being enabled, and 128 simultaneous I/O operations are allowed for syncing a shadow disk (we can spare the I/O on our fast host server, right?). After both nodes have their proper parameters, run autogen and reboot.

@SYS$UPDATE:AUTOGEN GETDATA REBOOT NOFEEDBACK

Once both machines are back up, lo and behold!

$ SHOW CLUSTER

View of Cluster from system ID 1155  node: WHISKY
┌───────────────────┬─────────┐
│      SYSTEMS      │ MEMBERS │
├────────┬──────────┼─────────┤
│  NODE  │ SOFTWARE │  STATUS │
├────────┼──────────┼─────────┤
│ WHISKY │ VMS V7.3 │ MEMBER  │
│ COGNAC │ VMS V7.3 │ MEMBER  │
└────────┴──────────┴─────────┘

Disk shadowing

One of the fun things of VMS clusters is their ability to mount any disk on any node, regardless of where that disk is located in the cluster - and their ability to replicate a disk between nodes. This is called disk shadowing, and it's a handy option to make sure your data doesn't die with your machine if something breaks.

Since we have a DUA1 on each of the nodes, we'll setup the shadowset between those.

$ INIT/ERASE $131$DUA1: SHADOWDATA
$ INIT/ERASE $132$DUA1: SHADOWDATA
$ MOUNT/SYSTEM/CLUSTER DSA0: /SHADOW=($131$DUA1,$132$DUA1) SHADOWDATA SHADOWDATA
%WBM-I-WBMINFO Deleting all bitmaps represented by this WBMB.
%MOUNT-I-MOUNTED, SHADOWDATA mounted on _DSA0:
%MOUNT-I-SHDWMEMSUCC, _$131$DUA1: (WHISKY) is now a valid member of the shadow set
%MOUNT-I-SHDWMEMCOPY, _$132$DUA1: (COGNAC) added to the shadow set with a copy operation

The commands speak for themselves: first, we initialise the disks, with extra option to erase them; then, we mount the disk on all systems in the cluster as device DSA0. DSA0 is a shadowset of $131$DUA1 and $132$DUA1; we mount it under label SHADOWDATA.

A few seconds later, OPCOM announces that the disk sync has started; ff you check the device now, you can see the shadow sync progress.

%%%%%%%%%%%  OPCOM  27-JAN-2012 20:03:07.93  %%%%%%%%%%%    (from node COGNAC at 27-JAN-2012 20:02:46.41)
Message from user SYSTEM on COGNAC
%SHADOW_SERVER-I-SSRVINICPY, initiating copy operation on _DSA0: at LBN: 0, I/O size: 127 blocks, ID number: 4500004B.

$ SHOW DEVICE SHADOWDATA

Device                  Device           Error    Volume         Free  Trans Mnt
Name                   Status           Count     Label        Blocks Count Cnt
DSA0:                   Mounted              0  SHADOWDATA     2375847     1   2
$131$DUA1:    (WHISKY)  ShadowSetMember      0  (member of DSA0:)
$132$DUA1:    (COGNAC)  ShadowCopying        0  (copy trgt DSA0:   4% copied)

And that's that. You can create, delete, modify, ... anything on SHADOWDATA: and it'll be immediately visible on the other node.

The last step: shared management

Now that we have a cluster and a cluster-wide shared disk, there's only one last thing to do. All nodes are still using their own, separate databases for virtually everything: their own user authorization files (UAF), rights lists, DECnet lists, ... This means that for every action we want to do on the cluster, we have to perform the commands on every node! This includes adding users; changing passwords; deleting users; ...

The solution is to put the files that keep the records for all these facilities on a common disk - like our shadowset. But since these files will be needed very early in the boot process, doing a MOUNT in SYSTARTUP_VMS.COM won't do: we'll have to hook into a file that is run much earlier. Create a dir on the SHADOWDATA volume to hold cluster-specific files, and edit SYLOGICALS.COM.

$ CREATE/DIRECTORY SHADOWDATA:[CLUSTER$CONFIG]
$ EDIT SYS$MANAGER:SYLOGICALS.COM

Yes, I always wanted to create a directory with a funky $ in it name :-) In SYLOGICALS.COM, search for the following line:

$! DEFINE/SYSTEM/EXECUTIVE SYSUAF                      SYS$SYSTEM:SYSUAF.DAT

As you can see, it's commented; this is the default location for SYSUAF.DAT. Below it, all other cluster-aware files are listed with their default locations. Our cluster disk will have to be mounted before we redefine these locations, so put these lines before the DEFINE list:

$ MOUNT/SYSTEM DSA0: /SHADOW=($131$DUA1,$132$DUA1) SHADOWDATA SHADOWDATA

Notice that we're NOT using the /CLUSTER switch here! Every node will mount this disk independently, to avoid complex issues if the node issuing the /CLUSTER mount is delayed during boot. Save the file (remember to do it on both nodes!) and reboot the entire cluster.

$ REBOOT

If all goes well, the shadowset will be mounted on both nodes at boottime.

$ SHOW DEV SHADOWDATA

Device                  Device           Error    Volume         Free  Trans Mnt
Name                    Status           Count     Label        Blocks Count Cnt
DSA0:                   Mounted              0  SHADOWDATA     2375847     1   2
$131$DUA1:    (WHISKY)  ShadowSetMember      0  (member of DSA0:)
$132$DUA1:    (COGNAC)  ShadowSetMember      0  (member of DSA0:)

Next, we're going to copy the needed files to the CLUSTER$CONFIG directory:

$ COPY SYS$SYSTEM:SYSUAF.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:SYSUAFALT.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:SYSALF.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:RIGHTSLIST.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:NETPROXY.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:NET$PROXY.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:NETOBJECT.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:NETNODE_REMOTE.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:LMF$LICENSE.LDB SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:VMSMAIL_PROFILE.DATA SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:VMS$OBJECTS.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$MANAGER:VMS$AUDIT_SERVER.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:VMS$PASSWORD_HISTORY.DATA SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$LIBRARY:VMS$PASSWORD_DICTIONARY.DATA SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$MANAGER:NETNODE_UPDATE.COM SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$LIBRARY:VMS$PASSWORD_POLICY.EXE SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$SYSTEM:LAN$NODE_DATABASE.DAT SHADOWDATA:[CLUSTER$CONFIG]
$ COPY SYS$STARTUP:SYLOGIN.TEMPLATE SHADOWDATA:[CLUSTER$CONFIG]SYLOGIN.COM

Now, don't be alarmed if not all commands succeed; this is normal. Not all files will already exist on your system, in which case you'll get an error like the following:

$ $ COPY SYS$SYSTEM:NETPROXY.DAT SHADOWDATA:[CLUSTER$CONFIG]
%COPY-E-OPENIN, error opening SYS$COMMON:[SYSEXE]NETPROXY.DAT; as input
-RMS-E-FNF, file not found

You can safely ignore this; if the file doesn't exist, there's nothing to copy. You'll also notice some files being locked and thus not copyable:

$ $ COPY SYS$SYSTEM:VMS$OBJECTS.DAT SHADOWDATA:[CLUSTER$CONFIG]
%COPY-E-OPENIN, error opening SYS$COMMON:[SYSEXE]VMS$OBJECTS.DAT;1 as input
-RMS-E-FLK, file currently locked by another user

In this case, you'll have to use a sneaky workaround (thanks, Hoff!) to get the contents of this file anyway:

$ CONVERT/SHARE SYS$SYSTEM:VMS$OBJECTS.DAT SHADOWDATA:[CLUSTER$CONFIG]VMS$OBJECTS.DAT

Once all files are either copied or found not to exist, all that remains to do is to modify SYLOGICALS.COM to actually point to the files on the shadowed disk. Open the file, locate the MOUNT command that you added earlier on, and add the following lines underneath it:

$ DEFINE/SYSTEM/EXECUTIVE SYSUAF          SHADOWDATA:[CLUSTER$CONFIG]SYSUAF.DAT
$ DEFINE/SYSTEM/EXECUTIVE SYSUAFALT       SHADOWDATA:[CLUSTER$CONFIG]SYSUAFALT.DAT
$ DEFINE/SYSTEM/EXECUTIVE SYSALF          SHADOWDATA:[CLUSTER$CONFIG]SYSALF.DAT
$ DEFINE/SYSTEM/EXECUTIVE RIGHTSLIST      SHADOWDATA:[CLUSTER$CONFIG]RIGHTSLIST.DAT
$ DEFINE/SYSTEM/EXECUTIVE NETPROXY        SHADOWDATA:[CLUSTER$CONFIG]NETPROXY.DAT
$ DEFINE/SYSTEM/EXECUTIVE NET$PROXY       SHADOWDATA:[CLUSTER$CONFIG]NET$PROXY.DAT
$ DEFINE/SYSTEM/EXECUTIVE NETOBJECT       SHADOWDATA:[CLUSTER$CONFIG]NETOBJECT.DAT
$ DEFINE/SYSTEM/EXECUTIVE NETNODE_REMOTE  SHADOWDATA:[CLUSTER$CONFIG]NETNODE_REMOTE.DAT
$ DEFINE/SYSTEM/EXECUTIVE LMF$LICENSE     SHADOWDATA:[CLUSTER$CONFIG]LMF$LICENSE.LDB
$ DEFINE/SYSTEM/EXECUTIVE VMSMAIL_PROFILE SHADOWDATA:[CLUSTER$CONFIG]VMSMAIL_PROFILE.DATA
$ DEFINE/SYSTEM/EXECUTIVE VMS$OBJECTS     SHADOWDATA:[CLUSTER$CONFIG]VMS$OBJECTS.DAT
$ DEFINE/SYSTEM/EXECUTIVE VMS$AUDIT_SERVER SHADOWDATA:[CLUSTER$CONFIG]VMS$AUDIT_SERVER.DAT
$ DEFINE/SYSTEM/EXECUTIVE VMS$PASSWORD_HISTORY SHADOWDATA:[CLUSTER$CONFIG]VMS$PASSWORD_HISTORY.DATA
$ DEFINE/SYSTEM/EXECUTIVE VMS$PASSWORD_DICTIONARY SHADOWDATA:[CLUSTER$CONFIG]VMS$PASSWORD_DICTIONARY.DATA
$ DEFINE/SYSTEM/EXECUTIVE NETNODE_UPDATE  SHADOWDATA:[CLUSTER$CONFIG]NETNODE_UPDATE.COM
$ DEFINE/SYSTEM/EXECUTIVE VMS$PASSWORD_POLICY SHADOWDATA:[CLUSTER$CONFIG]VMS$PASSWORD_POLICY.EXE
$ DEFINE/SYSTEM/EXECUTIVE LAN$NODE_DATABASE SHADOWDATA:[CLUSTER$CONFIG]LAN$NODE_DATABASE.DAT

Save SYLOGICALS.COM. As you have noticed, I've copied one extra file to the CLUSTER$CONFIG directory: SYLOGIN.COM. This is basically the "/etc/profile" of VMS: you can define startup commands here that will be executed for every DCL login - be it local console or telnet, SYSTEM or user. This file is defined in SYSTARTUP_VMS.COM:

$ DEFINE/SYSTEM/EXECUTIVE SYS$SYLOGIN SHADOWDATA:[CLUSTER$CONFIG]SYLOGIN.COM

This does mean that the file itself must be readable by all users, and also that the directory must be accessible:

$ SET DEF SHADOWDATA:[000000]
$ SET PROTECTION=(S:RWED,O:RWED,G:RE,W:RE) SHADOWDATA:[CLUSTER$CONFIG]SYLOGIN.COM
$ SET PROTECTION=(S:RWED,O:RWED,G:RE,W:RE) CLUSTER$CONFIG.DIR

Do note the funky syntax to go to the "root" directory of a filesystem; this root dir is called 000000. We're adding the World (W:) permissions Read and Execute on both dir and file. Edit the SYLOGIN.COM file, and add a VT100 term definition and a nicer custom prompt:

$ SET TERM/VT100 
$ SET PROMPT="''F$GETSYI("DECNET_FULLNAME")'''F$GETJPI("","USERNAME")'$ "

Reboot both nodes. Crossing fingers, you'll have a common UAF in a minute.

Welcome to OpenVMS (TM) VAX Operating System, Version V7.3   

Username: system
Password:
%LICENSE-I-NOLICENSE, no license is active for this software product
%LOGIN-S-LOGOPRCON, login allowed from OPA0:
Welcome to OpenVMS (TM) VAX Operating System, Version V7.3 on node COGNAC
    Last interactive login on Friday, 27-JAN-2012 21:12
COGNAC::ALVER       $

Wait, what? Yes, of course. The license db is one of the files that was moved to the cluster disk, and since we took the copy of the master node, we'll have to re-apply all licenses of the slave node. Easiest way is to FTP the license files over to the machine, and @ them.

COGNAC::ALVER       $ @ COGNAC-VMS.TXT
COGNAC::ALVER       $ @ COGNAC-PAK.TXT
COGNAC::ALVER       $ LICENSE MODIFY/INCLUDE=COGNAC VAX-VMS
%LICENSE-W-AMBIG, information provided was ambiguous; multiple licenses were found for VAX-VMS

This last error is to show that since there are now two licenses for each software on the cluster, you'll need to be more specific about which specific license you'll want to MODIFY:

COGNAC::ALVER $ LICENSE MODIFY/INCLUDE=COGNAC /AUTHORIZATION=DECUS-BEL-006150000-170XXXX VAX-VMS 
COGNAC::ALVER $ LICENSE LOAD

Now, let's check the terminal type:

COGNAC::ALVER       $ SHOW TERMINAL
Terminal: _OPA0:      Device_Type: VT100         Owner: SYSTEM

   Input:     300     LFfill:  0      Width: 132      Parity: None
   Output:    300     CRfill:  0      Page:   24     

Terminal Characteristics:
   Interactive        Echo               Type_ahead         No Escape
   No Hostsync        TTsync             Lowercase          Tab
   Wrap               Scope              No Remote          No Eightbit
   Broadcast          No Readsync        No Form            Fulldup
   No Modem           No Local_echo      No Autobaud        No Hangup
   No Brdcstmbx       No DMA             No Altypeahd       Set_speed
   No Commsync        Line Editing       Overstrike editing No Fallback
   No Dialup          No Secure server   No Disconnect      No Pasthru
   No Syspassword     No SIXEL Graphics  No Soft Characters No Printer Port
   Numeric Keypad     ANSI_CRT           No Regis           No Block_mode
   Advanced_video     No Edit_mode       DEC_CRT            No DEC_CRT2
   No DEC_CRT3        No DEC_CRT4        No DEC_CRT5        No Ansi_Color
   VMS Style Input

Yup: VT100. And that pretty much concludes it. Two-node VMS cluster, running in emulated SIMH VAXes, on a single Linux host, with shadowed disk and common UAF.

Some notes

What if you'd want to expand your cluster and shadowset to three nodes, there's a few things to keep in mind:
- EXPECTED_VOTES has to be adjusted to 3 on all nodes
- VOTES has to be adjusted to 1 on the original 'slave' node
- DSA0 needs to be expanded with an extra shadow member
- SYLOGICALS.COM needs to be adjusted on the original two nodes to reflect the extra shadow member

Other than that... just try :-) and remember, if you think that what you're going to do might just screw up your entire setup... do a shutdown first, and take a plain file backup of the VMS-RQ* disks for each node. Easy, no?

Enjoy!

Credits & thanks

Even more than the earlier article, this second part of my project would, with a 100% certainty, have ended in a giant bloody trainwreck if it weren't for the guidance and VMS voodoo of Steve "Hoff" Hoffman, the Deathrow VMS cluster people and everyone in #vms on irc.2600.net. Thanks!

Lennert Fri, 27/01/2012 - 21:49

Installing VMS in SIMH on Linux

Lennert — Thu, 26 Jan 2012 21:19:57 +0000

Installing VMS in SIMH on Linux

Last week, I decided to pick up a pet project I started once but never got around to finish succesfully; building my own OpenVMS cluster. OpenVMS is one of the oldest operating systems that is still actively developed; it was originally created as VMS by Digital Equipment Corp (DEC, in short), which was bought by Compaq, which was bought by Hewlett-Packard. While most people know Linux and its UNIX roots, this system is a totally different cup of tea.

Three different architectures of machines can run VMS: the oldest being VAX, which was replaced by Alpha, which was in turn replaced by Itanium (the HP Integrity line of machines). Getting that kind of hardware for home use is not impossible, but would be both costly to purchase and hurting your power bill - and they're not exactly quiet either. Emulation is the way to go. Hence, a new goal: getting an emulated VMS cluster up and running. In this article we'll create two VMS machines and prepare them for clustering.

The Emulator

The emulation of choice is SIMH. SIMH supports a ton of different architectures, and one of those is a VAXserver 3900. The good thing about VAX machines is that they didn't have much memory - 16 or 32MB wasn't uncommon - so if we give each of our cluster nodes 64MB, it'll be plenty.

Either you build it by hand, or you use your distribution's package; I built mine myself and carry it around since it's just one binary file anyway. Just make sure you have the ka655x.bin file around; that one's needed to properly operate the SIMH VAX.

Networking

SIMH uses libpcap for its network connectivity. That means that while you *can* assign a network interface to a SIMH instance, you won't be able to connect to it from your own host. Silly - so this link explains how you can work around it. I chose to go for Hans Rosenfeld's approach: you create a bridge on one interface, launch the taptap binary, and bring up two tunX devices. Twice, once for each machine.

With newer SIMH releases (4.0-beta and up) this is no longer necessary.

This is the debian network config that I'm using:

[alver@ziff ~]$ tail -13 /etc/network/interfaces
auto br0 
iface br0 inet static
    pre-up /srv/vax/taptap &
    pre-up ifconfig tun0 up
    pre-up ifconfig tun1 up
    pre-up /srv/vax/taptap &
    pre-up ifconfig tun2 up
    pre-up ifconfig tun3 up
    bridge_ports eth1 tun0 tun2
        address 10.20.0.131
        netmask 255.255.0.0
        network 10.20.0.0
        broadcast 10.20.255.255

This leaves two interfaces to be used by the SIMH instances: tun1 (which is tunneled to tun0 by taptap) and tun3 (which is tunneled to tun2).

Prerequisites

SIMH may be opensource and free, but VMS isn't. Luckily, there's the OpenVMS Hobbyist Program; this program provides free OpenVMS Product Authorization Keys (PAK's) to any member of a participating HP User Group. In addition, the OpenVMS Hobbyist Program offers media kits containing OpenVMS Base O/S software and selected Layered Products for a low price.

So... if you're not yet a member of a local DECUS chapter, get registered there first (it's usually free); get your hands on a VMS software kit; and register for two sets of licenses for your SIMH/VAXes.

SIMH config

To form my cluster, I created two SIMH VAX instances. VMS hostnames are limited to six characters (yes, six). This makes finding a proper hostname less easy than you'd like. Following an obvious naming theme, my two nodes were baptized WHISKY and COGNAC. This is the config for WHISKY:

[simh@ziff ~]$ cat WHISKY/vax.ini
LOAD -r /srv/vax/ka655x.bin
ATTACH nvr /srv/vax/WHISKY/nvram.bin

SET RQ0 RA90
SET RQ1 RA90
ATTACH RQ0 /srv/vax/WHISKY/VMS-RQ0.dsk
ATTACH RQ1 /srv/vax/WHISKY/VMS-RQ1.dsk
SET RQ2 RRD40
ATTACH RQ2 /srv/vax/VMS-VAX.iso
SET RL DISABLE
SET TS DISABLE

SET CPU 64M
SET CPU IDLE

SET XQ MAC=00:11:22:33:44:55
ATTACH XQ tun1

BOOT CPU

COGNAC has a totally similar config, except for (obviously) different disk-files, tun3 instead of tun1, and a different (random) MAC address. Let's see what this config does:

LOAD -r /srv/vax/ka655x.bin
ATTACH nvr /srv/vax/WHISKY/nvram.bin

This loads the VAXserver 3900 microcode, and attaches a machine-specific file to be used as virtual NVRAM - to save eg. boot settings.

SET RQ0 RA90
SET RQ1 RA90
ATTACH RQ0 /srv/vax/WHISKY/VMS-RQ0.dsk
ATTACH RQ1 /srv/vax/WHISKY/VMS-RQ1.dsk
SET RQ2 RRD40
ATTACH RQ2 /srv/vax/VMS-VAX.iso
SET RL DISABLE
SET TS DISABLE

Two block devices RQ0 and RQ1 are defined, both of type RA90, which is an 1.2GB type disk. Two actual files on the host system are attached to these devices.
Warning: these files WILL grow to the size of the disktype specified! Even if you create the file as 1.2GB - if you define RQ0 as RA92, which is 8GB, it will expand as the system uses it. I prefer to define them as RA90 and immediately dd a 1.2GB file for each. 1GB is, for most uses, plenty.

A third block device is then defined - our CD drive, which is attached to an iso of the software kit CD. After that, unused devices are disabled.

SET CPU 64M
SET CPU IDLE

The VM gets 64M RAM; "idle" CPU detection is enabled. Earlier, without this option, SIMH would hog a full CPU core for 100% nonstop - even if the guest OS wasn't doing anything. Setting this option saves a lot of resources on the host.

SET XQ MAC=00:11:22:33:44:55
ATTACH XQ tun1

Fairly obvious: A network device with MAC address 00:11:22:33:44:55 is defined, and attached to device tun1 on the host.

BOOT CPU

...and boot the VM. Extra options will be added later to enable auto-boot. But for now, it'll do.

Installing VMS

Time to fire up the first VAX:

[simh@ziff vax]$ ./vax WHISKY/vax.ini

VAX simulator V3.8-1
NVR: creating new file
NVR: buffering file in memory

KA655-B V5.3, VMB 2.7
Performing normal system tests.
40..39..38..37..36..35..34..33..32..31..30..29..28..27..26..25..
24..23..22..21..20..19..18..17..16..15..14..13..12..11..10..09..
08..07..06..05..04..03..
Tests completed.
>>>

We're at the console prompt of the virtual VAX. We'll want to boot from the third device, being the cdrom drive; this device is called DUA2 here (after DUA0 and DUA1, the two disk devices).

>>>BOOT DUA2
(BOOT/R5:0 DUA2

  2..
-DUA2
  1..0..

%SYSBOOT-I-SYSBOOT Mapping the SYSDUMP.DMP on the System Disk
%SYSBOOT-W-SYSBOOT Can not map SYSDUMP.DMP on the System Disk
%SYSBOOT-W-SYSBOOT Can not map PAGEFILE.SYS on the System Disk
   OpenVMS (TM) VAX Version X7G7 Major version id = 1 Minor version id = 0
%WBM-I-WBMINFO Write Bitmap has successfully completed initialization.
PLEASE ENTER DATE AND TIME (DD-MMM-YYYY  HH:MM)

Enter current date and time. Remember to use the month's name abbreviations, like JAN, FEB, etc. When the installer asks to confirm if all devices are present, enter YES. Of course everything is there... we're running emulated, after all.

Configuring devices . . . 
Now configuring HSC, RF, and MSCP-served devices . . . 

Please check the names of the devices which have been configured,
to make sure that ALL remote devices which you intend to use have
been configured.

If any device does not show up, please take action now to make it
available.

Available device  DUA0:                            device type RA90
Available device  DUA1:                            device type RA90
Available device  DUA2:                            device type RRD40
Available device  DUA3:                            device type RX50
Available device  DYA0:                            device type RX02
Available device  DYA1:                            device type RX02
Available device  MUA0:                            device type TK50
Available device  MUA1:                            device type TK50
Available device  MUA2:                            device type TK50
Available device  MUA3:                            device type TK50

Enter "YES" when all needed devices are available: YES 

%BACKUP-I-IDENT, Stand-alone BACKUP T7.2; the date is 18-JAN-2012 19:44:12.80

$

And then you're at the first prompt. You'll want to tell the installer environment to restore a "backup saveset" of VMS, which is located on the cdrom at DUA2, onto the disk you're going to boot from, which is DUA0. After it's done,

$ BACKUP DUA2:VMS073.B/SAVE_SET DUA0:

%BACKUP-I-PROCDONE, operation completed.  Processing finished at 18-JAN-2012 19:45:57.83
If you do not want to perform another standalone BACKUP operation,
use the console to halt the system.

If you do want to perform another standalone BACKUP operation,
ensure the standalone application volume is online and ready.

Enter "YES" to continue: YES

%BACKUP-I-IDENT, Stand-alone BACKUP T7.2; the date is 18-JAN-2012 19:47:56.29

At this point, you have to exit the emulator by pressing CTRL-E. Remember this combination, you may need it later on. Restart the emulator; we're now going to configure auto-boot. At the console prompt, configure DUA0 to be auto-booted disk:

>>>SET BOOT DUA0

Exit the emulator again. Now we'll edit the vax.ini file, and replace the following line:

BOOT CPU

... by this:

SET CPU CONHALT
DEP BDR 0
BOOT CPU
EXIT

This tells the CPU to return control to the emulated VAX console, instead of the emulator itself. DEP BDR 0 is a black magic trick which writes a zero-byte to the BDR register, effectively causing the machine to auto-boot. This combined with the previous option allows us to reboot the VAX from within VMS, without having to touch the emulator. And, in case something goes wrong that the console can't handle, SIMH is told to gracefully EXIT.

Right... time to fire up the VAX once more to continue the installation process.

(...)
%SYSBOOT-I-SYSBOOT Mapping the SYSDUMP.DMP on the System Disk
%SYSBOOT-W-SYSBOOT Can not map SYSDUMP.DMP on the System Disk
%SYSBOOT-I-SYSBOOT Mapping PAGEFILE.SYS on the System Disk
%SYSBOOT-I-SYSBOOT SAVEDUMP parameter not set to protect the PAGEFILE.SYS
   OpenVMS (TM) VAX Version BI73-7G7 Major version id = 1 Minor version id = 0
%WBM-I-WBMINFO Write Bitmap has successfully completed initialization.

           OpenVMS VAX V7.3 Installation Procedure

                         Model: VAXserver 3900 Series
                 System device: RA90 - _DUA0:
                   Free Blocks: 2854566
                      CPU type: 10-01

* Please enter the date and time (DD-MMM-YYYY HH:MM) 18-JAN-2012 19:53
* Enter the volume label for this system disk [OVMSVAXSYS]: WHISKYSYS
* Enter name of drive holding the OpenVMS distribution media: DUA2
* Is the OpenVMS media ready to be mounted? [N] Y

A few basic things to configure, and one important one too: if you're going to cluster, you really don't want to have all your machines' system disks called "OVMSVAXSYS". That's just asking for confusion afterwards. I went for "WHISKYSYS" and "COGNACSYS". The VMS media are on DUA2, and are of course ready to be mounted.

%MOUNT-I-MOUNTED, VAXVMS073 mounted on _DUA2:

Select optional software you want to install.  You can install one
or more of the following OpenVMS or DECwindows components:

o OpenVMS library                              -  52200 blocks
o OpenVMS optional                             -  19000 blocks
o OpenVMS Help Message                         -  10400 blocks
o OpenVMS Management Station                   -  20000 blocks
o DECwindows base support                      -   4400 blocks
o DECwindows workstation support               -  23800 blocks
-  75 dots per inch video fonts          -    (included)
- 100 dots per inch video fonts          -   6200 blocks
o DECnet-Plus networking                       -  80000 blocks
o DECnet Phase IV networking                   -    800 blocks

Space remaining on system disk:  2854377 blocks

At this point the system will ask you what to install. You will want the library and optional files:

* Do you want to install the OpenVMS library files? (Y/N) Y

Space remaining on system disk:  2802177 blocks

* Do you want to install the OpenVMS optional files? (Y/N) Y

Space remaining on system disk:  2783177 blocks

And yes, you will really, really want the HELP database! Just have it install in the default location:

The Help Message utility (MSGHLP) provides online explanations
and user actions for OpenVMS messages in place of the hardcopy
OpenVMS System Messages and Recovery Procedures Reference Manual,
which is now separately orderable.

The MSGHLP database file, MSGHLP$LIBRARY.MSGHLP$DATA,
consumes approximately 10400 blocks and will be
placed by default on your system disk in SYS$COMMON:[SYSHLP]
unless you specify an alternate device when prompted.

* Do you want to install the MSGHLP database? (Y/N) Y

You can install this database on your system disk in SYS$COMMON:[SYSHLP]
or on an alternate device.  If you specify an alternate device, but no
directory, MSGHLP$LIBRARY.MSGHLP$DATA is placed in [HELP_MESSAGE].  When
prompted, take the default of the system disk or specify an alternate
device using this format:

device:[directory]

* Where do you want to install the MSGHLP database?
[SYS$COMMON:[SYSHLP]]

Space remaining on system disk:  2772777 blocks

We don't want the OpenVMS Management Station files - why would we want to manage our VMS from a Windows NT machine?

The OpenVMS Management Station is a client-server application that
provides OpenVMS system management capabilities through a client
application on a personal computer (PC) running Microsoft Windows.

The server application runs on OpenVMS systems and is automatically
installed as part of the OpenVMS operating system.

This option provides the files used to install the PC client software.
If you want to use the OpenVMS Management Station, you must install
these optional files on at least one OpenVMS system and then use one or
both of them to install the PC client on one or more PCs.  There are two
files:  TNT030_I.EXE for Intel systems (Windows 95 and Windows NT), and
TNT030_A.EXE for Alpha Windows NT systems.

The OpenVMS Management Station optional files consume approximately 20000
blocks and will be placed on your system disk in SYS$COMMON:[TNT.CLIENT].

* Do you want to install the optional OpenVMS Management Station files? (Y/N) N

DECwindows is the GUI of VMS. Since we're running in an emulator without graphics, it would just be a waste of diskspace to install it.

You can select DECwindows now, or you can use the DECW$TAILOR utility
to provide or remove DECwindows support after the installation.

Some media, TK50s in particular, can be very slow when tailoring on files.
You might want to select DECwindows now and tailor off unwanted files later.

NOTE: This kit does NOT contain full DECwindows.
To obtain full DECwindows, you must also install the separate
layered product, DECwindows Motif for OpenVMS VAX.
V1.2-3 is the minimum version of DECwindows Motif for OpenVMS VAX
that can be used with OpenVMS VAX V7.3.

The DECwindows components provided in this kit requires approximately
34400 blocks, broken down as follows:

o DECwindows base support                    -   4400 blocks
o DECwindows workstation support             -  23800 blocks
  -  75 dots per inch video fonts            -    (included)
  - 100 dots per inch video fonts (optional) -   6200 blocks

You must select the DECwindows base support option if
- you plan to run DECwindows software, or
- you are installing this kit on
  * a workstation or
  * an OpenVMS Cluster that contains workstations, or
- you want to provide font files for Xterminals.

If you are installing this kit on a system that includes Xterminals
and you do NOT select DECwindows base support, then you will have to use
the DECW$TAILOR utility to provide font files.

* Do you want the DECwindows base support? (Y/N) N

Here we arrive at the interesting part. DECnet is a non-IP network protocol, used by VMS for various tasks. There are multiple flavours of DECnet; at this point, there's two options to choose from. DECnet Phase IV is the "older" branch; DECnet-Plus is the most "modern" version with a lot of new features, which need a lot more insight and offer a plethora of new ways to screw things up. For our purposes - and to make sure we don't get any more confused - I'll be going with the old DECnet Phase IV.

Beginning with OpenVMS V7.1, the DECnet-Plus kit is provided with
the OpenVMS operating system kit.  Compaq strongly recommends that
DECnet users install DECnet-Plus.  DECnet Phase IV applications are
supported by DECnet-Plus.

DECnet Phase IV is also provided as an option.  Support for DECnet
Phase IV is available through a Prior Version Support Contract.

If you install DECnet-Plus and TCP/IP you can run DECnet
applications over a TCP/IP network.  Please see the OpenVMS
Management Guide for information on running DECnet over TCI/IP.

If you plan to install DECnet Phase IV do NOT select DECnet-Plus.

* Do you want to install DECnet-Plus? (Y/N) N
* Do you want to install DECnet Phase IV? (Y/N) Y

Space remaining on system disk:  2771977 blocks

And that's about it for choices. A summary is given, and you give the go.

The following options will be provided:

OpenVMS library
OpenVMS optional
OpenVMS Help Message
DECnet Phase IV

Space remaining on system disk:  2771977 blocks

* Is this correct? (Y/N) Y

It'll start to unpack your selections. For DECnet, it'll ask for confirmation once more.

Restoring OpenVMS library save set ...
%BACKUP-I-STARTVERIFY, starting verification pass

Restoring OpenVMS optional save set ...
%BACKUP-I-STARTVERIFY, starting verification pass

Restoring OpenVMS Help Message save set ...
%BACKUP-I-STARTVERIFY, starting verification pass

Now registering the OpenVMS operating system in the
POLYCENTER Software Installation product database

The following product will be registered:
DEC VAXVMS VMS V7.3                    DISK$VAXVMSV73:[VMS$COMMON.]

The following product has been registered:
DEC VAXVMS VMS V7.3                    Transition (registration)

Installing DECnet Phase IV...
%MOUNT-I-MOUNTED, VAXVMS073 mounted on _DUA2:

The following product has been selected:
DEC VAXVMS DECNET_PHASE_IV V7.3        Layered Product

Configuration phase starting ...

You will be asked to choose options, if any, for each selected product and for
any products that may be installed to satisfy software dependency requirements.

DEC VAXVMS DECNET_PHASE_IV V7.3: DECNET_PHASE_IV

COPYRIGHT (c)  2-APR-2001 -- All rights reserved

Digital Equipment Corporation

* This product does not have any configuration options.

Support addendum to DECnet Phase IV service contract required

Do you want to continue? [YES]

Execution phase starting ...

The following product will be installed to destination:
DEC VAXVMS DECNET_PHASE_IV V7.3        DISK$VAXVMSV73:[VMS$COMMON.]

Portion done: 0%...10%...30%...40%...50%...60%...70%...80%...100%

The following product has been installed:
DEC VAXVMS DECNET_PHASE_IV V7.3        Layered Product

You can now remove the distribution kit from DUA2:.

In an OpenVMS Cluster, you can run multiple systems sharing all files
except PAGEFILE.SYS, SWAPFILE.SYS, SYSDUMP.DMP, and VAXVMSSYS.PAR.

Cluster configuration cannot be done at this time because no network
is present.  In order to configure a cluster you must FIRST do one
or both of the following:

o Install DECnet-Plus (or DECnet Phase IV), or
o Execute SYS$STARTUP:LAN$STARTUP.COM by removing the
  comment delimiter ("!") from the line

$! @SYS$STARTUP:LAN$STARTUP

in SYS$MANAGER:SYSTARTUP_VMS.COM.

Then configure the cluster by executing the following command:

@ @SYS$MANAGER:CLUSTER_CONFIG

See the OpenVMS System Manager's Manual: Essentials for more information.

Make sure to write down your passwords. SYSTEM is an account you'll be needing at least for the next few steps; for SYSTEST and FIELD I provided a random 32-char string as pass.

Now we will ask you for new passwords for the following accounts:

SYSTEM, SYSTEST, FIELD

Passwords must be a minimum of 8 characters in length.  All passwords
will be checked and verified.  Any passwords that can be guessed easily
will not be accepted.

* Enter password for SYSTEM:
* Re-enter for verification:

%UAF-I-MDFYMSG, user record(s) updated
%VMS-I-PWD_OKAY, account password for SYSTEM verified

* Enter password for SYSTEST:
* Re-enter for verification:

%UAF-I-MDFYMSG, user record(s) updated
%VMS-I-PWD_OKAY, account password for SYSTEST verified

The SYSTEST_CLIG account will be disabled.  You must re-enable
it before running UETP but do not assign a password.

%UAF-I-PWDLESSMIN, new password is shorter than minimum password length
%UAF-I-MDFYMSG, user record(s) updated

* Enter password for FIELD:
* Re-enter for verification:

%UAF-I-MDFYMSG, user record(s) updated
%VMS-I-PWD_OKAY, account password for FIELD verified

Creating RIGHTS database file, SYS$SYSTEM:RIGHTSLIST.DAT
Ignore any "-SYSTEM-F-DUPIDENT, duplicate identifier" errors.

%UAF-I-RDBCREMSG, rights database created
%UAF-I-RDBADDMSGU, identifier DEFAULT value [000200,000200] added to rights database
%UAF-I-RDBADDMSGU, identifier FIELD value [000001,000010] added to rights database
%UAF-I-RDBADDMSGU, identifier SYSTEM value [000001,000004] added to rights database
%UAF-I-RDBADDMSGU, identifier SYSTEST value [000001,000007] added to rights database
%UAF-E-RDBADDERRU, unable to add SYSTEST_CLIG value [000001,000007] to rights database
-SYSTEM-F-DUPIDENT, duplicate identifier
%UAF-I-NOMODS, no modifications made to system authorization file
%UAF-I-RDBDONEMSG, rights database modified

Creating MODPARAMS.DAT database file, SYS$SYSTEM:MODPARAMS.DAT

For the SCSNODE name - the hostname, if you will - you'll have to remember to use only 6 characters or less. The SCSSYSTEMID is a bit more complex. It's calculated through the formula X * 1024 + Y, where X is the DECnet area number, and Y the DECnet node number in that area.

Since we're starting from scratch, we can use X=1; for the node numbers, it makes sense to think this through properly. Useful tip: use the last octet of the IP address you want to assign to this machine! Since I want to give IPs 10.20.1.131 and 10.20.1.132 to the nodes, I'll give WHISKY Y=131, COGNAC Y=132. This leaves us with a SCSSYSTEMID of 1155 for WHISKY, and DECnet code 1.131. COGNAC will get 1156 as SCSSYSTEMID and DECnet code 1.132.

If you want machines to communicate with others over DECnet, you really, really should communicate with those people on how to decide on DECnet addresses. If you would want to join my cluster, you'll HAVE to use DECnet area 1, and you'll HAVE to use a DECnet node number that's not 131 or 132.

Why? Because DECnet overwrites your MAC address, based on the DECnet address. The 16-bit decimal address is converted to a hexadecimal number and appended to the address AA:00:04:00 in byte-swapped order, with the least significant byte first. For example, DECnet address 12.75 becomes 12363 (base 10), which equals 304B (base 16). After this byte-swapped address is appended to the standard DECnet MAC address prefix, the resulting address is AA:00:04:00:4B:30 (thanks to Cisco for that brief explanation). And, in the end, we really don't want to have two identical MAC addresses on the same network, do we?

* Please enter the SCSNODE name: WHISKY
* Please enter the SCSSYSTEMID:  1155

After the installation finishes, you might want to do one or more of the
following tasks:

o DECOMPRESS THE SYSTEM LIBRARIES - To save space, many of the system
libraries are shipped in a data-compressed format.  If you have
enough disk space, you can decompress the libraries for faster access.
To data expand the libraries, type:

$ @SYS$UPDATE:LIBDECOMP.COM

If you do not decompress these libraries, you will experience
slower response to the HELP and LINK commands.

o BUILD A STANDALONE BACKUP KIT - You can build a standalone backup kit
using the procedure described in the "Backup Procedures" chapter of
tye upgrade and installation supplement provided for your VAX computer.

o TAILOR THE SYSTEM DISK - You might want to review the files provided or
not provided during this installation.  If you find there are files
you want to remove from the system disk (TAILOR OFF) or files you want
to add (TAILOR ON), use the following utilities to perform the
desired tailoring.

OpenVMS tailoring:          $ RUN SYS$UPDATE:VMSTAILOR

DECwindows tailoring:       $ RUN SYS$UPDATE:DECW$TAILOR

NOTE:  The tailor procedure cannot be used to TAILOR ON or TAILOR OFF
files located on an alternate disk.

=================================================================
Continuing with OpenVMS VAX V7.3 Installation Procedure.

Configuring all devices on the system ...

The question will come up whether or not you want to register licenses (PAKs) now. I always say 'no' here; it's much, much easier to copy/paste the licenses from a different terminal to the SIMH console, than it is to manually type in all those license checksums.

* Do you want to register any Product Authorization Keys? (Y/N): N

********************************************************************************

After the system has rebooted you must register any Product
Authorization Keys (PAKs) that you have received with this kit.
You can register these PAKs by executing the following procedure:

$ @SYS$UPDATE:VMSLICENSE

See the OpenVMS License Management Utility Manual for any additional
information you need.

********************************************************************************

Enter your timezone. For me, that would be 37: WET, and we're in daylight savings time now.

%UTC-I-UPDTIME, updating Time Zone information in SYS$COMMON:[SYSEXE]

Configuring the Local Time Zone

TIME ZONE SPECIFICATION -- Main Time Zone Menu

1) Australia       11) GMT             21) Mexico          31) Turkey
2) Brazil          12) Greenwich       22) NZ              32) UCT
3) CET             13) Hong Kong       23) NZ-CHAT         33) US
4) Canada          14) Iceland         24) Navajo          34) UTC
5) Chile           15) Iran            25) PRC             35) Universal
6) Cuba            16) Israel          26) Poland          36) W-SU
7) EET             17) Jamaica         27) ROC             37) WET
8) Egypt           18) Japan           28) ROK             38) Zulu
9) Factory         19) Libya           29) Singapore
10) GB-Eire         20) MET             30) SystemV

0) None of the above

Select the number above that best describes your location: 37

You selected WET as your time zone.

Is this correct? (Yes/No) [YES]:

Default Time Differential Factor for standard time is 0:00.
Default Time Differential Factor for daylight saving time is 1:00.


The Time Differential Factor (TDF) is the difference between your
system time and Coordinated Universal Time (UTC).  UTC is similar
in most repects to Greenwich Mean Time (GMT).

The TDF is expressed as hours and minutes, and should be entered
in the hh:mm format.  TDFs for the Americas will be negative
(-3:00, -4:00, etc.); TDFs for Europe, Africa, Asia and Australia
will be positive (1:00, 2:00, etc.).

Is Daylight Savings time in effect? (Yes/No): YES

Enter the Time Differential Factor [1:00]:

NEW SYSTEM TIME DIFFERENTIAL FACTOR = 1:00.

Is this correct? [Y]:

At this point, the last configurations will be done, and the system will auto-reboot. Installation: finished. After it comes back up, log in with SYSTEM and the password you provided before. You'll be greeted with a nice warning about licenses; let's get those sorted out first.

%LICENSE-I-NOLICENSE, no license is active for this software product
%LOGIN-S-LOGOPRCON, login allowed from OPA0:
Welcome to OpenVMS (TM) VAX Operating System, Version V7.3

$

A license (PAK) looks like this:

$ LICENSE REGISTER VAX-VMS - 
/ISSUER=OPENVMS_HOBBYIST - 
/AUTHORIZATION=DECUS-BEL-006150000-1700000 - 
/PRODUCER=DEC - 
/UNITS=0 - 
/TERMINATION_DATE=30-JAN-2013  - 
/ACTIVITY=A - 
/OPTIONS=(NO_SHARE) - 
/CHECKSUM=2-EIDC-LMLD-EJGL-XXXX

Notice that the license itself is provided as a DCL "LICENSE" command line? You can just copy the license, paste it in your prompt, and it'll be registered. And you can copy/paste the entire file there - it'll take a while to get through, but it'll work. Another option is to only paste the basic VAX-VMS and networking licenses, and transfer the license file to the machine over the network, after which you can simply run it as a command file.

After licenses have been entered, MODIFY the VAX-VMS license to include your nodename, then LOAD them all.

$ LICENSE MODIFY/INCLUDE=WHISKY VAX-VMS
$ LICENSE LOAD

Something I tend to do after this step is decompressing the libraries, which speeds things up quite a bit.

$ @SYS$UPDATE:LIBDECOMP.COM

                 OpenVMS Library Decompression Utility

     1  HELPLIB.HLB          15  SDA.HLB              28  PHONEHELP.HLB         
     2  STARLET.OLB          16  SHWCLHELP.HLB        29  LIB.MLB               
     3  ACLEDT.HLB           17  SYSGEN.HLB           30  STARLET.MLB           
     4  ANLRMSHLP.HLB        18  ANALAUDIT$HELP.HLB   31  STARLETSD.TLB         
     5  DBG$HELP.HLB         19  SYSMANHELP.HLB       32  SYS$STARLET_C.TLB     
     6  DBG$UIHELP.HLB       20  TFF$TFUHELP.HLB      33  ERFLIB.TLB            
     7  DISKQUOTA.HLB        21  EXCHNGHLP.HLB        34  VAXCCURSE.OLB         
     8  EDFHLP.HLB           22  TPUHELP.HLB          35  VAXCRTL.OLB           
     9  INSTALHLP.HLB        23  EVE$HELP.HLB         36  VAXCRTLG.OLB          
    10  LATCP$HELP.HLB       24  EVE$KEYHELP.HLB      37  IMAGELIB.OLB          
    11  MAILHELP.HLB         25  UAFHELP.HLB          38  DECCCURSE.OLB         
    12  MNRHELP.HLB          26  TECO.HLB             39  DECCRTL.OLB           
    13  EDTHELP.HLB          27  PATCHHELP.HLB        40  DECCRTLG.OLB          
    14  NCPHELP.HLB                                                             

          A  ALL libraries to be decompressed
          E  EXIT this procedure

* Enter letter or number(s) of libraries to be decompressed
  (Separate multiple entries with a comma)a

Just let it do all libraries - yes, it'll fit in that 1.2GB easily. After a bit it's done, and there we are; ready to setup VMS networking.

VMS Networking

Before configuring DECnet and TCPIP, a couple of system parameters need to be changed. This can be done via the MODPARAMS.DAT file, which is read by AUTOGEN. While not all of the parameters you'll want to change require a reboot, some do; in the end, I tried to group some of the parameters together, and reboot anyway for good measure.

$ SET DEF SYS$SYSTEM
$ EDIT MODPARAMS.DAT

Add the following parameters to the end of the file, CTRL-Z to save and exit, and then run AUTOGEN:

ADD_GBLPAGES=10000
ADD_GBLSECTIONS=100
ADD_NPAGEDYN=800000
ADD_NPAGEVIR=800000
MIN_SPTREQ=6000
INTSTKPAGES=20

9 lines written to file SYS$SYSROOT:[SYSEXE]MODPARAMS.DAT;2

$ @SYS$UPDATE:AUTOGEN GETDATA REBOOT NOFEEDBACK

Next up: run the DECnet configuration utility. The defaults are usually all fine; you just need to enter your DECnet address that you decided on earlier.

$ @SYS$MANAGER:NETCONFIG.COM

DECnet for OpenVMS network configuration procedure

This procedure will help you define the parameters needed to get DECnet
running on this machine.  You will be shown the changes before they are
executed, in case you wish to perform them manually.

What do you want your DECnet node name to be?        [WHISKY]:
What do you want your DECnet address to be?                  : 1.131   
Do you want to operate as a router?         [NO (nonrouting)]:
Do you want a default DECnet account?                    [NO]:
Do you want a default account for the MAIL object?      [YES]:
Do you want a default account for the FAL object?        [NO]:
Do you want a default account for the PHONE object?     [YES]:
Do you want a default account for the NML object?       [YES]:
Do you want a default account for the MIRROR object?    [YES]:
Do you want a default account for the VPM object?       [YES]:

It will spew out a LOT of DCL that will execute if you confirm. Since we're starting from brand new operating systems, there's little point in not accepting them, so just give it a YES.

Do you want these commands to be executed?  [YES]:

Confirm to have DECnet started manually this time (supposing you really did input and load that license), and watch the magic happen!

The changes have been made.

If you have not already registered the DECnet-VAX key, then do so now.

After the key has been registered, you should invoke the procedure
SYS$MANAGER:STARTNET.COM to startup DECnet-VAX with these changes.

(If the key is already registered) Do you want DECnet started?  [YES]:

%%%%%%%%%%%  OPCOM  18-JAN-2012 20:37:24.45  %%%%%%%%%%%
Message from user DECNET on WHISKY
DECnet starting

%RUN-S-PROC_ID, identification of created process is 00000211
%%%%%%%%%%%  OPCOM  18-JAN-2012 20:37:24.98  %%%%%%%%%%%
Message from user DECNET on WHISKY
Error opening permanent proxy database

%NCP-I-NOINFO, No information in database
%RUN-S-PROC_ID, identification of created process is 00000213
$
%%%%%%%%%%%  OPCOM  18-JAN-2012 20:37:29.20  %%%%%%%%%%%
Message from user DECNET on WHISKY
DECnet event 4.10, circuit up
From node 1.131 (WHISKY), 18-JAN-2012 20:37:26.89
Circuit QNA-0

Sweet! DECnet is running. Once you are at this point for both machines, you'll want to sync the node databases:

$ RUN SYS$SYSTEM:NCP
NCP> COPY KNOWN NODES FROM 1.132 TO BOTH
NCP> EXIT

It's time to install TCPIP now. This is also located on the VMS iso, which is DUA2 in VMS:

$ MOUNT/OVER=ID DUA2

%MOUNT-I-WRITELOCK, volume is write locked
%MOUNT-I-MOUNTED, VAXVMS073 mounted on _WHISKY$DUA2:

$ SET DEF DUA2:[TCPIP_VAX051.KIT]
$ PRODUCT INSTALL *

The following product has been selected:
DEC VAXVMS TCPIP V5.1-15               Layered Product

Do you want to continue? [YES]

When asked, tell it to stick with defaults, and that there's no need to review these defaults.

Configuration phase starting ...

You will be asked to choose options, if any, for each selected product and for
any products that may be installed to satisfy software dependency requirements.

DEC VAXVMS TCPIP V5.1-15: Compaq TCP/IP Services for OpenVMS.
(c) Compaq Computer Corporation 2000. All Rights Reserved.
Compaq Computer Corporation
Compaq TCP/IP Services for OpenVMS offers several license options.

Do you want the defaults for all options? [YES]
Do you want to review the options? [NO]

Execution phase starting ...

The following product will be installed to destination:
DEC VAXVMS TCPIP V5.1-15               DISK$WHISKYSYS:[VMS$COMMON.]

Portion done: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%

%PCSI-I-PRCOUTPUT, output from subprocess follows ...
% - HELP has been updated. You may purge SYS$COMMON:[SYSHLP]HELPLIB.HLB
%
%PCSI-I-PRCOUTPUT, output from subprocess follows ...
% TCPIP-W-PCSI_INSTALL
% - Execute SYS$MANAGER:TCPIP$CONFIG.COM to proceed with configuration of
%   Compaq TCP/IP Services.
%
Portion done: 100%

The following product has been installed:
DEC VAXVMS TCPIP V5.1-15               Layered Product

DEC VAXVMS TCPIP V5.1-15: Compaq TCP/IP Services for OpenVMS.

Check the release notes for current status of the product.

Once installed, run the TCPIP configuration utility:

$ @SYS$MANAGER:TCPIP$CONFIG

I don't think it's necessary to review all IP configuration options here - you know pretty well what kind of options you want for networking. But since we did the smart thing and plan to use the same numbers for DECnet and IP, we'll give each node 10.20.[DECnet address] as IP. Handy to remember, and it tends to avoid problems when the cluster grows with other people's nodes.

Wrapping up and testing

Once you've completed your IP setup (and verified it's working; try telnetting to your machine) all that is left to do is making sure that both DECnet and TCPIP are properly started at boot time.

$ EDIT SYS$MANAGER:SYSTARTUP_VMS.COM

Add the following two lines just before EXIT, at the end of the file:

$ @SYS$MANAGER:STARTNET.COM
$ @SYS$STARTUP:TCPIP$STARTUP

Make sure not to put them in the other way round; DECnet really needs to do its MAC address black magic first, and only then TCPIP can kick in. Reboot the machine, and check if all goes well.

$ REBOOT

After both VAX instances are back up, try telnetting to the machines. Log in to both, and try PHONE'ing to yourself from one node to the other!

$ PHONE COGNAC::SYSTEM

Look in COGNACs terminal... ha!

COGNAC::SYSTEM is phoning you on WHISKY::     (09:29:24)
$ PHONE ANSWER

PHONE: VMS' instant messaging over DECnet. :-)

                            OpenVMS Phone Facility                  27-JAN-2012
%
-------------------------------------------------------------------------------
                                WHISKY::SYSTEM
Do be do be do



-------------------------------------------------------------------------------
                                COGNAC::SYSTEM
You bet.



-------------------------------------------------------------------------------

And with that, you're ready to play with your two new VMS systems. You can add new users to the system, but I would recommend not doing that until you have a fully working cluster. If you want to anyway, here's how to create a new user with 'full' privileges:

$ SET DEFAULT SYS$SYSTEM
$ RUN AUTHORIZE
UAF> ADD ALVER/PASSWORD=TEMP/OWNER="Lennert Van Alboom"/DEV=DUA0/DIR=[ALVER]/UIC=[200,201]/FLAG=NODISUSER/PRIV=ALL
%UAF-I-PWDLESSMIN, new password is shorter than minimum password length
%UAF-I-ADDMSG, user record successfully added

*IF* you create a user, make sure to create his/her homedirectory and set the proper permissions:

$ CREATE/DIRECTORY DUA0:[ALVER]
$ SET DIRECTORY/OWNER=ALVER DUA0:[ALVER]

Coming up later: joining the two VAX-VMS machines in a VAXcluster. Enjoy!

Credits & thanks

Large parts of this howto are based on the RetroComputing SIMHNetworking information and Running VAX/VMS Under Linux Using SIMH by Phillip Wherry.

I wouldn't have gotten anywhere without the countless tips and VMS insights of Steve "Hoff" Hoffman, the Deathrow VMS cluster people and everyone in #vms on irc.2600.net. Thanks!

Lennert Thu, 26/01/2012 - 23:19

HPUX and the Quest for a Modern Desktop

Lennert — Sat, 12 Feb 2011 19:56:09 +0000

HPUX and the Quest for a Modern Desktop

Over the years, there have been a good number of decent RISC workstations: DEC, IBM, SGI, Sun and HP all had their time. Nowadays there's hardly any left - pretty much all of the large UNIX manufacturers' workstations have been discontinued. I still have mine though, and I like them too much to let go already.

Open source software offers a way to get decent, modern applications on an older machine: if you have the source, you can build it. But building every bit of software from source, tracking down dependencies, takes a lot of effort and time... or does it? Here's where pkgsrc comes into play. New quest unlocked: turn HPUX 11.11 in a desktop that is usable for day-to-day tasks.

The NetBSD Packages Collection (pkgsrc) is a framework for building third-party software on NetBSD and other UNIX-like systems, currently containing over 8000 packages. It is used to enable freely available software to be configured and built easily on supported platforms. Now, supported and easy are terms that will be filled in differently depending on whom you ask, but... why not give it a shot?

This is the machine I've been using:

It's a HP C8000 workstation, sporting two dual-core PA8900 "Shortfin" CPUs at 1.1GHz, 4GB RAM, four U320 SCSI disks and three gigabit ethernet ports. It is configured with HPUX 11.11, patched to the last goldpak release (12/2009). Graphics have been setup, and CDE is working on it (ahhh, CDE nostalgia...). Both OS and data disks are in a mirrored LVM setup, so both performance and safety should be okay.

First I downloaded and unpacked the gcc compiler, conveniently prebuilt for HPUX, which you can find here - many thanks to Tobias Nygren for making this possible! After unpacking gcc, one has to run the mkheaders script:

$ /usr/local/gcc4/libexec/gcc/hppa2.0w-hp-hpux11.11/4.2.3/install-tools/mkheaders

Add /usr/local/gcc4/bin to your $PATH, and make sure that your $CC is set to gcc. If your OS has the needed bundles to do development - HP C/aC++ Developer's Bundle comes to mind; if you're reading this article, you're probably familiar with getting HPUX in development-ready shape - you should be good to go. Next was downloading pkgsrc itself. I picked the most recent "stable" release at the time of writing, which was pkgsrc-2010Q4. Beware; untarring this can be problematic. Standard UNIX tar cannot handle the length of paths inside the tarball, so you may want to untar on a machine that has GNU tar (like Linux, BSD, ...). I created a dedicated LV of 8GB for this pkgsrc tree, mounted it on /mnt/alverware-2010q4, and created a symlink:

$ ln -s /mnt/alverware-2010q4 /alverware

This will allow me to easily switch between different (stable, development, ...) pkgsrc releases; all it takes is changing that symlink. The newly unpacked pkgsrc tree was put under /alverware/pkgsrc. I'll keep my own structure throughout this article; it speaks for itself that you can put it wherever you like.

Bootstrapping pkgsrc turned out to have a small glitch: the unit-tests for bmake don't like HPUX much (or don't like *my* HPUX much... can anyone confirm?). Quick and dirty solution is to edit the Makefile.in to exclude unit-tests altogether.

$ vi /alverware/pkgsrc/devel/bmake/files/Makefile.in

Comment out the last line (starting with "cd ${.CURDIR}/unit-tests"). Next, you can bootstrap pkgsrc for HPUX, using gcc as default compiler and with a 32bit ABI. Don't use a 64bit ABI; it'll work partially, but subtle errors will appear (python, for an example, will not function at all). You'll also notice the --unprivileged option: this allows pkgsrc to run the "bmake install" phase as nonroot user, which I prefer a lot over giving the root password every five minutes. All files will be built under /alverware/pkg.

$ cd /alverware/pkgsrc/bootstrap 
$ ./bootstrap --prefix /alverware/pkg --pkgdbdir /alverware/pkg/var/db/pkg --sysconfdir /alverware/pkg/etc --compiler gcc --abi 32 --unprivileged

After roughly 10 minutes (on my machine), the bootstrap will have completed succesfully, and you will be back at the prompt. Update the vulnerabilities database:

$ /alverware/pkg/sbin/pkg_admin -K /alverware/pkg/var/db/pkg fetch-pkg-vulnerabilities

You can now start to tweak your mk.conf - the master configuration file for pkgsrc - to take advantage of your hardware, and add a few extra options:

.ifdef BSD_PKG_MK               # begin pkgsrc settings

ABI=                            32 
PKGSRC_COMPILER=                gcc
USE_NATIVE_GCC=                 yes
MAKE_JOBS=                      8 
   
CFLAGS+=                        -D__STDC_EXT__ -D_XOPEN_SOURCE_EXTENDED -D_INCLUDE_XOPEN_SOURCE_520 -D_INCLUDE_HPUX_SOURCE -D_INCLUDE__STDC__ -D_INCLUDE__STDC_A1_SOURCE -D_REENTRANT
   
UNPRIVILEGED=                   yes
PKG_DBDIR=                      /alverware/pkg/var/db/pkg
LOCALBASE=                      /alverware/pkg
VARBASE=                        /alverware/pkg/var
PKG_TOOLS_BIN=                  /alverware/pkg/sbin
PKGMANDIR=                      man
PYTHON_VERSION_DEFAULT=         26 
PYPACKAGE=                      python26
PASSIVE_FETCH=                  0  
   
TOOLS_PLATFORM.install?=        /alverware/pkg/bin/install-sh
TOOLS_PLATFORM.awk?=            /alverware/pkg/bin/nawk
TOOLS_PLATFORM.sed?=            /alverware/pkg/bin/nbsed
 
.endif                          # end pkgsrc settings

Notice the USE_NATIVE_GCC which prevents pkgsrc software from pulling in pkgsrc' own gcc as a dependency, and the biggest time saver of all: MAKE_JOBS. I have it set to 8 (as in make -j8); two jobs per cpu core seems to give decent results here. Next on the list is building GNU tar (didn't you wish you had that before?):

$ cd /alverware/pkgsrc/archivers/gtar
$ bmake && bmake install && bmake clean && bmake clean-depends

This should take quite a while - it will pull in a gigantic heap of dependencies of all kinds. It should complete without problems, after which you can add the following line to mk.conf:

TOOLS_PLATFORM.tar?= /alverware/pkg/bin/gtar

Once this is done you can start building as you wish. I'll document my own building track here, including various problems encountered while building, and a fix if available. Do note that I am NOT a developer, and my fixes may be quicker and dirtier than you'd like. A proper developer might be able to figure out the actual root cause (if I couldn't), and create a proper patch for upstream (which I can't).

Following the fixes documented below, I'm now working happily from my HPUX desktop. The following packages are installed on my system (and new ones are being added daily):

$ ls /alverware/pkg/var/db/pkg
aalib-1.4.0.5nb2                libart-2.3.21                   mkfontdir-1.0.5
aterm-1.0.0nb8                  libast-0.7nb3                   mkfontscale-1.0.7
atk-1.32.0                      libaudiofile-0.2.7              mng-1.0.10nb2
autoconf-2.68                   libcfg+-0.6.2nb3                monafonts-ttf-2.90
automake-1.11.1nb1              libcroco-0.6.2                  mplayer-1.0rc20100913nb2
autoswc-1.6                     libdvdnav-4.1.3                 mplayer-fonts-20030714nb1
babl-0.1.2                      libdvdread-4.1.3                mplayer-share-1.0rc20100913
bash-4.1nb1                     libdv-1.0.0                     ms-ttf-20020306nb5
bdftopcf-1.0.2                  libexif-0.6.20                  p5-gettext-1.05nb5
bigreqsproto-1.1.0              libffi-3.0.9                    p5-XML-Parser-2.40
bison-2.4.3                     libfontenc-1.0.5                pango-1.28.3
bmake-20100808                  libgcrypt-1.4.6                 pax-20080110
bootstrap-mk-files-20090807nb2  libgetopt-1.4.4                 pcre-8.11
bzip2-1.0.6                     libgpg-error-1.10               perl-5.12.2nb1
cabextract-1.3                  libICE-1.0.6                    pidgin-2.7.3nb2
cairo-1.10.2                    libiconv-1.13.1                 pixman-0.18.4
compat_headers-0.2              libidn-1.19                     pkgdb.byfile.db
compositeproto-0.4.1            libmad-0.15.1bnb1               pkg-config-0.25nb1
db4-4.8.30                      libogg-1.2.1nb1                 pkg-vulnerabilities
dejavu-ttf-2.32                 libpaper-1.1.24                 pkg_install-20101212
desktop-file-utils-0.15         libpurple-2.7.3nb2              pkg_install-info-4.5nb3
digest-20080510                 librsvg-2.32.1nb1               png-1.4.5
encodings-1.0.3                 libSM-1.1.1nb1                  popt-1.16
esound-0.2.41                   libtasn1-2.9                    pornview-0.2.0pre1nb7
eterm-0.9.5nb3                  libtheora-1.1.1nb1              printproto-1.0.4
expat-2.0.1nb2                  libtool-base-2.2.6bnb4          psutils-1.17nb2
gawk-3.1.8                      libungif-4.1.4nb1               py26-expat-0nb4
gdb-5.3nb5                      libvorbis-1.3.2                 py26-xcbgen-1.6nb1
gdk-pixbuf2-2.22.1              libwmf-0.2.8.4nb8               python26-2.6.6nb5
gegl-0.1.2nb3                   libX11-1.3.5                    randrproto-1.3.1
gettext-lib-0.14.6              libXau-1.0.6                    readline-6.1
gettext-tools-0.14.6nb1         libXaw-1.0.7                    renderproto-0.11
ghostscript-8.71nb6             libxcb-1.7                      resourceproto-1.1.0
ghostscript-fonts-8.11nb3       libXcomposite-0.4.2             screen-4.0.3nb4
gimp-2.6.11nb2                  libXcursor-1.1.10               scrnsaverproto-1.2.0
glib2-2.26.1nb1                 libXdmcp-1.0.3                  SDL-1.2.14nb1
glu-7.4.4nb2                    libXext-1.1.1                   shared-mime-info-0.90
gmake-3.82nb1                   libXfixes-4.0.5                 startup-notification-0.10
gmplayer-1.0rc20100913nb2       libXfont-1.4.2                  tiff-3.9.4nb1
gnutls-2.10.4                   libXft-2.1.14                   tnftp-20070806
gperf-3.0.4                     libXinerama-1.1                 unzip-6.0
groff-1.20.1nb2                 libXi-1.3.2                     vera-ttf-1.10nb4
gtar-1.25                       libxml2-2.7.8nb2                videoproto-2.3.1
gtar-base-1.25nb3               libXmu-1.0.5                    vim-7.2.446
gtar-info-1.25                  libXpm-3.5.8                    vim-share-7.2.446
gtexinfo-4.13                   libXp-1.0.0                     watch-3.2.6nb1
gtk2+-2.22.1                    libXrandr-1.3.0                 xcb-proto-1.6
ilmbase-1.0.2nb1                libXrender-0.9.6                xcb-util-0.3.6
imake-1.0.3                     libXres-1.0.4                   xcmiscproto-1.2.0
imlib2-1.4.2nb5                 libXScrnSaver-1.2.0             xextproto-7.1.2
inputproto-2.0                  libxslt-1.1.26                  xf86bigfontproto-1.2.0
install-sh-20100824             libXt-1.0.8                     xf86dgaproto-2.1
intltool-0.40.6                 libXvMC-1.0.6                   xf86vidmodeproto-2.3
jasper-1.900.1nb5               libXv-1.0.5                     xineramaproto-1.2
ja-sazanami-ttf-20040629nb1     libXxf86dga-1.1.1               xmlcatmgr-2.2nb1
jpeg-8b                         libXxf86vm-1.1.0                xorg-cf-files-1.0.3
kbproto-1.0.5                   lzo-2.04                        xproto-7.0.18
kochi-ttf-20030809nb5           m4-1.4.15                       xtrans-1.2.5
lcms-1.19                       makedepend-1.0.3                zlib-1.2.3
less-418                        MesaLib-7.4.4nb3

You'll notice that there is NO FIREFOX there. First of all, building firefox is a very, very difficult task, and one I'm not capable of doing. Second, HP is offering a prebuilt depot of Firefox 3.5.9 for HPUX. Their version will undoubtedly be a magnitude faster and more stable than anything I could hack together, so... I don't touch it.

• x11/libX11:

The default options for libX11 have changed since the last pkgsrc release, causing all X11 applications to fail on HPUX' libc poll(). This can be fixed by doing the following edit:

Makefile: 
    Replace CONFIGURE_ARGS+= --with-xcb by CONFIGURE_ARGS+= --without-xcb

After this, I tend to build libX11, which pulls in a lot of X11 dependencies as well.

• devel/gtexinfo:

Dependency for graphics/netpbm → print/ghostscript → devel/ncurses. Ncurses are good; you want them. gtexinfo fails to build with the linker complaining about missing symbols BC, PC and ospeed. Edit the failing code:

$ vi work/*/info/terminal.c

Search for ospeed; you'll notice the three mentioned symbols are defined inside an #ifdef block. Move the definitions out - HPUX hasn't got those symbols, and the code fails to detect that, causing the ifdefs to skip the defines. After this hack, it builds and installs just fine.

• graphics/netpbm:

Dependency for print/ghostscript → devel/ncurses. Before you start the build, edit the top level Makefile, and comment out the following line:

USE_FEATURES= snprintf

This suffices for netpbm to build.

• print/ghostscript:

This piece of software has some ugly problems. First, start the build, and wait until it fails miserably. It's unclear to me why, but somehow the build system doesn't pick up the basic facts that

HPUX does not use .so for its shared libraries, like Linux does, but .sl
HP's own linker does not follow the same syntax as GNU ld, causing ugly files named "name=libgs.so" to be created in the totally wrong place.

To force-override these two vital details, change the following things in this specific file:

work/*/base/unix-dll.mak: 
  GS_SOEXT=so needs to become GS_SOEXT=sl 
  LDFLAGS_SO=-shared -Wl,-soname=$(GS_SONAME_MAJOR) needs to become LDFLAGS_SO=-shared -Wl,+h,$(GS_SONAME_MAJOR)

Once you have done these two edits, it will complete succesfully.

• devel/ncurses:

If you have built the problematic dependencies mentioned above, ncurses should install without a problem. Make sure to clean everything afterwards.

 $ bmake && bmake install && bmake clean && bmake clean-depends

Once it's done, you can add another semi-crucial line to your mk.conf:

 CURSES_DEFAULT= ncurses

• editors/vim:

The only real editor in the world! This doesn't require any fixes, except you need to accept the license in mk.conf by adding the following line:

 ACCEPTABLE_LICENSES+= vim-license

At this point, I also included a big default options selection:

 PKG_DEFAULT_OPTIONS= -hal -fam -daap -evolution -dbus -avahi -farsight -gstreamer -gnome -gtkspell -arts -pulseaudio -nas -x264 ncurses

This enables or disables all dependencies mentioned above for all applications to be built, supposing an application offers the option of enabling/disabling it.

• multimedia/libdvdnav:

Dependency for multimedia/mplayer. This will fail during configure due to an obscure pthreads detection bug which seems to have haunted OSX as well. Pthreads shouldn't be an issue, so we'll force configure to continue, regardless of what it says:

work/*/configure: 
    Search for the following code: return pthreads_create(); return 0; 
    Replace by: return 0;

Dirty? Yes. But it'll continue and finish building.

• devel/SDL:

Dependency for multimedia/mplayer. Start the build with bmake, and wait for it to fail, then edit:

 work/*/include/SDL_stdinc.h: remove the "strings.h" include

Restart the build. Fixed.

• multimedia/mplayer:

Start the build with bmake, and wait for it to fail. Edit:

work/*/config.h: add the following lines: 
    #define lrintf(x) ((int)(rint(x)) 
    #define __open64 open 
    #define SIZE_MAX ((size_t)-1) 
    #define atoll(x) strtoll(x, (char **)NULL, 10) 
    #define ldexpf(x, y) (float)ldexp(x, y) 
work/*/libavutil/libm.h: comment out the following function entirely: 
    static av_always_inline av_const long int lrintf(float x)

After these edits, it'll build. And it works too:

MUPPET SHOW ON HPUX! I bet that's a world first, right here.

• wm/fluxbox:

Fluxbox acts a bit weird when being built in an unicode shell on UX. No idea why, but I had to start with:

$ LC_ALL=C bmake

Even then it fails on the polish translation, so I removed it, plus another quick hack to keep the UX linker happy:

nls/Makefile: 
    remove pl_PL from building (sorry, polish people) 

src/Makefile: 
    change "LDADD = FbTk/libFbTk.a FbTk/LogicCommands.o defaults.$(OBJEXT)" to "LDADD = FbTk/libFbTk.a defaults.$(OBJEXT)"

After these edits, rerun bmake and it'll be fine. Now, you might wonder, how do you get HPUX' login manager to start fluxbox instead of CDE? The trick is setting xdmMode in the dt configfile, then restart it, and having a good .xsession:

$ grep xdmMode /etc/dt/config/Xconfig Dtlogin*xdmMode: True 
$ /sbin/init.d/dtlogin.rc reset 
$ cat .xsession exec /alverware/pkg/bin/startfluxbox 
$ chmod 700 .xsession

... and next time you login, you get a nice fluxbox desktop. Supposing you don't want to sit behind your HPUX screen all day (building pkgsrc takes some time after all), you can test your X11 desktop using Xnest from a Linux or BSD machine:

$ Xnest -depth 24 -geometry 1024x768 -name Bladnoch -query Bladnoch :1

This will get you the HPUX desktop via XDMCP in a window, like the screenshot above.

• graphics/ilmbase:

Dependency for graphics/gegl → graphics/gimp. .This contains some lowlevel math routines, some of which depend on math.h functions that aren't available on PARISC HPUX (they apparently are on IA64, but that doesn't help us, does it?). Edit:

work/*/Imath/ImathMath.h: Find the function defines and change them to match this: 
    static float modf (float x, float *y) {return (float)::modf ((double)x, (double*)y);} 
    static float ceil (float x) {return (float)::ceil (x);} 
    static float floor (float x) {return (float)::floor (x);} 
    static float hypot (float x, float y) {return (float)::hypot (x, y);}

• graphics/gimp:

Gimp builds fairly easy, although it'll take a while due to the whole of X11 libs and GTK2 being pulled in. The build will eventually fail in gimp itself with a linker error; symbol "finite" is unknown. Edit:

work/*/config.h: Add a new define: 
    #define finite(x) isfinite(x)

Restart the build, and it'll succeed.

• x11/aterm:

Aterm has a hardcoded utmp reference in pkgsrc, which we have to remove in order to be able to build:

Makefile: 
    remove the "--enable-utmp" reference

Just like with fluxbox, I had to start with:

$ LC_ALL=C bmake

After that you're all set.

• misc/watch:

Silly little tool, but I still like it. It'll fail when building, which you can fix easily:

work/*/watch.c: replace strtof(x) by (float)strtod(x)

• security/libgcrypt:

Dependency for security/gnutls → chat/libpurple. It has a very subtle problem which, if you don't fix it upfront, will cause anything gnutls related to fail. Start the build, but stop during the configure phase, then edit:

/work/*/random/rndunix.c: add #include

• chat/libpurple:

Dependency for chat/pidgin and chat/finch. You'll have to explicitly enable gnutls to get SSL support, and perl for non-C plugins:

PKG_OPTIONS.libpurple= gnutls perl

It builds cleanly except for one protocol:

protocols/gg/lib/libgadu.h: 
    find the code block that defines int8_t; 
    expand the "#if !defined(__CYGWIN__) && !defined(__SunOS)" 
    with "&& !defined(_INCLUDE_HPUX_SOURCE)"

Once this is done, you can continue building pidgin.

• chat/finch:

Something in pkgsrc and/or finch doesn't agree with wide character support. Getting finch to build on my machine demanded a very ugly hack:

Makefile: 
     set --with-ncurses-headers=/alverware/pkg/include/ncursesw 

/alverware/pkg/include/ncursesw/ncurses.h: 
    add "typedef unsigned int wint_t;"

Looking through the ncursesw code, it looks as if something went wrong - large and vital parts of the code are between "if 0" blocks, effectively eliminating them.

• textproc/rarian:

This build will error out on a duplicate definition of strrstr. Remove the one provided by rarian itself:

work/*/librarian/rarian-man.c: 
    comment out entire function "static char *strrstr (char *s, char *wanted)".

• net/ORBit2:

This package uses IPv6 calls, but doesn't add the proper linker flag. Edit:

Makefile: 
    add LDFLAGS+= -lipv6

You do need the IPv6 bundle for this, naturally! You can get the bundle from HP for free at this page.

• sysutils/dbus:

Some things really do depend on dbus, so what the heck... just build it. Build will fail on missing symbol "vsyslog"; this is easily fixed:

work/*/dbus/dbus-sysdeps-util-unix.c: 
    replace all invocations of "vsyslog" by "syslog"

• multimedia/gstreamer-0.10:

I would have preferred not building this, but some things had hardcoded dependencies on it. Anyway, to get it to build, the following define is needed:

work/*/config.h: 
    add #define MSG_DONTWAIT 0

• textproc/gnome-doc-utils:

It's fairly ironic, but the thing in gnome-doc-utils that fails is... the docs. I guess we can live without the docs for gnome-doc-utils, can't we?

work/*/Makefile: 
    remove 'doc' from DIST_SUBDIRS and SUBDIRS variables

• sysutils/eggdbus:

This package feeds bad GNU linker options to HP's ld. Won't work, of course...

work/*/src/eggdbus: 
    remove "-Wl,--as-needed" from LDFLAGS 

work/*/src/tests: 
    remove "-Wl,--as-needed" from LDFLAGS

• security/libgnome-keyring:

Not much to do here; one define for a differently named symbol:

work/*/config.h: 
    add #define MAP_ANON MAP_ANONYMOUS

• textproc/enchant:

Even though size_t and gsize seem to have the same type, automatic casting doesn't work too well. Therefor a specific cast:

work/*/src/myspell/myspell_checker.cpp:
    164: size_t result = g_iconv(m_translate_in, &in, (gsize*)&len_in, &out, (gsize*)&len_out);
    190: size_t result = g_iconv(m_translate_in, &in, (gsize*)&len_in, &out, (gsize*)&len_out);
    206: if ((size_t)-1 == g_iconv(m_translate_out, &in, (gsize*)&len_in, &out, (gsize*)&len_out)) {

• editors/abiword:

To get abiword to build, quite a few fixes are needed. First of all, abiword defines a symbol which already exists in HPUX. We're not using this native symbol, so we undefine it first to allow it to be defined without error.

work/*/config.h: 
    add #undef m_unit

The 'main' makefile contains a few references to a linker option that is not valid for HP's ld, only GNU. Remove it:

work/*/src/Makefile: 
    remove all references to "--fno-undefined"

Another makefile has an issue with the plugins: we are not building any plugins, yet it wants to do a for-loop over them. Fails badly:

work/*/src/plugins/Makefile: 
    remove all "for plugins in ... done" blocks

Next, we get to apply the same size_t/gsize fix we had before:

work/*/src/af/util/xp/ut_iconv.cpp: line 314: 
    return g_iconv((GIConv)cd, (char **)inbuf, (gsize*)inbytesleft, outbuf, (gsize*)outbytesleft);

Finally, we have one file that uses strtof, which we haven't got:

work/abiword-2.8.6/src/wp/ap/gtk/ap_UnixDialog_Tab.cpp: 
    replace all instances of "strtof" by "(float)strtod"

EDIT: even though this builds, it doesn't actually run - it's lacking a symbol. Will look into it. For the time being it's not worth building this.

• net/libtorrent:

Libtorrent depends on a syscall mincore which indicates whether or not a specific page is in memory. This apparently strongly reduces page faults, boosting performance. HPUX hasn't got it, so we'll have to build without it.

Makefile: 
    add --disable-mincore to CONFIGURE_ARGS

Next, we run into another example of an application redefining a system macro:

work/*/rak/priority_queue_default.h: 
    add #undef m_slot

• net/rtorrent:

Some symbols are only defined if the proper compiler flag (_REENTRANT) is given on HPUX. At this point, I've added it to PKG_DEFAULT_OPTIONS (see above). Next, two symbols are missing on HPUX, and m_slot needs to be undefined again:

work/*/config.h: 
    #define AF_LOCAL AF_UNIX 
    #define PF_LOCAL PF_UNIX 

/work/*/rak/priority_queue_default.h, /work/*/src/option_parser.cc, /work/*/src/display/text_element_string.h: 
    #undef m_slot

Lennert Sat, 12/02/2011 - 21:56

MediaTomb and PS3: The Lazy Man's solution (Part II)

Lennert — Sun, 08 Aug 2010 17:40:11 +0000

MediaTomb and PS3: The Lazy Man's solution (Part II)

A while back I did an article on a catch-all solution for streaming content to the Playstation 3 using MediaTomb and a big transcoding script. It served me well: every movie I had - be it avi, mkv, mp4, dvd iso, RealMedia, ... was properly handled. I didn't care for HD content, mainly because my TV was a plain old CRT tube and couldn't tell the difference between a 720x480 mpeg2 and a 1080p h.264 movie anyway.

Last month I did get me a HDTV finally, and my original transcoding script wasn't ready for it.

Last week I finally got me a Boxee Box. Since this thing does everything I want and more when it comes to movies and series, I don't have any use for mediatomb anymore. So, while I'll keep the articles online... I mostlikely won't be making any changes/bugfixes to the script anymore :-)

In hindsight, there were two crucial things wrong with my original setup:

I didn't check which codecs were used in the source file. Everything was transcoded to MPEG-2, even if the source codec was supported by the PS3.
Transcoding HD content was problematic: transcoding 720p worked (barely!), but 1080p was plain impossible, cpu-wise.

So I had two major points to fix: first of all checking the source file's internals, and related to that, remuxing where possible instead of transcoding. It took me a few days of trial-and-error, but this is the result for now:

#!/bin/bash
#
# General all-covering MediaTomb transcoding script.
#
#############################################################################
# Edit the parameters below to suit your needs.
#############################################################################

# Subtitles imply transcoding; set to 1 to disable subtitle rendering.
# For divx this doesn't matter much but for mp4, mkv and DVD it does.
DISABLESUBS=1

# Change this to enable different DVD subtitle languages.
SUBS="nl,en"

# Change this line to set the average bitrate.
# Use something like 8000 for wired connections; lower to 2000 for wireless.
AVBIT=8000

# Change this line to set the maximum bitrate.
# Use something like 12000 for wired connections; lower to 5000 for wireless.
MVBIT=12000

# Change this line to set the MPEG audio bitrate in kbps. AC3 is fixed to 384.
AABIT=256

# Change this line to set your favourite subtitle font.
SFONT="/etc/mediatomb/DejaVuSans.ttf"

# Change this line to set the size of subtitles. 20-25 is okay.
SUBSIZE=20

# Enable downscaling of transcoded content to 720 pixels wide (DVD format)?
DOWNSCALE=1

# If downscaling is enabled, anything over this width (pixels) will be downscaled.
MAXSIZE=900

# Enable logging to file?
LOGGING=1

# If logging is enabled, log to which file?
LOGFILE="/var/log/mediatomb-transcode.log"

#############################################################################
# Do not change anything below this line.
#############################################################################
# Variables
#############################################################################

FILE=$1
VERSION="0.12"

MENCODER=$(which mencoder)
MEDIAINFO=$(which mediainfo)
FFMPEG=$(which ffmpeg)
LSDVD=$(which lsdvd)
XML=$(which xmlstarlet)

M_TR_M="-oac lavc -ovc lavc -of mpeg -lavcopts \
    abitrate=${AABIT}:vcodec=mpeg2video:keyint=1:vbitrate=${AVBIT}:\
    vrc_maxrate=${MVBIT}:vrc_buf_size=1835 \
    -mpegopts muxrate=12000 -af lavcresample=44100 "
M_TR_A="-oac lavc -ovc copy -of mpeg -lavcopts \
    abitrate=${AABIT} -af lavcresample=44100 "
M_RE_M="-oac copy -ovc copy -of mpeg -mpegopts format=dvd -noskip -mc 0 "
F_TR_M="-acodec ac3 -ab 384k -vcodec copy -vbsf h264_mp4toannexb -f mpegts -y "
F_RE_M="-acodec copy -vcodec copy -vbsf h264_mp4toannexb -f mpegts -y "
SUBOPTS="-slang ${SUBS} "
SRTOPTS="-font ${SFONT} -subfont-autoscale 0 \
    -subfont-text-scale ${SUBSIZE} -subpos 100 "
SIZEOPTS="-vf harddup,scale=720:-2 "
NOSIZEOPTS="-vf harddup "
S24FPS="23.976"
S24OPT="-ofps 24000/1001 "
S30FPS="29.970"
S30OPT="-ofps 30000/1001 "

VCODEC=""
ACODEC=""
VWIDTH=""
VFPS=""
QPEL=""
AVCPROF=""

OPTS=("")

declare -i MODE=0

#############################################################################
# Functions
#############################################################################

function log {
        if [ "${LOGGING}" == "1" ] ; then
                echo -e "$(date +'%Y/%m/%d %H:%m:%S') \t $1" >> ${LOGFILE}
        fi
}

function mediainfo {
        MIOUT=$(mktemp /tmp/tmp.mediainfo.XXXXXX)
        log "Logging mediainfo XML to ${MIOUT}."
        ${MEDIAINFO} --output=xml "${FILE}" > ${MIOUT}
        VCODEC=$(${XML} sel -t -m ".//track[@type='Video']" -v "Format" ${MIOUT} )
        ACODEC=$(${XML} sel -t -m ".//track[@type='Audio']" -v "Format" ${MIOUT} )
        VWIDTH=$(${XML} sel -t -m ".//track[@type='Video']" -v "Width" \ 
            ${MIOUT} | sed 's/ pixels//' )
        VFPS=$(${XML} sel -t -m ".//track[@type='Video']" -v "Frame_rate" \ 
            ${MIOUT} | sed 's/ fps//' )
        AVCPROF=$(${XML} sel -t -m ".//track[@type='Video']" -v "Format_profile" \
            ${MIOUT} | sed 's/[^0-9]//g' )
        QPEL=$(${XML} sel -t -m ".//track[@type='Video']" -v "Format_settings__QPel" \
            ${MIOUT} )
        log "Variables found: \
            ${VCODEC} | ${ACODEC} | ${VWIDTH} | ${VFPS} | ${AVCPROF} | ${QPEL} "
        rm -f ${MIOUT}
}

function tropts {
        if [ "${DOWNSCALE}" == "1" -a ${VWIDTH} -gt ${MAXSIZE} ] ; then
                log "Rescaling to 720 pixels wide."
                OPTS+=(${SIZEOPTS})
        else
                log "Rescaling disabled or file within limits."
                OPTS+=(${NOSIZEOPTS})
        fi
        if [ "${VFPS}" == "${S24FPS}" ] ; then
                log "Framerate adjusted for mencoder."
                OPTS+=(${S24OPT})
        else if [ "${VFPS}" == "${S30FPS}" ] ; then
                log "Framerate adjusted for mencoder."
                OPTS+=(${S30OPT})
        else
                log "Framerate acceptable for mencoder."
        fi
        fi
}

function getmode {
        # Fixed case: DVD ISO.
        if [ "${FEXT}" == "ISO" ] ; then
                CHAPTER=$(${LSDVD} "${FILE}" | grep Longest | sed 's/.* //')
                log "DVD iso image found: Longest chapter is ${CHAPTER}."
                MODE+=${DISABLESUBS}1000000
                return 0
        fi
        # Fixed case: subtitle found: transcode by default.
        if [ "${DISABLESUBS}" == "0" -a -e "$(echo $FILE | sed 's/...$/sub/')" ] ; then
                log "SRT subtitle found."
                SUB=$(echo $FILE | sed 's/...$/sub/')
                MODE+=100000
                return 0
        elif [ "${DISABLESUBS}" == "0" -a -e "$(echo $FILE | sed 's/...$/srt/')" ] ; then
                log "SUB subtitle found."
                SUB=$(echo $FILE | sed 's/...$/srt/')
                MODE+=100000
                return 0
        fi

        log "No subtitles found, or subtitle rendering disabled."
        mediainfo

        case ${VCODEC} in
        "AVC")
                if [ "${AVCPROF}" -gt "41" ] ; then
                        # Cannot handle h.264 4.1+
                        MODE+=10000     
                else          
                        # We can handle the rest                          
                        MODE+=1         
                fi ;;
        "MPEG-4 Visual")
                if [ "${QPEL}" == "No" ] ; then
                        # No QPEL: we could remux the video         
                        MODE+=100       
                else            
                        # QPEL: just transcode it all                        
                        MODE+=10000     
                fi ;;
        * )
                        # Transcode everything we don't know
                        MODE+=10000 ;;  
        esac

        case ${ACODEC} in
        "AC-3" | "MPEG Audio" )      
                        # These should be wellknown                   
                        MODE+=1 ;;      
        * )
                if [ "${MODE}" -lt "100" ] ; then    
                        # If video is AVC, transcode audio in m2ts   
                        MODE+=10        
                else     
                        # Otherwise in other container                       
                        MODE+=1000      
                fi ;;
        esac

}

function processmode {
        log "Mode is ${MODE}."
        if [ ! "${MODE}" -lt "10000000" ] ; then
                EXEC="${MENCODER} -dvd-device"
                OPTS+=(dvd://${CHAPTER} ${M_RE_M} -o )
        elif [ ! "${MODE}" -lt "1000000" ] ; then
                EXEC="${MENCODER} -dvd-device"
                OPTS+=(dvd://${CHAPTER} ${SUBOPTS} ${M_TR_M} -o )
        elif [ ! "${MODE}" -lt "100000" ] ; then
                EXEC=${MENCODER}
                tropts
                OPTS+=(${M_TR_M} -sub ${SUB} ${SRTOPTS} -o )
        elif [ ! "${MODE}" -lt "10000" ] ; then
                EXEC=${MENCODER}
                tropts
                OPTS+=(${M_TR_M} -o )
        elif [ ! "${MODE}" -lt "1000" ] ; then
                EXEC=${MENCODER}
                tropts
                OPTS+=(${M_TR_M} -o)
        elif [ ! "${MODE}" -lt "100" ] ; then
                EXEC=${MENCODER}
                OPTS+=(${M_TR_M} -o)
        elif [ ! "${MODE}" -lt "10" ] ; then
                EXEC="${FFMPEG} -i"
                OPTS+=(${F_TR_M})
        elif [ ! "${MODE}" -lt "1" ] ; then
                EXEC="${FFMPEG} -i"
                OPTS+=(${F_RE_M})
        else
                log "I'm sorry Dave, I'm afraid I can't do mode=0."
        fi
}

#############################################################################
# Main method
#############################################################################

log "Starting MediaTomb Multifunctional Transcoder (version ${VERSION})."
FEXT=$(echo $FILE | sed 's/.*\.//' | tr [a-z] [A-Z])
log "${FEXT} file specified: \"${FILE}\""

getmode
processmode

log "Starting exec:"
log "${EXEC} \"${FILE}\" ${OPTS[@]} ${2} &>/dev/null"
exec ${EXEC} "${FILE}" ${OPTS[@]} "${2}" &>/dev/null

Some explanation on how this works: first of all, the script checks the source file extension. Unlike the old script however, this is no longer the final source of info. A variable "MODE" has been introduced, which is being modified by various checks based on mediainfo output of the source file. In short: every digit has a meaning. If MODE has one digit, it means that both audio and video have been checked, and can be remuxed to a container the PS3 supports. Most common example: h.264 video, AC3 audio. If MODE has two digits after running through the checks, the video stream can be remuxed, but the audio stream needs to be transcoded. This happens quite a lot with MP4s where the video codec works well, but the PS3 has issues with the audio stream.

Three or four digits is a case I haven't cracked yet. Basically, three digits means that both audio and video are MPEG-4 (DivX, XviD) codecs that are natively supported by the PS3, so in theory, we might remux those into a container and serve it. You'll notice that most AVIs will have such a properly supported video stream and MP3 audio; they end up in this case. Four digits means only the audio needs to be transcoded, and the video can stay. Big problem there: the PS3 doesn't like it when you serve it an AVI that you've remuxed... something for me to investigate. For now, they get the same treatment as the five-digit MODEs - transcoded into a big ol' MPEG-2 stream, like all the other unsupported crap out there.

Six digits means any random media with a subtitle, which results in MPEG-2. Seven means a transcoded DVD iso (with subtitles); eight is a remuxed DVD iso (without subtitles). And nothing stops you from adding more subclasses if you'd see a need for it.

Once the MODE is set, all the script needs to do is pick the proper encoder (ffmpeg or mencoder), add the proper options, and exec it.

You'll also notice an extra dependency was introduced: XMLStarlet. This is needed to propery parse the XML dump that is created by running mediatomb on a source file. Well, 'needed'... it makes life a hell of a lot easier. And this way you don't have to call mediainfo half a dozen times on the same file to pull it through grep.

All in all, I'm happy with the result. I still need to investigate on how to remux an avi using this script (which has mimetype video/mpeg in config.xml; might have something to do with it?) but that's something for later.

Enjoy!

Lennert Sun, 08/08/2010 - 20:40

Trac, ACLs and OpenLDAP

Lennert — Wed, 07 Jul 2010 22:33:56 +0000

Trac, ACLs and OpenLDAP

I'm not a developer, but I like Trac. I like its wiki syntax, and the way you can create a simple, effective ticketing system without a billion options you'll never use anyway. I like how you can disable whole chunks of the system - like the code repository subsystem, which I don't have a use for. It's easy to modify (it's python, after all) and has a billion plugins available.

What we were looking for was a system that does wiki, ticketing, has a way to set ACLs on both, and supports LDAP.

LDAP support for Trac is twofold: first of all, you need to have authentication. Trac doesn't do LDAP auth; Apache does, however. For this, you'll need mod_ldap and mod_authnz_ldap. In your Apache vhost configuration, you'll want to put this for your Trac:

        
                AuthType Basic
                AuthName "Trac"
                AuthBasicProvider ldap
                AuthLDAPURL "ldaps://1.2.3.4:636/ou=users,dc=company,dc=local?uid?sub?(objectClass=MyUserClass)"
                AuthzLDAPAuthoritative Off
                AuthLDAPGroupAttribute memberUid
                AuthLDAPGroupAttributeIsDN off
                require valid-user

The code speaks for itself: you'll be authenticating against LDAP server 1.2.3.4, using secure LDAP (SSL) on port 636, looking for users in ou=users,dc=company,dc=local, comparing the UID with what the user provided, and only objects of class MyUserClass are being allowed. For most uses, this objectClass will be posixAccount. The AuthLDAPGroupAttributeIsDN setting is set to "off"; if we're using posixGroups, the groups only list UIDs. If your LDAP groups are of objectClass groupOfNames, you'll want this to be on.

So now we can authenticate to Trac via Apache and LDAP. But this only gives us access to the builtin "anonymous" and "authenticated" groups; there's still no link between LDAP groups and Trac groups. Cue the Trac LDAP Plugin: this should map LDAP groups to Trac.

This LDAP Plugin has some minor issues: it doesn't want to work on Trac 0.12, which is the most recent version, so just doing "easy_install ldapplugin" will pull in Trac 0.11 as a dependency. To get it working anyway, a two-line patch is needed:

$ svn co http://trac-hacks.org/svn/ldapplugin/0.11/ ldapplugin

Edit the setup.py and change this:

description = 'LDAP extensions for Trac 0.11',
(...)
install_requires = [ 'Trac>=0.11', 'Trac

... into this:

description = 'LDAP extensions for Trac 0.12',
(...)
install_requires = [ 'Trac>=0.12', 'Trac

Next, edit the ldapplugin/api.py file:

--- ldapplugin/api.py
+++ ldapplugin/api.py
@@ -62,7 +62,7 @@
         self._ldapcfg = {}
         for name,value in self.config.options('ldap'):
             if name in LDAP_DIRECTORY_PARAMS:
-                self._ldapcfg[name] = value
+                self._ldapcfg[str(name)] = value
         # user entry local cache
         self._cache = {}
         # max time to live for a cache entry
@@ -178,7 +178,7 @@
         self._ldapcfg = {}
         for name,value in self.config.options('ldap'):
             if name in LDAP_DIRECTORY_PARAMS:
-                self._ldapcfg[name] = value
+                self._ldapcfg[str(name)] = value
         # user entry local cache
         self._cache = {}
         # max time to live for a cache entry


Next is to install it:

$ python ./setup.py install


And voilà. You have a proper LdapPlugin for Trac 0.12. Now we have to set it up in trac.ini:

[components]
ldapplugin.api.ldappermissiongroupprovider = enabled
ldapplugin.api.ldappermissionstore = enabled

[ldap]
basedn = dc=company,dc=local
bind_passwd = myverysecurepassword
bind_user = cn=proxy,dc=company,dc=local
enable = true
group_bind = true
group_rdn = ou=groups
groupmember = memberUid
groupname = posixGroup
groupmemberisdn = false
host = 1.2.3.4
port = 636
use_tls = true
user_rdn = ou=users
uidattr = cn


The LDAP options deserve a closer look: lots of LDAP servers do not allow inspecting the tree anonymously (which is a sane approach; would you want strangers to nose through your address books?). So you define group_bind, create a proxy user, and tell Trac to use that user/password to get its info. The same distinction between groupOfNames and posixGroup appears here: usually you'll want to use posixGroup, in which case the group members are not in DN format, and the membership attribute is called memberUID. In case of a groupOfNames, group members are in DN format, and the groupmember attribute is simply called "member".

Now you can assign Trac permissions to LDAP users! Use trac-admin to do this, and keep in mind that LDAP groups are prefixed with an "@" to make the distinction with local groups:

$ trac-admin /path/to/my/project
Welcome to trac-admin 0.12
Interactive Trac administration console.
Copyright (c) 2003-2010 Edgewall Software

Type:  '?' or 'help' for help on commands.

Trac [/path/to/my/project]> permission add @administrators TRAC_ADMIN


You can give different groups different Trac permissions, just like you'd do with local groups. But then the last requirement...

One of the (shipped-by-default) plugins is the Trac Fine Grained Permissions system, which allows you to set ACLs on (amongst others) wiki pages. It uses an svn-style authz file for permissions, and allows wildcards. Lovely. However, this system only looks at internal groups! No integration whatsoever. Sounds bad? It is. And since this plugin is part of Trac itself (in tracopt - optional, sure, but still inside the Trac egg!) you'll have to build your own Trac. Not like that's so hard, of course.

$ svn co http://svn.edgewall.com/repos/trac/trunk/ trac


Edit the file tracopt/perm/authz_policy.py, and replace the function authz_permissions with the following code:

   def authz_permissions(self, resource_key, username):
        if username and username != 'anonymous':
            valid_users = ['*', 'authenticated', username]
            perms = LdapPermissionGroupProvider(self.env).get_permission_groups(username)
            valid_users += perms
        else:
            valid_users = ['*', 'anonymous']
        for resource_section in self.authz.sections:
            resource_glob = to_unicode(resource_section)
            if '@' not in resource_glob:
                resource_glob += '@*'
            if fnmatch(resource_key, resource_glob):
                section = self.authz[resource_section]
                for who, permissions in section.iteritems():
                    if who in valid_users:
                        if isinstance(permissions, basestring):
                            return [permissions]
                        else:
                            return permissions
                    else:
                        self.log.debug('%s does not match any of valid_users: %s', who, valid_users)
        return None


Install Trac from this tree (python ./setup.py install) and you're set. What does this code do? It tosses out the clunky authz-internal groups (yes, you won't be able to use internal groups anymore for ACLs... but there is no reason why you'd want that) and gets the LDAP groups for the given username via LdapPermissionGroupProvider. Easy. All there is left to do now is configure Trac itself to check for Authz ACLs on objects:

[components]
tracopt.perm.authz_policy.* = enabled

[authz_policy]
authz_file = /path/to/my/project/conf/authzpolicy.conf

[trac]
permission_policies = AuthzPolicy, DefaultPermissionPolicy, LegacyAttachmentPolicy


In authzpolicy.conf, you can then define your ACLs, like for an example when you want all users (including anonymous!) to be able to view the main WikiStart page, and only users in LDAP groups "treasury" and "trustedusers" to access the wiki section "financials", where "treasury" has full wiki rights and "trustedusers" read-only:

[wiki:WikiStart]
* = WIKI_VIEW

[wiki:financials/*]
@treasury = WIKI_ADMIN
@trustedusers = WIKI_VIEW
* =


Restart Trac (or Apache, since you're using Trac via Apache, right?) and there you go... Trac is being fully managed by LDAP.

Enjoy!

Lennert Thu, 08/07/2010 - 01:33

Authenticating Web Apps with your USB ActiveKey from Linux

Lennert — Mon, 17 May 2010 18:56:31 +0000

Authenticating Web Apps with your USB ActiveKey from Linux

After the previous story about authentication of Linux services against your USB ActiveKey, I received a lot of questions about using actual web applications with that same key. Is it possible? Well, as some people already pointed out in the comments, sure it is.

For a starter, I'll assume you've already followed the previous article and have it working on Linux. If you haven't, I doubt these next few things will work either.

First of all, you'll have to get the proper security device installed. This will be the CoolKey device that you configured before. In FireFox (or IceWeasel... what's in a name) go to: Edit → Preferences → Advanced → Encryption → Security Devices. In the Device Manager, load a new device named "CoolKey" (or whatever you want it to be) and select as module:

/usr/lib/pkcs11/libcoolkeypk11.so

If the module is named differently (this is on Debian), change as necessary. You should get the following kind of view:

Your key should appear below:

...and that's that. Now you can try connecting to a key-enabled website, like your company's Web Communicator site. If all goes well, the CoolKey module will prompt you for your key "master password":

Once you have entered it, the website will prompt you to provide your certificate.

And boom... the website loads. Brilliant.

What's even more surprising: this thing actually loads in Firefox - the only minor annoyance is the address bar in that tiny window. But hey, I would have at least expected it to barf on a non-IE browser. :-)

Enjoy!

Lennert Mon, 17/05/2010 - 21:56

Authenticating Linux with your USB ActiveKey

Lennert — Thu, 14 Jan 2010 12:11:50 +0000

Authenticating Linux with your USB ActiveKey

As I started working for my new employer, I got a new laptop. Linux is allowed here (thank goodness for that!) so I decided to go for a Debian install. I also got this nifty USB ActiveKey token, which can be used to authenticate to a plethora of services - Windows domain login, SAP login, ... fun stuff. But what about Linux? It turns out you can make good use of this key for Linux authentication too.

For simplicity's sake I'll assume here that the ActiveKey you have has already been initialized and you have stored your certificate on it using the standard (Windows) tooling.

First thing you'll need is a set of tools to read the ActiveKey itself.

apt-get install coolkey pcscd pcsc-tools libpam-pkcs11

Start the pcscd daemon and insert your ActiveKey. You'll notice that the light goes from red to green. Check if everything is detected properly:

[alver@Talisker ~]$ pcsc_scan 
PC/SC device scanner
V 1.4.16 (c) 2001-2009, Ludovic Rousseau 
Compiled with PC/SC lite version: 1.5.5
Scanning present readers...
0: Activkey Sim 00 00

Thu Jan 14 12:00:25 2010
 Reader 0: Activkey Sim 00 00
  Card state: Card inserted, Shared Mode,

... and so on. Good, the reader is found, and it knows there's a card inserted. The command will stop without returning to shell because it doesn't know anything about the reader or card - that's okay. Just ctrl-c back to shell.

Now it's time to setup pam_pkcs11. Just installing the package above won't do much; you'll have to manually create the necessary directories and create a config file.

mkdir -p /etc/pam_pkcs11/cacerts
zcat /usr/share/doc/libpam-pkcs11/examples/pam_pkcs11.conf.example.example.gz > /etc/pam_pkcs11/pam_pkcs11.conf

Edit /etc/pam_pkcs11/pam_pkcs11.conf to match the following (not pasting the complete config file, just the relevant bits to change):

use_pkcs11_module = coolkey;

# Coolkey Support
pkcs11_module coolkey {
  module = /usr/lib/pkcs11/libcoolkeypk11.so 
  description = "Coolkey";
  slot_num = 0;
  support_threads = false;
  ca_dir = /etc/pam_pkcs11/cacerts;
  cert_policy = ca;
}

use_mappers = subject;

You can setup a subject-to-user mapping now. You'll have to get your hands on the company Issuer CA certificate first though, otherwise all attempts to validate your personal certificate will (naturally) fail. Copy this certificate file in base64 format to /etc/pam_pkcs11/cacerts.

Last step is mapping your personal certificate to your unix user. Print the certificate listed on your ActiveKey to check if it's found and no certificate errors are given:

[alver@Talisker pam_pkcs11]$ pkcs11_listcerts 
PIN for token: 
Found '1' certificate(s)
Certificate #1:
- Subject:   /O=Your Company/OU=Yada Yada/OU=Foo/CN=Your Name Goes Here/emailAddress=your.mail@your.com.pa.ny
- Issuer:    /O=your.com.pa.ny/OU=Bar/C=XX/O=Your Company/CN=Your Company Primary Certification Authority
- Algorithm: rsaEncryption

Okay, looks fine. You already copied the CA file so the Issuer should be accepted, and the Subject... well, that should be you. If it's not, bail out. Map this Subject to your local user:

echo "$(pkcs11_listcerts | grep Subject | sed 's,^[^/]*/,/,') -> youruser" > /tmp/subject_mapping

Copy this file over to /etc/pam_pkcs11/. Okay... done. Verify that the mapping is properly detected:

[alver@Talisker pam_pkcs11]$ pklogin_finder
alver

Last thing you need to do is telling pam that you want to authenticate using your ActiveKey. The pam config line is this:

auth	sufficient	pam_pkcs11.so debug=false config_file=/etc/pam_pkcs11/pam_pkcs11.conf

I personally put it in /etc/pam.d/gdm and /etc/pam.d/gnome-screensaver. GDM will still ask for a username though (it's not fully smart card aware) so you leave the field blank and press enter; it will ask for a pincode next. Gnome-screensaver doesn't have such a quirk.

Enjoy!

Lennert Thu, 14/01/2010 - 14:11

MediaTomb and PS3: Playing DVD ISO files

Lennert — Mon, 07 Dec 2009 12:57:27 +0000

MediaTomb and PS3: Playing DVD ISO files

Since mediatomb doesn't care much about what it serves, it's not that hard to have it serve dvd movies from an ISO file. A PlayStation 3 however is rather picky when it comes to video formats, so getting the three to play along can get interesting.

First of all, you need to tell mediatomb that anything with a .iso extensions is something it should look into:

Then tell it to transcode this particular mimetype:

All you need then is a transcoding profile which handles the iso files.


        video/mpeg
        yes
        yes
        yes

If you're unfamiliar with the mediatomb config.xml structure, my full config file is up here. Finally, here's the helper script that does the actual transcoding of the dvd iso (with subtitles!) to something the PS3 knows and likes:

#!/bin/bash
CHAPTER=`lsdvd "$1" | grep Longest | sed 's/.* //'`

exec mencoder -dvd-device "$1" \
dvd://$CHAPTER -slang nl,en -oac copy -ovc lavc -of mpeg \
-lavcopts vcodec=mpeg2video:keyint=1:vbitrate=200000:vrc_maxrate=12000:vrc_buf_size=1835 \
-mpegopts muxrate=12000 -vf harddup,scale=720:-2 \
-o "$2" &>/dev/null

This script uses "lsdvd" to find the longest chapter of the dvd first - we'll have to assume there that the longest chapter is indeed the one with the main movie. A dvd iso containing two movies, or containing various episodes of a series for an example, won't work too well with this script - which is fine by me, since I don't have that kind of isos. This script Works For Me™ but feel free to change mencoder parameters to suit your needs. If you're not interested in subtitles, you could try without the -slang nl,en and using -ovc copy, essentially remuxing the media stream without actually transcoding it. YMMV.

So there you have it - iso files show up in the PS3's menu, and they get transcoded on the fly with dutch subtitles, or english subtitles if dutch ones aren't available in the iso. And you don't even need mediatomb 0.12 for it to work.

Lennert Mon, 07/12/2009 - 14:57

MediaTomb and PS3

Lennert — Fri, 04 Dec 2009 15:55:27 +0000

MediaTomb and PS3

Mediatomb is an open source (GPL) UPnP MediaServer with a nice web user interface. It allows you to stream your digital media through your home network and listen to/watch it on a variety of UPnP compatible devices.

Sony's PlayStation 3 is one of those UPnP compatible devices - but it's got its quirks. It took me a while to get my setup working the way I like it, so I guessed I might just as well post my config up here.

First of all the config file, located in /etc/mediatomb/config.xml:



  
    eth0
    50000
    
    
      
        
      
    
    MyMediatomb
    uuid:4105837b-0437-4848-9bea-xxxxxxxxxxxx
    /var/lib/mediatomb
    /usr/share/mediatomb/web
    
      
        sqlite3.db
      
      
        localhost
        mediatomb
        mediatomb
      
    
    
  
  
    UTF-8
    ISO-8859-15
    
      /usr/share/mediatomb/js/common.js
      /usr/share/mediatomb/js/playlists.js
      
        /etc/mediatomb/import-custom.js
      
    
    
      
        
        
        
        
        
        
        
        
        
        
        
      
      
        
        
        
      
      
        
        
        
      
    
  
  
    
      
      
      
      
      
      
    
    
      
        video/mpeg
        yes
        yes
        yes
        
        
      
      
        video/mpeg
        yes
        yes
        yes
        
        
      
      
        video/mpeg
        yes
        yes
       yes
        
        
      
      
        video/mpeg
        yes
        yes
        yes

As you can tell, there's a lot of transcoding profiles in there. The mencoder-int profile is used to transcode anything that should play, but doesn't - DIVX movies with QPeL, wonky audio codecs, realmedia files, ... All of those are tagged "video/transcode" - some automatically, like realmedia, others manually in the mediatomb web UI - and piped through mencoder: audio and video become MPEG2.

The mencoder-720 profile is used only for funky container formats here, such as OGM or MKV. The ones I have usually contain more than one audio track and a few subtitles, so I tell mencoder to select the first audio and subtitle track. Don't worry about the -ofps option; that one was needed to fit one particular set of files.

Lennert Fri, 04/12/2009 - 17:55