Mosquitto working

Some rework had to be done
Brett sent me answers on what to do about SSL 1.x.x, and the problem I had with one of the sample programs: SSL 1.0.x has been staticly linked into the shareable image, so linking against that file (libmqttv3c$shr.exe) instead of the object library (libmqttv3c.olb) would solve the problem.
He also added a somewhat reworked version of paho_c_pub.c, I had to do some re-editing to solve issues that were the result of retrieving this source from the message and store it in my (WIN10) workstation, and moving is (ASCII) to the VMS box. But once these were addressed, the program starts and runs.

Next issue was to figure out how to pass a message from the VMS box to the application on my mobile….
First do it the easy way:

  • On my mobile (in MQTT Dashboard) define a connection giving it a easy name enter login nredentials as connecting to the server (since I am using Brett’s additional module for authentication), and let it listen to any subject (“+” so I don’t have to bother on that to begin with. Start the connection on mobile, it will start listening.
  • On the server start paho_c_pub, any topic will now do; Enter the connection on the mobile as clientid;

  • So there the commands and results are:
    $ pub test --clientid (Name on Mobile) --username (username) --password (password)
    Using topic test


    Any test that is now entered, will show up in MQTT Dashboard in this client, under topic “test”

    Backlink has to be done – will be (I guess) paho_c_sub.c; Same setup for subject, it will show all that is sent – together with subject – until cancelled – it will listen to any topic (due to wildcard “+”):

    $ sub + --clientid (Name on Mobile) --username (username) --password (password)
    Subscribing to topic + with client (clientid) at QoS 2
    test redo
    Oktest redo
    Oktest redo
    Oktest Done

    These messages show up in the subcsscription area in MQTTDashboard as well – obvious, since I’m listening to any subject 🙂
    I also added –delimiter \n to add delimiter beyond, but that does not work the way I anticipated. But is newline is included in the message it will show up in output:

    $ sub + --clientid (Name on Mobile) --username (username) --password (password) --delimiter \n
    topic is +
    Subscribing to topic + with client htc at QoS 2
    test Done \test Done
    \test With crlf

    But that is possibly a matter of understanding on how to read the documentation, it might well be the way it should work.

    If sub is started without clientid,

    sub + --username (username) --password (password)
    topic is +
    Subscribing to topic + with client stdout-subscriber-async at QoS 2

    messages from my mobile are not accepted. Which is fine for what I’m planning to use it for.

    Now the first hurdles have been taken: Connection works both ways and I have a fair impression on how the use it both ways. Next is creating API-calls (that Brett already started to do) and use these.

    Paho re-installed

    Paho reinstalled
    I have overlooked a few things: I installed the previous version (that process removes what’s on the system…) and checked the contents of the object libraries in that version; but what is in the latest version was already saved – and the new lobrraies contain the same routines.
    Re-installed the latest version of Paho (1.2) and rebuilt the samples using the supplied procedure – which may be different from the 1.o version.
    Now all but two images can be compiles; the programs that run in an SSL environment compile but link fails.

    $ link/threads paho_cs_pub.obj,paho$root:[lib]libmqttv3cs.olb/lib,sys$input/opt

    stating that sys$library:ssl1$libssl_shr32.exe/share cannot be found.
    Of course not: I have no support contract with HPE an d so I cannot update my system anymore…

    So I used the old files instead, which gives me undefines symbols:

    $ link/threads paho_cs_sub.obj,paho$root:[lib]libmqttv3cs.olb/lib,sys$input/opt
    %LINK-W-NUDFSYMS, 2 undefined symbols:
    %LINK-W-USEUNDEF, undefined symbol CRYPTO_THREADID_SET_NUMERIC referenced
            in psect $LINK$ offset %X000007E0
            in module SSLSocket file PAHO$ROOT:[lib]libmqttv3cs.olb;1 %LINK-W-USEUNDEF, undefined symbol CRYPTO_THREADID_SET_CALLBACK referenced
            in psect $LINK$ offset %X00000890
            in module SSLSocket file PAHO$ROOT:[lib]libmqttv3cs.olb;1

    This is not a big issue – as long as the programs that do not require encrypted communication can be build. And that is Ok.

    So I started the publication program:

    $ sho symb pub
    $ pub test --clientid ExampleClientPub
    Using topic test
    Failed to start connect, return code -13

    It might be that I need to add credentials:

    $ pub test --clientid ExampleClientPub --username (username) --password (password)

    but that gives the same result

    Checked the log (after I enabled logging of connections) it shows a warning – which may point the cause:

    $ type dka0:[mqtt.mosquitto.logs]MOSQUITTO.LOG
    $ Set NoOn
    $ mosquitto :== $mosquitto$root:[bin]mosquitto.exe
    $ mosquitto "-c" mosquitto$root:[conf]mosquitto.conf
    1533837960: mosquitto version 1.4.14 (build date 29-MAR-2018 02:52:02.10) starting
    1533837960: Config loaded from mosquitto$root:[conf]mosquitto.conf.
    1533837960: Opening ipv4 listen socket on port 1883.
    1533837960: Opening ipv4 listen socket on port 1883.
    1533837960: Warning: protocol wrong type for socket

    I did some more investigation – the subscriber program does connect – but wont’t show a result when a message is posted from the mobile phone (using MQTT Dashboard):

    $ sho sym sub
    $ sub ExampleClientsub --username (username) --password (password)

    and it will wait until cancelled)

    log shows it all: Connection failures and successes:

    1533841611: New connection from on port 1883.
    1533841611: New client connected from as ExampleClientPub1533841611246 (c1, k60, u'Willem ').
    1533841630: New connection from on port 1883.
    1533841630: New client connected from as ExampleClientSub1533841630615 (c1, k60, u'Willem ').
    1533841655: New connection from on port 1883.
    1533841655: New client connected from as stdout-subscriber-async (c1, k10, u'willem').
    1533841708: Client stdout-subscriber-async disconnected.
    1533841713: New connection from on port 1883.
    1533841713: New client connected from as stdout-subscriber-async (c1, k10, u'willem').
    1533841749: Client stdout-subscriber-async disconnected.
    1533841888: Client ExampleClientPub1533841611246 has exceeded timeout, disconnecting.
    1533841888: Socket error on client ExampleClientPub1533841611246, disconnecting.
    1533841892: Client ExampleClientSub1533841630615 has exceeded timeout, disconnecting.
    1533841892: Socket error on client ExampleClientSub1533841630615, disconnecting.
    1533841948: New connection from on port 1883.
    1533841948: New client connected from as stdout-subscriber-async (c1, k10, u'willem').
    1533841956: Client stdout-subscriber-async disconnected.
    1533841962: New connection from on port 1883.
    1533841962: New client connected from as stdout-subscriber-async (c1, k10, u'willem').
    1533841966: Client stdout-subscriber-async disconnected.

    (connectios on address are from my mobile, is from my VMS development system)

    and in TCPIP I see both connections when starting the reader (sub):

    TCPIP> sho dev

                                Port                       Remote
    Device_socket  Type    Local  Remote  Service           Host

      bg581       STREAM    1883       0                   *
      bg678       STREAM   49159    1883         
      bg679       STREAM    1883   49159         

    I tried to compile and link the publisher program – but the problem is within the MQTT library which has no debug information, nor do I have the Mosquitto code with the package, so some assistance of Brett Cameron is required.



    BEWARE: This is a technical issue on how to analyze a program error with (built-in!!) options to compiler and linker. To understand, you’ll need some knowledge of hoe a program on VMS looks like internally.
    Listing fragments have been slightly edited to fit on the screen.

    The issue itself – and what’s related – can be found on the WASD mailing list for 2018 in entry 8 and following. This file contains the data of the analysis as taken from the files (not edited).

    As mentioned, HyperSpi++ collector didn’t collect any data since last reboot. The log showed whyL

    %SYSTEM-F-HPARITH, high performance arithmetic trap,
     Imask=00000000, Fmask=00080000, summary=02, PC=0000000000031448, PS=0000001B
    -SYSTEM-F-FLTINV, floating invalid operation,
     PC=0000000000031448, PS=0000001B

      Improperly handled condition, image exit forced.
        Signal arguments:   Number = 0000000000000006
                            Name   = 0000000000000504

    Register dump:
    R0 =0000000000040990 R1 =00B2816B42BD488F R2 =0000000000010250
    R3 =0000000000000001 R4 =00000000000201F0 R5 =0000000000040860
    R6 =0000000000040870 R7 =0000000000040870 R8 =0000000000000005
    R9 =0000000000040907 R10=00000000000408F5 R11=000000007FFCDC18
    R12=000000007FFCDA98 R13=000000007AEC33A0 R14=0000000000000000
    R15=000000007AEC29B0 R16=0000000000000000 R17=0000000000010848
    R18=000000007BF260F8 R19=FFFFFFFFFFFFFFF9 R20=FF4D7E94BD42B771
    R21=000000007B63DA10 R22=000000007ADB9940 R23=0000000000000002
    R24=0000000000000001 R25=000000000006C004 R26=FFFFFFFF80FAFB60
    R27=000000007BEF4820 R28=FFFFFFFF80CAFE40 R29=000000007ADB9930
    SP =000000007ADB9860 PC =0000000000031448 PS =200000000000001B

    but since the program wasn’t built with the options: /LIS/MACHINE_CODE on compilation, and /MAP/FULL on link. you’re in the dark. IMHO, these options should be mandantory for programs that run without user intervention, as detached program or as a service.

    But the kit contains the command procedures to build the agent, so I added these options, built the agent and ran it – interactively. Same error (of course) but now I could pinpoint the location.

    The MAP files shows the sections (PSECT) of a program – and the offsets in the image file. The important part is pretty well in the beginning:

      0     5    00010000    3 0 READ WRITE NON-SHAREABLE ADDRESS DATA
      0     2    00020000    8 0 READ WRITE COPY ON REF
      0    30    00030000   10 0 READ ONLY  EXECUTABLE    < <<<-------   0     5    00040000    0 0 READ WRITE DEMAND ZERO   0     2    00050000   40 0 READ WRITE FIXUP VECTORS 253    20    7FFF0000    0 0 READ WRITE DEMAND ZERO

    Most important here is the third line, where the last colums states "EXECUTABLE" This is where executable code - so what the program does - is located. It starts at offset 30000.

    Since the crash line states the process counter (PC) where the error happens:

    %SYSTEM-F-HPARITH, high performance arithmetic trap, Imask=00000000, Fmask=00080000, summary=02, PC=0000000000031448, PS=0000001B
    -SYSTEM-F-FLTINV, floating invalid operation, PC=0000000000031448, PS=0000001B

    it means the program counter refers to offset 31448 – within that section (obvious: it happens when the program executes an instruction) at offset 1448 in that section (31448 – 30000).

    But what instruction – and which codeline?
    Now the listing is important – and especially the /MACHONE_CODE switch. It lists the actual instructions that the program executes – located at their offsets in the executable section of the program. In the listing, this is the second part; it shows instructions, offset (within the section), the instruction and the codeline: For this particular case, look around offset 1448:

    0234  L$227:
    1420          LDAH    R25, 7(R31)                         ; 040554
    1424          LDL     R16, (R4)                           ; 040553
    1428          LDQ     R17, -528(R2)                       ; 040554
    142C          LDA     R25, -16380(R25)
    1430          LDQ     R18, -552(R2)
    1434          LDQ     R26, -544(R2)
    1438          LDF     F1, (R0)                            ; 040552
    143C          LDQ     R27, -536(R2)                       ; 040554
    1440          LDA     R17, 936(R17)
    1444          ADDF    F1, F18, F19                        ; 040552
    1448          STF     F19, (R0)
    144C          UNOP
    1450          BEQ     R16, L$122                          ; 040553
    1454          LDL     R16, DECC$GA_STDOUT ; R16, (R18)    ; 040554
    1458          JSR     R26, DECC$GXFPRINTF ; R26, R26
    145C  L$122:                                              ; 040555

    The first column shows the offset in the section – starting at 30000, which is offset 0. So the offending instruction seems to be on offset 1448 – the STF instruction.
    The last column – not on all lines – is the line number of the code that the programmer wrote. This will, in this case, be the instruction on line 40552: on the instruction just above this instruction. This is in the first part of the file, so look around this linenummber:

    1   40538 /**************************/
    1   40539 /* calculate elapsed time */
    1   40540 /**************************/
    1   40541
    1   40542 lib$sub_times (&CurrentBinTime, 
    1   40543
    1X  40544 #ifdef __RMIDEF_LOADED
    1X  40545 status = lib$cvts_from_internal_time 
                      (&cvtf_mode, &DeltaSeconds,
    1X  40546          &DiffBinTime);
    1X  40547 #else
    1   40548 status = lib$cvtf_from_internal_time
                     (&cvtf_mode, &DeltaSeconds,
    1   40549         &DiffBinTime);
    1   40550 #endif
    1   40551
    1   40552 TotalDeltaSeconds += DeltaSeconds;   < <===HERE 1   40553 if (Debug) 1   40554  fprintf (stdout,           "DeltaSeconds: %f TotalDeltaSeconds: %f\n", 1   40555  DeltaSeconds, TotalDeltaSeconds); 1   40556

    Both TotalDeltaSeconds and DeltaSeconds are defined as FLOAT - which is consistent with the error text. The question is: WHAT IS WRONG? So you will have to do some digging; compile the code with option /DEBUG, and set a breakpoint on, or just above this line, to find out what is going on.

    In this case, it turned out that quite likely compiler switch /OPTIMIMZE (perhaps implied by /NODEBUG) made the error occur; using /NOOPTIMZE solved the problem and the program executed flawlessly.


    MariaDB active
    Tonight I started MariaDB and had it ‘active’ for at least 15 minutes – longer than the previous version could stay alive. But nothing happened, so I did some SQL work, that would cause the server to die before. But it stayed on.
    So I changed access for the Trips, Tracks and Travels blog to use MariaDB – there was no change in that blog after I moved it to MariaDB. had to do some updates but they al seemed Ok. (Changed the Admin password, for a change :)).

    For now, it looks fine; Time will tell, but if everything is going as it should, this blog will be the next – which will require moving the whole MySQL database again, but now I know that the process works, it should not be a problem. Except that the blogs will be unavailable for an hour or so.

    Stay tuned….


    Apart from what happened last month, there is not much to look at. PMAS does its work:
    PMAS statistics for February
    Total messages    :   2534 = 100.0 o/o
    DNS Blacklisted   :      0 =    .0 o/o (Files:  0)
    Relay attempts    :     73 =   2.8 o/o (Files: 28)
    Accepted by PMAS  :   2461 =  97.1 o/o (Files: 28)
      Handled by explicit rule
             Rejected :   1682 =  68.3 o/o (processed),  66.3 o/o (all)
             Accepted :    200 =   8.1 o/o (processed),   7.8 o/o (all)
      Handled by content
            Discarded :    278 =  11.2 o/o (processed),  10.9 o/o (all)
         Quarantained :    264 =  10.7 o/o (processed),  10.4 o/o (all)
            Delivered :     37 =   1.5 o/o (processed),   1.4 o/o (all)

    Just the number of rejected messages is larger than normal, I’ve seen them coming in in the last week only; 26-Feb-2017 being most: Looking at the size of operator.log, it was over 1200 blocks on that day; the others that week were over 400, except for one, where less than 100 should be normal. The number of relay attempts was little; only on 08-Feb the attempts resulted in a log that was over 4 blocks in size, and mainly from one source – trying to relay. It could have been a test, sending from address as “”, “” or “antispam@[]” to a number of addresses all related to the same organization. I checked this address and it is listed in a number of blacklists, so it might as well be an attempt to see if there is a way to abuse the mailserver. But that checks the IP address of the sending server and if it not from my own address, and the recipient is outside my domain, it will fail.
    As simple as that.

    I had some trouble installing OpenVMS on the Itanium servers – one at least. It would find the DVD, start reading it but then halt. First, I thought it could have been a bad DVD, so I tried to read it on my old PWS, and I could access the files (not run the executables because these at I64 images, not AXP…). The DVD wasn’t as bad as I thought it was….
    So first I could try to make it accessible over the network: meaning I would have to start Infoserver on the PWS (which should be possible since that runs VMS 8.4). At last, found the documentation on the Internet to set it up, but the docs were a bit limited but with the examples, I think I had it all right, but in the end, I couldn’t start Infoserver on that box – still have to find out what caused it to fail.
    Second, I found out that I need to enable the iLO unit first – meaning I had to connect it the the network (it will be set up using DHCP to begin with), see what address was assigned to it – with dhcpdbdump, the MAC address is recognizable as from the Itanium server) and access MP on the iLO using telnet (it’s on the local network so why bother about security?). Now I could boot from internal DVD and installation did start and finish. The only thing yet to do is to install the licenses and do the TCPIP configuration (and give the iLO its own. fixed address (and name)). BTW: The first Itanium is now named Iris.

    No news on the PHP front
    There are a few things to consider.
    First, in the WASD global configuration I have a line:



    causing PHPWASD to run whenever there is an extension .PHP found. I’m pretty sure it will run the executable the logical PHPWASD refers to, because otherwise there would have a version conflict with PHPSHR.EXE: Each version I have on the system resides in its own directory, referred to by logical PHP_ROOT:. This is the item that is used when I run PHP_INFO.PHP, there is no other reference elsewhere to this script; And it shows all the correct data – with the correct version at that time.

    However, in the mapping, the blogs have their own reference:

    redirect /sysblog /sysblog/index.php
    map /sysblog**/ /sysblog*/index.php
    exec+ /sysblog/**.php* (phpwasd:)/sysblog/*.php* ods=5
    pass /sysblog/000000/* /sysblog/* ods=5 search=none dir=noaccess
    pass /sysblog/* /sysblog/* ods=5 search=none dir=noaccess

    which runs PHPWASD (the logical – explicitly via the logical) by it’s own rule.
    Now what happens if BOTH start running? It might mean that the one that I expect to run (the one in the mapping) will run into an I/o error, or it might not be able to access a sourcefile; or it may finbd it’s access to the database is blocked, causing a timeout in database access (as is shown in the PUP error log file).

    Another possibility that has crossed my mind: In the past I noted that PHP 5.3 and up will run a larger number of sub-processes. It might be that I have run out of process slots, or that the parsing, translating and execution of PHP code takes a far larger amount of system resources, causing the script not finishing in time (that is noted too, so I doubled the maximum execution time), but it might be the time to finish database transactions may have to be increased too. Another (and perhaps, better) solution is to increase the working set of the PHPWASD: processes: It hasn’t been changed since I started running PHP on a 256Mb system….Now I’ve seen that the peak working set can be increased – given the number of pagefaults…

    So these are paths to consider. I didn’t get an answer of Mark Berryman yet.


    WordPress 3.5 test – a final updare
    In the last two weeks, I’ve been busy with testing the new version of WordPress (at least, verison 3.5) to find out how to set things up in the WASD configuration, and it’s pretty close to be finalized. Two issues have been found – apart from problems within WordPress itself: The tests revealed an error in the PHP port, that could have caused the problem in cleanibg up the PHP environment after status has been returned to the server (a times that isn’t cleared when it should) but that is easily circumvented by specifying PHP.INI parameter max_execution_time to zero – meaning”indefentely”. In this case, it’s not a problem since WordPress won’t run into that situation :).
    The other problem is to be solved by HP.
    PHPWASD calls c-function chdir to change the ‘current directory’ for the running image, and next calls c-function setenv to change environment variable PATH to the same value. chdir succeeds – but setenv fails when the new value is longer than the last issued, or the default value that is passed when it wasn’t set before (being the current default). In such a case, PATH is set to NULL, the advantage being that from that moment on, any value is accepted, no matter what size it has, compared to the previously specified value.
    There is a simple workaround: just re-specify the new value one more time. If all is well, it will than be accepted. The right solution if for HP to solve the issue. I created a reproducer and sent it to OpenVMS engineering.
    (The full communication on the WordPress issue can be found on the WASD mailing list for 2013, thread on “WordPress: Config? UPDATE”. When applicable, some data may be downloadable from this site – links are included in the messages. The message to HP – including the reproducer, can be found on the same list – search for “setenv issue”).
    Updates to be applied
    This means I can now start thinking of an update of the blogs. Besides this one, I’m also thinking of updating Diana to VMS 8.4. One thing that may have caused the problem with PHP 5.3 on this system may be caused by a version difference in CRTL – or even VMS: if PHP5.3 is linked on 8.4 and run as-is on 8.3, it may have lead to the End-Of-Files issues I encountered before.


    Update on WordPress upgrade
    Finally, the new version of WordPress – that is: 3.5) runs on Daphne, under WASD 10.2 and PHP 5.3.14, in a basic theme, but it required a very different mapping than I used originally:

    redirect /wptest /wptest/index.php
    map /wptest**/ /wptest*/index.php
    exec+ /wptest/**.php* (cgi-bin:[000000]phpwasd.exe)/wptest/*.php* ods=5
    pass /wptest/* /wptest/* ods=5 search=none dir=noaccess

    There might be some additions, thanks to UMA, but this setting does the (most basic) job.
    Nevertheless, there are still a few things to take care of – but since I don’t control the PHP-port, PHPWASD or C-RTL, the few things that still don’t work as expected, or wanted, cannot be resolved immediately.
    Mark Berryman identified an error in PHP but there is a workaround: setting max_execution_time in PHP.INI to zero will probably solve some timer-related issues – as the sort-of keep-alive of PHPWASD.EXE and the PHP environment. He will solve this issue, so there is some more testing required.
    Not that this workaround solved all the issues – when WordPress admin page is started, it may take a while before they show up – and the number of open channels keep going up, to 150 or so. But later on, it’s MUCH faster, and requires less channels, even when PHPWASD is restarted from scratch. This has to do with paging, I guess, because Daphne is small – compared to the systems that Mark Berryman and Mark Daniel appear to be using.
    More problematic is a dysfunctional C function.
    As Mark Daniel has explained, PHPWASD.EXE uses the chdir function to change the location where the images should run – in the Unix-way of thinking – in practice, the directory specified in the URL that starts the PHP code. So http://<site>/wptest/index.php will cause a chdir to /wptest, and http://<site>/wptest/wp-admin/index.php will chdir to /wptest/wp-admin. That is working fine. But next, PHPWASD.EXE calls function setenv to set environment variable PATH to the same value. In itself, that’s fine also – but this fails when the second assignment uses a string longer than the preivious one. This happens within WordPress: On the first URL (http://<site>/wptest), nothing is wrong chdir ("/wptest") is executed as well as setenv("PATH","/wptest"), but after the login sequence, when http://<site>/wptest/wp-admin/index.php is to be excuted (which is reflected in the address bar of the browser), chdir ("/sptest/wp-admin") succeeds, but setenv ("PATH","/wptest/wp-admin") fails and PHPWASD exits after returning a 502 status to the server, causing a non-cgi-conform response….
    When you next start the same URL, all is well, and the pages show up nicely. However, you may encounter similar problems on executing other PHP code that re-executes setenv.

    After Mark Daniel had specified a test program for chdir, getenv and setenv, I adapted this program to do some testing. And my assumption that size maters, becomes clear: the length of the last assignment determines the maximum size of the next string! If longer, the status of setenv is -1 – stating an error, and the environment variable is set to null:

    $ run test3
    1 - Longest (/wptest/wp-includes)
    getenv PATH == sys$sysroot:[sysmgr]
    setenv PATH == 0
    getenv PATH == /wptest/wp-includes
    chdir PATH == 0
    getenv PATH == /wptest/wp-includes
    2 - Midsize (/wptest/wp-admin)
    setenv PATH == 0
    getenv PATH == /wptest/wp-admin
    chdir PATH == 0
    getenv PATH == /wptest/wp-admin
    3 - shortest (/wptest)
    setenv PATH == 0
    getenv PATH == /wptest
    chdir PATH == 0
    getenv PATH == /wptest
    4 - Midsize again (/wptest/wp-admin)
    setenv PATH == -1
    getenv PATH == (null)
    chdir PATH == -1
    getenv PATH == (null)

    I would call this a bug. It might have been introduced in an UPDATE on VMS 8.3 and 8.4, Jeremy Begg has located that there has been an update on this function to prevent a memory leak. Quit possible that the way it has been solved causes this erroneous behavior.
    Alas, there is no way to get around this, except that PHPWASD should NOT change PATH this way. So I would need the PHPWASD-code for PHP 5.3.14 (The version for PHP 5.2 has another interface to PHP which is not compatible…).

    But for the normal user – who reads the blogs – there seems to be no a problem. So the next step is to try the newest release of WordPress (3.5.1) and the themes I would like to use, and see if it all works fine.

    Finally, there may be another thing to consider, and this may have caused the end-of-file problems I encountered on Diana with PHP 5.3: If the code is built on VMS 8.4, it may cause problems when executed on VS 8.3. It shouldn’t, but because CRTL is involved, it could cause problems.

    All information on the issue can be found on the WASD mailing list, look for “wordpress: CONFIG?” in the 2012 and 2013 directories, and the test programs I used are available here. Note this is a VMS-created ZIPfile so you’ll have to move the file to VMS (binary!) and unzip it there….(The other files in this directory are Windows-based ZIPs containing NOTAPAD files.)


    Locating the right layout
    Now I have VWCMS running, it’s time to locate the right format for my homepage. Of course, I could fall back onto the current one, but it would be nice if there is a new look and feel on the site 🙂
    So I searched for free website templates and I came along They offer a nice tool to create your site – but it has to run on their servers, and to link the domain to the site, it’s no longer free….
    Second sources is that offer a number of ‘free’ templates to be created on…
    Nevertehless, they offer downloadable, CSS-based templates that I can now install to have a look. Since this installation is rather standard, it is possible to automate the basics, and si I now can just run a procedure to do just that:

    Create a directory tree under VWSCM_ROOT – if needed. Some packages have the directory spec in the ZIP-file, others don’t. By examining the location of index.html in the zip file; if there is no directory in front, it must be created and it must be mentioned as target directory. Otherwise, UNZIP will create the tree, so the target is VWCMS:[000000]
    Next, locate two files form VWCMS_ROOT:[STARTER} to that directory.
    Now the site must be added to the configuration:
    Add new lines to VMSCMS_CONFIG: and add them to WASD_CONFIG_MAP.

    It meant a small change in the WASD mapping configuration:

    # VWCMS

    pass /tinymce/* /tinymce/*

    [includefile] wasd_config:vwcms_test.conf

    # Public web

    and this file wasd_config:vwcms_test.conf is extended by the procedure, as well as the VWCMS configuration file, by open/append and writing the appropriate lines.

    One other thing is needed: INDEX.HTML must be renamed to _SITE.HTML, and a line must be added:
    <link href="_vwcms.css" rel="stylesheet" type="text/css" />
    must be added directly after
    <link href="style.css" rel="stylesheet" type="text/css" />

    I did this by copying each line of INDEX.HTML to _SITE.HTML, and add this extra line if the line read contains “HREF=STYLE.CSS” (I uppercased the content for the equation).
    In most cases, this worked, but in some, the line read spans multiple lines in the view, so you need a small edit of _SITE.HTML, but that shows immediately..

    LAst but not least: Change protection of all files to W:RE, and indeed, in most cases the templates can be visisted.

    This makes it easy to adapt the template to fit my needs.


    IP forensics
    The firewall in the router on the edge of the network has the ability to block 192 objects: single addresses, address ranges, or complete networks. I use this to block a number of single addresses and one or two ranges, based on the attempts to break into the FTP or web server. Since these attempts fail because of the use of wrong username or password, these are logged in TCPIP$FTP_RUN.LOG, the WASD access logs or in the accounting file. All mention the IP address from where the access originates, so it is easy to block them: Add the address as an object, add it to a group and you’re done.
    Good side-effects are that these can neiter abuse the web- or mail server anymore: It limits the amount of spam messages as well, and the number of break-in atempts on the webserver are less frequent. However, non-evil access is also blocked – but I don’t care for that: Who intentionally and repeatedly misbehaves should not be surprised to be blocked for non-evil access as well.
    These additions have to be done by hand, it would be nice to automate that, preferably on a daily basis :).
    So I’m now working on a set of procedures to extract data from a number of files.
    Primary log is the log file of the firewall: ALL access is logged in that file, and because the port is shown, it is possible to mention what type of access it is: is it mail, ftp or web access? But it doesn’t mention the intention: is it an acceptable access, or abusive? The amount of consequetive accesses from one address could be a hint for the second, especially where it concerns file transfer or mail, but for web access, it can be very valid. A real determination can only be achieved by examining the log files for the application that handles the protocol: the PMAS logfiles for incoming mail, access logs for the web or the logfiles. Other resources are the operator log and the accounting file.
    Once the output files exists, these can be concatenated and the output sorted, so I can correlate the data in the protocol-based output against the router log – which is a challenge by itself. Hopefully I have all data at hand. Luckily, all protocols are handled on the same (VMS) system, and the router get its time from there as well. It’s good to have one timebase!
    The only possible drawback is that access of blocked addresses or networks is not logged by the router, so once a site has been blocked for a period, it will show up as ‘non-evil’ the next time since there has been no noted access. At least: I didn’t see such a message in the router logfile.

    Today I finished procedures that convert the log files of the router, the spam filter and the webservice. They all have the same record layout. But there is one difference: where PMAS store the full date and time – including hundreds of a second – the router and web access logs show time in seconds; the router doesn’t even show the year; The latter has been handled – quite primitively, but it works. However, as ASCII sort will not produce the right order so I will have think of a way to get around that issue.

    Next stop is examining accounting.dat, on LOGFAIL records (giving me output for FTP access, for instance), and the operator.log file for mail that passed the spam filter; the basics are there already, I just need to reformat the output so that it fits the other files.
    I can also scan the FTP-server log file – but that is opened for write so I’m not certain I can access that file immediately. To be kept in mind…

    Once that is done, the data can be loaded into a database of some sort, and it will be simple to retriever the top-100 abusers and their IP addresses: these to be loaded as objects to be blocked, and that’s it!
    Or use them otherwise 🙂


    Work at hand
    Apart from the PHP issues, there are a few other things under construction: A new homepage, and a suite to process network-related logfiles.
    For the new homepage I plan to use Mark Daniel’s VmsWasdContentManagementSystem – a native VMS executable that can handle this type of posts – even blogging is an option (perhaps, any blog on this site may be redesigned using this package). I had the beta installed, so I removed it to prevent problems that coud arise; downloaded the latest version, (both the sources and the AXP objects), built and installed it. It does require some configuration, and mapping in WASD, and to get famliar with it (and because of the recommendation) I set up the example as in the documentation. But either I don’t understand or mis-interpret the docs, or these are inconclusive (incomplete of plein wrong – I cannot tell), I ended up with a message:
    ERROR 403 -  reported by VWcms
    Site directory not configured!

    To be investigated….
    Network logging
    It’s an idea for quite some time: Scan all incoming network access, find out who’s attempting to hack, or abuse the systems, and shut the door for these people.
    I started today with a program to scan the SYSLOGD logfiles on Diana: the firewall on the dge of the domain logs all access in this file, and when it is over 25.000 blocks in size, it’s cycled, and all cyccled files are stored in a zip file during the monthly maintenance process. Other files to process are the PMAS and FTP logfiles, and the access logs of the webserver.
    So I need a program to convert these files into data that can be stored and analyzed, and that is also capable of updating the firewall with the top-100 addresses; the Vigor is capable of storing 192 single addresses, address ranges or networks that can be denied access – at the gate.
    I started with a DCL-procedure that splits the SYSLOGD output – either active or archived – into incoming and outgoing traffic; each of which is next split into protocol-specific files; so at that moment, I have all lines of logging for every protocol, either incoming or outgoing – in exactly the same, fixed format. Therfore, it’s very easy to extract the required data from these files: date and time of access, the source and destination address and port – and the protocol.
    Since there is quite a number of archives to process, I also created a procedure to scan a directory for these files – put there by hand of by unzipping an archive – and have each file processed that way. I’ve taken a decision to mark each final output file by the date it is created, and once created (if not existing) it will be extended with each SYSLOGD file that is processed.
    This works fine now – next is the extraction of the same data from the PMAS logfiles, but IIRC, that has been done already, I just have to look fro them; otherwise, it is not a lot of work to do the same for these files. The same applies to the web-server access logfiles: Create a procedure that can handle one, and I’m done (just add a wrapper that passes the filename of the file to be processed.).
    And, of course, a program to store this data into a database, a program to analyze the data, and one to update the firewall accoringly.
    A few days ago, I found out – by accident – that the PMAS license expires tomorrow. I sent a request for a new license to the address I know exsists for that type of message – but it bounced. Next, I sent it to the address of Hunter Goatley – who’s in charge of the hobbyist licenses – and that bounced as well. So I sent it to the support desk of Process Software, but since I have a free license, they couldn’t help me; in stead they passed another address – which bounced also, so I was advised to contact Hunter directly – which didn’t bounce for the next hour. So it is likely to arrive; hopefully Hunter is not on holiday, and the license arrives is time – or I’ll be buried under all the messages that PMAS is now blocking ro rejecting…Fingers crossed….