10-Mar-2018

Database running on IRIS
Downloaded the latest version of MariaDB from Mark Berryman (5.5.59) which installed flawlessly (as expected) and running the MySQL_Install_db script went fine, until I (again) encountered an error, but now it shows the reason:

180310 18:43:30 [Note] $3$lda2:[000000.mysql055.][bin.ia64]mysqld.exe;1 (mysqld 5.5.59-MariaDB) starting as process 555746351 ...
180310 18:43:30 InnoDB: The InnoDB memory heap is disabled
180310 18:43:30 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
180310 18:43:30 InnoDB: Compressed tables use zlib 1.2.11
180310 18:43:31 InnoDB: Initializing buffer pool, size = 128.0M
180310 18:43:32 InnoDB: Assertion failure in thread 2070851264 in file [freeware.mariadb-5^.5^.59.storage.xtradb.os]os0sync.c;1 line 123
InnoDB: Failing assertion: pthread_cond_init(cond, NULL) == 0
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to https://jira.mariadb.org/
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
%DEBUGBOOT-W-EXPGFLQUOTA, exceeded pagefile quota

though the user (myqsl051_Svr) should have enough (2.000.000) blocks. Doubling it made no difference…

Checking sys$system:pagefile.sys showed the reason: it was just 604064 blocks in size. WAY to small, so I had to increase it’s size to match the requirement – for now, this is 3000000. Perhaps a second pagefile should have been netter, but this was the easy fix. Required a reboot of the system, after which there was no problem at all running the script. At least, up to starting the final startup of the server: This failed, tgime after time. So added a logfile to be created (the SUBMIT statement has “/nolog” option, changed that to “/log=mysql055_root:[MySQL_server]” to get it there) But: No file created….
Mind-wave: take a look into accounting – and behold:

10-MAR-2018 19:17:21 LOGFAIL             MYSQL051_SRV 2140042D          00D3810C

Same for every attempt.
$ ACC/FULL/TYPE=LOGFAIL to get more detail:

LOGIN FAILURE
-------------
Username:          MYSQL051_SRV      UIC:               [MYSQL051,MYSQL051_SRV]
Account:                      Finish time:       10-MAR-2018 19:17:21.34
Process ID:        2140042D          Start time:        10-MAR-2018 19:17:21.29
Owner ID:                            Elapsed time:                0 00:00:00.04
Terminal name:                       Processor time:              0 00:00:00.01
Remote node addr:                    Priority:          4
Remote node name:                    Privilege <31-00>: 00108000
Remote ID:                           Privilege <63-32>: 00000000
Remote full name:
Posix UID:         -2                Posix GID:         -2 (%XFFFFFFFE)
Queue entry:       2                 Final status code: 00D3810C
Queue name:        SYS$BATCH
Job name:          start_mysqld
Final status text: %LOGIN-F-DISUSER, account is disabled
Page faults:              164        Direct IO:                 12
Page fault reads:           4        Buffered IO:               14
Peak working set:        3056        Volumes mounted:            0
Peak page file:        173680        Images executed:            1

Of course. Creating a user requires “/flags=NODISUSER” to activate. And I forgot that one.
So
mc authorize mod mysql051_srv/flags=nodisuser

and redo startup. And this time:

$ sho sys/proc=mar*
OpenVMS V8.4  on node IRIS   10-MAR-2018 20:00:40.07   Uptime  0 00:50:34
  Pid    Process Name    State  Pri      I/O       CPU       Page flts  Pages
21400434 MariaDB_Server  HIB      6     2914   0 00:00:00.96     14466  15845 M

So now I do have a database running, to be filled: Get the backup from Diana (which is an SQL script) and run it. (tomorrow – than this entry will be there as well).

Now the database is running, get WASD – 11.2 (latest), which I will install on DIANA as well.

04-Mar-2018

Setting up IRIS
Getting on setting up IRIS, one of the big Itanium boxes – to hold the same software as DIANA (the DS10), involved new software to be installed. Started with the MariaDB database (5.5.58) as downloaded (and installed on DIANA). Setting up the software is no problem, but when starting the script to create an empty database, this failed when creating certificates: the script refers to SSLROOT: which doesn’t exist, and later on, the server program crashes. But I made a mistake in the beginning so no database was created.
Missing SSLROOT is easily solved:
$ DEFINE SSLROOT SSL$ROOT:
and I restarted the script – now without errors. Creating certificates was no problem – but the server still crashed:

Running MySQL for the first time...
Using Mailbox MBA564:

180304 20:10:41 InnoDB: The InnoDB memory heap is disabled
180304 20:10:41 InnoDB: Mutexes and rw_locks use InnoDB's own implementation
180304 20:10:41 InnoDB: Compressed tables use zlib 1.2.6
180304 20:10:42 InnoDB: Initializing buffer pool, size = 128.0M
180304 20:10:43 InnoDB: Assertion failure in thread 2070851264 in file [freeware.mariadb-5^.5^.25.storage.xtradb.os]os0sync.c;1 li4
InnoDB: Failing assertion: 0 == pthread_mutex_init(fast_mutex, MY_MUTEX_INIT_FAST)
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
%SYSTEM-F-OPCCUS, opcode reserved to customer fault at PC=FFFFFFFF848DDC60, PS=0000001B
%TRACE-F-TRACEBACK, symbolic stack dump follows
%TRACE-I-END, end of TRACE stack dump
%RMS-E-EOF, end of file detected

There is a new version of the software available (5.5.59), so that may solve these problems. And a new version of WASD webserver as well, to be installed on both AXP as Itanium, later this week.

Silence
Diana’s fan now has stalled…But it’s pretty cool up here, and the system isn’t very busy, so it won’t be much of a problem for a short time. A new fan has been ordered – waiting to be delivered this week.

01-Mar-2018

Maintenance
Monthly maintenance in itself showed no problems. Mail was as usual:

PMAS statistics for February
Total messages    :   2456 = 100.0 o/o
DNS Blacklisted   :      0 =    .0 o/o (Files:  0)
Relay attempts    :   1280 =  52.1 o/o (Files: 28)
Accepted by PMAS  :   1176 =  47.8 o/o (Files: 28)
  Handled by explicit rule
         Rejected :    842 =  71.5 o/o (processed),  34.2 o/o (all)
         Accepted :    116 =   9.8 o/o (processed),   4.7 o/o (all)
  Handled by content
        Discarded :    161 =  13.6 o/o (processed),   6.5 o/o (all)
     Quarantained :     49 =   4.1 o/o (processed),   1.9 o/o (all)
        Delivered :      8 =    .6 o/o (processed),    .3 o/o (all)

but there has been something strange in the antirelay logfiles: between 10-Fen and today, these are a mixture of PMAS.log content and PMAS_ANTIRELAY.LOG. All files were therefore over 60 blocks in size. Searching through these files for the antirelay status (571) held just two files that actually required inspection: the ones of 11th and 25th:

11-FEB-2018 between 21:24:54.19 and 21:28:00.87, from address 23.254.211.233 (190 entries)
25-FEB-2018 between 20:29:42.78 and 25-FEB-2018, from address 23.254.203.20 (146 entries).

In both cases, used a bogus user from this domain, trying to relay to locotrones1029@gmail.com. Once again, hosted by Hostwinds.com; so I’ll warn them again.

Last Itanium system into cluster
The two ‘big’ servers (IRIS and INDRA) are added to the cluster already, but one (INGE) was still to do.
Other than I initially thought, these two big servers have two single-core processors, without hyperthreading. Being similar in hardware (and from the same supplier), it looks these are the older ones of the three. That only CPU #1 is added to the set of CPU’s, is therefore obvious.
INGE, once booted, shows that CPU’s #1, 2 and 3 are added, so it is a newer one. However, it is the one causing problems. I had the machine running when I added the other systems but I couldn’t do that one because before, there as an issue with the SCSI controller so the machine didn’t boot when the system disk was in slot 0 (PUN0, LUN 0) – where is was when I installed VMS on it. Moved it to slot # 2 (PUN2, LUN 0) where I could boot is when accessing EFI and started it by accessing this disk directly and start FSO:\EFI\VMS\VMS_LOADER). But on the EFI boor menu, it was still residing on PUN0 LUN0. So I need to do it interactively.

However, trying to access the ILO yesterday failed: The ILO was non-responsive to PING, TELNET and HTTP on the designated address. So today it was a matter of finding out the cause. Thinking it might be the battery failing (though the supplier said he replaced it) so I took the IKO out, put in a new battery, and re-installed it. Since DHCP is enabled by default, I could figure out what address it would have – but on DIANA, $ DHCPDBDUMP didn’t show any address that did work. $ TCPIP SHO HOST gave me a hint – since the name of the management port is MP<macaddress> now I could access MP, set the configuration to what it should be, and work from there.: Booted the machine from bay#2 – and that worked. Next, shutdown the machine to find out what message was given if the disk was in slot#0 – where it should be. This time, there was nothing wrong; perhaps because when the ILO was out, I could check the connector, may have moved it a bit… SO I added the machine to the cluster. Now being able to access the disk containing the new licenses, this machine is set to work until 12-Mar-2019 – like all others.
Hopefully, this wail keep working – fingers crossed.
Next stage is setting this machine up (as well as INDRA) as DIANA and IRIS, and a quorum node on a laptop running a 64-bit version of Windows (so not the current one).

Fan bearings gone
The main fan of the DS10 server (DIANA) is getting very loud – sounds the bearings are gone. A new one has been ordered but it is uncertain when it will arrive….