23-May-2008

Performance statistics
The OpenVMS bootcamp must have caused quite some load on Diana – MySQL crashed almost every two days with the known error: “not enough core”. Internet access was very, very slow at times but the hotel network was probably heavily loaded as well.
The web server usage statistics shows some increase, but due to the scaling of both output and number of accesses, it’s not that clear:
web statistics

Regardless the exact cause – that needs to be determined using the T4 output of these days – it is more visible in the HyperSpy output:

CPU load
Note the increment in CPU load from May 14th, and, foremost, May 18th, when the bootcamp really started.

Memory usage
memory load seems pretty much the same, but in detail, it’sfar more peaking then before – and page file usage – the spiky line below – almost doubled.

Processes
It looks rather flat – but that is caused by the spike on the left: a problem that at ties occurs with my Windows-based FTP client. With more detail, it’s notable that the number of processes is far more flat since May 18th.

Paging
As noted with the pagefile usage, the pagefault rate has gone up during the bootcamp – to be expected with an increasing number of processes and heavy database load.

Buffered IO
By far most dramatic: Buffered IO. The only explanation I can give is the amount of data written to SYS$OUTPUT by the PHP engine that drives the blogs.

Direct IO
Because data comes off the disk, it’s obvious that the direct IO rate increased as well – tripled – during the bootcamp. (The high spikes mark the end-of-week processing on Monday, the low ones the end-of-day processing)

I expect that the load will drift to normal proportins in the week after the bootcamp. At that time I will be able to correlate all aspects of the system using T4 output.

20-May-2008

MySQL problem may be tracked down
After a relatively long time, MySQL has stopped again. something I expected to happen: I raised the pagefile quota of MYSQL_SERVER – the account that runs the MySQL server – from 3.000.000 to 4.500.000, because it looked like the crash previously occurred when the amount of virtual memory used was about that amount. I’ve seen it gradually increasing over the week, and yesterday it was approaching the new boundary. And this morning I learned it had restarted (looking at the watchdog log). It’s again about that size, so I wouldn’t be surprised if there will be a restart today again. It grows to this size faster than before since the blogs are more regularly updated (the bootcamp 2008 blog gets new entries every day) and more regularly read as well.
So my suspicion is there is a memory leak in MySQL_Server. Well, upgrade to 5.1 would possibly be the best way to go.

FTP attempts
They do not happen very often in these days, but once in a while there is a more genuine attempt – not a simple script, or one that does check outcome.

In yesterday’s OPERATOR.LOG there was this entry:

%%%%%%%%%%%  OPCOM  19-MAY-2008 20:06:48.38  %%%%%%%%%%%
Message from user TCPIP$FTP on DIANA
        User Name: anonymous
        Source:         59.22.140.17
        Status:         NOPRIV -- File access violation
        Object:         WEB_DISK2:[public.anonymous.test]

TCPIP$FTP_RUN.LOG shows that this is just it, after a timeout (it may be that the server was too busy to respond) and the attempt to create a directory, this user logs out:

%TCPIP-I-FTP_SESCON, FTP SERVER: session connection from 59.22.140.17 at 19-MAY-2008 20:03:36.57
%TCPIP-I-FTP_NODE, client host name: 59.22.140.17
%TCPIP-I-FTP_USER, user name: anonymous
%TCPIP-I-FTP_OBJ, object: 59.22.140.17
%TCPIP-I-FTP_CHINFO, TCPIP$FTPC00039: Can't open data connection
%SYSTEM-F-TIMEOUT, device timeout
%TCPIP-I-FTP_NODE, client host name: 59.22.140.17
%TCPIP-I-FTP_USER, user name: anonymous
%TCPIP-I-FTP_OBJ, object: WEB_DISK2:[public.anonymous.test]
%TCPIP-I-FTP_CHINFO, TCPIP$FTPC00039: Failed to create directory
%SYSTEM-F-NOPRIV, insufficient privilege or object protection violation
%TCPIP-I-FTP_USER, user name: anonymous
%TCPIP-I-FTP_SESDCN, FTP SERVER: session disconnection from 59.22.140.17 at 19-MAY-2008 20:06:48.71

This is reflected in the TCPIP$FTP_ANONYMOUS.LOG:

19-MAY-2008 20:03:37.96 User:anonymous logged in ident:bleh@blah.co.uk from Host:59.22.140.17 
19-MAY-2008 20:03:38.73 User:anonymous ident:bleh@blah.co.uk status:00010001 CWD dir:WEB_DISK2:[public.anonymous]
19-MAY-2008 20:06:48.69 User:anonymous ident:bleh@blah.co.uk logged out

This IP address resides in Korea:

inetnum: 59.0.0.0 - 59.31.255.255
netname: KORNET-KR
descr: Korea Telecom
country: KR
admin-c: IA9-KR
tech-c: IM9-KR
status: ALLOCATED PORTABLE
mnt-by: MNT-KRNIC-AP

Might be an anonymizer, a hacked system, or something else.

Googlebot
keeps trying to access a non-existing directory in the anonymous FTP site:

7-MAY-2008 10:43:44.98 User:anonymous logged in ident:googlebot@google.com from Host:crawl-66-249-73-115.googlebot.com
7-MAY-2008 10:43:45.67 User:anonymous ident:googlebot@google.com status:07649912 CWD dir:perl
7-MAY-2008 10:43:46.29 User:anonymous ident:googlebot@google.com logged out

Sometimes a few days in a row, sometimes a few times a week, and then it backs out for weeks.

18-May-2008

(well, 17-May-2008 at almost 19:00, local time in the US)
PHP
Running the new PHP engine now for well over a week, and no HPARITHM errors so far, nor the weird DEBUG messages. But in accessing MySQL there are these messages in the admin panel:

WordPress database error: [Unknown MySQL error]
SELECT option_value FROM sm_options WHERE option_name = 'rss_0ff4b43bd116a9d8720d689c80e7dfd4' LIMIT 1

just before the WordPress Development Blog chapter, and these:

WordPress database error: [Unknown MySQL error]
SELECT option_value FROM sm_options WHERE option_name = 'rss_867bd5c64f85878d03a060509cd2f92c' LIMIT 1

WordPress database error: [Lost connection to MySQL server during query]
SELECT option_name FROM sm_options WHERE option_name = 'rss_867bd5c64f85878d03a060509cd2f92c'

occur just before the WordPress news header.

I tried to check this with phpMyAdmin, and found there that the current agent is, again, MySQL 3:

MySQL client version: 3.23.49

So I do have to rebuild the MySQL engine again.
But since the blogs have no problem with this, I leave it for now.

15-May-2008

XP SP3
A requirement for the OpenVMS bootcamp is to have your system (that connects to the bootcamp LAN) to be up-to-date, and therefore Demeter, the company laptop, has been upgraded: ServicePack 3 has been installed. It takes quite a lot of time to download, verify (especially that part!) and install the package. For safety, a backup has been made of the Windows directory, so reversing should not be a problem.
After installation and reboot, it seems all is well. Auto-update has been left switched off – I like to keep things in my own hand :). Just wait a while, but when all is really well, the backup can be removed – it takes over 1 Gb on a not-to-big disk,

Remote management
Although I’m not at my normal location – way from that – I’ll be able to keep an eye on the systems, the only thing I won’t be able to do is a manual start the system. When needed, there are other ways to get that done.
But scanning log files and do some activity that does not require a reboot (something to be avoided anyway, and never an issue on Diana :)) can be done over the Internet.
Some weird messages were returned on the Admin pages: Unknown MySQL errors accessing the blog database, on one of the rss feeds. Something to look into, but for the rest it all works.

08-May-2008

PHP ECO2 problem solved
One of the suggestions on the ITRC was to use the undocumented SET WATCH FILE facility to find out what exactly was loaded. As it turned out, there was definitively something wrong in loading the PHP extensions. Whatever I tried, it always stopped at some point – after loading the first or second. It didn’t matter what I left in, or out, I always got the ACCVIO error.
This is the log I kept of these tests.

To keep all information into one file, I created a small test procedure to be run in batch, producing lots of output but having all data at hand in one logfile. One action taken was examining the link date of every executable:

$loop1:
$ f = f$search("PHP_ROOT:[extensions]*.EXE;1")
$ if f .eqs. "" then goto loop1end
$ write sys$output 'f'PHP_ROOT:[extensions]*.EXE;1
$ pipe analyze/image 'file' | search sys$pipe link
$ goto loop1
$loop1end:

PHP_ROOT:[extensions]*.EXE;1 would be the old PHP version, and PHP_ROOT:[extensions]*.EXE;2 the new one.
Drytesting the analysis in a simple test on the CLI-prompt – on both the old and new versions – revealed the dates were 100% equal. Comparing the files reveled NO DIFFERENCES….Something has gone wrong in setting up the environment!

Next, I checked the newly installed version in the Apache environment – and found there were new versions of every extension available, dated on the same date as the PHP and PHPSHR they belonged to: 3-Apr-2008, where the working environment only contained the ECO1 routines, dated 26-Sep-2006. Excepot for the module I built myself, of course.

So I copied all new files, reset the test environment to run the ECO-2 PHP executables, and behold: IT WORKED. So the problem was solved – so far. Next was of course renaming the new versions to become the valid ones, and just test the blog to run. It did – and so I entered this entry using ECO2.

The only problem that might occur is that the MySQL module used is built against the old PHPSHR. Reading the database was found not to cause a problem, and because saving the draft doesn’t moan either. write access is no problem either.

So it might well work. Publishing now….

Update
It does. Good news!

Next: admit the error on ITRC…

What I’ll do in the next update: Keep the PHP environment in the Apache location and run it from there. It would have prevented these problems. Or: In case of a new version, set PHP_ROOT to the Apache installation directory, in case of error change it back to the WASD environment. It means I still need to copy the files form [BIN] and [EXTENSIONS] to the WASD environment BEFORE updating PHP. But that is a way I have walked before.