31-Aug-2012

IP forensics
The firewall in the router on the edge of the network has the ability to block 192 objects: single addresses, address ranges, or complete networks. I use this to block a number of single addresses and one or two ranges, based on the attempts to break into the FTP or web server. Since these attempts fail because of the use of wrong username or password, these are logged in TCPIP$FTP_RUN.LOG, the WASD access logs or in the accounting file. All mention the IP address from where the access originates, so it is easy to block them: Add the address as an object, add it to a group and you’re done.
Good side-effects are that these can neiter abuse the web- or mail server anymore: It limits the amount of spam messages as well, and the number of break-in atempts on the webserver are less frequent. However, non-evil access is also blocked – but I don’t care for that: Who intentionally and repeatedly misbehaves should not be surprised to be blocked for non-evil access as well.
These additions have to be done by hand, it would be nice to automate that, preferably on a daily basis :).
So I’m now working on a set of procedures to extract data from a number of files.
Primary log is the log file of the firewall: ALL access is logged in that file, and because the port is shown, it is possible to mention what type of access it is: is it mail, ftp or web access? But it doesn’t mention the intention: is it an acceptable access, or abusive? The amount of consequetive accesses from one address could be a hint for the second, especially where it concerns file transfer or mail, but for web access, it can be very valid. A real determination can only be achieved by examining the log files for the application that handles the protocol: the PMAS logfiles for incoming mail, access logs for the web or the logfiles. Other resources are the operator log and the accounting file.
Once the output files exists, these can be concatenated and the output sorted, so I can correlate the data in the protocol-based output against the router log – which is a challenge by itself. Hopefully I have all data at hand. Luckily, all protocols are handled on the same (VMS) system, and the router get its time from there as well. It’s good to have one timebase!
The only possible drawback is that access of blocked addresses or networks is not logged by the router, so once a site has been blocked for a period, it will show up as ‘non-evil’ the next time since there has been no noted access. At least: I didn’t see such a message in the router logfile.

Today I finished procedures that convert the log files of the router, the spam filter and the webservice. They all have the same record layout. But there is one difference: where PMAS store the full date and time – including hundreds of a second – the router and web access logs show time in seconds; the router doesn’t even show the year; The latter has been handled – quite primitively, but it works. However, as ASCII sort will not produce the right order so I will have think of a way to get around that issue.

Next stop is examining accounting.dat, on LOGFAIL records (giving me output for FTP access, for instance), and the operator.log file for mail that passed the spam filter; the basics are there already, I just need to reformat the output so that it fits the other files.
I can also scan the FTP-server log file – but that is opened for write so I’m not certain I can access that file immediately. To be kept in mind…

Once that is done, the data can be loaded into a database of some sort, and it will be simple to retriever the top-100 abusers and their IP addresses: these to be loaded as objects to be blocked, and that’s it!
Or use them otherwise 🙂

30-Aug-2012

Weird PHP issue
All of a sudden, any blog that has run fine for years, run into an error:

Warning: require(/sysblog/000000/wp-includes/class.wp-dependencies.php) [function.require]:
failed to open stream: no such file or directory in /sysblog/wp-includes/script-loader.php on
line 3

Fatal error: require() [function.require]: Failed opening required
'/sysblog/000000/wp-includes/class.wp-dependencies.php' (include_path='.:/php_root/000000') in
/sysblog/wp-includes/script-loader.php on line 3

This happened also on the Trips,Tracks and Travels blog, and that runs WP 263 (where thos blog uses WP 265).
If I would have changed something, it could be the result of that change. But I didn’t….

So I tried a different approach, since it might be a web-issue, and ran the blog immediately under PHP:
$ php :== $php_root:[bin]php
$ set def sysblog:[000000]
$ php index.php

and that gave exactly the same outcome.

Also checked the (no longer published but still present) Boorcamp blogs: same result.
What the heck was going on here?

Looking back – on line – when this could have started, I found the earliest:
...[30/Aug/2012:06:25:59 +0000] "GET /sysblog/index.php?p=333 HTTP/1.1" 200 767
and to look even further back, I have to cycle the access log….

So I mailed the WASD mailing list on the issue, and tried again to find out some more information, and accessed the Trips,Tracks and Tavels blog again.
BEHOLD: NO PROBLEM….
I again accessed the sysmgr blog (this one), and here as well, it all worked again like before. That is also shown in the access log:
...[30/Aug/2012:08:41:06 +0000] "GET /sysblog/index.php HTTP/1.1" 200 37951
Mind the last numer: the size of the reply…

Who knows the answer, please inform me…
It must be someting in the system – but what could it be?

(There is something not entirely right: because the admin pages seem to require a re-load. Even on posting. But that can be worked around)

27-Aug-2012

WASD update additions
The update left two links in error, both concerning performance logging: HyperSpixx, showing stored performance data, and Mondesi, for real-time control. The first one requires an agent that stores data in files, and these are presented uing a CGI-based execuatble. That program can no longer determine the nodename for which data is to be displayed, and when I restarted the agent, it failed without notice…
The second program is started from the browser, and requests data from the system using the system API to collect performance data, and displayes it in the browser window. The link does display the window but slightly distorted, but no centent….
Today, I installed the latest versions of these programs.
Mondesi – now version 2.1 – was installed without a problem, and it now shows the correct data.
HyperSpi – the WASD version – has been buiklt and installed, the agent started; it does start and created a file. But displaying the data still fails: No node supplied….
The version by Jean-François Piéronne however causes problems in building. It comes as a complete package that must now be compiled and linked – where originally, the object files were supplied and LINK was sufficient. so you now need a C-compiler, and, for the libraries used, you must use MMS (or Hunter Goatley’s MMK), if you want to use the BUILD.COM procedure.
I found the DESCRIP.MMS files used in buidling the library are incomplete – so I had to update the files before I could create the agent and web-application. The agent has been installed and works – but the web-application still gives an error ‘No node supplied’ – and the documentation that is supplied doesn’t show a reason….Or how to define the node. But here, there has been no change….
So still some work to do…
DCLInABox works
The terminal amulation over port 80 – using WebSockets – works, once a [world] entry was added to WASD_CONFIG_AUTH.CONF:
[world]
if (websocket:) * r+w,https:

before the authentication for the site. And this addition prevents direct access to the terminal connection., since the site itself is still acessed over SSL, and requires login just for that. In the admin page however, you won’t see the user that started the application. But that could well be handled by the logger.

24-Aug-2012

WASD updated
It took some preparation because quite a lot of basic config of the server (not the sites) has changed and so the process needed some time: Another naming convention and location of logicals, and a change in configuration-schema made this update less straight forward than normally is the case. Including full startup (and shutdown) of the web server – and surrounding software: like the PHP engine and mail support.
Not forgetting the daily, weekly and monthly processing – it all needed to be overhauled.

To be sure I could reverse, if needed, the whole installation was to be performed on a new directory tree, fully separated – as far as the server environment was involved – from the old version.
Most preparation could be done before the server was updated, but it could only be checked after the new version was installed. That process itself required the running (9.3) server to be shut down. So I did, and it will have caused some timeout when the content was accessed.
I could have done simpler, perhaps – since there is a CLONE option, but that did not exist in pre-v10 versions. i think.
The update itself was flawless, as could be expected. Restarting the server itself was neither – but since the installation did copy sample configuration into the environment, I couldn’t get to the site itself. I had to remove these files ($ delete WASD*.CONf;0), then stop and restart the server before I could access the sites. Onl;y a few tweaks were required in the configuration (things did change….), and I could start the site used for system management: handy, because that also faciliates the server admin pages – including the WATCH facility. But other sites – including the normal site, were not, and the WATCH output immediately showed why: I still had to copy a number of files from the previous environment; including some executables and files that reside within the server environment.
Now, within an hour, at least most of the most important features seem to work. Just mail access doesn’t work, but that will be solved soon…
Update
The last hurdles have been taken…I had made a minor adjustment to the SoyMail configuration file, and that causes a double-quote to get lost. That’s why the footer didn’t show – and I couldn’t display the messages…Problem solved.
Other minor issues were easily solved by copying some files from the old to new environment. It’s just the Java-based terminal emulator that doesn;t work – but that is useless anyway so I can get rid of it. There is an alternative I can use with this version of WASD – and latest Chrome browser (and next versions of Firefox and IE), based on websockets…. So next action is to build this HTTP-based terminal emulator

Update 2
Done, and included in the system management site – which is (as can be expected) protected. It does work – running Chrome since that’s currently the only browser on my system that supports WebSockets – but connecting to the site will fail on authentication. Checked that with Mark Daniel who provided the code: it’s not fit -yet – to be run in an authenticated environment.
So now I have to find a way to make that path free from authentication and still make the access impossible ig you’re not authenticated. Can be done, but requires good thinking….

15-Aug-2012

More investigations
I have done a bit more investigation on the PHP issue.
First, I created a procedure in which I can set the PHP environment as /JOB logicals so I can do the tests without interferendce with the access over the Internet.
Next, I checked the WordPress versions I haev installed. Apart from 2.6.3, I have installed 2.6.5, 2.8.2 and 3.4.1. I tested each of them after setting the PHP version to use (either 5.2.6 of 5.3.14), directly on the root directory of that WP version – so VERY basic.
Even using 5.2.6 I had problems – where this wasn’t the case a few days ago: On each of the blogs, PHP complained a file could not be located:

$ set def WP:[000000] ! the root directory of that WP version
$ php index.php

X-Powered-By: PHP/5.2.6
Content-type: text/html

<br />
<b>Warning</b>: require(/WEB_DISK2/private/wp263/wp-includes/class.wp-dependencies.php) [<a
href='function.require'>function.require</a>]: failed to open stream: no such file or directory in
<b>/WEB_DISK2/private/wp263/WP-INCLUDES/script-loader.php</b> on line <b>3</b><br />
<br />
<b>Fatal error</b>: require() [<a href='function.require'>function.require</a>]: Failed
opening required '/WEB_DISK2/private/wp263/wp-includes/class.wp-dependencies.php' (include_path='.:/php_root/000000') in
<b>/WEB_DISK2/private/wp263/WP-INCLUDES/script-loader.php</b> on line <b>3</b><br />

The problem here is that this filename contains a dot: the file does exists:

$ dir [.wp-includes]class.wp-dependencies.php

Directory WEB_DISK2:[private.wp263.WP-INCLUDES]

class^
⁁.wp-dependencies.php;1

Total of 1 file.
(except in WP34, but that is to be expected: I didn’t change the file there)

The only reason I can think of is that the tests a few days ago – or work done yesterday – has removed something….

The same runs using PHP 5.3.14 didn’t run into this issue, even with the older WP versions; apart from the ‘depricated’ messages, any version runs into error:
PHP Parse error: syntax error, unexpected $end in
/sysblog/000000/wp-includes/class.wp-styles.php on line 72

It means the file – including the dotted name – is actually found.
WP 2.8.2 didn’t show the ‘depricated’ messages, so that one seemed to be Ok for PHP 5.3. But there was this same error in the end…
The WP3.4 gave a slightly different message:
PHP Parse error: syntax error, unexpected '>' in /web_disk2/private/wp34/wp-blog-header.php on line 19
So not ‘unexpected $end’; it could mean theer si something else wrong – but since it’s off-the-sheld, ist simply should work…

Bu the web has no trouble with it – so I tried – with PHP 5.2.6 – and removed the private PHP logicals, and used the /SYSTEM/EXEC defined once: Same problems….But for the web access, nothing changed: it simply runs like it has done for months….

Next, I redefined thet SYSMGR blog to run from WP 2.8.2 – being the latest version before WP 3.0; And by access from the web, I remembered why it wasn’t released before: It uses some form of Redirect which causes a loop when the mapping in WASD isn’t altered: accoring the WATCH output, it is obvious that is does: time after time, it runs into the bolded line:

redirect /sysblog /sysblog/index.php
redirect /sysblog/ /sysblog/index.php?
redirect /sysblog/**/ /sysblog/*/index.php?
exec /sysblog/**.php (phpwasd:)/sysblog/*.php \
ods=5 script=syntax=unix script=query=none map=once
pass /sysblog/* /sysblog/* ods=5 search=none dir=noaccess

Quite likely that WP 3.4 runs into the same problem

So I set things up to use 2.6.5, and now the blogs run on that version; there is yet another issue to be tested: Does WP 3.4 require another mapping…

PMAS license
The new license has arrived and is now installed. Just in time 🙂