Month: May 2019

Got account on VSI’s CTRL-C cluster to do the real work – officially I cannot do anything ‘commercial’ on my hobbyist license, but investigation of how to build open-source packages are allowed….
set thing up – for now – environments for OpenBLAS ans Numy: copied ZIP’s of these packages and extracted them here. Start with building OpenBLAS using Jouk Jansen’s MMS files – MMS can handle the make-files which makes life a lot easier. But almost directly there is a problem:
JULIA$ mms

ifndef TOPDIR
%MMS-W-DRVPARSERR, Parser error: "syntax error" in file ./MAKEFILE.SYSTEM, line 8.
%MMS-F-DRVBADPARSE, Parser detected a fatal syntax error in the description file.
JULIA$
That is:
ifndef TOPDIR
TOPDIR = .
Endif
So either ifndef is not understood – or ‘TOPDIR’ or ‘TOPDIR = .’. Perhaps the ALPHA code is designed for TRU64 (or Linux?) so some adjustments must be made for OpenVMS. Take into account that code seems to explicitly uses cache-settings (for optimation). That is: the code for ALPHA is still there but it may no longer be supported – the ALPHA processor is left out of supported CPU’s. Probably just TRU64 or Linux at the time – same applies for IA64. It is however questionable whether these are needed, because the Numpy docs state:

Linear Algebra libraries
NumPy does not require any external linear algebra libraries to be installed. However, if these are available, NumPy’s setup script can detect them and use them for building. A number of different LAPACK library setups can be used, including optimized LAPACK libraries such as ATLAS, MKL or the Accelerate/vecLib framework on OS X.

So whatever |Jouk Jansen has done already should be sufficient – easy enough

Disabling ATLAS and other accelerated libraries¶
Usage of ATLAS and other accelerated libraries in NumPy can be disabled via:
BLAS=None LAPACK=None ATLAS=None python setup.py build

so theer mey be need for a procedure to set things up for building (setup.py). This is a requirement for building the whole lot using Python. So I need Python on the cluster as well – with DisUtils installed/ It must be installed with ZLIB; but this is likely standard. Tried it on Diana: it looks like:

$ blas="none"
$ lapack="none"
$ atlas="none"
$ python setup.py build
Running from numpy source directory.
Traceback (most recent call last):
File "setup.py", line 415, in
setup_package()
File "setup.py", line 394, in setup_package
from numpy.distutils.core import setup
File "/USER/etw/numpy-1.16.3/numpy/distutils/__init__.py", line 6, in
from . import ccompiler
File "/USER/etw/numpy-1.16.3/numpy/distutils/ccompiler.py", line 18, in
from numpy.distutils import log
File "/USER/etw/numpy-1.16.3/numpy/distutils/log.py", line 10, in
from .misc_util import (red_text, default_text, cyan_text, green_text,
File "/USER/etw/numpy-1.16.3/numpy/distutils/misc_util.py", line 12, in
import multiprocessing
File "/python_root/lib/multiprocessing/__init__.py", line 84, in
import _multiprocessing
ImportError: No module named _multiprocessing

so somewhere this fails in basic code:

#
# Imports
#

import os
import sys

from multiprocessing.process import Process, current_process, active_children
from multiprocessing.util import SUBDEBUG, SUBWARNING

#set def
# Exceptions
#

class ProcessError(Exception):
pass

class BufferTooShort(ProcessError):
pass

class TimeoutError(ProcessError):
pass

class AuthenticationError(ProcessError):
pass

# This is down here because _multiprocessing uses BufferTooShort
import _multiprocessing < <<-----------------------ERROR # # Definitions not depending on native semaphores #

This seems more Python specific. Question JFP: What's going on here? Can it be suppressed?

Downloaded OpenBLAS. This can be used because LAPACK (and BLAS) are included – latest versions (3.6.0) so this can be build uising Jouk Jansen’s files. Next it is finding out what can be done in Numpy.
Also got MatPlotLib – this contains three C/C++ packages that are required – and these may also be covered by Jouk Jansen – and for matPlotlib it is just a number of C sources, it is mainly a number of Python aources. This should also be possible.
NumPy itself is mostly Python – just some C sources

Check what Jean-François Piéronne states in his building procedures: Python_root:[vms]python_compiler_options.com sets the options for C, build_modules.py and build_cython_modules.py actually build the modules but also add them directoy into pythonshr.exe – I doubt this is required for these routines, there is a large number of C and Fortran sources that need to be compiled first. Numpy is the first module to build because it is a requirement for SciPy.
Also downloaded the next part requested – again an older version (2.2.4) because of the limitation to Python 2.7. Didn’t install it yet, first process NumPy and SciPy. According the building procedure, in site.py, it is required to locate them in python_root:[lib.site-packaes] :

“””Append module search paths for third-party packages to sys.path.

****************************************************************
* This module is automatically imported during initialization. *
****************************************************************

In earlier versions of Python (up to 1.5a3), scripts or modules that
needed to use site-specific modules would place “import site”
somewhere near the top of their code. Because of the automatic
import, this is no longer necessary (but code that does it still
works).

This will append site-specific paths to the module search path. On
Unix (including Mac OSX), it starts with sys.prefix and
sys.exec_prefix (if different) and appends
lib/python/site-packages as well as lib/site-python.
On other platforms (such as Windows), it tries each of the
prefixes directly, as well as with lib/site-packages appended. The
resulting directories, if they exist, are appended to sys.path, and
also inspected for path configuration files.

A path configuration file is a file whose name has the form .pth; its contents are additional directories (one per line)
to be added to sys.path. Non-existing directories (or
non-directories) are never added to sys.path; no directory is added to
sys.path more than once. Blank lines and lines beginning with
‘#’ are skipped. Lines starting with ‘import’ are executed.

For example, suppose sys.prefix and sys.exec_prefix are set to
/usr/local and there is a directory /usr/local/lib/python2.5/site-packages
with three subdirectories, foo, bar and spam, and two path
configuration files, foo.pth and bar.pth. Assume foo.pth contains the
following:

# foo package configuration
foo
bar
bletch

and bar.pth contains:

# bar package configuration
bar

Then the following directories are added to sys.path, in this order:

/usr/local/lib/python2.5/site-packages/bar
/usr/local/lib/python2.5/site-packages/foo

Note that bletch is omitted because it doesn’t exist; bar precedes foo
because bar.pth comes alphabetically before foo.pth; and spam is
omitted because it is not mentioned in either path configuration file.

After these path manipulations, an attempt is made to import a module
named sitecustomize, which can perform arbitrary additional
site-specific customizations. If this import fails with an
ImportError exception, it is silently ignored.

“””

So boyh Numpy and SciPy should have a .pfh file – but in neither of them such a file exists – or the need to be created.

Second: both packages seem to require ATLAS and OpenBLAS, LAPACK and BLAS are not mentioned but a solution should be possible by placing blas-routines in a library (as LAPACK already has). using the mentioned options and adjustments to the build-procedures, it should be possible to build Numpy (found info here, it also contains references to the building tools like setuptools.

Searched for the versions I can use; latest versions are (as to be expected) based on Python3.x, so I’ll have to deal with an older one. The last usable is version 1.2.1 for SciPy, for NumPy 1/16/3 is the lastest and according the specification, that should work with Python 2.7 as well. But where to find that one? Asked on StackOverflow and got all of them
Next I put all packages in a working environment, and looked into the Linux/Unix makefiles how to build it, and asked on OpenVMS.org.
To build the packages, Python seems a requirement, so downloaded the latest disk-images for Alpha from the Python_on_VMS download pages and installed them, changed startup and login procedures and got it running. But following the SciPy building instructions, it doesn’t work:

Compiling /python_root/local/mpmath/libmp/exec_py3.py ...
File "/python_root/local/mpmath/libmp/exec_py3.py", line 1
exec_ = exec
^
SyntaxError: invalid syntax

and

Compiling /python_root/local/mpmath/tests/extratest_gamma.py ...
File "/python_root/local/mpmath/tests/extratest_gamma.py", line 50
print("%s ok;" % name, end=' ')
^
SyntaxError: invalid syntax


probably these are significant: it looks as if these are related to Python 3 or a test of balancing colors on screen. These need to be solved of course.

Next check how SciPy and Numpy are built: are C and Fortran used ‘under the hood’?