Posted on by SYSMGR

Building OpenBLAS – and got a production environment at VSI

Got account on VSI’s CTRL-C cluster to do the real work – officially I cannot do anything ‘commercial’ on my hobbyist license, but investigation of how to build open-source packages are allowed….
set thing up – for now – environments for OpenBLAS ans Numy: copied ZIP’s of these packages and extracted them here. Start with building OpenBLAS using Jouk Jansen’s MMS files – MMS can handle the make-files which makes life a lot easier. But almost directly there is a problem:
JULIA$ mms

ifndef TOPDIR
%MMS-W-DRVPARSERR, Parser error: "syntax error" in file ./MAKEFILE.SYSTEM, line 8.
%MMS-F-DRVBADPARSE, Parser detected a fatal syntax error in the description file.
JULIA$
That is:
ifndef TOPDIR
TOPDIR = .
Endif
So either ifndef is not understood – or ‘TOPDIR’ or ‘TOPDIR = .’. Perhaps the ALPHA code is designed for TRU64 (or Linux?) so some adjustments must be made for OpenVMS. Take into account that code seems to explicitly uses cache-settings (for optimation). That is: the code for ALPHA is still there but it may no longer be supported – the ALPHA processor is left out of supported CPU’s. Probably just TRU64 or Linux at the time – same applies for IA64. It is however questionable whether these are needed, because the Numpy docs state:

Linear Algebra libraries
NumPy does not require any external linear algebra libraries to be installed. However, if these are available, NumPy’s setup script can detect them and use them for building. A number of different LAPACK library setups can be used, including optimized LAPACK libraries such as ATLAS, MKL or the Accelerate/vecLib framework on OS X.

So whatever |Jouk Jansen has done already should be sufficient – easy enough

Disabling ATLAS and other accelerated libraries¶
Usage of ATLAS and other accelerated libraries in NumPy can be disabled via:
BLAS=None LAPACK=None ATLAS=None python setup.py build

so theer mey be need for a procedure to set things up for building (setup.py). This is a requirement for building the whole lot using Python. So I need Python on the cluster as well – with DisUtils installed/ It must be installed with ZLIB; but this is likely standard. Tried it on Diana: it looks like:

$ blas="none"
$ lapack="none"
$ atlas="none"
$ python setup.py build
Running from numpy source directory.
Traceback (most recent call last):
File "setup.py", line 415, in
setup_package()
File "setup.py", line 394, in setup_package
from numpy.distutils.core import setup
File "/USER/etw/numpy-1.16.3/numpy/distutils/__init__.py", line 6, in
from . import ccompiler
File "/USER/etw/numpy-1.16.3/numpy/distutils/ccompiler.py", line 18, in
from numpy.distutils import log
File "/USER/etw/numpy-1.16.3/numpy/distutils/log.py", line 10, in
from .misc_util import (red_text, default_text, cyan_text, green_text,
File "/USER/etw/numpy-1.16.3/numpy/distutils/misc_util.py", line 12, in
import multiprocessing
File "/python_root/lib/multiprocessing/__init__.py", line 84, in
import _multiprocessing
ImportError: No module named _multiprocessing

so somewhere this fails in basic code:

#
# Imports
#

import os
import sys

from multiprocessing.process import Process, current_process, active_children
from multiprocessing.util import SUBDEBUG, SUBWARNING

#set def
# Exceptions
#

class ProcessError(Exception):
pass

class BufferTooShort(ProcessError):
pass

class TimeoutError(ProcessError):
pass

class AuthenticationError(ProcessError):
pass

# This is down here because _multiprocessing uses BufferTooShort
import _multiprocessing < <<-----------------------ERROR # # Definitions not depending on native semaphores #

This seems more Python specific. Question JFP: What's going on here? Can it be suppressed?