Tag: SciPy

So adapt VMS_autu64.for: Remove the directory from the name of the mapfile (which is input from commandline) and use what’s left to add to the line: scan this name from back to fron to “:” or “]”, or beginning. If not at beginning,take the part that was scanned (leaving what’s in front alone), otherwise use the string.
It’s a matter of getting used to Fortran after so many years, especially where it is Fortran90 (not FORTRAN77)

Adapting makefiles is a huge undertaking, and not without problems because of nesting.
So fall back on ‘basic’ compilation – but it seems OpenBLAS may nog include all subprojects. the 3.8.0 version of LAPACK taken from GitHub contains LAPACK, BLAS and LAPACKE/ It may be sufficient to use these in stead of what is included in OpenBLAS, at least it is possible te refer to libraries on different locations.
So use the lasts LAPACK (and therefore BLAS) version, add the adaptions of Jouk Jansen in directory [.VMS] and define logicals referring this one before the base ones. Also split the output: .OBJ files into [.OBJ], list- and mapfiles in [.LIS], libarties in [.LIB] and executables in [.EXE]/ Because Jouk Jansen’s procedure puts all files in [.SRC]. be it, in this configuration, the ones in the [.VMS] structure. But it is (in my opinion) a bad habit….
It means the MMS files need to be adapted so the results are put into the right location.

Created procedure SetETW.com that sets the required logicals, and these must be mentioned in the MMS files. This means adaptation but (hopelfully) straight forward. First part is the [.INSTALL] environment: but here something fails in FORTRAN files that are to contain the 64-bit versions, compiled with C: the directory is added – and the right name is truncated:

$ mms

cc/define=(VMS)/comment=as_is/prep=LAPACK_INS:LAPACK_version.f_ LAPACK_INS:LAPACK_version.f
fortran/name=lower/float=ieee/ieee=denorm/list=LAPACK_LIS:/show=all/obj=LAPACK_OBJ:LAPACK_version.obj LAPACK_INS:LAPACK_version.f_
pipe link/map=LAPACK_LIS:/full/exec=nl: LAPACK_OBJ:LAPACK_version | copy sys$input nl:
mc etwexe:vms_auto64 LAPACK_LIS:LAPACK_version.map LAPACK_LIS:LAPACK_version.lis
normal end
cc/define=(VMS)/comment=as_is/prep=LAPACK_INS:LAPACK_version_64.f_ LAPACK_INS:LAPACK_version_64.f
fortran/name=lower/float=ieee/ieee=denorm/obj=LAPACK_OBJ:LAPACK_version_64.obj LAPACK_INS:LAPACK_version_64.f_

subroutine dummy__LAPACK_LIS:LAPA0940_64__
....................................^
%F90-E-ERROR, Syntax error, found IDENTIFIER 'LAPA0940_64__' when expecting one of: DO FORALL SELECT SELECTCASE WHERE IF
at line number 1 in file LAPACK_VMS:[INSTALL]LAPACK_version_64.f;3

subroutine dummy__LAPACK_LIS:LAPA0940_64__
^
%F90-E-ERROR, An unterminated block exists.
at line number 1 in file LAPACK_VMS:[INSTALL]LAPACK_version_64.f;3
%MMS-F-ABORT, For target LAPACK_version.obj, CLI returned abort status: %X186A1262.
$
The file starts:

subroutine dummy__LAPACK_LIS:LAPA0940_64__
end
# 1 "LAPACK_VMS:[INSTALL]LAPACK_version.f;1"
*> \brief \b LAPACK_VERSION

and these lines are added by:

mc etwexe:vms_auto64 LAPACK_LIS:$(MMS$TARGET_NAME).map LAPACK_LIS:$(MMS$TARGET_NAME).lis

or is this in line:

cc/define=(VMS)/comment=as_is/prep=LAPACK_INS:$(MMS$TARGET_NAME).f_\
LAPACK_INS:$(MMS$TARGET_NAME).f

No: it is vms_auto64, the other creates the .f_ file but that is correct. So vms_aut064 must be examined, it should use just the filename. Another option is to leave the 64-bit files out of compilations – for now

New versions of VMS patches by TU Delft: do OpenBLAS first, in principle can be done using MMS but makefile needs to be adapted first so the statement that cannot be handled (and are not accessed) are bypassed by adding:

ifneq($(OSNAME), VMS)

what should disable the part beyond this until according endif. But for some reason it didn’t work: MMS fails on

ifndef


No reply by JFP???

Because the standard method of SciPy using Python doesn’t seem to be usable, tried JFP’s method as entry point but then the structure of NumPy must be scrutinized. Apart from OpenBLAS (effectively standalone) there are C-sources to be compiled, and the order in which this done, may be crucial. So examine the building procedures.

Because OpenBLAS is said to be a requirement for NumPy, take a look there first. For LAPACK and BLAS there are VMS-specific procedures so keep these outside building OpenBLAS. There is a makefile for Alpha (and one for IA64) where there is no notion of OS, bt it is to be expected that TRU64 or Linux were in mind, but it also mentions a Comapq C-compiler so it might be it works for VMS as well. ATLAS is mentioned here as well (version 7.8) but given the location it is expected to reside on, it could be this is for TRU64 or Linux:

LIBATLAS = -L/usr/lib/atlas3.7.8 -lf77blas -latlas -lm

Building shows VMS isn’t supported:

$ mms

ifndef TOPDIR
%MMS-W-DRVPARSERR, Parser error: "syntax error" in file ./Makefile.system, line 8.
%MMS-F-DRVBADPARSE, Parser detected a fatal syntax error in the description file.
$...
NETLIB_LAPACK_DIR = $(TOPDIR)/lapack-netlib


So it might be handy to have GNV at hand – installed and activated on Diana, it should solve this. However, this gives yet another error:

$ bash
bash$ make
Makefile.system:15: Extraneous text after `else' directive
Makefile.system:17: Extraneous text after `else' directive
Makefile.system:17: *** only one `else' per conditional. Stop.
bash$

This is in the makefile – it can be left out (= commented out) because it is of no use for Alpha. But then it fails similarly on line 620:

bash$ make
Makefile.system:207: Makefile.conf: no such file or directory
Makefile.system:620: Extraneous text after `else' directive
Makefile.system:759: Extraneous text after `else' directive
Makefile.system:620: Extraneous text after `else' directive
Makefile.system:759: Extraneous text after `else' directive
Makefile.system:620: Extraneous text after `else' directive
Makefile.system:759: Extraneous text after `else' directive

so stopped with CTRL-C: same here:

ifeq ($(ARCH), $(filter $(ARCH),mips64))
ifdef BINARY64
CCOMMON_OPT += -mabi=64
else
CCOMMON_OPT += -mabi=n32
endif
BINARY_DEFINED = 1
else ifeq ($(ARCH), $(filter $(ARCH),mips)) < <<<------------- ERROR CCOMMON_OPT += -mabi=32 BINARY_DEFINED = 1 endif

and further down trhe code (line 759):

ifdef NO_BINARY_MODE
ifeq ($(ARCH), $(filter $(ARCH),mips64))
ifdef BINARY64
FCOMMON_OPT += -mabi=64
else
FCOMMON_OPT += -mabi=n32
endif
else ifeq ($(ARCH), $(filter $(ARCH),mips)) < <<<---------------- ERROR FCOMMON_OPT += -mabi=32 endif else ifdef BINARY64 ifneq ($(OSNAME), AIX) FCOMMON_OPT += -m64 endif

It seems GNV's make cannot handle nested ifdef - else idef … constructs. Mention this as a bug in the GNV ticket list, because this needs to eb solved.
Checked this on the cluster: Neither Python nor GNV are installed here: check with Robert.

Got account on VSI’s CTRL-C cluster to do the real work – officially I cannot do anything ‘commercial’ on my hobbyist license, but investigation of how to build open-source packages are allowed….
set thing up – for now – environments for OpenBLAS ans Numy: copied ZIP’s of these packages and extracted them here. Start with building OpenBLAS using Jouk Jansen’s MMS files – MMS can handle the make-files which makes life a lot easier. But almost directly there is a problem:
JULIA$ mms

ifndef TOPDIR
%MMS-W-DRVPARSERR, Parser error: "syntax error" in file ./MAKEFILE.SYSTEM, line 8.
%MMS-F-DRVBADPARSE, Parser detected a fatal syntax error in the description file.
JULIA$
That is:
ifndef TOPDIR
TOPDIR = .
Endif
So either ifndef is not understood – or ‘TOPDIR’ or ‘TOPDIR = .’. Perhaps the ALPHA code is designed for TRU64 (or Linux?) so some adjustments must be made for OpenVMS. Take into account that code seems to explicitly uses cache-settings (for optimation). That is: the code for ALPHA is still there but it may no longer be supported – the ALPHA processor is left out of supported CPU’s. Probably just TRU64 or Linux at the time – same applies for IA64. It is however questionable whether these are needed, because the Numpy docs state:

Linear Algebra libraries
NumPy does not require any external linear algebra libraries to be installed. However, if these are available, NumPy’s setup script can detect them and use them for building. A number of different LAPACK library setups can be used, including optimized LAPACK libraries such as ATLAS, MKL or the Accelerate/vecLib framework on OS X.

So whatever |Jouk Jansen has done already should be sufficient – easy enough

Disabling ATLAS and other accelerated libraries¶
Usage of ATLAS and other accelerated libraries in NumPy can be disabled via:
BLAS=None LAPACK=None ATLAS=None python setup.py build

so theer mey be need for a procedure to set things up for building (setup.py). This is a requirement for building the whole lot using Python. So I need Python on the cluster as well – with DisUtils installed/ It must be installed with ZLIB; but this is likely standard. Tried it on Diana: it looks like:

$ blas="none"
$ lapack="none"
$ atlas="none"
$ python setup.py build
Running from numpy source directory.
Traceback (most recent call last):
File "setup.py", line 415, in
setup_package()
File "setup.py", line 394, in setup_package
from numpy.distutils.core import setup
File "/USER/etw/numpy-1.16.3/numpy/distutils/__init__.py", line 6, in
from . import ccompiler
File "/USER/etw/numpy-1.16.3/numpy/distutils/ccompiler.py", line 18, in
from numpy.distutils import log
File "/USER/etw/numpy-1.16.3/numpy/distutils/log.py", line 10, in
from .misc_util import (red_text, default_text, cyan_text, green_text,
File "/USER/etw/numpy-1.16.3/numpy/distutils/misc_util.py", line 12, in
import multiprocessing
File "/python_root/lib/multiprocessing/__init__.py", line 84, in
import _multiprocessing
ImportError: No module named _multiprocessing

so somewhere this fails in basic code:

#
# Imports
#

import os
import sys

from multiprocessing.process import Process, current_process, active_children
from multiprocessing.util import SUBDEBUG, SUBWARNING

#set def
# Exceptions
#

class ProcessError(Exception):
pass

class BufferTooShort(ProcessError):
pass

class TimeoutError(ProcessError):
pass

class AuthenticationError(ProcessError):
pass

# This is down here because _multiprocessing uses BufferTooShort
import _multiprocessing < <<-----------------------ERROR # # Definitions not depending on native semaphores #

This seems more Python specific. Question JFP: What's going on here? Can it be suppressed?