There still wasn’t a complete build of Numpy using just lapack and blas as an example., si I had to redo the build – from scratch – on the Linux system. Done that and have output to guide me – but whether it is complete and fit for VMS on Alpha, remains to be seen. Combined with Brett’s DCL-procedure, it should be possible to get on.
Start with Numpy
Now XBLAS and LAPACk are done, I can start with Numpy. Brett Cameron has done this before, I can use that as the base for the command procedures that will be like the one I did with SCipy: part by part. Some files have been added by Brett and I added these files to my environment. But there were still some files to be adapted, where internal definitions in the compilers differ: gcc would use “__alpha__” where my (Compaq) C version (7.3) uses “__alpha” to set the CPU type. Otherwise, the CPU type could not be determined. I guess this will nor be the only one.
Once these changes were included, I could start building core.common – the first one already fails beyond repair (all is the same as defined by Brett – and that seemed to work. I just added /deb/noopt, /lis= and /obj= options because that conforms my environment):
$ cc/accept=novaxc_keywords/noopt/deb/names=(as_is,shortened)/define=(_OSF_SOURCE,_USE_STD_STAT)/include=cc$include/warn=disable=(
QUESTCOMPARE,QUESTCOMPARE1)/lis=NUMPY_LIS:/mach/obj=NUMPY_OBJ:/arch=host array_assign.c
static NPY_INLINE int PyInt_Check(PyObject *op) {
......................^
%CC-E-CLOSEPAREN, Missing ")".
at line number 35 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
if (!PyLong_Check(op)) {
.........^
%CC-E-BADEXPR, Invalid expression.
at line number 37 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
#define PyInt_AS_LONG PyLong_AsLong
......................^
%CC-W-MACROREDEF, The redefinition of the macro "PyInt_AS_LONG" conflicts with a current definition because one is object-like and t
he other is function-like. The redefinition is now in effect.
at line number 46 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
#define PyString_Check PyBytes_Check
.......................^
%CC-W-MACROREDEF, The redefinition of the macro "PyString_Check" conflicts with a current definition because one is object-like and
the other is function-like. The redefinition is now in effect.
at line number 89 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
#define PyString_AS_STRING PyBytes_AS_STRING
...........................^
%CC-W-MACROREDEF, The redefinition of the macro "PyString_AS_STRING" conflicts with a current definition because one is object-like
and the other is function-like. The redefinition is now in effect.
at line number 93 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
#define PyString_GET_SIZE PyBytes_GET_SIZE
..........................^
%CC-W-MACROREDEF, The redefinition of the macro "PyString_GET_SIZE" conflicts with a current definition because one is object-like a
nd the other is function-like. The redefinition is now in effect.
at line number 99 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
PyUnicode_ConcatAndDel(PyObject **left, PyObject *right)
................................^
%CC-E-NOTEXPECTING, Error parsing parameter list. Found "*" when expecting one of: ",", ")".
at line number 152 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_SETREF(*left, PyUnicode_Concat(*left, right));
....^
%CC-E-BADEXPR, Invalid expression.
at line number 154 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_SETREF(*left, PyUnicode_Concat(*left, right));
....^
%CC-E-BADSTMT, Invalid statement.
at line number 154 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(right);
....^
%CC-E-BADEXPR, Invalid expression.
at line number 155 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(right);
....^
%CC-E-BADSTMT, Invalid statement.
at line number 155 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
PyUnicode_Concat2(PyObject **left, PyObject *right)
...........................^
%CC-E-NOTEXPECTING, Error parsing parameter list. Found "*" when expecting one of: ",", ")".
at line number 159 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_SETREF(*left, PyUnicode_Concat(*left, right));
....^
%CC-E-BADEXPR, Invalid expression.
at line number 161 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_SETREF(*left, PyUnicode_Concat(*left, right));
....^
%CC-E-BADSTMT, Invalid statement.
at line number 161 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
npy_PyFile_Dup2(PyObject *file, char *mode, npy_off_t *orig_pos)
.........................^
%CC-E-NOTEXPECTING, Error parsing parameter list. Found "*" when expecting one of: ",", ")".
at line number 172 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(ret);
....^
%CC-E-BADEXPR, Invalid expression.
at line number 191 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(ret);
....^
%CC-E-BADSTMT, Invalid statement.
at line number 191 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(os);
....^
%CC-E-BADEXPR, Invalid expression.
at line number 206 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(os);
....^
%CC-E-BADSTMT, Invalid statement.
at line number 206 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(ret);
....^
%CC-E-BADEXPR, Invalid expression.
at line number 211 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(ret);
....^
%CC-E-BADSTMT, Invalid statement.
at line number 211 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io);
........^
%CC-E-BADEXPR, Invalid expression.
at line number 236 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io);
........^
%CC-E-BADSTMT, Invalid statement.
at line number 236 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io_raw);
........^
%CC-E-BADEXPR, Invalid expression.
at line number 242 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io_raw);
........^
%CC-E-BADSTMT, Invalid statement.
at line number 242 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(ret);
....^
%CC-E-BADEXPR, Invalid expression.
at line number 261 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(ret);
....^
%CC-E-BADSTMT, Invalid statement.
at line number 261 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
npy_PyFile_DupClose2(PyObject *file, FILE* handle, npy_off_t orig_pos)
..............................^
%CC-E-NOTEXPECTING, Error parsing parameter list. Found "*" when expecting one of: ",", ")".
at line number 278 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io);
........^
%CC-E-BADEXPR, Invalid expression.
at line number 314 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io);
........^
%CC-E-BADSTMT, Invalid statement.
at line number 314 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io_raw);
........^
%CC-E-BADEXPR, Invalid expression.
at line number 319 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(io_raw);
........^
%CC-E-BADSTMT, Invalid statement.
at line number 319 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
Py_DECREF(ret);
....^
%CC-E-BADEXPR, Invalid expression.
at line number 340 in file NUMPY_ROOT:[numpy.core.include.numpy]npy_3kcompat.h;1
%CC-F-TOOMANYERR, More than 30 errors were encountered in the course of compilation.
Here I must consult Bret on how to get around this – he must have mastered this before.
Building LAPACK, BLAS and NumPy ‘by hand’
Using the document I found to build SciPy-requirements, started with BLAS. Needed to changed a thing or two, because the Makefiles have changed in the last 12 years – again some research required. In the end I found the command to build the library should be:
gcc -shared -Wl,-soname,libblas.so -L . -lf2c -o libblas.so ./BLAS/SRC/*.o
-L… = define the location where the files to include (-l…) did the job…
Same for LAPACK, but this failed because BLAS wasn’t build correctly, the change above should be done in the ./BLAS/SRC/Makefile, not in ./BLAS/Makefile, and refer to the right (relative) location. It is a fast compilation so simply to redo. Now LAPACK compiles as well, and I have libblas.so and liblapack.so; copied both to a local environment (ETW/local) to be referred to by Numpy (to begin with) and later in ScipY.
Next is Numpy. that should be located here as well (option --prefix $HOME/ETW/local) but alas: specified --PREFIX and this is Unix-like, and therefore case-sensitive
Running from numpy source directory.
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: option --PREFIX not recognized
But in lowercase, it didn’t work either:
error: bad install directory or PYTHONPATH
You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:
/home/wgrooters/ETW/local/lib64/python2.7/site-packages
and your PYTHONPATH environment variable currently contains:
''
Here are some of your options for correcting the problem:
* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files
* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)
* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:
https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations
Please make the appropriate changes for your system and try again.
Used other options:
$ export ATLAS=None; export BLAS=/home/wgrooters/ETW/local/libblas.so; export LAPACK=/home/wgrooters/ETW/local/liblapack.so; export PYTHONUSERBASE $HOME/ETW/local; python setup.py install --user >bld13.log 2> bld13.err
showed other issues in bld3.err:
/home/wgrooters/ETW/numpy-1.16.3/numpy/distutils/system_info.py:721: UserWarning: Specified path /home/wgrooters/ETW/local/lib is invalid.
return self.get_paths(self.section, key)
/home/wgrooters/ETW/numpy-1.16.3/numpy/distutils/system_info.py:724: UserWarning: Specified path /home/wgrooters/ETW/local/lib is invalid.
path = self.get_paths(self.section, key)
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'define_macros'
warnings.warn(msg)
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’ [enabled by default]
int exp (void);
^
_configtest.o: In function `main':
/home/wgrooters/ETW/numpy-1.16.3/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’ [enabled by default]
int exp (void);
^
but looking in bld3.log, compilations did run and both libraries are found and used. Found the numpy libraries in another location: all below /home/wgrooters/.local/lib/python2.7/site-packages//numpy-1.16.3-py2.7-linux-x86_64.egg/numpy/.
So it should be possible to build SciPy…
Numpy built on Linux – got my commands. And Brett’s DCL file
Build Numpy on the Linux machine (just numpy, leave the rest for now):
python setup.py build -j 1 install --prefix $HOME/.local >build.log 2>build.err
and copied the resulting outputfiles to workstation to examine.
Brett has corrected the login on CTRL-C so I can login again; rolled back the backup file he placed there, and copied that as well to find out what he created. His environment is Python 3.5 – not 2.7 – but that should not be a problem – some changes will be needed anyway. Checked this against what my own build on Linux made of it: pretty much the same (this is Python 2.5 – but the build procedure did not complain)
Setting up Linux environment as a testbed for compilation commands
To build the Numpy C files, I need to get some documentation on C and C++ – but this is missing on the VSI site. Asked Brett, got that – plus what he had done on this in his spare time: He installed the bunch on a Windows PC running Cygwin, started a build (in C only) and trapped the output in a file; so he could transfer that into a plain, straightforward DCL procedure that compiled it all, placed modules in a library, and created the shared image. He passed it all in a backup file on the CTRL-C cluster – but this has been updated to VMS OpenVMS, and my login failed – because I was denied access to two files.. Asked him to adjust….
In the mean time, this is the way to go: Build in an environment that is known to work (some way or another), pick up the output and translate that into DCL – straight-forward, without any intelligence to be gin with.
So I set up a virtual machine running CentOS 7, installed all updates which caused a bunch of issues; but finally I had a more or less stable Linux environment where I installed the packages (OpenBLAS, NumPy en SciPy) and on VMS, I installed latest versions of MMK and Make from the VMS Freeware CD’s (that I have online) – may not be the latest but for now this will do.
Had a mail from Guido: proceed with this set up: Base and additions apart, add another layer for my own changes – that means that the adapted MMS files will reside in yet another layer.