Posted on

Fortran compilations on the way

Now I know how to set things up – in real;
SCIPY_BASE: refers to the package as it has been extracted from the .ZIP file
SCIPY_VMS: refers to the same structure to contain sources that have been adapted to compile on OpenVMS (well, only if needed, not now)
SCIPY_ROOT: refers to both of SCIPY_VMS, followed by SCIPY_BASE (currently just SCIPY_BASE because SCIPY_VMS is still empty).
SCIPY_WG: refers to the location where I create all files to set things up,define them and build the package; all code references are to SCIPY_ROOT so it code requires updates to be compiled in openVMS, I can still use the build proceduee without changes.

one of these files is – obviously – BUILD.COM. In the process of adding subsystems, I added the ability to build ALL or one package, so addition scan be tested without the requirement of redoing all what already had been done.

I could not refer to the logical name of the disk (USER:) but had to user the translation, otherwise I could do $ SET DEF but $ DIR would fail…

Now it has been determined what Fortran options will be used, it is time to work my way forward through all the Fortran compilations, subsystem by subsystem: Add each of the the build-proceduree and run them. As some point there is C-code, but this is bypassed for now: there are a LOT of options in the Linux build output. But for the Fortran code: it seems straight forward for now. I got the the point now where it seems that some numpy code is included – but Fortran code compiles with no problems where this is missing.

Now: just over halfway the fortran code (estimated).

Posted on

First fortran compilations

It has been quite some time ago since I did compile Fortran programs, these were administrative in stead of mathematical, so I needed to check the Fortran manual for the right options; Found what I could extract from the outputted building-log I got from Linux: The compilation succeeded with no error or warning, so that is Ok. Now getting on in building SciPy, subsystem by subsystem; where I need Numpy objects, I can use Brett Cameron’s file to create them: although this is C-only (since Cygwin seems to lack a Fortran compiler) this is fine to start with. And where I need LAPACK and BLAS, I can always fall back on what’s done by Jouk Janssen.
Since SciPy holds a number of ‘subsystems’ for several types of calculations, I intend to take them one by one and have then compiled successfully, and create an object library for each of them. In the end a shared image, but for that, I may have to do the same for numpy – it also contains a number of Fortran sources.

Now I’m in full control of what directory structure is to be used in building it all, I will set things up the way I think is right:
SCIPY_BASE: is the original package, and SCIPY_WG is created aside (not inside) it. Here, I place all files needed to build what’s in SCIPY_BASE, all results come in SCIPY_WG: object files in [.OBJ]. list and map-files in [.LIS], libraries in [.LIB] and executables in [.EXE]. SCIPY_ROOT: will refer to SCIPY_BASE. if no changes to code are required.
Whenever SpiPy code needs to be adjusted, there will also be a SCIPY_VMS: that will need to have the very same structure as SCIPY_BASE; SCIPY_ROOT: will then be referring SCIPY_VMS:,SCIPY_BASE:

The same will apply to Numpy, so the original environments will not be contaminated.

LAPACK is another story. I could do the same there, but that requires quite a lot of work the way it is set up now: the VMS-structure is inside the package, and it requires quite a lot of extensions in both structures before LAPACK_ROOT: can refer to LAPACK_VMS:,LAPACK_BASE:. So I leave it as it is – for now.

In reading the Fortran manual (of 2005, I have no more recent version) I came along the CXML – C(ompaq?) Extended Math Library – that contains LAPACK and BLAS, but also parts of SciPy. The doc mentioned “Alpha only”, so its current (and future) state are questionable. Anyway, I will proceed with the plan – using CXML is bi big deal – just the object(s) in LINK command will change.

Posted on

Start translating

Now I got the output (3 weeks ago :), almost 6000 lines were outputted. I started locating the real building commands: every line noting gfortran, gcc or g++. The real work seems to start around line 4000, where there is a line
building 'dfftpack' library
From there on, the real works seems to start, library after library. That is: in Linux terms.
All lines before seem to set up a building environment – probably depending on something that is not clear in the output; though it seems mostly copying sources to a BUILD directory, and all compilations are executed on these files. Therefore, first focus is on the compilations, not the copy – lines.
Now I have located the real work (set up the compiler options, execute the compiler and following activities like adding the output to a library), it’s time to translate these into the right DCL commands…

(It took so long because there were quite a few other s=things I needed to get my hands on…)

Posted on

Building SciPy itself

Now trying SciPy. Changed site.cfg to refer the libraries I built:
[openblas]
libraries = blas,lapack
library_dirs = home/wgrooters/ETW/local,home/wgrooters/.local/lib/python2.7/site_packages/nimpy-1.16.3py2.7-linux-x86_64.EGG/numpy
###include_dirs = home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/include
runtime_library_dirs = home/wgrooters/ETW/local,home/wgrooters/.local/lib/python2.7/site_packages/nimpy-1.16.3py2.7-linux-x86_64.EGG/numpy
and both BLAS and LAPACK environment variables. I should have done others as well, probably.
SciPy has no option –user or –prefix like Numpy. But there is –dry-run, that does it all, except building (according the documents). As long as I know what compilation looks like, that’s what I need; though it seems it actually DOES the compilations, the scripts create a Build directory from where the actual build is done; all files that get in here are copied from the directories below the scipy directory – and that are quite a lot of files. And the log seems to contain just what I need to create building procedures on OpenVMS. That’s the next phase.

Posted on

Building LAPACK, BLAS and NumPy ‘by hand’

Using the document I found to build SciPy-requirements, started with BLAS. Needed to changed a thing or two, because the Makefiles have changed in the last 12 years – again some research required. In the end I found the command to build the library should be:
gcc -shared -Wl,-soname,libblas.so -L . -lf2c -o libblas.so ./BLAS/SRC/*.o
-L… = define the location where the files to include (-l…) did the job…
Same for LAPACK, but this failed because BLAS wasn’t build correctly, the change above should be done in the ./BLAS/SRC/Makefile, not in ./BLAS/Makefile, and refer to the right (relative) location. It is a fast compilation so simply to redo. Now LAPACK compiles as well, and I have libblas.so and liblapack.so; copied both to a local environment (ETW/local) to be referred to by Numpy (to begin with) and later in ScipY.
Next is Numpy. that should be located here as well (option --prefix $HOME/ETW/local) but alas: specified --PREFIX and this is Unix-like, and therefore case-sensitive
Running from numpy source directory.
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help

error: option --PREFIX not recognized
But in lowercase, it didn’t work either:
error: bad install directory or PYTHONPATH

You are attempting to install a package to a directory that is not
on PYTHONPATH and which Python does not read ".pth" files from. The
installation directory you specified (via --install-dir, --prefix, or
the distutils default setting) was:

/home/wgrooters/ETW/local/lib64/python2.7/site-packages

and your PYTHONPATH environment variable currently contains:

''

Here are some of your options for correcting the problem:

* You can choose a different installation directory, i.e., one that is
on PYTHONPATH or supports .pth files

* You can add the installation directory to the PYTHONPATH environment
variable. (It must then also be on PYTHONPATH whenever you run
Python and want to use the package(s) you are installing.)

* You can set up the installation directory to support ".pth" files by
using one of the approaches described here:

https://pythonhosted.org/setuptools/easy_install.html#custom-installation-locations

Please make the appropriate changes for your system and try again.
Used other options:
$ export ATLAS=None; export BLAS=/home/wgrooters/ETW/local/libblas.so; export LAPACK=/home/wgrooters/ETW/local/liblapack.so; export PYTHONUSERBASE $HOME/ETW/local; python setup.py install --user >bld13.log 2> bld13.err
showed other issues in bld3.err:
/home/wgrooters/ETW/numpy-1.16.3/numpy/distutils/system_info.py:721: UserWarning: Specified path /home/wgrooters/ETW/local/lib is invalid.
return self.get_paths(self.section, key)
/home/wgrooters/ETW/numpy-1.16.3/numpy/distutils/system_info.py:724: UserWarning: Specified path /home/wgrooters/ETW/local/lib is invalid.
path = self.get_paths(self.section, key)
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'define_macros'
warnings.warn(msg)
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’ [enabled by default]
int exp (void);
^
_configtest.o: In function `main':
/home/wgrooters/ETW/numpy-1.16.3/_configtest.c:6: undefined reference to `exp'
collect2: error: ld returned 1 exit status
_configtest.c:1:5: warning: conflicting types for built-in function ‘exp’ [enabled by default]
int exp (void);
^
but looking in bld3.log, compilations did run and both libraries are found and used. Found the numpy libraries in another location: all below /home/wgrooters/.local/lib/python2.7/site-packages//numpy-1.16.3-py2.7-linux-x86_64.egg/numpy/.

So it should be possible to build SciPy…