Tag: SciPy

Now trying SciPy. Changed site.cfg to refer the libraries I built:
[openblas]
libraries = blas,lapack
library_dirs = home/wgrooters/ETW/local,home/wgrooters/.local/lib/python2.7/site_packages/nimpy-1.16.3py2.7-linux-x86_64.EGG/numpy
###include_dirs = home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/include
runtime_library_dirs = home/wgrooters/ETW/local,home/wgrooters/.local/lib/python2.7/site_packages/nimpy-1.16.3py2.7-linux-x86_64.EGG/numpy
and both BLAS and LAPACK environment variables. I should have done others as well, probably.
SciPy has no option –user or –prefix like Numpy. But there is –dry-run, that does it all, except building (according the documents). As long as I know what compilation looks like, that’s what I need; though it seems it actually DOES the compilations, the scripts create a Build directory from where the actual build is done; all files that get in here are copied from the directories below the scipy directory – and that are quite a lot of files. And the log seems to contain just what I need to create building procedures on OpenVMS. That’s the next phase.

Scipy site.cfg adjusted again, and started a build – with the same result: numpy fails, so redone what I did before: all swithed off except LAPACK and BLAS:
Export BLAS=/home/wgrooters/ETW/lapack-release-lapack-3.8.0; export LAPACK=/home/wgrooters/ETW/lapack-release-lapack-3.8.0; ATLAS=None; export BLAS_SRC=/home/wgrooters/ETW/lapack-release-lapack-3.8.0/BLAS; export LAPACK_SRC=/home/wgrooters/ETW/lapack-release-lapack-3.8.0/SRC; python setup.py build;
Or use openBLAS?
export BLAS=/home/wgrooters/ETW/xianyti-OpenBLAS-15cb124; export LAPACK=/home/wgrooters/ETW/xianyti-OpenBLAS-15cb124; ATLAS=None; export BLAS_SRC=/home/wgrooters/ETW/xianyti-OpenBLAS-15cb124/lapack-netlib/BLAS; export LAPACK_SRC=/home/wgrooters/ETW/ xianyti-OpenBLAS-15cb124/lapack-netlib/SRC; python setup.py build;
Sources are found this time, but not the libraries. Of course, I should specify the correct locations:
export BLAS=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124; export LAPACK=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/; export ATLAS=None; export BLAS_SRC=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC; export LAPACK_SRC=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/SRC
because the sources are on e deeper level (/SRC). That has a better result on config command:
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:491: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
self.calc_info()
blas_src_info:
FOUND:
sources = ['/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC/caxpy.f', '/home/wgrooters/ETW/xianyi-OpenBLAS-15cb...
...
...S-15cb124/lapack-netlib/BLAS/SRC/zsymm.f', '/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC/ztrsm.f']
language = f77

FOUND:
libraries = [('flapack_src', {'sources': ['/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/SRC/sbdsdc.f', '/home/wgrooters...
...
...b/BLAS/SRC/zsymm.f', '/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC/ztrsm.f'], 'language': 'f77'})]
define_macros = [('NO_ATLAS_INFO', 1)]
but there seems to be more to exclude:
Export MKL=None;
added. Didn’t help.
Should I put a question on a SciPy forum?
Changed the name of the OpenBlas directory – it contained quite a lot of extra naming that has no meaning here, but that didn’t solve the problem, all is put into one library. So I built the separate LAPACK and BLAS libraries using the MakeFile:
make clean
make lapacklib
make blaslib
but that didn’t give the intended result: There is a library liblapack-sandybridge.0.3.6 containing the objects of both LAPACK and BLAS. Not what I wanted: separate libraries.
Searched the Internet and found a description on how to build numpy using BLAS and LAPCK: http://www.tfinley.net/notes/blas-lapack/ . Published in 2007, but it may help.

According the on-line documentation and site.cfg.example OpenBlas is a requirement but LAPACK and BLAS are included – but last in the sequence to be used. So I could use the standard packages as well and rule-out the other ones. Just have to tell the compilation scripts that just these two are to be used: Adapted make.rule and start a build – output into files. See what happens.

I examined the lof and error files created yesterday and has done some editing on site.cfg, because I have created static libraries of LAPACK and BLAS. Given the comment on the SciPy site on building numpy, where it stated:

export BLAS=/path/to/libblas.so
export LAPACK=/path/to/liblapack.so
export ATLAS=/path/to/libatlas.so
python setup.py …………

If you don’t want to any LAPACK, just do “export LAPACK=”.

what I have to do is exporting these libraries as value to the according packages, or nothing if I don’t
need them:
$ export ATLAS=
$ export LAPACK=/home/wgrooters/ETW/lib/liblapack.a
$ export BLAS=/home/wgrooters/ETW/lib/librefblas.a
and run the config command again.
Had to export one more:
$ export OPENBLAS=
and no matter what, ATLAS error keeps showing up:
Running from numpy source directory.
setup.py:390: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
run_build = parse_setuppy_commands()
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
/home/wgrooters/ETW/numpy-1.16.3/numpy/distutils/system_info.py:638: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
self.calc_info()
/tmp/tmpMujJ2q/source.c:1:19: fatal error: cblas.h: No such file or directory
#include
^
compilation terminated.
left that alone for now, just started building:
$ python setup.py build > bld.log 2>bld.err
The result is a lot of warnings on variables that are defined but not used, conflicting types for built-in functions, statements wit no effect, just a few real compilation errors: two about include files (cblas.h, endian.h) not found (first is also mentioned in config command), and one undefined reference to ‘exp’, and an exit status 1 on collect2. In the end things go wrong in linking the shared object – need to recompile with -fPIC parameter. But the build log shows most sources have been compiled using that parameter; or does that have to with LAPACK and BLAS code?

According the numpy documentation, LAPACK and BLAS (and ATLAS) are not a requirement, so iy can be built without them: just disable them (following the doc):
$ export LAPACK=None
$ export BLAS=None
$ export ATLAS=None
and execute the build function as before. Again, it takes a while, the number of errors is far lower – the same as in the first config, the error on ‘exp’ as before, and a few warnings on an unused variable. The log – far smaller as well – shows no further issues – but need to check on the commands given: Does gcc do just the compilation, or does it create shareable objects as well? It seems so (man pages agree)

Now find out what libraries (static or not) have been created in either method, then start with Scipy. Here the same method must be used: define where the libraries are located and see if that builds.
Since there are building procedures for LAPACK/BLAS and Numpy already, this is the last hurdle: Find out what build commands are used to build Scipy. That is what this Linux exercise is all about….