Category: Software

Scipy site.cfg adjusted again, and started a build – with the same result: numpy fails, so redone what I did before: all swithed off except LAPACK and BLAS:
Export BLAS=/home/wgrooters/ETW/lapack-release-lapack-3.8.0; export LAPACK=/home/wgrooters/ETW/lapack-release-lapack-3.8.0; ATLAS=None; export BLAS_SRC=/home/wgrooters/ETW/lapack-release-lapack-3.8.0/BLAS; export LAPACK_SRC=/home/wgrooters/ETW/lapack-release-lapack-3.8.0/SRC; python setup.py build;
Or use openBLAS?
export BLAS=/home/wgrooters/ETW/xianyti-OpenBLAS-15cb124; export LAPACK=/home/wgrooters/ETW/xianyti-OpenBLAS-15cb124; ATLAS=None; export BLAS_SRC=/home/wgrooters/ETW/xianyti-OpenBLAS-15cb124/lapack-netlib/BLAS; export LAPACK_SRC=/home/wgrooters/ETW/ xianyti-OpenBLAS-15cb124/lapack-netlib/SRC; python setup.py build;
Sources are found this time, but not the libraries. Of course, I should specify the correct locations:
export BLAS=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124; export LAPACK=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/; export ATLAS=None; export BLAS_SRC=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC; export LAPACK_SRC=/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/SRC
because the sources are on e deeper level (/SRC). That has a better result on config command:
/usr/lib64/python2.7/site-packages/numpy/distutils/system_info.py:491: UserWarning:
Blas (http://www.netlib.org/blas/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [blas]) or by setting
the BLAS environment variable.
self.calc_info()
blas_src_info:
FOUND:
sources = ['/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC/caxpy.f', '/home/wgrooters/ETW/xianyi-OpenBLAS-15cb...
...
...S-15cb124/lapack-netlib/BLAS/SRC/zsymm.f', '/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC/ztrsm.f']
language = f77

FOUND:
libraries = [('flapack_src', {'sources': ['/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/SRC/sbdsdc.f', '/home/wgrooters...
...
...b/BLAS/SRC/zsymm.f', '/home/wgrooters/ETW/xianyi-OpenBLAS-15cb124/lapack-netlib/BLAS/SRC/ztrsm.f'], 'language': 'f77'})]
define_macros = [('NO_ATLAS_INFO', 1)]
but there seems to be more to exclude:
Export MKL=None;
added. Didn’t help.
Should I put a question on a SciPy forum?
Changed the name of the OpenBlas directory – it contained quite a lot of extra naming that has no meaning here, but that didn’t solve the problem, all is put into one library. So I built the separate LAPACK and BLAS libraries using the MakeFile:
make clean
make lapacklib
make blaslib
but that didn’t give the intended result: There is a library liblapack-sandybridge.0.3.6 containing the objects of both LAPACK and BLAS. Not what I wanted: separate libraries.
Searched the Internet and found a description on how to build numpy using BLAS and LAPCK: http://www.tfinley.net/notes/blas-lapack/ . Published in 2007, but it may help.

According the on-line documentation and site.cfg.example OpenBlas is a requirement but LAPACK and BLAS are included – but last in the sequence to be used. So I could use the standard packages as well and rule-out the other ones. Just have to tell the compilation scripts that just these two are to be used: Adapted make.rule and start a build – output into files. See what happens.

I examined the lof and error files created yesterday and has done some editing on site.cfg, because I have created static libraries of LAPACK and BLAS. Given the comment on the SciPy site on building numpy, where it stated:

export BLAS=/path/to/libblas.so
export LAPACK=/path/to/liblapack.so
export ATLAS=/path/to/libatlas.so
python setup.py …………

If you don’t want to any LAPACK, just do “export LAPACK=”.

what I have to do is exporting these libraries as value to the according packages, or nothing if I don’t
need them:
$ export ATLAS=
$ export LAPACK=/home/wgrooters/ETW/lib/liblapack.a
$ export BLAS=/home/wgrooters/ETW/lib/librefblas.a
and run the config command again.
Had to export one more:
$ export OPENBLAS=
and no matter what, ATLAS error keeps showing up:
Running from numpy source directory.
setup.py:390: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
run_build = parse_setuppy_commands()
/usr/lib64/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'python_requires'
warnings.warn(msg)
/home/wgrooters/ETW/numpy-1.16.3/numpy/distutils/system_info.py:638: UserWarning:
Atlas (http://math-atlas.sourceforge.net/) libraries not found.
Directories to search for the libraries can be specified in the
numpy/distutils/site.cfg file (section [atlas]) or by setting
the ATLAS environment variable.
self.calc_info()
/tmp/tmpMujJ2q/source.c:1:19: fatal error: cblas.h: No such file or directory
#include
^
compilation terminated.
left that alone for now, just started building:
$ python setup.py build > bld.log 2>bld.err
The result is a lot of warnings on variables that are defined but not used, conflicting types for built-in functions, statements wit no effect, just a few real compilation errors: two about include files (cblas.h, endian.h) not found (first is also mentioned in config command), and one undefined reference to ‘exp’, and an exit status 1 on collect2. In the end things go wrong in linking the shared object – need to recompile with -fPIC parameter. But the build log shows most sources have been compiled using that parameter; or does that have to with LAPACK and BLAS code?

According the numpy documentation, LAPACK and BLAS (and ATLAS) are not a requirement, so iy can be built without them: just disable them (following the doc):
$ export LAPACK=None
$ export BLAS=None
$ export ATLAS=None
and execute the build function as before. Again, it takes a while, the number of errors is far lower – the same as in the first config, the error on ‘exp’ as before, and a few warnings on an unused variable. The log – far smaller as well – shows no further issues – but need to check on the commands given: Does gcc do just the compilation, or does it create shareable objects as well? It seems so (man pages agree)

Now find out what libraries (static or not) have been created in either method, then start with Scipy. Here the same method must be used: define where the libraries are located and see if that builds.
Since there are building procedures for LAPACK/BLAS and Numpy already, this is the last hurdle: Find out what build commands are used to build Scipy. That is what this Linux exercise is all about….

started the ETW VM, had to install some updates (who said Linux doesn’t require as many updates as Windows? It still isn’t as good as VMS, with 2 or 3 a year….at most)
Next, moved the libraries that were created in building LAPACK to a location above the packages, so they can easily be accessed from anywhere during build.
Next give building numpy a try. From the project-root:

$ cd numpy.1.16.3
$ python setup.py config >bldcfg.log 2>bldfg.err

and look at the results: errors because of files that were not found – on their default locations (/usr/lib64). So I need to create site.cfg from the example fila and adjust that to match the locations where the files were put. And check the building docs – on the Internet….

For that, I copies all files I need to examine and adjust to the shared folder so I can do so on the workstation, and pass things over to the VM.