by Shai Revzen, ca. 2006. These instructions refer to old versions of libraries, etc., but may nonetheless provide help for those installing everything from source code on Gentoo linux.
This page provides a step by-step record of an experience installing a full PyDSTool system from scratch on a Linux Gentoo distribution.
I've taken the approach of installing all the PyDSTool requisites in their own filesystem branch, without overwriting the main Gentoo tools. It is advisable to use this approach because Gentoo tools are constantly updated and may break backward compatibility with PyDSTool, even if were possible to get them to work. Furthermore, so far PyDSTool does not work with any of the "cutting edge" versions of Python used in Gentoo.
A second significant installation choice is that I'm installing as root for shared use computer -- i.e. setting things up so that multiple students can use PyDSTool from their own accounts independently and without steppin g on each other's toes. If you are installing PyDSTool only for one user, there might be easier ways to do it.
Based on the instructions in this wiki, I downloaded the following:
Numeric-23.8.tar.gz
ipython-0.7.2.tar.gz
numarray-1.4.1.tar.gz
fftw-2.1.5.tar.gz
SciPy_complete-0.3.2.tar.gz
matplotlib-0.80.tar.gz
scipy_distutils-latest.tar.gz
Python-2.3.6.tar.bz2
PyDSTool-0.83.3.060921.zip
We'll use Gentoo to provide the optimized ATLAS/LAPACK library. The Gentoo library already deals with combining the ATLAS and LAPACK libraries to provide an optimized LAPACK (off-the-web ATLAS does not provide a full LAPACK, and just installing both libraries will not make LAPACK algorithms use the ATLAS optimization),
Gentoo also provides swig and gnuplot at no extra effort.
interactive=1 emerge blas-atlas interactive=1 emerge lapack-atlas emerge swig emerge gnuplot
Unpack python and configure it to use an alternative install path:
tar xjvf Python-2.3.6.tar.bz2 cd Python-2.3.6 ./configure --prefix=/usr/local/pydstool/ --enable-shared make && make install
Make our new python the default for the current shell:
export PATH=/usr/local/pydstool/bin/:$PATH ldconfig /usr/local/pydstool/lib which python python -c 'print "ok"'
The which python command should return
/usr/local/pydstool/bin/python
If that worked, then the python -c 'print "ok"' command should print ok. If you get a library error, try to give
ldconfig /usr/local/pydstool/lib/ && python -c "print 'ok'"
Gentoo calls the Fortran ABI blas library libblas.so, instead of the libf77blas.so that numarray expects. The easy solution is to make a symlink to alias libf77blas.so to libblas.so, but I didn't want to pollute my system directories with manually generated symlinks. You'd think that the --library-dirs option to distutil's config would be pretty reliable, and we could use that to make distutil look in the current directory. No such luck: although documented, the option has no effect at all for numarray.
So we make a mildly intrusive set of symlinks, just to make numarray happy, setup with "external LAPACK" and test:
ln -s /usr/lib/blas/threaded-atlas/libblas.so /usr/local/lib/libf77blas.so ln -s /usr/include/atlas/ /usr/local/include/atlas python setup.py --use_lapack config --gencode install python -c 'import numarray.testall as testall; testall.test()'
The good news about installing Numeric is that it works easily, just by using python setup.py install
The bad news is that all configuration *MUST* be done by patching the setup.py file. Here's a shell snippet that creates the patch:
cat >Numeric-23.8-setup.patch <<ENDS-PATCH
--- Numeric-23.8/setup.py 2005-02-08 14:58:25.000000000 -0800
+++ Numeric-23.8-setup.py 2006-12-24 21:34:36.000000000 -0800
@@ -19,7 +19,7 @@
MA_version = version
headers = glob (os.path.join ("Include","Numeric","*.h"))
-extra_compile_args = [] # You could put "-O4" etc. here.
+extra_compile_args = ["-O3"] # You could put "-O4" etc. here.
mathlibs = ['m']
define_macros = [('HAVE_INVERSE_HYPERBOLIC',None)]
undef_macros = []
@@ -32,20 +32,20 @@
mathlibs = []
# delete all but the first one in this list if using your own LAPACK/BLAS
-sourcelist = [os.path.join('Src', 'lapack_litemodule.c'),
- os.path.join('Src', 'blas_lite.c'),
- os.path.join('Src', 'f2c_lite.c'),
- os.path.join('Src', 'zlapack_lite.c'),
- os.path.join('Src', 'dlapack_lite.c')
- ]
+sourcelist = [os.path.join('Src', 'lapack_litemodule.c') ] #,
+# os.path.join('Src', 'blas_lite.c'),
+# os.path.join('Src', 'f2c_lite.c'),
+# os.path.join('Src', 'zlapack_lite.c'),
+# os.path.join('Src', 'dlapack_lite.c')
+# ]
# set these to use your own BLAS;
-library_dirs_list = []#'/usr/lib/atlas']
-libraries_list = []#'lapack', 'cblas', 'f77blas', 'atlas', 'g2c']
+library_dirs_list = ['/usr/lib/blas/threaded-atlas/']#'/usr/lib/atlas']
+libraries_list = ['lapack','cblas']#'lapack', 'cblas', 'f77blas', 'atlas', 'g2c']
# set to true (1), if you also want BLAS optimized matrixmultiply/dot/innerproduct
-use_dotblas = 0
-include_dirs = []#'/usr/include/atlas']
+use_dotblas = 1 # 0
ENDS-PATCH
Now we apply the patch and install Numeric:
tar xzvf Numeric-23.8.tar.gz cd Numeric-23.8.tar.gz patch < ../Numeric-23.8-setup.patch python setup.py install
The unit test is rather slow. It is found by running:
python Test/test.py
A nice "sanity" test that Numeric is installed is
python Demo/life.py
This one is "pure" python, so it's easy:
python setup.py install
That's it. Sometimes it just works.
wxPython is useful for scipy GUI and matplotlib. So as to prevent it from conflicting with the Gentoo package, I installed from source and let it use the Gentoo wxGTK -- to no avail -- the Gentoo wxGTK is a wx 2.6 GTK-2 -- way too advanced for the installer to recognize.
So we build out own wxGTK:
export WXPREF=/usr/local/pydstool/wxPython
cd wxPythonSrc-2.4.2.4
mkdir build
cd build
../configure --with-gtk \
--prefix=$WXPREF \
--enable-rpath=$WXPREF/lib \
--with-opengl \
--enable-geometry \
--enable-optimise \
--enable-debug_flag
make
make install
cd ../wxPython
python setup.py \
BUILD_GLCANVAS=0 \
WX_CONFIG=/usr/local/pydstool/bin/wx-config \
build install
cd demo
python demo.py
fftw needs to be installed twice - once for double and once for single precision numbers. The library must be made from the old 2.x version sources (3.x API is not backward compatible).
Un-tar the code in the obvious way, change to the tar directory and:
./configure \
--enable-shared --enable-threads \
--enable-static \
--enable-type-prefix --prefix=/usr/local/pydstool/
make
make install
./configure \
--enable-shared --enable-threads \
--enable-static --enable-float \
--enable-type-prefix --prefix=/usr/local/pydstool/
make
make install
Surprisingly well behaved, it installed with:
tar xzvf SciPy_complete-0.3.2.tar.gz
cd SciPy_complete-0.3.2
python setup.py install
Test it with:
ipython
import scipy
scipy.test(level=1, verbosity=2)
If you are brave, run the full test:
scipy.test(level=10, verbosity=2)
It runs 986 tests in 82.397s on my machine
Similarly easy is matplotlib, except for a little annoying setup bug (see below)
tar xzvf matplotlib-0.80.tar.gz
cd matplotlib-0.80
python setup.py build install
Expect a lot of warnings. If all worked well, you can now see a graph:
cd ..
ipython
from pylab import *
t = arrayrange(0,20,0.05)
s = sin(t)
plot(t,s)
show()
If this crashed at the import, make sure the right rc file is being loaded:
import matplotlib
matplotlib.matplotlib_fname()
The result should be:
/usr/local/pydstool/share/matplotlib/.matplotlibrc
If it isn't, you probably didn't believe me that the cd .. is needed, but there is a .matplotlibrc with a bogus configuration in the matplotlib-0.80 root directory. Guess how many hours I spent to figure that out....
Make sure that the following entries are in the rc file:
backend : WXAgg
interactive : True
If you really need help doing this, you can give the following shell commands to apply a patch:
cd /usr/local/pydstool/share/matplotlib/ cat >matplotlibrc.patch <<ENDS-PATCH --- .matplotlibrc 2005-04-07 11:56:27.000000000 -0700 +++ /usr/local/pydstool/share/matplotlib/.matplotlibrc 2007-01-17 00:06:14.000000000 -0800 @@ -28,9 +28,9 @@ # - a legal html color name, eg red, blue, darkslategray #### CONFIGURATION BEGINS HERE -backend : GTKAgg # the default backend +backend : WXAgg # the default backend numerix : Numeric # Numeric or numarray -interactive : False # see http://matplotlib.sourceforge.net/interactive.html +interactive : True # see http://matplotlib.sourceforge.net/interactive.html toolbar : toolbar2 # None | classic | toolbar2 timezone : UTC # a pytz timezone string, eg US/Central or Europe/Paris ENDS-PATCH patch <matplotlibrc.patch && rm matplotlibrc.patch
Since this machine has a "dedicated" python for PyDSTool, it makes sense to put the library in the site-packages of that python instance:
export P=`pwd`/PyDSTool*zip
cd /usr/local/pydstool/lib/python2.3/site-packages
unzip $P
Now test our newfound glory with:
cp -rv /usr/local/pydstool/lib/python2.3/site-packages/PyDSTool/tests ~
cd ~/tests
ipython
run run_all_tests.py
We have to run the tests from our homedir? because they require write permissions for temporary files, and are not using the /tmp directory or python's tempfile interface
If you want all your users to have access to the PyDSTool package when they run ipython, there's an option that will actually make life easier: add into the /etc/env.d so that Gentoo's own env-update scripts configure the libraries and path:
As root, type:
cat > /etc/env.d/99pydstool <<END
LDPATH="/usr/local/pydstool/lib"
PATH="/usr/local/pydstool/bin"
END
This will set up a file called /etc/env.d/99pydstool that tells Gentoo's env-update script to configure our new tools at the very end of the search path. If you want to check whether this worked, open a new shell that was not configured for out PyDSTool and type
env-update
. /etc/profile
ipython
You should get a PyDSTool capable ipython, running from /usr/local/pydstool/bin. A nice corollary is that everything will work just fine after emerge-ing new packages, i.e. you won't need to run ldconfig.
WARNING: I haven't tested this thoroughly yet... mileage may vary.
Debugging distutils is a pain in the ass. The first thing to try is adding DISTUTILS_DEBUG=1 to your shell environment. But here's a tiny change that made me much happier: in distutils/spawn.py modify the function spawn to add a debug printout on the first line after the docstring:
print "#: "," ".join(cmd) #!!!
This causes distutils to generate make-like output showing all the commands that are spawned. It's pretty useful when things go bad, and keeps you from falling asleep when the build is slow.
Another misadventure was due to a Gentoo rather than a PyDSTool problem, but it is hard to track: the PyDSTool's tests failed when using scipy.gplt, with a useless error message (Broken Pipe). It turned out that gnuplot failed consistentently because it wanted libXaw.so.8 - which Gentoo doesn't have at this point in time. This turned out to me a depedency through libplot.so, left over from pre-modular xorg (if this means nothing to you - be very happy!).
Solution:
emerge plotutils
emerge gnuplot
And try again. Everything works!