Compiling the NFINDR cython version on WindowsΒΆ
The endmembers extraction algorithm NFINDR exist in two different versions. The first version is a pure Python implementation. It live in the nfindr.py file. The second one is a cython implementation, it live in the nfindr.pyx file and is compiled as nfindr.pyd. Both have the same class interface defined in the file eea_int.py.
There is tree main differences between these versions. The first one is speed, on the test data set the cython version is 2x faster than the pure python version. The second one is the use of float64 by the cython version for the volume expansion calculation giving a more precise result. The pure python version use float32.
But it is at the implementation that they differ the most. As usual, the pure python version is to be run everywhere. And, the goal of the cython version is speed improvement. The NFINDR algorithm is a good candidate for speed improvement. They are:
- by vector operations (using GPU or AVX slates or something else).
- by parallelism, the algoritm’s extern loop is highly parallel.
- by optimizing the memory use, almost all the computation is done in a small array.
For the current version, a tight memory management is implemented. The idea is to avoid the frenzy buffers creation and deletion associated when doing the lapack determinant function call. The solution is to reuse the memory allocated. The determinant and LU decomposition was rewrote in cython in a way that give the chance to reallocate the memory. It give a near 2x speedup when we compare to a direct lapack call (numpy-MKL is use for the benchmarks). The lapack determinant function wrapped by numpy is not very good at small matrix and zillion calls.
Following is a step by step tutorial on how to compile NFINDR on Windows 7 (it may be right for vista and Windows 8.x).
Installing the C/C++ compiler on Windows.
Following is for the official Python 2.7 release on windows (32 bits and 64 bits versions). The very first step is to install the C/C++ compiler. The official Python 2.7 was compiled with MSVC 2008. This is the version of the C/C++ compiler to install for sure if we want things to work. A good overview is at the page http://wiki.cython.org/64BitCythonExtensionsOnWindows. It is the main source of information used for this howto. Note that this installation can build binaries for 32 and 64 bits modules extension.
You have to install two packages. Here some comments that can be helpful:
To find the “Microsoft Windows SDK for Windows 7 and .NET Framework 3.5 SP1” start Google and type: “GRMSDK_EN_DVD.iso” and not GRMSDKX_EN_DVD.iso. With chance, you can grab the first link and go to the good Microsoft download page. The name of the file is truly GRMSDK_EN_DVD.iso; take this one.
I am not sure if MSVC 2008 needs to be installed or not. It is already on my computer. To find it Google “Visual C++ Express Edition 2008” and take some main software distribution sites, like www.01net.com.
An interesting post on stackoverflow is: http://stackoverflow.com/questions/15318560/visual-c-2008-express-download-link-dead
Don’t install the MSVC 2008 with SP1
Set the variable DISTUTILS_USE_SDK to 1
To open the Windows SDK Command Prompt window: Click Start, point to All Programs, point to Microsoft Windows SDK , and then click SDK Command Prompt or CMD Shell.
If you want to compile a 32-bits extension, with the yellow text command prompt, go to the folder where the cython code is and execute the standard command to compile your Cython code (python setup.py build_ext –inplace). Note that you can use a vanilla command prompt here.
If you want to compile a 64-bits extension, execute in the yellow text command prompt the two lines:
set DISTUTILS_USE_SDK=1
setenv /x64 /release
Avoid the first line if you have permanently set DISTUTILS_USE_SDK
The text turn to green and you can proceed to the compilation.
I tested Cython+openmp on this setup and it work fine with both 32 and 64 bits.
And finally!
The last step is to compile the nfindr.pyx file.
cd to the directory where nfindr.pyx live and call setup_nfindr.py like this: python setup_nfindr.py build_ext –inplace
Your done, the result is the nfindr.pyd file.