===== Relion =====



==== Version 1.1 ====


Small bug in line 658 in src/backprojector.cpp:

change:
<code>
double myfsc = XMIPP_MAX(0., DIRECT_A1D_ELEM(fsc, i));
</code>

to:
<code>
double myfsc = XMIPP_MAX(0.001, DIRECT_A1D_ELEM(fsc, i));
</code>

== Configure VSC-1 ==
on VSC-1 only gcc works currently well. Intel 10.x and 11.x have problems with some part of the code.

<code>
CPPFLAGS="-I/usr/include " \
CFLAGS="-L/usr/lib64 -lpthread" \
LDFLAGS="-L/usr/lib64 -lpthread" \
CXXFLAGS="-L/usr/lib64 -lpthread" \
MPICXX="mpicxx -DMPICH_IGNORE_CXX_SEEK " \
CXX=g++ \
CC="gcc" CPPFLAGS="-I/usr/include " \
CFLAGS="-L/usr/lib64  -lpthread" \
./configure --enable-mpi \
--prefix=/home/lv70172/blaas/software/relion-1.1
</code>


== Configure VSC-2 ==
Configure: (Note '\' characters are on all lines, i.e. following paragraph is treated like one single line by bash):

With INTEL MPI + IntelCompiler:
<code>
CPPFLAGS="-I/opt/fftw/3.3/intel_12.1_O3_pic/include " \
CFLAGS="-L/opt/fftw/3.3/intel_12.1_O3_pic/lib -lpthread" \
LDFLAGS="-L/opt/fftw/3.3/intel_12.1_O3_pic/lib -lpthread" \
CXXFLAGS="-L/opt/fftw/3.3/intel_12.1_O3_pic/lib -lpthread" \
MPICXX="mpicxx -DMPICH_IGNORE_CXX_SEEK " \
CXX=icpc \
CC="icc" CPPFLAGS="-I/opt/fftw/3.3/intel_12.1_O3/include " \
CFLAGS="-L/opt/fftw/3.3/intel_12.1_O3/lib  -lpthread" \
./configure --enable-mpi \
--prefix=/home/lv70172/blaas/software/relion-1.1
</code>


With gcc and intelmpi, first use INSTALL.sh to compile fftw for relion itself. FFTW on VSC-2 althoug configured very similarly, did not work here:
<code>
CPPFLAGS="-I/home/lv70172/blaas/software/relion-1.1_gcc/include " \
CFLAGS="-L/home/lv70172/blaas/software/relion-1.1_gcc/lib -lpthread" \
LDFLAGS="-L/home/lv70172/blaas/software/relion-1.1_gcc/lib -lpthread" \
CXXFLAGS="-L/home/lv70172/blaas/software/relion-1.1_gcc/lib -lpthread" \
MPICXX="mpigxx -DMPICH_IGNORE_CXX_SEEK" \
CXX=g++ \
CC="gcc" CPPFLAGS="-I/home/lv70172/blaas/software/relion-1.1_gcc/include " \
CFLAGS="-L/home/lv70172/blaas/software/relion-1.1_gcc/lib  -lpthread" \
./configure --enable-mpi \
--prefix=/home/lv70172/blaas/software/relion-1.1_gcc


</code>

== Compiling and Installation (VSC-1 and VSC-2) ==
Then:
<code>
make
make install 
</code>

For installing the GUI edit INSTALL.sh:
<code>
BUILD_FFTW=false
BUILD_FLTK=true
BUILD_RELION=false
BUILD_RELION_GUI=true
</code>

Finally execute:
<code>
./INSTALL.sh
</code>



When running jobs on the queue, place a file 'qsub.bash' in 
the 'gui' directory of relion:
<code>
#!/bin/bash
#$ -pe XXXqueueXXX XXXmpinodesXXX
#$ -e XXXerrfileXXX
#$ -o XXXoutfileXXX
#$ -cwd
#$ -V

#needed on VSC-1:
export IPATH_NO_CPUAFFINITY=1

mpirun -machinefile $TMPDIR/machines  -np XXXmpinodesXXX  XXXcommandXXX
</code>


Settings in the GUI:
  * queue name: mpich


==== Version 1.2 ====

== VSC-2: GCC and INTEL MPI ==

Compile FFTW from relion sources, because VSC FFTW showed some errors:

in INSTALL.sh set:
<code>
BUILD_FFTW=true
BUILD_FLTK=false
BUILD_RELION=false
BUILD_RELION_GUI=false
</code>

execute:
<code>
#might be needed to remove path to pgi because fftw does not compile, at least on vsc-1
PATH=`echo $PATH | sed -e s?/opt/sw/pgi/linux86-64/13.7/bin:??`

./INSTALL.sh
</code>

Somewhere a missing 'include' directive. Add to 'src/mpi.cpp' at the very beginning of the file:
<code>
#include <stdio.h>
</code>


Configure and install  Relion:
RELION_PREFIX should be the source directory of the current version

=== GNU compiler ===

For gcc use:<code>
#export RELION_PREFIX=/home/lv70172/blaas/software/relion-1.2_gcc
export RELION_PREFIX=`pwd`

CPPFLAGS="-I${RELION_PREFIX}/include " \
CFLAGS="-L${RELION_PREFIX}/lib -lpthread" \
LDFLAGS="-L${RELION_PREFIX}/lib -lpthread" \
CXXFLAGS="-L${RELION_PREFIX}/lib -lpthread" \
MPICXX="mpigxx -DMPICH_IGNORE_CXX_SEEK" \
CXX=g++ \
CC="gcc" CPPFLAGS="-I${RELION_PREFIX}/include " \
CFLAGS="-L${RELION_PREFIX}/lib  -lpthread" \
./configure --enable-mpi \
--prefix=${RELION_PREFIX}


make 
make install
</code>

=== Intel Compiler ===
For the Intel compiler first compile and install the FFTW libraries by editing the following lines in INSTALL.sh:<code>
BUILD_FFTW=true
BUILD_FLTK=false
BUILD_RELION=false
BUILD_RELION_GUI=false

...

#################### FFTW ###########################
if $BUILD_FFTW; then
  echo -e "$GREEN Compiling $VFFTW ...$ENDC"
  echo -e "See $RELION_HOME/external/fftw_build.log for details"
  cd external
  tar -zxf $VFFTW.tar.gz
  cd $VFFTW
  CC=icc CPP=cpp MPICC=mpiicc F77=ifort ./configure --enable-openmp --enable-shared prefix=$PREFIX > $RELION_HOME/external/fftw_build.log
  make $N_THREADS >> $RELION_HOME/external/fftw_build.log
  make install >> $RELION_HOME/external/fftw_build.log
  cd ../..
fi</code>
Then run:
<code>
./INSTALL.sh
</code>
Then execute<code>
export RELION_PREFIX=`pwd`

CPPFLAGS="-I${RELION_PREFIX}/include "\
CFLAGS="-L${RELION_PREFIX}/lib -openmp -lpthread -liomp5" \
LDFLAGS="-L${RELION_PREFIX}/lib -lpthread" \
CXXFLAGS="-L${RELION_PREFIX}/lib -openmp -lpthread -liomp5" \
MPICXX="mpiicpc -DMPICH_IGNORE_CXX_SEEK" \
CXX=icpc \
CC="icc" \
CPPFLAGS="-I${RELION_PREFIX}/include " \
CFLAGS="-L${RELION_PREFIX}/lib -openmp -lpthread -liomp5" \
./configure --enable-mpi --prefix=${RELION_PREFIX}

make 
make install
</code>
Finally recompile ctf by editing the appropriate Makefile in ''software/ctf'':<code>
all: ctffind3.exe ctftilt.exe

ctffind3.exe: ctffind3.o iof.o ioc.o
        ifort -xHost -openmp -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
        ctffind3.o iof.o ioc.o -o ctffind3.exe

ctftilt.exe: ctftilt.o iof.o ioc.o
        ifort -xHost -openmp -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
        ctftilt.o iof.o ioc.o -o ctftilt.exe
        \rm *.o

ctffind3.o: ctffind3.f
        ifort -xHost -openmp \
        -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
        -w -c ctffind3.f

ctftilt.o: ctftilt.f
        ifort -xHost -openmp \
        -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
        -w -c ctftilt.f

iof.o: iof.f
        ifort -xHost -openmp \
        -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE \
        -w -c iof.f

ioc.o: ioc.c
        icc -DPROTOTYPE -Dalliant -D_FILE_OFFSET_BITS=64 \
        -D_LARGEFILE_SOURCE -w -c ioc.c</code>

Then execute ''make -f Makefile''


For installing the GUI edit INSTALL.sh:
<code>
BUILD_FFTW=false
BUILD_FLTK=true
BUILD_RELION=false
BUILD_RELION_GUI=true
</code>

Finally execute:
<code>
./INSTALL.sh
</code>



When running jobs on the queue, place a file 'qsub.bash' in 
the 'gui' directory of relion:
<code>
#!/bin/bash
#$ -pe XXXqueueXXX XXXmpinodesXXX
#$ -e XXXerrfileXXX
#$ -o XXXoutfileXXX
#$ -cwd
#$ -V

#needed on VSC-1:
export IPATH_NO_CPUAFFINITY=1

mpirun -machinefile $TMPDIR/machines  -np XXXmpinodesXXX  XXXcommandXXX
</code>

The generated submit script should be modified to look like this:<code>
#!/bin/bash
#$ -pe mpich 128
#$ -e particles.err
#$ -o particles.out
#$ -cwd
#$ -V

export I_MPI_DAT_LIBRARY=/usr/lib64/libdat2.so.2
export OMP_NUM_THREADS=1
export I_MPI_FABRICS=shm:dapl
export I_MPI_FALLBACK
export I_MPI_CPUINFO=proc
export I_MPI_PIN_PROCESSOR_LIST=0-15
export I_MPI_JOB_FAST_STARTUP=0
unset PE_HOSTFILE


mpirun -machinefile $TMPDIR/machines -np $NSLOTS `which relion_preprocess_mpi` --o particles --ctffind "Micrographs/*.mrc" --ctfWin -1 --CS 2 --HT 300 --AmpCnst 0.1 --XMAG 79372 --DStep 15 --Box 512 --ResMin 100 --ResMax 7 --dFMin 5000 --dFMax 50000 --FStep 500 --ctffind3_exe /home/lv60000/jz/users/blaas/software/ctf/ctffind3.exe --coord_files "Micrographs/*.poscorr" --extract --coord_format xmipp2 --extract_size 240 --norm --bg_radius 100 --white_dust 3.5 --black_dust 3.5 --invert_contrast
</code>
Settings in the GUI:
  * queue name: mpich

==== Version 1.3 ====

Installation Procedure with Intelcompilers and Intelmpi:
<code>
umask 0022
my_src=/opt/src/relion/relion-1.3
my_inst=INSTALL_vsc.sh
my_prefix=/opt/sw/relion/1.3-intel/
cp_dirs="bin include lib share"

cd $my_src
sed '3 i export MPICXX=mpiicpc ; export CC=icc; export CXX=icpc ' INSTALL.sh > $my_inst
bash $my_inst

mkdir -p $my_prefix
for d in $cp_dirs
do
	cp -r $my_src/$d $my_prefix
done

mkdir -p $my_prefix/gui
echo '
#!/bin/bash
#$ -pe XXXqueueXXX XXXmpinodesXXX
#$ -e XXXerrfileXXX
#$ -o XXXoutfileXXX
#$ -cwd
#$ -V

#needed on VSC-1:
#export IPATH_NO_CPUAFFINITY=1

mpirun -machinefile $TMPDIR/machines  -np XXXmpinodesXXX  XXXcommandXXX
'> $my_prefix/gui/qsub.bash


</code>

Add this to .bashrc:
<code>
RELION_PREFIX=/opt/sw/relion/1.3-intel
export PATH=${RELION_PREFIX}/bin:$PATH
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${RELION_PREFIX}/lib
export RELION_QSUB_TEMPLATE="$RELION_PREFIX/gui/qsub.bash"
export RELION_CTFFIND3_EXECUTABLE="/opt/sw/ctf/ctffind3.exe"

</code>