How to configure CUDA MKL GCC in Linux Properly?

option, parallelism,...

Moderators: fgoudreault, mcote

Forum rules
Please have a look at ~abinit/doc/config/build-config.ac in the source package for detailed and up-to-date information about the configuration of Abinit 8 builds.
For a video explanation on how to build Abinit 7.x for Linux, please go to: http://www.youtube.com/watch?v=DppLQ-KQA68.
IMPORTANT: when an answer solves your problem, please check the little green V-like button on its upper-right corner to accept it.
Locked
Dominic
Posts: 18
Joined: Mon Jan 21, 2013 4:34 pm

How to configure CUDA MKL GCC in Linux Properly?

Post by Dominic » Wed Dec 21, 2016 9:52 am

I compiled Abinit 8.0.8b with the following configure options

Code: Select all

enable_mpi="yes"
with_mpi_prefix="/home/user/MineOS/openmpi2"

enable_gpu="yes"

with_gpu_prefix="/usr/local/cuda"

with_linalg_flavor="mkl+magma"
with_linalg_libs="-L/home/user/MineOS/magma/lib -Wl,--start-group -lmagma -lmagma_sparse -Wl,--end-group -L/opt/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_sequential -lmkl_core"


build was successful but testing using the 'runtests.py' results to a lot of failures, building with MKL and openmpi2 was fine, so definitely no problem with those (I guess) but when I add CUDA, then testing failures starts to crop up, I am sure that this problem is most likely related to my configuration options for CUDA, my Cuda version is Cuda8 and my GCC is 4.9, It was said that GPU support was experimental but I see some people on the net enabling this for supercomputers and other studies already, so there is no point for me having not making it work, still I get lots of tests failures compared to Non-CUDA build, can anybody pls help?
Top
User avatar
jbeuken
Posts: 365
Joined: Tue Aug 18, 2009 9:24 pm
Contact:
Contact jbeuken

Re: How to configure CUDA MKL GCC in Linux Properly?

Post by jbeuken » Wed Dec 21, 2016 6:43 pm

( update of post viewtopic.php?f=2&t=2677&p=8233&hilit=cuda#p8233 )

Hello,

we have a testfarm and one of the bot tests the cuda functionality ( this bot has 2 x K40 and 2 x Tesla C1060 )
we use gcc47 + mkl + magma ( > 1.6.x ) + cuda 6.5

and compile abinit with ( among others, as you can find in config/spec/testfarm.conf ) :

Code: Select all

FC_LDFLAGS_EXTRA="-Wl,-z,muldefs"
enable_mpi="yes"
enable_mpi_io="yes"
with_mpi_prefix="/usr/local/openmpi-1.6.5_gcc47"
enable_gpu="yes"
with_gpu_flavor="cuda-double"
NVCC_CFLAGS="-O3 -arch=sm_13 -Xptxas=-v --use_fast_math --compiler-options -O3,-fPIC"
with_trio_flavor="netcdf+etsf_io"
with_etsf_io_incs="-I/usr/local/fallbacks/gnu/4.7/mpi/include"
with_etsf_io_libs="-L/usr/local/fallbacks/gnu/4.7/mpi/lib -letsf_io_low_level -letsf_io_utils -letsf_io"
with_netcdf_incs="-I/usr/local/fallbacks/gnu/4.7/mpi/include"
with_netcdf_libs="-L/usr/local/fallbacks/gnu/4.7/mpi/lib -lnetcdf"
with_linalg_flavor="mkl+magma"
with_linalg_incs="-I${MAGMA_ROOT}/include -I${MKLROOT}/include"
with_linalg_libs="-L${MAGMA_ROOT}/lib -Wl,--start-group -lmagma -lcuda -Wl,--end-group -L${MKLROOT}/lib/intel64 -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lgomp -lpthread -lm"
with_dft_flavor="none"
enable_gw_dpc="yes"
enable_memory_profiling="no"
enable_maintainer_checks="no"


there are only 7 tests "tested" with cuda to validate the "gcc/cuda" part of abinit ( ./runtests.py gpu )

Code: Select all

==========================================================================
          Serie   #failed   #passed  #succes  #skip  |   #CPU      #WALL
==========================================================================
            gpu |     0   |    0   |    7   |    0   |   201.2  |   202.0
==========================================================================
Analysis 8.5.4 ( python v(2, 7, 8) )


all other tests are not validated, although the results may be correct…

the gpu is not yet fully officially supported ... but we're working/using … ;)

my 50¢

jmb
------
Jean-Michel Beuken
Computer Scientist
Top
Dominic
Posts: 18
Joined: Mon Jan 21, 2013 4:34 pm

Re: How to configure CUDA MKL GCC in Linux Properly?

Post by Dominic » Thu Dec 22, 2016 5:58 am

Thank you for keeping up the CUDA support in Abinit, I have seen people having success on it! Can I please request for your Magma configuration of make.inc?

I dont know if the problem is my configuration of magma, but as far as Magma compilation and testing, checks have been passed. Here is what I have:

Code: Select all


GPU_TARGET = Pascal

CC        = gcc
CXX       = g++
NVCC      = /usr/local/cuda/bin/nvcc
FORT      = gfortran

ARCH      = ar
ARCHFLAGS = cr
RANLIB    = ranlib

FPIC      = -fPIC

CFLAGS    = -O3 $(FPIC) -fopenmp -DNDEBUG -DADD_ -Wall -Wshadow -DMAGMA_WITH_MKL
FFLAGS    = -O3 $(FPIC)          -DNDEBUG -DADD_ -Wall -Wno-unused-dummy-argument
F90FLAGS  = -O3 $(FPIC)          -DNDEBUG -DADD_ -Wall -Wno-unused-dummy-argument -x f95-cpp-input
NVCCFLAGS = -O3                  -DNDEBUG -DADD_ -Xcompiler "$(FPIC) -Wall -Wno-unused-function"
LDFLAGS   =     $(FPIC) -fopenmp

CXXFLAGS := $(CFLAGS) -std=c++11
CFLAGS   += -std=c99

# see MKL Link Advisor at http://software.intel.com/sites/products/mkl/
# gcc/gfortran with MKL 10.3, GNU OpenMP threads (use -fopenmp in CFLAGS, LDFLAGS)
LIB       = -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lpthread -lstdc++ -lm -lgfortran

LIB      += -lcublas -lcusparse -lcudart -lcudadevrt

MKLROOT = /opt/intel/compilers_and_libraries_2017.0.098/linux/mkl
CUDADIR = /usr/local/cuda
-include make.check-mkl
-include make.check-cuda

LIBDIR    = -L$(CUDADIR)/lib64 \
            -L$(MKLROOT)/lib/intel64


this make file passed the checks and tests of MAGMA, so I am not sure were might the problem be?
Top
Locked

Return to “Configuring and compiling ABINIT”