Page 1 of 1
How to configure CUDA MKL GCC in Linux Properly?
Posted: Wed Dec 21, 2016 9:52 am
by Dominic
I compiled Abinit 8.0.8b with the following configure options
Code: Select all
enable_mpi="yes"
with_mpi_prefix="/home/user/MineOS/openmpi2"
enable_gpu="yes"
with_gpu_prefix="/usr/local/cuda"
with_linalg_flavor="mkl+magma"
with_linalg_libs="-L/home/user/MineOS/magma/lib -Wl,--start-group -lmagma -lmagma_sparse -Wl,--end-group -L/opt/intel/compilers_and_libraries_2017.0.098/linux/mkl/lib/intel64 -lmkl_gf_lp64 -lmkl_sequential -lmkl_core"
build was successful but testing using the 'runtests.py' results to a lot of failures, building with MKL and openmpi2 was fine, so definitely no problem with those (I guess) but when I add CUDA, then testing failures starts to crop up, I am sure that this problem is most likely related to my configuration options for CUDA, my Cuda version is Cuda8 and my GCC is 4.9, It was said that GPU support was experimental but I see some people on the net enabling this for supercomputers and other studies already, so there is no point for me having not making it work, still I get lots of tests failures compared to Non-CUDA build,
can anybody pls help?
Re: How to configure CUDA MKL GCC in Linux Properly?
Posted: Wed Dec 21, 2016 6:43 pm
by jbeuken
( update of post
viewtopic.php?f=2&t=2677&p=8233&hilit=cuda#p8233 )
Hello,
we have a testfarm and one of the bot tests the cuda functionality ( this bot has 2 x K40 and 2 x Tesla C1060 )
we use gcc47 + mkl + magma ( > 1.6.x ) + cuda 6.5
and compile abinit with ( among others, as you can find in config/spec/testfarm.conf ) :
Code: Select all
FC_LDFLAGS_EXTRA="-Wl,-z,muldefs"
enable_mpi="yes"
enable_mpi_io="yes"
with_mpi_prefix="/usr/local/openmpi-1.6.5_gcc47"
enable_gpu="yes"
with_gpu_flavor="cuda-double"
NVCC_CFLAGS="-O3 -arch=sm_13 -Xptxas=-v --use_fast_math --compiler-options -O3,-fPIC"
with_trio_flavor="netcdf+etsf_io"
with_etsf_io_incs="-I/usr/local/fallbacks/gnu/4.7/mpi/include"
with_etsf_io_libs="-L/usr/local/fallbacks/gnu/4.7/mpi/lib -letsf_io_low_level -letsf_io_utils -letsf_io"
with_netcdf_incs="-I/usr/local/fallbacks/gnu/4.7/mpi/include"
with_netcdf_libs="-L/usr/local/fallbacks/gnu/4.7/mpi/lib -lnetcdf"
with_linalg_flavor="mkl+magma"
with_linalg_incs="-I${MAGMA_ROOT}/include -I${MKLROOT}/include"
with_linalg_libs="-L${MAGMA_ROOT}/lib -Wl,--start-group -lmagma -lcuda -Wl,--end-group -L${MKLROOT}/lib/intel64 -lmkl_gf_lp64 -lmkl_sequential -lmkl_core -lgomp -lpthread -lm"
with_dft_flavor="none"
enable_gw_dpc="yes"
enable_memory_profiling="no"
enable_maintainer_checks="no"
there are only 7 tests "tested" with cuda to validate the "gcc/cuda" part of abinit ( ./runtests.py gpu )
Code: Select all
==========================================================================
Serie #failed #passed #succes #skip | #CPU #WALL
==========================================================================
gpu | 0 | 0 | 7 | 0 | 201.2 | 202.0
==========================================================================
Analysis 8.5.4 ( python v(2, 7, 8) )
all other tests are not validated, although the results may be correct…
the gpu is not yet fully officially supported ... but we're working/using …
my 50¢
jmb
Re: How to configure CUDA MKL GCC in Linux Properly?
Posted: Thu Dec 22, 2016 5:58 am
by Dominic
Thank you for keeping up the CUDA support in Abinit, I have seen people having success on it! Can I please request for your Magma configuration of make.inc?
I dont know if the problem is my configuration of magma, but as far as Magma compilation and testing, checks have been passed. Here is what I have:
Code: Select all
GPU_TARGET = Pascal
CC = gcc
CXX = g++
NVCC = /usr/local/cuda/bin/nvcc
FORT = gfortran
ARCH = ar
ARCHFLAGS = cr
RANLIB = ranlib
FPIC = -fPIC
CFLAGS = -O3 $(FPIC) -fopenmp -DNDEBUG -DADD_ -Wall -Wshadow -DMAGMA_WITH_MKL
FFLAGS = -O3 $(FPIC) -DNDEBUG -DADD_ -Wall -Wno-unused-dummy-argument
F90FLAGS = -O3 $(FPIC) -DNDEBUG -DADD_ -Wall -Wno-unused-dummy-argument -x f95-cpp-input
NVCCFLAGS = -O3 -DNDEBUG -DADD_ -Xcompiler "$(FPIC) -Wall -Wno-unused-function"
LDFLAGS = $(FPIC) -fopenmp
CXXFLAGS := $(CFLAGS) -std=c++11
CFLAGS += -std=c99
# see MKL Link Advisor at http://software.intel.com/sites/products/mkl/
# gcc/gfortran with MKL 10.3, GNU OpenMP threads (use -fopenmp in CFLAGS, LDFLAGS)
LIB = -lmkl_gf_lp64 -lmkl_gnu_thread -lmkl_core -lpthread -lstdc++ -lm -lgfortran
LIB += -lcublas -lcusparse -lcudart -lcudadevrt
MKLROOT = /opt/intel/compilers_and_libraries_2017.0.098/linux/mkl
CUDADIR = /usr/local/cuda
-include make.check-mkl
-include make.check-cuda
LIBDIR = -L$(CUDADIR)/lib64 \
-L$(MKLROOT)/lib/intel64
this make file passed the checks and tests of MAGMA,
so I am not sure were might the problem be?