ABINIT Discussion Forums

Hy

I'm admin of a computing cluster. A user had this error with a "big" job (366 atomes).
> my_nspinor: 1, mpw: 200091, mband: 1708, mkmem: 11, nsppol: 1
> Note: Compiling with large int (int64) requires a full software stack (MPI/FFTW/BLAS...) compiled in int64 mode

It appears that this message comes from `m_gstate.f90`:
> 696 & "Default integer is not wide enough to store the size of the wavefunction array (mcg).",ch10,&
> 697 & "This usually happens when paral_kgb == 0 and there are not enough procs to distribute kpts and spins",ch10,&
> 698 & "Action: if paral_kgb == 0, use nprocs = nkpt * nsppol to reduce the memory per node.",ch10,&
> 699 & "If this does not solve the problem, use paral_kgb 1 with nprocs > nkpt * nsppol and use npfft/npband/npspinor",ch10,&
> 700 & "to decrease the memory requirements. Consider also OpenMP threads."

So, I'm not sure compiling using int64 is the solution. May be more MPI instances is the solution ?
I mean, the problem may be an out of memory one ?

Whatever, I tryed to compile abinit 8.10.2, forcing int64. Following is what I tryed and/or noticed so far:
* An opened topic on the subject, but with (possible ?) solution in private gitlab > viewtopic.php?f=3&t=3607

EDIT : I don't have access to this gitlab, I don't know if a solution was found.

* Dedicated 64bits arguments are not viewed by `configure` (ie CPPFLAGS_64BITS, CFLAGS_64BITS, CXXFLAGS_64BITS, and FCFLAGS_64BITS, as mentioned in `abinit/doc/build/config-template.ac`). I used FCFLAGS (and C/C++ equivalents) argument to transmit compile flags (see what follows).
* I'm using intel compilers. Using `-i8` compile flag is recommended by intel documentation. But compilation fails for `12_hide_mpi/m_xmpi.F90`. Functions `xmpi_max_int0d_i4b` and `xmpi_max_int0d_i8b` are declared ambigus. From what I understand, -i8 force compiler to use int64 all the time. So `xmpi_max_int0d_i4b` and `xmpi_max_int0d_i8b` are both viewed as 64bits version. Should I disable one of those ? The existence of these two versions tells me the int64 compilation is may not needed again.
* Compiling using only `-ilp64` compile flag (partial ILP64 support says mpiifort doc) returns data type errors.
* Same with `-no_ilp64 -i8` (as mentioned in mpiifort doc).

Any help about my user's original error, or int64 compilation would be appreciated.
Thanks,
Baptiste

So, I'm not sure compiling using int64 is the solution. May be more MPI instances is the solution ?
I mean, the problem may be an out of memory one ?

Right, compiling with int64 is not the recommended solution.
The calculation is relatively large but some dimensions scale with the number of MPI processes
provided the correct algorithm is used.

First of all, 11 kpoints (mkmem 11) for a system with 366 atoms looks weird to me.

> my_nspinor: 1, mpw: 200091, mband: 1708, mkmem: 11, nsppol: 1

Calculations with large systems are usually done with one k-point (Gamma-only) in which one can use symmetry tricks
to halve the number of G-vectors. Hence using Gamma-only sampling allows one to decrease the memory requirement of an
hypothetical sequential job by a factor ~22

Secondly, I hope the user is using the paral_kgb = 1 solver because the default algorithm (paral_kgb = 0) has limited scalability.
paral_kgb 1 can distribute the memory over several levels thanks to the npfft, npband, npkpt input variables that define
the number of MPI processes for the different levels.

I would suggest the user to experiment a bit with these parameters following the discussion available
in this tutorial: https://docs.abinit.org/tutorial/paral_gspw/
If the problem persists, I would suggest that he/she shares the input file on the forum so that we have a better understanding
of the kind of calculation that is being performed.

Matteo

Hy,
I have forwarded these recommendations to the user, I will keep you informed of any progress.
Many thanks,
Baptiste