ABINIT Discussion Forums

Hallo to everybody,

I have some question on how the KSS is computed in version 6.12.
Actually the reason I am asking it is that I tried to compute the KSS on a system with fairly low parameters
npwkss = 600
nbandkss = 100
nkpt = 1156
I thought it would have been very fast, but even when running on 1184 processors,
the calculation lasted more than 20h.
Indeed after 20h, the wall clock time was over and in the log was stopped here:

Writing out eigenvalues/vectors for ikpt=505
Occupation numbers for ikpt=505
505 2.0000 2.0000 2.0000 2.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000
0.0000

So my question is, how it is possible that it took so much time?
The writing of the file is made during the diagonalisation, or the KSS file is written only at the end of the calculation?
I hope my question is well formulated, otherwise I can add more elements to understand my point.

Thanks.

Hi Lorenzo,

So my question is, how it is possible that it took so much time?

Because the code that implements the output of the KSS file is highly inefficient, in particular
when the run is done with so many processors. Performing IO with many processors is not trivial
and outkss still uses Fortran-IO to write the results plus point-to-point MPI communications to
send the data to master. The inefficiency quickly increases with the number of MPI node and this
is what you are experiencing.

This is the reason why massively parallel calculations should always use MPI-IO for reading/writing
the wavefunctions.
Note that the new version of the GW code can read WFK files (instead of KSS files), as a consequence
one can use MPI-IO to reduce the bottleneck due to IO.

You can use the following trick if you need a KSS file for performing calculations with DP, EXC or other codes.

1) Produce a WFK file with the GS part of abinit (no kssform)
2) run a second calculation that reads the WFK file and produces the KSS file (nstep=0, kssform=3, small number of MPI processors to avoid the IO issue)

Hope it helps,
Ciao

Thanks a lot gmatteo,
especially for the hint!