TDDFT in parallel
Posted: Thu Apr 20, 2017 3:32 pm
Dear all,
which is the best way (or the correct way) to run a TDDFT calculation in parallel with Abinit?
It is not clear to me if such a TDDFT calculation can / has to be parallelized over the number of excitations (= n_occ_states * n_unocc_states), or rather on bands, kpoints, fft as e.g. in an SCF run.
I would like to use the Abinit7.10.2 version (I know it is not so up to date...) which is installed on the CINECA machines (Galileo).
Can/should I use the autoparal keyword (which however is described as a keyword for ground state calculations), or rather the paral_kgb one? Or are there any specific parallelization keywords for TDDFT calculations?
I have an isolated molecule, therefore 1 k point only, but a rather large number of bands: 27 occupied bands + 50 or more unoccupied ones, and a rather large cell (e.g. 17x16x20 in Angstrom), therefore parallelization is almost vital in order to keep these runs within a reasonable duration. For the case with 50 unoccupied bands (i.e. 27+50=77 bands in total), I have tried a run with “autoparal 1” in the input file, and fixing NP=64 in the job script: Abinit writes the following relevant information in the .log file:
Computing all possible proc distrib for this input with nproc less than 64
npimage| npkpt| npspinor| npfft| npband| bandpp | nproc| weight|
1 -> 1| 1 -> 1| 1 -> 1| 1 -> 37| 1 -> 64| 1 -> 8| 2 -> 64| 1 -> 64|
1| 1| 1| 8| 7| 1| 56| 26.52 |
1| 1| 1| 4| 7| 1| 28| 10.07 |
1| 1| 1| 4| 11| 1| 44| 8.80 |
1| 1| 1| 2| 7| 1| 14| 4.15 |
1| 1| 1| 2| 11| 1| 22| 4.05 |
npfft, npband, npspinor and npkpt: 8 7 1 1
..which means, if I understood well, that it finds (npfft=8, npband=7, npspinor=1, npkpt=1) as the best distribution and uses it → therefore 7*8=56 processes. Indeed npband=7 does not surprise me, with nband=77.
Accordingly, later on, just before the “TDDFT : computation of excited states”, it writes “TDDFT 56 CPU synchronized”.
But then it crashes:
*** TDDFT : computation of excited states ***
Splitting of 77 states in 27 occupied states, and 50 unoccupied states,
giving 1350 excitations.
Nr of states to Fourier transform : 77
DATA TYPE INFORMATION:
REAL: Data type name: REAL(DP)
Kind value: 8
Precision: 15
Smallest nonnegligible quantity relative to 1: 0.22204460E-15
Smallest positive number: 0.22250739-307
Largest representable number: 0.17976931+309
INTEGER: Data type name: INTEGER(default)
Kind value: 4
Bit size: 32
Largest representable number: 2147483647
LOGICAL: Data type name: LOGICAL
Kind value: 4
CHARACTER: Data type name: CHARACTER Kind value: 1
==== Using MPI-2 specifications ====
MPI-IO support is ON
xmpi_tag_ub ................ 536870911
xmpi_bsize_ch .............. 1
xmpi_bsize_int ............. 4
xmpi_bsize_sp .............. 4
xmpi_bsize_dp .............. 8
xmpi_bsize_spc ............. 8
xmpi_bsize_dpc ............. 16
xmpio_bsize_frm ............ 4
xmpi_address_kind .......... 8
xmpi_offset_kind ........... 8
MPI_WTICK .................. 1.000000000000000E-006
(…..)
--- !BUG
message: |
Unable to find an allocated distrib for this fft grid
src_file: m_mpinfo.F90
src_line: 746
…
Image PC Routine Line Source
libifcore.so.5 00007F1AC92FC706 Unknown Unknown Unknown
abinit 00000000021DF0DF Unknown Unknown Unknown
abinit 000000000193695A Unknown Unknown Unknown
abinit 00000000017D541E Unknown Unknown Unknown
abinit 00000000006215CA Unknown Unknown Unknown
abinit 00000000005F7A76 Unknown Unknown Unknown
abinit 000000000059A294 Unknown Unknown Unknown
abinit 0000000000590168 Unknown Unknown Unknown
abinit 000000000053413D Unknown Unknown Unknown
abinit 000000000042F471 Unknown Unknown Unknown
abinit 0000000000413FEB Unknown Unknown Unknown
abinit 0000000000408F51 Unknown Unknown Unknown
abinit 0000000000406FBE Unknown Unknown Unknown
libc.so.6 00007F1AC7633B15 Unknown Unknown Unknown
abinit 0000000000406EB9 Unknown Unknown Unknown
leave_new : decision taken to exit ...
application called MPI_Abort(MPI_COMM_WORLD, 13) - process 0
What is the matter?
Also from previous tests using paral_kgb and several combinations of npfft, npband etc (instead of using autoparal), I wonder if Abinit performs a first part of the run (“Non-SCF iterations” with the specified number of bands, starting from DEN and WFK files from an SCF run with fewer bands) parallelizing on bands and ffts (finding in my case npfft=8, npband=7 as the best combination), and then, when it has to perform the real TDDFT computation, it “realizes” that the previously chosen set of paral keywords is not correct at this stage. This may also be a too “imaginative” interpretation on my part...
Thanks a lot in advance for any suggestions
Best regards
Elena Molteni
Physics Department
Universita' degli Studi di Milano
via Celoria, 16
I-20133, Milan, Italy
and European Theoretical Spectroscopy Facility (ETSF)
http://www.etsf.eu
which is the best way (or the correct way) to run a TDDFT calculation in parallel with Abinit?
It is not clear to me if such a TDDFT calculation can / has to be parallelized over the number of excitations (= n_occ_states * n_unocc_states), or rather on bands, kpoints, fft as e.g. in an SCF run.
I would like to use the Abinit7.10.2 version (I know it is not so up to date...) which is installed on the CINECA machines (Galileo).
Can/should I use the autoparal keyword (which however is described as a keyword for ground state calculations), or rather the paral_kgb one? Or are there any specific parallelization keywords for TDDFT calculations?
I have an isolated molecule, therefore 1 k point only, but a rather large number of bands: 27 occupied bands + 50 or more unoccupied ones, and a rather large cell (e.g. 17x16x20 in Angstrom), therefore parallelization is almost vital in order to keep these runs within a reasonable duration. For the case with 50 unoccupied bands (i.e. 27+50=77 bands in total), I have tried a run with “autoparal 1” in the input file, and fixing NP=64 in the job script: Abinit writes the following relevant information in the .log file:
Computing all possible proc distrib for this input with nproc less than 64
npimage| npkpt| npspinor| npfft| npband| bandpp | nproc| weight|
1 -> 1| 1 -> 1| 1 -> 1| 1 -> 37| 1 -> 64| 1 -> 8| 2 -> 64| 1 -> 64|
1| 1| 1| 8| 7| 1| 56| 26.52 |
1| 1| 1| 4| 7| 1| 28| 10.07 |
1| 1| 1| 4| 11| 1| 44| 8.80 |
1| 1| 1| 2| 7| 1| 14| 4.15 |
1| 1| 1| 2| 11| 1| 22| 4.05 |
npfft, npband, npspinor and npkpt: 8 7 1 1
..which means, if I understood well, that it finds (npfft=8, npband=7, npspinor=1, npkpt=1) as the best distribution and uses it → therefore 7*8=56 processes. Indeed npband=7 does not surprise me, with nband=77.
Accordingly, later on, just before the “TDDFT : computation of excited states”, it writes “TDDFT 56 CPU synchronized”.
But then it crashes:
*** TDDFT : computation of excited states ***
Splitting of 77 states in 27 occupied states, and 50 unoccupied states,
giving 1350 excitations.
Nr of states to Fourier transform : 77
DATA TYPE INFORMATION:
REAL: Data type name: REAL(DP)
Kind value: 8
Precision: 15
Smallest nonnegligible quantity relative to 1: 0.22204460E-15
Smallest positive number: 0.22250739-307
Largest representable number: 0.17976931+309
INTEGER: Data type name: INTEGER(default)
Kind value: 4
Bit size: 32
Largest representable number: 2147483647
LOGICAL: Data type name: LOGICAL
Kind value: 4
CHARACTER: Data type name: CHARACTER Kind value: 1
==== Using MPI-2 specifications ====
MPI-IO support is ON
xmpi_tag_ub ................ 536870911
xmpi_bsize_ch .............. 1
xmpi_bsize_int ............. 4
xmpi_bsize_sp .............. 4
xmpi_bsize_dp .............. 8
xmpi_bsize_spc ............. 8
xmpi_bsize_dpc ............. 16
xmpio_bsize_frm ............ 4
xmpi_address_kind .......... 8
xmpi_offset_kind ........... 8
MPI_WTICK .................. 1.000000000000000E-006
(…..)
--- !BUG
message: |
Unable to find an allocated distrib for this fft grid
src_file: m_mpinfo.F90
src_line: 746
…
Image PC Routine Line Source
libifcore.so.5 00007F1AC92FC706 Unknown Unknown Unknown
abinit 00000000021DF0DF Unknown Unknown Unknown
abinit 000000000193695A Unknown Unknown Unknown
abinit 00000000017D541E Unknown Unknown Unknown
abinit 00000000006215CA Unknown Unknown Unknown
abinit 00000000005F7A76 Unknown Unknown Unknown
abinit 000000000059A294 Unknown Unknown Unknown
abinit 0000000000590168 Unknown Unknown Unknown
abinit 000000000053413D Unknown Unknown Unknown
abinit 000000000042F471 Unknown Unknown Unknown
abinit 0000000000413FEB Unknown Unknown Unknown
abinit 0000000000408F51 Unknown Unknown Unknown
abinit 0000000000406FBE Unknown Unknown Unknown
libc.so.6 00007F1AC7633B15 Unknown Unknown Unknown
abinit 0000000000406EB9 Unknown Unknown Unknown
leave_new : decision taken to exit ...
application called MPI_Abort(MPI_COMM_WORLD, 13) - process 0
What is the matter?
Also from previous tests using paral_kgb and several combinations of npfft, npband etc (instead of using autoparal), I wonder if Abinit performs a first part of the run (“Non-SCF iterations” with the specified number of bands, starting from DEN and WFK files from an SCF run with fewer bands) parallelizing on bands and ffts (finding in my case npfft=8, npband=7 as the best combination), and then, when it has to perform the real TDDFT computation, it “realizes” that the previously chosen set of paral keywords is not correct at this stage. This may also be a too “imaginative” interpretation on my part...
Thanks a lot in advance for any suggestions
Best regards
Elena Molteni
Physics Department
Universita' degli Studi di Milano
via Celoria, 16
I-20133, Milan, Italy
and European Theoretical Spectroscopy Facility (ETSF)
http://www.etsf.eu