abiflows.core package

Submodules

abiflows.core.controllers module

Controllers

class abiflows.core.controllers.AbinitController(critical_events=None, handlers=None)

Bases: abiflows.core.mastermind_abc.Controller

General handler for abinit’s events. Determines whether the calculation ended correctly or not and fixes errors (including unconverged) if Abinit error handlers are available.

is_handler = True

is_validator = True

can_validate = True

process(**kwargs)

Returns the ControllerNote

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

as_dict()

A JSON serializable dict representation of an object.

get_event_report(ofile, mpiabort_file)

Analyzes the main output file for possible Errors or Warnings.

Returns

EventReport instance or None if the main output file does not exist.

fix_abicritical(report, abiinput, outdir, queue_adapter=None)

method to fix crashes/error caused by abinit

Returns

1 if task has been fixed else 0. reset: True if at least one of the corrections applied requires a reset

Return type

retcode

classmethod from_helper(helper)

class abiflows.core.controllers.QueueControllerMixin

Bases: object

get_queue_errors(**kwargs)

class abiflows.core.controllers.WalltimeController(max_timelimit=None, timelimit_increase=None)

Bases: abiflows.core.mastermind_abc.Controller, abiflows.core.controllers.QueueControllerMixin

Controller for walltime infringements of the resource manager.

is_handler = True

as_dict()

A JSON serializable dict representation of an object.

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

property skip_remaining_handlers

property skip_lower_priority_controllers

process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

class abiflows.core.controllers.MemoryController(max_mem_per_proc_mb=8000, mem_per_proc_increase_mb=1000, max_master_mem_overhead_mb=8000, master_mem_overhead_increase_mb=1000, memory_policy='physical_memory')

Bases: abiflows.core.mastermind_abc.Controller, abiflows.core.controllers.QueueControllerMixin

Controller for memory infringements of the resource manager. The handler should be able to handle the possible overhead of the master process.

is_handler = True

property memory_policy

as_dict()

A JSON serializable dict representation of an object.

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

property skip_remaining_handlers

property skip_lower_priority_controllers

process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

class abiflows.core.controllers.AbinitZenobeSlaveMemoryController(max_mem_per_proc_mb=8000, mem_per_proc_increase_mb=1000, memory_policy='physical_memory')

Bases: abiflows.core.mastermind_abc.Controller, abiflows.core.controllers.QueueControllerMixin

Controller for memory infringements of the resource manager. The handler should be able to handle the possible overhead of the master process.

is_handler = True

property memory_policy

as_dict()

A JSON serializable dict representation of an object.

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

property skip_remaining_handlers

property skip_lower_priority_controllers

process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

class abiflows.core.controllers.UltimateMemoryController(max_mem_per_proc_mb=8000, mem_per_proc_increase_mb=1000, max_master_mem_overhead_mb=8000, master_mem_overhead_increase_mb=1000, memory_policy='physical_memory')

Bases: abiflows.core.mastermind_abc.Controller, abiflows.core.controllers.QueueControllerMixin

Controller for memory infringements of the resource manager. The handler should be able to handle the possible overhead of the master process.

is_handler = True

property memory_policy

as_dict()

A JSON serializable dict representation of an object.

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

property skip_remaining_handlers

property skip_lower_priority_controllers

process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

class abiflows.core.controllers.SimpleValidatorController

Bases: abiflows.core.mastermind_abc.Controller

Simple validator controller to be applied after all other controllers (PRIORITY_LOWEST). This validator controller can be used when no “real” validator exists, but just handlers/monitors and that we suppose that if nothing is found by the handlers/monitors, then it means that it is ok.

can_validate = True

as_dict()

A JSON serializable dict representation of an object.

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

property skip_remaining_handlers

property skip_lower_priority_controllers

process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

property validated

class abiflows.core.controllers.VaspXMLValidatorController

Bases: abiflows.core.mastermind_abc.Controller

Checks that a valid vasprun.xml was generated

can_validate = True

as_dict()

A JSON serializable dict representation of an object.

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

property skip_remaining_handlers

property skip_lower_priority_controllers

process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

property validated

class abiflows.core.controllers.VaspNEBValidatorController

Bases: abiflows.core.mastermind_abc.Controller

Checks that a valid vasprun.xml was generated

can_validate = True

as_dict()

A JSON serializable dict representation of an object.

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

property skip_remaining_handlers

property skip_lower_priority_controllers

process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

property validated

abiflows.core.mastermind_abc module

Abstract base classes for controllers, monitors, and other possible tools/utils/objects used to manage, correct, monitor and check tasks, events, results, objects, …

class abiflows.core.mastermind_abc.ControlProcedure(controllers, monitors=None, sorting=None)

Bases: monty.json.MSONable

set_controlled_item_type(controlled_item_type)

setup_controllers(controlled_item_type)

add_controller(controller)

add_controllers(controllers)

process(**kwargs)

property ncontrollers

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

as_dict()

A JSON serializable dict representation of an object.

class abiflows.core.mastermind_abc.ControlledItemType(item_type)

Bases: monty.json.MSONable

ALLOWED_CONTROL_ITEMS = {'FILE': ['MISSING', 'EMPTY', 'CORRUPTED', 'ERRONEOUS', 'VALID'], 'OBJECT': ['CORRUPTED', 'ERRONEOUS', 'VALID'], 'TASK': ['READY', 'NOT_READY', 'VALID'], 'TASK_ABORTED': ['RECOVERABLE', 'UNRECOVERABLE'], 'TASK_BEFORE_EXECUTION': ['READY', 'NOT_READY'], 'TASK_COMPLETED': ['VALID', 'RECOVERABLE', 'UNRECOVERABLE'], 'TASK_FAILED': ['RECOVERABLE', 'UNRECOVERABLE', 'UNKNOWN_REASON'], 'TASK_RUNNING': ['VALID', 'RECOVERABLE']}

classmethod task()

classmethod task_running()

classmethod task_aborted()

classmethod task_failed()

classmethod task_completed()

classmethod file()

classmethod object()

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

as_dict()

A JSON serializable dict representation of an object.

class abiflows.core.mastermind_abc.Monitor

Bases: object

class abiflows.core.mastermind_abc.Controller

Bases: monty.json.MSONable

Abstract base class for controlling a task, an event, a result, an output, an object, …

can_validate = False

abstract from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

abstract as_dict()

A JSON serializable dict representation of an object.

abstract process(**kwargs)

Main function used to make the actual control/check of a list of inputs/outputs. The function should return a ControllerNote object containing the main conclusion of the controller, i.e. whether something important has been detected, as well as the possible actions/corrections to be done in order to carry on/continue/restart a task.

set_priority(priority)

property priority

property controlled_item_types

property skip_remaining_controllers

property skip_lower_priority_controllers

property validated

class abiflows.core.mastermind_abc.Cleaner(dirs_and_patterns)

Bases: monty.json.MSONable

clean(root_directory)

class abiflows.core.mastermind_abc.Manager

Bases: monty.json.MSONable

abstract manage()

class abiflows.core.mastermind_abc.ControllerNote(controller, state=None, problems=None, actions=None, restart=None, is_valid=None)

Bases: monty.json.MSONable

EVERYTHING_OK = 'EVERYTHING_OK'

NOTHING_FOUND = 'NOTHING_FOUND'

ERROR_UNRECOVERABLE = 'ERROR_UNRECOVERABLE'

ERROR_NOFIX = 'ERROR_NOFIX'

ERROR_RECOVERABLE = 'ERROR_RECOVERABLE'

STATES = ['EVERYTHING_OK', 'NOTHING_FOUND', 'ERROR_UNRECOVERABLE', 'ERROR_NOFIX', 'ERROR_RECOVERABLE']

ERROR_STATES = ['ERROR_UNRECOVERABLE', 'ERROR_NOFIX', 'ERROR_RECOVERABLE']

RESTART_FROM_SCRATCH = 'RESTART_FROM_SCRATCH'

RESET_RESTART = 'RESET_RESTART'

SIMPLE_RESTART = 'SIMPLE_RESTART'

property actions

add_problem(problem)

property state

property has_errors_recoverable

property has_errors_unrecoverable

property is_recoverable

property has_errors

property is_valid

set_restart(restart)

simple_restart()

reset_restart()

restart_from_scratch()

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

as_dict()

A JSON serializable dict representation of an object.

class abiflows.core.mastermind_abc.ControlReport(controller_notes=None)

Bases: monty.json.MSONable

FINALIZED = 'FINALIZED'

UNRECOVERABLE = 'UNRECOVERABLE'

RECOVERABLE = 'RECOVERABLE'

ONGOING = 'ONGOING'

NONE = 'NONE'

STATES = ['FINALIZED', 'UNRECOVERABLE', 'RECOVERABLE', 'ONGOING', 'NONE']

add_controller_notes(controller_notes)

add_controller_note(controller_note)

property state

update_state_from_controller_notes()

property finalized

property unrecoverable

property actions

property restart_info

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

as_dict()

A JSON serializable dict representation of an object.

class abiflows.core.mastermind_abc.Action(callable, **kwargs)

Bases: monty.json.MSONable

apply(object)

classmethod from_dict(d)

Parameters

d – Dict representation.

Returns

MSONable class.

as_dict()

A JSON serializable dict representation of an object.

classmethod from_string(callable_string, **kwargs)

abiflows.core.models module

Object-Document mapper

class abiflows.core.models.AbiGridFSProxy(grid_id=None, key=None, instance=None, db_alias='default', collection_name='fs')

Bases: mongoengine.fields.GridFSProxy

abiopen()

Dump the gridfs data to a temporary file and use abiopen to open the file.

class abiflows.core.models.AbiFileField(**kwargs)

Bases: mongoengine.fields.FileField

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

proxy_class

alias of AbiGridFSProxy

get_proxy_obj(**kwargs)

to_python(value)

Convert a MongoDB-compatible type to a Python type.

class abiflows.core.models.GzipGridFSProxy(grid_id=None, key=None, instance=None, db_alias='default', collection_name='fs')

Bases: mongoengine.fields.GridFSProxy

Proxy object to handle writing and reading of files to and from GridFS. Files are compressed with gzip before being saved in the GridFS. To decompress the object exposes an unzip method.

put(file_obj, **kwargs)

write(*args, **kwargs)

writelines(*args, **kwargs)

to_gzip()

unzip(filepath=None, mode='w+b')

class abiflows.core.models.GzipFileField(compresslevel=9, **kwargs)

Bases: mongoengine.fields.FileField

A GridFS storage field with automatic compression of the file with gzip.

proxy_class

alias of GzipGridFSProxy

class abiflows.core.models.AbiGzipFSProxy(grid_id=None, key=None, instance=None, db_alias='default', collection_name='fs')

Bases: abiflows.core.models.GzipGridFSProxy

Proxy object to handle writing and reading of abinit related files to and from GridFS. Files are compressed with gzip before being saved in the GridFS. To decompress the object exposes an unzip method.

abiopen()

Dump the unzipped gridfs data to a temporary file and use abiopen to open the file.

class abiflows.core.models.AbiGzipFileField(abiext, abiform, compresslevel=9, **kwargs)

Bases: abiflows.core.models.GzipFileField

A GridFS storage field for abinit related files with automatic compression of the file with gzip.

proxy_class

alias of AbiGzipFSProxy

class abiflows.core.models.MongoFiles(*args, **kwargs)

Bases: mongoengine.document.EmbeddedDocument

Document with the output files produced by the AbiPy pymatgen.io.abinit.tasks.Task (references to GridFs files)

gsr

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

hist

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

phbst

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

phdos

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

sigres

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

mdf

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

ddb

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

output_file

Extend FileField. Use customized version of proxy_class so that we can use abiopen to construct the AbiPy object from the gridfs content.

classmethod from_node(node)

Add to GridFs the files produced in the outdir of the node.

delete()

Delete gridFs files

class abiflows.core.models.MSONDict(dict_items, instance, name)

Bases: mongoengine.base.datastructures.BaseDict

to_mgobj()

class abiflows.core.models.MSONField(field=None, *args, **kwargs)

Bases: mongoengine.fields.DictField

class abiflows.core.models.MongoTaskResults(*args, **kwargs)

Bases: mongoengine.document.EmbeddedDocument

Document with the most important results produced by the AbiPy pymatgen.io.abinit.tasks.Task

initial_structure

final_structure

The final relaxed structure in a dict format.

classmethod from_task(task)

class abiflows.core.models.MongoNode(*args, **values)

Bases: mongoengine.document.Document

node_id

64-bit integer field.

node_class

A unicode string field.

status

A unicode string field.

workdir

A unicode string field.

classmethod from_node(node)

exception DoesNotExist

Bases: mongoengine.errors.DoesNotExist

exception MultipleObjectsReturned

Bases: mongoengine.errors.MultipleObjectsReturned

id

A field wrapper around MongoDB’s ObjectIds.

objects

The default QuerySet Manager.

Custom QuerySet Manager functions can extend this class and users can add extra queryset functionality. Any custom manager methods must accept a Document class as its first argument, and a QuerySet as its second argument.

The method function should return a QuerySet , probably the same one that was passed in, but modified in some way.

class abiflows.core.models.MongoEmbeddedNode(*args, **kwargs)

Bases: mongoengine.document.EmbeddedDocument

node_id

64-bit integer field.

node_class

A unicode string field.

status

A unicode string field.

workdir

A unicode string field.

classmethod from_node(node)

class abiflows.core.models.MongoTask(*args, **kwargs)

Bases: abiflows.core.models.MongoEmbeddedNode

Document associated to an AbiPy pymatgen.io.abinit.tasks.Task

input

A dictionary field that wraps a standard Python dictionary. This is similar to an embedded document, but the structure is not defined.

Note

Required means it cannot be empty - as the default for DictFields is {}

New in version 0.3.

Changed in version 0.5: - Can now handle complex / varying types of data

input_str

A unicode string field.

report

A dictionary field that wraps a standard Python dictionary. This is similar to an embedded document, but the structure is not defined.

Note

Required means it cannot be empty - as the default for DictFields is {}

New in version 0.3.

Changed in version 0.5: - Can now handle complex / varying types of data

num_warnings

32-bit integer field.

num_errors

32-bit integer field.

num_comments

32-bit integer field.

results

An embedded document field - with a declared document_type. Only valid values are subclasses of EmbeddedDocument.

outfiles

An embedded document field - with a declared document_type. Only valid values are subclasses of EmbeddedDocument.

classmethod from_task(task)

Build the document from an AbiPy pymatgen.io.abinit.tasks.Task object.

class abiflows.core.models.MongoWork(*args, **kwargs)

Bases: abiflows.core.models.MongoEmbeddedNode

Document associated to an AbiPy pymatgen.io.abinit.works.Work

tasks

List of tasks.

outfiles

Output files produced by the work.

classmethod from_work(work)

Build and return the document from a AbiPy pymatgen.io.abinit.works.Work instance.

class abiflows.core.models.MongoFlow(*args, **values)

Bases: abiflows.core.models.MongoNode

Document associated to an AbiPy pymatgen.io.abinit.flows.Flow

Assumptions:

All the tasks must have the same list of pseudos, same chemical formula.

works

List of works

outfiles

Output files produced by the flow.

classmethod from_flow(flow)

Build and return the document from a AbiPy pymatgen.io.abinit.flows.Flow instance.

pickle_load()

Load the pickle file from the working directory of the flow.

Return: AbiPy pymatgen.io.abinit.flows.Flow object.

delete()

Delete the Document from the database. This will only take effect if the document has been previously saved.

Parameters

• signal_kwargs – (optional) kwargs dictionary to be passed to the signal calls.

• write_concern – Extra keyword arguments are passed down which will be used as options for the resultant getLastError command. For example, save(..., w: 2, fsync: True) will wait until at least two servers have recorded the write and will force an fsync on the primary server.

Changed in version 0.10.7: Add signal_kwargs argument

completed

The default QuerySet Manager.

Custom QuerySet Manager functions can extend this class and users can add extra queryset functionality. Any custom manager methods must accept a Document class as its first argument, and a QuerySet as its second argument.

The method function should return a QuerySet , probably the same one that was passed in, but modified in some way.

exception DoesNotExist

Bases: abiflows.core.models.DoesNotExist

exception MultipleObjectsReturned

Bases: abiflows.core.models.MultipleObjectsReturned

abiflows.core.release module

Release data for the abiflows project.

abiflows.core.testing module

Common test support for abiflows test scripts.

This single module should provide all the common functionality for abiflows tests in a single location, so that test scripts can just import it and work right away. This module heavily depends on the abipy.testing module/

class abiflows.core.testing.AbiflowsTest(methodName='runTest')

Bases: abipy.core.testing.AbipyTest

Extends AbipyTest with methods specific for the testing of workflows

assertFwSerializable(obj)

classmethod setup_fireworks()

Sets up the fworker and launchpad if a connection to a local mongodb is available. cls.lp is set to None if not available

classmethod teardown_fireworks(module_dir=None)

Removes the fireworks test database if cls.lp is present and deletes all the launcher directories

classmethod setup_mongoengine()

classmethod teardown_mongoengine()

get_document_class_from_mixin(mixin_cls)

Utility function to generate a mongoengine Document class from the mixin. Needed to save the object in the db with mongoengine

Module contents