ramp_engine.dispatcher.Dispatcher

class ramp_engine.dispatcher.Dispatcher(config, event_config, worker=None, n_workers=1, n_threads=None, hunger_policy=None, time_between_collection=1)

Dispatcher which schedule workers and communicate with the database.

The dispatcher uses two queues: a queue containing containing the workers which should be launched and a queue containing the workers which are being processed. The latter queue has a limited size defined by n_workers. Note that these workers can run simultaneously.

Parameters

configdict or str

A configuration YAML file containing the information about the database.

event_configdict or str

A RAMP configuration YAML file with information regarding the worker and the ramp event.

workerWorker, default=CondaEnvWorker

The type of worker to launch. By default, we launch local worker which uses conda.

n_workersint, default=1

Maximum number of workers which can run submissions simultaneously.

n_threadsNone or int

The number of threads that each worker can use. By default, there is no limit imposed.

hunger_policy{None, ‘sleep’, ‘exit’}

Policy to apply in case that there is no anymore workers to be processed:

• if None: the dispatcher will work without interruption;

• if ‘sleep’: the dispatcher will sleep for 5 seconds before to check for new submission;

• if ‘exit’: the dispatcher will stop after collecting the results of the last submissions.

time_between_collectionint, default=1

The amount of time in seconds to wait before checking if we can collect results from worker.

Note

This parameter is important when using a cloud platform to run submissions, as the check for collection will be done through SSH. Thus, if the time between checks is too small, the repetitive SSH requests may be potentially blocked by the cloud provider.

__init__(config, event_config, worker=None, n_workers=1, n_threads=None, hunger_policy=None, time_between_collection=1)

Initialize self. See help(type(self)) for accurate signature.