bag.concurrent.lsf

This module define utility classes for launching subprocesses via IBM Load Sharing Facility (LSF).

Module Contents

Classes

LSFSubProcessManager

A class that provides methods to run multiple subprocesses in parallel using IBM Load Sharing Facility (LSF).
class bag.concurrent.lsf.LSFSubProcessManager(queue: str, options: Optional[List[str]] = None, max_workers: int = 0, cancel_timeout: float = 10.0)[source]

Bases: bag.concurrent.core.SubProcessManager

A class that provides methods to run multiple subprocesses in parallel using IBM Load Sharing Facility (LSF).

Parameters:

  • queue (str) – name of LSF queue to use for submitting jobs.

  • options (Optional[List[str]]) – list of additional command line arguments to pass into the bsub command.

  • max_workers (Optional[int]) – number of maximum allowed subprocesses. If None, defaults to system CPU count.

  • cancel_timeout (float) – Number of seconds to wait for a process to terminate once SIGTERM or SIGKILL is issued. Defaults to 10 seconds.

async async_new_subprocess(args: Union[str, Sequence[str]], log: str, env: Optional[Dict[str, str]] = None, cwd: Optional[str] = None, run_local: bool = False, **kwargs: Any) → Optional[int][source]

A coroutine which starts a subprocess.

If this coroutine is cancelled, it will shut down the subprocess gracefully using SIGTERM/SIGKILL, then raise CancelledError.

Parameters:

  • args (Union[str, Sequence[str]]) – command to run, as string or sequence of strings.

  • log (str) – the log file name.

  • env (Optional[Dict[str, str]]) – an optional dictionary of environment variables. None to inherit from parent.

  • cwd (Optional[str]) – the working directory. None to inherit from parent.

  • run_local (bool) – True to run locally (not use LSF). False to use LSF. Default is False.

Returns:

retcode – the return code of the subprocess.

Return type:

Optional[int]

async async_new_subprocess_flow(proc_info_list: Sequence[bag.concurrent.core.FlowInfo], run_local: bool = False, **kwargs: Any) → Any[source]

A coroutine which runs a series of subprocesses.

If this coroutine is cancelled, it will shut down the current subprocess gracefully using SIGTERM/SIGKILL, then raise CancelledError.

Parameters:

  • proc_info_list (Sequence[FlowInfo]) –

    a list of processes to execute in series. Each element is a tuple of:

    argsUnion[str, Sequence[str]]

    command to run, as string or list of string arguments.

    logstr

    log file name.

    envOptional[Dict[str, str]]

    environment variable dictionary. None to inherit from parent.

    cwdOptional[str]

    working directory path. None to inherit from parent.

    vfunSequence[Callable[[Optional[int], str], Any]]

    a function to validate if it is ok to execute the next process. The output of the last function is returned. The first argument is the return code, the second argument is the log file name.

  • run_local (bool) – True to run locally (not use LSF). False to use LSF. Default is False.

Returns:

result – the return value of the last validate function. None if validate function returns False.

Return type:

Any

static get_njobs_per_user(queue)[source]
static get_njobs_running(queue)[source]