bag.io.process

This module provides functions to help you run external processes.

Module Contents

Classes

ProcessManager

A class that manages subprocesses.

Functions

run_proc_with_quit(proc_id, quit_dict, args[, ...])

run_and_wait(args[, timeout, logfile, append, env, cwd])

Run a command in a subprocess, then wait for it to finish.
bag.io.process.run_proc_with_quit(proc_id, quit_dict, args, logfile=None, append=False, env=None, cwd=None)[source]
bag.io.process.run_and_wait(args, timeout=None, logfile=None, append=False, env=None, cwd=None)[source]

Run a command in a subprocess, then wait for it to finish.

Parameters:

  • args (string or list[string]) – the command to run. Should be either a command string or a list of command string and its arguments as strings. A list is preferred; see Python subprocess documentation.

  • timeout (float or None) – the amount of time to wait for the command to finish, in seconds. If None, waits indefinitely.

  • logfile (string or None) – If given, stdout and stderr will be written to this file.

  • append (bool) – True to append to the logfile. Defaults to False.

  • env (dict[string, any]) – If not None, environment variables of the subprocess will be set according to this dictionary instead of inheriting from current process.

  • cwd (string or None) – The current working directory of the subprocess.

Returns:

output – the standard output and standard error from the command.

Return type:

string

Raises:

subprocess.CalledProcessError – if any error occurred in the subprocess.

class bag.io.process.ProcessManager(max_workers=None, cancel_timeout=10.0)[source]

Bases: object

A class that manages subprocesses.

This class is for starting processes that you do not need to wait on, and allows you to query for their status or terminate/kill them if needed.

Parameters:

  • max_workers (int or None) – number of maximum allowed subprocesses. If None, defaults to system CPU count.

  • cancel_timeout (float or None) – Number of seconds to wait for a process to terminate once SIGTERM or SIGKILL is issued. Defaults to 10 seconds.

close(timeout=10.0)[source]

Cancel all processes.

Parameters:

timeout (float) – time to wait in seconds for each process to terminate.

new_thread(fun, basename=None, callback=None)[source]

Put a new custom task in queue.

Execute the given function in a thread asynchronously. The given function must take two arguments, The first argument is a unique string that represents this task, and the second argument is a dictionary. The dictionary will map the unique string to a timeout (in second) if this task is being cancelled. The function should periodically check the dictionary and terminate gracefully.

Before function returns, it should also delete the unique string from dictionary if it exists.

Parameters:

  • fun (callable) – the function to execute in a thread, as described above.

  • basename (string or None) – If given, this will be used as the basis for generating the unique process ID.

  • callback (callable) – If given, this function will automatically be executed when the process finished. This function should take a single argument, which is a Future object that returns the return code of the process.

Returns:

proc_id – a unique string representing this process. Can be used later to query process status or cancel process.

Return type:

string

new_process(args, basename=None, logfile=None, append=False, env=None, cwd=None, callback=None)[source]

Put a new process in queue.

When the process is done, its return code will be returned.

Parameters:

  • args (string or list[string]) – the command to run as a string or list of string arguments. See Python subprocess documentation. list of string format is preferred.

  • basename (string or None) – If given, this will be used as the basis for generating the unique process ID.

  • logfile (string or None) – If given, stdout and stderr will be written to this file. Otherwise, they will be redirected to os.devnull.

  • append (bool) – True to append to logfile instead of overwritng it.

  • env (dict[string, string] or None) – If given, environment variables of the process will be set according to this dictionary.

  • cwd (string or None) – current working directory of the process.

  • callback (callable) – If given, this function will automatically be executed when the process finished. This function should take a single argument, which is a Future object that returns the return code of the process.

Returns:

proc_id – a unique string representing this process. Can be used later to query process status or cancel process.

Return type:

string

static _get_output(future, timeout=None)[source]

Get output from future. Return None when exception.

cancel(proc_id, timeout=None)[source]

Cancel the given process.

If the process haven’t started, this method prevents it from started. Otherwise, we first send a SIGTERM signal to kill the process. If after timeout seconds the process is still alive, we will send a SIGKILL signal. If after another timeout seconds the process is still alive, an Exception will be raised.

Parameters:

  • proc_id (string) – the process ID to cancel.

  • timeout (float or None) – number of seconds to wait for cancellation. If None, use default timeout.

Returns:

output of the thread if it successfully terminates. Otherwise, return None.

Return type:

output

done(proc_id)[source]

Returns True if the given process finished or is cancelled successfully.

Parameters:

proc_id (string) – the process ID.

Returns:

done – True if the process is cancelled or completed.

Return type:

bool

wait(proc_id, timeout=None, cancel_timeout=None)[source]

Wait for the given process to finish, then return its return code.

If timeout is None, waits indefinitely. Otherwise, if after timeout seconds the process is still running, a concurrent.futures.TimeoutError will be raised. However, it is safe to catch this error and call wait again.

If Ctrl-C is pressed before process finish or before timeout is reached, the process will be cancelled.

Parameters:

  • proc_id (string) – the process ID.

  • timeout (float or None) – number of seconds to wait. If None, waits indefinitely.

  • cancel_timeout (float or None) – number of seconds to wait for process cancellation. If None, use default timeout.

Returns:

output of the thread if it successfully terminates. Otherwise return None.

Return type:

output

_start_cmd(args, proc_id, logfile=None, append=False, env=None, cwd=None)[source]

The function that actually starts the subprocess. Executed by thread.