aiscalator.core package

Submodules

aiscalator.core.config module

Handles configurations files for the application

class aiscalator.core.config.AiscalatorConfig(user_config_override=None, step_config=None, steps_selection=None)[source]

Bases: object

A configuration object for the Aiscalator application.

This object stores:
  • global configuration for the whole application
  • configuration for a particular context specified in a step configuration file.
  • In this case, we might even focus on a particular step.

Variables:
  • _user_config_override (str) – path to the specified user config
  • _app_conf – global configuration object for the application
  • _step_config (str) – path to the step configuration (or plain configuration as string)
  • _focused_steps (list) – list of selected steps
  • _step_it (int) – index of the step being processed
  • _step – configuration object for the currently processed step

Methods

app_config()
returns:str – the configuration object for the aiscalator application
app_config_has(field) Tests if the applicatin config has a configuration value for the field.
container_name() Return the docker container name to execute this step
extract_parameters() Returns a list of docker parameters
file_path(string) Returns absolute path of a file from a field of the currently focused step.
find_user_config_file(filename) Looks for configuration files in the user configuration folder
has_step_field(field) Tests if the currently focused step has a configuration value for the field.
next_step() Iterates to the next configuration step from the list of selected steps
notebook_output_path(notebook) Generates the name of the output notebook
root_dir()
returns:str – Returns the path to the folder containing the step
setup_app_config() Setup global application configuration.
step_config_path()
returns:str – Returns the path to the step configuration file.
step_field(field) Returns the value associated with the field for the currently focused step.
step_name() Returns the name of the currently focused step
timestamp_now() Depending on how the timezone is configured, returns the timestamp for this instant.
user_env_file() Find a list of env files to pass to docker containers
user_id()
returns:str – the user id stored when the application was first setup
validate_config() Check if all the fields in the reference config are defined in focused steps too.
app_config()[source]
Returns:str – the configuration object for the aiscalator application
app_config_has(field) → bool[source]

Tests if the applicatin config has a configuration value for the field.

container_name() → str[source]

Return the docker container name to execute this step

extract_parameters() → list[source]

Returns a list of docker parameters

file_path(string)[source]

Returns absolute path of a file from a field of the currently focused step.

find_user_config_file(filename) → str[source]

Looks for configuration files in the user configuration folder

Parameters:filename (str) – file to search for
Returns:str – path to the filename in the user configuration folder
has_step_field(field) → bool[source]

Tests if the currently focused step has a configuration value for the field.

next_step()[source]

Iterates to the next configuration step from the list of selected steps

Returns:the next configuration object
notebook_output_path(notebook) → str[source]

Generates the name of the output notebook

root_dir()[source]
Returns:str – Returns the path to the folder containing the step configuration file
setup_app_config()[source]

Setup global application configuration. If not found in the default location, this method will generate a brand new one.

step_config_path()[source]
Returns:str – Returns the path to the step configuration file. If it was an URL, it will return the path to the temporary downloaded version of it. If it was a plain string, then returns None
step_field(field)[source]

Returns the value associated with the field for the currently focused step.

step_name()[source]

Returns the name of the currently focused step

timestamp_now() → str[source]

Depending on how the timezone is configured, returns the timestamp for this instant.

user_env_file() → list[source]

Find a list of env files to pass to docker containers

Returns:List – env files
user_id() → str[source]
Returns:str – the user id stored when the application was first setup
validate_config()[source]

Check if all the fields in the reference config are defined in focused steps too. Otherwise raise an Exception (either pyhocon.ConfigMissingException or pyhocon.ConfigWrongTypeException)

aiscalator.core.config.convert_to_format(file: str, output: str, output_format: str)[source]

Converts a HOCON file to another format

Parameters:
  • file (str) – hocon file to convert
  • output (str) – output file to produce
  • output_format (str) – format of the output file
Returns:

str – the output file

aiscalator.core.config.find_tasks(tree: pyhocon.config_tree.ConfigTree, path='')[source]

Find all Tasks objects in the Configuration object and report their paths.

Parameters:
  • tree (pyhocon.ConfigTree) – Configuration object
  • path (str) – path that was traversed to get to this tree
Returns:

list – list of names of Configuration objects containing a definition of a section ‘task’

aiscalator.core.config.generate_global_config() → str[source]

Generate a standard configuration file for the application in the user’s home folder ~/.aiscalator/config/aiscalator.conf from the template file in aiscalator/config/template/aiscalator.conf

aiscalator.core.config.generate_user_id() → str[source]
Returns:str – Returns a string identifying this user when the setup was run first
aiscalator.core.config.load_logging_conf(file)[source]

Reads and loads the logging configuration file

aiscalator.core.config.parse_step_config(step_config)[source]

Interpret the step_config to produce a step configuration object. It could be provided as: - a path to a local file - a url to a remote file - the plain configuration stored as string

Returns:Step configuration object
aiscalator.core.config.select_steps(step_conf, steps_selection: list) → list[source]

Extract the list of step objects corresponding to the list of names provided.

Parameters:
  • step_conf – step configuration object
  • steps_selection (list) – list of names of step to extract
Returns:

list – list of tuples of (step_name, step) of selected configuration objects

aiscalator.core.config.setup_logging()[source]

Setup the logging configuration of the application

aiscalator.core.config.validate_configs(test, reference, path, missing_exception=True, type_mismatch_exception=True)[source]

Recursively check two configs if they match

Parameters:
  • test – configuration object to test
  • reference – reference configuration object
  • path (str) – this accumulates the recursive path for details in Exceptions
  • missing_exception (bool) – when a missing field is found, raise xception?
  • type_mismatch_exception (bool) – when a field has type mismatch, raise xception?

aiscalator.core.log_regex_analyzer module

Class to parse output logs from subprocess and catch particular expressions

class aiscalator.core.log_regex_analyzer.LogRegexAnalyzer(pattern=None)[source]

Bases: object

A regular expression analyzer object to parse logs and extract values from patterns in the logs. …

Variables:
  • _artifact (str) – Value of the pattern found in the logs
  • _pattern (bytes) – Regular expression to search for in the logs

Methods

artifact() Returns the artifact extracted from the logs.
grep_logs(pipe) Reads the logs and extract values defined by the pattern
artifact()[source]

Returns the artifact extracted from the logs.

grep_logs(pipe)[source]

Reads the logs and extract values defined by the pattern

Parameters:pipe – Stream of logs to analyze

aiscalator.core.utils module

Various Utility functions

class aiscalator.core.utils.BackgroundThreadRunner(command, log_function, no_redirect=False)[source]

Bases: object

Worker Thread to run logging output in the background

Variables:
  • _process – Process object of the command running in the background
  • _log_function (function(stream -> bool)) – callback function to log the output of the command
  • _no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs
  • _worker (Thread) – Thread object

Methods

process() Returns the process object.
run() Starts the Thread, process the output of the process.
process()[source]

Returns the process object.

run()[source]

Starts the Thread, process the output of the process.

aiscalator.core.utils.copy_replace(src, dst, pattern='', replace_value='')[source]

Copies a file from src to dst replacing pattern by replace_value

Parameters:
  • src (string) – Path to the source filename to copy from
  • dst (string) – Path to the output filename to copy to
  • pattern (string) – Pattern to replace inside the src file
  • replace_value – Value to replace by in the dst file
aiscalator.core.utils.data_file(path)[source]

Utility function to find resources data file packaged along with code

Parameters:path (path) – path to the resource file in the package
Returns:absolute path to the resource data file
aiscalator.core.utils.find(collection, item, field='name')[source]

Finds an element in a collection which has a field equal to particular item value

Parameters:
  • collection (Set) – Collection of objects
  • item – value of the item that we are looking for
  • field (string) – Name of the field from the object to inspect
Returns:

object – Corresponding element that has a field matching item in the collection

aiscalator.core.utils.log_info(pipe)[source]

Default logging function

aiscalator.core.utils.subprocess_run(command, log_function=<function log_info>, no_redirect=False, wait=True)[source]

Run command in a subprocess while redirecting output to log_function.

The subprocess either runs synchroneoulsy or in the background depending on the wait parameter.

Parameters:
  • command (List) – Command to run in the subprocess
  • log_function (function) – Callback function to log the output of the subprocess
  • no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs
  • wait (bool) – Whether the subprocess should be run synchroneously or in the background
Returns:

  • int – return code of the subprocess
  • BackgroundThreadRunner – the thread running in the background

Module contents