aiscalator.core package¶
Submodules¶
aiscalator.core.config module¶
Handles configurations files for the application
-
class
aiscalator.core.config.
AiscalatorConfig
(user_config_override=None, step_config=None, steps_selection=None)[source]¶ Bases:
object
A configuration object for the Aiscalator application.
- This object stores:
- global configuration for the whole application
- configuration for a particular context specified in a step configuration file.
- In this case, we might even focus on a particular step.
…
Variables: - _user_config_override (str) – path to the specified user config
- _app_conf – global configuration object for the application
- _step_config (str) – path to the step configuration (or plain configuration as string)
- _focused_steps (list) – list of selected steps
- _step_it (int) – index of the step being processed
- _step – configuration object for the currently processed step
Methods
app_config
()returns: str – the configuration object for the aiscalator application app_config_has
(field)Tests if the applicatin config has a configuration value for the field. container_name
()Return the docker container name to execute this step extract_parameters
()Returns a list of docker parameters file_path
(string)Returns absolute path of a file from a field of the currently focused step. find_user_config_file
(filename)Looks for configuration files in the user configuration folder has_step_field
(field)Tests if the currently focused step has a configuration value for the field. next_step
()Iterates to the next configuration step from the list of selected steps notebook_output_path
(notebook)Generates the name of the output notebook root_dir
()returns: str – Returns the path to the folder containing the step setup_app_config
()Setup global application configuration. step_config_path
()returns: str – Returns the path to the step configuration file. step_field
(field)Returns the value associated with the field for the currently focused step. step_name
()Returns the name of the currently focused step timestamp_now
()Depending on how the timezone is configured, returns the timestamp for this instant. user_env_file
()Find a list of env files to pass to docker containers user_id
()returns: str – the user id stored when the application was first setup validate_config
()Check if all the fields in the reference config are defined in focused steps too. -
app_config_has
(field) → bool[source]¶ Tests if the applicatin config has a configuration value for the field.
-
file_path
(string)[source]¶ Returns absolute path of a file from a field of the currently focused step.
-
find_user_config_file
(filename) → str[source]¶ Looks for configuration files in the user configuration folder
Parameters: filename (str) – file to search for Returns: str – path to the filename in the user configuration folder
-
has_step_field
(field) → bool[source]¶ Tests if the currently focused step has a configuration value for the field.
-
next_step
()[source]¶ Iterates to the next configuration step from the list of selected steps
Returns: the next configuration object
-
root_dir
()[source]¶ Returns: str – Returns the path to the folder containing the step configuration file
-
setup_app_config
()[source]¶ Setup global application configuration. If not found in the default location, this method will generate a brand new one.
-
step_config_path
()[source]¶ Returns: str – Returns the path to the step configuration file. If it was an URL, it will return the path to the temporary downloaded version of it. If it was a plain string, then returns None
-
step_field
(field)[source]¶ Returns the value associated with the field for the currently focused step.
-
timestamp_now
() → str[source]¶ Depending on how the timezone is configured, returns the timestamp for this instant.
-
aiscalator.core.config.
convert_to_format
(file: str, output: str, output_format: str)[source]¶ Converts a HOCON file to another format
Parameters: - file (str) – hocon file to convert
- output (str) – output file to produce
- output_format (str) – format of the output file
Returns: str – the output file
-
aiscalator.core.config.
find_tasks
(tree: pyhocon.config_tree.ConfigTree, path='')[source]¶ Find all Tasks objects in the Configuration object and report their paths.
Parameters: - tree (pyhocon.ConfigTree) – Configuration object
- path (str) – path that was traversed to get to this tree
Returns: list – list of names of Configuration objects containing a definition of a section ‘task’
-
aiscalator.core.config.
generate_global_config
() → str[source]¶ Generate a standard configuration file for the application in the user’s home folder ~/.aiscalator/config/aiscalator.conf from the template file in aiscalator/config/template/aiscalator.conf
-
aiscalator.core.config.
generate_user_id
() → str[source]¶ Returns: str – Returns a string identifying this user when the setup was run first
-
aiscalator.core.config.
load_logging_conf
(file)[source]¶ Reads and loads the logging configuration file
-
aiscalator.core.config.
parse_step_config
(step_config)[source]¶ Interpret the step_config to produce a step configuration object. It could be provided as: - a path to a local file - a url to a remote file - the plain configuration stored as string
Returns: Step configuration object
-
aiscalator.core.config.
select_steps
(step_conf, steps_selection: list) → list[source]¶ Extract the list of step objects corresponding to the list of names provided.
Parameters: - step_conf – step configuration object
- steps_selection (list) – list of names of step to extract
Returns: list – list of tuples of (step_name, step) of selected configuration objects
-
aiscalator.core.config.
validate_configs
(test, reference, path, missing_exception=True, type_mismatch_exception=True)[source]¶ Recursively check two configs if they match
Parameters: - test – configuration object to test
- reference – reference configuration object
- path (str) – this accumulates the recursive path for details in Exceptions
- missing_exception (bool) – when a missing field is found, raise xception?
- type_mismatch_exception (bool) – when a field has type mismatch, raise xception?
aiscalator.core.log_regex_analyzer module¶
Class to parse output logs from subprocess and catch particular expressions
-
class
aiscalator.core.log_regex_analyzer.
LogRegexAnalyzer
(pattern=None)[source]¶ Bases:
object
A regular expression analyzer object to parse logs and extract values from patterns in the logs. …
Variables: - _artifact (str) – Value of the pattern found in the logs
- _pattern (bytes) – Regular expression to search for in the logs
Methods
artifact
()Returns the artifact extracted from the logs. grep_logs
(pipe)Reads the logs and extract values defined by the pattern
aiscalator.core.utils module¶
Various Utility functions
-
class
aiscalator.core.utils.
BackgroundThreadRunner
(command, log_function, no_redirect=False)[source]¶ Bases:
object
Worker Thread to run logging output in the background
…
Variables: - _process – Process object of the command running in the background
- _log_function (function(stream -> bool)) – callback function to log the output of the command
- _no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs
- _worker (Thread) – Thread object
Methods
process
()Returns the process object. run
()Starts the Thread, process the output of the process.
-
aiscalator.core.utils.
copy_replace
(src, dst, pattern='', replace_value='')[source]¶ Copies a file from src to dst replacing pattern by replace_value
Parameters: - src (string) – Path to the source filename to copy from
- dst (string) – Path to the output filename to copy to
- pattern (string) – Pattern to replace inside the src file
- replace_value – Value to replace by in the dst file
-
aiscalator.core.utils.
data_file
(path)[source]¶ Utility function to find resources data file packaged along with code
Parameters: path (path) – path to the resource file in the package Returns: absolute path to the resource data file
-
aiscalator.core.utils.
find
(collection, item, field='name')[source]¶ Finds an element in a collection which has a field equal to particular item value
Parameters: - collection (Set) – Collection of objects
- item – value of the item that we are looking for
- field (string) – Name of the field from the object to inspect
Returns: object – Corresponding element that has a field matching item in the collection
-
aiscalator.core.utils.
subprocess_run
(command, log_function=<function log_info>, no_redirect=False, wait=True)[source]¶ Run command in a subprocess while redirecting output to log_function.
The subprocess either runs synchroneoulsy or in the background depending on the wait parameter.
Parameters: - command (List) – Command to run in the subprocess
- log_function (function) – Callback function to log the output of the subprocess
- no_redirect (bool) – whether the subprocess STDOUT and STDERR should be redirected to logs
- wait (bool) – Whether the subprocess should be run synchroneously or in the background
Returns: - int – return code of the subprocess
- BackgroundThreadRunner – the thread running in the background