yaecs.config

yaecs.config.config module

class Configuration(name='main', overwriting_regime='auto-save', config_path_or_dictionary=None, nesting_hierarchy=None, state=None, main_config=None, variation=None, do_not_pre_process=False, do_not_post_process=False, verbose=True, **kwargs)

Bases: _ConfigurationBase

Superclass for YAECS configurations. The superclass implements constructors, while the behaviour is spread across the parent classes in the following way :

_ConfigurationBase implements the most basic functionalities such as creation and merging operations. It inherits from all other parent classes as Mixins ;
ConfigHooksMixin implements processing functions whose name start with register_as_ and are decorated by the yaecs_utils.hook decorator. Users can use those processing either as pre- or post-processing functions, which will have the added effect of tagging the processed parameters as playing a certain pre-defined role in the config ;
ConfigGettersMixin implements public getters to access some private attributes as well as other values which require boilerplate code to be rigorously queried, such as user-defined parameters ;
ConfigSettersMixin implements a few setters which the config uses internally to manipulate private attributes of other Configuration instances (such as sub-configs) ;
ConfigConvenienceMixin implements all functions which are not of central importance to the behaviour of the config but wrap convenient functionalities which can be used either internally of by the user.

Should never be called directly by the user. Please use one of the constructors instead (load_config, build_from_configs, build_from_argv), or the utils.make_config convenience function.

Parameters:

name (str) – name for the config or sub-config
overwriting_regime (str) – can be “auto-save” (default, when a param is overwritten it is merged instead and the config is saved automatically if it had been saved previously), “locked” (params can’t be overwritten except using merge explicitly) or “unsafe” (params can be freely overwritten but reproducibility is not guaranteed).
config_path_or_dictionary (Union[str, dict, None]) – path or dictionary to create the config from
nesting_hierarchy (Optional[List[str]]) – list containing the names of all the configs in the sub-config chain leading to this config
state (Optional[List[str]]) – processing state used for state tracking and debugging
main_config (Optional[Configuration]) – main config corresponding to this sub-config, or None if this config is the main config
variation (Optional[str]) – the name of the variation being created
do_not_pre_process (bool) – if true, pre-processing is deactivated in this initialization
do_not_post_process (bool) – if true, post-processing is deactivated in this initialization
verbose (bool) – if set to false, message logging by this config will be deactivated

Raises:

ValueError – if the overwriting regime is not valid

Returns:

none

config_metadata: dict

classmethod load_config(*configs, default_config_path=None, overwriting_regime='auto-save', do_not_merge_command_line=False, do_not_pre_process=False, do_not_post_process=False, **kwargs)

First creates a config using the default config, then merges config_path into it. If config_path is a list, successively merges all configs in the list instead from index 0 to the last.

Parameters:

configs (Union[List[Union[str, dict]], str, dict]) – config’s path or dictionary, or list of default config’s paths or dictionaries to merge
default_config_path (Union[str, dict, None]) – default config’s path or dictionary
overwriting_regime (str) – can be “auto-save” (default, when a param is overwritten it is merged instead and the config is saved automatically if it had been saved previously), “locked” (params can’t be overwritten except using merge explicitly) or “unsafe” (params can be freely overwritten but reproducibility is not guaranteed).
do_not_merge_command_line (bool) – if True, does not try to merge the command line parameters
do_not_pre_process (bool) – if true, pre-processing is deactivated in this initialization
do_not_post_process (bool) – if true, post-processing is deactivated in this initialization
kwargs – additional parameters to pass to Configuration

Return type:

Configuration

Returns:

instance of Configuration object containing the desired config

classmethod build_from_configs(*configs, overwriting_regime='auto-save', do_not_merge_command_line=False, do_not_pre_process=False, do_not_post_process=False, **kwargs)

First creates a config using the first config provided (or the first config in the provided list), then merges the subsequent configs into it from index 1 to the last.

Parameters:

configs (Union[List[Union[str, dict]], str, dict]) – config’s path or dictionary, or list of default config’s paths or dictionaries to merge
overwriting_regime (str) – can be “auto-save” (default, when a param is overwritten it is merged instead and the config is saved automatically if it had been saved previously), “locked” (params can’t be overwritten except using merge explicitly) or “unsafe” (params can be freely overwritten but reproducibility is not guaranteed).
do_not_merge_command_line (bool) – if True, does not try to merge the command line parameters
do_not_pre_process (bool) – if true, pre-processing is deactivated in this initialization
do_not_post_process (bool) – if true, post-processing is deactivated in this initialization
kwargs – additional parameters to pass to Configuration

Raises:

TypeError – if configs is invalid

Return type:

Configuration

Returns:

instance of Configuration object containing the desired config

classmethod build_from_argv(*configs, fallback=None, pattern='--config', default_config_path=None, overwriting_regime='auto-save', do_not_merge_command_line=False, do_not_pre_process=False, do_not_post_process=False, **kwargs)

Assumes a pattern of the form ‘–config <path_to_config>’ or ‘–config [<path1>,<path2>,…]’ in sys.argv (the brackets are optional), and builds a config from the specified paths by merging them into the default config in the specified order.

Parameters:

configs (Union[List[Union[str, dict]], str, dict]) – config’s path or dictionary, or list of config paths or dictionaries to merge. Those will be merged to the default config before the config from the command line.
fallback (Union[List[Union[str, dict]], str, dict, None]) – config path or dictionary, or list of config paths or dictionaries to fall back to if no config was detected in the argv
pattern (str) – pattern to look for in sys.argv
default_config_path (Union[str, dict, None]) – default config’s path or dictionary
overwriting_regime (str) – can be “auto-save” (default, when a param is overwritten it is merged instead and the config is saved automatically if it had been saved previously), “locked” (params can’t be overwritten except using merge explicitly) or “unsafe” (params can be freely overwritten but reproducibility is not guaranteed).
do_not_merge_command_line (bool) – if True, does not try to merge the command line parameters
do_not_pre_process (bool) – if true, pre-processing is deactivated in this initialization
do_not_post_process (bool) – if true, post-processing is deactivated in this initialization
kwargs – additional parameters to pass to Configuration

Raises:

TypeError – if –config is not found and no fallback

Return type:

Configuration

Returns:

instance of Configuration object containing the desired config

create_variations()

Creates a list of configs that are derived from the current config using the internally tracked variations and grids registered via the corresponding functions (register_as_config_variations and register_as_grid).

Raises:: TypeError – if grid dimension is empty or not registered
Return type:: List[Configuration]
Returns:: the list of configs corresponding to the tracked variations

static get_default_config_path()

Returns the path to the default config of the project. This function must be implemented at project-level.

Return type:: str
Returns:: string corresponding to the path to the default config of the project

parameters_pre_processing()

Returns a dictionary where the keys are parameter names (supporting the ‘*’ character as a replacement for any number of characters) and the items are functions. The pre-processing functions need to take a single argument and return the new value of the parameter after pre-processing. During pre-processing, all parameters corresponding to the parameter name are passed to the corresponding function and their value is replaced by the value returned by the corresponding function. This function must be implemented at project-level.

Using this is advised when an action needs to happen during the ongoing creation or merging operation, such as the register_as_additional_config_file processing function, or when a parameter is stored on disk using a format that you would prefer to not be used within the config, as the pre-processing function will be performed before the parameter even enters the Configuration object.

Conversions to non-YAML-readable types are forbidden using pre-processing. Please use post-processing for those functions.

Return type:: Dict[str, Callable]
Returns:: dictionary of the pre-processing functions

parameters_post_processing()

Returns a dictionary where the keys are parameter names (supporting the ‘*’ character as a replacement for any number of characters) and the values are functions. The post-processing functions need to take a single argument and return the new value of the parameter after post-processing. After any creation or merging operation, parameters which were modified by said operation get post-processed according to the specified functions. This function can be implemented at project-level.

Using this is advised for type-changing processing functions or processing functions which have consequences beyond the value of that parameter (for example if they rely on another parameter being initialised, or if they would create directories). A notable exception to this rule of thumb is the register_as_additional_config_file processing function, which should almost always be called as a pre-processing function. Since the value of the parameter should no longer change after post-processing, you can also use post-processing to check if the value of the parameter has the correct type or is in the correct range.

Return type:: Dict[str, Callable]
Returns:: dictionary of the post-processing functions

set_variation_name(name, deep=False)

Sets the variation index of the config. This function is not intended to be used by the user.

Parameters:

name (str) – index to set the variation index with
deep (bool) – whether to also recursively set the variation name of all sub-configs

Return type:

None

yaecs.config.config_base module

yaecs.config.config_convenience module

class ConfigConvenienceMixin(*args, **kwargs)

Bases: object

Convenience functions Mixin class for YAECS configurations.

config_metadata: dict

get: Callable[[str, Any], Any]

get_dict: Callable[[bool], dict]

get_main_config: Callable[[], Configuration]

get_name: Callable[[], str]

get_parameter_names: Callable[[bool], List[str]]

get_pre_post_processing_values: Callable[[], Dict[str, Any]]

compare(other, reduce=False)

Returns a list of tuples, where each tuple represents a parameter that is different between the “self” configuration and the “other” configuration. Tuples are written in the form : (parameter_name, parameter_value_in_other). If parameter_name does not exist in other, (parameter_name, None) is given instead.

Parameters:

other (Configuration) – config to compare self with
reduce (bool) – tries to reduce the size of the output text as much as possible

Return type:

List[Tuple[str, Optional[Any]]]

Returns:

difference list

copy()

Returns a safe, independent copy of the config

Return type:: ConfigConvenienceMixin
Returns:: instance of Configuration that is a deep copy of the config

details(show_only=None, expand_only=None, no_show=None, no_expand=None)

Creates and returns a string describing all the parameters in the config and its sub-configs.

Parameters:

show_only (Union[str, List[str], None]) – if not None, list of names referring to params. Only params in the list are displayed in the details.
expand_only (Union[str, List[str], None]) – if not None, list of names referring to sub-configs. Only sub-configs in the list are unrolled in the details.
no_show (Union[str, List[str], None]) – if not None, list of names referring to params. Params in the list are not displayed in the details.
no_expand (Union[str, List[str], None]) – if not None, list of names referring to sub-configs. Sub-configs in the list are not unrolled in the details.

Return type:

str

Returns:

string containing the details

items(deep=False)

Behaves as dict.items(). If deep is False, sub-configs remain sub-configs in the items. Otherwise, they are converted to dict.

Parameters:: deep (bool) – how to return sub-configs that would appear among the items. If False, do not convert them, else recursively convert them to dict
Return type:: ItemsView
Returns:: the items of the config as in dict.items()

keys()

Behaves as dict.keys(), returning a _dict_keys instance containing the names of the params of the config.

Return type:: KeysView
Returns:: the keys if the config as in dict.keys()

match_params(*patterns)

For a string, a list of strings or several strings, returns all params matching at least one of the input strings.

Parameters:: patterns (Union[str, List[str], None]) – string, a list of strings or several strings
Return type:: Optional[List[str]]
Returns:: all params matching at least one of the input string

save(filename=None, save_header=True, save_hierarchy=True)

Saves the current config at the provided location. The saving format allows for a perfect recovery of the config by using : config = Configuration.load_config(filename). If no filename is given, overwrites the last save.

Parameters:

filename (Optional[str]) – path to the saving location of the config
save_header (bool) – whether to save the config metadata as the fist parameter. This will tag the saved file as a saved config in the eye of the config system when it gets merged, which will deactivate pre-processing
save_hierarchy (str) – whether to save config hierarchy as a *_hierarchy.yaml file

Raises:

RuntimeError – no file name is provided and there is no previous save to overwrite

Return type:

None

values(deep=False)

Behaves as dict.values(). If deep is False, sub-configs remain sub-configs in the values. Otherwise, they are converted to dict.

Parameters:: deep (bool) – how to return sub-configs that would appear among the values. If False, do not convert them, else recursively convert them to dict
Return type:: ValuesView
Returns:: the values of the config as in dict.values()

yaecs.config.config_getters module

class ConfigGettersMixin(*args, **kwargs)

Bases: object

Getters Mixin class for YAECS configurations.

get(parameter_name, default_value)

Behaves similarly to dict.get(parameter_name, default_value)

Parameters:

parameter_name (str) – parameter to query
default_value (Any) – value to return if the parameter does not exist

Return type:

Any

Returns:

queried value

get_all_linked_sub_configs()

Returns the list of all sub-configs that are directly linked to the root config by a chain of other sub-configs. For this to be the case, all of those sub-configs need to be contained directly in a parameter of another sub-config. For example, a sub-config stored in a list that is a parameter of a sub-config is not linked.

Return type:: List[Configuration]
Returns:: list corresponding to the linked sub-configs

get_all_sub_configs()

Returns the list of all sub-configs, including sub-configs of other sub-configs

Return type:: List[Configuration]
Returns:: list corresponding to the sub-configs

get_command_line_argument(deep=True, do_return_string=False)

Returns a list of command line parameters that can be used in a bash shell to re-create this exact config from the default. Can alternatively return the string itself with do_return_string=True.

Parameters:

deep (bool) – whether to also take the sub-config parameters into account
do_return_string (bool) – whether to return a string (True) or a list of strings (False, default)

Return type:

Union[List[str], str]

Returns:

list or string containing the parameters

get_dict(deep=True)

Returns a dictionary corresponding to the config.

Parameters:: deep (bool) – whether to recursively turn sub-configs into dicts or keep them as sub-configs
Return type:: dict
Returns:: dictionary corresponding to the config

get_main_config()

Getter for the main config corresponding to this config or sub-config. Using this is often hacky.

Return type:: Configuration
Returns:: the main config

get_master_switch(processing_type)

Getter for either the pre- or post-processing master switch depending on processing_type

Parameters:: processing_type (str) – can be ‘pre’ or ‘post’ : processing type to get the master switch for
Return type:: bool
Returns:: the main config

get_modified_buffer()

Getter for the buffer of modified parameters corresponding to this config or sub-config. This gets filled during a creation or merging operation to keep track of all the parameters modified by this operation. Then, it is emptied as all modified parameters get post-processed before the end of the operation.

Return type:: List[str]
Returns:: the buffer of modified elements

get_name()

Returns the name of the config. It is composed of a specified part (or ‘main’ when unspecified) and an indicator of its index in the list of variations of its parent if it is a variation of a config. This indicator is prefixed by ‘_VARIATION_’.

Return type:: str
Returns:: string corresponding to the name

get_nesting_hierarchy()

Returns the nesting hierarchy of the config

Return type:: List[str]
Returns:: list corresponding to the nesting hierarchy

get_parameter_names(deep=True, no_sub_config=False)

Returns the list of the names all parameters in this config. If deep is true, also returns the names of the parameters in the sub-configs using the dot convention.

Parameters:

deep (bool) – whether to also return the names of the parameters in the sub-configs
no_sub_config (bool) – if True, exclude names of sub-configs and only return real parameters

Return type:

List[str]

Returns:

the list of the names of all parameters

get_pre_post_processing_values()

Returns a dictionary containing :

as keys : all the names of the parameters which have been post-processed
as values : the values those parameters had before the post-processing operation

In particular, those values are the ones used when saving the config.

Return type:: Dict[str, Any]
Returns:: dictionary of values before post-processing

get_processed_param_name(full_path=True)

When used in a processing function, returns the full path to that param in the config, or its path in self if full_path is False.

Parameters:: full_path (bool) – if True, returns the path to the param in the main config, otherwise the path to the param in self.
Return type:: str

get_save_file()

If the config was saved previously, returns the path to this save. Otherwise, returns None.

Return type:: Optional[str]
Returns:: the path to the save if it exists, None otherwise

get_reference_folder()

If a reference folder has been registered, returns it. Otherwise, returns None.

Return type:: Optional[str]
Returns:: the reference folder if it exists, None otherwise

get_type_hint(param_name)

Returns the type hint for this param, or 0 if it has no type hint.

Return type:: Union[type, tuple, list, dict, set, int]
Returns:: possible types for this param according to its type hint, 0 if no type hint defined

get_type_hints()

Returns all the type hints for this config

Return type:: Dict[str, Union[type, tuple, list, dict, set, int]]
Returns:: dictionary with keys being parameter names and values being their possible types

get_variation_name()

Returns the variation name of the config

Return type:: str
Returns:: variation name

is_in_operation()

Returns whether the config is currently in a creation or merging process.

Return type:: bool
Returns:: True if the config is in a creation or merging process, False otherwise

yaecs.config.config_hooks module

class ConfigHooksMixin(*args, **kwargs)

Bases: object

Hooks Mixin class for YAECS configurations. Implements processing functions whose name start with register_as_ and are decorated by the yaecs_utils.hook decorator. Users can use those processing either as pre- or post-processing functions, which will have the added effect of tagging the processed parameters as playing a certain pre-defined role in the config

get_processed_param_name: Callable[[bool], str]

get_variation_name: Callable[[], str]

init_from_config: Callable[[Union[str, dict]], None]

add_currently_processed_param_as_hook(hook_name)

Used within _ConfigurationBase._process_parameter to add the param currently being processed as a hook with given name. Instead of using this directly, users should consider decorating their hooking function with the yaecs_utils.hook decorator.

Parameters:: hook_name (str) – name of the hook to add.
Return type:: None

get_experiment_path()

Returns the value of the parameter registered as the experiment path.

Raises:

RuntimeError – when no experiment path has been registered in the config
RuntimeError – when more than one experiment path has been registered in the config

Return type:

str

Returns:

experiment path

get_hook(hook_name=None)

Returns the parameters registered with given hook.

Parameters:: hook_name (Optional[str]) – name of the hook, or None to get the dict of all hooked parameters
Return type:: Union[Dict[str, List[str]], List[str]]
Returns:: list of hooked parameter names

yaecs.config.config_processing_functions module

class ConfigProcessingFunctionsMixin(*args, **kwargs)

Bases: object

Pre- and Post-processing functions Mixin class for YAECS configurations.

get_main_config: Callable[[], Configuration]

get_processed_param_name: Callable[[bool], str]

check_param_in_list(list_of_choices)

Returns a pre-processing function that checks if a param value belongs to a list. Returned function has order OFTEN_FIRST (-10). For example in your pre-processing dict there could be a line such as this : “mode”: self.check_param_in_list([“train”, “val”, “test”, “infer”])

Parameters:: list_of_choices (List[Any]) – list of valid parameter values
Return type:: Callable
Returns:: checking function

copy_param(path_to_copy)

This pre-processing function declares a param as being an exact copy of another param. Its default value must be name of the param it copies. It will be protected against modifications by further config merges (only the param being copied can be modified), but can still pe post-processed further after its copy by post-processing functions with an order above OFTEN_LAST (10). Priority : ALWAYS_LAST (20) YAML tag : copy

Parameters:: path_to_copy (str) – path of param from which to copy the value
Return type:: str
Returns:: path_to_copy

check_number_in_range(minimum=-inf, maximum=inf)

Returns a pre-processing function that checks if a numerical param value is in a range. Returned function has order OFTEN_FIRST (-10). For example in your pre-processing dict there could be a line such as this : “probability”: self.number_in_range(minimum=0, maximum=1)

Parameters:

minimum (Number) – minimal valid value for the parameter
maximum (Number) – maximal valid value for the parameter

Return type:

Callable

Returns:

checking function

protected_param(param)

This pre-processing function declares a param as being protected against modifications. This means that only the default config can change its value, and any attempt to set it from another config will raise an error. Priority : ALWAYS_LAST (20) YAML tag : protected

Parameters:: param (Any) – value of the param to protect
Return type:: Any
Returns:: param

folder_in_experiment(folder)

Returns a post-processing function that extends a path assuming it is located in the experiment path, then the corresponding folder is created. Requires an experiment_folder to have been declared. Priority : OFTEN_LAST (10) YAML tag : sub_folder

Parameters:: folder (Optional[str]) – path of the subfolder within the experiment folder
Return type:: Optional[str]
Returns:: checking function

folder_in_experiment_if(condition_list=None)

Returns a post-processing function that extends a path assuming it is located in the experiment path. Then, if the conditions listed in condition_list ar all met, the corresponding folder is created. Returned function has order OFTEN_LAST (10). condition_list is a list of conditions represented by tuples of length 2 or 3 :

when a condition is represented by a tuple of length 2, the first element should be the path to parameters in the main config and the second element should be a value for those parameters. The condition will return true if the corresponding parameters have the corresponding value ;
when a condition is represented by a tuple of length 3, the behaviour is the same except the third element is a function to apply to the parameters values before they are compared to the second element. For example in your post-processing dict there could be a line such as this : “save_weights_path”: self.folder_in_experiment(condition_list=[(“mode”, “train”)])

Parameters:: condition_list (Optional[List[tuple]]) – list of conditions to be met for the path to be created
Return type:: Callable
Returns:: checking function

yaecs.config.config_setters module

class ConfigSettersMixin(*args, **kwargs)

Bases: object

Setters Mixin class for YAECS configurations.

add_processing_function(param_name, function_to_add, processing_type)

If given parameter does not already have a post-processing function with the same name, adds given function as a post-processing function to parameters with the given name.

Return type:: None

add_processing_function_all(param_name, function_to_add, processing_type)

Triggers add_processing_function on the main config and all defined and future sub-configs.

Parameters:

param_name (str) – parameter(s) to which to add a postprocessing function. Expects paths with respect to the main config.
function_to_add (Union[str, Callable]) – postprocessing function to add, using the generic name “function” if it has no name
processing_type (str) – choose between ‘pre’ to add a pre-processing function or ‘post’ to add a post-processing function

Return type:

None

add_type_hint(name, type_hint)

Adds a type hint for a parameter to the list of type hints for automatic type checks.

Parameters:

name (str) – full path of the param in the main config
type_hint (Union[type, tuple, list, dict, set, int]) – type of the param

Return type:

None

set_sub_config(sub_config)

Registers a new sub-config to the main config.

Parameters:: sub_config (Configuration) – sub-config to register
Return type:: None

remove_value_before_postprocessing(name)

Function used for bookkeeping : it remove a parameter from the pre-post-processing archive.

Parameters:: name (str) – name of the parameter using the dot convention
Return type:: None

remove_type_hint(param_name)

Removes a registered type hint from a param with given name.

Parameters:: param_name (str) – param from which to remove the type hint
Return type:: None

save_value_before_postprocessing(name, value)

Function used for bookkeeping : it saves the value a parameter had before its post-processing.

Parameters:

name (str) – name of the parameter using the dot convention
value (Any) – value of the parameter before post-processing

Return type:

None

set_post_processing(value=True)

Sets the state of the master switch for pre-processing across the entire config object. Calling this for a sub-config will also affect the main config and all other sub-configs.

Parameters:: value (bool) – value to set the pre-processing to
Return type:: None

set_pre_processing(value=True)

Sets the state of the master switch for pre-processing across the entire config object. Calling this for a sub-config will also affect the main config and all other sub-configs.

Parameters:: value (bool) – value to set the pre-processing to
Return type:: None

unset_sub_config(sub_config)

Registers a new sub-config to the main config.

Parameters:: sub_config (Configuration) – sub-config to register
Return type:: None