scatlastb_utils.pipeline.PipelineConfig

scatlastb_utils.pipeline.PipelineConfig#

class scatlastb_utils.pipeline.PipelineConfig(config)#

Class for overall pipeline configuration.

Parameters:

config (dict)

Methods table#

update_file_for_module_param(dataset, ...[, ...])

Update a file path in the config for a specific module parameter.

update_input_files_per_dataset(dataset, ...)

Update input files for a given dataset and module.

Methods#

PipelineConfig.update_file_for_module_param(dataset, module_name, key, subset_dict=None, config_class_map=None, config_kwargs=None)#

Update a file path in the config for a specific module parameter.

This function checks if the file pattern for a module parameter is in the input files of the dataset.

Parameters:

  • dataset (str) – dataset name

  • module_name (str) – module name

  • config – config dict

  • key (str) – key in module that should specify a file name

  • subset_dict (dict (default: None)) – dictionary with subset parameters, e.g. {“dataset”: “dataset_name”}

  • config_class_map (dict[slice(<class ‘str’>, <class ‘scatlastb_utils.pipeline.ModuleConfig.ModuleConfig’>, None)] (default: None)) – mapping of module names to ModuleConfig (or inherited) classes, in cases where there are module-specific config classes

  • config_kwargs (dict[slice(<class ‘str’>, <class ‘dict’>, None)] (default: None)) – kwargs for initializing ModuleConfig classes

PipelineConfig.update_input_files_per_dataset(dataset, module_name, config, first_module=None, config_class_map=None, config_kwargs=None)#

Update input files for a given dataset and module.

Parameters:

  • dataset (str) – dataset name

  • module_name (str) – module name

  • first_module (str (default: None)) – starting module name

  • config_class_map (dict[slice(<class ‘str’>, <class ‘scatlastb_utils.pipeline.ModuleConfig.ModuleConfig’>, None)] (default: None)) – mapping of module names to ModuleConfig (or inherited) classes, in cases where there are module-specific config classes

  • config_kwargs (dict[slice(<class ‘str’>, <class ‘dict’>, None)] (default: None)) – kwargs for initializing ModuleConfig classes

  • config (dict)

Return type:

dict