Commit 0d883320 authored by Médéric Boquien's avatar Médéric Boquien

Pratical implementation of the validation of the parameters. The patch is...

Pratical implementation of the validation of the parameters. The patch is quite long as it has a direct effect on the structure of the configuration dictionary. The validation has the advantage of automatically convert the parameters to the right type. Therefore rather than building a dictionary ourselves, we rather use the ready-made dictionary from ConfigObj. Because the names of the sections are not the same, quite a bit of code had to be adapted. Finally, note that the validation file containing the specification of each variable, pcigale.ini.spec, is created while building the pcigale.ini file. Also because it is needed to convert the data to the right type, one cannot run cigale without a correct validation file.
parent 784a1690
...@@ -6,6 +6,7 @@ ...@@ -6,6 +6,7 @@
- Similarly to the savefluxes module, in the pdf_analysis module if the list of physical properties is left empty, all physical parameters are now analysed. (Médéric Boquien) - Similarly to the savefluxes module, in the pdf_analysis module if the list of physical properties is left empty, all physical parameters are now analysed. (Médéric Boquien)
- It is now possible to pass the parameters of the models to be computed from a file rather than having to indicate them in pcigale.ini. This means that the models do not necessarily need to be computed on a systematic grid of parameters. The name of the file is passed as an argument to the parameters\_file keyword in pcigale.ini. If this is done, the creation\_modules argument is ignored. Finally, the file must be formatted as following: each row is a different model and each column a different parameter. They must follow the naming scheme: module\_name.parameter\_name, that is "bc03.imf" for instance. (Médéric Boquien) - It is now possible to pass the parameters of the models to be computed from a file rather than having to indicate them in pcigale.ini. This means that the models do not necessarily need to be computed on a systematic grid of parameters. The name of the file is passed as an argument to the parameters\_file keyword in pcigale.ini. If this is done, the creation\_modules argument is ignored. Finally, the file must be formatted as following: each row is a different model and each column a different parameter. They must follow the naming scheme: module\_name.parameter\_name, that is "bc03.imf" for instance. (Médéric Boquien)
- Addition of the schreiber2016 SED creation module implementing the Schreiber et al. (2016) dust models. (Laure Ciesla) - Addition of the schreiber2016 SED creation module implementing the Schreiber et al. (2016) dust models. (Laure Ciesla)
- The physical parameters provided in pcigale.ini were not checked at startup against what the modules could accept. This could lead to a runtime crash if an unexpected value was passed to the module. Now the parameters are checked at startup. If an issue is found, it is indicated and the user is asked to fix it before launching cigale again. The validation file is build at the same time as pcigale.ini. (Médéric Boquien)
### Changed ### Changed
- The estimates of the physical parameters from the analysis of the PDF and from the best fit were recorded in separate files. This can be bothersome when trying to compare quantities from different files. Rather, we generate a single file containing all quantities. The ones estimated from the analysis of the PDF are prefixed with "bayes" and the ones from the best fit with "best". (Médéric Boquien) - The estimates of the physical parameters from the analysis of the PDF and from the best fit were recorded in separate files. This can be bothersome when trying to compare quantities from different files. Rather, we generate a single file containing all quantities. The ones estimated from the analysis of the PDF are prefixed with "bayes" and the ones from the best fit with "best". (Médéric Boquien)
......
...@@ -37,16 +37,20 @@ def check(config): ...@@ -37,16 +37,20 @@ def check(config):
# TODO: Check if all the parameters that don't have default values are # TODO: Check if all the parameters that don't have default values are
# given for each module. # given for each module.
configuration = config.configuration configuration = config.configuration
print("With this configuration cigale will compute {} "
"models.".format(ParametersHandler(configuration).size)) if configuration:
print("With this configuration cigale will compute {} "
"models.".format(ParametersHandler(configuration).size))
def run(config): def run(config):
"""Run the analysis. """Run the analysis.
""" """
configuration = config.configuration configuration = config.configuration
analysis_module = get_module(configuration['analysis_method'])
analysis_module.process(configuration) if configuration:
analysis_module = get_module(configuration['analysis_method'])
analysis_module.process(configuration)
def main(): def main():
......
...@@ -71,7 +71,7 @@ class AnalysisModule(object): ...@@ -71,7 +71,7 @@ class AnalysisModule(object):
KeyError: when not all the needed parameters are given. KeyError: when not all the needed parameters are given.
""" """
parameters = configuration['analysis_method_params'] parameters = configuration['analysis_params']
# For parameters that are present on the parameter_list with a default # For parameters that are present on the parameter_list with a default
# value and that are not in the parameters dictionary, we add them # value and that are not in the parameters dictionary, we add them
# with their default value. # with their default value.
......
...@@ -66,13 +66,13 @@ class PdfAnalysis(AnalysisModule): ...@@ -66,13 +66,13 @@ class PdfAnalysis(AnalysisModule):
False False
)), )),
("save_chi2", ( ("save_chi2", (
"boolean{}", "boolean()",
"If true, for each observation and each analysed variable save " "If true, for each observation and each analysed variable save "
"the reduced chi2.", "the reduced chi2.",
False False
)), )),
("save_pdf", ( ("save_pdf", (
"boolean{}", "boolean()",
"If true, for each observation and each analysed variable save " "If true, for each observation and each analysed variable save "
"the probability density function.", "the probability density function.",
False False
...@@ -115,16 +115,16 @@ class PdfAnalysis(AnalysisModule): ...@@ -115,16 +115,16 @@ class PdfAnalysis(AnalysisModule):
# Initalise variables from input arguments. # Initalise variables from input arguments.
creation_modules = conf['creation_modules'] creation_modules = conf['creation_modules']
creation_modules_params = conf['creation_modules_params'] creation_modules_params = conf['sed_modules_params']
analysed_variables = conf['analysis_method_params']["analysed_variables"] analysed_variables = conf['analysis_params']["analysed_variables"]
analysed_variables_nolog = [variable[:-4] if variable.endswith('_log') analysed_variables_nolog = [variable[:-4] if variable.endswith('_log')
else variable for variable in else variable for variable in
analysed_variables] analysed_variables]
n_variables = len(analysed_variables) n_variables = len(analysed_variables)
save = {key: conf['analysis_method_params']["save_{}".format(key)].lower() == "true" save = {key: conf['analysis_params']["save_{}".format(key)] for key in
for key in ["best_sed", "chi2", "pdf"]} ["best_sed", "chi2", "pdf"]}
lim_flag = conf['analysis_method_params']["lim_flag"].lower() == "true" lim_flag = conf['analysis_params']["lim_flag"]
mock_flag = conf['analysis_method_params']["mock_flag"].lower() == "true" mock_flag = conf['analysis_params']["mock_flag"]
filters = [name for name in conf['column_list'] if not filters = [name for name in conf['column_list'] if not
name.endswith('_err')] name.endswith('_err')]
...@@ -137,8 +137,7 @@ class PdfAnalysis(AnalysisModule): ...@@ -137,8 +137,7 @@ class PdfAnalysis(AnalysisModule):
lim_flag) lim_flag)
n_obs = len(obs_table) n_obs = len(obs_table)
w_redshifting = creation_modules.index('redshifting') z = np.array(creation_modules_params['redshifting']['redshift'])
z = np.array(creation_modules_params[w_redshifting]['redshift'])
# The parameters handler allows us to retrieve the models parameters # The parameters handler allows us to retrieve the models parameters
# from a 1D index. This is useful in that we do not have to create # from a 1D index. This is useful in that we do not have to create
......
...@@ -81,9 +81,9 @@ class SaveFluxes(AnalysisModule): ...@@ -81,9 +81,9 @@ class SaveFluxes(AnalysisModule):
# Rename the output directory if it exists # Rename the output directory if it exists
backup_dir() backup_dir()
out_file = conf['analysis_method_params']['output_file'] out_file = conf['analysis_params']['output_file']
out_format = conf['analysis_method_params']['output_format'] out_format = conf['analysis_params']['output_format']
save_sed = conf['analysis_method_params']['save_sed'].lower() == "true" save_sed = conf['analysis_params']['save_sed']
filters = [name for name in conf['column_list'] if not filters = [name for name in conf['column_list'] if not
name.endswith('_err')] name.endswith('_err')]
...@@ -97,7 +97,7 @@ class SaveFluxes(AnalysisModule): ...@@ -97,7 +97,7 @@ class SaveFluxes(AnalysisModule):
params = ParametersHandler(conf) params = ParametersHandler(conf)
n_params = params.size n_params = params.size
info = conf['analysis_method_params']['variables'] info = conf['analysis_params']['variables']
n_info = len(info) n_info = len(info)
model_fluxes = (RawArray(ctypes.c_double, n_params * n_filters), model_fluxes = (RawArray(ctypes.c_double, n_params * n_filters),
......
...@@ -33,6 +33,7 @@ def backup_dir(directory=OUT_DIR): ...@@ -33,6 +33,7 @@ def backup_dir(directory=OUT_DIR):
)) ))
os.mkdir(directory) os.mkdir(directory)
shutil.copyfile('pcigale.ini', directory + 'pcigale.ini') shutil.copyfile('pcigale.ini', directory + 'pcigale.ini')
shutil.copyfile('pcigale.ini.spec', directory + 'pcigale.ini.spec')
def save_fluxes(model_fluxes, model_parameters, filters, names, filename, def save_fluxes(model_fluxes, model_parameters, filters, names, filename,
......
...@@ -46,8 +46,8 @@ class ParametersHandlerGrid(object): ...@@ -46,8 +46,8 @@ class ParametersHandlerGrid(object):
""" """
self.modules = configuration['creation_modules'] self.modules = configuration['creation_modules']
self.parameters = [self._param_dict_combine(dictionary) for dictionary self.parameters = [self._param_dict_combine(configuration['sed_modules_params'][module])
in configuration['creation_modules_params']] for module in self.modules]
self.shape = tuple(len(parameter) for parameter in self.parameters) self.shape = tuple(len(parameter) for parameter in self.parameters)
self.size = int(np.product(self.shape)) self.size = int(np.product(self.shape))
......
...@@ -12,6 +12,7 @@ import configobj ...@@ -12,6 +12,7 @@ import configobj
from glob import glob # To allow the use of glob() in "eval..." from glob import glob # To allow the use of glob() in "eval..."
import pkg_resources import pkg_resources
import numpy as np import numpy as np
import validate
from ..handlers.parameters_handler import ParametersHandler from ..handlers.parameters_handler import ParametersHandler
from ..data import Database from ..data import Database
...@@ -19,64 +20,13 @@ from ..utils import read_table ...@@ -19,64 +20,13 @@ from ..utils import read_table
from .. import creation_modules from .. import creation_modules
from .. import analysis_modules from .. import analysis_modules
from ..warehouse import SedWarehouse from ..warehouse import SedWarehouse
from . import validation
# Limit the redshift to this number of decimals # Limit the redshift to this number of decimals
REDSHIFT_DECIMALS = 2 REDSHIFT_DECIMALS = 2
def evaluate_description(description):
"""Evaluate a description from the config file as a list.
The description is read from the config file by configobj that transforms
coma separated value in a list. From this description, this function try
to evaluate the desired list of values:
- If the description is a string beginning with 'eval ', then its content
(without 'eval ') is evaluated as Python code and its result returned.
An array is expected.
- If the description is a string beginning by 'range', the start, step and
stop values are then expected and the range is evaluated (stop included
if reached.
- Then the function tries to evaluate the description as a Numpy array of
float and returns the mere list if this fails.
Parameters
----------
description: string or list
The description to be evaluated.
Returns
-------
results: list
The evaluated list of values.
"""
results = description
if type(description) == str:
if description.startswith('eval '):
results = eval(description[4:])
# If the evaluation lead to a single value, we put it in a list.
if not isinstance(results, Iterable):
results = [results]
elif description.startswith('range '):
start, stop, step = [float(item) for item
in description[5:].split()]
results = np.arange(start, stop+step, step)
else:
# We need to return a list to combine the list of possible values
# for each parameter.
results = [results]
# We prefer to evaluate the parameter as a numpy array of floats if
# possible.
try:
results = np.array(results, float)
except ValueError:
pass
return results
class Configuration(object): class Configuration(object):
"""This class manages the configuration of pcigale. """This class manages the configuration of pcigale.
""" """
...@@ -90,10 +40,22 @@ class Configuration(object): ...@@ -90,10 +40,22 @@ class Configuration(object):
Name of the configuration file (pcigale.conf by default). Name of the configuration file (pcigale.conf by default).
""" """
self.spec = configobj.ConfigObj(filename+'.spec',
write_empty_values=True,
indent_type=' ',
encoding='UTF8',
list_values=False,
_inspec=True)
self.config = configobj.ConfigObj(filename, self.config = configobj.ConfigObj(filename,
write_empty_values=True, write_empty_values=True,
indent_type=' ', indent_type=' ',
encoding='UTF8') encoding='UTF8',
configspec=self.spec)
# We validate the configuration so that the variables are converted to
# the expected that. We do not handle errors at the point but only when
# we actually return the configuration file from the property() method.
self.config.validate(validate.Validator(validation.functions))
def create_blank_conf(self): def create_blank_conf(self):
"""Create the initial configuration file """Create the initial configuration file
...@@ -112,6 +74,7 @@ class Configuration(object): ...@@ -112,6 +74,7 @@ class Configuration(object):
"'_err' suffix for the uncertainties. The fluxes and the " "'_err' suffix for the uncertainties. The fluxes and the "
"uncertainties must be in mJy. This file is optional to generate " "uncertainties must be in mJy. This file is optional to generate "
"the configuration file, in particular for the savefluxes module.") "the configuration file, in particular for the savefluxes module.")
self.spec['data_file'] = "string"
self.config['parameters_file'] = "" self.config['parameters_file'] = ""
self.config.comments['parameters_file'] = [""] + wrap( self.config.comments['parameters_file'] = [""] + wrap(
...@@ -122,6 +85,7 @@ class Configuration(object): ...@@ -122,6 +85,7 @@ class Configuration(object):
"one. Finally, if this parameters is not left empty, cigale will " "one. Finally, if this parameters is not left empty, cigale will "
"not interpret the configuration parameters given in pcigale.ini. " "not interpret the configuration parameters given in pcigale.ini. "
"They will be given only for information.") "They will be given only for information.")
self.spec['parameters_file'] = "string()"
self.config['creation_modules'] = [] self.config['creation_modules'] = []
self.config.comments['creation_modules'] = ([""] + self.config.comments['creation_modules'] = ([""] +
...@@ -134,18 +98,22 @@ class Configuration(object): ...@@ -134,18 +98,22 @@ class Configuration(object):
["AGN: dale2014, fritz2006"] + ["AGN: dale2014, fritz2006"] +
["Radio: radio"] + ["Radio: radio"] +
["Redshift: redshifting (mandatory!)"]) ["Redshift: redshifting (mandatory!)"])
self.spec['creation_modules'] = "cigale_string_list()"
self.config['analysis_method'] = "" self.config['analysis_method'] = ""
self.config.comments['analysis_method'] = [""] + wrap( self.config.comments['analysis_method'] = [""] + wrap(
"Method used for statistical analysis. Available methods: " "Method used for statistical analysis. Available methods: "
"pdf_analysis, savefluxes.") "pdf_analysis, savefluxes.")
self.spec['analysis_method'] = "string()"
self.config['cores'] = "" self.config['cores'] = ""
self.config.comments['cores'] = [""] + wrap( self.config.comments['cores'] = [""] + wrap(
"Number of CPU cores available. This computer has {} cores." "Number of CPU cores available. This computer has {} cores."
.format(mp.cpu_count())) .format(mp.cpu_count()))
self.spec['cores'] = "integer(min=1)"
self.config.write() self.config.write()
self.spec.write()
def generate_conf(self): def generate_conf(self):
"""Generate the full configuration file """Generate the full configuration file
...@@ -192,16 +160,20 @@ class Configuration(object): ...@@ -192,16 +160,20 @@ class Configuration(object):
self.config.comments['column_list'] = [""] + wrap( self.config.comments['column_list'] = [""] + wrap(
"List of the columns in the observation data file to use for " "List of the columns in the observation data file to use for "
"the fitting.") "the fitting.")
self.spec['column_list'] = "cigale_string_list()"
# SED creation modules configurations. For each module, we generate # SED creation modules configurations. For each module, we generate
# the configuration section from its parameter list. # the configuration section from its parameter list.
self.config['sed_creation_modules'] = {} self.config['sed_modules_params'] = {}
self.config.comments['sed_creation_modules'] = ["", ""] + wrap( self.config.comments['sed_modules_params'] = ["", ""] + wrap(
"Configuration of the SED creation modules.") "Configuration of the SED creation modules.")
self.spec['sed_modules_params'] = {}
for module_name in self.config['creation_modules']: for module_name in self.config['creation_modules']:
self.config["sed_creation_modules"][module_name] = {} self.config['sed_modules_params'][module_name] = {}
sub_config = self.config["sed_creation_modules"][module_name] self.spec['sed_modules_params'][module_name] = {}
sub_config = self.config['sed_modules_params'][module_name]
sub_spec = self.spec['sed_modules_params'][module_name]
for name, (typ, description, default) in \ for name, (typ, description, default) in \
creation_modules.get_module( creation_modules.get_module(
...@@ -211,93 +183,62 @@ class Configuration(object): ...@@ -211,93 +183,62 @@ class Configuration(object):
default = '' default = ''
sub_config[name] = default sub_config[name] = default
sub_config.comments[name] = wrap(description) sub_config.comments[name] = wrap(description)
sub_spec[name] = typ
self.config['sed_creation_modules'].comments[module_name] = [ self.config['sed_modules_params'].comments[module_name] = [
creation_modules.get_module(module_name, blank=True).comments] creation_modules.get_module(module_name, blank=True).comments]
self.check_modules() self.check_modules()
# Configuration for the analysis method # Configuration for the analysis method
self.config['analysis_configuration'] = {} self.config['analysis_params'] = {}
self.config.comments['analysis_configuration'] = ["", ""] + wrap( self.config.comments['analysis_params'] = ["", ""] + wrap(
"Configuration of the statistical analysis method.") "Configuration of the statistical analysis method.")
self.spec['analysis_params'] = {}
module_name = self.config['analysis_method'] module_name = self.config['analysis_method']
for name, (typ, desc, default) in \ for name, (typ, desc, default) in \
analysis_modules.get_module(module_name).parameter_list.items(): analysis_modules.get_module(module_name).parameter_list.items():
if default is None: if default is None:
default = '' default = ''
self.config['analysis_configuration'][name] = default self.config['analysis_params'][name] = default
self.config['analysis_configuration'].comments[name] = wrap(desc) self.config['analysis_params'].comments[name] = wrap(desc)
self.spec['analysis_params'][name] = typ
self.config.write() self.config.write()
self.spec.write()
@property @property
def configuration(self): def configuration(self):
"""Returns a dictionary for the session configuration. """Returns a dictionary for the session configuration if it is valid.
Otherwise, print the erroneous keys.
Returns Returns
------- -------
configuration['data_file']: string configuration: dictionary
File containing the observations to fit. Dictionary containing the information provided in pcigale.ini.
configuration['column_list']: list of strings
List of the columns of data_file to use in the fitting.
configuration['creation_modules']: list of strings
List of the modules (in the right order) used to create the SEDs.
configuration['creation_modules_params']: list of dictionaries
Configuration parameters for each module. To each parameter, the
dictionary associates a list of possible values (possibly only
one).
configuration['analysis_method']: string
Statistical analysis module used to fit the data.
configuration['analysis_method_params']: dictionary
Parameters for the statistical analysis module. To each parameter
is associated a list of possible values.
""" """
configuration = {} self.complete_redshifts()
self.complete_analysed_parameters()
# Before building the configuration dictionary, we ensure that all the
# fields are filled vdt = validate.Validator(validation.functions)
if not self.config['parameters_file']: validity = self.config.validate(vdt, preserve_errors=True)
self.complete_redshifts()
if validity is not True:
for section in ['data_file', 'parameters_file', 'column_list', print("The following issues have been found in pcigale.ini:")
'creation_modules', 'analysis_method']: for module, param, message in configobj.flatten_errors(self.config,
configuration[section] = self.config[section] validity):
configuration['cores'] = int(self.config['cores']) if len(module) > 0:
print("Module {}, parameter {}: {}".format('/'.join(module),
# Parsing the SED modules parameters param, message))
configuration['creation_modules_params'] = [] else:
for module in self.config['creation_modules']: print("Parameter {}: {}".format(param, message))
module_params = {} print("Run the same command after having fixed pcigale.ini. If you"
for key, value in \ " want to disable error checking, simply remove the "
self.config['sed_creation_modules'][module].items(): "pcigale.ini.spec file.")
module_params[key] = evaluate_description(value)
configuration['creation_modules_params'].append(module_params)
if (self.config['analysis_method'] == 'savefluxes' and
not self.config['analysis_configuration']['variables']):
warehouse = SedWarehouse()
params = ParametersHandler(configuration)
sed = warehouse.get_sed(params.modules,
params.from_index(0))
info = list(sed.info.keys())
info.sort()
self.config['analysis_configuration']['variables'] = info
elif (self.config['analysis_method'] == 'pdf_analysis' and
not self.config['analysis_configuration']['analysed_variables']):
warehouse = SedWarehouse()
params = ParametersHandler(configuration)
sed = warehouse.get_sed(params.modules,
params.from_index(0))
info = list(sed.info.keys())
info.sort()
self.config['analysis_configuration']['analysed_variables'] = info
# Analysis method parameters return None
configuration['analysis_method_params'] = \
self.config['analysis_configuration']
return configuration return self.config.dict()
def check_modules(self): def check_modules(self):
"""Make a basic check to ensure that some required modules are present. """Make a basic check to ensure that some required modules are present.
...@@ -340,13 +281,31 @@ class Configuration(object): ...@@ -340,13 +281,31 @@ class Configuration(object):
configuration file and must be extracted from the input flux file. configuration file and must be extracted from the input flux file.
""" """
z_mod = self.config['sed_creation_modules']['redshifting']['redshift'] z_mod = self.config['sed_modules_params']['redshifting']['redshift']
if type(z_mod) is str and not z_mod: if type(z_mod) is str and not z_mod:
if self.config['data_file']: if self.config['data_file']:
obs_table = read_table(self.config['data_file']) obs_table = read_table(self.config['data_file'])
z = np.unique(np.around(obs_table['redshift'], z = list(np.unique(np.around(obs_table['redshift'],
decimals=REDSHIFT_DECIMALS)) decimals=REDSHIFT_DECIMALS)))
self.config['sed_creation_modules']['redshifting']['redshift'] = z self.config['sed_modules_params']['redshifting']['redshift'] = z
else: else:
raise Exception("No flux file and no redshift indicated. " raise Exception("No flux file and no redshift indicated. "
"The spectra cannot be computed. Aborting.") "The spectra cannot be computed. Aborting.")
def complete_analysed_parameters(self):
"""Complete the configuration when the variables are missing from the
configuration file and must be extract from a dummy run."""
if self.config['analysis_method'] == 'savefluxes':
name = 'variables'
elif self.config['analysis_method'] == 'pdf_analysis':
name = 'analysed_variables'
else:
raise Exception("Unknown analysis method")
if not self.config['analysis_params'][name]:
warehouse = SedWarehouse()
params = ParametersHandler(self.config.dict())
sed = warehouse.get_sed(params.modules, params.from_index(0))
info = list(sed.info.keys())
info.sort()
self.config['analysis_params'][name] = info
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment