Commit 58e65f0e authored by Médéric Boquien's avatar Médéric Boquien

Merge branch 'release/v0.9.0'

parents b8a59999 c502664a
# Change Log
## 0.9.0 (2016-04-04)
### Added
- When using the `savefluxes` module, all the output parameters were saved. This is not efficient when the user is only interested in some of the output parameters but not all. We introduce the "variables" configuration parameter for `savefluxes` to list the output parameters the user wants to save. If the list is left empty, all parameters are saved, preserving the current behaviour. This should increase the speed substantially when saving memory. (Médéric Boquien)
- Similarly to the `savefluxes` module, in the `pdf_analysis` module if the list of physical properties is left empty, all physical parameters are now analysed. (Médéric Boquien)
- It is now possible to pass the parameters of the models to be computed from a file rather than having to indicate them in pcigale.ini. This means that the models do not necessarily need to be computed on a systematic grid of parameters. The name of the file is passed as an argument to the parameters\_file keyword in pcigale.ini. If this is done, the creation\_modules argument is ignored. Finally, the file must be formatted as following: each row is a different model and each column a different parameter. They must follow the naming scheme: module\_name.parameter\_name, that is "bc03.imf" for instance. (Médéric Boquien)
- Addition of the `schreiber2016` SED creation module implementing the Schreiber et al. (2016) dust models. (Laure Ciesla)
- The physical parameters provided in pcigale.ini were not checked at startup against what the modules could accept. This could lead to a runtime crash if an unexpected value was passed to the module. Now the parameters are checked at startup. If an issue is found, it is indicated and the user is asked to fix it before launching cigale again. The validation file is build at the same time as pcigale.ini. (Médéric Boquien)
- There is a new attenuation module (`dustatt_2powerlaws`) based on a double power law: a birth cloud power law applied only to the young star population and an ISM power law applied to both the young and the old star population. (Yannick Roehlly & Véronique Buat)
### Changed
- The estimates of the physical parameters from the analysis of the PDF and from the best fit were recorded in separate files. This can be bothersome when trying to compare quantities from different files. Rather, we generate a single file containing all quantities. The ones estimated from the analysis of the PDF are prefixed with "bayes" and the ones from the best fit with "best". (Médéric Boquien)
- To homogenize input and output files, the "observation\_id" has been changed to "id" in the output files. (Médéric Boquien)
- The output files providing estimates of the physical properties are now generated both in the form of text and FITS files. (Médéric Boquien)
- When using the `dustatt_calzleit module`, choosing ẟ≠0 leads to an effective E(B-V) different from the one set by the user. Now the E(B-V) will always correspond to the one specified by the user. This means that at fixed E(B-V), A(V) depends on ẟ. (Médéric Boquien)
- The pcigale-mock tool has been merged into pcigale-plots; the mock plots can be obtained with the "mock" command.
- The `sfhdelayed` module is now initialised with _init_code() to be consistent with the way things are done in other modules. This should give a slight speedup under some sircumstances too. (Médéric Boquien)
- In `sfhfromfile`, the specification of the time grid was vague and therefore could lead to incorrect results if it was not properly formatted by the end user. The description has been clarified and we now check that the time starts from 0 and that the time step is always 1 Myr. If it is not the case we raise an exception. (Médéric Boquien)
- When the redshift is not indicated in pcigale.ini, the analysis module fills the list of redshifts from the redshifts indicated in the input flux file. This is inefficient as analysis modules should not have to modify the configuration. Now this is done when interpreting pcigale.ini before calling the relevant analysis module. As a side effect, "pigale check" now returns the total number of models that cigale will compute rather than the number of models per redshift bin. (Médéric Boquien)
- The optionally saved spectra in the `pdf_analysis` and `savefluxes` modules were saved in the VO-table format. The most important downside is that it is very slow to write to, which proves to be a major bottleneck in the computing speed. To overcome this issue, we rather save the spectra using the FITS formation. Instead of having one file containing the spectra (including the various components) and the SFH in a single file, now we have one file for the spectra and one file for the SFH.
### Fixed
- To estimate parameters in log, pcigale determines which variables end with the "\_log" string and removed it to compute the models. However in some circumstances, it was overzealous. This has been fixed. (Médéric Boquien)
- When estimating a parameter in log, these were not scaled appropriately and taken in log when saving the related χ² and PDF. (Médéric Boquien)
- In the presence of upper limits, correct the scaling factor of the models to the observations before computing the χ², not after. (Médéric Boquien)
- When called without arguments, pcigale-plots would crash and display the backtrace. Now it displays the a short help showing how to use it. (Médéric Boquien)
- For `sfh2exp`, when setting the scale of the SFH with sfr0, the normalisation was incorrect by a factor exp(-1/tau\_main). (Médéric Boquien)
- The mass-dependent physical properties are computed assuming the redshift of the model. However because we round the observed redshifts to two decimals, there can be a difference of 0.005 in redshift between the models and the actual observation if CIGALE computes the list of redshifts itself. At low redshift, this can cause a discrepancy in the mass-dependent physical properties: ~0.35 dex at z=0.010 vs 0.015 for instance. Therefore we now evaluate these physical quantities at the observed redshift at full precision. (Médéric Boquien, issue reported by Samir Salim)
- In the `sfhfromfile` module, an extraneous offset in the column index made that it took the previous column as the SFR rather than the selected column. (Médéric Boquien)
- In `sfhfromfile`, if the SFR is made of integers cigale crashed. Now we systematically convert it to float. (Médéric Boquien)
- The order of the parameters for the analysis modules would change each time a new pcigale.ini was generated. Now the order is fixed. (Médéric Boquien)
- In the output the sfh.age parameter would correspond to the input value minus 1. Now both values are consistent with one another. (Laure Ciesla & Médéric Boquien)
- In rare circumstances requiring a specific distribution of redshifts the integration of the spectrum in some filters was not done correctly, inducing relative errors of ~10¯⁵-10¯⁶. (Médéric Boquien)
- The absorption of the Lyman continuum from old stars tended to be overestimated leading to some “negative fluxes” for the Lyman continuum. (Médéric Boquien)
### Optimised
- Prior to version 0.7.0, we needed to maintain the list of redshifts for all the computed models. Past 0.7.0 we just infer the redshift from a list unique redshifts. This means that we can now discard the list of redshifts for all the models and only keep the list of unique redshifts. This saves ~8 MB of memory for every 10⁶ models. the models should be computed slightly faster but it is in the measurement noise. (Médéric Boquien)
- The `sfhfromfile` module is now fully initialised when it is instantiated rather than doing so when processing the SED. This should be especially sensitive when processing different SED. (Médéric Boquien)
- We do not store the time grid in the SED anymore given that we assume it starts at 0 Myr with steps of 1 Myr, we can easily reconstruct to save it if needed. It should save a little bit of memory and it should go a little bit faster. (Médéric Boquien)
- To compute the stellar spectrum of the young component, do not pass the full SFH with the old part set to 0. Rather, only pass the corresponding part of the SFH. This nearly doubles the computing speed of the stellar spectrum (Médéric Boquien)
- Computers are much better at multiplying than at dividing. Therefore to correct the spectral emission when redshifting we multiply by 1/(1+z) rather than dividing by 1+z. (Médéric Boquien)
---
## 0.8.1 (2015-12-07)
### Fixed
- To estimate parameters in log, pcigale determines which variables end with the "_log" string and removed it to compute the models. However in some circumstances, it was overzealous. This has been fixed. (Médéric Boquien)
- To estimate parameters in log, pcigale determines which variables end with the "\_log" string and removed it to compute the models. However in some circumstances, it was overzealous. This has been fixed. (Médéric Boquien)
## 0.8.0 (2015-11-01)
## 0.8.0 (2015-12-01)
### Added
- The evaluation of the parameters is always done linearly. This can be a problem when estimating the SFR or the stellar mass for instance as it is usual to estimate their log rather. Because the log is non-linear, the likelihood-weigthed mean of the log is not the log of the likelihood-weighted mean. Therefore the estimation of the log of these parameters has to be done during the analysis step. This is now possible. The variables to be analysed in log just need to be indicated with the suffix "_log", for instance "stellar.m_star_log". (Médéric Boquien, idea suggested by Samir Salim)
- The evaluation of the parameters is always done linearly. This can be a problem when estimating the SFR or the stellar mass for instance as it is usual to estimate their log rather. Because the log is non-linear, the likelihood-weighted mean of the log is not the log of the likelihood-weighted mean. Therefore the estimation of the log of these parameters has to be done during the analysis step. This is now possible. The variables to be analysed in log just need to be indicated with the suffix "_log", for instance "stellar.m\_star\_log". (Médéric Boquien, idea suggested by Samir Salim)
### Fixed
- Running the scripts in parallel trigger a deadlock on OS X with python 3.5. A workaround has been implemented. (Médéric Boquien)
......@@ -16,21 +59,23 @@
### Changed
- The attenuation.ebvs\_main and attenuation.ebvs\_old parameters are no longer present as they were duplicates of attenuation.E\_BVs.stellar.old and attenuation.E\_BVs.stellar.young (that are still available).
---
## 0.7.0 (2015-11-19)
### Added
- The pcigale-mock utility has been added to generate plots comparing the exact and pcigale-estimated parameters. This requires pcigale to be run beforehand with the pdf_analysis module and the mock_flag option set to True. (Denis Burgarella and Médéric Boquien)
- The pcigale-mock utility has been added to generate plots comparing the exact and pcigale-estimated parameters. This requires pcigale to be run beforehand with the `pdf_analysis` module and the mock\_flag option set to True. (Denis Burgarella and Médéric Boquien)
- The pcigale-filter utility has been added to easily list, plot, add, and remove filters without having the rebuild the database entirely. (Médéric Boquien)
- It is now possible to analyse the flux in a band as a regular parameter. It can be useful for flux predictions. (Yannick Roehlly)
- The redshift can be a now used as a free parameter, enabling pcigale to estimate the photometric redshift. (Médéric Boquien)
- When running "pcigale genconf", the list of modules is automatically checked against the list of official modules. If modules are missing, information is printed on the screen indicating the level of severity (information, warning, or error) and the list of modules that can be used. (Médéric Boquien)
### Changed
- The galaxy_mass parameter was very ambiguous. In reality it corresponds to the integral of the SFH. Consequently it has been renamed sfh.integrated. (Médéric Boquien)
- The galaxy\_mass parameter was very ambiguous. In reality it corresponds to the integral of the SFH. Consequently it has been renamed sfh.integrated. (Médéric Boquien)
- In the Calzetti attenuation module, add a warning saying that is the power law slope is different than 0, E(B-V) will no longer be the real one. (Yannick Roehlly)
- Add "B_B90" to the list of computed attenuation so that users can calculate the effective E(B-V). (Yannick Roehlly)
- Add "B\_B90" to the list of computed attenuation so that users can calculate the effective E(B-V). (Yannick Roehlly)
- Computing the parameters and their uncertainties through the histogram of the PDF is slow and can introduce biases in some cases. Rather, now the estimated values of the parameters and the corresponding uncertainties are simply computed from the weighted mean and standard deviation of the models that are at least 0.1% as likely as the best model to reproduce the observations. The differences in the estimates are very small except when very few models are used. (Médéric Boquien)
- Magic values to indicate invalid values (e.g. values lower than -99) are difficult to handle safely. They have been replaced with NaN wherever appropriate. The logic of the input flux file stays the same for the time being but the magic values are converted internally after reading it. Users are advised to replace magic values with NaN. The output files now use NaN instead of magic number to indicate invalid values. (Médéric Boquien)
- Rename the the AGN faction added by dale2014 module from agn.fracAGN to agn.fracAGN_dale2014 to avoid conflict with fritz2006 module. (Yannick Roehlly)
- Rename the the AGN faction added by dale2014 module from agn.fracAGN to agn.fracAGN\_dale2014 to avoid conflict with fritz2006 module. (Yannick Roehlly)
- Remove the radio component from the dale2014 model so that it can be used with the more flexible radio module, courtesy Daniel Dale. (Laure Ciesla and Médéric Boquien)
### Fixed
......@@ -56,8 +101,9 @@
- Each core copied the subset of models corresponding to the redshift of the object to be analysed. This is a problem as it can strongly increase memory usage with the number of cores, especially when there are many models and just one redshift. Rather than making a copy, we use a view, which not only saves a considerable amount of memory but is also faster as there is no need to allocate new, large arrays. This is made possible as models are regularly ordered with redshift. (Médéric Boquien)
- Various minor optimisations. (Médéric Boquien)
## 0.6.0 (2015-09-07)
---
## 0.6.0 (2015-09-07)
### Added
- New module to compute a star formation history as described in Buat et al. 2008. (Yannick Roehlly)
- New module to compute a periodic SFH. Each star formation episode can be exponential, "delayed", or rectangular. (Médéric Boquien and Denis Burgarella)
......@@ -68,13 +114,13 @@
- The dust masses are now computed for the draine2007 and draine2014 modules. (Médéric Boquien)
### Changed
- The nebular_lines_width parameter of the nebular module is now called lines_width as the nebular prefix was redundant. (Médéric Boquien)
- The nebular\_lines\_width parameter of the nebular module is now called lines\_width as the nebular prefix was redundant. (Médéric Boquien)
- Prefix the ouput variables of SFH-related modules with "sfh" to facilitate their identification in the output files. (Médéric Boquien)
- Prefix the output variables of the fritz2006 AGN module with "agn" to facilitate their identification in the output files. (Médéric Boquien)
- Prefix the redshift with "universe". (Médéric Boquien)
- With pcigale-plot, draw the spectra only to λ=50 cm as the models do not extend much further and there is very rarely any observation beyond λ=21 cm. (Médéric Boquien)
- As pcigale is getting much faster, display the number of computed models every 250 models rather than every 100 models. (Médéric Boquien)
- Give default values for the dl2014, sfh_buat, and sfhdelayed modules to allow for quick test runs. (Médéric Boquien)
- Give default values for the dl2014, sfh\_buat, and sfhdelayed modules to allow for quick test runs. (Médéric Boquien)
- Now pcigale-plots plots errors on upper limits. (Denis Burgarella)
### Fixed
......@@ -84,7 +130,7 @@
- Improve the readability of the pcigale-plots generated spectra by drawing the observed fluxes on top of other lines. (Médéric Boquien)
- The nebular components are now plotted with pcigale-plots. (Médéric Boquien)
- When a filter that was not in the database was called, pcigale would crash ungracefully as the exception invoked does not exist in Python 3.x. Now use a Python 3.x exception. (Médéric Boquien)
- The dustatt_powerlaw module could not identify which physical components to attenuate when the nebular module was called. (Médéric Boquien)
- The dustatt\_powerlaw module could not identify which physical components to attenuate when the nebular module was called. (Médéric Boquien)
- The displayed counter for the number of objects already analysed could be slightly offset from the number of models actually computed. (Médéric Boquien)
- Change the method to compute the χ² in the presence of upper limits as the previous method did not always converge. (Denis Burgarella and Médéric Boquien)
......@@ -100,25 +146,34 @@
- The output parameters of a model were stored in an ordered dictionary. While convenient to keep the order of insertion it is very slow as it is implemented in pure Python for versions up to 3.4. Rather we use a regular dictionary and we reorder the parameters alphabetically. (Médéric Boquien)
- To store the SED in memory and retrieve them later, we index them with the list of parameters used to compute them. We serialise those using JSON. However JSON is slow. As these data are purely internal, rather use marshal, which is much faster than JSON. (Médéric Boquien)
## 0.5.1 (2015-04-28)
---
## 0.5.1 (2015-04-28)
### Changed
- Set the default dale2014 AGN fraction to 0 to avoid the accidentl inclusion of AGN. (Denis Burgarella)
- Set the default dale2014 AGN fraction to 0 to avoid the accidental inclusion of AGN. (Denis Burgarella)
- Modify the name of the averaged SFR: two averaged SFRs over 10 (sfh.sfr10Myrs) and 100Myrs (sfh.sfr100Myrs). (Denis Burgarella)
- Improve the documentation of the savefluxes module. (Denis Burgarella)
- Improve the documentation of the `savefluxes` module. (Denis Burgarella)
### Fixed
- Correction of the x-axis limits. (Denis Burgarella)
- Fix the detection of the presence of the agn.fritz2006_therm in pcigale-plots. (Denis Burgarella)
- Fix the detection of the presence of the agn.fritz2006\_therm in pcigale-plots. (Denis Burgarella)
- Correct the wavelength in the SCUBA 450 μm filter. (Denis Burgarella)
- Install the ancillary data required to make plots. (Yannick Roehlly)
## 0.5.0 (2015-04-02)
---
## 0.4.0 (2014-10-09)
---
## 0.3.0 (2014-07-06)
---
## 0.2.0 (2014-06-10)
---
## 0.1.0 (2014-05-26)
......@@ -19,9 +19,10 @@ import itertools
import numpy as np
from scipy import interpolate
import scipy.constants as cst
from astropy.table import Table
from pcigale.data import (Database, Filter, M2005, BC03, Fritz2006,
Dale2014, DL2007, DL2014, NebularLines,
NebularContinuum)
NebularContinuum, Schreiber2016)
def read_bc03_ssp(filename):
......@@ -664,6 +665,34 @@ def build_nebular(base):
base.add_nebular_continuum(models_cont)
base.add_nebular_lines(models_lines)
def build_schreiber2016(base):
models = []
schreiber2016_dir = os.path.join(os.path.dirname(__file__),
'schreiber2016/')
print("Importing {}...".format(schreiber2016_dir + 'g15_pah.fits'))
pah = Table.read(schreiber2016_dir + 'g15_pah.fits')
print("Importing {}...".format(schreiber2016_dir + 'g15_dust.fits'))
dust = Table.read(schreiber2016_dir + 'g15_dust.fits')
# Getting the lambda grid for the templates and convert from μm to nm.
wave = dust['LAM'][0, 0, :].data * 1e3
for td in np.arange(15., 100.):
# Find the closest temperature in the model list of tdust
tsed = np.argmin(np.absolute(dust['TDUST'][0].data-td))
# The models are in νFν. We convert this to W/nm.
lumin_dust = dust['SED'][0, tsed, :].data / wave
lumin_pah = pah['SED'][0, tsed, :].data / wave
models.append(Schreiber2016(0, td, wave, lumin_dust))
models.append(Schreiber2016(1, td, wave, lumin_pah))
base.add_schreiber2016(models)
def build_base():
base = Database(writable=True)
base.upgrade_base()
......@@ -709,6 +738,11 @@ def build_base():
print("\nDONE\n")
print('#' * 78)
print("9- Importing Schreiber et al (2016) models\n")
build_schreiber2016(base)
print("\nDONE\n")
print('#' * 78)
base.session.close_all()
......
......@@ -2,10 +2,10 @@
# energy
# H beta pseudo filter
4827.87 0.0
4827.875 -0.2857959417043478
4847.87 -0.2857959417043478
4847.875 0.34782608695652173
4876.625 0.34782608695652173
4876.63 -0.2857959417043478
4891.625 -0.2857959417043478
4827.875 0.2857959417043478
4847.87 0.2857959417043478
4847.875 -0.34782608695652173
4876.625 -0.34782608695652173
4876.63 0.2857959417043478
4891.625 0.2857959417043478
4891.63 0.0
......@@ -2,14 +2,14 @@
# energy
# H delta pseudo filter
4041.5 0.0
4041.6000000000004 -0.1415428166967742
4079.75 -0.1415428166967742
4041.6000000000004 0.1415428166967742
4079.75 0.1415428166967742
4079.8 0.0
4083.3999999999996 0.0
4083.5 0.25806451612903225
4122.25 0.25806451612903225
4083.5 -0.25806451612903225
4122.25 -0.25806451612903225
4122.3 0.0
4128.4 0.0
4128.5 -0.1415428166967742
4161.0 -0.1415428166967742
4128.5 0.1415428166967742
4161.0 0.1415428166967742
4161.1 0.0
......@@ -2,12 +2,12 @@
# energy
# H gamma pseudo filter
4283.4 0.0
4283.5 -0.11268875366857142
4319.74 -0.11268875366857142
4319.75 0.22857142857142856
4363.5 0.22857142857142856
4283.5 0.11268875366857142
4319.74 0.11268875366857142
4319.75 -0.22857142857142856
4363.5 -0.22857142857142856
4363.6 0.0
4367.200000000001 0.0
4367.25 -0.11268875366857142
4419.75 -0.11268875366857142
4367.25 0.11268875366857142
4419.75 0.11268875366857142
4419.8 0.0
......@@ -2,14 +2,14 @@
# energy
# Mg2 pseudo filter
4895.12 0.0
4895.125 -0.07843137254117646
4957.625 -0.07843137254117646
4895.125 0.07843137254117646
4957.625 0.07843137254117646
4957.63 0.0
5154.120000000001 0.0
5154.125 0.23529411764705882
5196.625 0.23529411764705882
5154.125 -0.23529411764705882
5196.625 -0.23529411764705882
5196.63 0.0
5301.12 0.0
5301.125 -0.07843137254117646
5366.125 -0.07843137254117646
5301.125 0.07843137254117646
5366.125 0.07843137254117646
5366.130000000001 0.0
......@@ -8,8 +8,8 @@ import multiprocessing as mp
import sys
from .session.configuration import Configuration
from .analysis_modules import get_module as get_analysis_module
from .analysis_modules.utils import ParametersHandler
from .analysis_modules import get_module
from .handlers.parameters_handler import ParametersHandler
__version__ = "0.1-alpha"
......@@ -36,28 +36,21 @@ def check(config):
"""
# TODO: Check if all the parameters that don't have default values are
# given for each module.
print("With this configuration, pcigale must compute {} "
"SEDs.".format(ParametersHandler(
config.configuration['creation_modules'],
config.configuration['creation_modules_params']
).size))
configuration = config.configuration
if configuration:
print("With this configuration cigale will compute {} "
"models.".format(ParametersHandler(configuration).size))
def run(config):
"""Run the analysis.
"""
data_file = config.configuration['data_file']
column_list = config.configuration['column_list']
creation_modules = config.configuration['creation_modules']
creation_modules_params = config.configuration['creation_modules_params']
analysis_module = get_analysis_module(config.configuration[
'analysis_method'])
analysis_module_params = config.configuration['analysis_method_params']
cores = config.configuration['cores']
analysis_module.process(data_file, column_list, creation_modules,
creation_modules_params, analysis_module_params,
cores)
configuration = config.configuration
if configuration:
analysis_module = get_module(configuration['analysis_method'])
analysis_module.process(configuration)
def main():
......
......@@ -31,8 +31,7 @@ class AnalysisModule(object):
# module parameter.
self.parameters = kwargs
def _process(self, data_file, column_list, creation_modules,
creation_modules_params, parameters):
def _process(self, configuration):
"""Do the actual analysis
This method is responsible for the fitting / analysis process
......@@ -40,19 +39,8 @@ class AnalysisModule(object):
Parameters
----------
data_file: string
Name of the file containing the observations to be fitted.
column_list: array of strings
Names of the columns from the data file to use in the analysis.
creation_modules: array of strings
Names (in the right order) of the modules to use to build the SED.
creation_modules_params: array of array of dictionaries
Array containing all the possible combinations of configurations
for the creation_modules. Each 'inner' array has the same length as
the creation_modules array and contains the configuration
dictionary for the corresponding module.
parameters: dictionary
Configuration for the module.
configuration: dictionary
Configuration file
Returns
-------
......@@ -61,8 +49,7 @@ class AnalysisModule(object):
"""
raise NotImplementedError()
def process(self, data_file, column_list, creation_modules,
creation_modules_params, parameters):
def process(self, configuration):
"""Process with the analysis
This method is responsible for checking the module parameters before
......@@ -72,19 +59,8 @@ class AnalysisModule(object):
Parameters
----------
data_file: string
Name of the file containing the observations to be fitted.
column_list: array of strings
Names of the columns from the data file to use in the analysis.
creation_modules: array of strings
Names (in the right order) of the modules to use to build the SED.
creation_modules_params: array of array of dictionaries
Array containing all the possible combinations of configurations
for the creation_modules. Each 'inner' array has the same length as
the creation_modules array and contains the configuration
dictionary for the corresponding module.
parameters: dictionary
Configuration for the module.
configuration: dictionary
Contents of pcigale.ini in the form of a dictionary
Returns
-------
......@@ -95,6 +71,7 @@ class AnalysisModule(object):
KeyError: when not all the needed parameters are given.
"""
parameters = configuration['analysis_params']
# For parameters that are present on the parameter_list with a default
# value and that are not in the parameters dictionary, we add them
# with their default value.
......@@ -124,8 +101,7 @@ class AnalysisModule(object):
"expected one." + message)
# We do the actual processing
self._process(data_file, column_list, creation_modules,
creation_modules_params, parameters)
self._process(configuration)
def get_module(module_name):
......
......@@ -25,6 +25,7 @@ reduced χ²) is given for each observation.
"""
from collections import OrderedDict
import ctypes
import multiprocessing as mp
from multiprocessing.sharedctypes import RawArray
......@@ -34,64 +35,63 @@ import numpy as np
from ...utils import read_table
from .. import AnalysisModule, complete_obs_table
from .utils import save_table_analysis, save_table_best, analyse_chi2
from .utils import save_results, analyse_chi2
from ...warehouse import SedWarehouse
from .workers import sed as worker_sed
from .workers import init_sed as init_worker_sed
from .workers import init_analysis as init_worker_analysis
from .workers import analysis as worker_analysis
from ..utils import ParametersHandler, backup_dir
from ..utils import backup_dir
from ...handlers.parameters_handler import ParametersHandler
# Tolerance threshold under which any flux or error is considered as 0.
TOLERANCE = 1e-12
# Limit the redshift to this number of decimals
REDSHIFT_DECIMALS = 2
class PdfAnalysis(AnalysisModule):
"""PDF analysis module"""
parameter_list = dict([
("analysed_variables", (
"array of strings",
"List of the variables (in the SEDs info dictionaries) for which "
"the statistical analysis will be done.",
parameter_list = OrderedDict([
("variables", (
"cigale_string_list()",
"List of the physical properties to estimate. Leave empty to "
"analyse all the physical properties (not recommended when there "
"are many models).",
["sfh.sfr", "sfh.sfr10Myrs", "sfh.sfr100Myrs"]
)),
("save_best_sed", (
"boolean",
"boolean()",
"If true, save the best SED for each observation to a file.",
False
)),
("save_chi2", (
"boolean",
"boolean()",
"If true, for each observation and each analysed variable save "
"the reduced chi2.",
False
)),
("save_pdf", (
"boolean",
"boolean()",
"If true, for each observation and each analysed variable save "
"the probability density function.",
False
)),
("lim_flag", (
"boolean",
"boolean()",
"If true, for each object check whether upper limits are present "
"and analyse them.",
False
)),
("mock_flag", (
"boolean",
"boolean()",
"If true, for each object we create a mock object "
"and analyse them.",
False
))
])
def process(self, data_file, column_list, creation_modules,
creation_modules_params, config, cores):
def process(self, conf):
"""Process with the psum analysis.
The analysis is done in two steps which can both run on multiple
......@@ -102,19 +102,8 @@ class PdfAnalysis(AnalysisModule):
Parameters
----------
data_file: string
Name of the file containing the observations to fit.
column_list: list of strings
Name of the columns from the data file to use for the analysis.
creation_modules: list of strings
List of the module names (in the right order) to use for creating
the SEDs.
creation_modules_params: list of dictionaries
List of the parameter dictionaries for each module.
config: dictionary
Dictionary containing the configuration.
core: integer
Number of cores to run the analysis on
conf: dictionary
Contents of pcigale.ini in the form of a dictionary
"""
np.seterr(invalid='ignore')
......@@ -125,42 +114,38 @@ class PdfAnalysis(AnalysisModule):
backup_dir()
# Initalise variables from input arguments.
analysed_variables = config["analysed_variables"]
analysed_variables_nolog = [''.join(variable.rsplit('_log', 1)) for
variable in analysed_variables]
n_variables = len(analysed_variables)
save = {key: config["save_{}".format(key)].lower() == "true"
for key in ["best_sed", "chi2", "pdf"]}
lim_flag = config["lim_flag"].lower() == "true"
mock_flag = config["mock_flag"].lower() == "true"
filters = [name for name in column_list if not name.endswith('_err')]
variables = conf['analysis_params']["variables"]
variables_nolog = [variable[:-4] if variable.endswith('_log') else
variable for variable in variables]
n_variables = len(variables)
save = {key: conf['analysis_params']["save_{}".format(key)] for key in
["best_sed", "chi2", "pdf"]}
lim_flag = conf['analysis_params']["lim_flag"]
filters = [name for name in conf['bands'] if not
name.endswith('_err')]
n_filters = len(filters)
# Read the observation table and complete it by adding error where
# none is provided and by adding the systematic deviation.
obs_table = complete_obs_table(read_table(data_file), column_list,
filters, TOLERANCE, lim_flag)
obs_table = complete_obs_table(read_table(conf['data_file']),
conf['bands'], filters, TOLERANCE,
lim_flag)
n_obs = len(obs_table)
w_redshifting = creation_modules.index('redshifting')
if list(creation_modules_params[w_redshifting]['redshift']) == ['']:
z = np.unique(np.around(obs_table['redshift'],
decimals=REDSHIFT_DECIMALS))
creation_modules_params[w_redshifting]['redshift'] = z
del z
z = np.array(conf['sed_modules_params']['redshifting']['redshift'])
# The parameters handler allows us to retrieve the models parameters
# from a 1D index. This is useful in that we do not have to create
# a list of parameters as they are computed on-the-fly. It also has
# nice goodies such as finding the index of the first parameter to
# have changed between two indices or the number of models.
params = ParametersHandler(creation_modules, creation_modules_params)
params = ParametersHandler(conf)
n_params = params.size
# Retrieve an arbitrary SED to obtain the list of output parameters
warehouse = SedWarehouse()
sed = warehouse.get_sed(creation_modules, params.from_index(0))
sed = warehouse.get_sed(conf['sed_modules'], params.from_index(0))
info = list(sed.info.keys())
info.sort()
n_info = len(info)
......@@ -175,24 +160,19 @@ class PdfAnalysis(AnalysisModule):
# not write on the same section.
# We put the shape in a tuple along with the RawArray because workers
# need to know the shape to create the numpy array from the RawArray.
model_redshifts = (RawArray(ctypes.c_double, n_params),
(n_params))
model_fluxes = (RawArray(ctypes.c_double,
n_params * n_filters),
model_fluxes = (RawArray(ctypes.c_double, n_params * n_filters),
(n_params, n_filters))
model_variables = (RawArray(ctypes.c_double,
n_params * n_variables),
model_variables = (RawArray(ctypes.c_double, n_params * n_variables),
(n_params, n_variables))
initargs = (params, filters, analysed_variables_nolog, model_redshifts,
model_fluxes, model_variables, time.time(),
mp.Value('i', 0))
if cores == 1: # Do not create a new process
initargs = (params, filters, variables_nolog, model_fluxes,
model_variables, time.time(), mp.Value('i', 0))
if conf['cores'] == 1: # Do not create a new process
init_worker_sed(*initargs)
for idx in range(n_params):