GitLab

Attempt to fix the upper limit by revert a clearly wrong commit from 2014 which did the opposite of what was said in the comment …

We need to sort the sed.info before saving the data. The reason is that the dictionary orders the keys with a hash so it appears random. In addition the order also changes from one process to another (I suspect the hash also depends on a seed). Thanks to Yannick for pointing out this issue.

We rely on OrderedDict to keep the order of the parameters. However given the way the code is structured currently, there is really no reason for this. So let's just switch to the standard dictionary. This has two advantages. First the standard dictionary is much faster than OrderedDict (implemented in C rather than in pure python). Then, standard dictionaries are marshalable, which makes them usable as keys for other dictionaries.

Move the simple reduced χ² analysis to a specific function and call this function. It helps keeping the code more readable.

Set the astropy log level to ERROR rather than WARNING to avoid clogging the terminal with unimportant messages.

simplification of the pdf analysis using fixed number of bins and a minimum error to 5% of the values

Estimate the parameters by binning the parameters into bins, with the height of the histogram based on the likelihood of the points in each bin. Adapted from an initial patch by Denis.

Ensure that the column names line is commented to make some external softwares happy when reading the file.

Centralise OUT_DIR in one location. One day we will have to have a second look at all these files and maybe rearrange them a little.

Move the backup dir function to the utils file common to all analysis modules so it can easily be used by all anlysis modules.

Get rid of the parameters array. Rather we generate parameters on-on-fly. This is done using a new class ParameterHandlers, which handles the generation of parameters from a given 1D index. While at it, also do away with global variables. This allows to spawn processes rather than fork them. Even if it takes a little time to do the spawn, this allows to solves some deadlocks with libraries which do not support being used on both sides of a forked process. Overall these modifications makes startup faster (no need to compute the cache cleaning parameters beforehand) and the generation speed is also a few percent faster when the number of models is large enough to compensate for the relatively slow spaning time.

Improve memory management. Several techniques are used. First, we use arrays in shared memory when computing the SEDs (this should also be done for the analysis later). This avoids using temporary objects that have to be sent back into a list from pool.starmap. Second, we do not save the sed.info data until the best sed has been computed. Only then we recompute the SED to get the info. Third, the list of changed parameters for cache cleaning was too big. It contained the key and the value of the changed parameter to identify models to be discarded. Now we simply return an array of integers giving the index of the module to be discarded. Much simpler, much smaller. There are also some small improvements here and there that I probably forget. The main culprit now is by far the list of parameters. It gets horribly big. 3.5 GB for 30M models. Anyway, now pcigale runs with 30M models.

The initial implementation of the parallel analysis ended up grinding pcigale to a halt. This was due to the numerous array transfers between the main process and subprocesses. To solve this problem, we now share arrays through a module. This has the advantage
that thanks to copy-on-write, we almost never need to actually copy arrays. Now all the subprocessing run at full speed. Quite a few changes for a patch that only starting with the activation of cache clearing.

Make pdf_analysis parallel. This required important changes. Functions saving data have been moved to the utils.py file. Workers for the parallel processes have been put in workers.py. To make sure we eliminate models incompatible with the age of the universe now the latter is indicated in sed.info for easy access. Ditto for the redshift.

Move the plotting functions out of the analysis modules. Plots are now generated using an external script named pcigale-plots. This script is ready for multiprocessing but it is not yet activated. Finally, we can now save the reduced χ² values (which is required to plot them).

Change the way redshifts are handled. Now all redshifts are specified in pcigale.ini. To compute the χ², we simply take the models corresponding to the closest χ².

Allow pcigale to run without stellar populations so we can carry out SED modelling of dust only for instance.

There is maybe a bug triggering the display of the best model figure in
a matplotlib interactive window.

Use a simplified gaussian kernel density estimation for computing the
probability density functions.

Maybe we should change the name.