GitLab

We rely on OrderedDict to keep the order of the parameters. However given the way the code is structured currently, there is really no reason for this. So let's just switch to the standard dictionary. This has two advantages. First the standard dictionary is much faster than OrderedDict (implemented in C rather than in pure python). Then, standard dictionaries are marshalable, which makes them usable as keys for other dictionaries.

Replace the SED.info OrderedDict() with a very light weight and speficically tailored reimplementation of an OrderedDict(). The reason is that an OrderedDict() takes an inordinate amount to time to copy. In the end this could amount to a very significant fraction of the total runtime. Do not use this new implementation anywhere or it will break your code. It makes very strong assumptions on how it is to be used.

Implement a cache for the filters to compute the fluxes. Rather than passing the transmission table as an argument to compute_fnu(), which makes caching difficult and/or slow to compute the hash, we rather pass the filter name. Then the SED object fetches the filter from the database and resamples it on the optimal wavelength grid. The result is stored in cache to avoid carrying out this operation repeatedly.

Update CIGALE to clean anything related to 'average_sfr' and replace it by 'sfr10Myrs' and 'sfr100Myrs'

Move the simple reduced χ² analysis to a specific function and call this function. It helps keeping the code more readable.

We use only a memory-based cache, so we get rid of the option. For the time being we keep the architecture in place.

Cleaned up version of the upper limit handling code written by Denis. Some further optimisations should be easy.

Add back the possibility to read the redshifts from the input file. To do so, do not give any redshift to the redshifting module. To avoid having too many redshifts, it is rounded to 2 decimals. This number can be changed in the analysis modules.

Centralise OUT_DIR in one location. One day we will have to have a second look at all these files and maybe rearrange them a little.

Move the backup dir function to the utils file common to all analysis modules so it can easily be used by all anlysis modules.

Get rid of the parameters array. Rather we generate parameters on-on-fly. This is done using a new class ParameterHandlers, which handles the generation of parameters from a given 1D index. While at it, also do away with global variables. This allows to spawn processes rather than fork them. Even if it takes a little time to do the spawn, this allows to solves some deadlocks with libraries which do not support being used on both sides of a forked process. Overall these modifications makes startup faster (no need to compute the cache cleaning parameters beforehand) and the generation speed is also a few percent faster when the number of models is large enough to compensate for the relatively slow spaning time.

If the code is to be run only on 1 core, then there is no need to fork a new process for that. As a side effect, it should at least allow people who have a multiprocessing code to run the code, even if only on one core.

Improve memory management. Several techniques are used. First, we use arrays in shared memory when computing the SEDs (this should also be done for the analysis later). This avoids using temporary objects that have to be sent back into a list from pool.starmap. Second, we do not save the sed.info data until the best sed has been computed. Only then we recompute the SED to get the info. Third, the list of changed parameters for cache cleaning was too big. It contained the key and the value of the changed parameter to identify models to be discarded. Now we simply return an array of integers giving the index of the module to be discarded. Much simpler, much smaller. There are also some small improvements here and there that I probably forget. The main culprit now is by far the list of parameters. It gets horribly big. 3.5 GB for 30M models. Anyway, now pcigale runs with 30M models.

Make Denis happy. Now we display the number of models/objects already computed/analysed along with the time spent and the speed.

Prevent the model computation worker from returning a view of sed.info.values(). This meant the entire dictionary was actually copied, wasting a lot of memory. Also adopt a usual list rather than a numpy array to store the info in pdf_analysis.

The initial implementation of the parallel analysis ended up grinding pcigale to a halt. This was due to the numerous array transfers between the main process and subprocesses. To solve this problem, we now share arrays through a module. This has the advantage
that thanks to copy-on-write, we almost never need to actually copy arrays. Now all the subprocessing run at full speed. Quite a few changes for a patch that only starting with the activation of cache clearing.

Make pdf_analysis parallel. This required important changes. Functions saving data have been moved to the utils.py file. Workers for the parallel processes have been put in workers.py. To make sure we eliminate models incompatible with the age of the universe now the latter is indicated in sed.info for easy access. Ditto for the redshift.

Move the plotting functions out of the analysis modules. Plots are now generated using an external script named pcigale-plots. This script is ready for multiprocessing but it is not yet activated. Finally, we can now save the reduced χ² values (which is required to plot them).

Tiny optimisation. Create arrays are empty rather than being zero-filled as we are going to assign all elements later on anyway.

Change the way redshifts are handled. Now all redshifts are specified in pcigale.ini. To compute the χ², we simply take the models corresponding to the closest χ².

Allow pcigale to run without stellar populations so we can carry out SED modelling of dust only for instance.

There is maybe a bug triggering the display of the best model figure in
a matplotlib interactive window.