1. 24 May, 2014 3 commits
  2. 23 May, 2014 2 commits
  3. 21 May, 2014 1 commit
  4. 20 May, 2014 2 commits
  5. 07 May, 2014 5 commits
  6. 06 May, 2014 2 commits
  7. 01 May, 2014 1 commit
  8. 16 Apr, 2014 2 commits
  9. 29 Mar, 2014 3 commits
  10. 23 Mar, 2014 1 commit
    • Médéric Boquien's avatar
      Improve memory management. Several techniques are used. First, we use arrays... · 83e13558
      Médéric Boquien authored
      Improve memory management. Several techniques are used. First, we use arrays in shared memory when computing the SEDs (this should also be done for the analysis later). This avoids using temporary objects that have to be sent back into a list from pool.starmap. Second, we do not save the sed.info data until the best sed has been computed. Only then we recompute the SED to get the info. Third, the list of changed parameters for cache cleaning was too big. It contained the key and the value of the changed parameter to identify models to be discarded. Now we simply return an array of integers giving the index of the module to be discarded. Much simpler, much smaller. There are also some small improvements here and there that I probably forget. The main culprit now is by far the list of parameters. It gets horribly big. 3.5 GB for 30M models. Anyway, now pcigale runs with 30M models.
      83e13558
  11. 21 Mar, 2014 1 commit
  12. 20 Mar, 2014 2 commits
  13. 11 Mar, 2014 2 commits
    • Médéric Boquien's avatar
      The initial implementation of the parallel analysis ended up grinding pcigale... · 31c30672
      Médéric Boquien authored
      The initial implementation of the parallel analysis ended up grinding pcigale to a halt. This was due to the numerous array transfers between the main process and subprocesses. To solve this problem, we now share arrays through a module. This has the advantage
      that thanks to copy-on-write, we almost never need to actually copy arrays. Now all the subprocessing run at full speed. Quite a few changes for a patch that only starting with the activation of cache clearing.
      31c30672
    • Médéric Boquien's avatar
      Make pdf_analysis parallel. This required important changes. Functions saving... · 3a44b65a
      Médéric Boquien authored
      Make pdf_analysis parallel. This required important changes. Functions saving data have been moved to the utils.py file. Workers for the parallel processes have been put in workers.py. To make sure we eliminate models incompatible with the age of the universe now the latter is indicated in sed.info for easy access. Ditto for the redshift.
      3a44b65a