A Distributed Learning Architecture for Scientific Imaging Problems


Authors: A. Panousopoulou, S. Farrens, K. Fotiadou, A. Woiselle, G. Tsagkatakis, J-L. Starck,  P. Tsakalides
Journal: arXiv
Year: 2018
Download: ADS | arXiv


Current trends in scientific imaging are challenged by the emerging need of integrating sophisticated machine learning with Big Data analytics platforms. This work proposes an in-memory distributed learning architecture for enabling sophisticated learning and optimization techniques on scientific imaging problems, which are characterized by the combination of variant information from different origins. We apply the resulting, Spark-compliant, architecture on two emerging use cases from the scientific imaging domain, namely: (a) the space variant deconvolution of galaxy imaging surveys (astrophysics), (b) the super-resolution based on coupled dictionary training (remote sensing). We conduct evaluation studies considering relevant datasets, and the results report at least 60\% improvement in time response against the conventional computing solutions. Ultimately, the offered discussion provides useful practical insights on the impact of key Spark tuning parameters on the speedup achieved, and the memory/disk footprint.

Cosmostat day Machine Learning in Astrophysics

Date: January the 24th, 2019

Organizer:  Joana Frontera-Pons  <joana.frontera-pons@cea.fr>


Local information

CEA Saclay is around 23 km South of Paris. The astrophysics division (DAp) is located at the CEA site at Orme des Merisiers, which is around 1 km South of the main CEA campus. See http://www.cosmostat.org/link/how-to-get-to-sap/ for detailed information on how to arrive.

On January the 24th, 2019, we organize the fourth day on machine learning in astrophysics at DAp, CEA Saclay. 


All talks are taking place at DAp, Salle Galilée (Building 713)

10:00 - 10:45h. TBD
10:45 - 11:15h. TBD
11:15 - 11:45h. TBD

12:00 - 13:30h. Lunch

13:30 - 14:00h. TBD
14:00 - 14:30h. TBD                                                            
14:30 - 15:00h. TBD
15:00 - 15:30h. TBD

15:30 - 16:00h. Coffe break

16:00 - 17:00h. Round table

 Previous Cosmostat Days on Machine Learning in Astrophysics :

Euclid - France atelier/workshop gravitational lensing

Date: October 22, 2018

Organizer:  Martin Kilbinger & Karim Benabed

Venue: IAP,  98bis bd Arago, 75014 Paris. Salle Entresol

Local information



Martin Kilbinger
Karim Benabed
Sandrine Codis
Eric Jullo
Francis Bernardeau
Yohan Dubois
Santiago Casas
Raphael Gavazzi
Alain Blanchard
Patrick Hudelot
Calum Murray
Matteo Rizzato
Samuel Farrens
Alexandre Barthelemy
Austin Peel
Nicolas Martinet
Morgan Schmitz
Virginia Ajani
Henry McCracken
Peter Taylor
Bertrand Morin
Céline Gouin



10:00   Café
10:30   Martin Kilbinger                Welcome, introduction, goals of the meeting, resources
10:45   Matteo Rizzato                   Information content in the weak lensing bispectrum
11:15   Eric Jullo                                 WLSWG work package “Galaxy-galaxy lensing”
11:45   Alexandre Barthelemy    One-point statistics of weak lensing maps
12:15    Peter Taylor                         k-cut Cosmic Shear: Tunable Power Spectrum Sensitivity to Test Gravity
12:45   Henry Joy McCracken    Euclid VIS activities and weak lensing requirements
13:00   Lunch
14:15   Austin Peel                           Peak counts: breaking degeneracies & machine learning
14:45   Nicolas Martinet               WL peak/mass mapping/shear calibration
15:15   Céline Gouin                       The impact of baryons on WL statistics
15:45   Bertrand Morin                  COSEBIs - Implementation of cosmic shear E-/B- modes
16:15   Martin Kilbinger, all          WL projects in Euclid-France, discussion, future plans
17:15   End



French-Chinese Days on Weak Lensing

Date: October 4-5, 2018

Organizer:  Jean-Luc Starck and Martin Kilbinger


Local information

CEA Saclay is around 23 km South of Paris. The astrophysics division (DAp) is located at the CEA site at Orme des Merisiers, which is around 1 km South of the main CEA campus. See http://www.cosmostat.org/link/how-to-get-to-sap/ for detailed information on how to arrive.

On 4 and 5 October, 2018, we are organizing the first French-Chinese weak-lensing meeting at DAp, CEA Saclay. 


All talks are taking place at DAp, Salle Kepler (Building 709)

Thursday, October 4

9:30 - 10:00h.  Café

10:00 - 10:15h.    Welcome & introductions

10:15 - 10:45h.   Hu Zhan,   Overview of CSS-OS

10:45 - 11:15h.   Martin Kilbinger, Overview of CFIS Weak Lensing

11:15 - 11:45h.  Jun Zhang, Fourier_Quad,  a shear measurement method in Fourier Space

11:45 - 14:00h.  Lunch at the Rotonde

14:00 - 14:30h. Morgan Schmitz,  PSF Modeling using a Graph Manifold

14:30 - 15:00h. Chengliang Wei, A full sky WL simulation with semi-analytic galaxy formation 

15:00 - 15:30h. Jean-Luc Starck,  WL Mass Mapping

15:30 - 16:00h. Zuhui Fan,  WL peak statistics

16:00 - 16:30h. Austin Peel,  Cosmology with Mass Maps

Friday, October 5

9:30 - 10:00h.      Café

10:00 - 10:30h.   Sam Farrens,   The CFIS pipeline

10:30 - 11:00h.  Ran Li,  Lensing studies of sub-structures

11:00 - 11:30h.  Axel  Guinot,  Preliminary CFIS results

11:30 - 12:00h.  Liping Fu, Shear measurement from VOICE deep survey

12:00 - 14:00h. Lunch at Les Algorithmes

14:00 - 14:30h. Jean-Charles Cuillandre, The Euclid mission and ground-based observations

14:30 - 15:00h.  Huanyuan Shan: KiDS WL studies (via skype)

15:00 - 15:30h.  Alexandre Bruckert, Machine learning for blended objects separation

15:30 - 16:00h.   Rebeca Araripe Furtado Cunha,  Optimal Transport and PSF Modeling

16:00 -  17:00h. Discussion


École Euclid de cosmologie 2018

Date: August 20 - September 1, 2018

Venue: Roscoff, Bretagne, France

Website: http://ecole-euclid.cnrs.fr/accueil-session-2018

Lecture ``Weak gravitational lensing'' (Le lentillage gravitationnel), Martin Kilbinger.

Find here links to the lecture notes, TD exercises, "tables rondes" topics, and other information.

  • Resources.
    • A great and detailed introduction to (weak) gravitational lensing are the 2005 Saas Fee lecture notes by Peter Schneider. Download Part I (Introduction to lensing) and Part III (Weak lensing) from my homepage.
    • Check out Sarah Bridle's video lectures on WL from 2014.
  • TD cycle 1+2, Data analysis.
    1.  We will work on a rather large (150 MB) weak-lensing catalogues from the public CFHTLenS web page. During the TD I will show instructions how to create and download this catalogue. These catalogues will also be available on the virtual machine for the school.
      If you like, you can however download the catalogue on your laptop at home. Please have a look at the instructions in the TD slides.
    2. If you want to do the TD on your laptop, you'll need to download and install athena (the newest version 1.7). Available on the VM.
    3.  For one of the bonus TD you'll need a new version of pallas.py (v 1.8beta). Download it here. Available on the VM.
  • Lecture notes and exercise classes.  You can already download the slides in one file (40 - 60 MB), but be ware that the content will still change slightly until the classes.
    • Part I (Cycle 1):    [all | day 1 (1/6)  |   day 2 (2/6) |  day 3 (3/6)]
    • Part II (Cycle 2):  [all | day 1 (4/6)   |   day 2 (5/6)  | day 3 (6/6)]
    • TD:                             [1/2 and 2/2]
    • Table Ronde sujet
  • Slack channel: ede2018.slack.com

DEDALE: Mathematical Tools to Help Navigate the Big Data Maze

Managing the huge volumes and varying streams of Big Data digital information presents formidable analytical challenges to anyone wanting to make sense of it. Consider the mapping of space, where scientists collect, process and transmit giga-scale data sets to generate accurate visual representations of millions of galaxies. Or consider the vast information being generated by genomics and bioinformatics as genomes are mapped and new drugs discovered. And soon the Internet of Things will bring millions of interconnected information-sensing and transmitting devices.