1. LOFAR & Python
Gijs Molenaar
gijs@pythonic.nl

2. Agenda
● LOFAR
● AARTFAAC / TKP
● Python
● MonetDB

3. Non Python stuff

4. What is LOFAR

5. What is LOFAR
● Low-Frequency Array
● Distributed sensor network
● No dishes
● Simple and cheap antenna's
● Omni-directional
● Concurrent observations
● Software telescope
– Flexible
– Requires supercomputer

6. What is LOFAR
Biggest Radio telescope
array in the world
● 7000 antennas
● 10 to 80 MHz (LBA)
● 120 to 240 MHz (HBA)
● Multiple small stations
● NL, DE, UK, FR, SE
● New (unique) design

7. New design
● Many unsolved problems
● Multi-terabit/s data scale
● How to
– Process
– Store
– Distribute
– Make results useful
● Small scale example for SKA

8. Usage
● Epoch of re-ionization
● Cosmic rays
● Extragalactic surveys
● Transients
● Pulsars

10. Correlation

11. Correlation
● IBM blue Gene/P
● Groningen
● Alotta computer power
● Will be replaced with
FPGA's (for our
purpose)
● No Python

12. Example output – Cygnus A

13. Actual data

14. Transient Key Project
● Static sources are boring
●
Transient – something that changes
● Detect sources
●
Store extracted information
●
Associate with existing observations
●
Did something change?
●
This is where the Python fun starts!

15. ● The glue
● Distributing computation
● Image processing
● Source extraction
● Source association
● Database interactions

16. Why Python
● A lot of existing code
● Good (scientific) libraries
● Good astronomical libraries
● Easy to use
● Get it working first, then fast
● NumPy is quite fast when used correctly

17. What do we use?
● Python
● NumPy / SciPy
● Django
● MonetDB
● Pyrap (casacore wrapper)
– Does all astro shizzle you can imagine

18. Distributed Computation
● Home made (ASTRON)
●
SSH based
●
Calling remote scripts
● Shared code base (NFS)
● Protocol for passing arguments and results
●
Uses logging module a lot
●
Difficult to debug
● Difficult to profile
●
Hadoop probably better

19. The Lofar Cluster
● 100 nodes
● 24 cores each
● 20 TB raid5 storage
● 40 Gbps InfiniBand
● But for AARTFAAC we build our own
● Eventually

20. Main loop
● Extract source properties
● Store properties
● Do association
● Detect transients

21. The data
● 1 datacube per second
● 10 frequency bands
● In the future 10 images per second
● In the future 4 different polarization
● Non stop

22. Source extraction

23. Source extraction & NumPy
● Calculate average noise
● Subtract
● Find 'islands'
● Fit gauss

24. Store the data

25. Database
● Move calculation to the data
● Independent data
● Naturally separable by sky coordinates
● ~100 TB/year
● 10.000 insert/second

26. MonetDB
● Don't believe the hype
● NoNoSQL
● Relational
● Column store DB
● Fast
● Auto tuning!
● Developers next door (CWI)
● Cross research funding

27. INSERT INTO tempbasesources
(xtrsrc_id
,datapoints
,I_peak_sum
,I_peak_sq_sum
,weight_peak_sum
,weight_I_peak_sum
,weight_I_peak_sq_sum
)
SELECT b0.xtrsrc_id
,b0.datapoints
+ 1 AS datapoints
,b0.I_peak_sum
+ x0.I_peak AS i_peak_sum
,b0.I_peak_sq_sum
+ x0.I_peak * x0.I_peak AS i_peak_sq_sum
,b0.weight_peak_sum
+ 1 / (x0.I_peak_err * x0.I_peak_err) AS weight_peak_sum
,b0.weight_I_peak_sum
+ x0.I_peak / (x0.I_peak_err * x0.I_peak_err)
AS weight_i_peak_sum
,b0.weight_I_peak_sq_sum
+ x0.I_peak * x0.I_peak / (x0.I_peak_err * x0.I_peak_err)
AS weight_i_peak_sq_sum
FROM basesources b0
,extractedsources x0
WHERE x0.image_id = @imageid
AND b0.zone BETWEEN CAST(FLOOR((x0.decl - @theta) / x0.zoneheight
) AS INTEGER)
AND CAST(FLOOR((x0.decl + @theta) / x0.zoneheight
) AS INTEGER)
AND ASIN(SQRT((x0.x - b0.x)*(x0.x - b0.x)
+(x0.y - b0.y)*(x0.y - b0.y)
+(x0.z - b0.z)*(x0.z - b0.z)
)/2
)
/
SQRT(x0.ra_err * x0.ra_err + b0.ra_err * b0.ra_err
+x0.decl_err * x0.decl_err + b0.decl_err * b0.decl_err)
< @assoc_r;

28. MonetDB vs MySQL

29. MonetDB and Python
● We maintain the MonetDB Python API
● http://pypi.python.org/pypi/python-monetdb/
● Problems? Ask me :)

30. Challenges
● Debugging queries
● MonetDB still in active development

31. Djonet
● MonetDB backend for Django
● https://github.com/gijzelaerr/djonet
● brew install monetdb
● pip install python-monetdb djonet
● Contributions are welcome!

32. VO events
● Standardized language
● Report observations of astronomical events
● Hey world, check this supernova out over
there
● http://comet.transientskp.org

33. VOevents

34. Visualisation
● Web interface
● Django!
● Not public (yet)

35. Source code?
● No sorry.
● Maybe some day
● Legal issues, bureaucracy
● UvA, Astron
●

36. More
● http://www.transientskp.org/
● http://www.lofar.org/
● http://www.aartfaac.org/

37. Questions?