Inhomogeneous Cosmologies III Software notes

See also:

• scalar averaging software - ramses-scalav: CosmoTorun17Software#workshop_notes_45_scalar_averaging_N_45body
• SageManifolds - CosmoTorun17Software#workshop_notes_45_Sage_Manifolds

Einstein Toolkit tutorial

• main URL 2017: http://cosmo.torun.pl/cosmotorun17_soft.html + notes at CosmoTorun17Software#workshop_notes_45_ET

Workshop notes - ET

• FIRST download:
  • as per instructions on: http://einsteintoolkit.org
  • or easier: Please try to install the ET (and perhaps reproduce a simple spacetime) prior to the workshop, following the instructions at https://docs.einsteintoolkit.org/et-docs/Simplified_Tutorial_for_New_Users.

Download FLRWSolver and add it to the ET

• Clone the public FLRWSolver version from bitbucket using: git clone https://github.com/hayleyjm/FLRWSolver_public
  • Warning: do not use the 'gh' command recommended by Github, since it aims at vendor lockin, also known as "destroying your academic freedom".
• Rename the folder flrwsolver, and move to Cactus/arrangements/EinsteinInitialData/
• Edit thornlists/einsteintoolkit.th in your top Cactus directory
• Find EinsteinInitialData
• Add the line: EinsteinInitialData/FLRWSolver beneath other thorns
• You may safely comment out the line Llama/WaveExtractL since it is heavy (RAM/cpu time) to compile and not needed.
• You may also safely comment out EinsteinInitialData/NoExcision, or any of the Meudon_BH thorns (if you run into errors)
• Compile: ./simfactory/bin/sim build --mdbkey make 'make -j2' --thornlist=thornlists/einsteintoolkit.th --optionlist=myoptionlist.cfg
  • where myoptionlist.cfg is the appropriate optionlist for your machine -- look in simfactory/mdb/optionlists for a list
• (You will probably first be asked to use sim setup ; you should create a directory where you want to store simulations, and then run ./simfactory/bin/sim setup ; for local usage, you can accept the defaults for the "key" values.)
• Run: ./simfactory/bin/sim submit FLRW1 --parfile=path/to/FLRW.par --procs=1 --walltime=0:30:0
• OR: navigate to your simulations/ directory and simply run: ./cactus_sim FLRW.par (where cactus_sim is the executable, located in Cactus/exe/. Create a softlink in your simulations directory with ln -s /path/to/Cactus/exe/cactus_sim . )

Checking your solutions

• You will also find a simple Python script exactsolns.py in the FLRWSolver repository to check how your simulation is working.
• Check the script, and make sure the indicated parameters match your simulation
• This will generate some PDF plots which you can see how well you match the FLRW solution
• If you get some weird-looking solutions (e.g. density dropping to very low values suddenly), do not fear, try increasing the parameter EOS_Omni::poly_k in your .par file and re-running the simulation (to a max value of 0.1)

More interesting runs

• We can repeat the above run using instead FLRW_perturb.par to simulate a simple single-mode perturbation to a background FLRW spacetime
• Once this is done, the same python script exactsolns.py can be edited to instead compare solutions for the growth of linear perturbations (set FLRW=False)
• Note that the perturbations may not match the exact solution very well -- this is simply because we are currently running a 10^3 grid, so increase the resolution (change dx,dy,dz in your parameter file) and you will see a better match!
• We can use SPLASH to visualise our 3D HDF5 data output from Cactus.

SPLASH

• Usually used for visualising particle distributions
• Daniel has modified it to read in Cactus HDF5 data
• We first need to modify the Cactus output to have one file (with all variables included) per timestep. To do this, run the Python script (also located in the FLRWSolver repository): python split_HDF5_per_iteration.py
• This is written using Python2.7, and you will need to install h5py and numpy
• Once you have run this, you will have some files named FLRW_perturb_it0.hdf5, FLRW_perturb_it10.hdf5... and so on.
• These files can then be visualised using SPLASH. Download from: http://users.monash.edu.au/~dprice/splash/ and first compile as per instructions on the webpage
• We then need to compile the Cactus capabilities separately: make SYSTEM=gfortran cactus
• You should now be able to visualise your data with: csplash-hdf5 FLRW_perturb_it0.hdf5 and select from the list which variable you want to look at!
• Enjoy!

Hints

• avoid filling home directory: To avoid creating a huge (733 Mb) directory in your home directory for the simplified tutorial for new users, create your target directory and soft link it to ~/simulations either before you start or before you start running a simulation.

Successful runs of tutorial

• CosmoTorun17: Krzysztof, Boude, Sven
• InhomCosmIII: Quentin

Gevolution hackathon

• gevolution:
  • https://github.com/gevolution-code/gevolution-1.1
  • ArXiv:1604.06065
  • makefile edit for Debian GNU/Linux (stretch):

    INCLUDE += -I../LATfield2 -I/usr/include/hdf5/serial -L/usr/lib/x86_64-linux-gnu/hdf5/serial

    or

    INCLUDE += -I../LATfield2 -I/usr/include/hdf5/openmpi -L/usr/lib/x86_64-linux-gnu/hdf5/openmpi

  • LATfield2 hack: see this hdf5.h hack to daverio LATfield2
  • if you unload both package directories at the same directory level, then in gevolution-1.1/ you can do ln -s ../LATfield2 . so that the gevolution makefile can find the LATfield2 files
  • Create an output directory mkdir output/
  • example libraries you will need: sudo aptitude install libfftw3-dev openmpi-bin libopenmpi-dev libhdf5-openmpi-dev libhdf5-dev hdf5-helpers
  • You should be able to compile gevolution with make and run the example with

    (time mpirun -np 4 ./gevolution -n 2 -m 2 -s settings.ini) > gevolution.log 2>&1

    in order to get a nice log file - on Boud's 4-core laptop this took about 140 seconds of user time and created many output files in output/ .
• Visualisation
  • SPLASH:
    • http://users.monash.edu.au/~dprice/splash/
    • Debian GNU/Linux derivatives (Ubuntu, Mint, ...): aptitude update && aptitude install splash
      • DebianBackportSplash
  • gadgetviewer (GPLv3): https://github.com/jchelly/gadgetviewer

Hints

• To test run on a machine with less than four cores, a hack to try is: mpirun -np 4 --map-by core:OVERSUBSCRIBE" ; you might get a segmentation fault at the end of the run, after output files are created

Maxima

• debian derivatives: aptitude install maxima
• http://maxima.sourceforge.net
• Szymon - general introduction - git clone https://bitbucket.org/broukema/maxima_flrw
• Boud - example of using ctensor: FLRWEquationsGR
• Malcolm MacCallum Living Review on Relativity on computer algebra systems

Ray-tracing: gyoto

• TODO: learn https://gyoto.obspm.fr for ray-tracing independent of ET and gevolution

• gevolution phi P(k) example: