Out of memory during plot_phase

[avivajpeyi]

gosh, these incessant memory errors! It's not even during sampling but during plotting!

nbconvert.preprocessors.execute.DeadKernelError: Kernel died
slurmstepd: error: Detected 1 oom-kill event(s) in step 26320284.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
srun: error: john16: task 0: Out Of Memory

Occurs at

plot_phase(tic_entry, inference_data, planet_transit_model)

[dfm]

Dang - this is probably related to this line:

tess-atlas/src/tess_atlas/plotting/plotting_utils.py

Line 192 in 6066e78

target=[model.lightcurve_models, model.gp_mu],

and the computation of the GP mean. I wouldn't expect it to be a huge memory cost, but I guess it is. Do you really need the GP prediction for all those samples?

[avivajpeyi]

tess-atlas/src/tess_atlas/notebook_templates/toi_template.py

Line 345 in d5b5ebd

my_planet_transit_model.gp_mu = gp.predict(

Maybe just remove t and this might just work ^TM

[avivajpeyi]

do we need to compute GP so many times?

[avivajpeyi]

probably not

[avivajpeyi]

probably just the median(GP)

[avivajpeyi]

Offt not fixed -- this is still occurring with 1.5k jobs!

Rather frustrating...

[avivajpeyi]

seems to be fixed now