wrapping gitpuller

[kcranston]

Some notes here about wrapping gitpuller to deal with #255 and #205

• gitpuller runs as a lifecycleHook and if it fails, the pod does not start. We can't fix the failure by cleaning up the file system because we can't access the unmounted volume for a pod that isn't running
• it is useful and desirable for students to be able to access their pod even if the gitpuller step fails
• if we wrap the gitpuller command in a python script that gracefully catches any exceptions, we allow the pod to startup no matter what happens to gitpuller (the python script exits with error code 0, so it looks like a pass to kubernetes)
• kubernetes only logs failure events for lifecycleHooks - see docs and also this issue, it does not log stdout / stderr from commands that run as lifecycle hooks
• so, when a wrapped gitpuller fails, it is completely silent. The student gets no notification, and there is nothing in the logs for the admins to check

[kcranston]

Based on response to this discourse post, there is no easy fix. Plan going forward:

• ensure that the python wrapper is as robust as possible
• have the wrapper fix errors where it can, for example, backing up an existing git repo on the pod and trying gitpuller again as a workaround for the case where gitpuller can't merge
• the python wrapper could also write a local log file to the pod, which would be available to users / admins (noting that this would fail if the user storage is full)

[lwasser]

@kcranston just an idea - is it possible to send an email or message somehow when this fails? that may be difficult i just figured i'd ask. this could even be writing a message on github in an issue and tagging X people. i agree it's useful to allow the pod to start! it's also useful for me to know there is a point of failure when it happens.

[lwasser]

i mention this only because i know we use the github API and as such i was wondering if adding to an issue or creating an issue is something we could do via python too! just an idea. maybe a bad one... or a complicated one? but an idea.

[kcranston]

I like this idea. Let me think about options.