On modern software

Today I noticed that gfdb updates were broken again. It was the third time it happened, so I knew this deserved a post.

I caught first update breakage purely by accident. One day I looked at the process list and noticed a chain of git commands sitting there doing nothing. Apparently at some point git decided it was time to do garbage collection, so it ran "git gc --auto", which, for reasons yet unknown to me, could not complete in 50 minutes on a fairly small and completely linear repository. This step should have happened before the push took place, but since it just hung there (with no cpu/disk activity, by the way) it was killed by task scheduler every hour to start new task, which repeated everything. This was going on for 5 days before I noticed. I run "git gc" manually and it finished in about a minute, cleared the clog and all was good (for a while).

Second time was exactly the same as first, except I was already aware it could happen, so I reacted sooner. Internet search brought nothing: either it was a new bug, or it was specific to some part of my configuration. Since I (mistakenly) already dismissed it I had to wait for another chance to debug and see what was happening.

However, today's problem was different. Call stack was not the same and had bash prompt for input at the top instead of gc command. After I killed it and manually tried to push all accumulated updates git asked me... for a username? What? Did github ban me, so my user-pass no longer worked? But why? I tried to login into github. Blah-blah, we see your device for the first time, how about you go fetch verification code we sent to your email? Oh great, fuck you very much, another service turned """""user-friendly""""". But whatever, here's your code.

Hmm, no notifications, the repo is there, everything looks pretty normal. I tried pushing again from the console. Typed the username and copypasted the password.

>remote: Invalid username or password.
>fatal: Authentication failed

What? Did I just mistype 5-letter username or miscopy a doubleclicked string? Try again:

>Enumerating objects: 14190, done.
>Counting objects: 100% (14190/14190), done.
>Delta compression using up to 12 threads
>Compressing objects: 100% (1834/1834), done.
>error: RPC failed; HTTP 401 curl 22 The requested URL returned error: 401
>fWatal: the remote end hung up unexpectedly

What?? Looks like github is having problems, should I try again?

>remote: Invalid username or password.
>fatal: Authentication failed

And then after one more attempt it accepted the changes I tried to push.

But what about git? Where is the password that was saved what, 3 months ago, and reused every hour since then? I went searching and encountered this magnificent piece of information:
>"git pull" will fail, because the password is incorrect, git then removes the offending user+password from the ~/.git-credentials file

Let me repeat that, so you can enjoy it one more time: whenever git discovers that any stored user+pass combination doesn't work it simply erases it from saved credentials file. I checked the file - yeah, 0 bytes. All puzzle pieces are in place.

What did we learn today?

• Shame on you, git. I did not expect this systemd-level bullshit from you, of all programs.
• Shame on you, github. Having transient authentication problems (caused by service overload, network partitioning or something else) is understandable, returning wrong error statuses and not notifying affected users is not.
• Modern software (and software as service doubly, if not triply so) ALWAYS needs to be propped up by humans. Even if you setup everything correctly and leave it unattended, then sooner or later, mostly sooner, something will fail.