Git – Fixing a botched commit / push to a bare repository

So, heya!! I know I am still taking time in between my posts. Long story short, let’s jump to the topic of this blog post 😉

I had my last weekend working with my entire ClientLinq team. We needed to complete a delivery and it appeared everything was progressing as per plan, when voila, my Manager called and told me no-one’s able to work anything with the Git repo’s origin. They are neither able to pull or push code to it. And the error reported was:

error: remote unpack failed: unpack-objects abnormal exit
To http://ourgitserver.com/OurRepo.git
! [remote rejected]       master -> master (unpacker error)
error: failed to push some refs to 'http://ourgitserver.com/OurRepo.git'

My immediate suspicion was two-fold:

1. Either someone had accidently made a large commit / push (e.g. committing a large binary file, like a dll, or image) to the origin repo.
2. There has been a botched commit / push where the commit / push did not complete successfully.

Based on my experience with Git errors, my intuition was 1) above had more probability. And the solution for the same was easy. You go back one or more commits on your local repo and force push the same to origin ignoring known or unknown changes (we are using TortoiseGit as our Git GUI). It has worked or a majority of times earlier with our Git repo corruptions.

However it failed this time.

The silver-lining was the error message was a bit more verbose than earlier:

git.exe push --progress "origin" master:master
Enumerating objects: 27, done.
Counting objects: 100% (27/27), done.
Delta compression using up to 8 threads
Compressing objects: 100% (16/16), done.
Writing objects: 100% (16/16), 5.13 KiB | 437.00 KiB/s, done.
Total 16 (delta 14), reused 0 (delta 0), pack-reused 0
remote: fatal: loose object 6f1d035e5085e391b479f74bb6fb25cce0a1e573 (stored in ./objects/6f/1d035e5085e391b479f74bb6fb25cce0a1e573) is corrupt
error: remote unpack failed: unpack-objects abnormal exit
To http://ourgitserver.com/OurRepo.git
! [remote rejected]       master -> master (unpacker error)
error: failed to push some refs to 'http://ourgitserver.com/OurRepo.git'

The part of the message remote: fatal: loose object 6f1d035e5085e391b479f74bb6fb25cce0a1e573 (stored in ./objects/6f/1d035e5085e391b479f74bb6fb25cce0a1e573) is corrupt obviously pointed there was a dangling commit on the remote repo. Our origin repo was completely stuck and nobody was able to do anything with it, meaning our build processes were failing too as they could not access the latest committed code.

A bit of a research as to how the problem originated brought forward the fact that it was caused by a bizarre power outage on the server. A dev was pushing the code to origin server when the electricity on the server snapped exactly in the middle of the push (coz of a backup generator malfunctioning).

I really would have expected this to be an easy fix with the solutions available on web, here, here and here etc. None of these worked. I hoped this would work, it involved a simple commit, push and pull by another dev. However the same failed miserably for me. This also looked like a really promising solution, again to no avail. git prune failed miserably too as did all its variations:

git gc --aggressive --prune=now

I finally thought this should do it. As it was a botched commit / push, I expected one or more files under .git/objects/ to be empty. I ran Windows size based search for empty files, to my surprise no file under .git/objects/ turned up empty. Finally, this link provided some insights which enabled me to cough up something to fix the issue.

So I first ran:

git fsck

It produced the following output:

error: unable to unpack header of H:/SomePath/SomeRepo/
objects/58/e58970c3bb0823ab8abbedc629f480417e47c7
error: 58e58970c3bb0823ab8abbedc629f480417e47c7: object corrupt or missing: H:/SomePath/SomeRepo/objects/58/e58970c3bb0823ab8abbedc629f
480417e47c7
error: inflate: data stream error (unknown compression method)
error: unable to unpack header of H:/SomePath/SomeRepo/
objects/6f/1d035e5085e391b479f74bb6fb25cce0a1e573
error: 6f1d035e5085e391b479f74bb6fb25cce0a1e573: object corrupt or missing: H:/SomePath/SomeRepo/objects/6f/1d035e5085e391b479f74bb6fb2
5cce0a1e573
error: inflate: data stream error (unknown compression method)
error: unable to unpack header of H:/SomePath/SomeRepo/
objects/e1/26a141beeba0c1347aca199fd5be4ce5153bf6
error: e126a141beeba0c1347aca199fd5be4ce5153bf6: object corrupt or missing: H:/SomePath/SomeRepo/objects/e1/26a141beeba0c1347aca199fd5b
e4ce5153bf6

I thought I was getting somewhere. I checked the following files:

1. objects/58/e58970c3bb0823ab8abbedc629f480417e47c7
2. objects/6f/1d035e5085e391b479f74bb6fb25cce0a1e573
3. objects/e1/26a141beeba0c1347aca199fd5be4ce5153bf6

All 3 of them were blank, but blank as in stuffed with spaces. None of them were a 0-byte file. And each had different number of spaces. I removed all the 3 files from the respective folders and ran git fsck again. This time it produced a different error:

error: refs/heads/master: invalid sha1 pointer e126a141beeba0c1347aca199fd5be4ce
5153bf6
error: HEAD: invalid sha1 pointer e126a141beeba0c1347aca199fd5be4ce5153bf6

This error I thought looked a lot familiar and manageable. So I opened packed-refs file in the git repo root (remember it was a bare repo on the server and I am performing all these operations on the origin server). And I immediately found the commit hash e126a141beeba0c1347aca199fd5be4ce5153bf6 against refs/heads/master (you should check against your desired branch). I looked at my local git repo, opened the git log for origin master branch, went back a single commit, which I knew was safe and everyone in the team had the same commit on their local copy pulled, and updated the hash of the commit to the packed-refs file on the server. Saved the file on the server, went back to my local machine, did a git pull and voila, all iz well 🙂

Was quite a scare though. Many people had suggested re-creating the repo from scratch by creating a new bare repo, and pulling / pushing against it. The same would have needed hours as it was a pretty big repo sizing in multiple GBs and would have impacted our build processes where there were multiple builds running against various branches of this repo.

In the end, the solution was quite simple, remove the blank (space stuffed files) and reset branch head to a known valid commit. There’s one interesting thing, the number of blank (space stuffed) files match the number of files that were being pushed to the origin server. So you should expect a differing number of empty files if you face the same issue. This might also depend upon how many commits and number of files in each commit that you are pushing that got botched during the push.

Happy troubleshooting!!!