So it happened recently that one of our in-house (Windows-based) servers which hosted most of our Git repositories went kaboom!! Upon inspection, it appeared that the power backup supporting the server was malfunctioning which caused abrupt shut-downs of our Windows Server a couple of times. And the last time it happened, someone was doing something with one of our most important Git repos hosted on this server. And that git repo’s master branch went completely inaccessible after reboot trying to do a Git Pull or Push from / to the repo respectively.
Anytime we tried to do a Pull / Push from / to the server, the same failed with the following error:
Could not read 8c68d0702af281a70d530d24e14b61d9def4b526
remote: fatal: bad tree object 8c68d0702af281a70d530d24e14b61d9def4b526
remote: aborting due to possible repository corruption on the remote side.
It initially appeared the solution would be very simple. We checked with all our devs who committed code to that repo, and figured out the one who last made a push to the repo successfully. We asked that dev to push again to the repo with Tortoise Git’s Force known changes (Git’s --force-with-lease) option. That Push succeeded and all other devs on the team were able to now pull and push from that repo.
However, as should be expected from a party-crasher, this issue refused to go away so easily. While Git pull / push worked for everyone, it failed on my specific machine. And I kept receiving the same error no matter what I did:
Could not read 8c68d0702af281a70d530d24e14b61d9def4b526
remote: fatal: bad tree object 8c68d0702af281a70d530d24e14b61d9def4b526
remote: aborting due to possible repository corruption on the remote side.
A very easy solution could have been just to re-clone my entire repo from the server again (or take a copy from one of my team members). However I really wanted to understand what was wrong with the remote repo and if there’s anything that could be done to salvage it.
So I switched my focus to our server now. Please note our server repo was a bare repo meaning it did not had any working tree.
I opened an elevated command prompt on the server and went to the repo’s base directory on the server. What I could make of from the error above was there was either an orphaned commit in the server branch (master branch in this case) or there was corrupted 0-byte file as many suggested online (like here).
However the master ref on the server (in info\refs or packed-refs) was not pointing to the bad object ref here (8c68d0702af281a70d530d24e14b61d9def4b526). In fact, a search of the entire bare git repo on the server for a reference to the commit 8c68d0702af281a70d530d24e14b61d9def4b526 returned 0-results.
Next I checked 8c folder on the server inside objects folder and there was no file with the name 68d0702af281a70d530d24e14b61d9def4b526 (object files in Git are stored inside a folder with the first 2 characters of the hash being the folder name inside objects folder and the remaining hash being the file name inside that folder). Unfortunately there was no such file either.
I tried and checked a lot of other things suggested online (like here and here) but none of the answers / suggestions worked.
I tried running git fsck on the server and to my utmost surprise even it failed with a similar error for repository corruption (I do not recall the exact error, nor do I have a copy of it handy. However I do recall the error had the same commit hash in it like the errors above).
My next attempt was was to try garbage clean the repo with git gc. Alas, the same failed horribly again with a similar repo corruption message.
My intuition then was to hard reset the bare repo to a commit a few days old, and then I would ask my team member to push to the repo again. So I executed:
git reset --hard <KNOWN-COMMIT-HASH-FROM-FEW-DAYS-AGO>
And what git gave me back was:
fatal: this operation must be run in a work tree
So, we cannot hard reset a bare git repo. What next. Quick googling gave me the following solution:
git update-ref HEAD <KNOWN-COMMIT-HASH-FROM-FEW-DAYS-AGO>
git gc still failed after this. However out of nowhere I decided to run git prune, which miraculously succeeded. Followed by which, git gc succeeded too, and so did git gc --aggressive --prune=now.
So, in a nut-shell, if your bare Git repo is facing corruption in a way where git gc or git prune are not working at all, the following steps should help to get the repo to a working state:
git update-ref HEAD <AN-OLDER-COMMIT-HASH>
git prune
git gc
Keep moving <AN-OLDER-COMMIT-HASH> backwards by a few commits if git prune does not immediately succeed after git update-ref. Once all the 3 steps complete successfully in order, you can push from your local repo to the remote repo to again bring it up-to-date with the latest commit.
Phew, a narrow escape 🙂
Recent Comments