loader: Add support for dumb HTTP transfer protocol

Git supports two HTTP based transfer protocols to exchange data between two repositories: the dumb protocol and the smart protocol.

Nowadays, the smart protocol is a common method of transferring data because it is more efficient but there is still some git servers in the wild (mostly cgit) that only support the dumb protocol.

Unfortunately the dulwich package does not support such protocol so this kind of git repository could not be loaded into the archive.

That commit adds support to load such git repository by fetching objects according to the dumb HTTP transfer protocol specification.

Below are some useful resources on the subject:

• Git HTTP transfer protocols specification
• Git Internals - Transfer Protocols
• Git on the Server - The Protocols
• Git Mirror Anywhere using the Dumb Http Protocol

I have successfully tested the loading with numerous git repositories only supporting dumb protocol, below are some of them found by inspecting the failed save code now requests for git in production:

• http://hdiff.luite.com/cgit/AC-Boolean
• https://git.stoutner.com/UltraPrivacy.git
• https://git.raptorengineering.com/git/hqemu
• https://forge.touhey.org/casio/libcasio.git
• https://forge.frm2.tum.de/cgit/cgit.cgi/frm2/mira/tlibs.git
• https://git.tuxfamily.org/harmonist/harmonist.git
• https://git.openembedded.org/openembedded-core
• https://depp.brause.cc/nov.el.git
• https://www.6809.org.uk/git/asm6809.git/

Debian packaging has also been successfully tested, git must added as build dependency though as it is used in newly added test suite.

Related to #2489 (closed)

Test Plan

A new test suite for the dumb protocol support has been added. The sample repository used in tests is bare cloned and served by a local HTTP server as specified by the dumb protocol.

---

Migrated from D6342 (view on Phabricator)