dumb: Add support for tree and blob refs

Previously the dumb git loader was only considering refs targeting
commits and tags but those can also occasionally target blobs and
trees, so ensure to support such refs.

Fixes #4756.