It exists some cases (nix and guix origins for instance) when a snapshot
branch targets a revision that is missing in the archive.

So ensure to remove Web UI components related to missing objects (revision
and directory) as they will lead to errors when following the links they
contain.

Piwik script should only be included in production website but checking
the value of the DJANGO_SETTINGS_MODULE variable is not reliable enough
(swh-web in docker uses Django production settings for instance).

The vault API returns CoreSWHID, but mocks in tests returned strings...

This broke Guix's client implementation:
https://lists.gnu.org/archive/html/guix-devel/2021-09/msg00115.html

https://docs.djangoproject.com/en/3.2/ref/templates/builtins/#filesizeformat

Instead of hardcoding the list of visit types that can be used in
search queries, retrieve it dynamically using swh-search.

Also remove visit_types from global template context processor
and move it to seach related views only to avoid sending a
request to elasticsearch on every page load.

Related to T3441

Antoine Lambert authored 3 years ago and Antoine Lambert committed 3 years ago

Add a new Django view that can be embedded in an iframe to display
contents and directories archived by Software Heritage in a minimalist
Web UI.

The URL targetting that view is /embed/(swhid)/.

The view enables to navigate in a source tree and to display SWHIDs
of currently browsed objects. It also has a fixed header with a link
to softwareheritage.org and a link to browse current object on the
main archive website. Code fragment highlighting for content objects
is also supported if the input SWHID has a lines qualifier.

Related to T3489

This partially replaces the existing 'Show metadata' that only works for
metadata written directly on revision objects.

It's always allowed with the API; but still restricted on the UI for now,
while we make sure that are no major issues.

While pull request branches are hidden by default in Web UI, they should
remain browsable.

That commit fixes the error that was previously raised when attempting
to browse such a branch.

When working from home, I run swh-web remotely, so it's not on localhost.

Allow any host makes this simpler.

this property is only used for the metadata search.
Naming it search_config.backend is confusing as it lets suppose
it's the global search configuration

Web API links generated by the urlize_links_and_mails template filter
are removed by highlight.js when highlighting JSON response.

So ensure to restore them once JSON highlighting has been performed.

Those were missed by continous integration because the httpdomain warnings
were disabled on Jenkins (see D6193).

It will simplify the adding of new visit types for Save code now.

Related to D5992

Both Firefox and Chromium replace ':' (with spaces and underscores respectively)

Add new origins types that we now archived.

Update source code provider logos to png only in order to ease the
integration of future ones

Split origins in multiple categories: listed, legacy and deposited.

Add global origins counts per origin type.

Add detailed origins counts for each listed forge, those details are
hidden by default and can be displayed on demand through collapsible
elements.

Remove dead code.

Related to T3127

In order to ease the integration of new visit types for the "Save code now"
service, filter out those with no associated task type in scheduler db.

It means that a new visit type can be added to "Save code now" implementation
but it will not be available until the associated scheduler load task type
gets created. It enables to test the new visit type in dev environments
and deploy safely to production if the associated scheduler load task type
did not get created yet.

Also remove the internal endpoint to get savable visit types, those can
be simply added in Django template.

As that change broke a lot of tests due to scheduler being previously
mocked, most of the tests related to "Save code now" got improved by
removing the mocks and using the swh_scheduler pytest fixture.

Related to D5992

When a content cannot be displayed (for instance due to a too large size),
remove script tag from template to prevent execution errors.

Closes T3454

Only revisions for now. We'll add other object types after
we make sure it works properly.

Using a single example skips a lot of useful tests...

v1.0.0 changed the interface and we need the new one.

It's brittle, and not the right place.

This is to match changes in the vault interface, see D6112 for the rationale.

All existing queries still work, through "fallback" views.  There are
some minor changes in returned fields (removed obj_type, and replace
obj_id with swhid), but I don't think anyone used them (since that info
is already available to them before they send the request).

So tests don't have side-effects

It should be imported from hypothesis.strategies instead from the internal
private module.

This fixes Python tests when building debian buster package.

Running metadata tests with multiple hyothesis examples uncovered those
issues in api-1-raw-extrinsic-metadata-swhid Web API view:

  - RawExtrinsincMetaData only targetting core SWHIds must be provided as test inputs.

  - link-next URL for pagination was invalid.

  - next_page_token must be encoded before providing it to urlsafe_b64encode.

When running swh-web tests using 'make test-full', multiple hypothesis examples
are provided as test inputs instead of a single one when running 'make test'.
In that case some tests were failing mostly due to the fact they were not stateless
between test runs.

That commit fixes those tests execution and ensures stateless test runs by:

  - turning some hypothesis strategies into stateless ones

  - turning the archive_data fixture into a function scope one

  - using subtest fixture from pytest-subtesthack when it is required to reset
    the archive_data fixture for a test between hypothesis example runs

As a consequence, tests will be longer to execute as global state will be reseted
between each test.

Nevertheless, metadata related tests are still failing when running with multiple
hypothesis examples, fix will be handled in next commit.

Related to T1695

Contents with mimetype starting with application/ but with
no detected programming language were not processed for
display but they can be regular text files.

Closes T3486