Add new origins types that we now archived.

Update source code provider logos to png only in order to ease the
integration of future ones

Split origins in multiple categories: listed, legacy and deposited.

Add global origins counts per origin type.

Add detailed origins counts for each listed forge, those details are
hidden by default and can be displayed on demand through collapsible
elements.

Remove dead code.

Related to T3127

In order to ease the integration of new visit types for the "Save code now"
service, filter out those with no associated task type in scheduler db.

It means that a new visit type can be added to "Save code now" implementation
but it will not be available until the associated scheduler load task type
gets created. It enables to test the new visit type in dev environments
and deploy safely to production if the associated scheduler load task type
did not get created yet.

Also remove the internal endpoint to get savable visit types, those can
be simply added in Django template.

As that change broke a lot of tests due to scheduler being previously
mocked, most of the tests related to "Save code now" got improved by
removing the mocks and using the swh_scheduler pytest fixture.

Related to D5992

When a content cannot be displayed (for instance due to a too large size),
remove script tag from template to prevent execution errors.

Closes T3454

Only revisions for now. We'll add other object types after
we make sure it works properly.

Using a single example skips a lot of useful tests...

v1.0.0 changed the interface and we need the new one.

It's brittle, and not the right place.

This is to match changes in the vault interface, see D6112 for the rationale.

All existing queries still work, through "fallback" views.  There are
some minor changes in returned fields (removed obj_type, and replace
obj_id with swhid), but I don't think anyone used them (since that info
is already available to them before they send the request).

So tests don't have side-effects

It should be imported from hypothesis.strategies instead from the internal
private module.

This fixes Python tests when building debian buster package.

Running metadata tests with multiple hyothesis examples uncovered those
issues in api-1-raw-extrinsic-metadata-swhid Web API view:

  - RawExtrinsincMetaData only targetting core SWHIds must be provided as test inputs.

  - link-next URL for pagination was invalid.

  - next_page_token must be encoded before providing it to urlsafe_b64encode.

When running swh-web tests using 'make test-full', multiple hypothesis examples
are provided as test inputs instead of a single one when running 'make test'.
In that case some tests were failing mostly due to the fact they were not stateless
between test runs.

That commit fixes those tests execution and ensures stateless test runs by:

  - turning some hypothesis strategies into stateless ones

  - turning the archive_data fixture into a function scope one

  - using subtest fixture from pytest-subtesthack when it is required to reset
    the archive_data fixture for a test between hypothesis example runs

As a consequence, tests will be longer to execute as global state will be reseted
between each test.

Nevertheless, metadata related tests are still failing when running with multiple
hypothesis examples, fix will be handled in next commit.

Related to T1695

Contents with mimetype starting with application/ but with
no detected programming language were not processed for
display but they can be regular text files.

Closes T3486

Useful for the dev environments (a connection to the vpn is needed)

This diff introduces a new config parameter "enable_ql"

If it's set to yes/True, the swh-search query language can be used for the archive search

While using the /origin/search API endpoint, "use_ql=true" can be
passed to use the url_pattern as the search query language input

api_raw_extrinsic_metadata_swhid: Make the docstring consistent with api_raw_extrinsic_metadata_swhid_authorities

Reword some sentences and add step showing SWHID with lines qualifer.

Fix redirection at end of tour when started from homepage.

Related to T3437

D5982 did not fix the flaky cypress test.

Removing cy.intercept hack required some times ago seems
to do the trick.

It will enable to exploit those data in the archive coverage widget
implementation to get deposit counts by origins type.

Related to T3127

Footer is put in cache by Varnish so an old version will be displayed
while the layout template is not updated.

So only display the version in development mode.

Antoine Lambert authored 3 years ago and Antoine Lambert committed 3 years ago

Add a Web UI guided tour implemented using the intro.js library.

The tour showcases the main browsing features offered by Software
Heritage web application by highlighting the relevant DOM elements
and displaying help texts in tooltips.

The tour can span multiple web pages and steps are configured in
a YAML file for help texts writing commodity.

Currently, the homepage, the main origin browsing view and content
browsing view are covered by the tour.

Related to T3202

Antoine Lambert authored 3 years ago and Antoine Lambert committed 3 years ago

This new major version of highlight.js brings notable improvements in
terms of language parsing.

As the github theme we were using has been updated to match the current
github.com one, we now use a custom theme adapted from the old github
one to keep the same look and feel for code highlighting as before.

Closes T3368

The visit is already recursive, we don't need to add this explicit
recursion.

Children nodes can be dispatched to visit_field_name and
visit_field_body instead.

Motivation:

This commit started as a simple change: I wanted to replace:

```
`<type> <IRI>`
```

with:

```
``<type> <IRI>``
```

Unfortunately, this syntax looks too much like XML for its own good,
so it was stripped by the `process_paragraph` method, because it reads
the docutils pseudo-XML representation and strips every tag it doesn't
know about.
(I'm saying pseudo-XML, because my poor `<type> <IRI>` string was not
escaped with XML entities, so it was in fact undistinguishable from
actual XML tags).

Changes:

Therefore, stops using the XML-like string representation of docutils
trees, and visits tree nodes directly instead.
Conveniently, this is already in a node visit, so we can reuse that;
simply by iterating recursively instead of stopping the recursion
as soon as we see a known node (ie. the visitors actually visited
only nodes very close to the root).

This means that we needed to add methods to handle each node type,
and produce its ReST output. And since we don't have a global view
anymore, we need to return the produced ReST instead of appending
directly to `self.data["description"]`, because handlers of parent
nodes may need to re-indent their children's output.o

This results in cleaner code (and also closer to what we expect from
a visitor transformer), so it's a win too.

This has some other nice side-effects:

* our custom role code is now neatly restricted in `visit_problematic`,
  so it can't overflow, because docutils runs `visit_problematic` with
  *only* the role's string as child
* it detects unexpected nodes, such as the `title_reference` roles,
  which is usually produced when accidentally using single-backquotes
  instead of double-backquotes to wrap inline code blocks (it happens
  a lot when one is used to markdown)

When clicking on the "Go to origin" icon, open a new browser tab to
keep Software Heritage one.

Cypress commands should not be invoked in a loop as it is an anti-pattern
that can produce flaky tests.

The issue was spotted by a test that start failing since
rDSEA2e1fb863871c82c5561b6a1cbd94c7064a1a392d got landed.

Hypothesis origin strategy in swh-model has been recently updated
to generate IRIs instead of just URIs.

It allows easier discovery for humans, as clients need to know the authority
type/url to list metadata on a given SWHID.