- Jun 26, 2019
-
-
Antoine R. Dumont authored
Related D1635
-
Antoine R. Dumont authored
instead of converting that column as a string As a side effect, bitbucket wise, we provided improperly the after query parameter as a date not url encoded. This resulted in improper api response from bitbucket's (we received from time to time the same next index as the current one). Related T1826
- Jun 25, 2019
-
-
Antoine R. Dumont authored
If nothing has been done prior to a full relisting, there is actually nothing to list. So the relister in question does nothing. In that context, the IndexingLister class's `db_partition_indices` method now returns an empty list instead of raising a ValueError when there is nothing to list. Related T1826 Related e129e48c
-
Antoine R. Dumont authored
Prior to this commit, indexing lister instances were flushing every packet of 20. This can now be defined per sub classes.
-
Antoine R. Dumont authored
- Remove unneeded return instructions - Clarify tests code regarding request_index computations
-
Antoine R. Dumont authored
-
- Jun 21, 2019
-
-
Antoine R. Dumont authored
This is independent but still, it somehow fixes the issue occurring on T1826. Related T1826
-
Antoine R. Dumont authored
Related T1826
-
- Jun 20, 2019
-
-
Antoine R. Dumont authored
It's redundant with the swh modules in itself.
-
Archit Agrawal authored
Add new page in lister tutorial which guides through the process of running a new lister in docker.
-
Archit Agrawal authored
Add testing section for lister. Also added how to run a new lister section which elaborates about the steps required to run the now lister in docker
-
- Jun 19, 2019
-
-
Archit Agrawal authored
Change origin type from 'gnu' to 'tar'
-
vlorentz authored
They are useless.
-
Nancy Chauhan authored
-
- Jun 18, 2019
-
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
Related T1824 Related P438
-
Antoine R. Dumont authored
As phabricator is an "instance" lister (there exists multiple instances of phabricator in the wild), we need to reference that information. In effect, this aligns phabricator lister with for example the gitlab one. Related T1801 Related P434
-
Antoine R. Dumont authored
Related P434
-
- Jun 17, 2019
-
-
Antoine R. Dumont authored
Prior to this commit, this expected the api.token to be provided at task initialization. That behavior has been kept for cli purposes. It's no good for production purposes though (as this leaks the credentials in the scheduler db). So now, the credentials is fetched from the lister's configuration file as the other listers do. Another change is the authentication mechanism which is slighly different. It's not using a basic `auth` mechanism. It's expecting an `api.token` query parameter so the `request_params` is overriden to provide that. Related T1809
-
- Jun 14, 2019
-
-
Antoine R. Dumont authored
This should have been removed along with the code in b8162127. The request authentication has been reworked so that all listers use the same credentials dict. Related b8162127 Related T1772
-
- Jun 13, 2019
-
-
Antoine R. Dumont authored
-
Antoine R. Dumont authored
Prior to this commit, lister did create origins as well in the archive. Now, we only schedule new origins for ingestion.
-
- Jun 11, 2019
-
-
Archit Agrawal authored
Add a lister to list all the CRAN packages . It uses the build-in API in R language to list the packages and get their metadata. Closes T1709
-
Archit Agrawal authored
Added a function which will derive the extension from filename and check if the fie extension match the type of file that is to be archived.
-
- Jun 08, 2019
-
-
Archit Agrawal authored
As discussed in T1389 to ingest all packages using base loader, it need a list of all the tarballs for a pakage. Hence modifified lister to recursively list all the tarballs for a package with their last updated time.
-
Archit Agrawal authored
Previously gnu lister was using same code as that of tarball loader to download, unzip and read tree.json file. To make the code consise the downloading method is changed to requests library.
-
Archit Agrawal authored
Implement first pass of gnu lister to list all the packages present in https://ftp.gnu.org/ Add GNU lister in README and cli.py Closes T1722
-
Archit Agrawal authored
Some of the new listers like GNU and CRAN do not follow the conventional way of making an HTTP request, hence they do not need some of the methods which are usually needed by in conventional HTTP request. But those method are marked abstractmethod in the core making them necessary to be present. So it is in best to remove abstractmethod to increase the readability of those listers.
-
- May 29, 2019
-
-
Antoine R. Dumont authored
This becomes a dictionary of key <lister-name>, value a dict of key <instance-name>, value list of dict username/password. Related T1772
-
- May 22, 2019
-
-
David Douard authored
also add a cli group named 'lister' for the sake of consistency with other swh packages and rename the command as 'db-init', like: swh lister db-init LISTER [...]
-
- May 21, 2019
-
-
Antoine Lambert authored
Related T1508
-
- May 20, 2019
-
-
David Douard authored
This is (hopefully) a temporary fix that can be removed as soon as https://github.com/pypa/pip/issues/6239 is fixed, probably thanks to https://github.com/pypa/pip/issues/988
-
- May 15, 2019
-
-
Archit Agrawal authored
Add a lister of all hosted repositories on a Phabricator instance Closes T808
-
Antoine Lambert authored
-
- May 14, 2019
-
-
Antoine Lambert authored
-
- May 13, 2019
-
-
Antoine Lambert authored
-
- Apr 12, 2019
-
-
Nicolas Dandrimont authored
-
- Apr 11, 2019
-
-
David Douard authored
the exception can also be an AttributeError. Also do not reraise this exception (in github/tasks.py). This promise saving feature is used for tests.
-
- Apr 10, 2019
-
-
Antoine Lambert authored
Related T1508
-