Commits · e02440b8640ea8476806ea4ad2c984034afea411 · Platform / Development / snippets

May 25, 2018
- Add snippets to ease the archival of existing tasks · e02440b8
  Antoine R. Dumont authored 6 years ago
  
  Related T1071
  Verified
  
  e02440b8
May 04, 2018
- seirl: add crossminer vault scheduler · 7caa7e70
  Antoine Pietri authored 7 years ago
  
  7caa7e70
Apr 26, 2018
- seirl: dedup: send sql inserts in batch · 300d534c
  Antoine Pietri authored 7 years ago
  
  300d534c
Apr 13, 2018
- morane/crossminer_launch: Make the script use starting range · 88eca5dd
  Antoine R. Dumont authored 7 years ago
  
  Unverified
  
  88eca5dd
- morane/crossminer_launch: Fix origin_scan function to use range · a546af39
  Antoine R. Dumont authored 7 years ago
  
  + add docstring
  Unverified
  
  a546af39
- morane/crossminer_launch: Fix pep8 violations · 7fbe9a1d
  Antoine R. Dumont authored 7 years ago
  
  Unverified
  
  7fbe9a1d
- Add crossminer_launch snippet · 777ab0c0
  Morane Otilia Gruenpeter authored 7 years ago
  
  777ab0c0
Apr 06, 2018
- seirl: swh-dedup: working prototype with preseeded methods · 9e7eb6aa
  Antoine Pietri authored 7 years ago
  
  9e7eb6aa
- seirl: swh-dedup: working prototype with preseeded methods · 57c3c638
  Antoine Pietri authored 7 years ago
  
  57c3c638
- seirl: ase-vault-test: update get_all_trees and add fetch_vault · ff7d7338
  Antoine Pietri authored 7 years ago
  
  ff7d7338
Mar 26, 2018
- seirl: ase-vault-test: add send_vault · ee08a2a4
  Antoine Pietri authored 7 years ago
  
  ee08a2a4
- seirl: ase-vault-test: add github queries · 028e25c3
  Antoine Pietri authored 7 years ago
  
  028e25c3
Mar 21, 2018
- Use last 50 characters to group by exception message · d550280d
  Antoine R. Dumont authored 7 years ago
  
  Unverified
  
  d550280d
- scheduling: Adapt mercurial/svndump/svn arguments setup · f30a6764
  Antoine R. Dumont authored 7 years ago
  
  Verified
  
  f30a6764
Mar 14, 2018
- seirl: add swh-dedup · b496be4e
  Antoine Pietri authored 7 years ago
  
  b496be4e
Feb 22, 2018
- add SQL snippets to dump (meaningful parts of) the origin table · d43f2eaf
  Stefano Zacchiroli authored 7 years ago
  
  d43f2eaf
- swh-dedup: fix db name in driver shell script · 3d51335f
  Stefano Zacchiroli authored 7 years ago
  
  3d51335f
Feb 12, 2018
- group_by_exception: Add mercurial loader type · f2dbd149
  Antoine R. Dumont authored 7 years ago
  
  Related T682
  Unverified
  
  f2dbd149
- kibana_fetch_logs: Add configuration abilities to fetching tool · 1cb7d409
  Antoine R. Dumont authored 7 years ago
  
  Related T682 Related P221
  Unverified
  
  1cb7d409
Feb 09, 2018
- scheduling: Fix mercurial arguments setup · 26ea29b2
  Antoine R. Dumont authored 7 years ago
  
  Related T329
  Verified
  
  26ea29b2
- scheduling: Fix default case scheduling · 4d182dbe
  Antoine R. Dumont authored 7 years ago
  
  Verified
  
  4d182dbe
Jan 12, 2018

scheduling: Check queues' state to trigger new task scheduling · 68c7d748

Use case: indexers.  To send new tasks for scheduling to the
orchestrator queue (input queue), we need to check the mimetype and
license indexers queues (not only the mimetype one).

Verified

68c7d748

Jan 11, 2018
- create_snapshots: be more efficient · 2c49fd80
  Nicolas Dandrimont authored 7 years ago
  
  2c49fd80
Jan 08, 2018
- swh-dedup-blocks: add sample driver script (run.sh) · 998d88cb
  Stefano Zacchiroli authored 7 years ago
  
  to streamline running experiments with different parameters
  998d88cb
- swh-dedup-blocks: add effective average block size to summary · 9b50858a
  Stefano Zacchiroli authored 7 years ago
  
  9b50858a
- swh-dedup-blocks: fix typo in avg block size configuration · 75e0a2d6
  Stefano Zacchiroli authored 7 years ago
  
  also, updated Rabin fingerprint parameters for better compression rates
  75e0a2d6
Jan 07, 2018
- swh-dedup-blocks: multiprocess implementation · 359fea82
  Stefano Zacchiroli authored 7 years ago
  
  down to 25m for linux.git, from 1h for the single process version, from 14h for the sqlite3 version
  359fea82
- swh-dedup-blocks: use postgres to store deduplication results · b9d742ee
  Stefano Zacchiroli authored 7 years ago
  
  instead of sqlite3 even if we're not using COPY, it is *much* faster
  b9d742ee
- move swh-dedup-blocks.py to dedicated dir · 567ad94a
  Stefano Zacchiroli authored 7 years ago
  
  567ad94a
- swh-dedup-blocks: new script to play with (rabin) deduplication · e26a6cee
  Stefano Zacchiroli authored 7 years ago
  
  e26a6cee
Jan 04, 2018
- sql/anon-dump: fix typo in SQL statement · d6ccd489
  Stefano Zacchiroli authored 7 years ago
  
  d6ccd489
Jan 03, 2018
- sql/anon-dump: fix bytea output format · d180b165
  Stefano Zacchiroli authored 7 years ago
  
  also, add driver shell script
  d180b165
- check-in id-to-path, to have it have a stable place · 10f626fd
  Stefano Zacchiroli authored 7 years ago
  
  10f626fd
- SQL queries (+ script) to produce an email-anonymized dump of the DB · 70bd9cbf
  Stefano Zacchiroli authored 7 years ago
  
  70bd9cbf
Dec 22, 2017
- Add snapshot migration snippet · 0118a624
  Nicolas Dandrimont authored 7 years ago
  
  0118a624
Dec 21, 2017
- Prepare googlecode mercurial origins scheduling · 4b224005
  Antoine R. Dumont authored 7 years ago
  
  Related T682
  Verified
  
  4b224005
- Update filtering tool to integrate the nature in the built url · 8bea10f0
  Antoine R. Dumont authored 7 years ago
  
  Related T682
  Unverified
  
  8bea10f0
Dec 14, 2017
- filter_path_clash: Update filtering path clash tool · 5c8391fa
  Antoine R. Dumont authored 7 years ago
  
  This will help to filter the origins that should not be googlecode ones but eclipselabs or apache-extras one
  Unverified
  
  5c8391fa
Dec 13, 2017
- volatile scheduling: Permit batch size override from cli · b85d77ec
  Antoine R. Dumont authored 7 years ago
  
  Related T864
  Verified
  
  b85d77ec
- volatile scheduling: Improve state information output · 3ee2a9cb
  Antoine R. Dumont authored 7 years ago
  
  Related T864
  Verified
  
  3ee2a9cb