Produce and encrypt recovery bundles

swh alter remove should produce backups with what has been removed from the archive. These backups should contain enough information to restore removed data in case of a mistake. Because they might contain sensitive data, they should be encrypted using a public encryption key. (The decryption key will be split between multiple entities.)

• Design a format for the recovery bundles (!2 (merged))
• Implement basic operations (creation, extraction, key management) to handle recovery bundles (!3 (merged))
• Implement backing up objects to recovery bundles from swh alter remove (!3 (merged))

changed milestone to %Tooling for takedown notices [Roadmap - Collect]

assigned to @lunar

unassigned @lunar

added activity::Epic label

changed title from Encrypt removal manifests to Produce and encrypt removal manifests

changed the description

changed title from Produce and encrypt removal manifests to Produce and encrypt removal records

changed the description

Open questions:

• What should be the proper format for the records?
• What needs to go in the manifest?
• Could we reuse the VITAM format? See swh/meta#3415
• What are the security requirements?

mentioned in merge request !1 (merged)

The Vitam format (SIP) seems terribly over-engineered for our needs. It also would require to implement both an exporter and an importer.

swh/devel/swh-dataset> contains an exporter for the ORC format that could be pretty appropriate. Do we have an importer already?

mentioned in issue #7 (closed)

mentioned in issue swh/meta#4803 (closed)

changed the description

changed title from Produce and encrypt removal records to Produce and encrypt removal backups

Updated to use the terminology decided during the 2023-06-05 meeting.

mentioned in merge request !2 (merged)

Readings for the encryption aspects:

• The PGP Problem
• age specification
• YubiKey plugin for age clients
• pyrage announcement

age-plugin-yubikey seems to have a pretty good UX.

$ age-plugin-yubikey 
✨ Let's get your YubiKey set up for age! ✨

This tool can create a new age identity in a free slot of your YubiKey.
It will generate an identity file that you can use with an age client,
along with the corresponding recipient. You can also do this directly
with:
    age-plugin-yubikey --generate

If you are already using a YubiKey with age, you can select an existing
slot to recreate its corresponding identity file and recipient.

When asked below to select an option, use the up/down arrow keys to
make your choice, or press [Esc] or [q] to quit.

🔑 Select a YubiKey: Yubico YubiKey OTP+FIDO+CCID 00 00 (Serial: 5229836)
🕳️  Select a slot for your age identity: Slot 1 (Empty)
📛 Name this identity [age identity TAG_HEX]: 
🔤 Select a PIN policy: Never  (A PIN is NOT required to decrypt)
👆 Select a touch policy: Cached (A physical touch is required for decryption, and is cached for 15 seconds)
Generate new identity in slot 1? yes

🎲 Generating key...

Enter PIN for YubiKey with serial 5229836 (default is 123456): [hidden]

🔏 Generating certificate...
👆 Please touch the YubiKey

📝 File name to write this identity to: age-yubikey-identity-c3fa08e1.txt

✅ Done! This YubiKey identity is ready to go.

🔑 Here's your shiny new YubiKey recipient:
  age1yubikey1q2e37f74zzazz75mtggzql3at66pegemfnul0dtd7axctahljkvsqezscaq

Here are some example things you can do with it:

- Encrypt a file to this identity:
  $ cat foo.txt | rage -r age1yubikey1q2e37f74zzazz75mtggzql3at66pegemfnul0dtd7axctahljkvsqezscaq -o foo.txt.age

- Decrypt a file with this identity:
  $ cat foo.txt.age | rage -d -i age-yubikey-identity-c3fa08e1.txt > foo.txt

- Recreate the identity file:
  $ age-plugin-yubikey -i --serial 5229836 --slot 1 > age-yubikey-identity-c3fa08e1.txt

- Recreate the recipient:
  $ age-plugin-yubikey -l --serial 5229836 --slot 1

💭 Remember: everything breaks, have a backup plan for when this YubiKey does.

With the above settings, running rage -d -i age-yubikey-identity-c3fa08e1.txt will make the YubiKey blink. A touch will enable the decryption to perform. No messages appear on the console though.

Creating an encrypted archive protected by a passphrase from Python, that can later be browsed through any terminal tools:

create.py:

import subprocess
import zipfile

TEST_CONTENT = "Salut! ⏚"

logger = logging.getLogger(__name__)

def main():
    age_proc = subprocess.Popen(["/home/lunar/.cargo/bin/rage", "-p", "-o", "bundle.zip.age", "-"], stdin=subprocess.PIPE, env={b"PINENTRY_PROGRAM": b"/tmp/pinentry-auto"})
    with zipfile.ZipFile(age_proc.stdin, "w") as bundle:
        bundle.writestr("test.txt", TEST_CONTENT.encode("utf-8"))
    age_proc.stdin.close()
    age_proc.wait()

if __name__ == "__main__":
    main()

Also: pinentry-auto

Then we can do:

$ python3 create.py
$ mkdir mnt
$ rage-mount -t zip bundle.zip.age mnt/
$ cat mnt/test.txt      # In another terminal, rage-mount runs in foreground
Salut! ⏚

Using python-shamir-mnemonic, splitting a secret in 2 shares:

$ shamir create -S deadbeefdeadbeefdeadbeefdeafbeef 2of2
Using master secret: deadbeefdeadbeefdeadbeefdeafbeef
Group 1 of 1 - 2 of 2 shares required:
clock recover academic acid disaster tension regular human grin force faint museum satoshi listen royal stadium river oasis decent benefit
clock recover academic agency course unkind traffic traveler lyrics armed august firefly empty cards listen darkness dream easy source spray

Encrypting them for two different YubiKeys:

echo -n "clock recover academic acid disaster tension regular human grin force faint museum satoshi listen royal stadium river oasis decent benefit" | rage -r age1yubikey1q2e37f74zzazz75mtggzql3at66pegemfnul0dtd7axctahljkvsqezscaq -o share-c3fa08e1.age
echo -n "clock recover academic agency course unkind traffic traveler lyrics armed august firefly empty cards listen darkness dream easy source spray" | rage -r age1yubikey1q04k5fs8cz6kypt7vjetl2gc2qtpz9lyzpxrvv2agzt6h8n3awmzk9sgd8v -o share-4c1bc5f1.age

decrypt.py:

import subprocess

def main():
    mnemonic_str_1 = subprocess.check_output(["/home/lunar/.cargo/bin/rage", "-d", "-i", "age-yubikey-identity-c3fa08e1.txt", "share-c3fa08e1.age"]).decode('utf-8')
    mnemonic_str_2 = subprocess.check_output(["/home/lunar/.cargo/bin/rage", "-d", "-i", "age-yubikey-identity-4c1bc5f1.txt", "share-4c1bc5f1.age"]).decode('utf-8')
    print(recover([mnemonic_str_1, mnemonic_str_2]).hex())

def recover(share_mnemonic_strs):
    from shamir_mnemonic.recovery import RecoveryState
    from shamir_mnemonic.share import Share

    recovery_state = RecoveryState()
    for mnemonic_str in share_mnemonic_strs:
        share = Share.from_mnemonic(mnemonic_str)
        recovery_state.add_share(share)
    assert recovery_state.is_complete()
    # no passphrase has been set so leave it empty
    passphrase = b""
    return recovery_state.recover(passphrase)

if __name__ == "__main__":
    main()

It works:

$ python3 decrypt.py 
deadbeefdeadbeefdeadbeefdeafbeef

Each YubiKey will blink and require a tap before we get the secret.

assigned to @lunar

changed title from Produce and encrypt removal backups to Produce and encrypt recovery bundles

changed the description

mentioned in commit lunar/swh-storage@f6a90533

mentioned in merge request swh-storage!1070 (merged)

mentioned in commit lunar/swh-storage@abbe06db

changed the description

Implemented in !3 (merged).

closed