Repository navigation
feat(sdk): add an account keystore and recovery-phrase accounts - #90
Conversation
Add aleo.keystore, which stores an Aleo account in ~/.aleo/accounts/<name>.json and resolves it again on later runs: - save, load, info, protect, export and migrate, with a KeystoreError carrying a stable code. - Four storage types: plaintext (the default, an owner-only file), encrypted (a password stretched with scrypt), keychain (the macOS login keychain) and none (the address only). - One lookup order: an explicit key, ALEO_PRIVATE_KEY, ALEO_PRIVATE_KEY_FILE (with the SHIELD_SWAP_ names as aliases), the account file, then a caller-supplied legacy key. A key from the environment is never written to disk, and lookup never generates one. - python -m aleo.keystore with create, import, show, protect, export and migrate. Commands that print a key or a recovery phrase run only at a terminal. Add recovery-phrase support to the account module: from_mnemonic, generate_mnemonic and create_with_mnemonic, using the derivation of the Shield wallet and the TypeScript SDK. The BIP-39 English word list ships as package data. The account file format and the sample files under tests/fixtures/keystore-vectors are shared byte for byte with the TypeScript SDK, so either SDK opens accounts the other stored.
- Create a new account file exclusively, through a hard link, so two processes storing different accounts under one name cannot replace each other's file. A save that loses the race removes the Keychain entries it wrote. - Remove Keychain entries written for an account file that could not be created. - protect: when a step after the new file is written fails, restore the original file and remove a Keychain key entry written for the new storage. Bring the saved password in line before the old Keychain key is removed, so a failure there leaves the original account working. - migrate: remove both key fields when a legacy file holds the same key in each, and refuse with KEY_CONFLICT when they hold different keys.
- Skip the owner-only re-mode of an account file where os.fchmod does not exist. Windows has no fchmod and does not enforce these modes. - Keep the shared sample files and the BIP-39 word list at LF line endings on every platform, since they are checked byte for byte. - Drop a test that asserted the package under test was imported from the source tree, which does not hold for an installed package.
iamalwaysuncomfortable
left a comment
There was a problem hiding this comment.
Found two issues that should be fixed before merging: shared Keychain entries can be deleted by another account file, and saved passwords collide between encrypted files for the same address.
Validation: the focused keystore, CLI, and mnemonic suite passed (353 passed, 1 skipped). Both issues were reproduced with a simulated Keychain; the real Keychain was untouched.
| def remove_written() -> None: | ||
| for service in written: | ||
| try: | ||
| delete_keychain(service, account.address) |
There was a problem hiding this comment.
[P1] Preserve shared Keychain entries during cleanup
Keychain entries are shared by address, but this cleanup assumes the failed save owns the entry. Reproduced: save an account as original with Keychain storage, then save it under another name while simulating a disk-write failure. The cleanup deletes the existing key, and loading original now fails with KEYCHAIN_ITEM_NOT_FOUND. This can erase the only persisted copy. The deletion in protect() also breaks other files sharing the entry. Preserve pre-existing/shared entries when rolling back or changing storage.
There was a problem hiding this comment.
Fixed in 7642b77. write_keychain now reads the entry first and reports whether the call created it, and rollback removes only entries the call created. An entry that already holds the key is not rewritten.
For protect(), the old key entry is kept when another account file in the same directory still points at it. Account files in other directories are not visible to the keystore, so a file elsewhere that shares the entry is still affected when the last local file leaves Keychain storage.
The same commit also puts the removal of the old secret inside the rollback, matching the fix in ProvableHQ/veil#221.
| key = dict(encrypt_key(private_key, password)) | ||
| if save_password and keychain_available(): | ||
| write_keychain(PASSWORD_SERVICE, address, password_to_keychain_value(password), location.name) | ||
| if written is not None: | ||
| written.append(PASSWORD_SERVICE) |
There was a problem hiding this comment.
[P2] Scope saved passwords to the encrypted file
The saved-password entry is keyed only by address, although separate files for that address can use different passwords. Reproduced: save the same account under two names with different passwords and save_password=True; both saves succeed, but loading the first now raises WRONG_PASSWORD because the second overwrote its saved password. That failure also prevents fallback to a terminal prompt. Associate saved passwords with the encrypted file rather than the address alone.
There was a problem hiding this comment.
Fixed in 7642b77 with a file-specific identity. The saved-password entry is now named <address>-<salt in hex>, where the salt is the encrypted file's own scrypt salt. Two files for one address have separate saved passwords, in any directory, and a load never consults another file's entry, so the first file's unattended load keeps working and a file with no saved password still reaches the terminal prompt. The account file format is unchanged.
Tests: test_gives_each_encrypted_file_its_own_saved_password, test_does_not_consult_the_saved_password_of_another_file_for_the_same_address.
A Keychain key entry is found by address, so every account file that holds one key in the Keychain points at the same entry. The keystore treated an entry as owned by the file being written, and could remove a key another file relied on. - Report whether a Keychain write created its entry, and undo only the entries a call created. A save that fails after finding an existing key entry leaves it in place, and an entry that already holds the key is not rewritten. - Give each encrypted file its own saved password. The entry is named after the address and the file's salt, so two files that hold one account under different passwords no longer overwrite each other's saved password, and a load never consults another file's. - protect: undo every step together. The new file, its saved password and the removal of the old secret now share one rollback, so a failed rotation keeps the original saved password and a failed removal of the old key restores the original storage. - Keep a Keychain entry that a neighbouring account file in the same directory still uses when another file leaves that storage.
iamalwaysuncomfortable
left a comment
There was a problem hiding this comment.
Re-reviewed 7642b77. The saved-password collision is fixed, and a failed save now preserves a Keychain entry that existed before the call. Shared Keychain deletion still has two failure cases: an overlapping writer can adopt a newly created entry before rollback, and account files in other directories are omitted from the ownership check. Both findings should be addressed before merging.
Validation: the focused keystore, CLI, and mnemonic suite passed (362 passed, 1 skipped). Both remaining failures were reproduced with a simulated Keychain; the real Keychain was untouched.
| except BaseException: | ||
| # No account file was created, so nothing points at the entries made here. | ||
| _remove_entries(created) |
There was a problem hiding this comment.
[P1] Preserve entries adopted by another writer before rollback
Creating an entry does not guarantee exclusive ownership until this call finishes. Reproduced this interleaving: writer A creates the Keychain entry; writer B successfully saves the same key under another name; A's file write then fails. This rollback deletes the entry B now relies on, leaving B's successful save unable to load (KEYCHAIN_ITEM_NOT_FOUND) and removing the only persisted key. Creation, adoption, and cleanup need coordinated ownership handling.
There was a problem hiding this comment.
Fixed in dc715a6 by removing the sharing, so there is nothing to adopt. Each account file now gets a Keychain key entry of its own, named <address>-<16 random bytes in hex> and recorded in the file's key.account. Writer A and writer B create different entries; when A fails it removes only the one it made, and B's is untouched.
Test: test_does_not_let_a_failed_save_remove_the_key_of_a_save_that_succeeds_alongside_it runs the second save between the first save's Keychain write and its file write.
| if current == "keychain" and storage != "keychain": | ||
| if not _neighbour_uses( | ||
| location, lambda file: file.get("address") == address and _key_of(file).get("type") == "keychain" | ||
| ): | ||
| delete_keychain(KEY_SERVICE, address) |
There was a problem hiding this comment.
[P2] Do not infer global ownership from neighbouring files
The Keychain entry is shared across directories, but this check searches only the selected directory. Reproduced: save the same account with Keychain storage in two project directories, then protect the first as plaintext. The operation succeeds but deletes the second project's key entry; its next load fails with KEYCHAIN_ITEM_NOT_FOUND. Copied encrypted files in another directory similarly lose their saved password. Preserve shared entries when their other users cannot be determined, or track ownership across directories.
There was a problem hiding this comment.
Fixed in dc715a6 the same way. Two account files stored separately no longer point at one entry, in the same directory or in different ones, so protect on one leaves the other's entry alone. The directory scan is no longer used to decide whether a separately stored account still needs an entry. Saved passwords were already per file (named after the file's salt).
Test: test_leaves_the_same_account_in_another_directory_working_when_one_leaves_the_keychain.
One case still shares an entry: a byte-for-byte copy of an account file, because the copy carries the same entry name or the same salt. A copy in the same directory keeps the entry alive. A copy kept in another directory is not seen, so removing the original's storage removes the entry that copy points at. The key is not lost in that case, since the original's new storage is read back before the entry is removed. Tracking copies in arbitrary locations would need a registry outside the account file, which I have not added.
A key entry was named after the address alone, so every account file that held one key in the Keychain shared a single entry. No check that looks at one directory can prove that such an entry is safe to remove: an overlapping save could adopt an entry that a failing save then removed, and the same account stored in another directory lost its key when the first left Keychain storage. - Name each key entry after the address and a random identifier chosen when the file is written, and record that name in the account file's `account` field. Entries are no longer shared between files that were stored separately, so creating, undoing or removing one file's entry cannot affect another's. A reader accepts only an entry name that starts with the file's own address. - A failed save removes the entries it made before anything else, so a later error cannot leave them behind. - Only a copy of an account file still shares that file's entries. An entry a copy in the same directory uses is kept. The shared sample file keychain.json now carries an entry name. A Keychain file written by the keystore matches it apart from the identifier it chooses.
| except BaseException: | ||
| write_file_atomic(location.path, existing.data) | ||
| _remove_entries(created) |
There was a problem hiding this comment.
[P1] Serialize storage changes before restoring an old account file
Rollback unconditionally restores the file captured before this operation, even if another process has since changed it successfully. Reproduced: A reads a Keychain-backed account; B changes that same file to plaintext and deletes the old Keychain entry; A's write then fails. A restores the stale Keychain pointer over B's plaintext file, leaving no persisted key and causing KEYCHAIN_ITEM_NOT_FOUND. Protect the entire read/change/cleanup/rollback sequence with an interprocess lock or equivalent concurrency control.
Validated against dc715a6 using a simulated Keychain and a deterministic interleaving with a simulated write failure. The focused keystore, CLI, and mnemonic suite passes (368 passed, 1 skipped); the real Keychain was untouched.
There was a problem hiding this comment.
Fixed in 2910aaf. protect() now holds an interprocess lock (<name>.json.lock, created exclusively next to the account file) for the whole write, read-back, cleanup and rollback sequence. Under the lock it reads the file again and stops with the new code ACCOUNT_BUSY if the bytes no longer match what it read, so in your interleaving A never writes and never restores: B's plaintext file stands.
Passwords are asked for before the lock is taken, so it is held only while files and the Keychain change. A caller that finds the lock held waits up to ten seconds, then fails with ACCOUNT_BUSY.
A lock is never removed automatically, however old. I first had a stale-lock timeout, but judging a lock abandoned and removing it cannot be one step, so two waiters could both remove it and both proceed. A lock left by a killed process stays until a person removes it, and the error names the file.
Tests: test_does_not_restore_a_stale_file_over_a_change_another_process_made reproduces the interleaving (it fails with the check disabled), plus test_waits_for_the_lock_and_gives_up_with_account_busy, test_never_removes_a_lock_on_its_own_however_old_it_is and test_releases_the_lock_when_the_change_fails. The same change is in ProvableHQ/veil#221, and I checked that a lock held by either SDK blocks the other.
protect() reads an account file, writes a new one, removes the old secret, and puts the old file back if a step fails. Two processes doing that to one file could interleave: the first reads a Keychain-backed file, the second moves it to plaintext and removes the Keychain entry, then the first fails and restores its stale copy, leaving a pointer to an entry that is gone and no stored key. - Hold an interprocess lock, <name>.json.lock, created exclusively, while the file and the Keychain change. A caller that finds it held waits up to ten seconds and then fails with the new code ACCOUNT_BUSY. Passwords are asked for before the lock is taken. - Under the lock, read the file again and stop with ACCOUNT_BUSY if it no longer matches what was read, so a change another process made in the meantime is never overwritten or undone. - Never remove a lock on another caller's behalf. Judging a lock to be abandoned and removing it cannot be one step, so two waiters could both remove it and both proceed. The error names the lock file so a person can remove one left by a killed process.
Summary
Adds
aleo.keystore, a small account store for the Aleo SDK, and recovery-phrase support on the account module. A bot or script can store an account once and resolve it on every run without carrying its own key-handling code.The account file format is shared with the TypeScript SDK (Veil), so an account stored by either SDK opens in the other. The companion change there, ProvableHQ/veil#221, adds the same keystore to
@provablehq/veil-aleo-sdk/node.What is added
aleo.keystoresave(account, …)~/.aleo/accounts/<name>.json. Never replaces an existing account file.load(…)info(…)protect(…)export(…)migrate(source, …)plaintext(the default; an owner-only file),encrypted(a password stretched with scrypt, then the SDK's own private-key ciphertext),keychain(the macOS login keychain) andnone(the address only).ALEO_PRIVATE_KEY,ALEO_PRIVATE_KEY_FILE, the account file, then a caller-supplied legacy key.SHIELD_SWAP_PRIVATE_KEYandSHIELD_SWAP_PRIVATE_KEY_FILEare accepted as aliases.KeystoreError(anAleoError) with a stablecode.python -m aleo.keystorewithcreate,import,show,protect,exportandmigrate.Recovery phrases on the client's account module:
from_mnemonic,generate_mnemonicandcreate_with_mnemonic, using the derivation of the Shield wallet and the TypeScript SDK. The BIP-39 English word list ships as package data.Behaviour worth reviewing
passwordargument,ALEO_ACCOUNT_PASSWORD, a password saved in the macOS Keychain, or a hidden terminal prompt.export,create --storage none,create --mnemonic) run only with a terminal attached and refuse--json.protectreads the new storage back before removing the old secret, and restores the original file if any later step fails.protectholds an interprocess lock (<name>.json.lock) while it changes an account file, and stops withACCOUNT_BUSYif another process holds the lock or changed the file in the meantime. A lock is never removed automatically; the error names the file.key.account. Files stored separately never share an entry, so creating, undoing or removing one file's entry cannot affect another's. The label carries the account name (aleo-account-key:<name>) for display only.No existing behaviour changes.
aleo.account.create(),from_private_key(),export_encrypted()andimport_encrypted()are untouched, and no runtime dependency is added.Not in this change
aleo_shield_swap(Profile.load_or_create()) still stores its key inprofile.json. Moving it onto the keystore is a follow-up.Testing
python -m pytest python/tests -m "not slow": 1503 passed, 14 skipped.pyright: 0 errors.test_keystore.py,test_keystore_cli.py,test_facade_mnemonic.py. They run in temporary directories with the key, account and password variables cleared and the Keychain tool replaced by a fake.ALEO_KEYSTORE_KEYCHAIN_TEST=1), with the public sample account only, and removes its entries afterwards. Run locally on macOS: passed, nothing left behind.Sample files
sdk/python/tests/fixtures/keystore-vectors/holds the shared sample files. The private key in them is the devnode key that@provablehq/veil-aleo-devnodealready publishes, and the recovery phrase is the standard BIP-39 test phrase. Neither is a new secret, and neither account should ever be funded.