diff --git a/rust/CLAUDE.md b/rust/CLAUDE.md index 572604aaa6..98ffd62dfc 100644 --- a/rust/CLAUDE.md +++ b/rust/CLAUDE.md @@ -24,19 +24,23 @@ cargo fmt --check ``` **Why CI might fail even after running these:** + - Skipping any command above - Assuming auto-fix tools handle everything (they don't) - Not fixing warnings that clippy reports - Making changes after running the checks ### Commit & PR Message Rules + **These rules apply to BOTH commit messages AND pull request descriptions** + - Keep messages CONCISE (no walls of text) - Subject line under 72 chars (no body text unless critical) - NO emojis - NO promotional text or links (except Co-Authored-By line) ### Additional Guidelines + - Check `git status` before committing to avoid adding temporary/binary files - Never commit to main branch - All CI checks must pass - run the checklist commands above before committing or creating PR @@ -74,61 +78,76 @@ The codebase must pass the GitHub Actions workflow at `../.github/workflows/rust ## Architecture Overview -This is a Rust CLI for ente.io, providing encrypted photo backup and export functionality. The project is migrating from a Go implementation and follows a modular architecture: +CLI for ente.io with end-to-end encryption, multi-account and multi-app support. ### Core Modules **`api/`** - HTTP client for ente.io API - `client.rs`: Base HTTP client with auth token management -- Authentication uses SRP (Secure Remote Password) protocol -- Handles retry logic and rate limiting +- `auth.rs`: SRP (Secure Remote Password) authentication implementation +- `methods.rs`: API method implementations (collections, files, user info) +- `models.rs`: API request/response data structures +- `retry.rs`: Retry logic with exponential backoff for rate limiting **`crypto/`** - Cryptographic operations using libsodium - `argon.rs`: Argon2id key derivation - `chacha.rs`: ChaCha20-Poly1305 encryption/decryption +- `stream.rs`: XChaCha20-Poly1305 streaming decryption for large files - `kdf.rs`: Blake2b key derivation - All crypto MUST use `libsodium-sys-stable` (statically linked) **`storage/`** - SQLite persistence layer - `schema.rs`: Database schema for accounts, files, collections -- `account.rs`: Account CRUD operations -- `sync.rs`: Sync state management +- `account.rs`: Account CRUD operations with multi-app support +- `sync.rs`: Sync state management (last sync times, file tracking) +- `config.rs`: Key-value configuration store - Uses `rusqlite` with bundled SQLite for portability -**`cli/`** - Command-line interface +**`cli/`** - Command-line argument parsing -- `account.rs`: Account management (add, list, update) -- `export.rs`: Photo export orchestration +- `account.rs`: Account command argument structures +- `export.rs`: Export command argument structures +- `version.rs`: Version information constants - Uses `clap` for argument parsing +**`commands/`** - Command implementation logic + +- `account.rs`: Account management implementation (add, list, update, get-token) +- `export.rs`: Photo export orchestration with filtering support +- `sync.rs`: Synchronization logic execution + **`models/`** - Data structures - `account.rs`: Account model with encrypted credentials - `file.rs`: File metadata and encryption info -- `collection.rs`: Albums/collections +- `collection.rs`: Albums/collections with sharing support +- `metadata.rs`: Decrypted file metadata (title, timestamps, location) +- `filter.rs`: Export filtering options (shared/hidden albums, emails) - `error.rs`: Error types using `thiserror` **`sync/`** - Synchronization engine -- Handles incremental sync with ente servers -- File download and decryption -- Collection management +- `engine.rs`: Core sync orchestration and state management +- `files.rs`: File synchronization logic +- `download.rs`: File download and decryption implementation + +**`utils/`** - Utility functions + +- `mod.rs`: Config directory management with platform-specific defaults ### Key Implementation Details -1. **Authentication Flow**: SRP-based authentication storing tokens in SQLite -2. **Multi-Account Support**: HashMap-based token storage per account -3. **File Organization**: Export to `YYYY/MM-Month/` directory structure -4. **Encryption**: All files encrypted with ChaCha20-Poly1305, keys derived via Argon2/Blake2b -5. **Async Runtime**: Uses tokio with full features -6. **Error Handling**: Propagate errors with `?` operator, use `anyhow` for context - -## Current Status - -The project has completed foundational components (crypto, storage, models) and is currently implementing the API client for authentication and sync. See `CONVERSION_PLAN.md` for detailed implementation roadmap. +1. **Authentication Flow**: SRP-based authentication storing tokens in SQLite, supports multiple apps (photos, locker, auth) +2. **Multi-Account Support**: SQLite-based storage with per-account/per-app token management +3. **File Organization**: Export to `export_dir/AlbumName/filename` structure (files in "Uncategorized" if no album) +4. **Export Filtering**: Support for filtering by shared/hidden albums and specific user emails +5. **Encryption**: Files encrypted with ChaCha20-Poly1305, streaming decryption for large files, keys derived via Argon2/Blake2b +6. **Async Runtime**: Uses tokio with full features for concurrent operations +7. **Error Handling**: Propagate errors with `?` operator, use `anyhow` for context +8. **Configuration**: Platform-specific config directories (Linux: ~/.config/ente-cli, macOS: ~/Library/Application Support/ente-cli, Windows: %APPDATA%/ente-cli) ## Security Guidelines @@ -144,5 +163,4 @@ The project has completed foundational components (crypto, storage, models) and ## Environment Variables - `ENTE_CLI_CONFIG_DIR`: Override default config directory -- `ENTE_LOG`: Set log level (debug, info, warn, error) -- `RUST_LOG`: Alternative log level configuration +- `RUST_LOG`: Set log level (debug, info, warn, error, trace) diff --git a/rust/CONVERSION_PLAN.md b/rust/CONVERSION_PLAN.md deleted file mode 100644 index 8f7bd5785c..0000000000 --- a/rust/CONVERSION_PLAN.md +++ /dev/null @@ -1,288 +0,0 @@ -# Ente CLI Rust Conversion Plan - -## Current Status -The Rust CLI has achieved **feature parity with the Go CLI for photos app** core functionality! Export, sync, and incremental updates are fully working with proper file decryption, metadata handling, progress indicators, and deduplication. - -### โœ… Photos App Core Features - COMPLETE -- **Export**: Full workflow with decryption, metadata, deduplication, live photos -- **Sync**: Full and incremental sync with downloads, progress tracking -- **Account**: Multi-account support with SRP authentication -- **Storage**: SQLite with efficient schema and indexing -- **Crypto**: All encryption/decryption working (Argon2, XChaCha20-Poly1305, XSalsa20-Poly1305) - -### ๐Ÿ“ Photos App Remaining Features -- Export filters (by album, date range, shared/hidden) -- Resume interrupted downloads -- EXIF/location data preservation -- Thumbnail generation -- Album symlinks - -### โŒ Not Planned (Auth App Features) -- Locker export -- 2FA/Auth export -These features are specific to the auth app and not needed for photos functionality. - -## Completed Components โœ… - -### Core Infrastructure -- โœ… Project structure and dependencies (libsodium-sys-stable for crypto) -- โœ… SQLite storage layer with schema for accounts, files, collections -- โœ… Data models (Account, File, Collection, Error types) -- โœ… CLI command structure (account, export, version commands) - -### Cryptography Module (`/rust/src/crypto/`) -- โœ… Argon2 key derivation (`argon.rs`) -- โœ… Blake2b login key derivation (`kdf.rs`) -- โœ… XSalsa20-Poly1305 (secret_box) for key decryption (`chacha.rs`) -- โœ… **Streaming XChaCha20-Poly1305** for file decryption (`stream.rs`) -- โœ… Chunked decryption for large files (4MB chunks) -- โœ… libsodium initialization and helpers - -### API Client (`/rust/src/api/`) -- โœ… **Base HTTP client with token management** (`client.rs`) -- โœ… **Request/Response models** (`models.rs`) - - โœ… Collection models - - โœ… File models with metadata - - โœ… User and auth response models -- โœ… **Core API methods** (`methods.rs`) - - โœ… `get_collections()` - Fetch collections - - โœ… `get_collection_files()` - Fetch files with pagination - - โœ… `download_file()` - Download encrypted files - -### Export Command (`/rust/src/commands/export.rs`) -- โœ… **Full export workflow implemented** -- โœ… Load stored credentials from SQLite -- โœ… Decrypt collection keys using master key -- โœ… Decrypt file keys using collection keys -- โœ… Decrypt file data using streaming XChaCha20-Poly1305 -- โœ… Decrypt and parse metadata for original filenames -- โœ… **Public magic metadata support for renamed files** -- โœ… Create date-based directory structure (YYYY/MM-Month/) -- โœ… Skip already exported files (local deduplication) -- โœ… **Hash-based deduplication to prevent duplicate exports** -- โœ… **Live photo extraction from ZIP archives** -- โœ… Progress indicators with file counts and progress bars -- โœ… Export summary with statistics - -### Account Management (`/rust/src/commands/account.rs`) -- โœ… **Account list** - Display all configured accounts -- โœ… **Account add** - Full SRP authentication implemented -- โœ… Store encrypted credentials in SQLite -- โœ… 2FA/OTP support -- โœ… Proper key derivation with Argon2 - -### Metadata Handling (`/rust/src/models/metadata.rs`, `/rust/src/models/file.rs`) -- โœ… **Metadata decryption and parsing** -- โœ… Extract original filenames -- โœ… File type detection (Image, Video, LivePhoto) -- โœ… Public magic metadata support for renamed files -- โœ… Edited name prioritization (public magic metadata โ†’ regular metadata) - -## Recently Completed ๐ŸŽ‰ - -### Sync Command (`/rust/src/commands/sync.rs`) - โœ… COMPLETE -- โœ… **Full sync workflow implemented** -- โœ… Metadata-only mode for fast syncing -- โœ… Full mode with file downloads -- โœ… Per-collection incremental sync tracking -- โœ… Store sync state in SQLite -- โœ… Handle deleted files/collections -- โœ… **Integrated file downloads with progress indicators** -- โœ… Hash-based deduplication during downloads -- โœ… Correct counting logic for new/updated/deleted items - -### File Download Manager (`/rust/src/sync/download.rs`) - โœ… COMPLETE -- โœ… Download individual files with decryption -- โœ… Parallel download infrastructure (tokio tasks) -- โœ… **Progress bars using indicatif** -- โœ… **Live photo extraction from ZIP archives** -- โœ… Proper error handling and retry logic -- โœ… Memory-efficient streaming downloads -- โœ… Hash-based deduplication - -## Remaining Components ๐Ÿ“ - -### Database and Storage (`/rust/src/storage/`) - โœ… COMPLETE -- โœ… **Platform-specific config directory** (`~/.config/ente-cli/`) -- โœ… Avoid conflicts with Go CLI path -- โœ… SQLite schema with proper foreign keys -- โœ… Collections and files storage -- โœ… Per-collection sync state tracking -- โœ… Content hash storage for deduplication -- โœ… Efficient indexes for lookups - -### Account Management Enhancements -- [ ] **Account remove** - Delete account and credentials -- [ ] **Token refresh** - Handle expired tokens - -### Advanced Export Features -- [ ] Export filters (by album, date range) -- [ ] Shared album support -- [ ] Hidden album handling -- โœ… **Live photos (ZIP file extraction)** - Implemented -- [ ] Thumbnail generation -- [ ] Export to different formats - -### API Client Enhancements -- [ ] Retry logic with exponential backoff -- [ ] Rate limiting (429 status codes) -- [ ] Request/response logging -- [ ] Error recovery -- [ ] Connection pooling - -### File Processing -- [ ] EXIF data extraction -- [ ] Location data handling -- [ ] Creation/modification time preservation -- [ ] Symlink creation for albums -- โœ… **Duplicate detection by hash** - Implemented with SHA-512 - -### Download Manager Enhancements -- โœ… **Parallel downloads** - Using tokio tasks -- [ ] Resume interrupted downloads -- [ ] Bandwidth throttling -- โœ… **Progress tracking per file** - Using indicatif progress bars -- [ ] Temp file management - -## Testing Status ๐Ÿงช - -### Successfully Tested โœ… -- โœ… Export with real account -- โœ… Small file decryption (JPEG images) -- โœ… Large file decryption (33MB RAW file) -- โœ… Metadata extraction for filenames -- โœ… Public magic metadata for renamed files -- โœ… Date-based directory creation -- โœ… File deduplication (local and hash-based) -- โœ… Incremental sync (per-collection) -- โœ… Live photo extraction from ZIP -- โœ… Progress indicators during downloads -- โœ… Hash-based duplicate detection - -### Manual Testing Checklist -- [x] Can export from existing Ente account -- [x] Lists all albums/collections correctly -- [x] Downloads files to correct folder structure (YYYY/MM-Month/) -- [x] Correctly decrypts files -- [x] Extracts original filenames from metadata -- [x] Handles renamed files from public magic metadata -- [x] Sync command fetches collections and files -- [x] Metadata-only sync mode works -- [x] Full sync mode with file downloads -- [x] Database stored in ~/.config/ente-cli/ -- [x] Handles incremental sync (only new files) -- [x] Hash-based deduplication prevents duplicates -- [x] Live photos extracted correctly -- [x] Progress bars show download progress -- [ ] Resumes interrupted downloads -- [ ] Multi-account export works -- [ ] Export filters (by album, date range) work - -## Migration from Go CLI - -### Feature Parity Progress (Photos App) -- [x] Multi-account support (storage) -- [x] Photos export (complete with all features) -- [x] Sync command (full implementation with downloads) -- [x] Album organization -- [x] Deduplicated storage (hash-based) -- [x] Platform-specific config paths -- [x] SRP authentication (fully implemented) -- [x] Full sync with file downloads -- [x] Incremental sync (per-collection tracking) -- [x] Public magic metadata support -- [x] Live photo extraction -- [x] Progress indicators -- [ ] Export filters (albums, shared, hidden) -- [ ] Shared album support - -### Not Planned (Auth App Features) -- [ ] Locker export (auth app) -- [ ] Auth (2FA) export (auth app) - -### Data Migration -- [ ] BoltDB to SQLite migration tool -- [ ] Preserve sync state -- [ ] Migrate account credentials - -## Recent Achievements ๐ŸŽ‰ - -1. **Full Sync Implementation with Downloads** - - Complete sync engine with per-collection tracking - - Incremental sync with proper timestamp management - - Integrated file downloads with progress bars - - Hash-based deduplication prevents duplicate downloads - - Live photo extraction from ZIP archives - -2. **Public Magic Metadata Support** - - Handles renamed files correctly - - Prioritizes edited names over original names - - Decrypts both regular and public magic metadata - -3. **Progress Indicators** - - Download progress bars using indicatif - - Real-time status updates during sync - - Accurate counting of new/updated/deleted items - -4. **Hash-Based Deduplication** - - SHA-512 content hashing for files - - Prevents duplicate exports across collections - - Efficient database indexing for hash lookups - - Tested and verified with duplicate files - -## Next Actions ๐Ÿš€ - -### Photos App - Remaining Features - -1. **Export Filters** - - Filter by album/collection name - - Filter by date range - - Export only specific albums - - Support for shared albums - - Support for hidden albums - -2. **Resume Capability** - - Track partially downloaded files - - Resume interrupted downloads - - Verify partial file integrity - -3. **Advanced Features** - - Thumbnail generation - - EXIF data preservation - - Location data handling - - Creation/modification time preservation - - Symlink creation for album organization - -4. **Performance Optimizations** - - Connection pooling for API requests - - Bandwidth throttling options - - Configurable parallel download limits - - Memory usage optimization for large exports - -### Infrastructure Improvements - -1. **Error Handling** - - Retry logic with exponential backoff - - Better rate limiting (429 handling) - - Graceful recovery from network errors - -2. **Account Management** - - Account remove command - - Token refresh mechanism - - Multiple endpoint support - -3. **Data Migration** - - BoltDB to SQLite migration tool - - Preserve sync state during migration - - Account credential migration - -## Environment Variables -- `ENTE_CLI_CONFIG_DIR` - Override config directory location -- `ENTE_LOG` / `RUST_LOG` - Set log level (debug, info, warn, error) - -## Key Implementation Notes -1. **Crypto**: Successfully using libsodium-sys-stable for all operations -2. **Streaming**: Proper streaming XChaCha20-Poly1305 implementation -3. **Storage**: SQLite working well for account and credential storage -4. **Async**: Tokio runtime properly configured -5. **Memory**: Chunked processing prevents memory issues with large files \ No newline at end of file diff --git a/rust/src/utils/mod.rs b/rust/src/utils/mod.rs index 434b450e8b..45841f7d90 100644 --- a/rust/src/utils/mod.rs +++ b/rust/src/utils/mod.rs @@ -8,11 +8,6 @@ pub fn get_cli_config_dir() -> crate::Result { return Ok(PathBuf::from(config_dir)); } - // For backward compatibility - if let Ok(config_dir) = std::env::var("ENTE_CLI_CONFIG_PATH") { - return Ok(PathBuf::from(config_dir)); - } - // Use platform-specific config directory // On Linux: ~/.config/ente-cli // On macOS: ~/Library/Application Support/ente-cli