delete rust install (already on runner env by default)

## Description

- Clear vectorDB index on logout
- Revert to using `view` on index
- Use `.usearch` for index file
- Minor design changes

## Tests

Tested in debug mode on my pixel phone.

.github/ISSUE_TEMPLATE/bug_report.yml

@@ -1,45 +1,22 @@
 name: Report a bug
-description: Let us know if something's not working the way you expected.
+description: Things that were working earlier but don't anymore
 labels: []
 body:
     - type: markdown
       attributes:
           value: |
-              Before opening a new bug report, please ensure
-              1. you are on the latest version (it might've already been fixed),
-              2. you've searched for existing issues (please add your observations as a comment there instead of creating a duplicate).
-
-              If you are self hosting, please create a community [Q&A](https://github.com/ente-io/ente/discussions/categories/q-a) instead.
+              **Checklist**
+              1. You've searched existing [issues](https://github.com/search?q=repo%3Aente-io%2Fente+&type=issues) and [discussions](https://github.com/search?q=repo%3Aente-io%2Fente+&type=discussions)
+              2. It was working earlier (otherwise use [enhancements](https://github.com/ente-io/ente/discussions/categories/enhancements))
+              3. It is not about self hosting (for those use [this](https://github.com/ente-io/ente/discussions/categories/q-a))
     - type: textarea
       attributes:
           label: Description
-          description: >
-              Please describe the bug. If possible, also include the steps to
-              reproduce the behaviour, and the expected behaviour (sometimes
-              bugs are just expectation mismatches, in which case this would be
-              a good fit for [feature
-              requests](https://github.com/ente-io/ente/discussions/categories/feature-requests)).
-      validations:
-          required: true
     - type: input
       attributes:
           label: Version
           description: The version can be seen at the bottom of settings.
           placeholder: e.g. v1.2.3
-    - type: input
-      attributes:
-          label: Last working version
-          description: >
-              The version where the feature was last known to be working. It is
-              fine if you don't remember the exact version (mention roughly
-              then), but if there just isn't a last known working version, then
-              it is likely that what is being reported is not an issue but a
-              feature request. The difference between the two categories is not
-              just semantic - feature requests use GitHub discussions and so can
-              be [upvoted by the
-              community](https://github.com/ente-io/ente/discussions/categories/feature-requests)
-              (issues can't be).
-          placeholder: e.g. v1.2.3
     - type: dropdown
       attributes:
           label: What product are you using?

.github/ISSUE_TEMPLATE/config.yml

@@ -1,5 +1,5 @@
 blank_issues_enabled: true
 contact_links:
-    - name: Feature requests and questions
+    - name: Enhacements, feature requests, feedback and questions
       url: https://github.com/ente-io/ente/discussions
       about: Please use Discussions for everything apart from the above.

.github/workflows/auth-crowdin-push.yml

@@ -5,7 +5,7 @@ on:
         branches: [main]
         paths:
             # Run workflow when auth's intl_en.arb is changed
-            - "auth/lib/l10n/arb/app_en.arb"
+            - "mobile/apps/auth/lib/l10n/arb/app_en.arb"
             # Or the workflow itself is changed
             - ".github/workflows/auth-crowdin.yml"
 
@@ -24,8 +24,8 @@ jobs:
             - name: Crowdin's action
               uses: crowdin/github-action@v2
               with:
-                  base_path: "auth/"
-                  config: "auth/crowdin.yml"
+                  base_path: "mobile/apps/auth/"
+                  config: "mobile/apps/auth/crowdin.yml"
                   upload_sources: true
                   upload_translations: false
                   download_translations: false

.github/workflows/auth-crowdin-sync.yml

@@ -23,8 +23,8 @@ jobs:
             - name: Crowdin's action
               uses: crowdin/github-action@v2
               with:
-                  base_path: "auth/"
-                  config: "auth/crowdin.yml"
+                  base_path: "mobile/apps/auth/"
+                  config: "mobile/apps/auth/crowdin.yml"
                   upload_sources: true
                   upload_translations: false
                   download_translations: true

.github/workflows/auth-internal-release.yml

@@ -4,7 +4,7 @@ on:
     workflow_dispatch: # Allow manually running the action
 
 env:
-    FLUTTER_VERSION: "3.24.3"
+    FLUTTER_VERSION: "3.32.8"
 
 permissions:
     contents: write
@@ -15,7 +15,7 @@ jobs:
 
         defaults:
             run:
-                working-directory: auth
+                working-directory: mobile/apps/auth
 
         steps:
             - name: Checkout code and submodules
@@ -55,7 +55,7 @@ jobs:
               with:
                   serviceAccountJsonPlainText: ${{ secrets.SERVICE_ACCOUNT_JSON }}
                   packageName: io.ente.auth
-                  releaseFiles: auth/build/app/outputs/bundle/playstoreRelease/app-playstore-release.aab
+                  releaseFiles: mobile/apps/auth/build/app/outputs/bundle/playstoreRelease/app-playstore-release.aab
                   track: internal
 
             - name: Notify Discord

.github/workflows/auth-lint.yml

@@ -4,11 +4,11 @@ on:
     # Run on every pull request (open or push to it) that changes auth/
     pull_request:
         paths:
-            - "auth/**"
+            - "mobile/apps/auth/**"
             - ".github/workflows/auth-lint.yml"
 
 env:
-    FLUTTER_VERSION: "3.24.3"
+    FLUTTER_VERSION: "3.32.8"
 
 permissions:
     contents: read
@@ -18,7 +18,7 @@ jobs:
         runs-on: ubuntu-latest
         defaults:
             run:
-                working-directory: auth
+                working-directory: mobile/apps/auth
         steps:
             - name: Checkout code and submodules
               uses: actions/checkout@v4

.github/workflows/auth-release.yml

@@ -29,18 +29,18 @@ on:
             - "auth-v*"
 
 env:
-    FLUTTER_VERSION: "3.24.3"
+    FLUTTER_VERSION: "3.32.8"
 
 permissions:
     contents: write
 
 jobs:
     build-linux-latest:
-        runs-on: ubuntu-latest
+        runs-on: ubuntu-22.04
 
         defaults:
             run:
-                working-directory: auth
+                working-directory: mobile/apps/auth
 
         steps:
             - name: Checkout code and submodules
@@ -83,7 +83,7 @@ jobs:
               # disable this step if release tag contains nightly or beta
               if: startsWith(github.ref, 'refs/tags/auth-v') && !contains(github.ref, 'nightly') && !contains(github.ref, 'beta')
               run: |
-                  flutter build appbundle --release --flavor playstore --dart-define=app.flavor=playstore
+                  flutter build appbundle --release --flavor playstore --dart-define=app.flavor=playstore --dart-define=cronetHttpNoPlay=true
               env:
                   SIGNING_KEY_PATH: "/home/runner/work/_temp/keystore/ente_auth_key.jks"
                   SIGNING_KEY_ALIAS: ${{ secrets.SIGNING_KEY_ALIAS }}
@@ -93,12 +93,12 @@ jobs:
             - name: Install dependencies for desktop build
               run: |
                   sudo apt-get update -y
-                  sudo apt-get install -y libsecret-1-dev libsodium-dev libfuse2 ninja-build libgtk-3-dev dpkg-dev pkg-config rpm patchelf libsqlite3-dev locate libayatana-appindicator3-dev libffi-dev libtiff6 xz-utils libarchive-tools libcurl4-openssl-dev
+                  sudo apt-get install -y libsecret-1-dev libsodium-dev libfuse2 ninja-build libgtk-3-dev dpkg-dev pkg-config rpm patchelf libsqlite3-dev locate libayatana-appindicator3-dev libffi-dev libtiff5 xz-utils libarchive-tools libcurl4-openssl-dev
                   sudo updatedb --localpaths='/usr/lib/x86_64-linux-gnu'
 
             - name: Install appimagetool
               run: |
-                  wget -O appimagetool "https://github.com/AppImage/AppImageKit/releases/download/continuous/appimagetool-x86_64.AppImage"
+                  wget -O appimagetool "https://github.com/AppImage/appimagetool/releases/download/continuous/appimagetool-x86_64.AppImage"
                   chmod +x appimagetool
                   mv appimagetool /usr/local/bin/
 
@@ -117,12 +117,14 @@ jobs:
                   mv dist/**/*-*-linux.deb artifacts/ente-${{ github.ref_name }}-x86_64.deb
 
             - name: Generate checksums
-              run: sha256sum artifacts/ente-* >> artifacts/sha256sum-rpm-appimage
+              run: |
+                sha256sum artifacts/ente-auth-*.apk >> artifacts/sha256sum-apk
+                sha256sum artifacts/ente-auth-*.deb artifacts/ente-auth-*.rpm artifacts/ente-auth-*.AppImage >> artifacts/sha256sum-linux
 
             - name: Create a draft GitHub release
               uses: ncipollo/release-action@v1
               with:
-                  artifacts: "auth/artifacts/*"
+                  artifacts: "mobile/apps/auth/artifacts/*"
                   draft: true
                   allowUpdates: true
                   updateOnlyUnreleased: true
@@ -134,15 +136,16 @@ jobs:
               with:
                   serviceAccountJsonPlainText: ${{ secrets.SERVICE_ACCOUNT_JSON }}
                   packageName: io.ente.auth
-                  releaseFiles: auth/build/app/outputs/bundle/playstoreRelease/app-playstore-release.aab
+                  releaseFiles: mobile/apps/auth/build/app/outputs/bundle/playstoreRelease/app-playstore-release.aab
                   track: internal
 
     build-windows:
         runs-on: windows-latest
+        environment: "auth-win-build"
 
         defaults:
             run:
-                working-directory: auth
+                working-directory: mobile/apps/auth
 
         steps:
             - name: Checkout code and submodules
@@ -172,14 +175,22 @@ jobs:
             - name: Retain Windows EXE and DLLs
               run: cp -r build/windows/x64/runner/Release ente-${{ github.ref_name }}-windows
 
-            - name: Code sign Windows installer and EXE
-              uses: dlemstra/code-sign-action@v1
+            - name: Sign files with Trusted Signing
+              uses: azure/trusted-signing-action@v0
               with:
-                  certificate: "${{ secrets.WINDOWS_CERTIFICATE }}"
-                  password: "${{ secrets.WINDOWS_CERTIFICATE_PASSWORD }}"
-                  files: |
-                      auth/artifacts/ente-${{ github.ref_name }}-installer.exe
-                      auth/ente-${{ github.ref_name }}-windows/auth.exe
+                azure-tenant-id: ${{ secrets.AZURE_TENANT_ID }}
+                azure-client-id: ${{ secrets.AZURE_CLIENT_ID }}
+                azure-client-secret: ${{ secrets.AZURE_CLIENT_SECRET }}
+                endpoint: ${{ secrets.AZURE_ENDPOINT }}
+                trusted-signing-account-name: ${{ secrets.AZURE_CODE_SIGNING_NAME }}
+                certificate-profile-name: ${{ secrets.AZURE_CERT_PROFILE_NAME }}
+                files: |
+                      ${{ github.workspace }}/mobile/apps/auth/artifacts/ente-${{ github.ref_name }}-installer.exe
+                      ${{ github.workspace }}/mobile/apps/auth/ente-${{ github.ref_name }}-windows/auth.exe
+                file-digest: SHA256
+                timestamp-rfc3161: http://timestamp.acs.microsoft.com
+                timestamp-digest: SHA256  
+
 
             - name: Zip Windows EXE and DLLs
               run: tar.exe -a -c -f artifacts/ente-${{ github.ref_name }}-windows.zip ente-${{ github.ref_name }}-windows
@@ -190,7 +201,7 @@ jobs:
             - name: Create a draft GitHub release
               uses: ncipollo/release-action@v1
               with:
-                  artifacts: "auth/artifacts/*"
+                  artifacts: "mobile/apps/auth/artifacts/*"
                   draft: true
                   allowUpdates: true
                   updateOnlyUnreleased: true
@@ -200,7 +211,7 @@ jobs:
 
         defaults:
             run:
-                working-directory: auth
+                working-directory: mobile/apps/auth
 
         steps:
             - name: Checkout code and submodules
@@ -287,7 +298,7 @@ jobs:
             - name: Create a draft GitHub release
               uses: ncipollo/release-action@v1
               with:
-                  artifacts: "auth/artifacts/*"
+                  artifacts: "mobile/apps/auth/artifacts/*"
                   draft: true
                   allowUpdates: true
                   updateOnlyUnreleased: true

.github/workflows/docs-lint.yml

@@ -0,0 +1,32 @@
+name: "Lint (docs)"
+
+on:
+    # Run on every pull request (open or push to it) that changes docs/
+    pull_request:
+        paths:
+            - "docs/**"
+            - ".github/workflows/docs-lint.yml"
+
+permissions:
+    contents: read
+
+jobs:
+    lint:
+        runs-on: ubuntu-latest
+        defaults:
+            run:
+                working-directory: docs
+        steps:
+            - name: Checkout code
+              uses: actions/checkout@v4
+
+            - name: Setup node and enable yarn caching
+              uses: actions/setup-node@v4
+              with:
+                  node-version: 22
+                  cache: "yarn"
+                  cache-dependency-path: "web/yarn.lock"
+
+            - run: yarn install
+
+            - run: yarn pretty:check

.github/workflows/mobile-crowdin-push.yml

@@ -5,7 +5,7 @@ on:
         branches: [main]
         paths:
             # Run workflow when mobiles's intl_en.arb is changed
-            - "mobile/lib/l10n/intl_en.arb"
+            - "mobile/apps/photos/lib/l10n/intl_en.arb"
             # Or the workflow itself is changed
             - ".github/workflows/mobile-crowdin.yml"
 
@@ -24,8 +24,8 @@ jobs:
             - name: Crowdin's action
               uses: crowdin/github-action@v2
               with:
-                  base_path: "mobile/"
-                  config: "mobile/crowdin.yml"
+                  base_path: "mobile/apps/photos/"
+                  config: "mobile/apps/photos/crowdin.yml"
                   upload_sources: true
                   upload_translations: false
                   download_translations: false

.github/workflows/mobile-crowdin-sync.yml

@@ -1,4 +1,4 @@
-name: "Sync Crowdin translations (mobile)"
+name: "Sync Crowdin translations (mobile/photos)"
 
 on:
     schedule:
@@ -23,14 +23,14 @@ jobs:
             - name: Crowdin's action
               uses: crowdin/github-action@v2
               with:
-                  base_path: "mobile/"
-                  config: "mobile/crowdin.yml"
+                  base_path: "mobile/apps/photos/"
+                  config: "mobile/apps/photos/crowdin.yml"
                   upload_sources: true
                   upload_translations: false
                   download_translations: true
                   localization_branch_name: translations/mobile
                   create_pull_request: true
-                  pull_request_title: "[mobile] New translations"
+                  pull_request_title: "[mobile/photos] New translations"
                   pull_request_body: "New translations from [Crowdin](https://crowdin.com/project/ente-photos-app)"
                   pull_request_base_branch_name: "main"
                   project_id: 574741

.github/workflows/mobile-daily-internal.yml

@@ -0,0 +1,147 @@
+name: "Internal release (photos)"
+
+on:
+    schedule:
+        # Runs daily at 12:30 UTC (6:00 PM IST)
+        - cron: "30 12 * * *"
+    workflow_dispatch: # Allow manual trigger
+
+env:
+    FLUTTER_VERSION: "3.32.8"
+    RUST_VERSION: "1.86.0"
+
+permissions:
+    contents: read
+
+jobs:
+    build:
+        runs-on: ubuntu-latest
+
+        defaults:
+            run:
+                working-directory: mobile/apps/photos
+
+        steps:
+            - name: Checkout code and submodules
+              uses: actions/checkout@v4
+              with:
+                  submodules: recursive
+
+            - name: Setup JDK 17
+              uses: actions/setup-java@v1
+              with:
+                  java-version: 17
+
+            - name: Install Flutter ${{ env.FLUTTER_VERSION }}
+              uses: subosito/flutter-action@v2
+              with:
+                  channel: "stable"
+                  flutter-version: ${{ env.FLUTTER_VERSION }}
+                  cache: true
+
+            - name: Install Flutter Rust Bridge
+              run: cargo install flutter_rust_bridge_codegen
+
+            - name: Generate Rust bindings
+              run: flutter_rust_bridge_codegen generate
+
+            - name: Increment version code for build
+              run: |
+                  CURRENT_VERSION=$(grep '^version:' pubspec.yaml | sed 's/version: //')
+                  VERSION_NAME=$(echo $CURRENT_VERSION | cut -d'+' -f1)
+                  CURRENT_BUILD=$(echo $CURRENT_VERSION | cut -d'+' -f2)
+                  NEW_BUILD=$((CURRENT_BUILD + ${{ github.run_number }}))
+                  NEW_VERSION="${VERSION_NAME}+${NEW_BUILD}"
+
+                  sed -i "s/^version: .*/version: $NEW_VERSION/" pubspec.yaml
+                  echo "Building with version ${NEW_VERSION}"
+                  
+                  # Store version for later use
+                  echo "NEW_VERSION=${NEW_VERSION}" >> $GITHUB_ENV
+
+            - name: Prepare and validate changelog for Play Store
+              run: |
+                  mkdir -p whatsnew
+                  CHANGELOG_FILE="scripts/store_changes.txt"
+                  DISCORD_FILE="scripts/internal_changes.txt"
+                  OUTPUT_FILE="whatsnew/whatsnew-en-US"
+
+                  # Use provided changelog or fallback
+                  if [ -f "$CHANGELOG_FILE" ]; then
+                          head -c 500 "$CHANGELOG_FILE" > "$OUTPUT_FILE"
+                  else
+                          echo "Bug fixes and improvements" > "$OUTPUT_FILE"
+                  fi
+
+                  # Validate: file exists
+                  if [ ! -s "$OUTPUT_FILE" ]; then
+                          echo "❌ Changelog is empty."
+                          exit 1
+                  fi
+
+                  # Validate: <= 500 chars
+                  LENGTH=$(wc -m < "$OUTPUT_FILE")
+                  if [ "$LENGTH" -gt 500 ]; then
+                          echo "❌ Changelog exceeds 500 characters ($LENGTH)."
+                          exit 1
+                  fi
+
+                  # Validate: no markdown or HTML
+                  if grep -Eq '[\*\_\<\>\[\]\(\)]' "$OUTPUT_FILE"; then
+                          echo "❌ Changelog contains markdown/HTML formatting."
+                          exit 1
+                  fi
+
+                  echo "✅ Changelog valid:"
+                  cat "$OUTPUT_FILE"
+                  
+                  # Store changelog for Play Store (with escaped newlines)
+                  CHANGELOG_PLAYSTORE=$(cat "$OUTPUT_FILE" | sed ':a;N;$!ba;s/\n/\\n/g' | sed 's/"/\\"/g')
+                  echo "CHANGELOG=${CHANGELOG_PLAYSTORE}" >> $GITHUB_ENV
+                  
+                  # Store changelog for Discord (with proper newlines)
+                  CHANGELOG_DISCORD=$(cat "$DISCORD_FILE" | sed 's/"/\\"/g')
+                  echo "CHANGELOG_DISCORD<<EOF" >> $GITHUB_ENV
+                  echo "$CHANGELOG_DISCORD" >> $GITHUB_ENV
+                  echo "EOF" >> $GITHUB_ENV
+
+            - name: Setup keys
+              uses: timheuer/base64-to-file@v1
+              with:
+                  fileName: "keystore/ente_photos_key.jks"
+                  encodedString: ${{ secrets.SIGNING_KEY_PHOTOS }}
+
+            - name: Build PlayStore AAB
+              run: |
+                  flutter build appbundle --dart-define=cronetHttpNoPlay=true --release --flavor playstore
+              env:
+                  SIGNING_KEY_PATH: "/home/runner/work/_temp/keystore/ente_photos_key.jks"
+                  SIGNING_KEY_ALIAS: ${{ secrets.SIGNING_KEY_ALIAS_PHOTOS }}
+                  SIGNING_KEY_PASSWORD: ${{ secrets.SIGNING_KEY_PASSWORD_PHOTOS }}
+                  SIGNING_STORE_PASSWORD: ${{ secrets.SIGNING_STORE_PASSWORD_PHOTOS }}
+
+            - name: Upload AAB to PlayStore
+              uses: r0adkll/upload-google-play@v1
+              with:
+                  serviceAccountJsonPlainText: ${{ secrets.SERVICE_ACCOUNT_JSON }}
+                  packageName: io.ente.photos
+                  releaseFiles: mobile/apps/photos/build/app/outputs/bundle/playstoreRelease/app-playstore-release.aab
+                  track: internal
+                  whatsNewDirectory: mobile/apps/photos/whatsnew
+                  mappingFile: mobile/apps/photos/build/app/outputs/mapping/playstoreRelease/mapping.txt
+
+            - name: Notify Discord
+              uses: sarisia/actions-status-discord@v1
+              with:
+                  webhook: ${{ secrets.DISCORD_INTERNAL_RELEASE_WEBHOOK }}
+                  nodetail: true
+                  title: "🏆 Daily release Photos v${{ env.NEW_VERSION }} (Branch: ${{ github.ref_name }})"
+                  description: |
+                      **Version:** ${{ env.NEW_VERSION }}
+                      **Flutter:** ${{ env.FLUTTER_VERSION }}
+                      **Commit:** [${{ github.sha }}](${{ github.server_url }}/${{ github.repository }}/commit/${{ github.sha }})
+                      **Download:** [Play Store](https://play.google.com/store/apps/details?id=io.ente.photos)
+
+                      **Changes:**
+                      ${{ env.CHANGELOG_DISCORD }}
+                  color: 0x00ff00

.github/workflows/mobile-internal-release.yml

@@ -1,68 +0,0 @@
-name: "Internal release (photos)"
-
-on:
-    workflow_dispatch: # Allow manually running the action
-
-env:
-    FLUTTER_VERSION: "3.24.3"
-
-permissions:
-    contents: write
-
-jobs:
-    build:
-        runs-on: ubuntu-latest
-
-        defaults:
-            run:
-                working-directory: mobile
-
-        steps:
-            - name: Checkout code and submodules
-              uses: actions/checkout@v4
-              with:
-                  submodules: recursive
-
-            - name: Setup JDK 17
-              uses: actions/setup-java@v1
-              with:
-                  java-version: 17
-
-            - name: Install Flutter ${{ env.FLUTTER_VERSION  }}
-              uses: subosito/flutter-action@v2
-              with:
-                  channel: "stable"
-                  flutter-version: ${{ env.FLUTTER_VERSION  }}
-                  cache: true
-
-            - name: Setup keys
-              uses: timheuer/base64-to-file@v1
-              with:
-                  fileName: "keystore/ente_photos_key.jks"
-                  encodedString: ${{ secrets.SIGNING_KEY_PHOTOS }}
-
-            - name: Build PlayStore AAB
-              run: |
-                  flutter build appbundle --dart-define=cronetHttpNoPlay=true --release --flavor playstore
-              env:
-                  SIGNING_KEY_PATH: "/home/runner/work/_temp/keystore/ente_photos_key.jks"
-                  SIGNING_KEY_ALIAS: ${{ secrets.SIGNING_KEY_ALIAS_PHOTOS }}
-                  SIGNING_KEY_PASSWORD: ${{ secrets.SIGNING_KEY_PASSWORD_PHOTOS }}
-                  SIGNING_STORE_PASSWORD: ${{ secrets.SIGNING_STORE_PASSWORD_PHOTOS }}
-
-            - name: Upload AAB to PlayStore
-              uses: r0adkll/upload-google-play@v1
-              with:
-                  serviceAccountJsonPlainText: ${{ secrets.SERVICE_ACCOUNT_JSON }}
-                  packageName: io.ente.photos
-                  releaseFiles: mobile/build/app/outputs/bundle/playstoreRelease/app-playstore-release.aab
-                  track: internal
-
-            - name: Notify Discord
-              uses: sarisia/actions-status-discord@v1
-              with:
-                  webhook: ${{ secrets.DISCORD_INTERNAL_RELEASE_WEBHOOK }}
-                  nodetail: true
-                  title: "🏆 Internal release available for Photos"
-                  description: "[Download](https://play.google.com/store/apps/details?id=io.ente.photos)"
-                  color: 0x00ff00

.github/workflows/mobile-lint.yml

@@ -4,11 +4,12 @@ on:
     # Run on every pull request (open or push to it) that changes mobile/
     pull_request:
         paths:
-            - "mobile/**"
+            - "mobile/apps/photos/**"
             - ".github/workflows/mobile-lint.yml"
 
 env:
-    FLUTTER_VERSION: "3.24.3"
+    FLUTTER_VERSION: "3.32.8"
+    RUST_VERSION: "1.86.0"
 
 permissions:
     contents: read
@@ -18,7 +19,7 @@ jobs:
         runs-on: ubuntu-latest
         defaults:
             run:
-                working-directory: mobile
+                working-directory: mobile/apps/photos
         steps:
             - name: Checkout code and submodules
               uses: actions/checkout@v4
@@ -31,7 +32,18 @@ jobs:
                   channel: "stable"
                   flutter-version: ${{ env.FLUTTER_VERSION  }}
                   cache: true
-
+                  
             - run: flutter pub get
+                  
+            - name: Install Rust ${{ env.RUST_VERSION }}
+              uses: dtolnay/rust-toolchain@master
+              with:
+                  toolchain: ${{ env.RUST_VERSION }}
+                  
+            - name: Install Flutter Rust Bridge
+              run: cargo install flutter_rust_bridge_codegen
+              
+            - name: Generate Rust bindings
+              run: flutter_rust_bridge_codegen generate
 
             - run: flutter analyze --no-fatal-infos

.github/workflows/mobile-release.yml

@@ -9,7 +9,7 @@ on:
             - "photos-v*"
 
 env:
-    FLUTTER_VERSION: "3.24.3"
+    FLUTTER_VERSION: "3.32.8"
 
 permissions:
     contents: write
@@ -20,7 +20,7 @@ jobs:
 
         defaults:
             run:
-                working-directory: mobile
+                working-directory: mobile/apps/photos
 
         steps:
             - name: Checkout code and submodules
@@ -62,5 +62,5 @@ jobs:
             - name: Create a draft GitHub release
               uses: ncipollo/release-action@v1
               with:
-                  artifacts: "mobile/build/app/outputs/flutter-apk/ente-${{ github.ref_name }}.apk,mobile/build/app/outputs/flutter-apk/sha256sum"
+                  artifacts: "mobile/apps/photos/build/app/outputs/flutter-apk/ente-${{ github.ref_name }}.apk,mobile/apps/photos/build/app/outputs/flutter-apk/sha256sum"
                   draft: true

.github/workflows/photos-internal-release.yml

@@ -0,0 +1,126 @@
+name: "Internal Release V2 (photos)"
+
+on:
+  workflow_dispatch: # Manual trigger only
+
+env:
+  FLUTTER_VERSION: "3.32.8"
+  ANDROID_KEYSTORE_PATH: "keystore/ente_photos_key.jks"
+
+jobs:
+  build:
+    runs-on: macos-latest # Required for iOS builds
+    environment: "ios-build"
+    permissions:
+      contents: write
+
+    defaults:
+      run:
+        working-directory: mobile/apps/photos
+
+    steps:
+      # Common Setup
+      - name: Checkout code
+        uses: actions/checkout@v4
+        with:
+          submodules: recursive
+
+      - name: Setup JDK 17
+        uses: actions/setup-java@v1
+        with:
+          java-version: 17
+
+      - name: Install Flutter
+        uses: subosito/flutter-action@v2
+        with:
+          flutter-version: ${{ env.FLUTTER_VERSION }}
+          cache: true
+
+      # Android Build 
+      - name: Setup Android signing key
+        uses: timheuer/base64-to-file@v1
+        with:
+          fileName: ${{ env.ANDROID_KEYSTORE_PATH }}
+          encodedString: ${{ secrets.SIGNING_KEY_PHOTOS }}
+
+      # - name: Build Android AAB
+      #   run: |
+      #     flutter build appbundle \
+      #       --dart-define=cronetHttpNoPlay=true \
+      #       --release \
+      #       --flavor playstore
+      #   env:
+      #     SIGNING_KEY_PATH: ${{ env.ANDROID_KEYSTORE_PATH }}
+      #     SIGNING_KEY_ALIAS: ${{ secrets.SIGNING_KEY_ALIAS_PHOTOS }}
+      #     SIGNING_KEY_PASSWORD: ${{ secrets.SIGNING_KEY_PASSWORD_PHOTOS }}
+      #     SIGNING_STORE_PASSWORD: ${{ secrets.SIGNING_STORE_PASSWORD_PHOTOS }}
+
+      # iOS Build (new secure implementation)
+      - name: Install fastlane
+        run: gem install fastlane
+      
+      - name: Create ExportOptions.plist
+        run: |
+          cat <<EOF > ios/ExportOptions.plist
+          <?xml version="1.0" encoding="UTF-8"?>
+          <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
+          <plist version="1.0">
+          <dict>
+              <key>method</key>
+              <string>app-store</string>
+              <key>teamID</key>
+              <string>${{ secrets.IOS_TEAM_ID }}</string>
+          </dict>
+          </plist>
+          EOF
+
+      - name: Setup App Store Connect API Key
+        run: |
+          echo '${{ secrets.IOS_API_KEY }}' > api_key.json
+          chmod 600 api_key.json
+
+      - name: Build iOS IPA
+        run: |
+          flutter build ipa \
+            --release \
+            --export-options-plist=ExportOptions.plist \
+            --dart-define=cronetHttpNoPlay=true
+        env:
+          SIGNING_TEAM_ID: ${{ secrets.IOS_TEAM_ID }}
+
+      # Uploads
+      # - name: Upload to Play Store
+      #   uses: r0adkll/upload-google-play@v1
+      #   with:
+      #     serviceAccountJsonPlainText: ${{ secrets.SERVICE_ACCOUNT_JSON }}
+      #     packageName: io.ente.photos
+      #     releaseFiles: build/app/outputs/bundle/playstoreRelease/app-playstore-release.aab
+      #     track: internal
+
+      - name: Upload to TestFlight
+        run: |
+          fastlane pilot upload \
+            --api_key_path api_key.json \
+            --ipa "build/ios/ipa/Ente Photos.ipa" \
+            --skip_waiting_for_build_processing \
+            --apple_id ${{ secrets.IOS_APPLE_ID }} \
+            --app_identifier "io.ente.photos"
+        env:
+          APP_STORE_CONNECT_API_KEY_ID: ${{ secrets.IOS_API_KEY_ID }}
+          APP_STORE_CONNECT_ISSUER_ID: ${{ secrets.IOS_ISSUER_ID }}
+          FASTLANE_APPLE_APPLICATION_SPECIFIC_PASSWORD: ${{ secrets.IOS_APP_SPECIFIC_PASSWORD }}
+
+      - name: Clean sensitive files
+        run: |
+          rm -f api_key.json
+          rm -f ${{ env.ANDROID_KEYSTORE_PATH }}
+
+      - name: Notify Discord
+        uses: sarisia/actions-status-discord@v1
+        with:
+          webhook: ${{ secrets.DISCORD_INTERNAL_RELEASE_WEBHOOK }}
+          title: "🚀 Dual Platform Release Uploaded"
+          description: |
+            **Android**: [Play Store Internal](https://play.google.com/store/apps/details?id=io.ente.photos)
+            **iOS**: TestFlight build processing
+          color: 0x00ff00

.github/workflows/rust-lint.yml

@@ -0,0 +1,44 @@
+name: "Lint (rust)"
+
+on:
+    # Run on every pull request (open or push to it) that changes rust/
+    pull_request:
+        paths:
+            - "rust/**"
+            - ".github/workflows/rust-lint.yml"
+
+permissions:
+    contents: read
+
+# Cancel in-progress lint runs when a new commit is pushed.
+concurrency:
+    group: ${{ github.workflow }}-${{ github.ref }}
+    cancel-in-progress: true
+
+env:
+    RUSTFLAGS: -D warnings
+
+jobs:
+    lint:
+        runs-on: ubuntu-latest
+        defaults:
+            run:
+                working-directory: rust
+        steps:
+            - name: Checkout code
+              uses: actions/checkout@v4
+
+            - name: Cache
+              uses: actions/cache@v4
+              with:
+                  path: |
+                      ~/.cargo/registry
+                      ~/.cargo/git
+                      target
+                  key: ${{ runner.os }}-cargo-${{ hashFiles('**/Cargo.lock') }}
+
+            - run: cargo fmt --check
+
+            - run: cargo clippy --all-targets --all-features
+
+            - run: cargo build

.github/workflows/web-deploy.yml

@@ -54,6 +54,18 @@ jobs:
                   apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
                   command: pages deploy --project-name=ente --commit-dirty=true --branch=deploy/photos web/apps/photos/out
 
+            - name: Build custom-albums
+              run: yarn build:photos
+              env:
+                  NEXT_PUBLIC_ENTE_ONLY_SERVE_ALBUMS_APP: 1
+
+            - name: Publish custom-albums
+              uses: cloudflare/wrangler-action@v3
+              with:
+                  accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+                  apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
+                  command: pages deploy --project-name=ente --commit-dirty=true --branch=deploy/custom-albums web/apps/photos/out
+
             - name: Build accounts
               run: yarn build:accounts
 
@@ -93,3 +105,13 @@ jobs:
                   accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
                   apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
                   command: pages deploy --project-name=ente --commit-dirty=true --branch=deploy/payments web/apps/payments/dist
+
+            - name: Build locker
+              run: yarn build:locker
+
+            - name: Publish locker
+              uses: cloudflare/wrangler-action@v3
+              with:
+                  accountId: ${{ secrets.CLOUDFLARE_ACCOUNT_ID }}
+                  apiToken: ${{ secrets.CLOUDFLARE_API_TOKEN }}
+                  command: pages deploy --project-name=ente --commit-dirty=true --branch=deploy/locker web/apps/locker/out

.gitmodules

@@ -3,9 +3,9 @@
 	url = https://github.com/ente-io/sentry-dart.git
 	branch = sentry_flutter_ente
 [submodule "auth/flutter"]
-	path = auth/flutter
+	path = mobile/apps/auth/flutter
 	url = https://github.com/flutter/flutter.git
 	branch = stable
 [submodule "auth/assets/simple-icons"]
-	path = auth/assets/simple-icons
+	path = mobile/apps/auth/assets/simple-icons
 	url = https://github.com/simple-icons/simple-icons.git

CONTRIBUTING.md

@@ -1,54 +1,42 @@
 # Contributing
 
-First and foremost, thank you for your interest in contributing to Ente 🙏
-
-There are many ways to contribute, and most of them don't require writing code.
-
-* [Spread the word](#spread-the-word)
-* [Engage with the community](#engage-with-the-community)
-* [Translate](#translate)
-* [Document](#document)
-
+- [Spread the word](#spread-the-word)
+- [Engage with the community](#engage-with-the-community)
+- [Translate](#translate)
+- [Document](#document)
 
 ## Spread the word
 
-This is perhaps the most impactful contribution you can make. [Spread the
-word](https://help.ente.io/photos/features/referral-program/). Online on your
-favorite social media channels. Offline to your friends and family who are
-looking for a privacy-friendly alternative to big tech.
+**This is the most impactful contribution you can make**.
+
+[Spread the word](https://help.ente.io/photos/features/referral-program/). Online on your favorite social media channels. Offline to your friends and family who are looking for a privacy-friendly alternative to big tech.
 
 ## Engage with the community
 
-Just hang around, enjoy the vibe. Answer someone's query on our
-[Discord](https://discord.gg/z2YVKkycX3), or pile on in the sporadic #off-topic
-rants there. Chuckle (or wince!) at our [Twitter](https://twitter.com/enteio)
-memes. Suggest a new feature in our [Github
-Discussions](https://github.com/ente-io/ente/discussions/new?category=feature-requests),
-or upvote the existing ones that you feel we should focus on first. Provide your
-opinion on existing threads.
+Just hang around, enjoy the vibe. The Ente community — the people who are building Ente, and the people who are using Ente — hang out at various places depending on their proclivity:
 
-These might seem like small things, but it provides us energy. Knowing that
-there is a community of people who care for what we are building.
+- [Discord](https://discord.ente.io)
+- [Mastodon](https://fosstodon.org/@ente)
+- [X / Twitter](https://twitter.com/enteio)
+- [Github Discussions](https://github.com/ente-io/ente/discussions)
+
+Just being around might seem a small thing, but it provides us energy. Knowing that there is a community of people who care for what we are building, **who want us to do better**.
 
 ## Translate
 
-If you're interested in helping out with translation, please visit our Crowdin
-projects to get started:
-
-| Project  |  |
-| ------------- | ------------- |
-| [Auth](https://crowdin.com/project/ente-authenticator-app)  | [![Crowdin](https://badges.crowdin.net/ente-authenticator-app/localized.svg)](https://crowdin.com/project/ente-authenticator-app)  |
-| [Photos](https://crowdin.com/project/ente-photos-app)  | [![Crowdin](https://badges.crowdin.net/ente-photos-app/localized.svg)](https://crowdin.com/project/ente-photos-app)  |
-| [Photos Web / Desktop](https://crowdin.com/project/ente-photos-web)  | [![Crowdin](https://badges.crowdin.net/ente-photos-web/localized.svg)](https://crowdin.com/project/ente-photos-web)  |
+Visit our Crowdin projects to help with translations:
 
+| Project                                                             |                                                                                                                                   |
+| ------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- |
+| [Auth](https://crowdin.com/project/ente-authenticator-app)          | [![Crowdin](https://badges.crowdin.net/ente-authenticator-app/localized.svg)](https://crowdin.com/project/ente-authenticator-app) |
+| [Photos](https://crowdin.com/project/ente-photos-app)               | [![Crowdin](https://badges.crowdin.net/ente-photos-app/localized.svg)](https://crowdin.com/project/ente-photos-app)               |
+| [Photos Web / Desktop](https://crowdin.com/project/ente-photos-web) | [![Crowdin](https://badges.crowdin.net/ente-photos-web/localized.svg)](https://crowdin.com/project/ente-photos-web)               |
 
 If your language is not listed for translation, please [create a GitHub
 issue](https://github.com/ente-io/ente/issues/new?title=Request+for+New+Language+Translation&body=Language+name%3A+%0AProject%3A+auth%2Fphotos%2Fboth)
 to have it added. It is okay to have partial translations. Once ~90% of the
 strings in a language get translated, we will start surfacing it in the apps.
 
-Thank you for your support.
-
 ## Document
 
 The help guides and FAQs for users of Ente products are also open source, and
@@ -60,25 +48,9 @@ See [docs/](docs/README.md) for how to edit these documents.
 
 ## Code contributions
 
-Code is a small aspect of community, and the ways mentioned above are more
-important in helping us. But if you'd _really_ like to contribute code, it is
-best to start small. Consider some well-scoped changes, say like adding more
-[custom icons to auth](auth/docs/adding-icons.md).
+If you'd like to contribute code, it is best to start small. Consider some well-scoped changes, say like adding more [custom icons to auth](mobile/apps/auth/docs/adding-icons.md), or fixing a specific bug.
 
-Each of the individual product/platform specific directories in this repository
-have instructions on setting up a dev environment.
-
-For anything beyond trivial bug fixes, please use [features requests and
-discussions](https://github.com/ente-io/ente/discussions) instead of performing
-code changes directly.
-
-> [!TIP]
->
-> Please remember that code is a important, but small, part of the overall big
-> picture that makes a product a joy to use. Something that's easy in code is
-> not necessarily the right choice for the product as a whole. So we'll repeat -
-> there are other ways to contribute than code that we'd request you to
-> consider.
+Code that changes the behaviour of the product might not get merged, at least not initially. The PR can serve as a discussion bed, but you might find it easier to just start a discussion instead, or post your perspective in the (likely) existing thread about the behaviour change or new feature you wish for.
 
 ## Leave a review or star
 

auth/.fvmrc

@@ -1,3 +0,0 @@
-{
-  "flutter": "3.24.3"
-}

auth/assets/custom-icons/icons/mistral.svg

@@ -1,15 +0,0 @@
-<svg width="500" height="500" viewBox="0 0 500 500" fill="none" xmlns="http://www.w3.org/2000/svg">
-<g style="mix-blend-mode:difference">
-<path fill-rule="evenodd" clip-rule="evenodd" d="M363.636 23H409.091V477.545H363.636V23ZM0 23H45.4545V477.545H0V23ZM227.273 295.727H181.818V386.636H227.273V295.727ZM272.727 113.909H318.182V204.818H272.727V113.909Z" fill="white"/>
-</g>
-<path d="M136.364 386.636H45.4545V477.545H136.364V386.636Z" fill="#EA3326"/>
-<path d="M500 386.636H409.091V477.545H500V386.636Z" fill="#EA3326"/>
-<path d="M136.364 295.727H45.4545V386.636H136.364V295.727Z" fill="#EB5829"/>
-<path d="M318.182 295.727H227.273V386.636H318.182V295.727Z" fill="#EB5829"/>
-<path d="M500 295.727H409.091V386.636H500V295.727Z" fill="#EB5829"/>
-<path d="M136.364 23H45.4545V113.909H136.364V23Z" fill="#F7D046"/>
-<path d="M500 23H409.091V113.909H500V23Z" fill="#F7D046"/>
-<path d="M227.273 113.909H45.4545V204.818H227.273V113.909Z" fill="#F2A73B"/>
-<path d="M500 113.909H318.182V204.818H500V113.909Z" fill="#F2A73B"/>
-<path d="M500 204.818H45.4545V295.727H500V204.818Z" fill="#EE792F"/>
-</svg>

auth/assets/custom-icons/icons/ubiquiti.svg

@@ -1,7 +0,0 @@
-<?xml version="1.0" encoding="utf-8"?><!-- Uploaded to: SVG Repo, www.svgrepo.com, Generator: SVG Repo Mixer Tools -->
-<svg xmlns="http://www.w3.org/2000/svg"
-aria-label="Ubiquiti" role="img"
-viewBox="0 0 512 512"><rect
-width="512" height="512"
-rx="15"
-fill="#399cdb"/><path d="M112 94v18h18V94h-18zm288 0c-82 0-90 31-90 61v172a147 147 0 01-3 28c43-9 72-36 86-82l7-23V94zm-234 18v18h18v-18h-18zm-18 18v18h18v-18h-18zm36 9v18h18v-18h-18zm-72 4v147c0 73 53 128 144 128 0 0-54-30-54-91V197h-18v66h-18v-39h-18v17h-18v-98h-18zm54 18v18h18v-18h-18zm-18 27v18h18v-18h-18zm252 87c-19 64-65 92-131 89-24-1-43-7-57-16 10 42 46 63 48 64l10 6c82-5 130-59 130-128v-15z" fill="#ffffff"/></svg>

auth/fastlane/metadata/android/en-US/full_description.txt

@@ -1,34 +0,0 @@
-ente's Authenticator app helps you generate and store 2 step verification (2FA) tokens on your mobile devices.
-
-
-FEATURES
-
-- Secure Backups
-ente provides end-to-end encrypted cloud backups so that you don't have to worry about losing your tokens. We use the same protocols Ente Photos uses to encrypt and preserve your data.
-
-- Multi Device Synchronization
-ente will automatically sync the 2FA tokens you add to your account, across all your devices. Every new device you sign into will have access to these tokens.
-
-- Web access
-You can access your 2FA code from any web browser by visiting https://auth.ente.io .
-
-- Offline Mode
-ente generates 2FA tokens offline, so your network connectivity will not get in the way of your workflow.
-
-- Import and Export Tokens
-You can add tokens to ente by one of the following methods: 
-1. Scanning a QR code
-2. Manually entering (copy-pasting) a 2FA secret
-3. Bulk importing from a file that contains a list of codes in the following format:
-
-otpauth://totp/provider.com:you@email.com?secret=YOUR_SECRET
-
-The codes maybe separated by new lines or commas.
-
-You can also export the codes you have added to ente, to an **unencrypted** text file, that adheres to the above format.
-
-
-SUPPORT
-
-If you need help, please visit @ https://github.com/ente-io/ente/discussions/new?category=q-a, and someone will get in touch with you.
-If you have feature requests, please create an issue @ https://github.com/ente-io/ente

auth/fastlane/metadata/android/en-US/short_description.txt

@@ -1 +0,0 @@
-ente is an end-to-end encrypted authenticator app

auth/fastlane/metadata/android/en-US/title.txt

@@ -1 +0,0 @@
-Ente Authenticator

auth/lib/events/codes_updated_event.dart

@@ -1,3 +0,0 @@
-import 'package:ente_auth/events/event.dart';
-
-class CodesUpdatedEvent extends Event {}

auth/lib/events/endpoint_updated_event.dart

@@ -1,3 +0,0 @@
-import 'package:ente_auth/events/event.dart';
-
-class EndpointUpdatedEvent extends Event {}

auth/lib/events/icons_changed_event.dart

@@ -1,3 +0,0 @@
-import 'package:ente_auth/events/event.dart';
-
-class IconsChangedEvent extends Event {}

auth/lib/events/notification_event.dart

@@ -1,5 +0,0 @@
-import 'package:ente_auth/events/event.dart';
-
-// NotificationEvent event is used to re-fresh the UI to show latest notification
-// (if any)
-class NotificationEvent extends Event {}

auth/lib/events/signed_in_event.dart

@@ -1,3 +0,0 @@
-import 'package:ente_auth/events/event.dart';
-
-class SignedInEvent extends Event {}

auth/lib/events/signed_out_event.dart

@@ -1,3 +0,0 @@
-import 'package:ente_auth/events/event.dart';
-
-class SignedOutEvent extends Event {}

auth/lib/events/trigger_logout_event.dart

@@ -1,3 +0,0 @@
-import 'package:ente_auth/events/event.dart';
-
-class TriggerLogoutEvent extends Event {}

auth/lib/l10n/arb/app_be.arb

@@ -1,97 +0,0 @@
-{
-  "account": "Уліковы запіс",
-  "unlock": "Разблакіраваць",
-  "recoveryKey": "Ключ аднаўлення",
-  "counterAppBarTitle": "Лічыльнік",
-  "@counterAppBarTitle": {
-    "description": "Text shown in the AppBar of the Counter Page"
-  },
-  "onBoardingBody": "Бяспечна зрабіць рэзервовую копію кодаў 2ФА",
-  "onBoardingGetStarted": "Пачаць",
-  "setupFirstAccount": "Наладзіць ваш першы ўліковы запіс",
-  "importScanQrCode": "Сканіраваць код QR-код",
-  "qrCode": "QR-код",
-  "importEnterSetupKey": "Увесці ключ наладжвання",
-  "importAccountPageTitle": "Увесці падрабязнасці ўліковага запісу",
-  "secretCanNotBeEmpty": "Сакрэт не можа быць пустым",
-  "bothIssuerAndAccountCanNotBeEmpty": "І выдавец, і ўліковы запіс не могуць быць пустымі",
-  "incorrectDetails": "Няправільныя падрабязнасці",
-  "pleaseVerifyDetails": "Праверце падрабязнасці і паспрабуйце яшчэ раз",
-  "codeIssuerHint": "Выдавец",
-  "codeSecretKeyHint": "Сакрэтны ключ",
-  "secret": "Сакрэт",
-  "all": "Усе",
-  "notes": "Нататкі",
-  "notesLengthLimit": "Максімальная колькасць сімвалаў у нататках не больш за {count}",
-  "@notesLengthLimit": {
-    "description": "Text to indicate the maximum number of characters allowed for notes",
-    "placeholders": {
-      "count": {
-        "description": "The maximum number of characters allowed for notes",
-        "type": "int",
-        "example": "100"
-      }
-    }
-  },
-  "codeAccountHint": "Уліковы запіс (vy@damen.com)",
-  "codeTagHint": "Тэг",
-  "accountKeyType": "Тып ключа",
-  "sessionExpired": "Сеанс завяршыўся",
-  "@sessionExpired": {
-    "description": "Title of the dialog when the users current session is invalid/expired"
-  },
-  "pleaseLoginAgain": "Аўтарызуйцеся яшчэ раз",
-  "loggingOut": "Выхад...",
-  "timeBasedKeyType": "Заснаваныя на часе (TOTP)",
-  "counterBasedKeyType": "Заснаваныя на лічыльніку (HOTP)",
-  "saveAction": "Захаваць",
-  "nextTotpTitle": "наступны",
-  "deleteCodeTitle": "Выдаліць код?",
-  "deleteCodeMessage": "Вы сапраўды хочаце выдаліць гэты код? Гэта дзеянне з'яўляецца незваротным.",
-  "trashCode": "Выдаліць код?",
-  "trashCodeMessage": "Вы сапраўды хочаце выдаліць код для {account}?",
-  "trash": "Сметніца",
-  "viewLogsAction": "Паглядзець журнал",
-  "preparingLogsTitle": "Падрыхтоўка журнала...",
-  "emailLogsTitle": "Адправіць журнал па электроннай пошце",
-  "exportLogsAction": "Экспартаваць журнал",
-  "reportABug": "Паведаміць пра памылку",
-  "reportBug": "Паведаміць пра памылку",
-  "contactSupport": "Звярнуцца ў службу падтрымкі",
-  "blog": "Блог",
-  "verifyPassword": "Праверыць пароль",
-  "pleaseWait": "Пачакайце...",
-  "generatingEncryptionKeysTitle": "Генерацыя ключоў шыфравання...",
-  "recreatePassword": "Стварыць пароль паўторна",
-  "useRecoveryKey": "Выкарыстоўваць ключ аднаўлення",
-  "incorrectPasswordTitle": "Няправільны пароль",
-  "welcomeBack": "З вяртаннем!",
-  "changePassword": "Змяніць пароль",
-  "data": "Даныя",
-  "importCodes": "Імпартаваць коды",
-  "passwordEmptyError": "Пароль не можа быць пустым",
-  "importFromApp": "Імпартаваць коды з {appName}",
-  "exportCodes": "Экспартаваць коды",
-  "importLabel": "Імпарт",
-  "selectFile": "Выбраць файл",
-  "yes": "Так",
-  "no": "Не",
-  "email": "Электронная пошта",
-  "support": "Падтрымка",
-  "settings": "Налады",
-  "pleaseTryAgain": "Калі ласка, паспрабуйце яшчэ раз",
-  "delete": "Выдаліць",
-  "enterYourPasswordHint": "Увядзіце ваш пароль",
-  "faq": "Частыя пытанні",
-  "deleteAccount": "Выдаліць уліковы запіс",
-  "noDeleteAccountAction": "Не, выдаліць уліковы запіс",
-  "sendEmail": "Адправіць ліст",
-  "createNewAccount": "Стварыць новы ўліковы запіс",
-  "confirmPassword": "Пацвердзіць пароль",
-  "close": "Закрыць",
-  "oopsSomethingWentWrong": "Штосьці пайшло не так.",
-  "language": "Мова",
-  "security": "Бяспека",
-  "searchHint": "Пошук...",
-  "search": "Пошук"
-}

auth/lib/l10n/arb/app_et.arb

@@ -1,4 +0,0 @@
-{
-  "loggingOut": "Väljalogimine...",
-  "useRecoveryKey": "Kasuta taastevõtit"
-}

auth/lib/l10n/l10n.dart

@@ -1,8 +0,0 @@
-import "package:flutter/widgets.dart";
-import "package:flutter_gen/gen_l10n/app_localizations.dart";
-
-export "package:flutter_gen/gen_l10n/app_localizations.dart";
-
-extension AppLocalizationsX on BuildContext {
-  AppLocalizations get l10n => AppLocalizations.of(this);
-}

auth/lib/models/subscription.dart

@@ -1,65 +0,0 @@
-const freeProductID = "free";
-const stripe = "stripe";
-const appStore = "appstore";
-const playStore = "playstore";
-
-class Subscription {
-  final String productID;
-  final int storage;
-  final String originalTransactionID;
-  final String paymentProvider;
-  final int expiryTime;
-  final String price;
-  final String period;
-  final Attributes? attributes;
-
-  Subscription({
-    required this.productID,
-    required this.storage,
-    required this.originalTransactionID,
-    required this.paymentProvider,
-    required this.expiryTime,
-    required this.price,
-    required this.period,
-    this.attributes,
-  });
-
-  bool isValid() {
-    return expiryTime > DateTime.now().microsecondsSinceEpoch;
-  }
-
-  bool isYearlyPlan() {
-    return 'year' == period;
-  }
-
-  static fromMap(Map<String, dynamic>? map) {
-    if (map == null) return null;
-    return Subscription(
-      productID: map['productID'],
-      storage: map['storage'],
-      originalTransactionID: map['originalTransactionID'],
-      paymentProvider: map['paymentProvider'],
-      expiryTime: map['expiryTime'],
-      price: map['price'],
-      period: map['period'],
-      attributes: map["attributes"] != null
-          ? Attributes.fromJson(map["attributes"])
-          : null,
-    );
-  }
-}
-
-class Attributes {
-  bool? isCancelled;
-  String? customerID;
-
-  Attributes({
-    this.isCancelled,
-    this.customerID,
-  });
-
-  Attributes.fromJson(dynamic json) {
-    isCancelled = json["isCancelled"];
-    customerID = json["customerID"];
-  }
-}

auth/lib/models/user_details.dart

@@ -1,146 +0,0 @@
-import 'dart:convert';
-import 'dart:math';
-
-import 'package:collection/collection.dart';
-import 'package:ente_auth/models/subscription.dart';
-
-class UserDetails {
-  final String email;
-  final int usage;
-  final int fileCount;
-  final int sharedCollectionsCount;
-  final Subscription subscription;
-  final FamilyData? familyData;
-  final ProfileData? profileData;
-
-  UserDetails(
-    this.email,
-    this.usage,
-    this.fileCount,
-    this.sharedCollectionsCount,
-    this.subscription,
-    this.familyData,
-      this.profileData,
-  );
-
-  bool isPartOfFamily() {
-    return familyData?.members?.isNotEmpty ?? false;
-  }
-
-  bool isFamilyAdmin() {
-    assert(isPartOfFamily(), "verify user is part of family before calling");
-    final FamilyMember currentUserMember = familyData!.members!
-        .firstWhere((element) => element.email.trim() == email.trim());
-    return currentUserMember.isAdmin;
-  }
-
-  // getFamilyOrPersonalUsage will return total usage for family if user
-  // belong to family group. Otherwise, it will return storage consumed by
-  // current user
-  int getFamilyOrPersonalUsage() {
-    return isPartOfFamily() ? familyData!.getTotalUsage() : usage;
-  }
-
-  int getFreeStorage() {
-    return max(
-      isPartOfFamily()
-          ? (familyData!.storage - familyData!.getTotalUsage())
-          : (subscription.storage - (usage)),
-      0,
-    );
-  }
-
-  int getTotalStorage() {
-    return isPartOfFamily() ? familyData!.storage : subscription.storage;
-  }
-
-  factory UserDetails.fromMap(Map<String, dynamic> map) {
-    return UserDetails(
-      map['email'] as String,
-      map['usage'] as int,
-      (map['fileCount'] ?? 0) as int,
-      (map['sharedCollectionsCount'] ?? 0) as int,
-      Subscription.fromMap(map['subscription']),
-      FamilyData.fromMap(map['familyData']),
-      ProfileData.fromJson(map['profileData']),
-    );
-  }
-
-}
-
-class FamilyMember {
-  final String email;
-  final int usage;
-  final String id;
-  final bool isAdmin;
-
-  FamilyMember(this.email, this.usage, this.id, this.isAdmin);
-
-  factory FamilyMember.fromMap(Map<String, dynamic> map) {
-    return FamilyMember(
-      (map['email'] ?? '') as String,
-      map['usage'] as int,
-      map['id'] as String,
-      map['isAdmin'] as bool,
-    );
-  }
-}
-class ProfileData {
-  bool canDisableEmailMFA;
-  bool isEmailMFAEnabled;
-  bool isTwoFactorEnabled;
-
-  // Constructor with default values
-  ProfileData({
-    this.canDisableEmailMFA = false,
-    this.isEmailMFAEnabled = false,
-    this.isTwoFactorEnabled = false,
-  });
-
-  // Factory method to create ProfileData instance from JSON
-  factory ProfileData.fromJson(Map<String, dynamic>? json) {
-    if (json == null) null;
-
-    return ProfileData(
-      canDisableEmailMFA: json!['canDisableEmailMFA'] ?? false,
-      isEmailMFAEnabled: json['isEmailMFAEnabled'] ?? false,
-      isTwoFactorEnabled: json['isTwoFactorEnabled'] ?? false,
-    );
-  }
-
-  // Method to convert ProfileData instance to JSON
-  Map<String, dynamic> toJson() {
-    return {
-      'canDisableEmailMFA': canDisableEmailMFA,
-      'isEmailMFAEnabled': isEmailMFAEnabled,
-      'isTwoFactorEnabled': isTwoFactorEnabled,
-    };
-  }
-  String toJsonString() => json.encode(toJson());
-}
-class FamilyData {
-  final List<FamilyMember>? members;
-
-  // Storage available based on the family plan
-  final int storage;
-  final int expiryTime;
-
-  FamilyData(this.members, this.storage, this.expiryTime);
-
-  int getTotalUsage() {
-    return members!.map((e) => e.usage).toList().sum;
-  }
-
-  static fromMap(Map<String, dynamic>? map) {
-    if (map == null) return null;
-    assert(map['members'] != null && map['members'].length >= 0);
-    final members = List<FamilyMember>.from(
-      map['members'].map((x) => FamilyMember.fromMap(x)),
-    );
-    return FamilyData(
-      members,
-      map['storage'] as int,
-      map['expiryTime'] as int,
-    );
-  }
-}

auth/lib/services/user_remote_flag_service.dart

@@ -1,141 +0,0 @@
-import 'dart:async';
-import 'dart:io';
-
-import 'package:collection/collection.dart';
-import 'package:dio/dio.dart';
-import 'package:ente_auth/core/configuration.dart';
-import 'package:ente_auth/core/event_bus.dart';
-import 'package:ente_auth/core/network.dart';
-import 'package:ente_auth/events/notification_event.dart';
-import 'package:ente_auth/services/user_service.dart';
-import 'package:logging/logging.dart';
-import 'package:shared_preferences/shared_preferences.dart';
-
-class UserRemoteFlagService {
-  final _dio = Network.instance.getDio();
-  final _logger = Logger((UserRemoteFlagService).toString());
-  final _config = Configuration.instance;
-  late SharedPreferences _prefs;
-
-  UserRemoteFlagService._privateConstructor();
-
-  static final UserRemoteFlagService instance =
-      UserRemoteFlagService._privateConstructor();
-
-  static const String recoveryVerificationFlag = "recoveryKeyVerified";
-  static const String needRecoveryKeyVerification =
-      "needRecoveryKeyVerification";
-
-  Future<void> init() async {
-    _prefs = await SharedPreferences.getInstance();
-  }
-
-  bool shouldShowRecoveryVerification() {
-    if (!_prefs.containsKey(needRecoveryKeyVerification)) {
-      // fetch the status from remote
-      unawaited(_refreshRecoveryVerificationFlag());
-      return false;
-    } else {
-      final bool shouldShow = _prefs.getBool(needRecoveryKeyVerification)!;
-      if (shouldShow) {
-        // refresh the status to check if user marked it as done on another device
-        unawaited(_refreshRecoveryVerificationFlag());
-      }
-      return shouldShow;
-    }
-  }
-
-  // markRecoveryVerificationAsDone is used to track if user has verified their
-  // recovery key in the past or not. This helps in avoid showing the same
-  // prompt to the user on re-install or signing into a different device
-  Future<void> markRecoveryVerificationAsDone() async {
-    await _updateKeyValue(recoveryVerificationFlag, true.toString());
-    await _prefs.setBool(needRecoveryKeyVerification, false);
-  }
-
-  Future<void> _refreshRecoveryVerificationFlag() async {
-    _logger.finest('refresh recovery key verification flag');
-    final remoteStatusValue =
-        await _getValue(recoveryVerificationFlag, "false");
-    final bool isNeedVerificationFlagSet =
-        _prefs.containsKey(needRecoveryKeyVerification);
-    if (remoteStatusValue.toLowerCase() == "true") {
-      await _prefs.setBool(needRecoveryKeyVerification, false);
-      // If the user verified on different device, then we should refresh
-      // the UI to dismiss the Notification.
-      if (isNeedVerificationFlagSet) {
-        Bus.instance.fire(NotificationEvent());
-      }
-    } else if (!isNeedVerificationFlagSet) {
-      // Verification is not done yet as remoteStatus is false and local flag to
-      // show notification isn't set. Set the flag to true if any active
-      // session is older than 1 day.
-      final activeSessions = await UserService.instance.getActiveSessions();
-      final int microSecondsInADay = const Duration(days: 1).inMicroseconds;
-      final bool anyActiveSessionOlderThanADay =
-          activeSessions.sessions.firstWhereOrNull(
-                (e) =>
-                    (e.creationTime + microSecondsInADay) <
-                    DateTime.now().microsecondsSinceEpoch,
-              ) !=
-              null;
-      if (anyActiveSessionOlderThanADay) {
-        await _prefs.setBool(needRecoveryKeyVerification, true);
-        Bus.instance.fire(NotificationEvent());
-      } else {
-        // continue defaulting to no verification prompt
-        _logger.finest('No active session older than 1 day');
-      }
-    }
-  }
-
-  Future<String> _getValue(String key, String? defaultValue) async {
-    try {
-      final Map<String, dynamic> queryParams = {"key": key};
-      if (defaultValue != null) {
-        queryParams["defaultValue"] = defaultValue;
-      }
-      final response = await _dio.get(
-        "${_config.getHttpEndpoint()}/remote-store",
-        queryParameters: queryParams,
-        options: Options(
-          headers: {
-            "X-Auth-Token": _config.getToken(),
-          },
-        ),
-      );
-      if (response.statusCode != HttpStatus.ok) {
-        throw Exception("Unexpected status code ${response.statusCode}");
-      }
-      return response.data["value"];
-    } catch (e) {
-      _logger.info("Error while fetching bool status for $key", e);
-      rethrow;
-    }
-  }
-
-  // _setBooleanFlag sets the corresponding flag on remote
-  // to mark recovery as completed
-  Future<void> _updateKeyValue(String key, String value) async {
-    try {
-      final response = await _dio.post(
-        "${_config.getHttpEndpoint()}/remote-store/update",
-        data: {
-          "key": key,
-          "value": value,
-        },
-        options: Options(
-          headers: {
-            "X-Auth-Token": _config.getToken(),
-          },
-        ),
-      );
-      if (response.statusCode != HttpStatus.ok) {
-        throw Exception("Unexpected state");
-      }
-    } catch (e) {
-      _logger.warning("Failed to set flag for $key", e);
-      rethrow;
-    }
-  }
-}

auth/lib/ui/account/verify_recovery_page.dart

@@ -1,219 +0,0 @@
-import 'package:bip39/bip39.dart' as bip39;
-import 'package:dio/dio.dart';
-import 'package:ente_auth/core/configuration.dart';
-import 'package:ente_auth/ente_theme_data.dart';
-import 'package:ente_auth/l10n/l10n.dart';
-import 'package:ente_auth/services/local_authentication_service.dart';
-import 'package:ente_auth/services/user_remote_flag_service.dart';
-import 'package:ente_auth/ui/account/recovery_key_page.dart';
-import 'package:ente_auth/ui/common/gradient_button.dart';
-import 'package:ente_auth/ui/components/buttons/button_widget.dart';
-import 'package:ente_auth/utils/dialog_util.dart';
-import 'package:ente_auth/utils/navigation_util.dart';
-import 'package:ente_auth/utils/platform_util.dart';
-import 'package:ente_crypto_dart/ente_crypto_dart.dart';
-import 'package:flutter/material.dart';
-import 'package:logging/logging.dart';
-
-class VerifyRecoveryPage extends StatefulWidget {
-  const VerifyRecoveryPage({super.key});
-
-  @override
-  State<VerifyRecoveryPage> createState() => _VerifyRecoveryPageState();
-}
-
-class _VerifyRecoveryPageState extends State<VerifyRecoveryPage> {
-  final _recoveryKey = TextEditingController();
-  final Logger _logger = Logger((_VerifyRecoveryPageState).toString());
-
-  void _verifyRecoveryKey() async {
-    final dialog =
-        createProgressDialog(context, context.l10n.verifyingRecoveryKey);
-    await dialog.show();
-    try {
-      final String inputKey = _recoveryKey.text.trim();
-      final String recoveryKey =
-          CryptoUtil.bin2hex(Configuration.instance.getRecoveryKey());
-      final String recoveryKeyWords = bip39.entropyToMnemonic(recoveryKey);
-      if (inputKey == recoveryKey || inputKey == recoveryKeyWords) {
-        try {
-          await UserRemoteFlagService.instance.markRecoveryVerificationAsDone();
-        } catch (e) {
-          await dialog.hide();
-          if (e is DioException && e.type == DioExceptionType.unknown) {
-            await showErrorDialog(
-              context,
-              "No internet connection",
-              "Please check your internet connection and try again.",
-            );
-          } else {
-            await showGenericErrorDialog(
-              context: context,
-              error: e,
-            );
-          }
-          return;
-        }
-
-        await dialog.hide();
-        // todo: change this as per figma once the component is ready
-        await showErrorDialog(
-          context,
-          context.l10n.recoveryKeyVerified,
-          context.l10n.recoveryKeySuccessBody,
-        );
-        Navigator.of(context).pop();
-      } else {
-        throw Exception("recovery key didn't match");
-      }
-    } catch (e, s) {
-      _logger.severe("failed to verify recovery key", e, s);
-      await dialog.hide();
-      final String errMessage = context.l10n.invalidRecoveryKey;
-      final result = await showChoiceDialog(
-        context,
-        title: context.l10n.invalidKey,
-        body: errMessage,
-        firstButtonLabel: context.l10n.tryAgain,
-        secondButtonLabel: context.l10n.viewRecoveryKey,
-        secondButtonAction: ButtonAction.second,
-      );
-      if (result!.action == ButtonAction.second) {
-        await _onViewRecoveryKeyClick();
-      }
-    }
-  }
-
-  Future<void> _onViewRecoveryKeyClick() async {
-    final hasAuthenticated =
-        await LocalAuthenticationService.instance.requestLocalAuthentication(
-      context,
-      "Please authenticate to view your recovery key",
-    );
-    await PlatformUtil.refocusWindows();
-
-    if (hasAuthenticated) {
-      String recoveryKey;
-      try {
-        recoveryKey =
-            CryptoUtil.bin2hex(Configuration.instance.getRecoveryKey());
-        await routeToPage(
-          context,
-          RecoveryKeyPage(
-            recoveryKey,
-            context.l10n.ok,
-            showAppBar: true,
-            onDone: () {
-              Navigator.of(context).pop();
-            },
-          ),
-        );
-      } catch (e) {
-        // ignore: unawaited_futures
-        showGenericErrorDialog(
-          context: context,
-          error: e,
-        );
-        return;
-      }
-    }
-  }
-
-  @override
-  Widget build(BuildContext context) {
-    final enteTheme = Theme.of(context).colorScheme.enteTheme;
-    return Scaffold(
-      appBar: AppBar(
-        elevation: 0,
-        leading: IconButton(
-          icon: const Icon(Icons.arrow_back),
-          color: Theme.of(context).iconTheme.color,
-          onPressed: () {
-            Navigator.of(context).pop();
-          },
-        ),
-      ),
-      body: Padding(
-        padding: const EdgeInsets.symmetric(horizontal: 16.0),
-        child: LayoutBuilder(
-          builder: (context, constraints) {
-            return SingleChildScrollView(
-              child: ConstrainedBox(
-                constraints: BoxConstraints(
-                  minWidth: constraints.maxWidth,
-                  minHeight: constraints.maxHeight,
-                ),
-                child: IntrinsicHeight(
-                  child: Column(
-                    children: [
-                      const SizedBox(height: 12),
-                      SizedBox(
-                        width: double.infinity,
-                        child: Text(
-                          context.l10n.confirmRecoveryKey,
-                          style: enteTheme.textTheme.h3Bold,
-                          textAlign: TextAlign.left,
-                        ),
-                      ),
-                      const SizedBox(height: 18),
-                      Text(
-                        context.l10n.recoveryKeyVerifyReason,
-                        style: enteTheme.textTheme.small
-                            .copyWith(color: enteTheme.colorScheme.textMuted),
-                      ),
-                      const SizedBox(height: 12),
-                      TextFormField(
-                        decoration: InputDecoration(
-                          filled: true,
-                          hintText: context.l10n.enterYourRecoveryKey,
-                          contentPadding: const EdgeInsets.all(20),
-                          border: UnderlineInputBorder(
-                            borderSide: BorderSide.none,
-                            borderRadius: BorderRadius.circular(6),
-                          ),
-                        ),
-                        style: const TextStyle(
-                          fontSize: 14,
-                          fontFeatures: [FontFeature.tabularFigures()],
-                        ),
-                        controller: _recoveryKey,
-                        autofocus: false,
-                        autocorrect: false,
-                        keyboardType: TextInputType.multiline,
-                        minLines: 4,
-                        maxLines: null,
-                        onChanged: (_) {
-                          setState(() {});
-                        },
-                      ),
-                      const SizedBox(height: 12),
-                      Expanded(
-                        child: Container(
-                          alignment: Alignment.bottomCenter,
-                          width: double.infinity,
-                          padding: const EdgeInsets.fromLTRB(0, 12, 0, 40),
-                          child: Column(
-                            mainAxisAlignment: MainAxisAlignment.end,
-                            crossAxisAlignment: CrossAxisAlignment.stretch,
-                            children: [
-                              GradientButton(
-                                onTap: _verifyRecoveryKey,
-                                text: context.l10n.confirm,
-                              ),
-                              const SizedBox(height: 8),
-                            ],
-                          ),
-                        ),
-                      ),
-                      const SizedBox(height: 20),
-                    ],
-                  ),
-                ),
-              ),
-            );
-          },
-        ),
-      ),
-    );
-  }
-}

cli/cmd/admin.go

@@ -142,6 +142,22 @@ var _updateFreeUserStorage = &cobra.Command{
 	},
 }
 
+var _sendMail = &cobra.Command{
+	Use:   "send-mail <to-email> <from-email> <from-name>",
+	Args:  cobra.ExactArgs(3),
+	Short: "Sends a test mail via the admin api",
+	RunE: func(cmd *cobra.Command, args []string) error {
+		recoverWithLog()
+		var flags = &model.AdminActionForUser{}
+		cmd.Flags().VisitAll(func(f *pflag.Flag) {
+			if f.Name == "admin-user" {
+				flags.AdminEmail = f.Value.String()
+			}
+		})
+		return ctrl.SendTestMail(context.Background(), *flags, args[0], args[1], args[2])
+	},
+}
+
 func init() {
 	rootCmd.AddCommand(_adminCmd)
 	_ = _userDetailsCmd.MarkFlagRequired("admin-user")
@@ -159,5 +175,6 @@ func init() {
 	_updateFreeUserStorage.Flags().StringP("user", "u", "", "The email of the user to update subscription for. (required)")
 	// add a flag with no value --no-limit
 	_updateFreeUserStorage.Flags().String("no-limit", "True", "When true, sets 100TB as storage limit, and expiry to current date + 100 years")
-	_adminCmd.AddCommand(_userDetailsCmd, _disable2faCmd, _disablePasskeyCmd, _updateFreeUserStorage, _listUsers, _deleteUser)
+	_sendMail.Flags().StringP("admin-user", "a", "", "The email of the admin user. ")
+	_adminCmd.AddCommand(_userDetailsCmd, _disable2faCmd, _disablePasskeyCmd, _updateFreeUserStorage, _listUsers, _deleteUser, _sendMail)
 }

cli/cmd/root.go

@@ -2,11 +2,12 @@ package cmd
 
 import (
 	"fmt"
-	"github.com/ente-io/cli/pkg"
-	"github.com/spf13/cobra/doc"
 	"os"
 	"runtime"
 
+	"github.com/ente-io/cli/pkg"
+	"github.com/spf13/cobra/doc"
+
 	"github.com/spf13/viper"
 
 	"github.com/spf13/cobra"
@@ -20,11 +21,6 @@ var ctrl *pkg.ClICtrl
 var rootCmd = &cobra.Command{
 	Use:   "ente",
 	Short: "CLI tool for exporting your photos from ente.io",
-	// Uncomment the following line if your bare application
-	// has an action associated with it:
-	Run: func(cmd *cobra.Command, args []string) {
-		fmt.Sprintf("Hello World")
-	},
 }
 
 func GenerateDocs() error {

cli/config.yaml.example

@@ -5,8 +5,6 @@
 
 endpoint:
   api: "http://localhost:8080"
-  # Endpoint for the account service for passkey
-  accounts: "http://localhost:3001"
 
 log:
  http: false # log status code & time taken by requests

cli/internal/api/admin.go

@@ -139,5 +139,28 @@ func (c *Client) UpdateFreePlanSub(ctx context.Context, userDetails *models.User
 		}
 	}
 	return nil
-
+}
+
+func (c *Client) SendTestMail(ctx context.Context, toEmail, fromEmail, fromName string) error {
+	body := map[string]interface{}{
+		"to":        []string{toEmail},
+		"fromName":  fromName,
+		"fromEmail": fromEmail,
+		"subject":   "Test mail from Ente",
+		"body":      "This is a test mail from Ente",
+	}
+	r, err := c.restClient.R().
+		SetContext(ctx).
+		SetBody(body).
+		Post("/admin/mail")
+	if err != nil {
+		return err
+	}
+	if r.IsError() {
+		return &ApiError{
+			StatusCode: r.StatusCode(),
+			Message:    r.String(),
+		}
+	}
+	return nil
 }

cli/internal/api/login.go

@@ -83,13 +83,14 @@ func (c *Client) VerifySRPSession(
 	return &res, nil
 }
 
-func (c *Client) SendEmailOTP(
+func (c *Client) SendLoginOTP(
 	ctx context.Context,
 	email string,
 ) error {
 	var res AuthorizationResponse
 	payload := map[string]interface{}{
-		"email": email,
+		"email":   email,
+		"purpose": "login",
 	}
 	r, err := c.restClient.R().
 		SetContext(ctx).

cli/internal/api/login_type.go

@@ -35,6 +35,7 @@ type AuthorizationResponse struct {
 	ID                 int64          `json:"id"`
 	KeyAttributes      *KeyAttributes `json:"keyAttributes,omitempty"`
 	EncryptedToken     string         `json:"encryptedToken,omitempty"`
+	AccountsUrl        string         `json:"accountsUrl"`
 	Token              string         `json:"token,omitempty"`
 	TwoFactorSessionID string         `json:"twoFactorSessionID"`
 	PassKeySessionID   string         `json:"passkeySessionID"`

cli/main.go

@@ -110,7 +110,6 @@ func initConfig(cliConfigDir string) {
 	viper.AddConfigPath(".")                // optionally look for config in the working directory
 
 	viper.SetDefault("endpoint.api", constants.EnteApiUrl)
-	viper.SetDefault("endpoint.accounts", constants.EnteAccountUrl)
 	viper.SetDefault("log.http", false)
 	if err := viper.ReadInConfig(); err != nil {
 		if _, ok := err.(viper.ConfigFileNotFoundError); ok {

cli/pkg/admin_actions.go

@@ -156,6 +156,23 @@ func (c *ClICtrl) UpdateFreeStorage(ctx context.Context, params model.AdminActio
 	return nil
 }
 
+func (c *ClICtrl) SendTestMail(ctx context.Context, params model.AdminActionForUser, to, from, fromName string) error {
+	accountCtx, err := c.buildAdminContext(ctx, params.AdminEmail)
+	if err != nil {
+		return err
+	}
+	err = c.Client.SendTestMail(accountCtx, to, from, fromName)
+	if err != nil {
+		if apiErr, ok := err.(*api.ApiError); ok && apiErr.StatusCode == 400 && strings.Contains(apiErr.Message, "Token is too old") {
+			fmt.Printf("Error: old admin token, please re-authenticate using `ente account add` \n")
+			return nil
+		}
+		return err
+	}
+	fmt.Printf("Successfully sent test email to %s\n", to)
+	return nil
+}
+
 func (c *ClICtrl) buildAdminContext(ctx context.Context, adminEmail string) (context.Context, error) {
 	accounts, err := c.GetAccounts(ctx)
 	if err != nil {

cli/pkg/sign_in.go

@@ -8,8 +8,8 @@ import (
 	eCrypto "github.com/ente-io/cli/internal/crypto"
 	"github.com/ente-io/cli/pkg/model"
 	"github.com/ente-io/cli/utils/browser"
+	"github.com/ente-io/cli/utils/constants"
 	"github.com/ente-io/cli/utils/encoding"
-	"github.com/spf13/viper"
 	"log"
 
 	"github.com/kong/go-srp"
@@ -145,7 +145,10 @@ func (c *ClICtrl) verifyPassKey(ctx context.Context, authResp *api.Authorization
 	if !authResp.IsPasskeyRequired() {
 		return authResp, nil
 	}
-	baseAccountUrl := viper.GetString("endpoint.accounts")
+	baseAccountUrl := constants.EnteAccountUrl
+	if authResp.AccountsUrl != "" {
+		baseAccountUrl = authResp.AccountsUrl
+	}
 	passkeyAuthUrl := fmt.Sprintf("%s/passkeys/verify?passkeySessionID=%s&redirect=ente-cli://passkey&clientPackage=%s", baseAccountUrl, authResp.PassKeySessionID, app.ClientPkg())
 	fmt.Printf("Open this url in browser to verify passkey: %s\n", passkeyAuthUrl)
 	err := browser.OpenURL(passkeyAuthUrl)
@@ -167,7 +170,7 @@ func (c *ClICtrl) verifyPassKey(ctx context.Context, authResp *api.Authorization
 }
 
 func (c *ClICtrl) validateEmail(ctx context.Context, email string) (*api.AuthorizationResponse, error) {
-	err := c.Client.SendEmailOTP(ctx, email)
+	err := c.Client.SendLoginOTP(ctx, email)
 	if err != nil {
 		return nil, err
 	}

desktop/.github/workflows/desktop-release.yml

@@ -44,8 +44,7 @@ jobs:
                   # If triggered by a tag, checkout photosd-$tag from the source
                   # repository. Otherwise checkout $source (default: "main").
                   repository: ente-io/ente
-                  ref:
-                      "${{ startsWith(github.ref, 'refs/tags/v') &&
+                  ref: "${{ startsWith(github.ref, 'refs/tags/v') &&
                       format('photosd-{0}', github.ref_name) || ( inputs.source
                       || 'main' ) }}"
 
@@ -110,9 +109,13 @@ jobs:
               env:
                   # macOS notarization credentials key details
                   APPLE_ID: ${{ secrets.APPLE_ID }}
-                  APPLE_APP_SPECIFIC_PASSWORD:
-                      ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
+                  APPLE_APP_SPECIFIC_PASSWORD: ${{ secrets.APPLE_APP_SPECIFIC_PASSWORD }}
                   APPLE_TEAM_ID: ${{ secrets.APPLE_TEAM_ID }}
+                  # Windows Azure Trusted Signing related values
+                  # https://www.electron.build/code-signing-win#using-azure-trusted-signing-beta
+                  AZURE_TENANT_ID: ${{ secrets.AZURE_TENANT_ID }}
+                  AZURE_CLIENT_ID: ${{ secrets.AZURE_CLIENT_ID }}
+                  AZURE_CLIENT_SECRET: ${{ secrets.AZURE_CLIENT_SECRET }}
                   # Default is "draft", but since our nightly builds update
                   # existing pre-releases, set this to "prerelease".
                   EP_PRE_RELEASE: true

desktop/.prettierrc.json

@@ -1,6 +1,5 @@
 {
     "tabWidth": 4,
-    "proseWrap": "always",
     "objectWrap": "collapse",
     "plugins": [
         "prettier-plugin-organize-imports",

desktop/CHANGELOG.md

@@ -1,9 +1,24 @@
 # CHANGELOG
 
-## v1.7.13 (Unreleased)
+## v1.7.15 (Unreleased)
 
+- Custom domains.
+- Support Czech translations.
 - .
 
+## v1.7.14
+
+- Increase file size limit to 10 GB.
+
+## v1.7.13
+
+- Generate streams for videos (beta)
+
+    > Streamable videos can be enabled in Preferences. For more details, see the
+    > [video streaming FAQ](https://help.ente.io/photos/faq/video-streaming).
+
+- Support Turkish translations.
+
 ## v1.7.12
 
 - Improved video player with streaming support (for already processed videos).

desktop/docs/dependencies.md

@@ -3,6 +3,7 @@
 - [Electron](#electron)
 - [Dev dependencies](#dev)
 - [Functionality](#functionality)
+- [Pinned](#pinned)
 
 ## Electron
 
@@ -140,3 +141,13 @@ handles to avoid reopening them for every operation.
 
 [chokidar](https://github.com/paulmillr/chokidar) is used as a file system
 watcher for the watch folders functionality.
+
+## Pinned
+
+- `electron-builder` is pinned to 26.0.14 because of
+  https://github.com/electron-userland/electron-builder/issues/9161
+
+    To reproduce this locally, add `x64ArchFiles: "ffmpeg"` to
+    `electron-builder.yml`, then run `node_modules/.bin/electron-builder --mac`.
+
+- `electron-store` is pinned to 8.2.0 because subsequent versions are ESM only.

desktop/electron-builder.yml

@@ -14,6 +14,11 @@ win:
     target:
         - target: nsis
           arch: [x64, arm64]
+    azureSignOptions:
+        publisherName: ENTE TECHNOLOGIES, INC.
+        endpoint: https://eus.codesigning.azure.net/
+        certificateProfileName: EnteTrustCertProfile
+        codeSigningAccountName: EnteTechnologiesInc
 nsis:
     deleteAppDataOnUninstall: true
 linux:

desktop/eslint.config.mjs

@@ -43,6 +43,11 @@ export default ts.config(
             //
             // See: [Note: non-null-assertions have better stack trace]
             "@typescript-eslint/no-non-null-assertion": "off",
+            // Allow `while(true)` etc.
+            "@typescript-eslint/no-unnecessary-condition": [
+                "error",
+                { allowConstantLoopConditions: true },
+            ],
         },
     },
 );

desktop/package.json

@@ -1,6 +1,6 @@
 {
     "name": "ente",
-    "version": "1.7.13-beta",
+    "version": "1.7.15-beta",
     "private": true,
     "description": "Desktop client for Ente Photos",
     "repository": "github:ente-io/photos-desktop",
@@ -21,9 +21,6 @@
         "lint-fix": "yarn prettier --write --log-level warn . && yarn eslint && yarn tsc",
         "prepare": "node scripts/vips.js"
     },
-    "resolutions": {
-        "jackspeak": "2.1.1"
-    },
     "dependencies": {
         "any-shell-escape": "^0.1.1",
         "auto-launch": "^5.0.6",
@@ -31,32 +28,32 @@
         "clip-bpe-js": "^0.0.6",
         "comlink": "^4.4.2",
         "compare-versions": "^6.1.1",
-        "electron-log": "^5.3.4",
+        "electron-log": "^5.4.1",
         "electron-store": "^8.2.0",
-        "electron-updater": "^6.6.2",
+        "electron-updater": "^6.6.5",
         "ffmpeg-static": "^5.2.0",
         "lru-cache": "^11.1.0",
         "next-electron-server": "^1.0.0",
         "node-stream-zip": "^1.15.0",
-        "onnxruntime-node": "^1.20.1"
+        "onnxruntime-node": "1.22.0-rev",
+        "zod": "^3.25.67"
     },
     "devDependencies": {
-        "@eslint/js": "^9.25.0",
-        "@tsconfig/node22": "^22.0.1",
+        "@eslint/js": "^9.30.1",
+        "@tsconfig/node22": "^22.0.2",
         "@types/auto-launch": "^5.0.5",
-        "@types/ffmpeg-static": "^3.0.3",
         "ajv": "^8.17.1",
-        "concurrently": "^9.1.2",
+        "concurrently": "^9.2.0",
         "cross-env": "^7.0.3",
-        "electron": "^35.2.0",
-        "electron-builder": "^26.0.12",
+        "electron": "^37.1.0",
+        "electron-builder": "26.0.14",
         "eslint": "^9",
-        "prettier": "3.5.3",
+        "prettier": "3.6.2",
         "prettier-plugin-organize-imports": "^4.1.0",
-        "prettier-plugin-packagejson": "^2.5.10",
-        "shx": "^0.3.4",
+        "prettier-plugin-packagejson": "^2.5.17",
+        "shx": "^0.4.0",
         "typescript": "^5.8.3",
-        "typescript-eslint": "^8.30.1"
+        "typescript-eslint": "^8.35.1"
     },
     "packageManager": "yarn@1.22.22",
     "productName": "ente"

desktop/src/main/ipc.ts

@@ -13,8 +13,10 @@ import type { BrowserWindow } from "electron";
 import { ipcMain } from "electron/main";
 import type {
     CollectionMapping,
+    FFmpegCommand,
     FolderWatch,
     PendingUploads,
+    UtilityProcessType,
     ZipItem,
 } from "../types/ipc";
 import { logToDisk } from "./log";
@@ -30,7 +32,7 @@ import {
     openLogDirectory,
     selectDirectory,
 } from "./services/dir";
-import { ffmpegExec } from "./services/ffmpeg";
+import { ffmpegDetermineVideoDuration, ffmpegExec } from "./services/ffmpeg";
 import {
     fsExists,
     fsFindFiles,
@@ -40,23 +42,23 @@ import {
     fsRename,
     fsRm,
     fsRmdir,
+    fsStatMtime,
     fsWriteFile,
     fsWriteFileViaBackup,
 } from "./services/fs";
 import { convertToJPEG, generateImageThumbnail } from "./services/image";
 import { logout } from "./services/logout";
-import { createMLWorker } from "./services/ml";
 import {
     lastShownChangelogVersion,
-    masterKeyB64,
-    saveMasterKeyB64,
+    masterKeyFromSafeStorage,
+    saveMasterKeyInSafeStorage,
     setLastShownChangelogVersion,
 } from "./services/store";
 import {
     clearPendingUploads,
     listZipItems,
-    markUploadedFiles,
-    markUploadedZipItems,
+    markUploadedFile,
+    markUploadedZipItem,
     pathOrZipItemSize,
     pendingUploads,
     setPendingUploads,
@@ -68,6 +70,7 @@ import {
     watchUpdateIgnoredFiles,
     watchUpdateSyncedFiles,
 } from "./services/watch";
+import { triggerCreateUtilityProcess } from "./services/workers";
 
 /**
  * Listen for IPC events sent/invoked by the renderer process, and route them to
@@ -105,10 +108,12 @@ export const attachIPCHandlers = () => {
 
     ipcMain.handle("selectDirectory", () => selectDirectory());
 
-    ipcMain.handle("masterKeyB64", () => masterKeyB64());
+    ipcMain.handle("masterKeyFromSafeStorage", () =>
+        masterKeyFromSafeStorage(),
+    );
 
-    ipcMain.handle("saveMasterKeyB64", (_, masterKeyB64: string) =>
-        saveMasterKeyB64(masterKeyB64),
+    ipcMain.handle("saveMasterKeyInSafeStorage", (_, masterKey: string) =>
+        saveMasterKeyInSafeStorage(masterKey),
     );
 
     ipcMain.handle("lastShownChangelogVersion", () =>
@@ -163,6 +168,8 @@ export const attachIPCHandlers = () => {
 
     ipcMain.handle("fsIsDir", (_, dirPath: string) => fsIsDir(dirPath));
 
+    ipcMain.handle("fsStatMtime", (_, path: string) => fsStatMtime(path));
+
     ipcMain.handle("fsFindFiles", (_, folderPath: string) =>
         fsFindFiles(folderPath),
     );
@@ -177,20 +184,26 @@ export const attachIPCHandlers = () => {
         "generateImageThumbnail",
         (
             _,
-            dataOrPathOrZipItem: Uint8Array | string | ZipItem,
+            pathOrZipItem: string | ZipItem,
             maxDimension: number,
             maxSize: number,
-        ) => generateImageThumbnail(dataOrPathOrZipItem, maxDimension, maxSize),
+        ) => generateImageThumbnail(pathOrZipItem, maxDimension, maxSize),
     );
 
     ipcMain.handle(
         "ffmpegExec",
         (
             _,
-            command: string[],
-            dataOrPathOrZipItem: Uint8Array | string | ZipItem,
+            command: FFmpegCommand,
+            pathOrZipItem: string | ZipItem,
             outputFileExtension: string,
-        ) => ffmpegExec(command, dataOrPathOrZipItem, outputFileExtension),
+        ) => ffmpegExec(command, pathOrZipItem, outputFileExtension),
+    );
+
+    ipcMain.handle(
+        "ffmpegDetermineVideoDuration",
+        (_, pathOrZipItem: string | ZipItem) =>
+            ffmpegDetermineVideoDuration(pathOrZipItem),
     );
 
     // - Upload
@@ -210,13 +223,15 @@ export const attachIPCHandlers = () => {
     );
 
     ipcMain.handle(
-        "markUploadedFiles",
-        (_, paths: PendingUploads["filePaths"]) => markUploadedFiles(paths),
+        "markUploadedFile",
+        (_, path: string, associatedPath: string | undefined) =>
+            markUploadedFile(path, associatedPath),
     );
 
     ipcMain.handle(
-        "markUploadedZipItems",
-        (_, items: PendingUploads["zipItems"]) => markUploadedZipItems(items),
+        "markUploadedZipItem",
+        (_, item: ZipItem, associatedItem: ZipItem | undefined) =>
+            markUploadedZipItem(item, associatedItem),
     );
 
     ipcMain.handle("clearPendingUploads", () => clearPendingUploads());
@@ -227,9 +242,11 @@ export const attachIPCHandlers = () => {
  * the main window to do their thing.
  */
 export const attachMainWindowIPCHandlers = (mainWindow: BrowserWindow) => {
-    // - ML
+    // - Utility processes
 
-    ipcMain.on("createMLWorker", () => createMLWorker(mainWindow));
+    ipcMain.on("triggerCreateUtilityProcess", (_, type: UtilityProcessType) =>
+        triggerCreateUtilityProcess(type, mainWindow),
+    );
 };
 
 /**

desktop/src/main/log-worker.ts

@@ -0,0 +1,59 @@
+/**
+ * A object that behaves similar to the default export of "./log", except this
+ * can be used from within a utility process.
+ *
+ * ---
+ *
+ * We cannot directly do
+ *
+ *     import log from "../log";
+ *
+ * because that requires the Electron APIs that are not available to a utility
+ * process (See: [Note: Using Electron APIs in UtilityProcess]).
+ *
+ * But even if that were to work, logging will still be problematic since we'd
+ * try opening the log file from two different Node.js processes (this one, and
+ * the main one), and I didn't find any indication in the electron-log
+ * repository that the log file's integrity would be maintained in such cases.
+ *
+ * So instead we provide this proxy log object that uses the
+ * `process.parentPort` to transport the logs over to the main process, where
+ * the {@link processUtilityProcessLogMessage} function in the main process is
+ * expected to handle these (sending them to the actual log).
+ */
+export default {
+    error: (s: string, e?: unknown) =>
+        mainProcess("log.errorString", messageWithError(s, e)),
+    warn: (s: string, e?: unknown) =>
+        mainProcess("log.warnString", messageWithError(s, e)),
+    info: (...ms: unknown[]) => mainProcess("log.info", ms),
+    /**
+     * Unlike the real {@link log.debug}, this is (a) eagerly evaluated, and (b)
+     * accepts only strings.
+     */
+    debugString: (s: string) => mainProcess("log.debugString", s),
+};
+
+/**
+ * Send a message to the main process using a barebones RPC protocol.
+ */
+const mainProcess = (method: string, param: unknown) =>
+    process.parentPort.postMessage({ method, p: param });
+
+// Duplicated verbatim from ./log.ts
+const messageWithError = (message: string, e?: unknown) => {
+    if (!e) return message;
+
+    let es: string;
+    if (e instanceof Error) {
+        // In practice, we expect ourselves to be called with Error objects, so
+        // this is the happy path so to say.
+        es = [`${e.name}: ${e.message}`, e.stack].filter((x) => x).join("\n");
+    } else {
+        // For the rest rare cases, use the default string serialization of e.
+        // eslint-disable-next-line @typescript-eslint/no-base-to-string
+        es = String(e);
+    }
+
+    return `${message}: ${es}`;
+};

desktop/src/main/log.ts

@@ -83,6 +83,56 @@ const logDebug = (param: () => unknown) => {
     }
 };
 
+/**
+ * Handle log messages posted from the utility process in the main process.
+ *
+ * See: [Note: Using Electron APIs in UtilityProcess]
+ *
+ * @param message The arbitrary message that was received as an argument to the
+ * "message" event invoked on a {@link UtilityProcess}.
+ *
+ * @returns true if the message was recognized and handled, and false otherwise.
+ */
+export const processUtilityProcessLogMessage = (
+    logTag: string,
+    message: unknown,
+) => {
+    const m = message; /* shorter alias */
+    if (m && typeof m == "object" && "method" in m && "p" in m) {
+        const p = m.p;
+        switch (m.method) {
+            case "log.errorString":
+                if (typeof p == "string") {
+                    logError(`${logTag} ${p}`);
+                    return true;
+                }
+                break;
+            case "log.warnString":
+                if (typeof p == "string") {
+                    logWarn(`${logTag} ${p}`);
+                    return true;
+                }
+                break;
+            case "log.info":
+                if (Array.isArray(p)) {
+                    // Need to cast from any[] to unknown[]
+                    logInfo(logTag, ...(p as unknown[]));
+                    return true;
+                }
+                break;
+            case "log.debugString":
+                if (typeof p == "string") {
+                    logDebug(() => `${logTag} ${p}`);
+                    return true;
+                }
+                break;
+            default:
+                break;
+        }
+    }
+    return false;
+};
+
 /**
  * Ente's logger.
  *

desktop/src/main/services/ffmpeg.ts

@@ -1,480 +1,77 @@
-import pathToFfmpeg from "ffmpeg-static";
-import { randomBytes } from "node:crypto";
+/**
+ * @file A bridge to the ffmpeg utility process. This code runs in the main
+ * process.
+ */
+
+import { wrap } from "comlink";
 import fs from "node:fs/promises";
-import path from "node:path";
-import type { ZipItem } from "../../types/ipc";
-import log from "../log";
-import { execAsync } from "../utils/electron";
+import type { FFmpegCommand, ZipItem } from "../../types/ipc";
 import {
     deleteTempFileIgnoringErrors,
-    makeFileForDataOrPathOrZipItem,
+    makeFileForStreamOrPathOrZipItem,
     makeTempFilePath,
 } from "../utils/temp";
-
-/* Ditto in the web app's code (used by the Wasm FFmpeg invocation). */
-const ffmpegPathPlaceholder = "FFMPEG";
-const inputPathPlaceholder = "INPUT";
-const outputPathPlaceholder = "OUTPUT";
+import type { FFmpegUtilityProcess } from "./ffmpeg-worker";
+import { ffmpegUtilityProcessEndpoint } from "./workers";
 
 /**
- * Run a FFmpeg command
- *
- * [Note: FFmpeg in Electron]
- *
- * There is a Wasm build of FFmpeg, but that is currently 10-20 times slower
- * that the native build. That is slow enough to be unusable for our purposes.
- * https://ffmpegwasm.netlify.app/docs/performance
- *
- * So the alternative is to bundle a FFmpeg executable binary with our app. e.g.
- *
- *     yarn add fluent-ffmpeg ffmpeg-static ffprobe-static
- *
- * (we only use ffmpeg-static, the rest are mentioned for completeness' sake).
- *
- * Interestingly, Electron already bundles an binary FFmpeg library (it comes
- * from the ffmpeg fork maintained by Chromium).
- * https://chromium.googlesource.com/chromium/third_party/ffmpeg
- * https://stackoverflow.com/questions/53963672/what-version-of-ffmpeg-is-bundled-inside-electron
- *
- * This can be found in (e.g. on macOS) at
- *
- *     $ file ente.app/Contents/Frameworks/Electron\ Framework.framework/Versions/Current/Libraries/libffmpeg.dylib
- *     .../libffmpeg.dylib: Mach-O 64-bit dynamically linked shared library arm64
- *
- * But I'm not sure if our code is supposed to be able to use it, and how.
+ * Return a handle to the ffmpeg utility process, starting it if needed.
+ */
+export const ffmpegUtilityProcess = () =>
+    ffmpegUtilityProcessEndpoint().then((port) =>
+        wrap<FFmpegUtilityProcess>(port),
+    );
+
+/**
+ * Implement the IPC "ffmpegExec" contract, writing the input and output to
+ * temporary files as needed, and then forward to the {@link ffmpegExec} running
+ * in the utility process.
  */
 export const ffmpegExec = async (
-    command: string[],
-    dataOrPathOrZipItem: Uint8Array | string | ZipItem,
+    command: FFmpegCommand,
+    pathOrZipItem: string | ZipItem,
     outputFileExtension: string,
-): Promise<Uint8Array> => {
+): Promise<Uint8Array> =>
+    withInputFile(pathOrZipItem, async (worker, inputFilePath) => {
+        const outputFilePath = await makeTempFilePath(outputFileExtension);
+        try {
+            await worker.ffmpegExec(command, inputFilePath, outputFilePath);
+            return await fs.readFile(outputFilePath);
+        } finally {
+            await deleteTempFileIgnoringErrors(outputFilePath);
+        }
+    });
+
+export const withInputFile = async <T>(
+    pathOrZipItem: string | ZipItem,
+    f: (worker: FFmpegUtilityProcess, inputFilePath: string) => Promise<T>,
+): Promise<T> => {
+    const worker = await ffmpegUtilityProcess();
+
     const {
         path: inputFilePath,
         isFileTemporary: isInputFileTemporary,
         writeToTemporaryFile: writeToTemporaryInputFile,
-    } = await makeFileForDataOrPathOrZipItem(dataOrPathOrZipItem);
+    } = await makeFileForStreamOrPathOrZipItem(pathOrZipItem);
 
-    const outputFilePath = await makeTempFilePath(outputFileExtension);
     try {
         await writeToTemporaryInputFile();
 
-        const cmd = substitutePlaceholders(
-            command,
-            inputFilePath,
-            outputFilePath,
-        );
-
-        await execAsync(cmd);
-
-        return await fs.readFile(outputFilePath);
+        return await f(worker, inputFilePath);
     } finally {
         if (isInputFileTemporary)
             await deleteTempFileIgnoringErrors(inputFilePath);
-        await deleteTempFileIgnoringErrors(outputFilePath);
     }
 };
 
-const substitutePlaceholders = (
-    command: string[],
-    inputFilePath: string,
-    outputFilePath: string,
-) =>
-    command.map((segment) => {
-        if (segment == ffmpegPathPlaceholder) {
-            return ffmpegBinaryPath();
-        } else if (segment == inputPathPlaceholder) {
-            return inputFilePath;
-        } else if (segment == outputPathPlaceholder) {
-            return outputFilePath;
-        } else {
-            return segment;
-        }
-    });
-
 /**
- * Return the path to the `ffmpeg` binary.
- *
- * At runtime, the FFmpeg binary is present in a path like (macOS example):
- * `ente.app/Contents/Resources/app.asar.unpacked/node_modules/ffmpeg-static/ffmpeg`
+ * Implement the IPC "ffmpegDetermineVideoDuration" contract, writing the input
+ * to temporary files as needed, and then forward to the
+ * {@link ffmpegDetermineVideoDuration} running in the utility process.
  */
-const ffmpegBinaryPath = () => {
-    // This substitution of app.asar by app.asar.unpacked is suggested by the
-    // ffmpeg-static library author themselves:
-    // https://github.com/eugeneware/ffmpeg-static/issues/16
-    return pathToFfmpeg!.replace("app.asar", "app.asar.unpacked");
-};
-
-/**
- * A variant of {@link ffmpegExec} adapted to work with streams so that it can
- * handle the MP4 conversion of large video files.
- *
- * @param inputFilePath The path to a file on the user's local file system. This
- * is the video we want to convert.
- *
- * @param outputFilePath The path to a file on the user's local file system where
- * we should write the converted MP4 video.
- */
-export const ffmpegConvertToMP4 = async (
-    inputFilePath: string,
-    outputFilePath: string,
-): Promise<void> => {
-    const command = [
-        ffmpegPathPlaceholder,
-        "-i",
-        inputPathPlaceholder,
-        "-preset",
-        "ultrafast",
-        outputPathPlaceholder,
-    ];
-
-    const cmd = substitutePlaceholders(command, inputFilePath, outputFilePath);
-
-    await execAsync(cmd);
-};
-
-export interface FFmpegGenerateHLSPlaylistAndSegmentsResult {
-    playlistPath: string;
-    videoPath: string;
-    dimensions: { width: number; height: number };
-}
-
-/**
- * A bespoke variant of {@link ffmpegExec} for generation of HLS playlists for
- * videos.
- *
- * See: [Note: Preview variant of videos]
-
- * @param inputFilePath The path to a file on the user's local file system. This
- * is the video we want to generate an streamable HLS playlist for.
- *
- * @param outputPathPrefix The path to unique, unused and temporary prefix on
- * the user's local file system. This function will write the generated HLS
- * playlist and video segments under this prefix.
- *
- * @returns The paths to two files on the user's local file system - one
- * containing the generated HLS playlist, and the other containing the
- * transcoded and encrypted video segments that the HLS playlist refers to.
- */
-export const ffmpegGenerateHLSPlaylistAndSegments = async (
-    inputFilePath: string,
-    outputPathPrefix: string,
-): Promise<FFmpegGenerateHLSPlaylistAndSegmentsResult> => {
-    // [Note: Tonemapping HDR to HD]
-    //
-    // BT.709 ("HD") is a standard that describes things like how color is
-    // encoded, the range of values, and their "meaning" - i.e. how to map the
-    // values in the video to the pixels on the screen.
-    //
-    // It is not the only such standard, there are three common examples:
-    //
-    // - BT.601 ("Standard-Definition" or SD)
-    // - BT.709 ("High-Definition" or HD)
-    // - BT.2020 ("Ultra-High-Definition" or UHD, aka HDR^).
-    //
-    // ^ HDR ("High-Dynamic-Range") is an addendum to BT.2020, but for our
-    //   purpose here we can treat it as as alias.
-    //
-    // BT.709 is the most common amongst these for older files out stored on
-    // computers, and they conform mostly to the standard (one notable exception
-    // is that the BT.709 standard also recommends using the yuv422p pixel
-    // format, but de facto yuv420p is used because many video players only
-    // support yuv420p).
-    //
-    // Since BT.709 is the most widely supported standard, we use it when
-    // generating the HLS playlist so to allow playback across the widest
-    // possible hardware/OS/browser combinations.
-    //
-    // If we convert HDR to HD without naively, then the colors look washed out
-    // compared to the original. To resolve this, we use a ffmpeg filterchain
-    // that uses the tonemap filter.
-    //
-    // However applying this tonemap to videos that are already HD leads to a
-    // brightness drop. So we conditionally apply this filter chain only if the
-    // colorspace is not already BT.709.
-    //
-    // Reference:
-    // - https://trac.ffmpeg.org/wiki/colorspace
-    const isBT709 = await detectIsBT709(inputFilePath);
-
-    // We want the generated playlist to refer to the chunks as "output.ts".
-    //
-    // So we arrange things accordingly: We use the `outputPathPrefix` as our
-    // working directory, and then ask ffmpeg to generate a playlist with the
-    // name "output.m3u8".
-    //
-    // ffmpeg will automatically place the segments in a file with the same base
-    // name as the playlist, but with a ".ts" extension. And since we use the
-    // "single_file" option, all the segments will be placed in a file named
-    // "output.ts".
-
-    await fs.mkdir(outputPathPrefix);
-
-    const playlistPath = path.join(outputPathPrefix, "output.m3u8");
-    const videoPath = path.join(outputPathPrefix, "output.ts");
-
-    // Generate a cryptographically secure random key (16 bytes).
-    const keyBytes = randomBytes(16);
-    const keyB64 = keyBytes.toString("base64");
-
-    // Convert it to a data: URI that will be added to the playlist.
-    const keyURI = `data:text/plain;base64,${keyB64}`;
-
-    // Determine two paths - one where we will write the key itself, and where
-    // we will write the "key info" that provides ffmpeg the `keyURI` and the
-    // `keyPath;.
-    const keyPath = playlistPath + ".key";
-    const keyInfoPath = playlistPath + ".key-info";
-
-    // Generate a "key info":
-    //
-    // - the first line specifies the key URI that is written into the playlist.
-    // - the second line specifies the path to the local filesystem file from
-    //   where ffmpeg should read the key.
-    const keyInfo = [keyURI, keyPath].join("\n");
-
-    // Overview:
-    //
-    // - H.264 video HD 720p 30fps.
-    // - AAC audio 128kbps.
-    // - Encrypted HLS playlist with a single file containing all the chunks.
-    //
-    // Reference:
-    // - `man ffmpeg-all`
-    // - https://trac.ffmpeg.org/wiki/Encode/H.264
-    //
-    const command = [
-        ffmpegBinaryPath(),
-        // Reduce the amount of output lines we have to parse.
-        ["-hide_banner"],
-        // Input file. We don't need any extra options that apply to the input file.
-        "-i",
-        inputFilePath,
-        // The remaining options apply to the next output file (`playlistPath`).
-        //
-        // ---
-        //
-        // `-vf` creates a filter graph for the video stream. This is a string
-        // of the form `filter1=key=value:key=value.filter2=key=value`, that is,
-        // a comma separated list of filters chained together.
-        [
-            "-vf",
-            [
-                // Scales the video to maximum 720p height, keeping aspect
-                // ratio, and keeping the calculated dimension divisible by 2
-                // (some of the other operations require an even pixel count).
-                "scale=-2:720",
-                // Convert the video to a constant 30 fps, duplicating or
-                // dropping frames as necessary.
-                "fps=30",
-                // If the video is not in the HD color space (bt709), convert
-                // it. Before conversion, tone map colors so that they work the
-                // same across the change in the dyamic range.
-                //
-                // 1. The tonemap filter only works linear light, so we first
-                //    use zscale with transfer=linear to linearize the input.
-                //
-                // 2. Then we use the tonemap, with the hable option that is
-                //    best for preserving details. desat=0 turns off the default
-                //    desaturation.
-                //
-                // 3. Use zscale again to "convert to BT.709" by asking it to
-                //    set the all three of color primaries, transfer
-                //    characteristics and colorspace matrix to 709 (Note: the
-                //    constants specified in the tonemap filter help do not
-                //    include the "bt" prefix)
-                //
-                // See: https://ffmpeg.org/ffmpeg-filters.html#tonemap-1
-                //
-                // See: [Note: Tonemapping HDR to HD]
-                isBT709
-                    ? []
-                    : [
-                          "zscale=transfer=linear",
-                          "tonemap=tonemap=hable:desat=0",
-                          "zscale=primaries=709:transfer=709:matrix=709",
-                      ],
-                // Output using the most widely supported pixel format: 8-bit
-                // YUV planar color space with 4:2:0 chroma subsampling.
-                "format=yuv420p",
-            ]
-                .flat()
-                .join(","),
-        ],
-        // Video codec H.264
-        //
-        // - `-c:v libx264` converts the video stream to use the H.264 codec.
-        //
-        // - We don't supply a bitrate, instead it uses the default CRF ("23")
-        //   as recommended in the ffmpeg trac.
-        //
-        // - We don't supply a preset, it'll use the default ("medium")
-        ["-c:v", "libx264"],
-        // Audio codec AAC
-        //
-        // - `-c:a aac` converts the audio stream to use the AAC codec
-        //
-        // - We don't supply a bitrate, it'll use the AAC default 128k bps.
-        ["-c:a", "aac"],
-        // Generate a HLS playlist.
-        ["-f", "hls"],
-        // Tell ffmpeg where to find the key, and the URI for the key to write
-        // into the generated playlist. Implies "-hls_enc 1".
-        ["-hls_key_info_file", keyInfoPath],
-        // Generate as many playlist entries as needed (default limit is 5).
-        ["-hls_list_size", "0"],
-        // Place all the video segments within the same .ts file (with the same
-        // path as the playlist file but with a ".ts" extension).
-        ["-hls_flags", "single_file"],
-        // Output path where the playlist should be generated.
-        playlistPath,
-    ].flat();
-
-    let dimensions: ReturnType<typeof detectVideoDimensions>;
-
-    try {
-        // Write the key and the keyInfo to their desired paths.
-        await Promise.all([
-            fs.writeFile(keyPath, keyBytes),
-            fs.writeFile(keyInfoPath, keyInfo, { encoding: "utf8" }),
-        ]);
-
-        // Run the ffmpeg command to generate the HLS playlist and segments.
-        //
-        // Note: Depending on the size of the input file, this may take long!
-        const { stderr: conversionStderr } = await execAsync(command);
-
-        // Determine the dimensions of the generated video from the stderr
-        // output produced by ffmpeg during the conversion.
-        dimensions = detectVideoDimensions(conversionStderr);
-    } catch (e) {
-        log.error("HLS generation failed", e);
-        await Promise.all([
-            deleteTempFileIgnoringErrors(playlistPath),
-            deleteTempFileIgnoringErrors(videoPath),
-        ]);
-        throw e;
-    } finally {
-        await Promise.all([
-            deleteTempFileIgnoringErrors(keyInfoPath),
-            deleteTempFileIgnoringErrors(keyPath),
-            // ffmpeg writes a /path/output.ts.tmp, clear it out too.
-            deleteTempFileIgnoringErrors(videoPath + ".tmp"),
-        ]);
-    }
-
-    return { playlistPath, videoPath, dimensions };
-};
-
-/**
- * A regex that matches the first line of the form
- *
- *     Stream #0:0: Video: h264 (High 10) ([27][0][0][0] / 0x001B), yuv420p10le(tv, bt2020nc/bt2020/arib-std-b67), 1920x1080, 30 fps, 30 tbr, 90k tbn
- *
- * The part after Video: is the first capture group.
- */
-const videoStreamLineRegex = /Stream #.+: Video:(.+)\n/;
-
-/** {@link videoStreamLineRegex}, but global. */
-const videoStreamLinesRegex = /Stream #.+: Video:(.+)\n/g;
-
-/**
- * A regex that matches <digits>x<digits> pair preceded by a space and followed
- * by a trailing comma. See {@link videoStreamLineRegex} for the context in
- * which it is used.
- */
-const videoDimensionsRegex = / (\d+)x(\d+),/;
-
-/**
- * Heuristically determine if the given video uses the BT.709 colorspace.
- *
- * This function tries to determine the input colorspace by scanning the ffmpeg
- * info output for the video stream line, and checking if it contains the string
- * "bt709". See: [Note: Parsing CLI output might break on ffmpeg updates].
- */
-const detectIsBT709 = async (inputFilePath: string) => {
-    const videoInfo = await pseudoFFProbeVideo(inputFilePath);
-    const videoStreamLine = videoStreamLineRegex.exec(videoInfo)?.at(1);
-    return !!videoStreamLine?.includes("bt709");
-};
-
-/**
- * Heuristically detect the dimensions of the given video from the log output of
- * the ffmpeg invocation during the HLS playlist generation.
- *
- * This function tries to determine the width and height of the generated video
- * from the output log written by ffmpeg on its stderr during the generation
- * process, scanning it for the last video stream line, and trying to match a
- * "<digits>x<digits>" regex.
- *
- * See: [Note: Parsing CLI output might break on ffmpeg updates].
- */
-const detectVideoDimensions = (conversionStderr: string) => {
-    // There is a nicer way to do it - by running `pseudoFFProbeVideo` on the
-    // generated playlist. However, that playlist includes a data URL that
-    // specifies the encryption info, and ffmpeg refuses to read that unless we
-    // specify the "-allowed_extensions ALL" or something to that effect.
-    //
-    // Unfortunately, our current ffmpeg binary (5.x) does not support that
-    // option. So we instead parse the conversion output itself.
-    //
-    // This is also nice, since it saves on an extra ffmpeg invocation. But we
-    // now need to be careful to find the right video stream line, since the
-    // conversion output includes both the input and output video stream lines.
-    //
-    // To match the right (output) video stream line, we use a global regex, and
-    // use the last match since that'd correspond to the single video stream
-    // written in the output.
-    const videoStreamLine = Array.from(
-        conversionStderr.matchAll(videoStreamLinesRegex),
-    )
-        .at(-1) /* Last Stream...: Video: line in the output */
-        ?.at(1); /* First capture group */
-    if (videoStreamLine) {
-        const [, ws, hs] = videoDimensionsRegex.exec(videoStreamLine) ?? [];
-        if (ws && hs) {
-            const w = parseInt(ws);
-            const h = parseInt(hs);
-            if (w && h) {
-                return { width: w, height: h };
-            }
-        }
-    }
-    throw new Error(
-        `Unable to detect video dimensions from stream line [${videoStreamLine ?? ""}]`,
+export const ffmpegDetermineVideoDuration = async (
+    pathOrZipItem: string | ZipItem,
+): Promise<number> =>
+    withInputFile(pathOrZipItem, async (worker, inputFilePath) =>
+        worker.ffmpegDetermineVideoDuration(inputFilePath),
     );
-};
-
-/**
- * We don't have the ffprobe binary at hand, so we make do by grepping the log
- * output of ffmpeg.
- *
- * > [Note: Parsing CLI output might break on ffmpeg updates]
- * >
- * > Needless to say, while this works currently, this is liable to break in the
- * > future. So if something stops working after updating ffmpeg, look here!
- *
- * @returns the stderr of ffmpeg after running it on the input file. The exact
- * command we run is:
- *
- *     ffmpeg -i in.mov -an -frames:v 0 -f null - 2>info.txt
- *
- * And the returned string is the contents of the `info.txt` thus produced.
- */
-const pseudoFFProbeVideo = async (inputFilePath: string) => {
-    const command = [
-        ffmpegPathPlaceholder,
-        // Reduce the amount of output lines we have to parse.
-        ["-hide_banner"],
-        ["-i", inputPathPlaceholder],
-        "-an",
-        ["-frames:v", "0"],
-        ["-f", "null"],
-        "-",
-    ].flat();
-
-    const cmd = substitutePlaceholders(command, inputFilePath, /* NA */ "");
-
-    const { stderr } = await execAsync(cmd);
-
-    return stderr;
-};

desktop/src/main/services/fs.ts

@@ -36,6 +36,17 @@ export const fsIsDir = async (dirPath: string) => {
     return stat.isDirectory();
 };
 
+export const fsStatMtime = (path: string) =>
+    // [Note: Integral last modified time]
+    //
+    // Whenever we need to find the modified time of a file, use the
+    // `mtime.getTime()` instead of `mtimeMs` of the stat; this way, it is
+    // guaranteed that the times are integral (we persist these values to remote
+    // in some cases, and the contract is for them to be integral; mtimeMs is a
+    // float with sub-millisecond precision), and that all places use the same
+    // value so that they're comparable.
+    fs.stat(path).then((st) => st.mtime.getTime());
+
 export const fsFindFiles = async (dirPath: string) => {
     const items = await fs.readdir(dirPath, { withFileTypes: true });
     let paths: string[] = [];

desktop/src/main/services/image.ts

@@ -6,7 +6,7 @@ import { type ZipItem } from "../../types/ipc";
 import { execAsync, isDev } from "../utils/electron";
 import {
     deleteTempFileIgnoringErrors,
-    makeFileForDataOrPathOrZipItem,
+    makeFileForStreamOrPathOrZipItem,
     makeTempFilePath,
 } from "../utils/temp";
 
@@ -61,7 +61,7 @@ const vipsPath = () =>
     );
 
 export const generateImageThumbnail = async (
-    dataOrPathOrZipItem: Uint8Array | string | ZipItem,
+    pathOrZipItem: string | ZipItem,
     maxDimension: number,
     maxSize: number,
 ): Promise<Uint8Array> => {
@@ -69,7 +69,7 @@ export const generateImageThumbnail = async (
         path: inputFilePath,
         isFileTemporary: isInputFileTemporary,
         writeToTemporaryFile: writeToTemporaryInputFile,
-    } = await makeFileForDataOrPathOrZipItem(dataOrPathOrZipItem);
+    } = await makeFileForStreamOrPathOrZipItem(pathOrZipItem);
 
     const outputFilePath = await makeTempFilePath("jpeg");
 

desktop/src/main/services/logout.ts

@@ -3,6 +3,7 @@ import log from "../log";
 import { clearPendingVideoResults } from "../stream";
 import { clearStores } from "./store";
 import { watchReset } from "./watch";
+import { terminateUtilityProcesses } from "./workers";
 import { clearOpenZipCache } from "./zip";
 
 /**
@@ -36,4 +37,9 @@ export const logout = (watcher: FSWatcher) => {
     } catch (e) {
         ignoreError("zip cache", e);
     }
+    try {
+        terminateUtilityProcesses();
+    } catch (e) {
+        ignoreError("utility processes", e);
+    }
 };

desktop/src/main/services/ml-worker.ts

@@ -15,46 +15,16 @@ import { existsSync } from "fs";
 import fs from "node:fs/promises";
 import path from "node:path";
 import * as ort from "onnxruntime-node";
+import { z } from "zod/v4";
+import log from "../log-worker";
 import { messagePortMainEndpoint } from "../utils/comlink";
 import { wait } from "../utils/common";
 import { writeStream } from "../utils/stream";
+import { fsStatMtime } from "./fs";
 
-/**
- * We cannot do
- *
- *     import log from "../log";
- *
- * because that requires the Electron APIs that are not available to a utility
- * process (See: [Note: Using Electron APIs in UtilityProcess]). But even if
- * that were to work, logging will still be problematic since we'd try opening
- * the log file from two different Node.js processes (this one, and the main
- * one), and I didn't find any indication in the electron-log repository that
- * the log file's integrity would be maintained in such cases.
- *
- * So instead we create this proxy log object that uses `process.parentPort` to
- * transport the logs over to the main process.
- */
-const log = {
-    /**
-     * Unlike the real {@link log.error}, this accepts only the first string
-     * argument, not the second optional error one.
-     */
-    errorString: (s: string) => mainProcess("log.errorString", s),
-    info: (...ms: unknown[]) => mainProcess("log.info", ms),
-    /**
-     * Unlike the real {@link log.debug}, this is (a) eagerly evaluated, and (b)
-     * accepts only strings.
-     */
-    debugString: (s: string) => mainProcess("log.debugString", s),
-};
+log.debugString("Started ML utility process");
 
-/**
- * Send a message to the main process using a barebones RPC protocol.
- */
-const mainProcess = (method: string, param: unknown) =>
-    process.parentPort.postMessage({ method, p: param });
-
-log.debugString(`Started ML worker process`);
+process.on("uncaughtException", (e, origin) => log.error(origin, e));
 
 process.parentPort.once("message", (e) => {
     // Initialize ourselves with the data we got from our parent.
@@ -63,6 +33,7 @@ process.parentPort.once("message", (e) => {
     // parent.
     expose(
         {
+            fsStatMtime,
             computeCLIPImageEmbedding,
             computeCLIPTextEmbeddingIfAvailable,
             detectFaces,
@@ -82,17 +53,10 @@ let _userDataPath: string | undefined;
 /** Equivalent to app.getPath("userData") */
 const userDataPath = () => _userDataPath!;
 
+const MLWorkerInitData = z.object({ userDataPath: z.string() });
+
 const parseInitData = (data: unknown) => {
-    if (
-        data &&
-        typeof data == "object" &&
-        "userDataPath" in data &&
-        typeof data.userDataPath == "string"
-    ) {
-        _userDataPath = data.userDataPath;
-    } else {
-        log.errorString("Unparseable initialization data");
-    }
+    _userDataPath = MLWorkerInitData.parse(data).userDataPath;
 };
 
 /**
@@ -159,7 +123,7 @@ const modelPathDownloadingIfNeeded = async (
     } else {
         const size = (await fs.stat(modelPath)).size;
         if (size !== expectedByteSize) {
-            log.errorString(
+            log.error(
                 `The size ${size} of model ${modelName} does not match the expected size, downloading again`,
             );
             await downloadModel(modelPath, modelName);
@@ -220,14 +184,13 @@ const downloadModel = async (saveLocation: string, name: string) => {
 /**
  * Create an ONNX {@link InferenceSession} with some defaults.
  */
-const createInferenceSession = async (modelPath: string) => {
-    return await ort.InferenceSession.create(modelPath, {
+const createInferenceSession = (modelPath: string) =>
+    ort.InferenceSession.create(modelPath, {
         // Restrict the number of threads to 1.
         intraOpNumThreads: 1,
         // Be more conservative with RAM usage.
         enableCpuMemArena: false,
     });
-};
 
 const cachedCLIPImageSession = makeCachedInferenceSession(
     "mobileclip_s2_image_opset18_rgba_opt.onnx",
@@ -269,9 +232,11 @@ const getTokenizer = () => (_tokenizer ??= new Tokenizer());
 export const computeCLIPTextEmbeddingIfAvailable = async (text: string) => {
     const sessionOrSkip = await Promise.race([
         cachedCLIPTextSession(),
-        // Wait for a tick to get the session promise to resolved the first time
-        // this code runs on each app start (and the model has been downloaded).
-        wait(0).then(() => 1),
+        // Wait a bit to get the session promise to resolved the first time this
+        // code runs on each app start (in these cases the model will already be
+        // downloaded, so session creation should take only a 1 or 2 ticks: file
+        // system stat, and ort.InferenceSession.create).
+        wait(50).then(() => 1),
     ]);
 
     // Don't wait for the download to complete.

desktop/src/main/services/ml.ts

@@ -1,147 +0,0 @@
-/**
- * @file ML related functionality. This code runs in the main process.
- */
-
-import {
-    MessageChannelMain,
-    type BrowserWindow,
-    type UtilityProcess,
-} from "electron";
-import { app, utilityProcess } from "electron/main";
-import path from "node:path";
-import log from "../log";
-
-/** The active ML worker (utility) process, if any. */
-let _child: UtilityProcess | undefined;
-
-/**
- * Create a new ML worker process, terminating the older ones (if any).
- *
- * [Note: ML IPC]
- *
- * The primary reason for doing ML tasks in the Node.js layer is so that we can
- * use the binary ONNX runtime, which is 10-20x faster than the Wasm one that
- * can be used directly on the web layer.
- *
- * For this to work, the main and renderer process need to communicate with each
- * other. Further, in the web layer the ML indexing runs in a web worker (so as
- * to not get in the way of the main thread). So the communication has 2 hops:
- *
- *     Node.js main <-> Renderer main <-> Renderer web worker
- *
- * This naive way works, but has a problem. The Node.js main process is in the
- * code path for delivering user events to the renderer process. The ML tasks we
- * do take in the order of 100-300 ms (possibly more) for each individual
- * inference. Thus, the Node.js main process is busy for those 100-300 ms, and
- * does not forward events to the renderer, causing the UI to jitter.
- *
- * The solution for this is to spawn an Electron UtilityProcess, which we can
- * think of a regular Node.js child process.  This frees up the Node.js main
- * process, and would remove the jitter.
- * https://www.electronjs.org/docs/latest/tutorial/process-model
- *
- * It would seem that this introduces another hop in our IPC
- *
- *     Node.js utility process <-> Node.js main <-> ...
- *
- * but here we can use the special bit about Electron utility processes that
- * separates them from regular Node.js child processes: their support for
- * message ports. https://www.electronjs.org/docs/latest/tutorial/message-ports
- *
- * As a brief summary, a MessagePort is a web feature that allows two contexts
- * to communicate. A pair of message ports is called a message channel. The cool
- * thing about these is that we can pass these ports themselves over IPC.
- *
- * > One caveat here is that the message ports can only be passed using the
- * > `postMessage` APIs, not the usual send/invoke APIs.
- *
- * So we
- *
- * 1. In the utility process create a message channel.
- * 2. Spawn a utility process, and send one port of the pair to it.
- * 3. Send the other port of the pair to the renderer.
- *
- * The renderer will forward that port to the web worker that is coordinating
- * the ML indexing on the web layer. Thereafter, the utility process and web
- * worker can directly talk to each other!
- *
- *     Node.js utility process <-> Renderer web worker
- *
- * The RPC protocol is handled using comlink on both ends. The port itself needs
- * to be relayed using `postMessage`.
- */
-export const createMLWorker = (window: BrowserWindow) => {
-    if (_child) {
-        log.debug(() => "Terminating previous ML worker process");
-        _child.kill();
-        _child = undefined;
-    }
-
-    const { port1, port2 } = new MessageChannelMain();
-
-    const child = utilityProcess.fork(path.join(__dirname, "ml-worker.js"));
-    const userDataPath = app.getPath("userData");
-    child.postMessage({ userDataPath }, [port1]);
-
-    window.webContents.postMessage("createMLWorker/port", undefined, [port2]);
-
-    handleMessagesFromUtilityProcess(child);
-
-    _child = child;
-};
-
-/**
- * Handle messages posted from the utility process.
- *
- * [Note: Using Electron APIs in UtilityProcess]
- *
- * Only a small subset of the Electron APIs are available to a UtilityProcess.
- * As of writing (Jul 2024, Electron 30), only the following are available:
- *
- * - net
- * - systemPreferences
- *
- * In particular, `app` is not available.
- *
- * We structure our code so that it doesn't need anything apart from `net`.
- *
- * For the other cases,
- *
- * -  Additional parameters to the utility process are passed alongwith the
- *    initial message where we provide it the message port.
- *
- * -  When we need to communicate from the utility process to the main process,
- *    we use the `parentPort` in the utility process.
- */
-const handleMessagesFromUtilityProcess = (child: UtilityProcess) => {
-    const logTag = "[ml-worker]";
-    child.on("message", (m: unknown) => {
-        if (m && typeof m == "object" && "method" in m && "p" in m) {
-            const p = m.p;
-            switch (m.method) {
-                case "log.errorString":
-                    if (typeof p == "string") {
-                        log.error(`${logTag} ${p}`);
-                        return;
-                    }
-                    break;
-                case "log.info":
-                    if (Array.isArray(p)) {
-                        // Need to cast from any[] to unknown[]
-                        log.info(logTag, ...(p as unknown[]));
-                        return;
-                    }
-                    break;
-                case "log.debugString":
-                    if (typeof p == "string") {
-                        log.debug(() => `${logTag} ${p}`);
-                        return;
-                    }
-                    break;
-                default:
-                    break;
-            }
-        }
-        log.info("Ignoring unknown message from ML worker", m);
-    });
-};

desktop/src/main/services/store.ts

@@ -24,17 +24,17 @@ export const clearStores = () => {
  * On macOS, `safeStorage` stores our data under a Keychain entry named
  * "<app-name> Safe Storage". In our case, "ente Safe Storage".
  */
-export const saveMasterKeyB64 = (masterKeyB64: string) => {
-    const encryptedKey = safeStorage.encryptString(masterKeyB64);
-    const b64EncryptedKey = Buffer.from(encryptedKey).toString("base64");
-    safeStorageStore.set("encryptionKey", b64EncryptedKey);
+export const saveMasterKeyInSafeStorage = (masterKey: string) => {
+    const encryptedKeyBuffer = safeStorage.encryptString(masterKey);
+    const encryptedKey = Buffer.from(encryptedKeyBuffer).toString("base64");
+    safeStorageStore.set("encryptionKey", encryptedKey);
 };
 
-export const masterKeyB64 = (): string | undefined => {
-    const b64EncryptedKey = safeStorageStore.get("encryptionKey");
-    if (!b64EncryptedKey) return undefined;
-    const keyBuffer = Buffer.from(b64EncryptedKey, "base64");
-    return safeStorage.decryptString(keyBuffer);
+export const masterKeyFromSafeStorage = (): string | undefined => {
+    const encryptedKey = safeStorageStore.get("encryptionKey");
+    if (!encryptedKey) return undefined;
+    const encryptedKeyBuffer = Buffer.from(encryptedKey, "base64");
+    return safeStorage.decryptString(encryptedKeyBuffer);
 };
 
 export const lastShownChangelogVersion = (): number | undefined =>

desktop/src/main/services/upload.ts

@@ -135,22 +135,35 @@ export const setPendingUploads = ({
     });
 };
 
-export const markUploadedFiles = (paths: string[]) => {
+export const markUploadedFile = (
+    path: string,
+    associatedPath: string | undefined,
+) => {
     const existing = uploadStatusStore.get("filePaths") ?? [];
-    const updated = existing.filter((p) => !paths.includes(p));
+    const updated = existing.filter((p) => p != path && p != associatedPath);
     uploadStatusStore.set("filePaths", updated);
+    // See: [Note: Integral last modified time]
+    return fs.stat(path).then((st) => st.mtime.getTime());
 };
 
-export const markUploadedZipItems = (
-    items: [zipPath: string, entryName: string][],
+export const markUploadedZipItem = (
+    item: ZipItem,
+    associatedItem: ZipItem | undefined,
 ) => {
     const existing = uploadStatusStore.get("zipItems") ?? [];
-    const updated = existing.filter(
-        (z) => !items.some((e) => z[0] == e[0] && z[1] == e[1]),
+    const updated = exceptZipItem(
+        exceptZipItem(existing, item),
+        associatedItem,
     );
     uploadStatusStore.set("zipItems", updated);
+    return fs.stat(item[0]).then((st) => st.mtime.getTime());
 };
 
+const exceptZipItem = (items: ZipItem[], item: ZipItem | undefined) =>
+    item
+        ? items.filter((zi) => !(zi[0] == item[0] && zi[1] == item[1]))
+        : items;
+
 export const clearPendingUploads = () => {
     uploadStatusStore.clear();
     clearOpenZipCache();

desktop/src/main/services/watch.ts

@@ -28,6 +28,13 @@ export const createWatcher = (mainWindow: BrowserWindow) => {
         // Ask the watcher to wait for a the file size to stabilize before
         // telling us about a new file. By default, it waits for 2 seconds.
         awaitWriteFinish: true,
+        // On macOS we start getting "EMFILE: too many open files" when watching
+        // large folders. This is a known regression in Chokidar v4:
+        // https://github.com/paulmillr/chokidar/issues/1385
+        //
+        // The recommended workaround for now is to enable usePolling. Since it
+        // comes at a performance cost, we only do it where needed (macOS).
+        ...(process.platform == "darwin" ? { usePolling: true } : {}),
     });
 
     watcher

desktop/src/main/services/workers.ts

@@ -0,0 +1,241 @@
+/**
+ * @file This main process code and interface for dealing with the various
+ * utility processes that we create.
+ */
+
+import type { Endpoint } from "comlink";
+import {
+    MessageChannelMain,
+    type BrowserWindow,
+    type UtilityProcess,
+} from "electron";
+import { app, utilityProcess } from "electron/main";
+import path from "node:path";
+import type { UtilityProcessType } from "../../types/ipc";
+import log, { processUtilityProcessLogMessage } from "../log";
+import { messagePortMainEndpoint } from "../utils/comlink";
+
+/**
+ * Terminate any existing utility processes if they're running.
+ *
+ * This function is called during the logout sequence.
+ */
+export const terminateUtilityProcesses = () => {
+    terminateMLProcessIfRunning();
+    terminateFFmpegProcessIfRunning();
+};
+
+/** The active ML utility process, if any. */
+let _utilityProcessML: UtilityProcess | undefined;
+
+/** The active FFmpeg utility process, if any. */
+let _utilityProcessFFmpeg: UtilityProcess | undefined;
+
+/**
+ * A promise to a comlink {@link Endpoint} that can be used to communicate with
+ * the active ffmpeg utility process (if any).
+ */
+let _utilityProcessFFmpegEndpoint: Promise<Endpoint> | undefined;
+
+/**
+ * Create a new utility process of the given {@link type}, terminating the older
+ * ones (if any).
+ *
+ * Currently the only type is "ml". The following note explains the reasoning
+ * why utility processes were used for the first workload (ML) that was handled
+ * this way. Similar reasoning applies to subsequent workloads (ffmpeg) that
+ * have been offloaded to utility processes in a slightly different manner to
+ * avoid stutter in the UI.
+ *
+ * [Note: ML IPC]
+ *
+ * The primary reason for doing ML tasks in the Node.js layer is so that we can
+ * use the binary ONNX runtime, which is 10-20x faster than the Wasm one that
+ * can be used directly on the web layer.
+ *
+ * For this to work, the main and renderer process need to communicate with each
+ * other. Further, in the web layer the ML indexing runs in a web worker (so as
+ * to not get in the way of the main thread). So the communication has 2 hops:
+ *
+ *     Node.js main <-> Renderer main <-> Renderer web worker
+ *
+ * This naive way works, but has a problem. The Node.js main process is in the
+ * code path for delivering user events to the renderer process. The ML tasks we
+ * do take in the order of 100-300 ms (possibly more) for each individual
+ * inference. Thus, the Node.js main process is busy for those 100-300 ms, and
+ * does not forward events to the renderer, causing the UI to jitter.
+ *
+ * The solution for this is to spawn an Electron UtilityProcess, which we can
+ * think of a regular Node.js child process. This frees up the Node.js main
+ * process, and would remove the jitter.
+ * https://www.electronjs.org/docs/latest/tutorial/process-model
+ *
+ * It would seem that this introduces another hop in our IPC
+ *
+ *     Node.js utility process <-> Node.js main <-> ...
+ *
+ * but here we can use the special bit about Electron utility processes that
+ * separates them from regular Node.js child processes: their support for
+ * message ports. https://www.electronjs.org/docs/latest/tutorial/message-ports
+ *
+ * As a brief summary, a MessagePort is a web feature that allows two contexts
+ * to communicate. A pair of message ports is called a message channel. The cool
+ * thing about these is that we can pass these ports themselves over IPC.
+ *
+ * > One caveat here is that the message ports can only be passed using the
+ * > `postMessage` APIs, not the usual send/invoke APIs.
+ *
+ * So we
+ *
+ * 1. In the utility process create a message channel.
+ * 2. Spawn a utility process, and send one port of the pair to it.
+ * 3. Send the other port of the pair to the renderer.
+ *
+ * The renderer will forward that port to the web worker that is coordinating
+ * the ML indexing on the web layer. Thereafter, the utility process and web
+ * worker can directly talk to each other!
+ *
+ *     Node.js utility process <-> Renderer web worker
+ *
+ * The RPC protocol is handled using comlink on both ends. The port itself needs
+ * to be relayed using `postMessage`.
+ */
+export const triggerCreateUtilityProcess = (
+    type: UtilityProcessType,
+    window: BrowserWindow,
+) => triggerCreateMLUtilityProcess(window);
+
+const terminateMLProcessIfRunning = () => {
+    if (_utilityProcessML) {
+        log.debug(() => "Terminating running ML utility process");
+        _utilityProcessML.kill();
+        _utilityProcessML = undefined;
+    }
+};
+
+export const triggerCreateMLUtilityProcess = (window: BrowserWindow) => {
+    terminateMLProcessIfRunning();
+
+    const { port1, port2 } = new MessageChannelMain();
+
+    const child = utilityProcess.fork(path.join(__dirname, "ml-worker.js"));
+    const userDataPath = app.getPath("userData");
+    child.postMessage(/* MLWorkerInitData */ { userDataPath }, [port1]);
+
+    window.webContents.postMessage("utilityProcessPort/ml", undefined, [port2]);
+
+    handleMessagesFromMLUtilityProcess(child);
+
+    _utilityProcessML = child;
+};
+
+/**
+ * Handle messages posted from the utility process.
+ *
+ * [Note: Using Electron APIs in UtilityProcess]
+ *
+ * Only a small subset of the Electron APIs are available to a UtilityProcess.
+ * As of writing (Jul 2024, Electron 30), only the following are available:
+ *
+ * - net
+ * - systemPreferences
+ *
+ * In particular, `app` is not available.
+ *
+ * We structure our code so that it doesn't need anything apart from `net`.
+ *
+ * For the other cases,
+ *
+ * -  Additional parameters to the utility process are passed alongwith the
+ *    initial message where we provide it the message port.
+ *
+ * -  When we need to communicate from the utility process to the main process,
+ *    we use the `parentPort` in the utility process.
+ */
+const handleMessagesFromMLUtilityProcess = (child: UtilityProcess) => {
+    child.on("message", (m: unknown) => {
+        if (processUtilityProcessLogMessage("[ml-worker]", m)) {
+            return;
+        }
+        log.info("Ignoring unknown message from ML utility process", m);
+    });
+};
+
+/**
+ * A comlink endpoint that can be used to communicate with the ffmpeg utility
+ * process. If there is no ffmpeg utility process, a new one is created on
+ * demand.
+ *
+ * See [Note: ML IPC] for a general outline of why utility processes are needed
+ * (tl;dr; to avoid stutter on the UI).
+ *
+ * In the case of ffmpeg, the IPC flow is a bit different: the utility process
+ * is not exposed to the web layer, and is internal to the node layer. The
+ * reason for this difference is that we need to create temporary files etc, and
+ * doing it a utility process requires access to the `app` module which are not
+ * accessible (See: [Note: Using Electron APIs in UtilityProcess]).
+ *
+ * There could've been possible reasonable workarounds, but the architecture
+ * we've adopted of three layers:
+ *
+ *     Renderer (web) <-> Node.js main <-> Node.js ffmpeg utility process
+ *
+ * The temporary file creation etc is handled in the Node.js main process, and
+ * paths to the files are forwarded to the ffmpeg utility process to act on.
+ *
+ * @returns an endpoint that can be used to communicate with the utility
+ * process. The utility process is expected to expose an object that conforms to
+ * the {@link ElectronFFmpegWorkerNode} interface on this endpoint.
+ */
+export const ffmpegUtilityProcessEndpoint = () =>
+    (_utilityProcessFFmpegEndpoint ??= createFFmpegUtilityProcessEndpoint());
+
+const terminateFFmpegProcessIfRunning = () => {
+    if (_utilityProcessFFmpeg) {
+        log.debug(() => "Terminating running FFmpeg utility process");
+        _utilityProcessFFmpeg.kill();
+        _utilityProcessFFmpeg = undefined;
+        _utilityProcessFFmpegEndpoint = undefined;
+    }
+};
+
+const createFFmpegUtilityProcessEndpoint = () => {
+    if (_utilityProcessFFmpeg) {
+        throw new Error("FFmpeg utility process is already running");
+    }
+
+    // Promise.withResolvers is currently in the node available to us.
+    let resolve: ((endpoint: Endpoint) => void) | undefined;
+    const promise = new Promise<Endpoint>((r) => (resolve = r));
+
+    const { port1, port2 } = new MessageChannelMain();
+
+    const child = utilityProcess.fork(path.join(__dirname, "ffmpeg-worker.js"));
+    // Send a handle to the port (one end of the message channel) to the utility
+    // process (alongwith any other init data). The utility process will reply
+    // with an "ack" when it get it.
+    const appVersion = app.getVersion();
+    child.postMessage(/* FFmpegWorkerInitData */ { appVersion }, [port1]);
+
+    child.on("message", (m: unknown) => {
+        if (m && typeof m == "object" && "method" in m) {
+            switch (m.method) {
+                case "ack":
+                    resolve!(messagePortMainEndpoint(port2));
+                    return;
+            }
+        }
+
+        if (processUtilityProcessLogMessage("[ffmpeg-worker]", m)) {
+            return;
+        }
+
+        log.info("Ignoring unknown message from ffmpeg utility process", m);
+    });
+
+    _utilityProcessFFmpeg = child;
+
+    // Resolve with the other end of the message channel (once we get an "ack"
+    // from the utility process).
+    return promise;
+};

desktop/src/main/stream.ts

@@ -7,16 +7,14 @@ import fs from "node:fs/promises";
 import { Writable } from "node:stream";
 import { pathToFileURL } from "node:url";
 import log from "./log";
-import {
-    ffmpegConvertToMP4,
-    ffmpegGenerateHLSPlaylistAndSegments,
-    type FFmpegGenerateHLSPlaylistAndSegmentsResult,
-} from "./services/ffmpeg";
+import { ffmpegUtilityProcess } from "./services/ffmpeg";
+import { type FFmpegGenerateHLSPlaylistAndSegmentsResult } from "./services/ffmpeg-worker";
 import { markClosableZip, openZip } from "./services/zip";
 import { writeStream } from "./utils/stream";
 import {
     deleteTempFile,
     deleteTempFileIgnoringErrors,
+    makeFileForStreamOrPathOrZipItem,
     makeTempFilePath,
 } from "./utils/temp";
 
@@ -78,7 +76,7 @@ const handleStreamRequest = async (request: Request): Promise<Response> => {
                     case "convert-to-mp4":
                         return handleConvertToMP4Write(request);
                     case "generate-hls":
-                        return handleGenerateHLSWrite(request);
+                        return handleGenerateHLSWrite(request, searchParams);
                     default:
                         return new Response(`Unknown op ${op}`, {
                             status: 404,
@@ -122,6 +120,7 @@ const handleRead = async (path: string) => {
         res.headers.set("Content-Length", `${fileSize}`);
 
         // Add the file's last modified time (as epoch milliseconds).
+        // See: [Note: Integral last modified time]
         const mtimeMs = stat.mtime.getTime();
         res.headers.set("X-Last-Modified-Ms", `${mtimeMs}`);
     }
@@ -230,12 +229,14 @@ export const clearPendingVideoResults = () => pendingVideoResults.clear();
  * See also: [Note: IPC streams]
  */
 const handleConvertToMP4Write = async (request: Request) => {
+    const worker = await ffmpegUtilityProcess();
+
     const inputTempFilePath = await makeTempFilePath();
     await writeStream(inputTempFilePath, request.body!);
 
     const outputTempFilePath = await makeTempFilePath("mp4");
     try {
-        await ffmpegConvertToMP4(inputTempFilePath, outputTempFilePath);
+        await worker.ffmpegConvertToMP4(inputTempFilePath, outputTempFilePath);
     } catch (e) {
         log.error("Conversion to MP4 failed", e);
         await deleteTempFileIgnoringErrors(outputTempFilePath);
@@ -246,7 +247,7 @@ const handleConvertToMP4Write = async (request: Request) => {
 
     const token = randomUUID();
     pendingVideoResults.set(token, outputTempFilePath);
-    return new Response(JSON.stringify([token]), { status: 200 });
+    return new Response(token, { status: 200 });
 };
 
 const handleVideoRead = async (token: string) => {
@@ -274,32 +275,83 @@ const handleVideoDone = async (token: string) => {
  * See: [Note: Convert to MP4] for the general architecture of commands that do
  * renderer <-> main I/O using streams.
  *
- * The difference here is that we the conversion generates two streams - one for
- * the HLS playlist itself, and one for the file containing the encrypted and
- * transcoded video chunks. So instead of returning a single token, we return a
- * JSON array containing two tokens so that the renderer can read them off
- * separately.
+ * The difference here is that we the conversion generates two streams^ - one
+ * for the HLS playlist itself, and one for the file containing the encrypted
+ * and transcoded video chunks. The video stream we write to the pre-signed
+ * object upload URL(s), and then we return a JSON object containing the token
+ * for the playlist, and other metadata for use by the renderer.
+ *
+ * ^ if the video doesn't require a stream to be generated (e.g. it is very
+ *   small and already uses a compatible codec) then a HTT 204 is returned and
+ *   no stream is generated.
  */
-const handleGenerateHLSWrite = async (request: Request) => {
-    const inputTempFilePath = await makeTempFilePath();
-    await writeStream(inputTempFilePath, request.body!);
+const handleGenerateHLSWrite = async (
+    request: Request,
+    params: URLSearchParams,
+) => {
+    const fileID = parseInt(params.get("fileID") ?? "", 10);
+    const fetchURL = params.get("fetchURL");
+    const authToken = params.get("authToken");
+    if (!fileID || !fetchURL || !authToken) throw new Error("Missing params");
 
-    const outputFilePathPrefix = await makeTempFilePath();
-    let paths: FFmpegGenerateHLSPlaylistAndSegmentsResult;
-    try {
-        paths = await ffmpegGenerateHLSPlaylistAndSegments(
-            inputTempFilePath,
-            outputFilePathPrefix,
-        );
-    } finally {
-        await deleteTempFileIgnoringErrors(inputTempFilePath);
+    let inputItem: Parameters<typeof makeFileForStreamOrPathOrZipItem>[0];
+    const path = params.get("path");
+    if (path) {
+        inputItem = path;
+    } else {
+        const zipPath = params.get("zipPath");
+        const entryName = params.get("entryName");
+        if (zipPath && entryName) {
+            inputItem = [zipPath, entryName];
+        } else {
+            const body = request.body;
+            if (!body) throw new Error("Missing body");
+            inputItem = body;
+        }
     }
 
-    const playlistToken = randomUUID();
-    const videoToken = randomUUID();
-    pendingVideoResults.set(playlistToken, paths.playlistPath);
-    pendingVideoResults.set(videoToken, paths.videoPath);
-    return new Response(JSON.stringify([playlistToken, videoToken]), {
-        status: 200,
-    });
+    const worker = await ffmpegUtilityProcess();
+
+    const {
+        path: inputFilePath,
+        isFileTemporary: isInputFileTemporary,
+        writeToTemporaryFile: writeToTemporaryInputFile,
+    } = await makeFileForStreamOrPathOrZipItem(inputItem);
+
+    const outputFilePathPrefix = await makeTempFilePath();
+    let result: FFmpegGenerateHLSPlaylistAndSegmentsResult | undefined;
+    try {
+        await writeToTemporaryInputFile();
+
+        result = await worker.ffmpegGenerateHLSPlaylistAndSegments(
+            inputFilePath,
+            outputFilePathPrefix,
+            fileID,
+            fetchURL,
+            authToken,
+        );
+
+        if (!result) {
+            // This video doesn't require stream generation.
+            return new Response(null, { status: 204 });
+        }
+
+        const { playlistPath, dimensions, videoSize, videoObjectID } = result;
+
+        const playlistToken = randomUUID();
+        pendingVideoResults.set(playlistToken, playlistPath);
+
+        return new Response(
+            JSON.stringify({
+                playlistToken,
+                dimensions,
+                videoSize,
+                videoObjectID,
+            }),
+            { status: 200 },
+        );
+    } finally {
+        if (isInputFileTemporary)
+            await deleteTempFileIgnoringErrors(inputFilePath);
+    }
 };

desktop/src/main/utils/comlink.ts

@@ -19,7 +19,7 @@ export const messagePortMainEndpoint = (mp: MessagePortMain): Endpoint => {
     const listeners = new WeakMap<NL, EL>();
     return {
         postMessage: (message, transfer) => {
-            mp.postMessage(message, transfer as unknown as MessagePortMain[]);
+            mp.postMessage(message, (transfer ?? []) as MessagePortMain[]);
         },
         addEventListener: (_, eh) => {
             const l: EL = (data) =>

desktop/src/main/utils/common.ts

@@ -10,6 +10,16 @@
  *
  * This function is a promisified `setTimeout`. It returns a promise that
  * resolves after {@link ms} milliseconds.
+ *
+ * Duplicated from `web/packages/utils/promise.ts`.
  */
 export const wait = (ms: number) =>
     new Promise((resolve) => setTimeout(resolve, ms));
+
+/**
+ * Convert `null` to `undefined`, passthrough everything else unchanged.
+ *
+ * Duplicated from `web/packages/utils/transform.ts`.
+ */
+export const nullToUndefined = <T>(v: T | null | undefined): T | undefined =>
+    v === null ? undefined : v;

desktop/src/main/utils/exec-worker.ts

@@ -0,0 +1,23 @@
+import shellescape from "any-shell-escape";
+import { exec } from "node:child_process";
+import { promisify } from "node:util";
+import log from "../log-worker";
+
+/**
+ * Run a shell command asynchronously (utility process edition).
+ *
+ * This is an almost verbatim copy of {@link execAsync} from `electron.ts`,
+ * except it is meant to be usable from a utility process where only a subset of
+ * imports are available. See [Note: Using Electron APIs in UtilityProcess].
+ */
+export const execAsyncWorker = async (command: string | string[]) => {
+    const escapedCommand = Array.isArray(command)
+        ? shellescape(command)
+        : command;
+    const startTime = Date.now();
+    const result = await execAsync_(escapedCommand);
+    log.debugString(`${escapedCommand} (${Date.now() - startTime} ms)`);
+    return result;
+};
+
+const execAsync_ = promisify(exec);

desktop/src/main/utils/http.ts

@@ -0,0 +1,31 @@
+export const clientPackageName = "io.ente.photos.desktop";
+
+/**
+ * Reimplementation of {@link publicRequestHeaders} from the web source.
+ *
+ * @param desktopAppVersion The desktop app's version. This will get passed on
+ * as the "X-Client-Version" header.
+ *
+ * We cannot directly use `app.getVersion()` to obtain this value since the
+ * {@link app} module is not accessible to Electron utility processes which also
+ * calls this function.
+ */
+export const publicRequestHeaders = (desktopAppVersion: string) => ({
+    "X-Client-Package": clientPackageName,
+    "X-Client-Version": desktopAppVersion,
+});
+
+/**
+ * Reimplementation of {@link authenticatedRequestHeaders} from the web source.
+ *
+ * This builds on top of {@link publicRequestHeaders} and takes the same
+ * parameters, and additionally also requires the {@link authToken} that will be
+ * passed as the "X-Auth-Token" header.
+ */
+export const authenticatedRequestHeaders = (
+    desktopAppVersion: string,
+    authToken: string,
+) => ({
+    ...publicRequestHeaders(desktopAppVersion),
+    "X-Auth-Token": authToken,
+});

desktop/src/main/utils/temp.ts

@@ -5,6 +5,7 @@ import path from "node:path";
 import type { ZipItem } from "../../types/ipc";
 import log from "../log";
 import { markClosableZip, openZip } from "../services/zip";
+import { writeStream } from "./stream";
 
 /**
  * Our very own directory within the system temp directory. Go crazy, but
@@ -79,8 +80,8 @@ export const deleteTempFileIgnoringErrors = async (tempFilePath: string) => {
     }
 };
 
-/** The result of {@link makeFileForDataOrPathOrZipItem}. */
-interface FileForDataOrPathOrZipItem {
+/** The result of {@link makeFileForStreamOrPathOrZipItem}. */
+interface FileForStreamOrPathOrZipItem {
     /**
      * The path to the file (possibly temporary).
      */
@@ -104,33 +105,32 @@ interface FileForDataOrPathOrZipItem {
 /**
  * Return the path to a file, a boolean indicating if this is a temporary path
  * that needs to be deleted after processing, and a function to write the given
- * {@link dataOrPathOrZipItem} into that temporary file if needed.
+ * {@link item} into that temporary file if needed.
  *
- * @param dataOrPathOrZipItem The contents of the file, or the path to an
- * existing file, or a (path to a zip file, name of an entry within that zip
- * file) tuple.
+ * @param item A {@link ReadableStream} with the contents of the file, or the
+ * path to an existing file, or a (path to a zip file, name of an entry within
+ * that zip file) tuple.
  */
-export const makeFileForDataOrPathOrZipItem = async (
-    dataOrPathOrZipItem: Uint8Array | string | ZipItem,
-): Promise<FileForDataOrPathOrZipItem> => {
+export const makeFileForStreamOrPathOrZipItem = async (
+    item: ReadableStream | string | ZipItem,
+): Promise<FileForStreamOrPathOrZipItem> => {
     let path: string;
     let isFileTemporary: boolean;
     let writeToTemporaryFile = async () => {
         /* no-op */
     };
 
-    if (typeof dataOrPathOrZipItem == "string") {
-        path = dataOrPathOrZipItem;
+    if (typeof item == "string") {
+        path = item;
         isFileTemporary = false;
     } else {
         path = await makeTempFilePath();
         isFileTemporary = true;
-        if (dataOrPathOrZipItem instanceof Uint8Array) {
-            writeToTemporaryFile = () =>
-                fs.writeFile(path, dataOrPathOrZipItem);
+        if (item instanceof ReadableStream) {
+            writeToTemporaryFile = () => writeStream(path, item);
         } else {
             writeToTemporaryFile = async () => {
-                const [zipPath, entryName] = dataOrPathOrZipItem;
+                const [zipPath, entryName] = item;
                 const zip = openZip(zipPath);
                 try {
                     await zip.extract(entryName, path);

desktop/src/preload.ts

@@ -66,8 +66,10 @@ import type { IpcRendererEvent } from "electron";
 import type {
     AppUpdate,
     CollectionMapping,
+    FFmpegCommand,
     FolderWatch,
     PendingUploads,
+    UtilityProcessType,
     ZipItem,
 } from "./types/ipc";
 
@@ -111,10 +113,11 @@ const logout = () => {
     return ipcRenderer.invoke("logout");
 };
 
-const masterKeyB64 = () => ipcRenderer.invoke("masterKeyB64");
+const masterKeyFromSafeStorage = () =>
+    ipcRenderer.invoke("masterKeyFromSafeStorage");
 
-const saveMasterKeyB64 = (masterKeyB64: string) =>
-    ipcRenderer.invoke("saveMasterKeyB64", masterKeyB64);
+const saveMasterKeyInSafeStorage = (masterKey: string) =>
+    ipcRenderer.invoke("saveMasterKeyInSafeStorage", masterKey);
 
 const lastShownChangelogVersion = () =>
     ipcRenderer.invoke("lastShownChangelogVersion");
@@ -183,47 +186,53 @@ const fsWriteFileViaBackup = (path: string, contents: string) =>
 
 const fsIsDir = (dirPath: string) => ipcRenderer.invoke("fsIsDir", dirPath);
 
+const fsStatMtime = (path: string) => ipcRenderer.invoke("fsStatMtime", path);
+
 // - Conversion
 
 const convertToJPEG = (imageData: Uint8Array) =>
     ipcRenderer.invoke("convertToJPEG", imageData);
 
 const generateImageThumbnail = (
-    dataOrPathOrZipItem: Uint8Array | string | ZipItem,
+    pathOrZipItem: string | ZipItem,
     maxDimension: number,
     maxSize: number,
 ) =>
     ipcRenderer.invoke(
         "generateImageThumbnail",
-        dataOrPathOrZipItem,
+        pathOrZipItem,
         maxDimension,
         maxSize,
     );
 
 const ffmpegExec = (
-    command: string[],
-    dataOrPathOrZipItem: Uint8Array | string | ZipItem,
+    command: FFmpegCommand,
+    pathOrZipItem: string | ZipItem,
     outputFileExtension: string,
 ) =>
     ipcRenderer.invoke(
         "ffmpegExec",
         command,
-        dataOrPathOrZipItem,
+        pathOrZipItem,
         outputFileExtension,
     );
 
-// - ML
+const ffmpegDetermineVideoDuration = (pathOrZipItem: string | ZipItem) =>
+    ipcRenderer.invoke("ffmpegDetermineVideoDuration", pathOrZipItem);
 
-const createMLWorker = () => {
+// - Utility processes
+
+const triggerCreateUtilityProcess = (type: UtilityProcessType) => {
+    const portEvent = `utilityProcessPort/${type}`;
     const l = (event: IpcRendererEvent) => {
         void windowLoaded.then(() => {
             // "*"" is the origin to send to.
-            window.postMessage("createMLWorker/port", "*", event.ports);
-            ipcRenderer.off("createMLWorker/port", l);
+            window.postMessage(portEvent, "*", event.ports);
+            ipcRenderer.off(portEvent, l);
         });
     };
-    ipcRenderer.on("createMLWorker/port", l);
-    ipcRenderer.send("createMLWorker");
+    ipcRenderer.on(portEvent, l);
+    ipcRenderer.send("triggerCreateUtilityProcess", type);
 };
 
 // - Watch
@@ -289,11 +298,11 @@ const pendingUploads = () => ipcRenderer.invoke("pendingUploads");
 const setPendingUploads = (pendingUploads: PendingUploads) =>
     ipcRenderer.invoke("setPendingUploads", pendingUploads);
 
-const markUploadedFiles = (paths: PendingUploads["filePaths"]) =>
-    ipcRenderer.invoke("markUploadedFiles", paths);
+const markUploadedFile = (path: string, associatedPath?: string) =>
+    ipcRenderer.invoke("markUploadedFile", path, associatedPath);
 
-const markUploadedZipItems = (items: PendingUploads["zipItems"]) =>
-    ipcRenderer.invoke("markUploadedZipItems", items);
+const markUploadedZipItem = (item: ZipItem, associatedItem?: ZipItem) =>
+    ipcRenderer.invoke("markUploadedZipItem", item, associatedItem);
 
 const clearPendingUploads = () => ipcRenderer.invoke("clearPendingUploads");
 
@@ -350,8 +359,8 @@ contextBridge.exposeInMainWorld("electron", {
     selectDirectory,
     pathForFile,
     logout,
-    masterKeyB64,
-    saveMasterKeyB64,
+    masterKeyFromSafeStorage,
+    saveMasterKeyInSafeStorage,
     lastShownChangelogVersion,
     setLastShownChangelogVersion,
     isAutoLaunchEnabled,
@@ -378,6 +387,7 @@ contextBridge.exposeInMainWorld("electron", {
         writeFile: fsWriteFile,
         writeFileViaBackup: fsWriteFileViaBackup,
         isDir: fsIsDir,
+        statMtime: fsStatMtime,
         findFiles: fsFindFiles,
     },
 
@@ -386,10 +396,11 @@ contextBridge.exposeInMainWorld("electron", {
     convertToJPEG,
     generateImageThumbnail,
     ffmpegExec,
+    ffmpegDetermineVideoDuration,
 
     // - ML
 
-    createMLWorker,
+    triggerCreateUtilityProcess,
 
     // - Watch
 
@@ -410,7 +421,7 @@ contextBridge.exposeInMainWorld("electron", {
     pathOrZipItemSize,
     pendingUploads,
     setPendingUploads,
-    markUploadedFiles,
-    markUploadedZipItems,
+    markUploadedFile,
+    markUploadedZipItem,
     clearPendingUploads,
 });

desktop/src/types/ipc.ts

@@ -5,6 +5,8 @@
  * See [Note: types.ts <-> preload.ts <-> ipc.ts]
  */
 
+export type UtilityProcessType = "ml";
+
 export interface AppUpdate {
     autoUpdatable: boolean;
     version: string;
@@ -32,3 +34,5 @@ export interface PendingUploads {
     filePaths: string[];
     zipItems: ZipItem[];
 }
+
+export type FFmpegCommand = string[] | { default: string[]; hdr: string[] };

docs/.prettierrc.json

@@ -1,4 +1,3 @@
 {
-    "tabWidth": 4,
-    "proseWrap": "always"
+    "tabWidth": 4
 }

docs/docs/.vitepress/config.ts

@@ -8,13 +8,6 @@ export default defineConfig({
     head: [["link", { rel: "icon", type: "image/png", href: "/favicon.png" }]],
     cleanUrls: true,
     ignoreDeadLinks: "localhostLinks",
-    vite: {
-      build: {
-        rollupOptions: {
-          external: ['client-museum-s3.png'] // Added to handle static asset import
-        }
-      }
-    },
     themeConfig: {
         // We use the default theme (with some CSS color overrides). This
         // themeConfig block can be used to further customize the default theme.
@@ -33,6 +26,9 @@ export default defineConfig({
             },
         },
         sidebar: sidebar,
+        outline: {
+            level: [2, 3],
+        },
         socialLinks: [
             { icon: "github", link: "https://github.com/ente-io/ente/" },
             { icon: "twitter", link: "https://twitter.com/enteio" },

docs/docs/.vitepress/sidebar.ts

@@ -26,6 +26,10 @@ export const sidebar = [
                         text: "Collecting photos",
                         link: "/photos/features/collect",
                     },
+                    {
+                        text: "Custom domains",
+                        link: "/photos/features/custom-domains/",
+                    },
                     {
                         text: "Deduplicate",
                         link: "/photos/features/deduplicate",
@@ -52,6 +56,10 @@ export const sidebar = [
                         link: "/photos/features/machine-learning",
                     },
                     { text: "Map", link: "/photos/features/map" },
+                    {
+                        text: "Notifications",
+                        link: "/photos/features/notifications",
+                    },
                     {
                         text: "Passkeys",
                         link: "/photos/features/passkeys",
@@ -182,6 +190,7 @@ export const sidebar = [
         text: "Auth",
         items: [
             { text: "Introduction", link: "/auth/" },
+            { text: "Features", link: "/auth/features/" },
             {
                 text: "FAQ",
                 collapsed: true,
@@ -202,27 +211,28 @@ export const sidebar = [
                 text: "Migration",
                 collapsed: true,
                 items: [
-                    { text: "Introduction", link: "/auth/migration-guides/" },
+                    { text: "Introduction", link: "/auth/migration/" },
                     {
                         text: "From Authy",
-                        link: "/auth/migration-guides/authy/",
+                        link: "/auth/migration/authy/",
                     },
                     {
                         text: "From Steam",
-                        link: "/auth/migration-guides/steam/",
+                        link: "/auth/migration/steam/",
                     },
                     {
                         text: "From others",
-                        link: "/auth/migration-guides/import",
+                        link: "/auth/migration/import",
                     },
                     {
                         text: "Exporting your data",
-                        link: "/auth/migration-guides/export",
+                        link: "/auth/migration/export",
                     },
                 ],
             },
             {
                 text: "Troubleshooting",
+                collapsed: true,
                 items: [
                     {
                         text: "Windows login",
@@ -233,129 +243,123 @@ export const sidebar = [
         ],
     },
     {
-        text: "Self hosting",
+        text: "Self-hosting",
         collapsed: true,
         items: [
-            { text: "Getting started", link: "/self-hosting/" },
             {
-                text: "Connect to custom server",
-                link: "/self-hosting/guides/custom-server/",
+                text: "Quickstart",
+                link: "/self-hosting/",
             },
             {
-                text: "Creating Accounts",
-                link: "/self-hosting/creating-accounts",
-            },
-            {
-                text: "Reverse Proxy",
-                link: "/self-hosting/reverse-proxy",
-            },
-            {
-                text: "Building your museum.yaml",
-                link: "/self-hosting/museum",
-            },
-            {
-                text: "Configuring S3",
-                link: "/self-hosting/guides/configuring-s3",
-            },
-            {
-                text: "Guides",
+                text: "Installation",
                 collapsed: true,
                 items: [
-                    { text: "Introduction", link: "/self-hosting/guides/" },
                     {
-                        text: "Hosting Ente from source",
-                        link: "/self-hosting/guides/from-source",
+                        text: "Requirements",
+                        link: "/self-hosting/installation/requirements",
                     },
                     {
-                        text: "Hosting Ente without Docker",
-                        link: "/self-hosting/guides/standalone-ente",
+                        text: "Quickstart script (Recommended)",
+                        link: "/self-hosting/installation/quickstart",
                     },
                     {
-                        text: "Administering your server",
-                        link: "/self-hosting/guides/admin",
+                        text: "Docker Compose",
+                        link: "/self-hosting/installation/compose",
                     },
                     {
-                        text: "Configure CLI for Self Hosted Instance",
-                        link: "/self-hosting/guides/selfhost-cli",
+                        text: "Manual setup (without Docker)",
+                        link: "/self-hosting/installation/manual",
                     },
                     {
-                        text: "DB migration",
-                        link: "/self-hosting/guides/db-migration",
+                        text: "Environment variables and defaults",
+                        link: "/self-hosting/installation/env-var",
                     },
                     {
-                        text: "Mobile build",
-                        link: "/self-hosting/guides/mobile-build",
+                        text: "Configuration",
+                        link: "/self-hosting/installation/config",
+                    },
+                    {
+                        text: "Post-installation steps",
+                        link: "/self-hosting/installation/post-install/",
+                    },
+                    {
+                        text: "Upgrade",
+                        link: "/self-hosting/installation/upgrade",
+                    },
+                ],
+            },
+            {
+                text: "Administration",
+                collapsed: true,
+                items: [
+                    {
+                        text: "User management",
+                        link: "/self-hosting/administration/users",
+                    },
+                    {
+                        text: "Reverse proxy",
+                        link: "/self-hosting/administration/reverse-proxy",
+                    },
+                    {
+                        text: "Object storage",
+                        link: "/self-hosting/administration/object-storage",
+                    },
+                    {
+                        text: "Ente CLI",
+                        link: "/self-hosting/administration/cli",
+                    },
+                    {
+                        text: "Backup",
+                        link: "/self-hosting/administration/backup",
+                    },
+                ],
+            },
+            {
+                text: "Development",
+                collapsed: true,
+                items: [
+                    {
+                        text: "Building mobile apps",
+                        link: "/self-hosting/development/mobile-build",
+                    },
+                ],
+            },
+            {
+                text: "Community Guides",
+                collapsed: true,
+                items: [
+                    {
+                        text: "Ente via Tailscale",
+                        link: "/self-hosting/guides/tailscale",
+                    },
+                    {
+                        text: "Running Ente using systemd",
+                        link: "/self-hosting/guides/systemd",
                     },
                 ],
             },
             {
                 text: "Troubleshooting",
+                collapsed: true,
                 items: [
                     {
                         text: "General",
                         link: "/self-hosting/troubleshooting/misc",
                     },
                     {
-                        text: "Bucket CORS",
-                        link: '/self-hosting/troubleshooting/bucket-cors'
+                        text: "Docker / quickstart",
+                        link: "/self-hosting/troubleshooting/docker",
                     },
                     {
                         text: "Uploads",
                         link: "/self-hosting/troubleshooting/uploads",
                     },
                     {
-                        text: "Docker / quickstart",
-                        link: "/self-hosting/troubleshooting/docker",
-                    },
-                    {
-                        text: "Yarn",
-                        link: "/self-hosting/troubleshooting/yarn",
-                    },
-                    {
-                        text: "Ente CLI Secrets",
-                        link: "/self-hosting/troubleshooting/keyring",
-                    },
-                ],
-            },
-            {
-                text: "Community Guides",
-                items :[
-                    {
-                        text: "Ente via Tailscale",
-                        link: "/self-hosting/guides/Tailscale",
-                    },
-                    {
-                        text: "Ente with External S3",
-                        link: "/self-hosting/guides/external-s3",
-                    }                
-                ]
-            },
-            {
-                text: "FAQ",
-                items: [
-                    { text: "General", link: "/self-hosting/faq/" },
-                    {
-                        text: "Verification code",
-                        link: "/self-hosting/faq/otp",
-                    },
-                    {
-                        text: "Shared albums",
-                        link: "/self-hosting/faq/sharing",
-                    },
-                    {
-                        text: "Backups",
-                        link: "/self-hosting/faq/backup",
+                        text: "Ente CLI",
+                        link: "/self-hosting/troubleshooting/cli",
                     },
                 ],
             },
         ],
     },
-    {
-        text: "About",
-        link: "/about/",
-    },
-    {
-        text: "Contribute",
-        link: "/about/contribute",
-    },
 ];

docs/docs/about/contribute.md

@@ -1,15 +0,0 @@
----
-title: Contribute
-description: Details about how to contribute to Ente's docs
----
-
-# Contributing
-
-To contribute to these docs, you can use the "Edit this page" button at the
-bottom of each page. This will allow you to directly edit the markdown file that
-is used to generate this documentation and open a quick pull request directly
-from GitHub's UI.
-
-If you're more comfortable in contributing with your text editor, see the
-`docs/` folder of our GitHub repository,
-[github.com/ente-io/ente](https://github.com/ente-io/ente).

docs/docs/about/index.md

@@ -1,72 +0,0 @@
----
-title: About Ente
-description: >
-    An overview of Ente: the company, and the people behind it, and the products
-    that we make.
----
-
-# About
-
-Ente is a end-to-end encrypted platform for privately, reliably, and securely
-storing your data on the cloud. On top of this platform, Ente offers two
-products:
-
-- **Ente Photos** - An alternative to Google Photos and Apple Photos
-
-- **Ente Auth** - A free 2FA alternative to Authy
-
-Both these apps are available for all desktop (Linux, Mac, Windows) and mobile
-(Android, iOS and F-Droid) platforms. They also work directly in your web
-browser without you needing to install anything.
-
-More products are in the pipeline.
-
-## History
-
-Ente was the founded by Vishnu Mohandas (he's also Ente's CEO) in response to
-privacy concerns with major tech companies. The underlying motivation was the
-understanding that big tech had no incentive to fix their act, but with
-end-to-end encrypted cross platform apps, there was a way for people to take
-back control over their own data without sacrificing on features.
-
-### What does Ente mean?
-
-In Malayalam, Vishnu's native language, "ente" means "mine". Thus "Ente Photos"
-has the literal meaning "my photos".
-
-This was a good name, but still Vishnu looked around for better ones. But one
-day, he discovered that "ente" means "duck" in German. This unexpected
-connection sealed the deal. We should ask him why he likes ducks so much, but
-apparently he does, so this dual meaning ("mine" / "duck") led him to finalize
-the name, and also led to the adoption of "Ducky", Ente's mascot:
-
-<div align="center">
-
-![Ente's mascot, Ducky](ducky.png){width=200px}
-
-</div>
-
-For the full origin story of Ducky you can check out
-[this blog post](https://ente.io/blog/ducky/).
-
-### How do I pronounce Ente?
-
-en-_tay_. Like ca<i>fe</i>.
-
-## Get in touch
-
-If you have a support query that is not answered by these docs, please reach out
-to our Customer Support by sending an email to support@ente.io
-
-To stay up to date with new product launches, and behind the scenes details of
-how we're building Ente, you can read our [blog](https://ente.io/blog) (or
-subscribe to it via [RSS](https://ente.io/blog/rss.xml))
-
-To suggest new features and/or offer your perspective on how we should design
-planned and upcoming features, use our
-[GitHub discussions](https://github.com/ente-io/ente/discussions)
-
-Or if you'd just like to hang out, join our
-[Discord](https://discord.gg/z2YVKkycX3), follow us on
-[Twitter](https://twitter.com/enteio) or give us a shout out on
-[Mastodon](https://mstdn.social/@ente)

docs/docs/auth/faq/index.md

@@ -41,6 +41,16 @@ Usually, this discrepancy occurs because the time in your browser might be
 incorrect. In particular, multiple users have reported that Firefox provides
 incorrect time when certain privacy settings are enabled.
 
+> [!TIP]
+>
+> Newer Ente Auth clients (upcoming 4.4.0+) will automatically try to correct
+> for incorrect system time, so you should be seeing correct codes even if your
+> system time is out of sync. However, this automatic correction will not work
+> if you're using Ente Auth in offline mode.
+>
+> If you've recently changed your system time and the codes are still incorrect,
+> try to refresh / restart the app if needed.
+
 ### Can I access my codes on web?
 
 You can access your codes on the web at [auth.ente.io](https://auth.ente.io).

docs/docs/auth/features/index.md

@@ -0,0 +1,138 @@
+---
+title: Features - Auth
+description: Features available in Ente Auth
+---
+
+# Features
+
+This page outlines the key features available in Ente Auth.
+
+### Icons
+
+Ente Auth supports the icon pack provided by
+[simple-icons](https://github.com/simple-icons/simple-icons). If an icon you
+need is missing, please refer to the
+[docs/adding-icons](https://github.com/ente-io/ente/blob/main/mobile/apps/auth/docs/adding-icons.md)
+guide for instructions on how to contribute.
+
+### Search
+
+Quickly find your codes by searching based on issuer or account name. You can
+also configure the app to focus the search bar automatically on app start by
+going to **Settings → General → Focus search on app start**.
+
+### Tags
+
+Organize and filter your codes with ease using tags.
+
+- **Creating a Tag:** When adding or editing a code, tap the orange (+) icon to
+  create a new tag.
+- **Adding an existing Tag:** When adding or editing a code, select the desired
+  tag from the list.
+
+### Pinning
+
+Highlight your frequently used services by pinning them to the top of your code
+list. To pin a code, long-press (mobile) or right-click (desktop) the code and
+select "Pin".
+
+### Notes
+
+Add additional information to your codes using notes. Notes can be added during
+the process of creating or modifying a code.
+
+### Sharing
+
+Securely share codes temporarily with others.
+
+- Long-press (mobile) or right-click (desktop) on a code and choose "Share".
+- Select a duration for the shared link: 2 minutes, 5 minutes, or 10 minutes.
+- This generates a unique, time-limited link. Recipients can view the codes for
+  the specified duration without gaining access to the underlying secret key.
+  After the link expires, the recipients will no longer be able to view new
+  codes.
+
+### Custom sorting
+
+Customize the order in which your codes are displayed. Ente Auth provides
+several sorting options:
+
+- Issuer name
+- Account name
+- Frequently used
+- Recently used
+- Manual (custom drag-and-drop order)
+
+Access the sort menu in the top-right corner (next to the search icon) to change
+your sorting preference.
+
+### Offline mode
+
+Ente Auth can be used entirely offline. Choose "Use without backups" on the
+login screen. In this mode, your codes are stored locally on your device.
+
+Unlike when using an account, data is not synced or backed up to the cloud. You
+are responsible for manually backing up your codes.
+
+### Display options
+
+Customize how your codes are displayed for optimal usability.
+
+- **Show large icons:** Display codes with larger icons for enhanced visibility.
+- **Compact mode:** Switch to a more compact layout to view more codes on the
+  screen simultaneously.
+- **Hide codes:** Hide the actual code values for extra privacy. Double-tap a
+  code to reveal it when needed.
+
+### App lock
+
+Add an additional layer of protection using the app lock. Choose from the
+following lock methods:
+
+- **Device lock:** Use the existing lock configured on your device (e.g., Face
+  ID, Touch ID, system password).
+- **PIN lock:** Set up a 4-digit PIN code to unlock the app.
+- **Password lock:** Set up a password to unlock the app.
+
+Additionally, configure **Auto lock** to automatically lock the app after a
+specified period of time (options: Immediately, 5s, 15s, 1m, 5m, 30m).
+
+### Import / Export
+
+Ente Auth offers various import and export options for your codes.
+
+- **Export:** Export your codes in plain text, as an encrypted file, or
+  automatically via the CLI.
+- **Import:** Import codes from various other authentication apps.
+
+For detailed instructions, refer to the [migration guides](../migration/).
+
+### Deduplicate codes
+
+If you import codes and end up with duplicates, you can easily remove them. Go
+to **Settings → Data → Duplicate codes** to find and remove duplicate codes.
+
+### Trash
+
+Manage unwanted codes by moving them to the Trash. The Trash is not cleared
+automatically, giving you the flexibility to restore or permanently delete codes
+at any time.
+
+- **Trashing a code:** Long-press (mobile) or right-click (desktop) on a code
+  and select "Trash" to move it to the Trash.
+- **Viewing trashed codes:** If you have trashed codes, you can view them by
+  selecting the Trash tag.
+- **Managing trashed codes:** In the Trash view, you can either permanently
+  delete codes or restore them back to your main list.
+
+### Scan QR
+
+Easily add or share entries using QR codes:
+
+- **Add by scanning (mobile):** On mobile, you can add a new entry by scanning
+  the QR code provided by the service. This quickly adds the entry to Ente Auth.
+- **Show entry as QR code:** On all apps, you can long-press (mobile) or
+  right-click (desktop) a code and select "QR". This allows you to easily share
+  the complete entry (including the secret) with others by letting them scan the
+  displayed QR code. This can also be used to easily add the same entry to
+  another authenticator app or service.

docs/docs/auth/migration/authy/index.md

@@ -10,8 +10,9 @@ A guide written by Green, an ente.io lover
 > [!WARNING]
 >
 > Authy has dropped all support for its desktop apps. It is no longer possible
-> to export data from Authy using methods 1 and 2. You will need either an iOS device
-> and computer (method 4) or a rooted Android phone (method 3) to follow this guide.
+> to export data from Authy using methods 1 and 2. You will need either an iOS
+> device and computer (method 4) or a rooted Android phone (method 3) to follow
+> this guide.
 
 ---
 
@@ -204,11 +205,24 @@ This uses the tool [Aegis Authenticator](https://getaegis.app/) from
 
 ## Method 4: Authy-iOS-MiTM
 
-**Who should use this?** Technical iOS users of Authy that cannot export their tokens with methods 1 or 2 (due to those methods being patched) or method 3 (due to that method requiring a rooted Android device).
+**Who should use this?** Technical iOS users of Authy that cannot export their
+tokens with methods 1 or 2 (due to those methods being patched) or method 3 (due
+to that method requiring a rooted Android device).
 
-This method works by intercepting the data the Authy app receives while logging in for the first time, which contains your encrypted authenticator tokens. After the encrypted authenticator tokens are dumped, you can decrypt them using your backup password and convert them to an Ente token file.
+This method works by intercepting the data the Authy app receives while logging
+in for the first time, which contains your encrypted authenticator tokens. After
+the encrypted authenticator tokens are dumped, you can decrypt them using your
+backup password and convert them to an Ente token file.
 
-For an up-to-date guide of how to retrieve the encrypted authenticator tokens and decrypt them, please see [Authy-iOS-MiTM](https://github.com/AlexTech01/Authy-iOS-MiTM). To convert the `decrypted_tokens.json` file from that guide into a format Ente Authenticator can recognize, use [this](https://gist.github.com/gboudreau/94bb0c11a6209c82418d01a59d958c93?permalink_comment_id=5317087#gistcomment-5317087) Python script. Once you have the `ente_auth_import.plain` file from that script, transfer it to your device and follow the instructions below to import it into Ente Authenticator.
+For an up-to-date guide of how to retrieve the encrypted authenticator tokens
+and decrypt them, please see
+[Authy-iOS-MiTM](https://github.com/AlexTech01/Authy-iOS-MiTM). To convert the
+`decrypted_tokens.json` file from that guide into a format Ente Authenticator
+can recognize, use
+[this](https://gist.github.com/gboudreau/94bb0c11a6209c82418d01a59d958c93?permalink_comment_id=5317087#gistcomment-5317087)
+Python script. Once you have the `ente_auth_import.plain` file from that script,
+transfer it to your device and follow the instructions below to import it into
+Ente Authenticator.
 
 ## Importing to Ente Authenticator (Method 1, method 2.1, method 4)
 

docs/docs/auth/migration/index.md

@@ -1,7 +1,6 @@
 ---
 title: Migrating to Ente Auth
-description:
-    Guides for migrating your existing 2FA tokens into or out of Ente Auth
+description: Guides for migrating your existing 2FA tokens into or out of Ente Auth
 ---
 
 # Migrating to/from Ente Auth

docs/docs/de/auth/index.md

@@ -10,4 +10,4 @@ Ende-zu-Ende-verschlüsselte Authenticator-App für jedermann. Wir sind froh, da
 du hier bist!
 
 **Please note that this German translation is currently just a placeholder.**
-Know German? [Help us fill this in!](/about/contribute).
+Know German? [Help us fill this in!](/#contribute).