[infra] web-based log parser for support (#7144)

## Description ## Tests
2025-09-11 14:30:22 +05:30
parent c2ba7c56be e1423f2030
commit 8810f88236
5 changed files with 2706 additions and 0 deletions
--- a/infra/experiments/logs-viewer/README.md
+++ b/infra/experiments/logs-viewer/README.md
@@ -0,0 +1,167 @@
+# Ente Log Viewer
+
+A web-based log viewer for analyzing Ente application logs. This tool provides similar functionality to the mobile log viewer, allowing you to upload, filter, and analyze log files from customer support requests.
+
+## Features
+
+### 📁 File Upload
+- Drag and drop ZIP files containing log files
+- Automatic extraction and parsing of log files
+- Support for daily log files format (YYYY-M-D.log)
+
+### 🔍 Search and Filtering
+- **Text Search**: Search through log messages, logger names, and error content
+- **Logger Filtering**: Use `logger:ServiceName` syntax to filter by specific loggers
+- **Wildcard Support**: Use `logger:Auth*` to match all loggers starting with "Auth"
+- **Level Filtering**: Filter by log levels (SEVERE, WARNING, INFO, etc.)
+- **Process Filtering**: Filter by foreground/background processes
+- **Timeline Filtering**: Filter by date/time ranges
+
+### 📊 Analytics
+- Logger statistics showing most active components
+- Log level distribution charts
+- Click-to-filter from analytics charts
+
+### 🎨 UI Features
+- **Color-coded log levels**: Red for SEVERE, orange for WARNING, etc.
+- **Process indicators**: Visual distinction between foreground and background processes
+- **Active filter chips**: Visual indication of applied filters with easy removal
+- **Log detail view**: Click any log entry for detailed information
+- **Sort options**: Sort by newest first or oldest first
+- **Responsive design**: Works on desktop and mobile devices
+
+### 📤 Export
+- Export filtered logs as text files
+- Copy individual log entries to clipboard
+- Maintain original formatting and error details
+
+## Usage
+
+### Starting the Application
+
+1. **Local Development**:
+   ```bash
+   cd infra/experiments/logs-viewer
+   python3 -m http.server 8080
+   ```
+   Open http://localhost:8080 in your browser
+
+2. **Upload Log Files**:
+   - Drag and drop a ZIP file containing `.log` files
+   - Or click "Choose ZIP File" to browse for files
+
+### Log Format Support
+
+The viewer understands the Ente log format as generated by `super_logging.dart`:
+
+```
+[process] [loggerName] [LEVEL] [timestamp] message
+```
+
+**Examples**:
+- `[bg] [SyncService] [INFO] [2025-08-24 01:36:03.677678] Syncing started`
+- `[CollectionsService] [WARNING] [2025-08-24 01:36:04.123456] Connection failed`
+
+**Multi-line Error Support**:
+- Automatically parses `⤷` error detail lines
+- Extracts stack traces and error IDs
+- Handles inline error messages and exceptions
+
+### Filtering Examples
+
+- **Search by text**: `connection failed`
+- **Filter by logger**: `logger:SyncService`
+- **Multiple loggers**: `logger:Sync* logger:Collection*`
+- **Combined search**: `logger:Auth* login failed`
+
+### Keyboard Shortcuts
+
+- **Search**: Click search bar or start typing
+- **Clear search**: Click X button or clear the input
+- **Sort toggle**: Click sort arrow button
+- **Filter dialog**: Click filter button
+
+## Technical Details
+
+### Supported Log Levels
+- **SHOUT**: Purple - Highest priority
+- **SEVERE**: Red - Errors and exceptions
+- **WARNING**: Orange - Warning conditions
+- **INFO**: Blue - Informational messages
+- **CONFIG**: Green - Configuration messages
+- **FINE/FINER/FINEST**: Gray - Debug messages
+
+### Process Types
+- **Foreground**: Main app processes
+- **Background (bg)**: Background tasks
+- **Firebase Background (fbg)**: Firebase-related background processes
+
+### Performance
+- Lazy loading: Only renders visible log entries
+- Efficient filtering: Client-side filtering with optimized algorithms
+- Memory management: Handles large log files (tested with 100k+ entries)
+
+## Sample Log Files
+
+For testing, you can use any ZIP file containing `.log` files from Ente mobile app logs.
+
+## Development
+
+### File Structure
+```
+logs-viewer/
+├── index.html          # Main HTML structure
+├── styles.css          # CSS styling
+├── script.js           # JavaScript logic
+├── README.md           # This documentation
+└── references/         # UI reference screenshots
+```
+
+### Key JavaScript Classes
+- `LogViewer`: Main application class
+- Log parsing logic in `parseLogFile()` and `parseLogLine()`
+- Filter management in `applyCurrentFilter()`
+- UI rendering in `renderLogs()` and `createLogEntryHTML()`
+
+### Adding Features
+1. **New Filter Types**: Extend `currentFilter` object and `matchesFilter()` method
+2. **New Log Formats**: Update parsing patterns in `parseLogLine()`
+3. **UI Components**: Add HTML elements and CSS classes, wire up in `initializeEventListeners()`
+
+## Troubleshooting
+
+### Common Issues
+
+1. **ZIP file not loading**:
+   - Ensure ZIP contains `.log` files
+   - Check browser console for errors
+   - Try a smaller file first
+
+2. **Logs not parsing correctly**:
+   - Check log format matches expected pattern
+   - Look for console warnings about parsing failures
+   - Verify timestamp format is correct
+
+3. **Performance issues with large files**:
+   - The viewer handles pagination (100 logs at a time)
+   - Use filters to reduce the dataset
+   - Close other browser tabs to free memory
+
+4. **Search not working**:
+   - Check for typos in logger names
+   - Use wildcard syntax: `logger:Service*`
+   - Search is case-insensitive for message content
+
+### Browser Compatibility
+- **Recommended**: Chrome 80+, Firefox 75+, Safari 13+
+- **Required**: ES6 support, Fetch API, File API
+- **Dependencies**: JSZip library for ZIP file handling
+
+## Future Enhancements
+
+- Real-time log streaming
+- Advanced regex search
+- Log correlation and grouping
+- Performance metrics dashboard
+- Dark theme support
+- Export to JSON/CSV formats
--- a/infra/experiments/logs-viewer/ente-theme.css
+++ b/infra/experiments/logs-viewer/ente-theme.css
@@ -0,0 +1,110 @@
+/* Ente Theme Variables based on web/packages/base/components/utils/theme.ts */
+:root {
+    /* Light theme colors */
+    --ente-color-accent-photos: #1db954;
+    --ente-color-accent-auth: #9610d6;
+    --ente-color-accent-locker: #5ba8ff;
+    
+    /* Background colors */
+    --ente-background-default: #fff;
+    --ente-background-paper: #fff;
+    --ente-background-paper2: #fbfbfb;
+    --ente-background-search: #f3f3f3;
+    
+    /* Text colors */
+    --ente-text-base: #000;
+    --ente-text-muted: rgba(0, 0, 0, 0.60);
+    --ente-text-faint: rgba(0, 0, 0, 0.50);
+    
+    /* Fill colors */
+    --ente-fill-base: #000;
+    --ente-fill-muted: rgba(0, 0, 0, 0.12);
+    --ente-fill-faint: rgba(0, 0, 0, 0.04);
+    --ente-fill-faint-hover: rgba(0, 0, 0, 0.08);
+    --ente-fill-fainter: rgba(0, 0, 0, 0.02);
+    
+    /* Stroke colors */
+    --ente-stroke-base: #000;
+    --ente-stroke-muted: rgba(0, 0, 0, 0.24);
+    --ente-stroke-faint: rgba(0, 0, 0, 0.12);
+    --ente-stroke-fainter: rgba(0, 0, 0, 0.06);
+    
+    /* Shadow */
+    --ente-shadow-paper: 0px 0px 10px rgba(0, 0, 0, 0.25);
+    --ente-shadow-menu: 0px 0px 6px rgba(0, 0, 0, 0.16), 0px 3px 6px rgba(0, 0, 0, 0.12);
+    --ente-shadow-button: 0px 4px 4px rgba(0, 0, 0, 0.25);
+    
+    /* Fixed colors */
+    --ente-success: #1db954;
+    --ente-warning: #ffc107;
+    --ente-danger: #ea3f3f;
+    --ente-danger-dark: #f53434;
+    --ente-danger-light: #ff6565;
+    
+    /* Secondary colors */
+    --ente-secondary-main: #f5f5f5;
+    --ente-secondary-hover: #e9e9e9;
+    
+    /* Action colors */
+    --ente-action-hover: rgba(0, 0, 0, 0.08);
+    --ente-action-disabled: rgba(0, 0, 0, 0.50);
+    
+    /* Typography */
+    --ente-font-family: "Inter Variable", sans-serif;
+    --ente-font-weight-regular: 500;
+    --ente-font-weight-medium: 600;
+    --ente-font-weight-bold: 700;
+    
+    /* Border radius */
+    --ente-border-radius: 8px;
+    --ente-border-radius-small: 4px;
+    
+    /* Spacing */
+    --ente-spacing-xs: 4px;
+    --ente-spacing-sm: 8px;
+    --ente-spacing-md: 12px;
+    --ente-spacing-lg: 16px;
+    --ente-spacing-xl: 24px;
+}
+
+/* Dark theme */
+@media (prefers-color-scheme: dark) {
+    :root {
+        /* Background colors */
+        --ente-background-default: #000;
+        --ente-background-paper: #1b1b1b;
+        --ente-background-paper2: #252525;
+        --ente-background-search: #1b1b1b;
+        
+        /* Text colors */
+        --ente-text-base: #fff;
+        --ente-text-muted: rgba(255, 255, 255, 0.70);
+        --ente-text-faint: rgba(255, 255, 255, 0.50);
+        
+        /* Fill colors */
+        --ente-fill-base: #fff;
+        --ente-fill-muted: rgba(255, 255, 255, 0.16);
+        --ente-fill-faint: rgba(255, 255, 255, 0.12);
+        --ente-fill-faint-hover: rgba(255, 255, 255, 0.16);
+        --ente-fill-fainter: rgba(255, 255, 255, 0.05);
+        
+        /* Stroke colors */
+        --ente-stroke-base: #fff;
+        --ente-stroke-muted: rgba(255, 255, 255, 0.24);
+        --ente-stroke-faint: rgba(255, 255, 255, 0.16);
+        --ente-stroke-fainter: rgba(255, 255, 255, 0.12);
+        
+        /* Shadow */
+        --ente-shadow-paper: 0px 2px 12px rgba(0, 0, 0, 0.75);
+        --ente-shadow-menu: 0px 0px 6px rgba(0, 0, 0, 0.50), 0px 3px 6px rgba(0, 0, 0, 0.25);
+        --ente-shadow-button: 0px 4px 4px rgba(0, 0, 0, 0.75);
+        
+        /* Secondary colors */
+        --ente-secondary-main: #2b2b2b;
+        --ente-secondary-hover: #373737;
+        
+        /* Action colors */
+        --ente-action-hover: rgba(255, 255, 255, 0.16);
+        --ente-action-disabled: rgba(255, 255, 255, 0.50);
+    }
+}
--- a/infra/experiments/logs-viewer/index.html
+++ b/infra/experiments/logs-viewer/index.html
@@ -0,0 +1,218 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Ente Log Viewer</title>
+    
+    <!-- Material-UI CSS -->
+    <link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Roboto:300,400,500,700&display=swap" />
+    <link rel="stylesheet" href="https://fonts.googleapis.com/icon?family=Material+Icons" />
+    <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/@mui/material@latest/dist/index.css" />
+    
+    <!-- Ente theme -->
+    <link rel="stylesheet" href="ente-theme.css">
+    
+    <!-- Custom styles -->
+    <link rel="stylesheet" href="styles.css">
+</head>
+<body>
+    <div class="app">
+        <!-- Header -->
+        <header class="header">
+            <div class="header-content">
+                <h1>📋 Ente Log Viewer</h1>
+                <div class="header-actions">
+                    <button id="filter-btn" class="mui-icon-btn" title="Filter logs">
+                        <span class="material-icons">filter_list</span>
+                        <span class="filter-count" id="filter-count" style="display: none;"></span>
+                    </button>
+                    <button id="sort-btn" class="mui-icon-btn" title="Sort order">
+                        <span class="material-icons">arrow_downward</span>
+                    </button>
+                    <div class="dropdown">
+                        <button class="mui-icon-btn dropdown-toggle" title="More actions">
+                            <span class="material-icons">more_vert</span>
+                        </button>
+                        <div class="dropdown-menu mui-menu">
+                            <button id="analytics-btn" class="dropdown-item mui-menu-item">
+                                <span class="material-icons">analytics</span>
+                                <span>Analytics</span>
+                            </button>
+                            <button id="export-btn" class="dropdown-item mui-menu-item">
+                                <span class="material-icons">download</span>
+                                <span>Export Logs</span>
+                            </button>
+                            <button id="clear-btn" class="dropdown-item mui-menu-item danger">
+                                <span class="material-icons">delete</span>
+                                <span>Clear Logs</span>
+                            </button>
+                        </div>
+                    </div>
+                </div>
+            </div>
+        </header>
+
+        <!-- Upload Section -->
+        <div class="upload-section" id="upload-section">
+            <div class="upload-area" id="upload-area">
+                <div class="upload-content">
+                    <div class="upload-icon">📁</div>
+                    <h2>Upload Log Files</h2>
+                    <p>Drag and drop a zip file containing log files, or click to browse</p>
+                    <input type="file" id="file-input" accept=".zip" hidden>
+                    <button id="browse-btn" class="primary-btn">Choose ZIP File</button>
+                </div>
+            </div>
+        </div>
+
+        <!-- Main Content -->
+        <div class="main-content" id="main-content" style="display: none;">
+            <!-- Search Bar -->
+            <div class="search-section">
+                <div class="search-bar mui-search-container">
+                    <div class="mui-textfield">
+                        <input type="text" id="search-input" placeholder="Search logs... (try 'logger:SyncService' or text search)" class="mui-input" />
+                        <span class="mui-search-icon material-icons">search</span>
+                        <button id="clear-search" class="mui-clear-btn" style="display: none;">
+                            <span class="material-icons">clear</span>
+                        </button>
+                    </div>
+                </div>
+            </div>
+
+            <!-- Timeline Filter -->
+            <div class="timeline-section" id="timeline-section" style="display: none;">
+                <div class="timeline-header">
+                    <span class="material-icons">timeline</span>
+                    <span>Timeline Filter</span>
+                    <button id="timeline-toggle" class="mui-icon-btn timeline-btn">
+                        <span class="material-icons">timeline</span>
+                    </button>
+                </div>
+                <div class="timeline-controls" id="timeline-controls" style="display: none;">
+                    <div class="timeline-range">
+                        <div class="mui-textfield">
+                            <input type="datetime-local" id="start-time" class="mui-input" />
+                        </div>
+                        <span class="range-separator">to</span>
+                        <div class="mui-textfield">
+                            <input type="datetime-local" id="end-time" class="mui-input" />
+                        </div>
+                        <button id="reset-timeline" class="mui-button secondary">
+                            <span class="material-icons">refresh</span>
+                            <span>Reset</span>
+                        </button>
+                    </div>
+                </div>
+            </div>
+
+            <!-- Active Filters -->
+            <div class="active-filters" id="active-filters" style="display: none;">
+                <div class="filter-chips" id="filter-chips"></div>
+            </div>
+
+            <!-- Log Stats -->
+            <div class="log-stats" id="log-stats">
+                <span id="log-count">0 logs loaded</span>
+                <span id="filtered-count"></span>
+            </div>
+
+            <!-- Log List -->
+            <div class="log-list-container">
+                <div class="log-list" id="log-list">
+                    <!-- Log entries will be populated here -->
+                </div>
+                <div class="loading" id="loading" style="display: none;">Loading...</div>
+                <div class="load-more" id="load-more" style="display: none;">
+                    <button class="secondary-btn">Load More</button>
+                </div>
+            </div>
+        </div>
+    </div>
+
+    <!-- Filter Dialog -->
+    <div class="dialog-overlay" id="filter-dialog" style="display: none;">
+        <div class="dialog">
+            <div class="dialog-header">
+                <h2>Filter Logs</h2>
+                <button class="close-btn" id="close-filter">✕</button>
+            </div>
+            <div class="dialog-content">
+                <!-- Log Levels -->
+                <div class="filter-section">
+                    <h3>Log Levels</h3>
+                    <div class="level-chips" id="level-chips">
+                        <!-- Level chips will be populated here -->
+                    </div>
+                </div>
+
+                <!-- Process -->
+                <div class="filter-section">
+                    <h3>Process</h3>
+                    <div class="process-list" id="process-list">
+                        <!-- Process checkboxes will be populated here -->
+                    </div>
+                </div>
+
+                <!-- Loggers -->
+                <div class="filter-section">
+                    <h3>Loggers</h3>
+                    <div class="logger-list" id="logger-list">
+                        <!-- Logger checkboxes will be populated here -->
+                    </div>
+                </div>
+            </div>
+            <div class="dialog-actions">
+                <button id="clear-filters" class="secondary-btn">Clear All</button>
+                <button id="cancel-filter" class="secondary-btn">Cancel</button>
+                <button id="apply-filters" class="primary-btn">Apply</button>
+            </div>
+        </div>
+    </div>
+
+    <!-- Analytics Dialog -->
+    <div class="dialog-overlay" id="analytics-dialog" style="display: none;">
+        <div class="dialog">
+            <div class="dialog-header">
+                <h2>Logger Analytics</h2>
+                <button class="close-btn" id="close-analytics">✕</button>
+            </div>
+            <div class="dialog-content">
+                <div id="analytics-content">
+                    <!-- Analytics charts will be populated here -->
+                </div>
+            </div>
+            <div class="dialog-actions">
+                <button id="close-analytics-btn" class="secondary-btn">Close</button>
+            </div>
+        </div>
+    </div>
+
+    <!-- Log Detail Dialog -->
+    <div class="dialog-overlay" id="detail-dialog" style="display: none;">
+        <div class="dialog large">
+            <div class="dialog-header">
+                <h2>Log Details</h2>
+                <button class="close-btn" id="close-detail">✕</button>
+            </div>
+            <div class="dialog-content">
+                <div id="detail-content">
+                    <!-- Log details will be populated here -->
+                </div>
+            </div>
+            <div class="dialog-actions">
+                <button id="copy-log" class="secondary-btn">Copy</button>
+                <button id="close-detail-btn" class="secondary-btn">Close</button>
+            </div>
+        </div>
+    </div>
+
+    <!-- Material-UI JavaScript -->
+    <script src="https://unpkg.com/react@18/umd/react.production.min.js"></script>
+    <script src="https://unpkg.com/react-dom@18/umd/react-dom.production.min.js"></script>
+    <script src="https://unpkg.com/@mui/material@latest/umd/material-ui.production.min.js"></script>
+    <script src="https://cdnjs.cloudflare.com/ajax/libs/jszip/3.10.1/jszip.min.js"></script>
+    <script src="script.js"></script>
+</body>
+</html>
--- a/infra/experiments/logs-viewer/script.js
+++ b/infra/experiments/logs-viewer/script.js
--- a/infra/experiments/logs-viewer/styles.css
+++ b/infra/experiments/logs-viewer/styles.css