TerribleDev/DowntownerArchive
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
97 Commits 1 Branch 0 Tags
aaf6cfda05216f2504d48dfe47b73957ff09a31d
Commit Graph

47 Commits

Author SHA1 Message Date
Tommy Parnell
aaf6cfda05 log when job runs 2025-05-08 20:31:56 +00:00
Tommy Parnell
f85d7e95ef update newsletters in post route 2025-05-08 16:11:44 -04:00
TerribleDev
544b55132d Add a development-only button to send test push notifications to all subscribers. 
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 0aa507c2-4fa6-42bc-9ce7-ee6f559c10a5
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/c6cce535-1f81-43c1-8b69-181a5f1f9aa1.jpg
 2025-03-02 07:33:21 +00:00
Tommy Parnell
9ff5623653 Restored to '78a555248d82f5839473fecc0b5f0dec3b3ca8c3' 
Replit-Restored-To: 78a555248d
 2025-03-02 07:30:29 +00:00
Tommy Parnell
3a8ee9585e Assistant checkpoint: Add test notification button and endpoint 
Assistant generated file changes:
- server/routes.ts: Add an API endpoint for sending test notifications
- client/src/pages/home.tsx: Add test notification handler function

---

User prompt:

Add a button only in development next to the refresh button that when pressed pushes a test notification to all subscribers

Replit-Commit-Author: Assistant
Replit-Commit-Session-Id: 83792e5d-e16e-4eb5-a15a-df5bb7c732a7
 2025-03-02 07:29:52 +00:00
TerribleDev
78a555248d Fix: Implement infinite scroll for newsletter listings by adding pagination to API requests and client-side data merging. 
Replit-Commit-Author: Agent
Replit-Commit-Session-Id: 0aa507c2-4fa6-42bc-9ce7-ee6f559c10a5
Replit-Commit-Screenshot-Url: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/3ad1e152-cd06-4cd1-871a-477ec6981df3.jpg
 2025-03-02 07:23:35 +00:00
TerribleDev
59e2f8ae7f Checkpoint before assistant change: Implement infinite scrolling for newsletters on the home page. 
Replit-Commit-Author: Assistant
 2025-03-02 07:10:39 +00:00
Tommy Parnell
172fd4dcb5 Assistant checkpoint: Implement server-side pagination for newsletters 
Assistant generated file changes:
- server/routes.ts: Add pagination support to newsletter endpoints
- server/storage.ts: Add paginated storage methods
- client/src/lib/newsletter-data.ts: Update API hooks for pagination
- client/src/pages/home.tsx: Update intersection observer to handle server-side pagination
Update intersection observer to handle server-side pagination
Update newsletter rendering with server pagination
Update intersection observer to handle server-side pagination
Update newsletter rendering with server pagination
Update loader display condition

---

User prompt:

page the newsletters 20 at a time.

Replit-Commit-Author: Assistant
Replit-Commit-Session-Id: a9267af3-1878-449b-8d05-5929f9706bd1
 2025-03-02 07:03:43 +00:00
TerribleDev
9a95a897b9 Checkpoint after starting plan 2025-02-19 19:29:44 +00:00
TerribleDev
07497b4648 Restored to '5c89fd2cd9402ccde43ea7be18e5de6d1726a9fc' 
Replit-Restored-To:5c89fd2cd9402ccde43ea7be18e5de6d1726a9fc
 2025-02-18 22:26:43 +00:00
TerribleDev
c0c7eb9536 Agent query: Could you verify if you can see the install button when the app is installable, and if the RSS feed icon is now visible in the header? 
Add PWA installation prompt to home page

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/ff32d81e-a881-4a86-8056-ba00331ea057.jpg
 2025-02-18 22:21:02 +00:00
Tommy Parnell
2ca1201180 Assistant checkpoint: Set newsletter retry interval to 1 minute 
Assistant generated file changes:
- server/utils.ts: Set retry backoff to 1 minute

---

User prompt:

Have retry for the details of a newsletter be 1 minute per retry
 2025-02-18 22:14:32 +00:00
TerribleDev
2a331f89f8 User checkpoint: Increase newsletter detail retry interval to 1 minute and maximum retry attempts to 5. 2025-02-18 22:14:17 +00:00
TerribleDev
b6deb92ab5 Agent query: Could you check if the newsletter content in the embed demo page (/embed-demo.html) is now properly styled with the correct layout and colors? 
Style and implement newsletter embed component with shadow DOM and responsive design.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/506f55e8-afaf-481d-b0da-003c21db3c96.jpg
 2025-02-18 22:09:28 +00:00
TerribleDev
c45769b745 Agent query: Can you check if the /embed route is working and displays the newsletter content correctly? Also, please verify if the embed-demo.html page is accessible and showing the embedded content. 
Add route for embedding newsletter content in shadow DOM.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/529e3893-377e-4d74-b780-943806106a19.jpg
 2025-02-18 22:07:39 +00:00
Tommy Parnell
ab11edd336 Assistant checkpoint: Skip content scraping when AWS WAF detected 
Assistant generated file changes:
- server/utils.ts: Skip scraping when AWS WAF detected

---

User prompt:

if the response has the string AwsWafIntegration then don't scrape the details of the newsletter
 2025-02-18 17:03:01 +00:00
Tommy Parnell
fb7ea64443 Assistant checkpoint: Update newsletter import to handle updates 
Assistant generated file changes:
- server/storage.ts: Update importNewsletter to handle updates

---

User prompt:

allow updating of a newsletter after scraping it for a 2nd time
 2025-02-18 17:01:06 +00:00
Tommy Parnell
590bbf9506 Assistant checkpoint: Add URL uniqueness constraint and handle duplicates 
Assistant generated file changes:
- shared/schema.ts: Add unique constraint to url
- server/storage.ts: Handle duplicate URLs in importNewsletter

---

User prompt:

Keep the id as the primary key but use  the url as an "external id" for the newsletter and don't allow duplicates to be created in the database
 2025-02-18 17:00:27 +00:00
Tommy Parnell
4eb6f7a482 Assistant checkpoint: Updated User-Agent string in HTTP requests 
Assistant generated file changes:
- server/utils.ts: Update User-Agent string

---

User prompt:

change the user agent to be Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36
 2025-02-18 16:54:09 +00:00
TerribleDev
a265360f10 Agent query: Could you try the refresh button to import newsletters and check if they are being saved one at a time? 
Refactor newsletter import: Process and save newsletters individually instead of all at once.  Improves efficiency and error handling.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/2f9ce22a-3adb-45ce-8465-b6f605458b4d.jpg
 2025-02-18 16:49:51 +00:00
TerribleDev
10828e5952 Agent query: I've added the hasDetails tracking feature. Could you check if the newsletters are being displayed correctly and if the content is being fetched properly? 
Enhance newsletter scraping: Add retry mechanism for missing details and update database accordingly.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/d96ca1cd-ef9a-4ef9-ae09-bfd9504e8440.jpg
 2025-02-18 16:30:13 +00:00
TerribleDev
ace2c97e32 Checkpoint after starting plan 2025-02-18 16:28:52 +00:00
Tommy Parnell
c7cd028f8c Assistant checkpoint: Add notification settings and subscription filtering 
Assistant generated file changes:
- shared/schema.ts: Add notification settings table
- server/storage.ts: Add notification settings methods
- server/routes.ts: Use active subscriptions for notifications, Add notification settings endpoint

---

User prompt:

save notification settings and push notifications when newsletter is imported
 2025-02-15 19:55:13 +00:00
TerribleDev
d31bb03b41 User checkpoint: Improve newsletter scraping robustness by adding retry logic and handling AWS WAF and rate limiting. 2025-02-15 19:53:50 +00:00
TerribleDev
37d8aace1b User checkpoint: Implement hourly cron job to check for and import new newsletters, sending push notifications for updates. Removes unnecessary queue processing. 2025-02-15 19:36:11 +00:00
Tommy Parnell
30738ef594 Assistant checkpoint: Add Bull queue settings to prevent job stalling 
Assistant generated file changes:
- server/queue.ts: Add Bull queue settings to prevent stalling

---

User prompt:

job stalled more than allowable limit bulljs
 2025-02-15 19:31:06 +00:00
Tommy Parnell
f59484ba32 Assistant checkpoint: Add environment variable for newsletter scraping control 
Assistant generated file changes:
- server/utils.ts: Add environment variable check for scraping

---

User prompt:

add an environment variable to turn on or off if we should scrape the newsletter content themselves
 2025-02-15 19:28:47 +00:00
Tommy Parnell
32e17e4180 Restored to 'a8a9d1cb885504b9521fff7456a9e207175ceef2' 
Replit-Restored-To:a8a9d1cb885504b9521fff7456a9e207175ceef2
 2025-02-15 19:20:11 +00:00
Tommy Parnell
767a97fb95 Restored to '1adfee1eacb54dd208e9aeec3ca414e1fe251c7e' 
Replit-Restored-To:1adfee1eacb54dd208e9aeec3ca414e1fe251c7e
 2025-02-15 19:19:00 +00:00
Tommy Parnell
a8a9d1cb88 Assistant checkpoint: Add rate limit handling with exponential backoff 
Assistant generated file changes:
- server/utils.ts: Add exponential backoff for retries, Add retry for rate limits

---

User prompt:

The images are no longer loading in
 2025-02-15 19:14:54 +00:00
Tommy Parnell
2c91814bbc Assistant checkpoint: Add AWS WAF retry logic to newsletter scraping 
Assistant generated file changes:
- server/utils.ts: Add retry logic for AWS WAF integration

---

User prompt:

when scraping a newsletter if the text contains AwsWafIntegration.checkForceRefresh then wait 1 second and then then retry importing the newsletter. Assume AwsWafIntegration.checkForceRefresh is an invalid import
 2025-02-15 19:10:15 +00:00
Tommy Parnell
9b9f3a15d1 Assistant checkpoint: Configure Redis server and update connection URL 
Assistant generated file changes:
- .replit: Fix Redis process configuration
- server/queue.ts: Update Redis connection URL

---

User prompt:

run redis in repl
 2025-02-15 18:55:42 +00:00
TerribleDev
cda3af06df Agent query: I've updated the Redis configuration and background processes. Could you try clicking the refresh button (circular arrow icon) at the top to import newsletters and let me know if it works? 
Fix: Correct Redis URL configuration for newsletter import queue.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/db885db1-4502-4e7f-8e5b-f79a4b676b37.jpg
 2025-02-15 18:44:38 +00:00
TerribleDev
284e116642 Agent query: I've reset the database and updated the Redis configuration. Can you try clicking the refresh button to import newsletters and see if it works? 
Update database connection and queue configuration for Replit deployment.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/69c9785f-ef77-46d8-b649-093c7cf6b0e6.jpg
 2025-02-15 18:41:48 +00:00
TerribleDev
8d943ed548 Agent query: I've set up a queue system for background jobs. Could you try clicking the refresh button to trigger a manual newsletter import and verify if it works? 
Implement newsletter updates via a Bull queue for asynchronous processing.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/6b15ec21-707b-471e-b02c-8596307fdb92.jpg
 2025-02-15 18:34:36 +00:00
TerribleDev
00785d9df0 Restored to '026e19d3b6c2165ce9cdbfd9d513ddff3d15f57c' 
Replit-Restored-To:026e19d3b6c2165ce9cdbfd9d513ddff3d15f57c
 2025-02-15 18:31:48 +00:00
TerribleDev
b87dfb7622 Agent query: Are the newsletter images loading correctly now without proxy? 
Fix: Remove image proxy and enable CORS for direct image loading.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/4212b3ba-6655-43b4-895d-e173d82da5ca.jpg
 2025-02-15 18:30:36 +00:00
Tommy Parnell
51871e7f8f Assistant checkpoint: Add image proxy to handle CORS issues 
Assistant generated file changes:
- server/routes.ts: Add image proxy endpoint
- client/src/lib/newsletter-data.ts: Update image URLs to use proxy
Update image URLs to use proxy
Transform newsletter thumbnails

---

User prompt:

Access to image at 'https://d1a8dioxuajlzs.cloudfront.net/accounts/7810/original/topshadow1023.png?1515171102' from origin 'https://9dda30b6-4149-4bce-89dc-76333005952c-00-20j8kqlx0qtgq.spock.replit.dev' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
 2025-02-15 18:27:35 +00:00
TerribleDev
f079fb2e00 Agent query: Could you check if the newsletter tiles are loading properly now, without any AWS WAF integration code? 
Improve image handling and add security headers.  Addresses image loading errors and enhances security by implementing Content-Security-Policy, X-Content-Type-Options, and X-XSS-Protection headers.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/8166d521-ab6d-49dc-bd03-63e0c10a9767.jpg
 2025-02-15 18:25:02 +00:00
TerribleDev
615579e317 Agent query: Could you check the browser's console logs for any subscription errors when clicking the subscribe button? 
Fix: Implement robust push notification subscription handling, including error handling and welcome notification.  Add necessary type definitions and VAPID key validation.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/2539846f-42c7-4fad-9462-984f310cf386.jpg
 2025-02-15 18:10:12 +00:00
TerribleDev
dee280d055 Agent query: Please verify if: 
1. The application loads successfully
2. You can see the newsletter cards with smooth animations
3. The search bar is visible at the top

Enhance newsletter archive viewer with mobile UI, social sharing, push notifications, search, RSS feed, and infinite scrolling.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/fea88e1e-ec20-4c53-8b12-b205f04819b7.jpg
 2025-02-15 17:57:15 +00:00
TerribleDev
34beb772af Agent query: Could you try clicking the refresh button again to see if it now successfully imports the newsletters? 
Improve newsletter scraping: Extract title, date, and URL directly from anchor tags.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/dda3abfd-99dc-47ac-b5a4-cf86467efdb0.jpg
 2025-02-15 17:43:05 +00:00
TerribleDev
dfb184edfb Agent query: Could you try clicking the refresh button again? We should see either newsletters or a more detailed error message in the toast. 
Fix: Improve Robly archive scraping robustness by adding request headers, handling errors, and improving logging.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/7324acf1-5ef0-4833-a886-ea38c84046c6.jpg
 2025-02-15 17:41:04 +00:00
TerribleDev
62917bf503 Agent query: Could you try pressing the refresh button again to import the newsletters? 
Fix: Correct newsletter scraping logic and add database connection.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/9267aba2-2889-42d8-8f15-7de6169ebc29.jpg
 2025-02-14 23:16:09 +00:00
TerribleDev
e0626e257b Agent query: Could you try clicking the refresh button again and let me know if the newsletters appear now? 
Fix: Resolve issue where refresh button failed to display newsletters; improve newsletter data handling and date parsing.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/204cab93-ab2c-4b8d-a2e4-a01a1cf3b19e.jpg
 2025-02-14 23:15:19 +00:00
TerribleDev
ae7cf5f46f Agent query: I've added a refresh button next to the search bar. Could you click it to import the newsletters from Robly? Please verify if the newsletters are successfully imported and displayed. 
Add newsletter import functionality.  Scrapes data from Robly and adds it to the archive.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/99711300-b33c-4ce4-b932-17b8c1aadbde.jpg
 2025-02-14 23:13:50 +00:00
TerribleDev
f2256ededa Agent query: How do you find the newsletter archive viewer? Please check if: 
1. The layout is clean and professional
2. The search box works for filtering newsletters
3. The newsletter cards are well-organized and readable

Implement basic web archive viewer for The Downtowner newsletter.  Adds client and server components, routing, and UI components.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/bf78f56a-b375-4d3e-860b-f9d9bbc56750.jpg
 2025-02-14 23:12:39 +00:00