TerribleDev/DowntownerArchive
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
69 Commits 1 Branch 0 Tags
e3607946e3f4ea52e00c30784b2e8f2adfa20085
Commit Graph

69 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
TerribleDev
e3607946e3 Restored to 'c0c7eb9536940fc88a15f117fd38aabec4d680d2' 
Replit-Restored-To:c0c7eb9536940fc88a15f117fd38aabec4d680d2
 2025-02-18 22:26:33 +00:00
TerribleDev
328cb1c8c9 Checkpoint before revert - Add a new /embed route that renders minimal page content and styles for shadow DOM integration 
Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/068967b0-c09c-479f-bd27-7d8e1cc8b519.jpg
 2025-02-18 22:26:32 +00:00
TerribleDev
668d858a91 Agent query: Could you check if you see an installation banner at the top of the page when the app is installable? 
Add install banner to home page prompting users to install the PWA.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/068967b0-c09c-479f-bd27-7d8e1cc8b519.jpg
 2025-02-18 22:23:57 +00:00
TerribleDev
c0c7eb9536 Agent query: Could you verify if you can see the install button when the app is installable, and if the RSS feed icon is now visible in the header? 
Add PWA installation prompt to home page

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/ff32d81e-a881-4a86-8056-ba00331ea057.jpg
 2025-02-18 22:21:02 +00:00
Tommy Parnell
2ca1201180 Assistant checkpoint: Set newsletter retry interval to 1 minute 
Assistant generated file changes:
- server/utils.ts: Set retry backoff to 1 minute

---

User prompt:

Have retry for the details of a newsletter be 1 minute per retry
 2025-02-18 22:14:32 +00:00
TerribleDev
2a331f89f8 User checkpoint: Increase newsletter detail retry interval to 1 minute and maximum retry attempts to 5. 2025-02-18 22:14:17 +00:00
Tommy Parnell
43dad3ea7d Assistant checkpoint: Created README with embed route documentation 
Assistant generated file changes:
- README.md: Add embed route documentation

---

User prompt:

write a readme on how to use the embed route
 2025-02-18 22:11:15 +00:00
TerribleDev
5c89fd2cd9 Agent query: Could you verify if the newsletter content at /embed endpoint is displaying properly with correct styling? 
Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/794eb54b-87a8-47c7-a6a6-5a11e1976ca4.jpg
 2025-02-18 22:10:17 +00:00
TerribleDev
b6deb92ab5 Agent query: Could you check if the newsletter content in the embed demo page (/embed-demo.html) is now properly styled with the correct layout and colors? 
Style and implement newsletter embed component with shadow DOM and responsive design.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/506f55e8-afaf-481d-b0da-003c21db3c96.jpg
 2025-02-18 22:09:28 +00:00
TerribleDev
c45769b745 Agent query: Can you check if the /embed route is working and displays the newsletter content correctly? Also, please verify if the embed-demo.html page is accessible and showing the embedded content. 
Add route for embedding newsletter content in shadow DOM.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/529e3893-377e-4d74-b780-943806106a19.jpg
 2025-02-18 22:07:39 +00:00
TerribleDev
0e99e215d6 Checkpoint after starting plan 2025-02-18 22:06:08 +00:00
Tommy Parnell
b962546004 Assistant checkpoint: Add isFetching from useNewsletters hook 
Assistant generated file changes:
- client/src/pages/home.tsx: Add isFetching from useNewsletters hook

---

User prompt:

isFetching is not defined
 2025-02-18 17:07:45 +00:00
Tommy Parnell
7aabf92c76 Assistant checkpoint: Added loading spinner visibility in production 
Assistant generated file changes:
- client/src/pages/home.tsx: Show loading state in production

---

User prompt:

show the refresh spinner in production
 2025-02-18 17:04:07 +00:00
Tommy Parnell
ab11edd336 Assistant checkpoint: Skip content scraping when AWS WAF detected 
Assistant generated file changes:
- server/utils.ts: Skip scraping when AWS WAF detected

---

User prompt:

if the response has the string AwsWafIntegration then don't scrape the details of the newsletter
 2025-02-18 17:03:01 +00:00
Tommy Parnell
fb7ea64443 Assistant checkpoint: Update newsletter import to handle updates 
Assistant generated file changes:
- server/storage.ts: Update importNewsletter to handle updates

---

User prompt:

allow updating of a newsletter after scraping it for a 2nd time
 2025-02-18 17:01:06 +00:00
Tommy Parnell
590bbf9506 Assistant checkpoint: Add URL uniqueness constraint and handle duplicates 
Assistant generated file changes:
- shared/schema.ts: Add unique constraint to url
- server/storage.ts: Handle duplicate URLs in importNewsletter

---

User prompt:

Keep the id as the primary key but use  the url as an "external id" for the newsletter and don't allow duplicates to be created in the database
 2025-02-18 17:00:27 +00:00
Tommy Parnell
4eb6f7a482 Assistant checkpoint: Updated User-Agent string in HTTP requests 
Assistant generated file changes:
- server/utils.ts: Update User-Agent string

---

User prompt:

change the user agent to be Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36
 2025-02-18 16:54:09 +00:00
TerribleDev
12923c3bd5 Agent query: Does the refresh button appear in development mode only (it should not be visible in production mode), and do newsletters get saved one at a time when importing? 
Implement incremental newsletter saving during processing.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/a694a8cd-6288-4fba-aa0c-65902233f177.jpg
 2025-02-18 16:50:09 +00:00
TerribleDev
a265360f10 Agent query: Could you try the refresh button to import newsletters and check if they are being saved one at a time? 
Refactor newsletter import: Process and save newsletters individually instead of all at once.  Improves efficiency and error handling.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/2f9ce22a-3adb-45ce-8465-b6f605458b4d.jpg
 2025-02-18 16:49:51 +00:00
TerribleDev
53227f34c0 Agent query: Please check if the refresh button is visible in development mode (it should be) and confirm that all other functionality (search, subscribe, RSS) still works correctly. 
Hide refresh button in production environment

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/7a0ed788-578a-4a62-a456-42adb7143259.jpg
 2025-02-18 16:44:37 +00:00
TerribleDev
10828e5952 Agent query: I've added the hasDetails tracking feature. Could you check if the newsletters are being displayed correctly and if the content is being fetched properly? 
Enhance newsletter scraping: Add retry mechanism for missing details and update database accordingly.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/d96ca1cd-ef9a-4ef9-ae09-bfd9504e8440.jpg
 2025-02-18 16:30:13 +00:00
TerribleDev
ace2c97e32 Checkpoint after starting plan 2025-02-18 16:28:52 +00:00
Tommy Parnell
c7cd028f8c Assistant checkpoint: Add notification settings and subscription filtering 
Assistant generated file changes:
- shared/schema.ts: Add notification settings table
- server/storage.ts: Add notification settings methods
- server/routes.ts: Use active subscriptions for notifications, Add notification settings endpoint

---

User prompt:

save notification settings and push notifications when newsletter is imported
 2025-02-15 19:55:13 +00:00
TerribleDev
d31bb03b41 User checkpoint: Improve newsletter scraping robustness by adding retry logic and handling AWS WAF and rate limiting. 2025-02-15 19:53:50 +00:00
TerribleDev
37d8aace1b User checkpoint: Implement hourly cron job to check for and import new newsletters, sending push notifications for updates. Removes unnecessary queue processing. 2025-02-15 19:36:11 +00:00
Tommy Parnell
30738ef594 Assistant checkpoint: Add Bull queue settings to prevent job stalling 
Assistant generated file changes:
- server/queue.ts: Add Bull queue settings to prevent stalling

---

User prompt:

job stalled more than allowable limit bulljs
 2025-02-15 19:31:06 +00:00
TerribleDev
99e9835e25 User checkpoint: Fix: Increase job timeout limit to prevent stalling in bulljs. 2025-02-15 19:30:54 +00:00
Tommy Parnell
f59484ba32 Assistant checkpoint: Add environment variable for newsletter scraping control 
Assistant generated file changes:
- server/utils.ts: Add environment variable check for scraping

---

User prompt:

add an environment variable to turn on or off if we should scrape the newsletter content themselves
 2025-02-15 19:28:47 +00:00
TerribleDev
de1c9603e5 User checkpoint: Add environment variable to control newsletter content scraping. 2025-02-15 19:28:34 +00:00
Tommy Parnell
32e17e4180 Restored to 'a8a9d1cb885504b9521fff7456a9e207175ceef2' 
Replit-Restored-To:a8a9d1cb885504b9521fff7456a9e207175ceef2
 2025-02-15 19:20:11 +00:00
Tommy Parnell
767a97fb95 Restored to '1adfee1eacb54dd208e9aeec3ca414e1fe251c7e' 
Replit-Restored-To:1adfee1eacb54dd208e9aeec3ca414e1fe251c7e
 2025-02-15 19:19:00 +00:00
Tommy Parnell
a8a9d1cb88 Assistant checkpoint: Add rate limit handling with exponential backoff 
Assistant generated file changes:
- server/utils.ts: Add exponential backoff for retries, Add retry for rate limits

---

User prompt:

The images are no longer loading in
 2025-02-15 19:14:54 +00:00
TerribleDev
1adfee1eac User checkpoint: Fix: Restore image loading by recreating database tables. 2025-02-15 19:14:39 +00:00
Tommy Parnell
9220d85f64 Assistant checkpoint: Create database reset script 
Assistant generated file changes:
- reset-db.sh: Create database reset script

---

User prompt:

generate a shell script to drop all the tables in the database and then recreate the database
 2025-02-15 19:11:23 +00:00
TerribleDev
37c150c01f User checkpoint: Update database schema by recreating it from dump.rdb 2025-02-15 19:10:51 +00:00
Tommy Parnell
2c91814bbc Assistant checkpoint: Add AWS WAF retry logic to newsletter scraping 
Assistant generated file changes:
- server/utils.ts: Add retry logic for AWS WAF integration

---

User prompt:

when scraping a newsletter if the text contains AwsWafIntegration.checkForceRefresh then wait 1 second and then then retry importing the newsletter. Assume AwsWafIntegration.checkForceRefresh is an invalid import
 2025-02-15 19:10:15 +00:00
TerribleDev
99a24e2573 User checkpoint: Add retry logic to newsletter scraping; Wait 1 second and retry if "AwsWafIntegration.checkForceRefresh" is detected in the text. 2025-02-15 19:10:03 +00:00
TerribleDev
56e3cfda5b User checkpoint: Configure Replit to run development server in background. 2025-02-15 18:56:38 +00:00
Tommy Parnell
9b9f3a15d1 Assistant checkpoint: Configure Redis server and update connection URL 
Assistant generated file changes:
- .replit: Fix Redis process configuration
- server/queue.ts: Update Redis connection URL

---

User prompt:

run redis in repl
 2025-02-15 18:55:42 +00:00
TerribleDev
a3d098b4f4 User checkpoint: Add Redis to Replit and implement newsletter scraping functionality. 2025-02-15 18:55:18 +00:00
TerribleDev
cda3af06df Agent query: I've updated the Redis configuration and background processes. Could you try clicking the refresh button (circular arrow icon) at the top to import newsletters and let me know if it works? 
Fix: Correct Redis URL configuration for newsletter import queue.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/db885db1-4502-4e7f-8e5b-f79a4b676b37.jpg
 2025-02-15 18:44:38 +00:00
TerribleDev
284e116642 Agent query: I've reset the database and updated the Redis configuration. Can you try clicking the refresh button to import newsletters and see if it works? 
Update database connection and queue configuration for Replit deployment.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/69c9785f-ef77-46d8-b649-093c7cf6b0e6.jpg
 2025-02-15 18:41:48 +00:00
TerribleDev
c389e0b0da Restored to '8d943ed548eefa23addcaae22962e07372f8dc83' 
Replit-Restored-To:8d943ed548eefa23addcaae22962e07372f8dc83
 2025-02-15 18:38:56 +00:00
TerribleDev
416c84d52e Checkpoint before revert - Build the initial prototype 
Remove --force flag from db:reset command.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/6b15ec21-707b-471e-b02c-8596307fdb92.jpg
 2025-02-15 18:38:55 +00:00
Tommy Parnell
d62bd85255 Assistant checkpoint: Added db:reset script to package.json 
Assistant generated file changes:
- package.json: Add db:reset script

---

User prompt:

add a command to the package json to reset the database
 2025-02-15 18:35:29 +00:00
TerribleDev
8d943ed548 Agent query: I've set up a queue system for background jobs. Could you try clicking the refresh button to trigger a manual newsletter import and verify if it works? 
Implement newsletter updates via a Bull queue for asynchronous processing.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/6b15ec21-707b-471e-b02c-8596307fdb92.jpg
 2025-02-15 18:34:36 +00:00
TerribleDev
00785d9df0 Restored to '026e19d3b6c2165ce9cdbfd9d513ddff3d15f57c' 
Replit-Restored-To:026e19d3b6c2165ce9cdbfd9d513ddff3d15f57c
 2025-02-15 18:31:48 +00:00
TerribleDev
9c27c42e16 Checkpoint before revert - Build the initial prototype 
Fix: Allow CORS for all origins in image proxy

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/4212b3ba-6655-43b4-895d-e173d82da5ca.jpg
 2025-02-15 18:31:46 +00:00
TerribleDev
b87dfb7622 Agent query: Are the newsletter images loading correctly now without proxy? 
Fix: Remove image proxy and enable CORS for direct image loading.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/4212b3ba-6655-43b4-895d-e173d82da5ca.jpg
 2025-02-15 18:30:36 +00:00
Tommy Parnell
51871e7f8f Assistant checkpoint: Add image proxy to handle CORS issues 
Assistant generated file changes:
- server/routes.ts: Add image proxy endpoint
- client/src/lib/newsletter-data.ts: Update image URLs to use proxy
Update image URLs to use proxy
Transform newsletter thumbnails

---

User prompt:

Access to image at 'https://d1a8dioxuajlzs.cloudfront.net/accounts/7810/original/topshadow1023.png?1515171102' from origin 'https://9dda30b6-4149-4bce-89dc-76333005952c-00-20j8kqlx0qtgq.spock.replit.dev' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.
 2025-02-15 18:27:35 +00:00