TerribleDev/DowntownerArchive
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
77 Commits 1 Branch 0 Tags
172fd4dcb5bc71c3974a0ec4b49b4ca5762a5c1c
Commit Graph

77 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Tommy Parnell
172fd4dcb5 Assistant checkpoint: Implement server-side pagination for newsletters 
Assistant generated file changes:
- server/routes.ts: Add pagination support to newsletter endpoints
- server/storage.ts: Add paginated storage methods
- client/src/lib/newsletter-data.ts: Update API hooks for pagination
- client/src/pages/home.tsx: Update intersection observer to handle server-side pagination
Update intersection observer to handle server-side pagination
Update newsletter rendering with server pagination
Update intersection observer to handle server-side pagination
Update newsletter rendering with server pagination
Update loader display condition

---

User prompt:

page the newsletters 20 at a time.

Replit-Commit-Author: Assistant
Replit-Commit-Session-Id: a9267af3-1878-449b-8d05-5929f9706bd1
 2025-03-02 07:03:43 +00:00
TerribleDev
6c5d7123e2 Checkpoint before assistant change: Refactor home page to paginate newsletters, displaying 20 at a time. Includes minor code style improvements. 
Replit-Commit-Author: Assistant
 2025-03-02 07:02:42 +00:00
Tommy Parnell
b965d5799d Assistant checkpoint: Make newsletter cards have consistent height 
Assistant generated file changes:
- client/src/pages/home.tsx: Fix the loading card skeletons to also have consistent height
Fix the loading card skeletons to also have consistent height
Fix the loading card skeletons to also have consistent height
Fix the loading card skeletons to also have consistent height
Fix the loading card skeletons to also have consistent height

---

User prompt:

Make all the cards that link to a newsletter have the same height in the page that lists all the newsletters

Replit-Commit-Author: Assistant
Replit-Commit-Session-Id: 912f8ca8-dc45-45a3-9b2d-257210cf9199
 2025-03-02 06:57:15 +00:00
TerribleDev
beb1a69b3b Checkpoint before assistant change: Update newsletter card height consistency. 
Replit-Commit-Author: Assistant
 2025-03-02 06:56:38 +00:00
TerribleDev
f383527b8b Agent query: Can you verify if: 1) The website shows an 'Add to Home Screen' or 'Install' option, 2) The service worker is registered successfully (check browser console)? 
Implement PWA functionality: added service worker for caching and push notifications, updated manifest and meta tags for installability.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/813fd020-c901-41bd-8d40-ee8624dc534d.jpg
 2025-02-19 19:30:23 +00:00
TerribleDev
9a95a897b9 Checkpoint after starting plan 2025-02-19 19:29:44 +00:00
TerribleDev
07497b4648 Restored to '5c89fd2cd9402ccde43ea7be18e5de6d1726a9fc' 
Replit-Restored-To:5c89fd2cd9402ccde43ea7be18e5de6d1726a9fc
 2025-02-18 22:26:43 +00:00
TerribleDev
6958c89e72 Checkpoint before revert - Add a new /embed route that renders minimal page content and styles for shadow DOM integration 
Add PWA installation instructions
 2025-02-18 22:26:42 +00:00
TerribleDev
e3607946e3 Restored to 'c0c7eb9536940fc88a15f117fd38aabec4d680d2' 
Replit-Restored-To:c0c7eb9536940fc88a15f117fd38aabec4d680d2
 2025-02-18 22:26:33 +00:00
TerribleDev
328cb1c8c9 Checkpoint before revert - Add a new /embed route that renders minimal page content and styles for shadow DOM integration 
Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/068967b0-c09c-479f-bd27-7d8e1cc8b519.jpg
 2025-02-18 22:26:32 +00:00
TerribleDev
668d858a91 Agent query: Could you check if you see an installation banner at the top of the page when the app is installable? 
Add install banner to home page prompting users to install the PWA.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/068967b0-c09c-479f-bd27-7d8e1cc8b519.jpg
 2025-02-18 22:23:57 +00:00
TerribleDev
c0c7eb9536 Agent query: Could you verify if you can see the install button when the app is installable, and if the RSS feed icon is now visible in the header? 
Add PWA installation prompt to home page

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/ff32d81e-a881-4a86-8056-ba00331ea057.jpg
 2025-02-18 22:21:02 +00:00
Tommy Parnell
2ca1201180 Assistant checkpoint: Set newsletter retry interval to 1 minute 
Assistant generated file changes:
- server/utils.ts: Set retry backoff to 1 minute

---

User prompt:

Have retry for the details of a newsletter be 1 minute per retry
 2025-02-18 22:14:32 +00:00
TerribleDev
2a331f89f8 User checkpoint: Increase newsletter detail retry interval to 1 minute and maximum retry attempts to 5. 2025-02-18 22:14:17 +00:00
Tommy Parnell
43dad3ea7d Assistant checkpoint: Created README with embed route documentation 
Assistant generated file changes:
- README.md: Add embed route documentation

---

User prompt:

write a readme on how to use the embed route
 2025-02-18 22:11:15 +00:00
TerribleDev
5c89fd2cd9 Agent query: Could you verify if the newsletter content at /embed endpoint is displaying properly with correct styling? 
Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/794eb54b-87a8-47c7-a6a6-5a11e1976ca4.jpg
 2025-02-18 22:10:17 +00:00
TerribleDev
b6deb92ab5 Agent query: Could you check if the newsletter content in the embed demo page (/embed-demo.html) is now properly styled with the correct layout and colors? 
Style and implement newsletter embed component with shadow DOM and responsive design.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/506f55e8-afaf-481d-b0da-003c21db3c96.jpg
 2025-02-18 22:09:28 +00:00
TerribleDev
c45769b745 Agent query: Can you check if the /embed route is working and displays the newsletter content correctly? Also, please verify if the embed-demo.html page is accessible and showing the embedded content. 
Add route for embedding newsletter content in shadow DOM.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/529e3893-377e-4d74-b780-943806106a19.jpg
 2025-02-18 22:07:39 +00:00
TerribleDev
0e99e215d6 Checkpoint after starting plan 2025-02-18 22:06:08 +00:00
Tommy Parnell
b962546004 Assistant checkpoint: Add isFetching from useNewsletters hook 
Assistant generated file changes:
- client/src/pages/home.tsx: Add isFetching from useNewsletters hook

---

User prompt:

isFetching is not defined
 2025-02-18 17:07:45 +00:00
Tommy Parnell
7aabf92c76 Assistant checkpoint: Added loading spinner visibility in production 
Assistant generated file changes:
- client/src/pages/home.tsx: Show loading state in production

---

User prompt:

show the refresh spinner in production
 2025-02-18 17:04:07 +00:00
Tommy Parnell
ab11edd336 Assistant checkpoint: Skip content scraping when AWS WAF detected 
Assistant generated file changes:
- server/utils.ts: Skip scraping when AWS WAF detected

---

User prompt:

if the response has the string AwsWafIntegration then don't scrape the details of the newsletter
 2025-02-18 17:03:01 +00:00
Tommy Parnell
fb7ea64443 Assistant checkpoint: Update newsletter import to handle updates 
Assistant generated file changes:
- server/storage.ts: Update importNewsletter to handle updates

---

User prompt:

allow updating of a newsletter after scraping it for a 2nd time
 2025-02-18 17:01:06 +00:00
Tommy Parnell
590bbf9506 Assistant checkpoint: Add URL uniqueness constraint and handle duplicates 
Assistant generated file changes:
- shared/schema.ts: Add unique constraint to url
- server/storage.ts: Handle duplicate URLs in importNewsletter

---

User prompt:

Keep the id as the primary key but use  the url as an "external id" for the newsletter and don't allow duplicates to be created in the database
 2025-02-18 17:00:27 +00:00
Tommy Parnell
4eb6f7a482 Assistant checkpoint: Updated User-Agent string in HTTP requests 
Assistant generated file changes:
- server/utils.ts: Update User-Agent string

---

User prompt:

change the user agent to be Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/133.0.0.0 Safari/537.36
 2025-02-18 16:54:09 +00:00
TerribleDev
12923c3bd5 Agent query: Does the refresh button appear in development mode only (it should not be visible in production mode), and do newsletters get saved one at a time when importing? 
Implement incremental newsletter saving during processing.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/a694a8cd-6288-4fba-aa0c-65902233f177.jpg
 2025-02-18 16:50:09 +00:00
TerribleDev
a265360f10 Agent query: Could you try the refresh button to import newsletters and check if they are being saved one at a time? 
Refactor newsletter import: Process and save newsletters individually instead of all at once.  Improves efficiency and error handling.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/2f9ce22a-3adb-45ce-8465-b6f605458b4d.jpg
 2025-02-18 16:49:51 +00:00
TerribleDev
53227f34c0 Agent query: Please check if the refresh button is visible in development mode (it should be) and confirm that all other functionality (search, subscribe, RSS) still works correctly. 
Hide refresh button in production environment

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/7a0ed788-578a-4a62-a456-42adb7143259.jpg
 2025-02-18 16:44:37 +00:00
TerribleDev
10828e5952 Agent query: I've added the hasDetails tracking feature. Could you check if the newsletters are being displayed correctly and if the content is being fetched properly? 
Enhance newsletter scraping: Add retry mechanism for missing details and update database accordingly.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/d96ca1cd-ef9a-4ef9-ae09-bfd9504e8440.jpg
 2025-02-18 16:30:13 +00:00
TerribleDev
ace2c97e32 Checkpoint after starting plan 2025-02-18 16:28:52 +00:00
Tommy Parnell
c7cd028f8c Assistant checkpoint: Add notification settings and subscription filtering 
Assistant generated file changes:
- shared/schema.ts: Add notification settings table
- server/storage.ts: Add notification settings methods
- server/routes.ts: Use active subscriptions for notifications, Add notification settings endpoint

---

User prompt:

save notification settings and push notifications when newsletter is imported
 2025-02-15 19:55:13 +00:00
TerribleDev
d31bb03b41 User checkpoint: Improve newsletter scraping robustness by adding retry logic and handling AWS WAF and rate limiting. 2025-02-15 19:53:50 +00:00
TerribleDev
37d8aace1b User checkpoint: Implement hourly cron job to check for and import new newsletters, sending push notifications for updates. Removes unnecessary queue processing. 2025-02-15 19:36:11 +00:00
Tommy Parnell
30738ef594 Assistant checkpoint: Add Bull queue settings to prevent job stalling 
Assistant generated file changes:
- server/queue.ts: Add Bull queue settings to prevent stalling

---

User prompt:

job stalled more than allowable limit bulljs
 2025-02-15 19:31:06 +00:00
TerribleDev
99e9835e25 User checkpoint: Fix: Increase job timeout limit to prevent stalling in bulljs. 2025-02-15 19:30:54 +00:00
Tommy Parnell
f59484ba32 Assistant checkpoint: Add environment variable for newsletter scraping control 
Assistant generated file changes:
- server/utils.ts: Add environment variable check for scraping

---

User prompt:

add an environment variable to turn on or off if we should scrape the newsletter content themselves
 2025-02-15 19:28:47 +00:00
TerribleDev
de1c9603e5 User checkpoint: Add environment variable to control newsletter content scraping. 2025-02-15 19:28:34 +00:00
Tommy Parnell
32e17e4180 Restored to 'a8a9d1cb885504b9521fff7456a9e207175ceef2' 
Replit-Restored-To:a8a9d1cb885504b9521fff7456a9e207175ceef2
 2025-02-15 19:20:11 +00:00
Tommy Parnell
767a97fb95 Restored to '1adfee1eacb54dd208e9aeec3ca414e1fe251c7e' 
Replit-Restored-To:1adfee1eacb54dd208e9aeec3ca414e1fe251c7e
 2025-02-15 19:19:00 +00:00
Tommy Parnell
a8a9d1cb88 Assistant checkpoint: Add rate limit handling with exponential backoff 
Assistant generated file changes:
- server/utils.ts: Add exponential backoff for retries, Add retry for rate limits

---

User prompt:

The images are no longer loading in
 2025-02-15 19:14:54 +00:00
TerribleDev
1adfee1eac User checkpoint: Fix: Restore image loading by recreating database tables. 2025-02-15 19:14:39 +00:00
Tommy Parnell
9220d85f64 Assistant checkpoint: Create database reset script 
Assistant generated file changes:
- reset-db.sh: Create database reset script

---

User prompt:

generate a shell script to drop all the tables in the database and then recreate the database
 2025-02-15 19:11:23 +00:00
TerribleDev
37c150c01f User checkpoint: Update database schema by recreating it from dump.rdb 2025-02-15 19:10:51 +00:00
Tommy Parnell
2c91814bbc Assistant checkpoint: Add AWS WAF retry logic to newsletter scraping 
Assistant generated file changes:
- server/utils.ts: Add retry logic for AWS WAF integration

---

User prompt:

when scraping a newsletter if the text contains AwsWafIntegration.checkForceRefresh then wait 1 second and then then retry importing the newsletter. Assume AwsWafIntegration.checkForceRefresh is an invalid import
 2025-02-15 19:10:15 +00:00
TerribleDev
99a24e2573 User checkpoint: Add retry logic to newsletter scraping; Wait 1 second and retry if "AwsWafIntegration.checkForceRefresh" is detected in the text. 2025-02-15 19:10:03 +00:00
TerribleDev
56e3cfda5b User checkpoint: Configure Replit to run development server in background. 2025-02-15 18:56:38 +00:00
Tommy Parnell
9b9f3a15d1 Assistant checkpoint: Configure Redis server and update connection URL 
Assistant generated file changes:
- .replit: Fix Redis process configuration
- server/queue.ts: Update Redis connection URL

---

User prompt:

run redis in repl
 2025-02-15 18:55:42 +00:00
TerribleDev
a3d098b4f4 User checkpoint: Add Redis to Replit and implement newsletter scraping functionality. 2025-02-15 18:55:18 +00:00
TerribleDev
cda3af06df Agent query: I've updated the Redis configuration and background processes. Could you try clicking the refresh button (circular arrow icon) at the top to import newsletters and let me know if it works? 
Fix: Correct Redis URL configuration for newsletter import queue.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/db885db1-4502-4e7f-8e5b-f79a4b676b37.jpg
 2025-02-15 18:44:38 +00:00
TerribleDev
284e116642 Agent query: I've reset the database and updated the Redis configuration. Can you try clicking the refresh button to import newsletters and see if it works? 
Update database connection and queue configuration for Replit deployment.

Screenshot: https://storage.googleapis.com/screenshot-production-us-central1/9dda30b6-4149-4bce-89dc-76333005952c/69c9785f-ef77-46d8-b649-093c7cf6b0e6.jpg
 2025-02-15 18:41:48 +00:00