View Issue Details

IDProjectCategoryView StatusLast Update
0006009Composrcorepublic2024-11-13 00:04
ReporterPatrick SchmalstigAssigned ToPatrick Schmalstig 
SeverityMinor-bug 
Status resolvedResolutionfixed 
Product Version11.beta4 
Fixed in Version 
Summary0006009: Add better handling of notification and mail crashes
DescriptionSometimes a notification digest or an e-mail will cause an out-of-memory error. When this happens, the whole process is halted:

* For digests, it halts / locks the entire hook.
* For the mail queue, the queue halts and no e-mails get sent out (except ones that bypass the queue).

Add a sub-locking mechanism within each using the elective value system. "Lock" a specific notification tin when we start processing, then delete the value when we are done. Do the same for the mail queue. Skip any records which are locked.

This will also help with troubleshooting because one can look into the values elective to see the record number that got locked.

Also add a health check that searches for the presence of any of these value prefixes (and also for background tasks), and fails if any are present and older than 5 minutes.
TagsRoadmap: v11
Time estimation (hours)
Sponsorship open

Activities

Patrick Schmalstig

2024-11-13 00:04

administrator   ~0009576

I took a slightly different approach:

* For mail, it will try up to 5 times (no more than once per hour) on failed mail. Permanently failed mail will be removed from the queue and dumped to a log file in data_custom/failed_mail. A health check will fail if that directory contains any log files.

* For notification digests, we're not using locks. Instead, two cut-offs are now used: item count and e-mail body size. If either is exceeded, the e-mail will be wrapped up. Remaining items will be sent in another digest one hour later [because the hook only runs once per hour] (truncation of items was a bad idea IMO because notification digests cannot easily be opened in full).

Issue History

Date Modified Username Field Change
2024-10-22 22:37 Patrick Schmalstig New Issue
2024-10-22 22:37 Patrick Schmalstig Status non-assigned => assigned
2024-10-22 22:37 Patrick Schmalstig Assigned To => Patrick Schmalstig
2024-10-22 22:37 Patrick Schmalstig Tag Attached: Roadmap: v11
2024-10-22 22:38 Patrick Schmalstig Description Updated View Revisions
2024-11-13 00:04 Patrick Schmalstig Status assigned => resolved
2024-11-13 00:04 Patrick Schmalstig Resolution open => fixed
2024-11-13 00:04 Patrick Schmalstig Note Added: 0009576