View Issue Details
| ID | Project | Category | View Status | Date Submitted | Last Update |
|---|---|---|---|---|---|
| 0002236 | Composr | core | public | 2016-03-01 12:30 | 2019-11-19 01:13 |
| Reporter | Chris Graham | Assigned To | Chris Graham | ||
| Severity | Feature-request | ||||
| Status | closed | Resolution | duplicate | ||
| Product Version | |||||
| Fixed in Version | |||||
| Summary | 0002236: Opportunistic scheduler and background caching | ||||
| Description | Code in a detector function to find if the server is under 'low load', based on I/O usage, CPU usage, and memory usage (configurable). Make the scheduler (Composr CRON system) use this to determine whether some low priority hooks should get done. For example, updating the sitemap. CRON hooks would specify: - a "frequency" (currently this logic is hard-coded) - a "fluff" factor (to determine how much things should be allowed to be late, e.g. 1.6x -- tasks being deferred if not in a low load state) - a "priority" (i.e. low priority tasks happen last and only if CRON has only been running less than say 20 seconds already) - a "ui_responsiveness" flag (set to false normally, see note about AJAX later) ...and the system would automatically launch CRON hooks accordingly. It would also handle locking (which we currently do with custom per-hook code). Code a new CRON hook that simply goes through and compiles any non-compiled templates, Comcode pages, blocks (by evaluating the Comcode pages after compiling them), blocks (just using their default parameters, or perhaps all permutations of parameters a new block method says should be pre-cached), and language files. It should focus on things from 'core' addons first. This hook has the 'ui_responsiveness' flag. Create a new AJAX system that *only* executes the ui_responsiveness flagged hooks. When an admin user is going around the site, this is being called in the background, and if the system is low-load, it is going through and compiling things. A good example of why this is nice, is on a fresh install, the Admin Zone is initially slow due to the sheer amount still to be compiled. As the user is reading through stuff, the stuff is getting compiled for them. Automatically determine which of the admin zone screens are slow. Put a threshold on it, e.g. no screen should be more than 1 second upon second load (i.e. cache-ready load). Create an automated test for that, ensuring base performance on anything on the sitemap. If anything isn't hitting it, this is an indicator we should be implementing a block (with caching) for that screen. For example, the "choose a theme image to edit" screen is slow due to massive numbers of theme images needing categorising and laying out, but a block could handle this. (Certain things may not be on the sitemap currently but should be, I think my example above isn't currently on the sitemap because it is per-theme, but it could be done anyway) | ||||
| Additional Information | I considered shipping pre-compiled stuff, and smarter (less harsh) decaching, but due to the number of different variables, neither of those are feasible. | ||||
| Tags | ocProducts client-work (likely), Type: Performance | ||||
| Time estimation (hours) | 40 | ||||
| Sponsorship open | |||||
| related to | 0003501 | resolved | Patrick Schmalstig | Poor-mans-Cron |
| related to | 0003958 | non-assigned | Opportunistic scheduler | |
| related to | 0003959 | non-assigned | Cache warm up via Cron |
|
|
Actually it was possible to take some of the harshness off decaching, particular template decaching, by focusing on sometimes only decaching a subset of templates based on some scannable dependencies. Also, I was able to make the Admin Zone dashboard blocks use deferred loading, which makes it much faster to load due to parallelisation and not having to wait for the whole page to come at once. |
|
|
Anything on the sitemap would be best populated into the guest static cache too. And anything expired from any cache, regenerated. |
|
|
I'm closing this, splitting this into 2 separate issues, and removing stuff already done or already covered by other issues. |
| Date Modified | Username | Field | Change |
|---|---|---|---|
| 2018-10-14 18:15 | Chris Graham | Note Added: 0005831 | |
| 2019-11-18 21:23 | Chris Graham | Relationship added | related to 0003501 |
| 2019-11-18 23:28 | Chris Graham | Tag Attached: ocProducts client-work (likely) | |
| 2019-11-19 01:11 | Chris Graham | Assigned To | => Chris Graham |
| 2019-11-19 01:11 | Chris Graham | Status | non-assigned => closed |
| 2019-11-19 01:11 | Chris Graham | Resolution | open => duplicate |
| 2019-11-19 01:11 | Chris Graham | Note Added: 0006163 | |
| 2019-11-19 01:13 | Chris Graham | Relationship added | related to 0003958 |
| 2019-11-19 01:13 | Chris Graham | Relationship added | related to 0003959 |
