Test release of v10, with higher I/O performance

Post

Posted
Rating:
Item has a rating of 5 (Liked by KingBast)
#4585 (In Topic #975)
Avatar
Site director
Chris Graham is in the usergroup ‘Administrators’
I invite you to help beta test this next patch version of Composr v10 that has improved I/O performance. Please don't test on a real live website.

I am providing both kinds of installer to choose from:
Attachment

composr_quick_installer-10.0.18 beta1.zip

Attachment

composr_manualextraction_installer-10.0.18 beta1.zip


Explanation

Over the years we've noticed that some Windows machines have significantly worse performance with Composr than if those same machines were running Linux. Specifically, the I/O performance sometimes is much worse on Windows – presumably as Linux either has smarter caching, or has less cruft filling up the cache, or a larger default cache.

Good I/O caching is crucial, as if Composr does 200 file reads to make a request, but each is having to do disk access with a latency of 5 milliseconds, that's 1 second just reading from the disk for that request. And 5 milliseconds is pretty conservative  – a stressed or fragmented system might be much higher. 200 reads seems like a lot, but actually with loading up the PHP code files, and checking cached files aren't stale, that's pretty normal (as it would be for other large systems). With a properly functioning I/O cache Composr should just touch the disk minimally, except when warming up – so file reads would be much less than a millisecond on average.

We've improved things over the years a lot even for systems with bad I/O – but I had a report on Friday about the Setup Wizard timing out on a Windows machine after 30 seconds. The Setup Wizard might touch 2000 files on the final step, so if I/O caching is not working well that's a problem. Normally the Setup Wizard is allowed 600 seconds to complete the step, but some PHP configurations do not allow overriding the timeout.

I decided to take the opportunity to approach optimising I/O performance a different way. I developed an intermediate layer (a PHP "stream wrapper") that would allow me to log exactly what I/O Composr was doing and why, so I could do deep optimisation on it – and so I could also artificially slow-down I/O for benchmarking purposes. Through this I identified quite a few optimisations to make, possibly doubling the speed of the Setup Wizard on systems that are bound by slow I/O.

I have the wizard finishing in 65 seconds for simulated 5ms I/O time on a git clone. That's with every bundled and non-bundled addon running, which roughly doubles the amount of I/O required. I expect this will allow it to finish well within 30 seconds on a Windows system with an I/O caching problem.

Let me know how it fairs. I don't think I've introduced any bugs, and all tests pass, but I might have overlooked something. Usually we wouldn't do deep optimisation for an upcoming patch release, but good performance on any real world system is very important.
I think you'll notice a speed up, even if you don't have any I/O problems.


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Follow me on Minds (where I am most active). Support me on Patreon

Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Online now: No Back to the top

Post

Posted
Rating:
#4586
Avatar
Site staff
KingBast is in the usergroup ‘Super-moderators’
"This server appears to have a relatively slow hard disk, or is overloaded – we had to skip checking the integrity of the uploaded files as it was taking too long." Didn't get that message in earlier versions of v10 and this still makes the initial load of the installer take longer than it should before deciding to show this alert. A refresh makes the alert disappear although I am not sure if the disk files have been checked via that refresh or it's just hiding the alert.

The site does seem snappier in general though clicking the proceed buttons manually doesn't seem as snappy as leaving them to auto-proceed when installing. Will also mention that the initial welcome page didn't seem to load correctly when I clicked Visit Website, I had to refresh there too, though this could be my relatively slow hard disk (on a relatively new PC). This is on localhost Bitnami Wampstack, not Windows based hosting.
 
Online now: No Back to the top

Post

Posted
Rating:
#4593
Avatar
Site director
Chris Graham is in the usergroup ‘Administrators’
Thanks,

The integrity check is only in the manual installer, as it's not realistically possibly to have corruption come out of the quick installer (if it was corrupt, that wouldn't run). I didn't change that, but I ran some calculations, and I can see that we're now really pushing it for this to finish. It would take around 35 seconds to finish for 5ms disk reads, as our file count has crept up over the years passed 6000. I've made some adjustments now so it doesn't check the .php files, as if those are corrupt you'll know about it anyway via some direct error message from that file. In the ideal world we'd still check them, but we have to be practical, and the upgrader's integrity check can still do it.

The automatically proceeding may be snappier simply because it gives the machine a bit more time to settle down.

The start page not loading right on first load, probably something random outside our control, I'm unlikely to be able to reproduce that one.


Become a fan of Composr on Facebook or add me as a friend. Add me on on Twitter. Follow me on Minds (where I am most active). Support me on Patreon

Was I helpful?
  • If not, please let us know how we can do better (please try and propose any bigger ideas in such a way that they are fundable and scalable).
  • If so, please let others know about Composr whenever you see the opportunity or support me on Patreon.
  • If my reply is too Vulcan or expressed too much in business-strategy terms, and not particularly personal, I apologise. As a company & project maintainer, time is very limited to me, so usually when I write a reply I try and make it generic advice to all readers. I'm also naturally a joined-up thinker, so I always express my thoughts in combined business and technical terms. I recognise not everyone likes that, don't let my Vulcan-thinking stop you enjoying Composr on fun personal projects.
  • If my response can inspire a community tutorial, that's a great way of giving back to the project as a user.
Online now: No Back to the top

Post

Posted
Rating:
#4631
Avatar
Standard member
ХорошаЯ is in the usergroup ‘Well-settled’

Я самая красивая из стаи синих птиц.
Online now: No Back to the top
1 guest and 0 members have just viewed this.

Statistics

Users online:

amit.nigam, Manu, 220 guests

Forum statistics:
  • 932 topics, 4,477 posts, 5,422 members
  • Our newest member is Sarawilliams7
Birthdays:
Back to Top