View Issue Details

IDProjectCategoryView StatusLast Update
0000159Composrcorepublic2016-06-26 22:50
ReporterChris GrahamAssigned To 
Status non-assignedResolutionopen 
Product Version 
Fixed in Version 
Summary0000159: Google AppEngine support
Description(See notes below, original description no longer valid)
TagsType: Performance
Attach Tags
Time estimation (hours)70
Sponsorship open


related to 0000143 closedChris Graham Addon disabling 
parent of 0001426 non-assigned Support uploading directly to Google Cloud Storage (GAE) 
parent of 0001429 non-assigned Google App Engine, support ticket email receipt 
Not all the children of this issue are yet resolved or closed.


Chris Graham

2013-05-17 00:19

administrator   ~0001397

Last edited: 2013-05-17 00:47

View 2 revisions

AppEngine now supports native PHP. There are some restrictions/quirks, however. You cannot write to the filesystem (well you can, but it must be partitioned - and you cannot write PHP code), and it's not a normal webserver - so custom request handling is required. It would be good to implement support for memcached and task queues.

Chris Graham

2013-05-17 00:51

administrator   ~0001398

Lots of limits still:

Chris Graham

2013-05-17 12:20

administrator   ~0001399

Last edited: 2013-05-20 14:14

View 2 revisions

I'd love to implement this, but there are very serious practical hurdles right now. It's not really just a matter of designing Composr around the appengine limits, it's also a matter of designing a website with appengine in mind and having a developer workflow to maintain that website.

Major issues include:
 - you cannot dynamically install new PHP files, so to add/remove addons or perform upgrades would be problematic (WORKAROUND: Produce an alternative way of specifying addons, such that files may remain when an addon is not installed)
 - Uncertainties about cost. The appengine prices went up a while back, and it may well be a lot cheaper just to buy dedicated servers. (Then again, you can get free hosting on appengine for low-load sites, which is great)
 - No custom PHP setup, e.g. your own choice of extensions, or additional backend services such as ffmpeg
 - No server command line, you have to go through web interfaces to access your files/data

Lesser issues we would need to workaround:
 - 32MB upload limit (stops videos, large downloads). You can workaround with the 'blob store', but it requires a custom uploader implementation
 - A new installer would be needed (probably a local tool that integrates with the push system)
 - A new upgrader would be needed (ditto, with some kind of 'system profile' generator to drive it)
 - Refactoring a lot to use tasks
  - no long running pages, without rewriting them to use the task system
  - no large downloads, so things like CSV download are questionable; we'd need to generate in the background then email a link
 - 1000 file in a directory limit, meaning we'd need to change how templates work
 - Separate writable files from PHP code files
 - Replacing .htaccess with an app.yaml file
 - Implement memcache as a persistent cache backend (as APC etc caching not support)
 - Must manually enable: phpversion, parse_str
 - Must workaround missing: disk_free_space
 - The fake filesystem does not actually properly support directories (although you can put stuff in directories, they are implicit within file paths used, not explicitly existent)
 - Must workaround missing: escapeshellarg, escapeshellcmd, exec, shell_exec
 - Must workaround missing: fsockopen
 - Must workaround missing: tmpfile, tempnam
 - No mbstring (i.e. botched unicode support)
 - Use of Cloud SQL means higher cost than datastore, without getting the performance of non-SQL's eventual consistency; yet using datastore would mean a significant effort (see the original description of this task)

It hurts that it is so developer-centric, because I think there's a great deal of synergy between App Engine and Composr. I love the approach. The 'Major issues' though just stop it being useful to the typical Composr user, and the lack of support for certain key things mean that larger projects need many decisions made around appengine's requirements.

Chris Graham

2013-05-17 12:38

administrator   ~0001400

Philosophically, this is why I love AppEngine...

The constraints it places actually help to guarantee the scalability it is built for. It forces you to do things the right way, then you reap the benefits. It's also pretty simple, you don't need to manage servers or configure infrastructure anymore, so long as you make sure you put your design decisions in line with the philosophy.

Composr has always been engineered to embrace these kinds of sensible rules, and adding more is fine if we can derive benefit from it. We can do it in this project because it is centrally designed, as opposed to other community-developed CMSs, where it is hard to enforce new standards because too many people have to be involved to make it work.

Chris Graham

2013-10-25 12:40

administrator   ~0001849

This is now implemented, but unfortunately cannot be tested due to a critical bug in GAE (a segment violation memory error in the production environment). Google are looking into it, and we're stuck until they finish. I'll therefore have to put it on freeze. It has led to some general improvements in Composr, so was not such a waste, but it's a bit frustrating.

The remaining todo list is...

Test live generally
Test live performance (any latency issues with storage remaining?)
Test automated admin login

Issue History

Date Modified Username Field Change