problem with thumb comcode tag and hyperlinks

I just upgraded from 10.0.31 to 10.0.33 two days ago and I just noticed that posts that were using the thumb comcode tag are no longer working.  I can see the image (in thumbnail size) but when clicking (to view the full size) nothing is happening.  This worked with point release 31 but not with the current point release (33).

This is the code for the thumb comcode tag (source view):


I recreated this code using Comcode Tag assistant but get the same results.

Has this already been reported or been resolved?

Thanks for your help!

Best,
Christopher


Edit:  I just noticed that the comcast code itself no longer appears in the code above (nor does it exist in the forum post on my site)   Additionally, I just discovered on my site that any hyperlinkable link no longer works!  In fact, ALL hyperlinks are gone.  That was not the case prior to installing the patch release (33).

Edit #2:  I went to add back in the hyperlink on one of the posts where I noticed it was gone.  It adds in the editor just fine.  I am able to preview the post and the hyperlink appears to work fine from preview.  But when I actually save the post the hyperlink is stripped and no longer there.

Edit #3: Later this evening while using the site I noticed these error messages when visiting the Polls Archive:

polls - errors.jpg

(Click to enlarge)

and I noticed that the images uploaded to represent each poll is broken.  Again, this all worked prior to updating to .33 as far as I can remember.

Edit #4:  It's been over a day and no response to my request for help so I decided to restore from the backup I took a couple of days ago (before upgrading to 10.0.33).  It looks like references to various images are now working again (i.e. images uploaded to represent each poll, for example).  However, hyperlinks appear to still be broken.  I think this is perhaps a permissions issue?  A user with administrative rights cannot see the hyperlinks in a News post from the home page, but if I go to edit the article the hyperlink is defined in the editor.  So it's there but not working after being saved (i.e. the home page, for example).  But a 'normal' user (without admin rights) cannot even see the hyperlink as defined in the editor.  Extremely weird, so I'm wondering if it's permissions related…and I have no idea even where to start looking.  Any help would be very much appreciated right now .  Also, I'm VERY nervous about upgrading to 10.0.33 as I'm very confident there was a problem with the upgrade that broke some posts (titles and content for both FAQ and Links for example had to be recreated as most of text was truncated).  The restore returned those posts back to their original content.

Hi Christopher,

I'm sorry you've experienced issues.

The custom fields issue was a very nasty bug in 10.0.33. 0004417: Catalogue field text truncated (including custom fields) - Composr CMS feature tracker
I'm sorry about that, it was a nasty blunder, and new automated tests have been added to stop that kind of mistake happening again.

As for the rest, I can't really think with any reasonable confidence what would cause that. I'd need to see a live example of it and work through it. Can you do an upgrade and reproduce the problem on some kind of separate test site? I can't think what would completely strip out all hyperlinks. I can imagine though some kind of JS issue stopping overlays opening up.

If you don't get a reply quick enough on the forum you can try going in the chat room. I stay logged in there, and if prodded I will usually try and have a quick look at something.

Chris Graham said

Hi Christopher,

I'm sorry you've experienced issues.

The custom fields issue was a very nasty bug in 10.0.33. 0004417: Catalogue field text truncated (including custom fields) - Composr CMS feature tracker
I'm sorry about that, it was a nasty blunder, and new automated tests have been added to stop that kind of mistake happening again.

From “Post #7,194”, 22nd October 2020, 5:28 pm

Thanks for the reply, Chris.  Much appreciated.  So is that fix already applied to the 10.0.33 patch update?  Or do I upgrade and then apply the hotfix?  

Chris Graham said

As for the rest, I can't really think with any reasonable confidence what would cause that. I'd need to see a live example of it and work through it. Can you do an upgrade and reproduce the problem on some kind of separate test site? I can't think what would completely strip out all hyperlinks. I can imagine though some kind of JS issue stopping overlays opening up.

If you don't get a reply quick enough on the forum you can try going in the chat room. I stay logged in there, and if prodded I will usually try and have a quick look at something.

From “Post #7,194”, 22nd October 2020, 5:28 pm

It's okay.  I was being a bit impatient as my site is now open and I have 20 active community members trying to use it regularly.  Some are quite new to a community web site so bumping into problems is difficult to manage when you have users who are also inexperienced.  Plus I'm trying to encourage their use of the site…so again, having issues doesn't instill much confidence.  But I know that 'stuff happens' so I'm definitely not pointing fingers.  Mostly just wanting to address issues quickly…before they become impactful  

With that said, I was able to 'fix' the hyperlink issue.  I'm honestly not sure of the cause but the fix was to simply go back into the post and re-save it.  I didn't even need to make a change…just had to re-save and then links started working again.  I'm wondering if there was some corruption somewhere?  Can't understand why something would get corrupted…my site is very new and not overly populated yet (but will be soon) so having corruption already is a bit disconcerting.

I don't have a 'test site' but will look into setting one up for future needs.  However, I do want to upgrade so if I experience anything new from that moment I will be sure to record it for you to review.

And thanks for the tip to go into the chat room is if something 'urgent' needs expediting.  This issue felt a bit urgent to me so I'll be sure to take advantage of that next time (should there be a next time   ).

Thanks again for the response and all the help!

Thanks for the reply, Chris.  Much appreciated.  So is that fix already applied to the 10.0.33 patch update?  Or do I upgrade and then apply the hotfix?  

You would apply the hot-fix after upgrading.

If you're back on the earlier Composr version and happy, I'm not going to push you to upgrade back to 10.0.33 (although reading your post perhaps you have already, I'm not sure). 10.0.34 will be out soon, as we need to put out a patch release to correct this nasty issue (and a nasty search bug), as well as add PHP 8 support.

just had to re-save and then links started working again

I am wondering if the user who originally posted the content is since deleted or demoted, or the group(s) they are in have lost privileges.
It sounds like they don't have "Subject to a more liberal HTML filter" privilege and they posted some anchor tag that didn't match the inclusion list we have hard-coded.
I'd need to see the exact HTML in the editor, and the exact HTML from the source of the rendered page, to give a clearer answer.

Re-saving would correct such an issue as it applies the Comcode/HTML privileges of the user doing the save, not the original content submitter.

Perhaps clearing the Comcode page cache (as a part of the upgrader) triggered the issue. Because that would force a re-render of the content with the new privileges taking effect.

Actually all this does sound like a very plausible explanation of the problem, but it's hard for me to speculate without looking in specific detail.

You would apply the hot-fix after upgrading.

10.0.34 will be out soon, as we need to put out a patch release to correct this nasty issue (and a nasty search bug), as well as add PHP 8 support

Thanks Chris.  I'll go ahead and wait for 10.0.34 before attempting to upgrade again.  Appreciate the tip.

I am wondering if the user who originally posted the content is since deleted or demoted, or the group(s) they are in have lost privileges.
It sounds like they don't have "Subject to a more liberal HTML filter" privilege and they posted some anchor tag that didn't match the inclusion list we have hard-coded.
I'd need to see the exact HTML in the editor, and the exact HTML from the source of the rendered page, to give a clearer answer.

Understood.  These assumptions are very plausible as I did have a couple of test accounts I was using and have recently deleted those.  Not sure if one of those test accounts created one of the problem posts.  I don't think so, but I'm not sure.  Regardless, I also had the Uploads limit issue I was trying to resolve, and do recall making some global permission changes in the hopes that might address the issue…so it is possible I may have changed the "Subject to a more liberal HTML filter" privilege in the process of troubleshooting.  I'll need to go back and check.

I could get you the HTML but I really don't want to waste your time.  But should it happen again I will capture those details for you as well.

My main concern now is how changing certain "permissions" can have a negative usability impact that goes beyond just permissions.  I'm wondering if a helpful warning when changing such permissions is warranted?  Otherwise things look 'broken' when they really are due to a permission change?

My main concern now is how changing certain "permissions" can have a negative usability impact that goes beyond just permissions.  I'm wondering if a helpful warning when changing such permissions is warranted?  Otherwise things look 'broken' when they really are due to a permission change?

It's a tricky problem.

I don't want to throw out warnings in the privileges UI if they're not likely to be relevant.
Scanning to see if they are relevant in any particular case isn't really feasible (too much code, and too long to run).
Storing the privileges-in-effect-at-time-of-editing with the Comcode isn't really feasible due to overhead and the need to retrofit every Comcode page field in the database with a new privileges field.
Mentioning if filtering happens via a visible message is not appropriate, as the filter may do trivial things and warning people (including regular visitors) that it's happening is alarmist and unprofessional. Plus it would only show the first time the content re-renders.
Putting a warning when going back to edit content is technically difficult, and not very relevant anyway because if you're an admin when you save no restrictions will apply.

Here's what we can do though…

1. Improve the HTML filter so that instead of just wiping out <a> and <img> tags, it'll strip them down to just the most basic valid form.
2. If something is filtered out, we can add an HTML comment to the page source mentioning it.
3. Add a new config option to specify that ANY Comcode Page will be treated as having full-Comcode-permissions, and default it to on for new installs. It's only Comcode Page's Comcode that is likely to get regenerated because clearing the Comcode Page cache is common (there are many reasons pages may need regenerating), but clearing ALL Comcode caching is never really done. EDIT: Or we could consider for v11 making a special case of storing the privileges-in-effect-at-time-of-editing only for Comcode Pages, and doing the cache clearance a different way that allows them to stay intact.

How does that sound?

Chris Graham said

My main concern now is how changing certain "permissions" can have a negative usability impact that goes beyond just permissions.  I'm wondering if a helpful warning when changing such permissions is warranted?  Otherwise things look 'broken' when they really are due to a permission change?

It's a tricky problem.

I don't want to throw out warnings in the privileges UI if they're not likely to be relevant.
Scanning to see if they are relevant in any particular case isn't really feasible (too much code, and too long to run).
Storing the privileges-in-effect-at-time-of-editing with the Comcode isn't really feasible due to overhead and the need to retrofit every Comcode page field in the database with a new privileges field.
Mentioning if filtering happens via a visible message is not appropriate, as the filter may do trivial things and warning people (including regular visitors) that it's happening is alarmist and unprofessional. Plus it would only show the first time the content re-renders.
Putting a warning when going back to edit content is technically difficult, and not very relevant anyway because if you're an admin when you save no restrictions will apply.

Here's what we can do though…

1. Improve the HTML filter so that instead of just wiping out <a> and <img> tags, it'll strip them down to just the most basic valid form.
2. If something is filtered out, we can add an HTML comment to the page source mentioning it.
3. Add a new config option to specify that ANY Comcode Page will be treated as having full-Comcode-permissions, and default it to on for new installs. It's only Comcode Page's Comcode that is likely to get regenerated because clearing the Comcode Page cache is common (there are many reasons pages may need regenerating), but clearing ALL Comcode caching is never really done. EDIT: Or we could consider for v11 making a special case of storing the privileges-in-effect-at-time-of-editing only for Comcode Pages, and doing the cache clearance a different way that allows them to stay intact.

How does that sound?

From “Post #7,224”, 24th October 2020, 4:27 am

Whatever you feel is best  

I've learned from this experience so it's unlikely to impact me the same way in the future.  Once I discovered the issue it took me quite some time to debug…so I'm mostly thinking about others and trying to save them some of the headache I experienced.  I completely understand now why it may have happened, but it is definitely an unwanted (or unanticipated) side-effect of changing certain permissions (and not necessarily knowing all the impacts).

I appreciate the consideration you're giving this.  Thanks!

Ok, I worked out a pretty good solution and included it for the next v10 release.
0004450: Improve handling around lost/missing Comcode permissions - Composr CMS feature tracker
It'll store the last known date of Comcode admin privileges when it saves a page for a user. It then uses that last known date as a reference point to override current privileges when re-rendering a page not currently in the cache.

Chris Graham said

Ok, I worked out a pretty good solution and included it for the next v10 release.
0004450: Improve handling around lost/missing Comcode permissions - Composr CMS feature tracker
It'll store the last known date of Comcode admin privileges when it saves a page for a user. It then uses that last known date as a reference point to override current privileges when re-rendering a page not currently in the cache.

From “Post #7,235”, 25th October 2020, 12:51 am

Sounds great.  Thanks Chris!  I look forward to the next update