200 outgoing mail limit

Since installing Composr last month, my site has 220 people. We averaging a steady 7-9 signups per day, and that seems to be increasing. On the plan I have with Namecheap, they seem to have limited outgoing emails to 200 per hour, so for the first time, sending out a newsletter choked at the 200 outgoing limit. There's about 20 that didn't get sent. I have a feeling that Composr will hold those (there was a message about that from Composr) for a while and try to send those again. The unsent 20. Is that assumption correct...that it will not try to send the first 200 that were successfully sent, and it will only attempt to resend the 20?

p.s. you can't get this sort of userbase this quick on Wordpress because Wordpress would have blown up already.

Terry
 

Was leaving this for one of the devs to answer with absolute certainty but I'm pretty sure Composr is smart enough to just continue from where it left it off and only send the 20 remaining. I am sure you have discovered the outcome by now though

Actually for newsletters, it isn't that smart (I had to check the code).

For regular email sends, it is smart.

The newsletter sending code is optimised for throughput and not tripping over itself with double-sends if two Cron scripts hit in parallel. The possibility of the mail server refusing an email wasn't considered, but I think it could be - I'll see if a hotfix is possible.

Often it wouldn't be an issue because there'd be a local SMTP server that would itself connect to a relay server - and the relay server would be imposing the throughput limit, with the local SMTP server responsible for the retries.

Thanks Chris, for looking at the code! I do believe the emails were rejected because NameCheap rejected them because my max-emails-out in an hour is 200. It's not an issue for us because this whole project is a learning experience for me.

There are a few strategies I can use until a full solution is worked in to the code, so it's definitely not a biggie. Plus I have until the next Full Moon (don't ask) until I send the next Newsletter out.

I'm going to throw in my two cents about how to code this in future versions. I have a long story about pre-internet days and BBS's and C++ and asynchronous communications, but I'll shorten it to this: buffers. Just an idea here, but if the Newsletters (or the entire Composr email system) used a buffer, it could regulate the number of emails sent. So, in Setup, there could be a number you set which represents the Maximum Number of Emails to Send in an Hour. Any emails that are to be sent to Members are put into a Queue (a buffer). If the number of emails to be sent out is larger (at the time) than the Max Allowed Per Hour, then it sets the end of the hour's Queue accordingly, and it sends the emails. Any overflow will have to wait until the next hour. Surprise Emails that are injected into the Queue are put at the end of the line and sent FIFO. I'd say that if this Queue method is used, it would have to incorporate ALL emails sent out, because otherwise one hand will not know what the other hand is doing, resulting in overflow.


 

Hi,

Quick bullet point reply as I'm rushing out the door.

1) I put a hot-fix in here: 0004126: Newsletter drip send (Cron) should only delete from send queue if send worked - Composr CMS feature tracker
It's a relatively simple change to the code.

2) I'm not concerned about double sends as I just double checked, and we do lock the Cron hooks to stop them double executing in parallel.

3) Most all of Composr e-mails go through either the normal send queue (logged_mail_message tables where m_queued=1) or the newsletter queue (newsletter_drip_send table).

4) However, some, such as account confirmation e-mails, send immediately as they are time-sensitive.

5) So you need to make sure you don't push right against the limit.

6) There are a couple of config options, Minutes between newsletter batches and E-mails per newsletter batch, you can use to control send speed.

7) We are using locking and queue database tables instead to effectively achieve what you describe. It isn't one single buffer though for the reasons I gave in 3/4 above. Particularly we don't want huge numbers of newsletters to get in the way of regular e-mails that need to be quick.

Fantastic!

On the same topic of sending bulk emails, we have another spammer. They gathered all Member's emails and at the same second of time, emailed everyone the typical kind of emails send. Looking at it, I want to share a few ideas.

1. There could be a way to ban an IP based on geography. That way I could just ban that general location the spammer is coming from.
2. Composr could have a Whitelist that contains who can bulk send emails. In general, I suppose the only thing on the Whitelist would be SYSTEM, meaning only Composr could bulk-send emails, but members could not. But the Whitelist could catch and suspend that suspicious activity before the emails got sent out. We now have over 260 Members, and they got spammed last night. In fact, I think that Members trying to bulk-send are very likely to be spammers, so Composr could auto-Probation their level.

I'll try to look through the code to locate where there is some value that contains the number of Emails in queue and who the email is sent From, to see if I can stop them. Any additional ideas about where this might be found would be excellent.

 

Spammers shouldn't be able to actually "gather all Member's emails" - that is protected by the "Edit member profiles/accounts (including banning/validating/promoting them)" privilege, assuming the default CNS_MEMBER_PROFILE_ABOUT.tpl template wasn't edited to remove the restriction, or the e-mails have been put out somewhere else by something new. So the critical thing is how they are sending the e-mails. If they are using the contact member form, them we've discussed how to better protect that with CAPTCHA (those hidden options, which will be regular options in v11). Did you try this? Or you can just completely uninstall the cns_contact_member addon, which I'd probably recommend (because everyone but guests can still send PTs to contact members). The actual content of the e-mails should indicated where they are being sent from because it'll explain the context of the e-mail around the text the spammer 'typed'.

Geographical IP bans can be done at the web server level (e.g. .htaccess file in Apache). However, I don't really like that because (a) it's ineffective (spammers often use proxies, VPNs, and relay through hacked servers). (b) Limiting access to parts of the world makes me sad for that part of the world.

There's no bulk send to members feature per se. The closest is when lots of members have notifications set up and a spammer posts to what they are tracking. You can also use the recommend form to e-mail multiple people random people referrals, but that requires their e-mail addresses, which as discussed should not actually be available. But I think you are talking about lots of individual requests sent close together. There's a tracker issue for limiting this 0000413: Limit message sending frequency - Composr CMS feature tracker

Perhaps this is an issue with permissions.

I have done a few tests.

I can log out of the website Completely. Then, as a Guest, click on Members ( https://witches.chat/index.php?page=members )

From there I can select a Member. For instance ( https://witches.chat/index.php?page=members&type=view&id=Malatesa# )

From there I can click on Email Member. Then I can email the Member. In fact, all Members. In a TCL script, this could be automated.

At that point, with a script, I can gather all Member's Emails.

It's starting to sound like I have incorrect permissions set.

If I create a New Member for testing, this same result exists. I can parse the site using Perl or Python or TCL to grab all of the Members Emails.

I think this is a permissions issue. Basically Guests should not have access to see the Member's Email Addresses (but they currently do). Also, there I need to limit New Members so that they can not see the other Member's Email Addresses (at least until they have 10 days of time and experience or POINTS earned)

Right now, all Email Addresses are exposed to Guests and certanly new Members.


 

Being able to fill in a form to e-mail a member is not the same as seeing their address ("all Email Addresses are exposed").

As long as the captcha addon is installed, it will not be possible to script this against a Guest account because any user will have to fill in the CAPTCHA, which a bot cannot do (assuming they don't have some advanced technique to break CAPTCHAs, which is very rare).

I also previously explained the hidden options you can use to enforce even registered members to be subject to that CAPTCHA - e.g. requiring them to have 20 forum posts and being a member for 20 days to be able to avoid the CAPTCHA.

I suspect you don't have captcha installed, so getting that back installed (it's a bundled addon) would be the solution.

Chris, you are correct again. This ended up being a settings issue on my part, not Composr's. I have someone helping me now that is familiar with Composr's admin pages. She figured out how to stop folks from spamming the other members. It was a setting I needed to set, and not a Composr issue. Thank you for your patience! When I get settled in my new job, (and stuff) I intend to focus on better programming standards and code a few add-ons.

Right. Possibly also guests had been given privilege to avoid CAPTCHA.
EDIT: There's no such privilege. But there is an "Use CAPTCHA" option that could also have been disabled. I'm just saying this for anyone else reading.
I'm going to have a little think if there are issues around CAPTCHA where we unintentionally give people unnecessary rope to hang themselves with.
EDIT: I think for v11 I'll add a new Health Check that will detect if CAPTCHA is not-installed or disabled for a public-facing website. I don't want to prevent people disabling CAPTCHA as it's valid on Intranets.