Plus many spam blacklists have blacklisted entire IPv4 ranges as "residential IPs" (meaning even if your ISP does allow SMTP (which many do not) then your emails may still be blocked as "spam.").
Only way around all of this is to pay for a "business" internet connection (with 500% markup for the same speed, single dedicated IP, and a pretty poor SLA).
I pay about $20-25 more for my business line than residential service, for the same speed and 5 dedicated IPs. Yes, it's more, but it's not 500% more....
>with 500% markup for the same speed, single dedicated IP, and a pretty poor SLA).
This is overly cynical. There's a world of difference between your crappy Comcast line at home and my fiber line at work. Unless you're actually responsible for an infrastructure, then I don't think its easy to appreciate the difference, but yes, its worth every penny. You may not realize it but jitter, latency, routing, etc are far, far better on proper business lines. Not to mention SLA's, dedicated bandwidth, and actually having someone to yell at on the phone who can do something for you instead of being yet another anonymous schmuck on discounted service that's oversold and runs like crap.
Oh, at previous gigs we tried "business cable" which was laughable. All the downsides of Comcast but somehow even worse uptimes and incompetence.
Personally, I agree that smtp traffic originating from residential blocks should be weighed much heavier than other blocks. Wanna see my mail logs at my spam filter edge? Most Windows computer in the world has been taken over in some fashion by malware. Its worse than you think out there.
I can't give real logs because of privacy concerns but in the past 7 days we've been at 79.2% spam. Roughly 2% of that is straight up malware like cryptolocker, various botnets, trojans, etc.
I've run a personal mail server on Comcast business class for about 2 years and haven't noticed this problem. Probably the majority of my outbound mail is to Gmail. I don't remember if I've checked the RBLs before, but I probably should re-check.
I have problems with several places that firewall the Comcast IPs - Symantec (messagelabs) is the worst because I don't know who uses their services until the message queue backs up and then I have to route around the damage. Really annoying.
To make it worse, I cannot find a way to contact them and ask them politely to whitelist our email server IP address.
Most mail services give a a rejection and, when I've asked to be whitelisted, they do it with no problem (Microsoft, for instance). Symantec simply doesn't respond on port 25 and has no contact information.
The big providers are actually less of a problem when it comes to RBLs than smaller ones, as the big providers have their own filtering and/or rely on reputable RBLs. The bigger problem in my experience is companies with overzealous IT people with the authority to largely do as they please, where you get some percentage that pick their favorite set of block lists with no consideration for whether or not said block lists are properly run.
We occasionally get on block lists, and though we do get onto reputable block lists now and again after mistakes by users, they're rarely a problem and quick to get off. But now and again we get on some shady block list where there's no quality assurance, and where delisting procedures are onerous. Every now and again one of them is an outright money-making scam ("pay $99 for express delisting, or wait for X days for the regular service; oh by the way we just randomly added you with no basis")
You certainly should check. As I'm sure you are aware, some RBL's areole aggressive than others. Some are down-right draconian, and some are more like a whitelist in that they leverage grey listing so any outbound mail is given a few good points in it's total possible spam score.
I ran a medium email ( late 90's ) mail server system on a dedicated 100mb/s line, on an older dual cpu Power Mac actually, but I pulled out all the GUI and tried to run it mostly as a server using the client OS and only the CLI, making it more like running FreeBSD or openBSD I suspect.
I got a /24 IP range out of Comcast which I think helped as it was a previously never used /24, or at least not in use long enough that it had fallen out of RBL's
I'm sure you are aware of the sites that checks your IP against all 200+ or so mainstream RBL's.
There are a few RBL's that block Comcast business as well as all cellular ranges and quite a bit more. I found RBL's to be more trouble than worth.
Ideally, you are dropping the connection at the very beginning, just after the EHLO/HELO., highly efficient, perhaps marginally more efficient CPU-wise than greylisting. Fail2Ban is probably your best friend. Do it closer to the network if possible.
The above, versus pushing the message through various spam filtering, address validation, and other measures. There's heavy CPU usage post greylisting, greylisting is great, but waiting that 5-15 minutes for the retry is too long, most users don't use a password manager. Their password manager is "forgot your password". Insane, but that's how I watch all my friends do it. They learned not to re-use passwords, but now they rely on third party email to send forgot password messages. At least people can tell which are not hashing/+salting their credentials. Though I doubt most blink an eye at a raw password in an email. Clients have email me their credit card data. I tell them delete their debt message, then dived 15 minutes doing tech support to help them locate the credit card sent email.
I was far too often adding whitelist entries, making certain clients ignore greylisting or drop down to 30 seconds, for which I saw no increase in spam. It's the retry value on the server that's set too high/long. Not to mention the 10% or so of older email servers that don't support greylisting so never retry. A clear RFC violation, but these are proprietary systems and not made with greylisting in mind. You could achieve it with a proxy running greylisting or ASSP if you want a full proxy with web admin and essentially a full MTA in a proxy all in one file of many tens of thousands of perl code. No version control, crazy version numbers, and I think still on SFNet ( Source Forge )
At some point you realize you have added aol, yahoo, gmail, etc., all the big guys, they need whitelisting because their IP ranges are huge, and ever changing. Keeping on top of which IP's they are publishing as public MTA based IP's, you can build a solid whitelistable IP range, but they change often enough it's not feasible. A SAAS that checked all this would be nice but latency may be too much. Less than the 5-30 minutes of greylisting though.
AOL fully ignores greylisting last I looked. Not to mention DNS TTL's are/were ignored. Sucked to set my MX TTL to 300 seconds, wait the 8 hours for my default DNS expiry, switch my DNS, and then wait 2 weeks for DNS proposition with aol. Not to mention all the paperwork that needs filling out to get into their MTA whitelist program which kicks your sending threshold up by a percentage so it's not a true whitelist. I had to renegotiate every time we added another 5000 to our outbound, it would flag me and I could see all the aol messages stuck in my outgoing queue. Two weeks was longer than my greylisting time, messages bounced because aol saw no server at the other end. I had to keep a "gateway" running, basically I set up a secondary MX to sit there and deal with the aol 2 week DNS propagation to happen.
Finally I gave up and added an external SMTP only machine in a colo. small machine, under 100.00 a month, all it did was SMTP. Deliverability was significantly better after that but ruined the notion of securing my data at my location. But it is relatively ephemeral in that its just SMTP, the data sends instantly, unless it hits greylisting servers or other things that initiate a SMTP retry. That leaves messages and retry logs on the remote server out of my control physically.
Most don't answer postmaster@ or abuse@ which are RFC. Some even bounce meaning they don't even have those addresses available. Having a support/postmaster/abuse @ address that you certainly should check. As I'm sure you are aware, some RBL's are more aggressive than others. Some are down-right draconian, and some are more like a whitelist in that they leverage grey listing so any outbound mail is given a few good points in it's total possible span score.
I ran a medium email server system on a dedicated 100mb/s line, on an older Power Mac actually, but I Pulled out all the GUI and tried to run it mostly as a server using client OS and only the CLI, making it more like running FreeBSD or openBSD I suspect. Added 2x SATA cards and an additional CPU to speed things up. IMAP is purely I/O limited and I wanted and had everyone in IMAP. but IMAP needs a few horses behind it, I wish SSD's were where they are now back in the 90's. That would have been great. I moved my 4GB IMAP account into pure ram disc. It was pretty amazing. Millisecond loading of a mailbox with push enabled in milliseconds to load many tens of thousands of messages and attachments.
I got a /24 out of Comcast which I think helped as it was a previously never used /24, or at least not in use long enough that it had fallen out of RBL's
I'm sure you are aware of the sites that check your IP against all 200+ or so mainstream RBL's.
There are a few RBL's that block Comcast business as well as all cellular ranges and quite a bit more. I found RBL's to be more trouble than they are worth. Good in theory and a great thing to run your own local RBL as it's such a nice way to manage it, through DNS. Want to whitelist or block, just add a DNS entry after reversing the IP. Simple, immediate, understandable.
Ideally, you are dropping the connection at the very beginning just after the EHLO/HELO. Versus pushing the message through various spam filtering, address validation, and other measures. There's heavy CPU usage post greylisting, greylisting is great, but wIting that 5-15 minutes for the retry is too long, most users don't use a password manager. Their password manager is the "I forgot my password" mechanism.
I was far too often adding whitelist entries, making certain clients ignore greylisting or drop down to 30 seconds, for which I saw no increase in spam. It's the return value on the severe that's set too high/long.
Eventually I ran out of granularity per user to set things how I wanted. Setting up a server for a subset of users was asking for trouble.
At some point you realize you have added aol, yahoo, gmail, etc., all the big guys, they need whitelisting because their IP's are huge, and ever changing. Keeping on top of which IP's they are publishing as public MTA based IP's, you hBe a solid whitelist able IP, but they change often.
AOL fully ignores greylisting last I looked. Most don't answer postmaster@ or abuse@ which are RFC required for postmaster and suggested strongly for abuse@.
Oracle ended up in spamhaus, after weeks, I finally got I touch with their admin. ( grind worked there, could not email him ) He'd ( Oracle MTA Admin ) been trying to figure it out for 4 months of bounced emails. How they never had a forwarded email from a bounce that shows the 5.5.0 code is all they would have needed.
But that poses another problem. You have to whitelist abuse@ and postmaster@ or the data you send them will trigger everything, anti-spam, and bounce. But then you get tons of spam, fully unfiltered on those two accounts.
Then there's DKIM and the rest plus the rest of the DNS based pseudo AUTH mechanisms. But those too are problematic. Some can't forward without bouncing, under SPF I believe. Oracle ended up in spamhaus, after weeks, I finally got I touch with their admin. He'd been trying to figure it out for 4 months or bounced emails. How they never had a forwarded email from a bounce that shows the 5.5.0 code is all they would have needed.
But that poses another problem. You have to whitelist abuse@ and postmaster@ or the data you send them will trigger everything from anti-spam and bounce. But then you get tons of spam, fully unfiltered on those two accounts.
In the end, I'm basically tailing logs, massaging them into a SNMP walkable data point so I can graph and chart. All to find out managing a mail server really is a near full time job. Add a secondary and it's even worse as spammers will direct target your mx2 Which means filtering, whitelist, blacklist, greylisting, geographically separated, and network role separated ideally, etc., full parity on two physically disparate boxes.
I was pushing about a million messages a day across a few 10's of thousands of accounts, forwards, aliases, etc. all over Comcast business.
The upsides: 4 hour support window 24/7/365. Direct phone line that a person answers. That person is an engineer of some sort or will transfer you to someone. I've spoken to engineers that maintain DNS, mail, web, etc. you get as close to he source as possible and they know how to unpack a gzipped or tar'd log file batch if you can even get a contact address for them.
Sorry if there's duplication of content. Either something is up with HN or my phone is being my phone as usual. I apologize for the illegibility in places.
Only way around all of this is to pay for a "business" internet connection (with 500% markup for the same speed, single dedicated IP, and a pretty poor SLA).