Category: Activities

Man, I hate RoadRunner/TimeWarner/Spectrum

I run a bunch of mailing lists. Most of them are very active, not too many members, and very chatty. They are also very low-maintenance. One of my mailing lists is an announcement list for our neighborhood association, and has 500+ members, many of them elderly. It gets about 3-4 posts a month, and it’s very high maintenance (see my previous remark about elderly members). A large number of those members (over 50 of them) have rochester.rr.com email addresses. RoadRunner was the first broadband in our neighborhood, offered by Time Warner. Sometime after I went around the streets knocking on doors trying to convince people to sign up for an independent fiber internet company (Greenlight), Time Warner became Spectrum.

I guess some of their infrastructure was owned by Charter? I don’t know, it’s just that the MX for rochester.rr.com points to a charter.net server. Probably due to an accquistion or something. Not relevant.

For my sins, I upgraded my server a few months ago to Debian 13, while simultaneously upgrading my mailing lists from Mailman 2 to Mailman 3. And a few days ago I discovered that since the change, a large proportion of rochester.rr.com users were getting booted off the list because their mail was bouncing. I found this out because a sweet little old lady got the message saying her subscription was being removed because of these bounces, and responded to the bounce message saying “but I don’t want to be removed”.

Since that time, I’ve been trying to diagnose and fix the problem. What I saw in the logs was that RoadRunner gives you a code when it defers or bounces, which you can look up

https://www.spectrum.net/support/internet/understanding-email-error-codes

And most of the deferrals and bounces were getting a 1300 or 1370 code, both of which mean too many concurrent connections, or too many recipients in one connection.

The first thing I found was that Mailman was VERPing every message, which obviously makes it easier for Mailman to determine who bounced, but also means that Postfix is making 500+ simultaneous outgoing connections. I decided it would be better if Mailman just passed the whole shebang off to Postfix and let Postfix pick that apart. That took a bit of doing, including adding a configuration parameter to my mailman.cfg file that `mailman conf` told me was already the default. *Sigh*

Ok, once I had all 500 members coming as one block to Postfix, I set up a separate transport “slow_smtp” just for rochester.rr.com. For that one, I set up 

slow_smtp_destination_concurrency_limit = 1
slow_smtp_destination_recipient_limit = 5

which I thought would mean it would make one connection at a time, and send 5 messages each time. Turns out the concurrency_limit wasn’t doing what I thought it would do – ie make sure that there’s only one connection at a time. I don’t actually know what it does because it looked like there were several connections at once. I showed my configuration on the Postfix Users mailing list, and Wietse Venema rather defensively said “The _destination_concurrency_limit and _smtp_destination_recipient_limit features are implemented by decades-old code that has not changed $forever.” And we all know that old code never has bugs in it. Besides, I was assuming I was configuring it wrong, not that the code was wrong.

Anyway, after some back and forths with Venema and another guy who thought I didn’t know how to read a log file (turns out he was partially right), I added

slow_smtp_destination_rate_delay = 5

which did seem to meant that it would delay starting a new connection for 5 seconds after the last one, and if I’m understanding this correctly, means that as long as the previous connection is processed in 5 seconds, the next one won’t be simultaneous. In practice, what I seem to be seeing is that the first batch of 5 gets sent, the second batch of 5 gets sent, and the rest get “status=deferred”. Some time later (about 8 minutes?) it send the third set of 5 and the forth set of 5, and deferred the rest. After 4 sets of retries, the last 19 users got “status=bounced” instead of “status=deferred”. I have no idea why they suddenly decided to start bouncing. I suspect it’s just Time Warner because arseholes.

I’m still searching for the magic configuration which will allow the non-RoadRunner users to keep going as normal, and RoadRunner users to trickle through in whatever configuration it takes.

Coming home

This is part 4 of 5 of our long North-ish Ontario trip:

We got a couple of nice weather days in Algonquin Park. On the first good day, Vicki and I went for a paddle. Paddling across Pog Lake didn’t fell like anything other than a good paddle with my wife and best friend, but when we turned the corner into the Madawaska River, and got into the corridor of trees and river, I suddenly felt a spiritual connection that I hadn’t felt in decades. A swallowtail butterfly nearly landed on my kayak, and Vicki said “that’s your mom”.

Madawaska River

We paddled down to the dam, and then turned around and paddled upstream to Lake of Two Rivers. There was a strong breeze and there were waves that shouldn’t have bothered me at all, except the thermarest pad I’ve been using for my sit bone pain raises my center of gravity just enough to made me feel like I’m paddling my Epic V12 instead of my Epic V10 Sport. So I decided discretion is the better part of valor, and turned around. And nearly dumped twice while turning around. Swallowtails showed up a few times as we paddled home.

The next day, we rode our bikes on the rail-to-trail path. Last time I visited Algonquin Park, part of that abandoned railroad was used for the Booth Rock interpretive trail, but now it’s a full blown trail for many miles.

We rode through the campground to a t-junction with the trail, and as we were standing there looking at a sign with historic information about the railroad, we heard an awful screeching. We looked over, and a couple on e-bikes were heading straight at us rather than going around (there was plenty of room to go around, like I said it was a t-junction). The screeching wasn’t them, it was that the woman still had the power on but also had the brakes on, so the brakes were screeching. She fell over (not sure if intentionally) which prevented her from plowing into us full power. I’d like to suggest that the EU is correct to require e-bikes to only have pedal assist, and not have a throttle.

The trail was mostly smooth, but with occasional wet spots. A couple of swallowtails showed up to come along with us. A couple of places we smelled something very close to the smell of horse poop. We figured it was probably moose or deer, until I actually saw one of the poops, and it was definitely bear. I figured I was safe, because if I couldn’t outrun the bear on my bike, I could at least out run Vicki.

During our time at Algonquin, one of my former class mates texted me out of the blue, saying he’d seen from a facebook post that we were in the Huntsville area and would we want to come spend some time at his cottage. Our next stop after Algonquin was supposed to be two nights in Parry Sound, so we thought maybe we’d shorten that to one night and spend one night with John and his wife.

But after the bike ride, both Vicki and I decided that we wanted another day in Algonquin. So we cancelled both days in Parry Sound, booked another night in Pog Lake (which involved moving to another campsite), and then going direct from there to John’s cottage. Unfortunately the new campsite was just overrun by swarms of mosquitos and blackflies and then it started to rain. But I’d rather spend a day in Algonquin with the screens open listening to the rain and the loons and smelling the fresh air than spending two days driving up to Parry Sound and back.

It was great catching up with John, and his wife is very nice as well. John was always the great organizer even back at uni, so I’m not surprised he was in touch with a great many former classmates as well and they have frequent get togethers. I’d like to get in on some of that action myself.

It’s done (I think)

The process I started with For my sins and documented my progress with Progress and Regress is now officially (probably) done. Mailman is up and running, mail is being delivered (although as usual, only one of my lists is getting much traffic). The new archives are very cool, with full text and all. I actually had to deal with a Y2K bug – some of the mail in the old archives had the year as “number of years after 1990” instead of “number of years after 0”, which meant I had to fix a bunch of dates with years 100, 101, and in the case of one person who refused to upgrade, 102 and 103. And I even put the websites behind https using the brilliant mod_md, which is even easier to use that certbot, which is what I’d used on some other domains.

I shut down my old linode, but just in case I need to recover something from it in the near future, I haven’t deleted it yet. Linode.com still charges the same rate for shut down servers as active ones, but it’s only by shutting it down I can make sure there isn’t something still relying on it that I forgot.

Two steps forward, three steps back

Since writing Progress in the new server, I’ve been working on the Mailman3 stuff. First I tried installing it from Debian packages. One of the dependencies was for a proxy server, and since it couldn’t resolve the one using Apache (even though mod_proxy is in the default Apache conf-available) it installed nginx, a whole different web server. Seems like overkill to me. Also, I couldn’t find installation instructions that seemed to correspond with what the package installed. I could find them for Debian 10, but things have changed since then.

So for the last 3 days I’ve been trying to install it the manual way, using pip install in a virtual environment. It’s been going very slowly. I finally got to the point where I could create a mailing list, but when I try to import the Mailman the configuration for that list from the old server using the `import21` command, I get a python error. As near as I can tell, the error message indicates that somebody is using the `with` command with a Path object, but the stack trace is coming from deep within a package that used by another package that used by another package that’s used by Django, so that doesn’t make a lot of sense to me. Surely I’m not the first person to try to import an old configuration?

I’m also seeing weird occasional errors from it trying to access an invalid url, like /api on the proxied server, and I have no idea why that’s happening.

I’m almost at the point where I want to kill this entire “manual” install and try the Debian packages again. Maybe the experience I’ve gained from these days of frustration will help me understand what was going wrong when I tried it that way.

Maybe this time I should document the things I found about how the documentation is wrong, and send them to somebody on the Mailman team?

I knew this wasn’t going to be easy, but I didn’t think it would be this hard.

Progress in the new server

As I wrote about in And for my sins…, I’m moving a bunch of stuff from the linode I set up back when Debian 5.0 was the current over onto a new server which I set up with Debian 13. And so far, it’s gone pretty well.

Moving the “alink” site took a bit of trouble, mostly because I made a mistake as where I put the git repository and so I had to change any URLs from $HOME/alink to $HOME/alink/alink. Unfortunately “alink” involves scraping flight data from another website, and currently the users of that site don’t have any flights on the other website so I can’t make sure the site is working right.

Moving the “navaid.com” site was easier, because it’s mostly written in Perl and nothing changes in Perl these days. The biggest change was that the new version of Debian uses a newer version of PostgreSQL, and a new version of PostGIS.

The next task was setting up email. That was mostly no big deal, except I decided to use rspamd instead of openDKIM to do mail signing. It took a few attempts to get it right, mostly because I wanted to use different signatures for the different domains and subdomains I send email from. I had a bit of a setup glitch because rspamd allows you to sign messages with a ED25519 key, but the tool I use for verifying DKIM signatures doesn’t understand them, so Ihad to switch back to RSA keys. rspamd will provide spam filtering on incoming mail as a bonus but I doubt that will be an issue. I left off all the filtering I used to do in postfix for now, figuring that rspamd will handle it instead.

The latest thing I did was move the news spool over to the new server. I’d anticipated that I’d have to rebuild the history and/or overview databases, because I’m pretty sure I had to last time I moved them. But in actual fact, I just made sure I used the rsync option that respected sparse files, and I didn’t have to rebuild. I did get a few errors about inode discrepancies which I fixed with tdx-util -F. I didn’t bring over my cleanfeed implementation because I hadn’t updated it since 2002 so there’s probably a better version out there.

The last thing I need to do is migrate the Mailman mailing lists over. This is the one I’ve been dreading, so I’ve put it off until last. I’m going to have to do more research on how to do it. But the biggest problem is I don’t want to shut down my mailing lists for too long.