Category: Geekery

Man, I hate RoadRunner/TimeWarner/Spectrum

I run a bunch of mailing lists. Most of them are very active, not too many members, and very chatty. They are also very low-maintenance. One of my mailing lists is an announcement list for our neighborhood association, and has 500+ members, many of them elderly. It gets about 3-4 posts a month, and it’s very high maintenance (see my previous remark about elderly members). A large number of those members (over 50 of them) have rochester.rr.com email addresses. RoadRunner was the first broadband in our neighborhood, offered by Time Warner. Sometime after I went around the streets knocking on doors trying to convince people to sign up for an independent fiber internet company (Greenlight), Time Warner became Spectrum.

I guess some of their infrastructure was owned by Charter? I don’t know, it’s just that the MX for rochester.rr.com points to a charter.net server. Probably due to an accquistion or something. Not relevant.

For my sins, I upgraded my server a few months ago to Debian 13, while simultaneously upgrading my mailing lists from Mailman 2 to Mailman 3. And a few days ago I discovered that since the change, a large proportion of rochester.rr.com users were getting booted off the list because their mail was bouncing. I found this out because a sweet little old lady got the message saying her subscription was being removed because of these bounces, and responded to the bounce message saying “but I don’t want to be removed”.

Since that time, I’ve been trying to diagnose and fix the problem. What I saw in the logs was that RoadRunner gives you a code when it defers or bounces, which you can look up

https://www.spectrum.net/support/internet/understanding-email-error-codes

And most of the deferrals and bounces were getting a 1300 or 1370 code, both of which mean too many concurrent connections, or too many recipients in one connection.

The first thing I found was that Mailman was VERPing every message, which obviously makes it easier for Mailman to determine who bounced, but also means that Postfix is making 500+ simultaneous outgoing connections. I decided it would be better if Mailman just passed the whole shebang off to Postfix and let Postfix pick that apart. That took a bit of doing, including adding a configuration parameter to my mailman.cfg file that `mailman conf` told me was already the default. *Sigh*

Ok, once I had all 500 members coming as one block to Postfix, I set up a separate transport “slow_smtp” just for rochester.rr.com. For that one, I set up 

slow_smtp_destination_concurrency_limit = 1
slow_smtp_destination_recipient_limit = 5

which I thought would mean it would make one connection at a time, and send 5 messages each time. Turns out the concurrency_limit wasn’t doing what I thought it would do – ie make sure that there’s only one connection at a time. I don’t actually know what it does because it looked like there were several connections at once. I showed my configuration on the Postfix Users mailing list, and Wietse Venema rather defensively said “The _destination_concurrency_limit and _smtp_destination_recipient_limit features are implemented by decades-old code that has not changed $forever.” And we all know that old code never has bugs in it. Besides, I was assuming I was configuring it wrong, not that the code was wrong.

Anyway, after some back and forths with Venema and another guy who thought I didn’t know how to read a log file (turns out he was partially right), I added

slow_smtp_destination_rate_delay = 5

which did seem to meant that it would delay starting a new connection for 5 seconds after the last one, and if I’m understanding this correctly, means that as long as the previous connection is processed in 5 seconds, the next one won’t be simultaneous. In practice, what I seem to be seeing is that the first batch of 5 gets sent, the second batch of 5 gets sent, and the rest get “status=deferred”. Some time later (about 8 minutes?) it send the third set of 5 and the forth set of 5, and deferred the rest. After 4 sets of retries, the last 19 users got “status=bounced” instead of “status=deferred”. I have no idea why they suddenly decided to start bouncing. I suspect it’s just Time Warner because arseholes.

I’m still searching for the magic configuration which will allow the non-RoadRunner users to keep going as normal, and RoadRunner users to trickle through in whatever configuration it takes.

It’s done (I think)

The process I started with For my sins and documented my progress with Progress and Regress is now officially (probably) done. Mailman is up and running, mail is being delivered (although as usual, only one of my lists is getting much traffic). The new archives are very cool, with full text and all. I actually had to deal with a Y2K bug – some of the mail in the old archives had the year as “number of years after 1990” instead of “number of years after 0”, which meant I had to fix a bunch of dates with years 100, 101, and in the case of one person who refused to upgrade, 102 and 103. And I even put the websites behind https using the brilliant mod_md, which is even easier to use that certbot, which is what I’d used on some other domains.

I shut down my old linode, but just in case I need to recover something from it in the near future, I haven’t deleted it yet. Linode.com still charges the same rate for shut down servers as active ones, but it’s only by shutting it down I can make sure there isn’t something still relying on it that I forgot.

Two steps forward, three steps back

Since writing Progress in the new server, I’ve been working on the Mailman3 stuff. First I tried installing it from Debian packages. One of the dependencies was for a proxy server, and since it couldn’t resolve the one using Apache (even though mod_proxy is in the default Apache conf-available) it installed nginx, a whole different web server. Seems like overkill to me. Also, I couldn’t find installation instructions that seemed to correspond with what the package installed. I could find them for Debian 10, but things have changed since then.

So for the last 3 days I’ve been trying to install it the manual way, using pip install in a virtual environment. It’s been going very slowly. I finally got to the point where I could create a mailing list, but when I try to import the Mailman the configuration for that list from the old server using the `import21` command, I get a python error. As near as I can tell, the error message indicates that somebody is using the `with` command with a Path object, but the stack trace is coming from deep within a package that used by another package that used by another package that’s used by Django, so that doesn’t make a lot of sense to me. Surely I’m not the first person to try to import an old configuration?

I’m also seeing weird occasional errors from it trying to access an invalid url, like /api on the proxied server, and I have no idea why that’s happening.

I’m almost at the point where I want to kill this entire “manual” install and try the Debian packages again. Maybe the experience I’ve gained from these days of frustration will help me understand what was going wrong when I tried it that way.

Maybe this time I should document the things I found about how the documentation is wrong, and send them to somebody on the Mailman team?

I knew this wasn’t going to be easy, but I didn’t think it would be this hard.

Progress in the new server

As I wrote about in And for my sins…, I’m moving a bunch of stuff from the linode I set up back when Debian 5.0 was the current over onto a new server which I set up with Debian 13. And so far, it’s gone pretty well.

Moving the “alink” site took a bit of trouble, mostly because I made a mistake as where I put the git repository and so I had to change any URLs from $HOME/alink to $HOME/alink/alink. Unfortunately “alink” involves scraping flight data from another website, and currently the users of that site don’t have any flights on the other website so I can’t make sure the site is working right.

Moving the “navaid.com” site was easier, because it’s mostly written in Perl and nothing changes in Perl these days. The biggest change was that the new version of Debian uses a newer version of PostgreSQL, and a new version of PostGIS.

The next task was setting up email. That was mostly no big deal, except I decided to use rspamd instead of openDKIM to do mail signing. It took a few attempts to get it right, mostly because I wanted to use different signatures for the different domains and subdomains I send email from. I had a bit of a setup glitch because rspamd allows you to sign messages with a ED25519 key, but the tool I use for verifying DKIM signatures doesn’t understand them, so Ihad to switch back to RSA keys. rspamd will provide spam filtering on incoming mail as a bonus but I doubt that will be an issue. I left off all the filtering I used to do in postfix for now, figuring that rspamd will handle it instead.

The latest thing I did was move the news spool over to the new server. I’d anticipated that I’d have to rebuild the history and/or overview databases, because I’m pretty sure I had to last time I moved them. But in actual fact, I just made sure I used the rsync option that respected sparse files, and I didn’t have to rebuild. I did get a few errors about inode discrepancies which I fixed with tdx-util -F. I didn’t bring over my cleanfeed implementation because I hadn’t updated it since 2002 so there’s probably a better version out there.

The last thing I need to do is migrate the Mailman mailing lists over. This is the one I’ve been dreading, so I’ve put it off until last. I’m going to have to do more research on how to do it. But the biggest problem is I don’t want to shut down my mailing lists for too long.

And for my sins…

I have a bunch of stuff running on a Virtual Private Server at a company called Linode. This instance first spooled up with Debian 5 was “stable”, and it’s currently running Debian 10. The only problem is that Debian 10 isn’t supported any more, and Debian 13 is now “stable”. At this point, I have two options:

  • I could upgrade incrementally from 10 to 11, 11 to 12, and 12 to 13, fixing all the problems that occur along the way or
  • I could do what the Linode tech support people have been begging me to do for a number of years, and spool up a new instance of Debian 13, and just transfer things one at a time to it.

I’m currently leaning towards the second option, but for two problems:

  • I would need to migrate my news spool. Last time I did that (I’m guessing sometime around 2010 or so?) I had to rebuild my overview database, which wasn’t a big deal, just that everything was kind of broken until I figured out how to do that. Also this time I’m using a new type of news spool, called a “timeCAF”. Don’t ask me what it means, I don’t recall. Only that the old way was one file for each article in each newsgroup, and this way makes files that contain multiple articles, and may or may not be “sparse” in the Linux definition of that. Which might complicate the move. I can’t seem to find any information on that yet, but I’m still looking.
  • The new Debian stable only has Mailman 3. I am currently running Mailman 2. I found a document about migrating from 2 to 3, but I think you need a computer with both installed, so I’m probably going to have to install the other one from source on one server or the other. My thought is that it’s probably better to upgrade on the old one so I don’t have to bring over any Mailman 2 cruft to the new server.

Other than that, I don’t anticipate any major trauma, just a lot of futzing around. I’ve done these upgrades in the past, and it’s usually been mostly a half a day of finding all the .dpkg-dist, .dpkg-old, and .dpkg-new files and diffing them with the corresponding config file and making whatever changes seem appropriate.