Author: Paul Tomblin

Archives finally satisfactory

When I last checked in I was having a little problem rebuilding my mailman archives.

After fixing up the corrupted mailing list (by finding a backup of the config.pck file), I decided I needed to blow away and retry building the archives again.

I should mention that the main reason this was such a hassle is that the file was too big to edit in vi. Everytime I tried, the machine would slow to a crawl as vi consumed all the memory and most of the swap.

First thing I discovered is that my modified script wasn’t doing the right thing in a lot of cases. But I also discovered that in the mailman distribution is a user contributed script called "bin/cleanarch", which uses "mailbox.UnixMailbox._fromlinepattern" from another package to recognize proper From lines and only proper From lines. It even looks to see if the next line is a mail header, in case somebody decided to include the From line from a different message.

I ran my mbox through bin/cleanarch. Then I ran the mailbox splitter awk script to split it into 500 message chunks. Then I blew away the archives, and run bin/arch on each chunk in turn. This took over an hour to finish, but at least it didn’t use up all the memory on the system. But I discovered that bin/arch was getting confused about 8 or so messages from early 2000 where a few people were using a non-Y2K compliant MUA that was filling in the date with a year of “100”.

So I fixed those dates using sed, and repeated the process. An hour and a half later, I discovered a couple of cases that bin/cleanarch didn’t handle, where somebody had quoted full mail or usenet news headers from an article.

So I fixed those cases individually using sed, and repeated the process. An hour and a half later, I discovered that there was one From line I missed. At this point, I said “to hell with it” and declared myself done.

I’m starting to thing it would be really nice if Postfix were to escape From lines in the middle of a message. It knows the boundary of a message already because it deals with the envelope. I wonder if that’s an existing Postfix option? Or maybe it could be done by whatever it is in Mailman that writes to the mbox file?

I thought I was the mightly debugging king…

…but I just handed my debugging crown to him.

Kris and I have been banging our heads on our desks because of problems we’re having with our JTreeTables. A JTreeTable is a class that we found on a Sun Java forum that combines the attributes of the JTable with a JTree – basically giving you JTree behaviour with columnar (table) data. It’s really handy. But frequently, and often in cases I could easily reproduce, the damn thing wasn’t updating correctly. Kris and I both made sure that our updates were properly protected by synchronization locks, and the events were being fired in the event loop, lessons we’ve both learned by hard experience. But it was still acting strangely. Kris spent a lot of time reading forum posts, running the debugger down deep in Java library code, and basically working this problem from all angles for days upon days.

Yesterday he found the problem. And the problem was in my code. When you fire an event, you need to give it an array of Objects that starts at the root node of the tree, and follows down through the tree to the node that actually changed. But of course, when you actually change a node, you’re already at the node that changed, and it’s pretty easy to trace up from node.parent() to node.parent() until you reach the top, so that’s what I do. And then I attempt to reverse the order of what I’ve got to make the required array. But it appears that I fundamentally misunderstood the Stack class, because pushing objects on the Stack and then doing a “toArray” on it doesn’t reverse the order, as I’d thought. So the view was getting a totally messed up event, and that was messing everything else up.

Kris changed my Stack.push into ArrayList.add(0, node), and everything works now. And I never thought about doing it that way because I thought List.add(0, Object) would replace the object at position 0, not push them all up.

And Kris’s small change (after big effort) closes three bug reports assigned to me, and a bunch that were assigned to him.

In my defence, I’d actually come up with the Stack thing while Vicki was driving us to Pittsburgh. So perhaps it wasn’t my best work.

NOOOOOOOOOOOOO!!!!

Yesterday, while trying to fix the problems with my mailman mailing list, I decided to rebuild the archives on the mailing list that was giving me problems. But I got the syntax of the “for i in *; do ... done” command and instead of running mailman’s arch command with carefully snipped out parts, it instead ran it with the whole archive. And that meant that arch quickly chewed up all the swap space available. I became unable to kill it, and quickly lost connections to both the domU in question, and the dom0. I couldn’t even ssh back into the dom0.

Not being clear of thought, I emailed the colo company asking if they could power cycle my box. 5 minutes later I realized that all my out-going mail goes through the colo box so it wasn’t going anywhere, and so I phoned them the request instead. They power cycled, I got control of my colo box again, and I got the list fixed up and the archives rebuilt.

But I noticed that this email to the colo company was still sitting in the outbound queue on the colo box, hours later. I didn’t think anything about it, until about 10 minutes ago I got a response to it, dated 10 minutes ago, saying “ok, I’ll power cycle it now”. I immediately fired back a “NOOOOOOOOOOO!!” email, but of course it was too late – the box went down, and now it’s back up.

And I notice that my email to the colo company is still just sitting there, with

88AB94F0AFB 1094 Tue Dec 5 10:54:49 ptomblin@xcski.com
(connect to mail.annexa.net[66.162.186.199]: Connection timed out)
annexa@annexa.net

Something tells me that email isn’t the best way to talk to these guys.

Oh oh. That’s not good.

It appears that something I did to my mailing lists yesterday may have totally fucked them up.

While I was at lunch, I got two emails, one from a member of list [PH] saying that all the email was coming tagged with the subject line tag for list [TH], and one from a member of list [TH] saying that all the list mail was coming tagged with the subject line tag for list [PH]. And so I go to the listinfo page for list [PH], and see the listinfo page for list [TH] instead. But the listinfo page for [TH] seems ok.

I looked at the archives page for [PH], and it’s totally corrupt – not just not archived correctly, I’m talking corrupt HTML here.

And the /var/log/mailman/errors file has these errors:

Dec 04 12:14:38 2006 (23894) Uncaught runner exception: could not find MARK
Dec 04 12:14:38 2006 (23894) Traceback (most recent call last):
File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 111, in _oneloop
self._onefile(msg, msgdata)
File "/usr/lib/mailman/Mailman/Queue/Runner.py", line 167, in _onefile
keepqueued = self._dispose(mlist, msg, msgdata)
File "/usr/lib/mailman/Mailman/Queue/ArchRunner.py", line 73, in _dispose
mlist.ArchiveMail(msg)
File "/usr/lib/mailman/Mailman/Archiver/Archiver.py", line 214, in ArchiveMail
h = HyperArch.HyperArchive(self)
File "/usr/lib/mailman/Mailman/Archiver/HyperArch.py", line 599, in __init__
self.__super_init(dir, reload=1, database=db)
File "/usr/lib/mailman/Mailman/Archiver/pipermail.py", line 289, in __init__
d = pickle.load(f)
UnpicklingError: could not find MARK

This can’t be good.