Rants and Revelations

Followup to Rants and Revelations » Bad news on the Linode front:

I ran the same generator task on Gradwell and the Linode. On Linode, it took 21m1.1s, on Gradwell, 1m0.5s. Kind of a huge difference, don’t you think? So I copied the database and code to my home machine, which has 1024Mb of RAM instead of 96Mb, and dual Athlon MP1800+ processors, and it still takes 7m46s.

So either Postgres is way slower than MySQL, or I’ve done something really wrong when I ported the code.

I guess my next move is to try the Gradwell MySQL code on my home server and see how long it takes.

I’ve started to port my navaid.com applications to run on the Linode. And I’ve got trouble. If I have three or four simultaneous generator processes running, they all will stall out. And I don’t mean that one will keep running while the rest stall, I mean that none of them will make any progress for 10-15 minutes, and then suddenly they’ll all start running again. I’m seeing load averages go up over 10 during the stall, and then come down to between two or three while they’re running, and then go back up and stall.

I think the problem is a lack of RAM. The only explanation for such high load averages and the stalls that I can find is that if the (virtual) machine is doing a lot of swapping - certainly while the load average is in the stratosphere “top” is reporting almost no CPU usage. And I can’t really see that paying $5/month for an extra 16Mb (bringing me up to 112Mb) is going to help a lot. What I really need is the sort of RAM I have on my home server (1024Mb) or a way to keep these processes from getting so big.

The individual “postmaster” processes are quite big - I wonder if turning on autocommit might shrink that. I’d turned it off, hoping that having the generator processes happening in a transaction would mean that if I load the data in a separate process while somebody is generating that they won’t get an inconsistent view.

And the CreateCoPilotDB processes are huge - at least part of that because the Palm::PDB code just puts everything into hash tables in memory until it’s all collected, and then writes it out en-mass at the end. There’s a reason for that - near the beginning of the PDB file is an index with the file offset of each individual record - and you can’t tell the offset to the first record until you know how big the index is going to be. But I had a thought about that last night - maybe I can write the actual records off into a temp file, and only store the relative offsets from the start of the first record in memory. Then at the end I can write the header and the index using (sizeof index + stored offset), and then append the temporary record file onto that file. Might be worth a try.

I have another problem where sometimes the web page progress meter will time-out and show a server error instead, but I can just raise the Apache TimeOut parameter from 30 seconds to 120 seconds like I have it at home.

I’m not sure what I’m going to do if I can’t fix the performance issues. Possibly bring the navaid site, or at least the Postgres database part of it, back onto my home server. I don’t like that idea at all.

…if it wasn’t for the damn users.

There is a guy who emails a complaint about anything remotely spammy that has any connection to NCF - either posted through NCF, or mentioning an NCF email address, or whatever. Half the time his complaints are pretty trivial - one of my lusers followed up to an off-topic post without putting it back on topic or something. But the other half of the time, he’s the only warning I have that one of my lusers is doing something wrong.

Today I got a spam complaint from him - somebody posted an ad for his “online pizza ordering” service to ott.events. Yeah, I saw it before he sent me the complaint. Sure it’s off-topic, but frankly if you got rid of all the off-topic posts in ott.events it would have only had two posts in the couple of years since it was created. So I didn’t pay any attention.

And then a few hours later, he sent me a bunch more complaints - the same guy was evidently spamming every newsgroup in Ottawa. So I do a search through the news spool:

find ../spool/overview/ -type f -print |xargs grep <idiots email> | sed -e 's?^../spool/overview?/usr/lib/news/spool/articles?' -e 's?.overview:??' -e 's/[ \t].*//’ > list

And sure enough there are 29 of his spams, all in ott.* newsgroups.

Time to get tough. And I just happened to have a copy of Dick Depew’s infamous ARMM bot which Dick himself send me about a year after the infamous ARMM debacle (where a bug in ARMM caused it to issue cancels for its own cancels, in an ever increasing cascade). I dusted it off, fixed the paths in it, and turned it lose. It generated the 29 cancels in no time.

And just to make sure it doesn’t happen again, I fixed bin/filter/nnrpd_auth.pl to reject any connection attempts from this guy. Of course all that means is that he’ll start spamming from Google, but at least then it’s not my problem.

I noticed a bunch of hits directly to pictures in my bird gallery, not going through the gallery interface. And they all had the same referrer string. So I went to the referrer, a MSN group, and was assaulted by the sort of schmaltzy, cornball, overwrought pretention that makes the people on alt.cuddle look sane and well adjusted in contrast. Pictures of fairies and unicorns and that sort of crap. Anyway, somebody put up a page on this group which just leeched my bird pictures, and other people’s as well, and titled it “My Beautiful Birds”.

First of all, they aren’t your beautiful birds, they’re mine. And secondly, didn’t your momma ever teach you about copyright?

So I went into my Apache configuration file, and a quick

RewriteEngine On
RewriteBase /albums
RewriteCond %{HTTP_REFERER} ^http://groups.msn.com/<cornball site>.*$ [NC]
RewriteRule .*\.(gif|jpg) http://xcski.com/~ptomblin/pork.jpg [R,L]

and the picture leechers are going to get a big surprise.

This morning, because of the pain in my elbows caused from overzealous paddling yesterday, I took some asseto^Waceto^Wecset^Wtylenol with codiene. At first I just felt this weird feeling of disassociation from my body - except the pain in my elbows, that penetrated like a million candlepower spot light through fog. But 5 minutes ago I broke out in a sweat and now I feel like I’m going to throw up. It’s going to be a fun day at work.

Update: Around 10:30 I had to literally RUN to the bathroom because I was about to throw up. I got there, but I didn’t throw up. I stood over the toilet for 15 minutes until the urge passed. It’s now 2pm, and I finally feel like I can actually think and work again, but of course my elbows are quite painful - although no worse than they were a few days ago. I used to have a large jar of ibuprophen in my desk drawer, but I took it out for some reason. I think I need to bring it back.

First I wrote the biggest check I’ve ever written in my life, and went to the bank to get a bank check for that amount. Then we went to the lawyer’s office to wear out a couple of pens signing and initialing several trees worth of paper.

Once that was over, I came home and went for a paddle. Without Vicki, so I paddled faster than I should have and took fewer rest stops. I tried to concentrate on making a good stroke without much elbow action, except when doing sweeps. I’m starting to feel a little twinge in my left elbow, so I’ll probably regret it tomorrow. There was a pretty brisk breeze blowing (the aviation weather report said gusting up to 19 knots), which was mostly acting opposite the stream, so paddling downwind and up river and then down river and into the wind, but the creek is twisty enough that sometimes I was fighting both stream and wind, and sometimes I was getting benefit from both.

I did the left fork up from the wier, which is a lot less used than the right fork, and much twistier and with a bunch of false creeks. Lots of paddling with the skeg up, maneuvering around, which probably isn’t so good for my elbows but quite fun.

At one point I coasted along a paddle’s length away from a Great Blue Heron whispering to it that I wasn’t a threat and it didn’t have to fly away. Another point I came around a corner and saw a 20 metre straight stretch of river with 5 geese poking their heads up out of the grass on the bank.

Today I had to explain how a filibuster works to the Republican woman mentioned in Rants and Revelations » Freedom from speech.

When I was a teen, one of the most important influences on my life and my personality was the time I spent at summer camp, at YMCA Camp Beausoliel, on Beausoliel Island in Georgian Bay. I think I went to this camp the first time when I was about 11, and went every summer until I was in the “senior cabin”, around 15 or 16 years old. The last time I went, I actually went for two or three straight terms instead of just one - when you’re there on the day in between the last batch leaving and the new batch arriving, you start to feel and act like one of the staff.

The most important part of every session at this camp was the canoe trip. The first time there I found it a little disorienting, because as soon as we arrived, we did the usual introductions to everybody in the cabin, and then we immediately started planning our menu for the trip. What did I know about planning menus for canoe trips? Not much. Neither did the other campers - so basically it was a monologue by the counsellor. The next day we had a lesson on basic paddling skills, and another lesson on basic wood craft, and the next day we set out on our trip. All the cabins from the 11 year olds to the 16 year olds set out on their trips that day. It must have been very quiet around the camp after that.

With 5 or 6 cabins full of boys setting out on simultaneous canoe trips, and a watercaft program for the more junior cabins, obviously the camp had a lot of canoes. All of them were wood and canvas except a couple of horrible wood and fiberglass canoes that nobody liked. There was a definite heirarchy of the canoes, and the heirarchy of cabins meant that the older you were, the more likely you were to do your trip in one of the Peterboroughs. A bit younger, and you might have had to muddle through with one of the Chesnuts. And the really young cabins ended up with Lakefields. To tell you the truth, I don’t think any of us knew the difference between the canoes, but all the staff fought over the Peterboroughs, and so that’s what we wanted too.

Peterboroughs were classic canoes, built in the glory days of classic canoe building in the 1920s and 30s, in (not too surprisingly) Peterborough Ontario. And when I was an active canoer, I always wanted one. Although, thanks to Bill Mason’s endorsement, I probably would have settled for a Chesnut. I built a “stripper” when I was in college, but even its racy lines and beautiful looks couldn’t still my desire for an old classic.

In the early 1990s, I was working on a contract job where I’d work for nothing for two months, and suddenly get a chunk of cash (with no tax taken out of it) - sometimes as much as $10,000. So I’d be practically starving maxing out my credit cards, and then suddenly I’d be flush with cash. And instead of budgetting for the next two months, I’d rush out and buy a bunch of stuff. One of the times when I was flush with cash, I saw an advert on Usenet. A woman in Kingston had a couple of Peterborough canoes that used to belong to the Royal Military College in Kingston, Ontario. I rushed down to look at them. All of them had been badly treated by the cadets - some had broken or missing boards, or badly done patches. worst of all, they’d slathered layers and layers of fiberglas over them. But they were Peterboroughs, and I was blind to the practicality, seeing only the possibilities of finally having one for myself. So I bought the one that seemed the least damaged - although in retrospect I probably should have taken the one that was more damaged, but which wasn’t missing the name plate.

I took my Peterborough home (and at the time I was living in a house I shared with other people, and didn’t have a proper wood shop). I bought some of those metal folding saw horses and a few tools. I figured the first thing to do would be to get the fiberglas off the outside and the horrible blue paint off the inside. I had it in my head that what I wanted to do was to strip it down to bare wood, and then give it a clear coat of ‘glas and epoxy resin, so rather than having the classic old (and hard to maintain) canvas, it would be durable but show the beauty of the underlying cedar. I got a lot of the fiberglas off - fortunately it was old and hadn’t adhered well to the underlying wood. Actually, it might have been applied on top of the old canvas - I don’t remember. But after working on it a bit, I discovered something I should have realized before I started: that with my bad knees, I can’t stand around a canoe project any more than I can run or ski. So I stopped working on it, and unfortunately the canoe has been sitting outside, unprotected, ever since.

Since we’re moving, I decided it’s time to finally get rid of this poor unfortunate mistreated canoe. I can’t stomach the thought of breaking it up for garbage. So I tried listing it as “free to good home” on the local Usenet forsale newsgroup, and the local paddler’s web forum. One person wrote to me to say that I should try the Wooden Canoe Heritage Association web site. So I did - I listed it this morning before I left for work, and in the next two hours I got 8 phone calls and emails. I guess I’m not the only person who fell in love with Peterboroughs.

As somebody who gets more than his fair share of spam (see this post for the gory details), I see several attempts a day to phish my Paypal account details. So a few days ago I was a little disconcerted to see something that met every criteria for being legitimate, telling me that somebody had requested a password change on my Paypal account. There were no fake and hidden URLs, the email came from an IP that belonged to Paypal, it used my full name, etc.

And it said that if this request didn’t come from me, I should go to the Paypal page to get the phone number for their fraud contact people. So I did, using my own login bookmark rather than the URL they gave me, in spite of me not being able to see anything wrong with the URL. In a fit of extra paranoia, I even looked at the security certificate on the site.

After making me step through a bunch of voice mail options relating to phishing rather that password change, I finally got to talk to somebody, who said that a glitch in their system sent out a bunch of these and I have nothing to worry about. Ok, fine, why didn’t you save us all some time and effort and put information about that system glitch on your web site?

Today, I got an email asking me to fill out a survey based on my call to Paypal customer support. The only problem is it came from a domain other than Paypal. I’m sure there are quite legitimate reasons why Paypal/eBay would decide not to run their own survey, but in this day and age there is no way in hell I’m going to give *any* sort of information about my interactions with Paypal to a third party. (Ok, this blog post is giving information about my interactions with Paypal to lots of third parties, but that’s different - this is “push”, not “pull”.) Paypal, if you want to survey me about your customer support, you’re going to have to do it from your own email servers and your own web servers.

The new departure procedure for Elmira airport, ELMIRA ONE, specifies the headings you’re supposed to fly to the hundredth of a degree.

I’m lucky if I can hold a heading within 5 degrees. Especially if there’s turbulence, and if there’s any wind at all at Elmira there’s going to be mechanical turbulence from the hills that surround the airport.

I’m home helping Vicki recover from her surgery. It appears to have gone well, and she’s not needing me to do much to help her - I expected her to be lying in bed all weekend weakly quavering out “bring me a cup of water, please”, but instead she’s sitting in her usual chair tapping away on her iBook.

While I’m home, though, I took the opportunity to move all my Mailman mailing lists from my home server to my linode. It was incredibly simple, if a bit time consuming. For each list, I copied over the /var/lib/mailman/lists/listname directory and the /var/lib/mailman/archives/private/listname.mbox/listname.mbox file. Then I fixed the permissions with
chown -R list.list lists/listname archives/private/listname.mbox
and fixed the internal pointers and stuff using
withlist -l -r fix_url listname
Then I trimmed down the archives (I don’t have enough room on the linode for the whole thing) using
mbox-purge --before 2004-01-01 archives/private/listname.mbox/listname.mbox
and rebuilt the archives using
su list -c "/var/lib/mailman/bin/arch listname /var/lib/mailman/archives/private/listname.mbox/listname.mbox"
and regenerated the aliases with genaliases. Then I went back to my home machine and removed the mailing lists there and put all the mailing list addresses in /etc/postfix/relocated.

Easy as pie. And now I don’t have to worry that the mailing lists will be down while we’re moving.

Here are the best of the pictures from Wednesday night’s kayaking.

Ah, Spring.

Last night Vicki and I went kayaking. It was great. Vicki used Baycreek’s new Hurricane Aquasports Tracer which looks like a really nice West Greenland style kayak, very similar lines to my Skerray or the Avocet she was using last year, but made of “Trylon” plastic using a new vacuum forming method. Surprisingly sharp bow for being plastic. So is my Skerray RMX, but Valley are famous for how good their rotomoulded kayaks are. We saw lots of geese, some guarding tiny fluffy goslings. Also saw swans, most of them in aggressive postures, swallows, red winged blackbirds, a flicker (which was a surprise) and a kingfisher (which was also a surprise). The river was fast, but not so high as to make the weir too challenging. The reeds are starting to come in, but they’re still low enough that we could see what was down the other branch when the creek diverged. We went pretty far, and my elbows aren’t that sore today.

As another highlight of spring, today was the first real mountain stage of the Giro D’Italia. It’s been great how the sprint stages have managed to avoid being “the Alessandro Petacchi show” that they were last year, but it’s good to be up to the part that matters, where the GC riders make or break. Basso did great today, and I was surprised to see Cunego lost ground to Simoni. I’m still mad that OLN TV isn’t covering it daily like they did last year. And the “live streaming” software only works on Windows and probably wouldn’t work through the company proxy server anyway, plus what’s the point without Phil Ligget and Paul Sherwin? It’s hard to get a real feel for what’s going on when all you’ve got is the web updates, but CyclingNews.com is doing a pretty good job.

And the third highlight is the Kodak Perigrine Falcons. I haven’t been following them as obsessively as I used to when I could compare notes with Maddy, but it’s good to see that they’ve hatched another 5 this year. Pigeons beware!

This is your digital cinema system: Some Tinseltown ‘Star Wars’ shows in new Kodak digital

This is your (competitors) digital cinema system on Windows: Midnight EP3 Horror Story in Maine

Any questions?

Back in December, the systems administrators at The National Capital Freenet gave me a new disk for the news spool. Since that time, the free disk space has been hovering around 75%, which is kind of ridiculous. So around the middle of ‘Week 16″ in the following graphs, I extended the retention of a bunch of newsgroup hierarchies (mostly the big8, the local ones, and Canadian regional heirarchies) by a couple of days. After waiting a week to see where it stabilized, it made almost no difference - it was now hovering around 70-72% free. Big fucking deal.

So on Week 18 I took a more drastic cut, and added 10-15 days to the retention time of all those groups. It has been scary for the last couple of weeks watching that graph on a steady downward trend, holding my breath hoping it would level out before it hit bottom.

Well, here it is Week 20, and it looks from the graphs that I’ve levelled out at around 50% free. That’s much better. That gives me room to deal with floods, but keeps most groups as long as is practical. Personally, I’ve never seen the point of keeping any groups except *.answers for more than 30 days - you know that if you chime in on a discussion that ended 30 days ago you’re going to be pissing off the majority of people who read it and moved on. And every point that’s made in a discussion that’s been going on for more than 30 days will have been repeated dozens of times in the last 30 days.

Anyway, I find these graphs fascinating. On the Monthly graph you can see this lovely sawtooth as the spool fills up during the day, then quickly goes the other way during the expire run. You can see the wierd little mini-spikes when I made my adjustments and run the news.daily expire run a couple of times in one day to make sure I hadn’t messed anything up, and then you can see the much smaller spikes while the groups whose retention time was increased stopped expiring anything while groups that I didnt’ mess with continued to expire things.

Free Disk-Space for ‘/usr/lib/news/spool/articles’ on theodyn

I like these graphs, and I wouldn’t mind having them for my home system and my linode, but on the other hand what I’ve seen of MRTG looks way too complicated for something as simple as monitoring disk space.