Category: Revelation

Bear with me here…

While most of my blog entries are an example to the world on how to write in a way that can appeal to everybody, this one is going to be mostly a reminder to myself.

I’m having problems with my waypoint generator on the Linode, mostly because with only 96Mb of real memory, each individual generator task quickly becomes too big and then tasks start swapping, and everything gets horribly I/O bound.

At first it seemed that things were dying right at the very end, and so I lept to the conclusion that it must be in the sort phase, where it takes all the records that it’s retrieved from the database and stuck into an array of references to hashs, and sorts the array by ID. I solicited some opinions on that, and got some good ideas on how to sort by ID in the database while still allowing the priority of datasources that I use now. The most interesting one said

select ...
from waypoints w1
where ....
and field(datasource 'FAA', 'DAFIF', 'Thompson')
= (SELECT min(field(w2.datasource 'FAA', 'DAFIF', 'Thompson'))
from waypoints w2
where w1.id=w2.id)
order by w1.id

But before I had a chance to implement it, I did some testing on my own machine using “ulimit -v” to simulate the reduced memory size. I ran an example query that produces a result file with 71197 records in it, honing in on the minimum memory size that would allow it to finish without getting an “Out of memory” error. Then I cut out the sort stage and did it again. And what I found surprised me. Cutting out the sort stage only saved me 375 bytes, reducing the memory size from 107625 to 107250 bytes. And made the time go from 1:46 to 1:35, a scant 10 seconds or 10%.

Looks like I’m going to have to find another way to reduce the memory footprint. And I keep coming back to this idea I had where I do the sorted query and write each record out to a temporary file as I retrieve it, storing only the id, PDB “unique id”, record number and the offset from the beginning of the temporary file. Then when that’s done, I go back and write the PDB file header, and the PDB file index (which consists of the offset from the beginning of the file, attibutes, category and the unique id), and then append the contents of the temporary file. That way I can avoid having the entire contents of the database in memory.

Side note about the PDB “unique id”: Each record in a PDB file has a 3 byte “unique id”. Normally when you’re creating a PDB file, you leave that as zero and the PDA itself fills it in when it loads the file. But when Laurie Davis created the CoPilot application, it used the unique id as the key to reference the waypoint records from the flight plans. So if I did leave them as zero and let the PDA fill them in, every time you reloaded your waypoint file your flight plans would get scrambled. So I maintain a table with a unique mapping between waypoint ids and “unique ids”. That way, even if you got, say, “KROC” from the FAA data this time and from the DAFIF data next time, your flight plans including KROC would still work, because both KROC ids would get the same “unique id”. That also means every time I load new data into the database, I have to find any ids that don’t currently have a “unique id” for them and generate some new ones. Occasionally I should purge no longer used ids and re-use their unique ids, because 3 bytes doesn’t give you a lot to play with.

Shopping for the person I fantasize that I am…

…not the person I really am.

At a charity auction the other day, I got a $500 gift certificate to a custom bespoke tailor shop, and another $500 gift certificate to a clothing store known for high quality and high prices. No, I’m not going to tell you how much I paid for those together, but it was more than I’ve spent on clothing in my life, except for the time we went wild at Men’s Warehouse before the cruise.

Today I had an appointment at the tailor shop. Imagine my surprise when I found out that their cheapest suit is around $1,200, and they go up from there to over $3,000. I wear a suit about once a year, and a good pair of pants and a blazer maybe three or four times.

But I get into that mood when I’m sitting in a store like that, and think that maybe I could turn into a sophisticated person. I saw a show once where they claimed that every man who tries on a tux thinks he’s James Bond. Me, I started to look at fabric bolts and half made suits and started imaging myself as the chairman of the board.

I could just see myself standing at a fancy cocktail party saying “why certainly, we could host the charity gala at our house if you can’t get it in at the yacht club”. (Of course the new house has a lot to do with the fantasy.) The only problem with that is that I don’t get invited to those sorts of parties, I can’t stand at those sorts of parties because of my bad knees, so I end up being ignored in the corner, and also I don’t drink cocktails. Also, I’m a slob. I can barely get around to shaving every couple of weeks, and I don’t remember to get my hair cut until it’s getting in my eyes and annoying me.

We went to the Red Barn and spent the other gift certificate on a pair of pants and a shirt. Very nice. But I have a feeling I don’t have a bespoke suit in my future.

Oh Canada

You know, there are times when I really miss living in Canada. Two examples from today:

Calling for my weather briefing for the flight home: Most briefers at the Buffalo AFSS read you the text, exactly as you get it from the computer yourself using DUATS (Direct User Access Terminal System), and read at approximately broadband speeds. If you get a better than average one, he might summarize all the local forecasts like “four to six thousand scattered to broken”, which is fine and a good time saver and short hand. I rarely get any insight or analysis. But every single time I talk to the briefers at Gatineau Flight Service, I get advice, I get analysis, I get exactly what I need. Today, I’d seen the TAFs showing the possibility of thunderstorms during the period, but only as temporary (TEMPO) or widely scattered. I looked at the Intellicast Northeast Radar Loop, and didn’t see any signs of these thunderstorms so I breathed a sign of relief. When I called for my briefing, the briefer told me what to expect (marginal VFR conditions, high ceilings but very low visibility in haze), and then he explained that when the TAFs were issued the forecasters were unsure about where the atmospheric instability was going, but it seemed to have dissipated and moved off north of Montreal, and in his opinion I wouldn’t have to worry about anything developing along my route during my flight. He twice asked me if I had any more weather questions. I couldn’t have asked for a better briefing, and while I could see the far off lightning on the Lance’s Stormscope on the way home, there was no evidence of any activity anywhere near my route.

The other example was listening to Cross Country Checkup on CBC radio. They were discussing a pretty controversial topic, the post release restrictions on Karla Homulka. My favourite Canadian expatriate, Maddy, hated Rex Murphy, but to me he’s what a radio host on a call in show should be. He took calls from “law and order” types who lamented the fact that Canada doesn’t have the death penalty, people who expressed a fond hope that she’ll be murdered after her release, all the way to people who felt that while it’s unfortunate that the police botched the investigation so badly that they had to make a deal with devil, but they made a deal with her and so it isn’t fair to impose further penalties after she served her sentence, either because the restrictions are a slippery slope that could lead to life-time restrictions on lesser and lesser crimes, or just because it would make it harder to make these sorts of deals in cases where the police really do need some cooperation to save lives. And with all these callers expressing all these varied views, Rex was polite, he found something to agree with them and amplify, and sometime to challenge with probing further questions. He gently kept them on topic and got them to say their piece and get off. He never yelled at anybody, he never shouted anybody down or cut off their microphones, he never insulted them or made disparaging remarks about them after they were off the show. Such a huge difference to the Air America show I heard on the drive home from the airport, or any other call in show on either left wing or right wing radio in America.

A Wikipedia for Navigation Data

I was reading A blog entry about airport entries on Wikipedia and COPA (Canadian Owners and Pilots Association), and reflecting on how thousands of volunteers, people with too much time on their hands mostly, have made Wikipedia such a comprehensive resource. And it struck me that if we could harness the same sort of dedication and talent from pilots around the world, the vast number of us who rely on DAFIF (Digital Aeronautical Flight Information File) data for our flight planning, flight simulation, moving map navigators and other applications won’t be left out in the cold when DAFIF goes away.

I was envisioning something based on the Wikipedia source, but instead of a free form text entry you’d have a structured form for entering the data, and it would be pre-filled with the data already available from DAFIF, FAA and other databases. Then interested people could regularly check that against their local Aeronautical Information Publication and update it as necessary. The resultant database would be available free to any person or program who currently uses FAA and DAFIF data, such as my own waypoint database generators.

Since I posted a variation of this idea on Usenet last night, I have been informed that one DAFIF using program has already attempting the volunteer approach. The problem with that is that while they will only target users of their one individual program and only make the results available to the users of their one individual program. I think it would be better to make one system that draws volunteers from users of all programs that use DAFIF, and which makes its results available to all users of DAFIF.

My experience trying to recruit volunteers for entering data for my navaid.com systems is that people express interest, but there are very few people who follow through, and most of them lose interest after a few update cycles. The only solution is to make it as easy as possible for people to contribute, and to draw from the largest possible pool of volunteers.

Update: I contacted the people doing the volunteer project for their own product, and they have no interest in doing something for the benefit of the entire aviation community because they can’t make money off of it, and made disparaging remarks about the types of volunteers that this project would attract. Since they’re being assholes, I took the link to their site off this posting.

Good day flying

I flew one of the club’s Archers to Batavia for an oil change and to get the wheel pants put on. I volunteers because I hadn’t flown in over a month and I needed to knock some rust off. A free flight in a plane that’s easier to fly than the Lance seemed like a good way to do it. The weather wasn’t great – there was a broken to overcast layer at about 6000 feet, and shafts of rain scattered all over the place. And as typical for a June afternoon, it was pretty bumpy.

After the oil change, I came back to Rochester and got the Lance ready to fly to Ottawa. I filed IFR at 9000 feet, and the Lance took its own sweet time getting up that high. Man that plane is getting aenemic. I barely got 300 fpm before I retracted the gear, and about 800 fpm afterwards, and getting worse and worse as I got higher. I climbed through a bit of a layer at 6,000 feet, and was in solid IMC at 9,000 feet for about 20 minutes before it cleared off and left me in pretty decent visual conditions. Man, I’m never going to keep IFR current in actual, am I?

I really wish I was able to fly more often to become a “better stick”. I have no problem with the procedures and the thinking part of flying, but I am just not happy with how well I can hold a course or an altitude. I hand flew the whole way for the practice, but even when I briefly put on the autopilot in order to dig out a chart, the autopilot did an absolutely horrible job of holding a heading – the turn coordinator is very very slow to react, and I think that makes the autopilot slow to respond and prone to overcorrect.

This was my first flight to Canada since getting my CANPASS registration. That means that I called beforehand, and didn’t have to call on arrival. Big whup. But it also means that in the future I won’t have to stop off in St. Catherines on the way to Oshawa (because Oshawa customs isn’t available after 4pm) because I don’t have to arrive at an airport of entry while customs is available. If I could have, but I didn’t, choose to land at Rockcliffe, Carp or Gatineau instead of Ottawa, and maybe save a few bucks on ramp fees. I didn’t, because I wasn’t sure if it was going to be IFR conditions and I wasn’t sure of the arrival and departure procedures for those outlying airports. Maybe next time.