Nav data update

I’m testing a new update script for my navaid.com waypoint database. The old update scripts were written for when I was running on MySQL, and I’ve switched to PostGIS to support the new iPhone version of the CoPilot flight planning program. One of the salient features of the new iPhone version is that it attempts to be smart about downloading waypoints as you need them. One of the ways it does that is by asking my server for all the points in a particular area that have changed since a given date. The app keeps track of all the “areas” it has seen, and when the last time it was updated, and asks for an update of those areas at certain intervals. But that means I have to keep track of when a point was last updated. It also means that I need to keep track of what “area” a point is in. For the areas, I use a pseudo-quadtree where I allow only 500 points in an “area”, and when it gets more than that I split the cell into four sub-cells and mark the original cell as “superseded”. The new sub-cells have a “supersedes” value, so if the app asks for an area X, and area X has been split, I can say “X has been superseded, and here are the area ids A, B, C, and D that supersede it.”

But all this means that my new update scripts have to get the new data for a waypoint, figure out which old waypoint it was equivalent to (even if the waypoint has been resurveyed and is at a slightly different location and/or it has changed id), and only save the point anew if something significant has changed. Oh, and if the new data is missing information that the old data has, try to be smart about keeping the old data – for instance, George Plews’ Airports In Canada web site has data for airports in Canada that I can’t get any other way, but it’s also got data for airports that either were in the DAFIF data or are in the FAA data, and those two data sources often have much more information about runways and communications frequencies that Plews doesn’t have. So I want his latest data, but I don’t want to lose the other stuff that he doesn’t.

One of the things I do to match up the old with the new data is look with a bit of geographic “slop” – in the case where the ID matches I look within 0.05 degrees latitude and 0.05 degrees longitude (which believe me, in Alaska is way too big an area), and if the IDs don’t match, I look within 0.025 degrees longitude and 0.025 degrees longitude. These numbers were chosen extremely arbitrarily, and still causes a bit of a problem with a couple of airports that are near the US/Canada border because when I’m loading the FAA data it changes some Canadian airport to the nearby US airport, and then when I load the Plews data it changes it back.

Testing out my load scripts, I discovered two things:

1. Sometimes the resurveyed point has moved enough that it’s in a different “area”. And that’s going to confuse the hell out of the algorithm that the app uses for getting updates, because it will ask for updates for the old area, and not get anything for that point. That’s going to require some thought to fix.
2. In the next FAA data load, they’ve actually moved a couple of airports by 1.0 degrees of latitude or longitude. And judging by what I’m seeing on the Our Airports site maps, it appear the new values are correct, so the old ones must have been a data entry error. In this case, my “match the old” algorithm didn’t find anything to match within its radius of action, so it made a new point and marked the old one as deleted. The app should deal with that nicely.

Hmmm. Need to think how to handle this…