Getting there, still some collateral damage

I spent much of the weekend working on my waypoint generator – more specifically on the code that loads new data into the waypoint generator database. As anybody who reads this blog probably heard (and forgot), the main source of data for outside the United States, the NGA ‘s DAFIF is no longer available, and so I’ve been working on ways that people can contribute data that fills in the gaps and updates the DAFIF data as it gets old.

The problem is how to integrate all these different sources of data of various levels of quality and completeness, and somehow pick out the “best” parts of them all. And that’s tricky, because points can move and their IDs can change. Worse still, the various data sources aren’t very consistent about a point’s “type” . Points can move because they’re re-surveyed, or because the navaids that define then have been re-aligned with magnetic north. IDs can change because an airport needed to get a ICAO compatible ID so that they could participate in weather reporting or international flights, or because of political expediency (there is an intersection near here named “PTAKI” – you can bet it wasn’t called that before George Pataki was governor of New York, and it probably won’t be called that after he’s gone). And types are the biggest problem – one source will have a “NDB/DME”, but another will have separate NDB and DME of the same ID at the same location, there is some mixing for “VOR/DME” and “VOR” for the same naaid, and one source uses the type “DVOR/DME” for what DAFIF would call a “VOR/DME”. The worst are the airborne points – the FAA and DAFIF could never agree on whether a point was a REP-PT or a CNF or a AWY-INTXN or a RNAV-WP.

And to make matters worse, one of the data that I used years ago was “waypoint.nl”, a site that catered to GPS users and Geocachers – it seemed likely to me that many of the airport locations were probably taken at the airport fence, but it had a bunch of airports that nobody else had, so I took them. Well, now that I’m getting better data, one of the things I’m finding is that their airport ids are frequently wrong. I’d like to just get rid of them entirely, but they cover countries that nobody else does.

So what all that leads to is if I’m loading somebody’s data, I can’t just delete all the existing points with the same id (or even the same id in the same general area) and insert what I’ve gotten. So I have to come up with a way to find the same point, even if it moved or changed ids, and use the old, existing data to fill in anything missing in the new data (like if the new data source doesn’t fill in runway or navigation frequency data). And hopefully, not lose any existing waypoints that might be “close”.

Because of the inconsistency of airborne waypoint points naming, I separate waypoint types into three types – “airport” and airport like, ground based navaids, and airborne waypoints. You’ll probably see why I mention that later.

The algorithm I settled on to find the existing point was to call an existing point the progenitor of the new point if the existing point was within 0.1 degrees of latitude and 0.1 degrees of longitude of the new point, it was in the same country (and in the case of the US and Canada, it was in the same state/province), and the same type (unless it was in the airborne waypoint category, where the same category was sufficient). In my first tests with some data I have for Ireland (after converting DVOR/DME to VOR/DME), that worked pretty well – it removed some garbage data from waypoints.nl, and updated some DAFIF data. But there was some collateral damage – a CNF named BAL86 was removed, because it was too close to a REP-PT called MARNO, a similar thing happened to GURGA near KEKUL, the SLG NDB/DME at Sligo was replaced by separate SLG NDB and DME, and who would ever have believe that they’d put waypoints named “TIMAR” and “TIMRA” within 0.17 nm of each other? Doesn’t that last one sound like a cockpit resource management nightmare?

I think I need to make a few small changes to the algorithm, but it’s mostly there. I hope. I need to make some bigger tests as well.