Category: Activities

Backing up

Unlike some minor internet celebrities, like one of the founders of StackOverflow, I am a big believer in making sure my data is backed up locally (for quick retrieval) and offsite in case of disaster. For the Mac, this is dead easy – I have a TimeMachine disk attached to my iMac, and I back it up offsite using BackBlaze Cloud Backup – $50 a year, and it’s unlimited so it even backs up my external drive full of my iMovie and Final Cut Pro projects. I also use Syncthing to replicate several important directories between my iMac and my Linux computer. Unlike Dropbox or Box or any of the other cloud replication services, Syncthing is free and doesn’t send your data to any third party.

But I also want the “two backups” system for the Linux computer here in my office, and also my virtual private server box that I have on Linode. And unfortunately Backblaze doesn’t support Linux yet. So for several years now, I’ve been doing a local hourly backup on my local computer to an external USB drive. My strategy for that is:

  • Make snapshots of every partition on the box using
    lvcreate -L 10G -s -n ${dir}-snapshot /dev/ssd/${dir}
    (I wish there was a way to snapshot them all at once to get a really consistent view of the file system, but I haven’t found a way.)
  • Mount them all under a root snapshot so I’ve basically got a static view of the system.
  • Back that up to the external drive using rsync. I use “--link-dest” so it really only backs up the files that have changed and makes hard links to the ones that haven’t, so it’s pretty fast and doesn’t take a lot of disk space to keep weeks of backups.
    rsync -vaSH --delete --exclude .gvfs /mnt/snapshot/ --link-dest=$PREV/ $DEST/allhats2/hour$HOUR/
  • unmount and lvremove the snapshots file systems.

As well, once a day I backup one of these snapshots to a third Linux system I haven’t mentioned yet. A while ago I discovered a company called Kimsufi that was advertising VPSes with Atom processors in Ireland or France (I’m actually not 100% sure which). Nothing too fancy in the CPU or RAM department, but with 1TB of disk space for 10 Euros a month. Not as cheap as Backblaze, but better than losing everything if Linode burns down. So once a day, I copy one of my hourly backups (still using rsync with the --link-dest option so multiple daily backups don’t take up much space, and I also backup my Linode the same way (except there’s no way to take a lvm snapshot inside the VPS so it’s just a simple rsync --link-dest of everything.

That’s been kicking along nicely, and it’s a good backup. But it’s brittle – if an hourly backup fails part way through and try to do an --link-dest the next hour using that one, I can end up with two full copies of the backup with no hard links. And there are other problems as well. Plus keeping the Kimsufi VPS just to back up to seems like a waste.

A while ago, a friend recommended Borg Backup as a better solution for regular backups than trying to roll my own. It also encrypts and compresses the backups. So I set up a parallel hourly and daily backup using Borg. The offsite one is still going to the Kimsufi VPS, but it is faster than rsync. So I was just about to pull the plug on the rsync backups and go to an all-Borg solution when that same friend then recommended Restic. Now Restic resembles Borg in a lot of ways, but it’s not very well documented. But one thing it does have is that it can backup to Amazon S3 and Backblaze B2 cloud storage. Doing some quick calculations with the B2 calculator, I think I can do my backups for way cheaper this way than maintaining a VPS in Ireland – like on the order of a dollar or two a month, rather than 10 Euros. The only problem is getting it to work.

And here’s the problem. Doing hourly restic backups to my local machine are pretty quick – about the same time as borg takes. The only problem comes when I want to delete yesterday’s hourly backups (or rather apply the rules to keep 24 hourlys, 8 dailys, 4 weeklies, and 12 monthlies). With borg, I apply these rules every hour when I back up because they’re quite fast. But they’re so damn slow with restic that had to start only doing it once per day – it takes just a few seconds to run the “forget” command to apply the rules and have it forget about the snapshots that it doesn’t want, but to run the “prune” command that actually reclaims that space, it takes over 30 minutes! And that’s on a local file system. I’m afraid to see how long it’s going to take on the B2 file system, and whether it bumps up the storage costs terribly.

I also had a bit of a teething problem with my first backup to B2 – I was doing it over a network link, and I forgot to do it in a tmux session, so when the network connection got interrupted, it left my first backup in a strange state. I did another full backup the next day, and then I was using twice the required storage on the B2 bucket. I did a restic prune which reclaimed the space, but it took 35 hours to do it. That’s not going to be useful. I need to do a couple of non-failing B2 backups and see how long prune takes in those cases – but if it’s going to take hours, I’m going to ditch restic and go back to borg to the VPS.

Oh, another thing I should mention about restic – It puts a ton of files in ~/.cache. Since I was backing up from the root account, I ended up having to resize my root partition from 4GB to 14GB just to accommodate all the cache files. Very annoying. Borg has 259MB of .cache, and restic has 7.4GB.

AWS Training

So over the Thanksgiving week, Udemy had a sale on video courses. Since my job search is going so slowly, I thought I’d maybe occupy the time by doing some of these courses, and I decided to start with the Amazon Web Services (AWS) certification training, starting with the AWS Certified Developer – Associate. Here are some impressions I’ve had after watching 10+ hours of course video:

  • The AWS offerings change so fast that it’s really easy for the course to fall behind reality. That might be one reason why they were selling the courses so cheap
  • AWS itself is very inconsistent in the way the UI for each individual web service is structured in the console. Some of them are very “here’s a list of options, pick one and configure it and launch it” and others are “here is a wizard to walk you through a bunch of steps that hopefully will lead to something you can launch”. It’s hard to describe exactly what I mean by that. That’s probably a result of how fast things are changing. Unfortunately, sometimes the course module was made when the console was the first way, but now it’s the second way and you basically have to watch half the video then try and remember everything he did on the first screen so you can do it on the Nth screen instead.
  • A couple of times, things in the AWS console have just failed. Sometimes it’s failed silently with no indication of why or even that it saw the “Create [blah]” button press. Other times it’s given very cryptic messages like “You have requested more instances (1) than your current instance limit of 0 allows”. (In that case, changing an obscure parameter from t2.small to t1.micro was the solution). The silent failure happened mostly in the Cloud Formation module when I was attempting to create a “Stack”, but after it appeared to fail silently (and nothing was shown in the list of stacks), and I tried to create it again, it complained that there was already a stack of that name and suddenly it’s there in the list of stacks again.
  • Other than the way the video is out of date in some place, my main complaint is that he is obviously reusing modules between the AWS Solutions Architect – Associate course and the AWS Developer – Associate course and so he’ll say “for this module, just use the S3 bucket named [blah] that you created in the last module” when you didn’t create anything of the sort. So then you have to hurriedly pause the video and create an S3 bucket and hope he didn’t expect any special permissions on it or any content in it.
  • A secondary complaint about that is that he never tells you when you’re done with a resource. I try to clean up all the S3 buckets and EC2 instance and whatever when it appears we’re done with them. I occasionally guess wrong. I wish at the end of a module he’d say “OK, we’re done with that bucket, feel free to delete it.” Sometimes he does, but mostly he doesn’t. I wonder if that’s an artifact of the fact that he’s mixing and matching modules? I’m probably over paranoid about leaving stuff around and getting charged for it, although when I started doing this course I discovered that a few years back I’d created an EC2 instance, stopped it, but never terminated it, so I guess their “free threshold” is high enough that I’m unlikely to hit it.

Long Lake Long Boat Regatta 2017 Video

I shot this one in 4K/30fps. 30 frames per second doesn’t capture the motion as well, but I think it does more justice to the amazing scenery. Also, it’s a bit easier to recognize people coming in the other direction – I had real problems with that doing the Lighthouse to Lighthouse video.

As I post this, YouTube is still working on it, so it’s not available in full resolution. If it doesn’t look really sharp, come back later.

It’s such a relief to have a GoPro on the front again. Hopefully, the new rig doesn’t allow in water the way my jury-rigged extra battery did.

Long Lake Long Boat Regatta 2017

I’ve done this race a bunch of times, and it remains one of my favorites. The Adirondacks are gorgeous this time of year, the weather is usually pretty good, and I love the new two-pronged course they’ve been using for the last couple of years.

This year, the weather was the best it’s ever been. The sun was bright, not a cloud in the sky, no wind, and temperatures that were warm but not hot at race time. Also, there was a bit of fall color in the trees. Just perfect.

Unfortunately, some people I know didn’t show. Todd, John, and Mike all stayed home in Rochester. Adirondack transplants Doug and Roger didn’t show. However, there were plenty of people I knew and several I didn’t. In unlimited kayak, as well as Jim and Royal there was a guy in an ICF sprint boat, as well as a couple in V10Ls, his was a GT and hers was an Ultra. Another guy had a Stellar SR18 but and it looks like he registered in unlimited class although he was probably eligible for touring. Eric, the guy I unsuccessfully chased last year was also in his touring class Epic 18x. They started the touring class boats in the first wave, so I wasn’t going to get a rematch with Eric, which was too bad because I’m pretty sure I’m faster this year. So, as usual, I figured I was probably going to finish about fourth or fifth in unlimited and I just hoped I would make a good showing against the strangers.

When we lined up for the second wave, there was a lot of lineup creep. Over on the right side, people were about two boat lengths in front of the line. I was hanging back anticipating that Brian would get people to move back, but when he told people to move back and they barely budged, I moved up level with them. I believe in rules, but I’m also not a sucker. I heard him give a number of seconds to the start but didn’t hear the number, so I started my GPS just in case it was a small number. After 15 seconds I heard “30 seconds”, and then 30 seconds after that, I heard “15 seconds”, so my GPS was reading 1 minute about when the gun went off.

At the start, I immediately latched onto the wake of the C-2 that started right beside me. I heard a war canoe on my left say “go right, Epic” but I don’t think he was talking to me. Maybe the video will show that (Update: it was the guy in the V10L GT). Under the bridge, the war canoes finally got up to speed and started passing us. One passed to my immediate left. Its wake threw off the C-2 and I went to the C-2’s right and passed them. I was getting a good ride from the war canoes wake, although I was working pretty hard to stay with them. There was another war canoe to my left and just slightly ahead of me – I was trying to stay far enough ahead on the first war canoes wake so I wasn’t getting thrown around by the second war canoe’s wake. Ahead of both war canoes and to the left, I could see the lead pack of Jim, Royal, and the ICF guy. I guess I’d left the two V10Ls behind, although I had no idea how far.

Things stayed like that for about a kilometer. I had initial fantasies of being pulled up to the lead pack of kayaks in this wake, but it didn’t happen like that. The second war canoe (can I just call them wcs? I’m getting tired of typing out “war canoe”) got close enough to the first one that his wake was throwing me around a bit, and I dropped off the first wc’s wake and got on the second, but I couldn’t hold him for long. Soon I was on my own, just as we were starting to pass the first wave boats.

Rounding the first turn boat, I didn’t think to look back to see where the two V10Ls were, but I didn’t see them coming in the other direction afterward so I should have known they were close. Unfortunately, I’d just discovered that my drink system wasn’t working right and I could only get a small sip after providing a lot of suction. On a hot day, that’s not ideal, and it was distracting. On the way back to the bridge, there were lots of boat wakes. I could see well up ahead that Jim and Royal were still pretty close together, but the ICF guy was tucked into the wake of the last wc and they were having to turn into every boat wake as they hit it. I was trying to figure out if I was getting closer or further away – I thought maybe they were fading or having problems with the wakes.

Just before the bridge, I decided I really needed a drink and I stopped to fiddle with my drink system. It didn’t help. Then coming under the bridge I picked up a weed on my bow. So much for catching the ICF boat. Fortunately, there were still boat wakes around so I cleared it after a bit. But maybe a kilometer after the bridge I grabbed my drink hose for one of those “high suction and low return” sips, and I felt a tap on my stern. It was the black V10L GT. No idea how long he’d been there, but obviously stopping to fiddle with my drink hose had been a big mistake. He came through, and I glommed onto his stern wake.

On his wake, sometimes I thought I was working too hard and other times I thought I was recovering. My heart rate was in the high 150s and then down into the mid 140s. I couldn’t figure it out. About a minute from the second turn, I could see that this C-4 we were catching was on a collision course. We were on the inside line to the buoy and we had the right of way, and I kept wondering why they weren’t steering away. As a matter of fact, they looked like they were on a line that would take them inside the turn buoy, not around it. We got closer and closer, and suddenly it became apparent that they had absolutely no intention of turning away – as they got so close that the people paddling on our side of their boat couldn’t get their paddles in the water without hitting us, the sternsman yelled “keep paddling” at them. Their boat gave the V10L a small push on his stern, steering him briefly to the right. I had to go the long way around them to the left. I was livid. I swore at them something fierce for deliberately interfering with another race (ie ours). Then out of breath from swearing at them, I had to sprint up to the stern wake of the V10L just in time for the turn. He turned sharper than me, so I had to sprint back up again. Then stupidly I decided I needed another sip of drink, losing me the wake and forcing a third sprint. I was pretty beat. That was about when my heart rate monitor decided to show numbers in the 80-90 range, which I knew was about 80 beats per minute too low. It’s done this before, so I figured it would work itself out and sure enough, it was soon back in the more believable 150-160 range (although I did get some more glitches later on). It was as much as I could do to hold on to the wake and try to recover a bit. But at least at the turn, I was able to see the woman in the V10L was far behind us.

I stayed in his stern wake most of the way home. With probably 2 km to go, he tried to outfox me. He slowed right down, causing me to pull out to his left side wake. I grabbed my drink hose to take a sip rather than be forced to take the lead, so he sprinted off to the right. He would have gotten away from me too, but for some reason, he veered left in front of me, and I was able to get back on. He tried a similar move with about 500 meters to go, but this time I tried to sprint side by side with him and he got away. I really didn’t have anything left for the sprint. Official results aren’t up, but I’m sure he got 30 seconds or more on me. (Update: Official results are up, he got me by 21 seconds. And I was 3:41 faster than Eric!)

And even though the conditions change and the length of the race changes depending on where they decide to anchor the turn boats, I’m extremely happy with the fact that I’m about 14 minutes faster this year than last, and my average speed went from 10.5 km/hr to 10.9 km/hr. Next year I’m going to break 11.

I took a quick look at the video from my two GoPros, and I think it’s going to be a good movie. Watch this space.

Lighthouse to Lighthouse 2017 Video

Unfortunately, I didn’t have a GoPro for the camera on the front of the boat, so I had to go back to my old Contour Roam 3. The Contour has good battery life, but unfortunately, it doesn’t do 1080p at 60fps and it doesn’t have image stabilization. As I found out in this race, it also doesn’t have a good strong mount so it wobbled around a bunch and then flopped over.

On the plus side, I have a new GoPro for the front for next weekend’s race – it arrived at home about the same time I was arriving in Norwalk.

The video is a little on the long side. What can I say? It was a long race and there were a lot of other paddlers to interact with.

By popular demand, I went back to using music. Hopefully, since Great Big Sea is basically broken up they won’t be siccing the YouTube police after me.

I’d appreciate any feedback anybody would like to offer. Are the videos too long? Should I show fewer incidents out on the course? Fewer jump cuts? Keep the cuts, but make each one shorter? And what about the music/no music question?