Category: Geekery

Oh yeah, /tmp is *temporary*

I was storing some files that were semi-important to the project I’m working on in /tmp. I knew that there is a process on some Unix computers that cleans out the stuff in /tmp either on boot or on a schedule, but I didn’t know if it did that on my Mac. So while I’d sort of had a flag in the back on my head to move that to somewhere less fragile, I never got around to it. And I got to working on another part of the project for a few days and forgot about them. And in the mean time, the files haven’t been touched, and I’ve installed an OS update and rebooted. And now I go back, and they’re gone. “Oh yeah”, I think, “/tmp is temporary”. So then I look to see if Time Machine has a backup, and of course Time Machine excludes /tmp because, oh yeah, /tmp is *temporary*.

I can recreate the files, but it’s a waste of a few hours. This time I’m going to recreate them in ~/data/.

This week’s interesting discoveries about Nutch

I’ve been working a lot with Nutch, the open source web crawler and indexer, and the first thing I found was that it was downloading web pages every day, instead of sending the “If-Modified-Since” header and only downloading ones that changed. Ok, I thought, I’ll fix that – since the information I want isn’t in the “datum.getModificationDate()”, I’ll use “datum.getFetchDate()”.

Second interesting discovery: Nutch then doesn’t index pages that returned 302 (not changed), and since the index merging code doesn’t seem to work, I can’t these pages that I cleverly managed to avoid downloading. Ok, I’ll fix IndexMapReduce and delete the code with the comment that says “// don’t index unmodified (empty) pages”, and resist the urge to send a cock-punch-over-ip to whoever wrote that comment for not realizing that “unmodified” does not mean “empty” by any stretch of the imagination.

Third interesting discovery: It turns out that some bright spark decided that when you’re crawling a page that’s never been loaded before, “datum.getFetchDate()” gets the current time, instead of any useful indication that it’s never been fetched before. So scratch my first fix, and go looking for why datum.getModifiedDate() isn’t set. And discover that it appears that datum.setModifiedDate() is never called except by code trying to force things to be recrawled. Yes, instead of forcing a new crawl by modifying the locally generated “fetch date”, they fuck around with the “modified date”, which is supposed to come originally from the server. My opinion of the quality of this crawler code is rapidly going down hill. But my patch to set the modification date according to the page’s metadata appears to be working. Sort of.

Fourth discovery, and one I can’t blame on Nutch: My Rochester Flying Club pages use shtml (Server Parsed HTML) so that I could include a standard header and navigation bar in each page. I could have used a perl script to automatically insert the header into the pages and regenerate them whenever anything changed, but this seemed a lot easier at the time. But one consequence that I’d never noticed before – the server doesn’t send a “Modification-Date” in the header meta data, so evidently these pages are never cached by any browser or crawler. Ooops.

Working Again!

I probably should have mentioned this earlier, but I’ve held back because

  1. It doesn’t feel completely real in some ways and
  2. I’m under NDA and don’t want to say too much

It’s a Java job, working with Nutch, Lucene and Solr. It’s a one person start-up, and the owner has expectations that she’ll need a Chief Technical Officer soon, so this contract I’m doing is sort of an audition for the CTO job. She has an existing code base that she paid a consulting company to write, but she has a long list of things she needs done to it, and I’m making my own list as I go along. (Unit tests and fixing the things FindBugs finds tops my list, but so is fixing the horribly manual deployment process.) The current contract is fixed price, but I should be able to do it quickly enough to make it a decent hourly wage. I’m currently working on my own at home, but she’s going to have a two person office in a week or so at a local technology incubator. I told her that as long as it has wifi, a big whiteboard and access to a fridge, I’d be happy. As an added bonus, the technology incubator is pretty close to where Vicki works, so we’ll be able to meet for lunch or go to the gym together.

Because I’m working on my own, but expecting to work with others in the future, the first thing I did with her existing source code was to put it into “git”. (Maybe it’s because git isn’t as obtrusive as something like ClearCase where you have to check things out, I haven’t found any decent Eclipse plug-ins for git.) As well as learning git, nutch, lucene and solr (not to mention the technologies they depend on, like Hadoop), I’m also learning a bit about making a build environment in ant.

All in all, it’s fun and interesting work, and whether the company fizzles out or grows to a hundred employees, it’s going to be a great learning and growth opportunity for me. And hell, even if it sucked as much as my last job (which it doesn’t) it would be better than unemployment.

How not to drum up business

There is a business here in Rochester that needs a lesson how to do business. I’m not going to give them the exposure (or Google rank) of putting their name here, but their name sounds a little like “Cock fire”. The business they are in is something that is actually of interest to me, something I currently use, and something that I recently solicited quotes from numerous companies in the business by going to a site that collects your requirements and sends them to registered providers. It’s also a business that members of a Linux Users Group, such as our own Linux Users Group of Rochester (LUGOR) might be more likely than the general public to want to do business with.

But “Cock fire”, instead of waiting for requests for quotes, or introducing themselves to the LUGOR group as a peer or contributor, instead decided to somehow mine our mailing list for email addresses, and then individually spammed the members of the list. When I got mine, I actually thought it was somehow related to my earlier request for quotes, until I realized that they’d sent it to both of the email addresses I’ve subscribed to the mailing list, not just the one I’d used in the RFQ. And then somebody else on the list mentioned that they’d gotten this spam to an address that they *only* use for the LUGOR list, and several other members piped up that they’d also gotten spammed, so we figured out what they’d done.

So well done, “Cock fire”. In spite of the fact that your product is actually $10 a month cheaper than what I currently pay your competitor, I’m not going to switch my existing use over, and neither am I going to recommend you to my current employer. Reap what you sow, assholes.

Update: I got a response from the email I sent them.

On behalf of [Cock fire] I would like to formally apologize for the e-mail marketing to your group. I was given 2 lists of e-mails from a Rochester Linux guru that said their group would be interested in Rochester based services. From the number of negative responses I have gotten back this was a mistake.

We have deleted all LUGOR e-mails and will not be in future communication. Please convey our apology to the group.

So it wasn’t his fault that they decided to spam, it was the fault of somebody who tempted him into it by giving him a list of email addresses. Oh, then that makes it all ok then? I don’t think so.

Do me a favour?

Update: Steve Robbins has modified his widget to use JSON, and I’ve gone back to using it because it works right at any text size.

The Stack Overflow team is sending me emails saying that my use of the Robbins Stack Overflow widget on my blog is putting an “unacceptable” load on their huge 48 Gb RAM, 8 processor box. So I’ve switched over to their preferred solution, which is an iframe containing their own “flair” page. The problem with the iframe option is that it requires me to tell my system exactly how many pixels high and wide it is – and when I change my text size, it starts putting scroll bars on it, and it looks like ass.

Can you please look at my Stack Overflow badge on the right side of my blog, and leave me a comment telling me yes or no if it has scroll bars for you? If you know, tell me what OS/browser and font size you’re using.