graphicsmagick and photobucket

I just discovered photobucket and have been uploading pictures there. I like that I can't even find (in some desultory FAQ searching) what their bandwidth or space limits are. And I really like that I can display those images from elsewhere (can't do that with flickr, need to surf to the flickr link).

I have no idea how these guys make money, but there must be money being made somehow.

When uploading images though, I've just been pushing up 2.5MB images (because my camera is set to take pictures at a very high resolution, have 1GB, will use it all ;-). Even at that high resolution I rarely ever run out of space on the memory stick. I've run out of battery 3 times more than I've run out of memory.

But it's impolite to push up images that large. I don't even dare show them on my other me because they're just too big. It took me all of five days to get up the initiative to install graphicsmagick. I used to use imagemagick and graphicsmagick is a fork of that project. Imagemagick is still around, but I'm just testing stuff out and decided to download graphicsmagick instead.

So now it's easy to increase the jpeg compression while at the same time scaling images to a lower resolution. I haven't decided yet what resolutions to use for thumbnail size and medium size, I'll look around and figure that out tomorrow. I'll ask around too, for what people think is a good "standard" thumbnail size.

mkdir medium
cd medium
for fn in ../*.jpg
do
gm convert -scale 800x600 -quality "75%" $fn `basename $fn`
done

cd ..
mkdir thumbnails
for fn in ../*.jpg
do
gm convert -scale [xw]x[yh] -quality "75%" $fn `basename $fn`
done

and just have a script to rotate left or right.

counting your money

I see that US$20 bills have RFID. And so do some euro banknotes.

I don't care that much since I don't intend to visit the U.S. or use U.S. currency for much of anything (if I do, it'll be to receive payment and convert it immediately). And I doubt if I'll ever have euros in any interesting amounts.

What I do wonder is, is it going to be possible for someone walking (or someone with an RFID long range gun) to count how much money is in someone's wallet? Or if not how much exactly, at least count the number of bills there?

Do all the bills radiate the same signal? If they do, then what help would that be with respect to counterfeiting? It shouldn't be hard to create a counterfeit that would radiate the same signal (it might be too expensive today, but I bet 10 years from now it'll cost nothing). And if the bills don't all radiate the same signal (e.g., they radiate something correlated to the bill's denomination and serial number), then it might be possible to deduce how much money is in one's wallet. If it's encrypted, that won't matter much. Banks will need to know how to decrypt it, and once it's in a few hundred bank branches, one of those decrypting boxes is going to get stolen (might even get stolen during a bank robbery) and reverse engineered.

Of course, though, it wouldn't be bad, in the eyes of any government, if people were to shy away from cash transactions and go with electronic transactions instead. Taxes are easier to collect that way.

I get pissed off when I look at old code like this:

1. No vertical whitespace between the pg_exec and the if.
Alright, the developer was just ignorant and was following the
bad standards in his environment.

$rs=pg_exec($common2_conn,$query);
if (pg_numrows($rs)==0) {
} else { # if (pg_numrows($rs)==0) {

2. WHAT IS THAT EMPTY BLOCK FOR? DID NO ONE TEACH THIS PROGRAMMER
(dammit, I want to change my title because of idiots like this).

DID NO ONE TEACH YOU != OR <> (yech, but at least it does
exist in PHP).

That code is wrogn anyway, pg_numrows will return -1 on error
(e.g., network goes down, db server reboots, db server daemon
dies, connection disconnected because the server restarts because
someone did a kill -9 on a client).

3. I'm sorry. K&R indention is dumb. Yes, it saves lines, but
we're not in the age of teletypes and 300baud modems anymore,
there's no great need to save screen refreshes. vertical space
is valuable, sure, but clarity and maintainability is more
important.

if (pg_numrows($rs)>0)
{
--- read the rows and do something with them
}

and suddenly there's no else to clutter things up.

There are a lot more stupidities in that program, but this scheisse makes me tired. If that developer were still around I'd fire his ass. Yes, I'm looking at you, A.M.

gmail problem

It's close to 8AM in the Philippines now, so around midnight UTC. We just had a short brownout. It's probably because of summer airconditioning power demand. Last night we had a short (less than 10 minutes) brownout too. This is better than when I came back to the Philippines in 1993 or so when there were 8-12 hour brownouts. Summer nights were terrible then.

I was in the middle of drafting an important (timmy's baptism requirements) email on gmail when the power cut out. Now that it's back (again, less than 10 minutes), I can't get to gmail. google.com and here, blogger, are fine. Gmail just isn't replying though. It isn't a DNS issue, the domain resolves. It just isn't replying yet. Ah well, i'll send myself the incomplete draft from one of my other web-based free email addresses.

highly effective study-avoidance technique

Stacking Pennies, Building bridges

pgsql COPY doesn't fire rules

I've got a very large database at $DAYJOB. Two and a half years worth of data takes up around 500GB. The largest table is around 80GB. I use postgresql, and with the right indexes, performance is very good. I've sort of become a specialist at database performance tuning since I've never had the benefit of working with anyone else who could avoid sequential scans on large tables.

So raw query or insert/update/delete performance isn't a problem. However, database management *is* a problem. On rare occasions, I need to vacuum a table, or many tables. That's very hard to do when the table is so large (I stopped the vacuum after 2 hours). It's the same thing when I have to backup a table. pg_dump -t takes a very long time to scan a very large table.

So I decided that I needed to partition the larger tables. Perhaps into per-month tables. My program to load data into the database uses COPY because that's the fastest way to load data into a table. Unfortunately, COPY doesn't fire rules or triggers, so the ON INSERT DO INSTEAD rules didn't fire and the data was still going into the base (parent) tables.

Just yesterday I was scanning through the pgsql-general list and saw Tom Lane say that COPY doesn't fire rules. It didn't strike me as significant at the time since I was thinking of something else. Tonight though, as I was working on it again and, again, finding that rows were going into the base table (or the whole transaction failing, since I decided to add a constraint on the parent so that no rows could insert into it at all), I finally put it together and knew why COPY wasn't doing what I wanted.

It was a simple thing to replace the COPY code with insert statements. Inserts will run more slowly now, (although I might get some performance gain by upgrading to 8.2 and using the new multi-row insert syntax) but the database will be much more maintainable. It'll probably run faster too since most queries will be against the most recent months, so indexes and whole tables for the most recent months will be more likely to fit in the OS cache/buffers and postgres' shared memory.

In any case, I'll keep the old non-partitioned database and the new partitioned database, inserting data into both. When I prove that the new partitioned database is stable and faster, I'll retire the old non-partitioned database.

My last concern is aesthetic. With new rules being added every month (and 2 or more years of data being loaded) a \d on a base table will yield a very ugly list of ON INSERT DO INSTEAD RULES. There's nothing to be done about that though, unless maybe I move the parent tables into a schema and replace them in the public schema with views :-).

gmailfs slow to connect - fix

I installed gmailfs using apt-get (on my Ubuntu Edgy laptop) because I thought it'd be interesting to see what file format gmail stores emails in. And I thought I'd download all my spam too, to see if my bogofilter classifies them correctly, and for those that it doesn't, train bogofilter with.

I thought there was something wrogn because it just wasn't mounting. A look at gmailfs.log shows:

04/12/07 19:28:15 ERROR gmailfs did not connect in less than 12 seconds, aborting...
04/12/07 19:28:15 WARNING Child process 24621 received SIGHUP, exiting...
04/12/07 19:28:15 INFO Successfully reaped child 24621

Clearly it was timing out. I tried a few more times and finally edited /usr/bin/mount.gmailfs. There's a define right there for the GMAILFS_MOUNTING_MAX_DELAY so I just set that to something higher (64, from the default of 12). Running mount.gmailfs under time shows that it's taking around 30s to complete.

Ah, except the filesystem looks empty. I can't see my email there. Copying files there makes email appear in my gmail Inbox, but I can't get my spam. Ah well. It's still cool, but I'm not sure I can see what use it's going to be to me :).

Who would win?

Somewhere at around 1/6th of the way down this Slashdot article Neal Stephenson describes a battle between himself and William Gibson.

Hi-freaking-larious.

In the first instance,

You don't have to settle for mere idle speculation. Let me tell you how it came out on the three occasions when we did fight.

The first time was a year or two after SNOW CRASH came out. I was doing a reading/signing at White Dwarf Books in Vancouver. Gibson stopped by to say hello and extended his hand as if to shake. But I remembered something Bruce Sterling had told me. For, at the time, Sterling and I had formed a pact to fight Gibson. Gibson had been regrown in a vat from scraps of DNA after Sterling had crashed an LNG tanker into Gibson's Stealth pleasure barge in the Straits of Juan de Fuca. During the regeneration process, telescoping Carbonite stilettos had been incorporated into Gibson's arms. Remembering this in the nick of time, I grabbed the signing table and flipped it up between us. Of course the Carbonite stilettos pierced it as if it were cork board, but this spoiled his aim long enough for me to whip my wakizashi out from between my shoulder blades and swing at his head. He deflected the blow with a force blast that sprained my wrist. The falling table knocked over a space heater and set fire to the store. Everyone else fled. Gibson and I dueled among blazing stacks of books for a while. Slowly I gained the upper hand, for, on defense, his Praying Mantis style was no match for my Flying Cloud technique. But I lost him behind a cloud of smoke. Then I had to get out of the place. The streets were crowded with his black-suited minions and I had to turn into a swarm of locusts and fly back to Seattle.

and, yes, there's quite a lot more ;-)

I haven't read all of Stephenson, although I read snowcrash (+5) and cryptonomicon (-3) several times. I might eventually read all of Stephenson's work, but I'm in no hurry. Jerry gave up on quicksilver, I think, at some point, with some disparaging comments on pace, continuity and point. Which is why there's no hurry. I don't agree with some things Jerry says. For instance, he says that U.S. nuclear waste should be dropped down into the subduction zone in the Surigao deep, without compensation to the philippine government, or even coordination or permission. That's more of the same thinking which brought about the Iraq tragedy (where U.S. intervention has killed more than half a million people and led to an exodus of the most productive parts of the Iraqi population). I'm aware that he thinks Iraq is a mistake, but that's just because the U.S. can't win in Iraq. In the Philippines he calculates that the U.S. can bully the government without much of a substantive cost, so he's willing to drop the nuclear waste in philippine territory without payment.

Well, Hell no. if the U.S. wants to drop nuclear waste here, LET THEM PAY FOR IT. And ask for the privilege. It may be completely safe, but we don't care. PAY, or keep the poison on your own land.

Crap, this post got away from me and turned into an anti-US screed again. Ah well, never mind. The U.S. certainly deserves it.

Greylisting -- bad

At Marco.org there's a short article discussing why greylisting is a bad thing.

I've always thought it wasn't much more than a stopgap. Marco says that his beef is the fact that email is no longer instantaneous because of the delays that SMTP graylisting injects. I agree that's important, but it's not that important to Me since I never send anything urgent in email (I'm aware of graylisting and delays) and I let my mail sit and ferment a few hours, usually, before reading it. Immediacy is important to other people, but it's not a big deal with me.

The reason it's a stopgap is that eventually botnet spammer software and viruses will be modified to retry. When that happens, we'll be back where we started, with spam classifiers.

Likely, then, there'll be some new mode of speeding things up, perhaps by putting fast but not so good classifiers in the front of the queue, passing through anything that clearly isn't spam, and passing the possibles to the heavyweight classifiers like spamassassin (which I can't stand because it's so heavy, I use bogofilter instead, it's simple and I don't mind training it from the command line, but that really doesn't scale and there should be an easy way to train so that non-geeks can use it without trembling).

Advice for young programmers

Jeremy Allison has good advice to himself, if he could travel back in time.

• If it's not what you love, don't do it


• Learn the architecture of the machine


• Reputation is important


• Proprietary environments are a trap


• The network really *is* the computer


• The community is more important than your employer

On an unrelated note, and a bit late for April 1, I was reading about how J Striegel fails the Turing Test. That's amusing in itself, but I was fantasizing about modifying gaim so that whenever it sees "cutie" or "sexy" it replaces the words with "moron" or similar. After all, what kind of people have cutie or sexy in their logins? This would be for my own amusement only. Hmmm, might be interesting to modify evolution the same way. If I receive mail from anyone with cutie or sexy in their email address, it's not like I'd actually want to reply to them, after all. And if I did, it would be a good thing if my mailer were to make any such attempts fail anyway.

dropping duplicate email in evolution

I recently had to reload all my mail from gmail again (I have an always-on computer that runs postfix and stores the email in maildir format, fetchmail gets the email from gmail and delivers locally, I then check my mail at that box regularly).

I had to reload because I decided to run a script that I have that removes duplicate emails from mboxes (evolution uses mbox, unfortunately). Due to a brain fart and Ctrl-C, and because the script isn't very smart, I lost some mboxes. It was simplest to just reload everything from gmail, even if it takes three days or so to get everything from gmail.

My scripts used formail and wrote temporary files which, once complete, were written back to the main mbox, with duplicates removed.

I'm switching to a safer alternative. In Evolution, I've created a filter thus:

Filter action is Pipe to Program
The program is /usr/bin/formail -D 9999999 /home/tiger/.evolution/dup-idcache
Action is move to some folder (e.g., ZZZZDups)

I've tested it with a mailbox that was horrendously bloated due to duplicate emails, and it found the dups, kept one email in the original mailbox and moved the duplicates to the designated dup folder). I double checked the emails (well, sampled, since there were too many) and ever dup found really was a dup (same email was in the original mailbox).

This is a much safer way to find dups, although I'll need to Ctrl-E the Inbox regularly, it's not that efficient (runs formail for every incoming email) and evolution doesn't much like the program specification (when I open filters, it shows an error with the filter with spaces converted to %20, etc., so URL-encoded). It's very convenient though and I'll use this until I can figure out a better way (probably hack up a script that hides the parameters and returns what formail returns).

The same technique works pretty well with bogofilter. I'd wondered how to integrate bogofilter into evolution. I'm OK with just finding spam and running bogofilter -s on it directly, so I don't need any fancy interface. For now (well, for my first test run earlier), bogofilter successfully found spam and moved them to a designated folder. I'll see if it catches more spam in the future.

I don't really get a lot of spam anymore. Gmail catches most of what goes there, and the mailserver at work has reasonably good spam filters. But sometimes some spam does get through. I'll see if bogofilter does a good job with those.

Wrong place, and how to transfer

Alright, The link to bacon wrapped inedible crepe was supposed to be on my other me. I'm going to waste some time trying to see if there's a quick way to move it over without having to copy and paste. And if there isn't, well, it may or may not move over, depending on how lazy I feel just then :-).

No Google Browser Sync

I used to use Google Browser Sync. I resisted it for a while, although Ian Sison at Q Software Research kept telling me how great it was. I resisted because I didn't think it would be useful (in the same way I didn't buy a celphone for years, and when I'd got one, I wouldn't send SMS messages because all the replies I got were illiterate). I also resisted using it because I was concerned about security. But BrowserSync won't sync passwords if you don't want it to.

When I finally installed it, it was just as with SMS. Eventually I was using it quite a lot (I also deal with fewer illiterates these days). I've given up on it though. I've removed it from all the firefox profiles I use. The main reason is that browser sync goes insane if the computer goes off without the browser being shut down nicely. Now, that doesn't happen a lot with me since I use linux, but I do occasionally use a computer that isn't on an AVR and when the power trips (gonna have a lot more of those for the next 3-4 months, it's summer now in the Philippines), BrowserSync loses its memory and tries to sync everything from the network when firefox comes back up. That takes too damn long, even on broadband. I don't want to wait 5 minutes or more for the browser to become usable again after a power outage. Once I went to sleep and in the morning it was still trying to sync. Something probably went wrogn with the bandwidth or the connection, but still...

In any case, i found myself cancelling the sync whenever this would happen. I was doing this enough that I finally realized there's no benefit to BrowserSync anymore. In any case, the only reason I really liked BrowserSync was, I could open a whole lot of windows in one browser, close it, go home (or to work, or to QSR, or wherever) start a browser, and have BrowserSync open the same windows/URLs. I was using it as a multi-client session synchronizer. After thinking about it though, I concluded that using it that way is anti-productive. Most of the time, when I'm reading articles from digg or reddit (or my feeds at bloglines) I'm just wasting time anyway. There's far too much to read, I spend too much time on it and not enough time thinking or coding.

Dropping BrowserSync increased my productivity by a bit because I didn't have to wait for syncs after power outages, and I didn't have to read non-work related articles at work. Of course I still read a lot at work. But it's more focused, it's more work related, and it's not synchronized all over the place :-).

bacon-wrapped, cheese-filled, battered and fried hot dogs!!!

If syphilis were a product this is what it would look like

But you'll have to follow the link and read the article. Or, hmmm, click here for the image. I would post an IMG link there, but again, that'd be rude.

Gmail Loader

Wow, I see that Mark Lyon has developed a Gmail Loader, which can take mail in standalone mail clients and load them into gmail. That's cool. I'd use it, but I've been using gmail so long that I don't even remember some of my old email addresses :-). I certainly don't have those mboxes around anymore.

It'll be very useful though, for when I convince everyone around me to upload their mail to gmail for archiving/backup purposes :-).

I should test it though to see what happens with executable attachments. Currently Gmail blocks executable attachments and even looks inside zip and .tar/.tar.gz files to see if they contain executables, so likely the same happens with uploaded email.

SQL Injection cheat sheet

SQL Injection cheatsheet is over there

Looks good, so far. Haven't read it all yet. I'm learning a lot though. There are a lot of tricks there that I might have seen before but didn't pay enough attention to.

Apple Not for me

A friend of mine bought himself a MacBook. That's a lovely computer and it's pretty rugged too. It survived having beer fizz up and explode all over the keyboard. Another friend is considering getting the same (MacBook, not the beer).

I've never been one to buy brands, so, while they'll certainly be very happy with their purchase(s), it's not a choice I would have made. At the price the MacBook is going for (close to $2000 in the Philippines), I could get two reasonably fast (but not dual-core) Windows/Linux capable laptops. And since I can get Winbook laptops (or Thinkpad, or HP) from my brother in the U.S., for even less than that, I could get three previously but lighly owned laptops for the price of one MacBook :-). Since I use Linux (it does all I need a computer to do) and didn't much like the Mac interface when I tested it out, there's nothing there to tempt me to go Mac. I might be slowly seduced by a Mac's smooth design (hardware and software) if I had one at work, or if I could borrow one and use it exclusively for a month :-). I doubt, though, if that seductive smoothness would get me to overlook the price difference though. I would enjoy the MacBook, but then forget it and switch back to Linux since $1000 is (to me) much more valuable and useful than whatever psychological benefits I would get from using a MacBook.

On the other hand, I'll work very hard to justify the cost, if I get one for free :-)

in-place upgrade successful but bumpy

I did the in-place upgrade to Edgy and it was mostly successful. I got stuck on one thing though. It seems to be a common bug with upgrade-in-place from Dapper to Edgy. A quick google search finds this:

https://launchpad.net/ubuntu/+source/apt/+bug/64615
http://www.nabble.com/Removing-a-Broken-Package-(courier-authdaemon)-t3331317.html

I only just found those links after I'd fixed the thing though. The fixes suggested there are too scary for me since I'm not an experienced debian user. Instead, I followed the hack that mhulboj suggested at this blog post

He noticed that the courier-authdaemon un-install script was trying to first stop the daemon and that it was getting stuck on the fact that /usr/sbin/courierlogger didn't exist anymore (probably a package dependency thing, something else removed courierlogger, could be fixed in the package by first checking if courierlogger even exists before removing it).

Anyway, the fix was:

sudo touch /usr/sbin/courierlogger
sudo chmod a+rx /usr/sbin/courierlogger

and continue with

dpkg --configure -a
apt-get remove courier-authdaemon

which successfully removed the package. I'm now continuing with the update in place (re-downloading a lot of packages since I also misunderstood something and did an aptitude autoclean :-).

Feeling Daring, in place upgrade to Edgy

I'm feeling daring and am going to do the in place upgrade to Edgy from Dapper.

Oh, there's a better link
here and running update-manager with -c and -d brings up the help page on upgrading to edgy

ten coolest numbers

There's a page that discusses ten cool numbers. Naturally, some of that judgement is subjective. I think the page itself is pretty cool.

I would write about e to the i pi and -1, except, I have no idea how to formulate that in HTML :-) and I'm too proud to read the HTML of CoolNumbers :-)

I like a lot of those numbers. I'm going to read all of that and understand perhaps 20% of it :-). hahahaha. I'm not enough of a number theory geek to know the math. I just know enough to enjoy what I see. Including the parts I just don't grok.