So you want to be a consultant

I saw So you want to be a consultant? long ago. I'm looking at it again since someone on a mailing list pointed at it.

I'm learning again. Some of those lessons I haven't internalized yet. It'll take a while, a lot of things do. I'll get there yet :).

rsync and compressed files

I did some testing and I find that generally, it is better to rsync uncompressed files rather than the corresponding compressed files or archives. at any rate, tar.gz archives are bad for rsync. tar files are OK.

1. i took a directory of source code and test data, around 9MB.
   
   
2. copied it to a remote box
   
   
3. tar cvzf on both sides to one file and also tar cvf to another file.
   
   
4. on the source box, edit one source file, insert only one line.
   
   
5. tar cvzf and tar cvf on the source box. the source box should have sources, tar and .tgz which vary in only one line in only one internal file.
   
   
6. rsync of the source gives a speedup of 450 (14K sent, 94 received), rsync of the tar file gives a speedup of 85000+ (78 bytes received, 20 bytes sent), rsync of the .tgz gives a speedup of 1.48, (2.4MB sent, 12K or so received).
   

so rsync of a tar file is best (because only one file needs to be analyzed to see where the differences are). rsync of a compressed file (at any rate of .tgz, but probably of any compressor) is bad. not sure why, but i wouldn't be surprised if the compressed representation of a lot of data depends on what has come before, and there may be other effects like that which
confound the difference finder since too much is found to be different.

Corporate peer-to-peer

is almost always a mistake. at any rate, it is anywhere where bandwidth is expensive.

I was just talking to someone at a company I do some consulting with. I was working remotely, and the link was ridiculously slow. Ping times were at around 1 second, and sometimes 1.5 seconds. I could still work (i've got some techniques involving rsync, for very bandwidth starved links, and i just type ahead), but I could work better if the bandwidth weren't so slow.


So I talked about the serious need in corporations to take steps to block p2p, and then, since it's impossible to block it completely, probably, to do as much as it can to monitor p2p and then to have a policy about p2p use (probably that it should not be allowed at all, and that it would be blocked and monitored, and violation would affect performance reviews).

That may sound draconian, but it's necessary.

1. bandwidth costs money. even if it were cheap, if peer to peer didn't soak bandwidth the company wouldn't need that much bandwidth and could contract for less, thus paying less every month. That's money that goes straight to the bottom line.


2. the company i'm using for my example runs its own publicly accessible mail and web servers and therefore their bandwidth is all fixed IP. That's a bit of a bug on the part of IT management, they could go with 80% dynamic IP bandwidth and then 20% fixed IP for mail and web. They would save quite a bit of money right there since fixed IP bandwidth carries a very high premium in the philippines. they would save more money just by buying dynamic bandwidth for staff time-wasting surfing and buying less fixed IP bandwidth for those services that require the bandwidth.


3. in a litigious world, it's for the company's good that peer to peer is blocked and violations monitored and punished. The same company has received a warning letter from a RIAA/MPAA related agency, apparently someone had left their bittorrent client on and had been downloading and serving enough files that they attracted someone's notice.

Naturally, this sort of thinking won't sit well with employees. But frankly, I don't think it matters. The staff aren't being monitored for wasteful surfing (of which, perhaps half of all surfing at the office is wasteful and not work related or only very peripherally work related), so their surfing for entertainment is a free benefit of employment. It's only fair that those online activities which might be damaging to the company be disabled so that other online activities of neutral or only mildly negative value may be allowed.

Normal distribution

I had to write a program that needed to simulate random data that was normally distributed (bell curve distribution).

Fortunately, I was able to grab a copy of Numerical Recipes in C" and found some code on p. 217. The code produces a normal distribution in the range -1.0 to 1.0 with a standard deviation of 1.

My program was PHP, so i translated it from C (K&R!, boy that's old :)

The PHP source is here

Unfortunately, I'm not a mathematician and I'll need to find ways to adapt that code or find some other code where I can adjust kurtosis, standard deviation, etc.

I'd look in Knuth's Seminumerical algorithms, but it might not be there. And anyway, my copy is in Mindanao. I won't be able to refer to that until I go in May.

Email from idiots is spam

I use gmail, and the spam marking feature, and the fact that it's so easy to use, is very nice.

Every once in a while I see vacation messages posted to mailing lists. Every single one of those I mark as spam in gmail. Partly I do that because people who don't know enough to set selective filters on their vacation messages are too dumb to listen to.

The gmail filter will learn from vacation messages which words score high as spam and perhaps future vacation messages will be marked spam and I'll see less of them. Also, the authors may start to score higher as spammers. That's a good thing too, for me. I'll see less of their mail since their mail will automatically go to the spam mailbox, and when I go in there to confirm which emails are spam, I get a chance to despam those emails which are important.

I don't think I've seen gmail do that yet though (filter mainly on the sender's email address), I've seen Bob Reyes' spams about his hosting service end up in the spam mailbox, but that's just because the email was spam, not because the emails were from him.

Good Article

Why your code sucks

Reading Multiple GMail accounts using the same computer user

I've had a bit of a hack in place for reading my two gmail accounts. I would open two copies of firefox (or mozilla), but using two different logins. The second would do an su and run the other firefox copy. E.g., I'd run firefox as user tiger, and then run firefox as user tiger1. I could then read my bopolissimus@gmail.com account as tiger, and the other firefox running as tiger1 would read bopolissimus.lists@gmail.com.

It worked out that way (couldn't read bopolissimus.lists@gmail.com as tiger) since when I'd surf to www.gmail.com, the browser would notice that it already had a cookie for there and would identify itself with that cookie.

I just noticed that it *is* possible to open two different gmail accounts as the same (linux logged in) user. Firefox has a -P parameter (for profile). The solution is:

1. start firefox with the -P parameter. This allows you to create,edit,delete profiles.
2. create two profiles (or one profile for the other account, use the default profile
for the first account).
3. when starting firefox, specify which profile to use with -P

This isn't as big a deal as it might seem, since the normal thing to do would
be to create a launcher (shortcut) on the desktop. For the main account,
set the program to run as "firefox -P default", and for the other account,
set it to "firefox -P [whatever_the_other_profile_is]".

Netscape has always had that feature, as does mozilla, I think. I just never
used profiles before. Now it comes in handy though.

Misc Articles

I read a lot. mostly online, but dead tree too. I often find
something worth passing on, but I rarely do. No time, lazy,
not online. http://www.jerrypournelle.com is a good source
for good articles and discussion.

Here's one. Not my country, don't care that much. One of
the things that I didn't like about life in the US though.

A Nation of Wimps

Focus on the first derivative

Things to do/learn

Things I need to do or learn when I find the time (not anytime soon, but maybe one thing at a time on weekends when current project is done).

1. Switch to Subversion from CVS (might do this soon).
2. Learn how to set up drbd (vmware or UML for testing).
3. Learn Postgresql Slony replication.
4. Set up bugzilla and mantisbt (the one just for fun, the other, probably for use).
5. Test setting up heartbeat and virtual servers (#2, maybe watch ian or cedric set this up?
More fun to figure it out on my own, but takes a lot longer :)
6. Test setting up ypserv and ypbind
7. openldap na rin for PAM and email and other auth...
8. Java this, Java that, lots of new stuff to look at there.
9. Test MySQL transactions (InnoDB) and subselects (latest version). Of course MySQL
really isn't usable yet for serious databases until something like plpgsql becomes
available for it. But it's getting there.

hahay, kailan pa ba magkaka-oras para sa lahat na iyan.

CVS on USB Flashdisk

I recently got a sandisk 128MB USB flash disk and I've decided to use it as my CVS repository (or arch, I'm looking into that, or svn, I'll look into that after I look at arch). I shuttle between multiple sites, and they're not all on the internet. Some of them are completely firewalled off from the world.

I don't share the repository with anyone, so there's no problem keeping it in usb. Of course that wouldn't work well for a team. But for me, well, I just like having all my code, editing history, and releases in a repository, so a USB disk is fine as a repository.

The only thing I'm worried about is the write cycle. Need to research on that. Some flash memories are supposed to have only a limited number of write cycles and beyond that limit it's not possible to write to the device anymore.

-- flash -- I decided on svn. I didn't like the arch interface (actually, I couldn't understand it, it's probably easy, I just didn't give it enough of a chance). I really like svn though. Performance isn't that great, but then my code base isn't that huge and svn has features that have been missing in cvs forever (file and directory rename, move, etc).

Porting Blogs

I've got another blog on another server but I'll be moving those posts here. Did one already, I'll be doing the rest slowly. Maybe one or two posts a day. I won't be doing the comments though. Not motivated enough for that. Hmmm, I should go over there and get the mysql database so I've got it here, in case the server goes away.

Consulting

I recently moved into consulting, after a few comfortable and enjoyable years developing software for an internet service provider.

I'm enjoying it a lot. It's not the money (I've seen more of that than I used to make, but I've also moved to where things are more expensive, so things even out). Rather, it's the fact that I work with many different technologies, so I get to learn more as I work.

Lately, I've been learning postgresql functions and triggers in plpgsql. I went back to something I used to do all the time, code generation. I wrote a multi-threaded program in C/C++ and learned the benefits of STL at the same time. Of course I already knew what the STL could give me, I just hadn't actually experienced the benefits yet.

After using STL for a few days, I dropped my handcrafted string and container classes.

Doxygen is great and I'll be getting into PHP+SOAP (it should really be java+SOAP or something similar, but I need quick wins, like results within a week, and java is too complicated for that kind of thing, maybe on my next project I'll work with java).

One thing about consulting is, there's always something different happening, so burnout due to boredom won't happen. Although burnout due to stress is always a possibility :). But then I can usually set my own schedule, so that's *less* likely than it seems.

Gmail Retroactive application

I've wanted to apply filters retroactively in Gmail (i.e., if I've got 10MB of mail in GMail and now I want to organize it into labels, I create the labels, create the filters to auto-archive and set labels, but how do I apply the same filter to old mail?).

There didn't seem to be an obvious way to do that. But then that's because I was thinking about it as "applying filters". So naturally I was looking for it in the Filter stuff (create, edit, delete, test, etc).

As it happens, there'ss a simple way to do retroactive filters in GMail. Just search for the relevant email using the search function. It'ss not going to be exactly the same as filtering, UNLESS one uses the "Show search options" link. That brings up a dialog box similar to create a filter, except it works only for one search.

After the relevant emails have been found, it's a simple matter to select them all and apply a label and (i like to do this so they go away from the inbox but are still in the label/folders) archive them. It's not quite as easy as it might be, but it works. Now I just need to remember that that's how it's done.

Spam Classification Results from an informal test

I'd been noticing that SpamAssassin, at a threshold of 4.5 and even with its built-in Bayesian scoring was just not performing as well as Bogofilter, which ONLY has Bayesian scoring (but of course, I tweaked the spam and ham cutoffs and other parameters around 3 months ago). I decided to do an informal test.

Procedure:

0. I used my already trained bogofilter and sa-learn setups. For about a month now, I've
been taking spam that bogofilter found but that spamassassin did not determine to be
spam, and I've been feeding them to sa-learn in hopes that spamassassin would eventually
score them as spam since spamassassin would learn through its bayesian test about the
spam that it had not found before. However, even after a month of this training, I see
the result documented below (i.e., spamassassin's bayesian component doesn't seem to
learn very well).

1. Get Mboxes from various sources. The Mboxes include spam and ham

2. Run the email through spamassassin and bogofilter. The bogofilter wordlist does not
include any spamassassin markup because all email is run through a filter that removes
such markup (and performs other cleanup, e.g., removing all lines with too many
consecutive characters without whitespace, the main effect of this is to throw away attachments
that are encoded via MIME, BASE-64 or other encoding schemes).

3. Have evolution group the email into ham, mail that only bogofilter thought was spam,
mail that only spamassassin thought was spam, and mail that both thought was spam.

4. Eyeball all that email (very quickly, mainly looking at from and subject lines, and then
viewing the body of suspicious email).

At the end of all that, I see the following numbers:

On the positive side for both:

• 1339 spam correctly classified by bogofilter


• 1337 spam correctly classified by both bogofilter and spamassassin


• 697 non-spam correctly classified by both bogofilter and spamassassin


• 0 false negatives by either bogofilter or spamassassin


• 0 false positives misclassified by bogofilter

On the minus side:

• 104 bogofilter false-negatives (spam that bogofilter didn‘t classify, all these false negatives were also misclassified as negatives by spamassassin)


• 90 false positives misclassified by spamassassin only (bogofilter correctly said they were not spam)

SpamAssassin has too high a false positive rate for me. Any false positives are a major problem since, with so much spam overwhelming the nonspam, false positives are very likely to hide in the spam noise and thus get lost. And while the rate here is very low in terms of probability, that is still too high for me.

False negatives aren't such a big deal since basically, the amount of spam is cut down to 1/100th or less of the true spam volume and the little spam left in inboxes is merely a nuisance and not the productivity destroyer that it used to be.

Given these results, where fully half of the spam I found is not correctly classified by SpamAssassin, I cannot afford to use only SpamAssassin. Of course, possibly my threshold of 4.5 is too high, but with the already too high levels of false positives now, lowering the threshold to catch more spam will mean that there will be an increase in false positives too. I‘ll continue my current system where both spamassassin and bogofilter are in use.

• Email that bogofilter doesn't flag as spam but spamassassin does, is examined and, if it's really spam, sent to bogofilter for training.


• If it's not really spam, then it's sent to sa-learn for training as –ham, so that the bayesian component will eventually learn that it isn't spam and, hopefully, contribute to decreasing the spamassassin scores of similar email in the future.


• Email that bogofilter flags as spam but spamassassin doesn't is examined and if it's really spam, is sent to sa-learn for training.


• If it isn't spam, then it's sent to sa-learn for training as –ham


• Email that neither bogofilter nor SA classifies as spam but which *are* spam (false negatives) are trained as spam in both


• I generally just delete email that is flagged as spam by both since my false positive rates are zero, I haven<'t seen any false positives from bogofilter, or from bogofilter+spamassassin in a year

Mailbomb DDOS and Postfix solution

We'resuddenly getting hit by a DDoS that's mailbombing our SMTP server with many simultaneous incoming emails for email addresses that don't exist. So we're getting a lot of errors in our logs about rejected email because of "User unknown in local recipient table". It took us a while to get a handle on this. We got part of the way with some hacks, but the server was still unstable. I posted questions on the Philippine Linux User's Group mailing list and the postfix-users mailing list, and I've got a recipe of things to mitigate the problem.

Orly at mozcom says to do:

disable_vrfy_command = yes
smtpd_banner = $myhostname NO UCE ESMTP
smtpd_delay_reject = no

# slowing down bad clients [added recommendations from wietse]
# we NEED hard_error_limit in order for dictionary-attack stoppage to work
smtpd_error_sleep_time = 0s
smtpd_soft_error_limit = 5
smtpd_hard_error_limit = 10
smtpd_timeout = 30s

and Victor Duchovni on the postfix mailing list gave me the smtp_error_sleep time thing too. Thanks to both. We've checked with upstream and downstream mailservers and they're not getting bombed. So it's probably a targetted DDoS. Some competitor in CDO is sufficiently worried about us that they're willing to pay real money to have thousands of zombie computers out there (many of the IPs resolve to dsl and cable companies in the states, so they're always-on, high bandwidth, cracked-wide-open windows boxes being orchestrated to attack us at the same time) attack us. We had a similar problem around midnight one night, very high UDP packets coming in. Ah well, there's probably no way to trace this back to the person or company that commissioned this short of going and finding the person/persons who cracked those zombie machines and, well, dismembering them little by little until they squeal.

IP Piracy in the Philippines – Irony

>I see that here, Bong Revilla Jr has been honored by an american (I mean, of course, USian) motion picture group for his efforts against piracy. I take it they are lauding him for his crusade, such as it is, against video piracy.

Now, I live in the Philippines, and it is ludicrous that any USian organization would laud any filipino for making a dent (just a dent, there is NO WAY he is making any real difference) in the pirating of movies. I live in Cagayan de Oro city. Pirated movies are everywhere. Pretty much the only movies I can’t find in Cagayan (although I’m sure they’re there, I’ve been approached often enough by hawkers who had the goods right in their hands) are pr0n. I’ve been to Davao, and there, **everything** is available. Finding Nemo, any movie not yet showing, lots of movies that have already show, Pr0n, etc, it’s all there. Now, certainly the last time I was in Manila was long before this award. But when I was there, any visitor to Virra Mall would be fucking overwhelmed by everything that was available. Software was available (anything, any version, any fucking language, even though there weren’t many russian speakers in Manila, you could get russian language anything), movie VCDs were available, DVDs were available (mostly they weren’t really the official DVDs,


although there were some of those too, mostly they were the VCD movies, but packaged on one DVD-medium rather than two VCD media).

It is a joke to give any filipino an anti-piracy award **UNTIL THE FUCKING CDs ARE OFF THE FUCKING STREETS**.

But hey, personally I don’t give a shit. Who cares about Bong Revilla, heya? Who, in the philippines, where the minimum wage (and even in Metro Manila, where the median wage is higher than anywhere else in the country) cares about multinational media companies? When the median wage in Metro Manila (did i say that it was higher than the median wage anywhere else in the country? Oh, I forgot to say that in some parts of the country people STARVE to death [no joke, that is literally true] because there isn’t enough to eat? well, during El Nino season anyway). And, by the way, the cost of any legal CD is at least twice, and often three times the daily salary of regular workers?

There is something wrong with USian media-capitalist values. Perhaps it’s simply ignorance. And if it isn’t, well, I’m not going to much care. And as a practical matter, heh, good luck on the anti-piracy campaign. You sure aren’t going to fucking make a dent in the Philippines until you bribe some corrupt congresspersons to make copyright infringement into a death penalty offence. And do you really want to do that? Oh, you do?!!! well, think about it again. The first pirate given the death penalty will be a cause celebre and will probably cause the DEATH of your fucking corporations.

Bloggers and the Dark

There are a lot of good bloggers. Too many to have in the links there. And really, I don't particularly want to link to people because then they'd link to me. And then this blog would become more self-conscious (or, actually, more other-conscious), and I'd actually have to think about what I say here.

But I should list bloggers, and links, here anyway. Just so that, maybe, I could find them again later. Thus:

Naomi Chana. I don't think we have anything in common. But those who write well I want to remember so that, when I'm dark and down and could use the help, I can visit them and all without their knowing, gain the benefit of their light.

Mandrake Weirdness

I’ve got quite a lot of Mandrake weirdness. On my dialup gateway, iptables wouldn’t load at all. I cleaned some stuff up (apparently I’d installed newer modutils and such, maybe when I was playing with some cooker RPMs, or when I was playing with a 2.5 kernel. Anyway, I got iptables to insmod. But it’s still not perfect, because I get invalid something errors when I try to add the masquerade rule. So iptables works, but masquerade won’t.

I have just the same problem on the laptop. With:

iptables -t nat -A POSTROUTING -j MASQUERADE

# yes, I know that’s not secure, it’s just for example and for simplifying things. In practice there would be filters there.

I get : Invalid argument.

I don’t get that in the vmware linux. So I guess it’s because of installing unsafe RPMs and such. I’ll have to reinstall

on both. Hahay. Better not forget anything when backing up! Heh, last time I lost two or three days worth of email

(had backed up just three days before).

Local Thesaurus and Dictionary - a technique

I’d love to have the contents of dictionary.reference.com and thesaurus.reference.com (and, for that matter, Encyclopedia Brittanica) on my local machines. So that i wouldn’t have be online to get access. Now, there’s not much chance of getting a CD copy to work since I work in linux and there aren’t any linux readers. Sure I’ve got win4lin, but win4lin has some limitations involving multimedia. And VMware is too slow on this box.

So I’m thinking, it’d be an interesting afternoon’s work to download a bunch of web pages, strip out the unique words in them, add the contents of the unix wordlist, maybe whack a lot of random blogs to a depth of 5 or so. That should cover almost all the words in the English language.

Then, after taking all the unique words out, run them through the thesaurus. Try each word in turn, except if it’s already appeared in a previous thesaurus run. That should give us almost everything right there.

And finally, with our new list of unique words, run them through dictionary.

I wonder what the legal repercussions would be though. Not that it matters. I’m probably not going to do this, and even if I do, it’ll be only for my own amusement. Heh, I probably won’t even use the data, probably deleting it after a week or so. But the legal status would be interesting.

So maybe opera isn't so broken after all

Here I whine that opera is unstable and does a lot of things wrong. Now I’m not so sure. My problems might be configuration related. I’m blogging this now from a different username, with a completely fresh .opera (didn’t exist before this session). I’m not seeing the problems I used to. For instance, while blogging, for instance, the text going under the scrollbar… well, for one thing, the scrollbar is always visible. Unlike in my other instance. Also, in my other setup, I have the zoom percentage set at a default of 120%. This is set at 100% and so i don’t see the text going under the scrollbar. One thing about that though… the text is rather too small for comfortable blogging.

I’ll test some more. For now I’m very interested to see if the browser will be unstable with very many documents open. And then I’ll start to do my fixes to bring it to parity with my other setup (bookmarks, remove the ad thing, switch to text only browser look (i don’t like the icons since i’m not a visual person).