Gutsy dist-upgrade again - slow

My wife has finally finished (well, officially, of course there'll be support calls for a month or so) a lucrative software development project that is funding our immigration to New Zealand. She uses Kubuntu and she's at Feisty. I'd delayed upgrading her laptop so that there'd be no delays with her project.

With the project done, I've started the Gutsy upgrade. There's a bug with the Feisty restricted updates packages list. It's supposed to be .gz, but surfing to it shows that it's actually just a text file. so the first stage of the dist-upgrade fails because gzip tried to decompress the file and failed.

I didn't want to fight it, so I just removed restricted from apt-get/sources.list and restarted the dist-upgrade.

It then said that / was too small. So I cleared out /var/log/messages and since that didn't help enough, I also moved /usr/share to /home and linked /home/usr-share to /usr/share. Standard trick when I underestimate how large / should be :-). I'll move it back later when the dist-upgrade is done.

That's going to be *MUCH* later though. The packages are downloading, but at a glacial 11-20 kiloBITS per second. I'll crash now. A few hours after I wake up the downloads should be done and the upgrade can continue.

I've been sick today but I'm excited about finally upgrading her laptop, which is why I'm sitting t her computer right now, close to midnight. I'd wanted to install xdebug and protoeditor a few days ago but again, in the interest of stability, held off. When the upgrade is done, I'll install protoeditor and she'll finally be able to debug PHP programs in linux. *THAT's* really what I've been excited about. The upgrade is just a logical forward step to take before I get to install the debugger.

Protoeditor still has some bugs (mainly involving displaying some array entries). It's already very useful though. Just the ability to animate the flow of execution is a big deal (e.g., for figuring out incredibly bad legacy code, or incredibly good but complex code I wrote but don't understand anymore :-). Finding bugs is great too :-).

php debugger finally

I've been looking around for free php debuggers. I never did get $DAYJOB to spring for nuedit phpED (and anyway, I couldn't get the demo to work when I downloaded and tried it). I've been struggling with Dmitri's DBG for a long time. Sometimes I'd get it working and sometimes not. Lately I had it pretty stable, but I never actually wanted to work with the command line debugger client since it was so ugly, not even being as nice as gdb, for instance.

I got xdebug working but didn't like the command line client. There are a lot of options though for clients. I could only test free clients though. I tried two windows clients under wine. Neither worked and I didn't care enough to work very hard at them.

Xdebug support is supposed to be in the PHP Development Tools but after trying that out for two hours or so last night (much of that time spent downloading eclipse, once from ubuntu, and once from PDT), I gave up and went to bed. I couldn't get the debugger to talk to the spawned command line program, or to a web program. I think that was mainly a failure of documentation though. It looks like a great plugin to eclipse, but I couldn't figure out how to make things work because I couldn't find any clear documentation (it may be there anyway, I just can't find it).

Timmy woke up early and I got him to sleep at 2:20AM. I went downstairs (the other hour of eclipse work) and, in between resizing videos I installed protoeditor. This was an install from source. I had to install a bunch of libraries it needed, but finally I got it compiled and running and actually stepping through a program (running under apache, and also under the command line).

This has got me sufficiently excited that I'm totally stressing my bandwidth at home and at work because I'm installing required packages on a centos 4.x vmware image at home and on my feisty box at work. Both downloads are going slow and are downloading huge numbers of packages. Ah well. I wouldn't install on the VM except I work with a team who all use a similar VM, so I need to be able to tell them how to install protoeditor on their VMs. They could install it on their host distros, but then they'd be on their own with that. Damn, I'm just at the initial yum update on the centos 4.x VM and it's downloading 95 packages!

Packages I needed to install (may not be complete since I've also previously manually installed a whole bunch of other software on this box, so it's not a virgin gutsy install anymore):

• xlibs-dev
  
• libqt3-headers
  
• libqt3-mt
  
• libqt3-mt-dev (not sure if really needed, I installed it anyway)
  
• kde-devel (which pulls in a boatload of stuff)
  

OK. I just installed that on my computer at work to see if protoeditor would ./configure. It is now. I'll test if protoeditor is really working tomorrow. I could test now, but I don't like running X applications remotely over my VPN :-).

It works great locally on my laptop though.

Don't use Union unless you know what you're doing

I got bitten by a little insect I didn't know about, the other day. I'd written a family of views to simplify something horribly complex. Except in the penultimate view I had used UNION instead of UNION ALL.

I *HAD* too read the postgresql documentation on SELECT which clearly states that:

The result of UNION does not contain any duplicate rows unless the ALL option is specified. ALL prevents elimination of duplicates. (Therefore, UNION ALL is usually significantly quicker than UNION; use ALL when you can.)

.

But clearly I hadn't internalized it and continued preferring UNION because it was one word instead of two :-).

The bug report set me straight though. Now I will *definitely* remember the difference between UNION and UNION ALL (in that UNION folds duplicate rows into just one).

Vista dialup plus winmodem -- can't help

A very good friend of mine bought a new computer. The new PC has Home Vista Basic on it. It works well enough for most things (that I try, anyway). There was some confusion with printer permissions for her children (she's the only administrator). I sorted that out a few weeks ago. Now she's complaining because there's a problem with the winmodem. I actually tested this a few days ago and, if I dialup to her (prepaid) ISP after a reboot, it works. I stay online for hours. But when I disconnect and try to dialup again it always fails. Diagnostics is useless (all it says is that the connection failed).

Rebooting fixes things. Just reboot, and I'll be able to connect again reliably. Unfortunately, after disconnecting, dialing up never works again until the next reboot.

I'm not able to help her because, much as I'd love to switch her to Ubuntu, I have never had any success with winmodems on linux and I'm not going to give her my US Robotics USB modem (if I could find it).

I guess she's just going to have to reboot between dialup connections to her ISP. It's the price to pay for going with windows. I'm sure things would go better with windows XP. Maybe I'll have her go back to where she bought her PC and get them to downgrade her to XP from Vista.

Microsoft, a whole bunch of bastards who force a pregnant woman to go to her PC store so they'll tell her they can't downgrade her to XP.

Catching up

The last 4 weeks or so have been hectic.

• Timmy's nanny had to go home to the province and then wasn't allowed to come back by her husband.
  
• Timmy got sick and we had to spend a work week in the hospital,
  
• I got to work but had to catch up on a lot of things (still not caught up),
  
• Preparations for Timmy's birthday, the actual birthday and then a long weekend (spent with family, so not much work got done)
  

Things are getting back to normal though. Mainly because of the Nanny and hospital issues, I stopped reading email for a while. I just sped through my work email. Fortunately I'm caught up there. Now I need to get caught up on my tasks. That's going to be more hecticity for another week or two.

Gutsy dist-upgrade

I waited a few weeks before upgrading to Ubuntu Gutsy Gibbon because I've always had trouble with Ubuntu dist-upgrades and I wanted to get past some milestones first (Timmy's birthday, for one) before trying to upgrade and possibly ending up with a broken system that I'd have to spend a day or more fixing.

I've never had success with any Ubuntu online distro upgrade, and I've been trying it since Dapper. Twice I've broken my system sufficiently badly that I had to reinstall from CD (I could probably have fixed it, but I'm not all that familiar with dpkg or grub, and both times I had problem with one or both of those subsystems).

I'm pretty sure that those failed online upgrades were my fault. I install packages from multiverse, universe and some lesser known repositories. I also will sometimes monkey with files that the packages require to exist or be in one or another format. In any case, I'd done some similar monkeying with Feisty, so I wasn't confident that the distro upgrade would work flawlessly.

I finally made the plunge last night after a marathon of converting images (renaming more logically/usefully, at my wife's request), videos (converting in kino to divx so that they're 1/4 to 1/6 original size), and backing up everything (rdiff-backup to my external hard drive, and the images and videos to my wife's external hard drive and laptop, because losing timmmy's pictures and videos is inconceivable).

As usual, I didn't get a flawless upgrade. Fortunately, however, this time there was just one major problem. Initially there was a weird minor issue when the distro upgrade seemed stuck. I did an strace on the process and noticed that it seemed to be in a loop working on the linux kernel image. So I killed that and after a

dpkg --configure -a

restarted the distro upgrade. It worked better after that, asking me what to do about config files that I'd manually modified. For some questions I kept my changes (always after viewing a diff of the config files), others I allowed the distro to replace my changes.

Again, though, the distro upgrade failed. This time on an openoffice.org package that involved mobile devices (not sure how I got that installed, probably with the openoffice.org evolution integration package). That was bad enough that I had to dpkg --configure -a again, and then remove all the openoffice.org packages so that the upgrade could continue.

I had to go to bed after some pressure from my wife, and then a 30 minute brownout almost broke the install. After more pressure from my wife (including her turning off and unplugging my laptop :-), I finally went to bed and restarted the distro upgrade in the morning. Fortunately, everything worked fine after that and I've got a usable Gutsy system now.

There is one problem though. Firefox 2.0.0.8 (even with all add-ons removed) crashes randomly on some sites, and consistently (as in EVERY SINGLE TIME) on blogger. Which is why I'm posting this using epiphany. I was going to use Opera, but I decided to try the different browsers available with Ubuntu first before switching to Opera. Galeon breaks in the same way Firefox does. Clearly it uses the same buggy library. Fortunately, epiphany works well. I think I'll switch to epiphany until Firefox is fixed. I'm sure going to miss adblock though, and noscript, etc.

I use gnomad2 (since we've got a creative nomad zen mp3 player which we use mainly to play lullabies for Timmy to sleep to), and that works perfectly. I'm hoping that kino and ffmpeg work a bit better in Gutsy. In Feisty, most files would convert to divx medium quality. But some files would produce zero length .avi files. Those I'd have to reconvert to the next higher quality level just so that it'd be divx (and still half the size of the original avi file). I'll have to test kino with ffmpeg again on those files :-).

Upgrade: I broke down and googled for the firefox problem (first I ran firefox in a terminal so I could see the error message when it died).

It's a bug with libhunspell. The workaround is to disable spell checking in firefox. Turn off Edit|Preferences|Advanced|Check my spelling as I type..

Cool. I was liking epiphany, but some things were irritating (no multiple home pages, no multiple rows of tabs, although I really liked how fast it was, but no NoScript or AdBlock).

Slow, interactive, but it works -- looking for scripts

I've got a lot of videos of my son. When imported from the camera they're AVI files (yes, I know that's just a container, but I don't know enough about video formats to know what the actual encoding is).

The videos are pretty large. I've found a good way to make the files smaller though. Using kino, I import them (I get a larger .dv file). And then convert it to MPEG4 AVI. That works pretty well. Large files (150MB or more) convert (at High Quality, full size, progressive, 2240 kb/s) to about 1/6th the original AVI size. Small files (30MB or so) only convert to half the original size. But that's pretty good, nevertheless. The proximate reason for looking for ways to encode the files to be smaller is because I post some of them to youtube and those original AVI files greater than 100MB get rejected.

Kino has a .FLV converter too. That's interesting. But I don't think I want to keep archives of .FLV files. I'm a bit conservative about file formats and I tend to go with standards (de-factor, or de-jure) when given the choice between file formats which are on different points on the standard-non-standard continuum.

I've looked around for scripts to do this conversion. Haven't looked hard enough though, apparently. I could read the manpages and learn enough about video and audio encodings and transcoding so that I could hack up my own scripts. Too lazy to do that though. I'll wait until something comes along that's better than what I've got.

This is what it feels like to be a mere user :-). Normally I'd just totally geek and learn what I needed to get something working. Taking care of my son, and taking videos, and juggling disk space so that the videos will fit and can be backed up though, that takes all my time :-).

I've looked (via ps auxw) at the command line commands that kino spews to do the conversion. I may test some of that out. I don't know if everything I need is in there though. If it is, and a test works tomorrow, I'll hack something up and possibly post it here. But possibly not, too. Nonworking holiday tomorrow, I'll be playing with my son :-).

Gmail Advanced

I just went and installed Better Gmail and Gmail Manager on my laptop, both firefox add-ons. I'm going to have to install both of those everywhere.

The Mail.app mac-like skin from Better Gmail looks great. I first tried Super Clean but abandoned it quickly since I found it unusable.

GMail manager lets me have two or more tabs to gmail, each tab being a different account. That's very useful and I may finally switch over to using web based gmail for my gmail accounts because of that (I'll still have to keep evolution for backup and for access to my IMAP email at work. But finally GMail is much more usable when I can have multiple tabs, one for each account.

I've also finally found a convenient way to download my spam (which I use to train a bogofilter adaptive filter, just for fun). I think Gmail must have fixed their POP3 support somehow. Previously, this technique didn't work but now it does. The solution is to:
A. go to the Spam folder
B. select all messages on the current page
C. add those messages to a label (e.g., ZSpamX)
D. mark the email not spam.
E. wait until they're downloaded by my fetchmail process running elsewhere.
F. once everything is downloaded, go into the label, select everything and mark them all Spam.
G. Go into Spam folder and delete all Spam.

Unfortunately, D doesn't work with the gmail "select everything" link that pops up as an option when we've pressed the (Select) All link. So if I've got several pages of spam, I need to do A-D for each page. That's not bad though since I've got gmail setup to show 100 emails at a time.

To be clear, that last is not a problem with either of the Gmail related plugins. It's an issue with how Gmail works to begin with. Probably as a bar against people making a dumb mistake that does too global a job to the selected email the "select everything" link that pops up when you click All does not all you to assign the same label to all matching emails, nor does it allow you to Unmark all matching emails Spam. I don't remember if it allows marking all matching email spam. I've already collected and remove all my spam, so now I don't remember whether marking all the spam messages spam again was one step or ten.

In any case, the C and F steps are vital. I don't think they previously worked. Or if they did, I didn't think to try them back then. It's great though, now, to be finally able to grab all my spam so I can train my own spam classifier. Sometimes spam gets past all the spam classifiers between me and the internet, then I'd like to have a personal spam classifier customized to my needs, similarly, sometimes mail from friends or mailing lists is marked spam by mistake. Seeing all the spam and running them through my own classifier gives me a better chance of finding the false positives since at least I control the classifier.

On 40 tips for optimizing PHP code

I saw something like Reinhold Weber's 40 tips for optimizing PHP code maybe a year ago. Recently, this particular set of 40 tips popped up on reddit. I decided to test. Now, my tests are trivial, on the command line, on Ubuntu. YMMV. Particularly with respect to "trivial", since the tests are so small they're likely to stay in L1 or L2 cache, so they may not look like other people's more complex tests since they're not likely to trigger cache misses.

Alright, after the first two tests, I've decided that this is probably not worth doing. I may continue writing some tests, but:

1 - use static if possible - 1.25 percent difference.
2 - echo is faster than print - yeah, by 1.35 percent.

Some of those tips might actually be pretty useful (I'll run some more tests over the next few days), but these two aren't very useful. Sure, if you applied 100 optimizations, each giving a 1 percent improvement you'd double the speed of your program. But I routinely improve the speed of programs by orders of magnitude by improving SQL queries, creating appropriate indexes and simplifying complex code. Most of those 40 tips sound like micro-optimizations which aren't worth the trouble to implement.

To be sure, error suppression with @, apart from being slow, is usually a mistake (turn on error logging in development, push error logging to a file or syslog rather than stdout in production), and $row[id] is just stupid and the sign of an incompetent developer, as are incrementing uninitialized variables and #31. Using mysql is almost always a mistake (it has its uses, but everywhere I've worked, mysql has always been the wrong tool and anyone who implemented it [i.e., me] always regretted the decision later).

3 - avoid magic (__get, __set, __autoload) functions - OK. magic __get and __set [together]
are half the speed of accessing the variables directly. So there's certainly some cost
to the indirection. It may be that high level frameworks could (and should) use the
magic functions to abstract away details. I've avoided them for now though since they
don't improve maintainability or quality of the code I currently work with, although I'd
be open to using them if I were to work on something significantly more complex than what
I work with now. I don't see anything that complex on the horizon though, I'd probably
switch to another language for anything complex (perhaps java, perhaps python, it'd depend
on the project).

More RAM in my next job -- and multitasking limits (not enough RAM in my brain)

I've got a slow computer (800Mhz) with a reasonable amount of RAM (640MB) at work. My laptop is twice as fast, with twice as much memory. But of course, the laptop hard drive is slower than the desktop hard drive.

I've been twiddling my thumbs quite a lot lately because a lot of my work has been very disk intensive. I could be speeding things up by not doing svn update of 300MB of source, instead going to just the directories that have been modified and doing the svn update there, but then that would require knowing intimately what the other developers are doing. And *they* don't necessarily remember all the directories they committed into.

So, in my preference for dumb solutions that work all the time (e.g., svn update at the root of the tree) versus smart solutions that can easily fail (svn update to targeted directories, possibly missing other directories with fixes), I just svn update at the root. I do similar things with very large databases too, and very large file copies (rsync). The dumb solution that works all the time but is not always optimal is more cost-effective than the smart solution that will take three weeks to shake all the bugs and corner cases out of.

I'll optimize (spending that three weeks later on), when the need arises.

If I'm going to be doing much the same in my next job, then I'm going to require my employer to get me the right hardware. So I'll have 4GB of RAM (or more), and the fastest SCSI drives money can buy, in RAID-0 with rdiff-backup to RAID-1 slow SATA drives. And then I'm going to do all my svn work in a 2GB ramdisk. Hahahahaha.

I was discussing the edges of this (in the context of multitasking) with the boss of my boss and he said, "well, so you can do other things while waiting for those long tasks to finish, right?". This was in the context of a query that ran for 22 hours (and yes, I got pissed off enough at that to set up an aggregated materialized view built via triggers [postgresql] so that it now runs in an hour). At the time I said, yes, so I can multitask and need not twiddle my thumbs.

I'm finding that there are limits to multitasking though. I get confused and forget to do important steps when the list of tasks hits more than 5 or so. So its best to stick with 4-5 concurrent tasks. So if all 4 or 5 of those tasks take more than an hour, I'm back to twiddling my thumbs because I just don't do well if I add another task. As designated mentor at work, I don't actually have to twiddle my thumbs though. Instead, I can stick my nose into my colleagues work and offer unsolicited advice. Sometimes it's even good advice :-).

Software is Hard

Kyle Wilson has a great article on software development, compromises, economics, complexity, lines of code, and why Software is Hard

I use "economics" the way I normally do though (which is not the way it usually is meant in RL, but possibly how it would be meant among economists). It's not just about money but about compromises among competing values and how different weights among the different values affect the outcome (which is usually also measured along several dimensions).

vim on slow links (sshfs)

Sometimes (as today), I need to edit files remotely. If the bandwidth is slow (PLDT myDSL has some occasional bogosity where my route to the office, which is normally just 5 hops, bounces between two internal routers whose IP addresses differ only by 1 in the last octet, so then my vpn goes to the other links and the hop count goes to 18 or so because it's overseas), editing remotely is intolerable.

Then I go with sshfs. I've got a script that mounts my working copy over sshfs. I always need to remember to setup vim though to not use swapfiles. So, from the man page,

-n No swap file will be used. Recovery after a crash will be impossible
Handy if you want to edit a file on a very slow medium (e.g. floppy).
Can also be done with ":set uc=0". Can be undone with ":set uc=200".

I've got an annotation in my ~/.vimrc (normally commented out) to set uc=0,
but I forget that it's there and I always google for sshfs vim swap file. And
then I go to this page, which doesn't have the answer :-).

I would post the solution so that the question doesn't stay unanswered indefinitely. If I could reply without joining the group I'd do that. But joining is required, and I'm no joiner. So the best I can do is point at it here.

Heheh, this will also help ME since I would normally query something like:

monotremetech sshfs vim swap

:-)

OLPC -- Inspiring

There's a great post on OLPC uptake and how it's changing the school experience in Arahuay, Peru.

This is very inspiring. If all (or even half of all) social/economic development projects were to end up 1/10th as effective as OLPC-Peru/Arahuay has been, the world would be incredibly good.
Antonio is repeating second grade, but:

The first day Antonio came to school with his XO laptop, instead of playing during recess, he kept on exploring the XO. He quickly became one of the most acquainted with it and empowered to help his classmates by telling them how to do this or that. The change was suddenly and highly noticeable. His teacher was amazed of how he had become more focused doing the class work and was helping his classmates with the computer activities.

And Diego's story is hard but heartwarming.

Diego is in second grade. He had stopped coming to school. However, he did come to pick up his XO laptop the day we handed them out. His teacher explained to me, that he has no father, and that his mother went to Lima, the capital, to work. And now, Diego and his two brothers are living by themselves, but a kind neighbor gives them their meals.

After that quote, there's a picture of Diego helping a first grader on using the OLPC.

Emilio is a hero:

“Once, when the teacher was explaining to the children how to look up some new words in the on-line Diccionario de la Real Academia Española [the best Spanish dictionary, costly and practically out of the children's reach in book form]. Emilio understood the procedure and quickly looked up the entire list of words while the teachers and the other students, together, slowly went through the process. The Internet connection went down. The teacher made the best of the situation and explained what the Internet was and about the satellite connection, for them to understand what was going on, and ended by saying, "We will have to wait a little bit until the connection is reestablished." To which Emilio replied, "No need, teacher. I got all the words, and everybody can copy them from me.”

But not just any hero. Anyone can shoot other people or order peons to shoot other peons. Emilio is a hero of the mind.

It's pretty clear that the OLPC is inspiring kids to stay in school. At best, they get interested in school and using the OLPC. At worst, they'll stay in school because if they don't, the OLPC will be taken from them.

high speed!

I've been very disappointed in my external USB laptop drive enclosure. Whenever I'd run my rdiff-backup or just a regular rsync backup (with --progress, because speed is what I'm nuts about) I'd get pissed off because the speed would only stay at arund 1-1.2MB per second.

Today though, I hit 5-5.5MB per second. I figure it was because I read some random advice (I think on the Philippine Linux users group mailing list) about plugging both USB jacks into the laptop because sometimes the power coming down one cable isn't enough (or the cable is substandard).

I'm a Psych major, so I never actually took any courses on electricity or electrical engineering (the computer science and computer engineering people do). I'm very glad though that, in fact, plugging in the second cable seems to have sped up the transfer rate. I may go buy a higher quality cable too as that is supposed to increase transfer rate too. For now though, ohhhh, 5+MBps is SOOoooo much better than 1+Mbps.

It's now fast enough that backups will be a joy, or only a minor nuisance, rather than the big production it used to be (I've got tens of gigabytes of data to transfer, it takes a while to rdiff-backup all of that, even if nothing's changed and all rdiff-backup has to do is read through the files to see if any bits have flipped).

[Update]
I don't think anymore it was the second cable (or the power issue). I think I probably got an kernel update that finally supports fast USB on this laptop. I experimented tonight, copying the data with both cables, and then with one cable. The speeds are the same. I'm still very glad to be getting 2.5-20MBps, all speeds are a lot faster than the 1.0-1.2MBps I'd get previously. But it's not the cable. That's too bad, since waiting for the kernel to get fixed for a particular laptop chipset is harder and less deterministic than just plugging in the other cable :-).

I'll keep the other cable plugged in though, still. I'm sure it'll help with stability, if the external drive gets enough power instead of sometimes getting too little.

Wrestling with pure eeevveeeel

Jeremy Miller has a whiny rant on a project he's working on.

The whiny part isn't from me, it's from his comments section.

I've been wrestling with pure evil too. It's not java, and it's a whole program that's too long, not a function. there's only one function in there, it's called doublenum, and it doesn't arithmetically double its parameter (which would be at least descriptive), instead it doubles its LENGTH. sometimes. by prepending a zero if it's only one digit wide.

so, applying some items from Jeremy's rant,

1. Very long method [check, although it's not even a method, it's just PHP code that's not in a function, so everything is global]


2. Variables are reused for different things throughout the method [check, but at least variable names aren't totally bogus, although it does use a class prepended with the initials of its author]


3. Lots of if/then exception cases [check - in fact, since there are about 8 common cases, the whole program is bunch of nested if/thens with one code path being one of the common cases, with exactly the same actions copy-pasted all over the place, plus deeply nested if/then exception cases peppered through the code (copy-pasted again), and at the end, just one of those common cases will be executed]


4. Deeply nested Arrowhead code [ check, except its several arrowheads ]


5. All data stored in Hashtable's -- and it's this data that's getting bamboozled somewhere [check - plus parallel PHP arrays (integer indexed), plus parallel (sort of) associative arrays where the keys are concatenated strings of the subkeys, plus some two dimensional arrays where the concatenated string keys previously mentioned are split up and make up the two dimensions]


6. Numbers that come out wrong - [ check, or, actually, a really obscure bug where orders get their order quantity zeroed ]

And for extra bogus points:

1. 
       $r=7-($cday-$x);
       $mt = sprintf("%2d",chop(`date +%m --date '$r days' `));
       $dy = sprintf("%2d",chop(`date +%d --date '$r days' `));
       $yr = sprintf("%2d",chop(`date +%Y --date '$r days' `));

   
   what moron calls a system function THREE TIMES to get the pieces of a date? If they didn't know anything about time functions in php (or perl, this crap is all over the perl code these people wrote too), they could at least have called date once, and then extracted the fields out of there. But then, if they were too dumb to learn how to use the time functions (or even just to learn that they exist), I suppose it's too much to expect that they would figure out, The function names aren't helpful, but that's not as much a bogosity as forking to get the time, three times. Even selecting the date from the database would be better than this stupidity, and if they couldn't wrap their brains around that, at least fork only once!,

   
       date +%m-%d-%Y --date '1 day'

   
   
2. oh, and that doublenum function? The only function defined in the whole program? It's actually inline in the code TWICE. But php doesn't do static analysis worth anything. Instead, if the two function definitions are in perfectly separate code-paths (it's never possible for both function definitions to be executed in the same run), then PHP sees no problem with that. PHP only points out function redefinition errors when the function definition is actually executed, e.g., no error here:
   

   
       function f()
       {
           return 0;
       }
   
       if(false)
       {
           function f()
           {
               return 31415;
   
           }
   
       }
   

My mistake with this abortion is that I've been trying to live with it, fixing it incrementally, when fixes needed to be done. I should have thrown this away long ago though, and just rewritten it so that it wouldn't be so bogus.

I was avoiding doing that because the horrendous date logic was totally obscured by the implementation, so it wasn't possible to extract what the date logic was supposed to do at all just from the source code (not possible for me anyway, I would get angry too early and give up). Now I'm going to have to understand the date logic and rewrite it all. Or if that can't be done, then use the current horrendous implementation as a test for proving that my implementation is "correct" (or at least complies with the current implementation, since I can't understand the current implementation, I'm not currently capable of proving *IT* correct. I can only take it as a given).

I'd better not meet any of the previous implementors in this or any future world. They're going to KNOW what I think of them, and then I'll hire them so I can fire them one second after.

screen multitasking

I've known about screen for a few years. I never did use it until recently though. When I tried it out a few years ago I came away with the impression that it wasn't worth the trouble since I'd have to learn another set of keystrokes. When I tried it back then, I didn't like Ctrl-A or any of the other keystrokes that came with it (all prefixed with Ctrl-A).

Of course the activating keystroke (Ctrl-A) can be configured to be something else, but the rest of the keystrokes still seemed unnecessary cruft to learn.

I recently looked at screen again though and I've fallen in love with it. I don't use it as efficiently as I might. I still have a problem with the keystrokes, so I learn the minimum that I need (Ctrl-Ad, Ctrl-AA, Ctrl-A[, Ctrl-A],Ctrl-An, Ctrl-Ap, Ctrl-A", Ctrl-AX) and leave the other commands alone until I find that I need them. I still only have limited space in my brain for keystrokes, so I economize.

The problem with screen now is that I've got 8 processes running in screen at work (I'm working remotely because my baby is sick and I need to be home to help take care of him). I can barely multi-task 3-4 processes, and now I'm running 8. Of course, they're all long running database processes and once they're set up they run independently until they complete. I could then move on to another set of 3-4 things to do. I avoid doing that too though since when any in the first set of tasks finish, I'll have forgotten what the purpose of it was :-). And frankly, some of these tasks are going to take hours. I can't afford to keep adding sets of 3-4 tasks until I can't count them anymore :-).

I do find screen very useful though. Mainly, as a replacement for nohup. Now I can start a screen session remotely, run the program in screen, give it any interactive commands it might need (hard with nohup) and then leave it running. Previously, I'd have problems with running a program over ssh (when the vpn would burp, I might lose the session, I certainly couldn't control it remotely anymore). And nohup has problems with interactivity. Screen is a great help there since I can get interactivity, and if ssh breaks because of network issues, I just ssh there again and re-attach to the screen.

I didn't know about -x back then too, when I was first looking at screen. I like working with 3 or 4 actual terminals on my monitor. That's one reason I don't like KDE, it's necessary to manually detach a tab. In gnome, I can just turn off the menus (in fact, I use aterm now, but for a while I was using gnome-terminal) and have terminal windows automatically open in new windows rather than tabs (when I do Ctrl-Shift-N in one terminal it creates a new terminal as a separate window.

For screen, I just ssh to the remote box, screen -x, do that 3 or 4 times, and then have each terminal go to a different screen sub-session (Ctrl-Alt-" or Ctrl-Alt-n a few times).

I've even used screen for discussing/team-programming. The several people in the discussion all connect to the same screen session and then discuss, view, and test code while all looking at the same editor sessions, debug screens, etc. That works for command line work, or just basic code slinging. of course, testing in a browser, or in some graphical environment is harder. I've avoided those complexities for now though, although of course vnc is waiting in the wings for when I need it.

robinson's free wifi - going downhill

I used to love sitting in the Cafe Mediterranean in Robinson's Galleria because of the good food, cold beer, and free wifi. I've also used it downstairs, by Bo's cafe. At the time, the wifi signal itself was good. The DNS was slow, as was the actual connection, but I didn't mind since I wasn't downloading huge files anyway. It was a pleasant place to surf and have coffee or a snack.

The wifi at Robinson's Pioneer was always bad, one could use it, but it was intermittent and slow when available.

I recently went to both Pioneer and galleria and checked out the wifi situation (at Cafe Med at galleria, and at Old Spaghetti House at pioneer). The wifi is much worse than it used to be. Possibly the hotspot has been moved at Galleria, but when I was last there, the signal wasn't available at all at Cafe Med. There were some secured networks, but no free wifi. I didn't go to Bo's to test there. It wasn't important enough.

At Pioneer, Old Spaghetti House charges a ridiculous PHP 50.00 per half-hour just for plugging in to the wall. And the wifi was very slow and unstable when I was there and it's completely down now (right now, my wife is meeting a fellow developer there). We liked the coffee and dessert at Old Spaghetti House, but the cost of plugging in is sufficiently stupid that I believe I'm going to tell anyone interested about how Old Spaghetti House is extremely unhospitable to laptop warriors. I think it's great that they have wall plugs at all. A more reasonable price or policy would be cool. For now though, absent either, the policy is just dumb and I'm not going to give them any of my money again. Although if my wife insists on going there, she can pay for lunch. The creme brulee WAS very good.

ionice - redux

I love ionice enough to give it thre exclamation points but I'd been having some problems with it. Mainly, I was optimizing too much. I was using -c 3 on the assumption that everything would run much better if svn update or git-svn rebase was taking too much IO bandwidth.

This was a mistake. I would have problems when the ionice -c 3 process was running since -c 3 (idle priority) requires root access, so if it created files, those files would be owned by root since ionice -c 3 needs to be run under sudo or directly, as root (sudo su, root login or /etc/crontab).

It took me more than a month of frustration (and actually stopping the use of ionice, or doing a chown -R at the end) to finally ask google :-). I must not have been all *that* frustrated.

There's an explanation here of why ionice -c 3 requires root access.

So finally I'm convinced and will now run large svn updates (or my huge rdiff-backup based backup system to a slow external USB drive that only runs at USB 1.0 speeds) under ionice -c 2 -n 7. A few quick tests shows that performance of the laptop (no locking of X, etc) is about as good as with -c 3 but there's no need to run under sudo, so no problems with file ownership or permissions later.

Downloading OpenSolaris sucks

After a good email from David Meyer, on the PLUG mailing list I decided to download OpenSolaris and test it out on vmware.

I chose to use the Sun Download Manager last night. I'm not going to do that again. The download was chugging along slowly but steadily but today it stopped. And the Sun Download Manager has this clunky, unintuitive interface and apparently some weird logic in there so that only one file can download at a time and there's a maximum number of retries and then a file will never be downloadable again. Possibly the internet went up and down today and it hit the maximum 10 retries so it stopped trying to download. I changed the retries to 20 though (the maximum, why is there a maximum??? If I want to keep downloading forever on an expensive and slow GPRS link, why not let me do it?) and it STILL wouldn't download anymore.

When I got back this evening it was still downloading, but at 5kBps, when I can normally get up to 80-120kBps on my DSL connection at home, I wasn't going to wait 70 hours or whatever it predicted the wait time to be.

Since I couldn't make it work anymore, I just nuked the whole Sun Download Manager thing and am currently downloading the files via the browser. I'm downloading all three files at the same time and they're all coming down at 30-80kBps. That should get faster as the night deepens.

This sucks though. I should get better speeds from Sun microsystems. I'll keep trying to download, but the piece of crap that is the Sun download manager doesn't inspire any confidence in me. Likely OpenSolaris will be very good. It's a very mature piece of work. I'll see if I get comfortable with it. But my expectations aren't very high just now. I hope I'm disappointed in my expectations.

svn mv and atomicity - lesson learned

I was moving some directories around in $DAYJOB's svn repository and I wasted a lot of time doing it. The problem was that I didn't realize that svn mv requires that both source and destination parent directories need to be committed at the same time.

I needed backward compatibility, so what I was doing was similar to:

svn mv {parentdir/srcdir} {elsewhere/destdir}
svn commit {parentdir} -m "some message"
ln -s {elsewhere/destdir} {parentdir/srcdir}
svn add {parentdir/srcdir}
svn commit {parentdir/srcdir} -m "some message"

svn commit {elsewhere/destdir}

usually I'd do that for a few directories before bunching the
commits of elsewhere.

I was then getting svn errors because the source wasn't around anymore. When
doing svn mv, it's necessary to commit the A (in the destination directory) and
the D (in the source directory) in the same commit, otherwise there will be
a lot of backfilling and rework necessary. What I had to do was find the
revision at which the given directories were deleted from the parentdir,

svn cp -r [rev] [URL] [destdir]

them (specifying a revision one less than the one they were deleted in), and
commit, bringing them back. Then I'm doing the svn mv operation again, but
this time committing both at the same time and only doing the "ln -s" at the end,
and in its own commit (can't be done along with the svn mv because I want the
link to have the same name as the source file, but the source file hasn't been
deleted yet, the ln -s and svn add of the link needs to be done after the
source file D operation has already been committed.

The links (for backward compatibility) are making a regular svn update fail.
This is because svn update needs to create the link, but the original file
is still there. svn doesn't replay the commit history, so it doesn't have
the opportunity to delete directory entries which are no longer the same type
(formerly directories, now links). I was going to just check out the whole
tree, but I think I'll wait on that, instead I'll first just remove all links
(find . -type l | xargs rm -f) and then svn update since the only same-name
files in there are links inserted for backward compatibility.