I was going to test out git-svn, but I couldn't make heads or tails of it. So I'm working with svnmerge and I like it very much. It looks like git would be very nice to work with, but since I can't understand it for now, I'm going to watch linus' youtube presentation on git.
Linus is pretty irritating in the presentation, and vlc won't fast forward over the flv file I've got, so I can't skip the pointless part. I'm just going to have to sit through the whole thing I guess. Maybe mostly listen to it and only look at it when he discusses something that looks like the slide is important.
Saturday, July 28, 2007
Thursday, July 26, 2007
backup and restore
I'm backing up my laptop's /home filesystem to my USB harddrive because I think it's time to change filesystems. /home is xfs. A few weeks ago though I found what seems to be a bad sector on that partition (an ISO image on /home had sections that were unreadable, although the same CD that the ISO was built from was perfect). Unfortunately, neither XFS nor JFS has the capacity to accept badblocks output so that the blocks found bad can be marked unusable (decreasing usable disk space a bit, but keeping data safe).
So I'm backing up my data (damn it's taking a long time to backup 37GB over a USB cable) so I can reformat /home as reiserfs (likely) or ext3 (less likely). And then it'll be another night of running badblocks so I can feed that to the new filesystem.
This isn't a big deal. I'm reading widely and playing chess against crafty (and losing every game). So it's not time lost. I'll go to bed soon and hope (but unconsciously) that the backup will be done by tomorrow morning, so I can format, badblocks, and slowly copy the data back.
This laptop doesn't have any S.M.A.R.T. options in the BIOS, so I can't take advantage of those options (apparently JFS and XFS rely on S.M.A.R.T. to handle bad blocks for them). Heck, since Ubuntu makes the IDE drive look like a SCSI drive, I can't even hdparm -d1 /dev/sda. I get:
Yech. There's probably a way to force Ubuntu to treat the drive as an IDE drive instead of emulating SCSI for it. I haven't gotten around to surfing for that solution though. It's not important enough just yet.
The next morning: Wow, badblocks has gone through just a fourth of the disk and the badblocks file is already 135MB. I think I'll have to go with ext3 instead of reiserfs so that I can take advantage of the -c parameter to both mkfs.ext3 and fsck.ext3. Yech, I've always had more problems with ext2/ext3 than with reiserfs/xfs with corruption after powerloss. But I'm going to have to bite the bullet here. And anyway, it's a laptop. The only time the OS wouldn't be shut down correctly would be if there were some sort of kernel panic (haven't got those in a long time, since I stopped compiling and using win4lin kernels) or if I were to turn the laptop off. The battery works as a UPS, so I won't get instant off when the wall power goes out.
I should really buy a new harddrive, of course. I'm not able to just yet though, so I'll limp along with this one and make weekly rdiff-backups to the USB drive, and another rdiff-backup everytime I transfer pictures and videos of Timmy from the digital camera. :-)
So I'm backing up my data (damn it's taking a long time to backup 37GB over a USB cable) so I can reformat /home as reiserfs (likely) or ext3 (less likely). And then it'll be another night of running badblocks so I can feed that to the new filesystem.
This isn't a big deal. I'm reading widely and playing chess against crafty (and losing every game). So it's not time lost. I'll go to bed soon and hope (but unconsciously) that the backup will be done by tomorrow morning, so I can format, badblocks, and slowly copy the data back.
This laptop doesn't have any S.M.A.R.T. options in the BIOS, so I can't take advantage of those options (apparently JFS and XFS rely on S.M.A.R.T. to handle bad blocks for them). Heck, since Ubuntu makes the IDE drive look like a SCSI drive, I can't even hdparm -d1 /dev/sda. I get:
/dev/sda:
setting using_dma to 1 (on)
HDIO_SET_DMA failed: Inappropriate ioctl for device
Yech. There's probably a way to force Ubuntu to treat the drive as an IDE drive instead of emulating SCSI for it. I haven't gotten around to surfing for that solution though. It's not important enough just yet.
The next morning: Wow, badblocks has gone through just a fourth of the disk and the badblocks file is already 135MB. I think I'll have to go with ext3 instead of reiserfs so that I can take advantage of the -c parameter to both mkfs.ext3 and fsck.ext3. Yech, I've always had more problems with ext2/ext3 than with reiserfs/xfs with corruption after powerloss. But I'm going to have to bite the bullet here. And anyway, it's a laptop. The only time the OS wouldn't be shut down correctly would be if there were some sort of kernel panic (haven't got those in a long time, since I stopped compiling and using win4lin kernels) or if I were to turn the laptop off. The battery works as a UPS, so I won't get instant off when the wall power goes out.
I should really buy a new harddrive, of course. I'm not able to just yet though, so I'll limp along with this one and make weekly rdiff-backups to the USB drive, and another rdiff-backup everytime I transfer pictures and videos of Timmy from the digital camera. :-)
Thinice
I thought I'd look for an Ubuntu (gnome) desktop theme I liked. So I looked around and tested the themes in System | Preferences | Theme. I didn't much like what I saw, but then I clicked on "Customize" and, testing the different themes there, found that I liked the *darkness* of the Thinice theme.
I tend to lower the brightness of the screen on any computer I use. Even on a laptop LCD, I prefer for the screen to be less bright. That isn't consistent with the dark text on light background of my terminal windows, but there it is, Very well then, ... I contain multitudes.
In any case, I liked thinice and I'm going to move all my computers (work and home) to that theme :-). The scrollbar sliders are cool too.
I tend to lower the brightness of the screen on any computer I use. Even on a laptop LCD, I prefer for the screen to be less bright. That isn't consistent with the dark text on light background of my terminal windows, but there it is, Very well then, ... I contain multitudes.
In any case, I liked thinice and I'm going to move all my computers (work and home) to that theme :-). The scrollbar sliders are cool too.
Monday, July 23, 2007
DRY
So this code I'm working through (it was supposed to be a trivial exercise in fixing brace style) has turned into a nightmare. I can understand when people copy and paste some code. Maybe copying and pasting the same code twice or three times is OK, although three times is stretching it. BUT GODDAMN IT, MUST THE SAME DAMN CODE BE COPIED 23 TIMES????
And it's not just one function it's more like 8 functions copied and slightly modified (or not modified) all over the place. If this had to be done at the beginning because there was no time to refactor, __NOW__ is the time to refactor because any bugs that come up (will need to be fixed 23 times. And I guarantee that at least of those bugs will be missed.
Parameterize, abstract, and remove redundancy. Dammit.
Oh yeah, that's Don't Repeat Yourself.
And it's not just one function it's more like 8 functions copied and slightly modified (or not modified) all over the place. If this had to be done at the beginning because there was no time to refactor, __NOW__ is the time to refactor because any bugs that come up (will need to be fixed 23 times. And I guarantee that at least of those bugs will be missed.
Parameterize, abstract, and remove redundancy. Dammit.
Oh yeah, that's Don't Repeat Yourself.
the root of all evil
Alright, it's not all that evil, but I'm looking at this PHP code:
And I'm thinking,
I don't care too much about the strings being hardcoded, the app doesn't need to be multilingual, yet. But I'm struck by the fact that in the attempt to optimize (in PHP, single quoted strings are faster than double quoted strings since the runtime doesn't need to try to do string interpolation (parse the string to find variables, and replace the variables with their values).
On the other hand, this code is going to send mail. This is a classic too early optimization. The network will be the bottleneck, or writing the file will be the bottleneck (if we write the emails into a file first for batch sending later [which isn't relevant for the concrete case, the email is sent immediately, see the mailto(...) function call]). And it's harder to read (mixing two kinds of strings always is, and in fact the code is mixing three kinds of strings, the third being variables. Code for maintainability first, code for performance only after profiling and finding the hotspots in the code.
$message = 'Hi, ' . "\r\n";
$message .= "\r\n";
$message .= 'Your account at ' . $this->domain . ' is awaiting activation.' . "\r\n";
$message .= "\r\n";
$message .= 'To activate your account, please visit ' . $this->domain . '/members/activate/' . $activation_key . "\r\n";
$message .= "\r\n";
mail($to, $subject, $message, $headers);
And I'm thinking,
- mixing single quoted strings and double quoted strings.
- mixing single quoted strings, double quoted strings, and concatenating variables instead of putting them inline
I don't care too much about the strings being hardcoded, the app doesn't need to be multilingual, yet. But I'm struck by the fact that in the attempt to optimize (in PHP, single quoted strings are faster than double quoted strings since the runtime doesn't need to try to do string interpolation (parse the string to find variables, and replace the variables with their values).
On the other hand, this code is going to send mail. This is a classic too early optimization. The network will be the bottleneck, or writing the file will be the bottleneck (if we write the emails into a file first for batch sending later [which isn't relevant for the concrete case, the email is sent immediately, see the mailto(...) function call]). And it's harder to read (mixing two kinds of strings always is, and in fact the code is mixing three kinds of strings, the third being variables. Code for maintainability first, code for performance only after profiling and finding the hotspots in the code.
Sunday, July 22, 2007
ionice!!!
Hot damn! ionice is incredible.
I was going to say "nice", but that would have been obvious and shallow.
I know, that's a link to a short tutorial on how to use ionice, I like short tutorial links though, so I can test something out quickly.
I'm installing something very IO heavy in vmware. This is on a single-core 1.5Ghz, 1.25GB Ram laptop (slow IDE hard drive). Before ionice, the browser and sometimes all of X would stall on me for some seconds (30-40) before getting more responsive. I don't mind if the vmware process takes a long time to run. If it finishes tomorrow, well, I'll just sleep through it. So it's very nice to be able to lower its IO priority so that it doesn't make the laptop completely unusable when it's shoveling gigabytes around on the disk.
Highly recommended. Requires at least kernel 2.6.13 though. I've already asked the system administrators at work for one of the debian servers at work to be upgraded to 2.6.13 or higher. That box runs a very long running doxygen process that makes everything else crawl for a while. It'll be great to get some of that performance back :-).
I was going to say "nice", but that would have been obvious and shallow.
I know, that's a link to a short tutorial on how to use ionice, I like short tutorial links though, so I can test something out quickly.
I'm installing something very IO heavy in vmware. This is on a single-core 1.5Ghz, 1.25GB Ram laptop (slow IDE hard drive). Before ionice, the browser and sometimes all of X would stall on me for some seconds (30-40) before getting more responsive. I don't mind if the vmware process takes a long time to run. If it finishes tomorrow, well, I'll just sleep through it. So it's very nice to be able to lower its IO priority so that it doesn't make the laptop completely unusable when it's shoveling gigabytes around on the disk.
Highly recommended. Requires at least kernel 2.6.13 though. I've already asked the system administrators at work for one of the debian servers at work to be upgraded to 2.6.13 or higher. That box runs a very long running doxygen process that makes everything else crawl for a while. It'll be great to get some of that performance back :-).
Saturday, July 21, 2007
trying out git-svn
I've got a pretty good solution to the merging problem in svn. I've also previously used svk for disconnected work. I couldn't stand the ugly commit messages svk created though. I'm now testing out git-svn. If the commit messages from git-svn are reasonable, and there are no problems with branches and merges and commits to svn, and it runs very fast, then I may personally switch to it.
If git-svn works out as well as promised then I'll switch to using it and the others on my team can decide whether they want to stick with pure svn or try out git-svn.
If git-svn works out as well as promised then I'll switch to using it and the others on my team can decide whether they want to stick with pure svn or try out git-svn.
svnmerge heaven
There are three major problems with subversion.
Git apparently solves all those problems. So says Linus on youtube.
I haven't used git and I'm not likely to at $DAYJOB. It took a long time to convince the developers and system administrators at $DAYJOB to actually use svn and I'm not done with the conversion yet, technical support isn't on the version control bandwagon yet. So there's just no way I'm going to spend another year or two convincing people to switch to git from subversion.
I don't worry too much about being online to do most operations. At the office I'm online to the svn server, and at home or anywhere else with an internet connection I can get through to the office on the VPN. If I'm not online, well, I just won't work. Or I'll work with svk if necessary.
The current full working copy (including branches) at work is pretty large, so an svn status on that *is* pretty slow. Even svn status on just the trunk is slow. But we work around that by avoiding working at the top of a branch unless it's really necessary. Working in individual subdirectories is much faster and the only time we ever need to work at the top of a branch is when we need to see what all the uncommitted changes have been to that branch, or when we need to do an svn log to see what all the committed changes have been to the branch.
For a long time though, I would make branched tags but would not actually do experimental or development branches because, while branching is easy, merging was painful. It could be done, but it was so inconvenient I never tried to do it.
Finally, svnmerge has landed. Svnmerge is a python program (on Ubuntu feisty, part of the subversion-tools package) that takes the tedium and complexity out of merging in svn. The link above has a good introduction to svnmerge. I learned the basics, though, from Ken Kinder's little howto for svnmerge.
I've tested it out once or twice with real branches at work, it works pretty well and hasn't failed me yet. I wouldn't be surprised if there were some weird or extreme corner cases where svnmerge could get confused and do the wrong thing, but I'll worry about that when I get there. I haven't seen any reports of that yet and I intend to stay away from the more complex features anyway unless I really need them (e.g., no bidirectional merge until at the end, when the experimental branch is stable and is ready to be fully merged back into the trunk). In any case, I expect that development on svnmerge will fix any such showstopper bugs before I ever see them since I intend to be as considerate and conservative as I can be when using svnmerge ;-).
I'm very happy that merging in svn is finally convenient. It's been a long time, I've done my share of hand-merging branches. I'm glad that tedium is going away.
- It's slow
- You have to be online to the repository to do many operations (not distributed)
- Merging is hard
Git apparently solves all those problems. So says Linus on youtube.
I haven't used git and I'm not likely to at $DAYJOB. It took a long time to convince the developers and system administrators at $DAYJOB to actually use svn and I'm not done with the conversion yet, technical support isn't on the version control bandwagon yet. So there's just no way I'm going to spend another year or two convincing people to switch to git from subversion.
I don't worry too much about being online to do most operations. At the office I'm online to the svn server, and at home or anywhere else with an internet connection I can get through to the office on the VPN. If I'm not online, well, I just won't work. Or I'll work with svk if necessary.
The current full working copy (including branches) at work is pretty large, so an svn status on that *is* pretty slow. Even svn status on just the trunk is slow. But we work around that by avoiding working at the top of a branch unless it's really necessary. Working in individual subdirectories is much faster and the only time we ever need to work at the top of a branch is when we need to see what all the uncommitted changes have been to that branch, or when we need to do an svn log to see what all the committed changes have been to the branch.
For a long time though, I would make branched tags but would not actually do experimental or development branches because, while branching is easy, merging was painful. It could be done, but it was so inconvenient I never tried to do it.
Finally, svnmerge has landed. Svnmerge is a python program (on Ubuntu feisty, part of the subversion-tools package) that takes the tedium and complexity out of merging in svn. The link above has a good introduction to svnmerge. I learned the basics, though, from Ken Kinder's little howto for svnmerge.
I've tested it out once or twice with real branches at work, it works pretty well and hasn't failed me yet. I wouldn't be surprised if there were some weird or extreme corner cases where svnmerge could get confused and do the wrong thing, but I'll worry about that when I get there. I haven't seen any reports of that yet and I intend to stay away from the more complex features anyway unless I really need them (e.g., no bidirectional merge until at the end, when the experimental branch is stable and is ready to be fully merged back into the trunk). In any case, I expect that development on svnmerge will fix any such showstopper bugs before I ever see them since I intend to be as considerate and conservative as I can be when using svnmerge ;-).
I'm very happy that merging in svn is finally convenient. It's been a long time, I've done my share of hand-merging branches. I'm glad that tedium is going away.
Wednesday, July 18, 2007
firefox multi-tab home
Long ago I knew that firefox supported multiple-tabs for the "Home Page". And then I forgot.
Just the other day I rediscovered this. It's very convenient. My "Home Page(s)" button now opens
Just the other day I rediscovered this. It's very convenient. My "Home Page(s)" button now opens
- a cacti tab to monitor the database server, our internal time tracking web page,
- a gmail tab,
- a tab that opens our internal doxygen documentation and,
- because I'm studying svnmerge, since I've been waiting for decent merge support in svn since early svn betas:
Ken Kinder on "Subversion merge tracking with svnmerge"
Tuesday, July 17, 2007
SQL subtotals, grandtotals
I really should google more. I sat down and thought for several hours trying to find a way to get running totals out of an SQL query. I should really give up earlier so I could have googled and found:
It's got some SQL-Server-isms in there but it's generic enough that I learned the basic technique pretty quickly and implemented it in postgresql. I got my query to use running totals, and used that as a springboard to getting running balance working. I haven't yet gotten around to getting subtotals and grandtotals working, but I'm sure I'll get around to that soon, SQL use where I work is getting more sophisticated. Years ago the code used monkey-see-monkey-do and mysql style treating the database as a dumb datastore. We're on the way to something between mysql style dumb queries and doing joins in php and oracle style "do everything in the database". I think both styles have their appropriate niches, but our systems are in neither of those niches, we need to be more moderate, using SQL where it's appropriate and business logic in applications where *that's* appropriate.
SQL Server: Calculating Running Totals, Subtotals and Grand Total Without a Cursor
It's got some SQL-Server-isms in there but it's generic enough that I learned the basic technique pretty quickly and implemented it in postgresql. I got my query to use running totals, and used that as a springboard to getting running balance working. I haven't yet gotten around to getting subtotals and grandtotals working, but I'm sure I'll get around to that soon, SQL use where I work is getting more sophisticated. Years ago the code used monkey-see-monkey-do and mysql style treating the database as a dumb datastore. We're on the way to something between mysql style dumb queries and doing joins in php and oracle style "do everything in the database". I think both styles have their appropriate niches, but our systems are in neither of those niches, we need to be more moderate, using SQL where it's appropriate and business logic in applications where *that's* appropriate.
Monday, July 16, 2007
minimal centos vmware image
I had to build a minimal centos 4.4 vmware image recently. Two projects I'm working on use centos because our team systems administrator is an RHCE, so he uses a familiar distribution. I don't much care which distribution I use. CentOS is fine as long as I can get reasonably new versions of the software I require (php, postgresql).
In fact CentOS 4.4 isn't that distribution, with php stuck at somewhere around 4.3 and postgresql at 7.4. But it's certainly stable. And I can build any necessary software from source, so I don't feel any great need to upgrade to 5 (although we should probably have a plan for that upgrade path for sometime in the next 6 months).
For this distribution, I had to install Oracle 10g Express Edition (for learning Oracle, and for prototyping). That makes the installed VM bloat up by a lot. I went through the installed RPMs and removed everything I didn't think I'd need (and a few things that I found I needed later on). Later, I had to reinstall some development packages since I needed to build php from source (because I couldn't figure out how to get CentOS' built-in php to talk to oracle). I eventually wound up installing php 5.3.2 since I couldn't get php 4.3.9 to do what I wanted (connect to oracle). That's probably pilot error, but I don't mind the upgrade, and if there are incompatibilities with the installed php 4.3.9 on the deployed live servers, well, I'll ask the team sysad to deploy 5.3.2 there (after some paranoid testing on the vmware image).
After building php, I removed the development packages again :-). For when I need the list again:
gcc apr-devel gcc-c++ libtool bison flex make
autoconf automake cpp db4-devel apr-devel
glibc-devel glibc-headers apr-util-devel
httpd-devel libstdc++-devel openldap-devel
openssl-devel pcre-devel postgresql-devel
zlib-devel libpng-devel libxml2-devel
The vmware image we use doesn't have any X anything in there except what is needed for vmware-toolbox (useful for resizing). I thought that I absolutely needed to have X actually running in the VM. That may have been true with older versions of vmware, but with the free vmware-server, I can safely do X11 forwarding over ssh and do the vmware-toolbox resizing using the X on the host. No need to have gnome and all the rest of the packages it pulls in just for resizing the vmware virtual drives.
In fact CentOS 4.4 isn't that distribution, with php stuck at somewhere around 4.3 and postgresql at 7.4. But it's certainly stable. And I can build any necessary software from source, so I don't feel any great need to upgrade to 5 (although we should probably have a plan for that upgrade path for sometime in the next 6 months).
For this distribution, I had to install Oracle 10g Express Edition (for learning Oracle, and for prototyping). That makes the installed VM bloat up by a lot. I went through the installed RPMs and removed everything I didn't think I'd need (and a few things that I found I needed later on). Later, I had to reinstall some development packages since I needed to build php from source (because I couldn't figure out how to get CentOS' built-in php to talk to oracle). I eventually wound up installing php 5.3.2 since I couldn't get php 4.3.9 to do what I wanted (connect to oracle). That's probably pilot error, but I don't mind the upgrade, and if there are incompatibilities with the installed php 4.3.9 on the deployed live servers, well, I'll ask the team sysad to deploy 5.3.2 there (after some paranoid testing on the vmware image).
After building php, I removed the development packages again :-). For when I need the list again:
gcc apr-devel gcc-c++ libtool bison flex make
autoconf automake cpp db4-devel apr-devel
glibc-devel glibc-headers apr-util-devel
httpd-devel libstdc++-devel openldap-devel
openssl-devel pcre-devel postgresql-devel
zlib-devel libpng-devel libxml2-devel
The vmware image we use doesn't have any X anything in there except what is needed for vmware-toolbox (useful for resizing). I thought that I absolutely needed to have X actually running in the VM. That may have been true with older versions of vmware, but with the free vmware-server, I can safely do X11 forwarding over ssh and do the vmware-toolbox resizing using the X on the host. No need to have gnome and all the rest of the packages it pulls in just for resizing the vmware virtual drives.
Tuesday, June 26, 2007
vmware networking - wireless bridge, nat problems
I was setting up a vmware image of Ubuntu Feisty the other night and I had a heck of a time with the networking. I went through and tried bridged, nat and host-only. I couldn't get bridged or nat to even talk to the host computer. host-only could talk to the host, I could even get it to NAT to the wifi-router, but I couldn't get it to go past that.
I wiped that image and redid the work and now I've got it working. The first basic problems was that I didn't look in dmesg. On ubuntu, it's necessary to use vmware-any-any to get vmware to work with the Ubuntu kernels (or something, in any case, something doesn't work, maybe compiler issues, and it's necessary to patch with vmware-any-any so that the vmware modules will compile).
After I finally looked in dmesg, there was a section that said that vmware bridge over a wifi device doesn't work with vmware-any-any. OK, next up was nat.
That didn't work either. It wasn't until I rebuilt the image (in fact, uninstalled and reinstalled free vmware-server) last night that I finally got nat working. Previously I had used vmware-config.pl to edit the networking settings (using the editor) and I had decided to always use vmnet0 for the bridge, and then later the nat, and later host-only and then, in the vmware client, chosen BRIDGE, NAT, HOST-ONLY as appropriate. That doesn't always work because apparently vmware has some sort of mapping for the vmnet numbers. Choosing NAT but having manually edited it at vmnet0 makes it not work because NAT starts at around vmnet8.
The trick, I found last night, was to select CUSTOM and then select the correct /dev/vmnet[0-9] entry. I've got networking working now (NAT though, since bridging on a wireless device still doesn't work) and have got the rest of the project done (install oracle 10g express edition with a small sample database, for use by my team in testing an oracle application we're developing for a large government owned corporation).
Oh yeah, and to solve it without a trick (i.e., canonically), just use the network connection wizard. It'll choose the correct number for the vmnet device type that you want to use. I like having things numbered sequentially though, which is why I got into trouble in the first place.
I wiped that image and redid the work and now I've got it working. The first basic problems was that I didn't look in dmesg. On ubuntu, it's necessary to use vmware-any-any to get vmware to work with the Ubuntu kernels (or something, in any case, something doesn't work, maybe compiler issues, and it's necessary to patch with vmware-any-any so that the vmware modules will compile).
After I finally looked in dmesg, there was a section that said that vmware bridge over a wifi device doesn't work with vmware-any-any. OK, next up was nat.
That didn't work either. It wasn't until I rebuilt the image (in fact, uninstalled and reinstalled free vmware-server) last night that I finally got nat working. Previously I had used vmware-config.pl to edit the networking settings (using the editor) and I had decided to always use vmnet0 for the bridge, and then later the nat, and later host-only and then, in the vmware client, chosen BRIDGE, NAT, HOST-ONLY as appropriate. That doesn't always work because apparently vmware has some sort of mapping for the vmnet numbers. Choosing NAT but having manually edited it at vmnet0 makes it not work because NAT starts at around vmnet8.
The trick, I found last night, was to select CUSTOM and then select the correct /dev/vmnet[0-9] entry. I've got networking working now (NAT though, since bridging on a wireless device still doesn't work) and have got the rest of the project done (install oracle 10g express edition with a small sample database, for use by my team in testing an oracle application we're developing for a large government owned corporation).
Oh yeah, and to solve it without a trick (i.e., canonically), just use the network connection wizard. It'll choose the correct number for the vmnet device type that you want to use. I like having things numbered sequentially though, which is why I got into trouble in the first place.
Sunday, June 24, 2007
Oracle bogosity - no boolean column type
Oracle has no boolean column type. That is, in a create table statement, you can't say:
MY_BOOLEAN_COLUMN boolean
I've seen some discussion of this and most of it boils down to what Tom Kyte says, that you don't need a boolean type since you can use an integer(1/0), a character(T/F,Y/N,even 1/0) or anything you like. I suppose you could use varchar(5) and use "true" and "false".
For a while I accepted this explanation, until I ran across an oracle database that had boolean types using all of these conventions (in different tables, although I did see T/F and Y/N for different columns in the same table!).
Of course it was implementer error that they didn't use check constraints on those pseudo-boolean fields. So in the integer field, it was possible to set the "boolean" to 314. and in the character fields, well, it was possible to have T/F/Y/N/1/0 all in the same column (fortunately, the character field was limited to one character, else I would have had even more fun with that stupidity).
I suppose Oracle is that way (bogus) because of legacy code, compatibility issues, and inertia. I can see internal discussions going sort of in the direction of creating a boolean type but stopping short of actually doing so because it just seems so hard, or no one wants to rock the boat and introduce potential compatibility issues (e.g., when oracle software uses boolean, and then won't work with oracle installations which didn't have the boolean type).
Another problem, of course, is that anyone confused enough to use all those different encodings for pseudo-booleans, and worse, IN THE SAME COLUMN with no check constraints, is going to continue being confused when the boolean type is introduced. They won't use the boolean type, they'll continue using their iodine-deficiency-generated solution. But just because idiots will continue in their idiocy is no reason to avoid doing the right thing. As it is, when a boolean field is concerned, there's going to be a discussion as to which convention to use and then there's going to be a random choice (unless the shop in question has advanced to the point of having a convention for boolean fields, in which case I say, bravo). If a true boolean column type were available, the canonical answer would be to use that. Iodine-deficient DBAs could still use their non-boolean hacks, but at least a canonically correct solution would exist, guiding iodine-sufficient (but trained by incompetent or iodine-deficient DBAs) in the right direction.
MY_BOOLEAN_COLUMN boolean
I've seen some discussion of this and most of it boils down to what Tom Kyte says, that you don't need a boolean type since you can use an integer(1/0), a character(T/F,Y/N,even 1/0) or anything you like. I suppose you could use varchar(5) and use "true" and "false".
For a while I accepted this explanation, until I ran across an oracle database that had boolean types using all of these conventions (in different tables, although I did see T/F and Y/N for different columns in the same table!).
Of course it was implementer error that they didn't use check constraints on those pseudo-boolean fields. So in the integer field, it was possible to set the "boolean" to 314. and in the character fields, well, it was possible to have T/F/Y/N/1/0 all in the same column (fortunately, the character field was limited to one character, else I would have had even more fun with that stupidity).
I suppose Oracle is that way (bogus) because of legacy code, compatibility issues, and inertia. I can see internal discussions going sort of in the direction of creating a boolean type but stopping short of actually doing so because it just seems so hard, or no one wants to rock the boat and introduce potential compatibility issues (e.g., when oracle software uses boolean, and then won't work with oracle installations which didn't have the boolean type).
Another problem, of course, is that anyone confused enough to use all those different encodings for pseudo-booleans, and worse, IN THE SAME COLUMN with no check constraints, is going to continue being confused when the boolean type is introduced. They won't use the boolean type, they'll continue using their iodine-deficiency-generated solution. But just because idiots will continue in their idiocy is no reason to avoid doing the right thing. As it is, when a boolean field is concerned, there's going to be a discussion as to which convention to use and then there's going to be a random choice (unless the shop in question has advanced to the point of having a convention for boolean fields, in which case I say, bravo). If a true boolean column type were available, the canonical answer would be to use that. Iodine-deficient DBAs could still use their non-boolean hacks, but at least a canonically correct solution would exist, guiding iodine-sufficient (but trained by incompetent or iodine-deficient DBAs) in the right direction.
Friday, June 22, 2007
auto-convert images for blog
Sol and I take a lot of pictures. The camera automatically names them (with zero prefixed integer sequential filenames) but it's very easy for filenames to collide (two different days can have 000001.jpg files). I also take pictures at a rather high resolution. The resulting images take up a LOT of space.
I wrote a shell script that, with the help of GraphicsMagick (or ImageMagick) takes all those jpgs and creates two files, one a thumbnail (ending in -th.jpg) and one a large images (ending in -1024.jpg). The date of file transfer (from camera to laptop) is prepended to the image name so that filename collisions become impossible.
irec
I originally put the thumbnails and -1024 files in separate directories. Now I put them in the same directory so that the files are easy to upload to photobucket (they're in the same directory, so clicking on the Browse to select an image button brings me to the same directory. It's very easy this way to upload the thumbnail and the -1024 together).
I still keep the thumb directory around but I populate it with links from the "smaller" directory. Keeping thumb around doesn't really make sense anymore. It's not like I have programs or scripts that depend on existence or contents of that directory. Just laziness, I guess.
If the script were named "picstoblog" (see below) I would do something like the ff to process all the directories in my Pics directory.
----
picstoblog script below, some things could be improved (mainly be simplifying/eliding, e.g., thumb), but i don't care enough to actually do the fixes)
-----
I wrote a shell script that, with the help of GraphicsMagick (or ImageMagick) takes all those jpgs and creates two files, one a thumbnail (ending in -th.jpg) and one a large images (ending in -1024.jpg). The date of file transfer (from camera to laptop) is prepended to the image name so that filename collisions become impossible.
irec
I originally put the thumbnails and -1024 files in separate directories. Now I put them in the same directory so that the files are easy to upload to photobucket (they're in the same directory, so clicking on the Browse to select an image button brings me to the same directory. It's very easy this way to upload the thumbnail and the -1024 together).
I still keep the thumb directory around but I populate it with links from the "smaller" directory. Keeping thumb around doesn't really make sense anymore. It's not like I have programs or scripts that depend on existence or contents of that directory. Just laziness, I guess.
If the script were named "picstoblog" (see below) I would do something like the ff to process all the directories in my Pics directory.
for fn in *
do
if [ -d $fn ]
then
pushd .
cd $fn
picstoblog
popd
fi
done
----
picstoblog script below, some things could be improved (mainly be simplifying/eliding, e.g., thumb), but i don't care enough to actually do the fixes)
-----
#!/bin/bash
if [ ! -d thumb ]
then
mkdir thumb
else
rm -f thumb/*
fi
if [ ! -d smaller ]
then
mkdir smaller
fi
for fn in *.jpg
do
base=`basename $fn | cut -f 1 -d '.'`
d=`pwd | cut -f 1-3 -d "-" | sed "s/-//g"`
d=`echo $d | basename $d`
sm=$d-$base-1024.jpg
th=$d-$base-th.jpg
if [ ! -f smaller/$sm ]
then
gm convert -scale 1024x768 -quality "70%" $fn smaller/$sm
fi
if [ ! -f smaller/$th ]
then
gm convert -scale 200x150 -quality "50%" $fn smaller/$th
fi
# for older files, remove if already there so can link
rm -f thumb/$sm
ln -s smaller/$sm thumb/$sm
ln -s smaller/$th thumb/$th
done
Sunday, June 10, 2007
php-oracle bogosity #1
That's not #1 as in it's the worst of its kind. I'm very new to playing with oracle on php, and it's only Oracle 10g Express Edition, so I am definitely not saying anything about oracle in general.
This might even be just a php-oracle thing, not necessarily in the oracle API at all (i tried to read that, got bored and came to no conclusion). In any case, I wasted a few days on this. I'm on vacation and getting online to fix something involves going to the internet cafe in the city. I only do that once a day. Of course, reading the documentation of oci_fetch_array would have helped :-), but I didn't realize oci_fetch_array could be this bogus.
In PHP, oci_fetch_array takes as parameters a resource statement (result of oci_parse) and some integer parameters. Apparently, Oracle (or the PHP-OCI8 library anyway) by default will not return null columns. It's necessary to specify OCI_RETURN_NULLS if you want null columns in the returned row. This is just bogus though. What does it cost to return some extra array entries, after all. Particularly if they're null or empty (which is a separate pure-oracle bogosity for later). It certainly creates an unnecessary asymmetry to have rows sometimes be 10 columns and sometimes 12 depending on whether certain columns in the returned row are null.
This might even be just a php-oracle thing, not necessarily in the oracle API at all (i tried to read that, got bored and came to no conclusion). In any case, I wasted a few days on this. I'm on vacation and getting online to fix something involves going to the internet cafe in the city. I only do that once a day. Of course, reading the documentation of oci_fetch_array would have helped :-), but I didn't realize oci_fetch_array could be this bogus.
In PHP, oci_fetch_array takes as parameters a resource statement (result of oci_parse) and some integer parameters. Apparently, Oracle (or the PHP-OCI8 library anyway) by default will not return null columns. It's necessary to specify OCI_RETURN_NULLS if you want null columns in the returned row. This is just bogus though. What does it cost to return some extra array entries, after all. Particularly if they're null or empty (which is a separate pure-oracle bogosity for later). It certainly creates an unnecessary asymmetry to have rows sometimes be 10 columns and sometimes 12 depending on whether certain columns in the returned row are null.
Monday, June 04, 2007
Maybe catching up
I've been away from mailing list email for months because the pressures of work don't allow me to even scan, let alone post, mailing lists. That's too bad. I really enjoy reading on the postgresql-general mailing list, for instance, since there's so much to learn and the signal to noise ratio there is incredibly high. On Plug I think the SNR has also been improving. But I'm not sure about that since, well, I just haven't been reading it. I've looked there once in a while and there seems to have been some improvement.
I'm going away for a two-week vacation with my family. We'll be in Cagayan de Oro, Duka Bay, and Camiguin. CDO, to visit, to prepare for timmy's baptism, and for the actual baptism, Dukay Bay for a break. My parents and brothers and sisters have been there many times, I never have, so we'll go. I may dive there, if they've got a mask with the right corrective lenses. If not, I'm sure we'll be going on the glass bottomed boat.
On camiguin, of course, we'll be diving No camiguin adventure tours> for us though. I've done most of them and timmy can't go with us. I'm really looking forward to diving. Sol hasn't been to Sunken Cemetery or Canyons yet and we've got a divecase now to go with the digital camera, so I want to take a lot of pictures :-). I don't know if the pictures will be good. Visibility might not be so great during rainy season, but it'll be great to take our own dive pictures and videos. Sol wants to do a night dive. That'd be great. The last and only time we did that, it was for qualifying for PADI advanced open water diver. It'll be good to do it again without having to worry about exercises, etc. The last time we had a really great time. There were incredible numbers of lionfish (I guess they come out at night, they're a common sight on camiguin, but they were really out in force that night, we even saw two that were doing the deed, or would have been, if they were mammals :-) and a huge cuttlefish, among a lot of other things. I can't wait to do that again. twice :-).
Oh, wait. This was about catching up. I get carried away sometimes. Back on topic, I'll have two weeks of free time. I'm sure I won't be in front of the laptop the whole time (although sol and I are both bringing laptops, we'll be working too while away) but I'll certainly spend 2 or 3 hours a day catching up.
I'm going away for a two-week vacation with my family. We'll be in Cagayan de Oro, Duka Bay, and Camiguin. CDO, to visit, to prepare for timmy's baptism, and for the actual baptism, Dukay Bay for a break. My parents and brothers and sisters have been there many times, I never have, so we'll go. I may dive there, if they've got a mask with the right corrective lenses. If not, I'm sure we'll be going on the glass bottomed boat.
On camiguin, of course, we'll be diving No camiguin adventure tours> for us though. I've done most of them and timmy can't go with us. I'm really looking forward to diving. Sol hasn't been to Sunken Cemetery or Canyons yet and we've got a divecase now to go with the digital camera, so I want to take a lot of pictures :-). I don't know if the pictures will be good. Visibility might not be so great during rainy season, but it'll be great to take our own dive pictures and videos. Sol wants to do a night dive. That'd be great. The last and only time we did that, it was for qualifying for PADI advanced open water diver. It'll be good to do it again without having to worry about exercises, etc. The last time we had a really great time. There were incredible numbers of lionfish (I guess they come out at night, they're a common sight on camiguin, but they were really out in force that night, we even saw two that were doing the deed, or would have been, if they were mammals :-) and a huge cuttlefish, among a lot of other things. I can't wait to do that again. twice :-).
Oh, wait. This was about catching up. I get carried away sometimes. Back on topic, I'll have two weeks of free time. I'm sure I won't be in front of the laptop the whole time (although sol and I are both bringing laptops, we'll be working too while away) but I'll certainly spend 2 or 3 hours a day catching up.
Saturday, May 26, 2007
Online feisty upgrade - washout again
I decided to do an online distro upgrade to Ubuntu feisty. As with my last online distro upgrade attempt (Dapper to Edgy), this one didn't go well either. Apache2 wouldn't upgrade cleanly, so I just removed all of apache2. I planned to install it from scratch when the distro upgrade was done.
Unfortunately, there was a problem with removing or upgrading the kernel and I went nuts and decided to completely throw away the Edgy kernel and give Feisty the change to install a few of its kernels. That was a mistake since Ubuntu likes grub and I prefer lilo. Something went wrong the kernel upgrade and I was left with a system that died with kernel panic when trying to mount the root filesystem.
A bit of work with a Dapper installer, and Edgy alternative installer, and a Mandriva 2006 installer gave me no joy. My old trick of chroot to the old mounted root and then mounting the other filesystems didn't work (so I couldn't get /boot to mount, nor could I lilo to write the MBR). There may be a way to do all of that, but I don't know what it is.
Finally I broke down, backed up my /etc and installed Feisty from a CD I'd previously made but not used (since I wanted to just update everything in place). I've spent a few hours downloading the other packages I need (postgresql, subversion, g++, apache2, a whole bunch of php, java, openvpn etc) and by morning I'll have a usable system.
I always have my /home on a partition separate from /. Occasions like this remind me of the wisdom of making that choice.
Unfortunately, there was a problem with removing or upgrading the kernel and I went nuts and decided to completely throw away the Edgy kernel and give Feisty the change to install a few of its kernels. That was a mistake since Ubuntu likes grub and I prefer lilo. Something went wrong the kernel upgrade and I was left with a system that died with kernel panic when trying to mount the root filesystem.
A bit of work with a Dapper installer, and Edgy alternative installer, and a Mandriva 2006 installer gave me no joy. My old trick of chroot to the old mounted root and then mounting the other filesystems didn't work (so I couldn't get /boot to mount, nor could I lilo to write the MBR). There may be a way to do all of that, but I don't know what it is.
Finally I broke down, backed up my /etc and installed Feisty from a CD I'd previously made but not used (since I wanted to just update everything in place). I've spent a few hours downloading the other packages I need (postgresql, subversion, g++, apache2, a whole bunch of php, java, openvpn etc) and by morning I'll have a usable system.
I always have my /home on a partition separate from /. Occasions like this remind me of the wisdom of making that choice.
Thursday, May 24, 2007
I broke my laptop
I'd been trying to upgrade to Ubuntu Feisty online but my laptop kept turning itself off after a few beeps. I realized that the fan was not turning on.
I had a spare fan (from a time when I thought the fan wasn't turning on but in fact was just a bit hard starting, somehow that fixed itself and stayed fixed for a year and a half or so). So I decided to replace the fan. Unfortunately, the laptop is an older Winbook model and apparently, in their first few iterations, the case design was rather obtusely inelegant. I've got a more recent Winbook and *that* is *very* much nicer to work with.
So I had to take out a whole bunch of screws (some of which I'm not able to put back in the same place again) and detach some cables and do a whole lot of other work just to get the *power* cable of the fan attached. Replacing the fan was trivial. It was connecting the power cable of the fan that was a pain. I should have just gone to the local cell phone repair shop and had the guy there solder the cut ends of the new fan to the cut ends of the old fan power cable.
I didn't do that though. Instead I actually tried to get the fan connected to power directly. Unfortunately, this involved opening up the laptop to a degree far greater than I had anticipated. And in doing all of that, I seem to have touched something and now the power won't go on. I might bring this laptop to a repair shop (or ship it to the U.S. to be fixed for a fee). It won't be anytime soon though. Instead, I just swapped the hard drive with a slower but much lighter winbook laptop I had lying around, and took the RAM from the bad laptop and put it into the new winbook (after doing some online research to make sure the memory was the same for both models, it was).
Now I've got a laptop that's half the speed (3Ghz to 1.5Ghz but with a bit more RAM 1GB to 1GB+256MB since I just moved the RAM over to the new laptop).
I like the smaller laptop. It's certainly a heck of a lot more convenient to lug around than the 3Ghz desktop replacement. I'm going to miss that 3Ghz though. I hope I can get it fixed. I'd love to keep working with it, despite its weight, for another 3-4 years :-). by then quad-core CPUs will be out and dual-core laptops will be as cheap as these winbooks were when I got them :-).
I had a spare fan (from a time when I thought the fan wasn't turning on but in fact was just a bit hard starting, somehow that fixed itself and stayed fixed for a year and a half or so). So I decided to replace the fan. Unfortunately, the laptop is an older Winbook model and apparently, in their first few iterations, the case design was rather obtusely inelegant. I've got a more recent Winbook and *that* is *very* much nicer to work with.
So I had to take out a whole bunch of screws (some of which I'm not able to put back in the same place again) and detach some cables and do a whole lot of other work just to get the *power* cable of the fan attached. Replacing the fan was trivial. It was connecting the power cable of the fan that was a pain. I should have just gone to the local cell phone repair shop and had the guy there solder the cut ends of the new fan to the cut ends of the old fan power cable.
I didn't do that though. Instead I actually tried to get the fan connected to power directly. Unfortunately, this involved opening up the laptop to a degree far greater than I had anticipated. And in doing all of that, I seem to have touched something and now the power won't go on. I might bring this laptop to a repair shop (or ship it to the U.S. to be fixed for a fee). It won't be anytime soon though. Instead, I just swapped the hard drive with a slower but much lighter winbook laptop I had lying around, and took the RAM from the bad laptop and put it into the new winbook (after doing some online research to make sure the memory was the same for both models, it was).
Now I've got a laptop that's half the speed (3Ghz to 1.5Ghz but with a bit more RAM 1GB to 1GB+256MB since I just moved the RAM over to the new laptop).
I like the smaller laptop. It's certainly a heck of a lot more convenient to lug around than the 3Ghz desktop replacement. I'm going to miss that 3Ghz though. I hope I can get it fixed. I'd love to keep working with it, despite its weight, for another 3-4 years :-). by then quad-core CPUs will be out and dual-core laptops will be as cheap as these winbooks were when I got them :-).
Wednesday, May 09, 2007
Friday, May 04, 2007
NEVER buy software that requires activation more than once
The company I work for has a custom program for handhelds. We bought tens of handhelds (might be a hundred by now, not sure) and installed the software on it. Unfortunately, we used Appforge to develop the software. The software requires real-time activation per-handheld device that the software is installed on. If we'd gotten the ISV license, no real-time activation per handheld would have been required, but the ISV license expires every year.
Now Appforge has closed (bankruptcy) and the activation servers have gone dark. We can't install on handhelds which we have licenses for because we have that activate-in-realtime license, not the ISV license.
Now, it wasn't my choice to go with Appforge, I'd have gone with J2ME or similar, probably. But we made the right decision at the time. Development in VB really *was* much faster than in J2ME or something else. We just didn't know that AppForge was in financial trouble. Next time we think about buying anything with similar "copy-protection" though, I'm going to put my foot down. NEVER buy anything that requires that the selling company still be around in order for you to use it. Open source is better, if it's available, but even for payware, if the company you buy it from can turn your service off at will (or when they go bankrupt), walk away.
Now Appforge has closed (bankruptcy) and the activation servers have gone dark. We can't install on handhelds which we have licenses for because we have that activate-in-realtime license, not the ISV license.
Now, it wasn't my choice to go with Appforge, I'd have gone with J2ME or similar, probably. But we made the right decision at the time. Development in VB really *was* much faster than in J2ME or something else. We just didn't know that AppForge was in financial trouble. Next time we think about buying anything with similar "copy-protection" though, I'm going to put my foot down. NEVER buy anything that requires that the selling company still be around in order for you to use it. Open source is better, if it's available, but even for payware, if the company you buy it from can turn your service off at will (or when they go bankrupt), walk away.
Subscribe to:
Posts (Atom)