Tuesday, February 21, 2006


I ran into a problem recently because PHP was dying when processing a query that returned some 20 million rows. PHP was running out of memory.

After some googling and running the program several more times (with some runs involving increased memory limit for PHP), it became clear that PHP's postgres support code is loading whole result sets into RAM (and feeding it to PHP one by one, when the PHP code calls pg_fetch_row or pg_fetch_array) rather than using cursors so that it can load one row at a time. I don't know if ADODB or PEAR do better. If they're still using the underlying postgres specific functions though, then they're still going to have the same problem.

The solution is to use cursors explicitly. There are some caveats.
1. Using a cursor requires using a transaction. This isn't onerous, just
something to remember.

2. It's necessary to check after the FETCH to see if there are no more rows.
There's probably a prettier way, but I just go with an infinite while loop
that breaks when there are no more rows.

// assume that $conn is the db connection and that $q is the query, the ff is typed
// straight into blogger, might contain minor errors.

pg_query($conn,"BEGIN WORK");

$cres=pg_query($conn,"DECLARE my_cursor CURSOR FOR $q FOR READ ONLY");

// query succeeded, fetch the rows from the cursor
$res=pg_query($conn,"FETCH my_cursor");

// quit loop if no more rows in cursor

// there's a row, get it and process it.


pg_query($conn,"CLOSE my_cursor");

pg_query($conn,"END WORK");

fetching rows one at a time, above, is for illustration only. of course,
for efficiency, one would fetch many rows at a time and loop through them.

Saturday, February 18, 2006

Title Wrong - ridiculously

So in my previous post the title says that java studio is ridiculously slow, and then I forgot to mention that fact in the rest of article. On Saturdays I lose my brains.

Anyway, when my colleague was showing me the class diagrams that he developed in Java Studio, I couldn't stand the fact that scrolling across the diagram space, zooming in, and dragging items from one place to another was slow. It wasn't really unusably slow, but it was irritating to anyone used to how fast dragging things in Konqueror (itself slow enough that I never use it, even on a fast box, but a good comparison because, despite it being slow, Java Studio is slower) is.

Maybe things will get better on my own computer though. I'm still waiting for the sun Developer Network site to let me in though, before I can download and test Java Studio Creator. Maybe under strace, in case that can show what's making it slow (I doubt if it'll help, but it's a weekend, I can spend the time looking at strace :-).

Java Studio - slow - ridiculously

A colleague at work was using Sun's Java Studio Creator for some prototyping. He was running it on the LTSP server and I noticed that it would sometimes eat 100% of the CPU (well, 50% of the CPU since it's a dual-CPU box, the culprit to be mentioned further on doesn't seem to be a multi-thread issue, at least) continuously, while it was doing nothing (everyone had left for home, no one was working in Java Studio anymore).

That seems like a stupidity, and I thought I'd download it to my own computer so I could see if the behavior would manifest there also.

I've subscribed to the Sun Developer's network though, and I can't log in to where I can download Java Studio. They haven't sent me a confirmation email yet, I can't log in. Seems like some slowness over there at Sun. probably due to java. Damn, I'm getting disillusioned with working in Java.

Friday, February 17, 2006

First rule of debugging

My wife and I came home from Tagaytay and there was no internet. The cable modem had the receive light blinking and the send and online lights off. OK, I had to get to work and didn't care enough to fix it right then (mistake, Sol wanted to work at home, but it's a short walk to work, so no huge deal). When I got home, the cable modem was still down. I texted the destiny cable help line and went to bed. In the morning, same thing.

I texted destiny again and they called me (but I didn't answer since my phone was on silent+vibrate).

And then I looked at the cable modem and realized that the cable wasn't connected. In removing the laptop power cable for the trip to Tagaytay, I had moved the cable enough that it disconnected from the modem.

I had violated the first rule of debugging, it's *MY* bug. Almost all bugs are my bug. First thing to do is check if I did something wrogn before reporting to anyone else that they have a problem with their system.

No big deal, but useful to be reminded.

Tuesday, February 07, 2006

cpufreq on an AMD 2400+ - strange

I rebuilt my 2.6.12-15mdk kernel, with a Pentium III/Celeron/Coppermine ... CPU processor family because when I built it with either AMD K6 or K7, it was just not showing the right CPU frequencies to choose from in cpufreq.

Or maybe it was, but I didn't like the choices as they all seemed too high.

cat /proc/cpuinfo says (among other, snipped, things):

vendor_id : AuthenticAMD
cpu family : 6
model : 10
model name : mobile AMD Athlon(tm) XP-M 2400+
stepping : 0

Maybe there's a problem with the detection of the CPU frequencies. Or maybe the frequencies it shows are right (unlikely, since the lowest frequence is around 1.3Ghz or so, and the highest is much higher than 2.4Ghz, even though the CPU only goes to around 1.7Ghz in actual clocks.

Anyway, no big deal. I've got a working kernel, it does swsusp, it does cpufreq, it does most of what I need. The only thing I'd really like now would be if I could get the damn ALSA sound working. When I load the ALSA sound driver for this VIA82cxxx, I get no sound, and when I load the OSS driver I can't change the volume in kmix. I can get around that by using headphones with volume control, but I just want the ALSA sound driver to work right :-).

I *did* get it working at some point, or maybe that was with the Mandriva 2005 kernel, 2.6.12 something.

Hmmm, might be time to build a generic kernel now.

Thursday, February 02, 2006

Kernel rebuilding week

I upgraded my Mandriva installation and some things are not working right, so it's kernel rebuilding week.

I use cpufreq to slow down the CPU when I'm not doing anything CPU intensive. So I needed that to work (/sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies or similar). With the default distro, I get an incorrect frequency list. So I need to build the kernel and tell it what CPU i've got. it looks like the kernel tries to use the powernow K7 module for cpufreq, but /proc/cpuinfo tells me that this CPU is AMD family 6, so I should probably be using the K6 powernow module).

I thought I could use the default distro kernel, but when I do that, the USB keyboard and mouse don't work (very strange). So my first kernel rebuild was just to get usbkbd and usbmouse working correctly. I built a kernel that was pure i386 (for vmplayer, going to test openmosix) and tested with AMD K7, now building one for AMD K6.

There is also a problem with sound. If I use the ALSA driver there is no sound, but if I use the OSS driver, there is sound but I can't change the volume. Not sure what that's about. OSS is fine with me, I don't care very much. There's probably something simple I'm missing there. I just need to figure that out and I'll have sound plus volume control.

Not that that matters very much. the sound on this laptop is ridiculously low in volume, so I'd need to get speakers anyway, and I could control the volume from there :-). But for now, it's a small challenge to either get ALSA working or to get OSS working with volume control. I *did* have ALSA working for a bit there, but I don't remember exactly what I was doing. I might have been playing with a 2.4 kernel for openmosix (but 2.4 doesn't have cpufreq built in and I was too lazy to apply that patch and then fight with openmosix' patch incompatibilities).

One of these days it'll all get done.